633 commits

Author	SHA1	Message	Date
Terrtia	2e763ba0aa	fix: [ModulesInformationV2] fix #244	2018-09-19 15:45:06 +02:00
Terrtia	5732d7b134	fix: [hashDecoded] range_type_json	2018-09-19 15:36:33 +02:00
Terrtia	ebad2af1ae	fix: [import_dir] fix #251, fix paste import ... if the date directory tree don't exist, he will be generated	2018-09-19 13:38:31 +02:00
Terrtia	653463a8c4	fix: [LAUNCH.sh] make pystemon optionnal ... AIL and Pystemon are independent, pystemon shouldn't be launch by default.	2018-09-19 09:52:27 +02:00
Terrtia	5b31b6e853	fix: [Crawler] save domain to crawl on splash error	2018-09-18 16:20:13 +02:00
Terrtia	f5b648d72a	pixelate paste screenshot	2018-09-18 11:03:40 +02:00
Terrtia	6aa1b99c2a	fix: [LAUNCH.sh] remove old decoders	2018-09-17 15:50:27 +02:00
Terrtia	0c63f2f24f	chg: [Crawler] catch server response	2018-09-17 15:35:06 +02:00
Alexandre Dulaunoy	a6c72a677d	Merge pull request #243 from kovacsbalu/docker-with-pystemon ... Docker build with pystemon	2018-09-12 21:50:11 +02:00
Terrtia	a94a0fe1a7	fix: [Install] add python requirements	2018-09-12 19:10:49 +02:00
Terrtia	204e996fc3	chg: [statistics] clean scripts	2018-09-12 11:21:11 +02:00
Terrtia	ea3d2c1977	Merge branch 'master' into statistic	2018-09-12 10:06:53 +02:00
Terrtia	6f0817365a	chg: [Crawler UI] display domain information	2018-09-12 09:55:49 +02:00
kovacsbalu	d19dd72f99	Install pystemon and start pystemon-feeder in docker	2018-09-03 16:01:39 +02:00
Terrtia	ca982e13e1	chg: [Crawled Domain] show crawled pastes by domain	2018-08-27 14:34:08 +02:00
Terrtia	823c33174a	Merge branch 'master' into onion_crawler	2018-08-27 11:20:51 +02:00
Terrtia	5c902640e9	Merge branch 'master' into onion_crawler	2018-08-27 11:05:25 +02:00
Terrtia	d42dd118a4	chg: [Domain crawled] add random screenshot	2018-08-27 11:02:39 +02:00
Terrtia	a586548047	Merge branch 'master' into base64	2018-08-24 10:35:28 +02:00
Terrtia	ced0b1e350	chg: [I2P] add default config	2018-08-24 10:24:03 +02:00
Terrtia	7e24943537	chg: [Crawler] crawler accept all kind of domains	2018-08-24 10:13:56 +02:00
Terrtia	e9580d6775	chg: [Crawler] change BDD, save i2p links	2018-08-21 15:54:53 +02:00
Terrtia	ed559d9f4a	chg: [Showpaste] add screenshot + improve onion db	2018-08-16 17:24:39 +02:00
Terrtia	6f2dbcb4e2	fix: [pystemon-feeder] fix #234	2018-08-13 10:22:20 +02:00
Terrtia	7652089433	chg: [Onion] change onion regex, fix crawler	2018-08-13 09:23:14 +02:00
Terrtia	8b1c10b38c	chg: [Onion] add onion splash crawler	2018-08-09 17:42:21 +02:00
Thirion Aurélien	000d5bf40a	fix: add miss config	2018-08-07 13:07:08 +02:00
Terrtia	d7545a46f0	chg: [Mixer] add submitted paste	2018-08-02 15:29:13 +02:00
Terrtia	449da63fd7	chg: [Keys] typo	2018-08-01 16:05:49 +02:00
Terrtia	67b0d973af	chg: [Keys] detect ssh2 private key	2018-08-01 16:04:06 +02:00
Terrtia	d0135c248d	add iban country statistic	2018-08-01 15:26:29 +02:00
Terrtia	940b96734a	Merge branch 'master' into statistic	2018-08-01 09:29:51 +02:00
Thirion Aurélien	f7f4422ff1	Merge pull request #232 from CIRCL/iban ... Add Iban detection	2018-08-01 08:57:48 +02:00
Terrtia	b5c1f281e6	fix: typo	2018-07-31 10:58:23 +02:00
Terrtia	61e98ed8fb	clean	2018-07-31 10:51:18 +02:00
Terrtia	e455d244d8	chg: [statistics] add domain statistics	2018-07-31 10:36:29 +02:00
Terrtia	5dcb4ebb58	chg: [statistics] add script to generate pie chart png	2018-07-30 16:36:34 +02:00
Terrtia	09fbc363f1	chg: [sglinjection Phone] add tld statistic, fix phone regex	2018-07-30 11:56:50 +02:00
Terrtia	c20e7d5ab4	chg: [Mail Credential] add tld statistic	2018-07-30 10:19:26 +02:00
Terrtia	39f54d4420	chg: [Mail] add country statistic	2018-07-30 09:21:22 +02:00
Terrtia	009f87ee10	[fix: [UI-hash] fix graph node zoom	2018-07-27 11:27:41 +02:00
Terrtia	153f9f38cc	chg: [BankAccount] regex	2018-07-27 10:20:03 +02:00
Terrtia	57b631233b	add: [BankAccount] iban detection	2018-07-26 15:31:58 +02:00
Terrtia	5ad5939bd5	chg: display last logs	2018-07-25 16:48:44 +02:00
Terrtia	03a37af5cf	chg: [Hash] add reprocess regex order	2018-07-25 10:47:36 +02:00
Terrtia	8beb63b6a8	fix: [Queues] module with no subscriber	2018-07-24 15:57:18 +02:00
Terrtia	8c97c70a3f	Merge branch 'master' into base64	2018-07-24 15:21:32 +02:00
Terrtia	4b8c26dec8	fix: [hive auto export] fix #221	2018-07-24 14:46:35 +02:00
Terrtia	54e8750c30	fix: [LAUNCH] script checking	2018-07-24 12:18:41 +02:00
Terrtia	d4f0c14620	fix: [LAUNCH] use envar, add options (launch all, flask, killall)	2018-07-24 10:41:10 +02:00
Terrtia	9c03c40482	chg: [UI-Hash] add decoder pie	2018-07-23 11:11:52 +02:00
Terrtia	1fbd4109c2	chg: [UI-Hash] add decoded type on graph	2018-07-20 15:48:56 +02:00
Terrtia	6d6de3ae54	fix: remove experimental modules	2018-07-20 10:38:04 +02:00
Terrtia	bd5f83f0eb	chg: refractor base64 encoded to hash	2018-07-20 10:32:52 +02:00
Terrtia	9a8e37fb0b	fix: decoder bdd	2018-07-20 09:43:09 +02:00
Terrtia	0e67b56906	add: Decoder module	2018-07-19 16:52:09 +02:00
Terrtia	ad26f016e3	add: Decoder Module, decode binary, hex and base64	2018-07-19 16:50:42 +02:00
Terrtia	280e9d8af8	fix: Binary decoder base2 string to bytes	2018-07-18 13:47:06 +02:00
Terrtia	e25dd52d64	add: Binary decoder	2018-07-18 11:45:19 +02:00
Terrtia	0cac3146f1	fix: remove hardcoded mail url	2018-07-17 15:11:25 +02:00
Terrtia	fa38434e04	add: new mail body and starttls	2018-07-16 15:51:37 +02:00
Terrtia	8b245d17d9	fix: add LibInjection to LAUNCH.sh	2018-07-16 15:45:55 +02:00
Terrtia	6f69da0c0d	fix: paste submit css	2018-07-16 10:32:41 +02:00
Terrtia	fce324e370	fix: release tiemeout alarm, chg:icon graph node	2018-07-13 11:54:14 +02:00
Terrtia	87b7facba8	add: base 64 node graph	2018-07-12 17:07:17 +02:00
bui	bd5c42caeb	add LibInjection Module	2018-07-04 14:28:10 +02:00
bui	362f72fe43	initial LibInjection module	2018-07-04 14:27:54 +02:00
Thirion Aurélien	f8a88cecec	fix: hive_feeder hiveApi declaration	2018-07-02 15:23:21 +02:00
Terrtia	44524b25a4	fix submit paste	2018-06-29 15:30:01 +02:00
Terrtia	a0a639276c	fix submit paste	2018-06-29 15:29:02 +02:00
Terrtia	98d0235fc3	Merge branch 'master' into base64	2018-06-29 14:57:46 +02:00
Terrtia	f5bda70e82	Merge branch 'master' of https://github.com/CIRCL/AIL-framework	2018-06-29 13:51:08 +02:00
Terrtia	72146f938e	change: feeder name separator	2018-06-29 13:50:41 +02:00
Thirion Aurélien	c41a165a21	fix: file submission, content type	2018-06-29 12:08:46 +02:00
Terrtia	407aa14244	fix: catch file error in paste submission	2018-06-29 11:32:52 +02:00
Terrtia	16c700ddb4	new: add base64 display on paste + base64 statistics	2018-06-29 10:02:29 +02:00
Terrtia	225776c69c	fix: tp and fp, errors messages on paste submission	2018-06-20 10:02:26 +02:00
Terrtia	6a89fac309	fix hive and misp error	2018-06-19 16:39:49 +02:00
Terrtia	af9db53171	add tag fp and tp	2018-06-19 15:09:26 +02:00
Terrtia	86c2cfa04b	update ail-leak object	2018-06-18 17:23:41 +02:00
Terrtia	634bf6e552	add img + fix upload api	2018-06-18 17:16:22 +02:00
Terrtia	704e6f0ad1	change default submit tag, event-path map, publish events	2018-06-18 13:58:31 +02:00
Terrtia	aee1b50ef1	fix duplicate export attribute + tag export whitelist	2018-06-15 17:25:43 +02:00
Terrtia	31fc33ab03	misp event creation, auto and manual + the hive alert auto creation, manual case creation	2018-06-14 16:51:06 +02:00
Terrtia	8522d50308	modify navbar + decompress file password	2018-06-11 10:21:11 +02:00
Terrtia	c64588f60e	add feature, user can submit paste on the web interface	2018-06-08 16:49:20 +02:00
Terrtia	6143bc3dce	submit users input paste	2018-06-05 16:58:04 +02:00
Terrtia	6287981668	clean	2018-06-01 09:49:26 +02:00
Terrtia	884d655b0c	taxonomies and galaxies update via LAUNCH	2018-05-31 13:53:29 +02:00
Terrtia	93dcdf4f23	fix duplicate	2018-05-31 09:01:47 +02:00
Terrtia	0d164b6f4c	add tag js and css	2018-05-16 14:48:15 +02:00
Terrtia	b1b7d33c96	tags	2018-05-16 14:39:01 +02:00
Terrtia	f66a528bc2	fix Duplicate, save list of duplicates on disk + prevent empty hash creation	2018-05-15 23:28:47 +02:00
Terrtia	225fe76c96	fix duplicate + get raw paste	2018-05-11 16:07:30 +02:00
Terrtia	d7ba5533be	fix duplicate + fix issue #200	2018-05-11 13:25:45 +02:00
Terrtia	8599d7142d	python 3 upgrade instructions	2018-05-09 13:32:25 +02:00
Terrtia	142c1dc9b0	clean	2018-05-09 13:03:46 +02:00
Terrtia	3841c159a0	change leveldb to ardb	2018-05-07 14:50:40 +02:00
Terrtia	c17b9402f7	clean	2018-05-04 14:25:47 +02:00
Terrtia	96a3df42bc	decode with redis connection	2018-05-04 13:53:29 +02:00
Terrtia	ae8f575879	use feeder name	2018-05-03 16:32:25 +02:00
Terrtia	a900d5f08d	add python 2 to 3 bash + fix Duplicate + clean	2018-05-03 16:21:33 +02:00
Terrtia	c52caebe7c	improve Duplicate + tlsh + add time out handler + debug + clean	2018-05-02 17:07:10 +02:00
Terrtia	92977201fd	create json metadata for base64 module	2018-04-27 15:46:41 +02:00
Terrtia	c3428df5dc	add apiKeys module	2018-04-26 14:42:39 +02:00
Terrtia	29c368250e	create new dashbord alert	2018-04-24 17:03:10 +02:00
Terrtia	8738b7cf75	fix track term and avoid duplicate mail address	2018-04-24 16:44:37 +02:00
Terrtia	a7bd01ed9a	Merge branch 'python3' of https://github.com/CIRCL/AIL-framework into python3	2018-04-23 16:22:51 +02:00
Terrtia	4e55f6ee90	fix Onion module	2018-04-23 16:22:23 +02:00
Thirion Aurélien	ef3c335b62	Delete config.cfg.backup	2018-04-23 14:54:23 +02:00
Terrtia	f93fe9aeb2	add bitcoin module	2018-04-23 14:51:53 +02:00
Terrtia	3fc48db903	add Base64 module	2018-04-20 10:48:44 +02:00
Terrtia	41eb250278	fix python 3 compqtibility issues	2018-04-20 10:42:19 +02:00
Terrtia	8571a86344	frontend python 3.5 upgrade	2018-04-17 16:06:32 +02:00
Terrtia	19c90339e0	clean	2018-04-17 09:43:45 +02:00
Terrtia	d0261233aa	fix ModulesInnformationV2	2018-04-17 09:37:58 +02:00
Terrtia	15ef02fe53	fix python3 types issue	2018-04-16 17:00:44 +02:00
Terrtia	9e07a0ebdb	fix python3 types issue	2018-04-16 15:04:22 +02:00
Terrtia	9c82dd90ec	python 3 backend upgrade	2018-04-16 14:50:04 +02:00
Terrtia	3395b16873	fix incoming encoding file issue	2018-04-13 09:17:56 +02:00
Terrtia	347588fdec	python3.5 Mixer and Global module	2018-04-12 17:06:57 +02:00
Terrtia	4386cbbd34	fix travis	2018-04-11 10:48:22 +02:00
Terrtia	14583f702f	create first test	2018-04-11 10:14:33 +02:00
Terrtia	e9b88b0656	Keys module improvement	2018-04-05 11:40:34 +02:00
CD Nguyen	96259c53bc	refactor NotificationHelper code	2018-04-04 09:41:13 +02:00
CD Nguyen	e9e3f17363	support non TSL SMTP servers	2018-03-30 11:35:37 +02:00
Philipp Schmied	29608ab29b	Fixed e-mail notificatins for sets and regexes	2018-02-28 11:31:16 +01:00
Philipp Schmied	fd329aaba9	config.cfg.sample: Merged overwritten changes from upstream	2018-02-27 16:16:57 +01:00
Philipp Schmied	5b1f0b0212	Implemented email notifications (bin: config.cfg additions, email sending via analyzer scripts; var: Changes to add notifications via terms management); terms_management: Fixed click handlers not being added to all tracked terms.	2018-02-27 15:12:02 +01:00
Alexandre Dulaunoy	abdf41fe23	Merge pull request #164 from mokaddem/showDiff ... Show differences between duplicates	2018-02-21 22:25:16 +01:00
Alexandre Dulaunoy	0263b25da8	Merge pull request #163 from mokaddem/iss160 ... Moving more to config	2018-02-21 22:24:50 +01:00
Sami Mokaddem	cbb19f95e4	fix: asciimatics introduced the '_required_height' property in the Label class. Just updating here to mirror the change	2018-02-16 14:35:09 +01:00
Sami Mokaddem	3e117195a2	fix: Added a backoff time if an IOError happens	2018-01-15 17:45:13 +01:00
Sami Mokaddem	9a66a6111b	Fixing no such file	2018-01-15 17:10:03 +01:00
Sami Mokaddem	c39e537cbe	Merge branch 'iss160' into showDiff ... Preemptively preventing merge-conflict with branch iss160 (moving to configuration)	2017-12-12 11:33:22 +01:00
Sami Mokaddem	05b22e8963	typo: added word + deleted trailing spaces	2017-12-12 11:19:37 +01:00
Sami Mokaddem	ef1bd16a72	typo: added word + deleted trailing spaces	2017-12-12 11:16:40 +01:00
Sami Mokaddem	304213cfa4	update: Possibility to change the authorized max line length in the config for the diff of duplicates	2017-12-12 11:00:26 +01:00
Sami Mokaddem	860acbf7cf	fix: Renamed Credentials into Credential sectiont title ... config change: By default, zmq listen to both local and CRF	2017-12-12 09:29:15 +01:00
Sami Mokaddem	9d0d0b4303	update: Moved filtering operation (thresholds, number of matching in the ... categ file) in the configuration file. It permits to better control the flow of pastes. Also set default mixer duplicate filtering to 3 (Do not filter)	2017-12-11 17:28:34 +01:00
Fabien Mathey	3105192312	Allow for multiple entries like 1 2 3 4 5 ... Should replace the need to type 1<Enter>2<Enter> etc. and allows for 1 2 3 4 5<Enter>	2017-11-28 15:15:12 +01:00
Alexandre Dulaunoy	71922f9def	add: Python script to test if the ZMQ feed works as expected	2017-11-28 09:46:06 +01:00
Sami Mokaddem	83e082e62a	update: removed useless comments	2017-11-24 08:57:41 +01:00
Sami Mokaddem	a0d07b1098	feature: try to send data to MISP if keys are provided, else do nothing	2017-11-23 13:47:02 +01:00
Sami Mokaddem	2a967c4d92	update/feature: Max number of duplicate push to MISP + duplicate are pushed as attachment	2017-11-23 07:13:44 +01:00
Sami Mokaddem	805171a8a5	Added a seemingly working version of ailleak to misp	2017-11-20 14:57:25 +01:00
Sami Mokaddem	e0b188c069	Better support of python3 in Helper.py	2017-11-20 14:55:36 +01:00
Sami Mokaddem	1e76e10c01	Support of add_object to a MISP instance	2017-11-20 12:12:53 +01:00
Sami Mokaddem	6f0227a78a	Starting support of python3	2017-11-20 12:12:06 +01:00
Sami Mokaddem	ee7759be91	Updated alertHandler and ailleakObject	2017-11-16 11:18:13 +01:00
Sami Mokaddem	5c7c07fbea	Merge branch 'launcher_update' into misp_obj	2017-11-16 10:09:05 +01:00
Sami Mokaddem	45a9897020	Added draft support of MISP ail-leak object	2017-11-16 09:52:37 +01:00
Sami Mokaddem	652b00f4b2	Renamed BrowseWarningPaste into alertHandler	2017-11-15 16:15:43 +01:00
Sami Mokaddem	ec7d2d49b2	Added seconds args in import_dir	2017-11-15 16:03:42 +01:00
Sami Mokaddem	13f1c676d6	Added '_AIL' to the end of screen session name to better check an kills sessions. Also, gracefully close redis server + Check 1 time if redis-servers are running.	2017-09-26 16:17:06 +02:00
=	a8dc9231c6	Typo2..	2017-08-24 16:43:42 +02:00
=	b2d4dd4a1f	Typo	2017-08-24 16:43:05 +02:00
=	74ff401e1d	Fixed bug: No CR between indexes in all_index.txt file	2017-08-24 16:41:30 +02:00
Alexandre Dulaunoy	bb128a5e94	Merge pull request #138 from mokaddem/import_dir_update ... Improvement of import_dir	2017-08-24 09:40:52 +02:00
=	429cc96d2a	Improvement of import_dir.py. Added feed name parameter, send the correct path (without potential ../) and gzipEncode it if needed.	2017-08-23 15:10:26 +02:00
=	e7c66d1aac	Updated individual launch scripts	2017-08-23 12:28:12 +02:00
=	d632335760	Automatically start redis_levelDB with the correct date	2017-08-23 10:16:56 +02:00
=	64ec44085d	Dynamically choose correct port for LevelDB in script	2017-08-22 17:52:15 +02:00
=	66d60a1f01	Added LevelDB Curve for 2017 (3017, used for backward compatibility)	2017-08-21 13:54:44 +02:00
=	92adea38d0	Semi-automated directory creation with date as name in LAUNCH and installing_deps	2017-08-21 13:48:02 +02:00
=	bc35747f00	Updated KillAll in LAUNCH	2017-08-21 13:19:26 +02:00
Mokaddem	c3632c7d86	Added operation mode 3 for Mixer (do not filter) and added possibility to recurse through dir for import_dir	2017-08-03 00:34:53 +02:00
Mokaddem	cbb2909156	Added comments	2017-07-20 10:24:48 +02:00
Mokaddem	05417f405c	Finished basic link web interface + solved some bugs	2017-07-20 10:04:30 +02:00
Mokaddem	ef741f85c0	Added similarity checks + started link webInterface	2017-07-19 11:52:06 +02:00
Mokaddem	882ab88b00	Draft module + web link	2017-07-18 16:57:15 +02:00
Mokaddem	9558aafe67	Prevent the loss of the 1 message in import_dir.py	2017-07-17 16:14:27 +02:00
Mokaddem	4d59871f00	Merged with previous changes for PR	2017-07-17 13:48:16 +02:00
Alexandre Dulaunoy	4986352099	Merge pull request #120 from PaulSec/master ... Added Python Phonenumbers module to parse numbers and retrieves stats…	2017-06-08 07:05:52 +02:00
Alexandre Dulaunoy	446a97c47a	Merge pull request #115 from mokaddem/regexInTerms ... New features for terms frequency	2017-06-08 07:04:17 +02:00
Mokaddem	3a4dcd691d	Improved description of modules inside the scripts	2017-05-09 11:13:16 +02:00
Paul A	747fff1df1	Added Python Phonenumbers module to parse numbers and retrieves stats from countrycode	2017-05-03 14:25:18 +02:00
Mokaddem	3ea0924120	Improved phone regex to handle end with 4 digits and presence of parenthesis.	2017-04-26 12:41:23 +02:00
Mokaddem	b60e164380	Added support of onion in browseWarningPaste	2017-04-26 11:47:02 +02:00
Mokaddem	318a557c56	Added possibility to keep a backup of the old config + check if config is valid before starting scripts.	2017-04-25 09:32:05 +02:00
Mokaddem	72c1d903eb	Merge branch 'newIndexer' into auto-conf	2017-04-25 08:14:31 +02:00
Mokaddem	eccc8fc88b	modified check_size timeout + utilisation of shutil instead of os.system()	2017-04-19 16:37:04 +02:00
Mokaddem	da081bae42	Merge branch 'regexInTerms' into production	2017-04-18 16:59:40 +02:00
Mokaddem	2d907ae784	Merge branch 'newIndexer' into auto-conf	2017-04-18 16:20:00 +02:00
Mokaddem	b0f1cf84b7	remplaced usage of global variable by a function param	2017-04-18 16:02:22 +02:00
Mokaddem	6e8509a0ba	Removed wierd line	2017-04-18 15:57:07 +02:00
Mokaddem	c8baabd882	Updated TermsFrequency related modules + Fixed bugs (encoding, behaviors, ...)	2017-04-18 15:28:21 +02:00
Mokaddem	2da4c572c7	New feature in Terms: regex and set of words support (draft)	2017-03-28 17:42:44 +02:00
Mokaddem	6e80eb0954	Typo	2017-03-28 09:55:59 +02:00
Mokaddem	c79d4d65c8	Keep original section order while updating	2017-03-28 09:54:24 +02:00
Mokaddem	e225090e07	Added script that checks configuration and may auto-fix it + added missing field in config.cfg.sample	2017-03-15 16:36:51 +01:00
Mokaddem	be93af6f5a	Merge branch 'terms-improvements'	2017-03-15 14:55:44 +01:00
Mokaddem	3261eca836	Fixed all_index newline bug + added print of what's happening	2017-03-15 14:29:49 +01:00
Mokaddem	5b7d047622	Changed os.system by os.mkdir + fixed bug during rotation	2017-03-15 14:05:13 +01:00
Mokaddem	adc37ab50e	Indexer max size now in config	2017-03-15 12:14:41 +01:00
Mokaddem	4005b72180	Added backward support + renamed index with timestamp	2017-03-15 11:51:35 +01:00
Mokaddem	248469d61e	Indexer now create an index_dir when it became too large (search in index not updated yet)	2017-03-14 10:37:31 +01:00
Alexandre Dulaunoy	1540b719ed	Merge pull request #108 from mokaddem/newModuleInformation ... New version of ModuleInformation	2017-03-14 08:29:09 +01:00
Mokaddem	3c9cdac39a	Merge branch 'terms-improvements' into production	2017-02-28 16:51:39 +01:00
Mokaddem	dadc872f9f	Merge branch 'newModuleInformation' into production	2017-02-28 16:51:08 +01:00