1122 commits

Author	SHA1	Message	Date
Terrtia	3a8531cafa	chg: [API + import] add API format + item_import refractor	2019-07-25 17:26:32 +02:00
Terrtia	653cb4a209	chg: [misp_modules] parse modules result	2019-07-24 15:51:06 +02:00
Terrtia	09ecc4d93f	chg: [Crawler] add default crawler config + update default user_agent	2019-07-24 10:18:10 +02:00
Terrtia	6295560216	chg: [misp_modules] add enrichment request	2019-07-23 17:09:29 +02:00
Terrtia	8cbfec51f6	chg: [misp_modules] create misp modules + generate config	2019-07-23 16:44:02 +02:00
Terrtia	26a4c7fd2c	fix: [Crawler] incorrect config	2019-07-10 09:42:20 +02:00
Terrtia	34786d54db	Merge branch 'master' into user_management	2019-07-02 14:12:28 +02:00
Terrtia	bd0da210e7	fix: [helper dump_crawler] fix files not found	2019-06-26 11:51:26 +02:00
Terrtia	06ab66ff57	fix: [helper dump_crawler] fix empty dict	2019-06-26 11:42:19 +02:00
Terrtia	5377369032	chg: [helper] dump crawler history by daterange	2019-06-26 11:29:28 +02:00
Terrtia	87c5e0f9ee	Merge branch 'master' into user_management	2019-06-24 16:06:08 +02:00
Terrtia	3943b3ed9a	fix: [Update] add default update script	2019-06-24 15:51:42 +02:00
Terrtia	03ed3c63a1	chg: [helper] generate self signed certificates	2019-06-24 14:41:54 +02:00
Terrtia	bb65179e50	fix: [user_management] fix tokens duplicate + check user_acl_integrity + add login errors messages	2019-06-20 15:49:40 +02:00
Terrtia	6eec0aabfe	Merge branch 'master' into user_management	2019-06-19 15:11:44 +02:00
Terrtia	34e57fe7af	chg: [user_managemant] clean code + check password and email length	2019-06-19 15:00:25 +02:00
Terrtia	e65f465f02	fix: [TheHive feeder] create_alert: push all items tags ... TODO check items status + add more item metadata	2019-06-19 10:42:36 +02:00
Terrtia	42d32ef310	fix: [domain history] fix domain status	2019-06-18 11:19:56 +02:00
Terrtia	e1bf59aded	chg: [UI crawler, show_domain] add domain history list + navigation	2019-06-18 10:58:37 +02:00
Terrtia	2b6cb86d55	fix: [PgpDump] catch UnicodeDecodeError error	2019-06-12 10:33:40 +02:00
Terrtia	b4f06c21f9	chg: [backend crawler] domains: download 1 archive by crawled (most recent)	2019-06-07 13:47:44 +02:00
Terrtia	3fe9d14e9f	chg: [user_management] create default admin user (temp passwd save in AIL_HOME) + change password UI + logout UI + create random password	2019-06-06 21:27:13 +02:00
Terrtia	99e35c51ec	Merge branch 'master' into user_management	2019-06-05 16:59:36 +02:00
Terrtia	17af705231	fix: [PgpDump] process large pgp blocks	2019-06-05 16:20:26 +02:00
Alexandre Dulaunoy	fae3530fec	chg: [PgpDump] PMF fix - longer keys are truncated ... TODO: Need to be updated by an async process handling the keys	2019-05-24 14:10:06 +02:00
Terrtia	4b0a7210b8	Merge branch 'master' of https://github.com/CIRCL/AIL-framework	2019-05-24 13:27:44 +02:00
Terrtia	eb9c81acef	chg: [correlation] clean files	2019-05-24 13:08:10 +02:00
Terrtia	4794cafe01	chg: [correlation] add cryptocurrency + refractor correlation	2019-05-24 12:02:43 +02:00
Terrtia	ff23a2bce2	chg: [Bitcoin] map cryptocurrency: bitcoin (DB pivot)	2019-05-21 16:14:09 +02:00
Terrtia	da5579875d	chg: [pgpdump] reprocess tagged items + fix pgpdump	2019-05-20 14:32:07 +02:00
Terrtia	49f7429c5f	chg: [Update] force update order	2019-05-20 13:02:09 +02:00
Terrtia	9639a51abd	chg: [PgpDump] fix graph + add new tags: pgp-signature pgp-public-key-block + avoid keys injection in pgp user_id	2019-05-20 11:57:49 +02:00
Thirion Aurélien	9c6a1ea8ec	Merge pull request #349 from kovacsbalu/fix-paste-encoding ... Fix #314	2019-05-16 14:30:33 +02:00
kovacsbalu	f61d830678	Use default encoding error from redis.	2019-05-16 14:24:03 +02:00
kovacsbalu	7765ab92e0	Hopp, single quote :)	2019-05-15 10:00:51 +02:00
kovacsbalu	6092f482e6	Fix crawler rotation ... Before this, crawler processed prioritized onions and after all starts prioritized regular.	2019-05-15 09:57:18 +02:00
Terrtia	6480744641	chg: [PgpDump] add PgpDump backend ... TODO: UI	2019-05-14 17:49:31 +02:00
kovacsbalu	05079c143c	Fix #314 ... Replace char on redis encoding error. Try to use local file on other error.	2019-05-14 16:06:20 +02:00
Terrtia	0389b9c23b	chg: [crawler] manual/auto crawler: always save screenshots	2019-05-13 14:24:16 +02:00
Terrtia	254441f193	chg: [crawler] manual/auto crawler: always save screenshots	2019-05-13 13:56:43 +02:00
Terrtia	2606220c2b	fix: [faup] fix new return types (bytes to str)	2019-05-06 13:38:13 +02:00
Terrtia	a4c03b4ba4	fix: [Crawler] force domains/subdomains lower case (rfc4343)	2019-05-06 11:46:20 +02:00
Terrtia	64ff94ce5f	chg: [user_management] create + check user password	2019-05-03 16:52:05 +02:00
Terrtia	1ab1a55a4f	chg: [UI] add basic user management	2019-05-02 17:31:14 +02:00
Steve Clement	9f65bc3788	chg: [travis] Travis has his own venv where it installs "stuff". Now we detect and us it in the launcher.	2019-05-01 21:31:47 +09:00
Terrtia	f30befa0eb	fix: [Onion] typo	2019-04-29 11:46:28 +02:00
Terrtia	a183695ddb	chg: [crawler UI] bootstrap 4 migration: show domains by daterange + bug fix hidden services tags	2019-04-26 15:14:29 +02:00
Terrtia	ff14fe6279	fix: [submit paste] remove item full path, use rel path	2019-04-26 10:22:02 +02:00
Terrtia	3e6fbc8f79	fix: [paste submit] update broken endpoint + get item date	2019-04-26 10:04:07 +02:00
Terrtia	3207949c5f	fix: [update v1.5] create update config file	2019-04-25 17:17:12 +02:00
Terrtia	41e6b4ec59	fix: [v1.5 background update screenshot] remove duplicate files	2019-04-25 14:39:38 +02:00
Terrtia	499c07a93e	fix: [update v1.5] remove unbuffer	2019-04-25 14:21:02 +02:00
Terrtia	fc2c1422ff	fix: [Crawler] unpack_url	2019-04-25 13:54:06 +02:00
Terrtia	50944b1d61	fix: [LAUNCH] fix typo	2019-04-24 16:37:44 +02:00
Terrtia	e6dca7f8bf	chg: [update v1.5] add background update: screenshots_crawled	2019-04-24 16:19:35 +02:00
Terrtia	9868833c77	chg: [crawled screenshot] use sha256 as filepath	2019-04-24 14:09:04 +02:00
Terrtia	99b9c95638	fix: [update] fix is_fork check	2019-04-23 15:30:47 +02:00
Terrtia	35c188829f	chg: [config update] refractor, auto update, create config if file doesnt exist	2019-04-23 15:04:45 +02:00
Terrtia	2a1cd4a009	chg: [Onion, crawler config] auto crawler: add config by url, fix onions tagging + filter subdomains	2019-04-23 11:15:34 +02:00
Terrtia	6fdf7c2123	chg: [UI crawler] status/remove auto crawler	2019-04-18 16:57:51 +02:00
Terrtia	ddaad5db93	chg: [v1.5 background update] create update stats	2019-04-17 17:07:09 +02:00
Terrtia	4bff603d4f	chg: [settings UI] add background update progress	2019-04-17 14:39:29 +02:00
Terrtia	68077316eb	chg: [git directory] add git helper	2019-04-16 17:26:35 +02:00
Terrtia	3b11df7f90	chg: [UI] add server Management, check if new update/version are available + background update status + link to new realease/commits	2019-04-16 17:24:59 +02:00
Terrtia	a3167a740a	fix: [Update DB] remove indexErrors	2019-04-12 17:32:17 +02:00
Terrtia	bf92a2f22f	fix: [background update] avoid multiple update on the same key	2019-04-12 16:07:40 +02:00
Terrtia	f2dc3bcbf4	fix: [Global] remove absolute path from item path	2019-04-12 15:13:36 +02:00
Terrtia	3a8631d44a	chg: [Update + Launcher] use background update + add new options in launcher	2019-04-12 15:06:24 +02:00
Terrtia	5c4db5f2f1	chg: [update] add update script Redis ARDB + add background update	2019-04-11 17:49:20 +02:00
Terrtia	3cc614a1ad	chg: [DB] update items tags metadata	2019-04-11 11:58:06 +02:00
Terrtia	d44acea04d	chg: [merge] merge update into advanced_crawler	2019-04-10 16:41:06 +02:00
Terrtia	2589fc2161	chg: [merge] merge master into update branche	2019-04-10 15:43:15 +02:00
Terrtia	b4bee29a33	Merge branch 'master' into advanced_crawler	2019-04-10 15:37:42 +02:00
Terrtia	f6d7d2ae16	chg: [Update] add debug	2019-04-02 09:46:04 +02:00
Thirion Aurélien	976b97bcd1	Merge pull request #338 from CIRCL/pre_update ... Merge Pre update	2019-03-26 16:51:26 +01:00
Terrtia	044933aa33	chg: [Launch] remove old launchers	2019-03-26 16:47:10 +01:00
Terrtia	cee2b2486f	chg: [Launch] remove old launchers	2019-03-26 16:45:01 +01:00
Terrtia	59664efe45	Merge branch 'master' into advanced_crawler	2019-03-26 16:03:42 +01:00
Terrtia	5165a5de2f	fix: [Crawler] fix index + redis history key	2019-03-22 17:14:27 +01:00
Terrtia	f64c385343	chg: [Crawler] handle port: crawling + history	2019-03-22 16:48:07 +01:00
Terrtia	f4cdddbc7f	chg: [UI + item submit] remove browse paste from UI + fix pasteSubmit tagging	2019-03-21 11:50:53 +01:00
Terrtia	c2bc72c72e	chg: [Tags alertHandler] remove alertHandler + add quick tag search sidebar + use tag last_seen to display tagged items	2019-03-21 11:11:53 +01:00
Terrtia	a9e86231ad	chg: [Tags] add backend, daterange selector	2019-03-20 13:25:02 +01:00
Terrtia	65e6eae00d	chg: [UI preview items] b4 migration + new template + better display + decrease item loading time + add basic animation/transition	2019-03-19 13:34:41 +01:00
Terrtia	23d9cb2265	chg: [UI Tags search] bootstrap 4 migration	2019-03-14 17:04:55 +01:00
Alexandre Dulaunoy	85d2ea578c	Merge pull request #335 from 5thColumn/docker ... Add docker-compose support	2019-03-13 21:38:12 +01:00
Jason Hedden	23fb91d326	Add docker-compose support ... This commit separates the AIL Framework services into individual Docker containers that can be managed with docker-compose. It's intended to ease the installation, development and troubleshooting procedures for all-in-one deployments. No changes to architecture or base code are required in this commit. Changes to existing files: .gitignore * Ignore pystemon archive directory Dockerfile: * Add AIL shell environment variables * Install pystemon and crawler pip requirements into AIL virtual environment Docker-compose notes: * All containers are participating in a service network `network_mode: service:flask`. This allows the containers to share the same IP namespace to accommodate hard-coded localhost entries. * By default persistent data is saved to the following directories in the local AIL framework git directory: PASTES, HASHS, CRAWLED_SCREENSHOTS, pystemon/archives This was tested with docker-ce on Ubuntu 16.04 and MacOS. A typical deployment would look like: ``` git clone https://github.com/CIRCL/AIL-framework.git cd AIL-framework cp bin/packages/config.cfg.docker-compose-sample bin/packages/config.cfg (optionally enable activate_crawler in config.cfg) docker-compose build docker-compose up -d ```	2019-03-11 19:46:34 -05:00
Terrtia	d3b3942ad7	chg: [Flask UI] refractor crawler frontend + show crawled domain	2019-03-06 15:05:58 +01:00
Terrtia	c0d72e7d2a	chg: [Crawler UI] Crawler major refractor (end) + basic UI for manual crawler	2019-02-26 14:50:48 +01:00
Terrtia	7b32d7f34e	chg: [Crawler] major refractor	2019-02-25 16:38:50 +01:00
Jason Hedden	9eae25bdf5	Fix current_selected amount and action vars ... This commit fixes issue #268 ``` File "./bin/ModulesInformationV2.py", line 319, in _setValue action = current_selected_action + " " + str(current_selected_amount) + "x" ``` `current_selected_action` was defined twice. The second occurance should be `current_selected_amount`. This also fixes the generic exception from failing when `current_selected_value` is not defined.	2019-02-22 12:39:27 -06:00
Terrtia	60f7645ac1	chg: [Crawler] refractor	2019-02-22 17:00:24 +01:00
Terrtia	3d36ddbc1e	fix: [CVE] fix stat regex	2019-02-21 10:51:05 +01:00
Terrtia	1114aa2ffd	chg: [CVE] add stat script	2019-02-21 09:58:10 +01:00
Terrtia	e5dca268a8	chg: [Crawler] refractor	2019-02-21 09:54:43 +01:00
Terrtia	da78d0552d	chg: [Crawler UI Tags] add tag by day + add crawler status + UI onion blacklist	2019-02-19 11:41:45 +01:00
Terrtia	6a8fa3d5d1	chg: [Tags] tags by day	2019-02-18 14:33:34 +01:00
Terrtia	709d5487b8	fix: [Onion] filter too many subdomain	2019-02-13 16:53:38 +01:00
Terrtia	b87707e8bc	fix: [Crawler] typo	2019-02-12 15:54:42 +01:00
Terrtia	37276e52a3	fix: [Crawler] typo	2019-02-12 15:53:40 +01:00
Terrtia	7cb03fc769	fix: [Crawler] typo	2019-02-12 15:51:19 +01:00
Terrtia	7a4989ce10	fix: [Global Crawler] max filename size	2019-02-12 15:45:58 +01:00
Terrtia	c2885589cf	chg: [UI] basic navbar + sidebar + refractor	2019-02-07 17:22:44 +01:00
Terrtia	516238025f	chg: [Crawler] add bootsrap4 src + refractor crawler	2019-02-05 17:16:44 +01:00
Thirion Aurélien	44c513dcbb	chg: [Crawler] add onion to blacklist	2019-01-31 16:56:45 +01:00
Xavier Mertens	43d08d0d3e	Cleanup	2019-01-30 22:36:10 +01:00
Xavier Mertens	1be91b88bf	Fix: reset alarm timeout	2019-01-30 22:30:07 +01:00
Xavier Mertens	0372a78704	Merge remote-tracking branch 'upstream/master'	2019-01-30 22:27:44 +01:00
Xavier Mertens	b099e2ae4a	Update LAUNCH.sh	2019-01-30 22:27:33 +01:00
Terrtia	92d192238b	fix: [Crawler] change max page crawled	2019-01-29 17:04:45 +01:00
Terrtia	6c7086f4eb	fix: [Crawler] first_seen	2019-01-29 16:54:39 +01:00
Terrtia	ad04cf7bb2	fix: [Onion] typo	2019-01-29 16:27:12 +01:00
Terrtia	88eaaeae93	chg: [Crawler] add priority queue, fix #263	2019-01-29 16:08:59 +01:00
Terrtia	c1b34bd99c	fix: [Crawler] limit max crawled pages	2019-01-29 15:38:00 +01:00
Terrtia	64ffdd52e8	fix: [Onion] loop	2019-01-29 14:39:57 +01:00
Terrtia	2dc0eca4a9	fix: [Crawler] fix crawler cache info	2019-01-29 12:09:19 +01:00
Terrtia	bb301a870c	fix: [Crawler] fix onion blacklist + add crawler info	2019-01-29 12:00:14 +01:00
Terrtia	535f44044d	fix: [Onion] add timeout signal	2019-01-29 09:46:03 +01:00
Thirion Aurélien	f4ba21e492	blacklist onion debian manpages	2019-01-14 11:08:53 +01:00
raggadhub	74c726d5fa	fix path_to_send path ... Hi, Missing slash between pastes_directory and paste in path_to_send = pastes_directory+paste path error example: /opt/AIL-framework/PASTESarchive/pastebin.com_pro/2019/01/07/cAgqsY2U.gz	2019-01-09 13:15:58 +01:00
Terrtia	b3b75ccbea	fix: [Crawler] Restart Splash on failure, limit unbound in memory cache (maxrss)	2019-01-04 15:51:08 +01:00
Terrtia	6ee4b7bf93	Merge branch 'master' into update	2018-12-19 15:23:34 +01:00
Terrtia	ca47764836	fix: [update v1.5]	2018-12-19 15:16:52 +01:00
Terrtia	198ee97d90	chg: [update 1.5] add update bash	2018-12-19 11:41:01 +01:00
Terrtia	2560260ce4	Merge branch 'master' into pre_update	2018-12-19 10:01:52 +01:00
Terrtia	4d04333f54	fix: [Splash server] add debug output	2018-12-19 09:30:24 +01:00
Terrtia	f842194c57	fix: [Crawler] retry when splash is not available	2018-12-17 16:04:12 +01:00
Terrtia	4e08aaa80f	chg: [UI Crawled Domains] show crawled domains by date range (options: domains tags, up domains, down domains) ... todo: Shows domain history	2018-12-17 15:58:48 +01:00
Steve Clement	b60fd55efe	chg: [bash] Small PATH precedence issue.	2018-11-22 16:16:31 +09:00
Steve Clement	f82c899e7b	fix: [bash] fixed log_subscriber path	2018-11-22 15:48:12 +09:00
Steve Clement	88e561aa20	chg: [bash] Replace '' with ""	2018-11-22 15:41:45 +09:00
Steve Clement	33bc23a297	fix: [bash] evoking bash -c and splitting vars in this fashion wants to be changed in future revisions.	2018-11-22 15:22:35 +09:00
Steve Clement	6522cffd57	chg: [doc] Added note that redis-server is bundled during install	2018-11-22 15:10:57 +09:00
Steve Clement	41a34b3519	fix: [bash] As we instantiate from a subdirectory, we need to take this into account.	2018-11-22 15:08:59 +09:00
Steve Clement	e4ced241eb	chg: [python] Added necessary checks in LAUNCH.sh to be able to launch the script without doing bin/activate ... chg: [doc] Added correct pystemon path fix: [zmq] Small ZMQ test fix. fix: [doc] Moved the print statement in feeder so it is less verbose but still useful	2018-11-22 14:57:24 +09:00
Terrtia	31a8dfe0b3	fix: [AIL] use only relative paths ... pastes duplicates are fixed on the fly	2018-11-21 16:45:25 +01:00
Terrtia	cb41c54135	fix: [lib_refine] add dns configuration	2018-11-21 08:49:58 +01:00
Terrtia	4e680aabf0	chg: [Overview] add doc	2018-11-20 14:39:45 +01:00
Terrtia	aaa277b8a0	chg: [Update] add additonal update	2018-11-15 14:26:41 +01:00
Terrtia	f6e86582c8	chg: [Update] generate current version	2018-11-15 13:48:44 +01:00
Terrtia	108fdb868e	chg: update Overview	2018-11-15 10:39:41 +01:00
Terrtia	d9b776d58a	merge master	2018-11-14 15:55:42 +01:00
Terrtia	aed8d65aef	fix: [LAUNCH] add update in helper	2018-11-14 15:17:56 +01:00
Terrtia	912b977bb4	chg: [LAUNCH] update: launch + check BDD	2018-11-13 16:54:39 +01:00
Terrtia	94fcf66d20	fix: [Update] cleaning	2018-11-13 15:52:33 +01:00
Terrtia	347986a271	chg: [LAUNCH] add AIL update by default	2018-11-13 15:45:22 +01:00
Terrtia	7aff45c507	chg [Update] update AIL clone and fork	2018-11-13 14:54:56 +01:00
Terrtia	60ff0b9cf7	chg: [Update] add update script	2018-11-12 17:10:31 +01:00
Terrtia	b31618eb6a	chg: [Mixer] config default feeder name #291	2018-11-09 15:26:26 +01:00
Terrtia	c3903ab203	fix: [MISP_The_Hive_feeder] handle not saved pastes	2018-11-09 13:58:09 +01:00
Sami Mokaddem	62e46b6dc1	Merge pull request #287 from CIRCL/term_management ... Term management upgrade	2018-11-08 14:10:25 +01:00
Terrtia	e3764ca575	fix: [MISP_the_Hive_feeder] force absolute path	2018-11-08 11:22:23 +01:00
Terrtia	979472df15	chg: [Terms] tag tracked terms	2018-11-06 13:38:37 +01:00
Terrtia	7ef5cc3205	fix:[MISP_The_Hive_feeder] typo	2018-11-05 16:40:25 +01:00
Terrtia	85f933fd81	chg: [NotificationHelper] add error traceback	2018-11-05 14:59:40 +01:00
Terrtia	877ac0636e	fix: typo	2018-11-05 14:30:03 +01:00
Terrtia	aa16c52cbf	fix: [NotificationHelper] add config	2018-11-05 14:20:12 +01:00
Terrtia	dad0365c67	fix:[MISP_The_Hive_feeder] set default values	2018-11-05 09:16:18 +01:00
Terrtia	5872cf9196	fix: [Scripts] Remove absolute path	2018-11-02 16:07:27 +01:00
Xavier Mertens	05de39e271	Fixed line counter (count all lines)	2018-11-02 15:55:55 +01:00
Xavier Mertens	ac95dd4ee8	Added timeout for Regex module	2018-11-02 15:50:38 +01:00
Xavier Mertens	9e7ca84581	Added timeout to avoid blocking regexes	2018-11-02 15:49:06 +01:00
Xavier Mertens	9bfb7a5fb6	Added module Regex.py	2018-10-26 17:13:26 +02:00
Terrtia	ea7fd8d54c	fix: [Duplicate] show duplicate with relative path	2018-10-24 15:47:03 +02:00
Terrtia	5afbc3179c	Merge branch 'master' of https://github.com/CIRCL/AIL-framework	2018-10-24 15:16:41 +02:00
Terrtia	8a6ac6be1e	fix: [Paste] create absolute path	2018-10-24 15:16:12 +02:00
Sami Mokaddem	d9bc2d6f44	Merge pull request #266 from kovacsbalu/fix-regexp-timeout ... Stop regexp processing after timeout (60sec)	2018-10-09 14:48:08 +02:00
kovacsbalu	c1c40e3e56	Fix config section name	2018-10-09 11:30:04 +02:00
kovacsbalu	272b0fe960	RegexpForTerm timeout from config ... Critical log removed	2018-10-09 09:32:32 +02:00
kovacsbalu	6d199f0150	Stop regexp processing after timeput (60sec) ... Minor pep8 fixes	2018-10-08 11:25:32 +02:00
Alexandre Dulaunoy	efaba35dec	Common onion addresses	2018-10-05 13:50:17 +02:00
Alexandre Dulaunoy	ed9f9e5dce	Merge pull request #262 from kovacsbalu/feat-smtp-test ... Test notification	2018-10-02 17:27:55 +02:00
Terrtia	44d19fbb80	fix: [pystemon-feeder] paste save path	2018-10-02 16:02:59 +02:00
Alexandre Dulaunoy	59df5f0e12	chg: [config] Tor crawler disabled by default in sample configuration ... (to match HOWTO when enabling it)	2018-10-02 14:17:58 +02:00
kovacsbalu	93a3c5779c	Test notification ... Minor pep8 fixes	2018-10-01 15:56:48 +02:00
Terrtia	6328cc22b7	chg: [Crawler] add domains blacklist	2018-09-28 16:29:09 +02:00
Terrtia	82e6df4b94	chg: [Crawler] domains stats + logs + clean	2018-09-28 15:23:27 +02:00
Terrtia	ecb2857151	chg: [Crawler] css + limit splash RAM	2018-09-27 16:47:48 +02:00
Terrtia	e357dce59b	fix: [Crawler] detect splash connection to proxy error	2018-09-27 15:43:03 +02:00
Terrtia	04b9d9fc1d	chg: [Crawler] add docs	2018-09-27 11:14:29 +02:00
Terrtia	c49e871ba8	chg: [crawler] add infos	2018-09-26 16:34:27 +02:00
Terrtia	e4a5f66a10	Merge branch 'master' into onion_crawler	2018-09-24 16:35:04 +02:00
Terrtia	874824a589	fix: [Crawler] clean	2018-09-24 16:28:55 +02:00
Terrtia	8eca0e0778	fix: [Crawler] clean	2018-09-24 16:24:30 +02:00
Terrtia	50c81773e9	chg: [Crawler] add launcher and install	2018-09-24 16:23:14 +02:00
Terrtia	b575efcc46	fix: [pystemon] pystemon encoding	2018-09-21 14:16:06 +02:00
Terrtia	ead4dba0d8	fix: [MISP feeder] typo filename-config name	2018-09-21 11:02:08 +02:00
Terrtia	6edc1ddbeb	chg: [HiddenServices] get domain related to other domains	2018-09-21 10:34:06 +02:00
Terrtia	5eadc2755f	chg: [Flask] add prefix in config to flask routes	2018-09-20 10:38:19 +02:00
Terrtia	2e763ba0aa	fix: [ModulesInformationV2] fix #244	2018-09-19 15:45:06 +02:00
Terrtia	5732d7b134	fix: [hashDecoded] range_type_json	2018-09-19 15:36:33 +02:00
Terrtia	ebad2af1ae	fix: [import_dir] fix #251, fix paste import ... if the date directory tree don't exist, he will be generated	2018-09-19 13:38:31 +02:00
Terrtia	653463a8c4	fix: [LAUNCH.sh] make pystemon optionnal ... AIL and Pystemon are independent, pystemon shouldn't be launch by default.	2018-09-19 09:52:27 +02:00
Terrtia	5b31b6e853	fix: [Crawler] save domain to crawl on splash error	2018-09-18 16:20:13 +02:00
Terrtia	f5b648d72a	pixelate paste screenshot	2018-09-18 11:03:40 +02:00
Terrtia	6aa1b99c2a	fix: [LAUNCH.sh] remove old decoders	2018-09-17 15:50:27 +02:00
Terrtia	0c63f2f24f	chg: [Crawler] catch server response	2018-09-17 15:35:06 +02:00
Alexandre Dulaunoy	a6c72a677d	Merge pull request #243 from kovacsbalu/docker-with-pystemon ... Docker build with pystemon	2018-09-12 21:50:11 +02:00
Terrtia	a94a0fe1a7	fix: [Install] add python requirements	2018-09-12 19:10:49 +02:00
Terrtia	204e996fc3	chg: [statistics] clean scripts	2018-09-12 11:21:11 +02:00
Terrtia	ea3d2c1977	Merge branch 'master' into statistic	2018-09-12 10:06:53 +02:00
Terrtia	6f0817365a	chg: [Crawler UI] display domain information	2018-09-12 09:55:49 +02:00
kovacsbalu	d19dd72f99	Install pystemon and start pystemon-feeder in docker	2018-09-03 16:01:39 +02:00
Terrtia	ca982e13e1	chg: [Crawled Domain] show crawled pastes by domain	2018-08-27 14:34:08 +02:00
Terrtia	823c33174a	Merge branch 'master' into onion_crawler	2018-08-27 11:20:51 +02:00
Terrtia	5c902640e9	Merge branch 'master' into onion_crawler	2018-08-27 11:05:25 +02:00
Terrtia	d42dd118a4	chg: [Domain crawled] add random screenshot	2018-08-27 11:02:39 +02:00
Terrtia	a586548047	Merge branch 'master' into base64	2018-08-24 10:35:28 +02:00
Terrtia	ced0b1e350	chg: [I2P] add default config	2018-08-24 10:24:03 +02:00
Terrtia	7e24943537	chg: [Crawler] crawler accept all kind of domains	2018-08-24 10:13:56 +02:00
Terrtia	e9580d6775	chg: [Crawler] change BDD, save i2p links	2018-08-21 15:54:53 +02:00
Terrtia	ed559d9f4a	chg: [Showpaste] add screenshot + improve onion db	2018-08-16 17:24:39 +02:00
Terrtia	6f2dbcb4e2	fix: [pystemon-feeder] fix #234	2018-08-13 10:22:20 +02:00
Terrtia	7652089433	chg: [Onion] change onion regex, fix crawler	2018-08-13 09:23:14 +02:00
Terrtia	8b1c10b38c	chg: [Onion] add onion splash crawler	2018-08-09 17:42:21 +02:00
Thirion Aurélien	000d5bf40a	fix: add miss config	2018-08-07 13:07:08 +02:00
Terrtia	d7545a46f0	chg: [Mixer] add submitted paste	2018-08-02 15:29:13 +02:00
Terrtia	449da63fd7	chg: [Keys] typo	2018-08-01 16:05:49 +02:00
Terrtia	67b0d973af	chg: [Keys] detect ssh2 private key	2018-08-01 16:04:06 +02:00
Terrtia	d0135c248d	add iban country statistic	2018-08-01 15:26:29 +02:00
Terrtia	940b96734a	Merge branch 'master' into statistic	2018-08-01 09:29:51 +02:00
Thirion Aurélien	f7f4422ff1	Merge pull request #232 from CIRCL/iban ... Add Iban detection	2018-08-01 08:57:48 +02:00
Terrtia	b5c1f281e6	fix: typo	2018-07-31 10:58:23 +02:00
Terrtia	61e98ed8fb	clean	2018-07-31 10:51:18 +02:00
Terrtia	e455d244d8	chg: [statistics] add domain statistics	2018-07-31 10:36:29 +02:00
Terrtia	5dcb4ebb58	chg: [statistics] add script to generate pie chart png	2018-07-30 16:36:34 +02:00
Terrtia	09fbc363f1	chg: [sglinjection Phone] add tld statistic, fix phone regex	2018-07-30 11:56:50 +02:00
Terrtia	c20e7d5ab4	chg: [Mail Credential] add tld statistic	2018-07-30 10:19:26 +02:00
Terrtia	39f54d4420	chg: [Mail] add country statistic	2018-07-30 09:21:22 +02:00
Terrtia	009f87ee10	[fix: [UI-hash] fix graph node zoom	2018-07-27 11:27:41 +02:00
Terrtia	153f9f38cc	chg: [BankAccount] regex	2018-07-27 10:20:03 +02:00
Terrtia	57b631233b	add: [BankAccount] iban detection	2018-07-26 15:31:58 +02:00
Terrtia	5ad5939bd5	chg: display last logs	2018-07-25 16:48:44 +02:00
Terrtia	03a37af5cf	chg: [Hash] add reprocess regex order	2018-07-25 10:47:36 +02:00
Terrtia	8beb63b6a8	fix: [Queues] module with no subscriber	2018-07-24 15:57:18 +02:00
Terrtia	8c97c70a3f	Merge branch 'master' into base64	2018-07-24 15:21:32 +02:00
Terrtia	4b8c26dec8	fix: [hive auto export] fix #221	2018-07-24 14:46:35 +02:00
Terrtia	54e8750c30	fix: [LAUNCH] script checking	2018-07-24 12:18:41 +02:00
Terrtia	d4f0c14620	fix: [LAUNCH] use envar, add options (launch all, flask, killall)	2018-07-24 10:41:10 +02:00
Terrtia	9c03c40482	chg: [UI-Hash] add decoder pie	2018-07-23 11:11:52 +02:00
Terrtia	1fbd4109c2	chg: [UI-Hash] add decoded type on graph	2018-07-20 15:48:56 +02:00
Terrtia	6d6de3ae54	fix: remove experimental modules	2018-07-20 10:38:04 +02:00
Terrtia	bd5f83f0eb	chg: refractor base64 encoded to hash	2018-07-20 10:32:52 +02:00
Terrtia	9a8e37fb0b	fix: decoder bdd	2018-07-20 09:43:09 +02:00
Terrtia	0e67b56906	add: Decoder module	2018-07-19 16:52:09 +02:00
Terrtia	ad26f016e3	add: Decoder Module, decode binary, hex and base64	2018-07-19 16:50:42 +02:00
Terrtia	280e9d8af8	fix: Binary decoder base2 string to bytes	2018-07-18 13:47:06 +02:00
Terrtia	e25dd52d64	add: Binary decoder	2018-07-18 11:45:19 +02:00
Terrtia	0cac3146f1	fix: remove hardcoded mail url	2018-07-17 15:11:25 +02:00
Terrtia	fa38434e04	add: new mail body and starttls	2018-07-16 15:51:37 +02:00
Terrtia	8b245d17d9	fix: add LibInjection to LAUNCH.sh	2018-07-16 15:45:55 +02:00
Terrtia	6f69da0c0d	fix: paste submit css	2018-07-16 10:32:41 +02:00
Terrtia	fce324e370	fix: release tiemeout alarm, chg:icon graph node	2018-07-13 11:54:14 +02:00
Terrtia	87b7facba8	add: base 64 node graph	2018-07-12 17:07:17 +02:00
bui	bd5c42caeb	add LibInjection Module	2018-07-04 14:28:10 +02:00
bui	362f72fe43	initial LibInjection module	2018-07-04 14:27:54 +02:00
Thirion Aurélien	f8a88cecec	fix: hive_feeder hiveApi declaration	2018-07-02 15:23:21 +02:00
Terrtia	44524b25a4	fix submit paste	2018-06-29 15:30:01 +02:00
Terrtia	a0a639276c	fix submit paste	2018-06-29 15:29:02 +02:00
Terrtia	98d0235fc3	Merge branch 'master' into base64	2018-06-29 14:57:46 +02:00
Terrtia	f5bda70e82	Merge branch 'master' of https://github.com/CIRCL/AIL-framework	2018-06-29 13:51:08 +02:00
Terrtia	72146f938e	change: feeder name separator	2018-06-29 13:50:41 +02:00
Thirion Aurélien	c41a165a21	fix: file submission, content type	2018-06-29 12:08:46 +02:00
Terrtia	407aa14244	fix: catch file error in paste submission	2018-06-29 11:32:52 +02:00
Terrtia	16c700ddb4	new: add base64 display on paste + base64 statistics	2018-06-29 10:02:29 +02:00
Terrtia	225776c69c	fix: tp and fp, errors messages on paste submission	2018-06-20 10:02:26 +02:00
Terrtia	6a89fac309	fix hive and misp error	2018-06-19 16:39:49 +02:00
Terrtia	af9db53171	add tag fp and tp	2018-06-19 15:09:26 +02:00
Terrtia	86c2cfa04b	update ail-leak object	2018-06-18 17:23:41 +02:00
Terrtia	634bf6e552	add img + fix upload api	2018-06-18 17:16:22 +02:00
Terrtia	704e6f0ad1	change default submit tag, event-path map, publish events	2018-06-18 13:58:31 +02:00
Terrtia	aee1b50ef1	fix duplicate export attribute + tag export whitelist	2018-06-15 17:25:43 +02:00
Terrtia	31fc33ab03	misp event creation, auto and manual + the hive alert auto creation, manual case creation	2018-06-14 16:51:06 +02:00
Terrtia	8522d50308	modify navbar + decompress file password	2018-06-11 10:21:11 +02:00
Terrtia	c64588f60e	add feature, user can submit paste on the web interface	2018-06-08 16:49:20 +02:00
Terrtia	6143bc3dce	submit users input paste	2018-06-05 16:58:04 +02:00
Terrtia	6287981668	clean	2018-06-01 09:49:26 +02:00
Terrtia	884d655b0c	taxonomies and galaxies update via LAUNCH	2018-05-31 13:53:29 +02:00
Terrtia	93dcdf4f23	fix duplicate	2018-05-31 09:01:47 +02:00
Terrtia	0d164b6f4c	add tag js and css	2018-05-16 14:48:15 +02:00
Terrtia	b1b7d33c96	tags	2018-05-16 14:39:01 +02:00
Terrtia	f66a528bc2	fix Duplicate, save list of duplicates on disk + prevent empty hash creation	2018-05-15 23:28:47 +02:00
Terrtia	225fe76c96	fix duplicate + get raw paste	2018-05-11 16:07:30 +02:00
Terrtia	d7ba5533be	fix duplicate + fix issue #200	2018-05-11 13:25:45 +02:00
Terrtia	8599d7142d	python 3 upgrade instructions	2018-05-09 13:32:25 +02:00
Terrtia	142c1dc9b0	clean	2018-05-09 13:03:46 +02:00
Terrtia	3841c159a0	change leveldb to ardb	2018-05-07 14:50:40 +02:00
Terrtia	c17b9402f7	clean	2018-05-04 14:25:47 +02:00
Terrtia	96a3df42bc	decode with redis connection	2018-05-04 13:53:29 +02:00
Terrtia	ae8f575879	use feeder name	2018-05-03 16:32:25 +02:00
Terrtia	a900d5f08d	add python 2 to 3 bash + fix Duplicate + clean	2018-05-03 16:21:33 +02:00
Terrtia	c52caebe7c	improve Duplicate + tlsh + add time out handler + debug + clean	2018-05-02 17:07:10 +02:00
Terrtia	92977201fd	create json metadata for base64 module	2018-04-27 15:46:41 +02:00
Terrtia	c3428df5dc	add apiKeys module	2018-04-26 14:42:39 +02:00
Terrtia	29c368250e	create new dashbord alert	2018-04-24 17:03:10 +02:00
Terrtia	8738b7cf75	fix track term and avoid duplicate mail address	2018-04-24 16:44:37 +02:00
Terrtia	a7bd01ed9a	Merge branch 'python3' of https://github.com/CIRCL/AIL-framework into python3	2018-04-23 16:22:51 +02:00
Terrtia	4e55f6ee90	fix Onion module	2018-04-23 16:22:23 +02:00
Thirion Aurélien	ef3c335b62	Delete config.cfg.backup	2018-04-23 14:54:23 +02:00
Terrtia	f93fe9aeb2	add bitcoin module	2018-04-23 14:51:53 +02:00
Terrtia	3fc48db903	add Base64 module	2018-04-20 10:48:44 +02:00
Terrtia	41eb250278	fix python 3 compqtibility issues	2018-04-20 10:42:19 +02:00
Terrtia	8571a86344	frontend python 3.5 upgrade	2018-04-17 16:06:32 +02:00
Terrtia	19c90339e0	clean	2018-04-17 09:43:45 +02:00
Terrtia	d0261233aa	fix ModulesInnformationV2	2018-04-17 09:37:58 +02:00
Terrtia	15ef02fe53	fix python3 types issue	2018-04-16 17:00:44 +02:00
Terrtia	9e07a0ebdb	fix python3 types issue	2018-04-16 15:04:22 +02:00
Terrtia	9c82dd90ec	python 3 backend upgrade	2018-04-16 14:50:04 +02:00
Terrtia	3395b16873	fix incoming encoding file issue	2018-04-13 09:17:56 +02:00
Terrtia	347588fdec	python3.5 Mixer and Global module	2018-04-12 17:06:57 +02:00
Terrtia	4386cbbd34	fix travis	2018-04-11 10:48:22 +02:00
Terrtia	14583f702f	create first test	2018-04-11 10:14:33 +02:00
Terrtia	e9b88b0656	Keys module improvement	2018-04-05 11:40:34 +02:00
CD Nguyen	96259c53bc	refactor NotificationHelper code	2018-04-04 09:41:13 +02:00
CD Nguyen	e9e3f17363	support non TSL SMTP servers	2018-03-30 11:35:37 +02:00
Philipp Schmied	29608ab29b	Fixed e-mail notificatins for sets and regexes	2018-02-28 11:31:16 +01:00
Philipp Schmied	fd329aaba9	config.cfg.sample: Merged overwritten changes from upstream	2018-02-27 16:16:57 +01:00
Philipp Schmied	5b1f0b0212	Implemented email notifications (bin: config.cfg additions, email sending via analyzer scripts; var: Changes to add notifications via terms management); terms_management: Fixed click handlers not being added to all tracked terms.	2018-02-27 15:12:02 +01:00
Alexandre Dulaunoy	abdf41fe23	Merge pull request #164 from mokaddem/showDiff ... Show differences between duplicates	2018-02-21 22:25:16 +01:00
Alexandre Dulaunoy	0263b25da8	Merge pull request #163 from mokaddem/iss160 ... Moving more to config	2018-02-21 22:24:50 +01:00
Sami Mokaddem	cbb19f95e4	fix: asciimatics introduced the '_required_height' property in the Label class. Just updating here to mirror the change	2018-02-16 14:35:09 +01:00
Sami Mokaddem	3e117195a2	fix: Added a backoff time if an IOError happens	2018-01-15 17:45:13 +01:00
Sami Mokaddem	9a66a6111b	Fixing no such file	2018-01-15 17:10:03 +01:00
Sami Mokaddem	c39e537cbe	Merge branch 'iss160' into showDiff ... Preemptively preventing merge-conflict with branch iss160 (moving to configuration)	2017-12-12 11:33:22 +01:00
Sami Mokaddem	05b22e8963	typo: added word + deleted trailing spaces	2017-12-12 11:19:37 +01:00
Sami Mokaddem	ef1bd16a72	typo: added word + deleted trailing spaces	2017-12-12 11:16:40 +01:00
Sami Mokaddem	304213cfa4	update: Possibility to change the authorized max line length in the config for the diff of duplicates	2017-12-12 11:00:26 +01:00
Sami Mokaddem	860acbf7cf	fix: Renamed Credentials into Credential sectiont title ... config change: By default, zmq listen to both local and CRF	2017-12-12 09:29:15 +01:00
Sami Mokaddem	9d0d0b4303	update: Moved filtering operation (thresholds, number of matching in the ... categ file) in the configuration file. It permits to better control the flow of pastes. Also set default mixer duplicate filtering to 3 (Do not filter)	2017-12-11 17:28:34 +01:00
Fabien Mathey	3105192312	Allow for multiple entries like 1 2 3 4 5 ... Should replace the need to type 1<Enter>2<Enter> etc. and allows for 1 2 3 4 5<Enter>	2017-11-28 15:15:12 +01:00
Alexandre Dulaunoy	71922f9def	add: Python script to test if the ZMQ feed works as expected	2017-11-28 09:46:06 +01:00
Sami Mokaddem	83e082e62a	update: removed useless comments	2017-11-24 08:57:41 +01:00
Sami Mokaddem	a0d07b1098	feature: try to send data to MISP if keys are provided, else do nothing	2017-11-23 13:47:02 +01:00
Sami Mokaddem	2a967c4d92	update/feature: Max number of duplicate push to MISP + duplicate are pushed as attachment	2017-11-23 07:13:44 +01:00
Sami Mokaddem	805171a8a5	Added a seemingly working version of ailleak to misp	2017-11-20 14:57:25 +01:00
Sami Mokaddem	e0b188c069	Better support of python3 in Helper.py	2017-11-20 14:55:36 +01:00
Sami Mokaddem	1e76e10c01	Support of add_object to a MISP instance	2017-11-20 12:12:53 +01:00
Sami Mokaddem	6f0227a78a	Starting support of python3	2017-11-20 12:12:06 +01:00
Sami Mokaddem	ee7759be91	Updated alertHandler and ailleakObject	2017-11-16 11:18:13 +01:00
Sami Mokaddem	5c7c07fbea	Merge branch 'launcher_update' into misp_obj	2017-11-16 10:09:05 +01:00
Sami Mokaddem	45a9897020	Added draft support of MISP ail-leak object	2017-11-16 09:52:37 +01:00
Sami Mokaddem	652b00f4b2	Renamed BrowseWarningPaste into alertHandler	2017-11-15 16:15:43 +01:00
Sami Mokaddem	ec7d2d49b2	Added seconds args in import_dir	2017-11-15 16:03:42 +01:00
Sami Mokaddem	13f1c676d6	Added '_AIL' to the end of screen session name to better check an kills sessions. Also, gracefully close redis server + Check 1 time if redis-servers are running.	2017-09-26 16:17:06 +02:00
=	a8dc9231c6	Typo2..	2017-08-24 16:43:42 +02:00
=	b2d4dd4a1f	Typo	2017-08-24 16:43:05 +02:00
=	74ff401e1d	Fixed bug: No CR between indexes in all_index.txt file	2017-08-24 16:41:30 +02:00
Alexandre Dulaunoy	bb128a5e94	Merge pull request #138 from mokaddem/import_dir_update ... Improvement of import_dir	2017-08-24 09:40:52 +02:00
=	429cc96d2a	Improvement of import_dir.py. Added feed name parameter, send the correct path (without potential ../) and gzipEncode it if needed.	2017-08-23 15:10:26 +02:00
=	e7c66d1aac	Updated individual launch scripts	2017-08-23 12:28:12 +02:00
=	d632335760	Automatically start redis_levelDB with the correct date	2017-08-23 10:16:56 +02:00
=	64ec44085d	Dynamically choose correct port for LevelDB in script	2017-08-22 17:52:15 +02:00
=	66d60a1f01	Added LevelDB Curve for 2017 (3017, used for backward compatibility)	2017-08-21 13:54:44 +02:00
=	92adea38d0	Semi-automated directory creation with date as name in LAUNCH and installing_deps	2017-08-21 13:48:02 +02:00
=	bc35747f00	Updated KillAll in LAUNCH	2017-08-21 13:19:26 +02:00
Mokaddem	c3632c7d86	Added operation mode 3 for Mixer (do not filter) and added possibility to recurse through dir for import_dir	2017-08-03 00:34:53 +02:00
Mokaddem	cbb2909156	Added comments	2017-07-20 10:24:48 +02:00
Mokaddem	05417f405c	Finished basic link web interface + solved some bugs	2017-07-20 10:04:30 +02:00
Mokaddem	ef741f85c0	Added similarity checks + started link webInterface	2017-07-19 11:52:06 +02:00
Mokaddem	882ab88b00	Draft module + web link	2017-07-18 16:57:15 +02:00
Mokaddem	9558aafe67	Prevent the loss of the 1 message in import_dir.py	2017-07-17 16:14:27 +02:00
Mokaddem	4d59871f00	Merged with previous changes for PR	2017-07-17 13:48:16 +02:00
Alexandre Dulaunoy	4986352099	Merge pull request #120 from PaulSec/master ... Added Python Phonenumbers module to parse numbers and retrieves stats…	2017-06-08 07:05:52 +02:00
Alexandre Dulaunoy	446a97c47a	Merge pull request #115 from mokaddem/regexInTerms ... New features for terms frequency	2017-06-08 07:04:17 +02:00
Mokaddem	3a4dcd691d	Improved description of modules inside the scripts	2017-05-09 11:13:16 +02:00
Paul A	747fff1df1	Added Python Phonenumbers module to parse numbers and retrieves stats from countrycode	2017-05-03 14:25:18 +02:00
Mokaddem	3ea0924120	Improved phone regex to handle end with 4 digits and presence of parenthesis.	2017-04-26 12:41:23 +02:00
Mokaddem	b60e164380	Added support of onion in browseWarningPaste	2017-04-26 11:47:02 +02:00
Mokaddem	318a557c56	Added possibility to keep a backup of the old config + check if config is valid before starting scripts.	2017-04-25 09:32:05 +02:00
Mokaddem	72c1d903eb	Merge branch 'newIndexer' into auto-conf	2017-04-25 08:14:31 +02:00
Mokaddem	eccc8fc88b	modified check_size timeout + utilisation of shutil instead of os.system()	2017-04-19 16:37:04 +02:00
Mokaddem	da081bae42	Merge branch 'regexInTerms' into production	2017-04-18 16:59:40 +02:00
Mokaddem	2d907ae784	Merge branch 'newIndexer' into auto-conf	2017-04-18 16:20:00 +02:00
Mokaddem	b0f1cf84b7	remplaced usage of global variable by a function param	2017-04-18 16:02:22 +02:00
Mokaddem	6e8509a0ba	Removed wierd line	2017-04-18 15:57:07 +02:00
Mokaddem	c8baabd882	Updated TermsFrequency related modules + Fixed bugs (encoding, behaviors, ...)	2017-04-18 15:28:21 +02:00
Mokaddem	2da4c572c7	New feature in Terms: regex and set of words support (draft)	2017-03-28 17:42:44 +02:00
Mokaddem	6e80eb0954	Typo	2017-03-28 09:55:59 +02:00
Mokaddem	c79d4d65c8	Keep original section order while updating	2017-03-28 09:54:24 +02:00
Mokaddem	e225090e07	Added script that checks configuration and may auto-fix it + added missing field in config.cfg.sample	2017-03-15 16:36:51 +01:00
Mokaddem	be93af6f5a	Merge branch 'terms-improvements'	2017-03-15 14:55:44 +01:00
Mokaddem	3261eca836	Fixed all_index newline bug + added print of what's happening	2017-03-15 14:29:49 +01:00
Mokaddem	5b7d047622	Changed os.system by os.mkdir + fixed bug during rotation	2017-03-15 14:05:13 +01:00
Mokaddem	adc37ab50e	Indexer max size now in config	2017-03-15 12:14:41 +01:00
Mokaddem	4005b72180	Added backward support + renamed index with timestamp	2017-03-15 11:51:35 +01:00
Mokaddem	248469d61e	Indexer now create an index_dir when it became too large (search in index not updated yet)	2017-03-14 10:37:31 +01:00
Alexandre Dulaunoy	1540b719ed	Merge pull request #108 from mokaddem/newModuleInformation ... New version of ModuleInformation	2017-03-14 08:29:09 +01:00
Mokaddem	3c9cdac39a	Merge branch 'terms-improvements' into production	2017-02-28 16:51:39 +01:00
Mokaddem	dadc872f9f	Merge branch 'newModuleInformation' into production	2017-02-28 16:51:08 +01:00
Mokaddem	a2849c1bcd	Added a catch for psutils if process get stopped by user	2017-02-28 16:13:01 +01:00
Mokaddem	4071bf2e91	fixed a bug in manage_term (usage of loop.index0) + fixed bug in top_set where graph was not displaying the correct per_paste value	2017-02-28 15:01:48 +01:00
Mokaddem	a1c47d7a13	Merge branch 'production' into terms-improvements	2017-02-28 11:15:56 +01:00
Mokaddem	fd4feb374f	Merge branch 'newModuleInformation' into production	2017-02-28 11:12:06 +01:00
Mokaddem	75b4037519	Fixed bug - If we move the cursor from a line that has dissapear	2017-02-28 11:11:17 +01:00
Mokaddem	d6e02591af	Merge branch 'newModuleInformation' into production	2017-02-28 10:23:45 +01:00
Mokaddem	975c1a37d9	ModuleInformation create the all_modules files if missing	2017-02-28 10:23:22 +01:00
Mokaddem	ccdf256383	Added possibility to show or not in dashboard text that is logged	2017-02-28 10:21:27 +01:00
Mokaddem	39104dea5e	Merge branch 'bugfix_categ' into production	2017-02-28 09:56:25 +01:00
Mokaddem	26eb5fdff0	Fixed bug closing parenthesis in categ + changed behavior of sending to browseWarningPaste in Mail	2017-02-28 09:23:32 +01:00
Mokaddem	4a0dac1fae	Merge branch 'production' into terms-improvements	2017-02-15 16:31:22 +01:00
Mokaddem	a696ccf83b	Merge branch 'newModuleInformation' into production	2017-02-15 16:30:44 +01:00
Mokaddem	85a6f6fa63	Fixed a bug (str+float)	2017-02-15 16:30:31 +01:00
Mokaddem	25adae0104	Added possibility to select in top_terms the data per_paste or not (need module testing)	2017-02-15 16:29:02 +01:00
Mokaddem	e98e92b0d4	Merge branch 'fix-linkPaste-debug' into production	2017-02-15 13:43:03 +01:00
Mokaddem	c552aced42	Merge branch 'newModuleInformation' into production	2017-02-15 13:42:46 +01:00
Mokaddem	1b0d9d9507	Fixed str and int concat + Added sorting by queue name in queue not running	2017-02-15 13:42:21 +01:00
Mokaddem	74e0bc7acb	Merge branch 'newModuleInformation' into production	2017-02-15 11:22:50 +01:00
Mokaddem	7fc9ba18d7	Added support of ENTER for navigation	2017-02-15 11:01:11 +01:00
Mokaddem	b84194542d	Moved logging into a function	2017-02-15 10:57:41 +01:00
Mokaddem	0fce3f5d65	Renamed variables + added comments	2017-02-15 10:37:59 +01:00
Mokaddem	9ca3e79e70	Removed useless comments + commented	2017-02-15 10:19:03 +01:00
Mokaddem	9199ea3f0e	Added comments	2017-02-15 10:04:51 +01:00
Mokaddem	ff1df07b5d	Added comments and moved portion of code for slightly better visibility	2017-02-15 09:43:18 +01:00
Mokaddem	b272b1b07b	Added a way to quit application with 'q' more or less properly	2017-02-15 09:35:03 +01:00
Mokaddem	0254c51164	Added end of paste marker + handling screen resize	2017-02-15 08:26:48 +01:00
Mokaddem	0b9609e4a9	Merge branch 'newModuleInformation' into production	2017-02-14 17:22:06 +01:00
Mokaddem	444f6071b7	Fixed screen bug by adding an input before starting asciimatics + improved path localisation in the logs from modules	2017-02-14 17:21:52 +01:00
Mokaddem	b04bac4731	Merge branch 'newModuleInformation' into production	2017-02-14 16:08:42 +01:00
Mokaddem	d520c45a15	Added support of display of the last paste processed by a module	2017-02-14 16:05:59 +01:00
Mokaddem	d3b4409156	Added path for debug in SQLInjection	2017-02-14 11:23:34 +01:00
Mokaddem	135a1457e0	Merge branch 'fix-linkPaste-debug' into production	2017-02-14 11:12:22 +01:00
Mokaddem	d4da3a316f	Added path of the paste in the log of Categ.py	2017-02-14 10:59:47 +01:00
Mokaddem	303575b2a1	Merge remote-tracking branch 'upstream/master' into production	2017-02-14 09:31:18 +01:00
Alexandre Dulaunoy	2f4e5b9016	Merge branch 'master' of github.com:CIRCL/AIL-framework	2017-01-26 07:16:44 +00:00
Alexandre Dulaunoy	cf903cc212	(partially fix #90 too) using a simple Alarm (SIGNAL) when exec-timeout ... Introducing a timer (in this case 60 seconds) to ensure that the execution time of the analysis takes less than 60 seconds. This is a simple and standard POSIX signal handler. If the timeout is reached, the module will process the next one. This approach fixes the specific issues we have currently with some inputs where the sentiment analysis takes too much time. This fix should be improved and be more generic: - Introducing statistics of content which timeouts. - Keeping a list/queue to further process those files using a different analysis approach. Maybe a set of "dirty" processes to handle the edge cases and to not impact the overall processing and analysis. - Make the timer configurable per module (at least for this one).	2017-01-26 07:11:18 +00:00
Mokaddem	c422db6e6d	Merge remote-tracking branch 'upstream/master' into production	2017-01-17 16:38:48 +01:00
mokaddem	b3a3616b0b	Merge pull request #102 from ptitdoc/bugfix1 ... Bug fixes in Workers and improvement of pystemon-feeder	2017-01-17 16:35:32 +01:00
Mokaddem	ea423fde0f	Added requirement, better behaviors and confirmation box	2017-01-17 15:58:16 +01:00
Mokaddem	692f13d18b	Improved module + Added support of click and keyboard navigation for killing and starting modules	2017-01-17 14:31:06 +01:00
Mokaddem	3b8149d840	Added action button and support of logs	2017-01-16 17:08:48 +01:00
Mokaddem	0ecfaf00f5	Added color in front of queue related to its state	2017-01-16 15:30:33 +01:00
Mokaddem	af3241c008	Overwritted listbox update function to add formating	2017-01-16 14:41:02 +01:00
Mokaddem	30f303cb99	Added new ModuleInformation	2017-01-16 12:18:23 +01:00
Olivier MEDOC	16044d4d36	pystemon-feeder: add the option bind in configuration file instead of address ... This is essentially because multiple feeds can be used in the address configuration variable.	2017-01-13 14:58:18 +01:00
Alexandre Dulaunoy	3b101ea8f5	(partially) Fix #91 using a simple Alarm (SIGNAL) when exec-timeout ... Introducing a timer (in this case 5 seconds) to ensure that the execution time of the tokenizer takes less than 5 seconds. This is a simple and standard POSIX signal handler. This approach fixes the specific issues we have currently with some inputs where the tokenization takes too much time. This fix should be improved and be more generic: - Introducing statistics of content which timeouts. - Keeping a list/queue to further process those files using a different tokenizer approach. Maybe a set of "dirty" processes to handle the edge cases and to not impact the overall processing and analysis. - Make the timer configurable per module (at least for this one).	2017-01-12 07:32:55 +00:00
Olivier MEDOC	9f9c265cb0	SentimentAnalysis: use lexicon file path from the ail configuration file	2017-01-11 11:07:08 +01:00
ptitdoc	c0fac82031	Add pystemon path in configuration sample	2017-01-10 18:33:46 +01:00
Olivier MEDOC	83db40104c	import pystemon-feeder configuration from the central configuration file	2017-01-10 18:26:36 +01:00
Olivier MEDOC	3dc014dad9	fix invalid variable propagation	2017-01-10 18:26:24 +01:00
Olivier MEDOC	68ca20db8f	fix to use the new pycountry API	2017-01-10 18:26:17 +01:00
Alexandre Dulaunoy	1950a2dc0e	Merge pull request #99 from mokaddem/more-feed ... Usage of sha1 to filter paste content in Mixer.py	2017-01-09 17:06:37 +01:00
Mokaddem	500608d0a1	Usage of sha1 to filter paste content in Mixer.py	2017-01-09 17:01:10 +01:00
Alexandre Dulaunoy	f5adb65cb3	Merge pull request #98 from mokaddem/more-feed ... Config 2017	2017-01-09 15:39:53 +01:00
Mokaddem	409c5b8b42	Added 2017	2017-01-09 15:36:50 +01:00
Alexandre Dulaunoy	a96ba40d94	Merge pull request #96 from mokaddem/more-feed ... Handling of multiple feeders	2017-01-09 15:29:28 +01:00
Mokaddem	0e39f516a9	Added sleep in helper for multiple listening interfaces (reduce useless work), Added a simple way to pre-process feed before sending it to the global feed.	2017-01-09 14:12:26 +01:00
Mokaddem	3a298f0c7c	Commented Mixer.py for further multiple feeds support.	2016-12-26 16:23:18 +01:00
Mokaddem	1c2169d3bc	Improved interface for multiple feeds and refresh_script.	2016-12-26 16:16:44 +01:00
Mokaddem	97292e0899	Updated web interface to handle new mixer module. and fixed one dependency bug.	2016-12-23 15:44:46 +01:00
Mokaddem	e70b9cd15c	Added basic mixer with confirugable behavior. It handles muliple feeders and performs some basic stats on them.	2016-12-23 10:31:26 +01:00
Alexandre Dulaunoy	ddfe3cf66a	Merge pull request #95 from mokaddem/module_handling ... Module handling	2016-12-22 22:18:31 +01:00
Alexandre Dulaunoy	3ce850e093	Merge pull request #94 from mokaddem/website-refacto ... Website refacto, support and fixes	2016-12-22 22:17:58 +01:00
Mokaddem	9758579753	Merge branch 'module_handling' into production	2016-12-22 10:24:30 +01:00
Mokaddem	7438f16c63	Added new config	2016-12-22 10:06:35 +01:00
Mokaddem	73938cf41b	Prepared moduleInfo to handle module click with curses (will come back later).	2016-12-22 08:45:53 +01:00
Mokaddem	d37b243e1d	Added legend + added more info if restarting disabled	2016-12-09 15:13:47 +01:00
Mokaddem	d3ed1bbf40	Merge branch 'website-improvement' into production	2016-12-08 10:05:55 +01:00
Mokaddem	73d4f9e082	Webstats should correctly updates top_progression_zset (Not fully tested because not enough data. Will be tested latter)	2016-12-08 10:05:07 +01:00
Mokaddem	570324060e	terms top_sets correctly supports blacklisted terms	2016-12-08 09:13:31 +01:00
Alexandre Dulaunoy	412b80c44d	Merge pull request #88 from mokaddem/Issue-Feeding-AIL ... Pystemon tuto	2016-11-26 12:45:28 +01:00
Mokaddem	d831a79614	Do not try to restart not running queue if autokill is not set	2016-11-25 16:00:46 +01:00
Mokaddem	4bf45eea36	ModuleInformation sort by name as well as elapsed time	2016-11-25 14:44:27 +01:00
Mokaddem	866be4e884	Added row coloring in ModuleInformation	2016-11-25 14:12:09 +01:00
Mokaddem	a3255d168c	ModuleInfo correctly handle CurveManageTopSets, ... Changed queue coloring in dashboard, ModuleInformation now have a history of executed command and better handle module killing.	2016-11-25 11:54:16 +01:00
Mokaddem	224fbc8084	Better handle stuck modules. Differentiate between not running and no info and tries to restart stuck ones.	2016-11-24 16:58:32 +01:00
Mokaddem	5d269ea1ee	Added date in Duplicate for better distinguish duplicate. Also, added a timeout for the dataTable in duplicate inside the modal.	2016-11-24 15:05:29 +01:00
Mokaddem	ac254e0e63	Added support of browsing concerned paste in dashboard for all modules + Show in the page what was the concerned module	2016-10-27 11:50:24 +02:00
Mokaddem	1826b170ec	Added support of local paste view in dashboard for Mails event only	2016-10-27 11:27:26 +02:00
Mokaddem	5a4ea734a2	Fixed path issues in doc + added dependencies related to ssdeep	2016-10-14 14:26:33 +02:00
Mokaddem	4eafca2936	Added pystemon instruction section	2016-10-13 17:13:08 +02:00
Alexandre Dulaunoy	eb14005fcf	Fix: as JSON decode needs a string, the empty list is a string.	2016-08-29 12:08:12 +00:00
Alexandre Dulaunoy	3b6e3e156e	Merge branch 'master' of github.com:CIRCL/AIL-framework	2016-08-26 15:57:45 +00:00
Mokaddem	0c760d763b	Added support of PID in ModuleInformation and index.html	2016-08-26 15:27:37 +02:00
Alexandre Dulaunoy	758f2c5334	Merge branch 'master' of github.com:CIRCL/AIL-framework	2016-08-26 08:30:11 +00:00
Mokaddem	494980c48e	Added comment in moduleInformation	2016-08-26 10:07:06 +02:00
Mokaddem	8b33b4e7ba	Added not running module info + swapped order of module launch	2016-08-26 09:21:02 +02:00
Mokaddem	813cc7031a	Changed lots of code into zrank	2016-08-25 17:48:12 +02:00
Mokaddem	aaac798db9	updated individual launch script	2016-08-25 16:51:57 +02:00
Mokaddem	78c991539e	Added number of same module running in ModuleInformation	2016-08-25 16:07:47 +02:00
Alexandre Dulaunoy	1ab08d4a05	Merge branch 'master' of github.com:CIRCL/AIL-framework	2016-08-25 12:33:31 +00:00
Mokaddem	f9bf00f022	Removed autokill feature if not specified	2016-08-25 11:09:31 +02:00
Mokaddem	2f136595fb	Removed autokill feature if not specified	2016-08-25 11:05:18 +02:00
Mokaddem	9146feab88	Fixed bug in WebStats not deleting correct key + fixed typo in CurveSetManager preventing connecting to redis + modified display in moduleInfo	2016-08-25 10:43:04 +02:00
Mokaddem	3313064e51	removed useless import.	2016-08-24 17:32:50 +02:00
Mokaddem	64bf481480	Added moduleInfo + Renamed modules	2016-08-24 17:28:39 +02:00
Mokaddem	8fb552b7da	ModuleInfo support autokill and auto restart after logging.	2016-08-24 16:52:01 +02:00
Mokaddem	3e88d0911d	ModuleInfo now support logs.	2016-08-24 15:53:00 +02:00
Mokaddem	ff6dcaf456	Fixed typo in creditcard + fixed bug in ModuleInformation (relative to date).	2016-08-24 15:35:23 +02:00
Mokaddem	deeebec2f5	Added new module 'ModuleInformation' which gives real time information on running module	2016-08-24 15:21:41 +02:00
Mokaddem	07856f3119	Added script module-dataflow-graph + fixed bug in module.cfg preventing the credential module to receive data.	2016-08-24 11:32:48 +02:00
Alexandre Dulaunoy	bde5f47ed0	Merge branch 'master' of github.com:CIRCL/AIL-framework ... Conflicts: bin/launch_scripts.sh	2016-08-23 14:20:39 +00:00
Mokaddem	42b3966499	Added new module in individual launch scripts	2016-08-23 16:01:11 +02:00
Mokaddem	697812dbff	Added config.sample	2016-08-23 09:59:39 +02:00
Mokaddem	f9b27e4013	Fixed bug in curve preventing to track concerned_pastes. Now all tracked words are put in lower case.	2016-08-23 09:50:03 +02:00
Mokaddem	047331df3e	Added save of paste_baseName in set for searches. This is done in ModuleStats to not surcharge the Global module.	2016-08-23 08:52:54 +02:00
Mokaddem	20fd191c20	Suppressed prints	2016-08-22 21:29:59 +02:00
Mokaddem	2fd2c8af39	Removed commented code	2016-08-22 20:59:56 +02:00
Mokaddem	003c72bd7c	Fixed bug with redis where the inserted key was too long, causing a crash + fixed bug taking min instead of max in terms-top-set	2016-08-22 16:01:42 +02:00
Mokaddem	2466f355ab	Fixed bug in manage-terms-top-set + added auto sorting in important-paste-by-module	2016-08-22 14:54:37 +02:00
Mokaddem	d3c0fc5f94	Added terms trending module and web interface: ... - Curve module has been rewritten - Added new module curve_manage_top_set - Added all web-pages	2016-08-22 13:35:49 +02:00