1612 commits

Author	SHA1	Message	Date
terrtia	9c6619aed3	fix: [crawler] log UNKNOWN timeout	2024-01-26 15:31:32 +01:00
Steve Clement	9cfd230661	chg: [LAUNCH] make sure reload works	2024-01-18 11:53:13 +01:00
Steve Clement	1a02b6d768	chg: [LAUNCH] Addded a "restart" option, killAll + launchAuto	2024-01-18 11:42:37 +01:00
terrtia	edf0c4c454	chg: [message] UI translate message object	2024-01-16 14:38:29 +01:00
terrtia	f586baa0c5	fix: [Language] fix language source	2024-01-16 12:04:39 +01:00
terrtia	1c46bb4296	chg: [Language] replace pycld3 by gcld3 + clean text before language detection	2024-01-15 14:17:15 +01:00
terrtia	d6d67f6a4c	chg: [hosts] filter onion	2024-01-09 14:31:55 +01:00
terrtia	bdaa4c51c9	fix: [hosts] fix number of hosts extracted	2024-01-09 12:15:40 +01:00
terrtia	5094b2dcbb	fix: [DomClassifier] improve perf	2024-01-09 11:38:54 +01:00
terrtia	7263a9777c	fix: [DomClassifier] improve perf	2024-01-09 11:26:05 +01:00
terrtia	5c25ec0fea	fix: [DomClassifier] improve perf	2024-01-09 11:24:54 +01:00
terrtia	f851cc9f42	fix: [queue] save last timout in cache	2024-01-09 11:19:01 +01:00
terrtia	bd2ca4b319	fix: [crawler] fix api create_task	2024-01-09 09:47:49 +01:00
terrtia	a14c0484af	fix: [tags] debug	2024-01-08 15:34:32 +01:00
terrtia	be4feb7799	fix: [tags] debug	2024-01-08 15:28:31 +01:00
terrtia	259f29c10c	fix: [tags] debug	2024-01-08 15:27:08 +01:00
terrtia	bfc018f929	fix: [tags] debug	2024-01-08 15:21:37 +01:00
terrtia	f95d32d6dc	fix: [tags] debug	2024-01-08 15:02:39 +01:00
terrtia	4b21cc2939	fix: [tags] debug	2024-01-08 15:01:40 +01:00
terrtia	07c51e111f	fix: [tags] debug	2024-01-08 14:59:58 +01:00
terrtia	efb8b2d0d3	fix: [tags] debug	2024-01-08 14:58:18 +01:00
terrtia	0abc3fee0e	fix: [tags] debug	2024-01-08 14:56:20 +01:00
terrtia	8bf67cf3b6	fix: [tags] remove invalid tags	2024-01-08 14:51:40 +01:00
terrtia	d7c8262653	fix: [keys module] fix tags	2024-01-08 14:24:51 +01:00
terrtia	9716351196	Merge branch 'master' into dev	2024-01-08 14:05:34 +01:00
terrtia	0af5ea9d48	chg: [queues] timeout obj after 2 days	2024-01-08 13:50:56 +01:00
terrtia	c05f4d7833	chg: [chats] get user message ids by chat	2024-01-02 17:15:45 +01:00
Markus Lassfolk	81f9a860d4	Fix IndexError in get_last_tag_from_remote function ... This commit adds a check to ensure that the output from the subprocess command in the get_last_tag_from_remote function has a sufficient number of lines before attempting to access specific indices. This change prevents the IndexError that occurred when the git command's output was shorter than expected.	2024-01-01 21:54:29 +01:00
terrtia	70bb6757f8	chg: [correlation] UI chats filters + correation user-account/username	2023-12-13 11:51:53 +01:00
terrtia	9221e532c4	fix: [crawlers] fix task start	2023-12-12 11:32:33 +01:00
terrtia	7e9ea48c81	fix: [crawler] debug	2023-12-12 11:20:21 +01:00
terrtia	cdfc9f64e5	fix: [crawler] debug	2023-12-12 11:13:58 +01:00
terrtia	847d004c13	fix: [crawler] debug	2023-12-12 11:05:23 +01:00
terrtia	c20c41c50f	fix: [libinjection] memory leak, disable module	2023-12-12 10:30:40 +01:00
terrtia	4529a76d13	fix: [zmq importer] fix object source name	2023-12-12 10:14:59 +01:00
terrtia	235539ea42	fix: [crawler] fix capture start time	2023-12-11 09:30:09 +01:00
terrtia	5fc9b1403f	chg: [chats] add pagination	2023-12-11 00:46:15 +01:00
terrtia	943a873124	chg: [tags] searech messages tags by daterange	2023-12-09 16:50:43 +01:00
terrtia	73185f19fd	chg: [categ] messages, bypass categ module + fix correlation	2023-12-08 15:40:05 +01:00
terrtia	5b808ed416	fix: [translate] fix exception	2023-12-08 14:38:55 +01:00
terrtia	1c52c187ad	fix: [api] fix add crawler capture return	2023-12-08 10:37:58 +01:00
terrtia	cea96863ba	fix: [language] libretranslate unreachable	2023-12-07 14:59:40 +01:00
terrtia	3e591d95bc	fix: [retro_hunt] fix daterange	2023-12-07 14:40:51 +01:00
terrtia	a382b572c6	chg: [crawler] push onion discovery capture_uuid to another AIL	2023-12-07 11:28:35 +01:00
terrtia	38ce17bc8a	chg: [chats] add chats participants + user-accounts basic template	2023-12-06 16:26:26 +01:00
terrtia	bef4e69a68	chg: [chats] translate messages on demand	2023-12-04 15:47:58 +01:00
terrtia	941838ab76	chg: [chats] add discord threads, Forum channel	2023-12-04 10:26:02 +01:00
terrtia	93ef541862	chg: [chats] add messages threads	2023-11-29 16:28:25 +01:00
terrtia	f766cbebda	chg: [chats] add message file-name object + str emoticon reactions	2023-11-27 16:25:09 +01:00
terrtia	fa4f1269cd	chg: [chat users] add users icons	2023-11-24 16:35:45 +01:00
terrtia	2b8e9b43f3	chg: [chats] factorise heatmap + chat icon	2023-11-24 15:05:19 +01:00
terrtia	9fbd3f4bb6	chg: [chats] add message and blur blocks	2023-11-16 14:50:03 +01:00
terrtia	2ea5f82760	chg: [queues] add new image queue + add exif module	2023-11-15 15:45:25 +01:00
terrtia	4142ad9884	chg: [chats] add image object + show message image	2023-11-15 14:12:50 +01:00
terrtia	7bf0fe8992	chg: [chats] add heatmap nb week messages by hour	2023-11-13 14:10:24 +01:00
terrtia	54c57ea35b	chg: [chats] message object template	2023-11-08 15:46:05 +01:00
terrtia	e0f70c5072	fix: [investigations] delete obj	2023-11-08 13:35:13 +01:00
terrtia	ce989adbd3	fix: [chats] fix username meta	2023-11-08 13:07:00 +01:00
terrtia	207a6524d7	fix: [languages] fix language module	2023-11-08 11:59:06 +01:00
terrtia	6c77ca5136	fix: [chats] fix chat username	2023-11-08 11:25:30 +01:00
terrtia	5068287c76	Merge branch 'master' into dev	2023-11-08 10:42:02 +01:00
terrtia	acef57bb36	fix: [tags] fix galaxies synonyms	2023-11-08 10:41:32 +01:00
terrtia	e7f060c23d	chg: [messages] refactor get_messages_meta + add basic message template	2023-11-08 10:31:51 +01:00
terrtia	4cc9608a3f	chg: [chats explorer] show chats/subchannels creation date	2023-11-07 15:24:40 +01:00
terrtia	3fb1d0ad74	fix: [chats] fix messages replies	2023-11-07 11:24:24 +01:00
terrtia	b1d5399607	chg: [chats] add UI shortcut + networks list + show chats/subchannels info	2023-11-06 16:38:31 +01:00
terrtia	789210bcba	chg: [chats] improve UI + fix importer	2023-11-06 14:08:23 +01:00
terrtia	9125119764	chg: [chats] add chats explorer v0	2023-11-02 16:28:33 +01:00
terrtia	c5cef5fd00	chg: [core] merge master + fix object subtype correlation stats	2023-10-12 13:53:00 +02:00
terrtia	6978764b02	fix: [module] fix module obj type: language + mail	2023-10-11 14:53:12 +02:00
terrtia	623ba455ff	fix: [queues] fix ended duplicate + sync queue	2023-10-11 14:31:13 +02:00
terrtia	676b0f84ef	chg: [module + queues] track + rename object global ID by module	2023-10-11 12:06:01 +02:00
terrtia	eae57fb813	chg: [importers obj_type] importers queues: add feeder source + object global ID	2023-10-05 16:24:28 +02:00
terrtia	daf9f6fb5d	fix: [chats] message css + reply ID	2023-10-04 14:40:13 +02:00
terrtia	fee3332edb	fix: [tracker] delete yara rule, fix filter by object type	2023-09-29 15:43:37 +02:00
terrtia	c19b1f34e3	chg: [ail queues] merge	2023-09-08 10:52:55 +02:00
terrtia	bb3dad2873	chg: [objs processed] xxhash messages	2023-09-07 10:38:03 +02:00
Jean-Louis Huynen	68c17c3fbc	chg: [crawlers] submit cookies to the crawler task API	2023-08-31 16:13:20 +02:00
Jean-Louis Huynen	ed0423118e	chg: [crawlers] submit a single cookie to the crawler task API	2023-08-31 15:42:44 +02:00
Terrtia	7c73f0944a	fix: [items source] filter invalid item sources	2023-08-29 14:03:26 +02:00
Terrtia	099253f854	fix: [json importer] fix empty source name	2023-08-29 13:50:16 +02:00
Terrtia	24969610cc	fix: [items source] fix empty sources list	2023-08-29 11:59:39 +02:00
Terrtia	b32f110285	chg: [chat + user-account] correlations + usernames timeline	2023-08-28 16:29:38 +02:00
Terrtia	546d6538fd	chg: [mail exporter] add obj content extract for each regex match	2023-08-24 14:37:50 +02:00
Terrtia	c01b806ae3	chg: [mail exporter] add obj content extract for each yara rule match	2023-08-24 11:11:57 +02:00
Terrtia	843b2d3134	fix: correlations	2023-08-23 16:13:20 +02:00
Terrtia	4e3784922c	fix: typo	2023-08-23 11:47:39 +02:00
Terrtia	2145eb7b8a	fix: [title] fix None title	2023-08-23 11:46:37 +02:00
Terrtia	f44c5509da	chg: [titles] add yara tracker on title + tags domains if unsafe title tags	2023-08-23 11:16:22 +02:00
Terrtia	045aab6f34	fix: [module pasties] fix module name	2023-08-21 15:52:33 +02:00
Terrtia	0cb7431e10	chg: [modules] crawl pasties domains	2023-08-21 15:49:32 +02:00
Terrtia	3c1813ba02	chg: [core] add telegram importer + Chat object + message Object + add timeline engine	2023-08-18 11:05:21 +02:00
Terrtia	f05c7b6a93	fix: [module extractor] fix invalid yara offset	2023-08-08 11:27:57 +02:00
Terrtia	4dc5527c1a	fix: [module extractor] fix invalid yara offset	2023-08-08 11:26:16 +02:00
Terrtia	529a24c191	chg: [module extrator] add debug	2023-08-08 10:40:44 +02:00
Terrtia	bd7aa979bd	chg: [module extrator] add debug	2023-08-08 10:36:58 +02:00
Terrtia	2691000d0c	chg: [telegram fedeer] use meta of the new feeder	2023-08-02 15:49:12 +02:00
Terrtia	14a76a91d9	fix: [tags ui] fix galaxy, get number of tags enabled + add toolip helper	2023-08-01 11:07:06 +02:00
Terrtia	1aa0bd8a0e	fix: [settings] fix edit user	2023-07-31 16:25:28 +02:00
Terrtia	64252ddf75	fix: [Tracker] edit tracker filters	2023-07-31 16:00:31 +02:00
Terrtia	4eb5b4c4b9	fix: [MailExporter] improve logs + SMTP over SSL: add an option to validate self signed CA ... This exporter use STARTTLS by defaults. If the SMTP server does not support the STARTTLS extension, the client will switch to SSL instead.	2023-07-28 11:10:21 +02:00
Terrtia	68dffcd26b	chg: [api crawler] fix response + add cookiejar, proxy and frequency parameters	2023-07-25 15:57:11 +02:00
Terrtia	fe2769308b	fix: [crawler] fix down domain + domain redirection history	2023-07-18 14:30:00 +02:00
Terrtia	77e8cc6c02	Merge branch 'master' of github.com:ail-project/ail-framework	2023-07-17 15:47:32 +02:00
Terrtia	a9485928db	chg: [HHHash] add HHHash object and correlation https://www.foo.be/2023/07/HTTP-Headers-Hashing_HHHash	2023-07-17 15:47:17 +02:00
fukusuket	22dff19004	fix: [domain search] returns an empty string when searched with invalid characters	2023-07-14 18:28:34 +09:00
Terrtia	780b4f9f9c	fix: [background update] fix logger	2023-07-12 14:37:59 +02:00
Terrtia	73bfe614df	chg: [updater] refactor background updater + add v5.2 update	2023-07-12 11:36:47 +02:00
Terrtia	28c647d370	chg: [crawler har] compress HAR	2023-07-10 15:56:34 +02:00
Terrtia	c719990125	fix: [crawler] add timeout to Unknown captures	2023-07-10 11:23:44 +02:00
Thirion Aurélien	8f0e7f1434	Merge pull request #174 from fukusuket/fix-500-erro-when-invalid-lacus-url ... fix: [crawler] add exception handing for `ping_lacus`	2023-07-10 10:05:20 +02:00
fukusuket	fcd6a98fb0	fix: [crawler] added existence check for screenshot set	2023-07-09 12:50:13 +09:00
fukusuket	e35924ec22	fix: [crawler] add exception handing for ping_lacus	2023-07-08 12:11:25 +09:00
Terrtia	482fc21b5e	chg: [correlation] correlation graph, add an option to hide an object/node by pressing H + reset correlation graph	2023-07-07 16:29:32 +02:00
Terrtia	450ebdd789	chg: [etag] add new etag object	2023-07-06 11:26:32 +02:00
Terrtia	35f678245f	fix: [decoded] fix download file	2023-06-30 16:22:30 +02:00
Terrtia	0fa0984ec4	fix: [updater] fix db checker	2023-06-30 16:19:38 +02:00
Terrtia	fa8b4a67d9	fix: [correlation tags] fix tag all objects	2023-06-29 15:02:57 +02:00
Terrtia	279ec00990	fix: [correlation card decoded meta] mimetype + size	2023-06-29 11:40:57 +02:00
Terrtia	fce63d52a0	fix: [updater] remove old ARDB env	2023-06-29 09:07:00 +02:00
Terrtia	d6a24c0357	fix: [hunter + misp export] fix misp event json export + retro hunt date search and description	2023-06-29 08:48:50 +02:00
Terrtia	b4d536f174	chg: [investigation] add object comment	2023-06-26 16:28:31 +02:00
Terrtia	a0686eefcf	fix: [pystemon importer] fix base64 encoding	2023-06-26 11:24:04 +02:00
Terrtia	f1f33d6c1b	chg: [pystemon importer] fix gzipped pastes	2023-06-22 16:34:14 +02:00
Terrtia	47e1343187	fix: [crawler] same capture uuid if a domain is already crawled	2023-06-22 16:09:18 +02:00
Terrtia	b459498db2	chg: [queues] track object + check if object processed	2023-06-22 15:38:04 +02:00
Terrtia	4567c9d400	chg: [correlation graph] show message if max_nodes reached + fix cookie-name sparkline	2023-06-20 11:23:58 +02:00
Terrtia	501d10bbbd	chg: [crawler] auto tag crawled domains	2023-06-20 08:11:44 +02:00
Terrtia	c02b9f2a81	fix: [IPAdress module] empty queue if no IP ranges provided	2023-06-19 15:17:41 +02:00
Terrtia	b64c70e0cd	fix: [retro hunt] fix object tag queue + decoded content	2023-06-19 10:53:15 +02:00
Terrtia	6c21a8f8cf	fix: [daterange object] fix objects by date	2023-06-19 08:55:56 +02:00
Terrtia	84bc2c0669	chg: [correlation] add an option to remove max number of nodes if max_node == 0	2023-06-18 16:09:57 +02:00
Terrtia	e9539e640b	fix: [crawler] fix incomplete response	2023-06-18 15:09:09 +02:00
Terrtia	f8fd037bd2	chg: [object cookie-name] add new cookie-name object + correlation	2023-06-16 15:39:13 +02:00
Terrtia	ee76d20277	fix: [user] fix get user token #163	2023-06-14 14:53:07 +02:00
Terrtia	b38819fdaa	fix: [user] fix user delete #163	2023-06-14 14:40:01 +02:00
Terrtia	d4152462f5	chg: [titles] add title IDs and contents search	2023-06-13 15:29:11 +02:00
Terrtia	94961f2eba	chg: [favicon object] add favicon object	2023-06-12 16:51:45 +02:00
Terrtia	580879ee5c	fix: [MISP export] fix ail object first/last seen + obj logger	2023-06-09 11:19:22 +02:00
Terrtia	f540df0ff2	chg: [sow item] show item investigations	2023-06-07 14:17:06 +02:00
Terrtia	70802205f2	fix: [d4] change enable d4	2023-06-07 11:03:32 +02:00
Terrtia	4cd2b6517b	chg: [kvrocks migration] mv update/v.50	2023-06-07 10:33:45 +02:00
Terrtia	a22e6f6180	fix: [kvrocks migration]	2023-06-07 10:21:49 +02:00
Terrtia	f3c3cb5d05	fix: [objects] fix investigation + ail2ail + screenshot MISP export	2023-06-04 17:34:21 +02:00
Terrtia	62880e3a1c	fix: [show domains] fix down domains	2023-06-02 13:30:59 +02:00
Terrtia	9efc348506	chg: [correlation] filter blank screenshots	2023-06-02 11:23:52 +02:00
Terrtia	7c7799564f	chg: [importers] improve abstract class and logs	2023-06-02 11:03:32 +02:00
Terrtia	50abff66b4	chg: [HOWTO] improve HOWTO	2023-05-30 14:48:06 +02:00
Terrtia	2ebe4845a7	fix: [module extractor] fix tracker extractor	2023-05-30 10:11:12 +02:00
Terrtia	8252d6b69e	fix: [tracker] fix tracker delete	2023-05-26 14:09:12 +02:00
Terrtia	1e7b527e41	fix: [tracker] fix webhook	2023-05-26 13:57:34 +02:00
Terrtia	b3cafd2a1d	chg: [correlation graph] update node legend	2023-05-26 11:44:29 +02:00
Terrtia	b4f1a43208	chg: [correlation] correlation graph: filter title objects	2023-05-26 10:47:58 +02:00
Terrtia	405d097024	fix: [crawler] fix undefined capture status	2023-05-25 16:26:48 +02:00
Terrtia	9a4feb93a0	fix: [correlation btc info] catch btc txs error	2023-05-25 16:11:55 +02:00
Terrtia	ee951ca948	chg: [correlation] add direct correlation stats	2023-05-25 16:00:27 +02:00
Terrtia	c008366f02	chg: [new title object] add new title object + correlation on page title	2023-05-25 14:33:12 +02:00
Terrtia	f7e0a357ea	fix: [Phone module] Filter Invalid Phone numbers	2023-05-24 13:31:10 +02:00
Terrtia	2c8b239eec	fix: [phone] fix phone module	2023-05-24 13:26:47 +02:00
Terrtia	19f154a81a	fix: [domain search] fix template domain types filter	2023-05-24 11:55:31 +02:00
Terrtia	a30cc1539d	fix: [MISP auto export] fix module input message	2023-05-24 11:12:02 +02:00
Terrtia	353b290899	chg: [Phone module] Filter Invalid Phone numbers + UI Show extracted	2023-05-24 10:48:29 +02:00
Terrtia	7a52aec884	fix: [tests] replace unmaintened nose by nose2	2023-05-23 14:47:02 +02:00
Terrtia	94d7eaf11d	fix: [tests] fix tests	2023-05-23 13:38:33 +02:00
Terrtia	af719d1d94	chg: [importers] add Dir/File Importer	2023-05-22 15:31:48 +02:00
Terrtia	175963e371	merge master	2023-05-17 16:07:32 +02:00
Terrtia	20fad97d6a	chg: [update] bump version < 5.0 roll back commit	2023-05-17 15:59:52 +02:00
Terrtia	2127b4363f	chg: [kvrocks] add migration notice	2023-05-17 15:46:44 +02:00
Terrtia	5b6a7f15ce	chg: [update] older versions: filter 5.0 updates	2023-05-17 11:40:42 +02:00
Terrtia	88d3870230	chg: [ail] cleanup	2023-05-17 10:26:54 +02:00
Terrtia	5ef29d8abb	chg: [MISP] refactor MISP thehive auto push	2023-05-16 16:34:22 +02:00
Terrtia	5d84b347bc	fix: [cves] fix cve search timeout	2023-05-16 13:28:31 +02:00
Terrtia	ed9682798e	chg: [IPAddress module] migrate IPAddress module	2023-05-15 11:39:16 +02:00
Terrtia	7669c16c74	fix: [Onion module] fix kvrocks sismeber	2023-05-15 10:42:46 +02:00
Terrtia	2ae3ffd6d2	fix: [core] fix instance ail:uuid	2023-05-15 10:37:03 +02:00
Terrtia	8ec4370043	fix: [telegram] fix regex	2023-05-15 10:12:58 +02:00
Terrtia	1aa63e8a08	fix: [logger] rename default logger file + fix log messages	2023-05-15 09:51:10 +02:00
Terrtia	e365555905	fix: [trackers] fix pgp yara tracker	2023-05-12 15:43:41 +02:00
Terrtia	9b2e8718d7	chg: [logs] add new logger	2023-05-12 15:29:53 +02:00
Terrtia	98cbaad1ad	fix: [cves] catch cve search errors	2023-05-12 09:59:18 +02:00
Terrtia	f4d20533cc	chg: [trackers UI] add admin endpoint	2023-05-12 09:25:17 +02:00
Terrtia	d4766c5adc	fix: [trackers] fix trackers edit	2023-05-11 16:49:09 +02:00
Terrtia	4473086f89	chg: [trackers] refactor trackers	2023-05-11 16:21:43 +02:00
Terrtia	6b60041db2	chg: [crawler] add unsafe tag if domain contain unsafe screenshot	2023-05-10 16:28:19 +02:00
Terrtia	37c71b8438	chg: [objects + retro hunt] refactor retro hunt + objects retro hunts + get objects generator by filters (date_from, sources, mimetypes, ...)	2023-05-10 16:26:46 +02:00
Terrtia	0daf5bad44	chg: [trackers] refactor trackers: track objects + filter by object types/sources/subtypes	2023-05-04 16:35:56 +02:00
Terrtia	8f65fd5fe7	fix: [search domains] fix search domains by daterange	2023-04-24 13:35:55 +02:00
Terrtia	10fbf228c1	fix: [domain explorer] fix search by daterange	2023-04-24 10:55:58 +02:00
Terrtia	3f8656a835	fix: [api] fix token length	2023-04-21 16:16:10 +02:00
Terrtia	e0899e6e0d	chg: [corellation graph] hide unsafe screenshots	2023-04-21 15:38:48 +02:00
Terrtia	6c6a3b48e2	fix: [modules] log level	2023-04-21 14:53:33 +02:00
Terrtia	6338268e8c	fix: [domain UI] fix correlation card + language search	2023-04-21 10:26:14 +02:00
Terrtia	79fa5dc30b	fix: [investigation] fix create investigation	2023-04-20 16:17:54 +02:00
Terrtia	7ed7b4c63e	fix: [module extractor] fix cache + fix yara match	2023-04-20 16:05:58 +02:00
Terrtia	86b1fda59b	fix: [MISP exporter] fix MISP exporter CIRCL/AIL-framework#590	2023-04-20 10:44:17 +02:00
Terrtia	ab7b2bdbab	chg: [pystemon importer] create new PystemonImporter module, refactor pystemon feeder	2023-04-14 14:43:07 +02:00
Terrtia	ccf935700a	fix: [objects] show/add subtype objects tags	2023-04-14 10:50:58 +02:00
Terrtia	02814e7f81	chg: [UI settings] add ail modules svg + fix admin view	2023-04-13 15:59:44 +02:00
Terrtia	6b5e833fc1	fix: [tracker modulew] fix duplicate module stats	2023-04-13 14:42:57 +02:00
Terrtia	6f9e0c2f66	chg: [AIL queues] rewrite module queues. remove PUBSUB	2023-04-13 14:25:02 +02:00