66 commits

Author	SHA1	Message	Date
Terrtia	4e3784922c	fix: typo	2023-08-23 11:47:39 +02:00
Terrtia	2145eb7b8a	fix: [title] fix None title	2023-08-23 11:46:37 +02:00
Terrtia	68dffcd26b	chg: [api crawler] fix response + add cookiejar, proxy and frequency parameters	2023-07-25 15:57:11 +02:00
Terrtia	a9485928db	chg: [HHHash] add HHHash object and correlation https://www.foo.be/2023/07/HTTP-Headers-Hashing_HHHash	2023-07-17 15:47:17 +02:00
Terrtia	73bfe614df	chg: [updater] refactor background updater + add v5.2 update	2023-07-12 11:36:47 +02:00
Terrtia	28c647d370	chg: [crawler har] compress HAR	2023-07-10 15:56:34 +02:00
Terrtia	c719990125	fix: [crawler] add timeout to Unknown captures	2023-07-10 11:23:44 +02:00
fukusuket	e35924ec22	fix: [crawler] add exception handing for ping_lacus	2023-07-08 12:11:25 +09:00
Terrtia	450ebdd789	chg: [etag] add new etag object	2023-07-06 11:26:32 +02:00
Terrtia	47e1343187	fix: [crawler] same capture uuid if a domain is already crawled	2023-06-22 16:09:18 +02:00
Terrtia	501d10bbbd	chg: [crawler] auto tag crawled domains	2023-06-20 08:11:44 +02:00
Terrtia	f8fd037bd2	chg: [object cookie-name] add new cookie-name object + correlation	2023-06-16 15:39:13 +02:00
Terrtia	94961f2eba	chg: [favicon object] add favicon object	2023-06-12 16:51:45 +02:00
Terrtia	405d097024	fix: [crawler] fix undefined capture status	2023-05-25 16:26:48 +02:00
Terrtia	c008366f02	chg: [new title object] add new title object + correlation on page title	2023-05-25 14:33:12 +02:00
Terrtia	7669c16c74	fix: [Onion module] fix kvrocks sismeber	2023-05-15 10:42:46 +02:00
Terrtia	54a0bcb022	chg: [crawler] update default user agent	2023-04-04 09:23:52 +02:00
Terrtia	47da4aa62c	chg: [crawle] migrate domains settings	2023-03-31 09:25:06 +02:00
Terrtia	126ecb2e39	fix: [core] fix merge	2023-03-16 16:49:53 +01:00
Terrtia	524a404dc8	chg: [core] merge conflict	2023-03-16 15:50:42 +01:00
Terrtia	925d67a35e	chg: [crawler] add crawler scheduler	2023-03-14 17:36:42 +01:00
Terrtia	6842efc15d	chg: [crawler] refactor crawler tasks + migrate cookiejars + add proxy option	2023-02-21 12:22:49 +01:00
Terrtia	c04bc7bb57	chg: [crawler] cookies migration + refactor	2023-02-17 14:50:20 +01:00
Terrtia	f9715408be	chg: [migration] migrate Item + Domain metas	2022-11-30 15:50:10 +01:00
Terrtia	73dbef2700	chg: [all] remove old objects + migrate cryptocurrencies module + cleanup code	2022-11-28 15:01:40 +01:00
Terrtia	aac024565f	chg: [tags] refactor tags + cleanup	2022-11-22 10:47:15 +01:00
Terrtia	104eaae793	chg: [crawler + core + cve] migrate crawler to lacus + add new CVE object and correlation + migrate core	2022-10-25 16:31:38 +02:00
Terrtia	1372b1ef68	fix: [api] fix crawler api response	2022-09-14 10:27:17 +02:00
Terrtia	1254c1c9c0	chg: [api] send url to crawler	2022-09-14 10:02:38 +02:00
Terrtia	aa6ba61050	chg: [statistics] ARDB migration	2022-09-08 10:31:57 +02:00
Terrtia	d27d47dc70	chg: [Kvrocks migration] rewrite obj tags + migration	2022-09-01 14:04:00 +02:00
Terrtia	9c1bfb7073	DB migration	2022-08-19 16:53:31 +02:00
Terrtia	ebcffd4b95	fix: [crawler] fix is_splash_manager_connected #133	2021-12-03 15:36:47 +01:00
Terrtia	cb45fe9fab	fix: [crawler] add comment	2021-11-26 16:35:51 +01:00
Terrtia	4e481603b5	Merge branch 'master' of github.com:ail-project/ail-framework	2021-10-14 14:23:24 +02:00
Terrtia	57fbacc49c	chg: [crawler] add auto crawler functions	2021-10-14 14:23:11 +02:00
osagit	fc2c3ea08f	fix: error message contains http protocol twice ... Error Can't connect to AIL Splash Manager, http://https://localhost:7001/	2021-09-07 11:57:17 +02:00
Terrtia	7a652b5195	fix: [crawler] fix new crawled item id	2021-07-14 15:48:17 +02:00
Terrtia	b29767a020	merge	2021-07-14 14:08:15 +02:00
Terrtia	ec727338e6	fix: [crawlers] get_all_splash return type	2021-06-16 10:06:04 +02:00
Terrtia	759ec73f84	fix: [Splash_Manager errors] catch invalid response	2021-06-15 17:25:51 +02:00
Terrtia	2abe5217aa	fix: [Splash_Manager errors] catch invalid response	2021-06-15 17:19:57 +02:00
Terrtia	4896db98a3	chg: [launcher + modules] add module tests (Onion module)	2021-05-17 18:03:30 +02:00
Terrtia	4bbff47989	chg: [AIL items + Onion] create AIL item objects + Onion module refactor	2021-05-14 14:42:16 +02:00
Terrtia	c0be210d2c	chg: [crawler] add test + relaunch crawlers + major fixs	2021-03-29 20:27:20 +02:00
Terrtia	503e7e33aa	fix: [crawler] typo	2021-03-05 18:52:14 +01:00
Terrtia	6daa750e3b	fix: [Crawler] faup	2021-03-05 18:47:38 +01:00
Terrtia	1f94c1c693	chg: [splash manager] update enpoints + use Splash name to restart docker	2021-03-04 09:26:28 +01:00
Terrtia	fc7a61f67c	chg: [merge master]	2021-02-10 15:50:48 +01:00
Terrtia	d941d8abb4	chg: [domains search] search domains by name	2021-02-05 17:42:33 +01:00