Commit graph

75 commits

Author SHA1 Message Date
terrtia
bd2ca4b319
fix: [crawler] fix api create_task 2024-01-09 09:47:49 +01:00
terrtia
9221e532c4
fix: [crawlers] fix task start 2023-12-12 11:32:33 +01:00
terrtia
235539ea42
fix: [crawler] fix capture start time 2023-12-11 09:30:09 +01:00
terrtia
1c52c187ad
fix: [api] fix add crawler capture return 2023-12-08 10:37:58 +01:00
terrtia
a382b572c6
chg: [crawler] push onion discovery capture_uuid to another AIL 2023-12-07 11:28:35 +01:00
terrtia
c5cef5fd00
chg: [core] merge master + fix object subtype correlation stats 2023-10-12 13:53:00 +02:00
Jean-Louis Huynen
68c17c3fbc
chg: [crawlers] submit cookies to the crawler task API 2023-08-31 16:13:20 +02:00
Jean-Louis Huynen
ed0423118e
chg: [crawlers] submit a single cookie to the crawler task API 2023-08-31 15:42:44 +02:00
Terrtia
b32f110285
chg: [chat + user-account] correlations + usernames timeline 2023-08-28 16:29:38 +02:00
Terrtia
4e3784922c
fix: typo 2023-08-23 11:47:39 +02:00
Terrtia
2145eb7b8a
fix: [title] fix None title 2023-08-23 11:46:37 +02:00
Terrtia
68dffcd26b
chg: [api crawler] fix response + add cookiejar, proxy and frequency parameters 2023-07-25 15:57:11 +02:00
Terrtia
a9485928db
chg: [HHHash] add HHHash object and correlation https://www.foo.be/2023/07/HTTP-Headers-Hashing_HHHash 2023-07-17 15:47:17 +02:00
Terrtia
73bfe614df
chg: [updater] refactor background updater + add v5.2 update 2023-07-12 11:36:47 +02:00
Terrtia
28c647d370
chg: [crawler har] compress HAR 2023-07-10 15:56:34 +02:00
Terrtia
c719990125
fix: [crawler] add timeout to Unknown captures 2023-07-10 11:23:44 +02:00
fukusuket
e35924ec22 fix: [crawler] add exception handing for ping_lacus 2023-07-08 12:11:25 +09:00
Terrtia
450ebdd789
chg: [etag] add new etag object 2023-07-06 11:26:32 +02:00
Terrtia
47e1343187
fix: [crawler] same capture uuid if a domain is already crawled 2023-06-22 16:09:18 +02:00
Terrtia
501d10bbbd
chg: [crawler] auto tag crawled domains 2023-06-20 08:11:44 +02:00
Terrtia
f8fd037bd2
chg: [object cookie-name] add new cookie-name object + correlation 2023-06-16 15:39:13 +02:00
Terrtia
94961f2eba
chg: [favicon object] add favicon object 2023-06-12 16:51:45 +02:00
Terrtia
405d097024
fix: [crawler] fix undefined capture status 2023-05-25 16:26:48 +02:00
Terrtia
c008366f02
chg: [new title object] add new title object + correlation on page title 2023-05-25 14:33:12 +02:00
Terrtia
7669c16c74
fix: [Onion module] fix kvrocks sismeber 2023-05-15 10:42:46 +02:00
Terrtia
54a0bcb022
chg: [crawler] update default user agent 2023-04-04 09:23:52 +02:00
Terrtia
47da4aa62c
chg: [crawle] migrate domains settings 2023-03-31 09:25:06 +02:00
Terrtia
126ecb2e39
fix: [core] fix merge 2023-03-16 16:49:53 +01:00
Terrtia
524a404dc8
chg: [core] merge conflict 2023-03-16 15:50:42 +01:00
Terrtia
925d67a35e
chg: [crawler] add crawler scheduler 2023-03-14 17:36:42 +01:00
Terrtia
6842efc15d
chg: [crawler] refactor crawler tasks + migrate cookiejars + add proxy option 2023-02-21 12:22:49 +01:00
Terrtia
c04bc7bb57
chg: [crawler] cookies migration + refactor 2023-02-17 14:50:20 +01:00
Terrtia
f9715408be
chg: [migration] migrate Item + Domain metas 2022-11-30 15:50:10 +01:00
Terrtia
73dbef2700
chg: [all] remove old objects + migrate cryptocurrencies module + cleanup code 2022-11-28 15:01:40 +01:00
Terrtia
aac024565f
chg: [tags] refactor tags + cleanup 2022-11-22 10:47:15 +01:00
Terrtia
104eaae793
chg: [crawler + core + cve] migrate crawler to lacus + add new CVE object and correlation + migrate core 2022-10-25 16:31:38 +02:00
Terrtia
1372b1ef68
fix: [api] fix crawler api response 2022-09-14 10:27:17 +02:00
Terrtia
1254c1c9c0
chg: [api] send url to crawler 2022-09-14 10:02:38 +02:00
Terrtia
aa6ba61050
chg: [statistics] ARDB migration 2022-09-08 10:31:57 +02:00
Terrtia
d27d47dc70
chg: [Kvrocks migration] rewrite obj tags + migration 2022-09-01 14:04:00 +02:00
Terrtia
9c1bfb7073
DB migration 2022-08-19 16:53:31 +02:00
Terrtia
ebcffd4b95
fix: [crawler] fix is_splash_manager_connected #133 2021-12-03 15:36:47 +01:00
Terrtia
cb45fe9fab
fix: [crawler] add comment 2021-11-26 16:35:51 +01:00
Terrtia
4e481603b5
Merge branch 'master' of github.com:ail-project/ail-framework 2021-10-14 14:23:24 +02:00
Terrtia
57fbacc49c
chg: [crawler] add auto crawler functions 2021-10-14 14:23:11 +02:00
osagit
fc2c3ea08f
fix: error message contains http protocol twice
Error Can't connect to AIL Splash Manager, http://https://localhost:7001/
2021-09-07 11:57:17 +02:00
Terrtia
7a652b5195
fix: [crawler] fix new crawled item id 2021-07-14 15:48:17 +02:00
Terrtia
b29767a020
merge 2021-07-14 14:08:15 +02:00
Terrtia
ec727338e6
fix: [crawlers] get_all_splash return type 2021-06-16 10:06:04 +02:00
Terrtia
759ec73f84
fix: [Splash_Manager errors] catch invalid response 2021-06-15 17:25:51 +02:00