Terrtia
762b517150
fix: [MISP auto export] fix MISP_TheHive_feeder
2020-06-19 13:36:03 +02:00
Terrtia
41cacf7129
chg: [crawler manager] get all splash dockers, proxies and launch all crawlers
2020-06-09 18:33:41 +02:00
Terrtia
d20ae35548
fix: [crawler] option to disable screenshots and har
2020-06-04 16:05:32 +02:00
Terrtia
d6897904d6
Merge branch 'master' of https://github.com/ail-project/ail-framework
2020-05-27 20:43:17 +02:00
Terrtia
6dc8f0a79d
chg: [API + item library] add a way to check if a user can access the API + item import (use item_basic)
2020-05-27 20:43:00 +02:00
a997e546cb
chg: [tor crawler] nyt added
2020-05-26 14:43:40 +02:00
Terrtia
eec579c5d4
fix: [crawlers lib] missing config file
2020-05-25 08:54:19 +02:00
6b7514f6f0
chg: [black-list onion] keybase added
2020-05-23 21:06:41 +02:00
Terrtia
9991570499
fix: [create au to crawler] save crawler_delta time
2020-05-22 16:20:36 +02:00
Terrtia
f9ebe26354
fix: [updater] return boolean
2020-05-22 15:49:06 +02:00
Terrtia
5a221cd996
chg: [core crawler] add screen script
...
create screen + windows + kill them + get list
2020-05-22 15:43:58 +02:00
Terrtia
3fd200be43
chg: [crawler proxy] interact with the splash manager API
...
(get list of proxies + splash containers + launch them)
2020-05-22 15:41:05 +02:00
Terrtia
5d7488f74a
fix: [Credential] fix timeout error
2020-05-20 17:29:51 +02:00
Terrtia
3e2d04bbd3
fix: [Credential] fix timeout error
2020-05-20 17:26:40 +02:00
Terrtia
acd564026e
fix: [Credential] force lowercase of mail address
2020-05-20 17:20:23 +02:00
Terrtia
f917357615
fix: [Credential] fix timeout message
2020-05-20 17:16:02 +02:00
Terrtia
4601003509
fix: [Mail module] regex timeout
2020-05-20 17:03:58 +02:00
Terrtia
85a0d944cc
chg: [launcher] add option to reset UI admin password
2020-05-19 10:07:27 +02:00
Terrtia
c675cb2eaa
chg: [UI dashboard] sort module by name
2020-05-16 22:35:24 +02:00
Terrtia
3bb1bf8d43
fix: [Mail module] remove debug
2020-05-14 14:31:35 +02:00
Terrtia
f8ff232c3e
fix: [Mail module] fix dns caching + use redis queue
2020-05-14 14:26:52 +02:00
Terrtia
5475660785
fix: [UI Decoded items] fix download decoded file
2020-05-14 10:26:54 +02:00
Terrtia
cfaa8ef2ba
chg: [LAUNCH] launch telegram module by default
2020-05-12 16:46:11 +02:00
Terrtia
3c49135081
fix: [Mail module] add debug output
2020-05-12 15:49:35 +02:00
Terrtia
3f6efadcc4
fix: [Mail module] replace signal by multiprocessing ( https://docs.python.org/3.4/library/signal.html#execution-of-python-signal-handlers )
2020-05-12 15:38:03 +02:00
Terrtia
c56d0a471e
fix: [Mail] debug signal
2020-05-12 10:02:38 +02:00
Terrtia
c2d47cdd61
fix: [Mail] debug
2020-05-12 09:53:31 +02:00
Terrtia
19f7d8c1e8
chg: [UI correlation] add username correlation graph
2020-05-11 18:11:38 +02:00
Terrtia
fc0c12417e
fix: [Mail module] remove test time.wait
2020-05-11 16:24:34 +02:00
Terrtia
f82fadba4a
fix: [Mail module] debug signal timeout
2020-05-11 16:20:36 +02:00
Terrtia
f9ce4fa280
fix: [Mail module] debug signal timeout
2020-05-11 16:13:03 +02:00
Terrtia
8f31b9b90c
fix: [telegram module] typo
2020-05-11 14:48:12 +02:00
Terrtia
013f32c7d1
fix: [telegram module] fix tagging
2020-05-11 14:47:08 +02:00
Terrtia
daf534d74b
chg: [telegram module] add new tag: telegram invite code
2020-05-11 14:44:50 +02:00
Terrtia
3af5c4bafa
fix: [telegram module] check username length
2020-05-11 14:28:18 +02:00
Terrtia
0d67ad80d1
fix: [telegram module] remove debug
2020-05-11 14:22:56 +02:00
Terrtia
297d5891a8
chg: [telegram backend] add username correlation + save invite hash
2020-05-11 14:21:10 +02:00
Terrtia
27554d8863
fix: [Mails] regex timeout
2020-05-11 11:33:07 +02:00
Terrtia
0dfddd318b
fix: [Mails] change module output
2020-05-11 11:22:09 +02:00
Terrtia
4d318643ce
fix: [Mails] remove print + test
2020-05-11 11:15:12 +02:00
Terrtia
0a525f369e
fix: [Mails] refactor Mail module
2020-05-11 11:11:05 +02:00
Terrtia
8c864fdba0
fix: [urlextract importer] fix parent map + replace set JSON queue by list
2020-05-06 16:57:13 +02:00
Terrtia
bae2fd79df
fix: [import urlextract parent] fix typo
2020-05-05 11:39:06 +02:00
Terrtia
074ada504a
chg: [importer] add map twitter id - item id, add parents link between twitter and url extracted
2020-05-05 11:24:33 +02:00
Terrtia
3f4447f9b1
fix: [Mails regex timeout] reduce default timeout
2020-05-04 14:36:48 +02:00
Terrtia
fa35bfc252
fix: [Mails dns resolver] update timeout exception
2020-05-04 14:30:11 +02:00
Terrtia
3f5fc85536
fix: [Mails] typo
2020-05-04 14:10:42 +02:00
Terrtia
855a246bbb
fix: [Mails] import
2020-05-04 14:09:14 +02:00
Terrtia
67b6f86d20
fix: [Mails] import
2020-05-04 14:08:12 +02:00
Terrtia
42d7a6cb78
fix: [Mails] add regex timeout
2020-05-04 14:05:04 +02:00
Terrtia
c3d37f2bbf
fix: [Credential] add regex timeout
2020-05-04 11:11:35 +02:00
Terrtia
52b7f13a6f
fix: [Credential] add regex timeout
2020-05-04 11:02:24 +02:00
Terrtia
64fae5d7df
fix: [urlexport importer] fix item name + redis config
2020-05-03 15:40:43 +02:00
Terrtia
b4cb14798c
fix: [urlexport importer] item_id, force str type
2020-05-02 20:33:25 +02:00
Terrtia
ecc042376d
fix: [urlexport importer] add missing import
2020-05-02 20:30:21 +02:00
Terrtia
15265d7909
fix: [urlextract importer] class name
2020-05-02 16:09:51 +02:00
Terrtia
d269a5b31b
chg: [import er url_extract] add item twitter parent
2020-04-30 15:46:38 +02:00
Terrtia
eddeb0b3b0
chg: [importer url_extract] fix item id
2020-04-30 09:57:26 +02:00
Terrtia
3d4131a193
chg: [importer] add url_extract importer
2020-04-30 09:54:26 +02:00
Terrtia
199df870be
fix: [importer] fix typo
2020-04-30 09:23:51 +02:00
Terrtia
a92b81034b
fix: [Global] extend ungzip error catching, catch invalid compressed file
2020-04-29 13:42:55 +02:00
Terrtia
ead24038d9
fix: [json import API] remove list input
2020-04-28 16:44:19 +02:00
Terrtia
6f3cb17beb
chg: [API json import] take list as imput
2020-04-28 15:02:22 +02:00
Terrtia
8d727cbd31
chg: [core import] add AIL JSON import format, API endpoint added (AIL feeders)
2020-04-28 14:14:45 +02:00
Terrtia
596dec3f62
chg: [update doc] update doc install + logo + fix updater
2020-04-20 17:50:40 +02:00
Terrtia
b6dc8e92b2
chg: [Updater] change default branch origin
2020-04-20 17:17:58 +02:00
Terrtia
bec7c52485
chg: [Updater] change default branch origin
2020-04-20 17:16:19 +02:00
Sunil D S
c8c96937cc
crawler_time -> crawler_delta
2020-04-17 20:06:40 +05:30
Terrtia
927a8c96c5
chg [telegram + correlation] new module: telegram (username + login code + join_chat) + add simple_correlation backend
2020-04-10 10:50:55 +02:00
Terrtia
ef3d145c87
chg [telegram + correlation] new module: telegram (username + login code + join_chat) + add simple_correlation backend
2020-04-10 10:49:21 +02:00
Terrtia
5ad529bc7f
fix: [crawler] typo
2020-04-06 10:57:51 +02:00
Terrtia
672bb02bbf
fix: [Crawler splash ResponseNeverReceived] add retry
2020-04-06 10:52:44 +02:00
Terrtia
179fba2ecc
fix: [crawler] error catcher
2020-04-01 14:58:27 +02:00
Terrtia
f97698ad44
Merge branch 'master' into crawler_v2
2020-04-01 09:59:33 +02:00
Terrtia
72f1f15659
chg: [crawler] edit cookie and cookiejar + add cookie to cookiejar + fix screenshot duplicate
2020-04-01 09:58:47 +02:00
Terrtia
5f289f04f3
chg: [Crawler core + UI] crawler lua: handle retry + fix cookie loader and selector
2020-03-30 18:43:50 +02:00
Terrtia
e3e543ba8b
chg: [Crawler] default docker memory usage
2020-03-30 09:42:14 +02:00
Terrtia
169c4a8ec7
chg: [cookiejar UI] add cookiejar + show all
2020-03-27 17:06:26 +01:00
Terrtia
720ffa1c9c
chg: [MISP export] export domain as domain-crawled object
2020-03-26 17:03:57 +01:00
Terrtia
d87ecff4a0
chg: [crawler - cookies] add/show/select cookies
2020-03-24 17:15:43 +01:00
Terrtia
1c45571042
chg: [crawler] add cookies list by user/global, save cookies from file + dict(name, value), TODO: API + handle errors
2020-03-23 18:00:09 +01:00
Terrtia
db634e8866
fix: [crawler] cleanup
2020-03-20 16:20:01 +01:00
Terrtia
6cfd3fe36d
chg: [crawler] bypass login: use cookie provided by user and accept cookie from server + refractor
2020-03-20 16:15:25 +01:00
Terrtia
42ea678b7a
chg: [Splash Crawler] use cookies to bypass login
2020-03-09 17:02:18 +01:00
Terrtia
4300fd7803
fix: [PgpDump parser] remove header comment (rfc4880) + remove empty lines
2020-03-05 11:19:43 +01:00
Terrtia
1944593a48
fix: [Pgp Dump] remove tool version
2020-03-04 16:37:00 +01:00
Terrtia
873797d87f
chg: [import_dir] fix is_gzip test, use magic number
2020-02-27 14:21:32 +01:00
Terrtia
998f8cc8e1
fix: [ZMQ Feeder] performance: replace zmq recv NOBLOCK by Poller
2020-02-27 13:23:40 +01:00
Terrtia
40b853cbe3
chg: [MISP Importer] files: handle missing sha1/sha256 attributes + fix Items and Screenshots dir
2020-02-26 13:45:47 +01:00
Terrtia
5ae22ec216
chg: [Updater] relauch updater on change (git pull)
2020-02-26 10:19:36 +01:00
Terrtia
e6c3e31343
fix: [pgpdump] fix subtype save
2020-02-26 09:24:35 +01:00
Terrtia
0d47c6efd3
fix: [Updater] fix current_tag parser
2020-02-25 15:57:07 +01:00
Terrtia
bdf2fce332
fix: [import_dir] remove special characters
2020-02-25 14:41:45 +01:00
Terrtia
72fe8a2461
fix: [import_dir] remove dir whitespaces #475
2020-02-25 11:34:55 +01:00
Terrtia
1f8c858c77
fix: [MISP export] get full list
2020-02-21 15:47:05 +01:00
Terrtia
32b023165a
fix: [MISP export] misp response parsing
2020-02-21 15:09:16 +01:00
Terrtia
b04b3dcfa2
fix: [MISP export import]
2020-02-21 15:03:48 +01:00
Terrtia
66dd0dec0b
chg: [MISP export] remove pythonify
2020-02-21 15:01:39 +01:00
Terrtia
cae4086208
chg: [MISP export] save event uuid - global_id
2020-02-21 14:42:18 +01:00
Terrtia
988f71035d
fix: [misp export] force event uuid
2020-02-21 14:35:34 +01:00
Terrtia
bdddb1e5b6
fix: [MISP export] fix pgp-meta template
2020-02-21 13:25:59 +01:00
Terrtia
7f64d57bbd
fix: [Update] force str tags
2020-02-21 13:08:39 +01:00
Terrtia
ac3a531987
fix: [Update] force str tags
2020-02-21 13:06:37 +01:00
Terrtia
07e3bfe396
fix: [update thirdparty] #473
2020-02-21 11:36:22 +01:00
Terrtia
e3f76c52fa
fix: [misp keys import] fix #471
2020-02-21 11:03:28 +01:00
Terrtia
fd4b3e04b6
fix: [Update] support subversion + fix #472
2020-02-21 10:41:34 +01:00
Terrtia
301e7e67d1
chg: [UI MISP export] add object export picker by user
2020-02-19 16:15:41 +01:00
Terrtia
62e3d95f22
merge master => import_export_v1
2020-02-19 09:35:00 +01:00
Terrtia
b8342cfa69
chg: [UI MISP export] export to MISP instance
2020-02-18 17:02:00 +01:00
Terrtia
b0dde78c2e
chg: [UI MISP import]show error +imported uuid/global_id
2020-02-18 13:47:47 +01:00
Terrtia
1b861f55ee
chg: [UI MISP export] add error output
2020-02-18 11:27:00 +01:00
Terrtia
4d0e9a94d0
chg: [UI MISP export] add MISP export
2020-02-17 17:01:16 +01:00
Terrtia
ee15a9b5c6
chg: [UI MISP import] add MISP import
2020-02-17 10:52:25 +01:00
Terrtia
b641be3b58
chg: [MISP import export] item with relationships (domain not handled)
2020-02-14 09:57:42 +01:00
Terrtia
4d2449cd61
chg: [MISP import export] decoded with relationships
2020-02-13 15:03:05 +01:00
Terrtia
eddb000890
chg: [MISP import-export] screenshots with relationships
2020-02-12 17:12:17 +01:00
Terrtia
ac114814fd
chg: [MISP import-export] pgp and cryptocurrency with relationships
2020-02-12 16:36:02 +01:00
Terrtia
c042a2a66f
chg: [MISP import] import files obj: decoded + screenshot
2020-02-11 15:50:56 +01:00
Terrtia
25e3022eab
chg: [MISP import] import files obj: decoded + screenshot
2020-02-11 15:48:30 +01:00
Terrtia
f950e700bd
chg: [SentimentAnalysis] clean
2020-02-10 11:06:39 +01:00
Terrtia
4097d95237
fix: [SentimentAnalysis] download nltk punkt
2020-02-10 11:04:24 +01:00
Terrtia
cf24c59e1d
fix: [IPAdress] remove leading zeros #465
2020-02-10 10:44:06 +01:00
Terrtia
f9856a1589
fix: [module Webstats + BankAccount-Decoder] fix faup return type + remove old Paste library from BankAccount-Decoder #465
2020-02-10 10:31:53 +01:00
Terrtia
6bbcef024b
chg: [add MISP import] correlation
...
TODO: create correlation
2020-02-07 15:08:41 +01:00
Terrtia
d8fbd72863
fix: [Global catch incomplete file error] fix #464
2020-02-07 10:53:45 +01:00
Terrtia
62ce4646e5
fix: [search items tag] fix daterange
2020-02-07 10:12:38 +01:00
Terrtia
73f98c0897
chg: [add MISP import](no correlations) import item + pgp(incomplete) + screenshot(incomplete)
2020-02-06 17:14:08 +01:00
Thirion Aurélien
c7979d81e3
Merge pull request #463 from petmi627/master
...
Added support for username as notification credentials
2020-02-06 09:49:43 +01:00
Terrtia
fc58940ed6
fix: [core Global] catch and log incomplete files
2020-02-06 09:41:43 +01:00
Mike Peters
7888f54902
Added support for username as notification credentials
2020-02-06 09:18:54 +01:00
Terrtia
b31fffa728
chg: [MISP export] add pgp object + add relationship: item + pgp + cryptocurrency
2020-02-04 13:04:09 +01:00
Terrtia
8770bf05d7
fix: [IPAddress] catch empty config error
2020-02-03 15:29:37 +01:00
Terrtia
4d8db3fcc4
fix: [Global: already saved filename] save updated + filter duplicated items
2020-02-03 14:51:51 +01:00
Terrtia
e808840f95
fix: [Global: filename provided by all feeders] avoid path tranversal
2020-02-03 10:32:20 +01:00
Terrtia
4a732ea9f3
chg: [export MISP] export Items + Screenshots objects with relationships
2020-01-31 17:01:47 +01:00
Terrtia
727bc55966
fix: [Tag core] check if item_date type is an integer
2020-01-30 11:31:33 +01:00
Terrtia
b63fa51166
chg: [domain explorer] domains explorer v2, filter domains by daterange
2020-01-24 15:03:04 +01:00
Terrtia
addb885674
chg: [domain core + UI] add domain explorer v1
2020-01-23 15:43:54 +01:00
Terrtia
08014f125a
fix: [tag core] typo
2020-01-21 15:27:40 +01:00
Terrtia
249b3d657f
fix: [tag core] item date type
2020-01-21 15:23:59 +01:00
Terrtia
997e79a476
fix: [Tag core] add tag, update tag last seen
2020-01-21 13:35:52 +01:00
Terrtia
f139e9564e
fix: [expand btc adress] filter empty addr fields
2020-01-21 10:59:11 +01:00
Terrtia
9357097d2f
chg: [UI + core] tag decoded items, fix : #455
2020-01-20 16:42:26 +01:00
Terrtia
2b735e3b97
chg: [UI item tags search] refractor: search item tag by object, use new functions
2020-01-20 15:13:03 +01:00
Terrtia
a77d16f6d7
fix: [UI] screenshot url
2020-01-14 16:46:53 +01:00
Terrtia
f06551f1a1
chg: [UI tags] add + delete image (screenshot) tags
2020-01-14 16:14:21 +01:00
Terrtia
fb320ca5d5
chg: [UI tags] search domains by tags
2020-01-13 11:10:03 +01:00
Terrtia
99897ffa9b
chg: [core + UI] search domain by tags
2020-01-10 16:52:55 +01:00
Terrtia
ce3d98d620
chg: [Update v2.7] add update v2.7 scripts
2020-01-09 15:07:51 +01:00
Terrtia
3a691a4af3
chg: [tags UI] edit object tags (delete tags)
2020-01-08 17:11:37 +01:00
Terrtia
85d5770e80
chg: [Tag core] objects tagging, Part 2/2
...
TODO: UI tags domain + screenshot + object rename paste=>item
2020-01-07 16:14:56 +01:00
Terrtia
b557b7fed8
chg: [Tag core] objects tagging, Part 1/2
...
TODO UI (tags) + rename paste=>item
2020-01-07 14:30:52 +01:00
Terrtia
2be06973ee
chg: [Tag core] objects tagging, Part 1/2
2020-01-06 17:07:52 +01:00
Terrtia
354a4fef7d
fix: [Crawler] typo
2019-12-19 16:58:36 +01:00
Terrtia
218f1af241
fix: [Crawler] fix screenshot-domain typo
2019-12-19 08:54:32 +01:00
Terrtia
3d81f30043
fix: [Crawler] fix screenshot-domain typo
2019-12-19 08:53:55 +01:00
Terrtia
363801fff7
fix: [Crawler] fix screenshot-domain map
2019-12-19 08:52:02 +01:00
Terrtia
7dbc96cee2
chg: [UI correlation graph + UI domain] correlation screenshot: show img in toolip + show hash in ShowDomain
...
TODO: pixelate images
2019-12-18 16:17:29 +01:00
Terrtia
056bad7a49
chg: [screenshot correlation + v2.6] add screenshot-domain correlation + v2.6 update
2019-12-17 15:13:36 +01:00
Terrtia
b4f49f9e81
fix: [UI showDomain] fix down domain history
2019-12-16 15:55:50 +01:00
Terrtia
fffb4464c6
fix: [Domain] domain was up
2019-12-16 15:24:47 +01:00
Terrtia
dee5e8490f
fix: [Domain] is_domain_up
2019-12-16 15:06:52 +01:00
Terrtia
28ece38d82
chg: [API] get domain min metadata (first up, last up) + get crawled domain by daterange and status
2019-12-16 14:31:31 +01:00
Terrtia
1f97159413
chg: [Domain + Date] get domain up range + get date days and months by daterange
2019-12-13 16:57:24 +01:00
Terrtia
b6cd16e663
chg: [Domain] get all/by month domains up
2019-12-12 14:29:44 +01:00
Terrtia
e71a181bb9
chg: [API] get domain metadata (minimal)
2019-12-11 13:58:43 +01:00
Terrtia
0fb4990d98
chg: [UI correlation graph] tooltip: show domain tags
2019-12-10 16:29:56 +01:00
Terrtia
362afc2253
fix: [Update] force manual update, fix #443
2019-12-10 16:12:05 +01:00
Terrtia
d45d02b667
chg: [UI correlation graph] popover: add loading status + chg css
2019-12-10 15:41:47 +01:00
Terrtia
0ec56cf1ed
fix: [PgpDump] catch bs4 error
2019-12-04 10:02:47 +01:00
Terrtia
3ec72b0430
chg: [pgpdump] add debug
2019-12-04 09:32:09 +01:00
Terrtia
9d652525c4
fix: [Item lib] fix import
2019-12-03 10:10:32 +01:00
Terrtia
2cf419fc26
fix: [Paste submit] fix tags unpack
2019-12-02 17:15:48 +01:00
Terrtia
2841c95321
chg: [Onion] add discovery queue
2019-11-26 16:43:04 +01:00
Terrtia
0af359e5e5
chg: [Showpaste] check if tags are safe (img) + fix domain link
2019-11-25 18:11:20 +01:00
Terrtia
bff9b45c26
fix: [Update] filter invalid tags
2019-11-25 15:54:53 +01:00
edcfb4fcaa
Merge pull request #422 from CIRCL/misp_modules
...
Misp modules
2019-11-25 10:12:32 +01:00
Terrtia
46f6f648a0
chg: [Correlation UI] btc address: get related transaction input/output
2019-11-22 15:30:11 +01:00
Terrtia
1573f5e495
Merge branch 'master' into misp_modules
2019-11-22 15:25:46 +01:00
Terrtia
00c2b139c6
fix: [Correlation UI] fix decoded file type + exist_correlation_id
2019-11-21 10:32:51 +01:00
Terrtia
96cbeddd92
chg: [MispModules] use ConfigLoader
2019-11-20 11:47:13 +01:00
Terrtia
f4fa811229
Merge branch 'master' into misp_modules
2019-11-20 11:42:39 +01:00
Terrtia
5fc5c152f1
chg: [MispModules] mv misp modules to lib
2019-11-20 11:42:25 +01:00
Terrtia
73afdfa227
chg: [Domain UI] add total_nb_correlation
2019-11-19 14:30:12 +01:00
Terrtia
15bd1cc78f
chg: [Domain UI] add unsafe_tags_check (taxo: dark-web + copine-scale) + hide unsafe domain img
2019-11-19 14:03:23 +01:00
Terrtia
ae9cd1d355
chg: [Corrrelation UI] add 404 return
2019-11-19 09:02:25 +01:00
Terrtia
7d442646c2
chg: [Correlation UI] add metadata card and item link: paste + domain
2019-11-18 16:35:40 +01:00
Terrtia
39be4ef916
chg: [Correlation UI + default taxonomies] add dark-web to default taxonomies + fix domain correlation url
2019-11-18 09:46:15 +01:00
Terrtia
cd4f81ec64
chg: [merge] merge master in misp_modules
2019-11-17 15:22:30 +01:00
Terrtia
2e921a12f2
chg: [UI] show correlation (pgp. decoded, cryptocurrency) migration
2019-11-15 17:22:50 +01:00
Terrtia
bb03ef532b
chg: [Correlation UI] add correlation blueprint + UI graph correlation
2019-11-14 17:05:58 +01:00
Terrtia
e42ec48380
chg: [Correlation Graph] add correlation by pgp, cryptocurrency, decoded and item + request correlation by type(pgp, ...) + objects(domain, paste)
2019-11-13 16:46:28 +01:00
Terrtia
7d7c318534
chg: [Correlation Graph] add limit nb max nodes
2019-11-13 10:20:11 +01:00
Terrtia
504e45a43b
chg: [Correlation domain] add correlation graph by domain (union + intersection)
2019-11-12 17:08:52 +01:00
Terrtia
35b63f7bf4
fix: [Domain tag] avoid crawled tags
2019-11-11 14:54:12 +01:00
Terrtia
c5510acb73
fix: [Cryptocurrency] fix import conflict
2019-11-11 10:29:36 +01:00
Terrtia
d765a83066
fix: [Decoded] add Decoded file
2019-11-08 16:27:55 +01:00
Terrtia
880c351c0c
chg: [update + show decoded items] add background update
2019-11-08 16:00:42 +01:00
Terrtia
6b9ba9d377
chg: [Domain] Show last origin
2019-11-08 09:25:09 +01:00
Terrtia
712de0d88a
chg: [Update v2.4] cp config
2019-11-05 16:44:10 +01:00
Terrtia
71718fb69f
Merge branch 'master' into domain_object
2019-11-05 16:03:42 +01:00
Terrtia
c8d5ce9a28
chg: [core] mv bin/packages/config.cfg configs/core.cfg + use ConfigLoader
2019-11-05 15:18:03 +01:00
Terrtia
3c6e424ac3
chg: [UI Domain] UI: tag domain
2019-11-05 09:49:51 +01:00
Terrtia
a3e5e44c9b
chg: [core + UI] refractor Domain + Show Domain, better handle of ports, select screenshot on the UI, ...
2019-10-31 17:14:23 +01:00
Terrtia
a2d6874417
chg: [Domain + UI Crawler] refractor show domain
2019-10-30 17:12:04 +01:00
Terrtia
44bb18a8be
chg: [Correlation] get correlation (crypto + pgp) by domain
2019-10-29 16:52:33 +01:00
Terrtia
1ca1f7975c
chg: [Cryptocurrency + Tools] launch by default + remove old Bitcoin module
2019-10-29 09:19:26 +01:00
Terrtia
4b389559ab
chg: [Domain] move Domain to lib/
2019-10-29 09:13:44 +01:00
Terrtia
cd9488d79a
Merge branch 'master' of https://github.com/CIRCL/AIL-framework
2019-10-28 14:38:15 +01:00
Terrtia
da581878c3
chg: [Keys module] detect public key
2019-10-28 14:37:50 +01:00
Terrtia
f1def65c89
chg: [lib ConfigLoader] add configLoader class
2019-10-28 13:48:43 +01:00
Terrtia
48abb89d28
chg: [Domain] add domain object: tag + correlation (decoded items + tags + pgp + cryptocurrency)
2019-10-17 16:39:43 +02:00
Thirion Aurélien
dc25cd98ff
Merge pull request #404 from WimpyMan/master
...
Added: IP matching module
2019-10-04 13:50:53 +02:00
Bastien Schils
de494aa460
LAUNCH.py: Added execution of script IPAddress.py
2019-10-02 21:35:30 +02:00
Bastien Schils
4586467962
IPAddress.py: use ipaddress module
...
Improved readibility, maintainability and use of standard module
2019-10-02 21:33:02 +02:00
Terrtia
e759b560db
fix: [Tool] fix searchsploit regex
2019-09-30 10:13:47 +02:00
Miroslav Stampar
4f3858045c
Adding more tools
2019-09-27 14:46:26 +02:00
Miroslav Stampar
34bfc6a44a
Covering special cases (on pastebin)
...
There is no need for checking `()` in case of (e.g.) Litecoin and Dash as those are also additionally checked with Bitcoin address verifier
2019-09-26 11:10:02 +02:00
Terrtia
e3040f8e6d
fix: [Tools] typo
2019-09-25 23:57:57 +02:00
Terrtia
804515179d
fix: [Tools] typo
2019-09-25 23:50:27 +02:00
Terrtia
b9c9187e3e
fix: [Tools] fix loop
2019-09-25 23:48:30 +02:00
Miroslav Stampar
ae5c0ace01
Adding tool regexes
2019-09-25 16:12:03 +02:00
Terrtia
987b56a224
Merge branch 'master' of https://github.com/CIRCL/AIL-framework
2019-09-25 10:26:47 +02:00
Terrtia
54bd00503f
chg: [Tools detection] add tool detection module
2019-09-25 10:26:06 +02:00
Miroslav Stampar
69e793f16a
Enforcing Base58 check on Litecoin and Dash addresses
2019-09-25 09:37:33 +02:00
Terrtia
734c94453a
Merge branch 'master' of https://github.com/CIRCL/AIL-framework
2019-09-24 15:34:15 +02:00
Terrtia
bfdcc13496
fix: [TermTracker] fix performance: disable token stats
2019-09-24 15:33:53 +02:00
WimpyMan
d5c3f28e47
modules.cfg: Minor: Added \n to seperate sections
2019-09-24 10:30:33 +02:00
WimpyMan
163fa4b95a
config.cfg.sample: Improved example for IP module
...
By default, the list of networks to monitor is now empty.
The previous value is now given as example.
2019-09-24 10:27:56 +02:00
Bastien Schils
c0171eba81
Added: IP matching module
2019-09-23 18:22:25 +02:00
Miroslav Stampar
9c51c582f9
Improvement of crypto-address regexes (lesser FPs)
2019-09-23 09:41:58 +02:00
Terrtia
839443500d
chg: [Cryptocurrency, RegexTracker] uptdate cryptocurrency list + fix: RegexTracker typo
2019-09-22 14:57:23 +02:00
Terrtia
30fd361e1c
chg: [Cryptocurrency] add private_key entry + fix dash regex
2019-09-22 14:44:37 +02:00
Miroslav Stampar
5136ef12e2
Implementation for different cryptocurrencies
2019-09-20 15:06:39 +02:00
Terrtia
3a9d0157e4
chg: [Cryptocurrency] add new Cryptocurrency module
2019-09-19 16:31:15 +02:00
Terrtia
8612d89275
fix: [SQL module] fix typo
2019-09-19 12:22:21 +02:00
Miroslav Stampar
ba46e38125
Improvement of SQLi detection
2019-09-19 12:08:11 +02:00
Terrtia
fed04e8e5a
chg: [Tracker] add optionnal description field
2019-09-13 16:33:34 +02:00
Terrtia
fb35be77f4
chg: [Tracker] add more info
2019-09-13 11:06:41 +02:00
35dcf428ff
Merge pull request #389 from CIRCL/term_trackerv1
...
Term trackerv1
2019-09-12 14:06:05 +02:00
Terrtia
01ff202ae8
chg: [update] add v2.2 update + fix default update + fix Empty Item.get_tags() + add new LAUNCHER options
2019-09-12 13:25:29 +02:00
Terrtia
0ced25f3be
chg: [trackers] filter trackers list by type + minor fix
2019-09-11 15:33:04 +02:00
Terrtia
21013a7935
fix: [BankAccount] fix #385
2019-09-04 17:08:28 +02:00
Terrtia
bd8735a6e5
chg: [merge] master
2019-09-04 10:24:46 +02:00
Terrtia
e0bb973989
fix: [API doc] get bitcoin metadata + list of items
2019-09-04 10:06:52 +02:00
Terrtia
dfd1128daa
chg: [api] add new endpoints: get bitcoin/pgp name/pgp keys/pgp mail metadata + items list
2019-08-16 17:59:44 +02:00
ab45ac0fef
Merge pull request #370 from CIRCL/api_v1
...
Api v1
2019-08-14 14:28:52 +02:00
Terrtia
7ed09bc923
chg: [UI term tracker] refractor term management: trackers list + show trackers + add new trackers
2019-08-14 09:44:49 +02:00
Terrtia
80f9535074
chg: [api] add endpoint: get tracked item_id by uuid and daterange
2019-08-09 17:07:54 +02:00
Terrtia
4b0819a644
chg: [api] add endpoint: delete tracker term (regex/set/word)
2019-08-09 15:22:57 +02:00
Terrtia
1008c7c4fe
chg: [Term Tracker] refractor term tracker word/set/regex modules + remove old modules
2019-08-09 14:20:13 +02:00
Terrtia
d9bdfecef3
fix: [Term Tracker module] chg module flow
2019-08-07 12:19:42 +02:00
Terrtia
bb6d3a6a26
chg: [Term tracker] add term tracker module (word + set) + API: add new term to track (word + set + regex)
2019-08-07 12:08:24 +02:00
Terrtia
28320a32a6
chg: [term] refractor + add new tracked word/set
2019-08-06 17:03:49 +02:00
Terrtia
e28d563523
chg: [api] use POST with parameters + add API unittest
2019-08-05 16:00:23 +02:00
Terrtia
5e1ae8a893
chg: [api] add new endpoints: get tag metadata + get all tags
2019-08-01 14:36:52 +02:00
Terrtia
98fb6ecef7
fix: [api doc] typo
2019-08-01 13:43:28 +02:00
Terrtia
4c20f58a52
chg: [api] add advanced get item via POST + use same query for each get item
2019-08-01 13:16:57 +02:00
Terrtia
8c02c1b00b
chg: [api] add 404 errors code + add future endpoints in doc
2019-08-01 09:45:59 +02:00
Terrtia
918b4c28ed
fix: [api] fix errors handler 404 405, return json
2019-07-31 13:24:43 +02:00
Terrtia
5f5e86bb13
fix: [Tags + api] fix dict keys name + fix documentation errors output
2019-07-31 11:15:34 +02:00
Terrtia
44cf5bb4af
chg: [API v1] add API documentation + update/delete items tags + Flask_tags refractor
2019-07-30 13:49:21 +02:00
Terrtia
6af9514a48
chg: [API] add GET: item metadata + item content + item tags
2019-07-26 15:44:29 +02:00
Terrtia
0a756294fe
chg: [API] import item (text)
2019-07-26 14:28:02 +02:00
Terrtia
3a8531cafa
chg: [API + import] add API format + item_import refractor
2019-07-25 17:26:32 +02:00
Terrtia
653cb4a209
chg: [misp_modules] parse modules result
2019-07-24 15:51:06 +02:00
Terrtia
09ecc4d93f
chg: [Crawler] add default crawler config + update default user_agent
2019-07-24 10:18:10 +02:00
Terrtia
6295560216
chg: [misp_modules] add enrichment request
2019-07-23 17:09:29 +02:00
Terrtia
8cbfec51f6
chg: [misp_modules] create misp modules + generate config
2019-07-23 16:44:02 +02:00
Terrtia
26a4c7fd2c
fix: [Crawler] incorrect config
2019-07-10 09:42:20 +02:00
Terrtia
34786d54db
Merge branch 'master' into user_management
2019-07-02 14:12:28 +02:00
Terrtia
bd0da210e7
fix: [helper dump_crawler] fix files not found
2019-06-26 11:51:26 +02:00
Terrtia
06ab66ff57
fix: [helper dump_crawler] fix empty dict
2019-06-26 11:42:19 +02:00
Terrtia
5377369032
chg: [helper] dump crawler history by daterange
2019-06-26 11:29:28 +02:00
Terrtia
87c5e0f9ee
Merge branch 'master' into user_management
2019-06-24 16:06:08 +02:00
Terrtia
3943b3ed9a
fix: [Update] add default update script
2019-06-24 15:51:42 +02:00
Terrtia
03ed3c63a1
chg: [helper] generate self signed certificates
2019-06-24 14:41:54 +02:00
Terrtia
bb65179e50
fix: [user_management] fix tokens duplicate + check user_acl_integrity + add login errors messages
2019-06-20 15:49:40 +02:00
Terrtia
6eec0aabfe
Merge branch 'master' into user_management
2019-06-19 15:11:44 +02:00
Terrtia
34e57fe7af
chg: [user_managemant] clean code + check password and email length
2019-06-19 15:00:25 +02:00
Terrtia
e65f465f02
fix: [TheHive feeder] create_alert: push all items tags
...
TODO check items status + add more item metadata
2019-06-19 10:42:36 +02:00
Terrtia
42d32ef310
fix: [domain history] fix domain status
2019-06-18 11:19:56 +02:00
Terrtia
e1bf59aded
chg: [UI crawler, show_domain] add domain history list + navigation
2019-06-18 10:58:37 +02:00
Terrtia
2b6cb86d55
fix: [PgpDump] catch UnicodeDecodeError error
2019-06-12 10:33:40 +02:00
Terrtia
b4f06c21f9
chg: [backend crawler] domains: download 1 archive by crawled (most recent)
2019-06-07 13:47:44 +02:00
Terrtia
3fe9d14e9f
chg: [user_management] create default admin user (temp passwd save in AIL_HOME) + change password UI + logout UI + create random password
2019-06-06 21:27:13 +02:00
Terrtia
99e35c51ec
Merge branch 'master' into user_management
2019-06-05 16:59:36 +02:00
Terrtia
17af705231
fix: [PgpDump] process large pgp blocks
2019-06-05 16:20:26 +02:00
fae3530fec
chg: [PgpDump] PMF fix - longer keys are truncated
...
TODO: Need to be updated by an async process handling the keys
2019-05-24 14:10:06 +02:00
Terrtia
4b0a7210b8
Merge branch 'master' of https://github.com/CIRCL/AIL-framework
2019-05-24 13:27:44 +02:00
Terrtia
eb9c81acef
chg: [correlation] clean files
2019-05-24 13:08:10 +02:00
Terrtia
4794cafe01
chg: [correlation] add cryptocurrency + refractor correlation
2019-05-24 12:02:43 +02:00
Terrtia
ff23a2bce2
chg: [Bitcoin] map cryptocurrency: bitcoin (DB pivot)
2019-05-21 16:14:09 +02:00
Terrtia
da5579875d
chg: [pgpdump] reprocess tagged items + fix pgpdump
2019-05-20 14:32:07 +02:00
Terrtia
49f7429c5f
chg: [Update] force update order
2019-05-20 13:02:09 +02:00
Terrtia
9639a51abd
chg: [PgpDump] fix graph + add new tags: pgp-signature pgp-public-key-block + avoid keys injection in pgp user_id
2019-05-20 11:57:49 +02:00
Thirion Aurélien
9c6a1ea8ec
Merge pull request #349 from kovacsbalu/fix-paste-encoding
...
Fix #314
2019-05-16 14:30:33 +02:00
kovacsbalu
f61d830678
Use default encoding error from redis.
2019-05-16 14:24:03 +02:00
kovacsbalu
7765ab92e0
Hopp, single quote :)
2019-05-15 10:00:51 +02:00
kovacsbalu
6092f482e6
Fix crawler rotation
...
Before this, crawler processed prioritized onions and after all starts prioritized regular.
2019-05-15 09:57:18 +02:00
Terrtia
6480744641
chg: [PgpDump] add PgpDump backend
...
TODO: UI
2019-05-14 17:49:31 +02:00
kovacsbalu
05079c143c
Fix #314
...
Replace char on redis encoding error.
Try to use local file on other error.
2019-05-14 16:06:20 +02:00
Terrtia
0389b9c23b
chg: [crawler] manual/auto crawler: always save screenshots
2019-05-13 14:24:16 +02:00
Terrtia
254441f193
chg: [crawler] manual/auto crawler: always save screenshots
2019-05-13 13:56:43 +02:00
Terrtia
2606220c2b
fix: [faup] fix new return types (bytes to str)
2019-05-06 13:38:13 +02:00
Terrtia
a4c03b4ba4
fix: [Crawler] force domains/subdomains lower case (rfc4343)
2019-05-06 11:46:20 +02:00
Terrtia
64ff94ce5f
chg: [user_management] create + check user password
2019-05-03 16:52:05 +02:00
Terrtia
1ab1a55a4f
chg: [UI] add basic user management
2019-05-02 17:31:14 +02:00
Steve Clement
9f65bc3788
chg: [travis] Travis has his own venv where it installs "stuff". Now we detect and us it in the launcher.
2019-05-01 21:31:47 +09:00
Terrtia
f30befa0eb
fix: [Onion] typo
2019-04-29 11:46:28 +02:00
Terrtia
a183695ddb
chg: [crawler UI] bootstrap 4 migration: show domains by daterange + bug fix hidden services tags
2019-04-26 15:14:29 +02:00
Terrtia
ff14fe6279
fix: [submit paste] remove item full path, use rel path
2019-04-26 10:22:02 +02:00
Terrtia
3e6fbc8f79
fix: [paste submit] update broken endpoint + get item date
2019-04-26 10:04:07 +02:00
Terrtia
3207949c5f
fix: [update v1.5] create update config file
2019-04-25 17:17:12 +02:00
Terrtia
41e6b4ec59
fix: [v1.5 background update screenshot] remove duplicate files
2019-04-25 14:39:38 +02:00
Terrtia
499c07a93e
fix: [update v1.5] remove unbuffer
2019-04-25 14:21:02 +02:00
Terrtia
fc2c1422ff
fix: [Crawler] unpack_url
2019-04-25 13:54:06 +02:00
Terrtia
50944b1d61
fix: [LAUNCH] fix typo
2019-04-24 16:37:44 +02:00
Terrtia
e6dca7f8bf
chg: [update v1.5] add background update: screenshots_crawled
2019-04-24 16:19:35 +02:00
Terrtia
9868833c77
chg: [crawled screenshot] use sha256 as filepath
2019-04-24 14:09:04 +02:00
Terrtia
99b9c95638
fix: [update] fix is_fork check
2019-04-23 15:30:47 +02:00
Terrtia
35c188829f
chg: [config update] refractor, auto update, create config if file doesnt exist
2019-04-23 15:04:45 +02:00
Terrtia
2a1cd4a009
chg: [Onion, crawler config] auto crawler: add config by url, fix onions tagging + filter subdomains
2019-04-23 11:15:34 +02:00
Terrtia
6fdf7c2123
chg: [UI crawler] status/remove auto crawler
2019-04-18 16:57:51 +02:00
Terrtia
ddaad5db93
chg: [v1.5 background update] create update stats
2019-04-17 17:07:09 +02:00
Terrtia
4bff603d4f
chg: [settings UI] add background update progress
2019-04-17 14:39:29 +02:00
Terrtia
68077316eb
chg: [git directory] add git helper
2019-04-16 17:26:35 +02:00
Terrtia
3b11df7f90
chg: [UI] add server Management, check if new update/version are available + background update status + link to new realease/commits
2019-04-16 17:24:59 +02:00
Terrtia
a3167a740a
fix: [Update DB] remove indexErrors
2019-04-12 17:32:17 +02:00
Terrtia
bf92a2f22f
fix: [background update] avoid multiple update on the same key
2019-04-12 16:07:40 +02:00
Terrtia
f2dc3bcbf4
fix: [Global] remove absolute path from item path
2019-04-12 15:13:36 +02:00
Terrtia
3a8631d44a
chg: [Update + Launcher] use background update + add new options in launcher
2019-04-12 15:06:24 +02:00
Terrtia
5c4db5f2f1
chg: [update] add update script Redis ARDB + add background update
2019-04-11 17:49:20 +02:00
Terrtia
3cc614a1ad
chg: [DB] update items tags metadata
2019-04-11 11:58:06 +02:00
Terrtia
d44acea04d
chg: [merge] merge update into advanced_crawler
2019-04-10 16:41:06 +02:00
Terrtia
2589fc2161
chg: [merge] merge master into update branche
2019-04-10 15:43:15 +02:00
Terrtia
b4bee29a33
Merge branch 'master' into advanced_crawler
2019-04-10 15:37:42 +02:00
Terrtia
f6d7d2ae16
chg: [Update] add debug
2019-04-02 09:46:04 +02:00
Thirion Aurélien
976b97bcd1
Merge pull request #338 from CIRCL/pre_update
...
Merge Pre update
2019-03-26 16:51:26 +01:00
Terrtia
044933aa33
chg: [Launch] remove old launchers
2019-03-26 16:47:10 +01:00
Terrtia
cee2b2486f
chg: [Launch] remove old launchers
2019-03-26 16:45:01 +01:00
Terrtia
59664efe45
Merge branch 'master' into advanced_crawler
2019-03-26 16:03:42 +01:00
Terrtia
5165a5de2f
fix: [Crawler] fix index + redis history key
2019-03-22 17:14:27 +01:00
Terrtia
f64c385343
chg: [Crawler] handle port: crawling + history
2019-03-22 16:48:07 +01:00
Terrtia
f4cdddbc7f
chg: [UI + item submit] remove browse paste from UI + fix pasteSubmit tagging
2019-03-21 11:50:53 +01:00
Terrtia
c2bc72c72e
chg: [Tags alertHandler] remove alertHandler + add quick tag search sidebar + use tag last_seen to display tagged items
2019-03-21 11:11:53 +01:00
Terrtia
a9e86231ad
chg: [Tags] add backend, daterange selector
2019-03-20 13:25:02 +01:00
Terrtia
65e6eae00d
chg: [UI preview items] b4 migration + new template + better display + decrease item loading time + add basic animation/transition
2019-03-19 13:34:41 +01:00
Terrtia
23d9cb2265
chg: [UI Tags search] bootstrap 4 migration
2019-03-14 17:04:55 +01:00
85d2ea578c
Merge pull request #335 from 5thColumn/docker
...
Add docker-compose support
2019-03-13 21:38:12 +01:00
Jason Hedden
23fb91d326
Add docker-compose support
...
This commit separates the AIL Framework services into individual Docker
containers that can be managed with docker-compose. It's intended to ease the
installation, development and troubleshooting procedures for all-in-one
deployments.
No changes to architecture or base code are required in this commit.
Changes to existing files:
.gitignore
* Ignore pystemon archive directory
Dockerfile:
* Add AIL shell environment variables
* Install pystemon and crawler pip requirements into AIL virtual environment
Docker-compose notes:
* All containers are participating in a service network `network_mode:
service:flask`. This allows the containers to share the same IP namespace
to accommodate hard-coded localhost entries.
* By default persistent data is saved to the following directories in the
local AIL framework git directory: PASTES, HASHS, CRAWLED_SCREENSHOTS,
pystemon/archives
This was tested with docker-ce on Ubuntu 16.04 and MacOS. A typical deployment
would look like:
```
git clone https://github.com/CIRCL/AIL-framework.git
cd AIL-framework
cp bin/packages/config.cfg.docker-compose-sample bin/packages/config.cfg
(optionally enable activate_crawler in config.cfg)
docker-compose build
docker-compose up -d
```
2019-03-11 19:46:34 -05:00
Terrtia
d3b3942ad7
chg: [Flask UI] refractor crawler frontend + show crawled domain
2019-03-06 15:05:58 +01:00
Terrtia
c0d72e7d2a
chg: [Crawler UI] Crawler major refractor (end) + basic UI for manual crawler
2019-02-26 14:50:48 +01:00
Terrtia
7b32d7f34e
chg: [Crawler] major refractor
2019-02-25 16:38:50 +01:00
Jason Hedden
9eae25bdf5
Fix current_selected amount and action vars
...
This commit fixes issue #268
```
File "./bin/ModulesInformationV2.py", line 319, in _setValue
action = current_selected_action + " " + str(current_selected_amount) + "x"
```
`current_selected_action` was defined twice. The second occurance should be `current_selected_amount`.
This also fixes the generic exception from failing when `current_selected_value` is not defined.
2019-02-22 12:39:27 -06:00
Terrtia
60f7645ac1
chg: [Crawler] refractor
2019-02-22 17:00:24 +01:00
Terrtia
3d36ddbc1e
fix: [CVE] fix stat regex
2019-02-21 10:51:05 +01:00
Terrtia
1114aa2ffd
chg: [CVE] add stat script
2019-02-21 09:58:10 +01:00
Terrtia
e5dca268a8
chg: [Crawler] refractor
2019-02-21 09:54:43 +01:00
Terrtia
da78d0552d
chg: [Crawler UI Tags] add tag by day + add crawler status + UI onion blacklist
2019-02-19 11:41:45 +01:00
Terrtia
6a8fa3d5d1
chg: [Tags] tags by day
2019-02-18 14:33:34 +01:00
Terrtia
709d5487b8
fix: [Onion] filter too many subdomain
2019-02-13 16:53:38 +01:00
Terrtia
b87707e8bc
fix: [Crawler] typo
2019-02-12 15:54:42 +01:00
Terrtia
37276e52a3
fix: [Crawler] typo
2019-02-12 15:53:40 +01:00
Terrtia
7cb03fc769
fix: [Crawler] typo
2019-02-12 15:51:19 +01:00
Terrtia
7a4989ce10
fix: [Global Crawler] max filename size
2019-02-12 15:45:58 +01:00
Terrtia
c2885589cf
chg: [UI] basic navbar + sidebar + refractor
2019-02-07 17:22:44 +01:00
Terrtia
516238025f
chg: [Crawler] add bootsrap4 src + refractor crawler
2019-02-05 17:16:44 +01:00
Thirion Aurélien
44c513dcbb
chg: [Crawler] add onion to blacklist
2019-01-31 16:56:45 +01:00
Xavier Mertens
43d08d0d3e
Cleanup
2019-01-30 22:36:10 +01:00
Xavier Mertens
1be91b88bf
Fix: reset alarm timeout
2019-01-30 22:30:07 +01:00
Xavier Mertens
0372a78704
Merge remote-tracking branch 'upstream/master'
2019-01-30 22:27:44 +01:00
Xavier Mertens
b099e2ae4a
Update LAUNCH.sh
2019-01-30 22:27:33 +01:00
Terrtia
92d192238b
fix: [Crawler] change max page crawled
2019-01-29 17:04:45 +01:00
Terrtia
6c7086f4eb
fix: [Crawler] first_seen
2019-01-29 16:54:39 +01:00
Terrtia
ad04cf7bb2
fix: [Onion] typo
2019-01-29 16:27:12 +01:00
Terrtia
88eaaeae93
chg: [Crawler] add priority queue, fix #263
2019-01-29 16:08:59 +01:00
Terrtia
c1b34bd99c
fix: [Crawler] limit max crawled pages
2019-01-29 15:38:00 +01:00
Terrtia
64ffdd52e8
fix: [Onion] loop
2019-01-29 14:39:57 +01:00
Terrtia
2dc0eca4a9
fix: [Crawler] fix crawler cache info
2019-01-29 12:09:19 +01:00
Terrtia
bb301a870c
fix: [Crawler] fix onion blacklist + add crawler info
2019-01-29 12:00:14 +01:00
Terrtia
535f44044d
fix: [Onion] add timeout signal
2019-01-29 09:46:03 +01:00
Thirion Aurélien
f4ba21e492
blacklist onion debian manpages
2019-01-14 11:08:53 +01:00
raggadhub
74c726d5fa
fix path_to_send path
...
Hi,
Missing slash between pastes_directory and paste in path_to_send = pastes_directory+paste
path error example:
/opt/AIL-framework/PASTESarchive/pastebin.com_pro/2019/01/07/cAgqsY2U.gz
2019-01-09 13:15:58 +01:00
Terrtia
b3b75ccbea
fix: [Crawler] Restart Splash on failure, limit unbound in memory cache (maxrss)
2019-01-04 15:51:08 +01:00
Terrtia
6ee4b7bf93
Merge branch 'master' into update
2018-12-19 15:23:34 +01:00
Terrtia
ca47764836
fix: [update v1.5]
2018-12-19 15:16:52 +01:00
Terrtia
198ee97d90
chg: [update 1.5] add update bash
2018-12-19 11:41:01 +01:00
Terrtia
2560260ce4
Merge branch 'master' into pre_update
2018-12-19 10:01:52 +01:00
Terrtia
4d04333f54
fix: [Splash server] add debug output
2018-12-19 09:30:24 +01:00
Terrtia
f842194c57
fix: [Crawler] retry when splash is not available
2018-12-17 16:04:12 +01:00
Terrtia
4e08aaa80f
chg: [UI Crawled Domains] show crawled domains by date range (options: domains tags, up domains, down domains)
...
todo: Shows domain history
2018-12-17 15:58:48 +01:00
Steve Clement
b60fd55efe
chg: [bash] Small PATH precedence issue.
2018-11-22 16:16:31 +09:00
Steve Clement
f82c899e7b
fix: [bash] fixed log_subscriber path
2018-11-22 15:48:12 +09:00
Steve Clement
88e561aa20
chg: [bash] Replace '' with ""
2018-11-22 15:41:45 +09:00
Steve Clement
33bc23a297
fix: [bash] evoking bash -c and splitting vars in this fashion wants to be changed in future revisions.
2018-11-22 15:22:35 +09:00
Steve Clement
6522cffd57
chg: [doc] Added note that redis-server is bundled during install
2018-11-22 15:10:57 +09:00
Steve Clement
41a34b3519
fix: [bash] As we instantiate from a subdirectory, we need to take this into account.
2018-11-22 15:08:59 +09:00
Steve Clement
e4ced241eb
chg: [python] Added necessary checks in LAUNCH.sh to be able to launch the script without doing bin/activate
...
chg: [doc] Added correct pystemon path
fix: [zmq] Small ZMQ test fix.
fix: [doc] Moved the print statement in feeder so it is less verbose but still useful
2018-11-22 14:57:24 +09:00
Terrtia
31a8dfe0b3
fix: [AIL] use only relative paths
...
pastes duplicates are fixed on the fly
2018-11-21 16:45:25 +01:00
Terrtia
cb41c54135
fix: [lib_refine] add dns configuration
2018-11-21 08:49:58 +01:00
Terrtia
4e680aabf0
chg: [Overview] add doc
2018-11-20 14:39:45 +01:00
Terrtia
aaa277b8a0
chg: [Update] add additonal update
2018-11-15 14:26:41 +01:00
Terrtia
f6e86582c8
chg: [Update] generate current version
2018-11-15 13:48:44 +01:00
Terrtia
108fdb868e
chg: update Overview
2018-11-15 10:39:41 +01:00
Terrtia
d9b776d58a
merge master
2018-11-14 15:55:42 +01:00
Terrtia
aed8d65aef
fix: [LAUNCH] add update in helper
2018-11-14 15:17:56 +01:00
Terrtia
912b977bb4
chg: [LAUNCH] update: launch + check BDD
2018-11-13 16:54:39 +01:00
Terrtia
94fcf66d20
fix: [Update] cleaning
2018-11-13 15:52:33 +01:00
Terrtia
347986a271
chg: [LAUNCH] add AIL update by default
2018-11-13 15:45:22 +01:00
Terrtia
7aff45c507
chg [Update] update AIL clone and fork
2018-11-13 14:54:56 +01:00
Terrtia
60ff0b9cf7
chg: [Update] add update script
2018-11-12 17:10:31 +01:00
Terrtia
b31618eb6a
chg: [Mixer] config default feeder name #291
2018-11-09 15:26:26 +01:00
Terrtia
c3903ab203
fix: [MISP_The_Hive_feeder] handle not saved pastes
2018-11-09 13:58:09 +01:00
Sami Mokaddem
62e46b6dc1
Merge pull request #287 from CIRCL/term_management
...
Term management upgrade
2018-11-08 14:10:25 +01:00
Terrtia
e3764ca575
fix: [MISP_the_Hive_feeder] force absolute path
2018-11-08 11:22:23 +01:00
Terrtia
979472df15
chg: [Terms] tag tracked terms
2018-11-06 13:38:37 +01:00
Terrtia
7ef5cc3205
fix:[MISP_The_Hive_feeder] typo
2018-11-05 16:40:25 +01:00
Terrtia
85f933fd81
chg: [NotificationHelper] add error traceback
2018-11-05 14:59:40 +01:00
Terrtia
877ac0636e
fix: typo
2018-11-05 14:30:03 +01:00
Terrtia
aa16c52cbf
fix: [NotificationHelper] add config
2018-11-05 14:20:12 +01:00
Terrtia
dad0365c67
fix:[MISP_The_Hive_feeder] set default values
2018-11-05 09:16:18 +01:00
Terrtia
5872cf9196
fix: [Scripts] Remove absolute path
2018-11-02 16:07:27 +01:00
Xavier Mertens
05de39e271
Fixed line counter (count all lines)
2018-11-02 15:55:55 +01:00
Xavier Mertens
ac95dd4ee8
Added timeout for Regex module
2018-11-02 15:50:38 +01:00
Xavier Mertens
9e7ca84581
Added timeout to avoid blocking regexes
2018-11-02 15:49:06 +01:00
Xavier Mertens
9bfb7a5fb6
Added module Regex.py
2018-10-26 17:13:26 +02:00
Terrtia
ea7fd8d54c
fix: [Duplicate] show duplicate with relative path
2018-10-24 15:47:03 +02:00
Terrtia
5afbc3179c
Merge branch 'master' of https://github.com/CIRCL/AIL-framework
2018-10-24 15:16:41 +02:00
Terrtia
8a6ac6be1e
fix: [Paste] create absolute path
2018-10-24 15:16:12 +02:00
Sami Mokaddem
d9bc2d6f44
Merge pull request #266 from kovacsbalu/fix-regexp-timeout
...
Stop regexp processing after timeout (60sec)
2018-10-09 14:48:08 +02:00
kovacsbalu
c1c40e3e56
Fix config section name
2018-10-09 11:30:04 +02:00
kovacsbalu
272b0fe960
RegexpForTerm timeout from config
...
Critical log removed
2018-10-09 09:32:32 +02:00
kovacsbalu
6d199f0150
Stop regexp processing after timeput (60sec)
...
Minor pep8 fixes
2018-10-08 11:25:32 +02:00
efaba35dec
Common onion addresses
2018-10-05 13:50:17 +02:00
ed9f9e5dce
Merge pull request #262 from kovacsbalu/feat-smtp-test
...
Test notification
2018-10-02 17:27:55 +02:00
Terrtia
44d19fbb80
fix: [pystemon-feeder] paste save path
2018-10-02 16:02:59 +02:00
59df5f0e12
chg: [config] Tor crawler disabled by default in sample configuration
...
(to match HOWTO when enabling it)
2018-10-02 14:17:58 +02:00
kovacsbalu
93a3c5779c
Test notification
...
Minor pep8 fixes
2018-10-01 15:56:48 +02:00
Terrtia
6328cc22b7
chg: [Crawler] add domains blacklist
2018-09-28 16:29:09 +02:00
Terrtia
82e6df4b94
chg: [Crawler] domains stats + logs + clean
2018-09-28 15:23:27 +02:00
Terrtia
ecb2857151
chg: [Crawler] css + limit splash RAM
2018-09-27 16:47:48 +02:00
Terrtia
e357dce59b
fix: [Crawler] detect splash connection to proxy error
2018-09-27 15:43:03 +02:00
Terrtia
04b9d9fc1d
chg: [Crawler] add docs
2018-09-27 11:14:29 +02:00
Terrtia
c49e871ba8
chg: [crawler] add infos
2018-09-26 16:34:27 +02:00
Terrtia
e4a5f66a10
Merge branch 'master' into onion_crawler
2018-09-24 16:35:04 +02:00
Terrtia
874824a589
fix: [Crawler] clean
2018-09-24 16:28:55 +02:00
Terrtia
8eca0e0778
fix: [Crawler] clean
2018-09-24 16:24:30 +02:00
Terrtia
50c81773e9
chg: [Crawler] add launcher and install
2018-09-24 16:23:14 +02:00
Terrtia
b575efcc46
fix: [pystemon] pystemon encoding
2018-09-21 14:16:06 +02:00
Terrtia
ead4dba0d8
fix: [MISP feeder] typo filename-config name
2018-09-21 11:02:08 +02:00
Terrtia
6edc1ddbeb
chg: [HiddenServices] get domain related to other domains
2018-09-21 10:34:06 +02:00
Terrtia
5eadc2755f
chg: [Flask] add prefix in config to flask routes
2018-09-20 10:38:19 +02:00
Terrtia
2e763ba0aa
fix: [ModulesInformationV2] fix #244
2018-09-19 15:45:06 +02:00
Terrtia
5732d7b134
fix: [hashDecoded] range_type_json
2018-09-19 15:36:33 +02:00
Terrtia
ebad2af1ae
fix: [import_dir] fix #251 , fix paste import
...
if the date directory tree don't exist, he will be generated
2018-09-19 13:38:31 +02:00
Terrtia
653463a8c4
fix: [LAUNCH.sh] make pystemon optionnal
...
AIL and Pystemon are independent, pystemon shouldn't be launch by default.
2018-09-19 09:52:27 +02:00
Terrtia
5b31b6e853
fix: [Crawler] save domain to crawl on splash error
2018-09-18 16:20:13 +02:00
Terrtia
f5b648d72a
pixelate paste screenshot
2018-09-18 11:03:40 +02:00
Terrtia
6aa1b99c2a
fix: [LAUNCH.sh] remove old decoders
2018-09-17 15:50:27 +02:00
Terrtia
0c63f2f24f
chg: [Crawler] catch server response
2018-09-17 15:35:06 +02:00
a6c72a677d
Merge pull request #243 from kovacsbalu/docker-with-pystemon
...
Docker build with pystemon
2018-09-12 21:50:11 +02:00
Terrtia
a94a0fe1a7
fix: [Install] add python requirements
2018-09-12 19:10:49 +02:00
Terrtia
204e996fc3
chg: [statistics] clean scripts
2018-09-12 11:21:11 +02:00
Terrtia
ea3d2c1977
Merge branch 'master' into statistic
2018-09-12 10:06:53 +02:00
Terrtia
6f0817365a
chg: [Crawler UI] display domain information
2018-09-12 09:55:49 +02:00
kovacsbalu
d19dd72f99
Install pystemon and start pystemon-feeder in docker
2018-09-03 16:01:39 +02:00
Terrtia
ca982e13e1
chg: [Crawled Domain] show crawled pastes by domain
2018-08-27 14:34:08 +02:00
Terrtia
823c33174a
Merge branch 'master' into onion_crawler
2018-08-27 11:20:51 +02:00
Terrtia
5c902640e9
Merge branch 'master' into onion_crawler
2018-08-27 11:05:25 +02:00
Terrtia
d42dd118a4
chg: [Domain crawled] add random screenshot
2018-08-27 11:02:39 +02:00
Terrtia
a586548047
Merge branch 'master' into base64
2018-08-24 10:35:28 +02:00
Terrtia
ced0b1e350
chg: [I2P] add default config
2018-08-24 10:24:03 +02:00
Terrtia
7e24943537
chg: [Crawler] crawler accept all kind of domains
2018-08-24 10:13:56 +02:00
Terrtia
e9580d6775
chg: [Crawler] change BDD, save i2p links
2018-08-21 15:54:53 +02:00
Terrtia
ed559d9f4a
chg: [Showpaste] add screenshot + improve onion db
2018-08-16 17:24:39 +02:00
Terrtia
6f2dbcb4e2
fix: [pystemon-feeder] fix #234
2018-08-13 10:22:20 +02:00
Terrtia
7652089433
chg: [Onion] change onion regex, fix crawler
2018-08-13 09:23:14 +02:00
Terrtia
8b1c10b38c
chg: [Onion] add onion splash crawler
2018-08-09 17:42:21 +02:00
Thirion Aurélien
000d5bf40a
fix: add miss config
2018-08-07 13:07:08 +02:00
Terrtia
d7545a46f0
chg: [Mixer] add submitted paste
2018-08-02 15:29:13 +02:00
Terrtia
449da63fd7
chg: [Keys] typo
2018-08-01 16:05:49 +02:00
Terrtia
67b0d973af
chg: [Keys] detect ssh2 private key
2018-08-01 16:04:06 +02:00
Terrtia
d0135c248d
add iban country statistic
2018-08-01 15:26:29 +02:00
Terrtia
940b96734a
Merge branch 'master' into statistic
2018-08-01 09:29:51 +02:00
Thirion Aurélien
f7f4422ff1
Merge pull request #232 from CIRCL/iban
...
Add Iban detection
2018-08-01 08:57:48 +02:00
Terrtia
b5c1f281e6
fix: typo
2018-07-31 10:58:23 +02:00
Terrtia
61e98ed8fb
clean
2018-07-31 10:51:18 +02:00
Terrtia
e455d244d8
chg: [statistics] add domain statistics
2018-07-31 10:36:29 +02:00
Terrtia
5dcb4ebb58
chg: [statistics] add script to generate pie chart png
2018-07-30 16:36:34 +02:00
Terrtia
09fbc363f1
chg: [sglinjection Phone] add tld statistic, fix phone regex
2018-07-30 11:56:50 +02:00
Terrtia
c20e7d5ab4
chg: [Mail Credential] add tld statistic
2018-07-30 10:19:26 +02:00
Terrtia
39f54d4420
chg: [Mail] add country statistic
2018-07-30 09:21:22 +02:00
Terrtia
009f87ee10
[fix: [UI-hash] fix graph node zoom
2018-07-27 11:27:41 +02:00
Terrtia
153f9f38cc
chg: [BankAccount] regex
2018-07-27 10:20:03 +02:00
Terrtia
57b631233b
add: [BankAccount] iban detection
2018-07-26 15:31:58 +02:00
Terrtia
5ad5939bd5
chg: display last logs
2018-07-25 16:48:44 +02:00
Terrtia
03a37af5cf
chg: [Hash] add reprocess regex order
2018-07-25 10:47:36 +02:00
Terrtia
8beb63b6a8
fix: [Queues] module with no subscriber
2018-07-24 15:57:18 +02:00
Terrtia
8c97c70a3f
Merge branch 'master' into base64
2018-07-24 15:21:32 +02:00
Terrtia
4b8c26dec8
fix: [hive auto export] fix #221
2018-07-24 14:46:35 +02:00
Terrtia
54e8750c30
fix: [LAUNCH] script checking
2018-07-24 12:18:41 +02:00
Terrtia
d4f0c14620
fix: [LAUNCH] use envar, add options (launch all, flask, killall)
2018-07-24 10:41:10 +02:00
Terrtia
9c03c40482
chg: [UI-Hash] add decoder pie
2018-07-23 11:11:52 +02:00
Terrtia
1fbd4109c2
chg: [UI-Hash] add decoded type on graph
2018-07-20 15:48:56 +02:00
Terrtia
6d6de3ae54
fix: remove experimental modules
2018-07-20 10:38:04 +02:00
Terrtia
bd5f83f0eb
chg: refractor base64 encoded to hash
2018-07-20 10:32:52 +02:00
Terrtia
9a8e37fb0b
fix: decoder bdd
2018-07-20 09:43:09 +02:00
Terrtia
0e67b56906
add: Decoder module
2018-07-19 16:52:09 +02:00
Terrtia
ad26f016e3
add: Decoder Module, decode binary, hex and base64
2018-07-19 16:50:42 +02:00
Terrtia
280e9d8af8
fix: Binary decoder base2 string to bytes
2018-07-18 13:47:06 +02:00
Terrtia
e25dd52d64
add: Binary decoder
2018-07-18 11:45:19 +02:00
Terrtia
0cac3146f1
fix: remove hardcoded mail url
2018-07-17 15:11:25 +02:00
Terrtia
fa38434e04
add: new mail body and starttls
2018-07-16 15:51:37 +02:00
Terrtia
8b245d17d9
fix: add LibInjection to LAUNCH.sh
2018-07-16 15:45:55 +02:00
Terrtia
6f69da0c0d
fix: paste submit css
2018-07-16 10:32:41 +02:00
Terrtia
fce324e370
fix: release tiemeout alarm, chg:icon graph node
2018-07-13 11:54:14 +02:00
Terrtia
87b7facba8
add: base 64 node graph
2018-07-12 17:07:17 +02:00
bui
bd5c42caeb
add LibInjection Module
2018-07-04 14:28:10 +02:00
bui
362f72fe43
initial LibInjection module
2018-07-04 14:27:54 +02:00
Thirion Aurélien
f8a88cecec
fix: hive_feeder hiveApi declaration
2018-07-02 15:23:21 +02:00
Terrtia
44524b25a4
fix submit paste
2018-06-29 15:30:01 +02:00
Terrtia
a0a639276c
fix submit paste
2018-06-29 15:29:02 +02:00
Terrtia
98d0235fc3
Merge branch 'master' into base64
2018-06-29 14:57:46 +02:00
Terrtia
f5bda70e82
Merge branch 'master' of https://github.com/CIRCL/AIL-framework
2018-06-29 13:51:08 +02:00
Terrtia
72146f938e
change: feeder name separator
2018-06-29 13:50:41 +02:00
Thirion Aurélien
c41a165a21
fix: file submission, content type
2018-06-29 12:08:46 +02:00
Terrtia
407aa14244
fix: catch file error in paste submission
2018-06-29 11:32:52 +02:00
Terrtia
16c700ddb4
new: add base64 display on paste + base64 statistics
2018-06-29 10:02:29 +02:00
Terrtia
225776c69c
fix: tp and fp, errors messages on paste submission
2018-06-20 10:02:26 +02:00
Terrtia
6a89fac309
fix hive and misp error
2018-06-19 16:39:49 +02:00
Terrtia
af9db53171
add tag fp and tp
2018-06-19 15:09:26 +02:00
Terrtia
86c2cfa04b
update ail-leak object
2018-06-18 17:23:41 +02:00
Terrtia
634bf6e552
add img + fix upload api
2018-06-18 17:16:22 +02:00
Terrtia
704e6f0ad1
change default submit tag, event-path map, publish events
2018-06-18 13:58:31 +02:00
Terrtia
aee1b50ef1
fix duplicate export attribute + tag export whitelist
2018-06-15 17:25:43 +02:00
Terrtia
31fc33ab03
misp event creation, auto and manual + the hive alert auto creation, manual case creation
2018-06-14 16:51:06 +02:00
Terrtia
8522d50308
modify navbar + decompress file password
2018-06-11 10:21:11 +02:00
Terrtia
c64588f60e
add feature, user can submit paste on the web interface
2018-06-08 16:49:20 +02:00
Terrtia
6143bc3dce
submit users input paste
2018-06-05 16:58:04 +02:00
Terrtia
6287981668
clean
2018-06-01 09:49:26 +02:00
Terrtia
884d655b0c
taxonomies and galaxies update via LAUNCH
2018-05-31 13:53:29 +02:00
Terrtia
93dcdf4f23
fix duplicate
2018-05-31 09:01:47 +02:00
Terrtia
0d164b6f4c
add tag js and css
2018-05-16 14:48:15 +02:00
Terrtia
b1b7d33c96
tags
2018-05-16 14:39:01 +02:00
Terrtia
f66a528bc2
fix Duplicate, save list of duplicates on disk + prevent empty hash creation
2018-05-15 23:28:47 +02:00
Terrtia
225fe76c96
fix duplicate + get raw paste
2018-05-11 16:07:30 +02:00
Terrtia
d7ba5533be
fix duplicate + fix issue #200
2018-05-11 13:25:45 +02:00
Terrtia
8599d7142d
python 3 upgrade instructions
2018-05-09 13:32:25 +02:00
Terrtia
142c1dc9b0
clean
2018-05-09 13:03:46 +02:00
Terrtia
3841c159a0
change leveldb to ardb
2018-05-07 14:50:40 +02:00
Terrtia
c17b9402f7
clean
2018-05-04 14:25:47 +02:00
Terrtia
96a3df42bc
decode with redis connection
2018-05-04 13:53:29 +02:00
Terrtia
ae8f575879
use feeder name
2018-05-03 16:32:25 +02:00
Terrtia
a900d5f08d
add python 2 to 3 bash + fix Duplicate + clean
2018-05-03 16:21:33 +02:00
Terrtia
c52caebe7c
improve Duplicate + tlsh + add time out handler + debug + clean
2018-05-02 17:07:10 +02:00
Terrtia
92977201fd
create json metadata for base64 module
2018-04-27 15:46:41 +02:00
Terrtia
c3428df5dc
add apiKeys module
2018-04-26 14:42:39 +02:00
Terrtia
29c368250e
create new dashbord alert
2018-04-24 17:03:10 +02:00
Terrtia
8738b7cf75
fix track term and avoid duplicate mail address
2018-04-24 16:44:37 +02:00
Terrtia
a7bd01ed9a
Merge branch 'python3' of https://github.com/CIRCL/AIL-framework into python3
2018-04-23 16:22:51 +02:00
Terrtia
4e55f6ee90
fix Onion module
2018-04-23 16:22:23 +02:00
Thirion Aurélien
ef3c335b62
Delete config.cfg.backup
2018-04-23 14:54:23 +02:00
Terrtia
f93fe9aeb2
add bitcoin module
2018-04-23 14:51:53 +02:00
Terrtia
3fc48db903
add Base64 module
2018-04-20 10:48:44 +02:00
Terrtia
41eb250278
fix python 3 compqtibility issues
2018-04-20 10:42:19 +02:00
Terrtia
8571a86344
frontend python 3.5 upgrade
2018-04-17 16:06:32 +02:00
Terrtia
19c90339e0
clean
2018-04-17 09:43:45 +02:00
Terrtia
d0261233aa
fix ModulesInnformationV2
2018-04-17 09:37:58 +02:00
Terrtia
15ef02fe53
fix python3 types issue
2018-04-16 17:00:44 +02:00
Terrtia
9e07a0ebdb
fix python3 types issue
2018-04-16 15:04:22 +02:00
Terrtia
9c82dd90ec
python 3 backend upgrade
2018-04-16 14:50:04 +02:00
Terrtia
3395b16873
fix incoming encoding file issue
2018-04-13 09:17:56 +02:00
Terrtia
347588fdec
python3.5 Mixer and Global module
2018-04-12 17:06:57 +02:00
Terrtia
4386cbbd34
fix travis
2018-04-11 10:48:22 +02:00
Terrtia
14583f702f
create first test
2018-04-11 10:14:33 +02:00
Terrtia
e9b88b0656
Keys module improvement
2018-04-05 11:40:34 +02:00
CD Nguyen
96259c53bc
refactor NotificationHelper code
2018-04-04 09:41:13 +02:00
CD Nguyen
e9e3f17363
support non TSL SMTP servers
2018-03-30 11:35:37 +02:00
Philipp Schmied
29608ab29b
Fixed e-mail notificatins for sets and regexes
2018-02-28 11:31:16 +01:00
Philipp Schmied
fd329aaba9
config.cfg.sample: Merged overwritten changes from upstream
2018-02-27 16:16:57 +01:00
Philipp Schmied
5b1f0b0212
Implemented email notifications (bin: config.cfg additions, email sending via analyzer scripts; var: Changes to add notifications via terms management); terms_management: Fixed click handlers not being added to all tracked terms.
2018-02-27 15:12:02 +01:00
abdf41fe23
Merge pull request #164 from mokaddem/showDiff
...
Show differences between duplicates
2018-02-21 22:25:16 +01:00
0263b25da8
Merge pull request #163 from mokaddem/iss160
...
Moving more to config
2018-02-21 22:24:50 +01:00
Sami Mokaddem
cbb19f95e4
fix: asciimatics introduced the '_required_height' property in the Label class. Just updating here to mirror the change
2018-02-16 14:35:09 +01:00
Sami Mokaddem
3e117195a2
fix: Added a backoff time if an IOError happens
2018-01-15 17:45:13 +01:00
Sami Mokaddem
9a66a6111b
Fixing no such file
2018-01-15 17:10:03 +01:00
Sami Mokaddem
c39e537cbe
Merge branch 'iss160' into showDiff
...
Preemptively preventing merge-conflict with branch iss160 (moving to
configuration)
2017-12-12 11:33:22 +01:00
Sami Mokaddem
05b22e8963
typo: added word + deleted trailing spaces
2017-12-12 11:19:37 +01:00
Sami Mokaddem
ef1bd16a72
typo: added word + deleted trailing spaces
2017-12-12 11:16:40 +01:00
Sami Mokaddem
304213cfa4
update: Possibility to change the authorized max line length in the config for the diff of duplicates
2017-12-12 11:00:26 +01:00
Sami Mokaddem
860acbf7cf
fix: Renamed Credentials into Credential sectiont title
...
config change: By default, zmq listen to both local and CRF
2017-12-12 09:29:15 +01:00
Sami Mokaddem
9d0d0b4303
update: Moved filtering operation (thresholds, number of matching in the
...
categ file) in the configuration file. It permits to better control the
flow of pastes.
Also set default mixer duplicate filtering to 3 (Do not filter)
2017-12-11 17:28:34 +01:00
Fabien Mathey
3105192312
Allow for multiple entries like 1 2 3 4 5
...
Should replace the need to type
1<Enter>2<Enter>
etc. and allows for
1 2 3 4 5<Enter>
2017-11-28 15:15:12 +01:00
71922f9def
add: Python script to test if the ZMQ feed works as expected
2017-11-28 09:46:06 +01:00
Sami Mokaddem
83e082e62a
update: removed useless comments
2017-11-24 08:57:41 +01:00
Sami Mokaddem
a0d07b1098
feature: try to send data to MISP if keys are provided, else do nothing
2017-11-23 13:47:02 +01:00
Sami Mokaddem
2a967c4d92
update/feature: Max number of duplicate push to MISP + duplicate are pushed as attachment
2017-11-23 07:13:44 +01:00
Sami Mokaddem
805171a8a5
Added a seemingly working version of ailleak to misp
2017-11-20 14:57:25 +01:00
Sami Mokaddem
e0b188c069
Better support of python3 in Helper.py
2017-11-20 14:55:36 +01:00
Sami Mokaddem
1e76e10c01
Support of add_object to a MISP instance
2017-11-20 12:12:53 +01:00
Sami Mokaddem
6f0227a78a
Starting support of python3
2017-11-20 12:12:06 +01:00
Sami Mokaddem
ee7759be91
Updated alertHandler and ailleakObject
2017-11-16 11:18:13 +01:00
Sami Mokaddem
5c7c07fbea
Merge branch 'launcher_update' into misp_obj
2017-11-16 10:09:05 +01:00
Sami Mokaddem
45a9897020
Added draft support of MISP ail-leak object
2017-11-16 09:52:37 +01:00
Sami Mokaddem
652b00f4b2
Renamed BrowseWarningPaste into alertHandler
2017-11-15 16:15:43 +01:00
Sami Mokaddem
ec7d2d49b2
Added seconds args in import_dir
2017-11-15 16:03:42 +01:00
Sami Mokaddem
13f1c676d6
Added '_AIL' to the end of screen session name to better check an kills sessions. Also, gracefully close redis server + Check 1 time if redis-servers are running.
2017-09-26 16:17:06 +02:00
=
a8dc9231c6
Typo2..
2017-08-24 16:43:42 +02:00
=
b2d4dd4a1f
Typo
2017-08-24 16:43:05 +02:00
=
74ff401e1d
Fixed bug: No CR between indexes in all_index.txt file
2017-08-24 16:41:30 +02:00
bb128a5e94
Merge pull request #138 from mokaddem/import_dir_update
...
Improvement of import_dir
2017-08-24 09:40:52 +02:00
=
429cc96d2a
Improvement of import_dir.py. Added feed name parameter, send the correct path (without potential ../) and gzipEncode it if needed.
2017-08-23 15:10:26 +02:00
=
e7c66d1aac
Updated individual launch scripts
2017-08-23 12:28:12 +02:00
=
d632335760
Automatically start redis_levelDB with the correct date
2017-08-23 10:16:56 +02:00
=
64ec44085d
Dynamically choose correct port for LevelDB in script
2017-08-22 17:52:15 +02:00
=
66d60a1f01
Added LevelDB Curve for 2017 (3017, used for backward compatibility)
2017-08-21 13:54:44 +02:00
=
92adea38d0
Semi-automated directory creation with date as name in LAUNCH and installing_deps
2017-08-21 13:48:02 +02:00
=
bc35747f00
Updated KillAll in LAUNCH
2017-08-21 13:19:26 +02:00
Mokaddem
c3632c7d86
Added operation mode 3 for Mixer (do not filter) and added possibility to recurse through dir for import_dir
2017-08-03 00:34:53 +02:00
Mokaddem
cbb2909156
Added comments
2017-07-20 10:24:48 +02:00
Mokaddem
05417f405c
Finished basic link web interface + solved some bugs
2017-07-20 10:04:30 +02:00
Mokaddem
ef741f85c0
Added similarity checks + started link webInterface
2017-07-19 11:52:06 +02:00
Mokaddem
882ab88b00
Draft module + web link
2017-07-18 16:57:15 +02:00
Mokaddem
9558aafe67
Prevent the loss of the 1 message in import_dir.py
2017-07-17 16:14:27 +02:00
Mokaddem
4d59871f00
Merged with previous changes for PR
2017-07-17 13:48:16 +02:00
4986352099
Merge pull request #120 from PaulSec/master
...
Added Python Phonenumbers module to parse numbers and retrieves stats…
2017-06-08 07:05:52 +02:00
446a97c47a
Merge pull request #115 from mokaddem/regexInTerms
...
New features for terms frequency
2017-06-08 07:04:17 +02:00
Mokaddem
3a4dcd691d
Improved description of modules inside the scripts
2017-05-09 11:13:16 +02:00
Paul A
747fff1df1
Added Python Phonenumbers module to parse numbers and retrieves stats from countrycode
2017-05-03 14:25:18 +02:00
Mokaddem
3ea0924120
Improved phone regex to handle end with 4 digits and presence of parenthesis.
2017-04-26 12:41:23 +02:00
Mokaddem
b60e164380
Added support of onion in browseWarningPaste
2017-04-26 11:47:02 +02:00
Mokaddem
318a557c56
Added possibility to keep a backup of the old config + check if config is valid before starting scripts.
2017-04-25 09:32:05 +02:00
Mokaddem
72c1d903eb
Merge branch 'newIndexer' into auto-conf
2017-04-25 08:14:31 +02:00
Mokaddem
eccc8fc88b
modified check_size timeout + utilisation of shutil instead of os.system()
2017-04-19 16:37:04 +02:00
Mokaddem
da081bae42
Merge branch 'regexInTerms' into production
2017-04-18 16:59:40 +02:00
Mokaddem
2d907ae784
Merge branch 'newIndexer' into auto-conf
2017-04-18 16:20:00 +02:00
Mokaddem
b0f1cf84b7
remplaced usage of global variable by a function param
2017-04-18 16:02:22 +02:00
Mokaddem
6e8509a0ba
Removed wierd line
2017-04-18 15:57:07 +02:00
Mokaddem
c8baabd882
Updated TermsFrequency related modules + Fixed bugs (encoding, behaviors, ...)
2017-04-18 15:28:21 +02:00
Mokaddem
2da4c572c7
New feature in Terms: regex and set of words support (draft)
2017-03-28 17:42:44 +02:00
Mokaddem
6e80eb0954
Typo
2017-03-28 09:55:59 +02:00
Mokaddem
c79d4d65c8
Keep original section order while updating
2017-03-28 09:54:24 +02:00
Mokaddem
e225090e07
Added script that checks configuration and may auto-fix it + added missing field in config.cfg.sample
2017-03-15 16:36:51 +01:00
Mokaddem
be93af6f5a
Merge branch 'terms-improvements'
2017-03-15 14:55:44 +01:00
Mokaddem
3261eca836
Fixed all_index newline bug + added print of what's happening
2017-03-15 14:29:49 +01:00
Mokaddem
5b7d047622
Changed os.system by os.mkdir + fixed bug during rotation
2017-03-15 14:05:13 +01:00
Mokaddem
adc37ab50e
Indexer max size now in config
2017-03-15 12:14:41 +01:00
Mokaddem
4005b72180
Added backward support + renamed index with timestamp
2017-03-15 11:51:35 +01:00
Mokaddem
248469d61e
Indexer now create an index_dir when it became too large (search in index not updated yet)
2017-03-14 10:37:31 +01:00
1540b719ed
Merge pull request #108 from mokaddem/newModuleInformation
...
New version of ModuleInformation
2017-03-14 08:29:09 +01:00
Mokaddem
3c9cdac39a
Merge branch 'terms-improvements' into production
2017-02-28 16:51:39 +01:00
Mokaddem
dadc872f9f
Merge branch 'newModuleInformation' into production
2017-02-28 16:51:08 +01:00
Mokaddem
a2849c1bcd
Added a catch for psutils if process get stopped by user
2017-02-28 16:13:01 +01:00
Mokaddem
4071bf2e91
fixed a bug in manage_term (usage of loop.index0) + fixed bug in top_set where graph was not displaying the correct per_paste value
2017-02-28 15:01:48 +01:00
Mokaddem
a1c47d7a13
Merge branch 'production' into terms-improvements
2017-02-28 11:15:56 +01:00
Mokaddem
fd4feb374f
Merge branch 'newModuleInformation' into production
2017-02-28 11:12:06 +01:00
Mokaddem
75b4037519
Fixed bug - If we move the cursor from a line that has dissapear
2017-02-28 11:11:17 +01:00
Mokaddem
d6e02591af
Merge branch 'newModuleInformation' into production
2017-02-28 10:23:45 +01:00
Mokaddem
975c1a37d9
ModuleInformation create the all_modules files if missing
2017-02-28 10:23:22 +01:00
Mokaddem
ccdf256383
Added possibility to show or not in dashboard text that is logged
2017-02-28 10:21:27 +01:00
Mokaddem
39104dea5e
Merge branch 'bugfix_categ' into production
2017-02-28 09:56:25 +01:00
Mokaddem
26eb5fdff0
Fixed bug closing parenthesis in categ + changed behavior of sending to browseWarningPaste in Mail
2017-02-28 09:23:32 +01:00
Mokaddem
4a0dac1fae
Merge branch 'production' into terms-improvements
2017-02-15 16:31:22 +01:00
Mokaddem
a696ccf83b
Merge branch 'newModuleInformation' into production
2017-02-15 16:30:44 +01:00
Mokaddem
85a6f6fa63
Fixed a bug (str+float)
2017-02-15 16:30:31 +01:00
Mokaddem
25adae0104
Added possibility to select in top_terms the data per_paste or not (need module testing)
2017-02-15 16:29:02 +01:00
Mokaddem
e98e92b0d4
Merge branch 'fix-linkPaste-debug' into production
2017-02-15 13:43:03 +01:00
Mokaddem
c552aced42
Merge branch 'newModuleInformation' into production
2017-02-15 13:42:46 +01:00
Mokaddem
1b0d9d9507
Fixed str and int concat + Added sorting by queue name in queue not running
2017-02-15 13:42:21 +01:00
Mokaddem
74e0bc7acb
Merge branch 'newModuleInformation' into production
2017-02-15 11:22:50 +01:00
Mokaddem
7fc9ba18d7
Added support of ENTER for navigation
2017-02-15 11:01:11 +01:00
Mokaddem
b84194542d
Moved logging into a function
2017-02-15 10:57:41 +01:00
Mokaddem
0fce3f5d65
Renamed variables + added comments
2017-02-15 10:37:59 +01:00
Mokaddem
9ca3e79e70
Removed useless comments + commented
2017-02-15 10:19:03 +01:00
Mokaddem
9199ea3f0e
Added comments
2017-02-15 10:04:51 +01:00
Mokaddem
ff1df07b5d
Added comments and moved portion of code for slightly better visibility
2017-02-15 09:43:18 +01:00
Mokaddem
b272b1b07b
Added a way to quit application with 'q' more or less properly
2017-02-15 09:35:03 +01:00
Mokaddem
0254c51164
Added end of paste marker + handling screen resize
2017-02-15 08:26:48 +01:00
Mokaddem
0b9609e4a9
Merge branch 'newModuleInformation' into production
2017-02-14 17:22:06 +01:00
Mokaddem
444f6071b7
Fixed screen bug by adding an input before starting asciimatics + improved path localisation in the logs from modules
2017-02-14 17:21:52 +01:00
Mokaddem
b04bac4731
Merge branch 'newModuleInformation' into production
2017-02-14 16:08:42 +01:00
Mokaddem
d520c45a15
Added support of display of the last paste processed by a module
2017-02-14 16:05:59 +01:00
Mokaddem
d3b4409156
Added path for debug in SQLInjection
2017-02-14 11:23:34 +01:00
Mokaddem
135a1457e0
Merge branch 'fix-linkPaste-debug' into production
2017-02-14 11:12:22 +01:00
Mokaddem
d4da3a316f
Added path of the paste in the log of Categ.py
2017-02-14 10:59:47 +01:00
Mokaddem
303575b2a1
Merge remote-tracking branch 'upstream/master' into production
2017-02-14 09:31:18 +01:00
2f4e5b9016
Merge branch 'master' of github.com:CIRCL/AIL-framework
2017-01-26 07:16:44 +00:00
cf903cc212
(partially fix #90 too) using a simple Alarm (SIGNAL) when exec-timeout
...
Introducing a timer (in this case 60 seconds) to ensure that the
execution time of the analysis takes less than 60 seconds. This
is a simple and standard POSIX signal handler. If the timeout
is reached, the module will process the next one.
This approach fixes the specific issues we have currently
with some inputs where the sentiment analysis takes too much time. This
fix should be improved and be more generic:
- Introducing statistics of content which timeouts.
- Keeping a list/queue to further process those files using a different
analysis approach. Maybe a set of "dirty" processes to handle the edge cases
and to not impact the overall processing and analysis.
- Make the timer configurable per module (at least for this one).
2017-01-26 07:11:18 +00:00
Mokaddem
c422db6e6d
Merge remote-tracking branch 'upstream/master' into production
2017-01-17 16:38:48 +01:00
mokaddem
b3a3616b0b
Merge pull request #102 from ptitdoc/bugfix1
...
Bug fixes in Workers and improvement of pystemon-feeder
2017-01-17 16:35:32 +01:00
Mokaddem
ea423fde0f
Added requirement, better behaviors and confirmation box
2017-01-17 15:58:16 +01:00
Mokaddem
692f13d18b
Improved module + Added support of click and keyboard navigation for killing and starting modules
2017-01-17 14:31:06 +01:00
Mokaddem
3b8149d840
Added action button and support of logs
2017-01-16 17:08:48 +01:00
Mokaddem
0ecfaf00f5
Added color in front of queue related to its state
2017-01-16 15:30:33 +01:00
Mokaddem
af3241c008
Overwritted listbox update function to add formating
2017-01-16 14:41:02 +01:00
Mokaddem
30f303cb99
Added new ModuleInformation
2017-01-16 12:18:23 +01:00
Olivier MEDOC
16044d4d36
pystemon-feeder: add the option bind in configuration file instead of address
...
This is essentially because multiple feeds can be used in the address
configuration variable.
2017-01-13 14:58:18 +01:00
3b101ea8f5
(partially) Fix #91 using a simple Alarm (SIGNAL) when exec-timeout
...
Introducing a timer (in this case 5 seconds) to ensure that the
execution time of the tokenizer takes less than 5 seconds. This
is a simple and standard POSIX signal handler.
This approach fixes the specific issues we have currently
with some inputs where the tokenization takes too much time. This
fix should be improved and be more generic:
- Introducing statistics of content which timeouts.
- Keeping a list/queue to further process those files using a different
tokenizer approach. Maybe a set of "dirty" processes to handle the edge cases
and to not impact the overall processing and analysis.
- Make the timer configurable per module (at least for this one).
2017-01-12 07:32:55 +00:00
Olivier MEDOC
9f9c265cb0
SentimentAnalysis: use lexicon file path from the ail configuration file
2017-01-11 11:07:08 +01:00
ptitdoc
c0fac82031
Add pystemon path in configuration sample
2017-01-10 18:33:46 +01:00
Olivier MEDOC
83db40104c
import pystemon-feeder configuration from the central configuration file
2017-01-10 18:26:36 +01:00
Olivier MEDOC
3dc014dad9
fix invalid variable propagation
2017-01-10 18:26:24 +01:00
Olivier MEDOC
68ca20db8f
fix to use the new pycountry API
2017-01-10 18:26:17 +01:00
1950a2dc0e
Merge pull request #99 from mokaddem/more-feed
...
Usage of sha1 to filter paste content in Mixer.py
2017-01-09 17:06:37 +01:00
Mokaddem
500608d0a1
Usage of sha1 to filter paste content in Mixer.py
2017-01-09 17:01:10 +01:00
f5adb65cb3
Merge pull request #98 from mokaddem/more-feed
...
Config 2017
2017-01-09 15:39:53 +01:00
Mokaddem
409c5b8b42
Added 2017
2017-01-09 15:36:50 +01:00
a96ba40d94
Merge pull request #96 from mokaddem/more-feed
...
Handling of multiple feeders
2017-01-09 15:29:28 +01:00
Mokaddem
0e39f516a9
Added sleep in helper for multiple listening interfaces (reduce useless work), Added a simple way to pre-process feed before sending it to the global feed.
2017-01-09 14:12:26 +01:00
Mokaddem
3a298f0c7c
Commented Mixer.py for further multiple feeds support.
2016-12-26 16:23:18 +01:00
Mokaddem
1c2169d3bc
Improved interface for multiple feeds and refresh_script.
2016-12-26 16:16:44 +01:00
Mokaddem
97292e0899
Updated web interface to handle new mixer module. and fixed one dependency bug.
2016-12-23 15:44:46 +01:00
Mokaddem
e70b9cd15c
Added basic mixer with confirugable behavior. It handles muliple feeders and performs some basic stats on them.
2016-12-23 10:31:26 +01:00
ddfe3cf66a
Merge pull request #95 from mokaddem/module_handling
...
Module handling
2016-12-22 22:18:31 +01:00
3ce850e093
Merge pull request #94 from mokaddem/website-refacto
...
Website refacto, support and fixes
2016-12-22 22:17:58 +01:00
Mokaddem
9758579753
Merge branch 'module_handling' into production
2016-12-22 10:24:30 +01:00
Mokaddem
7438f16c63
Added new config
2016-12-22 10:06:35 +01:00
Mokaddem
73938cf41b
Prepared moduleInfo to handle module click with curses (will come back later).
2016-12-22 08:45:53 +01:00
Mokaddem
d37b243e1d
Added legend + added more info if restarting disabled
2016-12-09 15:13:47 +01:00
Mokaddem
d3ed1bbf40
Merge branch 'website-improvement' into production
2016-12-08 10:05:55 +01:00
Mokaddem
73d4f9e082
Webstats should correctly updates top_progression_zset (Not fully tested because not enough data. Will be tested latter)
2016-12-08 10:05:07 +01:00
Mokaddem
570324060e
terms top_sets correctly supports blacklisted terms
2016-12-08 09:13:31 +01:00
412b80c44d
Merge pull request #88 from mokaddem/Issue-Feeding-AIL
...
Pystemon tuto
2016-11-26 12:45:28 +01:00
Mokaddem
d831a79614
Do not try to restart not running queue if autokill is not set
2016-11-25 16:00:46 +01:00
Mokaddem
4bf45eea36
ModuleInformation sort by name as well as elapsed time
2016-11-25 14:44:27 +01:00
Mokaddem
866be4e884
Added row coloring in ModuleInformation
2016-11-25 14:12:09 +01:00
Mokaddem
a3255d168c
ModuleInfo correctly handle CurveManageTopSets,
...
Changed queue coloring in dashboard,
ModuleInformation now have a history of executed command and better handle module killing.
2016-11-25 11:54:16 +01:00
Mokaddem
224fbc8084
Better handle stuck modules. Differentiate between not running and no info and tries to restart stuck ones.
2016-11-24 16:58:32 +01:00
Mokaddem
5d269ea1ee
Added date in Duplicate for better distinguish duplicate. Also, added a timeout for the dataTable in duplicate inside the modal.
2016-11-24 15:05:29 +01:00
Mokaddem
ac254e0e63
Added support of browsing concerned paste in dashboard for all modules + Show in the page what was the concerned module
2016-10-27 11:50:24 +02:00
Mokaddem
1826b170ec
Added support of local paste view in dashboard for Mails event only
2016-10-27 11:27:26 +02:00
Mokaddem
5a4ea734a2
Fixed path issues in doc + added dependencies related to ssdeep
2016-10-14 14:26:33 +02:00
Mokaddem
4eafca2936
Added pystemon instruction section
2016-10-13 17:13:08 +02:00
eb14005fcf
Fix: as JSON decode needs a string, the empty list is a string.
2016-08-29 12:08:12 +00:00
3b6e3e156e
Merge branch 'master' of github.com:CIRCL/AIL-framework
2016-08-26 15:57:45 +00:00
Mokaddem
0c760d763b
Added support of PID in ModuleInformation and index.html
2016-08-26 15:27:37 +02:00
758f2c5334
Merge branch 'master' of github.com:CIRCL/AIL-framework
2016-08-26 08:30:11 +00:00
Mokaddem
494980c48e
Added comment in moduleInformation
2016-08-26 10:07:06 +02:00
Mokaddem
8b33b4e7ba
Added not running module info + swapped order of module launch
2016-08-26 09:21:02 +02:00
Mokaddem
813cc7031a
Changed lots of code into zrank
2016-08-25 17:48:12 +02:00
Mokaddem
aaac798db9
updated individual launch script
2016-08-25 16:51:57 +02:00
Mokaddem
78c991539e
Added number of same module running in ModuleInformation
2016-08-25 16:07:47 +02:00
1ab08d4a05
Merge branch 'master' of github.com:CIRCL/AIL-framework
2016-08-25 12:33:31 +00:00
Mokaddem
f9bf00f022
Removed autokill feature if not specified
2016-08-25 11:09:31 +02:00
Mokaddem
2f136595fb
Removed autokill feature if not specified
2016-08-25 11:05:18 +02:00
Mokaddem
9146feab88
Fixed bug in WebStats not deleting correct key + fixed typo in CurveSetManager preventing connecting to redis + modified display in moduleInfo
2016-08-25 10:43:04 +02:00
Mokaddem
3313064e51
removed useless import.
2016-08-24 17:32:50 +02:00
Mokaddem
64bf481480
Added moduleInfo + Renamed modules
2016-08-24 17:28:39 +02:00
Mokaddem
8fb552b7da
ModuleInfo support autokill and auto restart after logging.
2016-08-24 16:52:01 +02:00