Commit graph

766 commits

Author SHA1 Message Date
Terrtia
d42dd118a4
chg: [Domain crawled] add random screenshot 2018-08-27 11:02:39 +02:00
Terrtia
a586548047
Merge branch 'master' into base64 2018-08-24 10:35:28 +02:00
Terrtia
ced0b1e350
chg: [I2P] add default config 2018-08-24 10:24:03 +02:00
Terrtia
7e24943537
chg: [Crawler] crawler accept all kind of domains 2018-08-24 10:13:56 +02:00
Terrtia
e9580d6775
chg: [Crawler] change BDD, save i2p links 2018-08-21 15:54:53 +02:00
Terrtia
ed559d9f4a
chg: [Showpaste] add screenshot + improve onion db 2018-08-16 17:24:39 +02:00
Terrtia
6f2dbcb4e2
fix: [pystemon-feeder] fix #234 2018-08-13 10:22:20 +02:00
Terrtia
7652089433
chg: [Onion] change onion regex, fix crawler 2018-08-13 09:23:14 +02:00
Terrtia
8b1c10b38c
chg: [Onion] add onion splash crawler 2018-08-09 17:42:21 +02:00
Thirion Aurélien
000d5bf40a
fix: add miss config 2018-08-07 13:07:08 +02:00
Terrtia
d7545a46f0
chg: [Mixer] add submitted paste 2018-08-02 15:29:13 +02:00
Terrtia
449da63fd7
chg: [Keys] typo 2018-08-01 16:05:49 +02:00
Terrtia
67b0d973af
chg: [Keys] detect ssh2 private key 2018-08-01 16:04:06 +02:00
Terrtia
d0135c248d
add iban country statistic 2018-08-01 15:26:29 +02:00
Terrtia
940b96734a
Merge branch 'master' into statistic 2018-08-01 09:29:51 +02:00
Thirion Aurélien
f7f4422ff1
Merge pull request #232 from CIRCL/iban
Add Iban detection
2018-08-01 08:57:48 +02:00
Terrtia
b5c1f281e6
fix: typo 2018-07-31 10:58:23 +02:00
Terrtia
61e98ed8fb
clean 2018-07-31 10:51:18 +02:00
Terrtia
e455d244d8
chg: [statistics] add domain statistics 2018-07-31 10:36:29 +02:00
Terrtia
5dcb4ebb58
chg: [statistics] add script to generate pie chart png 2018-07-30 16:36:34 +02:00
Terrtia
09fbc363f1
chg: [sglinjection Phone] add tld statistic, fix phone regex 2018-07-30 11:56:50 +02:00
Terrtia
c20e7d5ab4
chg: [Mail Credential] add tld statistic 2018-07-30 10:19:26 +02:00
Terrtia
39f54d4420
chg: [Mail] add country statistic 2018-07-30 09:21:22 +02:00
Terrtia
009f87ee10
[fix: [UI-hash] fix graph node zoom 2018-07-27 11:27:41 +02:00
Terrtia
153f9f38cc
chg: [BankAccount] regex 2018-07-27 10:20:03 +02:00
Terrtia
57b631233b
add: [BankAccount] iban detection 2018-07-26 15:31:58 +02:00
Terrtia
5ad5939bd5
chg: display last logs 2018-07-25 16:48:44 +02:00
Terrtia
03a37af5cf
chg: [Hash] add reprocess regex order 2018-07-25 10:47:36 +02:00
Terrtia
8beb63b6a8
fix: [Queues] module with no subscriber 2018-07-24 15:57:18 +02:00
Terrtia
8c97c70a3f
Merge branch 'master' into base64 2018-07-24 15:21:32 +02:00
Terrtia
4b8c26dec8
fix: [hive auto export] fix #221 2018-07-24 14:46:35 +02:00
Terrtia
54e8750c30
fix: [LAUNCH] script checking 2018-07-24 12:18:41 +02:00
Terrtia
d4f0c14620
fix: [LAUNCH] use envar, add options (launch all, flask, killall) 2018-07-24 10:41:10 +02:00
Terrtia
9c03c40482
chg: [UI-Hash] add decoder pie 2018-07-23 11:11:52 +02:00
Terrtia
1fbd4109c2
chg: [UI-Hash] add decoded type on graph 2018-07-20 15:48:56 +02:00
Terrtia
6d6de3ae54
fix: remove experimental modules 2018-07-20 10:38:04 +02:00
Terrtia
bd5f83f0eb
chg: refractor base64 encoded to hash 2018-07-20 10:32:52 +02:00
Terrtia
9a8e37fb0b
fix: decoder bdd 2018-07-20 09:43:09 +02:00
Terrtia
0e67b56906
add: Decoder module 2018-07-19 16:52:09 +02:00
Terrtia
ad26f016e3
add: Decoder Module, decode binary, hex and base64 2018-07-19 16:50:42 +02:00
Terrtia
280e9d8af8
fix: Binary decoder base2 string to bytes 2018-07-18 13:47:06 +02:00
Terrtia
e25dd52d64
add: Binary decoder 2018-07-18 11:45:19 +02:00
Terrtia
0cac3146f1
fix: remove hardcoded mail url 2018-07-17 15:11:25 +02:00
Terrtia
fa38434e04
add: new mail body and starttls 2018-07-16 15:51:37 +02:00
Terrtia
8b245d17d9
fix: add LibInjection to LAUNCH.sh 2018-07-16 15:45:55 +02:00
Terrtia
6f69da0c0d
fix: paste submit css 2018-07-16 10:32:41 +02:00
Terrtia
fce324e370
fix: release tiemeout alarm, chg:icon graph node 2018-07-13 11:54:14 +02:00
Terrtia
87b7facba8
add: base 64 node graph 2018-07-12 17:07:17 +02:00
bui
bd5c42caeb add LibInjection Module 2018-07-04 14:28:10 +02:00
bui
362f72fe43 initial LibInjection module 2018-07-04 14:27:54 +02:00
Thirion Aurélien
f8a88cecec
fix: hive_feeder hiveApi declaration 2018-07-02 15:23:21 +02:00
Terrtia
44524b25a4
fix submit paste 2018-06-29 15:30:01 +02:00
Terrtia
a0a639276c
fix submit paste 2018-06-29 15:29:02 +02:00
Terrtia
98d0235fc3
Merge branch 'master' into base64 2018-06-29 14:57:46 +02:00
Terrtia
f5bda70e82
Merge branch 'master' of https://github.com/CIRCL/AIL-framework 2018-06-29 13:51:08 +02:00
Terrtia
72146f938e
change: feeder name separator 2018-06-29 13:50:41 +02:00
Thirion Aurélien
c41a165a21
fix: file submission, content type 2018-06-29 12:08:46 +02:00
Terrtia
407aa14244
fix: catch file error in paste submission 2018-06-29 11:32:52 +02:00
Terrtia
16c700ddb4
new: add base64 display on paste + base64 statistics 2018-06-29 10:02:29 +02:00
Terrtia
225776c69c
fix: tp and fp, errors messages on paste submission 2018-06-20 10:02:26 +02:00
Terrtia
6a89fac309
fix hive and misp error 2018-06-19 16:39:49 +02:00
Terrtia
af9db53171
add tag fp and tp 2018-06-19 15:09:26 +02:00
Terrtia
86c2cfa04b
update ail-leak object 2018-06-18 17:23:41 +02:00
Terrtia
634bf6e552
add img + fix upload api 2018-06-18 17:16:22 +02:00
Terrtia
704e6f0ad1
change default submit tag, event-path map, publish events 2018-06-18 13:58:31 +02:00
Terrtia
aee1b50ef1
fix duplicate export attribute + tag export whitelist 2018-06-15 17:25:43 +02:00
Terrtia
31fc33ab03
misp event creation, auto and manual + the hive alert auto creation, manual case creation 2018-06-14 16:51:06 +02:00
Terrtia
8522d50308
modify navbar + decompress file password 2018-06-11 10:21:11 +02:00
Terrtia
c64588f60e
add feature, user can submit paste on the web interface 2018-06-08 16:49:20 +02:00
Terrtia
6143bc3dce
submit users input paste 2018-06-05 16:58:04 +02:00
Terrtia
6287981668
clean 2018-06-01 09:49:26 +02:00
Terrtia
884d655b0c
taxonomies and galaxies update via LAUNCH 2018-05-31 13:53:29 +02:00
Terrtia
93dcdf4f23
fix duplicate 2018-05-31 09:01:47 +02:00
Terrtia
0d164b6f4c
add tag js and css 2018-05-16 14:48:15 +02:00
Terrtia
b1b7d33c96
tags 2018-05-16 14:39:01 +02:00
Terrtia
f66a528bc2
fix Duplicate, save list of duplicates on disk + prevent empty hash creation 2018-05-15 23:28:47 +02:00
Terrtia
225fe76c96
fix duplicate + get raw paste 2018-05-11 16:07:30 +02:00
Terrtia
d7ba5533be
fix duplicate + fix issue #200 2018-05-11 13:25:45 +02:00
Terrtia
8599d7142d
python 3 upgrade instructions 2018-05-09 13:32:25 +02:00
Terrtia
142c1dc9b0
clean 2018-05-09 13:03:46 +02:00
Terrtia
3841c159a0
change leveldb to ardb 2018-05-07 14:50:40 +02:00
Terrtia
c17b9402f7
clean 2018-05-04 14:25:47 +02:00
Terrtia
96a3df42bc
decode with redis connection 2018-05-04 13:53:29 +02:00
Terrtia
ae8f575879
use feeder name 2018-05-03 16:32:25 +02:00
Terrtia
a900d5f08d
add python 2 to 3 bash + fix Duplicate + clean 2018-05-03 16:21:33 +02:00
Terrtia
c52caebe7c
improve Duplicate + tlsh + add time out handler + debug + clean 2018-05-02 17:07:10 +02:00
Terrtia
92977201fd create json metadata for base64 module 2018-04-27 15:46:41 +02:00
Terrtia
c3428df5dc add apiKeys module 2018-04-26 14:42:39 +02:00
Terrtia
29c368250e create new dashbord alert 2018-04-24 17:03:10 +02:00
Terrtia
8738b7cf75 fix track term and avoid duplicate mail address 2018-04-24 16:44:37 +02:00
Terrtia
a7bd01ed9a Merge branch 'python3' of https://github.com/CIRCL/AIL-framework into python3 2018-04-23 16:22:51 +02:00
Terrtia
4e55f6ee90 fix Onion module 2018-04-23 16:22:23 +02:00
Thirion Aurélien
ef3c335b62
Delete config.cfg.backup 2018-04-23 14:54:23 +02:00
Terrtia
f93fe9aeb2 add bitcoin module 2018-04-23 14:51:53 +02:00
Terrtia
3fc48db903 add Base64 module 2018-04-20 10:48:44 +02:00
Terrtia
41eb250278 fix python 3 compqtibility issues 2018-04-20 10:42:19 +02:00
Terrtia
8571a86344 frontend python 3.5 upgrade 2018-04-17 16:06:32 +02:00
Terrtia
19c90339e0 clean 2018-04-17 09:43:45 +02:00
Terrtia
d0261233aa fix ModulesInnformationV2 2018-04-17 09:37:58 +02:00
Terrtia
15ef02fe53 fix python3 types issue 2018-04-16 17:00:44 +02:00
Terrtia
9e07a0ebdb fix python3 types issue 2018-04-16 15:04:22 +02:00
Terrtia
9c82dd90ec python 3 backend upgrade 2018-04-16 14:50:04 +02:00
Terrtia
3395b16873 fix incoming encoding file issue 2018-04-13 09:17:56 +02:00
Terrtia
347588fdec python3.5 Mixer and Global module 2018-04-12 17:06:57 +02:00
Terrtia
4386cbbd34 fix travis 2018-04-11 10:48:22 +02:00
Terrtia
14583f702f create first test 2018-04-11 10:14:33 +02:00
Terrtia
e9b88b0656 Keys module improvement 2018-04-05 11:40:34 +02:00
CD Nguyen
96259c53bc refactor NotificationHelper code 2018-04-04 09:41:13 +02:00
CD Nguyen
e9e3f17363 support non TSL SMTP servers 2018-03-30 11:35:37 +02:00
Philipp Schmied
29608ab29b Fixed e-mail notificatins for sets and regexes 2018-02-28 11:31:16 +01:00
Philipp Schmied
fd329aaba9 config.cfg.sample: Merged overwritten changes from upstream 2018-02-27 16:16:57 +01:00
Philipp Schmied
5b1f0b0212 Implemented email notifications (bin: config.cfg additions, email sending via analyzer scripts; var: Changes to add notifications via terms management); terms_management: Fixed click handlers not being added to all tracked terms. 2018-02-27 15:12:02 +01:00
abdf41fe23
Merge pull request #164 from mokaddem/showDiff
Show differences between duplicates
2018-02-21 22:25:16 +01:00
0263b25da8
Merge pull request #163 from mokaddem/iss160
Moving more to config
2018-02-21 22:24:50 +01:00
Sami Mokaddem
cbb19f95e4 fix: asciimatics introduced the '_required_height' property in the Label class. Just updating here to mirror the change 2018-02-16 14:35:09 +01:00
Sami Mokaddem
3e117195a2 fix: Added a backoff time if an IOError happens 2018-01-15 17:45:13 +01:00
Sami Mokaddem
9a66a6111b Fixing no such file 2018-01-15 17:10:03 +01:00
Sami Mokaddem
c39e537cbe Merge branch 'iss160' into showDiff
Preemptively preventing merge-conflict with branch iss160 (moving to
configuration)
2017-12-12 11:33:22 +01:00
Sami Mokaddem
05b22e8963 typo: added word + deleted trailing spaces 2017-12-12 11:19:37 +01:00
Sami Mokaddem
ef1bd16a72 typo: added word + deleted trailing spaces 2017-12-12 11:16:40 +01:00
Sami Mokaddem
304213cfa4 update: Possibility to change the authorized max line length in the config for the diff of duplicates 2017-12-12 11:00:26 +01:00
Sami Mokaddem
860acbf7cf fix: Renamed Credentials into Credential sectiont title
config change: By default, zmq listen to both local and CRF
2017-12-12 09:29:15 +01:00
Sami Mokaddem
9d0d0b4303 update: Moved filtering operation (thresholds, number of matching in the
categ file) in the configuration file. It permits to better control the
flow of pastes.
Also set default mixer duplicate filtering to 3 (Do not filter)
2017-12-11 17:28:34 +01:00
Fabien Mathey
3105192312
Allow for multiple entries like 1 2 3 4 5
Should replace the need to type 
1<Enter>2<Enter> 
etc. and allows for 
1 2 3 4 5<Enter>
2017-11-28 15:15:12 +01:00
71922f9def
add: Python script to test if the ZMQ feed works as expected 2017-11-28 09:46:06 +01:00
Sami Mokaddem
83e082e62a update: removed useless comments 2017-11-24 08:57:41 +01:00
Sami Mokaddem
a0d07b1098 feature: try to send data to MISP if keys are provided, else do nothing 2017-11-23 13:47:02 +01:00
Sami Mokaddem
2a967c4d92 update/feature: Max number of duplicate push to MISP + duplicate are pushed as attachment 2017-11-23 07:13:44 +01:00
Sami Mokaddem
805171a8a5 Added a seemingly working version of ailleak to misp 2017-11-20 14:57:25 +01:00
Sami Mokaddem
e0b188c069 Better support of python3 in Helper.py 2017-11-20 14:55:36 +01:00
Sami Mokaddem
1e76e10c01 Support of add_object to a MISP instance 2017-11-20 12:12:53 +01:00
Sami Mokaddem
6f0227a78a Starting support of python3 2017-11-20 12:12:06 +01:00
Sami Mokaddem
ee7759be91 Updated alertHandler and ailleakObject 2017-11-16 11:18:13 +01:00
Sami Mokaddem
5c7c07fbea Merge branch 'launcher_update' into misp_obj 2017-11-16 10:09:05 +01:00
Sami Mokaddem
45a9897020 Added draft support of MISP ail-leak object 2017-11-16 09:52:37 +01:00
Sami Mokaddem
652b00f4b2 Renamed BrowseWarningPaste into alertHandler 2017-11-15 16:15:43 +01:00
Sami Mokaddem
ec7d2d49b2 Added seconds args in import_dir 2017-11-15 16:03:42 +01:00
Sami Mokaddem
13f1c676d6 Added '_AIL' to the end of screen session name to better check an kills sessions. Also, gracefully close redis server + Check 1 time if redis-servers are running. 2017-09-26 16:17:06 +02:00
=
a8dc9231c6 Typo2.. 2017-08-24 16:43:42 +02:00
=
b2d4dd4a1f Typo 2017-08-24 16:43:05 +02:00
=
74ff401e1d Fixed bug: No CR between indexes in all_index.txt file 2017-08-24 16:41:30 +02:00
bb128a5e94 Merge pull request #138 from mokaddem/import_dir_update
Improvement of import_dir
2017-08-24 09:40:52 +02:00
=
429cc96d2a Improvement of import_dir.py. Added feed name parameter, send the correct path (without potential ../) and gzipEncode it if needed. 2017-08-23 15:10:26 +02:00
=
e7c66d1aac Updated individual launch scripts 2017-08-23 12:28:12 +02:00
=
d632335760 Automatically start redis_levelDB with the correct date 2017-08-23 10:16:56 +02:00
=
64ec44085d Dynamically choose correct port for LevelDB in script 2017-08-22 17:52:15 +02:00
=
66d60a1f01 Added LevelDB Curve for 2017 (3017, used for backward compatibility) 2017-08-21 13:54:44 +02:00
=
92adea38d0 Semi-automated directory creation with date as name in LAUNCH and installing_deps 2017-08-21 13:48:02 +02:00
=
bc35747f00 Updated KillAll in LAUNCH 2017-08-21 13:19:26 +02:00
Mokaddem
c3632c7d86 Added operation mode 3 for Mixer (do not filter) and added possibility to recurse through dir for import_dir 2017-08-03 00:34:53 +02:00
Mokaddem
cbb2909156 Added comments 2017-07-20 10:24:48 +02:00
Mokaddem
05417f405c Finished basic link web interface + solved some bugs 2017-07-20 10:04:30 +02:00
Mokaddem
ef741f85c0 Added similarity checks + started link webInterface 2017-07-19 11:52:06 +02:00
Mokaddem
882ab88b00 Draft module + web link 2017-07-18 16:57:15 +02:00
Mokaddem
9558aafe67 Prevent the loss of the 1 message in import_dir.py 2017-07-17 16:14:27 +02:00
Mokaddem
4d59871f00 Merged with previous changes for PR 2017-07-17 13:48:16 +02:00
4986352099 Merge pull request #120 from PaulSec/master
Added Python Phonenumbers module to parse numbers and retrieves stats…
2017-06-08 07:05:52 +02:00
446a97c47a Merge pull request #115 from mokaddem/regexInTerms
New features for terms frequency
2017-06-08 07:04:17 +02:00
Mokaddem
3a4dcd691d Improved description of modules inside the scripts 2017-05-09 11:13:16 +02:00
Paul A
747fff1df1 Added Python Phonenumbers module to parse numbers and retrieves stats from countrycode 2017-05-03 14:25:18 +02:00
Mokaddem
3ea0924120 Improved phone regex to handle end with 4 digits and presence of parenthesis. 2017-04-26 12:41:23 +02:00
Mokaddem
b60e164380 Added support of onion in browseWarningPaste 2017-04-26 11:47:02 +02:00
Mokaddem
318a557c56 Added possibility to keep a backup of the old config + check if config is valid before starting scripts. 2017-04-25 09:32:05 +02:00
Mokaddem
72c1d903eb Merge branch 'newIndexer' into auto-conf 2017-04-25 08:14:31 +02:00
Mokaddem
eccc8fc88b modified check_size timeout + utilisation of shutil instead of os.system() 2017-04-19 16:37:04 +02:00
Mokaddem
da081bae42 Merge branch 'regexInTerms' into production 2017-04-18 16:59:40 +02:00
Mokaddem
2d907ae784 Merge branch 'newIndexer' into auto-conf 2017-04-18 16:20:00 +02:00
Mokaddem
b0f1cf84b7 remplaced usage of global variable by a function param 2017-04-18 16:02:22 +02:00
Mokaddem
6e8509a0ba Removed wierd line 2017-04-18 15:57:07 +02:00
Mokaddem
c8baabd882 Updated TermsFrequency related modules + Fixed bugs (encoding, behaviors, ...) 2017-04-18 15:28:21 +02:00
Mokaddem
2da4c572c7 New feature in Terms: regex and set of words support (draft) 2017-03-28 17:42:44 +02:00
Mokaddem
6e80eb0954 Typo 2017-03-28 09:55:59 +02:00
Mokaddem
c79d4d65c8 Keep original section order while updating 2017-03-28 09:54:24 +02:00
Mokaddem
e225090e07 Added script that checks configuration and may auto-fix it + added missing field in config.cfg.sample 2017-03-15 16:36:51 +01:00
Mokaddem
be93af6f5a Merge branch 'terms-improvements' 2017-03-15 14:55:44 +01:00
Mokaddem
3261eca836 Fixed all_index newline bug + added print of what's happening 2017-03-15 14:29:49 +01:00
Mokaddem
5b7d047622 Changed os.system by os.mkdir + fixed bug during rotation 2017-03-15 14:05:13 +01:00
Mokaddem
adc37ab50e Indexer max size now in config 2017-03-15 12:14:41 +01:00
Mokaddem
4005b72180 Added backward support + renamed index with timestamp 2017-03-15 11:51:35 +01:00
Mokaddem
248469d61e Indexer now create an index_dir when it became too large (search in index not updated yet) 2017-03-14 10:37:31 +01:00
1540b719ed Merge pull request #108 from mokaddem/newModuleInformation
New version of ModuleInformation
2017-03-14 08:29:09 +01:00
Mokaddem
3c9cdac39a Merge branch 'terms-improvements' into production 2017-02-28 16:51:39 +01:00
Mokaddem
dadc872f9f Merge branch 'newModuleInformation' into production 2017-02-28 16:51:08 +01:00
Mokaddem
a2849c1bcd Added a catch for psutils if process get stopped by user 2017-02-28 16:13:01 +01:00
Mokaddem
4071bf2e91 fixed a bug in manage_term (usage of loop.index0) + fixed bug in top_set where graph was not displaying the correct per_paste value 2017-02-28 15:01:48 +01:00
Mokaddem
a1c47d7a13 Merge branch 'production' into terms-improvements 2017-02-28 11:15:56 +01:00
Mokaddem
fd4feb374f Merge branch 'newModuleInformation' into production 2017-02-28 11:12:06 +01:00
Mokaddem
75b4037519 Fixed bug - If we move the cursor from a line that has dissapear 2017-02-28 11:11:17 +01:00
Mokaddem
d6e02591af Merge branch 'newModuleInformation' into production 2017-02-28 10:23:45 +01:00
Mokaddem
975c1a37d9 ModuleInformation create the all_modules files if missing 2017-02-28 10:23:22 +01:00
Mokaddem
ccdf256383 Added possibility to show or not in dashboard text that is logged 2017-02-28 10:21:27 +01:00
Mokaddem
39104dea5e Merge branch 'bugfix_categ' into production 2017-02-28 09:56:25 +01:00
Mokaddem
26eb5fdff0 Fixed bug closing parenthesis in categ + changed behavior of sending to browseWarningPaste in Mail 2017-02-28 09:23:32 +01:00
Mokaddem
4a0dac1fae Merge branch 'production' into terms-improvements 2017-02-15 16:31:22 +01:00
Mokaddem
a696ccf83b Merge branch 'newModuleInformation' into production 2017-02-15 16:30:44 +01:00
Mokaddem
85a6f6fa63 Fixed a bug (str+float) 2017-02-15 16:30:31 +01:00
Mokaddem
25adae0104 Added possibility to select in top_terms the data per_paste or not (need module testing) 2017-02-15 16:29:02 +01:00
Mokaddem
e98e92b0d4 Merge branch 'fix-linkPaste-debug' into production 2017-02-15 13:43:03 +01:00
Mokaddem
c552aced42 Merge branch 'newModuleInformation' into production 2017-02-15 13:42:46 +01:00
Mokaddem
1b0d9d9507 Fixed str and int concat + Added sorting by queue name in queue not running 2017-02-15 13:42:21 +01:00
Mokaddem
74e0bc7acb Merge branch 'newModuleInformation' into production 2017-02-15 11:22:50 +01:00
Mokaddem
7fc9ba18d7 Added support of ENTER for navigation 2017-02-15 11:01:11 +01:00
Mokaddem
b84194542d Moved logging into a function 2017-02-15 10:57:41 +01:00
Mokaddem
0fce3f5d65 Renamed variables + added comments 2017-02-15 10:37:59 +01:00
Mokaddem
9ca3e79e70 Removed useless comments + commented 2017-02-15 10:19:03 +01:00
Mokaddem
9199ea3f0e Added comments 2017-02-15 10:04:51 +01:00
Mokaddem
ff1df07b5d Added comments and moved portion of code for slightly better visibility 2017-02-15 09:43:18 +01:00
Mokaddem
b272b1b07b Added a way to quit application with 'q' more or less properly 2017-02-15 09:35:03 +01:00
Mokaddem
0254c51164 Added end of paste marker + handling screen resize 2017-02-15 08:26:48 +01:00
Mokaddem
0b9609e4a9 Merge branch 'newModuleInformation' into production 2017-02-14 17:22:06 +01:00
Mokaddem
444f6071b7 Fixed screen bug by adding an input before starting asciimatics + improved path localisation in the logs from modules 2017-02-14 17:21:52 +01:00
Mokaddem
b04bac4731 Merge branch 'newModuleInformation' into production 2017-02-14 16:08:42 +01:00
Mokaddem
d520c45a15 Added support of display of the last paste processed by a module 2017-02-14 16:05:59 +01:00
Mokaddem
d3b4409156 Added path for debug in SQLInjection 2017-02-14 11:23:34 +01:00
Mokaddem
135a1457e0 Merge branch 'fix-linkPaste-debug' into production 2017-02-14 11:12:22 +01:00
Mokaddem
d4da3a316f Added path of the paste in the log of Categ.py 2017-02-14 10:59:47 +01:00
Mokaddem
303575b2a1 Merge remote-tracking branch 'upstream/master' into production 2017-02-14 09:31:18 +01:00
2f4e5b9016 Merge branch 'master' of github.com:CIRCL/AIL-framework 2017-01-26 07:16:44 +00:00
cf903cc212 (partially fix #90 too) using a simple Alarm (SIGNAL) when exec-timeout
Introducing a timer (in this case 60 seconds) to ensure that the
execution time of the analysis takes less than 60 seconds. This
is a simple and standard POSIX signal handler. If the timeout
is reached, the module will process the next one.

This approach fixes the specific issues we have currently
with some inputs where the sentiment analysis takes too much time. This
fix should be improved and be more generic:

 - Introducing statistics of content which timeouts.
 - Keeping a list/queue to further process those files using a different
   analysis approach. Maybe a set of "dirty" processes to handle the edge cases
   and to not impact the overall processing and analysis.
 - Make the timer configurable per module (at least for this one).
2017-01-26 07:11:18 +00:00
Mokaddem
c422db6e6d Merge remote-tracking branch 'upstream/master' into production 2017-01-17 16:38:48 +01:00
mokaddem
b3a3616b0b Merge pull request #102 from ptitdoc/bugfix1
Bug fixes in Workers and improvement of pystemon-feeder
2017-01-17 16:35:32 +01:00
Mokaddem
ea423fde0f Added requirement, better behaviors and confirmation box 2017-01-17 15:58:16 +01:00
Mokaddem
692f13d18b Improved module + Added support of click and keyboard navigation for killing and starting modules 2017-01-17 14:31:06 +01:00
Mokaddem
3b8149d840 Added action button and support of logs 2017-01-16 17:08:48 +01:00
Mokaddem
0ecfaf00f5 Added color in front of queue related to its state 2017-01-16 15:30:33 +01:00
Mokaddem
af3241c008 Overwritted listbox update function to add formating 2017-01-16 14:41:02 +01:00
Mokaddem
30f303cb99 Added new ModuleInformation 2017-01-16 12:18:23 +01:00
Olivier MEDOC
16044d4d36 pystemon-feeder: add the option bind in configuration file instead of address
This is essentially because multiple feeds can be used in the address
configuration variable.
2017-01-13 14:58:18 +01:00
3b101ea8f5 (partially) Fix #91 using a simple Alarm (SIGNAL) when exec-timeout
Introducing a timer (in this case 5 seconds) to ensure that the
execution time of the tokenizer takes less than 5 seconds. This
is a simple and standard POSIX signal handler.

This approach fixes the specific issues we have currently
with some inputs where the tokenization takes too much time. This
fix should be improved and be more generic:

 - Introducing statistics of content which timeouts.
 - Keeping a list/queue to further process those files using a different
   tokenizer approach. Maybe a set of "dirty" processes to handle the edge cases
   and to not impact the overall processing and analysis.
 - Make the timer configurable per module (at least for this one).
2017-01-12 07:32:55 +00:00
Olivier MEDOC
9f9c265cb0 SentimentAnalysis: use lexicon file path from the ail configuration file 2017-01-11 11:07:08 +01:00
ptitdoc
c0fac82031 Add pystemon path in configuration sample 2017-01-10 18:33:46 +01:00
Olivier MEDOC
83db40104c import pystemon-feeder configuration from the central configuration file 2017-01-10 18:26:36 +01:00
Olivier MEDOC
3dc014dad9 fix invalid variable propagation 2017-01-10 18:26:24 +01:00
Olivier MEDOC
68ca20db8f fix to use the new pycountry API 2017-01-10 18:26:17 +01:00
1950a2dc0e Merge pull request #99 from mokaddem/more-feed
Usage of sha1 to filter paste content in Mixer.py
2017-01-09 17:06:37 +01:00
Mokaddem
500608d0a1 Usage of sha1 to filter paste content in Mixer.py 2017-01-09 17:01:10 +01:00
f5adb65cb3 Merge pull request #98 from mokaddem/more-feed
Config 2017
2017-01-09 15:39:53 +01:00
Mokaddem
409c5b8b42 Added 2017 2017-01-09 15:36:50 +01:00
a96ba40d94 Merge pull request #96 from mokaddem/more-feed
Handling of multiple feeders
2017-01-09 15:29:28 +01:00
Mokaddem
0e39f516a9 Added sleep in helper for multiple listening interfaces (reduce useless work), Added a simple way to pre-process feed before sending it to the global feed. 2017-01-09 14:12:26 +01:00
Mokaddem
3a298f0c7c Commented Mixer.py for further multiple feeds support. 2016-12-26 16:23:18 +01:00
Mokaddem
1c2169d3bc Improved interface for multiple feeds and refresh_script. 2016-12-26 16:16:44 +01:00
Mokaddem
97292e0899 Updated web interface to handle new mixer module. and fixed one dependency bug. 2016-12-23 15:44:46 +01:00
Mokaddem
e70b9cd15c Added basic mixer with confirugable behavior. It handles muliple feeders and performs some basic stats on them. 2016-12-23 10:31:26 +01:00
ddfe3cf66a Merge pull request #95 from mokaddem/module_handling
Module handling
2016-12-22 22:18:31 +01:00
3ce850e093 Merge pull request #94 from mokaddem/website-refacto
Website refacto, support and fixes
2016-12-22 22:17:58 +01:00
Mokaddem
9758579753 Merge branch 'module_handling' into production 2016-12-22 10:24:30 +01:00
Mokaddem
7438f16c63 Added new config 2016-12-22 10:06:35 +01:00
Mokaddem
73938cf41b Prepared moduleInfo to handle module click with curses (will come back later). 2016-12-22 08:45:53 +01:00
Mokaddem
d37b243e1d Added legend + added more info if restarting disabled 2016-12-09 15:13:47 +01:00
Mokaddem
d3ed1bbf40 Merge branch 'website-improvement' into production 2016-12-08 10:05:55 +01:00
Mokaddem
73d4f9e082 Webstats should correctly updates top_progression_zset (Not fully tested because not enough data. Will be tested latter) 2016-12-08 10:05:07 +01:00
Mokaddem
570324060e terms top_sets correctly supports blacklisted terms 2016-12-08 09:13:31 +01:00
412b80c44d Merge pull request #88 from mokaddem/Issue-Feeding-AIL
Pystemon tuto
2016-11-26 12:45:28 +01:00
Mokaddem
d831a79614 Do not try to restart not running queue if autokill is not set 2016-11-25 16:00:46 +01:00
Mokaddem
4bf45eea36 ModuleInformation sort by name as well as elapsed time 2016-11-25 14:44:27 +01:00
Mokaddem
866be4e884 Added row coloring in ModuleInformation 2016-11-25 14:12:09 +01:00
Mokaddem
a3255d168c ModuleInfo correctly handle CurveManageTopSets,
Changed queue coloring in dashboard,
ModuleInformation now have a history of executed command and better handle module killing.
2016-11-25 11:54:16 +01:00
Mokaddem
224fbc8084 Better handle stuck modules. Differentiate between not running and no info and tries to restart stuck ones. 2016-11-24 16:58:32 +01:00
Mokaddem
5d269ea1ee Added date in Duplicate for better distinguish duplicate. Also, added a timeout for the dataTable in duplicate inside the modal. 2016-11-24 15:05:29 +01:00
Mokaddem
ac254e0e63 Added support of browsing concerned paste in dashboard for all modules + Show in the page what was the concerned module 2016-10-27 11:50:24 +02:00
Mokaddem
1826b170ec Added support of local paste view in dashboard for Mails event only 2016-10-27 11:27:26 +02:00
Mokaddem
5a4ea734a2 Fixed path issues in doc + added dependencies related to ssdeep 2016-10-14 14:26:33 +02:00
Mokaddem
4eafca2936 Added pystemon instruction section 2016-10-13 17:13:08 +02:00
eb14005fcf Fix: as JSON decode needs a string, the empty list is a string. 2016-08-29 12:08:12 +00:00
3b6e3e156e Merge branch 'master' of github.com:CIRCL/AIL-framework 2016-08-26 15:57:45 +00:00
Mokaddem
0c760d763b Added support of PID in ModuleInformation and index.html 2016-08-26 15:27:37 +02:00
758f2c5334 Merge branch 'master' of github.com:CIRCL/AIL-framework 2016-08-26 08:30:11 +00:00
Mokaddem
494980c48e Added comment in moduleInformation 2016-08-26 10:07:06 +02:00
Mokaddem
8b33b4e7ba Added not running module info + swapped order of module launch 2016-08-26 09:21:02 +02:00
Mokaddem
813cc7031a Changed lots of code into zrank 2016-08-25 17:48:12 +02:00
Mokaddem
aaac798db9 updated individual launch script 2016-08-25 16:51:57 +02:00
Mokaddem
78c991539e Added number of same module running in ModuleInformation 2016-08-25 16:07:47 +02:00
1ab08d4a05 Merge branch 'master' of github.com:CIRCL/AIL-framework 2016-08-25 12:33:31 +00:00
Mokaddem
f9bf00f022 Removed autokill feature if not specified 2016-08-25 11:09:31 +02:00
Mokaddem
2f136595fb Removed autokill feature if not specified 2016-08-25 11:05:18 +02:00
Mokaddem
9146feab88 Fixed bug in WebStats not deleting correct key + fixed typo in CurveSetManager preventing connecting to redis + modified display in moduleInfo 2016-08-25 10:43:04 +02:00
Mokaddem
3313064e51 removed useless import. 2016-08-24 17:32:50 +02:00
Mokaddem
64bf481480 Added moduleInfo + Renamed modules 2016-08-24 17:28:39 +02:00
Mokaddem
8fb552b7da ModuleInfo support autokill and auto restart after logging. 2016-08-24 16:52:01 +02:00
Mokaddem
3e88d0911d ModuleInfo now support logs. 2016-08-24 15:53:00 +02:00
Mokaddem
ff6dcaf456 Fixed typo in creditcard + fixed bug in ModuleInformation (relative to date). 2016-08-24 15:35:23 +02:00
Mokaddem
deeebec2f5 Added new module 'ModuleInformation' which gives real time information on running module 2016-08-24 15:21:41 +02:00
Mokaddem
07856f3119 Added script module-dataflow-graph + fixed bug in module.cfg preventing the credential module to receive data. 2016-08-24 11:32:48 +02:00
bde5f47ed0 Merge branch 'master' of github.com:CIRCL/AIL-framework
Conflicts:
	bin/launch_scripts.sh
2016-08-23 14:20:39 +00:00
Mokaddem
42b3966499 Added new module in individual launch scripts 2016-08-23 16:01:11 +02:00
Mokaddem
697812dbff Added config.sample 2016-08-23 09:59:39 +02:00
Mokaddem
f9b27e4013 Fixed bug in curve preventing to track concerned_pastes. Now all tracked words are put in lower case. 2016-08-23 09:50:03 +02:00
Mokaddem
047331df3e Added save of paste_baseName in set for searches. This is done in ModuleStats to not surcharge the Global module. 2016-08-23 08:52:54 +02:00
Mokaddem
20fd191c20 Suppressed prints 2016-08-22 21:29:59 +02:00
Mokaddem
2fd2c8af39 Removed commented code 2016-08-22 20:59:56 +02:00
Mokaddem
003c72bd7c Fixed bug with redis where the inserted key was too long, causing a crash + fixed bug taking min instead of max in terms-top-set 2016-08-22 16:01:42 +02:00
Mokaddem
2466f355ab Fixed bug in manage-terms-top-set + added auto sorting in important-paste-by-module 2016-08-22 14:54:37 +02:00
Mokaddem
d3c0fc5f94 Added terms trending module and web interface:
- Curve module has been rewritten
- Added new module curve_manage_top_set
- Added all web-pages
2016-08-22 13:35:49 +02:00
Mokaddem
0b6828fb16 Correctly handle and display add or remove a term in term-frequency + started sketch of terms-plot tool. 2016-08-19 16:53:46 +02:00
Mokaddem
b78effe779 Added draft of terms_management + Added copyright in readme 2016-08-19 13:34:02 +02:00
Mokaddem
52b7124a6b - Modified redis connection (from levelDB to redis).
- Added term frequency in curve.
- Modified ModuleStats and WebStats to use redis special command (incrby instead of get-set and zset) and Flask to perform the correct queries.
- Added panel color in sentiment trending
2016-08-18 15:34:19 +02:00
Mokaddem
902b7e39f4 Probably fixed bug in creditcard (not sending to duplicate) 2016-08-17 14:40:13 +02:00
Mokaddem
c10f28d9df Updated isolated launch scripts + Added switch displaying queue in index 2016-08-17 13:46:01 +02:00
Mokaddem
dbc164f4e8 modified + into append 2016-08-17 11:26:07 +02:00
Mokaddem
c12e67a35a Added tooltips + slight gauge modification 2016-08-17 10:43:43 +02:00
Mokaddem
894b9efda9 Added drop of really long line in sentiment-analysis module + Added description of sentiment module. Also, fixed bug in webpage sentiement-trending concerning avg and date range. 2016-08-17 09:46:25 +02:00
Mokaddem
1084e45f1b Added draft of filter in sentiment analysis (Discard syntaxical languages) + Added nice tooltip for sparkline. Trending displays avg in function of the number of elements processed and not for the complete week + fixed bug in gauge and canvasjs (was performing avg with only 1 graph instead of all 8). 2016-08-16 16:33:02 +02:00
Mokaddem
cdd08d7df0 Added all providers for plot_tool in moduleStats + fixed date range bug 2016-08-16 11:07:36 +02:00
Mokaddem
1a5158b081 Added sentiment analyser module (draft) 2016-08-13 15:24:57 +02:00
Mokaddem
c028ca80da Separated options from the plot funcion for sparkline. 2016-08-11 17:57:23 +02:00
95a22abe9c Merge branch 'master' of github.com:CIRCL/AIL-framework
Conflicts:
	bin/packages/modules.cfg
2016-08-11 08:06:57 +00:00
Mokaddem
cd74e6d1c5 Added module sentimentAnalyser 2016-08-11 09:40:42 +02:00
Mokaddem
64fe32b29b Fixed small bug in SQLInjection module 2016-08-10 14:23:36 +02:00
Mokaddem
feafa8ffe2 Slight refacto + sql comments are now placed in injection suspision. 2016-08-10 09:06:37 +02:00
Mokaddem
943016d6ca Fixed one of the small display bug in DUplicate_ssdeep_v2 (was displaying [tlsh, tlsh] instead of simply tlsh) 2016-08-09 15:06:55 +02:00
Mokaddem
a4595010c0 Added whole config in config.sample 2016-08-09 14:23:36 +02:00
Mokaddem
a80f6ca0c2 Bug fix related with redis:
Fixed typo key in redis for module creditcard and sqlinjection
Modified Curve redisLvlDb server
Modified Url.py so that it forwards name of protocol from saved protocolsfile
Added Cache control in Flask
Modified key-tab name into keys-tab
2016-08-09 11:59:36 +02:00
Mokaddem
60512a30a2 Fixeed bug in Duplicate (The comparison value was not saved correctly in redis) + Modified to progression detection algo 2016-08-08 15:28:26 +02:00
Mokaddem
aedb3198cd Fixed bug in CVE (always sending to browse_warning even if there was no match) 2016-08-08 11:54:27 +02:00
Mokaddem
14062a549c Fixed bug in credential (not sending correctly to ModuleStat) + Fixed date bug in ModuleStats 2016-08-08 11:37:18 +02:00
Mokaddem
4e56f76f62 Fixed bug CVE,
CVE was not sending its data to BrowseWarningPaste.
2016-08-08 10:05:21 +02:00
Mokaddem
1c40ad522f Merge branch 'master' into production 2016-08-08 09:26:58 +02:00
Mokaddem
78c611fead Added warning_paste module and created related webpages.
Fixed a Faup bug in credential (multiple instanciation) and added correc populate_set_out in concerned modules (creditcard, credential, ...).
Linked browse_warning_paste module and Flask function with redis (created new sets).
2016-08-08 09:17:44 +02:00
Mokaddem
3523d79a05 Started browse important paste feature. Still need to link with redis. + Fixed a bug in Flask (String to list parser) 2016-08-04 16:57:34 +02:00
Mokaddem
55ef917ef1 Added few comments 2016-08-04 12:10:56 +02:00
Mokaddem
d9316771cd Draft: added new duplicate hash comparison - tlsh 2016-08-04 11:55:38 +02:00
Mokaddem
455911fc26 Better display printing 2016-08-02 16:21:52 +02:00
Mokaddem
3169dc27b0 Added SQLInjectionDetection module 2016-08-02 15:43:11 +02:00
Mokaddem
a1b2329a9b Fixed bug in case of null elements arrising because of a new month 2016-08-01 10:32:24 +02:00
Mokaddem
a9637ec7cf removed prints 2016-07-29 16:54:25 +02:00
Mokaddem
feb1717dc6 fixed size bug, which in some cases display the total size instead of the average + few cleanup + commented out log scale in moduleStats.js and added large number division in case of div by 0 2016-07-29 16:52:50 +02:00
Mokaddem
e8fae17cfd Fixed small race condition bug in ModuleStat.py and Added highlight of bar belonging to the same serie in moduleStats.js 2016-07-29 10:36:44 +02:00
Mokaddem
240bdd9cca Added number of paste by provider in module stats 2016-07-27 09:27:32 +02:00
Mokaddem
baa042827d Removed useless comments + prevent trying to add an already present element in the set 2016-07-26 11:06:31 +02:00
Mokaddem
cc1b49baaf Fixed variable bug in moduleStats and added draft of credential stats 2016-07-26 10:45:02 +02:00
Mokaddem
06be1f129a Added Draft of ModuleStats + Paste size average 2016-07-25 16:38:57 +02:00
d013546643 Merge branch 'master' of github.com:CIRCL/AIL-framework
Conflicts:
	bin/packages/modules.cfg
2016-07-25 07:56:46 +00:00
722634adf4 Prod config 2016-07-25 07:49:14 +00:00
Raphaël Vinot
4aa484a91f Merge pull request #63 from mokaddem/uptodate-duplicate-module
Uptodate duplicate module
2016-07-25 09:44:49 +02:00
Mokaddem
34a00abdc2 Merge branch 'uptodate-duplicate-module' into production 2016-07-22 10:31:10 +02:00
Mokaddem
c686f69ca6 Clean not compatible ssdeep hash encountered 2016-07-22 10:04:58 +02:00
Raphaël Vinot
ea8703e608 Fix path to tldsfile 2016-07-22 09:32:13 +02:00
Mokaddem
dad05073fd Fixed bug introduced in merge-conflict 2016-07-22 09:25:05 +02:00
Mokaddem
7c4f4a2f79 Merge branch 'statistics-v2' into uptodate-statistics-v2 2016-07-21 15:16:03 +02:00
Mokaddem
b685f81752 Merge branch 'duplicate-module' into uptodate-duplicate-module 2016-07-21 14:59:52 +02:00
Mokaddem
0ab7560477 Added top_progression chart for tld, domain and scheme + Small modification in config file. 2016-07-21 13:44:22 +02:00
Raphaël Vinot
ede46ec799 Fix default config. 2016-07-21 09:19:13 +02:00
Raphaël Vinot
781b441bb8 Fix path in config 2016-07-20 14:12:50 +02:00
Raphaël Vinot
620d69467f Add WebStats in launcher 2016-07-20 14:12:37 +02:00
Raphaël Vinot
62eef44ca8 Fix pep8 2016-07-20 14:12:18 +02:00
Raphaël Vinot
8a69110c8e Merge branch 'webpage-splitting' of https://github.com/mokaddem/AIL-framework into mokaddem-webpage-splitting 2016-07-19 16:35:31 +02:00
Mokaddem
a6996c0b23 Added related functions and display of duplicated paste in search.py 2016-07-19 10:48:44 +02:00
Mokaddem
996c0e02de Duplicate module takes its messages from other modules and no more from Global. 2016-07-18 16:22:33 +02:00
Mokaddem
6805ed6488 Added default config 2016-07-18 15:52:53 +02:00