AIL/ail-framework
7
0
Fork 0
mirror of https://github.com/ail-project/ail-framework.git synced 2024-11-14 02:28:23 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
685 commits 7 branches 43 tags 99 MiB
Commit graph

318 commits

Author SHA1 Message Date
Mokaddem
73938cf41b Prepared moduleInfo to handle module click with curses (will come back later). 2016-12-22 08:45:53 +01:00
Mokaddem
d37b243e1d Added legend + added more info if restarting disabled 2016-12-09 15:13:47 +01:00
Mokaddem
d3ed1bbf40 Merge branch 'website-improvement' into production 2016-12-08 10:05:55 +01:00
Mokaddem
73d4f9e082 Webstats should correctly updates top_progression_zset (Not fully tested because not enough data. Will be tested latter) 2016-12-08 10:05:07 +01:00
Mokaddem
570324060e terms top_sets correctly supports blacklisted terms 2016-12-08 09:13:31 +01:00
Alexandre Dulaunoy
412b80c44d Merge pull request #88 from mokaddem/Issue-Feeding-AIL 
Pystemon tuto
 2016-11-26 12:45:28 +01:00
Mokaddem
d831a79614 Do not try to restart not running queue if autokill is not set 2016-11-25 16:00:46 +01:00
Mokaddem
4bf45eea36 ModuleInformation sort by name as well as elapsed time 2016-11-25 14:44:27 +01:00
Mokaddem
866be4e884 Added row coloring in ModuleInformation 2016-11-25 14:12:09 +01:00
Mokaddem
a3255d168c ModuleInfo correctly handle CurveManageTopSets, 
Changed queue coloring in dashboard,
ModuleInformation now have a history of executed command and better handle module killing.
 2016-11-25 11:54:16 +01:00
Mokaddem
224fbc8084 Better handle stuck modules. Differentiate between not running and no info and tries to restart stuck ones. 2016-11-24 16:58:32 +01:00
Mokaddem
5d269ea1ee Added date in Duplicate for better distinguish duplicate. Also, added a timeout for the dataTable in duplicate inside the modal. 2016-11-24 15:05:29 +01:00
Mokaddem
ac254e0e63 Added support of browsing concerned paste in dashboard for all modules + Show in the page what was the concerned module 2016-10-27 11:50:24 +02:00
Mokaddem
1826b170ec Added support of local paste view in dashboard for Mails event only 2016-10-27 11:27:26 +02:00
Mokaddem
5a4ea734a2 Fixed path issues in doc + added dependencies related to ssdeep 2016-10-14 14:26:33 +02:00
Mokaddem
4eafca2936 Added pystemon instruction section 2016-10-13 17:13:08 +02:00
Alexandre Dulaunoy
eb14005fcf Fix: as JSON decode needs a string, the empty list is a string. 2016-08-29 12:08:12 +00:00
Alexandre Dulaunoy
3b6e3e156e Merge branch 'master' of github.com:CIRCL/AIL-framework 2016-08-26 15:57:45 +00:00
Mokaddem
0c760d763b Added support of PID in ModuleInformation and index.html 2016-08-26 15:27:37 +02:00
Alexandre Dulaunoy
758f2c5334 Merge branch 'master' of github.com:CIRCL/AIL-framework 2016-08-26 08:30:11 +00:00
Mokaddem
494980c48e Added comment in moduleInformation 2016-08-26 10:07:06 +02:00
Mokaddem
8b33b4e7ba Added not running module info + swapped order of module launch 2016-08-26 09:21:02 +02:00
Mokaddem
813cc7031a Changed lots of code into zrank 2016-08-25 17:48:12 +02:00
Mokaddem
aaac798db9 updated individual launch script 2016-08-25 16:51:57 +02:00
Mokaddem
78c991539e Added number of same module running in ModuleInformation 2016-08-25 16:07:47 +02:00
Alexandre Dulaunoy
1ab08d4a05 Merge branch 'master' of github.com:CIRCL/AIL-framework 2016-08-25 12:33:31 +00:00
Mokaddem
f9bf00f022 Removed autokill feature if not specified 2016-08-25 11:09:31 +02:00
Mokaddem
2f136595fb Removed autokill feature if not specified 2016-08-25 11:05:18 +02:00
Mokaddem
9146feab88 Fixed bug in WebStats not deleting correct key + fixed typo in CurveSetManager preventing connecting to redis + modified display in moduleInfo 2016-08-25 10:43:04 +02:00
Mokaddem
3313064e51 removed useless import. 2016-08-24 17:32:50 +02:00
Mokaddem
64bf481480 Added moduleInfo + Renamed modules 2016-08-24 17:28:39 +02:00
Mokaddem
8fb552b7da ModuleInfo support autokill and auto restart after logging. 2016-08-24 16:52:01 +02:00
Mokaddem
3e88d0911d ModuleInfo now support logs. 2016-08-24 15:53:00 +02:00
Mokaddem
ff6dcaf456 Fixed typo in creditcard + fixed bug in ModuleInformation (relative to date). 2016-08-24 15:35:23 +02:00
Mokaddem
deeebec2f5 Added new module 'ModuleInformation' which gives real time information on running module 2016-08-24 15:21:41 +02:00
Mokaddem
07856f3119 Added script module-dataflow-graph + fixed bug in module.cfg preventing the credential module to receive data. 2016-08-24 11:32:48 +02:00
Alexandre Dulaunoy
bde5f47ed0 Merge branch 'master' of github.com:CIRCL/AIL-framework 
Conflicts:
	bin/launch_scripts.sh
 2016-08-23 14:20:39 +00:00
Mokaddem
42b3966499 Added new module in individual launch scripts 2016-08-23 16:01:11 +02:00
Mokaddem
697812dbff Added config.sample 2016-08-23 09:59:39 +02:00
Mokaddem
f9b27e4013 Fixed bug in curve preventing to track concerned_pastes. Now all tracked words are put in lower case. 2016-08-23 09:50:03 +02:00
Mokaddem
047331df3e Added save of paste_baseName in set for searches. This is done in ModuleStats to not surcharge the Global module. 2016-08-23 08:52:54 +02:00
Mokaddem
20fd191c20 Suppressed prints 2016-08-22 21:29:59 +02:00
Mokaddem
2fd2c8af39 Removed commented code 2016-08-22 20:59:56 +02:00
Mokaddem
003c72bd7c Fixed bug with redis where the inserted key was too long, causing a crash + fixed bug taking min instead of max in terms-top-set 2016-08-22 16:01:42 +02:00
Mokaddem
2466f355ab Fixed bug in manage-terms-top-set + added auto sorting in important-paste-by-module 2016-08-22 14:54:37 +02:00
Mokaddem
d3c0fc5f94 Added terms trending module and web interface: 
- Curve module has been rewritten
- Added new module curve_manage_top_set
- Added all web-pages
 2016-08-22 13:35:49 +02:00
Mokaddem
0b6828fb16 Correctly handle and display add or remove a term in term-frequency + started sketch of terms-plot tool. 2016-08-19 16:53:46 +02:00
Mokaddem
b78effe779 Added draft of terms_management + Added copyright in readme 2016-08-19 13:34:02 +02:00
Mokaddem
52b7124a6b - Modified redis connection (from levelDB to redis). 
- Added term frequency in curve.
- Modified ModuleStats and WebStats to use redis special command (incrby instead of get-set and zset) and Flask to perform the correct queries.
- Added panel color in sentiment trending
 2016-08-18 15:34:19 +02:00
Mokaddem
902b7e39f4 Probably fixed bug in creditcard (not sending to duplicate) 2016-08-17 14:40:13 +02:00
Mokaddem
c10f28d9df Updated isolated launch scripts + Added switch displaying queue in index 2016-08-17 13:46:01 +02:00
Mokaddem
dbc164f4e8 modified + into append 2016-08-17 11:26:07 +02:00
Mokaddem
c12e67a35a Added tooltips + slight gauge modification 2016-08-17 10:43:43 +02:00
Mokaddem
894b9efda9 Added drop of really long line in sentiment-analysis module + Added description of sentiment module. Also, fixed bug in webpage sentiement-trending concerning avg and date range. 2016-08-17 09:46:25 +02:00
Mokaddem
1084e45f1b Added draft of filter in sentiment analysis (Discard syntaxical languages) + Added nice tooltip for sparkline. Trending displays avg in function of the number of elements processed and not for the complete week + fixed bug in gauge and canvasjs (was performing avg with only 1 graph instead of all 8). 2016-08-16 16:33:02 +02:00
Mokaddem
cdd08d7df0 Added all providers for plot_tool in moduleStats + fixed date range bug 2016-08-16 11:07:36 +02:00
Mokaddem
1a5158b081 Added sentiment analyser module (draft) 2016-08-13 15:24:57 +02:00
Mokaddem
c028ca80da Separated options from the plot funcion for sparkline. 2016-08-11 17:57:23 +02:00
Alexandre Dulaunoy
95a22abe9c Merge branch 'master' of github.com:CIRCL/AIL-framework 
Conflicts:
	bin/packages/modules.cfg
 2016-08-11 08:06:57 +00:00
Mokaddem
cd74e6d1c5 Added module sentimentAnalyser 2016-08-11 09:40:42 +02:00
Mokaddem
64fe32b29b Fixed small bug in SQLInjection module 2016-08-10 14:23:36 +02:00
Mokaddem
feafa8ffe2 Slight refacto + sql comments are now placed in injection suspision. 2016-08-10 09:06:37 +02:00
Mokaddem
943016d6ca Fixed one of the small display bug in DUplicate_ssdeep_v2 (was displaying [tlsh, tlsh] instead of simply tlsh) 2016-08-09 15:06:55 +02:00
Mokaddem
a4595010c0 Added whole config in config.sample 2016-08-09 14:23:36 +02:00
Mokaddem
a80f6ca0c2 Bug fix related with redis: 
Fixed typo key in redis for module creditcard and sqlinjection
Modified Curve redisLvlDb server
Modified Url.py so that it forwards name of protocol from saved protocolsfile
Added Cache control in Flask
Modified key-tab name into keys-tab
 2016-08-09 11:59:36 +02:00
Mokaddem
60512a30a2 Fixeed bug in Duplicate (The comparison value was not saved correctly in redis) + Modified to progression detection algo 2016-08-08 15:28:26 +02:00
Mokaddem
aedb3198cd Fixed bug in CVE (always sending to browse_warning even if there was no match) 2016-08-08 11:54:27 +02:00
Mokaddem
14062a549c Fixed bug in credential (not sending correctly to ModuleStat) + Fixed date bug in ModuleStats 2016-08-08 11:37:18 +02:00
Mokaddem
4e56f76f62 Fixed bug CVE, 
CVE was not sending its data to BrowseWarningPaste.
 2016-08-08 10:05:21 +02:00
Mokaddem
1c40ad522f Merge branch 'master' into production 2016-08-08 09:26:58 +02:00
Mokaddem
78c611fead Added warning_paste module and created related webpages. 
Fixed a Faup bug in credential (multiple instanciation) and added correc populate_set_out in concerned modules (creditcard, credential, ...).
Linked browse_warning_paste module and Flask function with redis (created new sets).
 2016-08-08 09:17:44 +02:00
Mokaddem
3523d79a05 Started browse important paste feature. Still need to link with redis. + Fixed a bug in Flask (String to list parser) 2016-08-04 16:57:34 +02:00
Mokaddem
55ef917ef1 Added few comments 2016-08-04 12:10:56 +02:00
Mokaddem
d9316771cd Draft: added new duplicate hash comparison - tlsh 2016-08-04 11:55:38 +02:00
Mokaddem
455911fc26 Better display printing 2016-08-02 16:21:52 +02:00
Mokaddem
3169dc27b0 Added SQLInjectionDetection module 2016-08-02 15:43:11 +02:00
Mokaddem
a1b2329a9b Fixed bug in case of null elements arrising because of a new month 2016-08-01 10:32:24 +02:00
Mokaddem
a9637ec7cf removed prints 2016-07-29 16:54:25 +02:00
Mokaddem
feb1717dc6 fixed size bug, which in some cases display the total size instead of the average + few cleanup + commented out log scale in moduleStats.js and added large number division in case of div by 0 2016-07-29 16:52:50 +02:00
Mokaddem
e8fae17cfd Fixed small race condition bug in ModuleStat.py and Added highlight of bar belonging to the same serie in moduleStats.js 2016-07-29 10:36:44 +02:00
Mokaddem
240bdd9cca Added number of paste by provider in module stats 2016-07-27 09:27:32 +02:00
Mokaddem
baa042827d Removed useless comments + prevent trying to add an already present element in the set 2016-07-26 11:06:31 +02:00
Mokaddem
cc1b49baaf Fixed variable bug in moduleStats and added draft of credential stats 2016-07-26 10:45:02 +02:00
Mokaddem
06be1f129a Added Draft of ModuleStats + Paste size average 2016-07-25 16:38:57 +02:00
Alexandre Dulaunoy
d013546643 Merge branch 'master' of github.com:CIRCL/AIL-framework 
Conflicts:
	bin/packages/modules.cfg
 2016-07-25 07:56:46 +00:00
Alexandre Dulaunoy
722634adf4 Prod config 2016-07-25 07:49:14 +00:00
Raphaël Vinot
4aa484a91f Merge pull request #63 from mokaddem/uptodate-duplicate-module 
Uptodate duplicate module
 2016-07-25 09:44:49 +02:00
Mokaddem
34a00abdc2 Merge branch 'uptodate-duplicate-module' into production 2016-07-22 10:31:10 +02:00
Mokaddem
c686f69ca6 Clean not compatible ssdeep hash encountered 2016-07-22 10:04:58 +02:00
Raphaël Vinot
ea8703e608 Fix path to tldsfile 2016-07-22 09:32:13 +02:00
Mokaddem
dad05073fd Fixed bug introduced in merge-conflict 2016-07-22 09:25:05 +02:00
Mokaddem
7c4f4a2f79 Merge branch 'statistics-v2' into uptodate-statistics-v2 2016-07-21 15:16:03 +02:00
Mokaddem
b685f81752 Merge branch 'duplicate-module' into uptodate-duplicate-module 2016-07-21 14:59:52 +02:00
Mokaddem
0ab7560477 Added top_progression chart for tld, domain and scheme + Small modification in config file. 2016-07-21 13:44:22 +02:00
Raphaël Vinot
ede46ec799 Fix default config. 2016-07-21 09:19:13 +02:00
Raphaël Vinot
781b441bb8 Fix path in config 2016-07-20 14:12:50 +02:00
Raphaël Vinot
620d69467f Add WebStats in launcher 2016-07-20 14:12:37 +02:00
Raphaël Vinot
62eef44ca8 Fix pep8 2016-07-20 14:12:18 +02:00
Raphaël Vinot
8a69110c8e Merge branch 'webpage-splitting' of https://github.com/mokaddem/AIL-framework into mokaddem-webpage-splitting 2016-07-19 16:35:31 +02:00
Mokaddem
a6996c0b23 Added related functions and display of duplicated paste in search.py 2016-07-19 10:48:44 +02:00
Mokaddem
996c0e02de Duplicate module takes its messages from other modules and no more from Global. 2016-07-18 16:22:33 +02:00
Mokaddem
6805ed6488 Added default config 2016-07-18 15:52:53 +02:00
Mokaddem
4f6813350b Added two new version of duplicate module. 
One with hashes are saved in json on disk
The other with only leveldb
 2016-07-18 15:50:41 +02:00
Mokaddem
14e9850dd6 Added new module for Duplicate paste. Seems working but has some small bug (re-check same paste twice) 2016-07-15 16:58:48 +02:00
Mokaddem
2383db022f Added default configuration 2016-07-15 09:10:44 +02:00
Mokaddem
9a9e07f600 Added default configuration 2016-07-15 09:08:38 +02:00
Mokaddem
0332f23579 Added SimHash library 2016-07-15 08:56:16 +02:00
Mokaddem
fba14bfb4b In index: Added number of processed pastes chart 2016-07-14 11:16:07 +02:00
Mokaddem
ab61e32399 Commented out get_language because it adds too much overhead 2016-07-14 11:15:15 +02:00
Mokaddem
594d2def35 In index: Added number of processed pastes chart 2016-07-13 15:57:33 +02:00
Mokaddem
56b6659d8b Commented out get_language because it adds too much overhead 2016-07-13 08:59:48 +02:00
Mokaddem
c51bdec8aa Merge branch 'mokaddem-testing' 2016-07-12 11:53:24 +02:00
Mokaddem
8a1247cf5d modified variable name str 2016-07-12 11:52:19 +02:00
Mokaddem
7e5ce0f17f Optimized create_plot and removed test commemts 2016-07-12 11:47:51 +02:00
Mokaddem
465244e1ce Added dynamic table sorting in search page. (Still need to add dependencies) 2016-07-07 16:38:00 +02:00
Mokaddem
7ff9b9a583 Added DomainTrending seems working. 
Started search features with related html pages, not finish yet.
 2016-07-05 16:53:03 +02:00
Mokaddem
5a9eca9291 Added few comments 2016-07-04 09:18:23 +02:00
Mokaddem
4b3101b7b6 Added template tld. Modified URL using Faup and refactored WebStats. 2016-07-01 16:59:08 +02:00
Mokaddem
beeeb76de9 Added new modules and started WebTrending web interface 2016-06-30 14:38:28 +02:00
Mokaddem
3dc356dc5e Getting Started: Initial configuration working on the laptop 2016-06-30 14:36:47 +02:00
MaximeStor
ab66cd255a Improve SourceCode, keywords and add description in /doc 2016-03-12 12:30:38 +01:00
MaximeStor
701d771aa5 Add first version of Source Code 2016-03-12 11:21:29 +01:00
Raphaël Vinot
be86737ca7 Because 0MQ fails. 2016-03-11 16:16:53 +01:00
Raphaël Vinot
f6e4ea2270 Fix logging, fix URL regex 2016-02-11 12:19:03 +01:00
Raphaël Vinot
d160e4a2c8 Add Credential in the scripts to launch 2016-02-10 17:33:16 +01:00
Raphaël Vinot
90e1b25426 Split filepath and count in credential module 2016-02-10 17:31:52 +01:00
Raphaël Vinot
4895ee9fa2 Add new category (Credential) 2016-02-10 16:39:56 +01:00
Raphaël Vinot
1da8675750 Refactoring on Credential, Phone and Release 2016-02-10 16:39:06 +01:00
Alexandre Dulaunoy
c68136b04b Merge branch 'pgp' of https://github.com/Rafiot/AIL-framework 
Conflicts:
	bin/packages/modules.cfg
 2016-02-08 10:13:44 +01:00
MaximeStor
b7d2b64a86 Merge branch 'master' of https://github.com/CIRCL/AIL-framework into module 2016-02-06 11:28:48 +01:00
Alexandre Dulaunoy
192ee7f0ec Merge pull request #49 from Alainfou/master 
Phone module added (regex needs optimization)
 2016-02-06 11:20:46 +01:00
Alain
ea52fd1068 Phone regex updated 
Might still need to be fixed / optimized, in case of maths or random numbers starting with a 0. Do not capture dates, hours, coordinates anymore. Captured formats are: e.g. +331234567890 ; 09 12 34 56 78 ; +4177/123.45.69 ; +352(0)6-23-23-23...
 2016-02-05 20:58:02 +01:00
Alain
fabbfd8ae9 Update module.cfg (adding Keys and Phone section) 2016-02-05 14:00:41 -05:00
Alain
43b3556588 Starting Phone number recognition 2016-02-05 13:58:21 -05:00
MaximeStor
07513a5b37 Add modules Credential and Release 2016-02-05 16:15:09 +01:00
Raphaël Vinot
9171d5b118 Add module to find PGP encrypted blobs 2016-02-05 16:03:37 +01:00
Raphaël Vinot
aef8ab0411 Listen locally for 0MQ 2016-02-04 15:32:50 +01:00
Raphaël Vinot
5ca13c42eb Launch redis and leveldb from local directory 2016-02-04 15:24:39 +01:00
Raphaël Vinot
12aca6b760 Add script to import from local directory, use local python from env 2016-02-04 15:22:51 +01:00
Raphaël Vinot
315cb48117 Add template file for writing a new module 2016-02-03 10:33:42 +01:00
Raphaël Vinot
0d6adc2063 Add initial Travis file 2016-01-19 11:43:34 +01:00
Alexandre Dulaunoy
cdd0725e88 -v option added to list the path 2015-12-22 21:37:05 +00:00
Alexandre Dulaunoy
e3971ac93a Onion fetching loop deactivated by default 2014-12-22 16:06:38 +00:00
Raphaël Vinot
08ceefc375 Re-add config option 2014-12-22 16:50:25 +01:00
Raphaël Vinot
50369c6706 Revert changes on the config file due to merging messup 2014-12-22 16:29:05 +01:00
Raphaël Vinot
f717f9fe89 Merge branch 'master' of github.com:CIRCL/AIL-framework 2014-12-22 15:32:48 +01:00
Raphaël Vinot
9ee61db2cf Add hotfixes 2014-12-22 15:27:02 +01:00
Raphaël Vinot
8803c8447a Publish the fetched onions on a ZMQ feed. 2014-09-30 16:55:16 +02:00
Alexandre Dulaunoy
25757b0fff A simple feeder script feeding data from pystemon to AIL. 
The configuration matches the default Redis parameters used
in the pystemon configuration.

https://github.com/cvandeplas/pystemon/blob/master/pystemon.yaml#L16
 2014-09-19 14:03:05 +02:00
Raphaël Vinot
65b9a01644 Add config file for DomainClassifier, proper reporting 2014-09-17 17:22:56 +02:00