Sami Mokaddem
ef1bd16a72
typo: added word + deleted trailing spaces
2017-12-12 11:16:40 +01:00
Sami Mokaddem
304213cfa4
update: Possibility to change the authorized max line length in the config for the diff of duplicates
2017-12-12 11:00:26 +01:00
Sami Mokaddem
860acbf7cf
fix: Renamed Credentials into Credential sectiont title
...
config change: By default, zmq listen to both local and CRF
2017-12-12 09:29:15 +01:00
Sami Mokaddem
9d0d0b4303
update: Moved filtering operation (thresholds, number of matching in the
...
categ file) in the configuration file. It permits to better control the
flow of pastes.
Also set default mixer duplicate filtering to 3 (Do not filter)
2017-12-11 17:28:34 +01:00
Fabien Mathey
3105192312
Allow for multiple entries like 1 2 3 4 5
...
Should replace the need to type
1<Enter>2<Enter>
etc. and allows for
1 2 3 4 5<Enter>
2017-11-28 15:15:12 +01:00
71922f9def
add: Python script to test if the ZMQ feed works as expected
2017-11-28 09:46:06 +01:00
Sami Mokaddem
83e082e62a
update: removed useless comments
2017-11-24 08:57:41 +01:00
Sami Mokaddem
a0d07b1098
feature: try to send data to MISP if keys are provided, else do nothing
2017-11-23 13:47:02 +01:00
Sami Mokaddem
2a967c4d92
update/feature: Max number of duplicate push to MISP + duplicate are pushed as attachment
2017-11-23 07:13:44 +01:00
Sami Mokaddem
805171a8a5
Added a seemingly working version of ailleak to misp
2017-11-20 14:57:25 +01:00
Sami Mokaddem
e0b188c069
Better support of python3 in Helper.py
2017-11-20 14:55:36 +01:00
Sami Mokaddem
1e76e10c01
Support of add_object to a MISP instance
2017-11-20 12:12:53 +01:00
Sami Mokaddem
6f0227a78a
Starting support of python3
2017-11-20 12:12:06 +01:00
Sami Mokaddem
ee7759be91
Updated alertHandler and ailleakObject
2017-11-16 11:18:13 +01:00
Sami Mokaddem
5c7c07fbea
Merge branch 'launcher_update' into misp_obj
2017-11-16 10:09:05 +01:00
Sami Mokaddem
45a9897020
Added draft support of MISP ail-leak object
2017-11-16 09:52:37 +01:00
Sami Mokaddem
652b00f4b2
Renamed BrowseWarningPaste into alertHandler
2017-11-15 16:15:43 +01:00
Sami Mokaddem
ec7d2d49b2
Added seconds args in import_dir
2017-11-15 16:03:42 +01:00
Sami Mokaddem
13f1c676d6
Added '_AIL' to the end of screen session name to better check an kills sessions. Also, gracefully close redis server + Check 1 time if redis-servers are running.
2017-09-26 16:17:06 +02:00
=
a8dc9231c6
Typo2..
2017-08-24 16:43:42 +02:00
=
b2d4dd4a1f
Typo
2017-08-24 16:43:05 +02:00
=
74ff401e1d
Fixed bug: No CR between indexes in all_index.txt file
2017-08-24 16:41:30 +02:00
bb128a5e94
Merge pull request #138 from mokaddem/import_dir_update
...
Improvement of import_dir
2017-08-24 09:40:52 +02:00
=
429cc96d2a
Improvement of import_dir.py. Added feed name parameter, send the correct path (without potential ../) and gzipEncode it if needed.
2017-08-23 15:10:26 +02:00
=
e7c66d1aac
Updated individual launch scripts
2017-08-23 12:28:12 +02:00
=
d632335760
Automatically start redis_levelDB with the correct date
2017-08-23 10:16:56 +02:00
=
64ec44085d
Dynamically choose correct port for LevelDB in script
2017-08-22 17:52:15 +02:00
=
66d60a1f01
Added LevelDB Curve for 2017 (3017, used for backward compatibility)
2017-08-21 13:54:44 +02:00
=
92adea38d0
Semi-automated directory creation with date as name in LAUNCH and installing_deps
2017-08-21 13:48:02 +02:00
=
bc35747f00
Updated KillAll in LAUNCH
2017-08-21 13:19:26 +02:00
Mokaddem
c3632c7d86
Added operation mode 3 for Mixer (do not filter) and added possibility to recurse through dir for import_dir
2017-08-03 00:34:53 +02:00
Mokaddem
cbb2909156
Added comments
2017-07-20 10:24:48 +02:00
Mokaddem
05417f405c
Finished basic link web interface + solved some bugs
2017-07-20 10:04:30 +02:00
Mokaddem
ef741f85c0
Added similarity checks + started link webInterface
2017-07-19 11:52:06 +02:00
Mokaddem
882ab88b00
Draft module + web link
2017-07-18 16:57:15 +02:00
Mokaddem
9558aafe67
Prevent the loss of the 1 message in import_dir.py
2017-07-17 16:14:27 +02:00
Mokaddem
4d59871f00
Merged with previous changes for PR
2017-07-17 13:48:16 +02:00
4986352099
Merge pull request #120 from PaulSec/master
...
Added Python Phonenumbers module to parse numbers and retrieves stats…
2017-06-08 07:05:52 +02:00
446a97c47a
Merge pull request #115 from mokaddem/regexInTerms
...
New features for terms frequency
2017-06-08 07:04:17 +02:00
Mokaddem
3a4dcd691d
Improved description of modules inside the scripts
2017-05-09 11:13:16 +02:00
Paul A
747fff1df1
Added Python Phonenumbers module to parse numbers and retrieves stats from countrycode
2017-05-03 14:25:18 +02:00
Mokaddem
3ea0924120
Improved phone regex to handle end with 4 digits and presence of parenthesis.
2017-04-26 12:41:23 +02:00
Mokaddem
b60e164380
Added support of onion in browseWarningPaste
2017-04-26 11:47:02 +02:00
Mokaddem
318a557c56
Added possibility to keep a backup of the old config + check if config is valid before starting scripts.
2017-04-25 09:32:05 +02:00
Mokaddem
72c1d903eb
Merge branch 'newIndexer' into auto-conf
2017-04-25 08:14:31 +02:00
Mokaddem
eccc8fc88b
modified check_size timeout + utilisation of shutil instead of os.system()
2017-04-19 16:37:04 +02:00
Mokaddem
da081bae42
Merge branch 'regexInTerms' into production
2017-04-18 16:59:40 +02:00
Mokaddem
2d907ae784
Merge branch 'newIndexer' into auto-conf
2017-04-18 16:20:00 +02:00
Mokaddem
b0f1cf84b7
remplaced usage of global variable by a function param
2017-04-18 16:02:22 +02:00
Mokaddem
6e8509a0ba
Removed wierd line
2017-04-18 15:57:07 +02:00
Mokaddem
c8baabd882
Updated TermsFrequency related modules + Fixed bugs (encoding, behaviors, ...)
2017-04-18 15:28:21 +02:00
Mokaddem
2da4c572c7
New feature in Terms: regex and set of words support (draft)
2017-03-28 17:42:44 +02:00
Mokaddem
6e80eb0954
Typo
2017-03-28 09:55:59 +02:00
Mokaddem
c79d4d65c8
Keep original section order while updating
2017-03-28 09:54:24 +02:00
Mokaddem
e225090e07
Added script that checks configuration and may auto-fix it + added missing field in config.cfg.sample
2017-03-15 16:36:51 +01:00
Mokaddem
be93af6f5a
Merge branch 'terms-improvements'
2017-03-15 14:55:44 +01:00
Mokaddem
3261eca836
Fixed all_index newline bug + added print of what's happening
2017-03-15 14:29:49 +01:00
Mokaddem
5b7d047622
Changed os.system by os.mkdir + fixed bug during rotation
2017-03-15 14:05:13 +01:00
Mokaddem
adc37ab50e
Indexer max size now in config
2017-03-15 12:14:41 +01:00
Mokaddem
4005b72180
Added backward support + renamed index with timestamp
2017-03-15 11:51:35 +01:00
Mokaddem
248469d61e
Indexer now create an index_dir when it became too large (search in index not updated yet)
2017-03-14 10:37:31 +01:00
1540b719ed
Merge pull request #108 from mokaddem/newModuleInformation
...
New version of ModuleInformation
2017-03-14 08:29:09 +01:00
Mokaddem
3c9cdac39a
Merge branch 'terms-improvements' into production
2017-02-28 16:51:39 +01:00
Mokaddem
dadc872f9f
Merge branch 'newModuleInformation' into production
2017-02-28 16:51:08 +01:00
Mokaddem
a2849c1bcd
Added a catch for psutils if process get stopped by user
2017-02-28 16:13:01 +01:00
Mokaddem
4071bf2e91
fixed a bug in manage_term (usage of loop.index0) + fixed bug in top_set where graph was not displaying the correct per_paste value
2017-02-28 15:01:48 +01:00
Mokaddem
a1c47d7a13
Merge branch 'production' into terms-improvements
2017-02-28 11:15:56 +01:00
Mokaddem
fd4feb374f
Merge branch 'newModuleInformation' into production
2017-02-28 11:12:06 +01:00
Mokaddem
75b4037519
Fixed bug - If we move the cursor from a line that has dissapear
2017-02-28 11:11:17 +01:00
Mokaddem
d6e02591af
Merge branch 'newModuleInformation' into production
2017-02-28 10:23:45 +01:00
Mokaddem
975c1a37d9
ModuleInformation create the all_modules files if missing
2017-02-28 10:23:22 +01:00
Mokaddem
ccdf256383
Added possibility to show or not in dashboard text that is logged
2017-02-28 10:21:27 +01:00
Mokaddem
39104dea5e
Merge branch 'bugfix_categ' into production
2017-02-28 09:56:25 +01:00
Mokaddem
26eb5fdff0
Fixed bug closing parenthesis in categ + changed behavior of sending to browseWarningPaste in Mail
2017-02-28 09:23:32 +01:00
Mokaddem
4a0dac1fae
Merge branch 'production' into terms-improvements
2017-02-15 16:31:22 +01:00
Mokaddem
a696ccf83b
Merge branch 'newModuleInformation' into production
2017-02-15 16:30:44 +01:00
Mokaddem
85a6f6fa63
Fixed a bug (str+float)
2017-02-15 16:30:31 +01:00
Mokaddem
25adae0104
Added possibility to select in top_terms the data per_paste or not (need module testing)
2017-02-15 16:29:02 +01:00
Mokaddem
e98e92b0d4
Merge branch 'fix-linkPaste-debug' into production
2017-02-15 13:43:03 +01:00
Mokaddem
c552aced42
Merge branch 'newModuleInformation' into production
2017-02-15 13:42:46 +01:00
Mokaddem
1b0d9d9507
Fixed str and int concat + Added sorting by queue name in queue not running
2017-02-15 13:42:21 +01:00
Mokaddem
74e0bc7acb
Merge branch 'newModuleInformation' into production
2017-02-15 11:22:50 +01:00
Mokaddem
7fc9ba18d7
Added support of ENTER for navigation
2017-02-15 11:01:11 +01:00
Mokaddem
b84194542d
Moved logging into a function
2017-02-15 10:57:41 +01:00
Mokaddem
0fce3f5d65
Renamed variables + added comments
2017-02-15 10:37:59 +01:00
Mokaddem
9ca3e79e70
Removed useless comments + commented
2017-02-15 10:19:03 +01:00
Mokaddem
9199ea3f0e
Added comments
2017-02-15 10:04:51 +01:00
Mokaddem
ff1df07b5d
Added comments and moved portion of code for slightly better visibility
2017-02-15 09:43:18 +01:00
Mokaddem
b272b1b07b
Added a way to quit application with 'q' more or less properly
2017-02-15 09:35:03 +01:00
Mokaddem
0254c51164
Added end of paste marker + handling screen resize
2017-02-15 08:26:48 +01:00
Mokaddem
0b9609e4a9
Merge branch 'newModuleInformation' into production
2017-02-14 17:22:06 +01:00
Mokaddem
444f6071b7
Fixed screen bug by adding an input before starting asciimatics + improved path localisation in the logs from modules
2017-02-14 17:21:52 +01:00
Mokaddem
b04bac4731
Merge branch 'newModuleInformation' into production
2017-02-14 16:08:42 +01:00
Mokaddem
d520c45a15
Added support of display of the last paste processed by a module
2017-02-14 16:05:59 +01:00
Mokaddem
d3b4409156
Added path for debug in SQLInjection
2017-02-14 11:23:34 +01:00
Mokaddem
135a1457e0
Merge branch 'fix-linkPaste-debug' into production
2017-02-14 11:12:22 +01:00
Mokaddem
d4da3a316f
Added path of the paste in the log of Categ.py
2017-02-14 10:59:47 +01:00
Mokaddem
303575b2a1
Merge remote-tracking branch 'upstream/master' into production
2017-02-14 09:31:18 +01:00
2f4e5b9016
Merge branch 'master' of github.com:CIRCL/AIL-framework
2017-01-26 07:16:44 +00:00
cf903cc212
(partially fix #90 too) using a simple Alarm (SIGNAL) when exec-timeout
...
Introducing a timer (in this case 60 seconds) to ensure that the
execution time of the analysis takes less than 60 seconds. This
is a simple and standard POSIX signal handler. If the timeout
is reached, the module will process the next one.
This approach fixes the specific issues we have currently
with some inputs where the sentiment analysis takes too much time. This
fix should be improved and be more generic:
- Introducing statistics of content which timeouts.
- Keeping a list/queue to further process those files using a different
analysis approach. Maybe a set of "dirty" processes to handle the edge cases
and to not impact the overall processing and analysis.
- Make the timer configurable per module (at least for this one).
2017-01-26 07:11:18 +00:00
