Mokaddem
3261eca836
Fixed all_index newline bug + added print of what's happening
2017-03-15 14:29:49 +01:00
Mokaddem
5b7d047622
Changed os.system by os.mkdir + fixed bug during rotation
2017-03-15 14:05:13 +01:00
Mokaddem
adc37ab50e
Indexer max size now in config
2017-03-15 12:14:41 +01:00
Mokaddem
4005b72180
Added backward support + renamed index with timestamp
2017-03-15 11:51:35 +01:00
Mokaddem
248469d61e
Indexer now create an index_dir when it became too large (search in index not updated yet)
2017-03-14 10:37:31 +01:00
1540b719ed
Merge pull request #108 from mokaddem/newModuleInformation
...
New version of ModuleInformation
2017-03-14 08:29:09 +01:00
Mokaddem
3c9cdac39a
Merge branch 'terms-improvements' into production
2017-02-28 16:51:39 +01:00
Mokaddem
dadc872f9f
Merge branch 'newModuleInformation' into production
2017-02-28 16:51:08 +01:00
Mokaddem
a2849c1bcd
Added a catch for psutils if process get stopped by user
2017-02-28 16:13:01 +01:00
Mokaddem
4071bf2e91
fixed a bug in manage_term (usage of loop.index0) + fixed bug in top_set where graph was not displaying the correct per_paste value
2017-02-28 15:01:48 +01:00
Mokaddem
a1c47d7a13
Merge branch 'production' into terms-improvements
2017-02-28 11:15:56 +01:00
Mokaddem
fd4feb374f
Merge branch 'newModuleInformation' into production
2017-02-28 11:12:06 +01:00
Mokaddem
75b4037519
Fixed bug - If we move the cursor from a line that has dissapear
2017-02-28 11:11:17 +01:00
Mokaddem
d6e02591af
Merge branch 'newModuleInformation' into production
2017-02-28 10:23:45 +01:00
Mokaddem
975c1a37d9
ModuleInformation create the all_modules files if missing
2017-02-28 10:23:22 +01:00
Mokaddem
ccdf256383
Added possibility to show or not in dashboard text that is logged
2017-02-28 10:21:27 +01:00
Mokaddem
39104dea5e
Merge branch 'bugfix_categ' into production
2017-02-28 09:56:25 +01:00
Mokaddem
26eb5fdff0
Fixed bug closing parenthesis in categ + changed behavior of sending to browseWarningPaste in Mail
2017-02-28 09:23:32 +01:00
Mokaddem
4a0dac1fae
Merge branch 'production' into terms-improvements
2017-02-15 16:31:22 +01:00
Mokaddem
a696ccf83b
Merge branch 'newModuleInformation' into production
2017-02-15 16:30:44 +01:00
Mokaddem
85a6f6fa63
Fixed a bug (str+float)
2017-02-15 16:30:31 +01:00
Mokaddem
25adae0104
Added possibility to select in top_terms the data per_paste or not (need module testing)
2017-02-15 16:29:02 +01:00
Mokaddem
e98e92b0d4
Merge branch 'fix-linkPaste-debug' into production
2017-02-15 13:43:03 +01:00
Mokaddem
c552aced42
Merge branch 'newModuleInformation' into production
2017-02-15 13:42:46 +01:00
Mokaddem
1b0d9d9507
Fixed str and int concat + Added sorting by queue name in queue not running
2017-02-15 13:42:21 +01:00
Mokaddem
74e0bc7acb
Merge branch 'newModuleInformation' into production
2017-02-15 11:22:50 +01:00
Mokaddem
7fc9ba18d7
Added support of ENTER for navigation
2017-02-15 11:01:11 +01:00
Mokaddem
b84194542d
Moved logging into a function
2017-02-15 10:57:41 +01:00
Mokaddem
0fce3f5d65
Renamed variables + added comments
2017-02-15 10:37:59 +01:00
Mokaddem
9ca3e79e70
Removed useless comments + commented
2017-02-15 10:19:03 +01:00
Mokaddem
9199ea3f0e
Added comments
2017-02-15 10:04:51 +01:00
Mokaddem
ff1df07b5d
Added comments and moved portion of code for slightly better visibility
2017-02-15 09:43:18 +01:00
Mokaddem
b272b1b07b
Added a way to quit application with 'q' more or less properly
2017-02-15 09:35:03 +01:00
Mokaddem
0254c51164
Added end of paste marker + handling screen resize
2017-02-15 08:26:48 +01:00
Mokaddem
0b9609e4a9
Merge branch 'newModuleInformation' into production
2017-02-14 17:22:06 +01:00
Mokaddem
444f6071b7
Fixed screen bug by adding an input before starting asciimatics + improved path localisation in the logs from modules
2017-02-14 17:21:52 +01:00
Mokaddem
b04bac4731
Merge branch 'newModuleInformation' into production
2017-02-14 16:08:42 +01:00
Mokaddem
d520c45a15
Added support of display of the last paste processed by a module
2017-02-14 16:05:59 +01:00
Mokaddem
d3b4409156
Added path for debug in SQLInjection
2017-02-14 11:23:34 +01:00
Mokaddem
135a1457e0
Merge branch 'fix-linkPaste-debug' into production
2017-02-14 11:12:22 +01:00
Mokaddem
d4da3a316f
Added path of the paste in the log of Categ.py
2017-02-14 10:59:47 +01:00
Mokaddem
303575b2a1
Merge remote-tracking branch 'upstream/master' into production
2017-02-14 09:31:18 +01:00
2f4e5b9016
Merge branch 'master' of github.com:CIRCL/AIL-framework
2017-01-26 07:16:44 +00:00
cf903cc212
(partially fix #90 too) using a simple Alarm (SIGNAL) when exec-timeout
...
Introducing a timer (in this case 60 seconds) to ensure that the
execution time of the analysis takes less than 60 seconds. This
is a simple and standard POSIX signal handler. If the timeout
is reached, the module will process the next one.
This approach fixes the specific issues we have currently
with some inputs where the sentiment analysis takes too much time. This
fix should be improved and be more generic:
- Introducing statistics of content which timeouts.
- Keeping a list/queue to further process those files using a different
analysis approach. Maybe a set of "dirty" processes to handle the edge cases
and to not impact the overall processing and analysis.
- Make the timer configurable per module (at least for this one).
2017-01-26 07:11:18 +00:00
Mokaddem
c422db6e6d
Merge remote-tracking branch 'upstream/master' into production
2017-01-17 16:38:48 +01:00
mokaddem
b3a3616b0b
Merge pull request #102 from ptitdoc/bugfix1
...
Bug fixes in Workers and improvement of pystemon-feeder
2017-01-17 16:35:32 +01:00
Mokaddem
ea423fde0f
Added requirement, better behaviors and confirmation box
2017-01-17 15:58:16 +01:00
Mokaddem
692f13d18b
Improved module + Added support of click and keyboard navigation for killing and starting modules
2017-01-17 14:31:06 +01:00
Mokaddem
3b8149d840
Added action button and support of logs
2017-01-16 17:08:48 +01:00
Mokaddem
0ecfaf00f5
Added color in front of queue related to its state
2017-01-16 15:30:33 +01:00
Mokaddem
af3241c008
Overwritted listbox update function to add formating
2017-01-16 14:41:02 +01:00
Mokaddem
30f303cb99
Added new ModuleInformation
2017-01-16 12:18:23 +01:00
Olivier MEDOC
16044d4d36
pystemon-feeder: add the option bind in configuration file instead of address
...
This is essentially because multiple feeds can be used in the address
configuration variable.
2017-01-13 14:58:18 +01:00
3b101ea8f5
(partially) Fix #91 using a simple Alarm (SIGNAL) when exec-timeout
...
Introducing a timer (in this case 5 seconds) to ensure that the
execution time of the tokenizer takes less than 5 seconds. This
is a simple and standard POSIX signal handler.
This approach fixes the specific issues we have currently
with some inputs where the tokenization takes too much time. This
fix should be improved and be more generic:
- Introducing statistics of content which timeouts.
- Keeping a list/queue to further process those files using a different
tokenizer approach. Maybe a set of "dirty" processes to handle the edge cases
and to not impact the overall processing and analysis.
- Make the timer configurable per module (at least for this one).
2017-01-12 07:32:55 +00:00
Olivier MEDOC
9f9c265cb0
SentimentAnalysis: use lexicon file path from the ail configuration file
2017-01-11 11:07:08 +01:00
ptitdoc
c0fac82031
Add pystemon path in configuration sample
2017-01-10 18:33:46 +01:00
Olivier MEDOC
83db40104c
import pystemon-feeder configuration from the central configuration file
2017-01-10 18:26:36 +01:00
Olivier MEDOC
3dc014dad9
fix invalid variable propagation
2017-01-10 18:26:24 +01:00
Olivier MEDOC
68ca20db8f
fix to use the new pycountry API
2017-01-10 18:26:17 +01:00
1950a2dc0e
Merge pull request #99 from mokaddem/more-feed
...
Usage of sha1 to filter paste content in Mixer.py
2017-01-09 17:06:37 +01:00
Mokaddem
500608d0a1
Usage of sha1 to filter paste content in Mixer.py
2017-01-09 17:01:10 +01:00
f5adb65cb3
Merge pull request #98 from mokaddem/more-feed
...
Config 2017
2017-01-09 15:39:53 +01:00
Mokaddem
409c5b8b42
Added 2017
2017-01-09 15:36:50 +01:00
a96ba40d94
Merge pull request #96 from mokaddem/more-feed
...
Handling of multiple feeders
2017-01-09 15:29:28 +01:00
Mokaddem
0e39f516a9
Added sleep in helper for multiple listening interfaces (reduce useless work), Added a simple way to pre-process feed before sending it to the global feed.
2017-01-09 14:12:26 +01:00
Mokaddem
3a298f0c7c
Commented Mixer.py for further multiple feeds support.
2016-12-26 16:23:18 +01:00
Mokaddem
1c2169d3bc
Improved interface for multiple feeds and refresh_script.
2016-12-26 16:16:44 +01:00
Mokaddem
97292e0899
Updated web interface to handle new mixer module. and fixed one dependency bug.
2016-12-23 15:44:46 +01:00
Mokaddem
e70b9cd15c
Added basic mixer with confirugable behavior. It handles muliple feeders and performs some basic stats on them.
2016-12-23 10:31:26 +01:00
ddfe3cf66a
Merge pull request #95 from mokaddem/module_handling
...
Module handling
2016-12-22 22:18:31 +01:00
3ce850e093
Merge pull request #94 from mokaddem/website-refacto
...
Website refacto, support and fixes
2016-12-22 22:17:58 +01:00
Mokaddem
9758579753
Merge branch 'module_handling' into production
2016-12-22 10:24:30 +01:00
Mokaddem
7438f16c63
Added new config
2016-12-22 10:06:35 +01:00
Mokaddem
73938cf41b
Prepared moduleInfo to handle module click with curses (will come back later).
2016-12-22 08:45:53 +01:00
Mokaddem
d37b243e1d
Added legend + added more info if restarting disabled
2016-12-09 15:13:47 +01:00
Mokaddem
d3ed1bbf40
Merge branch 'website-improvement' into production
2016-12-08 10:05:55 +01:00
Mokaddem
73d4f9e082
Webstats should correctly updates top_progression_zset (Not fully tested because not enough data. Will be tested latter)
2016-12-08 10:05:07 +01:00
Mokaddem
570324060e
terms top_sets correctly supports blacklisted terms
2016-12-08 09:13:31 +01:00
412b80c44d
Merge pull request #88 from mokaddem/Issue-Feeding-AIL
...
Pystemon tuto
2016-11-26 12:45:28 +01:00
Mokaddem
d831a79614
Do not try to restart not running queue if autokill is not set
2016-11-25 16:00:46 +01:00
Mokaddem
4bf45eea36
ModuleInformation sort by name as well as elapsed time
2016-11-25 14:44:27 +01:00
Mokaddem
866be4e884
Added row coloring in ModuleInformation
2016-11-25 14:12:09 +01:00
Mokaddem
a3255d168c
ModuleInfo correctly handle CurveManageTopSets,
...
Changed queue coloring in dashboard,
ModuleInformation now have a history of executed command and better handle module killing.
2016-11-25 11:54:16 +01:00
Mokaddem
224fbc8084
Better handle stuck modules. Differentiate between not running and no info and tries to restart stuck ones.
2016-11-24 16:58:32 +01:00
Mokaddem
5d269ea1ee
Added date in Duplicate for better distinguish duplicate. Also, added a timeout for the dataTable in duplicate inside the modal.
2016-11-24 15:05:29 +01:00
Mokaddem
ac254e0e63
Added support of browsing concerned paste in dashboard for all modules + Show in the page what was the concerned module
2016-10-27 11:50:24 +02:00
Mokaddem
1826b170ec
Added support of local paste view in dashboard for Mails event only
2016-10-27 11:27:26 +02:00
Mokaddem
5a4ea734a2
Fixed path issues in doc + added dependencies related to ssdeep
2016-10-14 14:26:33 +02:00
Mokaddem
4eafca2936
Added pystemon instruction section
2016-10-13 17:13:08 +02:00
eb14005fcf
Fix: as JSON decode needs a string, the empty list is a string.
2016-08-29 12:08:12 +00:00
3b6e3e156e
Merge branch 'master' of github.com:CIRCL/AIL-framework
2016-08-26 15:57:45 +00:00
Mokaddem
0c760d763b
Added support of PID in ModuleInformation and index.html
2016-08-26 15:27:37 +02:00
758f2c5334
Merge branch 'master' of github.com:CIRCL/AIL-framework
2016-08-26 08:30:11 +00:00
Mokaddem
494980c48e
Added comment in moduleInformation
2016-08-26 10:07:06 +02:00
Mokaddem
8b33b4e7ba
Added not running module info + swapped order of module launch
2016-08-26 09:21:02 +02:00
Mokaddem
813cc7031a
Changed lots of code into zrank
2016-08-25 17:48:12 +02:00
Mokaddem
aaac798db9
updated individual launch script
2016-08-25 16:51:57 +02:00
Mokaddem
78c991539e
Added number of same module running in ModuleInformation
2016-08-25 16:07:47 +02:00
1ab08d4a05
Merge branch 'master' of github.com:CIRCL/AIL-framework
2016-08-25 12:33:31 +00:00
Mokaddem
f9bf00f022
Removed autokill feature if not specified
2016-08-25 11:09:31 +02:00
