cf903cc212
(partially fix #90 too) using a simple Alarm (SIGNAL) when exec-timeout
...
Introducing a timer (in this case 60 seconds) to ensure that the
execution time of the analysis takes less than 60 seconds. This
is a simple and standard POSIX signal handler. If the timeout
is reached, the module will process the next one.
This approach fixes the specific issues we have currently
with some inputs where the sentiment analysis takes too much time. This
fix should be improved and be more generic:
- Introducing statistics of content which timeouts.
- Keeping a list/queue to further process those files using a different
analysis approach. Maybe a set of "dirty" processes to handle the edge cases
and to not impact the overall processing and analysis.
- Make the timer configurable per module (at least for this one).
2017-01-26 07:11:18 +00:00
3b101ea8f5
(partially) Fix #91 using a simple Alarm (SIGNAL) when exec-timeout
...
Introducing a timer (in this case 5 seconds) to ensure that the
execution time of the tokenizer takes less than 5 seconds. This
is a simple and standard POSIX signal handler.
This approach fixes the specific issues we have currently
with some inputs where the tokenization takes too much time. This
fix should be improved and be more generic:
- Introducing statistics of content which timeouts.
- Keeping a list/queue to further process those files using a different
tokenizer approach. Maybe a set of "dirty" processes to handle the edge cases
and to not impact the overall processing and analysis.
- Make the timer configurable per module (at least for this one).
2017-01-12 07:32:55 +00:00
1950a2dc0e
Merge pull request #99 from mokaddem/more-feed
...
Usage of sha1 to filter paste content in Mixer.py
2017-01-09 17:06:37 +01:00
Mokaddem
500608d0a1
Usage of sha1 to filter paste content in Mixer.py
2017-01-09 17:01:10 +01:00
f5adb65cb3
Merge pull request #98 from mokaddem/more-feed
...
Config 2017
2017-01-09 15:39:53 +01:00
Mokaddem
f8c28796d2
Merge branch 'more-feed' of github.com:mokaddem/AIL-framework into more-feed
2017-01-09 15:37:58 +01:00
Mokaddem
409c5b8b42
Added 2017
2017-01-09 15:36:50 +01:00
a96ba40d94
Merge pull request #96 from mokaddem/more-feed
...
Handling of multiple feeders
2017-01-09 15:29:28 +01:00
Mokaddem
29241eb49f
Added a line separator between the two graphs in dashboard.
2017-01-09 14:44:14 +01:00
Mokaddem
0e39f516a9
Added sleep in helper for multiple listening interfaces (reduce useless work), Added a simple way to pre-process feed before sending it to the global feed.
2017-01-09 14:12:26 +01:00
Mokaddem
24c5621d29
BUG FIX in Flask_trendingmodule, Fixed member_set when get_top_relevant_data is None.
2016-12-26 16:33:10 +01:00
Mokaddem
3a298f0c7c
Commented Mixer.py for further multiple feeds support.
2016-12-26 16:23:18 +01:00
Mokaddem
1c2169d3bc
Improved interface for multiple feeds and refresh_script.
2016-12-26 16:16:44 +01:00
Mokaddem
a18c046dea
Fixed bug multiple refresh instances and harmonized interface.
2016-12-23 16:15:05 +01:00
Mokaddem
03dddbc359
Reduced refresh rate of processed_pastes, synchro graphs and adjusted max on each graphs.
2016-12-23 16:01:30 +01:00
Mokaddem
97292e0899
Updated web interface to handle new mixer module. and fixed one dependency bug.
2016-12-23 15:44:46 +01:00
Mokaddem
e70b9cd15c
Added basic mixer with confirugable behavior. It handles muliple feeders and performs some basic stats on them.
2016-12-23 10:31:26 +01:00
ddfe3cf66a
Merge pull request #95 from mokaddem/module_handling
...
Module handling
2016-12-22 22:18:31 +01:00
3ce850e093
Merge pull request #94 from mokaddem/website-refacto
...
Website refacto, support and fixes
2016-12-22 22:17:58 +01:00
Mokaddem
9758579753
Merge branch 'module_handling' into production
2016-12-22 10:24:30 +01:00
Mokaddem
7438f16c63
Added new config
2016-12-22 10:06:35 +01:00
Mokaddem
faa8f8302d
Removed typo in the number of remaining pastes to display in browseImportantPaste
2016-12-22 09:28:55 +01:00
Mokaddem
9df12cdae8
Added sub-flask in a directory
2016-12-22 08:57:45 +01:00
Mokaddem
73938cf41b
Prepared moduleInfo to handle module click with curses (will come back later).
2016-12-22 08:45:53 +01:00
Mokaddem
d37b243e1d
Added legend + added more info if restarting disabled
2016-12-09 15:13:47 +01:00
Mokaddem
d30f3ca6c7
Added missing depency datetime in dashboard
2016-12-09 13:53:57 +01:00
Mokaddem
0054be5bab
Added missing json dependency in dashboard
2016-12-09 08:50:36 +01:00
Mokaddem
8daa72789e
Splitted Flask_server into module related to website sections
2016-12-09 08:46:37 +01:00
Mokaddem
d3ed1bbf40
Merge branch 'website-improvement' into production
2016-12-08 10:05:55 +01:00
Mokaddem
73d4f9e082
Webstats should correctly updates top_progression_zset (Not fully tested because not enough data. Will be tested latter)
2016-12-08 10:05:07 +01:00
Mokaddem
570324060e
terms top_sets correctly supports blacklisted terms
2016-12-08 09:13:31 +01:00
Mokaddem
1abba4dcf9
Added support of re-plotting the plotted terms
2016-12-08 08:44:10 +01:00
412b80c44d
Merge pull request #88 from mokaddem/Issue-Feeding-AIL
...
Pystemon tuto
2016-11-26 12:45:28 +01:00
Mokaddem
84dcd5586f
Started plotting same data with different dates
2016-11-25 16:42:22 +01:00
Mokaddem
c2e5d69e21
Merge branch 'module_handling' into production
2016-11-25 16:01:55 +01:00
Mokaddem
0cabd5c4fe
Merge branch 'website-improvement' into production
2016-11-25 16:01:40 +01:00
Mokaddem
d831a79614
Do not try to restart not running queue if autokill is not set
2016-11-25 16:00:46 +01:00
Mokaddem
4bf45eea36
ModuleInformation sort by name as well as elapsed time
2016-11-25 14:44:27 +01:00
Mokaddem
866be4e884
Added row coloring in ModuleInformation
2016-11-25 14:12:09 +01:00
Mokaddem
a3255d168c
ModuleInfo correctly handle CurveManageTopSets,
...
Changed queue coloring in dashboard,
ModuleInformation now have a history of executed command and better handle module killing.
2016-11-25 11:54:16 +01:00
Mokaddem
224fbc8084
Better handle stuck modules. Differentiate between not running and no info and tries to restart stuck ones.
2016-11-24 16:58:32 +01:00
Mokaddem
5d269ea1ee
Added date in Duplicate for better distinguish duplicate. Also, added a timeout for the dataTable in duplicate inside the modal.
2016-11-24 15:05:29 +01:00
Mokaddem
df8d978088
Added dynamic data loading in dataTable in search.html
2016-11-24 13:31:31 +01:00
Mokaddem
80ca49f018
Prevent browser to request reconnection for the eventStream + Added dynamic data loading in the dataTable of BrowseImportantPastes
2016-11-11 16:51:39 +01:00
Mokaddem
768fb6f2df
removed server overhead for paste preview data
2016-11-10 16:41:07 +01:00
Mokaddem
699c819387
fixed bug in show concerned pastes previex en (terms managements)
2016-11-10 16:33:05 +01:00
Mokaddem
94e3034e7d
Fixed bun in terms management, where action_button was not bind in other dataTable pages
2016-11-10 15:53:22 +01:00
Mokaddem
7763bfb4c7
Pastes dataTable now display only the latest clicked paste even if there were discarded queries
2016-11-10 15:39:45 +01:00
Mokaddem
2dd69fbb25
Fixed modal background stacking bug + button leading to saved paste bug fixed in search
2016-11-10 14:56:25 +01:00
Mokaddem
317005c2b4
tryed to fix html display problem in tooltip + fixed problem button showPaste in modal not pointing to the paste
2016-11-10 10:30:29 +01:00
