e3971ac93a
Onion fetching loop deactivated by default
2014-12-22 16:06:38 +00:00
Raphaël Vinot
08ceefc375
Re-add config option
2014-12-22 16:50:25 +01:00
Raphaël Vinot
50369c6706
Revert changes on the config file due to merging messup
2014-12-22 16:29:05 +01:00
Raphaël Vinot
f717f9fe89
Merge branch 'master' of github.com:CIRCL/AIL-framework
2014-12-22 15:32:48 +01:00
Raphaël Vinot
9ee61db2cf
Add hotfixes
2014-12-22 15:27:02 +01:00
Raphaël Vinot
8803c8447a
Publish the fetched onions on a ZMQ feed.
2014-09-30 16:55:16 +02:00
25757b0fff
A simple feeder script feeding data from pystemon to AIL.
...
The configuration matches the default Redis parameters used
in the pystemon configuration.
https://github.com/cvandeplas/pystemon/blob/master/pystemon.yaml#L16
2014-09-19 14:03:05 +02:00
Raphaël Vinot
65b9a01644
Add config file for DomainClassifier, proper reporting
2014-09-17 17:22:56 +02:00
27b134ec03
Add proper publisher for classified domains/hostnames
2014-09-10 09:27:47 +02:00
Raphaël Vinot
f017680365
fix onions, cc and domain classifier modules
2014-09-08 16:51:43 +02:00
de6e21d5a7
DomainClassifier sample configuration added
2014-09-08 16:44:05 +02:00
246621f663
First version of the DomainClassifier
2014-09-08 16:43:21 +02:00
1397db9691
Global queue for DomainClassifier
2014-09-08 11:07:45 +02:00
Raphaël Vinot
e983c839ad
Categ now listen to the Global queue
2014-09-05 17:05:45 +02:00
Raphaël Vinot
46f27ada4e
More cleanup
2014-09-05 10:42:01 +02:00
Raphaël Vinot
fca00beed9
Add Domain Classifier module.
...
Cleanup in the config files.
2014-09-05 10:41:00 +02:00
Raphaël Vinot
b7c9e489c9
Fix the exceptions
2014-09-04 11:46:07 +02:00
Raphaël Vinot
9e8611a42d
stop killing the disk when creating the word curve
2014-09-02 18:20:28 +02:00
Raphaël Vinot
7542eaf739
Update starting script.
2014-09-02 15:21:36 +02:00
Raphaël Vinot
0c6b09f379
Fix the onion module, log the valid onions.
2014-09-01 16:18:06 +02:00
Raphaël Vinot
f4b89669fc
The onion module now fetches the URLs it finds.
2014-08-31 22:42:12 +02:00
Raphaël Vinot
abfe13436b
Big refactoring, make the queues more flexible
2014-08-29 19:37:56 +02:00
Raphaël Vinot
623e876f3b
Cleanup.
...
* Remove useless subscriber
* Fix typo in the config file
* Update Helper accordingly
2014-08-26 17:36:57 +02:00
3b499a2ec8
ZMQ Publisher removed
...
ZMQ Publisher removed to allow concurrent use of the scripts.
In short term, we would replace all publishing part within AIL
into pub-sub Redis to avoid ZMQ limitation.
2014-08-26 14:38:49 +02:00
f070ac2005
cymruwhois uses dotted decimal format
2014-08-25 10:05:36 +02:00
Raphaël Vinot
3886d1b834
Small fixes to make the refactoring production ready
...
* the port for the logging is 6380
* use os.environ properly
* fix typos
2014-08-22 17:35:40 +02:00
Raphaël Vinot
78125db4ea
Use env variables everywhere
2014-08-22 14:52:02 +02:00
Raphaël Vinot
277d138a5d
cleanup, add FIXME
2014-08-21 14:39:17 +02:00
Raphaël Vinot
63b29176c1
move Redis_Data_Merging to Paste
2014-08-21 12:22:07 +02:00
Raphaël Vinot
50cfac857e
Update config
...
Make all paths in the config file relative to the home directory.
2014-08-20 16:00:56 +02:00
Raphaël Vinot
a68f5b6a0e
fix subscriber names, update default config
2014-08-20 15:54:21 +02:00
Raphaël Vinot
2485ba5df2
Merge remote-tracking branch 'origin/master' into testing
...
Conflicts:
bin/ZMQ_Sub_Urls.py
2014-08-20 15:24:10 +02:00
Raphaël Vinot
99c8cc7941
completely remove ZMQ_PubSub.py
2014-08-20 15:14:57 +02:00
1d64dc44c8
MIME type guessing - removed one duplicate call to libmagic
2014-08-20 10:22:33 +02:00
Raphaël Vinot
8d9ffbaa53
Do not create a ZMQ sub if it is not required.
2014-08-19 19:53:33 +02:00
Raphaël Vinot
45b0bf3983
Improve the cleanup. Still some to do.
2014-08-19 19:07:07 +02:00
Raphaël Vinot
f1753d67c6
Cleanup the queues.
2014-08-19 16:05:37 +02:00
e8fcea6cd6
Remove undeclared variable
2014-08-18 16:17:36 +02:00
7d8ee102a3
Assignment before use (if Enumerate fails)
2014-08-18 15:58:06 +02:00
4304c6858e
Configuration path fixed
2014-08-18 09:02:08 +02:00
Raphaël Vinot
078c8ea836
Big cleanup, pep8
2014-08-14 18:07:18 +02:00
Jules
ab6765315e
Merge pull request #13 from adulau/master
...
Log where URLs are hosted - cc_critical option added
2014-08-14 14:28:01 +02:00
762def3a23
Log where URLs are hosted - cc_critical option added
...
It logs where the hostname of the URL is hosted (ASN and geographic location).
A simple option cc_critical added to set the country code to log as critical.
2014-08-14 14:22:11 +02:00
Raphaël Vinot
4a1f300a1a
Cleanup (remove unused imports, more pep8 compatible)
2014-08-14 14:11:07 +02:00
Starow
04a8f1bdf2
maxi cleanup old code :'(
2014-08-14 11:48:46 +02:00
Starow
29b24b6466
printing set of domain for debugging
2014-08-13 16:35:27 +02:00
Raphaël Vinot
ece3bc173e
Cleanup of main Paste module
2014-08-13 11:56:22 +02:00
Raphaël Vinot
5b17d416c8
remove script installed by pubsublogger
2014-08-13 11:55:59 +02:00
Raphaël Vinot
935e51c961
Remove 3rd party code (pubsublogger), add it in the deps.
2014-08-13 10:19:43 +02:00
Starow
37033ca3a6
Minor logs modifications
2014-08-13 10:08:44 +02:00
