Sami Mokaddem
c39e537cbe
Merge branch 'iss160' into showDiff
...
Preemptively preventing merge-conflict with branch iss160 (moving to
configuration)
2017-12-12 11:33:22 +01:00
Sami Mokaddem
304213cfa4
update: Possibility to change the authorized max line length in the config for the diff of duplicates
2017-12-12 11:00:26 +01:00
Sami Mokaddem
860acbf7cf
fix: Renamed Credentials into Credential sectiont title
...
config change: By default, zmq listen to both local and CRF
2017-12-12 09:29:15 +01:00
Sami Mokaddem
9d0d0b4303
update: Moved filtering operation (thresholds, number of matching in the
...
categ file) in the configuration file. It permits to better control the
flow of pastes.
Also set default mixer duplicate filtering to 3 (Do not filter)
2017-12-11 17:28:34 +01:00
Sami Mokaddem
2a967c4d92
update/feature: Max number of duplicate push to MISP + duplicate are pushed as attachment
2017-11-23 07:13:44 +01:00
=
64ec44085d
Dynamically choose correct port for LevelDB in script
2017-08-22 17:52:15 +02:00
Mokaddem
882ab88b00
Draft module + web link
2017-07-18 16:57:15 +02:00
Mokaddem
e225090e07
Added script that checks configuration and may auto-fix it + added missing field in config.cfg.sample
2017-03-15 16:36:51 +01:00
Mokaddem
adc37ab50e
Indexer max size now in config
2017-03-15 12:14:41 +01:00
Olivier MEDOC
16044d4d36
pystemon-feeder: add the option bind in configuration file instead of address
...
This is essentially because multiple feeds can be used in the address
configuration variable.
2017-01-13 14:58:18 +01:00
Olivier MEDOC
9f9c265cb0
SentimentAnalysis: use lexicon file path from the ail configuration file
2017-01-11 11:07:08 +01:00
ptitdoc
c0fac82031
Add pystemon path in configuration sample
2017-01-10 18:33:46 +01:00
Mokaddem
0e39f516a9
Added sleep in helper for multiple listening interfaces (reduce useless work), Added a simple way to pre-process feed before sending it to the global feed.
2017-01-09 14:12:26 +01:00
Mokaddem
e70b9cd15c
Added basic mixer with confirugable behavior. It handles muliple feeders and performs some basic stats on them.
2016-12-23 10:31:26 +01:00
Mokaddem
7438f16c63
Added new config
2016-12-22 10:06:35 +01:00
Mokaddem
697812dbff
Added config.sample
2016-08-23 09:59:39 +02:00
Mokaddem
a4595010c0
Added whole config in config.sample
2016-08-09 14:23:36 +02:00
Mokaddem
1c40ad522f
Merge branch 'master' into production
2016-08-08 09:26:58 +02:00
Raphaël Vinot
4aa484a91f
Merge pull request #63 from mokaddem/uptodate-duplicate-module
...
Uptodate duplicate module
2016-07-25 09:44:49 +02:00
Raphaël Vinot
ea8703e608
Fix path to tldsfile
2016-07-22 09:32:13 +02:00
Mokaddem
7c4f4a2f79
Merge branch 'statistics-v2' into uptodate-statistics-v2
2016-07-21 15:16:03 +02:00
Mokaddem
b685f81752
Merge branch 'duplicate-module' into uptodate-duplicate-module
2016-07-21 14:59:52 +02:00
Mokaddem
0ab7560477
Added top_progression chart for tld, domain and scheme + Small modification in config file.
2016-07-21 13:44:22 +02:00
Raphaël Vinot
ede46ec799
Fix default config.
2016-07-21 09:19:13 +02:00
Raphaël Vinot
781b441bb8
Fix path in config
2016-07-20 14:12:50 +02:00
Raphaël Vinot
8a69110c8e
Merge branch 'webpage-splitting' of https://github.com/mokaddem/AIL-framework into mokaddem-webpage-splitting
2016-07-19 16:35:31 +02:00
Mokaddem
6805ed6488
Added default config
2016-07-18 15:52:53 +02:00
Mokaddem
4f6813350b
Added two new version of duplicate module.
...
One with hashes are saved in json on disk
The other with only leveldb
2016-07-18 15:50:41 +02:00
Mokaddem
2383db022f
Added default configuration
2016-07-15 09:10:44 +02:00
Mokaddem
9a9e07f600
Added default configuration
2016-07-15 09:08:38 +02:00
Raphaël Vinot
aef8ab0411
Listen locally for 0MQ
2016-02-04 15:32:50 +01:00
Raphaël Vinot
08ceefc375
Re-add config option
2014-12-22 16:50:25 +01:00
Raphaël Vinot
50369c6706
Revert changes on the config file due to merging messup
2014-12-22 16:29:05 +01:00
Raphaël Vinot
f717f9fe89
Merge branch 'master' of github.com:CIRCL/AIL-framework
2014-12-22 15:32:48 +01:00
Raphaël Vinot
8803c8447a
Publish the fetched onions on a ZMQ feed.
2014-09-30 16:55:16 +02:00
25757b0fff
A simple feeder script feeding data from pystemon to AIL.
...
The configuration matches the default Redis parameters used
in the pystemon configuration.
https://github.com/cvandeplas/pystemon/blob/master/pystemon.yaml#L16
2014-09-19 14:03:05 +02:00
Raphaël Vinot
65b9a01644
Add config file for DomainClassifier, proper reporting
2014-09-17 17:22:56 +02:00
de6e21d5a7
DomainClassifier sample configuration added
2014-09-08 16:44:05 +02:00
Raphaël Vinot
fca00beed9
Add Domain Classifier module.
...
Cleanup in the config files.
2014-09-05 10:41:00 +02:00
Raphaël Vinot
abfe13436b
Big refactoring, make the queues more flexible
2014-08-29 19:37:56 +02:00
Raphaël Vinot
623e876f3b
Cleanup.
...
* Remove useless subscriber
* Fix typo in the config file
* Update Helper accordingly
2014-08-26 17:36:57 +02:00
Raphaël Vinot
50cfac857e
Update config
...
Make all paths in the config file relative to the home directory.
2014-08-20 16:00:56 +02:00
Raphaël Vinot
a68f5b6a0e
fix subscriber names, update default config
2014-08-20 15:54:21 +02:00
Raphaël Vinot
45b0bf3983
Improve the cleanup. Still some to do.
2014-08-19 19:07:07 +02:00
762def3a23
Log where URLs are hosted - cc_critical option added
...
It logs where the hostname of the URL is hosted (ASN and geographic location).
A simple option cc_critical added to set the country code to log as critical.
2014-08-14 14:22:11 +02:00
Starow
079db6f80c
Hardcoded path from ZMQ_Curve are now referring correctly in config.cfg.sample fix #6
2014-08-11 11:33:18 +02:00
7bdd4a41a5
Indexer module added - initial version with Whoosh full-text indexer
...
The indexer module indexes all the pastes using Whoosh. The module
can be extended to support additional full-text indexers in the future.
2014-08-11 11:04:09 +02:00
Starow
1379ef705a
Initial import of AIL framework - Analysis Information Leak framework
...
AIL is a modular framework to analyse potential information leak from unstructured data source like pastes from Past
ebin or similar services. AIL framework is flexible and can be extended to support other functionalities to mine sen
sitive information
2014-08-06 11:43:40 +02:00
