Commit graph

404 commits

Author SHA1 Message Date
Terrtia
64ff94ce5f
chg: [user_management] create + check user password 2019-05-03 16:52:05 +02:00
Terrtia
1ab1a55a4f
chg: [UI] add basic user management 2019-05-02 17:31:14 +02:00
Terrtia
a183695ddb
chg: [crawler UI] bootstrap 4 migration: show domains by daterange + bug fix hidden services tags 2019-04-26 15:14:29 +02:00
Terrtia
9868833c77
chg: [crawled screenshot] use sha256 as filepath 2019-04-24 14:09:04 +02:00
Terrtia
68077316eb
chg: [git directory] add git helper 2019-04-16 17:26:35 +02:00
Terrtia
3cc614a1ad
chg: [DB] update items tags metadata 2019-04-11 11:58:06 +02:00
Terrtia
d44acea04d
chg: [merge] merge update into advanced_crawler 2019-04-10 16:41:06 +02:00
Terrtia
2589fc2161
chg: [merge] merge master into update branche 2019-04-10 15:43:15 +02:00
Terrtia
b4bee29a33
Merge branch 'master' into advanced_crawler 2019-04-10 15:37:42 +02:00
Terrtia
5165a5de2f
fix: [Crawler] fix index + redis history key 2019-03-22 17:14:27 +01:00
Terrtia
f64c385343
chg: [Crawler] handle port: crawling + history 2019-03-22 16:48:07 +01:00
Terrtia
c2bc72c72e
chg: [Tags alertHandler] remove alertHandler + add quick tag search sidebar + use tag last_seen to display tagged items 2019-03-21 11:11:53 +01:00
Terrtia
65e6eae00d
chg: [UI preview items] b4 migration + new template + better display + decrease item loading time + add basic animation/transition 2019-03-19 13:34:41 +01:00
Jason Hedden
23fb91d326 Add docker-compose support
This commit separates the AIL Framework services into individual Docker
containers that can be managed with docker-compose. It's intended to ease the
installation, development and troubleshooting procedures for all-in-one
deployments.

No changes to architecture or base code are required in this commit.

Changes to existing files:
.gitignore
  * Ignore pystemon archive directory

Dockerfile:
  * Add AIL shell environment variables
  * Install pystemon and crawler pip requirements into AIL virtual environment

Docker-compose notes:
  * All containers are participating in a service network `network_mode:
  service:flask`.  This allows the containers to share the same IP namespace
  to accommodate hard-coded localhost entries.
  * By default persistent data is saved to the following directories in the
  local AIL framework git directory: PASTES, HASHS, CRAWLED_SCREENSHOTS,
  pystemon/archives

This was tested with docker-ce on Ubuntu 16.04 and MacOS. A typical deployment
would look like:
```
git clone https://github.com/CIRCL/AIL-framework.git
cd AIL-framework
cp bin/packages/config.cfg.docker-compose-sample bin/packages/config.cfg
	(optionally enable activate_crawler in config.cfg)
docker-compose build
docker-compose up -d
```
2019-03-11 19:46:34 -05:00
Terrtia
d3b3942ad7
chg: [Flask UI] refractor crawler frontend + show crawled domain 2019-03-06 15:05:58 +01:00
Terrtia
7b32d7f34e
chg: [Crawler] major refractor 2019-02-25 16:38:50 +01:00
Terrtia
da78d0552d
chg: [Crawler UI Tags] add tag by day + add crawler status + UI onion blacklist 2019-02-19 11:41:45 +01:00
Terrtia
516238025f
chg: [Crawler] add bootsrap4 src + refractor crawler 2019-02-05 17:16:44 +01:00
Terrtia
535f44044d
fix: [Onion] add timeout signal 2019-01-29 09:46:03 +01:00
Terrtia
6ee4b7bf93
Merge branch 'master' into update 2018-12-19 15:23:34 +01:00
Terrtia
4e08aaa80f
chg: [UI Crawled Domains] show crawled domains by date range (options: domains tags, up domains, down domains)
todo: Shows domain history
2018-12-17 15:58:48 +01:00
Terrtia
31a8dfe0b3
fix: [AIL] use only relative paths
pastes duplicates are fixed on the fly
2018-11-21 16:45:25 +01:00
Terrtia
cb41c54135
fix: [lib_refine] add dns configuration 2018-11-21 08:49:58 +01:00
Terrtia
d9b776d58a
merge master 2018-11-14 15:55:42 +01:00
Terrtia
b31618eb6a
chg: [Mixer] config default feeder name #291 2018-11-09 15:26:26 +01:00
Terrtia
979472df15
chg: [Terms] tag tracked terms 2018-11-06 13:38:37 +01:00
Terrtia
aa16c52cbf
fix: [NotificationHelper] add config 2018-11-05 14:20:12 +01:00
Terrtia
5872cf9196
fix: [Scripts] Remove absolute path 2018-11-02 16:07:27 +01:00
Terrtia
ea7fd8d54c
fix: [Duplicate] show duplicate with relative path 2018-10-24 15:47:03 +02:00
Terrtia
5afbc3179c
Merge branch 'master' of https://github.com/CIRCL/AIL-framework 2018-10-24 15:16:41 +02:00
Terrtia
8a6ac6be1e
fix: [Paste] create absolute path 2018-10-24 15:16:12 +02:00
Sami Mokaddem
d9bc2d6f44
Merge pull request #266 from kovacsbalu/fix-regexp-timeout
Stop regexp processing after timeout (60sec)
2018-10-09 14:48:08 +02:00
kovacsbalu
272b0fe960 RegexpForTerm timeout from config
Critical log removed
2018-10-09 09:32:32 +02:00
Terrtia
44d19fbb80
fix: [pystemon-feeder] paste save path 2018-10-02 16:02:59 +02:00
59df5f0e12
chg: [config] Tor crawler disabled by default in sample configuration
(to match HOWTO when enabling it)
2018-10-02 14:17:58 +02:00
Terrtia
04b9d9fc1d
chg: [Crawler] add docs 2018-09-27 11:14:29 +02:00
Terrtia
e4a5f66a10
Merge branch 'master' into onion_crawler 2018-09-24 16:35:04 +02:00
Terrtia
50c81773e9
chg: [Crawler] add launcher and install 2018-09-24 16:23:14 +02:00
Terrtia
ead4dba0d8
fix: [MISP feeder] typo filename-config name 2018-09-21 11:02:08 +02:00
Terrtia
6edc1ddbeb
chg: [HiddenServices] get domain related to other domains 2018-09-21 10:34:06 +02:00
Terrtia
5eadc2755f
chg: [Flask] add prefix in config to flask routes 2018-09-20 10:38:19 +02:00
Terrtia
ea3d2c1977
Merge branch 'master' into statistic 2018-09-12 10:06:53 +02:00
Terrtia
6f0817365a
chg: [Crawler UI] display domain information 2018-09-12 09:55:49 +02:00
Terrtia
ca982e13e1
chg: [Crawled Domain] show crawled pastes by domain 2018-08-27 14:34:08 +02:00
Terrtia
823c33174a
Merge branch 'master' into onion_crawler 2018-08-27 11:20:51 +02:00
Terrtia
5c902640e9
Merge branch 'master' into onion_crawler 2018-08-27 11:05:25 +02:00
Terrtia
d42dd118a4
chg: [Domain crawled] add random screenshot 2018-08-27 11:02:39 +02:00
Terrtia
a586548047
Merge branch 'master' into base64 2018-08-24 10:35:28 +02:00
Terrtia
ced0b1e350
chg: [I2P] add default config 2018-08-24 10:24:03 +02:00
Terrtia
e9580d6775
chg: [Crawler] change BDD, save i2p links 2018-08-21 15:54:53 +02:00
Terrtia
ed559d9f4a
chg: [Showpaste] add screenshot + improve onion db 2018-08-16 17:24:39 +02:00
Terrtia
8b1c10b38c
chg: [Onion] add onion splash crawler 2018-08-09 17:42:21 +02:00
Thirion Aurélien
000d5bf40a
fix: add miss config 2018-08-07 13:07:08 +02:00
Terrtia
940b96734a
Merge branch 'master' into statistic 2018-08-01 09:29:51 +02:00
Terrtia
39f54d4420
chg: [Mail] add country statistic 2018-07-30 09:21:22 +02:00
Terrtia
57b631233b
add: [BankAccount] iban detection 2018-07-26 15:31:58 +02:00
Terrtia
8beb63b6a8
fix: [Queues] module with no subscriber 2018-07-24 15:57:18 +02:00
Terrtia
8c97c70a3f
Merge branch 'master' into base64 2018-07-24 15:21:32 +02:00
Terrtia
bd5f83f0eb
chg: refractor base64 encoded to hash 2018-07-20 10:32:52 +02:00
Terrtia
9a8e37fb0b
fix: decoder bdd 2018-07-20 09:43:09 +02:00
Terrtia
ad26f016e3
add: Decoder Module, decode binary, hex and base64 2018-07-19 16:50:42 +02:00
Terrtia
e25dd52d64
add: Binary decoder 2018-07-18 11:45:19 +02:00
Terrtia
0cac3146f1
fix: remove hardcoded mail url 2018-07-17 15:11:25 +02:00
bui
bd5c42caeb add LibInjection Module 2018-07-04 14:28:10 +02:00
Terrtia
16c700ddb4
new: add base64 display on paste + base64 statistics 2018-06-29 10:02:29 +02:00
Terrtia
af9db53171
add tag fp and tp 2018-06-19 15:09:26 +02:00
Terrtia
31fc33ab03
misp event creation, auto and manual + the hive alert auto creation, manual case creation 2018-06-14 16:51:06 +02:00
Terrtia
6143bc3dce
submit users input paste 2018-06-05 16:58:04 +02:00
Terrtia
93dcdf4f23
fix duplicate 2018-05-31 09:01:47 +02:00
Terrtia
0d164b6f4c
add tag js and css 2018-05-16 14:48:15 +02:00
Terrtia
f66a528bc2
fix Duplicate, save list of duplicates on disk + prevent empty hash creation 2018-05-15 23:28:47 +02:00
Terrtia
225fe76c96
fix duplicate + get raw paste 2018-05-11 16:07:30 +02:00
Terrtia
d7ba5533be
fix duplicate + fix issue #200 2018-05-11 13:25:45 +02:00
Terrtia
8599d7142d
python 3 upgrade instructions 2018-05-09 13:32:25 +02:00
Terrtia
142c1dc9b0
clean 2018-05-09 13:03:46 +02:00
Terrtia
3841c159a0
change leveldb to ardb 2018-05-07 14:50:40 +02:00
Terrtia
96a3df42bc
decode with redis connection 2018-05-04 13:53:29 +02:00
Terrtia
a900d5f08d
add python 2 to 3 bash + fix Duplicate + clean 2018-05-03 16:21:33 +02:00
Terrtia
c52caebe7c
improve Duplicate + tlsh + add time out handler + debug + clean 2018-05-02 17:07:10 +02:00
Terrtia
c3428df5dc add apiKeys module 2018-04-26 14:42:39 +02:00
Terrtia
8738b7cf75 fix track term and avoid duplicate mail address 2018-04-24 16:44:37 +02:00
Thirion Aurélien
ef3c335b62
Delete config.cfg.backup 2018-04-23 14:54:23 +02:00
Terrtia
f93fe9aeb2 add bitcoin module 2018-04-23 14:51:53 +02:00
Terrtia
3fc48db903 add Base64 module 2018-04-20 10:48:44 +02:00
Terrtia
8571a86344 frontend python 3.5 upgrade 2018-04-17 16:06:32 +02:00
Terrtia
9c82dd90ec python 3 backend upgrade 2018-04-16 14:50:04 +02:00
CD Nguyen
e9e3f17363 support non TSL SMTP servers 2018-03-30 11:35:37 +02:00
Philipp Schmied
fd329aaba9 config.cfg.sample: Merged overwritten changes from upstream 2018-02-27 16:16:57 +01:00
Philipp Schmied
5b1f0b0212 Implemented email notifications (bin: config.cfg additions, email sending via analyzer scripts; var: Changes to add notifications via terms management); terms_management: Fixed click handlers not being added to all tracked terms. 2018-02-27 15:12:02 +01:00
Sami Mokaddem
c39e537cbe Merge branch 'iss160' into showDiff
Preemptively preventing merge-conflict with branch iss160 (moving to
configuration)
2017-12-12 11:33:22 +01:00
Sami Mokaddem
304213cfa4 update: Possibility to change the authorized max line length in the config for the diff of duplicates 2017-12-12 11:00:26 +01:00
Sami Mokaddem
860acbf7cf fix: Renamed Credentials into Credential sectiont title
config change: By default, zmq listen to both local and CRF
2017-12-12 09:29:15 +01:00
Sami Mokaddem
9d0d0b4303 update: Moved filtering operation (thresholds, number of matching in the
categ file) in the configuration file. It permits to better control the
flow of pastes.
Also set default mixer duplicate filtering to 3 (Do not filter)
2017-12-11 17:28:34 +01:00
Sami Mokaddem
2a967c4d92 update/feature: Max number of duplicate push to MISP + duplicate are pushed as attachment 2017-11-23 07:13:44 +01:00
Sami Mokaddem
6f0227a78a Starting support of python3 2017-11-20 12:12:06 +01:00
Sami Mokaddem
652b00f4b2 Renamed BrowseWarningPaste into alertHandler 2017-11-15 16:15:43 +01:00
=
64ec44085d Dynamically choose correct port for LevelDB in script 2017-08-22 17:52:15 +02:00
Mokaddem
882ab88b00 Draft module + web link 2017-07-18 16:57:15 +02:00
Mokaddem
b60e164380 Added support of onion in browseWarningPaste 2017-04-26 11:47:02 +02:00
Mokaddem
da081bae42 Merge branch 'regexInTerms' into production 2017-04-18 16:59:40 +02:00
Mokaddem
c8baabd882 Updated TermsFrequency related modules + Fixed bugs (encoding, behaviors, ...) 2017-04-18 15:28:21 +02:00
Mokaddem
2da4c572c7 New feature in Terms: regex and set of words support (draft) 2017-03-28 17:42:44 +02:00
Mokaddem
e225090e07 Added script that checks configuration and may auto-fix it + added missing field in config.cfg.sample 2017-03-15 16:36:51 +01:00
Mokaddem
adc37ab50e Indexer max size now in config 2017-03-15 12:14:41 +01:00
Olivier MEDOC
16044d4d36 pystemon-feeder: add the option bind in configuration file instead of address
This is essentially because multiple feeds can be used in the address
configuration variable.
2017-01-13 14:58:18 +01:00
Olivier MEDOC
9f9c265cb0 SentimentAnalysis: use lexicon file path from the ail configuration file 2017-01-11 11:07:08 +01:00
ptitdoc
c0fac82031 Add pystemon path in configuration sample 2017-01-10 18:33:46 +01:00
Mokaddem
0e39f516a9 Added sleep in helper for multiple listening interfaces (reduce useless work), Added a simple way to pre-process feed before sending it to the global feed. 2017-01-09 14:12:26 +01:00
Mokaddem
e70b9cd15c Added basic mixer with confirugable behavior. It handles muliple feeders and performs some basic stats on them. 2016-12-23 10:31:26 +01:00
Mokaddem
7438f16c63 Added new config 2016-12-22 10:06:35 +01:00
eb14005fcf Fix: as JSON decode needs a string, the empty list is a string. 2016-08-29 12:08:12 +00:00
3b6e3e156e Merge branch 'master' of github.com:CIRCL/AIL-framework 2016-08-26 15:57:45 +00:00
Mokaddem
0c760d763b Added support of PID in ModuleInformation and index.html 2016-08-26 15:27:37 +02:00
1ab08d4a05 Merge branch 'master' of github.com:CIRCL/AIL-framework 2016-08-25 12:33:31 +00:00
Mokaddem
07856f3119 Added script module-dataflow-graph + fixed bug in module.cfg preventing the credential module to receive data. 2016-08-24 11:32:48 +02:00
bde5f47ed0 Merge branch 'master' of github.com:CIRCL/AIL-framework
Conflicts:
	bin/launch_scripts.sh
2016-08-23 14:20:39 +00:00
Mokaddem
697812dbff Added config.sample 2016-08-23 09:59:39 +02:00
Mokaddem
d3c0fc5f94 Added terms trending module and web interface:
- Curve module has been rewritten
- Added new module curve_manage_top_set
- Added all web-pages
2016-08-22 13:35:49 +02:00
Mokaddem
c12e67a35a Added tooltips + slight gauge modification 2016-08-17 10:43:43 +02:00
Mokaddem
894b9efda9 Added drop of really long line in sentiment-analysis module + Added description of sentiment module. Also, fixed bug in webpage sentiement-trending concerning avg and date range. 2016-08-17 09:46:25 +02:00
Mokaddem
1084e45f1b Added draft of filter in sentiment analysis (Discard syntaxical languages) + Added nice tooltip for sparkline. Trending displays avg in function of the number of elements processed and not for the complete week + fixed bug in gauge and canvasjs (was performing avg with only 1 graph instead of all 8). 2016-08-16 16:33:02 +02:00
Mokaddem
1a5158b081 Added sentiment analyser module (draft) 2016-08-13 15:24:57 +02:00
95a22abe9c Merge branch 'master' of github.com:CIRCL/AIL-framework
Conflicts:
	bin/packages/modules.cfg
2016-08-11 08:06:57 +00:00
Mokaddem
a4595010c0 Added whole config in config.sample 2016-08-09 14:23:36 +02:00
Mokaddem
4e56f76f62 Fixed bug CVE,
CVE was not sending its data to BrowseWarningPaste.
2016-08-08 10:05:21 +02:00
Mokaddem
1c40ad522f Merge branch 'master' into production 2016-08-08 09:26:58 +02:00
Mokaddem
78c611fead Added warning_paste module and created related webpages.
Fixed a Faup bug in credential (multiple instanciation) and added correc populate_set_out in concerned modules (creditcard, credential, ...).
Linked browse_warning_paste module and Flask function with redis (created new sets).
2016-08-08 09:17:44 +02:00
Mokaddem
55ef917ef1 Added few comments 2016-08-04 12:10:56 +02:00
Mokaddem
d9316771cd Draft: added new duplicate hash comparison - tlsh 2016-08-04 11:55:38 +02:00
Mokaddem
3169dc27b0 Added SQLInjectionDetection module 2016-08-02 15:43:11 +02:00
Mokaddem
06be1f129a Added Draft of ModuleStats + Paste size average 2016-07-25 16:38:57 +02:00
d013546643 Merge branch 'master' of github.com:CIRCL/AIL-framework
Conflicts:
	bin/packages/modules.cfg
2016-07-25 07:56:46 +00:00
722634adf4 Prod config 2016-07-25 07:49:14 +00:00
Raphaël Vinot
4aa484a91f Merge pull request #63 from mokaddem/uptodate-duplicate-module
Uptodate duplicate module
2016-07-25 09:44:49 +02:00
Raphaël Vinot
ea8703e608 Fix path to tldsfile 2016-07-22 09:32:13 +02:00
Mokaddem
7c4f4a2f79 Merge branch 'statistics-v2' into uptodate-statistics-v2 2016-07-21 15:16:03 +02:00
Mokaddem
b685f81752 Merge branch 'duplicate-module' into uptodate-duplicate-module 2016-07-21 14:59:52 +02:00
Mokaddem
0ab7560477 Added top_progression chart for tld, domain and scheme + Small modification in config file. 2016-07-21 13:44:22 +02:00
Raphaël Vinot
ede46ec799 Fix default config. 2016-07-21 09:19:13 +02:00
Raphaël Vinot
781b441bb8 Fix path in config 2016-07-20 14:12:50 +02:00
Raphaël Vinot
8a69110c8e Merge branch 'webpage-splitting' of https://github.com/mokaddem/AIL-framework into mokaddem-webpage-splitting 2016-07-19 16:35:31 +02:00
Mokaddem
a6996c0b23 Added related functions and display of duplicated paste in search.py 2016-07-19 10:48:44 +02:00
Mokaddem
996c0e02de Duplicate module takes its messages from other modules and no more from Global. 2016-07-18 16:22:33 +02:00
Mokaddem
6805ed6488 Added default config 2016-07-18 15:52:53 +02:00
Mokaddem
4f6813350b Added two new version of duplicate module.
One with hashes are saved in json on disk
The other with only leveldb
2016-07-18 15:50:41 +02:00
Mokaddem
14e9850dd6 Added new module for Duplicate paste. Seems working but has some small bug (re-check same paste twice) 2016-07-15 16:58:48 +02:00
Mokaddem
2383db022f Added default configuration 2016-07-15 09:10:44 +02:00
Mokaddem
9a9e07f600 Added default configuration 2016-07-15 09:08:38 +02:00
Mokaddem
0332f23579 Added SimHash library 2016-07-15 08:56:16 +02:00
Mokaddem
465244e1ce Added dynamic table sorting in search page. (Still need to add dependencies) 2016-07-07 16:38:00 +02:00
Mokaddem
7ff9b9a583 Added DomainTrending seems working.
Started search features with related html pages, not finish yet.
2016-07-05 16:53:03 +02:00
Mokaddem
3dc356dc5e Getting Started: Initial configuration working on the laptop 2016-06-30 14:36:47 +02:00
MaximeStor
701d771aa5 Add first version of Source Code 2016-03-12 11:21:29 +01:00
Raphaël Vinot
4895ee9fa2 Add new category (Credential) 2016-02-10 16:39:56 +01:00
c68136b04b Merge branch 'pgp' of https://github.com/Rafiot/AIL-framework
Conflicts:
	bin/packages/modules.cfg
2016-02-08 10:13:44 +01:00
MaximeStor
b7d2b64a86 Merge branch 'master' of https://github.com/CIRCL/AIL-framework into module 2016-02-06 11:28:48 +01:00
192ee7f0ec Merge pull request #49 from Alainfou/master
Phone module added (regex needs optimization)
2016-02-06 11:20:46 +01:00
Alain
fabbfd8ae9 Update module.cfg (adding Keys and Phone section) 2016-02-05 14:00:41 -05:00
MaximeStor
07513a5b37 Add modules Credential and Release 2016-02-05 16:15:09 +01:00
Raphaël Vinot
9171d5b118 Add module to find PGP encrypted blobs 2016-02-05 16:03:37 +01:00
Raphaël Vinot
aef8ab0411 Listen locally for 0MQ 2016-02-04 15:32:50 +01:00
e3971ac93a Onion fetching loop deactivated by default 2014-12-22 16:06:38 +00:00
Raphaël Vinot
08ceefc375 Re-add config option 2014-12-22 16:50:25 +01:00
Raphaël Vinot
50369c6706 Revert changes on the config file due to merging messup 2014-12-22 16:29:05 +01:00
Raphaël Vinot
f717f9fe89 Merge branch 'master' of github.com:CIRCL/AIL-framework 2014-12-22 15:32:48 +01:00
Raphaël Vinot
9ee61db2cf Add hotfixes 2014-12-22 15:27:02 +01:00
Raphaël Vinot
8803c8447a Publish the fetched onions on a ZMQ feed. 2014-09-30 16:55:16 +02:00
25757b0fff A simple feeder script feeding data from pystemon to AIL.
The configuration matches the default Redis parameters used
in the pystemon configuration.

https://github.com/cvandeplas/pystemon/blob/master/pystemon.yaml#L16
2014-09-19 14:03:05 +02:00
Raphaël Vinot
65b9a01644 Add config file for DomainClassifier, proper reporting 2014-09-17 17:22:56 +02:00
de6e21d5a7 DomainClassifier sample configuration added 2014-09-08 16:44:05 +02:00
Raphaël Vinot
e983c839ad Categ now listen to the Global queue 2014-09-05 17:05:45 +02:00
Raphaël Vinot
fca00beed9 Add Domain Classifier module.
Cleanup in the config files.
2014-09-05 10:41:00 +02:00
Raphaël Vinot
b7c9e489c9 Fix the exceptions 2014-09-04 11:46:07 +02:00
Raphaël Vinot
9e8611a42d stop killing the disk when creating the word curve 2014-09-02 18:20:28 +02:00
Raphaël Vinot
0c6b09f379 Fix the onion module, log the valid onions. 2014-09-01 16:18:06 +02:00
Raphaël Vinot
f4b89669fc The onion module now fetches the URLs it finds. 2014-08-31 22:42:12 +02:00
Raphaël Vinot
abfe13436b Big refactoring, make the queues more flexible 2014-08-29 19:37:56 +02:00
Raphaël Vinot
623e876f3b Cleanup.
* Remove useless subscriber
* Fix typo in the config file
* Update Helper accordingly
2014-08-26 17:36:57 +02:00
Raphaël Vinot
3886d1b834 Small fixes to make the refactoring production ready
* the port for the logging is 6380
* use os.environ properly
* fix typos
2014-08-22 17:35:40 +02:00
Raphaël Vinot
78125db4ea Use env variables everywhere 2014-08-22 14:52:02 +02:00
Raphaël Vinot
63b29176c1 move Redis_Data_Merging to Paste 2014-08-21 12:22:07 +02:00
Raphaël Vinot
50cfac857e Update config
Make all paths in the config file relative to the home directory.
2014-08-20 16:00:56 +02:00
Raphaël Vinot
a68f5b6a0e fix subscriber names, update default config 2014-08-20 15:54:21 +02:00
Raphaël Vinot
2485ba5df2 Merge remote-tracking branch 'origin/master' into testing
Conflicts:
	bin/ZMQ_Sub_Urls.py
2014-08-20 15:24:10 +02:00
Raphaël Vinot
99c8cc7941 completely remove ZMQ_PubSub.py 2014-08-20 15:14:57 +02:00
1d64dc44c8 MIME type guessing - removed one duplicate call to libmagic 2014-08-20 10:22:33 +02:00
Raphaël Vinot
45b0bf3983 Improve the cleanup. Still some to do. 2014-08-19 19:07:07 +02:00
Raphaël Vinot
f1753d67c6 Cleanup the queues. 2014-08-19 16:05:37 +02:00
7d8ee102a3 Assignment before use (if Enumerate fails) 2014-08-18 15:58:06 +02:00
Jules
ab6765315e Merge pull request #13 from adulau/master
Log where URLs are hosted - cc_critical option added
2014-08-14 14:28:01 +02:00
762def3a23 Log where URLs are hosted - cc_critical option added
It logs where the hostname of the URL is hosted (ASN and geographic location).
A simple option cc_critical added to set the country code to log as critical.
2014-08-14 14:22:11 +02:00
Raphaël Vinot
4a1f300a1a Cleanup (remove unused imports, more pep8 compatible) 2014-08-14 14:11:07 +02:00
Starow
04a8f1bdf2 maxi cleanup old code :'( 2014-08-14 11:48:46 +02:00
Raphaël Vinot
ece3bc173e Cleanup of main Paste module 2014-08-13 11:56:22 +02:00
Starow
079db6f80c Hardcoded path from ZMQ_Curve are now referring correctly in config.cfg.sample fix #6 2014-08-11 11:33:18 +02:00
7bdd4a41a5 Indexer module added - initial version with Whoosh full-text indexer
The indexer module indexes all the pastes using Whoosh. The module
can be extended to support additional full-text indexers in the future.
2014-08-11 11:04:09 +02:00
Starow
eb603e8762 Fixing a bug about caching paste inside Redis :) 2014-08-08 17:23:51 +02:00
Starow
7a1db94f9e Adding a letter (s) 2014-08-08 17:19:42 +02:00
Starow
043800287a adding a . 2014-08-08 17:18:03 +02:00
Starow
bf682c4b44 Fixing last commit ... 2014-08-08 17:13:18 +02:00
Starow
503c23ca3b Fixing last commit 2014-08-08 17:08:41 +02:00
Starow
c9e1eaf182 Improving cache code 2014-08-08 17:04:25 +02:00
Starow
44addf1afe Redis cache added fix #5
The paste will be add in Redis during 5min and also saved on disk.
Now if a module want to get the paste for further processing, it will first try to get it in the cache
instead of getting it directly on the disk and wasting I/O.
2014-08-08 16:48:02 +02:00
Starow
1379ef705a Initial import of AIL framework - Analysis Information Leak framework
AIL is a modular framework to analyse potential information leak from unstructured data source like pastes from Past
ebin or similar services. AIL framework is flexible and can be extended to support other functionalities to mine sen
sitive information
2014-08-06 11:43:40 +02:00