Terrtia
3207949c5f
fix: [update v1.5] create update config file
2019-04-25 17:17:12 +02:00
Terrtia
41e6b4ec59
fix: [v1.5 background update screenshot] remove duplicate files
2019-04-25 14:39:38 +02:00
Terrtia
499c07a93e
fix: [update v1.5] remove unbuffer
2019-04-25 14:21:02 +02:00
Terrtia
fc2c1422ff
fix: [Crawler] unpack_url
2019-04-25 13:54:06 +02:00
Terrtia
50944b1d61
fix: [LAUNCH] fix typo
2019-04-24 16:37:44 +02:00
Terrtia
e6dca7f8bf
chg: [update v1.5] add background update: screenshots_crawled
2019-04-24 16:19:35 +02:00
Terrtia
9868833c77
chg: [crawled screenshot] use sha256 as filepath
2019-04-24 14:09:04 +02:00
Terrtia
99b9c95638
fix: [update] fix is_fork check
2019-04-23 15:30:47 +02:00
Terrtia
35c188829f
chg: [config update] refractor, auto update, create config if file doesnt exist
2019-04-23 15:04:45 +02:00
Terrtia
2a1cd4a009
chg: [Onion, crawler config] auto crawler: add config by url, fix onions tagging + filter subdomains
2019-04-23 11:15:34 +02:00
Terrtia
6fdf7c2123
chg: [UI crawler] status/remove auto crawler
2019-04-18 16:57:51 +02:00
Terrtia
ddaad5db93
chg: [v1.5 background update] create update stats
2019-04-17 17:07:09 +02:00
Terrtia
4bff603d4f
chg: [settings UI] add background update progress
2019-04-17 14:39:29 +02:00
Terrtia
68077316eb
chg: [git directory] add git helper
2019-04-16 17:26:35 +02:00
Terrtia
3b11df7f90
chg: [UI] add server Management, check if new update/version are available + background update status + link to new realease/commits
2019-04-16 17:24:59 +02:00
Terrtia
a3167a740a
fix: [Update DB] remove indexErrors
2019-04-12 17:32:17 +02:00
Terrtia
bf92a2f22f
fix: [background update] avoid multiple update on the same key
2019-04-12 16:07:40 +02:00
Terrtia
f2dc3bcbf4
fix: [Global] remove absolute path from item path
2019-04-12 15:13:36 +02:00
Terrtia
3a8631d44a
chg: [Update + Launcher] use background update + add new options in launcher
2019-04-12 15:06:24 +02:00
Terrtia
5c4db5f2f1
chg: [update] add update script Redis ARDB + add background update
2019-04-11 17:49:20 +02:00
Terrtia
3cc614a1ad
chg: [DB] update items tags metadata
2019-04-11 11:58:06 +02:00
Terrtia
d44acea04d
chg: [merge] merge update into advanced_crawler
2019-04-10 16:41:06 +02:00
Terrtia
2589fc2161
chg: [merge] merge master into update branche
2019-04-10 15:43:15 +02:00
Terrtia
b4bee29a33
Merge branch 'master' into advanced_crawler
2019-04-10 15:37:42 +02:00
Terrtia
f6d7d2ae16
chg: [Update] add debug
2019-04-02 09:46:04 +02:00
Thirion Aurélien
976b97bcd1
Merge pull request #338 from CIRCL/pre_update
...
Merge Pre update
2019-03-26 16:51:26 +01:00
Terrtia
044933aa33
chg: [Launch] remove old launchers
2019-03-26 16:47:10 +01:00
Terrtia
cee2b2486f
chg: [Launch] remove old launchers
2019-03-26 16:45:01 +01:00
Terrtia
59664efe45
Merge branch 'master' into advanced_crawler
2019-03-26 16:03:42 +01:00
Terrtia
5165a5de2f
fix: [Crawler] fix index + redis history key
2019-03-22 17:14:27 +01:00
Terrtia
f64c385343
chg: [Crawler] handle port: crawling + history
2019-03-22 16:48:07 +01:00
Terrtia
f4cdddbc7f
chg: [UI + item submit] remove browse paste from UI + fix pasteSubmit tagging
2019-03-21 11:50:53 +01:00
Terrtia
c2bc72c72e
chg: [Tags alertHandler] remove alertHandler + add quick tag search sidebar + use tag last_seen to display tagged items
2019-03-21 11:11:53 +01:00
Terrtia
a9e86231ad
chg: [Tags] add backend, daterange selector
2019-03-20 13:25:02 +01:00
Terrtia
65e6eae00d
chg: [UI preview items] b4 migration + new template + better display + decrease item loading time + add basic animation/transition
2019-03-19 13:34:41 +01:00
Terrtia
23d9cb2265
chg: [UI Tags search] bootstrap 4 migration
2019-03-14 17:04:55 +01:00
85d2ea578c
Merge pull request #335 from 5thColumn/docker
...
Add docker-compose support
2019-03-13 21:38:12 +01:00
Jason Hedden
23fb91d326
Add docker-compose support
...
This commit separates the AIL Framework services into individual Docker
containers that can be managed with docker-compose. It's intended to ease the
installation, development and troubleshooting procedures for all-in-one
deployments.
No changes to architecture or base code are required in this commit.
Changes to existing files:
.gitignore
* Ignore pystemon archive directory
Dockerfile:
* Add AIL shell environment variables
* Install pystemon and crawler pip requirements into AIL virtual environment
Docker-compose notes:
* All containers are participating in a service network `network_mode:
service:flask`. This allows the containers to share the same IP namespace
to accommodate hard-coded localhost entries.
* By default persistent data is saved to the following directories in the
local AIL framework git directory: PASTES, HASHS, CRAWLED_SCREENSHOTS,
pystemon/archives
This was tested with docker-ce on Ubuntu 16.04 and MacOS. A typical deployment
would look like:
```
git clone https://github.com/CIRCL/AIL-framework.git
cd AIL-framework
cp bin/packages/config.cfg.docker-compose-sample bin/packages/config.cfg
(optionally enable activate_crawler in config.cfg)
docker-compose build
docker-compose up -d
```
2019-03-11 19:46:34 -05:00
Terrtia
d3b3942ad7
chg: [Flask UI] refractor crawler frontend + show crawled domain
2019-03-06 15:05:58 +01:00
Terrtia
c0d72e7d2a
chg: [Crawler UI] Crawler major refractor (end) + basic UI for manual crawler
2019-02-26 14:50:48 +01:00
Terrtia
7b32d7f34e
chg: [Crawler] major refractor
2019-02-25 16:38:50 +01:00
Jason Hedden
9eae25bdf5
Fix current_selected amount and action vars
...
This commit fixes issue #268
```
File "./bin/ModulesInformationV2.py", line 319, in _setValue
action = current_selected_action + " " + str(current_selected_amount) + "x"
```
`current_selected_action` was defined twice. The second occurance should be `current_selected_amount`.
This also fixes the generic exception from failing when `current_selected_value` is not defined.
2019-02-22 12:39:27 -06:00
Terrtia
60f7645ac1
chg: [Crawler] refractor
2019-02-22 17:00:24 +01:00
Terrtia
3d36ddbc1e
fix: [CVE] fix stat regex
2019-02-21 10:51:05 +01:00
Terrtia
1114aa2ffd
chg: [CVE] add stat script
2019-02-21 09:58:10 +01:00
Terrtia
e5dca268a8
chg: [Crawler] refractor
2019-02-21 09:54:43 +01:00
Terrtia
da78d0552d
chg: [Crawler UI Tags] add tag by day + add crawler status + UI onion blacklist
2019-02-19 11:41:45 +01:00
Terrtia
6a8fa3d5d1
chg: [Tags] tags by day
2019-02-18 14:33:34 +01:00
Terrtia
709d5487b8
fix: [Onion] filter too many subdomain
2019-02-13 16:53:38 +01:00
Terrtia
b87707e8bc
fix: [Crawler] typo
2019-02-12 15:54:42 +01:00
Terrtia
37276e52a3
fix: [Crawler] typo
2019-02-12 15:53:40 +01:00
Terrtia
7cb03fc769
fix: [Crawler] typo
2019-02-12 15:51:19 +01:00
Terrtia
7a4989ce10
fix: [Global Crawler] max filename size
2019-02-12 15:45:58 +01:00
Terrtia
c2885589cf
chg: [UI] basic navbar + sidebar + refractor
2019-02-07 17:22:44 +01:00
Terrtia
516238025f
chg: [Crawler] add bootsrap4 src + refractor crawler
2019-02-05 17:16:44 +01:00
Thirion Aurélien
44c513dcbb
chg: [Crawler] add onion to blacklist
2019-01-31 16:56:45 +01:00
Xavier Mertens
43d08d0d3e
Cleanup
2019-01-30 22:36:10 +01:00
Xavier Mertens
1be91b88bf
Fix: reset alarm timeout
2019-01-30 22:30:07 +01:00
Xavier Mertens
0372a78704
Merge remote-tracking branch 'upstream/master'
2019-01-30 22:27:44 +01:00
Xavier Mertens
b099e2ae4a
Update LAUNCH.sh
2019-01-30 22:27:33 +01:00
Terrtia
92d192238b
fix: [Crawler] change max page crawled
2019-01-29 17:04:45 +01:00
Terrtia
6c7086f4eb
fix: [Crawler] first_seen
2019-01-29 16:54:39 +01:00
Terrtia
ad04cf7bb2
fix: [Onion] typo
2019-01-29 16:27:12 +01:00
Terrtia
88eaaeae93
chg: [Crawler] add priority queue, fix #263
2019-01-29 16:08:59 +01:00
Terrtia
c1b34bd99c
fix: [Crawler] limit max crawled pages
2019-01-29 15:38:00 +01:00
Terrtia
64ffdd52e8
fix: [Onion] loop
2019-01-29 14:39:57 +01:00
Terrtia
2dc0eca4a9
fix: [Crawler] fix crawler cache info
2019-01-29 12:09:19 +01:00
Terrtia
bb301a870c
fix: [Crawler] fix onion blacklist + add crawler info
2019-01-29 12:00:14 +01:00
Terrtia
535f44044d
fix: [Onion] add timeout signal
2019-01-29 09:46:03 +01:00
Thirion Aurélien
f4ba21e492
blacklist onion debian manpages
2019-01-14 11:08:53 +01:00
raggadhub
74c726d5fa
fix path_to_send path
...
Hi,
Missing slash between pastes_directory and paste in path_to_send = pastes_directory+paste
path error example:
/opt/AIL-framework/PASTESarchive/pastebin.com_pro/2019/01/07/cAgqsY2U.gz
2019-01-09 13:15:58 +01:00
Terrtia
b3b75ccbea
fix: [Crawler] Restart Splash on failure, limit unbound in memory cache (maxrss)
2019-01-04 15:51:08 +01:00
Terrtia
6ee4b7bf93
Merge branch 'master' into update
2018-12-19 15:23:34 +01:00
Terrtia
ca47764836
fix: [update v1.5]
2018-12-19 15:16:52 +01:00
Terrtia
198ee97d90
chg: [update 1.5] add update bash
2018-12-19 11:41:01 +01:00
Terrtia
2560260ce4
Merge branch 'master' into pre_update
2018-12-19 10:01:52 +01:00
Terrtia
4d04333f54
fix: [Splash server] add debug output
2018-12-19 09:30:24 +01:00
Terrtia
f842194c57
fix: [Crawler] retry when splash is not available
2018-12-17 16:04:12 +01:00
Terrtia
4e08aaa80f
chg: [UI Crawled Domains] show crawled domains by date range (options: domains tags, up domains, down domains)
...
todo: Shows domain history
2018-12-17 15:58:48 +01:00
Steve Clement
b60fd55efe
chg: [bash] Small PATH precedence issue.
2018-11-22 16:16:31 +09:00
Steve Clement
f82c899e7b
fix: [bash] fixed log_subscriber path
2018-11-22 15:48:12 +09:00
Steve Clement
88e561aa20
chg: [bash] Replace '' with ""
2018-11-22 15:41:45 +09:00
Steve Clement
33bc23a297
fix: [bash] evoking bash -c and splitting vars in this fashion wants to be changed in future revisions.
2018-11-22 15:22:35 +09:00
Steve Clement
6522cffd57
chg: [doc] Added note that redis-server is bundled during install
2018-11-22 15:10:57 +09:00
Steve Clement
41a34b3519
fix: [bash] As we instantiate from a subdirectory, we need to take this into account.
2018-11-22 15:08:59 +09:00
Steve Clement
e4ced241eb
chg: [python] Added necessary checks in LAUNCH.sh to be able to launch the script without doing bin/activate
...
chg: [doc] Added correct pystemon path
fix: [zmq] Small ZMQ test fix.
fix: [doc] Moved the print statement in feeder so it is less verbose but still useful
2018-11-22 14:57:24 +09:00
Terrtia
31a8dfe0b3
fix: [AIL] use only relative paths
...
pastes duplicates are fixed on the fly
2018-11-21 16:45:25 +01:00
Terrtia
cb41c54135
fix: [lib_refine] add dns configuration
2018-11-21 08:49:58 +01:00
Terrtia
4e680aabf0
chg: [Overview] add doc
2018-11-20 14:39:45 +01:00
Terrtia
aaa277b8a0
chg: [Update] add additonal update
2018-11-15 14:26:41 +01:00
Terrtia
f6e86582c8
chg: [Update] generate current version
2018-11-15 13:48:44 +01:00
Terrtia
108fdb868e
chg: update Overview
2018-11-15 10:39:41 +01:00
Terrtia
d9b776d58a
merge master
2018-11-14 15:55:42 +01:00
Terrtia
aed8d65aef
fix: [LAUNCH] add update in helper
2018-11-14 15:17:56 +01:00
Terrtia
912b977bb4
chg: [LAUNCH] update: launch + check BDD
2018-11-13 16:54:39 +01:00
Terrtia
94fcf66d20
fix: [Update] cleaning
2018-11-13 15:52:33 +01:00
Terrtia
347986a271
chg: [LAUNCH] add AIL update by default
2018-11-13 15:45:22 +01:00
Terrtia
7aff45c507
chg [Update] update AIL clone and fork
2018-11-13 14:54:56 +01:00
Terrtia
60ff0b9cf7
chg: [Update] add update script
2018-11-12 17:10:31 +01:00
Terrtia
b31618eb6a
chg: [Mixer] config default feeder name #291
2018-11-09 15:26:26 +01:00
Terrtia
c3903ab203
fix: [MISP_The_Hive_feeder] handle not saved pastes
2018-11-09 13:58:09 +01:00
Sami Mokaddem
62e46b6dc1
Merge pull request #287 from CIRCL/term_management
...
Term management upgrade
2018-11-08 14:10:25 +01:00
Terrtia
e3764ca575
fix: [MISP_the_Hive_feeder] force absolute path
2018-11-08 11:22:23 +01:00
Terrtia
979472df15
chg: [Terms] tag tracked terms
2018-11-06 13:38:37 +01:00
Terrtia
7ef5cc3205
fix:[MISP_The_Hive_feeder] typo
2018-11-05 16:40:25 +01:00
Terrtia
85f933fd81
chg: [NotificationHelper] add error traceback
2018-11-05 14:59:40 +01:00
Terrtia
877ac0636e
fix: typo
2018-11-05 14:30:03 +01:00
Terrtia
aa16c52cbf
fix: [NotificationHelper] add config
2018-11-05 14:20:12 +01:00
Terrtia
dad0365c67
fix:[MISP_The_Hive_feeder] set default values
2018-11-05 09:16:18 +01:00
Terrtia
5872cf9196
fix: [Scripts] Remove absolute path
2018-11-02 16:07:27 +01:00
Xavier Mertens
05de39e271
Fixed line counter (count all lines)
2018-11-02 15:55:55 +01:00
Xavier Mertens
ac95dd4ee8
Added timeout for Regex module
2018-11-02 15:50:38 +01:00
Xavier Mertens
9e7ca84581
Added timeout to avoid blocking regexes
2018-11-02 15:49:06 +01:00
Xavier Mertens
9bfb7a5fb6
Added module Regex.py
2018-10-26 17:13:26 +02:00
Terrtia
ea7fd8d54c
fix: [Duplicate] show duplicate with relative path
2018-10-24 15:47:03 +02:00
Terrtia
5afbc3179c
Merge branch 'master' of https://github.com/CIRCL/AIL-framework
2018-10-24 15:16:41 +02:00
Terrtia
8a6ac6be1e
fix: [Paste] create absolute path
2018-10-24 15:16:12 +02:00
Sami Mokaddem
d9bc2d6f44
Merge pull request #266 from kovacsbalu/fix-regexp-timeout
...
Stop regexp processing after timeout (60sec)
2018-10-09 14:48:08 +02:00
kovacsbalu
c1c40e3e56
Fix config section name
2018-10-09 11:30:04 +02:00
kovacsbalu
272b0fe960
RegexpForTerm timeout from config
...
Critical log removed
2018-10-09 09:32:32 +02:00
kovacsbalu
6d199f0150
Stop regexp processing after timeput (60sec)
...
Minor pep8 fixes
2018-10-08 11:25:32 +02:00
efaba35dec
Common onion addresses
2018-10-05 13:50:17 +02:00
ed9f9e5dce
Merge pull request #262 from kovacsbalu/feat-smtp-test
...
Test notification
2018-10-02 17:27:55 +02:00
Terrtia
44d19fbb80
fix: [pystemon-feeder] paste save path
2018-10-02 16:02:59 +02:00
59df5f0e12
chg: [config] Tor crawler disabled by default in sample configuration
...
(to match HOWTO when enabling it)
2018-10-02 14:17:58 +02:00
kovacsbalu
93a3c5779c
Test notification
...
Minor pep8 fixes
2018-10-01 15:56:48 +02:00
Terrtia
6328cc22b7
chg: [Crawler] add domains blacklist
2018-09-28 16:29:09 +02:00
Terrtia
82e6df4b94
chg: [Crawler] domains stats + logs + clean
2018-09-28 15:23:27 +02:00
Terrtia
ecb2857151
chg: [Crawler] css + limit splash RAM
2018-09-27 16:47:48 +02:00
Terrtia
e357dce59b
fix: [Crawler] detect splash connection to proxy error
2018-09-27 15:43:03 +02:00
Terrtia
04b9d9fc1d
chg: [Crawler] add docs
2018-09-27 11:14:29 +02:00
Terrtia
c49e871ba8
chg: [crawler] add infos
2018-09-26 16:34:27 +02:00
Terrtia
e4a5f66a10
Merge branch 'master' into onion_crawler
2018-09-24 16:35:04 +02:00
Terrtia
874824a589
fix: [Crawler] clean
2018-09-24 16:28:55 +02:00
Terrtia
8eca0e0778
fix: [Crawler] clean
2018-09-24 16:24:30 +02:00
Terrtia
50c81773e9
chg: [Crawler] add launcher and install
2018-09-24 16:23:14 +02:00
Terrtia
b575efcc46
fix: [pystemon] pystemon encoding
2018-09-21 14:16:06 +02:00
Terrtia
ead4dba0d8
fix: [MISP feeder] typo filename-config name
2018-09-21 11:02:08 +02:00
Terrtia
6edc1ddbeb
chg: [HiddenServices] get domain related to other domains
2018-09-21 10:34:06 +02:00
Terrtia
5eadc2755f
chg: [Flask] add prefix in config to flask routes
2018-09-20 10:38:19 +02:00
Terrtia
2e763ba0aa
fix: [ModulesInformationV2] fix #244
2018-09-19 15:45:06 +02:00
Terrtia
5732d7b134
fix: [hashDecoded] range_type_json
2018-09-19 15:36:33 +02:00
Terrtia
ebad2af1ae
fix: [import_dir] fix #251 , fix paste import
...
if the date directory tree don't exist, he will be generated
2018-09-19 13:38:31 +02:00
Terrtia
653463a8c4
fix: [LAUNCH.sh] make pystemon optionnal
...
AIL and Pystemon are independent, pystemon shouldn't be launch by default.
2018-09-19 09:52:27 +02:00
Terrtia
5b31b6e853
fix: [Crawler] save domain to crawl on splash error
2018-09-18 16:20:13 +02:00
Terrtia
f5b648d72a
pixelate paste screenshot
2018-09-18 11:03:40 +02:00
Terrtia
6aa1b99c2a
fix: [LAUNCH.sh] remove old decoders
2018-09-17 15:50:27 +02:00
Terrtia
0c63f2f24f
chg: [Crawler] catch server response
2018-09-17 15:35:06 +02:00
a6c72a677d
Merge pull request #243 from kovacsbalu/docker-with-pystemon
...
Docker build with pystemon
2018-09-12 21:50:11 +02:00
Terrtia
a94a0fe1a7
fix: [Install] add python requirements
2018-09-12 19:10:49 +02:00
Terrtia
204e996fc3
chg: [statistics] clean scripts
2018-09-12 11:21:11 +02:00
Terrtia
ea3d2c1977
Merge branch 'master' into statistic
2018-09-12 10:06:53 +02:00
Terrtia
6f0817365a
chg: [Crawler UI] display domain information
2018-09-12 09:55:49 +02:00
kovacsbalu
d19dd72f99
Install pystemon and start pystemon-feeder in docker
2018-09-03 16:01:39 +02:00
Terrtia
ca982e13e1
chg: [Crawled Domain] show crawled pastes by domain
2018-08-27 14:34:08 +02:00
Terrtia
823c33174a
Merge branch 'master' into onion_crawler
2018-08-27 11:20:51 +02:00
Terrtia
5c902640e9
Merge branch 'master' into onion_crawler
2018-08-27 11:05:25 +02:00
Terrtia
d42dd118a4
chg: [Domain crawled] add random screenshot
2018-08-27 11:02:39 +02:00
Terrtia
a586548047
Merge branch 'master' into base64
2018-08-24 10:35:28 +02:00
Terrtia
ced0b1e350
chg: [I2P] add default config
2018-08-24 10:24:03 +02:00
Terrtia
7e24943537
chg: [Crawler] crawler accept all kind of domains
2018-08-24 10:13:56 +02:00
Terrtia
e9580d6775
chg: [Crawler] change BDD, save i2p links
2018-08-21 15:54:53 +02:00
Terrtia
ed559d9f4a
chg: [Showpaste] add screenshot + improve onion db
2018-08-16 17:24:39 +02:00
Terrtia
6f2dbcb4e2
fix: [pystemon-feeder] fix #234
2018-08-13 10:22:20 +02:00
Terrtia
7652089433
chg: [Onion] change onion regex, fix crawler
2018-08-13 09:23:14 +02:00
Terrtia
8b1c10b38c
chg: [Onion] add onion splash crawler
2018-08-09 17:42:21 +02:00
Thirion Aurélien
000d5bf40a
fix: add miss config
2018-08-07 13:07:08 +02:00
Terrtia
d7545a46f0
chg: [Mixer] add submitted paste
2018-08-02 15:29:13 +02:00
Terrtia
449da63fd7
chg: [Keys] typo
2018-08-01 16:05:49 +02:00
Terrtia
67b0d973af
chg: [Keys] detect ssh2 private key
2018-08-01 16:04:06 +02:00
Terrtia
d0135c248d
add iban country statistic
2018-08-01 15:26:29 +02:00
Terrtia
940b96734a
Merge branch 'master' into statistic
2018-08-01 09:29:51 +02:00
Thirion Aurélien
f7f4422ff1
Merge pull request #232 from CIRCL/iban
...
Add Iban detection
2018-08-01 08:57:48 +02:00
Terrtia
b5c1f281e6
fix: typo
2018-07-31 10:58:23 +02:00
Terrtia
61e98ed8fb
clean
2018-07-31 10:51:18 +02:00
Terrtia
e455d244d8
chg: [statistics] add domain statistics
2018-07-31 10:36:29 +02:00
Terrtia
5dcb4ebb58
chg: [statistics] add script to generate pie chart png
2018-07-30 16:36:34 +02:00
Terrtia
09fbc363f1
chg: [sglinjection Phone] add tld statistic, fix phone regex
2018-07-30 11:56:50 +02:00
Terrtia
c20e7d5ab4
chg: [Mail Credential] add tld statistic
2018-07-30 10:19:26 +02:00
Terrtia
39f54d4420
chg: [Mail] add country statistic
2018-07-30 09:21:22 +02:00
Terrtia
009f87ee10
[fix: [UI-hash] fix graph node zoom
2018-07-27 11:27:41 +02:00
Terrtia
153f9f38cc
chg: [BankAccount] regex
2018-07-27 10:20:03 +02:00
Terrtia
57b631233b
add: [BankAccount] iban detection
2018-07-26 15:31:58 +02:00
Terrtia
5ad5939bd5
chg: display last logs
2018-07-25 16:48:44 +02:00
Terrtia
03a37af5cf
chg: [Hash] add reprocess regex order
2018-07-25 10:47:36 +02:00
Terrtia
8beb63b6a8
fix: [Queues] module with no subscriber
2018-07-24 15:57:18 +02:00
Terrtia
8c97c70a3f
Merge branch 'master' into base64
2018-07-24 15:21:32 +02:00
Terrtia
4b8c26dec8
fix: [hive auto export] fix #221
2018-07-24 14:46:35 +02:00
Terrtia
54e8750c30
fix: [LAUNCH] script checking
2018-07-24 12:18:41 +02:00
Terrtia
d4f0c14620
fix: [LAUNCH] use envar, add options (launch all, flask, killall)
2018-07-24 10:41:10 +02:00
Terrtia
9c03c40482
chg: [UI-Hash] add decoder pie
2018-07-23 11:11:52 +02:00
Terrtia
1fbd4109c2
chg: [UI-Hash] add decoded type on graph
2018-07-20 15:48:56 +02:00
Terrtia
6d6de3ae54
fix: remove experimental modules
2018-07-20 10:38:04 +02:00
Terrtia
bd5f83f0eb
chg: refractor base64 encoded to hash
2018-07-20 10:32:52 +02:00
Terrtia
9a8e37fb0b
fix: decoder bdd
2018-07-20 09:43:09 +02:00
Terrtia
0e67b56906
add: Decoder module
2018-07-19 16:52:09 +02:00
Terrtia
ad26f016e3
add: Decoder Module, decode binary, hex and base64
2018-07-19 16:50:42 +02:00
Terrtia
280e9d8af8
fix: Binary decoder base2 string to bytes
2018-07-18 13:47:06 +02:00
Terrtia
e25dd52d64
add: Binary decoder
2018-07-18 11:45:19 +02:00
Terrtia
0cac3146f1
fix: remove hardcoded mail url
2018-07-17 15:11:25 +02:00
Terrtia
fa38434e04
add: new mail body and starttls
2018-07-16 15:51:37 +02:00
Terrtia
8b245d17d9
fix: add LibInjection to LAUNCH.sh
2018-07-16 15:45:55 +02:00
Terrtia
6f69da0c0d
fix: paste submit css
2018-07-16 10:32:41 +02:00
Terrtia
fce324e370
fix: release tiemeout alarm, chg:icon graph node
2018-07-13 11:54:14 +02:00
Terrtia
87b7facba8
add: base 64 node graph
2018-07-12 17:07:17 +02:00
bui
bd5c42caeb
add LibInjection Module
2018-07-04 14:28:10 +02:00
bui
362f72fe43
initial LibInjection module
2018-07-04 14:27:54 +02:00
Thirion Aurélien
f8a88cecec
fix: hive_feeder hiveApi declaration
2018-07-02 15:23:21 +02:00
Terrtia
44524b25a4
fix submit paste
2018-06-29 15:30:01 +02:00
Terrtia
a0a639276c
fix submit paste
2018-06-29 15:29:02 +02:00
Terrtia
98d0235fc3
Merge branch 'master' into base64
2018-06-29 14:57:46 +02:00
Terrtia
f5bda70e82
Merge branch 'master' of https://github.com/CIRCL/AIL-framework
2018-06-29 13:51:08 +02:00
Terrtia
72146f938e
change: feeder name separator
2018-06-29 13:50:41 +02:00
Thirion Aurélien
c41a165a21
fix: file submission, content type
2018-06-29 12:08:46 +02:00
Terrtia
407aa14244
fix: catch file error in paste submission
2018-06-29 11:32:52 +02:00
Terrtia
16c700ddb4
new: add base64 display on paste + base64 statistics
2018-06-29 10:02:29 +02:00
Terrtia
225776c69c
fix: tp and fp, errors messages on paste submission
2018-06-20 10:02:26 +02:00
Terrtia
6a89fac309
fix hive and misp error
2018-06-19 16:39:49 +02:00
Terrtia
af9db53171
add tag fp and tp
2018-06-19 15:09:26 +02:00
Terrtia
86c2cfa04b
update ail-leak object
2018-06-18 17:23:41 +02:00
Terrtia
634bf6e552
add img + fix upload api
2018-06-18 17:16:22 +02:00
Terrtia
704e6f0ad1
change default submit tag, event-path map, publish events
2018-06-18 13:58:31 +02:00
Terrtia
aee1b50ef1
fix duplicate export attribute + tag export whitelist
2018-06-15 17:25:43 +02:00
Terrtia
31fc33ab03
misp event creation, auto and manual + the hive alert auto creation, manual case creation
2018-06-14 16:51:06 +02:00
Terrtia
8522d50308
modify navbar + decompress file password
2018-06-11 10:21:11 +02:00
Terrtia
c64588f60e
add feature, user can submit paste on the web interface
2018-06-08 16:49:20 +02:00
Terrtia
6143bc3dce
submit users input paste
2018-06-05 16:58:04 +02:00
Terrtia
6287981668
clean
2018-06-01 09:49:26 +02:00
Terrtia
884d655b0c
taxonomies and galaxies update via LAUNCH
2018-05-31 13:53:29 +02:00
Terrtia
93dcdf4f23
fix duplicate
2018-05-31 09:01:47 +02:00
Terrtia
0d164b6f4c
add tag js and css
2018-05-16 14:48:15 +02:00
Terrtia
b1b7d33c96
tags
2018-05-16 14:39:01 +02:00
Terrtia
f66a528bc2
fix Duplicate, save list of duplicates on disk + prevent empty hash creation
2018-05-15 23:28:47 +02:00
Terrtia
225fe76c96
fix duplicate + get raw paste
2018-05-11 16:07:30 +02:00
Terrtia
d7ba5533be
fix duplicate + fix issue #200
2018-05-11 13:25:45 +02:00
Terrtia
8599d7142d
python 3 upgrade instructions
2018-05-09 13:32:25 +02:00
Terrtia
142c1dc9b0
clean
2018-05-09 13:03:46 +02:00
Terrtia
3841c159a0
change leveldb to ardb
2018-05-07 14:50:40 +02:00
Terrtia
c17b9402f7
clean
2018-05-04 14:25:47 +02:00
Terrtia
96a3df42bc
decode with redis connection
2018-05-04 13:53:29 +02:00
Terrtia
ae8f575879
use feeder name
2018-05-03 16:32:25 +02:00
Terrtia
a900d5f08d
add python 2 to 3 bash + fix Duplicate + clean
2018-05-03 16:21:33 +02:00
Terrtia
c52caebe7c
improve Duplicate + tlsh + add time out handler + debug + clean
2018-05-02 17:07:10 +02:00
Terrtia
92977201fd
create json metadata for base64 module
2018-04-27 15:46:41 +02:00
Terrtia
c3428df5dc
add apiKeys module
2018-04-26 14:42:39 +02:00
Terrtia
29c368250e
create new dashbord alert
2018-04-24 17:03:10 +02:00
Terrtia
8738b7cf75
fix track term and avoid duplicate mail address
2018-04-24 16:44:37 +02:00
Terrtia
a7bd01ed9a
Merge branch 'python3' of https://github.com/CIRCL/AIL-framework into python3
2018-04-23 16:22:51 +02:00
Terrtia
4e55f6ee90
fix Onion module
2018-04-23 16:22:23 +02:00
Thirion Aurélien
ef3c335b62
Delete config.cfg.backup
2018-04-23 14:54:23 +02:00
Terrtia
f93fe9aeb2
add bitcoin module
2018-04-23 14:51:53 +02:00
Terrtia
3fc48db903
add Base64 module
2018-04-20 10:48:44 +02:00
Terrtia
41eb250278
fix python 3 compqtibility issues
2018-04-20 10:42:19 +02:00
Terrtia
8571a86344
frontend python 3.5 upgrade
2018-04-17 16:06:32 +02:00
Terrtia
19c90339e0
clean
2018-04-17 09:43:45 +02:00
Terrtia
d0261233aa
fix ModulesInnformationV2
2018-04-17 09:37:58 +02:00
Terrtia
15ef02fe53
fix python3 types issue
2018-04-16 17:00:44 +02:00
Terrtia
9e07a0ebdb
fix python3 types issue
2018-04-16 15:04:22 +02:00
Terrtia
9c82dd90ec
python 3 backend upgrade
2018-04-16 14:50:04 +02:00
Terrtia
3395b16873
fix incoming encoding file issue
2018-04-13 09:17:56 +02:00
Terrtia
347588fdec
python3.5 Mixer and Global module
2018-04-12 17:06:57 +02:00
Terrtia
4386cbbd34
fix travis
2018-04-11 10:48:22 +02:00
Terrtia
14583f702f
create first test
2018-04-11 10:14:33 +02:00
Terrtia
e9b88b0656
Keys module improvement
2018-04-05 11:40:34 +02:00
CD Nguyen
96259c53bc
refactor NotificationHelper code
2018-04-04 09:41:13 +02:00
CD Nguyen
e9e3f17363
support non TSL SMTP servers
2018-03-30 11:35:37 +02:00
Philipp Schmied
29608ab29b
Fixed e-mail notificatins for sets and regexes
2018-02-28 11:31:16 +01:00
Philipp Schmied
fd329aaba9
config.cfg.sample: Merged overwritten changes from upstream
2018-02-27 16:16:57 +01:00
Philipp Schmied
5b1f0b0212
Implemented email notifications (bin: config.cfg additions, email sending via analyzer scripts; var: Changes to add notifications via terms management); terms_management: Fixed click handlers not being added to all tracked terms.
2018-02-27 15:12:02 +01:00
abdf41fe23
Merge pull request #164 from mokaddem/showDiff
...
Show differences between duplicates
2018-02-21 22:25:16 +01:00
0263b25da8
Merge pull request #163 from mokaddem/iss160
...
Moving more to config
2018-02-21 22:24:50 +01:00
Sami Mokaddem
cbb19f95e4
fix: asciimatics introduced the '_required_height' property in the Label class. Just updating here to mirror the change
2018-02-16 14:35:09 +01:00
Sami Mokaddem
3e117195a2
fix: Added a backoff time if an IOError happens
2018-01-15 17:45:13 +01:00
Sami Mokaddem
9a66a6111b
Fixing no such file
2018-01-15 17:10:03 +01:00
Sami Mokaddem
c39e537cbe
Merge branch 'iss160' into showDiff
...
Preemptively preventing merge-conflict with branch iss160 (moving to
configuration)
2017-12-12 11:33:22 +01:00
Sami Mokaddem
05b22e8963
typo: added word + deleted trailing spaces
2017-12-12 11:19:37 +01:00
Sami Mokaddem
ef1bd16a72
typo: added word + deleted trailing spaces
2017-12-12 11:16:40 +01:00
Sami Mokaddem
304213cfa4
update: Possibility to change the authorized max line length in the config for the diff of duplicates
2017-12-12 11:00:26 +01:00
Sami Mokaddem
860acbf7cf
fix: Renamed Credentials into Credential sectiont title
...
config change: By default, zmq listen to both local and CRF
2017-12-12 09:29:15 +01:00
Sami Mokaddem
9d0d0b4303
update: Moved filtering operation (thresholds, number of matching in the
...
categ file) in the configuration file. It permits to better control the
flow of pastes.
Also set default mixer duplicate filtering to 3 (Do not filter)
2017-12-11 17:28:34 +01:00
Fabien Mathey
3105192312
Allow for multiple entries like 1 2 3 4 5
...
Should replace the need to type
1<Enter>2<Enter>
etc. and allows for
1 2 3 4 5<Enter>
2017-11-28 15:15:12 +01:00
71922f9def
add: Python script to test if the ZMQ feed works as expected
2017-11-28 09:46:06 +01:00
Sami Mokaddem
83e082e62a
update: removed useless comments
2017-11-24 08:57:41 +01:00
Sami Mokaddem
a0d07b1098
feature: try to send data to MISP if keys are provided, else do nothing
2017-11-23 13:47:02 +01:00
Sami Mokaddem
2a967c4d92
update/feature: Max number of duplicate push to MISP + duplicate are pushed as attachment
2017-11-23 07:13:44 +01:00
Sami Mokaddem
805171a8a5
Added a seemingly working version of ailleak to misp
2017-11-20 14:57:25 +01:00
Sami Mokaddem
e0b188c069
Better support of python3 in Helper.py
2017-11-20 14:55:36 +01:00
Sami Mokaddem
1e76e10c01
Support of add_object to a MISP instance
2017-11-20 12:12:53 +01:00
Sami Mokaddem
6f0227a78a
Starting support of python3
2017-11-20 12:12:06 +01:00
Sami Mokaddem
ee7759be91
Updated alertHandler and ailleakObject
2017-11-16 11:18:13 +01:00
Sami Mokaddem
5c7c07fbea
Merge branch 'launcher_update' into misp_obj
2017-11-16 10:09:05 +01:00
Sami Mokaddem
45a9897020
Added draft support of MISP ail-leak object
2017-11-16 09:52:37 +01:00
Sami Mokaddem
652b00f4b2
Renamed BrowseWarningPaste into alertHandler
2017-11-15 16:15:43 +01:00
Sami Mokaddem
ec7d2d49b2
Added seconds args in import_dir
2017-11-15 16:03:42 +01:00
Sami Mokaddem
13f1c676d6
Added '_AIL' to the end of screen session name to better check an kills sessions. Also, gracefully close redis server + Check 1 time if redis-servers are running.
2017-09-26 16:17:06 +02:00
=
a8dc9231c6
Typo2..
2017-08-24 16:43:42 +02:00
=
b2d4dd4a1f
Typo
2017-08-24 16:43:05 +02:00
=
74ff401e1d
Fixed bug: No CR between indexes in all_index.txt file
2017-08-24 16:41:30 +02:00
bb128a5e94
Merge pull request #138 from mokaddem/import_dir_update
...
Improvement of import_dir
2017-08-24 09:40:52 +02:00
=
429cc96d2a
Improvement of import_dir.py. Added feed name parameter, send the correct path (without potential ../) and gzipEncode it if needed.
2017-08-23 15:10:26 +02:00
=
e7c66d1aac
Updated individual launch scripts
2017-08-23 12:28:12 +02:00
=
d632335760
Automatically start redis_levelDB with the correct date
2017-08-23 10:16:56 +02:00
=
64ec44085d
Dynamically choose correct port for LevelDB in script
2017-08-22 17:52:15 +02:00
=
66d60a1f01
Added LevelDB Curve for 2017 (3017, used for backward compatibility)
2017-08-21 13:54:44 +02:00
=
92adea38d0
Semi-automated directory creation with date as name in LAUNCH and installing_deps
2017-08-21 13:48:02 +02:00
=
bc35747f00
Updated KillAll in LAUNCH
2017-08-21 13:19:26 +02:00
Mokaddem
c3632c7d86
Added operation mode 3 for Mixer (do not filter) and added possibility to recurse through dir for import_dir
2017-08-03 00:34:53 +02:00
Mokaddem
cbb2909156
Added comments
2017-07-20 10:24:48 +02:00
Mokaddem
05417f405c
Finished basic link web interface + solved some bugs
2017-07-20 10:04:30 +02:00
Mokaddem
ef741f85c0
Added similarity checks + started link webInterface
2017-07-19 11:52:06 +02:00
Mokaddem
882ab88b00
Draft module + web link
2017-07-18 16:57:15 +02:00
Mokaddem
9558aafe67
Prevent the loss of the 1 message in import_dir.py
2017-07-17 16:14:27 +02:00
Mokaddem
4d59871f00
Merged with previous changes for PR
2017-07-17 13:48:16 +02:00
4986352099
Merge pull request #120 from PaulSec/master
...
Added Python Phonenumbers module to parse numbers and retrieves stats…
2017-06-08 07:05:52 +02:00
446a97c47a
Merge pull request #115 from mokaddem/regexInTerms
...
New features for terms frequency
2017-06-08 07:04:17 +02:00
Mokaddem
3a4dcd691d
Improved description of modules inside the scripts
2017-05-09 11:13:16 +02:00
Paul A
747fff1df1
Added Python Phonenumbers module to parse numbers and retrieves stats from countrycode
2017-05-03 14:25:18 +02:00
Mokaddem
3ea0924120
Improved phone regex to handle end with 4 digits and presence of parenthesis.
2017-04-26 12:41:23 +02:00
Mokaddem
b60e164380
Added support of onion in browseWarningPaste
2017-04-26 11:47:02 +02:00
Mokaddem
318a557c56
Added possibility to keep a backup of the old config + check if config is valid before starting scripts.
2017-04-25 09:32:05 +02:00
Mokaddem
72c1d903eb
Merge branch 'newIndexer' into auto-conf
2017-04-25 08:14:31 +02:00
Mokaddem
eccc8fc88b
modified check_size timeout + utilisation of shutil instead of os.system()
2017-04-19 16:37:04 +02:00
Mokaddem
da081bae42
Merge branch 'regexInTerms' into production
2017-04-18 16:59:40 +02:00
Mokaddem
2d907ae784
Merge branch 'newIndexer' into auto-conf
2017-04-18 16:20:00 +02:00
Mokaddem
b0f1cf84b7
remplaced usage of global variable by a function param
2017-04-18 16:02:22 +02:00
Mokaddem
6e8509a0ba
Removed wierd line
2017-04-18 15:57:07 +02:00
Mokaddem
c8baabd882
Updated TermsFrequency related modules + Fixed bugs (encoding, behaviors, ...)
2017-04-18 15:28:21 +02:00
Mokaddem
2da4c572c7
New feature in Terms: regex and set of words support (draft)
2017-03-28 17:42:44 +02:00
Mokaddem
6e80eb0954
Typo
2017-03-28 09:55:59 +02:00
Mokaddem
c79d4d65c8
Keep original section order while updating
2017-03-28 09:54:24 +02:00
Mokaddem
e225090e07
Added script that checks configuration and may auto-fix it + added missing field in config.cfg.sample
2017-03-15 16:36:51 +01:00
Mokaddem
be93af6f5a
Merge branch 'terms-improvements'
2017-03-15 14:55:44 +01:00
Mokaddem
3261eca836
Fixed all_index newline bug + added print of what's happening
2017-03-15 14:29:49 +01:00
Mokaddem
5b7d047622
Changed os.system by os.mkdir + fixed bug during rotation
2017-03-15 14:05:13 +01:00
Mokaddem
adc37ab50e
Indexer max size now in config
2017-03-15 12:14:41 +01:00
Mokaddem
4005b72180
Added backward support + renamed index with timestamp
2017-03-15 11:51:35 +01:00
Mokaddem
248469d61e
Indexer now create an index_dir when it became too large (search in index not updated yet)
2017-03-14 10:37:31 +01:00
1540b719ed
Merge pull request #108 from mokaddem/newModuleInformation
...
New version of ModuleInformation
2017-03-14 08:29:09 +01:00
Mokaddem
3c9cdac39a
Merge branch 'terms-improvements' into production
2017-02-28 16:51:39 +01:00
Mokaddem
dadc872f9f
Merge branch 'newModuleInformation' into production
2017-02-28 16:51:08 +01:00
Mokaddem
a2849c1bcd
Added a catch for psutils if process get stopped by user
2017-02-28 16:13:01 +01:00
Mokaddem
4071bf2e91
fixed a bug in manage_term (usage of loop.index0) + fixed bug in top_set where graph was not displaying the correct per_paste value
2017-02-28 15:01:48 +01:00
Mokaddem
a1c47d7a13
Merge branch 'production' into terms-improvements
2017-02-28 11:15:56 +01:00
Mokaddem
fd4feb374f
Merge branch 'newModuleInformation' into production
2017-02-28 11:12:06 +01:00
Mokaddem
75b4037519
Fixed bug - If we move the cursor from a line that has dissapear
2017-02-28 11:11:17 +01:00
Mokaddem
d6e02591af
Merge branch 'newModuleInformation' into production
2017-02-28 10:23:45 +01:00
Mokaddem
975c1a37d9
ModuleInformation create the all_modules files if missing
2017-02-28 10:23:22 +01:00
Mokaddem
ccdf256383
Added possibility to show or not in dashboard text that is logged
2017-02-28 10:21:27 +01:00
Mokaddem
39104dea5e
Merge branch 'bugfix_categ' into production
2017-02-28 09:56:25 +01:00
Mokaddem
26eb5fdff0
Fixed bug closing parenthesis in categ + changed behavior of sending to browseWarningPaste in Mail
2017-02-28 09:23:32 +01:00
Mokaddem
4a0dac1fae
Merge branch 'production' into terms-improvements
2017-02-15 16:31:22 +01:00
Mokaddem
a696ccf83b
Merge branch 'newModuleInformation' into production
2017-02-15 16:30:44 +01:00
Mokaddem
85a6f6fa63
Fixed a bug (str+float)
2017-02-15 16:30:31 +01:00
Mokaddem
25adae0104
Added possibility to select in top_terms the data per_paste or not (need module testing)
2017-02-15 16:29:02 +01:00
Mokaddem
e98e92b0d4
Merge branch 'fix-linkPaste-debug' into production
2017-02-15 13:43:03 +01:00
Mokaddem
c552aced42
Merge branch 'newModuleInformation' into production
2017-02-15 13:42:46 +01:00
Mokaddem
1b0d9d9507
Fixed str and int concat + Added sorting by queue name in queue not running
2017-02-15 13:42:21 +01:00
Mokaddem
74e0bc7acb
Merge branch 'newModuleInformation' into production
2017-02-15 11:22:50 +01:00
Mokaddem
7fc9ba18d7
Added support of ENTER for navigation
2017-02-15 11:01:11 +01:00
Mokaddem
b84194542d
Moved logging into a function
2017-02-15 10:57:41 +01:00
Mokaddem
0fce3f5d65
Renamed variables + added comments
2017-02-15 10:37:59 +01:00
Mokaddem
9ca3e79e70
Removed useless comments + commented
2017-02-15 10:19:03 +01:00
Mokaddem
9199ea3f0e
Added comments
2017-02-15 10:04:51 +01:00
Mokaddem
ff1df07b5d
Added comments and moved portion of code for slightly better visibility
2017-02-15 09:43:18 +01:00
Mokaddem
b272b1b07b
Added a way to quit application with 'q' more or less properly
2017-02-15 09:35:03 +01:00
Mokaddem
0254c51164
Added end of paste marker + handling screen resize
2017-02-15 08:26:48 +01:00
Mokaddem
0b9609e4a9
Merge branch 'newModuleInformation' into production
2017-02-14 17:22:06 +01:00
Mokaddem
444f6071b7
Fixed screen bug by adding an input before starting asciimatics + improved path localisation in the logs from modules
2017-02-14 17:21:52 +01:00
Mokaddem
b04bac4731
Merge branch 'newModuleInformation' into production
2017-02-14 16:08:42 +01:00
Mokaddem
d520c45a15
Added support of display of the last paste processed by a module
2017-02-14 16:05:59 +01:00
Mokaddem
d3b4409156
Added path for debug in SQLInjection
2017-02-14 11:23:34 +01:00
Mokaddem
135a1457e0
Merge branch 'fix-linkPaste-debug' into production
2017-02-14 11:12:22 +01:00
Mokaddem
d4da3a316f
Added path of the paste in the log of Categ.py
2017-02-14 10:59:47 +01:00
Mokaddem
303575b2a1
Merge remote-tracking branch 'upstream/master' into production
2017-02-14 09:31:18 +01:00
2f4e5b9016
Merge branch 'master' of github.com:CIRCL/AIL-framework
2017-01-26 07:16:44 +00:00
cf903cc212
(partially fix #90 too) using a simple Alarm (SIGNAL) when exec-timeout
...
Introducing a timer (in this case 60 seconds) to ensure that the
execution time of the analysis takes less than 60 seconds. This
is a simple and standard POSIX signal handler. If the timeout
is reached, the module will process the next one.
This approach fixes the specific issues we have currently
with some inputs where the sentiment analysis takes too much time. This
fix should be improved and be more generic:
- Introducing statistics of content which timeouts.
- Keeping a list/queue to further process those files using a different
analysis approach. Maybe a set of "dirty" processes to handle the edge cases
and to not impact the overall processing and analysis.
- Make the timer configurable per module (at least for this one).
2017-01-26 07:11:18 +00:00
Mokaddem
c422db6e6d
Merge remote-tracking branch 'upstream/master' into production
2017-01-17 16:38:48 +01:00
mokaddem
b3a3616b0b
Merge pull request #102 from ptitdoc/bugfix1
...
Bug fixes in Workers and improvement of pystemon-feeder
2017-01-17 16:35:32 +01:00
Mokaddem
ea423fde0f
Added requirement, better behaviors and confirmation box
2017-01-17 15:58:16 +01:00
Mokaddem
692f13d18b
Improved module + Added support of click and keyboard navigation for killing and starting modules
2017-01-17 14:31:06 +01:00
Mokaddem
3b8149d840
Added action button and support of logs
2017-01-16 17:08:48 +01:00
Mokaddem
0ecfaf00f5
Added color in front of queue related to its state
2017-01-16 15:30:33 +01:00
Mokaddem
af3241c008
Overwritted listbox update function to add formating
2017-01-16 14:41:02 +01:00
Mokaddem
30f303cb99
Added new ModuleInformation
2017-01-16 12:18:23 +01:00
Olivier MEDOC
16044d4d36
pystemon-feeder: add the option bind in configuration file instead of address
...
This is essentially because multiple feeds can be used in the address
configuration variable.
2017-01-13 14:58:18 +01:00
3b101ea8f5
(partially) Fix #91 using a simple Alarm (SIGNAL) when exec-timeout
...
Introducing a timer (in this case 5 seconds) to ensure that the
execution time of the tokenizer takes less than 5 seconds. This
is a simple and standard POSIX signal handler.
This approach fixes the specific issues we have currently
with some inputs where the tokenization takes too much time. This
fix should be improved and be more generic:
- Introducing statistics of content which timeouts.
- Keeping a list/queue to further process those files using a different
tokenizer approach. Maybe a set of "dirty" processes to handle the edge cases
and to not impact the overall processing and analysis.
- Make the timer configurable per module (at least for this one).
2017-01-12 07:32:55 +00:00
Olivier MEDOC
9f9c265cb0
SentimentAnalysis: use lexicon file path from the ail configuration file
2017-01-11 11:07:08 +01:00
ptitdoc
c0fac82031
Add pystemon path in configuration sample
2017-01-10 18:33:46 +01:00
Olivier MEDOC
83db40104c
import pystemon-feeder configuration from the central configuration file
2017-01-10 18:26:36 +01:00
Olivier MEDOC
3dc014dad9
fix invalid variable propagation
2017-01-10 18:26:24 +01:00
Olivier MEDOC
68ca20db8f
fix to use the new pycountry API
2017-01-10 18:26:17 +01:00
1950a2dc0e
Merge pull request #99 from mokaddem/more-feed
...
Usage of sha1 to filter paste content in Mixer.py
2017-01-09 17:06:37 +01:00
Mokaddem
500608d0a1
Usage of sha1 to filter paste content in Mixer.py
2017-01-09 17:01:10 +01:00
f5adb65cb3
Merge pull request #98 from mokaddem/more-feed
...
Config 2017
2017-01-09 15:39:53 +01:00
Mokaddem
409c5b8b42
Added 2017
2017-01-09 15:36:50 +01:00
a96ba40d94
Merge pull request #96 from mokaddem/more-feed
...
Handling of multiple feeders
2017-01-09 15:29:28 +01:00
Mokaddem
0e39f516a9
Added sleep in helper for multiple listening interfaces (reduce useless work), Added a simple way to pre-process feed before sending it to the global feed.
2017-01-09 14:12:26 +01:00
Mokaddem
3a298f0c7c
Commented Mixer.py for further multiple feeds support.
2016-12-26 16:23:18 +01:00
Mokaddem
1c2169d3bc
Improved interface for multiple feeds and refresh_script.
2016-12-26 16:16:44 +01:00
Mokaddem
97292e0899
Updated web interface to handle new mixer module. and fixed one dependency bug.
2016-12-23 15:44:46 +01:00
Mokaddem
e70b9cd15c
Added basic mixer with confirugable behavior. It handles muliple feeders and performs some basic stats on them.
2016-12-23 10:31:26 +01:00
ddfe3cf66a
Merge pull request #95 from mokaddem/module_handling
...
Module handling
2016-12-22 22:18:31 +01:00
3ce850e093
Merge pull request #94 from mokaddem/website-refacto
...
Website refacto, support and fixes
2016-12-22 22:17:58 +01:00
Mokaddem
9758579753
Merge branch 'module_handling' into production
2016-12-22 10:24:30 +01:00
Mokaddem
7438f16c63
Added new config
2016-12-22 10:06:35 +01:00
Mokaddem
73938cf41b
Prepared moduleInfo to handle module click with curses (will come back later).
2016-12-22 08:45:53 +01:00
Mokaddem
d37b243e1d
Added legend + added more info if restarting disabled
2016-12-09 15:13:47 +01:00
Mokaddem
d3ed1bbf40
Merge branch 'website-improvement' into production
2016-12-08 10:05:55 +01:00
Mokaddem
73d4f9e082
Webstats should correctly updates top_progression_zset (Not fully tested because not enough data. Will be tested latter)
2016-12-08 10:05:07 +01:00
Mokaddem
570324060e
terms top_sets correctly supports blacklisted terms
2016-12-08 09:13:31 +01:00
412b80c44d
Merge pull request #88 from mokaddem/Issue-Feeding-AIL
...
Pystemon tuto
2016-11-26 12:45:28 +01:00
Mokaddem
d831a79614
Do not try to restart not running queue if autokill is not set
2016-11-25 16:00:46 +01:00
Mokaddem
4bf45eea36
ModuleInformation sort by name as well as elapsed time
2016-11-25 14:44:27 +01:00
Mokaddem
866be4e884
Added row coloring in ModuleInformation
2016-11-25 14:12:09 +01:00
Mokaddem
a3255d168c
ModuleInfo correctly handle CurveManageTopSets,
...
Changed queue coloring in dashboard,
ModuleInformation now have a history of executed command and better handle module killing.
2016-11-25 11:54:16 +01:00
Mokaddem
224fbc8084
Better handle stuck modules. Differentiate between not running and no info and tries to restart stuck ones.
2016-11-24 16:58:32 +01:00
Mokaddem
5d269ea1ee
Added date in Duplicate for better distinguish duplicate. Also, added a timeout for the dataTable in duplicate inside the modal.
2016-11-24 15:05:29 +01:00
Mokaddem
ac254e0e63
Added support of browsing concerned paste in dashboard for all modules + Show in the page what was the concerned module
2016-10-27 11:50:24 +02:00
Mokaddem
1826b170ec
Added support of local paste view in dashboard for Mails event only
2016-10-27 11:27:26 +02:00
Mokaddem
5a4ea734a2
Fixed path issues in doc + added dependencies related to ssdeep
2016-10-14 14:26:33 +02:00
Mokaddem
4eafca2936
Added pystemon instruction section
2016-10-13 17:13:08 +02:00
eb14005fcf
Fix: as JSON decode needs a string, the empty list is a string.
2016-08-29 12:08:12 +00:00
3b6e3e156e
Merge branch 'master' of github.com:CIRCL/AIL-framework
2016-08-26 15:57:45 +00:00
Mokaddem
0c760d763b
Added support of PID in ModuleInformation and index.html
2016-08-26 15:27:37 +02:00
758f2c5334
Merge branch 'master' of github.com:CIRCL/AIL-framework
2016-08-26 08:30:11 +00:00
Mokaddem
494980c48e
Added comment in moduleInformation
2016-08-26 10:07:06 +02:00
Mokaddem
8b33b4e7ba
Added not running module info + swapped order of module launch
2016-08-26 09:21:02 +02:00
Mokaddem
813cc7031a
Changed lots of code into zrank
2016-08-25 17:48:12 +02:00
Mokaddem
aaac798db9
updated individual launch script
2016-08-25 16:51:57 +02:00
Mokaddem
78c991539e
Added number of same module running in ModuleInformation
2016-08-25 16:07:47 +02:00
1ab08d4a05
Merge branch 'master' of github.com:CIRCL/AIL-framework
2016-08-25 12:33:31 +00:00
Mokaddem
f9bf00f022
Removed autokill feature if not specified
2016-08-25 11:09:31 +02:00
Mokaddem
2f136595fb
Removed autokill feature if not specified
2016-08-25 11:05:18 +02:00
Mokaddem
9146feab88
Fixed bug in WebStats not deleting correct key + fixed typo in CurveSetManager preventing connecting to redis + modified display in moduleInfo
2016-08-25 10:43:04 +02:00
Mokaddem
3313064e51
removed useless import.
2016-08-24 17:32:50 +02:00
Mokaddem
64bf481480
Added moduleInfo + Renamed modules
2016-08-24 17:28:39 +02:00
Mokaddem
8fb552b7da
ModuleInfo support autokill and auto restart after logging.
2016-08-24 16:52:01 +02:00
Mokaddem
3e88d0911d
ModuleInfo now support logs.
2016-08-24 15:53:00 +02:00
Mokaddem
ff6dcaf456
Fixed typo in creditcard + fixed bug in ModuleInformation (relative to date).
2016-08-24 15:35:23 +02:00
Mokaddem
deeebec2f5
Added new module 'ModuleInformation' which gives real time information on running module
2016-08-24 15:21:41 +02:00
Mokaddem
07856f3119
Added script module-dataflow-graph + fixed bug in module.cfg preventing the credential module to receive data.
2016-08-24 11:32:48 +02:00
bde5f47ed0
Merge branch 'master' of github.com:CIRCL/AIL-framework
...
Conflicts:
bin/launch_scripts.sh
2016-08-23 14:20:39 +00:00
Mokaddem
42b3966499
Added new module in individual launch scripts
2016-08-23 16:01:11 +02:00
Mokaddem
697812dbff
Added config.sample
2016-08-23 09:59:39 +02:00
Mokaddem
f9b27e4013
Fixed bug in curve preventing to track concerned_pastes. Now all tracked words are put in lower case.
2016-08-23 09:50:03 +02:00
Mokaddem
047331df3e
Added save of paste_baseName in set for searches. This is done in ModuleStats to not surcharge the Global module.
2016-08-23 08:52:54 +02:00
Mokaddem
20fd191c20
Suppressed prints
2016-08-22 21:29:59 +02:00
Mokaddem
2fd2c8af39
Removed commented code
2016-08-22 20:59:56 +02:00
Mokaddem
003c72bd7c
Fixed bug with redis where the inserted key was too long, causing a crash + fixed bug taking min instead of max in terms-top-set
2016-08-22 16:01:42 +02:00
Mokaddem
2466f355ab
Fixed bug in manage-terms-top-set + added auto sorting in important-paste-by-module
2016-08-22 14:54:37 +02:00
Mokaddem
d3c0fc5f94
Added terms trending module and web interface:
...
- Curve module has been rewritten
- Added new module curve_manage_top_set
- Added all web-pages
2016-08-22 13:35:49 +02:00
Mokaddem
0b6828fb16
Correctly handle and display add or remove a term in term-frequency + started sketch of terms-plot tool.
2016-08-19 16:53:46 +02:00
Mokaddem
b78effe779
Added draft of terms_management + Added copyright in readme
2016-08-19 13:34:02 +02:00
Mokaddem
52b7124a6b
- Modified redis connection (from levelDB to redis).
...
- Added term frequency in curve.
- Modified ModuleStats and WebStats to use redis special command (incrby instead of get-set and zset) and Flask to perform the correct queries.
- Added panel color in sentiment trending
2016-08-18 15:34:19 +02:00
Mokaddem
902b7e39f4
Probably fixed bug in creditcard (not sending to duplicate)
2016-08-17 14:40:13 +02:00
Mokaddem
c10f28d9df
Updated isolated launch scripts + Added switch displaying queue in index
2016-08-17 13:46:01 +02:00
Mokaddem
dbc164f4e8
modified + into append
2016-08-17 11:26:07 +02:00
Mokaddem
c12e67a35a
Added tooltips + slight gauge modification
2016-08-17 10:43:43 +02:00
Mokaddem
894b9efda9
Added drop of really long line in sentiment-analysis module + Added description of sentiment module. Also, fixed bug in webpage sentiement-trending concerning avg and date range.
2016-08-17 09:46:25 +02:00
Mokaddem
1084e45f1b
Added draft of filter in sentiment analysis (Discard syntaxical languages) + Added nice tooltip for sparkline. Trending displays avg in function of the number of elements processed and not for the complete week + fixed bug in gauge and canvasjs (was performing avg with only 1 graph instead of all 8).
2016-08-16 16:33:02 +02:00
Mokaddem
cdd08d7df0
Added all providers for plot_tool in moduleStats + fixed date range bug
2016-08-16 11:07:36 +02:00
Mokaddem
1a5158b081
Added sentiment analyser module (draft)
2016-08-13 15:24:57 +02:00
Mokaddem
c028ca80da
Separated options from the plot funcion for sparkline.
2016-08-11 17:57:23 +02:00
95a22abe9c
Merge branch 'master' of github.com:CIRCL/AIL-framework
...
Conflicts:
bin/packages/modules.cfg
2016-08-11 08:06:57 +00:00
Mokaddem
cd74e6d1c5
Added module sentimentAnalyser
2016-08-11 09:40:42 +02:00
Mokaddem
64fe32b29b
Fixed small bug in SQLInjection module
2016-08-10 14:23:36 +02:00
Mokaddem
feafa8ffe2
Slight refacto + sql comments are now placed in injection suspision.
2016-08-10 09:06:37 +02:00
Mokaddem
943016d6ca
Fixed one of the small display bug in DUplicate_ssdeep_v2 (was displaying [tlsh, tlsh] instead of simply tlsh)
2016-08-09 15:06:55 +02:00
Mokaddem
a4595010c0
Added whole config in config.sample
2016-08-09 14:23:36 +02:00
Mokaddem
a80f6ca0c2
Bug fix related with redis:
...
Fixed typo key in redis for module creditcard and sqlinjection
Modified Curve redisLvlDb server
Modified Url.py so that it forwards name of protocol from saved protocolsfile
Added Cache control in Flask
Modified key-tab name into keys-tab
2016-08-09 11:59:36 +02:00
Mokaddem
60512a30a2
Fixeed bug in Duplicate (The comparison value was not saved correctly in redis) + Modified to progression detection algo
2016-08-08 15:28:26 +02:00
Mokaddem
aedb3198cd
Fixed bug in CVE (always sending to browse_warning even if there was no match)
2016-08-08 11:54:27 +02:00
Mokaddem
14062a549c
Fixed bug in credential (not sending correctly to ModuleStat) + Fixed date bug in ModuleStats
2016-08-08 11:37:18 +02:00
Mokaddem
4e56f76f62
Fixed bug CVE,
...
CVE was not sending its data to BrowseWarningPaste.
2016-08-08 10:05:21 +02:00
Mokaddem
1c40ad522f
Merge branch 'master' into production
2016-08-08 09:26:58 +02:00
Mokaddem
78c611fead
Added warning_paste module and created related webpages.
...
Fixed a Faup bug in credential (multiple instanciation) and added correc populate_set_out in concerned modules (creditcard, credential, ...).
Linked browse_warning_paste module and Flask function with redis (created new sets).
2016-08-08 09:17:44 +02:00
Mokaddem
3523d79a05
Started browse important paste feature. Still need to link with redis. + Fixed a bug in Flask (String to list parser)
2016-08-04 16:57:34 +02:00
Mokaddem
55ef917ef1
Added few comments
2016-08-04 12:10:56 +02:00
Mokaddem
d9316771cd
Draft: added new duplicate hash comparison - tlsh
2016-08-04 11:55:38 +02:00
Mokaddem
455911fc26
Better display printing
2016-08-02 16:21:52 +02:00
Mokaddem
3169dc27b0
Added SQLInjectionDetection module
2016-08-02 15:43:11 +02:00
Mokaddem
a1b2329a9b
Fixed bug in case of null elements arrising because of a new month
2016-08-01 10:32:24 +02:00
Mokaddem
a9637ec7cf
removed prints
2016-07-29 16:54:25 +02:00
Mokaddem
feb1717dc6
fixed size bug, which in some cases display the total size instead of the average + few cleanup + commented out log scale in moduleStats.js and added large number division in case of div by 0
2016-07-29 16:52:50 +02:00
Mokaddem
e8fae17cfd
Fixed small race condition bug in ModuleStat.py and Added highlight of bar belonging to the same serie in moduleStats.js
2016-07-29 10:36:44 +02:00
Mokaddem
240bdd9cca
Added number of paste by provider in module stats
2016-07-27 09:27:32 +02:00
Mokaddem
baa042827d
Removed useless comments + prevent trying to add an already present element in the set
2016-07-26 11:06:31 +02:00
Mokaddem
cc1b49baaf
Fixed variable bug in moduleStats and added draft of credential stats
2016-07-26 10:45:02 +02:00
Mokaddem
06be1f129a
Added Draft of ModuleStats + Paste size average
2016-07-25 16:38:57 +02:00
d013546643
Merge branch 'master' of github.com:CIRCL/AIL-framework
...
Conflicts:
bin/packages/modules.cfg
2016-07-25 07:56:46 +00:00
722634adf4
Prod config
2016-07-25 07:49:14 +00:00
Raphaël Vinot
4aa484a91f
Merge pull request #63 from mokaddem/uptodate-duplicate-module
...
Uptodate duplicate module
2016-07-25 09:44:49 +02:00
Mokaddem
34a00abdc2
Merge branch 'uptodate-duplicate-module' into production
2016-07-22 10:31:10 +02:00
Mokaddem
c686f69ca6
Clean not compatible ssdeep hash encountered
2016-07-22 10:04:58 +02:00
Raphaël Vinot
ea8703e608
Fix path to tldsfile
2016-07-22 09:32:13 +02:00
Mokaddem
dad05073fd
Fixed bug introduced in merge-conflict
2016-07-22 09:25:05 +02:00
Mokaddem
7c4f4a2f79
Merge branch 'statistics-v2' into uptodate-statistics-v2
2016-07-21 15:16:03 +02:00
Mokaddem
b685f81752
Merge branch 'duplicate-module' into uptodate-duplicate-module
2016-07-21 14:59:52 +02:00
Mokaddem
0ab7560477
Added top_progression chart for tld, domain and scheme + Small modification in config file.
2016-07-21 13:44:22 +02:00
Raphaël Vinot
ede46ec799
Fix default config.
2016-07-21 09:19:13 +02:00
Raphaël Vinot
781b441bb8
Fix path in config
2016-07-20 14:12:50 +02:00
Raphaël Vinot
620d69467f
Add WebStats in launcher
2016-07-20 14:12:37 +02:00
Raphaël Vinot
62eef44ca8
Fix pep8
2016-07-20 14:12:18 +02:00
Raphaël Vinot
8a69110c8e
Merge branch 'webpage-splitting' of https://github.com/mokaddem/AIL-framework into mokaddem-webpage-splitting
2016-07-19 16:35:31 +02:00
Mokaddem
a6996c0b23
Added related functions and display of duplicated paste in search.py
2016-07-19 10:48:44 +02:00
Mokaddem
996c0e02de
Duplicate module takes its messages from other modules and no more from Global.
2016-07-18 16:22:33 +02:00
Mokaddem
6805ed6488
Added default config
2016-07-18 15:52:53 +02:00
Mokaddem
4f6813350b
Added two new version of duplicate module.
...
One with hashes are saved in json on disk
The other with only leveldb
2016-07-18 15:50:41 +02:00
Mokaddem
14e9850dd6
Added new module for Duplicate paste. Seems working but has some small bug (re-check same paste twice)
2016-07-15 16:58:48 +02:00
Mokaddem
2383db022f
Added default configuration
2016-07-15 09:10:44 +02:00
Mokaddem
9a9e07f600
Added default configuration
2016-07-15 09:08:38 +02:00
Mokaddem
0332f23579
Added SimHash library
2016-07-15 08:56:16 +02:00
Mokaddem
fba14bfb4b
In index: Added number of processed pastes chart
2016-07-14 11:16:07 +02:00
Mokaddem
ab61e32399
Commented out get_language because it adds too much overhead
2016-07-14 11:15:15 +02:00
Mokaddem
594d2def35
In index: Added number of processed pastes chart
2016-07-13 15:57:33 +02:00
Mokaddem
56b6659d8b
Commented out get_language because it adds too much overhead
2016-07-13 08:59:48 +02:00
Mokaddem
c51bdec8aa
Merge branch 'mokaddem-testing'
2016-07-12 11:53:24 +02:00
Mokaddem
8a1247cf5d
modified variable name str
2016-07-12 11:52:19 +02:00
Mokaddem
7e5ce0f17f
Optimized create_plot and removed test commemts
2016-07-12 11:47:51 +02:00
Mokaddem
465244e1ce
Added dynamic table sorting in search page. (Still need to add dependencies)
2016-07-07 16:38:00 +02:00
Mokaddem
7ff9b9a583
Added DomainTrending seems working.
...
Started search features with related html pages, not finish yet.
2016-07-05 16:53:03 +02:00
Mokaddem
5a9eca9291
Added few comments
2016-07-04 09:18:23 +02:00
Mokaddem
4b3101b7b6
Added template tld. Modified URL using Faup and refactored WebStats.
2016-07-01 16:59:08 +02:00
Mokaddem
beeeb76de9
Added new modules and started WebTrending web interface
2016-06-30 14:38:28 +02:00
Mokaddem
3dc356dc5e
Getting Started: Initial configuration working on the laptop
2016-06-30 14:36:47 +02:00
MaximeStor
ab66cd255a
Improve SourceCode, keywords and add description in /doc
2016-03-12 12:30:38 +01:00
MaximeStor
701d771aa5
Add first version of Source Code
2016-03-12 11:21:29 +01:00
Raphaël Vinot
be86737ca7
Because 0MQ fails.
2016-03-11 16:16:53 +01:00
Raphaël Vinot
f6e4ea2270
Fix logging, fix URL regex
2016-02-11 12:19:03 +01:00
Raphaël Vinot
d160e4a2c8
Add Credential in the scripts to launch
2016-02-10 17:33:16 +01:00
Raphaël Vinot
90e1b25426
Split filepath and count in credential module
2016-02-10 17:31:52 +01:00
Raphaël Vinot
4895ee9fa2
Add new category (Credential)
2016-02-10 16:39:56 +01:00
Raphaël Vinot
1da8675750
Refactoring on Credential, Phone and Release
2016-02-10 16:39:06 +01:00
c68136b04b
Merge branch 'pgp' of https://github.com/Rafiot/AIL-framework
...
Conflicts:
bin/packages/modules.cfg
2016-02-08 10:13:44 +01:00
MaximeStor
b7d2b64a86
Merge branch 'master' of https://github.com/CIRCL/AIL-framework into module
2016-02-06 11:28:48 +01:00
192ee7f0ec
Merge pull request #49 from Alainfou/master
...
Phone module added (regex needs optimization)
2016-02-06 11:20:46 +01:00
Alain
ea52fd1068
Phone regex updated
...
Might still need to be fixed / optimized, in case of maths or random numbers starting with a 0. Do not capture dates, hours, coordinates anymore. Captured formats are: e.g. +331234567890 ; 09 12 34 56 78 ; +4177/123.45.69 ; +352(0)6-23-23-23...
2016-02-05 20:58:02 +01:00
Alain
fabbfd8ae9
Update module.cfg (adding Keys and Phone section)
2016-02-05 14:00:41 -05:00
Alain
43b3556588
Starting Phone number recognition
2016-02-05 13:58:21 -05:00
MaximeStor
07513a5b37
Add modules Credential and Release
2016-02-05 16:15:09 +01:00
Raphaël Vinot
9171d5b118
Add module to find PGP encrypted blobs
2016-02-05 16:03:37 +01:00
Raphaël Vinot
aef8ab0411
Listen locally for 0MQ
2016-02-04 15:32:50 +01:00
Raphaël Vinot
5ca13c42eb
Launch redis and leveldb from local directory
2016-02-04 15:24:39 +01:00
Raphaël Vinot
12aca6b760
Add script to import from local directory, use local python from env
2016-02-04 15:22:51 +01:00
Raphaël Vinot
315cb48117
Add template file for writing a new module
2016-02-03 10:33:42 +01:00
Raphaël Vinot
0d6adc2063
Add initial Travis file
2016-01-19 11:43:34 +01:00
cdd0725e88
-v option added to list the path
2015-12-22 21:37:05 +00:00
e3971ac93a
Onion fetching loop deactivated by default
2014-12-22 16:06:38 +00:00
Raphaël Vinot
08ceefc375
Re-add config option
2014-12-22 16:50:25 +01:00
Raphaël Vinot
50369c6706
Revert changes on the config file due to merging messup
2014-12-22 16:29:05 +01:00
Raphaël Vinot
f717f9fe89
Merge branch 'master' of github.com:CIRCL/AIL-framework
2014-12-22 15:32:48 +01:00
Raphaël Vinot
9ee61db2cf
Add hotfixes
2014-12-22 15:27:02 +01:00
Raphaël Vinot
8803c8447a
Publish the fetched onions on a ZMQ feed.
2014-09-30 16:55:16 +02:00
25757b0fff
A simple feeder script feeding data from pystemon to AIL.
...
The configuration matches the default Redis parameters used
in the pystemon configuration.
https://github.com/cvandeplas/pystemon/blob/master/pystemon.yaml#L16
2014-09-19 14:03:05 +02:00
Raphaël Vinot
65b9a01644
Add config file for DomainClassifier, proper reporting
2014-09-17 17:22:56 +02:00
27b134ec03
Add proper publisher for classified domains/hostnames
2014-09-10 09:27:47 +02:00
Raphaël Vinot
f017680365
fix onions, cc and domain classifier modules
2014-09-08 16:51:43 +02:00
de6e21d5a7
DomainClassifier sample configuration added
2014-09-08 16:44:05 +02:00
246621f663
First version of the DomainClassifier
2014-09-08 16:43:21 +02:00
1397db9691
Global queue for DomainClassifier
2014-09-08 11:07:45 +02:00
Raphaël Vinot
e983c839ad
Categ now listen to the Global queue
2014-09-05 17:05:45 +02:00
Raphaël Vinot
46f27ada4e
More cleanup
2014-09-05 10:42:01 +02:00
Raphaël Vinot
fca00beed9
Add Domain Classifier module.
...
Cleanup in the config files.
2014-09-05 10:41:00 +02:00
Raphaël Vinot
b7c9e489c9
Fix the exceptions
2014-09-04 11:46:07 +02:00
Raphaël Vinot
9e8611a42d
stop killing the disk when creating the word curve
2014-09-02 18:20:28 +02:00
Raphaël Vinot
7542eaf739
Update starting script.
2014-09-02 15:21:36 +02:00
Raphaël Vinot
0c6b09f379
Fix the onion module, log the valid onions.
2014-09-01 16:18:06 +02:00
Raphaël Vinot
f4b89669fc
The onion module now fetches the URLs it finds.
2014-08-31 22:42:12 +02:00
Raphaël Vinot
abfe13436b
Big refactoring, make the queues more flexible
2014-08-29 19:37:56 +02:00
Raphaël Vinot
623e876f3b
Cleanup.
...
* Remove useless subscriber
* Fix typo in the config file
* Update Helper accordingly
2014-08-26 17:36:57 +02:00
3b499a2ec8
ZMQ Publisher removed
...
ZMQ Publisher removed to allow concurrent use of the scripts.
In short term, we would replace all publishing part within AIL
into pub-sub Redis to avoid ZMQ limitation.
2014-08-26 14:38:49 +02:00
f070ac2005
cymruwhois uses dotted decimal format
2014-08-25 10:05:36 +02:00
Raphaël Vinot
3886d1b834
Small fixes to make the refactoring production ready
...
* the port for the logging is 6380
* use os.environ properly
* fix typos
2014-08-22 17:35:40 +02:00
Raphaël Vinot
78125db4ea
Use env variables everywhere
2014-08-22 14:52:02 +02:00
Raphaël Vinot
277d138a5d
cleanup, add FIXME
2014-08-21 14:39:17 +02:00
Raphaël Vinot
63b29176c1
move Redis_Data_Merging to Paste
2014-08-21 12:22:07 +02:00
Raphaël Vinot
50cfac857e
Update config
...
Make all paths in the config file relative to the home directory.
2014-08-20 16:00:56 +02:00
Raphaël Vinot
a68f5b6a0e
fix subscriber names, update default config
2014-08-20 15:54:21 +02:00
Raphaël Vinot
2485ba5df2
Merge remote-tracking branch 'origin/master' into testing
...
Conflicts:
bin/ZMQ_Sub_Urls.py
2014-08-20 15:24:10 +02:00
Raphaël Vinot
99c8cc7941
completely remove ZMQ_PubSub.py
2014-08-20 15:14:57 +02:00
1d64dc44c8
MIME type guessing - removed one duplicate call to libmagic
2014-08-20 10:22:33 +02:00
Raphaël Vinot
8d9ffbaa53
Do not create a ZMQ sub if it is not required.
2014-08-19 19:53:33 +02:00
Raphaël Vinot
45b0bf3983
Improve the cleanup. Still some to do.
2014-08-19 19:07:07 +02:00
Raphaël Vinot
f1753d67c6
Cleanup the queues.
2014-08-19 16:05:37 +02:00
e8fcea6cd6
Remove undeclared variable
2014-08-18 16:17:36 +02:00
7d8ee102a3
Assignment before use (if Enumerate fails)
2014-08-18 15:58:06 +02:00
4304c6858e
Configuration path fixed
2014-08-18 09:02:08 +02:00
Raphaël Vinot
078c8ea836
Big cleanup, pep8
2014-08-14 18:07:18 +02:00
Jules
ab6765315e
Merge pull request #13 from adulau/master
...
Log where URLs are hosted - cc_critical option added
2014-08-14 14:28:01 +02:00
762def3a23
Log where URLs are hosted - cc_critical option added
...
It logs where the hostname of the URL is hosted (ASN and geographic location).
A simple option cc_critical added to set the country code to log as critical.
2014-08-14 14:22:11 +02:00
Raphaël Vinot
4a1f300a1a
Cleanup (remove unused imports, more pep8 compatible)
2014-08-14 14:11:07 +02:00
Starow
04a8f1bdf2
maxi cleanup old code :'(
2014-08-14 11:48:46 +02:00
Starow
29b24b6466
printing set of domain for debugging
2014-08-13 16:35:27 +02:00
Raphaël Vinot
ece3bc173e
Cleanup of main Paste module
2014-08-13 11:56:22 +02:00
Raphaël Vinot
5b17d416c8
remove script installed by pubsublogger
2014-08-13 11:55:59 +02:00
Raphaël Vinot
935e51c961
Remove 3rd party code (pubsublogger), add it in the deps.
2014-08-13 10:19:43 +02:00
Starow
37033ca3a6
Minor logs modifications
2014-08-13 10:08:44 +02:00
Starow
6aa4d7cb7d
Harmonising logs messages + Changing some dygraph options
2014-08-12 15:42:16 +02:00
0b4a80b7ea
-s option added to find similar documents
...
By default, the index is not storing the vector of the document (Whoosh
document schema). It won't work if you don't change the schema of the
index for the content. It depends of your storage strategy.
2014-08-12 13:42:26 +02:00
fd6e1a8436
-f option added: dump full document for each match
2014-08-12 13:26:56 +02:00
0a6664ffba
Indexer: Some index statistics added
...
usage: indexer_lookup.py [-h] [-q Q] [-n] [-t] [-l]
Fulltext search for AIL
optional arguments:
-h, --help show this help message and exit
-q Q query to lookup (one or more)
-n return number of indexed documents
-t dump top 500 terms
-l dump all terms encountered in indexed documents
2014-08-11 15:07:12 +02:00
f65a94d47b
-l added -> dumping all terms indexed
2014-08-11 14:56:15 +02:00
f3d1ca052e
Return the number of indexed documents
2014-08-11 14:50:35 +02:00
611d2a466f
Configuration that should not be there...
2014-08-11 14:24:27 +02:00
2b8f2689bf
Indexer queue and script added to "BBS-like" LAUNCH script
2014-08-11 14:06:52 +02:00
9657c6bf80
Merge branch 'master' of https://github.com/CIRCL/AIL-framework
2014-08-11 13:46:37 +02:00
b1053af3cd
Indexer module: script to query the index
...
Test script to query the index generated from the Indexer module.
python indexer_lookup.py -q Visa -q Mastercard
2014-08-11 12:03:27 +02:00
Starow
079db6f80c
Hardcoded path from ZMQ_Curve are now referring correctly in config.cfg.sample fix #6
2014-08-11 11:33:18 +02:00
7bdd4a41a5
Indexer module added - initial version with Whoosh full-text indexer
...
The indexer module indexes all the pastes using Whoosh. The module
can be extended to support additional full-text indexers in the future.
2014-08-11 11:04:09 +02:00
Starow
d1d4b2ebe0
Importing dns.exeption fix #4 fix #7
2014-08-11 09:27:50 +02:00
Starow
192074e569
Merge branch 'master' of https://github.com/CIRCL/AIL-framework
2014-08-11 09:21:09 +02:00
Starow
a5c1d59d29
Catching the exception dns.exception.Timeout fix #7
2014-08-11 09:18:55 +02:00
Starow
54091a2174
Catching the exception dns.exception.Timeout fix #4
2014-08-11 09:08:28 +02:00
Starow
eb603e8762
Fixing a bug about caching paste inside Redis :)
2014-08-08 17:23:51 +02:00
Starow
7a1db94f9e
Adding a letter (s)
2014-08-08 17:19:42 +02:00
Starow
043800287a
adding a .
2014-08-08 17:18:03 +02:00
Starow
bf682c4b44
Fixing last commit ...
2014-08-08 17:13:18 +02:00
Starow
503c23ca3b
Fixing last commit
2014-08-08 17:08:41 +02:00
Starow
c9e1eaf182
Improving cache code
2014-08-08 17:04:25 +02:00
Starow
44addf1afe
Redis cache added fix #5
...
The paste will be add in Redis during 5min and also saved on disk.
Now if a module want to get the paste for further processing, it will first try to get it in the cache
instead of getting it directly on the disk and wasting I/O.
2014-08-08 16:48:02 +02:00
Starow
97f3a3df9e
update pubsublogger with the last version
2014-08-07 14:49:34 +02:00
Starow
c10003a630
Changing ZMQ Curve Module comment
2014-08-07 14:46:43 +02:00
Starow
1379ef705a
Initial import of AIL framework - Analysis Information Leak framework
...
AIL is a modular framework to analyse potential information leak from unstructured data source like pastes from Past
ebin or similar services. AIL framework is flexible and can be extended to support other functionalities to mine sen
sitive information
2014-08-06 11:43:40 +02:00
