AIL/ail-framework
7
0
Fork 0
mirror of https://github.com/ail-project/ail-framework.git synced 2024-11-30 09:47:17 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
1476 commits 8 branches 43 tags 99 MiB
Commit graph

246 commits

Author SHA1 Message Date
Terrtia
b4bee29a33
Merge branch 'master' into advanced_crawler 2019-04-10 15:37:42 +02:00
Terrtia
5165a5de2f
fix: [Crawler] fix index + redis history key 2019-03-22 17:14:27 +01:00
Terrtia
f64c385343
chg: [Crawler] handle port: crawling + history 2019-03-22 16:48:07 +01:00
Terrtia
c2bc72c72e
chg: [Tags alertHandler] remove alertHandler + add quick tag search sidebar + use tag last_seen to display tagged items 2019-03-21 11:11:53 +01:00
Terrtia
65e6eae00d
chg: [UI preview items] b4 migration + new template + better display + decrease item loading time + add basic animation/transition 2019-03-19 13:34:41 +01:00
Jason Hedden
23fb91d326 Add docker-compose support 
This commit separates the AIL Framework services into individual Docker
containers that can be managed with docker-compose. It's intended to ease the
installation, development and troubleshooting procedures for all-in-one
deployments.

No changes to architecture or base code are required in this commit.

Changes to existing files:
.gitignore
  * Ignore pystemon archive directory

Dockerfile:
  * Add AIL shell environment variables
  * Install pystemon and crawler pip requirements into AIL virtual environment

Docker-compose notes:
  * All containers are participating in a service network `network_mode:
  service:flask`.  This allows the containers to share the same IP namespace
  to accommodate hard-coded localhost entries.
  * By default persistent data is saved to the following directories in the
  local AIL framework git directory: PASTES, HASHS, CRAWLED_SCREENSHOTS,
  pystemon/archives

This was tested with docker-ce on Ubuntu 16.04 and MacOS. A typical deployment
would look like:
```
git clone https://github.com/CIRCL/AIL-framework.git
cd AIL-framework
cp bin/packages/config.cfg.docker-compose-sample bin/packages/config.cfg
	(optionally enable activate_crawler in config.cfg)
docker-compose build
docker-compose up -d
```
 2019-03-11 19:46:34 -05:00
Terrtia
d3b3942ad7
chg: [Flask UI] refractor crawler frontend + show crawled domain 2019-03-06 15:05:58 +01:00
Terrtia
7b32d7f34e
chg: [Crawler] major refractor 2019-02-25 16:38:50 +01:00
Terrtia
da78d0552d
chg: [Crawler UI Tags] add tag by day + add crawler status + UI onion blacklist 2019-02-19 11:41:45 +01:00
Terrtia
516238025f
chg: [Crawler] add bootsrap4 src + refractor crawler 2019-02-05 17:16:44 +01:00
Terrtia
535f44044d
fix: [Onion] add timeout signal 2019-01-29 09:46:03 +01:00
Terrtia
6ee4b7bf93
Merge branch 'master' into update 2018-12-19 15:23:34 +01:00
Terrtia
4e08aaa80f
chg: [UI Crawled Domains] show crawled domains by date range (options: domains tags, up domains, down domains) 
todo: Shows domain history
 2018-12-17 15:58:48 +01:00
Terrtia
31a8dfe0b3
fix: [AIL] use only relative paths 
pastes duplicates are fixed on the fly
 2018-11-21 16:45:25 +01:00
Terrtia
cb41c54135
fix: [lib_refine] add dns configuration 2018-11-21 08:49:58 +01:00
Terrtia
d9b776d58a
merge master 2018-11-14 15:55:42 +01:00
Terrtia
b31618eb6a
chg: [Mixer] config default feeder name #291 2018-11-09 15:26:26 +01:00
Terrtia
979472df15
chg: [Terms] tag tracked terms 2018-11-06 13:38:37 +01:00
Terrtia
aa16c52cbf
fix: [NotificationHelper] add config 2018-11-05 14:20:12 +01:00
Terrtia
5872cf9196
fix: [Scripts] Remove absolute path 2018-11-02 16:07:27 +01:00
Terrtia
ea7fd8d54c
fix: [Duplicate] show duplicate with relative path 2018-10-24 15:47:03 +02:00
Terrtia
5afbc3179c
Merge branch 'master' of https://github.com/CIRCL/AIL-framework 2018-10-24 15:16:41 +02:00
Terrtia
8a6ac6be1e
fix: [Paste] create absolute path 2018-10-24 15:16:12 +02:00
Sami Mokaddem
d9bc2d6f44
Merge pull request #266 from kovacsbalu/fix-regexp-timeout 
Stop regexp processing after timeout (60sec)
 2018-10-09 14:48:08 +02:00
kovacsbalu
272b0fe960 RegexpForTerm timeout from config 
Critical log removed
 2018-10-09 09:32:32 +02:00
Terrtia
44d19fbb80
fix: [pystemon-feeder] paste save path 2018-10-02 16:02:59 +02:00
Alexandre Dulaunoy
59df5f0e12
chg: [config] Tor crawler disabled by default in sample configuration 
(to match HOWTO when enabling it)
 2018-10-02 14:17:58 +02:00
Terrtia
04b9d9fc1d
chg: [Crawler] add docs 2018-09-27 11:14:29 +02:00
Terrtia
e4a5f66a10
Merge branch 'master' into onion_crawler 2018-09-24 16:35:04 +02:00
Terrtia
50c81773e9
chg: [Crawler] add launcher and install 2018-09-24 16:23:14 +02:00
Terrtia
ead4dba0d8
fix: [MISP feeder] typo filename-config name 2018-09-21 11:02:08 +02:00
Terrtia
6edc1ddbeb
chg: [HiddenServices] get domain related to other domains 2018-09-21 10:34:06 +02:00
Terrtia
5eadc2755f
chg: [Flask] add prefix in config to flask routes 2018-09-20 10:38:19 +02:00
Terrtia
ea3d2c1977
Merge branch 'master' into statistic 2018-09-12 10:06:53 +02:00
Terrtia
6f0817365a
chg: [Crawler UI] display domain information 2018-09-12 09:55:49 +02:00
Terrtia
ca982e13e1
chg: [Crawled Domain] show crawled pastes by domain 2018-08-27 14:34:08 +02:00
Terrtia
823c33174a
Merge branch 'master' into onion_crawler 2018-08-27 11:20:51 +02:00
Terrtia
5c902640e9
Merge branch 'master' into onion_crawler 2018-08-27 11:05:25 +02:00
Terrtia
d42dd118a4
chg: [Domain crawled] add random screenshot 2018-08-27 11:02:39 +02:00
Terrtia
a586548047
Merge branch 'master' into base64 2018-08-24 10:35:28 +02:00
Terrtia
ced0b1e350
chg: [I2P] add default config 2018-08-24 10:24:03 +02:00
Terrtia
e9580d6775
chg: [Crawler] change BDD, save i2p links 2018-08-21 15:54:53 +02:00
Terrtia
ed559d9f4a
chg: [Showpaste] add screenshot + improve onion db 2018-08-16 17:24:39 +02:00
Terrtia
8b1c10b38c
chg: [Onion] add onion splash crawler 2018-08-09 17:42:21 +02:00
Thirion Aurélien
000d5bf40a
fix: add miss config 2018-08-07 13:07:08 +02:00
Terrtia
940b96734a
Merge branch 'master' into statistic 2018-08-01 09:29:51 +02:00
Terrtia
39f54d4420
chg: [Mail] add country statistic 2018-07-30 09:21:22 +02:00
Terrtia
57b631233b
add: [BankAccount] iban detection 2018-07-26 15:31:58 +02:00
Terrtia
8beb63b6a8
fix: [Queues] module with no subscriber 2018-07-24 15:57:18 +02:00
Terrtia
8c97c70a3f
Merge branch 'master' into base64 2018-07-24 15:21:32 +02:00
Terrtia
bd5f83f0eb
chg: refractor base64 encoded to hash 2018-07-20 10:32:52 +02:00
Terrtia
9a8e37fb0b
fix: decoder bdd 2018-07-20 09:43:09 +02:00
Terrtia
ad26f016e3
add: Decoder Module, decode binary, hex and base64 2018-07-19 16:50:42 +02:00
Terrtia
e25dd52d64
add: Binary decoder 2018-07-18 11:45:19 +02:00
Terrtia
0cac3146f1
fix: remove hardcoded mail url 2018-07-17 15:11:25 +02:00
bui
bd5c42caeb add LibInjection Module 2018-07-04 14:28:10 +02:00
Terrtia
16c700ddb4
new: add base64 display on paste + base64 statistics 2018-06-29 10:02:29 +02:00
Terrtia
af9db53171
add tag fp and tp 2018-06-19 15:09:26 +02:00
Terrtia
31fc33ab03
misp event creation, auto and manual + the hive alert auto creation, manual case creation 2018-06-14 16:51:06 +02:00
Terrtia
6143bc3dce
submit users input paste 2018-06-05 16:58:04 +02:00
Terrtia
93dcdf4f23
fix duplicate 2018-05-31 09:01:47 +02:00
Terrtia
0d164b6f4c
add tag js and css 2018-05-16 14:48:15 +02:00
Terrtia
f66a528bc2
fix Duplicate, save list of duplicates on disk + prevent empty hash creation 2018-05-15 23:28:47 +02:00
Terrtia
225fe76c96
fix duplicate + get raw paste 2018-05-11 16:07:30 +02:00
Terrtia
d7ba5533be
fix duplicate + fix issue #200 2018-05-11 13:25:45 +02:00
Terrtia
8599d7142d
python 3 upgrade instructions 2018-05-09 13:32:25 +02:00
Terrtia
142c1dc9b0
clean 2018-05-09 13:03:46 +02:00
Terrtia
3841c159a0
change leveldb to ardb 2018-05-07 14:50:40 +02:00
Terrtia
96a3df42bc
decode with redis connection 2018-05-04 13:53:29 +02:00
Terrtia
a900d5f08d
add python 2 to 3 bash + fix Duplicate + clean 2018-05-03 16:21:33 +02:00
Terrtia
c52caebe7c
improve Duplicate + tlsh + add time out handler + debug + clean 2018-05-02 17:07:10 +02:00
Terrtia
c3428df5dc add apiKeys module 2018-04-26 14:42:39 +02:00
Terrtia
8738b7cf75 fix track term and avoid duplicate mail address 2018-04-24 16:44:37 +02:00
Thirion Aurélien
ef3c335b62
Delete config.cfg.backup 2018-04-23 14:54:23 +02:00
Terrtia
f93fe9aeb2 add bitcoin module 2018-04-23 14:51:53 +02:00
Terrtia
3fc48db903 add Base64 module 2018-04-20 10:48:44 +02:00
Terrtia
8571a86344 frontend python 3.5 upgrade 2018-04-17 16:06:32 +02:00
Terrtia
9c82dd90ec python 3 backend upgrade 2018-04-16 14:50:04 +02:00
CD Nguyen
e9e3f17363 support non TSL SMTP servers 2018-03-30 11:35:37 +02:00
Philipp Schmied
fd329aaba9 config.cfg.sample: Merged overwritten changes from upstream 2018-02-27 16:16:57 +01:00
Philipp Schmied
5b1f0b0212 Implemented email notifications (bin: config.cfg additions, email sending via analyzer scripts; var: Changes to add notifications via terms management); terms_management: Fixed click handlers not being added to all tracked terms. 2018-02-27 15:12:02 +01:00
Sami Mokaddem
c39e537cbe Merge branch 'iss160' into showDiff 
Preemptively preventing merge-conflict with branch iss160 (moving to
configuration)
 2017-12-12 11:33:22 +01:00
Sami Mokaddem
304213cfa4 update: Possibility to change the authorized max line length in the config for the diff of duplicates 2017-12-12 11:00:26 +01:00
Sami Mokaddem
860acbf7cf fix: Renamed Credentials into Credential sectiont title 
config change: By default, zmq listen to both local and CRF
 2017-12-12 09:29:15 +01:00
Sami Mokaddem
9d0d0b4303 update: Moved filtering operation (thresholds, number of matching in the 
categ file) in the configuration file. It permits to better control the
flow of pastes.
Also set default mixer duplicate filtering to 3 (Do not filter)
 2017-12-11 17:28:34 +01:00
Sami Mokaddem
2a967c4d92 update/feature: Max number of duplicate push to MISP + duplicate are pushed as attachment 2017-11-23 07:13:44 +01:00
Sami Mokaddem
6f0227a78a Starting support of python3 2017-11-20 12:12:06 +01:00
Sami Mokaddem
652b00f4b2 Renamed BrowseWarningPaste into alertHandler 2017-11-15 16:15:43 +01:00
=
64ec44085d Dynamically choose correct port for LevelDB in script 2017-08-22 17:52:15 +02:00
Mokaddem
882ab88b00 Draft module + web link 2017-07-18 16:57:15 +02:00
Mokaddem
b60e164380 Added support of onion in browseWarningPaste 2017-04-26 11:47:02 +02:00
Mokaddem
da081bae42 Merge branch 'regexInTerms' into production 2017-04-18 16:59:40 +02:00
Mokaddem
c8baabd882 Updated TermsFrequency related modules + Fixed bugs (encoding, behaviors, ...) 2017-04-18 15:28:21 +02:00
Mokaddem
2da4c572c7 New feature in Terms: regex and set of words support (draft) 2017-03-28 17:42:44 +02:00
Mokaddem
e225090e07 Added script that checks configuration and may auto-fix it + added missing field in config.cfg.sample 2017-03-15 16:36:51 +01:00
Mokaddem
adc37ab50e Indexer max size now in config 2017-03-15 12:14:41 +01:00
Olivier MEDOC
16044d4d36 pystemon-feeder: add the option bind in configuration file instead of address 
This is essentially because multiple feeds can be used in the address
configuration variable.
 2017-01-13 14:58:18 +01:00
Olivier MEDOC
9f9c265cb0 SentimentAnalysis: use lexicon file path from the ail configuration file 2017-01-11 11:07:08 +01:00
ptitdoc
c0fac82031 Add pystemon path in configuration sample 2017-01-10 18:33:46 +01:00
Mokaddem
0e39f516a9 Added sleep in helper for multiple listening interfaces (reduce useless work), Added a simple way to pre-process feed before sending it to the global feed. 2017-01-09 14:12:26 +01:00