AIL/ail-framework
7
0
Fork 0
mirror of https://github.com/ail-project/ail-framework.git synced 2024-12-02 10:47:18 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
1528 commits 8 branches 43 tags 99 MiB
Commit graph

262 commits

Author SHA1 Message Date
Terrtia
e1bf59aded
chg: [UI crawler, show_domain] add domain history list + navigation 2019-06-18 10:58:37 +02:00
Terrtia
b4f06c21f9
chg: [backend crawler] domains: download 1 archive by crawled (most recent) 2019-06-07 13:47:44 +02:00
Terrtia
3fe9d14e9f
chg: [user_management] create default admin user (temp passwd save in AIL_HOME) + change password UI + logout UI + create random password 2019-06-06 21:27:13 +02:00
Terrtia
99e35c51ec
Merge branch 'master' into user_management 2019-06-05 16:59:36 +02:00
Terrtia
4b0a7210b8
Merge branch 'master' of https://github.com/CIRCL/AIL-framework 2019-05-24 13:27:44 +02:00
kovacsbalu
f61d830678 Use default encoding error from redis. 2019-05-16 14:24:03 +02:00
Terrtia
6480744641
chg: [PgpDump] add PgpDump backend 
TODO: UI
 2019-05-14 17:49:31 +02:00
kovacsbalu
05079c143c Fix #314 
Replace char on redis encoding error.
Try to use local file on other error.
 2019-05-14 16:06:20 +02:00
Terrtia
64ff94ce5f
chg: [user_management] create + check user password 2019-05-03 16:52:05 +02:00
Terrtia
1ab1a55a4f
chg: [UI] add basic user management 2019-05-02 17:31:14 +02:00
Terrtia
a183695ddb
chg: [crawler UI] bootstrap 4 migration: show domains by daterange + bug fix hidden services tags 2019-04-26 15:14:29 +02:00
Terrtia
9868833c77
chg: [crawled screenshot] use sha256 as filepath 2019-04-24 14:09:04 +02:00
Terrtia
68077316eb
chg: [git directory] add git helper 2019-04-16 17:26:35 +02:00
Terrtia
3cc614a1ad
chg: [DB] update items tags metadata 2019-04-11 11:58:06 +02:00
Terrtia
d44acea04d
chg: [merge] merge update into advanced_crawler 2019-04-10 16:41:06 +02:00
Terrtia
2589fc2161
chg: [merge] merge master into update branche 2019-04-10 15:43:15 +02:00
Terrtia
b4bee29a33
Merge branch 'master' into advanced_crawler 2019-04-10 15:37:42 +02:00
Terrtia
5165a5de2f
fix: [Crawler] fix index + redis history key 2019-03-22 17:14:27 +01:00
Terrtia
f64c385343
chg: [Crawler] handle port: crawling + history 2019-03-22 16:48:07 +01:00
Terrtia
c2bc72c72e
chg: [Tags alertHandler] remove alertHandler + add quick tag search sidebar + use tag last_seen to display tagged items 2019-03-21 11:11:53 +01:00
Terrtia
65e6eae00d
chg: [UI preview items] b4 migration + new template + better display + decrease item loading time + add basic animation/transition 2019-03-19 13:34:41 +01:00
Jason Hedden
23fb91d326 Add docker-compose support 
This commit separates the AIL Framework services into individual Docker
containers that can be managed with docker-compose. It's intended to ease the
installation, development and troubleshooting procedures for all-in-one
deployments.

No changes to architecture or base code are required in this commit.

Changes to existing files:
.gitignore
  * Ignore pystemon archive directory

Dockerfile:
  * Add AIL shell environment variables
  * Install pystemon and crawler pip requirements into AIL virtual environment

Docker-compose notes:
  * All containers are participating in a service network `network_mode:
  service:flask`.  This allows the containers to share the same IP namespace
  to accommodate hard-coded localhost entries.
  * By default persistent data is saved to the following directories in the
  local AIL framework git directory: PASTES, HASHS, CRAWLED_SCREENSHOTS,
  pystemon/archives

This was tested with docker-ce on Ubuntu 16.04 and MacOS. A typical deployment
would look like:
```
git clone https://github.com/CIRCL/AIL-framework.git
cd AIL-framework
cp bin/packages/config.cfg.docker-compose-sample bin/packages/config.cfg
	(optionally enable activate_crawler in config.cfg)
docker-compose build
docker-compose up -d
```
 2019-03-11 19:46:34 -05:00
Terrtia
d3b3942ad7
chg: [Flask UI] refractor crawler frontend + show crawled domain 2019-03-06 15:05:58 +01:00
Terrtia
7b32d7f34e
chg: [Crawler] major refractor 2019-02-25 16:38:50 +01:00
Terrtia
da78d0552d
chg: [Crawler UI Tags] add tag by day + add crawler status + UI onion blacklist 2019-02-19 11:41:45 +01:00
Terrtia
516238025f
chg: [Crawler] add bootsrap4 src + refractor crawler 2019-02-05 17:16:44 +01:00
Terrtia
535f44044d
fix: [Onion] add timeout signal 2019-01-29 09:46:03 +01:00
Terrtia
6ee4b7bf93
Merge branch 'master' into update 2018-12-19 15:23:34 +01:00
Terrtia
4e08aaa80f
chg: [UI Crawled Domains] show crawled domains by date range (options: domains tags, up domains, down domains) 
todo: Shows domain history
 2018-12-17 15:58:48 +01:00
Terrtia
31a8dfe0b3
fix: [AIL] use only relative paths 
pastes duplicates are fixed on the fly
 2018-11-21 16:45:25 +01:00
Terrtia
cb41c54135
fix: [lib_refine] add dns configuration 2018-11-21 08:49:58 +01:00
Terrtia
d9b776d58a
merge master 2018-11-14 15:55:42 +01:00
Terrtia
b31618eb6a
chg: [Mixer] config default feeder name #291 2018-11-09 15:26:26 +01:00
Terrtia
979472df15
chg: [Terms] tag tracked terms 2018-11-06 13:38:37 +01:00
Terrtia
aa16c52cbf
fix: [NotificationHelper] add config 2018-11-05 14:20:12 +01:00
Terrtia
5872cf9196
fix: [Scripts] Remove absolute path 2018-11-02 16:07:27 +01:00
Terrtia
ea7fd8d54c
fix: [Duplicate] show duplicate with relative path 2018-10-24 15:47:03 +02:00
Terrtia
5afbc3179c
Merge branch 'master' of https://github.com/CIRCL/AIL-framework 2018-10-24 15:16:41 +02:00
Terrtia
8a6ac6be1e
fix: [Paste] create absolute path 2018-10-24 15:16:12 +02:00
Sami Mokaddem
d9bc2d6f44
Merge pull request #266 from kovacsbalu/fix-regexp-timeout 
Stop regexp processing after timeout (60sec)
 2018-10-09 14:48:08 +02:00
kovacsbalu
272b0fe960 RegexpForTerm timeout from config 
Critical log removed
 2018-10-09 09:32:32 +02:00
Terrtia
44d19fbb80
fix: [pystemon-feeder] paste save path 2018-10-02 16:02:59 +02:00
Alexandre Dulaunoy
59df5f0e12
chg: [config] Tor crawler disabled by default in sample configuration 
(to match HOWTO when enabling it)
 2018-10-02 14:17:58 +02:00
Terrtia
04b9d9fc1d
chg: [Crawler] add docs 2018-09-27 11:14:29 +02:00
Terrtia
e4a5f66a10
Merge branch 'master' into onion_crawler 2018-09-24 16:35:04 +02:00
Terrtia
50c81773e9
chg: [Crawler] add launcher and install 2018-09-24 16:23:14 +02:00
Terrtia
ead4dba0d8
fix: [MISP feeder] typo filename-config name 2018-09-21 11:02:08 +02:00
Terrtia
6edc1ddbeb
chg: [HiddenServices] get domain related to other domains 2018-09-21 10:34:06 +02:00
Terrtia
5eadc2755f
chg: [Flask] add prefix in config to flask routes 2018-09-20 10:38:19 +02:00
Terrtia
ea3d2c1977
Merge branch 'master' into statistic 2018-09-12 10:06:53 +02:00
Terrtia
6f0817365a
chg: [Crawler UI] display domain information 2018-09-12 09:55:49 +02:00
Terrtia
ca982e13e1
chg: [Crawled Domain] show crawled pastes by domain 2018-08-27 14:34:08 +02:00
Terrtia
823c33174a
Merge branch 'master' into onion_crawler 2018-08-27 11:20:51 +02:00
Terrtia
5c902640e9
Merge branch 'master' into onion_crawler 2018-08-27 11:05:25 +02:00
Terrtia
d42dd118a4
chg: [Domain crawled] add random screenshot 2018-08-27 11:02:39 +02:00
Terrtia
a586548047
Merge branch 'master' into base64 2018-08-24 10:35:28 +02:00
Terrtia
ced0b1e350
chg: [I2P] add default config 2018-08-24 10:24:03 +02:00
Terrtia
e9580d6775
chg: [Crawler] change BDD, save i2p links 2018-08-21 15:54:53 +02:00
Terrtia
ed559d9f4a
chg: [Showpaste] add screenshot + improve onion db 2018-08-16 17:24:39 +02:00
Terrtia
8b1c10b38c
chg: [Onion] add onion splash crawler 2018-08-09 17:42:21 +02:00
Thirion Aurélien
000d5bf40a
fix: add miss config 2018-08-07 13:07:08 +02:00
Terrtia
940b96734a
Merge branch 'master' into statistic 2018-08-01 09:29:51 +02:00
Terrtia
39f54d4420
chg: [Mail] add country statistic 2018-07-30 09:21:22 +02:00
Terrtia
57b631233b
add: [BankAccount] iban detection 2018-07-26 15:31:58 +02:00
Terrtia
8beb63b6a8
fix: [Queues] module with no subscriber 2018-07-24 15:57:18 +02:00
Terrtia
8c97c70a3f
Merge branch 'master' into base64 2018-07-24 15:21:32 +02:00
Terrtia
bd5f83f0eb
chg: refractor base64 encoded to hash 2018-07-20 10:32:52 +02:00
Terrtia
9a8e37fb0b
fix: decoder bdd 2018-07-20 09:43:09 +02:00
Terrtia
ad26f016e3
add: Decoder Module, decode binary, hex and base64 2018-07-19 16:50:42 +02:00
Terrtia
e25dd52d64
add: Binary decoder 2018-07-18 11:45:19 +02:00
Terrtia
0cac3146f1
fix: remove hardcoded mail url 2018-07-17 15:11:25 +02:00
bui
bd5c42caeb add LibInjection Module 2018-07-04 14:28:10 +02:00
Terrtia
16c700ddb4
new: add base64 display on paste + base64 statistics 2018-06-29 10:02:29 +02:00
Terrtia
af9db53171
add tag fp and tp 2018-06-19 15:09:26 +02:00
Terrtia
31fc33ab03
misp event creation, auto and manual + the hive alert auto creation, manual case creation 2018-06-14 16:51:06 +02:00
Terrtia
6143bc3dce
submit users input paste 2018-06-05 16:58:04 +02:00
Terrtia
93dcdf4f23
fix duplicate 2018-05-31 09:01:47 +02:00
Terrtia
0d164b6f4c
add tag js and css 2018-05-16 14:48:15 +02:00
Terrtia
f66a528bc2
fix Duplicate, save list of duplicates on disk + prevent empty hash creation 2018-05-15 23:28:47 +02:00
Terrtia
225fe76c96
fix duplicate + get raw paste 2018-05-11 16:07:30 +02:00
Terrtia
d7ba5533be
fix duplicate + fix issue #200 2018-05-11 13:25:45 +02:00
Terrtia
8599d7142d
python 3 upgrade instructions 2018-05-09 13:32:25 +02:00
Terrtia
142c1dc9b0
clean 2018-05-09 13:03:46 +02:00
Terrtia
3841c159a0
change leveldb to ardb 2018-05-07 14:50:40 +02:00
Terrtia
96a3df42bc
decode with redis connection 2018-05-04 13:53:29 +02:00
Terrtia
a900d5f08d
add python 2 to 3 bash + fix Duplicate + clean 2018-05-03 16:21:33 +02:00
Terrtia
c52caebe7c
improve Duplicate + tlsh + add time out handler + debug + clean 2018-05-02 17:07:10 +02:00
Terrtia
c3428df5dc add apiKeys module 2018-04-26 14:42:39 +02:00
Terrtia
8738b7cf75 fix track term and avoid duplicate mail address 2018-04-24 16:44:37 +02:00
Thirion Aurélien
ef3c335b62
Delete config.cfg.backup 2018-04-23 14:54:23 +02:00
Terrtia
f93fe9aeb2 add bitcoin module 2018-04-23 14:51:53 +02:00
Terrtia
3fc48db903 add Base64 module 2018-04-20 10:48:44 +02:00
Terrtia
8571a86344 frontend python 3.5 upgrade 2018-04-17 16:06:32 +02:00
Terrtia
9c82dd90ec python 3 backend upgrade 2018-04-16 14:50:04 +02:00
CD Nguyen
e9e3f17363 support non TSL SMTP servers 2018-03-30 11:35:37 +02:00
Philipp Schmied
fd329aaba9 config.cfg.sample: Merged overwritten changes from upstream 2018-02-27 16:16:57 +01:00
Philipp Schmied
5b1f0b0212 Implemented email notifications (bin: config.cfg additions, email sending via analyzer scripts; var: Changes to add notifications via terms management); terms_management: Fixed click handlers not being added to all tracked terms. 2018-02-27 15:12:02 +01:00
Sami Mokaddem
c39e537cbe Merge branch 'iss160' into showDiff 
Preemptively preventing merge-conflict with branch iss160 (moving to
configuration)
 2017-12-12 11:33:22 +01:00
Sami Mokaddem
304213cfa4 update: Possibility to change the authorized max line length in the config for the diff of duplicates 2017-12-12 11:00:26 +01:00
Sami Mokaddem
860acbf7cf fix: Renamed Credentials into Credential sectiont title 
config change: By default, zmq listen to both local and CRF
 2017-12-12 09:29:15 +01:00