AIL/ail-framework
7
0
Fork 0
mirror of https://github.com/ail-project/ail-framework.git synced 2024-11-23 14:37:17 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
2095 commits 8 branches 43 tags 99 MiB
Commit graph

346 commits

Author SHA1 Message Date
Bastien Schils
c0171eba81 Added: IP matching module 2019-09-23 18:22:25 +02:00
Terrtia
30fd361e1c
chg: [Cryptocurrency] add private_key entry + fix dash regex 2019-09-22 14:44:37 +02:00
Terrtia
3a9d0157e4
chg: [Cryptocurrency] add new Cryptocurrency module 2019-09-19 16:31:15 +02:00
Terrtia
fed04e8e5a
chg: [Tracker] add optionnal description field 2019-09-13 16:33:34 +02:00
Terrtia
fb35be77f4
chg: [Tracker] add more info 2019-09-13 11:06:41 +02:00
Terrtia
01ff202ae8
chg: [update] add v2.2 update + fix default update + fix Empty Item.get_tags() + add new LAUNCHER options 2019-09-12 13:25:29 +02:00
Terrtia
0ced25f3be
chg: [trackers] filter trackers list by type + minor fix 2019-09-11 15:33:04 +02:00
Terrtia
bd8735a6e5
chg: [merge] master 2019-09-04 10:24:46 +02:00
Terrtia
e0bb973989
fix: [API doc] get bitcoin metadata + list of items 2019-09-04 10:06:52 +02:00
Terrtia
dfd1128daa
chg: [api] add new endpoints: get bitcoin/pgp name/pgp keys/pgp mail metadata + items list 2019-08-16 17:59:44 +02:00
Alexandre Dulaunoy
ab45ac0fef
Merge pull request #370 from CIRCL/api_v1 
Api v1
 2019-08-14 14:28:52 +02:00
Terrtia
7ed09bc923
chg: [UI term tracker] refractor term management: trackers list + show trackers + add new trackers 2019-08-14 09:44:49 +02:00
Terrtia
80f9535074
chg: [api] add endpoint: get tracked item_id by uuid and daterange 2019-08-09 17:07:54 +02:00
Terrtia
4b0819a644
chg: [api] add endpoint: delete tracker term (regex/set/word) 2019-08-09 15:22:57 +02:00
Terrtia
1008c7c4fe
chg: [Term Tracker] refractor term tracker word/set/regex modules + remove old modules 2019-08-09 14:20:13 +02:00
Terrtia
d9bdfecef3
fix: [Term Tracker module] chg module flow 2019-08-07 12:19:42 +02:00
Terrtia
bb6d3a6a26
chg: [Term tracker] add term tracker module (word + set) + API: add new term to track (word + set + regex) 2019-08-07 12:08:24 +02:00
Terrtia
28320a32a6
chg: [term] refractor + add new tracked word/set 2019-08-06 17:03:49 +02:00
Terrtia
e28d563523
chg: [api] use POST with parameters + add API unittest 2019-08-05 16:00:23 +02:00
Terrtia
5e1ae8a893
chg: [api] add new endpoints: get tag metadata + get all tags 2019-08-01 14:36:52 +02:00
Terrtia
98fb6ecef7
fix: [api doc] typo 2019-08-01 13:43:28 +02:00
Terrtia
4c20f58a52
chg: [api] add advanced get item via POST + use same query for each get item 2019-08-01 13:16:57 +02:00
Terrtia
8c02c1b00b
chg: [api] add 404 errors code + add future endpoints in doc 2019-08-01 09:45:59 +02:00
Terrtia
918b4c28ed
fix: [api] fix errors handler 404 405, return json 2019-07-31 13:24:43 +02:00
Terrtia
5f5e86bb13
fix: [Tags + api] fix dict keys name + fix documentation errors output 2019-07-31 11:15:34 +02:00
Terrtia
44cf5bb4af
chg: [API v1] add API documentation + update/delete items tags + Flask_tags refractor 2019-07-30 13:49:21 +02:00
Terrtia
6af9514a48
chg: [API] add GET: item metadata + item content + item tags 2019-07-26 15:44:29 +02:00
Terrtia
0a756294fe
chg: [API] import item (text) 2019-07-26 14:28:02 +02:00
Terrtia
3a8531cafa
chg: [API + import] add API format + item_import refractor 2019-07-25 17:26:32 +02:00
Terrtia
09ecc4d93f
chg: [Crawler] add default crawler config + update default user_agent 2019-07-24 10:18:10 +02:00
Terrtia
bb65179e50
fix: [user_management] fix tokens duplicate + check user_acl_integrity + add login errors messages 2019-06-20 15:49:40 +02:00
Terrtia
6eec0aabfe
Merge branch 'master' into user_management 2019-06-19 15:11:44 +02:00
Terrtia
34e57fe7af
chg: [user_managemant] clean code + check password and email length 2019-06-19 15:00:25 +02:00
Terrtia
42d32ef310
fix: [domain history] fix domain status 2019-06-18 11:19:56 +02:00
Terrtia
e1bf59aded
chg: [UI crawler, show_domain] add domain history list + navigation 2019-06-18 10:58:37 +02:00
Terrtia
b4f06c21f9
chg: [backend crawler] domains: download 1 archive by crawled (most recent) 2019-06-07 13:47:44 +02:00
Terrtia
3fe9d14e9f
chg: [user_management] create default admin user (temp passwd save in AIL_HOME) + change password UI + logout UI + create random password 2019-06-06 21:27:13 +02:00
Terrtia
99e35c51ec
Merge branch 'master' into user_management 2019-06-05 16:59:36 +02:00
Terrtia
4b0a7210b8
Merge branch 'master' of https://github.com/CIRCL/AIL-framework 2019-05-24 13:27:44 +02:00
kovacsbalu
f61d830678 Use default encoding error from redis. 2019-05-16 14:24:03 +02:00
Terrtia
6480744641
chg: [PgpDump] add PgpDump backend 
TODO: UI
 2019-05-14 17:49:31 +02:00
kovacsbalu
05079c143c Fix #314 
Replace char on redis encoding error.
Try to use local file on other error.
 2019-05-14 16:06:20 +02:00
Terrtia
64ff94ce5f
chg: [user_management] create + check user password 2019-05-03 16:52:05 +02:00
Terrtia
1ab1a55a4f
chg: [UI] add basic user management 2019-05-02 17:31:14 +02:00
Terrtia
a183695ddb
chg: [crawler UI] bootstrap 4 migration: show domains by daterange + bug fix hidden services tags 2019-04-26 15:14:29 +02:00
Terrtia
9868833c77
chg: [crawled screenshot] use sha256 as filepath 2019-04-24 14:09:04 +02:00
Terrtia
68077316eb
chg: [git directory] add git helper 2019-04-16 17:26:35 +02:00
Terrtia
3cc614a1ad
chg: [DB] update items tags metadata 2019-04-11 11:58:06 +02:00
Terrtia
d44acea04d
chg: [merge] merge update into advanced_crawler 2019-04-10 16:41:06 +02:00
Terrtia
2589fc2161
chg: [merge] merge master into update branche 2019-04-10 15:43:15 +02:00
Terrtia
b4bee29a33
Merge branch 'master' into advanced_crawler 2019-04-10 15:37:42 +02:00
Terrtia
5165a5de2f
fix: [Crawler] fix index + redis history key 2019-03-22 17:14:27 +01:00
Terrtia
f64c385343
chg: [Crawler] handle port: crawling + history 2019-03-22 16:48:07 +01:00
Terrtia
c2bc72c72e
chg: [Tags alertHandler] remove alertHandler + add quick tag search sidebar + use tag last_seen to display tagged items 2019-03-21 11:11:53 +01:00
Terrtia
65e6eae00d
chg: [UI preview items] b4 migration + new template + better display + decrease item loading time + add basic animation/transition 2019-03-19 13:34:41 +01:00
Jason Hedden
23fb91d326 Add docker-compose support 
This commit separates the AIL Framework services into individual Docker
containers that can be managed with docker-compose. It's intended to ease the
installation, development and troubleshooting procedures for all-in-one
deployments.

No changes to architecture or base code are required in this commit.

Changes to existing files:
.gitignore
  * Ignore pystemon archive directory

Dockerfile:
  * Add AIL shell environment variables
  * Install pystemon and crawler pip requirements into AIL virtual environment

Docker-compose notes:
  * All containers are participating in a service network `network_mode:
  service:flask`.  This allows the containers to share the same IP namespace
  to accommodate hard-coded localhost entries.
  * By default persistent data is saved to the following directories in the
  local AIL framework git directory: PASTES, HASHS, CRAWLED_SCREENSHOTS,
  pystemon/archives

This was tested with docker-ce on Ubuntu 16.04 and MacOS. A typical deployment
would look like:
```
git clone https://github.com/CIRCL/AIL-framework.git
cd AIL-framework
cp bin/packages/config.cfg.docker-compose-sample bin/packages/config.cfg
	(optionally enable activate_crawler in config.cfg)
docker-compose build
docker-compose up -d
```
 2019-03-11 19:46:34 -05:00
Terrtia
d3b3942ad7
chg: [Flask UI] refractor crawler frontend + show crawled domain 2019-03-06 15:05:58 +01:00
Terrtia
7b32d7f34e
chg: [Crawler] major refractor 2019-02-25 16:38:50 +01:00
Terrtia
da78d0552d
chg: [Crawler UI Tags] add tag by day + add crawler status + UI onion blacklist 2019-02-19 11:41:45 +01:00
Terrtia
516238025f
chg: [Crawler] add bootsrap4 src + refractor crawler 2019-02-05 17:16:44 +01:00
Terrtia
535f44044d
fix: [Onion] add timeout signal 2019-01-29 09:46:03 +01:00
Terrtia
6ee4b7bf93
Merge branch 'master' into update 2018-12-19 15:23:34 +01:00
Terrtia
4e08aaa80f
chg: [UI Crawled Domains] show crawled domains by date range (options: domains tags, up domains, down domains) 
todo: Shows domain history
 2018-12-17 15:58:48 +01:00
Terrtia
31a8dfe0b3
fix: [AIL] use only relative paths 
pastes duplicates are fixed on the fly
 2018-11-21 16:45:25 +01:00
Terrtia
cb41c54135
fix: [lib_refine] add dns configuration 2018-11-21 08:49:58 +01:00
Terrtia
d9b776d58a
merge master 2018-11-14 15:55:42 +01:00
Terrtia
b31618eb6a
chg: [Mixer] config default feeder name #291 2018-11-09 15:26:26 +01:00
Terrtia
979472df15
chg: [Terms] tag tracked terms 2018-11-06 13:38:37 +01:00
Terrtia
aa16c52cbf
fix: [NotificationHelper] add config 2018-11-05 14:20:12 +01:00
Terrtia
5872cf9196
fix: [Scripts] Remove absolute path 2018-11-02 16:07:27 +01:00
Terrtia
ea7fd8d54c
fix: [Duplicate] show duplicate with relative path 2018-10-24 15:47:03 +02:00
Terrtia
5afbc3179c
Merge branch 'master' of https://github.com/CIRCL/AIL-framework 2018-10-24 15:16:41 +02:00
Terrtia
8a6ac6be1e
fix: [Paste] create absolute path 2018-10-24 15:16:12 +02:00
Sami Mokaddem
d9bc2d6f44
Merge pull request #266 from kovacsbalu/fix-regexp-timeout 
Stop regexp processing after timeout (60sec)
 2018-10-09 14:48:08 +02:00
kovacsbalu
272b0fe960 RegexpForTerm timeout from config 
Critical log removed
 2018-10-09 09:32:32 +02:00
Terrtia
44d19fbb80
fix: [pystemon-feeder] paste save path 2018-10-02 16:02:59 +02:00
Alexandre Dulaunoy
59df5f0e12
chg: [config] Tor crawler disabled by default in sample configuration 
(to match HOWTO when enabling it)
 2018-10-02 14:17:58 +02:00
Terrtia
04b9d9fc1d
chg: [Crawler] add docs 2018-09-27 11:14:29 +02:00
Terrtia
e4a5f66a10
Merge branch 'master' into onion_crawler 2018-09-24 16:35:04 +02:00
Terrtia
50c81773e9
chg: [Crawler] add launcher and install 2018-09-24 16:23:14 +02:00
Terrtia
ead4dba0d8
fix: [MISP feeder] typo filename-config name 2018-09-21 11:02:08 +02:00
Terrtia
6edc1ddbeb
chg: [HiddenServices] get domain related to other domains 2018-09-21 10:34:06 +02:00
Terrtia
5eadc2755f
chg: [Flask] add prefix in config to flask routes 2018-09-20 10:38:19 +02:00
Terrtia
ea3d2c1977
Merge branch 'master' into statistic 2018-09-12 10:06:53 +02:00
Terrtia
6f0817365a
chg: [Crawler UI] display domain information 2018-09-12 09:55:49 +02:00
Terrtia
ca982e13e1
chg: [Crawled Domain] show crawled pastes by domain 2018-08-27 14:34:08 +02:00
Terrtia
823c33174a
Merge branch 'master' into onion_crawler 2018-08-27 11:20:51 +02:00
Terrtia
5c902640e9
Merge branch 'master' into onion_crawler 2018-08-27 11:05:25 +02:00
Terrtia
d42dd118a4
chg: [Domain crawled] add random screenshot 2018-08-27 11:02:39 +02:00
Terrtia
a586548047
Merge branch 'master' into base64 2018-08-24 10:35:28 +02:00
Terrtia
ced0b1e350
chg: [I2P] add default config 2018-08-24 10:24:03 +02:00
Terrtia
e9580d6775
chg: [Crawler] change BDD, save i2p links 2018-08-21 15:54:53 +02:00
Terrtia
ed559d9f4a
chg: [Showpaste] add screenshot + improve onion db 2018-08-16 17:24:39 +02:00
Terrtia
8b1c10b38c
chg: [Onion] add onion splash crawler 2018-08-09 17:42:21 +02:00
Thirion Aurélien
000d5bf40a
fix: add miss config 2018-08-07 13:07:08 +02:00
Terrtia
940b96734a
Merge branch 'master' into statistic 2018-08-01 09:29:51 +02:00
Terrtia
39f54d4420
chg: [Mail] add country statistic 2018-07-30 09:21:22 +02:00
Terrtia
57b631233b
add: [BankAccount] iban detection 2018-07-26 15:31:58 +02:00
Terrtia
8beb63b6a8
fix: [Queues] module with no subscriber 2018-07-24 15:57:18 +02:00
Terrtia
8c97c70a3f
Merge branch 'master' into base64 2018-07-24 15:21:32 +02:00
Terrtia
bd5f83f0eb
chg: refractor base64 encoded to hash 2018-07-20 10:32:52 +02:00
Terrtia
9a8e37fb0b
fix: decoder bdd 2018-07-20 09:43:09 +02:00
Terrtia
ad26f016e3
add: Decoder Module, decode binary, hex and base64 2018-07-19 16:50:42 +02:00
Terrtia
e25dd52d64
add: Binary decoder 2018-07-18 11:45:19 +02:00
Terrtia
0cac3146f1
fix: remove hardcoded mail url 2018-07-17 15:11:25 +02:00
bui
bd5c42caeb add LibInjection Module 2018-07-04 14:28:10 +02:00
Terrtia
16c700ddb4
new: add base64 display on paste + base64 statistics 2018-06-29 10:02:29 +02:00
Terrtia
af9db53171
add tag fp and tp 2018-06-19 15:09:26 +02:00
Terrtia
31fc33ab03
misp event creation, auto and manual + the hive alert auto creation, manual case creation 2018-06-14 16:51:06 +02:00
Terrtia
6143bc3dce
submit users input paste 2018-06-05 16:58:04 +02:00
Terrtia
93dcdf4f23
fix duplicate 2018-05-31 09:01:47 +02:00
Terrtia
0d164b6f4c
add tag js and css 2018-05-16 14:48:15 +02:00
Terrtia
f66a528bc2
fix Duplicate, save list of duplicates on disk + prevent empty hash creation 2018-05-15 23:28:47 +02:00
Terrtia
225fe76c96
fix duplicate + get raw paste 2018-05-11 16:07:30 +02:00
Terrtia
d7ba5533be
fix duplicate + fix issue #200 2018-05-11 13:25:45 +02:00
Terrtia
8599d7142d
python 3 upgrade instructions 2018-05-09 13:32:25 +02:00
Terrtia
142c1dc9b0
clean 2018-05-09 13:03:46 +02:00
Terrtia
3841c159a0
change leveldb to ardb 2018-05-07 14:50:40 +02:00
Terrtia
96a3df42bc
decode with redis connection 2018-05-04 13:53:29 +02:00
Terrtia
a900d5f08d
add python 2 to 3 bash + fix Duplicate + clean 2018-05-03 16:21:33 +02:00
Terrtia
c52caebe7c
improve Duplicate + tlsh + add time out handler + debug + clean 2018-05-02 17:07:10 +02:00
Terrtia
c3428df5dc add apiKeys module 2018-04-26 14:42:39 +02:00
Terrtia
8738b7cf75 fix track term and avoid duplicate mail address 2018-04-24 16:44:37 +02:00
Thirion Aurélien
ef3c335b62
Delete config.cfg.backup 2018-04-23 14:54:23 +02:00
Terrtia
f93fe9aeb2 add bitcoin module 2018-04-23 14:51:53 +02:00
Terrtia
3fc48db903 add Base64 module 2018-04-20 10:48:44 +02:00
Terrtia
8571a86344 frontend python 3.5 upgrade 2018-04-17 16:06:32 +02:00
Terrtia
9c82dd90ec python 3 backend upgrade 2018-04-16 14:50:04 +02:00
CD Nguyen
e9e3f17363 support non TSL SMTP servers 2018-03-30 11:35:37 +02:00
Philipp Schmied
fd329aaba9 config.cfg.sample: Merged overwritten changes from upstream 2018-02-27 16:16:57 +01:00
Philipp Schmied
5b1f0b0212 Implemented email notifications (bin: config.cfg additions, email sending via analyzer scripts; var: Changes to add notifications via terms management); terms_management: Fixed click handlers not being added to all tracked terms. 2018-02-27 15:12:02 +01:00
Sami Mokaddem
c39e537cbe Merge branch 'iss160' into showDiff 
Preemptively preventing merge-conflict with branch iss160 (moving to
configuration)
 2017-12-12 11:33:22 +01:00
Sami Mokaddem
304213cfa4 update: Possibility to change the authorized max line length in the config for the diff of duplicates 2017-12-12 11:00:26 +01:00
Sami Mokaddem
860acbf7cf fix: Renamed Credentials into Credential sectiont title 
config change: By default, zmq listen to both local and CRF
 2017-12-12 09:29:15 +01:00
Sami Mokaddem
9d0d0b4303 update: Moved filtering operation (thresholds, number of matching in the 
categ file) in the configuration file. It permits to better control the
flow of pastes.
Also set default mixer duplicate filtering to 3 (Do not filter)
 2017-12-11 17:28:34 +01:00
Sami Mokaddem
2a967c4d92 update/feature: Max number of duplicate push to MISP + duplicate are pushed as attachment 2017-11-23 07:13:44 +01:00
Sami Mokaddem
6f0227a78a Starting support of python3 2017-11-20 12:12:06 +01:00
Sami Mokaddem
652b00f4b2 Renamed BrowseWarningPaste into alertHandler 2017-11-15 16:15:43 +01:00
=
64ec44085d Dynamically choose correct port for LevelDB in script 2017-08-22 17:52:15 +02:00
Mokaddem
882ab88b00 Draft module + web link 2017-07-18 16:57:15 +02:00
Mokaddem
b60e164380 Added support of onion in browseWarningPaste 2017-04-26 11:47:02 +02:00
Mokaddem
da081bae42 Merge branch 'regexInTerms' into production 2017-04-18 16:59:40 +02:00
Mokaddem
c8baabd882 Updated TermsFrequency related modules + Fixed bugs (encoding, behaviors, ...) 2017-04-18 15:28:21 +02:00
Mokaddem
2da4c572c7 New feature in Terms: regex and set of words support (draft) 2017-03-28 17:42:44 +02:00
Mokaddem
e225090e07 Added script that checks configuration and may auto-fix it + added missing field in config.cfg.sample 2017-03-15 16:36:51 +01:00
Mokaddem
adc37ab50e Indexer max size now in config 2017-03-15 12:14:41 +01:00
Olivier MEDOC
16044d4d36 pystemon-feeder: add the option bind in configuration file instead of address 
This is essentially because multiple feeds can be used in the address
configuration variable.
 2017-01-13 14:58:18 +01:00
Olivier MEDOC
9f9c265cb0 SentimentAnalysis: use lexicon file path from the ail configuration file 2017-01-11 11:07:08 +01:00
ptitdoc
c0fac82031 Add pystemon path in configuration sample 2017-01-10 18:33:46 +01:00
Mokaddem
0e39f516a9 Added sleep in helper for multiple listening interfaces (reduce useless work), Added a simple way to pre-process feed before sending it to the global feed. 2017-01-09 14:12:26 +01:00