Terrtia
0a756294fe
chg: [API] import item (text)
2019-07-26 14:28:02 +02:00
Terrtia
3a8531cafa
chg: [API + import] add API format + item_import refractor
2019-07-25 17:26:32 +02:00
Terrtia
09ecc4d93f
chg: [Crawler] add default crawler config + update default user_agent
2019-07-24 10:18:10 +02:00
Terrtia
bb65179e50
fix: [user_management] fix tokens duplicate + check user_acl_integrity + add login errors messages
2019-06-20 15:49:40 +02:00
Terrtia
6eec0aabfe
Merge branch 'master' into user_management
2019-06-19 15:11:44 +02:00
Terrtia
34e57fe7af
chg: [user_managemant] clean code + check password and email length
2019-06-19 15:00:25 +02:00
Terrtia
42d32ef310
fix: [domain history] fix domain status
2019-06-18 11:19:56 +02:00
Terrtia
e1bf59aded
chg: [UI crawler, show_domain] add domain history list + navigation
2019-06-18 10:58:37 +02:00
Terrtia
b4f06c21f9
chg: [backend crawler] domains: download 1 archive by crawled (most recent)
2019-06-07 13:47:44 +02:00
Terrtia
3fe9d14e9f
chg: [user_management] create default admin user (temp passwd save in AIL_HOME) + change password UI + logout UI + create random password
2019-06-06 21:27:13 +02:00
Terrtia
99e35c51ec
Merge branch 'master' into user_management
2019-06-05 16:59:36 +02:00
Terrtia
4b0a7210b8
Merge branch 'master' of https://github.com/CIRCL/AIL-framework
2019-05-24 13:27:44 +02:00
kovacsbalu
f61d830678
Use default encoding error from redis.
2019-05-16 14:24:03 +02:00
Terrtia
6480744641
chg: [PgpDump] add PgpDump backend
...
TODO: UI
2019-05-14 17:49:31 +02:00
kovacsbalu
05079c143c
Fix #314
...
Replace char on redis encoding error.
Try to use local file on other error.
2019-05-14 16:06:20 +02:00
Terrtia
64ff94ce5f
chg: [user_management] create + check user password
2019-05-03 16:52:05 +02:00
Terrtia
1ab1a55a4f
chg: [UI] add basic user management
2019-05-02 17:31:14 +02:00
Terrtia
a183695ddb
chg: [crawler UI] bootstrap 4 migration: show domains by daterange + bug fix hidden services tags
2019-04-26 15:14:29 +02:00
Terrtia
9868833c77
chg: [crawled screenshot] use sha256 as filepath
2019-04-24 14:09:04 +02:00
Terrtia
68077316eb
chg: [git directory] add git helper
2019-04-16 17:26:35 +02:00
Terrtia
3cc614a1ad
chg: [DB] update items tags metadata
2019-04-11 11:58:06 +02:00
Terrtia
d44acea04d
chg: [merge] merge update into advanced_crawler
2019-04-10 16:41:06 +02:00
Terrtia
2589fc2161
chg: [merge] merge master into update branche
2019-04-10 15:43:15 +02:00
Terrtia
b4bee29a33
Merge branch 'master' into advanced_crawler
2019-04-10 15:37:42 +02:00
Terrtia
5165a5de2f
fix: [Crawler] fix index + redis history key
2019-03-22 17:14:27 +01:00
Terrtia
f64c385343
chg: [Crawler] handle port: crawling + history
2019-03-22 16:48:07 +01:00
Terrtia
c2bc72c72e
chg: [Tags alertHandler] remove alertHandler + add quick tag search sidebar + use tag last_seen to display tagged items
2019-03-21 11:11:53 +01:00
Terrtia
65e6eae00d
chg: [UI preview items] b4 migration + new template + better display + decrease item loading time + add basic animation/transition
2019-03-19 13:34:41 +01:00
Jason Hedden
23fb91d326
Add docker-compose support
...
This commit separates the AIL Framework services into individual Docker
containers that can be managed with docker-compose. It's intended to ease the
installation, development and troubleshooting procedures for all-in-one
deployments.
No changes to architecture or base code are required in this commit.
Changes to existing files:
.gitignore
* Ignore pystemon archive directory
Dockerfile:
* Add AIL shell environment variables
* Install pystemon and crawler pip requirements into AIL virtual environment
Docker-compose notes:
* All containers are participating in a service network `network_mode:
service:flask`. This allows the containers to share the same IP namespace
to accommodate hard-coded localhost entries.
* By default persistent data is saved to the following directories in the
local AIL framework git directory: PASTES, HASHS, CRAWLED_SCREENSHOTS,
pystemon/archives
This was tested with docker-ce on Ubuntu 16.04 and MacOS. A typical deployment
would look like:
```
git clone https://github.com/CIRCL/AIL-framework.git
cd AIL-framework
cp bin/packages/config.cfg.docker-compose-sample bin/packages/config.cfg
(optionally enable activate_crawler in config.cfg)
docker-compose build
docker-compose up -d
```
2019-03-11 19:46:34 -05:00
Terrtia
d3b3942ad7
chg: [Flask UI] refractor crawler frontend + show crawled domain
2019-03-06 15:05:58 +01:00
Terrtia
7b32d7f34e
chg: [Crawler] major refractor
2019-02-25 16:38:50 +01:00
Terrtia
da78d0552d
chg: [Crawler UI Tags] add tag by day + add crawler status + UI onion blacklist
2019-02-19 11:41:45 +01:00
Terrtia
516238025f
chg: [Crawler] add bootsrap4 src + refractor crawler
2019-02-05 17:16:44 +01:00
Terrtia
535f44044d
fix: [Onion] add timeout signal
2019-01-29 09:46:03 +01:00
Terrtia
6ee4b7bf93
Merge branch 'master' into update
2018-12-19 15:23:34 +01:00
Terrtia
4e08aaa80f
chg: [UI Crawled Domains] show crawled domains by date range (options: domains tags, up domains, down domains)
...
todo: Shows domain history
2018-12-17 15:58:48 +01:00
Terrtia
31a8dfe0b3
fix: [AIL] use only relative paths
...
pastes duplicates are fixed on the fly
2018-11-21 16:45:25 +01:00
Terrtia
cb41c54135
fix: [lib_refine] add dns configuration
2018-11-21 08:49:58 +01:00
Terrtia
d9b776d58a
merge master
2018-11-14 15:55:42 +01:00
Terrtia
b31618eb6a
chg: [Mixer] config default feeder name #291
2018-11-09 15:26:26 +01:00
Terrtia
979472df15
chg: [Terms] tag tracked terms
2018-11-06 13:38:37 +01:00
Terrtia
aa16c52cbf
fix: [NotificationHelper] add config
2018-11-05 14:20:12 +01:00
Terrtia
5872cf9196
fix: [Scripts] Remove absolute path
2018-11-02 16:07:27 +01:00
Terrtia
ea7fd8d54c
fix: [Duplicate] show duplicate with relative path
2018-10-24 15:47:03 +02:00
Terrtia
5afbc3179c
Merge branch 'master' of https://github.com/CIRCL/AIL-framework
2018-10-24 15:16:41 +02:00
Terrtia
8a6ac6be1e
fix: [Paste] create absolute path
2018-10-24 15:16:12 +02:00
Sami Mokaddem
d9bc2d6f44
Merge pull request #266 from kovacsbalu/fix-regexp-timeout
...
Stop regexp processing after timeout (60sec)
2018-10-09 14:48:08 +02:00
kovacsbalu
272b0fe960
RegexpForTerm timeout from config
...
Critical log removed
2018-10-09 09:32:32 +02:00
Terrtia
44d19fbb80
fix: [pystemon-feeder] paste save path
2018-10-02 16:02:59 +02:00
59df5f0e12
chg: [config] Tor crawler disabled by default in sample configuration
...
(to match HOWTO when enabling it)
2018-10-02 14:17:58 +02:00
Terrtia
04b9d9fc1d
chg: [Crawler] add docs
2018-09-27 11:14:29 +02:00
Terrtia
e4a5f66a10
Merge branch 'master' into onion_crawler
2018-09-24 16:35:04 +02:00
Terrtia
50c81773e9
chg: [Crawler] add launcher and install
2018-09-24 16:23:14 +02:00
Terrtia
ead4dba0d8
fix: [MISP feeder] typo filename-config name
2018-09-21 11:02:08 +02:00
Terrtia
6edc1ddbeb
chg: [HiddenServices] get domain related to other domains
2018-09-21 10:34:06 +02:00
Terrtia
5eadc2755f
chg: [Flask] add prefix in config to flask routes
2018-09-20 10:38:19 +02:00
Terrtia
ea3d2c1977
Merge branch 'master' into statistic
2018-09-12 10:06:53 +02:00
Terrtia
6f0817365a
chg: [Crawler UI] display domain information
2018-09-12 09:55:49 +02:00
Terrtia
ca982e13e1
chg: [Crawled Domain] show crawled pastes by domain
2018-08-27 14:34:08 +02:00
Terrtia
823c33174a
Merge branch 'master' into onion_crawler
2018-08-27 11:20:51 +02:00
Terrtia
5c902640e9
Merge branch 'master' into onion_crawler
2018-08-27 11:05:25 +02:00
Terrtia
d42dd118a4
chg: [Domain crawled] add random screenshot
2018-08-27 11:02:39 +02:00
Terrtia
a586548047
Merge branch 'master' into base64
2018-08-24 10:35:28 +02:00
Terrtia
ced0b1e350
chg: [I2P] add default config
2018-08-24 10:24:03 +02:00
Terrtia
e9580d6775
chg: [Crawler] change BDD, save i2p links
2018-08-21 15:54:53 +02:00
Terrtia
ed559d9f4a
chg: [Showpaste] add screenshot + improve onion db
2018-08-16 17:24:39 +02:00
Terrtia
8b1c10b38c
chg: [Onion] add onion splash crawler
2018-08-09 17:42:21 +02:00
Thirion Aurélien
000d5bf40a
fix: add miss config
2018-08-07 13:07:08 +02:00
Terrtia
940b96734a
Merge branch 'master' into statistic
2018-08-01 09:29:51 +02:00
Terrtia
39f54d4420
chg: [Mail] add country statistic
2018-07-30 09:21:22 +02:00
Terrtia
57b631233b
add: [BankAccount] iban detection
2018-07-26 15:31:58 +02:00
Terrtia
8beb63b6a8
fix: [Queues] module with no subscriber
2018-07-24 15:57:18 +02:00
Terrtia
8c97c70a3f
Merge branch 'master' into base64
2018-07-24 15:21:32 +02:00
Terrtia
bd5f83f0eb
chg: refractor base64 encoded to hash
2018-07-20 10:32:52 +02:00
Terrtia
9a8e37fb0b
fix: decoder bdd
2018-07-20 09:43:09 +02:00
Terrtia
ad26f016e3
add: Decoder Module, decode binary, hex and base64
2018-07-19 16:50:42 +02:00
Terrtia
e25dd52d64
add: Binary decoder
2018-07-18 11:45:19 +02:00
Terrtia
0cac3146f1
fix: remove hardcoded mail url
2018-07-17 15:11:25 +02:00
bui
bd5c42caeb
add LibInjection Module
2018-07-04 14:28:10 +02:00
Terrtia
16c700ddb4
new: add base64 display on paste + base64 statistics
2018-06-29 10:02:29 +02:00
Terrtia
af9db53171
add tag fp and tp
2018-06-19 15:09:26 +02:00
Terrtia
31fc33ab03
misp event creation, auto and manual + the hive alert auto creation, manual case creation
2018-06-14 16:51:06 +02:00
Terrtia
6143bc3dce
submit users input paste
2018-06-05 16:58:04 +02:00
Terrtia
93dcdf4f23
fix duplicate
2018-05-31 09:01:47 +02:00
Terrtia
0d164b6f4c
add tag js and css
2018-05-16 14:48:15 +02:00
Terrtia
f66a528bc2
fix Duplicate, save list of duplicates on disk + prevent empty hash creation
2018-05-15 23:28:47 +02:00
Terrtia
225fe76c96
fix duplicate + get raw paste
2018-05-11 16:07:30 +02:00
Terrtia
d7ba5533be
fix duplicate + fix issue #200
2018-05-11 13:25:45 +02:00
Terrtia
8599d7142d
python 3 upgrade instructions
2018-05-09 13:32:25 +02:00
Terrtia
142c1dc9b0
clean
2018-05-09 13:03:46 +02:00
Terrtia
3841c159a0
change leveldb to ardb
2018-05-07 14:50:40 +02:00
Terrtia
96a3df42bc
decode with redis connection
2018-05-04 13:53:29 +02:00
Terrtia
a900d5f08d
add python 2 to 3 bash + fix Duplicate + clean
2018-05-03 16:21:33 +02:00
Terrtia
c52caebe7c
improve Duplicate + tlsh + add time out handler + debug + clean
2018-05-02 17:07:10 +02:00
Terrtia
c3428df5dc
add apiKeys module
2018-04-26 14:42:39 +02:00
Terrtia
8738b7cf75
fix track term and avoid duplicate mail address
2018-04-24 16:44:37 +02:00
Thirion Aurélien
ef3c335b62
Delete config.cfg.backup
2018-04-23 14:54:23 +02:00
Terrtia
f93fe9aeb2
add bitcoin module
2018-04-23 14:51:53 +02:00
Terrtia
3fc48db903
add Base64 module
2018-04-20 10:48:44 +02:00
Terrtia
8571a86344
frontend python 3.5 upgrade
2018-04-17 16:06:32 +02:00
Terrtia
9c82dd90ec
python 3 backend upgrade
2018-04-16 14:50:04 +02:00
CD Nguyen
e9e3f17363
support non TSL SMTP servers
2018-03-30 11:35:37 +02:00
Philipp Schmied
fd329aaba9
config.cfg.sample: Merged overwritten changes from upstream
2018-02-27 16:16:57 +01:00
Philipp Schmied
5b1f0b0212
Implemented email notifications (bin: config.cfg additions, email sending via analyzer scripts; var: Changes to add notifications via terms management); terms_management: Fixed click handlers not being added to all tracked terms.
2018-02-27 15:12:02 +01:00
Sami Mokaddem
c39e537cbe
Merge branch 'iss160' into showDiff
...
Preemptively preventing merge-conflict with branch iss160 (moving to
configuration)
2017-12-12 11:33:22 +01:00
Sami Mokaddem
304213cfa4
update: Possibility to change the authorized max line length in the config for the diff of duplicates
2017-12-12 11:00:26 +01:00
Sami Mokaddem
860acbf7cf
fix: Renamed Credentials into Credential sectiont title
...
config change: By default, zmq listen to both local and CRF
2017-12-12 09:29:15 +01:00
Sami Mokaddem
9d0d0b4303
update: Moved filtering operation (thresholds, number of matching in the
...
categ file) in the configuration file. It permits to better control the
flow of pastes.
Also set default mixer duplicate filtering to 3 (Do not filter)
2017-12-11 17:28:34 +01:00
Sami Mokaddem
2a967c4d92
update/feature: Max number of duplicate push to MISP + duplicate are pushed as attachment
2017-11-23 07:13:44 +01:00
Sami Mokaddem
6f0227a78a
Starting support of python3
2017-11-20 12:12:06 +01:00
Sami Mokaddem
652b00f4b2
Renamed BrowseWarningPaste into alertHandler
2017-11-15 16:15:43 +01:00
=
64ec44085d
Dynamically choose correct port for LevelDB in script
2017-08-22 17:52:15 +02:00
Mokaddem
882ab88b00
Draft module + web link
2017-07-18 16:57:15 +02:00
Mokaddem
b60e164380
Added support of onion in browseWarningPaste
2017-04-26 11:47:02 +02:00
Mokaddem
da081bae42
Merge branch 'regexInTerms' into production
2017-04-18 16:59:40 +02:00
Mokaddem
c8baabd882
Updated TermsFrequency related modules + Fixed bugs (encoding, behaviors, ...)
2017-04-18 15:28:21 +02:00
Mokaddem
2da4c572c7
New feature in Terms: regex and set of words support (draft)
2017-03-28 17:42:44 +02:00
Mokaddem
e225090e07
Added script that checks configuration and may auto-fix it + added missing field in config.cfg.sample
2017-03-15 16:36:51 +01:00
Mokaddem
adc37ab50e
Indexer max size now in config
2017-03-15 12:14:41 +01:00
Olivier MEDOC
16044d4d36
pystemon-feeder: add the option bind in configuration file instead of address
...
This is essentially because multiple feeds can be used in the address
configuration variable.
2017-01-13 14:58:18 +01:00
Olivier MEDOC
9f9c265cb0
SentimentAnalysis: use lexicon file path from the ail configuration file
2017-01-11 11:07:08 +01:00
ptitdoc
c0fac82031
Add pystemon path in configuration sample
2017-01-10 18:33:46 +01:00
Mokaddem
0e39f516a9
Added sleep in helper for multiple listening interfaces (reduce useless work), Added a simple way to pre-process feed before sending it to the global feed.
2017-01-09 14:12:26 +01:00
Mokaddem
e70b9cd15c
Added basic mixer with confirugable behavior. It handles muliple feeders and performs some basic stats on them.
2016-12-23 10:31:26 +01:00
Mokaddem
7438f16c63
Added new config
2016-12-22 10:06:35 +01:00
eb14005fcf
Fix: as JSON decode needs a string, the empty list is a string.
2016-08-29 12:08:12 +00:00
3b6e3e156e
Merge branch 'master' of github.com:CIRCL/AIL-framework
2016-08-26 15:57:45 +00:00
Mokaddem
0c760d763b
Added support of PID in ModuleInformation and index.html
2016-08-26 15:27:37 +02:00
1ab08d4a05
Merge branch 'master' of github.com:CIRCL/AIL-framework
2016-08-25 12:33:31 +00:00
Mokaddem
07856f3119
Added script module-dataflow-graph + fixed bug in module.cfg preventing the credential module to receive data.
2016-08-24 11:32:48 +02:00
bde5f47ed0
Merge branch 'master' of github.com:CIRCL/AIL-framework
...
Conflicts:
bin/launch_scripts.sh
2016-08-23 14:20:39 +00:00
Mokaddem
697812dbff
Added config.sample
2016-08-23 09:59:39 +02:00
Mokaddem
d3c0fc5f94
Added terms trending module and web interface:
...
- Curve module has been rewritten
- Added new module curve_manage_top_set
- Added all web-pages
2016-08-22 13:35:49 +02:00
Mokaddem
c12e67a35a
Added tooltips + slight gauge modification
2016-08-17 10:43:43 +02:00
Mokaddem
894b9efda9
Added drop of really long line in sentiment-analysis module + Added description of sentiment module. Also, fixed bug in webpage sentiement-trending concerning avg and date range.
2016-08-17 09:46:25 +02:00
Mokaddem
1084e45f1b
Added draft of filter in sentiment analysis (Discard syntaxical languages) + Added nice tooltip for sparkline. Trending displays avg in function of the number of elements processed and not for the complete week + fixed bug in gauge and canvasjs (was performing avg with only 1 graph instead of all 8).
2016-08-16 16:33:02 +02:00
Mokaddem
1a5158b081
Added sentiment analyser module (draft)
2016-08-13 15:24:57 +02:00
95a22abe9c
Merge branch 'master' of github.com:CIRCL/AIL-framework
...
Conflicts:
bin/packages/modules.cfg
2016-08-11 08:06:57 +00:00
Mokaddem
a4595010c0
Added whole config in config.sample
2016-08-09 14:23:36 +02:00
Mokaddem
4e56f76f62
Fixed bug CVE,
...
CVE was not sending its data to BrowseWarningPaste.
2016-08-08 10:05:21 +02:00
Mokaddem
1c40ad522f
Merge branch 'master' into production
2016-08-08 09:26:58 +02:00
Mokaddem
78c611fead
Added warning_paste module and created related webpages.
...
Fixed a Faup bug in credential (multiple instanciation) and added correc populate_set_out in concerned modules (creditcard, credential, ...).
Linked browse_warning_paste module and Flask function with redis (created new sets).
2016-08-08 09:17:44 +02:00
Mokaddem
55ef917ef1
Added few comments
2016-08-04 12:10:56 +02:00
Mokaddem
d9316771cd
Draft: added new duplicate hash comparison - tlsh
2016-08-04 11:55:38 +02:00
Mokaddem
3169dc27b0
Added SQLInjectionDetection module
2016-08-02 15:43:11 +02:00
Mokaddem
06be1f129a
Added Draft of ModuleStats + Paste size average
2016-07-25 16:38:57 +02:00
d013546643
Merge branch 'master' of github.com:CIRCL/AIL-framework
...
Conflicts:
bin/packages/modules.cfg
2016-07-25 07:56:46 +00:00
722634adf4
Prod config
2016-07-25 07:49:14 +00:00
Raphaël Vinot
4aa484a91f
Merge pull request #63 from mokaddem/uptodate-duplicate-module
...
Uptodate duplicate module
2016-07-25 09:44:49 +02:00
Raphaël Vinot
ea8703e608
Fix path to tldsfile
2016-07-22 09:32:13 +02:00
