Commit graph

1553 commits

Author SHA1 Message Date
Terrtia
a3167a740a
fix: [Update DB] remove indexErrors 2019-04-12 17:32:17 +02:00
Terrtia
bf92a2f22f
fix: [background update] avoid multiple update on the same key 2019-04-12 16:07:40 +02:00
Terrtia
f2dc3bcbf4
fix: [Global] remove absolute path from item path 2019-04-12 15:13:36 +02:00
Terrtia
3a8631d44a
chg: [Update + Launcher] use background update + add new options in launcher 2019-04-12 15:06:24 +02:00
Terrtia
5c4db5f2f1
chg: [update] add update script Redis ARDB + add background update 2019-04-11 17:49:20 +02:00
Terrtia
1ed9468881
fix: [UI showpaste] min preview 2019-04-11 13:44:20 +02:00
Terrtia
3cc614a1ad
chg: [DB] update items tags metadata 2019-04-11 11:58:06 +02:00
Terrtia
e83174327a
chg: [DB] remove absolute path 2019-04-10 17:47:40 +02:00
Terrtia
d44acea04d
chg: [merge] merge update into advanced_crawler 2019-04-10 16:41:06 +02:00
Terrtia
2589fc2161
chg: [merge] merge master into update branche 2019-04-10 15:43:15 +02:00
Terrtia
b4bee29a33
Merge branch 'master' into advanced_crawler 2019-04-10 15:37:42 +02:00
Terrtia
6165987ec3
chg: [DB Migration] add hash dynamic update 2019-04-10 15:36:41 +02:00
Terrtia
0519b4a437
chg: [update DB] add migration script 2019-04-08 17:04:09 +02:00
Terrtia
870fa1981c
fix: [update_thirdparty] detect and use virtualenv 2019-04-02 09:50:52 +02:00
Terrtia
f6d7d2ae16
chg: [Update] add debug 2019-04-02 09:46:04 +02:00
Thirion Aurélien
976b97bcd1
Merge pull request #338 from CIRCL/pre_update
Merge Pre update
2019-03-26 16:51:26 +01:00
Terrtia
044933aa33
chg: [Launch] remove old launchers 2019-03-26 16:47:10 +01:00
Terrtia
cee2b2486f
chg: [Launch] remove old launchers 2019-03-26 16:45:01 +01:00
Terrtia
59664efe45
Merge branch 'master' into advanced_crawler 2019-03-26 16:03:42 +01:00
Terrtia
5165a5de2f
fix: [Crawler] fix index + redis history key 2019-03-22 17:14:27 +01:00
Terrtia
f64c385343
chg: [Crawler] handle port: crawling + history 2019-03-22 16:48:07 +01:00
Terrtia
f4cdddbc7f
chg: [UI + item submit] remove browse paste from UI + fix pasteSubmit tagging 2019-03-21 11:50:53 +01:00
Terrtia
51eb951a47
fix: [UI terms] update tags endpoint 2019-03-21 11:34:21 +01:00
Terrtia
c2bc72c72e
chg: [Tags alertHandler] remove alertHandler + add quick tag search sidebar + use tag last_seen to display tagged items 2019-03-21 11:11:53 +01:00
Terrtia
ce6475e96e
chg: [tags] remove/add tags by daterange + improve item loading time 2019-03-20 15:57:55 +01:00
Terrtia
8b5266706e
fix: [Tags] cast nb_max_page to int 2019-03-20 13:33:12 +01:00
Terrtia
a9e86231ad
chg: [Tags] add backend, daterange selector 2019-03-20 13:25:02 +01:00
Terrtia
65e6eae00d
chg: [UI preview items] b4 migration + new template + better display + decrease item loading time + add basic animation/transition 2019-03-19 13:34:41 +01:00
Terrtia
e9bac098c9
fix: [UI Tags] paste preview 2019-03-15 11:16:44 +01:00
Terrtia
23d9cb2265
chg: [UI Tags search] bootstrap 4 migration 2019-03-14 17:04:55 +01:00
Thirion Aurélien
c87f0ffcb1
Merge pull request #336 from 5thColumn/docker
Add indexdir volume to docker-compose
2019-03-14 09:35:19 +01:00
Jason Hedden
9279f56048 Add indexdir volume to docker-compose
Make the indexdir data persistent and shared with Flask for searching pastes.
2019-03-13 17:05:54 -05:00
85d2ea578c
Merge pull request #335 from 5thColumn/docker
Add docker-compose support
2019-03-13 21:38:12 +01:00
Terrtia
9e4813ca4b
chg: [UI Tags] start b4 migration + add date range selector 2019-03-12 13:37:15 +01:00
Jason Hedden
23fb91d326 Add docker-compose support
This commit separates the AIL Framework services into individual Docker
containers that can be managed with docker-compose. It's intended to ease the
installation, development and troubleshooting procedures for all-in-one
deployments.

No changes to architecture or base code are required in this commit.

Changes to existing files:
.gitignore
  * Ignore pystemon archive directory

Dockerfile:
  * Add AIL shell environment variables
  * Install pystemon and crawler pip requirements into AIL virtual environment

Docker-compose notes:
  * All containers are participating in a service network `network_mode:
  service:flask`.  This allows the containers to share the same IP namespace
  to accommodate hard-coded localhost entries.
  * By default persistent data is saved to the following directories in the
  local AIL framework git directory: PASTES, HASHS, CRAWLED_SCREENSHOTS,
  pystemon/archives

This was tested with docker-ce on Ubuntu 16.04 and MacOS. A typical deployment
would look like:
```
git clone https://github.com/CIRCL/AIL-framework.git
cd AIL-framework
cp bin/packages/config.cfg.docker-compose-sample bin/packages/config.cfg
	(optionally enable activate_crawler in config.cfg)
docker-compose build
docker-compose up -d
```
2019-03-11 19:46:34 -05:00
d54c4e2d86
Merge pull request #334 from 5thColumn/pybgpranking
Add pybgpranking to install deps
2019-03-11 07:46:46 +01:00
Jason Hedden
c825ea6b2b Add pybgpranking to install deps
The python module DomainClassifier requires pybgpranking which is not
available in the community PyPI repository. This commit installs pybgpranking
from git using the same version required by DomainClassifier https://github.com/adulau/DomainClassifier

Fixes the following installation error when using installing_deps.sh:
```
Could not find a version that satisfies the requirement pybgpranking (from DomainClassifier->-r pip3_packages_requirement.txt (line 50)) (from versions: )
No matching distribution found for pybgpranking (from DomainClassifier->-r pip3_packages_requirement.txt (line 50))
```
2019-03-10 22:40:00 -05:00
Terrtia
d3b3942ad7
chg: [Flask UI] refractor crawler frontend + show crawled domain 2019-03-06 15:05:58 +01:00
Terrtia
6f5f880b7d
chg: [Crawler UI] add basic tooltips 2019-03-01 11:40:43 +01:00
Terrtia
93c7db86e4
chg: [Crawler UI Flask] refractor functions, templates 2019-02-28 14:56:14 +01:00
Terrtia
460f70fee6
chg: [css] add basic toggle sidebar 2019-02-27 16:58:17 +01:00
Terrtia
aaf4712877
fix: [UI crawler manual] fix min input 2019-02-26 15:39:52 +01:00
Terrtia
c0d72e7d2a
chg: [Crawler UI] Crawler major refractor (end) + basic UI for manual crawler 2019-02-26 14:50:48 +01:00
Terrtia
7b32d7f34e
chg: [Crawler] major refractor 2019-02-25 16:38:50 +01:00
Thirion Aurélien
77e4078f3d
Merge pull request #333 from 5thColumn/fix_modulesinformationv2
Fix current_selected amount and action vars
2019-02-25 08:53:03 +01:00
Jason Hedden
9eae25bdf5 Fix current_selected amount and action vars
This commit fixes issue #268

```
  File "./bin/ModulesInformationV2.py", line 319, in _setValue
    action = current_selected_action + " " + str(current_selected_amount) + "x"
```

`current_selected_action` was defined twice. The second occurance should be `current_selected_amount`.

This also fixes the generic exception from failing when `current_selected_value` is not defined.
2019-02-22 12:39:27 -06:00
Terrtia
60f7645ac1
chg: [Crawler] refractor 2019-02-22 17:00:24 +01:00
Terrtia
3d36ddbc1e
fix: [CVE] fix stat regex 2019-02-21 10:51:05 +01:00
Terrtia
1114aa2ffd
chg: [CVE] add stat script 2019-02-21 09:58:10 +01:00
Terrtia
e5dca268a8
chg: [Crawler] refractor 2019-02-21 09:54:43 +01:00