mirror of
https://github.com/ail-project/ail-framework.git
synced 2024-11-10 00:28:22 +00:00
chg: [doc] add AIL v5.0 + objects + Importers + sync
This commit is contained in:
parent
f3c3cb5d05
commit
683d52dfb8
5 changed files with 1539 additions and 1352 deletions
67
HOWTO.md
67
HOWTO.md
|
@ -1,73 +1,18 @@
|
|||
|
||||
# Feeding, adding new features and contributing
|
||||
|
||||
## [Documentation AIL Importers](./doc/README.md#ail-importers)
|
||||
|
||||
[Documentation AIL Importers](./doc/README.md#ail-importers)
|
||||
|
||||
## How to feed the AIL framework
|
||||
|
||||
Currently, there are three different ways to feed data into AIL:
|
||||
|
||||
1. Be a collaborator of CIRCL and ask to access our feed. It will be sent to the static IP you are using for AIL.
|
||||
|
||||
2. You can setup [pystemon](https://github.com/cvandeplas/pystemon) and use the custom feeder provided by AIL (see below).
|
||||
|
||||
3. You can feed your own data using the [./tool/file_dir_importer.py](./tool/file_dir_importer.py) script.
|
||||
|
||||
### Feeding AIL with pystemon
|
||||
|
||||
AIL is an analysis tool, not a collector!
|
||||
However, if you want to collect some pastes and feed them to AIL, the procedure is described below. Nevertheless, moderate your queries!
|
||||
|
||||
Feed data to AIL:
|
||||
1. [AIL Importers](./doc/README.md#ail-importers)
|
||||
|
||||
1. Clone the [pystemon's git repository](https://github.com/cvandeplas/pystemon):
|
||||
```
|
||||
git clone https://github.com/cvandeplas/pystemon.git
|
||||
```
|
||||
|
||||
2. Edit configuration file for pystemon ```pystemon/pystemon.yaml```:
|
||||
- Configure the storage section according to your needs:
|
||||
```
|
||||
storage:
|
||||
archive:
|
||||
storage-classname: FileStorage
|
||||
save: yes
|
||||
save-all: yes
|
||||
dir: "alerts"
|
||||
dir-all: "archive"
|
||||
compress: yes
|
||||
|
||||
redis:
|
||||
storage-classname: RedisStorage
|
||||
save: yes
|
||||
save-all: yes
|
||||
server: "localhost"
|
||||
port: 6379
|
||||
database: 10
|
||||
lookup: no
|
||||
```
|
||||
- Adjust the configuration for paste-sites based on your requirements (remember to throttle download and update times).
|
||||
|
||||
3. Install python dependencies inside the virtual environment:
|
||||
```shell
|
||||
cd ail-framework/
|
||||
. ./AILENV/bin/activate
|
||||
cd pystemon/
|
||||
pip install -U -r requirements.txt
|
||||
```
|
||||
4. Edit the configuration file ```ail-framework/configs/core.cfg```:
|
||||
- Modify the "pystemonpath" path accordingly.
|
||||
|
||||
5. Launch ail-framework, pystemon and PystemonImporter.py (all within the virtual environment):
|
||||
- Option 1 (recommended):
|
||||
```
|
||||
./ail-framework/bin/LAUNCH.py -l #starts ail-framework
|
||||
./ail-framework/bin/LAUNCH.py -f #starts pystemon and the PystemonImporter.py
|
||||
```
|
||||
- Option 2 (may require two terminal windows):
|
||||
```
|
||||
./ail-framework/bin/LAUNCH.py -l #starts ail-framework
|
||||
./pystemon/pystemon.py
|
||||
./ail-framework/bin/importer/PystemonImporter.py
|
||||
```
|
||||
2. ZMQ: Be a collaborator of CIRCL and ask to access our feed. It will be sent to the static IP you are using for AIL.
|
||||
|
||||
## How to create a new module
|
||||
|
||||
|
|
19
README.md
19
README.md
|
@ -34,6 +34,25 @@ AIL is a modular framework to analyse potential information leaks from unstructu
|
|||
|
||||
![Finding webshells with AIL](./doc/screenshots/webshells.gif?raw=true "Finding webshells with AIL")
|
||||
|
||||
## AIL V5.0 Version:
|
||||
|
||||
AIL v5.0 introduces significant improvements and new features:
|
||||
|
||||
- **Codebase Rewrite**: The codebase has undergone a substantial rewrite,
|
||||
resulting in enhanced performance and speed improvements.
|
||||
- **Database Upgrade**: The database has been migrated from ARDB to Kvrocks.
|
||||
- **New Correlation Engine**: AIL v5.0 introduces a new powerful correlation engine with two new correlation types: CVE and Title.
|
||||
- **Enhanced Logging**: The logging system has been improved to provide better troubleshooting capabilities.
|
||||
- **Tagging Support**: [AIL objects](./doc/README.md#ail-objects) now support tagging,
|
||||
allowing users to categorize and label extracted information for easier analysis and organization.
|
||||
- **Trackers**: Improved objects filtering, PGP and decoded tracking added.
|
||||
- **UI Content Visualization**: The user interface has been upgraded to visualize extracted and tracked information.
|
||||
- **New Crawler Lacus**: improve crawling capabilities.
|
||||
- **Modular Importers and Exporters**: New importers (ZMQ, AIL Feeders) and exporters (MISP, Mail, TheHive) modular design.
|
||||
Allow easy creation and customization by extending an abstract class.
|
||||
- **Module Queues**: improved the queuing mechanism between detection modules.
|
||||
- **New Object CVE and Title**: Extract an correlate CVE IDs and web page titles.
|
||||
|
||||
## Features
|
||||
|
||||
- Modular architecture to handle streams of unstructured or structured information
|
||||
|
|
1510
doc/README.md
1510
doc/README.md
File diff suppressed because it is too large
Load diff
BIN
doc/ail_modules_queues.png
Normal file
BIN
doc/ail_modules_queues.png
Normal file
Binary file not shown.
After Width: | Height: | Size: 143 KiB |
1295
doc/api.md
Normal file
1295
doc/api.md
Normal file
File diff suppressed because it is too large
Load diff
Loading…
Reference in a new issue