chg: [doc] add AIL v5.0 + objects + Importers + sync

This commit is contained in:
Terrtia 2023-06-05 16:14:29 +02:00
parent f3c3cb5d05
commit 683d52dfb8
No known key found for this signature in database
GPG key ID: 1E1B1F50D84613D0
5 changed files with 1539 additions and 1352 deletions

View file

@ -1,73 +1,18 @@
# Feeding, adding new features and contributing
## [Documentation AIL Importers](./doc/README.md#ail-importers)
[Documentation AIL Importers](./doc/README.md#ail-importers)
## How to feed the AIL framework
Currently, there are three different ways to feed data into AIL:
1. Be a collaborator of CIRCL and ask to access our feed. It will be sent to the static IP you are using for AIL.
2. You can setup [pystemon](https://github.com/cvandeplas/pystemon) and use the custom feeder provided by AIL (see below).
3. You can feed your own data using the [./tool/file_dir_importer.py](./tool/file_dir_importer.py) script.
### Feeding AIL with pystemon
AIL is an analysis tool, not a collector!
However, if you want to collect some pastes and feed them to AIL, the procedure is described below. Nevertheless, moderate your queries!
Feed data to AIL:
1. [AIL Importers](./doc/README.md#ail-importers)
1. Clone the [pystemon's git repository](https://github.com/cvandeplas/pystemon):
```
git clone https://github.com/cvandeplas/pystemon.git
```
2. Edit configuration file for pystemon ```pystemon/pystemon.yaml```:
- Configure the storage section according to your needs:
```
storage:
archive:
storage-classname: FileStorage
save: yes
save-all: yes
dir: "alerts"
dir-all: "archive"
compress: yes
redis:
storage-classname: RedisStorage
save: yes
save-all: yes
server: "localhost"
port: 6379
database: 10
lookup: no
```
- Adjust the configuration for paste-sites based on your requirements (remember to throttle download and update times).
3. Install python dependencies inside the virtual environment:
```shell
cd ail-framework/
. ./AILENV/bin/activate
cd pystemon/
pip install -U -r requirements.txt
```
4. Edit the configuration file ```ail-framework/configs/core.cfg```:
- Modify the "pystemonpath" path accordingly.
5. Launch ail-framework, pystemon and PystemonImporter.py (all within the virtual environment):
- Option 1 (recommended):
```
./ail-framework/bin/LAUNCH.py -l #starts ail-framework
./ail-framework/bin/LAUNCH.py -f #starts pystemon and the PystemonImporter.py
```
- Option 2 (may require two terminal windows):
```
./ail-framework/bin/LAUNCH.py -l #starts ail-framework
./pystemon/pystemon.py
./ail-framework/bin/importer/PystemonImporter.py
```
2. ZMQ: Be a collaborator of CIRCL and ask to access our feed. It will be sent to the static IP you are using for AIL.
## How to create a new module

View file

@ -34,6 +34,25 @@ AIL is a modular framework to analyse potential information leaks from unstructu
![Finding webshells with AIL](./doc/screenshots/webshells.gif?raw=true "Finding webshells with AIL")
## AIL V5.0 Version:
AIL v5.0 introduces significant improvements and new features:
- **Codebase Rewrite**: The codebase has undergone a substantial rewrite,
resulting in enhanced performance and speed improvements.
- **Database Upgrade**: The database has been migrated from ARDB to Kvrocks.
- **New Correlation Engine**: AIL v5.0 introduces a new powerful correlation engine with two new correlation types: CVE and Title.
- **Enhanced Logging**: The logging system has been improved to provide better troubleshooting capabilities.
- **Tagging Support**: [AIL objects](./doc/README.md#ail-objects) now support tagging,
allowing users to categorize and label extracted information for easier analysis and organization.
- **Trackers**: Improved objects filtering, PGP and decoded tracking added.
- **UI Content Visualization**: The user interface has been upgraded to visualize extracted and tracked information.
- **New Crawler Lacus**: improve crawling capabilities.
- **Modular Importers and Exporters**: New importers (ZMQ, AIL Feeders) and exporters (MISP, Mail, TheHive) modular design.
Allow easy creation and customization by extending an abstract class.
- **Module Queues**: improved the queuing mechanism between detection modules.
- **New Object CVE and Title**: Extract an correlate CVE IDs and web page titles.
## Features
- Modular architecture to handle streams of unstructured or structured information

File diff suppressed because it is too large Load diff

BIN
doc/ail_modules_queues.png Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 143 KiB

1295
doc/api.md Normal file

File diff suppressed because it is too large Load diff