chg: [README.md] update

README.md

@@ -1,5 +1,4 @@
-AIL
-===
+# AIL framework
 
 <p align="center">
 <img src="https://raw.githubusercontent.com/ail-project/ail-framework/master/var/www/static/image/ail-icon.png" height="250" />
@@ -34,53 +33,49 @@ AIL framework - Framework for Analysis of Information Leaks
 
 AIL is a modular framework to analyse potential information leaks from unstructured data sources like pastes from Pastebin or similar services or unstructured data streams. AIL framework is flexible and can be extended to support other functionalities to mine or process sensitive information (e.g. data leak prevention).
 
-![Dashboard](./doc/screenshots/dashboard.png?raw=true "AIL framework dashboard")
+![Dashboard](./doc/screenshots/dashboard0.png?raw=true "AIL framework dashboard")
 
 
-![Finding webshells with AIL](./doc/screenshots/webshells.gif?raw=true "Finding websheels with AIL")
+![Finding webshells with AIL](./doc/screenshots/webshells.gif?raw=true "Finding webshells with AIL")
 
-Features
---------
+## Features
 
-* Modular architecture to handle streams of unstructured or structured information
-* Default support for external ZMQ feeds, such as provided by CIRCL or other providers
-* Multiple feed support
-* Each module can process and reprocess the information already processed by AIL
-* Detecting and extracting URLs including their geographical location (e.g. IP address location)
-* Extracting and validating potential leaks of credit card numbers, credentials, ...
-* Extracting and validating leaked email addresses, including DNS MX validation
-* Module for extracting Tor .onion addresses (to be further processed for analysis)
-* Keep tracks of duplicates (and diffing between each duplicate found)
-* Extracting and validating potential hostnames (e.g. to feed Passive DNS systems)
-* A full-text indexer module to index unstructured information
-* Statistics on modules and web
-* Real-time modules manager in terminal
-* Global sentiment analysis for each providers based on nltk vader module
-* Terms, Set of terms and Regex tracking and occurrence
-* Many more modules for extracting phone numbers, credentials and others
-* Alerting to [MISP](https://github.com/MISP/MISP) to share found leaks within a threat intelligence platform using [MISP standard](https://www.misp-project.org/objects.html#_ail_leak)
-* Detect and decode encoded file (Base64, hex encoded or your own decoding scheme) and store files
-* Detect Amazon AWS and Google API keys
-* Detect Bitcoin address and Bitcoin private keys
-* Detect private keys, certificate, keys (including SSH, OpenVPN)
-* Detect IBAN bank accounts
-* Tagging system with [MISP Galaxy](https://github.com/MISP/misp-galaxy) and [MISP Taxonomies](https://github.com/MISP/misp-taxonomies) tags
-* UI paste submission
-* Create events on [MISP](https://github.com/MISP/MISP) and cases on [The Hive](https://github.com/TheHive-Project/TheHive)
-* Automatic paste export at detection on [MISP](https://github.com/MISP/MISP) (events) and [The Hive](https://github.com/TheHive-Project/TheHive) (alerts) on selected tags
-* Extracted and decoded files can be searched by date range, type of file (mime-type) and encoding discovered
-* Graph relationships between decoded file (hashes), similar PGP UIDs and addresses of cryptocurrencies
-* Tor hidden services crawler to crawl and parse output
-* Tor onion availability is monitored to detect up and down of hidden services
-* Browser hidden services are screenshot and integrated in the analysed output including a blurring screenshot interface (to avoid "burning the eyes" of the security analysis with specific content)
-* Tor hidden services is part of the standard framework, all the AIL modules are available to the crawled hidden services
-* Generic web crawler to trigger crawling on demand or at regular interval URL or Tor hidden services
+- Modular architecture to handle streams of unstructured or structured information
+- Default support for external ZMQ feeds, such as provided by CIRCL or other providers
+- Multiple Importers and feeds support
+- Each module can process and reprocess the information already analyzed by AIL
+- Detecting and extracting URLs including their geographical location (e.g. IP address location)
+- Extracting and validating potential leaks of credit card numbers, credentials, ...
+- Extracting and validating leaked email addresses, including DNS MX validation
+- Module for extracting Tor .onion addresses for further analysis
+- Keep tracks of credentials duplicates (and diffing between each duplicate found)
+- Extracting and validating potential hostnames (e.g. to feed Passive DNS systems)
+- A full-text indexer module to index unstructured information
+- Terms, Set of terms, Regex, typo squatting and YARA tracking and occurrence
+- YARA Retro Hunt
+- Many more modules for extracting phone numbers, credentials, and more
+- Alerting to [MISP](https://github.com/MISP/MISP) to share found leaks within a threat intelligence platform using [MISP standard](https://www.misp-project.org/objects.html#_ail_leak)
+- Detecting and decoding encoded file (Base64, hex encoded or your own decoding scheme) and storing files
+- Detecting Amazon AWS and Google API keys
+- Detecting Bitcoin address and Bitcoin private keys
+- Detecting private keys, certificate, keys (including SSH, OpenVPN)
+- Detecting IBAN bank accounts
+- Tagging system with [MISP Galaxy](https://github.com/MISP/misp-galaxy) and [MISP Taxonomies](https://github.com/MISP/misp-taxonomies) tags
+- UI submission
+- Create events on [MISP](https://github.com/MISP/MISP) and cases on [The Hive](https://github.com/TheHive-Project/TheHive)
+- Automatic export on detection with [MISP](https://github.com/MISP/MISP) (events) and [The Hive](https://github.com/TheHive-Project/TheHive) (alerts) on selected tags
+- Extracted and decoded files can be searched by date range, type of file (mime-type) and encoding discovered
+- Correlations engine and Graph to visualize relationships between decoded files (hashes), PGP UIDs, domains, username, and cryptocurrencies addresses
+- Websites, Forums and Tor Hidden-Services hidden services crawler to crawl and parse output
+- Domain availability monitoring to detect up and down of websites and hidden services
+- Browsed hidden services are automatically captured and integrated into the analyzed output, including a blurring screenshot interface (to avoid "burning the eyes" of security analysts with sensitive content)
+- Tor hidden services is part of the standard framework, all the AIL modules are available to the crawled hidden services
+- Crawler scheduler to trigger crawling on demand or at regular intervals for URLs or Tor hidden services
 
 
-Installation
-------------
+## Installation
 
-Type these command lines for a fully automated installation and start AIL framework:
+To install the AIL framework, run the following commands:
 ```bash
 # Clone the repo first
 git clone https://github.com/ail-project/ail-framework.git
@@ -89,10 +84,6 @@ cd ail-framework
 # For Debian and Ubuntu based distributions
 ./installing_deps.sh
 
-# For Centos based distributions (Tested: Centos 8)
-chmod u+x centos_installing_deps.sh
-./centos_installing_deps.sh
-
 # Launch ail
 cd ~/ail-framework/
 cd bin/
@@ -101,59 +92,52 @@ cd bin/
 
 The default [installing_deps.sh](./installing_deps.sh) is for Debian and Ubuntu based distributions.
 
-There is also a [Travis file](.travis.yml) used for automating the installation that can be used to build and install AIL on other systems.
-
 Requirement:
-- Python 3.6+
+- Python 3.7+
 
-Installation Notes
-------------
+## Installation Notes
 
-In order to use AIL combined with **ZFS** or **unprivileged LXC** it's necessary to disable Direct I/O in `$AIL_HOME/configs/6382.conf` by changing the value of the directive `use_direct_io_for_flush_and_compaction` to `false`.
+For Lacus Crawler installation instructions, refer to the [HOWTO](https://github.com/ail-project/ail-framework/blob/master/HOWTO.md#crawler)
 
-Tor installation instructions can be found in the [HOWTO](https://github.com/ail-project/ail-framework/blob/master/HOWTO.md#installationconfiguration)
+## Starting AIL
 
-Starting AIL
---------------------------
+To start AIL, use the following commands:
 
 ```bash
 cd bin/
 ./LAUNCH.sh -l
 ```
 
-Eventually you can browse the status of the AIL framework website at the following URL:
+You can access the AIL framework web interface at the following URL: 
 
 ```
 https://localhost:7000/
 ```
 
-The default credentials for the web interface are located in ``DEFAULT_PASSWORD``. This file is removed when you change your password.
+The default credentials for the web interface are located in the ``DEFAULT_PASSWORD``file, which is deleted when you change your password.
 
-Training
---------
+## Training
 
-CIRCL organises training on how to use or extend the AIL framework. AIL training materials are available at [https://www.circl.lu/services/ail-training-materials/](https://www.circl.lu/services/ail-training-materials/).
+CIRCL organises training on how to use or extend the AIL framework. AIL training materials are available at [https://github.com/ail-project/ail-training](https://github.com/ail-project/ail-training).
 
 
-API
------
+## API
 
 The API documentation is available in [doc/README.md](doc/README.md)
 
-HOWTO
------
+## HOWTO
 
 HOWTO are available in [HOWTO.md](HOWTO.md)
 
-Privacy and GDPR
-----------------
+## Privacy and GDPR
 
-[AIL information leaks analysis and the GDPR in the context of collection, analysis and sharing information leaks](https://www.circl.lu/assets/files/information-leaks-analysis-and-gdpr.pdf) document provides an overview how to use AIL in a lawfulness context especially in the scope of General Data Protection Regulation.
+For information on AIL's compliance with GDPR and privacy considerations, refer to the [AIL information leaks analysis and the GDPR in the context of collection, analysis and sharing information leaks](https://www.circl.lu/assets/files/information-leaks-analysis-and-gdpr.pdf) document.
 
-Research using AIL
-------------------
+this document provides an overview how to use AIL in a lawfulness context especially in the scope of General Data Protection Regulation.
 
-If you write academic paper, relying or using AIL, it can be cited with the following BibTeX:
+## Research using AIL
+
+If you use or reference AIL in an academic paper, you can cite it using the following BibTeX:
 
 ~~~~
 @inproceedings{mokaddem2018ail,
@@ -166,75 +150,66 @@ If you write academic paper, relying or using AIL, it can be cited with the foll
 }
 ~~~~
 
-Screenshots
-===========
+## Screenshots
 
 
-Tor hidden service crawler
---------------------------
+### Websites, Forums and Tor Hidden-Services
 
-![Tor hidden service](./doc/screenshots/ail-bitcoinmixer.png?raw=true "Tor hidden service crawler")
+![Domain CIRCL](./doc/screenshots/domain_circl.png?raw=true "Tor hidden service crawler")
 
-Trending charts
----------------
+#### Login protected, pre-recorded session cookies:
+![Domain cookiejar](./doc/screenshots/crawler-cookiejar-domain-crawled.png?raw=true "Tor hidden service crawler")
 
-![Trending-Modules](./doc/screenshots/trending-module.png?raw=true "AIL framework modulestrending")
+### Extracted encoded files from items
 
-Extracted encoded files from pastes
------------------------------------
+![Extracted files](./doc/screenshots/decodeds_dashboard.png?raw=true "AIL extracted decoded files statistics")
 
-![Extracted files from pastes](./doc/screenshots/ail-hashedfiles.png?raw=true "AIL extracted decoded files statistics")
-![Relationships between extracted files from encoded file in unstructured data](./doc/screenshots/hashedfile-graph.png?raw=true "Relationships between extracted files from encoded file in unstructured data")
+### Correlation Engine
 
-Browsing
---------
+![Correlation decoded image](./doc/screenshots/correlation_decoded_image.png?raw=true "Correlation decoded image")
 
-![Browse-Pastes](./doc/screenshots/browse-important.png?raw=true "AIL framework browseImportantPastes")
+### Investigation
 
-Tagging system
---------
+![Investigation](./doc/screenshots/investigation_mixer.png?raw=true "AIL framework cookiejar")
 
-![Tags](./doc/screenshots/tags.png?raw=true "AIL framework tags")
+### Tagging system
 
-MISP and The Hive, automatic events and alerts creation
---------
+![Tags](./doc/screenshots/tags_search.png?raw=true "AIL framework tags")
 
-![paste_submit](./doc/screenshots/tag_auto_export.png?raw=true "AIL framework MISP and Hive auto export")
+![Tags search](./doc/screenshots/tags_search_items.png?raw=true "AIL framework tags items search")
 
-Paste submission
---------
+### MISP Export
 
-![paste_submit](./doc/screenshots/paste_submit.png?raw=true "AIL framework paste submission")
+![misp_export](./doc/screenshots/misp_export.png?raw=true "AIL framework MISP Export")
 
-Sentiment analysis
-------------------
+### MISP and The Hive, automatic events and alerts creation
 
-![Sentiment](./doc/screenshots/sentiment.png?raw=true "AIL framework sentimentanalysis")
+![tags_misp_auto](./doc/screenshots/tags_misp_auto.png?raw=true "AIL framework MISP and Hive auto export")
 
-Terms tracker
----------------------------
+### UI submission
 
-![Term-tracker](./doc/screenshots/term-tracker.png?raw=true "AIL framework termManager")
+![ui_submit](./doc/screenshots/ui_submit.png?raw=true "AIL framework UI importer")
 
+### Trackers
+
+![tracker-create](./doc/screenshots/tracker_create.png?raw=true "AIL framework create tracker")
+
+![tracker-yara](./doc/screenshots/tracker_yara.png?raw=true "AIL framework Yara tracker")
+
+![retro-hunt](./doc/screenshots/retro_hunt.png?raw=true "AIL framework Retro Hunt")
 
 [AIL framework screencast](https://www.youtube.com/watch?v=1_ZrZkRKmNo)
 
-Command line module manager
----------------------------
-
-![Module-Manager](./doc/screenshots/module_information.png?raw=true "AIL framework ModuleInformationV2.py")
-
-License
-=======
+## License
 
 ```
     Copyright (C) 2014 Jules Debra
-    Copyright (C) 2014-2021 CIRCL - Computer Incident Response Center Luxembourg (c/o smile, security made in Lëtzebuerg, Groupement d'Intérêt Economique)
-    Copyright (c) 2014-2021 Raphaël Vinot
-    Copyright (c) 2014-2021 Alexandre Dulaunoy
-    Copyright (c) 2016-2021 Sami Mokaddem
-    Copyright (c) 2018-2021 Thirion Aurélien
     Copyright (c) 2021 Olivier Sagit
+    Copyright (C) 2014-2023 CIRCL - Computer Incident Response Center Luxembourg (c/o smile, security made in Lëtzebuerg, Groupement d'Intérêt Economique)
+    Copyright (c) 2014-2023 Raphaël Vinot
+    Copyright (c) 2014-2023 Alexandre Dulaunoy
+    Copyright (c) 2016-2023 Sami Mokaddem
+    Copyright (c) 2018-2023 Thirion Aurélien
 
     This program is free software: you can redistribute it and/or modify
     it under the terms of the GNU Affero General Public License as published by

doc/SourceCode.info

@@ -1,8 +0,0 @@
-SourceCode listens to Global and select only keywords that are relevants to AIL's purpose (CVE, Exploits, Vulnerability,...), then send matching file to a new queue.
-
-SourceCode.py search for differents languages such as C, PHP, Python, BASH and some Unix shells with default configuration.
-
-Every records is send to the warning log because filters are high enough (hence the critical var set to 0 but can be changed).
-
-FOR NOW : Still have troubles detecting ASM 
-

doc/statistics/create_graph_by_tld.py

@@ -1,184 +0,0 @@
-#!/usr/bin/env python3
-# -*-coding:UTF-8 -*
-
-'''
-Create statistics pie charts by tld
-
-Default tld: lu
-'''
-
-import matplotlib
-matplotlib.use('Agg')
-import os
-import sys
-import redis
-import argparse
-import datetime
-import heapq
-import operator
-import matplotlib.pyplot as plt
-import numpy as np
-
-sys.path.append(os.environ['AIL_BIN'])
-
-from Helper import Process
-
-def create_pie_chart(country ,db_key, date, pie_title, path, save_name):
-
-    monthly_credential_by_tld = server_statistics.hkeys(db_key + date)
-
-    l_tld = []
-    for tld in monthly_credential_by_tld:
-        nb_tld = server_statistics.hget(db_key + date, tld)
-        if nb_tld is not None:
-            nb_tld = int(nb_tld)
-        else:
-            nb_tld = 0
-        l_tld.append( (tld, nb_tld) )
-
-    mail_tld_top5 = heapq.nlargest(5, l_tld, key=operator.itemgetter(1))
-
-    # Pie chart, where the slices will be ordered and plotted counter-clockwise:
-    labels = []
-    sizes = []
-    explode = []  # only "explode" the 2nd slice (i.e. 'Hogs')
-    explode_value = 0
-    for tld in mail_tld_top5:
-        labels.append(tld[0] +' ('+str(tld[1])+')')
-        sizes.append(tld[1])
-        explode.append(explode_value)
-        explode_value = explode_value +0.1
-
-    nb_tld = server_statistics.hget(db_key + date, country)
-    if nb_tld is not None:
-        nb_tld = int(nb_tld)
-    else:
-        nb_tld = 0
-    country_label = country + ' ('+str(nb_tld)+')'
-    if country_label not in labels:
-        labels.append(country_label)
-        sizes.append(nb_tld)
-        explode.append(explode_value)
-    explode = tuple(explode)
-
-    fig1, ax1 = plt.subplots()
-    ax1.pie(sizes, explode=explode, labels=labels, autopct='%1.1f%%',
-            shadow=True, startangle=90)
-    ax1.axis('equal')  # Equal aspect ratio ensures that pie is drawn as a circle.
-
-    ax1.set_title(pie_title)
-    #plt.show()
-    plt.savefig(os.path.join(path,save_name))
-    plt.close(fig1)
-
-def create_donut_chart(db_key, date, pie_title, path, save_name):
-
-    monthly_credential_by_tld = server_statistics.hkeys(db_key + date)
-    print()
-
-    l_tld = []
-    for tld in monthly_credential_by_tld:
-        nb_tld = server_statistics.hget(db_key + date, tld)
-        if nb_tld is not None:
-            nb_tld = int(nb_tld)
-        else:
-            nb_tld = 0
-        l_tld.append( (tld, nb_tld) )
-
-    mail_tld_top5 = heapq.nlargest(5, l_tld, key=operator.itemgetter(1))
-
-    # Pie chart, where the slices will be ordered and plotted counter-clockwise:
-    recipe = []
-    data = []
-    for tld in mail_tld_top5:
-        recipe.append(tld[0])
-        data.append(tld[1])
-
-    nb_tld = server_statistics.hget(db_key + date, country)
-    if nb_tld is not None:
-        nb_tld = int(nb_tld)
-    else:
-        nb_tld = 0
-    if country not in recipe:
-        recipe.append(country)
-        data.append(nb_tld)
-
-    fig1, ax1 = plt.subplots(figsize=(6, 3), subplot_kw=dict(aspect="equal"))
-
-    wedges, texts = ax1.pie(data, wedgeprops=dict(width=0.5), startangle=-40)
-
-    bbox_props = dict(boxstyle="square,pad=0.3", fc="w", ec="k", lw=0.72)
-    kw = dict(xycoords='data', textcoords='data', arrowprops=dict(arrowstyle="-"),
-              bbox=bbox_props, zorder=0, va="center")
-
-    for i, p in enumerate(wedges):
-        ang = (p.theta2 - p.theta1)/2. + p.theta1
-        y = np.sin(np.deg2rad(ang))
-        x = np.cos(np.deg2rad(ang))
-        horizontalalignment = {-1: "right", 1: "left"}[int(np.sign(x))]
-        connectionstyle = "angle,angleA=0,angleB={}".format(ang)
-        kw["arrowprops"].update({"connectionstyle": connectionstyle})
-        ax1.annotate(recipe[i], xy=(x, y), xytext=(1.35*np.sign(x), 1.4*y),
-                     horizontalalignment=horizontalalignment, **kw)
-
-    ax1.set_title(pie_title)
-    #plt.show()
-    plt.savefig(os.path.join(path, save_name))
-    plt.close(fig1)
-
-if __name__ == '__main__':
-
-    parser = argparse.ArgumentParser(
-        description='''This script is a part of the Analysis Information Leak
-        framework. Create statistics pie charts".''',
-        epilog='Example: ./create_lu_graph.py 0 lu now, create_lu_graph.py 0 lu 201807')
-
-    parser.add_argument('type', type=int, default=0,
-                        help='''The graph type (default 0),
-                        0: all,
-                        1: credential_pie,
-                        2: mail_pie
-                        3: sqlinjection_pie,
-                        4: iban_pie,''',
-                        choices=[0, 1, 2, 3, 4], action='store')
-
-    parser.add_argument('country', type=str, default="lu",
-                        help='''The country code, lu:default''',
-                        action='store')
-
-    parser.add_argument('date', type=str, default="now",
-                        help='''month %Y%m, example: 201810''', action='store')
-
-    args = parser.parse_args()
-
-    path = os.path.join(os.environ['AIL_HOME'], 'doc', 'statistics') # save path
-
-    config_section = 'ARDB_Statistics'
-
-    p = Process(config_section, False)
-
-    # ARDB #
-    server_statistics = redis.StrictRedis(
-        host=p.config.get("ARDB_Statistics", "host"),
-        port=p.config.getint("ARDB_Statistics", "port"),
-        db=p.config.getint("ARDB_Statistics", "db"),
-        decode_responses=True)
-
-    if args.date == 'now' or len(args.date) != 6:
-        date = datetime.datetime.now().strftime("%Y%m")
-    else:
-        date = args.date
-
-    if args.type == 0:
-        create_pie_chart(args.country, 'credential_by_tld:', date, "AIL: Credential leak by tld", path, 'AIL_credential_by_tld.png')
-        create_pie_chart(args.country, 'mail_by_tld:', date, "AIL: mail leak by tld", path, 'AIL_mail_by_tld.png')
-        create_pie_chart(args.country, 'SQLInjection_by_tld:', date, "AIL: SQLInjection by tld", path, 'AIL_SQLInjection_by_tld.png')
-        create_pie_chart(args.country.upper(), 'iban_by_country:', date, "AIL: Iban by country", path, 'AIL_iban_by_country.png')
-    elif args.type == 1:
-        create_pie_chart(args.country, 'credential_by_tld:', date, "AIL: Credential leak by tld", path, 'AIL_credential_by_tld.png')
-    elif args.type == 2:
-        create_pie_chart(args.country, 'mail_by_tld:', date, "AIL: mail leak by tld", path, 'AIL_mail_by_tld.png')
-    elif args.type == 3:
-        create_pie_chart(args.country, 'SQLInjection_by_tld:', date, "AIL: sqlInjection by tld", path, 'AIL_sqlInjectionl_by_tld.png')
-    elif args.type == 4:
-        create_pie_chart(args.country.upper(), 'iban_by_country:', date, "AIL: Iban by country", path, 'AIL_iban_by_country.png')