diff --git a/README.md b/README.md index 0dd7a325..95398f3b 100644 --- a/README.md +++ b/README.md @@ -9,8 +9,34 @@ AIL framework - Framework for Analysis of Information Leaks AIL is a modular framework to analyse potential information leaks from unstructured data sources like pastes from Pastebin or similar services. AIL framework is flexible and can be extended to support other functionalities to mine sensitive information. -![Dashboard](./doc/screenshots/DashboardAIL.png?raw=true "AIL framework dashboard") -![Trending](./doc/screenshots/WordtrendingAIL.png?raw=true "AIL framework wordtrending") +![Dashboard](./doc/screenshots/dashboard.png?raw=true "AIL framework dashboard") + +Trending charts +--------------- + +![Trending-Web](./doc/screenshots/trending-web.png?raw=true "AIL framework webtrending") +![Trending-Modules](./doc/screenshots/trending-module.png?raw=true "AIL framework modulestrending") + +Browsing +-------- + +![Browse-Pastes](./doc/screenshots/browse-important.png?raw=true "AIL framework browseImportantPastes") + +Sentiment analysis +------------------ + +![Sentiment](./doc/screenshots/sentiment.png?raw=true "AIL framework sentimentanalysis") + +Terms manager and occurence +--------------------------- + +![Term-Manager](./doc/screenshots/terms-manager.png?raw=true "AIL framework termManager") + +## Top terms + +![Term-Top](./doc/screenshots/terms-top.png?raw=true "AIL framework termTop") +![Term-Plot](./doc/screenshots/terms-plot.png?raw=true "AIL framework termPlot") + AIL framework screencast: https://www.youtube.com/watch?v=9idfHCIMzBY @@ -26,6 +52,9 @@ Features * Module for extracting Tor .onion addresses (to be further processed for analysis) * Extracting and validating potential hostnames (e.g. to feed Passive DNS systems) * A full-text indexer module to index unstructured information +* Modules and web statistics +* Global sentiment analysis for each providers based on nltk vader module +* Terms tracking and occurence * Many more modules for extracting phone numbers, credentials and others Installation @@ -48,6 +77,7 @@ linux based distributions, you can replace it with [installing_deps_archlinux.sh There is also a [Travis file](.travis.yml) used for automating the installation that can be used to build and install AIL on other systems. + Starting AIL web interface -------------------------- @@ -94,6 +124,7 @@ Redis and LevelDB overview * DB 0 - Cache hostname/dns * Redis on TCP port 6380 - Redis Pub-Sub only * Redis on TCP port 6381 - DB 0 - Queue and Paste content LRU cache +* Redis on TCP port 6382 - DB 1-4 - Trending, terms and sentiments * LevelDB on TCP port - Lines duplicate LICENSE diff --git a/doc/screenshots/browse-important.png b/doc/screenshots/browse-important.png new file mode 100644 index 00000000..0490c03e Binary files /dev/null and b/doc/screenshots/browse-important.png differ diff --git a/doc/screenshots/dashboard.png b/doc/screenshots/dashboard.png new file mode 100644 index 00000000..cd633473 Binary files /dev/null and b/doc/screenshots/dashboard.png differ diff --git a/doc/screenshots/sentiment.png b/doc/screenshots/sentiment.png new file mode 100644 index 00000000..c4edee90 Binary files /dev/null and b/doc/screenshots/sentiment.png differ diff --git a/doc/screenshots/terms-manager.png b/doc/screenshots/terms-manager.png new file mode 100644 index 00000000..15d76804 Binary files /dev/null and b/doc/screenshots/terms-manager.png differ diff --git a/doc/screenshots/terms-plot.png b/doc/screenshots/terms-plot.png new file mode 100644 index 00000000..32647906 Binary files /dev/null and b/doc/screenshots/terms-plot.png differ diff --git a/doc/screenshots/terms-top.png b/doc/screenshots/terms-top.png new file mode 100644 index 00000000..e833ac35 Binary files /dev/null and b/doc/screenshots/terms-top.png differ diff --git a/doc/screenshots/trending-module.png b/doc/screenshots/trending-module.png new file mode 100644 index 00000000..1330ead8 Binary files /dev/null and b/doc/screenshots/trending-module.png differ diff --git a/doc/screenshots/trending-web.png b/doc/screenshots/trending-web.png new file mode 100644 index 00000000..cd7e510b Binary files /dev/null and b/doc/screenshots/trending-web.png differ diff --git a/installing_deps.sh b/installing_deps.sh index 805818e1..6af62795 100755 --- a/installing_deps.sh +++ b/installing_deps.sh @@ -83,5 +83,6 @@ pushd tlsh/py_ext python setup.py build python setup.py install -# Download the necessary NLTK corpora +# Download the necessary NLTK corpora and sentiment vader HOME=$(pwd) python -m textblob.download_corpora +python -m nltk.downloader vader_lexicon