AIL/ail-framework
7
0
Fork 0
mirror of https://github.com/ail-project/ail-framework.git synced 2024-11-27 00:07:16 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

chg: [HOWTO] add a warning regarding the Tor configuration and SOCKS5

Browse source 
proxy
This commit is contained in:
Alexandre Dulaunoy 2018-10-02 17:22:21 +02:00
parent 4dde62f26e
commit 033aa5ec71
Signed by: adulau
GPG key ID: 09E2CD4944E6CBCD
1 changed files with 7 additions and 5 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

12
HOWTO.md
View file

@ -51,9 +51,9 @@ If you want to add a new webpage for a module in AIL, follow these simple steps:
3. Edit the created html files under the template folder as well as the Flask_* python script so that they fit your needs. 3. Edit the created html files under the template folder as well as the Flask_* python script so that they fit your needs.
4. You can change the order of your module in the top navigation header in the file [./var/www/templates/header_base.html](./var/www/templates/header_base.html) 4. You can change the order of your module in the top navigation header in the file [./var/www/templates/header_base.html](./var/www/templates/header_base.html)
5. You can ignore module, and so, not display them in the top navigation header by adding the module name in the file [./var/www/templates/ignored_modules.txt](./var/www/templates/ignored_modules.txt) 5. You can ignore module, and so, not display them in the top navigation header by adding the module name in the file [./var/www/templates/ignored_modules.txt](./var/www/templates/ignored_modules.txt)
How to contribute a module How to contribute a module
-------------------------- --------------------------
@ -100,7 +100,8 @@ In AIL, you can track terms, set of terms and even regexes without creating a de
Crawler Crawler
--------------------- ---------------------
In AIL, you can crawl hidden services.
In AIL, you can crawl Tor hidden services. Don't forget to review the proxy configuration of your Tor client and especially if you enabled the SOCKS5 proxy and binding on the appropriate IP address reachable via the dockers where Splash runs.
There are two types of installation. You can install a *local* or a *remote* Splash server. There are two types of installation. You can install a *local* or a *remote* Splash server.
``(Splash host) = the server running the splash service`` ``(Splash host) = the server running the splash service``
@ -110,7 +111,7 @@ There are two types of installation. You can install a *local* or a *remote* Spl
1. *(Splash host)* Launch ``crawler_hidden_services_install.sh`` to install all requirements (type ``y`` if a localhost splah server is used or use the ``-y`` option) 1. *(Splash host)* Launch ``crawler_hidden_services_install.sh`` to install all requirements (type ``y`` if a localhost splah server is used or use the ``-y`` option)
2. *(Splash host)* To install and setup your tor proxy: 2. *(Splash host)* To install and setup your tor proxy:
- Install the tor proxy: ``sudo apt-get install tor -y`` - Install the tor proxy: ``sudo apt-get install tor -y``
(Not required if ``Splah host == AIL host`` - The tor proxy is installed by default in AIL) (Not required if ``Splah host == AIL host`` - The tor proxy is installed by default in AIL)
- Add the following line ``SOCKSPolicy accept 172.17.0.0/16`` in ``/etc/tor/torrc`` - Add the following line ``SOCKSPolicy accept 172.17.0.0/16`` in ``/etc/tor/torrc``
@ -126,7 +127,7 @@ There are two types of installation. You can install a *local* or a *remote* Spl
### Starting the scripts ### Starting the scripts
- *(Splash host)* Launch all Splash servers with: - *(Splash host)* Launch all Splash servers with:
```sudo ./bin/torcrawler/launch_splash_crawler.sh -f <config absolute_path> -p <port_start> -n <number_of_splash>``` ```sudo ./bin/torcrawler/launch_splash_crawler.sh -f <config absolute_path> -p <port_start> -n <number_of_splash>```
With ``<port_start>`` and ``<number_of_splash>`` matching those specified at ``splash_onion_port`` in the configuration file of point 3 (``/bin/packages/config.cfg``) With ``<port_start>`` and ``<number_of_splash>`` matching those specified at ``splash_onion_port`` in the configuration file of point 3 (``/bin/packages/config.cfg``)
@ -137,6 +138,7 @@ All Splash dockers are launched inside the ``Docker_Splash`` screen. You can use
### TL;DR - Local setup ### TL;DR - Local setup
#### Installation #### Installation
- ```crawler_hidden_services_install.sh -y``` - ```crawler_hidden_services_install.sh -y```
- Add the following line in ``SOCKSPolicy accept 172.17.0.0/16`` in ``/etc/tor/torrc`` - Add the following line in ``SOCKSPolicy accept 172.17.0.0/16`` in ``/etc/tor/torrc``