AIL/ail-framework
7
0
Fork 0
mirror of https://github.com/ail-project/ail-framework.git synced 2024-11-12 17:48:22 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

merge master

Browse source
This commit is contained in:
Terrtia 2023-05-17 16:07:32 +02:00
commit 175963e371
No known key found for this signature in database
GPG key ID: 1E1B1F50D84613D0
4 changed files with 87 additions and 87 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

65
HOWTO.md
View file

@ -78,21 +78,6 @@ If you want to add a new processing or analysis module in AIL, follow these simp
2. Use [./bin/template.py](./bin/template.py) as a sample module and create a new file in bin/ with the module name used in the modules.cfg configuration. 2. Use [./bin/template.py](./bin/template.py) as a sample module and create a new file in bin/ with the module name used in the modules.cfg configuration.
How to create a new webpage
---------------------------
If you want to add a new webpage for a module in AIL, follow these simple steps:
1. Launch [./var/www/create_new_web_module.py](./var/www/create_new_web_module.py) and enter the name to use for your webpage (Usually, your newly created python module).
2. A template and flask skeleton has been created for your new webpage in [./var/www/modules/](./var/www/modules/)
3. Edit the created html files under the template folder as well as the Flask_* python script so that they fit your needs.
4. You can change the order of your module in the top navigation header in the file [./var/www/templates/header_base.html](./var/www/templates/header_base.html)
5. You can ignore module, and so, not display them in the top navigation header by adding the module name in the file [./var/www/templates/ignored_modules.txt](./var/www/templates/ignored_modules.txt)
How to contribute a module How to contribute a module
-------------------------- --------------------------
@ -104,30 +89,6 @@ To contribute your module, feel free to pull your contribution.
Additional information Additional information
====================== ======================
Manage modules: ModulesInformationV2.py
---------------------------------------
You can do a lots of things easily with the [./bin/ModulesInformationV2](./bin/ModulesInformationV2) script:
- Monitor the health of other modules
- Monitor the ressources comsumption of other modules
- Start one or more modules
- Kill running modules
- Restart automatically stuck modules
- Show the paste currently processed by a module
### Navigation
You can navigate into the interface by using arrow keys. In order to perform an action on a selected module, you can either press <ENTER> or <SPACE> to show the dialog box.
To change list, you can press the <TAB> key.
Also, you can quickly stop or start modules by clicking on the ``<K>`` or ``<S>`` symbol respectively. These are located in the _Action_ column.
Finally, you can quit this program by pressing either ``<q>`` or ``<C-c>``.
Crawler Crawler
--------------------- ---------------------
@ -145,6 +106,7 @@ In AIL, you can crawl websites and Tor hidden services. Don't forget to review t
1. Lacus URL: 1. Lacus URL:
In the webinterface, go to ``Crawlers>Settings`` and click on the Edit button In the webinterface, go to ``Crawlers>Settings`` and click on the Edit button
![Splash Manager Config](./doc/screenshots/lacus_config.png?raw=true "AIL Lacus Config") ![Splash Manager Config](./doc/screenshots/lacus_config.png?raw=true "AIL Lacus Config")
![Splash Manager Config](./doc/screenshots/lacus_config_edit.png?raw=true "AIL Lacus Config") ![Splash Manager Config](./doc/screenshots/lacus_config_edit.png?raw=true "AIL Lacus Config")
@ -156,9 +118,26 @@ Choose the number of crawlers you want to launch
![Splash Manager Nb Crawlers Config](./doc/screenshots/crawler_nb_captures_edit.png?raw=true "AIL Lacus Nb Crawlers Config") ![Splash Manager Nb Crawlers Config](./doc/screenshots/crawler_nb_captures_edit.png?raw=true "AIL Lacus Nb Crawlers Config")
Kvrocks Migration
---------------------
**Important Note:
We are currently working on a [migration script](https://github.com/ail-project/ail-framework/blob/master/bin/DB_KVROCKS_MIGRATION.py) to facilitate the migration to Kvrocks.
Once this script is ready, AIL version 5.0 will be released.**
#### Old updates Please note that the current version of this migration script only supports migrating the database on the same server.
(If you plan to migrate to another server, we will provide additional instructions in this section once the migration script is completed)
##### Python 3 Upgrade To migrate your database to Kvrocks:
1. Launch ARDB and Kvrocks
To upgrade from an existing AIL installation, you have to launch [python3_upgrade.sh](./python3_upgrade.sh), this script will delete and create a new virtual environment. The script **will upgrade the packages but won't keep your previous data** (neverthless the data is copied into a directory called `old`). If you install from scratch, you don't require to launch the [python3_upgrade.sh](./python3_upgrade.sh). 2. Pull from remote
```
git checkout master
git pull
```
3. Launch the migration script:
```
git checkout master
git pull
cd bin/
./DB_KVROCKS_MIGRATION.py
```

38
bin/Update.py
View file

@ -37,7 +37,8 @@ def auto_update_enabled(cfg):
# check if files are modify locally # check if files are modify locally
def check_if_files_modified(): def check_if_files_modified():
process = subprocess.run(['git', 'ls-files', '-m'], stdout=subprocess.PIPE, stderr=subprocess.PIPE) # return True
process = subprocess.run(['git', 'ls-files' ,'-m'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
if process.returncode == 0: if process.returncode == 0:
modified_files = process.stdout modified_files = process.stdout
if modified_files: if modified_files:
@ -61,6 +62,7 @@ def check_if_files_modified():
sys.exit(1) sys.exit(1)
def repo_is_fork(): def repo_is_fork():
# return False
print('Check if this repository is a fork:') print('Check if this repository is a fork:')
process = subprocess.run(['git', 'remote', '-v'], stdout=subprocess.PIPE, stderr=subprocess.PIPE) process = subprocess.run(['git', 'remote', '-v'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
@ -234,9 +236,9 @@ def get_git_upper_tags_remote(current_tag, is_fork):
except ValueError: except ValueError:
continue continue
if float(current_tag) < 4.2: if float(current_tag) < 5.0:
# add tag with last commit # add tag with last commit
if float(current_tag_val) <= float(tag_val) < float(4.2): if float(current_tag_val) <= float(tag_val) < float(5.0):
dict_tags_commit[tag] = commit dict_tags_commit[tag] = commit
else: else:
# add tag with last commit # add tag with last commit
@ -270,13 +272,28 @@ def update_ail(current_tag, list_upper_tags_remote, current_version_path, is_for
update_submodules() update_submodules()
print('{}git pull:{}'.format(TERMINAL_YELLOW, TERMINAL_DEFAULT)) temp_current_tag = current_tag.replace('v', '')
process = subprocess.run(['git', 'pull'], stdout=subprocess.PIPE, stderr=subprocess.PIPE) if temp_current_tag.count('.') > 1:
temp_current_tag = temp_current_tag.rsplit('.', 1)
temp_current_tag = ''.join(temp_current_tag)
if process.returncode == 0: if float(temp_current_tag) < 5.0:
output = process.stdout.decode() roll_back_update('2c65194b94dab95df9b8da19c88d65239f398355')
print(output) pulled = True
else:
print('{}git pull:{}'.format(TERMINAL_YELLOW, TERMINAL_DEFAULT))
process = subprocess.run(['git', 'pull'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
if process.returncode == 0:
output = process.stdout.decode()
print(output)
pulled = True
else:
print('{}{}{}'.format(TERMINAL_RED, process.stderr.decode(), TERMINAL_DEFAULT))
aborting_update()
pulled = False
sys.exit(1)
if pulled:
# CHECK IF UPDATER Update # CHECK IF UPDATER Update
if float(os.stat(UPDATER_FILENAME).st_mtime) > UPDATER_LAST_MODIFICATION: if float(os.stat(UPDATER_FILENAME).st_mtime) > UPDATER_LAST_MODIFICATION:
# request updater relaunch # request updater relaunch
@ -321,10 +338,7 @@ def update_ail(current_tag, list_upper_tags_remote, current_version_path, is_for
print(f'{TERMINAL_YELLOW}**************** AIL Successfully Updated *****************{TERMINAL_DEFAULT}') print(f'{TERMINAL_YELLOW}**************** AIL Successfully Updated *****************{TERMINAL_DEFAULT}')
print() print()
sys.exit(0) sys.exit(0)
else:
print('{}{}{}'.format(TERMINAL_RED, process.stderr.decode(), TERMINAL_DEFAULT))
aborting_update()
sys.exit(1)
else: else:
print('{}{}{}'.format(TERMINAL_RED, process.stderr.decode(), TERMINAL_DEFAULT)) print('{}{}{}'.format(TERMINAL_RED, process.stderr.decode(), TERMINAL_DEFAULT))
aborting_update() aborting_update()

4
var/www/modules/dashboard/templates/index.html
View file

@ -74,9 +74,7 @@
{%endif%} {%endif%}
<!-- TODO: Add users messages --> <!-- TODO: Add users messages -->
<!-- {%if update_note%} {# {% include 'dashboard/update_modal.html' %}#}
{% include 'dashboard/update_modal.html' %}
{%endif%} -->
<div class="row my-2"> <div class="row my-2">

67
var/www/templates/dashboard/update_modal.html
View file

@ -1,40 +1,49 @@
<div class="modal fade" id="update_modal" tabindex="-1" role="dialog" aria-labelledby="exampleModalLabel" aria-hidden="true"> <div class="modal fade" id="update_modal" tabindex="-1" role="dialog" aria-labelledby="exampleModalLabel" aria-hidden="true">
<div class="modal-dialog modal-lg" role="document"> <div class="modal-dialog modal-lg" role="document">
<div class="modal-content"> <div class="modal-content">
<div class="modal-header bg-secondary text-white"> <div class="modal-header bg-secondary text-white">
<h5 class="modal-title" id="exampleModalLabel">Update Note: v3.5 - Splash Manager</h5> <h5 class="modal-title" id="exampleModalLabel">Update Note: <b>v5.0 - Kvrocks Migration</b></h5>
<button type="button" class="close" data-dismiss="modal" aria-label="Close"> <button type="button" class="close" data-dismiss="modal" aria-label="Close">
<span aria-hidden="true">&times;</span> <span aria-hidden="true">&times;</span>
</button> </button>
</div> </div>
<div class="modal-body"> <div class="modal-body">
<div class="alert alert-danger text-danger" role="alert">All Splash Crawlers have been removed from the core.</div> <div>
AIL is using a new Crawler manager to start and launch dockers and tor/web crawlers In order to improve the performance and reliability of AIL, we have migrated our database from ARDB to <a href="https://github.com/apache/incubator-kvrocks">Kvrocks</a>,
and the AIL crawler has been upgraded to <a href="https://github.com/ail-project/lacus">Lacus</a>.
</div>
<hr>
<ul class="list-group my-3"> <div class="alert alert-danger text-danger" role="alert">Kvrocks DB Migration.</div>
<li class="list-group-item active">Splash Manager Features:</li> <div>
<li class="list-group-item">Install and run Splash crawlers on another server</li> As a result of this change, it is necessary to manually update your AIL installation.
<li class="list-group-item">Handle proxies (Web and tor)</li> Instructions on how to perform the update can be found <a href="https://github.com/ail-project/ail-framework/blob/master/HOWTO.md#kvrocks-migration">here</a>.
<li class="list-group-item">Launch/Kill Splash Dockers</li> </div>
<li class="list-group-item">Restart crawlers on crash</li>
</ul>
<div class="d-flex justify-content-center"> <br><br>
<a class="btn btn-info" href="https://github.com/ail-project/ail-splash-manager" role="button">
<i class="fab fa-github"></i> Install and Configure AIL-Splash-Manager <div class="alert alert-danger text-danger" role="alert">AIL Crawlers Migration: Lacus</div>
</a> If you wish to crawl Tor or website domains, you need to install Lacus.
<div class="d-flex justify-content-center">
<a class="btn btn-info" href="https://github.com/ail-project/lacus" role="button">
<i class="fab fa-github"></i> Install and Configure Lacus
</a>
</div>
How to configure the AIL crawler:
<a href="https://github.com/ail-project/ail-framework/blob/master/HOWTO.md#crawler">HOWTO</a>
<br><br><br>
If you encounter any difficulties, you can open an <a href="https://github.com/ail-project/ail-framework/issues">issue</a>
</div>
<div class="modal-footer">
<button type="button" class="btn btn-secondary" data-dismiss="modal">Close</button>
</div>
</div> </div>
</div>
<div class="modal-footer">
<button type="button" class="btn btn-secondary" data-dismiss="modal">Close</button>
</div>
</div> </div>
</div>
</div> </div>
<script> <script>
$(window).on('load', function() { $(window).on('load', function() {
$('#update_modal').modal('show'); $('#update_modal').modal('show');