{% include 'nav_bar.html' %}
{% include 'crawler/menu_sidebar.html' %}
{% if is_manager_connected['status'] %}
Connected
{% else %}
Error
{% endif %}

AIL Lacus Crawler

{%if not is_manager_connected['status']%} {% include 'crawler/crawler_disabled.html' %} {%endif%}
{# #} {# #}
Lacus URL {{lacus_url}}
API Key#} {# {{api_key}}#} {# #} {#
{#
#} {#
#} {#
All Proxies:
#} {# #} {# #} {# #} {# #} {# #} {# #} {# #} {# #} {# #} {# #} {# {% for proxy_name in all_proxies %}#} {# #} {# #} {# #} {# #} {# #} {# #} {# {% endfor %}#} {# #} {#
Proxy nameURLDescription
#} {# {{proxy_name}}#} {# #} {# {{all_proxies[proxy_name]['url']}}#} {# #} {# {{all_proxies[proxy_name]['description']}}#} {# #} {#
#} {# #} {#
#} {#
#} {#
#} {#
#}
{% if is_crawler_working %}
{{crawler_error_mess}}
{% else %}
Error
{% endif %}

Crawlers

----------------------------
- TOR CRAWLER TEST OUTPUT: -
----------------------------

{{crawler_error_mess}}
Number of Concurrent Crawlers to Launch:   {{ nb_captures }}
Filter Unsafe Onion:   {{ is_onion_filter_enabled }}

This option enables filtering of onion domains that are considered unsafe due to containing violent content, child sexual abuse material (CSAM), or other harmful materials. When enabled, the system will attempt to identify and exclude such domains from crawling.

Disabling this option may result in crawling and downloading content that includes CSAM, extreme violence, or other harmful materials.
Users are strongly advised to keep this feature enabled to avoid unintentional exposure to such content.

🔍 How It Works: The filtering mechanism leverages known blocklists, heuristics, and automated detection techniques to reduce the risk of crawling unsafe content. While no filtering system is perfect, we continuously strive to improve detection and minimize exposure to harmful materials.

By using this feature, you benefit from an added layer of protection, but please note that some unsafe onion domains may still bypass detection due to evolving content and obfuscation techniques.
We encourage users to remain cautious and use this feature as an additional safeguard.

{% if is_onion_filter_enabled %} {% else %} {% endif %}
Crawl Unknown Onion:   {% if is_onion_filter_unknown %}Enabled{% else %}Disabled{% endif %}

This option controls whether the crawler should proceed with crawling onion domains that have not yet been classified as safe or unsafe.

  • If disabled: The crawler will process domains that have never been checked, potentially discovering new useful content but also increasing the risk of encountering unsafe materials.
  • If enabled: The crawler will only process domains that have been explicitly identified as safe, reducing risk but potentially missing new, unclassified domains.

This option is useful for users who want to explore uncharted domains while still benefiting from the filter_unsafe_onion protection. However, enabling this option increases the likelihood of encountering harmful content, so caution is advised.

{% if is_onion_filter_unknown %} {% else %} {% endif %}