{% include 'nav_bar.html' %}
{% include 'crawler/menu_sidebar.html' %}
{% include 'crawler/crawler_disabled.html' %}
Crawl a Domain
Enter an url or a domain and choose what kind of option you want.
Crawler Type
Proxy
{%for crawler_type in crawlers_types%}
{{crawler_type}}
{%endfor%}
Manual
Scheduler
Hourly
Daily
Weekly
Monthly
Custom
Adjust crawling interval as needed
Months
Weeks
Days
Hours
Minutes
Tags
{% include 'tags/block_tags_selector.html' %}
HTML
Screenshot
HAR
Depth Limit
{# TEMPORARY DISABLED #} {#
#} {#
#} {#
#} {#
#} {#
#} {#
#} {#
Max Pages
#} {#
#} {#
#}
Cookiejar:
Don't use any cookiejar
{%for cookiejar in l_cookiejar%}
{{cookiejar}}
{%endfor%}
Send to Spider