AIL/CommonCrawl-Ingestor
Watch
10
0
Fork
You've already forked CommonCrawl-Ingestor
 0
Code Issues 6 Pull requests Projects Releases Packages Wiki Activity Actions
Ingest Common Crawl index in ClickHouse
75 commits 1 branch 0 tags 1.5 MiB
  • Python 99.8%
  • Shell 0.2%
Find a file
Exact
Repository files (latest commit first)
Filename Latest commit message Latest commit date
Paul Jung (Thanat0s) 7760f6c75c Report active jobs on result resume
2026-08-03 15:54:30 +02:00
client Report active jobs on result resume 2026-08-03 15:54:30 +02:00
documentation Report active jobs on result resume 2026-08-03 15:54:30 +02:00
ingestion Move Common Crawl tools to ingestion package 2026-07-31 15:20:48 +02:00
svr Derive search start year from indexed metadata 2026-07-31 16:57:34 +02:00
tests Report active jobs on result resume 2026-08-03 15:54:30 +02:00
.gitignore Move shared libraries under client 2026-07-31 10:36:59 +02:00
.pylintrc Fix physical crawl table search timeouts 2026-07-30 15:27:25 +02:00
AGENTS.md Move shared libraries under client 2026-07-31 10:36:59 +02:00
config_cc4ail.yaml.sample Honor configured server listen address 2026-07-30 13:33:10 +02:00
config_client.yaml.sample Update client service port default 2026-07-30 13:39:22 +02:00
config_svr.yaml.sample Add per-client queue controls 2026-07-30 17:57:50 +02:00
config_svr_token.yaml.sample Add per-client queue controls 2026-07-30 17:57:50 +02:00
LICENSE Initial commit 2025-10-15 09:51:39 +00:00
pytest.ini Split backend server and client modules 2026-07-29 12:02:19 +02:00
README.md Move Common Crawl tools to ingestion package 2026-07-31 15:20:48 +02:00
release_notes.md Report active jobs on result resume 2026-08-03 15:54:30 +02:00
requirements-dev.txt Split client and server dependencies 2026-07-30 10:24:12 +02:00
requirements-svr.txt Split client and server dependencies 2026-07-30 10:24:12 +02:00
requirements.txt Add YAML client configuration 2026-07-30 10:43:54 +02:00

README.md

CommonCrawl-Ingestor

###########   XXXX    ##########      ___
####### XXXXXXXXXXXXXXXX #######    / ___|___  _ __ ___  _ __ ___   ___  _ __
#### XXXXXXXXXXXXXXXXXXXXXX ####   | |   / _ \| '_ ` _ \| '_ ` _ \ / _ \| '_ \
#####  XXXXXXXXXXXXXXXXXXXX ####   | |__| (_) | | | | | | | | | | | (_) | | | |
#####     XXXXXXXXXXXXXXXX #####    \____\___/|_| |_| |_|_| |_| |_|\___/|_| |_|
####          XXXXXXXXXXX   ####          / ___|_ __ __ ___      _| |
XXXXXXXXXXX     XXXXXXX      X #         | |   | '__/ _` \ \ /\ / / |
XXXXXXXXXXX         XX    XXXXX#         | |___| | | (_| |\ V  V /| |
XXXXXXXXXXXX             XXXXXXX          \____|_|  \__,_| \_/\_/ |_|  _
XXXXXXXXXXX        XXXXXXXXXXXX#                  __      __/ ___| ___| |_
XXXXXXXXXXX    XXXXXXXXXXXXXXX #                  \ \ /\ / / |  _ / _ \ __|
# XXXXXXXXXX  XXXXXXXXXXXXXXXXX#                   \ V  V /| |_| |  __/ |_
## XXXXXXXXX   XXXXXXXXXXXXXXX##                    \_/\_/  \____|\___|\__|
### XXXXXXXX     XXXXXXXXXXX ###
##### XXXXXX      XXXXXXXX #####                                       circl.lu
######## XXX ##### XXXX  #######

Common Crawl metadata ingestion and search using ClickHouse. Search archived records by URL, FQDN, domain, or SHA-1; retrieve response bodies from Common Crawl WARC storage.

Full guides: documentation/.

Install

python3 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.txt
cp config_svr.yaml.sample config_svr.yaml

For SVR/SRVAIL servers, install requirements-svr.txt. Developers should install requirements-dev.txt, which includes both runtime sets and Black, Pylint, pytest, and coverage tools.

Client configuration is optional:

cp config_client.yaml.sample config_client.yaml

Basic search

# Download archived URL
python client/ccwget-local.py https://circl.lu/pub/tr-73/

# Show metadata without downloading
python client/ccwget-local.py https://circl.lu/pub/tr-73/ --info

# List URLs for FQDN
python client/ccwget-local.py --list-fqdn www.circl.lu

# Enumerate subdomains across all crawls
python client/ccwget-local.py --domain-enumeration .circl.lu --alltime

# Search SHA-1
python client/ccwget-local.py --sha1 67df35fd332c2956c96771fd68a75680de5df4a4 --info

# Search one year
python client/ccwget-local.py --list-fqdn www.circl.lu --year 2024

# Authenticated queued client (set CCWGET_TOKEN first)
python client/ccwget-remote.py https://circl.lu/pub/tr-73/

# Download every archived occurrence instead of the first one
python client/ccwget-remote.py --all https://circl.lu/pub/tr-73/

# Extract passive-DNS observations from archived WARC headers
python client/ccwget-remote.py -pdns www.circl.lu

Basic operations

# Ingest newest Common Crawl dataset
python -m ingestion.getlatest

# Build or refresh SHA-1 index for one year
python -m ingestion.create_idx 2025

# Export FQDN lists
python -m svr.export_ccmain_fqdn

Current ingestion and HTTP services have important operational limits. Read operator runbook before ingestion or network exposure.