CommonCrawl-Ingestor

Common Crawl ingestion and search backend using ClickHouse, with one remote customer CLI at client/ccwget.py.

Install

python3 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.txt
cp config_svr.yaml.sample config_svr.yaml

Developers should install requirements-dev.txt for the server runtime and quality tools.

Configure remote client access with config_client.yaml or environment variables, then run searches with python client/ccwget.py. The same command supports -async, -jobs, -status JOB_ID, and -flush.

For the production Gunicorn/systemd installation, see gunicorn/README.md. System directories must be created as root, while the application and its runtime files belong to ccrawl.

Request logs default to log/ in the repository. Set LOG_DIR in config_svr.yaml to override this path, for example /var/log/ccrawlwget.

SQLite locations are configurable in the same file:

QUEUE_DB: "/var/lib/commoncrawl-ccwget/jobs.sqlite3"
CACHE_DB: "/var/lib/commoncrawl-ccwget/cache.sqlite3"

QUEUE_DB stores jobs; CACHE_DB stores downloaded objects. Create their parent directory and grant write access to ccrawl.

Operations

python -m ingestion.getlatest
python -m ingestion.create_idx 2025
python -m svr.svc_ccwget

SHA-1 Bloom filters

Maintain one global, persistent Bloom filter by adding one crawl archive at a time:

python -m ingestion.shatashabloom --init
python -m ingestion.shatashabloom --add CC-MAIN-2024-33
python -m ingestion.shatashabloom --add CC-MAIN-2025-47
python -m ingestion.shatashabloom --year 2025 --apply-workers 4
python -m ingestion.shatashabloom --sha 67df35fd332c2956c96771fd68a75680de5df4a4
python -m ingestion.shatashabloom --status

--add also accepts physical table form such as CCMAIN202547; legacy -build remains an alias. --year YYYY discovers matching CCMAINYYYYNN tables, sorts them by crawl number, and ingests them sequentially. Already completed archives are skipped. --apply-workers N controls parallel Bloom shard application. Each table keeps the two Building Bloom and Applying Bloom progress phases; year mode also reports per-table and cumulative timings. The command streams every digest without DISTINCT and records archive completion in an atomic manifest. Interrupted imports resume safely, and a writer lock prevents parallel mutation.

SHA-1 input accepts 40-character hexadecimal or 32-character Base32 form. The command reads CLICKHOUSE_HOST, CLICKHOUSE_PORT, CLICKHOUSE_DB, CLICKHOUSE_USER, and CLICKHOUSE_PASSWORD from config_svr.yaml. It prints Maybe Present or Absent; Maybe Present requires confirmation with a normal database lookup. Data uses 256 first-byte shards and 4 KiB Blocked Bloom blocks under ignored repository directory bloomdata/. The default base is sized for 300 billion values and requires roughly 351 GiB free before initialization.

The obsolete config_cc4ail.yaml file is not read by the current code and is intentionally ignored by Git. Do not use it for deployments; copy config_svr.yaml.sample to config_svr.yaml instead.

See SHA-1 Bloom usage for capacity, recovery, and operational details.

See documentation/README.md and the operator runbook for deployment, ClickHouse, indexing, authentication, and service configuration details.