Compress collected objects for efficient database storage and repush #17

New issue

Open

opened 2026-07-31 15:13:21 +00:00 by thanat0s · 0 comments

thanat0s commented

2026-07-31 15:13:21 +00:00

Owner

Problem

Collected Common Crawl objects are currently stored as raw payload bytes in the SQLite object_cache table implemented by svr/object_cache.py. The cache is used by svr/object_routes.py and svr/svc_ccwget.py for server downloads and authenticated client cache uploads.

The object payloads can be large, and storing them without an explicit compression strategy increases SQLite and future ClickHouse storage costs. The compressed representation should be compatible with the original Common Crawl WARC storage where possible. The system must also retain a way to produce the original uncompressed bytes when an object needs to be re-pushed or served.

Proposed approach

Add fast object compression for persisted database payloads while preserving the Common Crawl representation and the ability to recover original bytes.

The implementation should:

Determine whether the fetched object is already compressed in the Common Crawl source and preserve that representation when it is the correct archival form.
Store compression metadata with every object so readers never guess the codec.
Use a fast, stream-friendly codec supported by the deployment, with the final codec and level selected from measurements of compression ratio, CPU cost, read latency, and repush behavior.
Keep an uncompressed or lossless-recovery path so the original object bytes can be reconstructed exactly for repush.
Apply the same behavior to server downloads and authenticated client cache uploads.
Provide a migration path for existing uncompressed SQLite rows and future ClickHouse object tables.
Keep capability validation and query binding unchanged.

Scope

Extend svr/object_cache.py storage metadata and read/write paths for compressed payloads.
Update svr/object_routes.py and svr/svc_ccwget.py to transparently decode objects before serving or repushing them.
Define the equivalent schema fields for the planned separate ClickHouse object tables.
Add a migration tool for existing SQLite cache rows.
Add configuration for compression enablement and codec settings, without hardcoding production-specific limits.
Add tests for compressed writes, reads, cache misses, client uploads, server downloads, legacy uncompressed rows, corrupted payloads, and exact byte recovery.
Document the selected codec, format compatibility, migration, retention, and rollback procedure.
Keep existing URL search, digest search, authorization, and object capability behavior unchanged.

Acceptance criteria

Newly stored objects use the selected fast compression format and record the codec and relevant format metadata.
Reading a compressed object returns byte-for-byte identical payload and headers to the source object.
Existing uncompressed SQLite cache rows remain readable during and after migration.
A migrated object can be decoded and re-pushed in the original Common Crawl-compatible representation.
Server downloads and authenticated client uploads use the same compression policy.
Corrupt or unsupported compressed rows fail safely with an actionable error and do not return silently altered data.
Repeated writes of the same object remain idempotent.
Compression and decompression performance, storage ratio, and repush compatibility are measured and documented.
Automated tests cover success, legacy data, corruption, unsupported codecs, duplicate writes, and authorization boundaries.
Rollback preserves access to the pre-migration SQLite data until verification is complete.

Dependencies

Decision on the target separate ClickHouse object-table schema.
Production measurements comparing candidate fast codecs and levels.
Availability of the selected codec in the supported runtime and ClickHouse deployment.
Existing SQLite cache files configured through CACHE_DB.

## Problem Collected Common Crawl objects are currently stored as raw payload bytes in the SQLite `object_cache` table implemented by `svr/object_cache.py`. The cache is used by `svr/object_routes.py` and `svr/svc_ccwget.py` for server downloads and authenticated client cache uploads. The object payloads can be large, and storing them without an explicit compression strategy increases SQLite and future ClickHouse storage costs. The compressed representation should be compatible with the original Common Crawl WARC storage where possible. The system must also retain a way to produce the original uncompressed bytes when an object needs to be re-pushed or served. ## Proposed approach Add fast object compression for persisted database payloads while preserving the Common Crawl representation and the ability to recover original bytes. The implementation should: - Determine whether the fetched object is already compressed in the Common Crawl source and preserve that representation when it is the correct archival form. - Store compression metadata with every object so readers never guess the codec. - Use a fast, stream-friendly codec supported by the deployment, with the final codec and level selected from measurements of compression ratio, CPU cost, read latency, and repush behavior. - Keep an uncompressed or lossless-recovery path so the original object bytes can be reconstructed exactly for repush. - Apply the same behavior to server downloads and authenticated client cache uploads. - Provide a migration path for existing uncompressed SQLite rows and future ClickHouse object tables. - Keep capability validation and query binding unchanged. ## Scope - Extend `svr/object_cache.py` storage metadata and read/write paths for compressed payloads. - Update `svr/object_routes.py` and `svr/svc_ccwget.py` to transparently decode objects before serving or repushing them. - Define the equivalent schema fields for the planned separate ClickHouse object tables. - Add a migration tool for existing SQLite cache rows. - Add configuration for compression enablement and codec settings, without hardcoding production-specific limits. - Add tests for compressed writes, reads, cache misses, client uploads, server downloads, legacy uncompressed rows, corrupted payloads, and exact byte recovery. - Document the selected codec, format compatibility, migration, retention, and rollback procedure. - Keep existing URL search, digest search, authorization, and object capability behavior unchanged. ## Acceptance criteria - Newly stored objects use the selected fast compression format and record the codec and relevant format metadata. - Reading a compressed object returns byte-for-byte identical payload and headers to the source object. - Existing uncompressed SQLite cache rows remain readable during and after migration. - A migrated object can be decoded and re-pushed in the original Common Crawl-compatible representation. - Server downloads and authenticated client uploads use the same compression policy. - Corrupt or unsupported compressed rows fail safely with an actionable error and do not return silently altered data. - Repeated writes of the same object remain idempotent. - Compression and decompression performance, storage ratio, and repush compatibility are measured and documented. - Automated tests cover success, legacy data, corruption, unsupported codecs, duplicate writes, and authorization boundaries. - Rollback preserves access to the pre-migration SQLite data until verification is complete. ## Dependencies - Decision on the target separate ClickHouse object-table schema. - Production measurements comparing candidate fast codecs and levels. - Availability of the selected codec in the supported runtime and ClickHouse deployment. - Existing SQLite cache files configured through `CACHE_DB`.

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

AIL/CommonCrawl-Ingestor#17

No description provided.

Rows
Columns