Add SQLite server object cache with L/S client upload behavior #3
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Problem\n\nThe current object path in fetches WARC ranges from Common Crawl for each request. The client-side mode downloads directly from , while the server-side mode downloads the object in the backend. Neither path persists downloaded objects on the server.\n\nThe object identity is already available in search records as (Common Crawl Base32 SHA-1), and object metadata includes the WARC filename, offset, and length. Repeated requests therefore redownload the same content and cannot reuse a server-side object cache.\n\nThe requested behavior is:\n\n- Add a server-side SQLite object cache keyed by an object SHA-based UID.\n- Cache objects fetched by the server.\n- After an client downloads an object directly, upload/store that object in the server cache.\n- If a cached object is returned without headers, do not claim that headers are available.\n- When is requested and cached headers are unavailable, redownload the full WARC object from Common Crawl to obtain and print the headers.\n- Preserve the existing server-download and client-download contract.\n\n## Proposed approach\n\nImplement a server-owned cache component under , backed by a dedicated SQLite database. Use the normalized as the stable UID when available; validate the uploaded/downloaded payload against the expected digest before storing it. Define the behavior for records without a digest explicitly rather than inventing a cache key.\n\nStore payload bytes, digest/UID, byte length, source WARC identity, creation/access timestamps, and an explicit headers-available flag. Use atomic transactions, SQLite WAL/concurrency settings, and bounded/observable retention values selected from deployment measurements.\n\nExtend the authenticated API with a cache upload/store operation for completed downloads. The server must authenticate the caller, validate object metadata and digest, avoid accepting arbitrary cache keys, and never log tokens or payload contents.\n\nFor requests, check the cache before fetching Common Crawl. Return a cache hit when payload data is sufficient. If requires headers and the cached entry does not contain them, fetch the complete WARC range, extract headers, refresh the cache entry, and return the payload. Cache misses and Common Crawl failures must have deterministic status/error behavior.\n\nFor requests, keep returning and let the client fetch the validated range directly. After a successful direct download, the client uploads the payload and required object identity to the cache API. The client must not upload tokens in URLs or silently cache a payload whose digest cannot be verified.\n\n## Scope\n\n- Add a server SQLite cache module under .\n- Add configuration for cache database path, retention/cleanup, maximum object size, and upload limits.\n- Add authenticated cache store/upload and cache lookup behavior to .\n- Extend and header retrieval semantics for cache hits, cache misses, and .\n- Update to upload successful downloads after direct retrieval.\n- Keep as the compatibility/reference client; do not make it the production cache client.\n- Preserve token modes: remains client-side download and remains server-side download.\n- Keep payload response bodies free of WARC headers; headers are returned only through the explicit header operation and printed only when requested.\n- Add tests for cache hit/miss, digest identity, upload validation, cache corruption, header-unavailable behavior, redownload, L upload, S caching, concurrent access, cleanup, restart recovery, and failure propagation.\n- Update README, , configuration samples, operator monitoring, migration, backup, cleanup, and rollback instructions.\n- Do not change ClickHouse tables or crawl data.\n\n## Acceptance criteria\n\n- Server-fetched objects are stored in the SQLite cache after successful retrieval.\n- A subsequent request for the same digest returns the cached payload without a Common Crawl request when headers are not requested.\n- A cache miss fetches Common Crawl, validates the object identity, stores it, and returns .\n- An client downloads directly, then performs an authenticated cache upload; upload failure is reported and cannot corrupt an existing cache entry.\n- Cache entries are addressed by the verified object SHA/digest, not by an arbitrary client-provided UID.\n- Cached responses never inject WARC/HTTP headers into HTTP response headers.\n- If headers are absent from cache, a normal cached payload response does not claim they exist.\n- With , the service redownloads the full WARC object when cached headers are unavailable, then the client prints headers.\n- Tokens are required for cache lookup/upload/object operations, never appear in URLs or logs, and cross-token cache access is denied according to existing authorization rules.\n- Invalid digest, mismatched payload, oversized upload, unknown object metadata, expired entry, corrupt SQLite/blob data, and Common Crawl failure produce documented errors without replacing valid cached data.\n- SQLite behavior is safe under concurrent requests and service restart; tests prove no partial object is visible.\n- Cache size, retention, cleanup, backup, restore, and rollback procedures are documented and observable.\n- Existing URL, SHA-1, year, time-range, queue, /, and no-result behavior remains unchanged outside caching.\n- Black, Pylint, and pytest pass with no score regression; every changed function and test has the required documentation.\n\n## Dependencies\n\n- Related implementation: , , , and .\n- Requires a deployment decision for SQLite cache location, backup, retention, and measured size limits.\n- No ClickHouse schema migration is expected.\n
Fixed in commit
c9aad48. Object access now requires a short-lived capability issued by an authenticated search and bound to client identity, WARC location, and digest. Server validates payload SHA-1, uses ClickHouse lookup only when query digest is absent, refreshes cached headers when unavailable, and enforces cache size/retention cleanup. Added tests and documentation. Verification: 115 pytest tests passed, Pylint 10/10.