Bound request and job resources to prevent search-service DoS #20

New issue

Open

opened 2026-08-03 14:56:11 +00:00 by thanat0s · 0 comments

thanat0s commented

2026-08-03 14:56:11 +00:00

Owner

Problem

The search service has several unbounded or late-bound resource inputs.

Confirmed behavior:

POST /jobs accepts an arbitrary JSON body and arbitrary-size arguments mapping after only checking that it is a dictionary.
Flask has no configured global request body limit in the service.
POST /getobject/cache checks Content-Length before reading when available, but otherwise calls request.get_data() and only then checks the payload size.
The legacy svr/scv_cc4ail.py service stores job state in a process-global jobs dictionary without retention cleanup.
Expensive searches can be submitted through direct compatibility routes and wait synchronously for up to the configured sync timeout.

An authenticated or currently unauthenticated caller can consume memory, queue capacity, ClickHouse time, request-worker time, or disk cache space with oversized or repeated inputs.

Proposed approach

Add layered, configuration-backed resource controls at HTTP, queue, input, and cache boundaries. Reuse existing queue quotas, ClickHouse execution timeout, cache byte limit, and object-range validation. Values that need production sizing must be measured and documented as provisional during implementation.

The service should reject oversized requests before application-level parsing where the framework supports it, bound argument sizes and string lengths before queue insertion, and retain only a bounded/expiring set of legacy job records. Rate limiting or equivalent admission control should cover expensive compatibility routes.

This requires server code, configuration samples, tests, documentation, and deployment guidance. No database schema migration should be required unless the existing queue cleanup model cannot support retention.

Scope

Configure a maximum request body size for Flask endpoints.
Enforce bounded JSON depth, field count, and input string lengths before queue insertion.
Ensure object cache uploads are rejected before large bodies are materialized whenever possible.
Review and bound result size, queue retention, synchronous wait behavior, and repeated submissions.
Add expiration/cleanup for legacy in-memory jobs or explicitly retire the legacy service path.
Add rate/admission controls for expensive search operations.
Add deterministic tests for oversized bodies, oversized arguments, repeated submissions, queue saturation, and cleanup.
Document provisional limits, monitoring signals, and operator tuning.
Preserve valid searches, object retrieval, existing quotas, and ClickHouse timeout behavior.
Full distributed rate-limiting infrastructure is outside scope unless required by deployment architecture.

Acceptance criteria

Oversized HTTP requests receive a deterministic 413-style response before expensive backend work starts.
Oversized job arguments are rejected and do not create queue rows.
Cache uploads cannot bypass configured byte limits through missing or misleading Content-Length.
Repeated expensive submissions are bounded by explicit admission controls.
Legacy job records cannot grow without bound; expiration behavior is tested and documented.
Queue saturation returns a controlled error without unbounded memory growth.
Valid requests and existing per-token quotas continue to work.
Tests cover success, boundary, failure, and repeated-submission cases without live ClickHouse or Common Crawl.
Black, Pylint, and pytest baselines do not regress.
Operational documentation includes configured limits, metrics/logs, and rollback steps.

Dependencies

Issue #7 and #18: current input normalization and validation behavior.
Issue #12: durable queue lifecycle and active-job reporting.
Deployment decision on whether svr/scv_cc4ail.py remains supported.

## Problem The search service has several unbounded or late-bound resource inputs. Confirmed behavior: - `POST /jobs` accepts an arbitrary JSON body and arbitrary-size `arguments` mapping after only checking that it is a dictionary. - Flask has no configured global request body limit in the service. - `POST /getobject/cache` checks `Content-Length` before reading when available, but otherwise calls `request.get_data()` and only then checks the payload size. - The legacy `svr/scv_cc4ail.py` service stores job state in a process-global `jobs` dictionary without retention cleanup. - Expensive searches can be submitted through direct compatibility routes and wait synchronously for up to the configured sync timeout. An authenticated or currently unauthenticated caller can consume memory, queue capacity, ClickHouse time, request-worker time, or disk cache space with oversized or repeated inputs. ## Proposed approach Add layered, configuration-backed resource controls at HTTP, queue, input, and cache boundaries. Reuse existing queue quotas, ClickHouse execution timeout, cache byte limit, and object-range validation. Values that need production sizing must be measured and documented as provisional during implementation. The service should reject oversized requests before application-level parsing where the framework supports it, bound argument sizes and string lengths before queue insertion, and retain only a bounded/expiring set of legacy job records. Rate limiting or equivalent admission control should cover expensive compatibility routes. This requires server code, configuration samples, tests, documentation, and deployment guidance. No database schema migration should be required unless the existing queue cleanup model cannot support retention. ## Scope - Configure a maximum request body size for Flask endpoints. - Enforce bounded JSON depth, field count, and input string lengths before queue insertion. - Ensure object cache uploads are rejected before large bodies are materialized whenever possible. - Review and bound result size, queue retention, synchronous wait behavior, and repeated submissions. - Add expiration/cleanup for legacy in-memory jobs or explicitly retire the legacy service path. - Add rate/admission controls for expensive search operations. - Add deterministic tests for oversized bodies, oversized arguments, repeated submissions, queue saturation, and cleanup. - Document provisional limits, monitoring signals, and operator tuning. - Preserve valid searches, object retrieval, existing quotas, and ClickHouse timeout behavior. - Full distributed rate-limiting infrastructure is outside scope unless required by deployment architecture. ## Acceptance criteria - Oversized HTTP requests receive a deterministic 413-style response before expensive backend work starts. - Oversized job arguments are rejected and do not create queue rows. - Cache uploads cannot bypass configured byte limits through missing or misleading `Content-Length`. - Repeated expensive submissions are bounded by explicit admission controls. - Legacy job records cannot grow without bound; expiration behavior is tested and documented. - Queue saturation returns a controlled error without unbounded memory growth. - Valid requests and existing per-token quotas continue to work. - Tests cover success, boundary, failure, and repeated-submission cases without live ClickHouse or Common Crawl. - Black, Pylint, and pytest baselines do not regress. - Operational documentation includes configured limits, metrics/logs, and rollback steps. ## Dependencies - Issue #7 and #18: current input normalization and validation behavior. - Issue #12: durable queue lifecycle and active-job reporting. - Deployment decision on whether `svr/scv_cc4ail.py` remains supported.

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

AIL/CommonCrawl-Ingestor#20

No description provided.

Rows
Columns