AIL/CommonCrawl-Ingestor
Watch
10
0
Fork
You've already forked CommonCrawl-Ingestor
 0
Code Issues 5 Pull requests Projects Releases Packages Wiki Activity Actions

Require bearer authentication on all direct search routes #19

New issue
Closed
opened 2026-08-03 14:56:11 +00:00 by thanat0s · 1 comment
thanat0s commented 2026-08-03 14:56:11 +00:00
Owner
Copy link

Problem

The compatibility search routes in svr/svc_ccwget.py accept requests without a bearer token.

Confirmed behavior:

  • /query, /list-fqdn, /list-domain, /domain-enum, and /sha1 call queue_sync_request() when _queued=1 is absent.
  • queue_sync_request() maps a missing or invalid identity to the literal client ID anonymous instead of returning the existing unauthorized response.
  • The request is then submitted and executed through the durable queue.
  • Existing test tests/test_auth_and_object.py::test_direct_search_route_uses_queue currently verifies the unauthenticated behavior.

This allows an unauthenticated network caller to submit expensive ClickHouse searches and retrieve indexed results, bypassing the authentication enforced by /jobs and object routes. It also permits anonymous resource consumption until queue limits are reached.

Proposed approach

Require successful bearer authentication before direct compatibility routes enqueue work. Reuse authenticate_request() and unauthorized_response(); do not introduce a second authentication mechanism.

Preserve the existing queued execution path and token-scoped queue ownership. If unauthenticated compatibility access is required for a trusted local deployment, make it an explicit configuration mode with a secure default rather than an implicit fallback.

This requires server code, tests, and documentation. No database migration is expected.

Scope

  • Enforce authentication in queue_sync_request() or at every direct search route.
  • Remove the anonymous identity fallback for network requests.
  • Preserve _queued=1 worker execution without attempting to authenticate the internal test request context.
  • Add regression tests for missing, malformed, and valid bearer tokens on every direct route.
  • Verify that authenticated jobs retain the correct token identity and quota.
  • Document the compatibility route authentication contract.
  • Keep query validation, queue behavior, and result schemas unchanged.
  • Anonymous public search access is outside scope unless explicitly designed and configured.

Acceptance criteria

  • Direct search requests without authentication return the existing 401 response.
  • Invalid bearer tokens return 401 and do not create queue rows.
  • Valid bearer tokens can use all current direct routes and receive the current response format.
  • Queued worker execution remains functional without an HTTP Authorization header in its internal request context.
  • Queue ownership and per-token limits are preserved.
  • Tests prove that unauthenticated requests cannot submit work or read results.
  • Black, Pylint, and pytest baselines do not regress.
  • Documentation describes required authentication for all search endpoints.
  • Rollback is documented and does not remove authentication from primary job or object routes.

Dependencies

  • Issue #13: authenticated client job listing.
  • Issue #12: asynchronous job status and token identity behavior.
## Problem The compatibility search routes in `svr/svc_ccwget.py` accept requests without a bearer token. Confirmed behavior: - `/query`, `/list-fqdn`, `/list-domain`, `/domain-enum`, and `/sha1` call `queue_sync_request()` when `_queued=1` is absent. - `queue_sync_request()` maps a missing or invalid identity to the literal client ID `anonymous` instead of returning the existing unauthorized response. - The request is then submitted and executed through the durable queue. - Existing test `tests/test_auth_and_object.py::test_direct_search_route_uses_queue` currently verifies the unauthenticated behavior. This allows an unauthenticated network caller to submit expensive ClickHouse searches and retrieve indexed results, bypassing the authentication enforced by `/jobs` and object routes. It also permits anonymous resource consumption until queue limits are reached. ## Proposed approach Require successful bearer authentication before direct compatibility routes enqueue work. Reuse `authenticate_request()` and `unauthorized_response()`; do not introduce a second authentication mechanism. Preserve the existing queued execution path and token-scoped queue ownership. If unauthenticated compatibility access is required for a trusted local deployment, make it an explicit configuration mode with a secure default rather than an implicit fallback. This requires server code, tests, and documentation. No database migration is expected. ## Scope - Enforce authentication in `queue_sync_request()` or at every direct search route. - Remove the `anonymous` identity fallback for network requests. - Preserve `_queued=1` worker execution without attempting to authenticate the internal test request context. - Add regression tests for missing, malformed, and valid bearer tokens on every direct route. - Verify that authenticated jobs retain the correct token identity and quota. - Document the compatibility route authentication contract. - Keep query validation, queue behavior, and result schemas unchanged. - Anonymous public search access is outside scope unless explicitly designed and configured. ## Acceptance criteria - Direct search requests without authentication return the existing 401 response. - Invalid bearer tokens return 401 and do not create queue rows. - Valid bearer tokens can use all current direct routes and receive the current response format. - Queued worker execution remains functional without an HTTP Authorization header in its internal request context. - Queue ownership and per-token limits are preserved. - Tests prove that unauthenticated requests cannot submit work or read results. - Black, Pylint, and pytest baselines do not regress. - Documentation describes required authentication for all search endpoints. - Rollback is documented and does not remove authentication from primary job or object routes. ## Dependencies - Issue #13: authenticated client job listing. - Issue #12: asynchronous job status and token identity behavior.
thanat0s commented 2026-08-04 11:24:42 +00:00
Author
Owner
Copy link

Implemented and pushed in commit 496c897.

Changes:

  • Direct /query, /list-fqdn, /list-domain, /domain-enum, and /sha1 routes now require bearer authentication.
  • The local client forwards its configured bearer token.
  • Added authentication regression tests for all direct search routes and authenticated queue execution.
  • Updated client and codebase documentation.

Verification: Black passed; 146 pytest tests passed; targeted Pylint score 10.00/10.

Implemented and pushed in commit 496c897. Changes: - Direct /query, /list-fqdn, /list-domain, /domain-enum, and /sha1 routes now require bearer authentication. - The local client forwards its configured bearer token. - Added authentication regression tests for all direct search routes and authenticated queue execution. - Updated client and codebase documentation. Verification: Black passed; 146 pytest tests passed; targeted Pylint score 10.00/10.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
No results found.
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
AIL/CommonCrawl-Ingestor#19
No description provided.