AIL/CommonCrawl-Ingestor
Watch
10
0
Fork
You've already forked CommonCrawl-Ingestor
 0
Code Issues 5 Pull requests Projects Releases Packages Wiki Activity Actions

Add per-token job limits and asynchronous status commands #12

New issue
Closed
opened 2026-07-30 15:29:47 +00:00 by thanat0s · 2 comments
thanat0s commented 2026-07-30 15:29:47 +00:00
Owner
Copy link

Problem

The queued remote client currently allows a token to submit jobs until the global queue capacity is reached. There is no per-token maximum for active jobs, so one client can consume the shared queue and reduce availability for other clients.

The client also always waits for a submitted job to reach a terminal result. There is no asynchronous submission mode and no command to inspect a queued job status without waiting for its result.

Confirmed implementation points:

  • svr/job_queue.py::JobQueue enforces only the global capacity.
  • svr/svc_ccwget.py::submit_job authenticates a token and submits a job.
  • client/ccwget-remote.py::submit polls until completion.
  • Token identity is already derived from the bearer token and can be used for per-token accounting.

Proposed approach

  • Add configurable per-token active-job limits in the server configuration.
  • Support a default limit and optional per-token overrides, without storing plaintext tokens in logs.
  • Count both WAITING and RUNNING jobs for the limit.
  • Return HTTP 429 with a clear retry message when a token reaches its limit.
  • Add a remote client -async option that submits the job and returns its job identifier immediately.
  • Add a remote client -status JOB_ID option that displays queue state, position, wait time, table progress, and terminal errors/results metadata without implicitly downloading an object.
  • Keep the existing synchronous behavior unchanged when neither option is supplied.
  • Preserve token authorization: a token may inspect only jobs belonging to its own identity.
  • Validate option combinations and provide clear errors.

Scope

  • Modify svr/job_queue.py to count active jobs by token identity.
  • Modify svr/svc_ccwget.py to load per-token limits from YAML and enforce them atomically during submission.
  • Extend the sanitized config_svr.yaml.sample with the default and override structure.
  • Extend client/ccwget-local.py parser help so the shared command contract is visible.
  • Extend client/ccwget-remote.py with asynchronous submission and status rendering.
  • Add deterministic tests for quota enforcement, token isolation, async submission, status polling, invalid combinations, and terminal errors.
  • Document configuration, examples, security considerations, and operational recovery.
  • Preserve existing synchronous search, queue ordering, authentication, and object-download behavior.
  • Do not change ClickHouse query semantics or the global single-worker queue.

Acceptance criteria

  • A token cannot have more than its configured number of WAITING plus RUNNING jobs.
  • Different tokens have independent quotas.
  • Quota checks are atomic and cannot be bypassed by concurrent submissions.
  • Exceeding a quota returns HTTP 429 with a clear error and does not create a job.
  • ./ccwget-remote.py -async URL returns successfully without waiting and prints a usable job identifier.
  • ./ccwget-remote.py -status JOB_ID reports WAITING, RUNNING, DONE, and ERROR states with queue position and progress.
  • A token cannot query another token’s job status.
  • Synchronous commands retain current behavior.
  • Invalid combinations such as -async -status fail with a clear client-side message.
  • Tests do not require live ClickHouse, network downloads, or real-time sleeps.
  • Black, Pylint, and pytest quality baselines do not regress.
  • Documentation includes rollback guidance for quota configuration changes.

Dependencies

  • Issue #3: Add SQLite server object cache with L/S client upload behavior (related object lifecycle work, not a blocker).
## Problem The queued remote client currently allows a token to submit jobs until the global queue capacity is reached. There is no per-token maximum for active jobs, so one client can consume the shared queue and reduce availability for other clients. The client also always waits for a submitted job to reach a terminal result. There is no asynchronous submission mode and no command to inspect a queued job status without waiting for its result. Confirmed implementation points: - `svr/job_queue.py::JobQueue` enforces only the global `capacity`. - `svr/svc_ccwget.py::submit_job` authenticates a token and submits a job. - `client/ccwget-remote.py::submit` polls until completion. - Token identity is already derived from the bearer token and can be used for per-token accounting. ## Proposed approach - Add configurable per-token active-job limits in the server configuration. - Support a default limit and optional per-token overrides, without storing plaintext tokens in logs. - Count both WAITING and RUNNING jobs for the limit. - Return HTTP 429 with a clear retry message when a token reaches its limit. - Add a remote client `-async` option that submits the job and returns its job identifier immediately. - Add a remote client `-status JOB_ID` option that displays queue state, position, wait time, table progress, and terminal errors/results metadata without implicitly downloading an object. - Keep the existing synchronous behavior unchanged when neither option is supplied. - Preserve token authorization: a token may inspect only jobs belonging to its own identity. - Validate option combinations and provide clear errors. ## Scope - Modify `svr/job_queue.py` to count active jobs by token identity. - Modify `svr/svc_ccwget.py` to load per-token limits from YAML and enforce them atomically during submission. - Extend the sanitized `config_svr.yaml.sample` with the default and override structure. - Extend `client/ccwget-local.py` parser help so the shared command contract is visible. - Extend `client/ccwget-remote.py` with asynchronous submission and status rendering. - Add deterministic tests for quota enforcement, token isolation, async submission, status polling, invalid combinations, and terminal errors. - Document configuration, examples, security considerations, and operational recovery. - Preserve existing synchronous search, queue ordering, authentication, and object-download behavior. - Do not change ClickHouse query semantics or the global single-worker queue. ## Acceptance criteria - A token cannot have more than its configured number of WAITING plus RUNNING jobs. - Different tokens have independent quotas. - Quota checks are atomic and cannot be bypassed by concurrent submissions. - Exceeding a quota returns HTTP 429 with a clear error and does not create a job. - `./ccwget-remote.py -async URL` returns successfully without waiting and prints a usable job identifier. - `./ccwget-remote.py -status JOB_ID` reports WAITING, RUNNING, DONE, and ERROR states with queue position and progress. - A token cannot query another token’s job status. - Synchronous commands retain current behavior. - Invalid combinations such as `-async -status` fail with a clear client-side message. - Tests do not require live ClickHouse, network downloads, or real-time sleeps. - Black, Pylint, and pytest quality baselines do not regress. - Documentation includes rollback guidance for quota configuration changes. ## Dependencies - Issue #3: Add SQLite server object cache with L/S client upload behavior (related object lifecycle work, not a blocker).
thanat0s commented 2026-07-30 15:31:01 +00:00
Author
Owner
Copy link

Additional requirements for issue #12:

Server operator commands

Extend the server binary with:

  • ./svr/svc_ccwget.py --status: show configured token key identities and active job counts, without exposing plaintext tokens.
  • ./svr/svc_ccwget.py --flush KEY: remove queued jobs for one key and cancel its running job if possible.
  • The flush operation must be explicit, authenticated at the local operator boundary, and clearly report how many WAITING and RUNNING jobs were affected.
  • Running-job cancellation must be cooperative and safe: the worker must not leave a job permanently RUNNING, and ClickHouse work must be released through the existing queue recovery path.
  • Add a safe --flush-all only if implementation requires it; never make a broad flush implicit.

Client command

Add a client flush command using the authenticated API:

  • ./ccwget-remote.py --flush flushes all jobs owned by the current token, including the running job.
  • The client must not be able to flush jobs belonging to another token.
  • Report cancelled WAITING/RUNNING counts and handle already completed jobs clearly.
  • Preserve the existing token security model and document the destructive nature of this operation.

Acceptance additions:

  • Server status never prints plaintext bearer tokens.
  • Flush tests cover queued jobs, the active job, completed jobs, unknown keys, and token isolation.
  • Flush recovery leaves no orphaned RUNNING job.
Additional requirements for issue #12: ## Server operator commands Extend the server binary with: - `./svr/svc_ccwget.py --status`: show configured token key identities and active job counts, without exposing plaintext tokens. - `./svr/svc_ccwget.py --flush KEY`: remove queued jobs for one key and cancel its running job if possible. - The flush operation must be explicit, authenticated at the local operator boundary, and clearly report how many WAITING and RUNNING jobs were affected. - Running-job cancellation must be cooperative and safe: the worker must not leave a job permanently RUNNING, and ClickHouse work must be released through the existing queue recovery path. - Add a safe `--flush-all` only if implementation requires it; never make a broad flush implicit. ## Client command Add a client flush command using the authenticated API: - `./ccwget-remote.py --flush` flushes all jobs owned by the current token, including the running job. - The client must not be able to flush jobs belonging to another token. - Report cancelled WAITING/RUNNING counts and handle already completed jobs clearly. - Preserve the existing token security model and document the destructive nature of this operation. Acceptance additions: - Server status never prints plaintext bearer tokens. - Flush tests cover queued jobs, the active job, completed jobs, unknown keys, and token isolation. - Flush recovery leaves no orphaned RUNNING job.
thanat0s commented 2026-08-03 13:54:39 +00:00
Author
Owner
Copy link

Fixed in commit 7760f6c. --result JOB_ID now reports WAITING, RUNNING, or CANCEL_REQUESTED immediately without polling, fetching results, or deleting active jobs. Terminal result behavior remains unchanged. Tests and documentation updated.

Fixed in commit 7760f6c. --result JOB_ID now reports WAITING, RUNNING, or CANCEL_REQUESTED immediately without polling, fetching results, or deleting active jobs. Terminal result behavior remains unchanged. Tests and documentation updated.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
No results found.
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
AIL/CommonCrawl-Ingestor#12
No description provided.