AIL/CommonCrawl-Ingestor
Watch
10
0
Fork
You've already forked CommonCrawl-Ingestor
 0
Code Issues 6 Pull requests Projects Releases Packages Wiki Activity Actions

Add -vvv JSON response tracing to the remote client #15

New issue
Open
opened 2026-07-31 08:44:29 +00:00 by thanat0s · 0 comments
thanat0s commented 2026-07-31 08:44:29 +00:00
Owner
Copy link

Problem

The remote client accepts repeated -v flags, but -vvv currently behaves exactly like -vv.

Confirmed implementation points:

  • client/lib/printout.py::build_client_parser defines -v/--verbose with action="count".
  • client/ccwget-local.py::configure_logging maps every verbosity value greater than or equal to two to logging.DEBUG.
  • client/lib/http.py::ServiceHttpClient.request logs the HTTP method and endpoint, while Requests/urllib3 debug logging shows connection and response status details.
  • JSON response bodies are decoded later by callers such as client/ccwget-remote.py::show_job_status, but are not logged.
  • Therefore a command such as ./ccwget-remote.py -status JOB_ID -vv shows transport activity and formatted status fields without the original JSON response.

This limits API troubleshooting because operators cannot compare the backend JSON contract with the rendered client output. JSON tracing must remain explicit and must not leak the bearer token or dump binary object responses.

Proposed approach

  • Define the client verbosity contract explicitly:
    • -v: user-facing operation and progress information.
    • -vv: existing HTTP transport diagnostics.
    • -vvv: HTTP diagnostics plus decoded JSON response bodies.
  • Add an explicit JSON-tracing setting to client/lib/http.py::ServiceHttpClient and enable it only when parsed verbosity is at least three.
  • Log successful and error JSON responses after receipt, before downstream rendering or error conversion.
  • Format JSON deterministically and readably so nested objects and arrays can be inspected.
  • Include request method, endpoint, and HTTP status with each JSON trace.
  • Never log the Authorization header, bearer token, client token configuration, request headers, or token-derived internal identity.
  • Do not decode or log binary/non-JSON responses such as /getobject payload bodies.
  • Treat malformed JSON as non-traceable data without changing existing request success/error behavior.
  • Keep -vv, normal output, quiet mode, API polling, object download, and server behavior unchanged.

This requires client code, tests, documentation, and a short release-note update. No server, configuration, queue, database, or deployment migration is required.

Scope

  • Extend client/lib/http.py with opt-in JSON response tracing.
  • Configure the trace flag from parsed arguments in client/ccwget-remote.py.
  • Preserve the shared help definition in client/lib/printout.py, while documenting the third verbosity level.
  • Add tests for status JSON, nested/list JSON, backend error JSON, malformed JSON, and non-JSON binary responses.
  • Add regression tests proving that -vv does not print JSON and -vvv does.
  • Add tests proving that Authorization and bearer-token values are absent from trace output.
  • Update documentation/customer-commands.md, documentation/operator-runbook.md, and release_notes.md.
  • Keep all server routes, response schemas, job semantics, and download decisions unchanged.
  • Request-body tracing, raw binary dumps, and server-side logging changes are outside scope.

Acceptance criteria

  • ./ccwget-remote.py -status JOB_ID -vvv prints the decoded JSON returned by GET /jobs/JOB_ID in addition to existing HTTP diagnostics and formatted status output.
  • -vvv traces JSON for successful and error API responses.
  • Each trace identifies HTTP method, endpoint, and response status.
  • Nested objects and arrays are rendered as valid readable JSON.
  • -vv retains current output and does not print response JSON.
  • -v, default mode, and -q behavior remain unchanged unless -vvv is explicitly supplied.
  • Binary /getobject response bytes are never logged as JSON or text.
  • Malformed or non-JSON responses do not create a new client failure.
  • Authorization headers, bearer tokens, token YAML values, salts, and internal client IDs are never added to trace output.
  • Existing status, job listing, async submission, flush, search, and object-download behavior remains unchanged.
  • Automated tests need no live backend, Common Crawl download, or ClickHouse instance.
  • Black, Pylint, and pytest baselines do not regress.
  • Documentation explains the three verbosity levels and warns that -vvv may expose returned search metadata in terminal logs.
  • Rollback consists of removing the opt-in response tracing without changing API contracts or stored state.

Dependencies

None identified.

## Problem The remote client accepts repeated `-v` flags, but `-vvv` currently behaves exactly like `-vv`. Confirmed implementation points: - `client/lib/printout.py::build_client_parser` defines `-v/--verbose` with `action="count"`. - `client/ccwget-local.py::configure_logging` maps every verbosity value greater than or equal to two to `logging.DEBUG`. - `client/lib/http.py::ServiceHttpClient.request` logs the HTTP method and endpoint, while Requests/urllib3 debug logging shows connection and response status details. - JSON response bodies are decoded later by callers such as `client/ccwget-remote.py::show_job_status`, but are not logged. - Therefore a command such as `./ccwget-remote.py -status JOB_ID -vv` shows transport activity and formatted status fields without the original JSON response. This limits API troubleshooting because operators cannot compare the backend JSON contract with the rendered client output. JSON tracing must remain explicit and must not leak the bearer token or dump binary object responses. ## Proposed approach - Define the client verbosity contract explicitly: - `-v`: user-facing operation and progress information. - `-vv`: existing HTTP transport diagnostics. - `-vvv`: HTTP diagnostics plus decoded JSON response bodies. - Add an explicit JSON-tracing setting to `client/lib/http.py::ServiceHttpClient` and enable it only when parsed verbosity is at least three. - Log successful and error JSON responses after receipt, before downstream rendering or error conversion. - Format JSON deterministically and readably so nested objects and arrays can be inspected. - Include request method, endpoint, and HTTP status with each JSON trace. - Never log the Authorization header, bearer token, client token configuration, request headers, or token-derived internal identity. - Do not decode or log binary/non-JSON responses such as `/getobject` payload bodies. - Treat malformed JSON as non-traceable data without changing existing request success/error behavior. - Keep `-vv`, normal output, quiet mode, API polling, object download, and server behavior unchanged. This requires client code, tests, documentation, and a short release-note update. No server, configuration, queue, database, or deployment migration is required. ## Scope - Extend `client/lib/http.py` with opt-in JSON response tracing. - Configure the trace flag from parsed arguments in `client/ccwget-remote.py`. - Preserve the shared help definition in `client/lib/printout.py`, while documenting the third verbosity level. - Add tests for status JSON, nested/list JSON, backend error JSON, malformed JSON, and non-JSON binary responses. - Add regression tests proving that `-vv` does not print JSON and `-vvv` does. - Add tests proving that Authorization and bearer-token values are absent from trace output. - Update `documentation/customer-commands.md`, `documentation/operator-runbook.md`, and `release_notes.md`. - Keep all server routes, response schemas, job semantics, and download decisions unchanged. - Request-body tracing, raw binary dumps, and server-side logging changes are outside scope. ## Acceptance criteria - `./ccwget-remote.py -status JOB_ID -vvv` prints the decoded JSON returned by `GET /jobs/JOB_ID` in addition to existing HTTP diagnostics and formatted status output. - `-vvv` traces JSON for successful and error API responses. - Each trace identifies HTTP method, endpoint, and response status. - Nested objects and arrays are rendered as valid readable JSON. - `-vv` retains current output and does not print response JSON. - `-v`, default mode, and `-q` behavior remain unchanged unless `-vvv` is explicitly supplied. - Binary `/getobject` response bytes are never logged as JSON or text. - Malformed or non-JSON responses do not create a new client failure. - Authorization headers, bearer tokens, token YAML values, salts, and internal client IDs are never added to trace output. - Existing status, job listing, async submission, flush, search, and object-download behavior remains unchanged. - Automated tests need no live backend, Common Crawl download, or ClickHouse instance. - Black, Pylint, and pytest baselines do not regress. - Documentation explains the three verbosity levels and warns that `-vvv` may expose returned search metadata in terminal logs. - Rollback consists of removing the opt-in response tracing without changing API contracts or stored state. ## Dependencies None identified.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
No results found.
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
AIL/CommonCrawl-Ingestor#15
No description provided.