AIL/CommonCrawl-Ingestor
Watch
10
0
Fork
You've already forked CommonCrawl-Ingestor
 0
Code Issues 5 Pull requests Projects Releases Packages Wiki Activity Actions

Show dataset row count for every synchronous client search #14

New issue
Closed
opened 2026-07-31 06:52:01 +00:00 by thanat0s · 1 comment
thanat0s commented 2026-07-31 06:52:01 +00:00
Owner
Copy link

Problem

Dataset-size reporting is implemented but is not displayed consistently by every remote search command.

Confirmed implementation points:

  • svr/svc_ccwget.py::report_dataset_rows sums cached system.parts row counts for selected physical tables and stores the total on the active queue job.
  • URL query, FQDN/domain listing, domain enumeration, and SHA-1 handlers call report_dataset_rows.
  • client/ccwget-local.py::SearchProgress.update prints Digging into ... web pages... only when total_rows is non-zero.
  • client/ccwget-remote.py::submit forwards row-count metadata to the renderer only while state is WAITING or RUNNING. A fast job can reach a terminal state between polls, causing the client to skip the count entirely.

Users therefore receive inconsistent dataset scope information across equivalent search modes. The fix must continue using cached table metadata and must not add count() scans against Common Crawl tables.

Proposed approach

  • Make the client process dataset row-count metadata from every observed job state, including terminal states reached between polls.
  • Ensure every queued search operation sets total_rows after physical table selection and before its first data query.
  • Render the count once for every non-quiet synchronous search: direct URL, -e, -l, -ld, -de, and -1.
  • Define explicit output for a valid zero-row selected dataset instead of treating zero as missing metadata.
  • Keep -q silent.
  • Keep -async non-blocking; its later -status and job-listing output should expose the stored dataset row count once available.
  • Reuse the existing three-hour metadata cache and TABLE_COUNT_REFRESH_SECONDS; do not execute full-table row counts.

This requires client progress fixes, validation of server operation ordering, tests, and documentation. No ClickHouse schema or queue DB migration is expected.

Scope

  • Update status handling in client/ccwget-remote.py::submit.
  • Update client/ccwget-local.py::SearchProgress so zero and unavailable counts are distinguishable.
  • Audit all operations registered by svr/svc_ccwget.py::submit_job to ensure row metadata is stored before data-table queries.
  • Preserve current 20-block progress bar, verbose logging, quiet mode, queue waiting display, search results, and download behavior.
  • Add parameterized automated tests covering every client search mode and fast terminal transitions.
  • Add tests for zero-row metadata, unavailable metadata, -q, -v, and -vv.
  • Update customer and operator documentation with count source and timing.
  • Changes to metadata refresh frequency, table selection policy, or ClickHouse indexing are outside scope.

Acceptance criteria

  • Every successful non-quiet synchronous search command prints exactly one dataset scope message with the selected physical-table row total.
  • Direct URL, -e, -l, -ld, -de, and -1 produce consistent count behavior.
  • A job that transitions from WAITING directly to DONE between client polls still prints the count.
  • A selected dataset with zero cached rows prints an explicit zero-page message.
  • Missing row-count metadata is distinguishable from a valid zero and does not print a misleading value.
  • -q emits no dataset count or progress output.
  • -v and -vv retain their current diagnostic behavior while showing the dataset scope once.
  • No Common Crawl table count() query is introduced; totals come only from cached physical-table metadata.
  • Existing result ordering, queue serialization, auth, timeframe filtering, and WARC download behavior remain unchanged.
  • Automated tests need no live ClickHouse, network download, or real-time sleep.
  • Black, Pylint, and pytest baselines do not regress.
  • Documentation includes rollback guidance: revert renderer/status changes without deleting cached metadata or queue records.

Dependencies

  • Issue #12: related queued job status and dataset progress metadata.
## Problem Dataset-size reporting is implemented but is not displayed consistently by every remote search command. Confirmed implementation points: - `svr/svc_ccwget.py::report_dataset_rows` sums cached `system.parts` row counts for selected physical tables and stores the total on the active queue job. - URL query, FQDN/domain listing, domain enumeration, and SHA-1 handlers call `report_dataset_rows`. - `client/ccwget-local.py::SearchProgress.update` prints `Digging into ... web pages...` only when `total_rows` is non-zero. - `client/ccwget-remote.py::submit` forwards row-count metadata to the renderer only while state is `WAITING` or `RUNNING`. A fast job can reach a terminal state between polls, causing the client to skip the count entirely. Users therefore receive inconsistent dataset scope information across equivalent search modes. The fix must continue using cached table metadata and must not add `count()` scans against Common Crawl tables. ## Proposed approach - Make the client process dataset row-count metadata from every observed job state, including terminal states reached between polls. - Ensure every queued search operation sets `total_rows` after physical table selection and before its first data query. - Render the count once for every non-quiet synchronous search: direct URL, `-e`, `-l`, `-ld`, `-de`, and `-1`. - Define explicit output for a valid zero-row selected dataset instead of treating zero as missing metadata. - Keep `-q` silent. - Keep `-async` non-blocking; its later `-status` and job-listing output should expose the stored dataset row count once available. - Reuse the existing three-hour metadata cache and `TABLE_COUNT_REFRESH_SECONDS`; do not execute full-table row counts. This requires client progress fixes, validation of server operation ordering, tests, and documentation. No ClickHouse schema or queue DB migration is expected. ## Scope - Update status handling in `client/ccwget-remote.py::submit`. - Update `client/ccwget-local.py::SearchProgress` so zero and unavailable counts are distinguishable. - Audit all operations registered by `svr/svc_ccwget.py::submit_job` to ensure row metadata is stored before data-table queries. - Preserve current 20-block progress bar, verbose logging, quiet mode, queue waiting display, search results, and download behavior. - Add parameterized automated tests covering every client search mode and fast terminal transitions. - Add tests for zero-row metadata, unavailable metadata, `-q`, `-v`, and `-vv`. - Update customer and operator documentation with count source and timing. - Changes to metadata refresh frequency, table selection policy, or ClickHouse indexing are outside scope. ## Acceptance criteria - Every successful non-quiet synchronous search command prints exactly one dataset scope message with the selected physical-table row total. - Direct URL, `-e`, `-l`, `-ld`, `-de`, and `-1` produce consistent count behavior. - A job that transitions from `WAITING` directly to `DONE` between client polls still prints the count. - A selected dataset with zero cached rows prints an explicit zero-page message. - Missing row-count metadata is distinguishable from a valid zero and does not print a misleading value. - `-q` emits no dataset count or progress output. - `-v` and `-vv` retain their current diagnostic behavior while showing the dataset scope once. - No Common Crawl table `count()` query is introduced; totals come only from cached physical-table metadata. - Existing result ordering, queue serialization, auth, timeframe filtering, and WARC download behavior remain unchanged. - Automated tests need no live ClickHouse, network download, or real-time sleep. - Black, Pylint, and pytest baselines do not regress. - Documentation includes rollback guidance: revert renderer/status changes without deleting cached metadata or queue records. ## Dependencies - Issue #12: related queued job status and dataset progress metadata.
thanat0s commented 2026-07-31 12:44:38 +00:00
Author
Owner
Copy link

Fixed in commit 61fdca2. Dataset row metadata now propagates through terminal synchronous states; valid zero-row datasets print 0, unavailable metadata does not fabricate 0. Added regression tests and operator/release documentation. Verification: 111 pytest tests passed, Black clean, Pylint 10/10.

Fixed in commit 61fdca2. Dataset row metadata now propagates through terminal synchronous states; valid zero-row datasets print 0, unavailable metadata does not fabricate 0. Added regression tests and operator/release documentation. Verification: 111 pytest tests passed, Black clean, Pylint 10/10.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
No results found.
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
AIL/CommonCrawl-Ingestor#14
No description provided.