AIL/CommonCrawl-Ingestor

Watch

Fork 0

Search www host when apex URL has no indexed record #10

New issue

Closed

opened 2026-07-30 13:55:27 +00:00 by thanat0s · 2 comments

thanat0s commented

2026-07-30 13:55:27 +00:00

Owner

Problem

The remote client returns no records for the apex URL:

./ccwget-remote.py https://trollprod.org/

The indexed records are stored under the www host, including:

The current server path in svr/svc_ccwget.py normalizes a missing scheme but queries one exact url_host_name value. It does not try the conventional www host variant. The default two-year timeframe is also displayed in the failing command and must continue to apply to every host candidate.

Issue #7 covers bare-FQDN scheme and root-path normalization; this issue covers apex/www host matching.

Proposed approach

Preserve the exact requested host as the first search candidate.
When the requested host is an apex domain with no www label, also search the corresponding www host.
Apply the same scheme, path, query, date range, year, and all-time options to both candidates.
Return exact-host results first; use the www candidate only when needed.
De-duplicate records when both host variants contain the same indexed result.
Keep explicit www input unchanged and avoid adding www to unrelated hostnames.
Keep all ClickHouse values parameterized and use the existing physical-table queue.
Add tests and documentation for apex/www fallback.

The fallback must not silently change a user-supplied subdomain or search arbitrary DNS aliases.

Scope

Update exact URL search handling in svr/svc_ccwget.py.
Reuse existing URL parsing and batched physical-table search helpers.
Add tests for apex fallback, exact-host precedence, explicit www input, paths, queries, date filters, and no-result behavior.
Update customer and codebase documentation.
Add a short release note.
Preserve API response shape, authentication, queueing, WARC retrieval, and default timeframe behavior.

Acceptance criteria

Searching https://trollprod.org/ returns indexed www.trollprod.org records when no apex record exists.
The listed news, archive, offers, and root URLs can be found through the apex URL fallback when indexed under www.
If an apex record exists, it is returned before www records.
Searching https://www.trollprod.org/ performs no duplicate www fallback.
URL paths and query strings remain unchanged while only host candidate changes.
Explicit non-www subdomains are not rewritten.
--after, --before, --time-range, --year, and --alltime apply identically to both candidates.
No duplicate records are returned when both candidates match.
Existing URL, FQDN listing, date, queue, authentication, and object tests continue to pass.

Dependencies

Related issue #7 covers URL scheme/root-path normalization. No blocking dependency identified.

## Problem The remote client returns no records for the apex URL: ./ccwget-remote.py https://trollprod.org/ The indexed records are stored under the www host, including: - https://www.trollprod.org/ - https://www.trollprod.org/?p=news&id=105&lang=fr - https://www.trollprod.org/?p=news&id=106&lang=fr - https://www.trollprod.org/?p=archive&lang=fr - https://www.trollprod.org/?p=offres - https://www.trollprod.org/?p=news&id=79&lang=fr - https://www.trollprod.org/?p=news&id=80&lang=fr - https://www.trollprod.org/?p=news&id=82&lang=fr - https://www.trollprod.org/?p=news&id=83&lang=fr The current server path in svr/svc_ccwget.py normalizes a missing scheme but queries one exact url_host_name value. It does not try the conventional www host variant. The default two-year timeframe is also displayed in the failing command and must continue to apply to every host candidate. Issue #7 covers bare-FQDN scheme and root-path normalization; this issue covers apex/www host matching. ## Proposed approach - Preserve the exact requested host as the first search candidate. - When the requested host is an apex domain with no www label, also search the corresponding www host. - Apply the same scheme, path, query, date range, year, and all-time options to both candidates. - Return exact-host results first; use the www candidate only when needed. - De-duplicate records when both host variants contain the same indexed result. - Keep explicit www input unchanged and avoid adding www to unrelated hostnames. - Keep all ClickHouse values parameterized and use the existing physical-table queue. - Add tests and documentation for apex/www fallback. The fallback must not silently change a user-supplied subdomain or search arbitrary DNS aliases. ## Scope - Update exact URL search handling in svr/svc_ccwget.py. - Reuse existing URL parsing and batched physical-table search helpers. - Add tests for apex fallback, exact-host precedence, explicit www input, paths, queries, date filters, and no-result behavior. - Update customer and codebase documentation. - Add a short release note. - Preserve API response shape, authentication, queueing, WARC retrieval, and default timeframe behavior. ## Acceptance criteria - Searching https://trollprod.org/ returns indexed www.trollprod.org records when no apex record exists. - The listed news, archive, offers, and root URLs can be found through the apex URL fallback when indexed under www. - If an apex record exists, it is returned before www records. - Searching https://www.trollprod.org/ performs no duplicate www fallback. - URL paths and query strings remain unchanged while only host candidate changes. - Explicit non-www subdomains are not rewritten. - --after, --before, --time-range, --year, and --alltime apply identically to both candidates. - No duplicate records are returned when both candidates match. - Existing URL, FQDN listing, date, queue, authentication, and object tests continue to pass. ## Dependencies Related issue #7 covers URL scheme/root-path normalization. No blocking dependency identified.

thanat0s commented

2026-07-30 13:57:17 +00:00

Author

Owner

Additional production evidence for #10:

The root page exists in ClickHouse under the www host:

url_protocol: https
url_host_name: www.trollprod.org
url_path: /
url_query: NULL
content_digest: LNROZ754BFN2KAFXOYXQR4TH25O56R2S
content_languages: fra
content_mime_detected: text/html
warc_filename: crawl-data/CC-MAIN-2026-04/segments/1768220469178.0/warc/CC-MAIN-20260114215951-20260115005951-00464.warc.gz
warc_record_offset: 913589201
warc_record_length: 11768

The failing request uses https://trollprod.org/. This confirms the apex/www host fallback must find the existing www record.

Additional production evidence for #10: The root page exists in ClickHouse under the www host: url_protocol: https url_host_name: www.trollprod.org url_path: / url_query: NULL content_digest: LNROZ754BFN2KAFXOYXQR4TH25O56R2S content_languages: fra content_mime_detected: text/html warc_filename: crawl-data/CC-MAIN-2026-04/segments/1768220469178.0/warc/CC-MAIN-20260114215951-20260115005951-00464.warc.gz warc_record_offset: 913589201 warc_record_length: 11768 The failing request uses https://trollprod.org/. This confirms the apex/www host fallback must find the existing www record.

thanat0s commented

2026-07-30 14:05:39 +00:00

Author

Owner

Fixed in commit 8c87e4d. Exact host matching is preserved; root URL queries now match both url_path='/' and NULL. Added tests and documentation.

Fixed in commit 8c87e4d. Exact host matching is preserved; root URL queries now match both url_path='/' and NULL. Added tests and documentation.

thanat0s closed this issue

2026-07-30 14:05:39 +00:00

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

AIL/CommonCrawl-Ingestor#10

No description provided.

Rows
Columns