AIL/CommonCrawl-Ingestor
Watch
10
0
Fork
You've already forked CommonCrawl-Ingestor
 0
Code Issues 5 Pull requests Projects Releases Packages Wiki Activity Actions

Normalize bare FQDNs to explicit HTTPS root URLs #7

New issue
Closed
opened 2026-07-30 11:46:17 +00:00 by thanat0s · 1 comment
thanat0s commented 2026-07-30 11:46:17 +00:00
Owner
Copy link

Problem

Running the remote client with a bare FQDN:

./ccwget-remote.py www.circl.lu

does not make the URL interpretation explicit to the user. A bare FQDN should be treated as the root HTTPS URL, and the client should warn that it will search:

https://www.circl.lu/

The normalized URL must then be used for the actual search request. This behavior should apply consistently with the shared client command contract and must not alter explicit schemes, paths, queries, or fragments.

Proposed approach

  • Detect a positional URL that is a valid bare FQDN.
  • Normalize it to https:/// before submitting the search.
  • Emit a clear warning in the default output explaining the normalized URL.
  • Preserve explicit http:// or https:// input unchanged, apart from only adding a root slash when no path is present.
  • Keep -q behavior silent and ensure -v/-vv show the warning without corrupting progress or HTTP diagnostics.
  • Reuse existing domain validation and URL parsing helpers where possible.

Scope

  • Update client URL parsing/normalization used by client/ccwget-remote.py and the shared local client contract if applicable.
  • Add tests for bare FQDN, explicit schemes, existing paths, query strings, invalid domains, and quiet mode.
  • Document default FQDN normalization and warning behavior.
  • Preserve API payload shape after normalization.

Acceptance criteria

  • ./ccwget-remote.py www.circl.lu warns that https://www.circl.lu/ will be searched.
  • Backend receives https://www.circl.lu/ for that invocation.
  • ./ccwget-remote.py https://www.circl.lu/ behaves unchanged.
  • Bare FQDN with no path always receives a trailing slash.
  • Explicit non-root paths and queries remain unchanged.
  • -q suppresses the warning; -v and -vv remain compatible with existing progress and debug output.
  • Automated tests cover normalization and warning behavior.
  • Existing client and server tests continue to pass.

Dependencies

None identified.

## Problem Running the remote client with a bare FQDN: ./ccwget-remote.py www.circl.lu does not make the URL interpretation explicit to the user. A bare FQDN should be treated as the root HTTPS URL, and the client should warn that it will search: https://www.circl.lu/ The normalized URL must then be used for the actual search request. This behavior should apply consistently with the shared client command contract and must not alter explicit schemes, paths, queries, or fragments. ## Proposed approach - Detect a positional URL that is a valid bare FQDN. - Normalize it to https://<fqdn>/ before submitting the search. - Emit a clear warning in the default output explaining the normalized URL. - Preserve explicit http:// or https:// input unchanged, apart from only adding a root slash when no path is present. - Keep -q behavior silent and ensure -v/-vv show the warning without corrupting progress or HTTP diagnostics. - Reuse existing domain validation and URL parsing helpers where possible. ## Scope - Update client URL parsing/normalization used by client/ccwget-remote.py and the shared local client contract if applicable. - Add tests for bare FQDN, explicit schemes, existing paths, query strings, invalid domains, and quiet mode. - Document default FQDN normalization and warning behavior. - Preserve API payload shape after normalization. ## Acceptance criteria - ./ccwget-remote.py www.circl.lu warns that https://www.circl.lu/ will be searched. - Backend receives https://www.circl.lu/ for that invocation. - ./ccwget-remote.py https://www.circl.lu/ behaves unchanged. - Bare FQDN with no path always receives a trailing slash. - Explicit non-root paths and queries remain unchanged. - -q suppresses the warning; -v and -vv remain compatible with existing progress and debug output. - Automated tests cover normalization and warning behavior. - Existing client and server tests continue to pass. ## Dependencies None identified.
thanat0s commented 2026-08-03 11:16:11 +00:00
Author
Owner
Copy link

Fixed in commit f034424. Positional bare FQDNs now normalize to https://fqdn/ before search and emit a warning unless -q is used. Explicit HTTP/HTTPS URLs preserve paths, queries, and fragments; tests and documentation added.

Fixed in commit f034424. Positional bare FQDNs now normalize to https://fqdn/ before search and emit a warning unless -q is used. Explicit HTTP/HTTPS URLs preserve paths, queries, and fragments; tests and documentation added.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
No results found.
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
AIL/CommonCrawl-Ingestor#7
No description provided.