fetch_util 0.5.0 → 0.5.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 1ede7613063d932c011dcc3923412b3b41bfbdeadab49df501af635b0b7ffbd0
-  data.tar.gz: 5bd369cd22d7ca45c63046f707b12f09ad817d3b71267b3f5ccfd83455fb2c81
+  metadata.gz: 5e878d705eb186cb4a26e70c4e64d18b4efaffab67712ac8cb2bf9f194689634
+  data.tar.gz: cb151c6c55a704d175d6eb170bb24ffebadbe1af02de7f4b681f4781608974e3
 SHA512:
-  metadata.gz: 5103c8780482705beb59a40e16f31e3635f0f6512dffe3522ef7996b79652a45e81e00c716f085b2bacf85685dcd61f717477310864e5459a895cc7b5e9312ca
-  data.tar.gz: 9118e9c1d177fbaf92bb19fac644b37419dee8712cd839586a350fe1f09b4f3434f9334c20a694693c0a2f6af9e57cef1f9aa1f61a5017c91c6a13e3344d8932
+  metadata.gz: f4532ee6ac78e84eb31012bc4bc901404255e101bf1682a8598c535e988df7d4a1e0f361596a97bdf4dd70d2a6d6356b8b171880baa93c2ef049f357c80eca5f
+  data.tar.gz: d3c2b4b08bb53e960273b6c4e204477397f9848978443b4645e8ce8df59ef9742636a6815da76986d1ffd1c5ddadb473e299ce51c391f3356810b4d458cf9bca

data/CHANGELOG.md

@@ -2,6 +2,31 @@
 
 ## Unreleased
 
+## v0.5.2 - 2026-07-21
+
+### Fixed
+
+- Preserve every visible row and row-local field in compact linked build and monitor tables, including hierarchical headers, short record identifiers, and accessible icon-only statuses, while protecting prose-dominant article tables.
+
+## v0.5.1 - 2026-07-15
+
+### Added
+
+- Allow browser fetches to wait within the configured timeout for site-delivered Anubis JavaScript/WebAssembly proof-of-work to resolve, while retaining explicit interstitial results when it does not.
+- Add Yahoo as a direct search source with source-specific organic-card parsing and safe result-wrapper decoding.
+
+### Fixed
+
+- Retry eligible Yahoo transport, HTTP 429, and HTTP 5xx failures up to two times after the initial request under the existing search deadline.
+- Preserve candidates when source relevance is uncertain, including valid later-ranked evidence, and normalize scoped, negated, snake_case, and camelCase query evidence without turning uncertainty into an empty query.
+- Stabilize default quoted, delimiter-bearing, and scoped searches with a healthy Yahoo authority set while preserving explicit source unions and normal fallback aggregation.
+
+### Changed
+
+- Define agent retrieval as the primary product contract while keeping custom solvers, CAPTCHA handling, access circumvention, and high-throughput scraping behavior out of scope.
+- Clarify that direct HTTP search challenges are diagnosed rather than executed.
+- Change the default direct search sources to Brave, Bing, and Yahoo.
+
 ## v0.5.0 - 2026-07-13
 
 ### Added

data/README.md

@@ -1,9 +1,11 @@
 # fetch_util
 
-Reliable browser-backed fetching for Ruby.
+Reliable browser-backed fetching for agents and Ruby applications.
 
 `fetch_util` renders modern pages, inspects the live DOM, classifies page shape, and returns compact markdown plus structured metadata.
 
+Its primary product contract is agent retrieval: use it to discover and fetch public web information, then inspect structured warnings when access is incomplete or ambiguous.
+
 It also provides a plain-Ruby regulatory inspector for machine-readable crawl, index, and text-and-data-mining signals such as `robots.txt`, `X-Robots-Tag`, robots meta tags, and TDM reservation metadata.
 
 It helps applications distinguish between content pages and access/interstitial states such as consent prompts, login-required pages, and challenge screens. When original content is not available, it returns a compact summary with warnings rather than pretending the page was extracted successfully.
@@ -12,11 +14,11 @@ It helps applications distinguish between content pages and access/interstitial
 
 The easiest way to explain `fetch_util` is in three steps:
 
-- `Render` - load the page in Chromium, inspect the rendered DOM, and read page metadata.
+- `Render` - load the page in Chromium, inspect the rendered DOM, and read page metadata. If a site delivers a JavaScript or WebAssembly proof-of-work challenge, the normal browser session may execute it and preserve its resulting cookies while waiting within the configured browser timeout.
 - `Classify` - identify whether the page is an article, list/index, docs page, search result, or an interstitial/access-limited state.
 - `Shape` - return compact markdown, normalized URLs, and warning metadata so the result is usable by agents, LLM workflows, and ordinary Ruby applications.
 
-In short: `fetch_util` makes the web easier to build on.
+This is bounded native browser execution, not a custom challenge solver. If the challenge does not resolve within the bound, the result remains an explicit interstitial with warnings.
 
 ## Installation
 
@@ -75,11 +77,15 @@ fetch_util regulatory https://example.com/article --sources=machine,human
 
 ### Search
 
-Search uses direct HTTP requests to the supported sources, in parallel, rather than the browser fetcher. The default sources are `brave` and `bing`; explicit `--source` values may be any of `brave`, `bing`, `duckduckgo`, `google`, or `ecosia`.
+Search uses direct HTTP requests to the supported sources in parallel. The default sources are `brave`, `bing`, and `yahoo`; explicit `--source` values may be any of `brave`, `bing`, `duckduckgo`, `google`, `ecosia`, or `yahoo`.
+
+Search always emits one JSON object. The normal payload is exactly `{ "query": ..., "results": [...] }`. Results are interleaved by source rank, deduplicated by normalized URL, and retain every eligible result unless an explicit `--limit N` is supplied. `--limit` is applied after aggregation; there is no default result cap. Known Bing, Google, DuckDuckGo, and Yahoo result wrappers are decoded before destination validation.
 
-Search always emits one JSON object. The normal payload is exactly `{ "query": ..., "results": [...] }`. Results are interleaved by source rank, deduplicated by normalized URL, and retain every eligible result unless an explicit `--limit N` is supplied. `--limit` is applied after aggregation; there is no default result cap. Known Bing, Google, and DuckDuckGo result wrappers are decoded before destination validation.
+Each search has one finite deadline shared by its source requests and parsing. After the initial Yahoo request, Yahoo may retry generic transport failures reported as `failed`, HTTP 429, or HTTP 5xx responses up to two times within that same deadline. Direct HTTP search challenges are diagnosed, not executed or bypassed. A source can be `ok`, `empty`, or `failed`; finite reasons include `challenge`, `failed`, `host`, `http_status`, `parse`, `query_mismatch`, `redirect`, `size`, and `timeout`.
 
-Each search has one finite deadline shared by its source requests and parsing. Challenges are reported, not bypassed. A source can be `ok`, `empty`, or `failed`; failure reasons include `challenge`, `failed`, `host`, `http_status`, `parse`, `query_mismatch`, `redirect`, `size`, and `timeout`. Normal source failures do not discard healthy peer results. With `--verbose-search`, the payload additionally contains ordered finite source `diagnostics`, and each result contains ordered `sources` and per-source `ranks`.
+`query_mismatch` is a candidate-bearing relevance warning, not a terminal query failure. The transport retains parsed candidates even when another source succeeds, so an uncertain lexical check cannot erase valid later-ranked evidence or force an empty query. Explicit source unions expose those retained candidates; the separate default structured-query source policy below may select Yahoo's URL set. Scoped operators and negated terms are excluded from lexical evidence, and snake_case/camelCase identifiers share token boundaries.
+
+For default structured queries containing quoted phrases, delimiters, or scoped operators, a healthy Yahoo response defines the returned URL set; matching peer URLs retain their source provenance and ranks. Explicit `--source` selections preserve the requested source union, and normal aggregation remains in effect when Yahoo is unavailable or uncertain. This source policy applies no result-count cap: every eligible authority result is returned unless the caller supplies `--limit`. With `--verbose-search`, the payload additionally contains ordered finite source `diagnostics`, and each result contains ordered `sources` and per-source `ranks`.
 
 For agent discovery, use an explicit first-pass budget, choose only the best 1-3 direct result URLs, then fetch those destinations and inspect JSON `warnings`, `suspect`, and `content_type` when needed. Add `--verbose-search` when results are empty or suspicious, or when source health matters:
 
@@ -109,8 +115,8 @@ Useful result fields:
 
 ## Common Options
 
-- `timeout:` browser timeout in seconds
-- `wait:` additional settle delay after page load
+- `timeout:` browser timeout in seconds; it is also the bounded observation budget for a delivered Anubis challenge
+- `wait:` settle interval used by applicable post-load stabilization paths; it does not control the challenge-completion budget
 - `wait_for_idle:` wait for Ferrum network idle before extraction
 - `limit:` search-only explicit maximum result count; omitted by default, search returns every result in the fetched responses
 - `idle_duration:` idle duration passed to Ferrum when `wait_for_idle` is enabled
@@ -180,14 +186,16 @@ pp FetchUtil.regulatory(
 
 - Extracts articles, list/index pages, and search pages into compact markdown.
 - Uses page classification to select extraction logic appropriate to the rendered page type.
-- Detects consent prompts, login-required pages, and challenge/interstitial screens and reports them with concise summaries and warning tags.
+- Detects consent prompts, login-required pages, and challenge/interstitial screens and reports them with concise summaries and warning tags. A delivered JavaScript/WebAssembly proof-of-work may complete in the normal browser session within `timeout`; unresolved challenges remain explicit interstitials.
 - Cleans up docs/reference pages aggressively enough for agent consumption.
 - Preserves `final_url`, `canonical_url`, and warning metadata so callers can reason about redirects, mismatches, and interstitials.
 - Extracts regulatory allow/disallow signals from `robots.txt`, page headers/meta tags, and TDM reservation metadata without caching raw page bodies.
 
 ## Compliance Boundaries
 
-`fetch_util` is for rendering and summarizing publicly delivered page output. It may identify consent prompts, login-required pages, and challenge/interstitial states and return warning metadata for them. It is not intended to bypass account requirements, paywalls, verification systems, or other access controls.
+`fetch_util` is primarily an agent retrieval tool for rendering and summarizing publicly delivered page output. Browser fetch may allow a site's delivered JavaScript/WebAssembly proof-of-work, such as Anubis, to complete within the configured browser timeout and normal cookie session. This is bounded native browser execution, not a custom solver.
+
+Unresolved challenges remain explicit interstitials. CAPTCHA solving, account or paywall circumvention, identity deception, and high-throughput scraping optimization are out of scope. Direct HTTP search challenges are diagnosed rather than executed.
 
 Browser-profile normalization is intentionally limited to reducing obvious runtime inconsistencies that would otherwise change page behavior during extraction.
 

data/SKILL.md

@@ -10,6 +10,7 @@ Use this skill for agent web retrieval.
 
 Start by loading this skill, then use `fetch_util` as the default first tool for ordinary web-page retrieval and search.
 Use it first because it usually returns cleaner, more agent-friendly markdown than generic web fetch, and it is cheap enough for repeated multi-pass retrieval.
+`fetch_util` is primarily an agent retrieval tool: prefer bounded access to public information and inspect warnings when content is incomplete.
 
 ## Default Rule
 
@@ -27,6 +28,13 @@ Use it first because it usually returns cleaner, more agent-friendly markdown th
 - searching the web and returning compact agent-friendly results
 - getting cleaner markdown from noisy pages, docs, search pages, or light interstitials
 
+## Browser Challenges
+
+- Browser fetch may let the site's delivered JavaScript or WebAssembly proof-of-work, such as Anubis, complete within the configured browser `timeout` and the normal cookie session.
+- This is bounded native browser execution, not a custom solver. `timeout` supplies the challenge-observation budget; `wait` only affects applicable post-load stabilization paths.
+- If the challenge remains unresolved, treat the result as an explicit interstitial and inspect `warnings`, `suspect`, and `content_type`.
+- CAPTCHA solving, account or paywall circumvention, identity deception, and high-throughput scraping optimization are out of scope.
+
 ## Core Commands
 
 Fetch known URLs (returns pure markdown by default):
@@ -62,9 +70,11 @@ bundle exec exe/fetch_util search ruby language --limit 8
 - if the user gives you URLs, use `fetch_util fetch` first
 - if the user needs discovery, use `fetch_util search` first; for a context-efficient first pass, consider an explicit budget such as `--limit 8`
 - add `--verbose-search` and inspect ordered source `diagnostics` when results are empty or suspicious, or when source health matters
-- search defaults to direct HTTP Brave and Bing; explicit sources are `brave`, `bing`, `duckduckgo`, `google`, and `ecosia`
+- search defaults to direct HTTP Brave, Bing, and Yahoo; explicit sources are `brave`, `bing`, `duckduckgo`, `google`, `ecosia`, and `yahoo`
 - search always emits one JSON object and normally returns exactly `{query, results}`; `--verbose-search` adds source diagnostics plus per-result source provenance and ranks
-- search has one finite shared source deadline, does not bypass challenges, decodes known engine wrappers, and preserves healthy peer results when a source fails; `empty` and `query_mismatch` are finite source outcomes
+- search has one finite shared source deadline, diagnoses direct HTTP challenges rather than executing or bypassing them, decodes known engine wrappers, and preserves healthy peer results when a source fails; after the initial request, Yahoo retries generic `failed`, HTTP 429, and HTTP 5xx outcomes up to two times within the same deadline
+- `query_mismatch` retains candidate evidence at the transport boundary instead of forcing an empty query; explicit source unions expose it, while the separate default structured-query policy may select Yahoo's URL set
+- for default quoted, delimiter-bearing, or scoped queries, a healthy Yahoo result set defines returned URLs while matching peers retain provenance; explicit `--source` selections preserve the requested union
 - after search, select only 1-3 direct result URLs, then run `fetch_util fetch` on those destinations; use `--format json` or `--format jsonl` and inspect `warnings`, `suspect`, and `content_type` as needed
 - if the task is a normal web roundup (for example, checking several news homepages), still use `fetch_util` first; do not skip straight to built-in web fetch just because the URLs are already known
 - if you are in a subagent without the `skill` tool, treat `fetch_util` as a normal installed CLI and call it directly