fetch_util 0.4.0 → 0.5.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 423504e91e312b0131bc310777b0ea99c140c6a14ae4a71ccc5d7ae6f79bcb94
-  data.tar.gz: d056c04bc10a76322408c05be13ac1024bd654918f66e01dc338450475d40abb
+  metadata.gz: 6d9dd7f8e6ba30f2d7ad5c9818fc214add21a18baa69f0e1081fe124d7781c35
+  data.tar.gz: 22e9bbc8ad10f85745e2791f196ce2633f4b0a795c95ac94f867ef3696d2053f
 SHA512:
-  metadata.gz: 69484fc563f44a42e6bcce821d139aa17cabb2027fc501a3e68847061933a5bdfbcf3da9b72d5f092725064bc64b6387e78abe2cab5279ba09bec9b1059a8506
-  data.tar.gz: c40a19ca492c9e75037921e1dbd73f999ab848ddee57a1b5b6a34e46a8c14eed1a6ed0b84348bd0843e526f595fb44f1dbfeef1f7cdd2add85adfd8f400b0481
+  metadata.gz: f0115bb99e4c882fd46a06ac57b8350aaa464e651b5f7fb3f38e869a25b48b6ddde29a96a1f0fa955805c3de5459f436cacea8180dbe8246c662b896c2a5c991
+  data.tar.gz: 1813278203b73f64ce2bb26e2936ee42374d7b72fc1365622cba926d736672ac078efc5f03e6972703d47616f531483d74ad4e75c444c6665bef246186798af1

data/CHANGELOG.md

@@ -2,6 +2,45 @@
 
 ## Unreleased
 
+## v0.5.1 - 2026-07-15
+
+### Added
+
+- Allow browser fetches to wait within the configured timeout for site-delivered Anubis JavaScript/WebAssembly proof-of-work to resolve, while retaining explicit interstitial results when it does not.
+- Add Yahoo as a direct search source with source-specific organic-card parsing and safe result-wrapper decoding.
+
+### Fixed
+
+- Retry eligible Yahoo transport, HTTP 429, and HTTP 5xx failures up to two times after the initial request under the existing search deadline.
+- Preserve candidates when source relevance is uncertain, including valid later-ranked evidence, and normalize scoped, negated, snake_case, and camelCase query evidence without turning uncertainty into an empty query.
+- Stabilize default quoted, delimiter-bearing, and scoped searches with a healthy Yahoo authority set while preserving explicit source unions and normal fallback aggregation.
+
+### Changed
+
+- Define agent retrieval as the primary product contract while keeping custom solvers, CAPTCHA handling, access circumvention, and high-throughput scraping behavior out of scope.
+- Clarify that direct HTTP search challenges are diagnosed rather than executed.
+- Change the default direct search sources to Brave, Bing, and Yahoo.
+
+## v0.5.0 - 2026-07-13
+
+### Added
+
+- Add direct, concurrent HTTP search transport for Brave, Bing, DuckDuckGo, Google, and Ecosia with finite source diagnostics, wrapper decoding, and shared deadline enforcement.
+
+### Fixed
+
+- Prevent generic-list ranking scores, malformed engine wrappers, non-SERP engine pages, challenges, and query-mismatched source responses from appearing as valid search results.
+
+### Changed
+
+- Change default search sources from DuckDuckGo and Google to Brave and Bing. Search now returns typed direct-source results instead of reparsing browser Markdown.
+- Apply `limit:` only after aggregation and deduplication, with no default result cap. `verbose: true` adds ordered source diagnostics and result provenance.
+- Remove browser-only search options such as `fetcher:`, `concurrency:`, waits, and reader mode from `FetchUtil.search` and `FetchUtil::Searcher`; use `sources:`, `limit:`, `timeout:`, and `verbose:` for direct search.
+
+### Performance
+
+- Remove Chromium startup and generic browser stabilization from the normal search path.
+
 ## v0.4.0 - 2026-07-11
 
 ### Added

data/README.md

@@ -1,9 +1,11 @@
 # fetch_util
 
-Reliable browser-backed fetching for Ruby.
+Reliable browser-backed fetching for agents and Ruby applications.
 
 `fetch_util` renders modern pages, inspects the live DOM, classifies page shape, and returns compact markdown plus structured metadata.
 
+Its primary product contract is agent retrieval: use it to discover and fetch public web information, then inspect structured warnings when access is incomplete or ambiguous.
+
 It also provides a plain-Ruby regulatory inspector for machine-readable crawl, index, and text-and-data-mining signals such as `robots.txt`, `X-Robots-Tag`, robots meta tags, and TDM reservation metadata.
 
 It helps applications distinguish between content pages and access/interstitial states such as consent prompts, login-required pages, and challenge screens. When original content is not available, it returns a compact summary with warnings rather than pretending the page was extracted successfully.
@@ -12,11 +14,11 @@ It helps applications distinguish between content pages and access/interstitial
 
 The easiest way to explain `fetch_util` is in three steps:
 
-- `Render` - load the page in Chromium, inspect the rendered DOM, and read page metadata.
+- `Render` - load the page in Chromium, inspect the rendered DOM, and read page metadata. If a site delivers a JavaScript or WebAssembly proof-of-work challenge, the normal browser session may execute it and preserve its resulting cookies while waiting within the configured browser timeout.
 - `Classify` - identify whether the page is an article, list/index, docs page, search result, or an interstitial/access-limited state.
 - `Shape` - return compact markdown, normalized URLs, and warning metadata so the result is usable by agents, LLM workflows, and ordinary Ruby applications.
 
-In short: `fetch_util` makes the web easier to build on.
+This is bounded native browser execution, not a custom challenge solver. If the challenge does not resolve within the bound, the result remains an explicit interstitial with warnings.
 
 ## Installation
 
@@ -60,7 +62,7 @@ Repo-local usage:
 ```sh
 bundle exec exe/fetch_util fetch https://example.com/article
 bundle exec exe/fetch_util fetch https://example.com/a https://example.com/b --format jsonl
-bundle exec exe/fetch_util search ruby language
+bundle exec exe/fetch_util search ruby language --limit 8
 bundle exec exe/fetch_util regulatory https://example.com
 bundle exec exe/fetch_util regulatory https://example.com/article --sources=machine,human
 ```
@@ -69,15 +71,35 @@ Installed gem usage:
 
 ```sh
 fetch_util fetch https://example.com/article
-fetch_util search ruby language
+fetch_util search ruby language --limit 8
 fetch_util regulatory https://example.com/article --sources=machine,human
 ```
 
+### Search
+
+Search uses direct HTTP requests to the supported sources in parallel. The default sources are `brave`, `bing`, and `yahoo`; explicit `--source` values may be any of `brave`, `bing`, `duckduckgo`, `google`, `ecosia`, or `yahoo`.
+
+Search always emits one JSON object. The normal payload is exactly `{ "query": ..., "results": [...] }`. Results are interleaved by source rank, deduplicated by normalized URL, and retain every eligible result unless an explicit `--limit N` is supplied. `--limit` is applied after aggregation; there is no default result cap. Known Bing, Google, DuckDuckGo, and Yahoo result wrappers are decoded before destination validation.
+
+Each search has one finite deadline shared by its source requests and parsing. After the initial Yahoo request, Yahoo may retry generic transport failures reported as `failed`, HTTP 429, or HTTP 5xx responses up to two times within that same deadline. Direct HTTP search challenges are diagnosed, not executed or bypassed. A source can be `ok`, `empty`, or `failed`; finite reasons include `challenge`, `failed`, `host`, `http_status`, `parse`, `query_mismatch`, `redirect`, `size`, and `timeout`.
+
+`query_mismatch` is a candidate-bearing relevance warning, not a terminal query failure. The transport retains parsed candidates even when another source succeeds, so an uncertain lexical check cannot erase valid later-ranked evidence or force an empty query. Explicit source unions expose those retained candidates; the separate default structured-query source policy below may select Yahoo's URL set. Scoped operators and negated terms are excluded from lexical evidence, and snake_case/camelCase identifiers share token boundaries.
+
+For default structured queries containing quoted phrases, delimiters, or scoped operators, a healthy Yahoo response defines the returned URL set; matching peer URLs retain their source provenance and ranks. Explicit `--source` selections preserve the requested source union, and normal aggregation remains in effect when Yahoo is unavailable or uncertain. This source policy applies no result-count cap: every eligible authority result is returned unless the caller supplies `--limit`. With `--verbose-search`, the payload additionally contains ordered finite source `diagnostics`, and each result contains ordered `sources` and per-source `ranks`.
+
+For agent discovery, use an explicit first-pass budget, choose only the best 1-3 direct result URLs, then fetch those destinations and inspect JSON `warnings`, `suspect`, and `content_type` when needed. Add `--verbose-search` when results are empty or suspicious, or when source health matters:
+
+```sh
+fetch_util search ruby language --limit 8
+fetch_util search ruby language --limit 8 --verbose-search
+fetch_util fetch https://example.com/selected --format json
+```
+
 ## API
 
 - `FetchUtil.fetch(url, **options)` returns a `FetchUtil::Result`
 - `FetchUtil.fetch_many(urls, **options)` fetches multiple URLs in parallel and preserves input order
-- `FetchUtil.search(query, **options)` returns aggregated search results from the fetched responses; pass `limit:` to request an explicit result cap
+- `FetchUtil.search(query, **options)` returns direct-source aggregated search results; `limit:` is an explicit post-aggregation cap and is omitted by default
 - `FetchUtil.regulatory(url, **options)` returns a source-keyed hash of allow/disallow signals for crawling, indexing, and TDM-style usage
 - `FetchUtil::Fetcher.new(**options).fetch(url)` exposes the instance API directly
 
@@ -93,8 +115,8 @@ Useful result fields:
 
 ## Common Options
 
-- `timeout:` browser timeout in seconds
-- `wait:` additional settle delay after page load
+- `timeout:` browser timeout in seconds; it is also the bounded observation budget for a delivered Anubis challenge
+- `wait:` settle interval used by applicable post-load stabilization paths; it does not control the challenge-completion budget
 - `wait_for_idle:` wait for Ferrum network idle before extraction
 - `limit:` search-only explicit maximum result count; omitted by default, search returns every result in the fetched responses
 - `idle_duration:` idle duration passed to Ferrum when `wait_for_idle` is enabled
@@ -164,14 +186,16 @@ pp FetchUtil.regulatory(
 
 - Extracts articles, list/index pages, and search pages into compact markdown.
 - Uses page classification to select extraction logic appropriate to the rendered page type.
-- Detects consent prompts, login-required pages, and challenge/interstitial screens and reports them with concise summaries and warning tags.
+- Detects consent prompts, login-required pages, and challenge/interstitial screens and reports them with concise summaries and warning tags. A delivered JavaScript/WebAssembly proof-of-work may complete in the normal browser session within `timeout`; unresolved challenges remain explicit interstitials.
 - Cleans up docs/reference pages aggressively enough for agent consumption.
 - Preserves `final_url`, `canonical_url`, and warning metadata so callers can reason about redirects, mismatches, and interstitials.
 - Extracts regulatory allow/disallow signals from `robots.txt`, page headers/meta tags, and TDM reservation metadata without caching raw page bodies.
 
 ## Compliance Boundaries
 
-`fetch_util` is for rendering and summarizing publicly delivered page output. It may identify consent prompts, login-required pages, and challenge/interstitial states and return warning metadata for them. It is not intended to bypass account requirements, paywalls, verification systems, or other access controls.
+`fetch_util` is primarily an agent retrieval tool for rendering and summarizing publicly delivered page output. Browser fetch may allow a site's delivered JavaScript/WebAssembly proof-of-work, such as Anubis, to complete within the configured browser timeout and normal cookie session. This is bounded native browser execution, not a custom solver.
+
+Unresolved challenges remain explicit interstitials. CAPTCHA solving, account or paywall circumvention, identity deception, and high-throughput scraping optimization are out of scope. Direct HTTP search challenges are diagnosed rather than executed.
 
 Browser-profile normalization is intentionally limited to reducing obvious runtime inconsistencies that would otherwise change page behavior during extraction.
 

data/SKILL.md

@@ -10,6 +10,7 @@ Use this skill for agent web retrieval.
 
 Start by loading this skill, then use `fetch_util` as the default first tool for ordinary web-page retrieval and search.
 Use it first because it usually returns cleaner, more agent-friendly markdown than generic web fetch, and it is cheap enough for repeated multi-pass retrieval.
+`fetch_util` is primarily an agent retrieval tool: prefer bounded access to public information and inspect warnings when content is incomplete.
 
 ## Default Rule
 
@@ -27,6 +28,13 @@ Use it first because it usually returns cleaner, more agent-friendly markdown th
 - searching the web and returning compact agent-friendly results
 - getting cleaner markdown from noisy pages, docs, search pages, or light interstitials
 
+## Browser Challenges
+
+- Browser fetch may let the site's delivered JavaScript or WebAssembly proof-of-work, such as Anubis, complete within the configured browser `timeout` and the normal cookie session.
+- This is bounded native browser execution, not a custom solver. `timeout` supplies the challenge-observation budget; `wait` only affects applicable post-load stabilization paths.
+- If the challenge remains unresolved, treat the result as an explicit interstitial and inspect `warnings`, `suspect`, and `content_type`.
+- CAPTCHA solving, account or paywall circumvention, identity deception, and high-throughput scraping optimization are out of scope.
+
 ## Core Commands
 
 Fetch known URLs (returns pure markdown by default):
@@ -46,26 +54,34 @@ fetch_util fetch https://example.com/a https://example.com/b --format jsonl
 Search first, then fetch selected results if needed:
 
 ```sh
-fetch_util search ruby language
-fetch_util search site:docs.python.org json dump --verbose-search
+fetch_util search ruby language --limit 8
+fetch_util search site:docs.python.org json dump --limit 8
 ```
 
 Repository-local development form:
 
 ```sh
 bundle exec exe/fetch_util fetch https://example.com
-bundle exec exe/fetch_util search ruby language
+bundle exec exe/fetch_util search ruby language --limit 8
 ```
 
 ## Agent Guidance
 
 - if the user gives you URLs, use `fetch_util fetch` first
-- if the user needs discovery, use `fetch_util search` first
+- if the user needs discovery, use `fetch_util search` first; for a context-efficient first pass, consider an explicit budget such as `--limit 8`
+- add `--verbose-search` and inspect ordered source `diagnostics` when results are empty or suspicious, or when source health matters
+- search defaults to direct HTTP Brave, Bing, and Yahoo; explicit sources are `brave`, `bing`, `duckduckgo`, `google`, `ecosia`, and `yahoo`
+- search always emits one JSON object and normally returns exactly `{query, results}`; `--verbose-search` adds source diagnostics plus per-result source provenance and ranks
+- search has one finite shared source deadline, diagnoses direct HTTP challenges rather than executing or bypassing them, decodes known engine wrappers, and preserves healthy peer results when a source fails; after the initial request, Yahoo retries generic `failed`, HTTP 429, and HTTP 5xx outcomes up to two times within the same deadline
+- `query_mismatch` retains candidate evidence at the transport boundary instead of forcing an empty query; explicit source unions expose it, while the separate default structured-query policy may select Yahoo's URL set
+- for default quoted, delimiter-bearing, or scoped queries, a healthy Yahoo result set defines returned URLs while matching peers retain provenance; explicit `--source` selections preserve the requested union
+- after search, select only 1-3 direct result URLs, then run `fetch_util fetch` on those destinations; use `--format json` or `--format jsonl` and inspect `warnings`, `suspect`, and `content_type` as needed
 - if the task is a normal web roundup (for example, checking several news homepages), still use `fetch_util` first; do not skip straight to built-in web fetch just because the URLs are already known
 - if you are in a subagent without the `skill` tool, treat `fetch_util` as a normal installed CLI and call it directly
 - use `fetch_util` first because its output is usually cleaner and more compact for agents than generic page fetch output
 - treat `fetch_util` as cheap to use; it is fine to make multiple fetch/search passes when that helps answer the task well
 - prefer the compact default output; use `--format json` when you need metadata, warnings, or content_type fields, and `--format jsonl` for multi-result pipelines
+- search has no default result cap; `--limit` is an explicit post-aggregation cap, not a hidden presentation limit
 - use `--include-html` only when raw HTML is actually needed
 - treat `suspect` and `warnings` as signals that the page may be an interstitial, challenge, or mismatch
 - only fall back to other web tooling after `fetch_util` is unavailable or clearly insufficient