fetch_util 0.4.0 → 0.5.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 423504e91e312b0131bc310777b0ea99c140c6a14ae4a71ccc5d7ae6f79bcb94
-  data.tar.gz: d056c04bc10a76322408c05be13ac1024bd654918f66e01dc338450475d40abb
+  metadata.gz: 1ede7613063d932c011dcc3923412b3b41bfbdeadab49df501af635b0b7ffbd0
+  data.tar.gz: 5bd369cd22d7ca45c63046f707b12f09ad817d3b71267b3f5ccfd83455fb2c81
 SHA512:
-  metadata.gz: 69484fc563f44a42e6bcce821d139aa17cabb2027fc501a3e68847061933a5bdfbcf3da9b72d5f092725064bc64b6387e78abe2cab5279ba09bec9b1059a8506
-  data.tar.gz: c40a19ca492c9e75037921e1dbd73f999ab848ddee57a1b5b6a34e46a8c14eed1a6ed0b84348bd0843e526f595fb44f1dbfeef1f7cdd2add85adfd8f400b0481
+  metadata.gz: 5103c8780482705beb59a40e16f31e3635f0f6512dffe3522ef7996b79652a45e81e00c716f085b2bacf85685dcd61f717477310864e5459a895cc7b5e9312ca
+  data.tar.gz: 9118e9c1d177fbaf92bb19fac644b37419dee8712cd839586a350fe1f09b4f3434f9334c20a694693c0a2f6af9e57cef1f9aa1f61a5017c91c6a13e3344d8932

data/CHANGELOG.md

@@ -2,6 +2,26 @@
 
 ## Unreleased
 
+## v0.5.0 - 2026-07-13
+
+### Added
+
+- Add direct, concurrent HTTP search transport for Brave, Bing, DuckDuckGo, Google, and Ecosia with finite source diagnostics, wrapper decoding, and shared deadline enforcement.
+
+### Fixed
+
+- Prevent generic-list ranking scores, malformed engine wrappers, non-SERP engine pages, challenges, and query-mismatched source responses from appearing as valid search results.
+
+### Changed
+
+- Change default search sources from DuckDuckGo and Google to Brave and Bing. Search now returns typed direct-source results instead of reparsing browser Markdown.
+- Apply `limit:` only after aggregation and deduplication, with no default result cap. `verbose: true` adds ordered source diagnostics and result provenance.
+- Remove browser-only search options such as `fetcher:`, `concurrency:`, waits, and reader mode from `FetchUtil.search` and `FetchUtil::Searcher`; use `sources:`, `limit:`, `timeout:`, and `verbose:` for direct search.
+
+### Performance
+
+- Remove Chromium startup and generic browser stabilization from the normal search path.
+
 ## v0.4.0 - 2026-07-11
 
 ### Added

data/README.md

@@ -60,7 +60,7 @@ Repo-local usage:
 ```sh
 bundle exec exe/fetch_util fetch https://example.com/article
 bundle exec exe/fetch_util fetch https://example.com/a https://example.com/b --format jsonl
-bundle exec exe/fetch_util search ruby language
+bundle exec exe/fetch_util search ruby language --limit 8
 bundle exec exe/fetch_util regulatory https://example.com
 bundle exec exe/fetch_util regulatory https://example.com/article --sources=machine,human
 ```
@@ -69,15 +69,31 @@ Installed gem usage:
 
 ```sh
 fetch_util fetch https://example.com/article
-fetch_util search ruby language
+fetch_util search ruby language --limit 8
 fetch_util regulatory https://example.com/article --sources=machine,human
 ```
 
+### Search
+
+Search uses direct HTTP requests to the supported sources, in parallel, rather than the browser fetcher. The default sources are `brave` and `bing`; explicit `--source` values may be any of `brave`, `bing`, `duckduckgo`, `google`, or `ecosia`.
+
+Search always emits one JSON object. The normal payload is exactly `{ "query": ..., "results": [...] }`. Results are interleaved by source rank, deduplicated by normalized URL, and retain every eligible result unless an explicit `--limit N` is supplied. `--limit` is applied after aggregation; there is no default result cap. Known Bing, Google, and DuckDuckGo result wrappers are decoded before destination validation.
+
+Each search has one finite deadline shared by its source requests and parsing. Challenges are reported, not bypassed. A source can be `ok`, `empty`, or `failed`; failure reasons include `challenge`, `failed`, `host`, `http_status`, `parse`, `query_mismatch`, `redirect`, `size`, and `timeout`. Normal source failures do not discard healthy peer results. With `--verbose-search`, the payload additionally contains ordered finite source `diagnostics`, and each result contains ordered `sources` and per-source `ranks`.
+
+For agent discovery, use an explicit first-pass budget, choose only the best 1-3 direct result URLs, then fetch those destinations and inspect JSON `warnings`, `suspect`, and `content_type` when needed. Add `--verbose-search` when results are empty or suspicious, or when source health matters:
+
+```sh
+fetch_util search ruby language --limit 8
+fetch_util search ruby language --limit 8 --verbose-search
+fetch_util fetch https://example.com/selected --format json
+```
+
 ## API
 
 - `FetchUtil.fetch(url, **options)` returns a `FetchUtil::Result`
 - `FetchUtil.fetch_many(urls, **options)` fetches multiple URLs in parallel and preserves input order
-- `FetchUtil.search(query, **options)` returns aggregated search results from the fetched responses; pass `limit:` to request an explicit result cap
+- `FetchUtil.search(query, **options)` returns direct-source aggregated search results; `limit:` is an explicit post-aggregation cap and is omitted by default
 - `FetchUtil.regulatory(url, **options)` returns a source-keyed hash of allow/disallow signals for crawling, indexing, and TDM-style usage
 - `FetchUtil::Fetcher.new(**options).fetch(url)` exposes the instance API directly
 

data/SKILL.md

@@ -46,26 +46,32 @@ fetch_util fetch https://example.com/a https://example.com/b --format jsonl
 Search first, then fetch selected results if needed:
 
 ```sh
-fetch_util search ruby language
-fetch_util search site:docs.python.org json dump --verbose-search
+fetch_util search ruby language --limit 8
+fetch_util search site:docs.python.org json dump --limit 8
 ```
 
 Repository-local development form:
 
 ```sh
 bundle exec exe/fetch_util fetch https://example.com
-bundle exec exe/fetch_util search ruby language
+bundle exec exe/fetch_util search ruby language --limit 8
 ```
 
 ## Agent Guidance
 
 - if the user gives you URLs, use `fetch_util fetch` first
-- if the user needs discovery, use `fetch_util search` first
+- if the user needs discovery, use `fetch_util search` first; for a context-efficient first pass, consider an explicit budget such as `--limit 8`
+- add `--verbose-search` and inspect ordered source `diagnostics` when results are empty or suspicious, or when source health matters
+- search defaults to direct HTTP Brave and Bing; explicit sources are `brave`, `bing`, `duckduckgo`, `google`, and `ecosia`
+- search always emits one JSON object and normally returns exactly `{query, results}`; `--verbose-search` adds source diagnostics plus per-result source provenance and ranks
+- search has one finite shared source deadline, does not bypass challenges, decodes known engine wrappers, and preserves healthy peer results when a source fails; `empty` and `query_mismatch` are finite source outcomes
+- after search, select only 1-3 direct result URLs, then run `fetch_util fetch` on those destinations; use `--format json` or `--format jsonl` and inspect `warnings`, `suspect`, and `content_type` as needed
 - if the task is a normal web roundup (for example, checking several news homepages), still use `fetch_util` first; do not skip straight to built-in web fetch just because the URLs are already known
 - if you are in a subagent without the `skill` tool, treat `fetch_util` as a normal installed CLI and call it directly
 - use `fetch_util` first because its output is usually cleaner and more compact for agents than generic page fetch output
 - treat `fetch_util` as cheap to use; it is fine to make multiple fetch/search passes when that helps answer the task well
 - prefer the compact default output; use `--format json` when you need metadata, warnings, or content_type fields, and `--format jsonl` for multi-result pipelines
+- search has no default result cap; `--limit` is an explicit post-aggregation cap, not a hidden presentation limit
 - use `--include-html` only when raw HTML is actually needed
 - treat `suspect` and `warnings` as signals that the page may be an interstitial, challenge, or mismatch
 - only fall back to other web tooling after `fetch_util` is unavailable or clearly insufficient