fetch_util 0.3.2 → 0.5.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 8f8adcce7137bb744a361c30f0a2f47ea9f26e751cab745f0b298148151c0ea8
-  data.tar.gz: edbff86bc67288848d4e4c2548c6e8c5b7b53bcc4c29cc10a3d70e9831d19dec
+  metadata.gz: 1ede7613063d932c011dcc3923412b3b41bfbdeadab49df501af635b0b7ffbd0
+  data.tar.gz: 5bd369cd22d7ca45c63046f707b12f09ad817d3b71267b3f5ccfd83455fb2c81
 SHA512:
-  metadata.gz: 85ce1b2472bb679c4f5941d3f828cfbc6455732b81c31952e1d9a5582fbfaa78af247e3ea88fe7a20fdfdcd60d6f50b124e662d6f139891c703060938ff4977d
-  data.tar.gz: b43a3c4ac0ab6d469c895ca8a12aca159b9d8112de63074497dec4e73178340b2bab666c5f2c7bb3a7ac9f1326c551816a16fed65733f06449a95bab1178a150
+  metadata.gz: 5103c8780482705beb59a40e16f31e3635f0f6512dffe3522ef7996b79652a45e81e00c716f085b2bacf85685dcd61f717477310864e5459a895cc7b5e9312ca
+  data.tar.gz: 9118e9c1d177fbaf92bb19fac644b37419dee8712cd839586a350fe1f09b4f3434f9334c20a694693c0a2f6af9e57cef1f9aa1f61a5017c91c6a13e3344d8932

data/CHANGELOG.md

@@ -2,6 +2,55 @@
 
 ## Unreleased
 
+## v0.5.0 - 2026-07-13
+
+### Added
+
+- Add direct, concurrent HTTP search transport for Brave, Bing, DuckDuckGo, Google, and Ecosia with finite source diagnostics, wrapper decoding, and shared deadline enforcement.
+
+### Fixed
+
+- Prevent generic-list ranking scores, malformed engine wrappers, non-SERP engine pages, challenges, and query-mismatched source responses from appearing as valid search results.
+
+### Changed
+
+- Change default search sources from DuckDuckGo and Google to Brave and Bing. Search now returns typed direct-source results instead of reparsing browser Markdown.
+- Apply `limit:` only after aggregation and deduplication, with no default result cap. `verbose: true` adds ordered source diagnostics and result provenance.
+- Remove browser-only search options such as `fetcher:`, `concurrency:`, waits, and reader mode from `FetchUtil.search` and `FetchUtil::Searcher`; use `sources:`, `limit:`, `timeout:`, and `verbose:` for direct search.
+
+### Performance
+
+- Remove Chromium startup and generic browser stabilization from the normal search path.
+
+## v0.4.0 - 2026-07-11
+
+### Added
+
+- Add dedicated result types and structured fields for social posts, threads, feeds, and profiles; products; properties; lodging; jobs; events; recipes; podcasts; sports; medical content; government notices; and press releases.
+- Add explicit language output, direct PDF results, YAML front matter for the default Markdown format, and opt-in URL fields through `--include-urls`.
+- Add public extraction support for Mastodon, Bluesky, Telegram, Hacker News, GitHub discussions, Reddit, Discourse, Stack Exchange, Stack Overflow, Wykop, and other social/community page shapes.
+- Add deterministic content-fidelity contracts covering section coverage, DOM order, card-local context, canonical deduplication, chrome leakage, and focal article structure.
+
+### Fixed
+
+- Preserve complete, DOM-ordered visible content across portal sections, feeds, search results, documentation indexes, social threads, structured content, financial links, glossary entries, large sports tables, and search snippets without silent presentation caps.
+- Improve generic homepage, article, list, documentation, warning, and content-ownership arbitration across multilingual news, reference, commerce, social, legal, medical, and JavaScript-heavy pages.
+- Improve WP and Onet homepage fidelity, Wykop home/tag feeds, Ringier article bodies, MDN reference hierarchy, Mastodon and Telegram live routes, and Polish portal warning semantics.
+- Prevent publisher-specific Ringier extraction from mutating the shared media engine; separate WP and Onet homepage ownership while retaining neutral shared list primitives.
+- Normalize IRI input, strengthen browser navigation resilience, constrain false financial appendices, and reduce mismatch, multi-topic, homepage-index, access-wall, and truncation false positives.
+
+### Changed
+
+- Move editable JavaScript and its ordered manifest to root `websieve/`; generated runtime assets remain packaged under `lib/fetch_util/assets/`.
+- Reorganize JavaScript by systems, content types, profiles, classifiers, and extraction concerns; split oversized modules and normalize manifest dependency order and registration ownership.
+- Change default fetch rendering to YAML-front-matter-prefixed Markdown. JSON, JSONL, and front matter now share one filtered field contract, with URL fields hidden unless requested.
+- Sanitize unreleased fixtures and assertions with synthetic content while preserving selectors, schemas, structure, scripts, counts, order, and regression intent.
+
+### Performance
+
+- Reduce browser stabilization and consent-handling latency while preserving retry behavior for transient navigation and pending-connection failures.
+- Remove redundant fixture resets and consolidate shared extraction, metadata, address, description, cleanup, and registration helpers.
+
 ## v0.3.2 - 2026-07-09
 
 ### Added

data/README.md

@@ -60,7 +60,7 @@ Repo-local usage:
 ```sh
 bundle exec exe/fetch_util fetch https://example.com/article
 bundle exec exe/fetch_util fetch https://example.com/a https://example.com/b --format jsonl
-bundle exec exe/fetch_util search ruby language
+bundle exec exe/fetch_util search ruby language --limit 8
 bundle exec exe/fetch_util regulatory https://example.com
 bundle exec exe/fetch_util regulatory https://example.com/article --sources=machine,human
 ```
@@ -69,15 +69,31 @@ Installed gem usage:
 
 ```sh
 fetch_util fetch https://example.com/article
-fetch_util search ruby language
+fetch_util search ruby language --limit 8
 fetch_util regulatory https://example.com/article --sources=machine,human
 ```
 
+### Search
+
+Search uses direct HTTP requests to the supported sources, in parallel, rather than the browser fetcher. The default sources are `brave` and `bing`; explicit `--source` values may be any of `brave`, `bing`, `duckduckgo`, `google`, or `ecosia`.
+
+Search always emits one JSON object. The normal payload is exactly `{ "query": ..., "results": [...] }`. Results are interleaved by source rank, deduplicated by normalized URL, and retain every eligible result unless an explicit `--limit N` is supplied. `--limit` is applied after aggregation; there is no default result cap. Known Bing, Google, and DuckDuckGo result wrappers are decoded before destination validation.
+
+Each search has one finite deadline shared by its source requests and parsing. Challenges are reported, not bypassed. A source can be `ok`, `empty`, or `failed`; failure reasons include `challenge`, `failed`, `host`, `http_status`, `parse`, `query_mismatch`, `redirect`, `size`, and `timeout`. Normal source failures do not discard healthy peer results. With `--verbose-search`, the payload additionally contains ordered finite source `diagnostics`, and each result contains ordered `sources` and per-source `ranks`.
+
+For agent discovery, use an explicit first-pass budget, choose only the best 1-3 direct result URLs, then fetch those destinations and inspect JSON `warnings`, `suspect`, and `content_type` when needed. Add `--verbose-search` when results are empty or suspicious, or when source health matters:
+
+```sh
+fetch_util search ruby language --limit 8
+fetch_util search ruby language --limit 8 --verbose-search
+fetch_util fetch https://example.com/selected --format json
+```
+
 ## API
 
 - `FetchUtil.fetch(url, **options)` returns a `FetchUtil::Result`
 - `FetchUtil.fetch_many(urls, **options)` fetches multiple URLs in parallel and preserves input order
-- `FetchUtil.search(query, **options)` returns compact aggregated search results
+- `FetchUtil.search(query, **options)` returns direct-source aggregated search results; `limit:` is an explicit post-aggregation cap and is omitted by default
 - `FetchUtil.regulatory(url, **options)` returns a source-keyed hash of allow/disallow signals for crawling, indexing, and TDM-style usage
 - `FetchUtil::Fetcher.new(**options).fetch(url)` exposes the instance API directly
 
@@ -96,6 +112,7 @@ Useful result fields:
 - `timeout:` browser timeout in seconds
 - `wait:` additional settle delay after page load
 - `wait_for_idle:` wait for Ferrum network idle before extraction
+- `limit:` search-only explicit maximum result count; omitted by default, search returns every result in the fetched responses
 - `idle_duration:` idle duration passed to Ferrum when `wait_for_idle` is enabled
 - `reader_mode:` prefer Readability before heuristic fallbacks
 - `viewport:` viewport hash with `:width` and `:height`
@@ -105,11 +122,8 @@ Useful result fields:
 
 ## Output Shape
 
-`fetch` defaults to compact JSON intended for downstream agent/tool consumption. The default payload keeps the fields that are usually most useful in practice:
+Structured `fetch` output is compact JSON intended for downstream agent/tool consumption. The default payload keeps the fields that are usually most useful in practice:
 
-- `url`
-- `final_url`
-- `canonical_url`
 - `title`
 - `byline`
 - `site_name`
@@ -119,7 +133,7 @@ Useful result fields:
 - `suspect`
 - `warnings`
 
-Pass `--include-html` when you explicitly need extracted HTML. Multiple fetch URLs can be streamed as JSON Lines with `--format jsonl`.
+The default `fetch URL` output is Markdown prefixed with standard YAML front matter. The front matter contains the same filtered fields as JSON except `markdown`; the Markdown body follows its closing `---` delimiter. URL fields are omitted from JSON, JSONL, and front matter by default. Pass `--include-urls` to include `url`, `final_url`, and `canonical_url`, and pass `--include-html` to include extracted HTML. Use `--format json` or `--format jsonl` for structured output; multiple Markdown results are emitted as separate front-matter documents.
 
 Both CLI commands append requests to `~/.local/state/fetch_util/requests.log` by default. Override with `FETCH_UTIL_REQUEST_LOG` or `--log-path`.
 
@@ -190,10 +204,10 @@ bundle exec rake
 ```
 
 - The shipped browser bundle is `lib/fetch_util/assets/extract.js`.
-- Source JS lives under `lib/fetch_util/assets/src/` and is ordered by `lib/fetch_util/assets/src/manifest.txt`.
+- Source JS lives under `websieve/` and is ordered by `websieve/manifest.txt`.
 - `bundle exec rake build_extract_assets` rebuilds the bundle and runs `npx terser -cm` before writing `extract.js`.
 - `bundle exec rake verify_extract_assets` checks that the built bundle matches the current sources.
 - The default `bundle exec rake` task runs asset verification, specs, and RuboCop.
 - Direct `bundle exec rspec` runs still check bundle freshness through `spec/build_extract_assets_spec.rb` and enforce the repo-wide SimpleCov minimum.
 
-Do not hand-edit `lib/fetch_util/assets/extract.js`; edit the source files under `lib/fetch_util/assets/src/` and rebuild.
+Do not hand-edit `lib/fetch_util/assets/extract.js`; edit the source files under `websieve/` and rebuild.

data/SKILL.md

@@ -46,26 +46,32 @@ fetch_util fetch https://example.com/a https://example.com/b --format jsonl
 Search first, then fetch selected results if needed:
 
 ```sh
-fetch_util search ruby language
-fetch_util search site:docs.python.org json dump --verbose-search
+fetch_util search ruby language --limit 8
+fetch_util search site:docs.python.org json dump --limit 8
 ```
 
 Repository-local development form:
 
 ```sh
 bundle exec exe/fetch_util fetch https://example.com
-bundle exec exe/fetch_util search ruby language
+bundle exec exe/fetch_util search ruby language --limit 8
 ```
 
 ## Agent Guidance
 
 - if the user gives you URLs, use `fetch_util fetch` first
-- if the user needs discovery, use `fetch_util search` first
+- if the user needs discovery, use `fetch_util search` first; for a context-efficient first pass, consider an explicit budget such as `--limit 8`
+- add `--verbose-search` and inspect ordered source `diagnostics` when results are empty or suspicious, or when source health matters
+- search defaults to direct HTTP Brave and Bing; explicit sources are `brave`, `bing`, `duckduckgo`, `google`, and `ecosia`
+- search always emits one JSON object and normally returns exactly `{query, results}`; `--verbose-search` adds source diagnostics plus per-result source provenance and ranks
+- search has one finite shared source deadline, does not bypass challenges, decodes known engine wrappers, and preserves healthy peer results when a source fails; `empty` and `query_mismatch` are finite source outcomes
+- after search, select only 1-3 direct result URLs, then run `fetch_util fetch` on those destinations; use `--format json` or `--format jsonl` and inspect `warnings`, `suspect`, and `content_type` as needed
 - if the task is a normal web roundup (for example, checking several news homepages), still use `fetch_util` first; do not skip straight to built-in web fetch just because the URLs are already known
 - if you are in a subagent without the `skill` tool, treat `fetch_util` as a normal installed CLI and call it directly
 - use `fetch_util` first because its output is usually cleaner and more compact for agents than generic page fetch output
 - treat `fetch_util` as cheap to use; it is fine to make multiple fetch/search passes when that helps answer the task well
 - prefer the compact default output; use `--format json` when you need metadata, warnings, or content_type fields, and `--format jsonl` for multi-result pipelines
+- search has no default result cap; `--limit` is an explicit post-aggregation cap, not a hidden presentation limit
 - use `--include-html` only when raw HTML is actually needed
 - treat `suspect` and `warnings` as signals that the page may be an interstitial, challenge, or mismatch
 - only fall back to other web tooling after `fetch_util` is unavailable or clearly insufficient