fetch_util 0.3.2 → 0.4.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 8f8adcce7137bb744a361c30f0a2f47ea9f26e751cab745f0b298148151c0ea8
-  data.tar.gz: edbff86bc67288848d4e4c2548c6e8c5b7b53bcc4c29cc10a3d70e9831d19dec
+  metadata.gz: 423504e91e312b0131bc310777b0ea99c140c6a14ae4a71ccc5d7ae6f79bcb94
+  data.tar.gz: d056c04bc10a76322408c05be13ac1024bd654918f66e01dc338450475d40abb
 SHA512:
-  metadata.gz: 85ce1b2472bb679c4f5941d3f828cfbc6455732b81c31952e1d9a5582fbfaa78af247e3ea88fe7a20fdfdcd60d6f50b124e662d6f139891c703060938ff4977d
-  data.tar.gz: b43a3c4ac0ab6d469c895ca8a12aca159b9d8112de63074497dec4e73178340b2bab666c5f2c7bb3a7ac9f1326c551816a16fed65733f06449a95bab1178a150
+  metadata.gz: 69484fc563f44a42e6bcce821d139aa17cabb2027fc501a3e68847061933a5bdfbcf3da9b72d5f092725064bc64b6387e78abe2cab5279ba09bec9b1059a8506
+  data.tar.gz: c40a19ca492c9e75037921e1dbd73f999ab848ddee57a1b5b6a34e46a8c14eed1a6ed0b84348bd0843e526f595fb44f1dbfeef1f7cdd2add85adfd8f400b0481

data/CHANGELOG.md

@@ -2,6 +2,35 @@
 
 ## Unreleased
 
+## v0.4.0 - 2026-07-11
+
+### Added
+
+- Add dedicated result types and structured fields for social posts, threads, feeds, and profiles; products; properties; lodging; jobs; events; recipes; podcasts; sports; medical content; government notices; and press releases.
+- Add explicit language output, direct PDF results, YAML front matter for the default Markdown format, and opt-in URL fields through `--include-urls`.
+- Add public extraction support for Mastodon, Bluesky, Telegram, Hacker News, GitHub discussions, Reddit, Discourse, Stack Exchange, Stack Overflow, Wykop, and other social/community page shapes.
+- Add deterministic content-fidelity contracts covering section coverage, DOM order, card-local context, canonical deduplication, chrome leakage, and focal article structure.
+
+### Fixed
+
+- Preserve complete, DOM-ordered visible content across portal sections, feeds, search results, documentation indexes, social threads, structured content, financial links, glossary entries, large sports tables, and search snippets without silent presentation caps.
+- Improve generic homepage, article, list, documentation, warning, and content-ownership arbitration across multilingual news, reference, commerce, social, legal, medical, and JavaScript-heavy pages.
+- Improve WP and Onet homepage fidelity, Wykop home/tag feeds, Ringier article bodies, MDN reference hierarchy, Mastodon and Telegram live routes, and Polish portal warning semantics.
+- Prevent publisher-specific Ringier extraction from mutating the shared media engine; separate WP and Onet homepage ownership while retaining neutral shared list primitives.
+- Normalize IRI input, strengthen browser navigation resilience, constrain false financial appendices, and reduce mismatch, multi-topic, homepage-index, access-wall, and truncation false positives.
+
+### Changed
+
+- Move editable JavaScript and its ordered manifest to root `websieve/`; generated runtime assets remain packaged under `lib/fetch_util/assets/`.
+- Reorganize JavaScript by systems, content types, profiles, classifiers, and extraction concerns; split oversized modules and normalize manifest dependency order and registration ownership.
+- Change default fetch rendering to YAML-front-matter-prefixed Markdown. JSON, JSONL, and front matter now share one filtered field contract, with URL fields hidden unless requested.
+- Sanitize unreleased fixtures and assertions with synthetic content while preserving selectors, schemas, structure, scripts, counts, order, and regression intent.
+
+### Performance
+
+- Reduce browser stabilization and consent-handling latency while preserving retry behavior for transient navigation and pending-connection failures.
+- Remove redundant fixture resets and consolidate shared extraction, metadata, address, description, cleanup, and registration helpers.
+
 ## v0.3.2 - 2026-07-09
 
 ### Added

data/README.md

@@ -77,7 +77,7 @@ fetch_util regulatory https://example.com/article --sources=machine,human
 
 - `FetchUtil.fetch(url, **options)` returns a `FetchUtil::Result`
 - `FetchUtil.fetch_many(urls, **options)` fetches multiple URLs in parallel and preserves input order
-- `FetchUtil.search(query, **options)` returns compact aggregated search results
+- `FetchUtil.search(query, **options)` returns aggregated search results from the fetched responses; pass `limit:` to request an explicit result cap
 - `FetchUtil.regulatory(url, **options)` returns a source-keyed hash of allow/disallow signals for crawling, indexing, and TDM-style usage
 - `FetchUtil::Fetcher.new(**options).fetch(url)` exposes the instance API directly
 
@@ -96,6 +96,7 @@ Useful result fields:
 - `timeout:` browser timeout in seconds
 - `wait:` additional settle delay after page load
 - `wait_for_idle:` wait for Ferrum network idle before extraction
+- `limit:` search-only explicit maximum result count; omitted by default, search returns every result in the fetched responses
 - `idle_duration:` idle duration passed to Ferrum when `wait_for_idle` is enabled
 - `reader_mode:` prefer Readability before heuristic fallbacks
 - `viewport:` viewport hash with `:width` and `:height`
@@ -105,11 +106,8 @@ Useful result fields:
 
 ## Output Shape
 
-`fetch` defaults to compact JSON intended for downstream agent/tool consumption. The default payload keeps the fields that are usually most useful in practice:
+Structured `fetch` output is compact JSON intended for downstream agent/tool consumption. The default payload keeps the fields that are usually most useful in practice:
 
-- `url`
-- `final_url`
-- `canonical_url`
 - `title`
 - `byline`
 - `site_name`
@@ -119,7 +117,7 @@ Useful result fields:
 - `suspect`
 - `warnings`
 
-Pass `--include-html` when you explicitly need extracted HTML. Multiple fetch URLs can be streamed as JSON Lines with `--format jsonl`.
+The default `fetch URL` output is Markdown prefixed with standard YAML front matter. The front matter contains the same filtered fields as JSON except `markdown`; the Markdown body follows its closing `---` delimiter. URL fields are omitted from JSON, JSONL, and front matter by default. Pass `--include-urls` to include `url`, `final_url`, and `canonical_url`, and pass `--include-html` to include extracted HTML. Use `--format json` or `--format jsonl` for structured output; multiple Markdown results are emitted as separate front-matter documents.
 
 Both CLI commands append requests to `~/.local/state/fetch_util/requests.log` by default. Override with `FETCH_UTIL_REQUEST_LOG` or `--log-path`.
 
@@ -190,10 +188,10 @@ bundle exec rake
 ```
 
 - The shipped browser bundle is `lib/fetch_util/assets/extract.js`.
-- Source JS lives under `lib/fetch_util/assets/src/` and is ordered by `lib/fetch_util/assets/src/manifest.txt`.
+- Source JS lives under `websieve/` and is ordered by `websieve/manifest.txt`.
 - `bundle exec rake build_extract_assets` rebuilds the bundle and runs `npx terser -cm` before writing `extract.js`.
 - `bundle exec rake verify_extract_assets` checks that the built bundle matches the current sources.
 - The default `bundle exec rake` task runs asset verification, specs, and RuboCop.
 - Direct `bundle exec rspec` runs still check bundle freshness through `spec/build_extract_assets_spec.rb` and enforce the repo-wide SimpleCov minimum.
 
-Do not hand-edit `lib/fetch_util/assets/extract.js`; edit the source files under `lib/fetch_util/assets/src/` and rebuild.
+Do not hand-edit `lib/fetch_util/assets/extract.js`; edit the source files under `websieve/` and rebuild.