fetch_util 0.3.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: fa6e99e3ed511c2cab332a2914dbf37ffce3e248bd2ba08b967ef744c672b806
+  data.tar.gz: 2e4a8d928de0a9f4b2d60d81373326743341fcc0454cd087a4a7ced4f52393bd
+SHA512:
+  metadata.gz: c0dc535d27ef97d4a6ee65e154f9c3503ed78ec5df93280e6ee14a90d1c83ec641a4a53e78419056718d9cd78480c8124cf132cc06854140296df481e7153e42
+  data.tar.gz: 54303e5889abeacb1341602125eb90a8e1e38a054b3e6a853bd0789e3070272c86f9e8058b88f2282c3734c9c63af8cee4ffa5cd56285aa49c8d25eee6068131

data/.rspec

@@ -0,0 +1,2 @@
+--require spec_helper
+--format documentation

data/.rubocop.yml

@@ -0,0 +1,97 @@
+AllCops:
+  TargetRubyVersion: 3.2
+  NewCops: disable
+
+Style/StringLiterals:
+  EnforcedStyle: double_quotes
+  Exclude:
+    - "spec/**/*"
+    - "fetch_util.gemspec"
+
+Style/StringLiteralsInInterpolation:
+  EnforcedStyle: double_quotes
+
+Style/FrozenStringLiteralComment:
+  Enabled: false
+
+Style/Documentation:
+  Enabled: false
+
+Layout/LineLength:
+  Max: 170
+
+Metrics/BlockLength:
+  Enabled: false
+Metrics/ClassLength:
+  Enabled: false
+Metrics/MethodLength:
+  Enabled: false
+Metrics/AbcSize:
+  Enabled: false
+Metrics/CyclomaticComplexity:
+  Enabled: false
+Metrics/PerceivedComplexity:
+  Enabled: false
+Metrics/ParameterLists:
+  Enabled: false
+
+Naming/MethodParameterName:
+  Enabled: false
+
+Lint/SuppressedException:
+  Enabled: false
+
+Style/MultilineBlockChain:
+  Enabled: false
+
+Style/PerlBackrefs:
+  Enabled: false
+
+Style/FormatStringToken:
+  Enabled: false
+Style/FormatString:
+  Enabled: false
+
+Layout/SpaceAfterComma:
+  Enabled: false
+Layout/SpaceAroundOperators:
+  Enabled: false
+Layout/TrailingWhitespace:
+  Enabled: false
+
+Style/IfUnlessModifier:
+  Enabled: false
+Style/NilComparison:
+  Enabled: false
+Style/NumericPredicate:
+  Enabled: false
+
+Lint/DuplicateMethods:
+  Enabled: false
+
+Lint/Void:
+  Enabled: false
+
+Layout/SpaceBeforeBlockBraces:
+  Enabled: false
+
+Lint/UnusedBlockArgument:
+  Enabled: false
+
+Naming/AccessorMethodName:
+  Enabled: false
+
+Style/SingleLineMethods:
+  Enabled: false
+
+Layout/EmptyLineAfterGuardClause:
+  Enabled: false
+
+Style/IfInsideElse:
+  Enabled: false
+
+Style/RedundantBegin:
+  Enabled: false
+
+Style/SymbolProc:
+  Enabled: false

data/CHANGELOG.md

@@ -0,0 +1,48 @@
+# Changelog
+
+## Unreleased
+
+## v0.3.0 - 2026-06-21
+
+- Add generic portal, marketplace, and booking homepage root selection so lead-story lists are extracted from shell pages the previous docs, repo, and community dispatchers missed; already-handled homepages are unaffected.
+- Improve dictionary, glossary, and citation definition root extraction with a reusable definition-reference metadata scorer, `dl`/`dt`/`dd` and `itemprop=description` container boosts, and repeated-sense dedupe that strips numbering, term prefixes, and citation tails.
+- Surface commerce product card details from JSON-LD `Product`, `Offer`, `AggregateRating`, and `ItemList` structures with conservative DOM fallback via itemprop, price and rating classes, aria-label, and visible stock text; non-commerce lists are unchanged.
+- Recover GitHub README content across multiple selector variants (`article.markdown-body`, `[data-testid='readme-content']`, aria containers) with a compact project-summary fallback when no README is present; GitLab behavior is unchanged.
+- Enrich Google-style consent summaries with visible headings, consent paragraphs, bullets, and option/control labels; prefer a visible heading over the page title so control labels are not pushed past the highlight cutoff. No bypass or dismissal behavior is added.
+- Deepen Antora landing card detail extraction with card-scoped boundaries and preserved relative hrefs; add nested STLDocs schema property groups and method-local parameter/response field bullets with shared docs-scoped text helpers.
+- Fix pre-existing RuboCop offenses: split the 724-character multilingual homepage-phrase regex in `fetcher.rb` into a `Regexp.new` with string continuations, and correct `Style/RaiseArgs` in `browser_spec.rb`.
+- Remove ~1,337 lines of manifest-ordered dead code where later source files silently overrode earlier ones: collapse `dom_base.js` (9 functions overridden by `dom_cleanup.js`), `lists.js` (18 functions overridden by `list_extraction.js`), and `generic_docs.js` (15 functions overridden by `generic_docs_frameworks.js`); remove `generic_docs.js` from the asset manifest.
+- Restore missing forum/thread selectors in the live `list_extraction.js` that were present only in the dead `lists.js` override: `[class*='thread']`, `[class*='topic-list']`, `.structItem`, `.discussionListItem`.
+- Merge mintlify docs selectors from the dead `generic_docs.js` into the live `generic_docs_frameworks.js`: table-of-contents, context menu, eyebrow, ctrl keybindings, ask-an-ai cleanup, and pagination selectors.
+- Extract and adopt shared JS helpers: `listContentResult` and `bodyInnerText` in `core/metadata.js`; adopt existing `docsHostSignature` (7 sites) and `cleanDocsHeadings` (6 sites); refactor `listChromeOrNavigationNode` and `parseInstagramStats`.
+- Extract a shared `COOKIE_CONSENT_KEYWORDS` multilingual constant in `challenges.js` consumed by `consentWallDominates`, `consentLikeInterstitial`, and `consentWallPage`; remove dead `challengeNoiseText`.
+- Extract shared Ruby helpers: `FetchUtil.strip_www_host` (4 sites), `Regulatory#fetch_record` (4 record methods), and `Regulatory#signal_sort_prefix` (2 sort methods); `tdm_rep.rb` now calls existing `extract_tdm_value_signals`.
+- Extract shared spec support (`fetcher_spec_helpers.rb`, `fixture_html.rb`) and split large spec files: 4 Nordic/Baltic consent examples from `content_quality_spec.rb` into `consent_language_walls_spec.rb`, 6 Reddit/Behance examples from `consent_and_social_walls_spec.rb` into `social_platform_walls_spec.rb`.
+- Remove `require "bundler/setup"` from the installed executable so `fetch_util` works from inside any Ruby project, even ones whose Gemfile does not list fetch_util. Bundler is still activated by `bundle exec` for repo-local development.
+- Change `fetch` default output format from JSON to pure markdown so agents get clean readable content without parsing JSON. Use `--format json` for structured output with metadata, warnings, and content_type fields.
+
+## v0.2.1 - 2026-04-09
+
+- Detect liveblog and briefing/digest content formats via structured data, DOM heuristics, and multilingual title patterns; expose `content_format` field and `multi_topic_page` warning.
+- Detect paywall signals via structured data, meta tags, DOM elements, and multilingual text patterns; expose `paywall_state` field and `paywall_partial_content` warning.
+- Add diacritics-aware slug matching to reduce false `url_content_mismatch` warnings for Polish, Turkish, Latvian, and other accented-language URLs.
+- Add ratio-based truncation detection and `content_completeness_ratio` field for better short-extraction diagnostics.
+- Expand consent button patterns for Finnish, Lithuanian, Macedonian, and Romanian; support trailing clause variants.
+- Strip related-content sections by multilingual heading detection across 20+ European languages.
+- Expand language stopword coverage from 7 to 22 languages for more accurate content-language heuristics.
+- Remove Gemfile.lock from version control.
+
+## v0.2.0 - 2026-04-08
+
+- Reuse browser process across fetches instead of spawning Chromium per URL, dramatically reducing batch fetch overhead.
+- Recover partial results on parallel fetch failures instead of discarding all progress.
+- Retry navigation on transient PendingConnectionsError and TimeoutError before raising.
+- Log per-URL fetch duration in the request log.
+- Add comprehensive Polish-language noise stripping to DOM cleanup, markdown post-processing, and sidebar heading detection.
+- Remove hard caps on index link extraction so all scored candidates are returned from generic and site-profile extractors.
+
+## v0.1.1 - 2026-04-06
+
+- Initial standalone `rbutils` gem scaffold for reader-friendly web fetch.
+- Added Ferrum-based page loading, Readability-first extraction, heuristic fallback extraction, and Turndown markdown conversion.
+- Added RSpec coverage, RuboCop config, and a basic end-to-end smoke verification path.

data/LICENSE.txt

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 rbutils contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,199 @@
+# fetch_util
+
+Reliable browser-backed fetching for Ruby.
+
+`fetch_util` renders modern pages, inspects the live DOM, classifies page shape, and returns compact markdown plus structured metadata.
+
+It also provides a plain-Ruby regulatory inspector for machine-readable crawl, index, and text-and-data-mining signals such as `robots.txt`, `X-Robots-Tag`, robots meta tags, and TDM reservation metadata.
+
+It helps applications distinguish between content pages and access/interstitial states such as consent prompts, login-required pages, and challenge screens. When original content is not available, it returns a compact summary with warnings rather than pretending the page was extracted successfully.
+
+## How It Works
+
+The easiest way to explain `fetch_util` is in three steps:
+
+- `Render` - load the page in Chromium, inspect the rendered DOM, and read page metadata.
+- `Classify` - identify whether the page is an article, list/index, docs page, search result, or an interstitial/access-limited state.
+- `Shape` - return compact markdown, normalized URLs, and warning metadata so the result is usable by agents, LLM workflows, and ordinary Ruby applications.
+
+In short: `fetch_util` makes the web easier to build on.
+
+## Installation
+
+Add the gem to your Gemfile:
+
+```ruby
+gem "fetch_util"
+```
+
+Then install dependencies:
+
+```sh
+bundle install
+```
+
+## Quick Start
+
+```ruby
+require "fetch_util"
+
+result = FetchUtil.fetch(
+  "https://example.com/article",
+  timeout: 20,
+  wait: 0.75,
+  wait_for_idle: true,
+  viewport: { width: 1366, height: 900 }
+)
+
+puts result.title
+puts result.markdown
+puts result.final_url
+puts result.canonical_url
+puts result.content_type
+puts result.warnings.inspect
+```
+
+## CLI
+
+Repo-local usage:
+
+```sh
+bundle exec exe/fetch_util fetch https://example.com/article
+bundle exec exe/fetch_util fetch https://example.com/a https://example.com/b --format jsonl
+bundle exec exe/fetch_util search ruby language
+bundle exec exe/fetch_util regulatory https://example.com
+bundle exec exe/fetch_util regulatory https://example.com/article --sources=machine,human
+```
+
+Installed gem usage:
+
+```sh
+fetch_util fetch https://example.com/article
+fetch_util search ruby language
+fetch_util regulatory https://example.com/article --sources=machine,human
+```
+
+## API
+
+- `FetchUtil.fetch(url, **options)` returns a `FetchUtil::Result`
+- `FetchUtil.fetch_many(urls, **options)` fetches multiple URLs in parallel and preserves input order
+- `FetchUtil.search(query, **options)` returns compact aggregated search results
+- `FetchUtil.regulatory(url, **options)` returns a source-keyed hash of allow/disallow signals for crawling, indexing, and TDM-style usage
+- `FetchUtil::Fetcher.new(**options).fetch(url)` exposes the instance API directly
+
+Useful result fields:
+
+- `title`
+- `markdown`
+- `final_url`
+- `canonical_url`
+- `content_type` (`article`, `list`, or `search`)
+- `suspect`
+- `warnings`
+
+## Common Options
+
+- `timeout:` browser timeout in seconds
+- `wait:` additional settle delay after page load
+- `wait_for_idle:` wait for Ferrum network idle before extraction
+- `idle_duration:` idle duration passed to Ferrum when `wait_for_idle` is enabled
+- `reader_mode:` prefer Readability before heuristic fallbacks
+- `viewport:` viewport hash with `:width` and `:height`
+- `user_agent:` override the browser user agent
+- `accept_language:` override request language headers
+- `browser_path:` explicit Chromium path
+
+## Output Shape
+
+`fetch` defaults to compact JSON intended for downstream agent/tool consumption. The default payload keeps the fields that are usually most useful in practice:
+
+- `url`
+- `final_url`
+- `canonical_url`
+- `title`
+- `byline`
+- `site_name`
+- `published_time`
+- `markdown`
+- `content_type`
+- `suspect`
+- `warnings`
+
+Pass `--include-html` when you explicitly need extracted HTML. Multiple fetch URLs can be streamed as JSON Lines with `--format jsonl`.
+
+Both CLI commands append requests to `~/.local/state/fetch_util/requests.log` by default. Override with `FETCH_UTIL_REQUEST_LOG` or `--log-path`.
+
+## Regulatory
+
+`regulatory` inspects machine-readable and rough human-readable signals about what a site allows or disallows for crawling, indexing, and text-and-data-mining style use.
+
+- default source class: `machine`
+- source selector syntax: `--sources=human,machine,-robotstxt`
+- current machine sources:
+  - `robotstxt`
+  - `contentsignal`
+  - `contentusagerobots`
+  - `contentusageheader`
+  - `trusttxt`
+  - `xrobotstag`
+  - `metarobots`
+  - `tdmrep`
+  - `tdmheaders`
+  - `tdmmeta`
+  - `tdmpolicy`
+- current human source:
+  - `human`
+- structured per-request cache path: `~/.local/state/fetch_util/regulatory-cache`
+
+The regulatory inspector now understands both Cloudflare-style `Content-Signal` robots rules and the emerging IETF AIPREF `Content-Usage` syntax in `robots.txt` and HTTP response headers.
+
+It also understands site-wide `trust.txt` declarations using `datatrainingallowed=yes|no`, with `/trust.txt` first and `/.well-known/trust.txt` as fallback.
+
+Origin-level queries such as `https://example.com` keep source paths in the output. Path/resource queries such as `https://example.com/article` filter to matching signals and omit the path field.
+
+Example Ruby usage:
+
+```ruby
+require "fetch_util"
+
+pp FetchUtil.regulatory(
+  "https://example.com/article",
+  sources: "machine,human"
+)
+```
+
+## Behavior
+
+- Extracts articles, list/index pages, and search pages into compact markdown.
+- Uses page classification to select extraction logic appropriate to the rendered page type.
+- Detects consent prompts, login-required pages, and challenge/interstitial screens and reports them with concise summaries and warning tags.
+- Cleans up docs/reference pages aggressively enough for agent consumption.
+- Preserves `final_url`, `canonical_url`, and warning metadata so callers can reason about redirects, mismatches, and interstitials.
+- Extracts regulatory allow/disallow signals from `robots.txt`, page headers/meta tags, and TDM reservation metadata without caching raw page bodies.
+
+## Compliance Boundaries
+
+`fetch_util` is for rendering and summarizing publicly delivered page output. It may identify consent prompts, login-required pages, and challenge/interstitial states and return warning metadata for them. It is not intended to bypass account requirements, paywalls, verification systems, or other access controls.
+
+Browser-profile normalization is intentionally limited to reducing obvious runtime inconsistencies that would otherwise change page behavior during extraction.
+
+## Development
+
+Run from `/srv/code/rbutils/fetch_util`:
+
+```sh
+bundle exec rake build_extract_assets
+bundle exec rake verify_extract_assets
+bundle exec rspec
+bundle exec rake rubocop
+bundle exec rake
+```
+
+- The shipped browser bundle is `lib/fetch_util/assets/extract.js`.
+- Source JS lives under `lib/fetch_util/assets/src/` and is ordered by `lib/fetch_util/assets/src/manifest.txt`.
+- `bundle exec rake build_extract_assets` rebuilds the bundle and runs `npx terser -cm` before writing `extract.js`.
+- `bundle exec rake verify_extract_assets` checks that the built bundle matches the current sources.
+- The default `bundle exec rake` task runs asset verification, specs, and RuboCop.
+- Direct `bundle exec rspec` runs still check bundle freshness through `spec/build_extract_assets_spec.rb` and enforce the repo-wide SimpleCov minimum.
+
+Do not hand-edit `lib/fetch_util/assets/extract.js`; edit the source files under `lib/fetch_util/assets/src/` and rebuild.

data/Rakefile

@@ -0,0 +1,18 @@
+# frozen_string_literal: true
+
+require "bundler/gem_tasks"
+require "rspec/core/rake_task"
+require "rubocop/rake_task"
+
+RSpec::Core::RakeTask.new(:spec)
+RuboCop::RakeTask.new
+
+task :build_extract_assets do
+  ruby "script/build_extract_assets.rb"
+end
+
+task :verify_extract_assets do
+  ruby "script/build_extract_assets.rb", "--check"
+end
+
+task default: %i[verify_extract_assets spec rubocop]

data/SKILL.md

@@ -0,0 +1,92 @@
+---
+name: fetch_util
+description: Use fetch_util first for ordinary web fetch/search because it returns cleaner, agent-friendly markdown; fall back only if unavailable or insufficient.
+license: MIT
+---
+
+# FetchUtil Skill
+
+Use this skill for agent web retrieval.
+
+Start by loading this skill, then use `fetch_util` as the default first tool for ordinary web-page retrieval and search.
+Use it first because it usually returns cleaner, more agent-friendly markdown than generic web fetch, and it is cheap enough for repeated multi-pass retrieval.
+
+## Default Rule
+
+- Use `fetch_util` first for ordinary web-page retrieval or search.
+- If `fetch_util` is installed on PATH, use the `fetch_util ...` command first from any working directory. It works inside any Ruby project, even ones whose Gemfile does not list fetch_util.
+- If you are developing inside the repository and want the local worktree version specifically, use `bundle exec exe/fetch_util ...` from `/srv/code/rbutils/fetch_util`.
+- Never use `bundle exec fetch_util` inside another Ruby project. Bundler restricts executable lookup to the project's Gemfile and will reject fetch_util if it is not listed there. Use bare `fetch_util ...` instead.
+- If you are running inside a delegated subagent that does not expose the `skill` tool, use the installed `fetch_util ...` CLI directly instead of falling back to built-in web fetch/search right away.
+- Use built-in `webfetch` or other web tooling only after `fetch_util` is unavailable, the target is not a normal web page, or the task needs something `fetch_util` does not provide.
+- `fetch_util` is cheap to use, so an agent can use this tool as much as it needs for normal retrieval, search, and follow-up fetches.
+
+## Use It For
+
+- fetching one or more known URLs
+- searching the web and returning compact agent-friendly results
+- getting cleaner markdown from noisy pages, docs, search pages, or light interstitials
+
+## Core Commands
+
+Fetch known URLs (returns pure markdown by default):
+
+```sh
+fetch_util fetch https://example.com
+fetch_util fetch https://example.com/a https://example.com/b
+```
+
+Fetch with structured JSON output (when you need metadata, warnings, or content_type):
+
+```sh
+fetch_util fetch https://example.com --format json
+fetch_util fetch https://example.com/a https://example.com/b --format jsonl
+```
+
+Search first, then fetch selected results if needed:
+
+```sh
+fetch_util search ruby language
+fetch_util search site:docs.python.org json dump --verbose-search
+```
+
+Repository-local development form:
+
+```sh
+bundle exec exe/fetch_util fetch https://example.com
+bundle exec exe/fetch_util search ruby language
+```
+
+## Agent Guidance
+
+- if the user gives you URLs, use `fetch_util fetch` first
+- if the user needs discovery, use `fetch_util search` first
+- if the task is a normal web roundup (for example, checking several news homepages), still use `fetch_util` first; do not skip straight to built-in web fetch just because the URLs are already known
+- if you are in a subagent without the `skill` tool, treat `fetch_util` as a normal installed CLI and call it directly
+- use `fetch_util` first because its output is usually cleaner and more compact for agents than generic page fetch output
+- treat `fetch_util` as cheap to use; it is fine to make multiple fetch/search passes when that helps answer the task well
+- prefer the compact default output; use `--format json` when you need metadata, warnings, or content_type fields, and `--format jsonl` for multi-result pipelines
+- use `--include-html` only when raw HTML is actually needed
+- treat `suspect` and `warnings` as signals that the page may be an interstitial, challenge, or mismatch
+- only fall back to other web tooling after `fetch_util` is unavailable or clearly insufficient
+- never prefix with `bundle exec` when running inside another Ruby project; use bare `fetch_util ...` instead
+
+## Installation
+
+If `fetch_util` is not available on the machine yet, install the gem first:
+
+```sh
+gem install fetch_util
+```
+
+OpenCode global install:
+
+```sh
+mkdir -p ~/.config/opencode/skills/fetch_util && curl -fsSL https://raw.githubusercontent.com/rbutils/fetch_util/master/SKILL.md -o ~/.config/opencode/skills/fetch_util/SKILL.md
+```
+
+Repo-local install:
+
+```sh
+mkdir -p .opencode/skills/fetch_util && cp SKILL.md .opencode/skills/fetch_util/SKILL.md
+```

data/exe/fetch_util

@@ -0,0 +1,6 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+require "fetch_util"
+
+FetchUtil::CLI.start(ARGV)