html2rss 0.28.0 → 0.29.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 38cdc153241c05684ee40477e4df06ca2c88f013c971387ddd0ee6d8163c178a
-  data.tar.gz: eda1c02f0199568f8f71d24b22480b5eaa9a83c7e48c23678e9695e98e2dab8f
+  metadata.gz: 32c090eccd93210c9631778ff50ec7d32d36c7eccaa711cdd1a0dd5c3ea53616
+  data.tar.gz: 93e4d2f253a019d7de585c8acaaa8d06143accb4f3a43a4f8d13f9c96d838390
 SHA512:
-  metadata.gz: 5e063256aa063027d53986e38703cc0065d12e7aaee45cdfa7724a724f599a0f645e836a0573c9633865d007532bb82c3c2a6632984c5385dcc6fafd5ed0409d
-  data.tar.gz: 146ab91ee7925fc678ecdbe16895bd830bf9d7620bfc5239ccf12e109ef0913142da27f77c1be34f839cce350f51552ebcf163a5c92448c0b836b6a9fd8d7e4f
+  metadata.gz: 93956c4dc3d35e06c81dff939d4ea5184da93834a0fb48f1b3dbd4dea21d615177ad45fa34e4f16abf396a184cc18acbe31c354df6764ba858ec340c78a3b08c
+  data.tar.gz: b906784ed2ae53b80aa2fc00fe813550befcfac723d97346ffb87cff7f01ca6b0f5db0a6e4a2430dd10485c20f6d4b251a54a1b9e0462998f2ab628c0123a809

data/README.md

@@ -35,13 +35,63 @@ Cloud development: [Open in GitHub Codespaces](https://github.com/codespaces/new
 Config -> Request -> Extraction -> Processing -> Building -> Output
 ```
 
+## CLI Usage
+
+| Verb     | Job                                                           |
+| -------- | ------------------------------------------------------------- |
+| inspect  | Cheap diagnostics (final URL, status, alternates, surface)    |
+| recon    | Verdict + native feed preference (`BUILD` / `DEFER` / `DROP`) |
+| capture  | YAML draft config                                             |
+| validate | Schema only                                                   |
+| test     | Schema + live extraction (min items)                          |
+| apply    | Ship RSS from config or URL                                   |
+| scrape   | Articles now (one-shot auto-source)                           |
+
+Golden path: optional **inspect → recon → capture → test → apply**. Side door: **validate**. One-shot: **scrape**.
+
+```bash
+# Diagnostics and reconnaissance
+html2rss inspect https://example.com/news
+html2rss recon https://example.com/news
+html2rss recon --file urls.txt --verdict BUILD --url-only
+
+# Composable pipes
+html2rss recon --file urls.txt --verdict BUILD --url-only | html2rss capture -
+html2rss capture https://example.com/news | html2rss test -
+
+# One-shot articles now
+html2rss scrape https://example.com/news
+html2rss scrape https://example.com/news --format jsonfeed --explain
+
+# Durable config workflow
+html2rss capture https://example.com/news --write feed.yml
+html2rss test feed.yml --min-items 5
+html2rss apply feed.yml
+
+# Schema validation (side door)
+html2rss validate config.yml
+html2rss validate "configs/**/*.yml"
+
+# Export JSON Schema
+html2rss schema --write schema/html2rss-config.schema.json
+```
+
+Historic CLI aliases: `feed` → `apply`, `auto` → `scrape`.
+
+### Inspect output
+
+Inspect follows redirects and reports the landing URL in `final_url`. CLI text shows a `Final:` line **only when** the landing URL differs from what you typed — that line means the redirect succeeded, not that inspect stopped early.
+
+Cross-host redirects (e.g. `https://apex.example/` → `https://www.example/`) set `Host` per hop via Faraday/Net::HTTP; html2rss does not pin the entry hostname. When `final_url` differs and status is 4xx, retry on `final_url` or pass the site's canonical hostname. Details: [`lib/html2rss/page_recon/README.md`](lib/html2rss/page_recon/README.md).
+
 ## Capture API
 
-The `Html2rss.capture` method analyzes any URL and produces a reusable feed config hash with an items selector and `enhance: true`. Use it to speed up writing feed configuration files.
+`Html2rss.capture` returns a `Capture::CaptureResult`. Use `result.yaml` or `result.config`.
 
 ```ruby
-config = Html2rss.capture('https://example.com/articles')
-File.write('my-feed.yml', Html2rss::Config.to_yaml(config))
+result = Html2rss.capture('https://example.com/articles')
+File.write('my-feed.yml', result.yaml)
+# or: File.write('my-feed.yml', Html2rss::Config.to_yaml(result.config))
 ```
 
 The CLI alias `html2rss capture` prints the generated config as YAML to stdout. See [`lib/html2rss/capture/README.md`](lib/html2rss/capture/README.md) for detailed documentation.
@@ -62,9 +112,9 @@ stdio uses stdout for JSON-RPC, so the daemon logs to **stderr**. It defaults to
 
 HTTP transport needs `rack`, `rackup`, and `webrick` (declared gem dependencies). It listens on `127.0.0.1` only; do not expose it on a public interface without your own auth and Host/Origin controls.
 
-**Strategy note:** MCP `scrape_url` / `capture_config` with `strategy: "auto"` run Faraday → Botasaurus AutoFallback. `inspect_url` uses Faraday when `auto` (cheap diagnostic); pin `botasaurus` when you need browser rendering for inspect.
+**Strategy note:** MCP `scrape` / `capture` with `strategy: "auto"` run Faraday → Botasaurus AutoFallback. `inspect` uses Faraday when `auto` (cheap diagnostic); pin `botasaurus` when you need browser rendering for inspect.
 
-**Tool-call budget:** `scrape_url` is 1 call (auto already hops). Durable config is `capture_config` → `validate_config` → `apply_config`. Call `inspect_url` only when scrape/capture is weak or you need recon (final URL, status, https→http, native RSS/Atom).
+**Tool-call budget:** `scrape` is 1 call (auto already hops). Durable config is `capture` → `test` → `apply` (or `validate` → `test` → `apply` when you already have YAML). Call `inspect` only when scrape/capture is weak or you need recon (final URL, status, https→http, native RSS/Atom).
 
 Cursor / Claude Desktop `mcp.json` must put Botasaurus on the **MCP process** (not only your shell):
 
@@ -82,17 +132,24 @@ Cursor / Claude Desktop `mcp.json` must put Botasaurus on the **MCP process** (n
 }
 ```
 
-Read `html2rss://runtime` for a boolean `botasaurus_configured` (the URL is never returned). Every tool result is a JSON envelope (`ok`, `next_step`, `guidance`, `payload`) in both the text body and `structuredContent`. Follow `next_step` / `guidance`; do not parse scrape text as a raw item array.
+Read `html2rss://runtime` for `version`, `mcp_contract_version`, `catalog_fingerprint`, `tools`, and `botasaurus_configured` (the scraper URL is never returned). Refresh `tools/list` when `catalog_fingerprint` differs from your cache. Every tool result is a JSON envelope (`ok`, `next_step`, `guidance`, `payload`) in both the text body and `structuredContent`. Follow `next_step` / `guidance`; do not parse scrape text as a raw item array.
+
+Module guide: [`lib/html2rss/mcp/README.md`](lib/html2rss/mcp/README.md).
 
 ### Tools
 
-| Name              | When to use                                                                 |
-| ----------------- | --------------------------------------------------------------------------- |
-| `scrape_url`      | One-shot articles now (`payload.items`; empty is still success)             |
-| `inspect_url`     | Weak scrape/capture or recon (final_url, status, scheme_downgrade, feeds)   |
-| `capture_config`  | YAML draft in `payload.yaml`; strive `enhance: true`                        |
-| `validate_config` | Schema-check a `config` hash XOR `yaml` string (`isError` on failure)       |
-| `apply_config`    | RSS in `payload.rss`; `isError` when zero items; confirm `payload.item_count` |
+| Name            | When to use                                                                    |
+| --------------- | ------------------------------------------------------------------------------ |
+| `scrape`        | One-shot articles now (`payload.items`; empty is still success)                |
+| `batch_scrape`  | Parallel one-shot scrape across multiple URLs (`urls`, `limit`, `concurrency`) |
+| `inspect`       | Weak scrape/capture or recon (final_url, status, scheme_downgrade, feeds)      |
+| `batch_inspect` | Parallel diagnostics across multiple URLs (`urls`, `strategy`, `concurrency`)  |
+| `recon`         | Verdict + native_feed preference                                               |
+| `batch_recon`   | Parallel recon across multiple URLs                                            |
+| `capture`       | YAML draft in `payload.yaml`; strive `enhance: true`                           |
+| `validate`      | Schema-check a `config` hash XOR `yaml` string (`isError` on failure)          |
+| `test`          | Schema + live extraction; `quality_report.enhance_gains` when enhance on; optional `compare_enhance` |
+| `apply`         | RSS in `payload.rss`; `isError` when zero items; `quality_report` may include `enhance_gains`          |
 
 ### Resources
 
@@ -100,15 +157,15 @@ Read `html2rss://runtime` for a boolean `botasaurus_configured` (the URL is neve
 | ----------------------- | --------------------------------------------------------------- |
 | `html2rss://schema`     | Full JSON Schema for feed configurations                        |
 | `html2rss://extractors` | Registered extractor **names** (options live in schema `$defs`) |
-| `html2rss://strategies` | Published MCP strategies (`auto`, `faraday`, `botasaurus`) |
-| `html2rss://runtime`    | `botasaurus_configured` boolean (never the scraper URL)         |
+| `html2rss://strategies` | Published MCP strategies (`auto`, `faraday`, `botasaurus`)      |
+| `html2rss://runtime`    | `version`, `mcp_contract_version`, `catalog_fingerprint`, `tools`, `botasaurus_configured` (never the scraper URL) |
 
 ### Prompts
 
-| Name                  | Description                                                          |
-| --------------------- | -------------------------------------------------------------------- |
-| `scrape-webpage`      | One `scrape_url` call; inspect only if weak or recon                 |
-| `capture-feed-config` | Capture YAML → validate → apply; catalog rewrite; strive enhance     |
+| Name                  | Description                                                  |
+| --------------------- | ------------------------------------------------------------ |
+| `scrape-webpage`      | One `scrape` call; `inspect` only if weak or recon needed    |
+| `capture-feed-config` | Capture YAML → test → apply; catalog rewrite; strive enhance |
 
 The MCP module (`Html2rss::MCP`) lazy-loads the `mcp` gem — no cost when the server is not running.
 
@@ -124,13 +181,13 @@ Set `BOTASAURUS_SCRAPER_URL` to `http://127.0.0.1:4010` and use strategy `botasa
 
 ## Request Strategies
 
-| Strategy     | Description                                                                   |
-| ------------ | ----------------------------------------------------------------------------- |
-| `auto`       | Tries `faraday`, falls back to `botasaurus` (default in gem/CLI/MCP scrape)   |
-| `faraday`    | Plain HTTP requests via Faraday                                               |
-| `botasaurus` | Puppeteer-backed scraping for JavaScript pages                                |
+| Strategy     | Description                                                                 |
+| ------------ | --------------------------------------------------------------------------- |
+| `auto`       | Tries `faraday`, falls back to `botasaurus` (default in gem/CLI/MCP scrape) |
+| `faraday`    | Plain HTTP requests via Faraday                                             |
+| `botasaurus` | Puppeteer-backed scraping for JavaScript pages                              |
 
-`inspect_url` keeps Faraday when `auto` for cheap diagnostics. Elsewhere, strategy can be set via CLI (`--strategy`), gem API keyword argument, or feed config `request.strategy`. See the [request strategies docs](https://html2rss.github.io/ruby-gem/reference/strategy) for more details.
+`inspect` keeps Faraday when `auto` for cheap diagnostics. Elsewhere, strategy can be set via CLI (`--strategy`), gem API keyword argument, or feed config `request.strategy`. See the [request strategies docs](https://html2rss.github.io/ruby-gem/reference/strategy) for more details.
 
 ## License
 

data/html2rss.gemspec

@@ -45,7 +45,6 @@ Gem::Specification.new do |spec|
   spec.add_dependency 'rack', '~> 3.0'
   spec.add_dependency 'rackup', '~> 2.0'
   spec.add_dependency 'regexp_parser'
-  spec.add_dependency 'reverse_markdown', '~> 3.0'
   spec.add_dependency 'rss'
   spec.add_dependency 'sanitize'
   spec.add_dependency 'thor'

data/lib/html2rss/article.rb

@@ -8,7 +8,7 @@ module Html2rss
   # It is enumerable and responds to all keys specified in PROVIDED_KEYS.
   #
   # Description and enclosure wire presentation live in {FeedBuilder::ItemPresentation}.
-  # rubocop:disable Metrics/ClassLength -- value object retains Marshal + defensive freeze helpers
+  # rubocop:disable-next Metrics/ClassLength -- value object retains Marshal + defensive freeze helpers
   class Article
     include Enumerable
     include Comparable
@@ -209,5 +209,4 @@ module Html2rss
       value
     end
   end
-  # rubocop:enable Metrics/ClassLength
 end

data/lib/html2rss/auto_source/README.md

@@ -32,14 +32,14 @@ Segmenter strategies: `:semantic` (leaf containers + primary link), `:list` (rep
 
 ## Nokogiri vs SST boundaries
 
-| Surface | Owns DOM |
-| --- | --- |
-| `Response#parsed_body` | Single HTML parse for the page |
-| Schema / Microdata / MF2 / JsonState / XhrArticles / MetaOembed / app-shell detection | Nokogiri |
-| Sitemap detection (CSS/XPath) | Nokogiri; URL list parsing uses raw `response.body` (XML string) |
-| Selectors path / Sanitize transformers | Nokogiri (unchanged) |
-| `SST::Normalizer` | Sole Nokogiri consumer on the heuristic auto-source path |
-| Segmenter, Scoring, `Html::SstArticleExtractor`, heuristic chrome (`SST::Tags` / `SST::Text`) | SST only |
+| Surface                                                                                       | Owns DOM                                                         |
+| --------------------------------------------------------------------------------------------- | ---------------------------------------------------------------- |
+| `Response#parsed_body`                                                                        | Single HTML parse for the page                                   |
+| Schema / Microdata / MF2 / JsonState / XhrArticles / MetaOembed / app-shell detection         | Nokogiri                                                         |
+| Sitemap detection (CSS/XPath)                                                                 | Nokogiri; URL list parsing uses raw `response.body` (XML string) |
+| Selectors path / Sanitize transformers                                                        | Nokogiri (unchanged)                                             |
+| `SST::Normalizer`                                                                             | Sole Nokogiri consumer on the heuristic auto-source path         |
+| Segmenter, Scoring, `Html::SstArticleExtractor`, heuristic chrome (`SST::Tags` / `SST::Text`) | SST only                                                         |
 
 Production heuristic scrapers should reuse one `SST::Document` memoized from `parsed_body` (or a shared Document passed in), not re-parse HTML strings.
 

data/lib/html2rss/auto_source/cleanup.rb

@@ -47,13 +47,22 @@ module Html2rss
         [:video_chrome, /\AClipped\s+From\s+Video\b/i],
         [:video_chrome, /\AVideo\s*[•·]/i],
         [:template, /\ACreated\s+from\s+Template\s+ID\b/i],
-        [:template, /(\{\{[^}]+\}\}|%\{\w+\})/]
+        [:template, /(\{\{[^}]+\}\}|%\{\w+\})/],
+        [:cta, /\Aread more\z/i],
+        [:cta, /\Alearn more\z/i],
+        [:cta, /\Apdf\z/i]
       ].freeze
       private_constant :JUNK_TITLE_RULES
 
+      # Minimum present-title length before short-title warnings.
+      MIN_TITLE_LENGTH = 4
+
       # Admitted articles plus reason → count tallies for drops.
       Result = Data.define(:articles, :drop_tallies)
 
+      # Read-only ship-quality audit on RSS/article-like items (no mutation).
+      AuditResult = Data.define(:warnings, :metrics, :violations)
+
       class << self
         # @param articles [Array<Article>] extracted article candidates
         # @param url [Html2rss::Url] feed source URL used for same-host filtering
@@ -88,8 +97,83 @@ module Html2rss
           JUNK_TITLE_RULES.find { |_, pattern| pattern.match?(normalized) }&.first
         end
 
+        # @param items [Array] RSS or article-like objects with title and link/url
+        # @return [AuditResult]
+        def audit_feed_items(items)
+          metrics = initial_audit_metrics(items.size)
+          violations = Hash.new(0)
+          warnings = audit_url_diversity(items, metrics, violations)
+          audit_item_titles(items, metrics, violations)
+          warnings = finalize_audit_warnings(warnings, metrics)
+          log_audit(metrics, warnings, violations)
+          AuditResult.new(warnings:, metrics: metrics.freeze, violations: violations.freeze)
+        end
+
         private
 
+        def initial_audit_metrics(item_count)
+          {
+            item_count:,
+            unique_url_count: 0,
+            junk_title_count: 0,
+            short_title_count: 0,
+            low_word_count: 0
+          }
+        end
+
+        def audit_url_diversity(items, metrics, violations)
+          unique_urls = items.filter_map { |item| url_identity(item_url(item)) }.uniq
+          metrics[:unique_url_count] = unique_urls.size
+          return [] unless items.size >= 2 && unique_urls.size < 2
+
+          violations[:duplicate_urls] += 1
+          [:duplicate_urls]
+        end
+
+        def audit_item_titles(items, metrics, violations)
+          items.each do |item|
+            title = normalize_title(item.title)
+            next if title.empty?
+
+            audit_short_title(title, metrics, violations)
+            audit_title_quality(title, metrics, violations)
+          end
+        end
+
+        def audit_short_title(title, metrics, violations)
+          return unless title.length < MIN_TITLE_LENGTH
+
+          metrics[:short_title_count] += 1
+          violations[:short_title] += 1
+        end
+
+        def audit_title_quality(title, metrics, violations)
+          reason = junk_reason(title)
+          if reason
+            metrics[:junk_title_count] += 1
+            violations[reason] += 1
+          elsif !word_count_at_least?(title, MIN_WORDS)
+            metrics[:low_word_count] += 1
+            violations[:low_word_count] += 1
+          end
+        end
+
+        def finalize_audit_warnings(warnings, metrics)
+          warnings = warnings.dup
+          warnings << :generic_titles if metrics[:junk_title_count].positive?
+          warnings << :short_titles if metrics[:short_title_count].positive?
+          warnings << :low_word_count if metrics[:low_word_count].positive?
+          warnings.freeze
+        end
+
+        def log_audit(metrics, warnings, violations)
+          Log.debug(
+            'Cleanup.audit_feed_items: ' \
+            "item_count=#{metrics[:item_count]} unique_urls=#{metrics[:unique_url_count]} " \
+            "warnings=#{warnings.join(',')} violations=#{violations.keys.join(',')}"
+          )
+        end
+
         def reject_invalid!(articles, tallies)
           tally_reject!(articles, tallies, 'invalid') { |article| !article.valid? }
         end
@@ -172,6 +256,13 @@ module Html2rss
           url&.without_fragment&.to_s
         end
 
+        def item_url(item)
+          raw = item.respond_to?(:link) ? item.link : item.url
+          raw.nil? || raw.to_s.empty? ? nil : Html2rss::Url.from_absolute(raw.to_s)
+        rescue ArgumentError
+          nil
+        end
+
         def normalize_title(title)
           title.to_s.strip.gsub(/\s+/, ' ')
         end

data/lib/html2rss/auto_source/scraper/json_state/document_scanner.rb

@@ -8,8 +8,6 @@ module Html2rss
       class JsonState
         # Scans DOM nodes for JSON payloads containing article data.
         module DocumentScanner # rubocop:disable Metrics/ModuleLength
-          # Selector for JSON-only script tags.
-          JSON_SCRIPT_SELECTOR = 'script[type="application/json"]'
           # Regex patterns for known global JavaScript state assignments.
           GLOBAL_ASSIGNMENT_PATTERNS = [
             /(?:window|self|globalThis)\.__NEXT_DATA__\s*=\s*/m,
@@ -35,16 +33,16 @@ module Html2rss
           def json_documents(parsed_body)
             # Use identity-based cache to avoid double-parsing of the same document.
             # WeakMap allows the Nokogiri Document (key) to be garbage collected.
-            # rubocop:disable ThreadSafety/ClassInstanceVariable
+            # rubocop:disable-next ThreadSafety/ClassInstanceVariable
             (@cache ||= ObjectSpace::WeakMap.new)[parsed_body] ||=
               script_documents(parsed_body) + assignment_documents(parsed_body)
-            # rubocop:enable ThreadSafety/ClassInstanceVariable
           end
 
           # @param parsed_body [Nokogiri::HTML::Document] parsed HTML document
           # @return [Array<Hash, Array>] JSON documents extracted from JSON script tags
           def script_documents(parsed_body)
-            parsed_body.css(JSON_SCRIPT_SELECTOR).filter_map { parse_json(_1.text) }
+            ::Html2rss::Html::Probe.scripts(parsed_body, ::Html2rss::Html::Probe::APPLICATION_JSON)
+                                   .filter_map { parse_json(_1.text) }
           end
 
           # @param parsed_body [Nokogiri::HTML::Document] parsed HTML document

data/lib/html2rss/auto_source/scraper/meta_oembed.rb

@@ -13,9 +13,6 @@ module Html2rss
 
         # Selector for OpenGraph meta tags.
         OG_META_SELECTOR = 'meta[property^="og:"], meta[property^="article:"], meta[name^="twitter:"]'
-        # Selector for oEmbed JSON link tag.
-        OEMBED_LINK_SELECTOR = 'link[rel="alternate"][type="application/json+oembed"][href]'
-
         # Mapping of meta property names to article attribute keys.
         META_MAP = {
           'og:title' => :title,
@@ -40,7 +37,11 @@ module Html2rss
             return false unless parsed_body
 
             !parsed_body.at_css('meta[property="og:title"]').nil? ||
-              !parsed_body.at_css(OEMBED_LINK_SELECTOR).nil?
+              ::Html2rss::Html::Probe.alternate_links(
+                parsed_body,
+                rel: 'alternate',
+                mime: ::Html2rss::Html::Probe::APPLICATION_JSON_OEMBED
+              ).any?
           end
         end
 
@@ -109,7 +110,7 @@ module Html2rss
 
         # @return [Hash{Symbol => Object}] oEmbed fields hash
         def fetch_oembed_data
-          return {} unless request_session && (link_node = parsed_body.at_css(OEMBED_LINK_SELECTOR))
+          return {} unless request_session && (link_node = oembed_link_node)
           return {} unless (oembed_url = resolve_url(link_node['href']))
 
           response = request_session.follow_up(url: oembed_url, relation: :auto_source, origin_url: url)
@@ -119,6 +120,15 @@ module Html2rss
           {}
         end
 
+        # @return [Nokogiri::XML::Element, nil] oEmbed descriptor link node
+        def oembed_link_node
+          ::Html2rss::Html::Probe.alternate_links(
+            parsed_body,
+            rel: 'alternate',
+            mime: ::Html2rss::Html::Probe::APPLICATION_JSON_OEMBED
+          ).first
+        end
+
         # @param response [Html2rss::RequestService::Response, nil] HTTP response
         # @return [Hash{Symbol => Object}] parsed oEmbed fields
         def parse_oembed_response(response)

data/lib/html2rss/auto_source/scraper/microdata.rb

@@ -39,15 +39,17 @@ module Html2rss
           # @param node [Nokogiri::XML::Element] itemscope candidate node
           # @return [String, nil] supported schema type name when present
           def supported_type_name(node)
-            normalized_types(node['itemtype']).find { SUPPORTED_TYPES.include?(_1) }
+            itemtype_type_names(node['itemtype']).find { SUPPORTED_TYPES.include?(_1) }
           end
 
           # @param itemtype [String, nil] raw itemtype attribute value
-          # @return [Array<String>] normalized schema type names
-          def normalized_types(itemtype)
+          # @return [Array<String>] canonical schema type names
+          def itemtype_type_names(itemtype)
             itemtype.to_s.split.filter_map do |value|
-              type = value.split('/').last.to_s.split('#').last.to_s
-              type unless type.empty?
+              short = value.split('/').last.to_s.split('#').last.to_s
+              next if short.empty?
+
+              Schema.canonicalize_type(short)
             end
           end
 
@@ -189,7 +191,7 @@ module Html2rss
             item = call(node)
             itemtype = node['itemtype']
             itemid = node['itemid']
-            item[:@type] = Microdata.normalized_types(itemtype).first if itemtype
+            item[:@type] = Microdata.itemtype_type_names(itemtype).first if itemtype
             item[:@id] = itemid if present?(itemid)
             item
           end

data/lib/html2rss/auto_source/scraper/native_feed.rb

@@ -31,8 +31,11 @@ module Html2rss
 
           parsed_body.css('head link[rel~="alternate"][href]').any? do |node|
             href = node['href'].to_s
-            type = node['type'].to_s.downcase
-            type.include?('rss') || type.include?('atom') || href.match?(/rss|atom|feed|\.xml/i)
+            ::Html2rss::Html::Probe.mime_match?(
+              node['type'],
+              ::Html2rss::Html::Probe::APPLICATION_RSS_XML,
+              ::Html2rss::Html::Probe::APPLICATION_ATOM_XML
+            ) || href.match?(/rss|atom|feed|\.xml/i)
           end
         end
 

data/lib/html2rss/auto_source/scraper/schema.rb

@@ -15,27 +15,33 @@ module Html2rss
       class Schema
         include Enumerable
 
-        # Selector for JSON-LD script tags containing Schema.org objects.
-        TAG_SELECTOR = 'script[type="application/ld+json"]'
+        # Matches a leading schema.org URL prefix on @type values (http or https).
+        SCHEMA_ORG_PREFIX_RE = %r{\Ahttps?://schema\.org/}i
+
+        # Container types that must never be emitted as feed items (walk children only).
+        DENIED_CONTAINER_TYPES = Set[
+          'ItemList', 'Blog', 'BreadcrumbList', 'WebPage', 'CollectionPage'
+        ].freeze
+
+        # Canonical Schema.org type names keyed by folded wire forms (case-insensitive lookup).
+        CANONICAL_BY_DOWNCASE = begin
+          canonical_types = Thing::SUPPORTED_TYPES | ItemList::SUPPORTED_TYPES | DENIED_CONTAINER_TYPES | Set['Product']
+          canonical_types.to_h { |type| [::Html2rss::Html::Probe.fold(type), type] }.freeze
+        end.freeze
 
         # Pre-compiled regex for supported schema types (short name or schema.org URL; string or array @type).
         # Allows preceding entries in a JSON @type array (e.g. ["WebPage","NewsArticle"]).
         SUPPORTED_TYPES_RE = begin
           types = Thing::SUPPORTED_TYPES | ItemList::SUPPORTED_TYPES
           type_re = Regexp.union(types.to_a)
-          %r{"@type"\s*:\s*(?:\[\s*(?:"[^"]*"\s*,\s*)*)?"(?:https?://schema\.org/)?(?:#{type_re.source})"}
+          %r{(?i)"@type"\s*:\s*(?:\[\s*(?:"[^"]*"\s*,\s*)*)?"(?:https?://schema\.org/)?(?:#{type_re.source})"}
         end.freeze
 
-        # Matches a leading schema.org URL prefix on @type values (http or https).
-        SCHEMA_ORG_PREFIX_RE = %r{\Ahttps?://schema\.org/}i
-
         # Prefer these keys when recursively walking unsupported container objects.
         COLLECTION_KEYS = %i[itemListElement blogPost mainEntity hasPart].freeze
 
-        # Container types that must never be emitted as feed items (walk children only).
-        DENIED_CONTAINER_TYPES = Set[
-          'ItemList', 'Blog', 'BreadcrumbList', 'WebPage', 'CollectionPage'
-        ].freeze
+        # Shared empty type set for nil/unsupported wire forms (avoids per-call Set.new).
+        EMPTY_TYPES = Set.new.freeze
 
         # @return [Symbol] scraper config key
         def self.options_key = :schema
@@ -44,7 +50,8 @@ module Html2rss
           # @param parsed_body [Nokogiri::HTML::Document] parsed HTML document
           # @return [Boolean] whether the page includes supported schema types
           def articles?(parsed_body)
-            parsed_body.css(TAG_SELECTOR).any? { |script| supported_schema_type?(script) }
+            ::Html2rss::Html::Probe.scripts(parsed_body, ::Html2rss::Html::Probe::APPLICATION_LD_JSON)
+                                   .any? { |script| supported_schema_type?(script) }
           end
 
           # @param script [Nokogiri::XML::Element] schema JSON-LD script tag
@@ -105,14 +112,26 @@ module Html2rss
             case object
             when Array
               object.each_with_object(Set.new) { |item, set| set.merge(normalize_types(item)) }
-            when String, Symbol
-              short = object.to_s.sub(SCHEMA_ORG_PREFIX_RE, '')
-              short.empty? ? Set.new : Set[short]
             else
-              Set.new
+              name = canonicalize_type(object)
+              name ? Set[name] : EMPTY_TYPES
             end
           end
 
+          # Canonical short Schema.org type name for a scalar wire form.
+          #
+          # @param object [String, Symbol, nil] raw `@type` / itemtype token
+          # @return [String, nil]
+          # @api private
+          def canonicalize_type(object)
+            return unless object.is_a?(String) || object.is_a?(Symbol)
+
+            short = object.to_s.sub(SCHEMA_ORG_PREFIX_RE, '')
+            return if short.empty?
+
+            CANONICAL_BY_DOWNCASE.fetch(::Html2rss::Html::Probe.fold(short), short)
+          end
+
           private
 
           # @param hash [Hash] candidate schema object
@@ -150,15 +169,15 @@ module Html2rss
         ##
         # @yield [Hash] Each scraped article_hash
         # @return [Array<Hash>] the scraped article_hashes
-        def each(&)
+        def each
           return enum_for(:each) unless block_given?
 
-          schema_objects.filter_map do |schema_object|
+          schema_objects.each do |schema_object|
             next unless (klass = self.class.scraper_for_schema_object(schema_object))
             next unless (results = klass.new(schema_object, url:).call)
 
             if results.is_a?(Array)
-              results.each { |result| yield(result) } # rubocop:disable Style/ExplicitBlockArgument
+              results.each { yield(_1) }
             else
               yield(results)
             end
@@ -168,9 +187,8 @@ module Html2rss
         private
 
         def schema_objects
-          @parsed_body.css(TAG_SELECTOR).flat_map do |tag|
-            Schema.from(tag)
-          end
+          ::Html2rss::Html::Probe.scripts(@parsed_body, ::Html2rss::Html::Probe::APPLICATION_LD_JSON)
+                                 .flat_map { |tag| Schema.from(tag) }
         end
 
         attr_reader :parsed_body, :url

data/lib/html2rss/auto_source/scraper.rb

@@ -118,7 +118,7 @@ module Html2rss
       # @return [SST::Document, nil]
       def self.normalize_sst(parsed_body)
         SST::Normalizer.call(parsed_body)
-      rescue ArgumentError
+      rescue SST::Normalizer::EmptyTree
         nil
       end
 
@@ -145,7 +145,7 @@ module Html2rss
       # @option opts [Hash] :html scraper toggle and configuration
       # @option opts [Hash] :sitemap scraper toggle and configuration
       # @return [Object, nil]
-      # rubocop:disable Metrics/ParameterLists, Metrics/MethodLength -- construction context for structured and heuristic scrapers
+      # rubocop:disable-next Metrics/ParameterLists, Metrics/MethodLength -- construction context for structured and heuristic scrapers
       def self.build_instance(scraper, parsed_body, opts:, url:, request_session: nil, body: nil, document: nil,
                               link_resolver: nil, captured_responses: [])
         return unless opts.dig(scraper.options_key, :enabled)
@@ -162,7 +162,6 @@ module Html2rss
         )
         scraper.new(parsed_body, url:, **kwargs, **scraper_opts)
       end
-      # rubocop:enable Metrics/ParameterLists, Metrics/MethodLength
 
       ##
       # @param instance [Object]

data/lib/html2rss/auto_source/segmenter/cluster.rb

@@ -64,7 +64,7 @@ module Html2rss
         def normalize_class(class_names)
           return '' if class_names.empty?
 
-          class_names.sort.join(' ')
+          class_names.map { ::Html2rss::Html::Probe.fold(_1) }.sort.join(' ')
         end
         module_function :normalize_class
         private_class_method :normalize_class
@@ -122,7 +122,7 @@ module Html2rss
             groups.sort_by { |_key, nodes| -nodes.size }.first(MAX_GROUPS).to_h
           end
 
-          # rubocop:disable Metrics/MethodLength
+          # rubocop:disable-next Metrics/MethodLength
           def container_of?(nodes_a, nodes_b)
             return false unless @layout_tags.include?(nodes_b.first.name)
 
@@ -139,7 +139,6 @@ module Html2rss
               count > 1
             end
           end
-          # rubocop:enable Metrics/MethodLength
 
           def resolve_1_to_1_overlap(cls_a, cls_b, groups, discarded)
             nodes_a = groups[cls_a]

data/lib/html2rss/auto_source/segmenter/primary_link.rb

@@ -37,7 +37,7 @@ module Html2rss
 
         private
 
-        # rubocop:disable Metrics/AbcSize, Metrics/CyclomaticComplexity, Metrics/MethodLength, Metrics/PerceivedComplexity
+        # rubocop:disable-next Metrics/AbcSize, Metrics/CyclomaticComplexity, Metrics/MethodLength, Metrics/PerceivedComplexity
         def candidate_facts(anchor, container)
           destination = @link_resolver.destination_facts(anchor)
           return unless destination
@@ -63,7 +63,6 @@ module Html2rss
 
           { anchor:, destination: destination.destination, score: }
         end
-        # rubocop:enable Metrics/AbcSize, Metrics/CyclomaticComplexity, Metrics/MethodLength, Metrics/PerceivedComplexity
 
         def first_heading(container)
           (@headings ||= {}.compare_by_identity)[container] ||= container.find(&:heading?)