html2rss 0.27.2 → 0.29.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: '09b4a391aa87101ab6612c07c0c434b9ebc6198fc3bed8c14c2f8b90bd61bc6a'
-  data.tar.gz: d7fac305311719a6b131be4e89fe78919aef1b45c763a22c471b24583b41d7fa
+  metadata.gz: 32c090eccd93210c9631778ff50ec7d32d36c7eccaa711cdd1a0dd5c3ea53616
+  data.tar.gz: 93e4d2f253a019d7de585c8acaaa8d06143accb4f3a43a4f8d13f9c96d838390
 SHA512:
-  metadata.gz: d5fb2cdfc2b95e64de5ac42ffbe1479c15fe63b01f4b922d721d3099ffdd418c9293a1672217838612d5cf4f7bbc2e153cf83ffbcc85e8f30bf8a1a17bb20a79
-  data.tar.gz: a9f51aa30dfdf8e3e222ed1a49216e857cd056f473408273ad3781e862d743f657a67473fe691423e1fb1252fbe2935ee6eb1cef009d3b2662d03b66f7bdb90c
+  metadata.gz: 93956c4dc3d35e06c81dff939d4ea5184da93834a0fb48f1b3dbd4dea21d615177ad45fa34e4f16abf396a184cc18acbe31c354df6764ba858ec340c78a3b08c
+  data.tar.gz: b906784ed2ae53b80aa2fc00fe813550befcfac723d97346ffb87cff7f01ca6b0f5db0a6e4a2430dd10485c20f6d4b251a54a1b9e0462998f2ab628c0123a809

data/README.md

@@ -35,13 +35,63 @@ Cloud development: [Open in GitHub Codespaces](https://github.com/codespaces/new
 Config -> Request -> Extraction -> Processing -> Building -> Output
 ```
 
+## CLI Usage
+
+| Verb     | Job                                                           |
+| -------- | ------------------------------------------------------------- |
+| inspect  | Cheap diagnostics (final URL, status, alternates, surface)    |
+| recon    | Verdict + native feed preference (`BUILD` / `DEFER` / `DROP`) |
+| capture  | YAML draft config                                             |
+| validate | Schema only                                                   |
+| test     | Schema + live extraction (min items)                          |
+| apply    | Ship RSS from config or URL                                   |
+| scrape   | Articles now (one-shot auto-source)                           |
+
+Golden path: optional **inspect → recon → capture → test → apply**. Side door: **validate**. One-shot: **scrape**.
+
+```bash
+# Diagnostics and reconnaissance
+html2rss inspect https://example.com/news
+html2rss recon https://example.com/news
+html2rss recon --file urls.txt --verdict BUILD --url-only
+
+# Composable pipes
+html2rss recon --file urls.txt --verdict BUILD --url-only | html2rss capture -
+html2rss capture https://example.com/news | html2rss test -
+
+# One-shot articles now
+html2rss scrape https://example.com/news
+html2rss scrape https://example.com/news --format jsonfeed --explain
+
+# Durable config workflow
+html2rss capture https://example.com/news --write feed.yml
+html2rss test feed.yml --min-items 5
+html2rss apply feed.yml
+
+# Schema validation (side door)
+html2rss validate config.yml
+html2rss validate "configs/**/*.yml"
+
+# Export JSON Schema
+html2rss schema --write schema/html2rss-config.schema.json
+```
+
+Historic CLI aliases: `feed` → `apply`, `auto` → `scrape`.
+
+### Inspect output
+
+Inspect follows redirects and reports the landing URL in `final_url`. CLI text shows a `Final:` line **only when** the landing URL differs from what you typed — that line means the redirect succeeded, not that inspect stopped early.
+
+Cross-host redirects (e.g. `https://apex.example/` → `https://www.example/`) set `Host` per hop via Faraday/Net::HTTP; html2rss does not pin the entry hostname. When `final_url` differs and status is 4xx, retry on `final_url` or pass the site's canonical hostname. Details: [`lib/html2rss/page_recon/README.md`](lib/html2rss/page_recon/README.md).
+
 ## Capture API
 
-The `Html2rss.capture` method analyzes any URL and produces a reusable feed config hash with an items selector and `enhance: true`. Use it to speed up writing feed configuration files.
+`Html2rss.capture` returns a `Capture::CaptureResult`. Use `result.yaml` or `result.config`.
 
 ```ruby
-config = Html2rss.capture('https://example.com/articles')
-File.write('my-feed.yml', Html2rss::Config.to_yaml(config))
+result = Html2rss.capture('https://example.com/articles')
+File.write('my-feed.yml', result.yaml)
+# or: File.write('my-feed.yml', Html2rss::Config.to_yaml(result.config))
 ```
 
 The CLI alias `html2rss capture` prints the generated config as YAML to stdout. See [`lib/html2rss/capture/README.md`](lib/html2rss/capture/README.md) for detailed documentation.
@@ -62,9 +112,9 @@ stdio uses stdout for JSON-RPC, so the daemon logs to **stderr**. It defaults to
 
 HTTP transport needs `rack`, `rackup`, and `webrick` (declared gem dependencies). It listens on `127.0.0.1` only; do not expose it on a public interface without your own auth and Host/Origin controls.
 
-**Strategy note:** MCP `scrape_url` / `capture_config` with `strategy: "auto"` run Faraday → Botasaurus AutoFallback. `inspect_url` uses Faraday when `auto` (cheap diagnostic); pin `botasaurus` when you need browser rendering for inspect.
+**Strategy note:** MCP `scrape` / `capture` with `strategy: "auto"` run Faraday → Botasaurus AutoFallback. `inspect` uses Faraday when `auto` (cheap diagnostic); pin `botasaurus` when you need browser rendering for inspect.
 
-**Tool-call budget:** `scrape_url` is 1 call (auto already hops). Durable config is `capture_config` → `validate_config` → `apply_config`. Call `inspect_url` only when scrape/capture is weak or you need recon (final URL, status, https→http, native RSS/Atom).
+**Tool-call budget:** `scrape` is 1 call (auto already hops). Durable config is `capture` → `test` → `apply` (or `validate` → `test` → `apply` when you already have YAML). Call `inspect` only when scrape/capture is weak or you need recon (final URL, status, https→http, native RSS/Atom).
 
 Cursor / Claude Desktop `mcp.json` must put Botasaurus on the **MCP process** (not only your shell):
 
@@ -82,17 +132,24 @@ Cursor / Claude Desktop `mcp.json` must put Botasaurus on the **MCP process** (n
 }
 ```
 
-Read `html2rss://runtime` for a boolean `botasaurus_configured` (the URL is never returned). Every tool result is a JSON envelope (`ok`, `next_step`, `guidance`, `payload`) in both the text body and `structuredContent`. Follow `next_step` / `guidance`; do not parse scrape text as a raw item array.
+Read `html2rss://runtime` for `version`, `mcp_contract_version`, `catalog_fingerprint`, `tools`, and `botasaurus_configured` (the scraper URL is never returned). Refresh `tools/list` when `catalog_fingerprint` differs from your cache. Every tool result is a JSON envelope (`ok`, `next_step`, `guidance`, `payload`) in both the text body and `structuredContent`. Follow `next_step` / `guidance`; do not parse scrape text as a raw item array.
+
+Module guide: [`lib/html2rss/mcp/README.md`](lib/html2rss/mcp/README.md).
 
 ### Tools
 
-| Name              | When to use                                                                 |
-| ----------------- | --------------------------------------------------------------------------- |
-| `scrape_url`      | One-shot articles now (`payload.items`; empty is still success)             |
-| `inspect_url`     | Weak scrape/capture or recon (final_url, status, scheme_downgrade, feeds)   |
-| `capture_config`  | YAML draft in `payload.yaml`; strive `enhance: true`                        |
-| `validate_config` | Schema-check a `config` hash XOR `yaml` string (`isError` on failure)       |
-| `apply_config`    | RSS in `payload.rss`; `isError` when zero items; confirm `payload.item_count` |
+| Name            | When to use                                                                    |
+| --------------- | ------------------------------------------------------------------------------ |
+| `scrape`        | One-shot articles now (`payload.items`; empty is still success)                |
+| `batch_scrape`  | Parallel one-shot scrape across multiple URLs (`urls`, `limit`, `concurrency`) |
+| `inspect`       | Weak scrape/capture or recon (final_url, status, scheme_downgrade, feeds)      |
+| `batch_inspect` | Parallel diagnostics across multiple URLs (`urls`, `strategy`, `concurrency`)  |
+| `recon`         | Verdict + native_feed preference                                               |
+| `batch_recon`   | Parallel recon across multiple URLs                                            |
+| `capture`       | YAML draft in `payload.yaml`; strive `enhance: true`                           |
+| `validate`      | Schema-check a `config` hash XOR `yaml` string (`isError` on failure)          |
+| `test`          | Schema + live extraction; `quality_report.enhance_gains` when enhance on; optional `compare_enhance` |
+| `apply`         | RSS in `payload.rss`; `isError` when zero items; `quality_report` may include `enhance_gains`          |
 
 ### Resources
 
@@ -100,15 +157,15 @@ Read `html2rss://runtime` for a boolean `botasaurus_configured` (the URL is neve
 | ----------------------- | --------------------------------------------------------------- |
 | `html2rss://schema`     | Full JSON Schema for feed configurations                        |
 | `html2rss://extractors` | Registered extractor **names** (options live in schema `$defs`) |
-| `html2rss://strategies` | Published MCP strategies (`auto`, `faraday`, `botasaurus`) |
-| `html2rss://runtime`    | `botasaurus_configured` boolean (never the scraper URL)         |
+| `html2rss://strategies` | Published MCP strategies (`auto`, `faraday`, `botasaurus`)      |
+| `html2rss://runtime`    | `version`, `mcp_contract_version`, `catalog_fingerprint`, `tools`, `botasaurus_configured` (never the scraper URL) |
 
 ### Prompts
 
-| Name                  | Description                                                          |
-| --------------------- | -------------------------------------------------------------------- |
-| `scrape-webpage`      | One `scrape_url` call; inspect only if weak or recon                 |
-| `capture-feed-config` | Capture YAML → validate → apply; catalog rewrite; strive enhance     |
+| Name                  | Description                                                  |
+| --------------------- | ------------------------------------------------------------ |
+| `scrape-webpage`      | One `scrape` call; `inspect` only if weak or recon needed    |
+| `capture-feed-config` | Capture YAML → test → apply; catalog rewrite; strive enhance |
 
 The MCP module (`Html2rss::MCP`) lazy-loads the `mcp` gem — no cost when the server is not running.
 
@@ -124,13 +181,13 @@ Set `BOTASAURUS_SCRAPER_URL` to `http://127.0.0.1:4010` and use strategy `botasa
 
 ## Request Strategies
 
-| Strategy     | Description                                                                   |
-| ------------ | ----------------------------------------------------------------------------- |
-| `auto`       | Tries `faraday`, falls back to `botasaurus` (default in gem/CLI/MCP scrape)   |
-| `faraday`    | Plain HTTP requests via Faraday                                               |
-| `botasaurus` | Puppeteer-backed scraping for JavaScript pages                                |
+| Strategy     | Description                                                                 |
+| ------------ | --------------------------------------------------------------------------- |
+| `auto`       | Tries `faraday`, falls back to `botasaurus` (default in gem/CLI/MCP scrape) |
+| `faraday`    | Plain HTTP requests via Faraday                                             |
+| `botasaurus` | Puppeteer-backed scraping for JavaScript pages                              |
 
-`inspect_url` keeps Faraday when `auto` for cheap diagnostics. Elsewhere, strategy can be set via CLI (`--strategy`), gem API keyword argument, or feed config `request.strategy`. See the [request strategies docs](https://html2rss.github.io/ruby-gem/reference/strategy) for more details.
+`inspect` keeps Faraday when `auto` for cheap diagnostics. Elsewhere, strategy can be set via CLI (`--strategy`), gem API keyword argument, or feed config `request.strategy`. See the [request strategies docs](https://html2rss.github.io/ruby-gem/reference/strategy) for more details.
 
 ## License
 

data/html2rss.gemspec

@@ -45,7 +45,6 @@ Gem::Specification.new do |spec|
   spec.add_dependency 'rack', '~> 3.0'
   spec.add_dependency 'rackup', '~> 2.0'
   spec.add_dependency 'regexp_parser'
-  spec.add_dependency 'reverse_markdown', '~> 3.0'
   spec.add_dependency 'rss'
   spec.add_dependency 'sanitize'
   spec.add_dependency 'thor'

data/lib/html2rss/article.rb

@@ -8,7 +8,7 @@ module Html2rss
   # It is enumerable and responds to all keys specified in PROVIDED_KEYS.
   #
   # Description and enclosure wire presentation live in {FeedBuilder::ItemPresentation}.
-  # rubocop:disable Metrics/ClassLength -- value object retains Marshal + defensive freeze helpers
+  # rubocop:disable-next Metrics/ClassLength -- value object retains Marshal + defensive freeze helpers
   class Article
     include Enumerable
     include Comparable
@@ -209,5 +209,4 @@ module Html2rss
       value
     end
   end
-  # rubocop:enable Metrics/ClassLength
 end

data/lib/html2rss/auto_source/README.md

@@ -10,9 +10,10 @@ Entry: `FeedPipeline` → `AutoSource#articles` → `Scraper.build_instance` →
 
 ## Live flow
 
-1. **Request** — `RequestSession` returns a `Response` with `body` (String) and `parsed_body` (Nokogiri HTML).
+1. **Request** — `RequestSession` returns a `Response` with `body` (String) and `parsed_body` (Nokogiri HTML). Direct syndication responses skip HTML scrapers and parse via `Syndication::Parser`.
 2. **Scraper tiers** — Enabled scrapers that claim the page (shallow `articles?` or instance `extractable?`) run in `Scraper::SCRAPER_TIERS` order. Merge within a tier, then stop when enough articles survive Cleanup:
 
+   0. Native feed: NativeFeed (head alternates + path discovery → RSS/Atom parse)
    1. In-page structured: Schema, Microdata, Microformats2, JsonState, XhrArticles
    2. Follow-up IO: WordPress API, Sitemap, MetaOembed
    3. Heuristic: SemanticHtml
@@ -25,18 +26,20 @@ Entry: `FeedPipeline` → `AutoSource#articles` → `Scraper.build_instance` →
 
 5. **Cleanup** — Merge, dedupe, hard-exclude non-article destinations (via `PathClassifier` facts), drop junk titles, and trim to `limit`. Html is skipped when earlier tiers already admitted clean items.
 
+6. **Entry URL resolution** (pipeline, not this class) — when AutoFallback sees a weak homepage extract, `FeedResolution` may rewrite the scrape URL to a listing/feed before escalating strategies. See {Html2rss::FeedPipeline} and {Html2rss::FeedResolution}.
+
 Segmenter strategies: `:semantic` (leaf containers + primary link), `:list` (repeated tag paths), `:cluster` (class/structure grids for anchorless cards). Scoring ranks and demotes; `LinkDestination::NoisePolicy` owns content-anchor eligibility. Cleanup owns feed-item admission.
 
 ## Nokogiri vs SST boundaries
 
-| Surface | Owns DOM |
-| --- | --- |
-| `Response#parsed_body` | Single HTML parse for the page |
-| Schema / Microdata / MF2 / JsonState / XhrArticles / MetaOembed / app-shell detection | Nokogiri |
-| Sitemap detection (CSS/XPath) | Nokogiri; URL list parsing uses raw `response.body` (XML string) |
-| Selectors path / Sanitize transformers | Nokogiri (unchanged) |
-| `SST::Normalizer` | Sole Nokogiri consumer on the heuristic auto-source path |
-| Segmenter, Scoring, `Html::SstArticleExtractor`, heuristic chrome (`SST::Tags` / `SST::Text`) | SST only |
+| Surface                                                                                       | Owns DOM                                                         |
+| --------------------------------------------------------------------------------------------- | ---------------------------------------------------------------- |
+| `Response#parsed_body`                                                                        | Single HTML parse for the page                                   |
+| Schema / Microdata / MF2 / JsonState / XhrArticles / MetaOembed / app-shell detection         | Nokogiri                                                         |
+| Sitemap detection (CSS/XPath)                                                                 | Nokogiri; URL list parsing uses raw `response.body` (XML string) |
+| Selectors path / Sanitize transformers                                                        | Nokogiri (unchanged)                                             |
+| `SST::Normalizer`                                                                             | Sole Nokogiri consumer on the heuristic auto-source path         |
+| Segmenter, Scoring, `Html::SstArticleExtractor`, heuristic chrome (`SST::Tags` / `SST::Text`) | SST only                                                         |
 
 Production heuristic scrapers should reuse one `SST::Document` memoized from `parsed_body` (or a shared Document passed in), not re-parse HTML strings.
 

data/lib/html2rss/auto_source/cleanup.rb

@@ -47,13 +47,22 @@ module Html2rss
         [:video_chrome, /\AClipped\s+From\s+Video\b/i],
         [:video_chrome, /\AVideo\s*[•·]/i],
         [:template, /\ACreated\s+from\s+Template\s+ID\b/i],
-        [:template, /(\{\{[^}]+\}\}|%\{\w+\})/]
+        [:template, /(\{\{[^}]+\}\}|%\{\w+\})/],
+        [:cta, /\Aread more\z/i],
+        [:cta, /\Alearn more\z/i],
+        [:cta, /\Apdf\z/i]
       ].freeze
       private_constant :JUNK_TITLE_RULES
 
+      # Minimum present-title length before short-title warnings.
+      MIN_TITLE_LENGTH = 4
+
       # Admitted articles plus reason → count tallies for drops.
       Result = Data.define(:articles, :drop_tallies)
 
+      # Read-only ship-quality audit on RSS/article-like items (no mutation).
+      AuditResult = Data.define(:warnings, :metrics, :violations)
+
       class << self
         # @param articles [Array<Article>] extracted article candidates
         # @param url [Html2rss::Url] feed source URL used for same-host filtering
@@ -88,8 +97,83 @@ module Html2rss
           JUNK_TITLE_RULES.find { |_, pattern| pattern.match?(normalized) }&.first
         end
 
+        # @param items [Array] RSS or article-like objects with title and link/url
+        # @return [AuditResult]
+        def audit_feed_items(items)
+          metrics = initial_audit_metrics(items.size)
+          violations = Hash.new(0)
+          warnings = audit_url_diversity(items, metrics, violations)
+          audit_item_titles(items, metrics, violations)
+          warnings = finalize_audit_warnings(warnings, metrics)
+          log_audit(metrics, warnings, violations)
+          AuditResult.new(warnings:, metrics: metrics.freeze, violations: violations.freeze)
+        end
+
         private
 
+        def initial_audit_metrics(item_count)
+          {
+            item_count:,
+            unique_url_count: 0,
+            junk_title_count: 0,
+            short_title_count: 0,
+            low_word_count: 0
+          }
+        end
+
+        def audit_url_diversity(items, metrics, violations)
+          unique_urls = items.filter_map { |item| url_identity(item_url(item)) }.uniq
+          metrics[:unique_url_count] = unique_urls.size
+          return [] unless items.size >= 2 && unique_urls.size < 2
+
+          violations[:duplicate_urls] += 1
+          [:duplicate_urls]
+        end
+
+        def audit_item_titles(items, metrics, violations)
+          items.each do |item|
+            title = normalize_title(item.title)
+            next if title.empty?
+
+            audit_short_title(title, metrics, violations)
+            audit_title_quality(title, metrics, violations)
+          end
+        end
+
+        def audit_short_title(title, metrics, violations)
+          return unless title.length < MIN_TITLE_LENGTH
+
+          metrics[:short_title_count] += 1
+          violations[:short_title] += 1
+        end
+
+        def audit_title_quality(title, metrics, violations)
+          reason = junk_reason(title)
+          if reason
+            metrics[:junk_title_count] += 1
+            violations[reason] += 1
+          elsif !word_count_at_least?(title, MIN_WORDS)
+            metrics[:low_word_count] += 1
+            violations[:low_word_count] += 1
+          end
+        end
+
+        def finalize_audit_warnings(warnings, metrics)
+          warnings = warnings.dup
+          warnings << :generic_titles if metrics[:junk_title_count].positive?
+          warnings << :short_titles if metrics[:short_title_count].positive?
+          warnings << :low_word_count if metrics[:low_word_count].positive?
+          warnings.freeze
+        end
+
+        def log_audit(metrics, warnings, violations)
+          Log.debug(
+            'Cleanup.audit_feed_items: ' \
+            "item_count=#{metrics[:item_count]} unique_urls=#{metrics[:unique_url_count]} " \
+            "warnings=#{warnings.join(',')} violations=#{violations.keys.join(',')}"
+          )
+        end
+
         def reject_invalid!(articles, tallies)
           tally_reject!(articles, tallies, 'invalid') { |article| !article.valid? }
         end
@@ -172,6 +256,13 @@ module Html2rss
           url&.without_fragment&.to_s
         end
 
+        def item_url(item)
+          raw = item.respond_to?(:link) ? item.link : item.url
+          raw.nil? || raw.to_s.empty? ? nil : Html2rss::Url.from_absolute(raw.to_s)
+        rescue ArgumentError
+          nil
+        end
+
         def normalize_title(title)
           title.to_s.strip.gsub(/\s+/, ' ')
         end

data/lib/html2rss/auto_source/scraper/json_state/document_scanner.rb

@@ -8,8 +8,6 @@ module Html2rss
       class JsonState
         # Scans DOM nodes for JSON payloads containing article data.
         module DocumentScanner # rubocop:disable Metrics/ModuleLength
-          # Selector for JSON-only script tags.
-          JSON_SCRIPT_SELECTOR = 'script[type="application/json"]'
           # Regex patterns for known global JavaScript state assignments.
           GLOBAL_ASSIGNMENT_PATTERNS = [
             /(?:window|self|globalThis)\.__NEXT_DATA__\s*=\s*/m,
@@ -35,16 +33,16 @@ module Html2rss
           def json_documents(parsed_body)
             # Use identity-based cache to avoid double-parsing of the same document.
             # WeakMap allows the Nokogiri Document (key) to be garbage collected.
-            # rubocop:disable ThreadSafety/ClassInstanceVariable
+            # rubocop:disable-next ThreadSafety/ClassInstanceVariable
             (@cache ||= ObjectSpace::WeakMap.new)[parsed_body] ||=
               script_documents(parsed_body) + assignment_documents(parsed_body)
-            # rubocop:enable ThreadSafety/ClassInstanceVariable
           end
 
           # @param parsed_body [Nokogiri::HTML::Document] parsed HTML document
           # @return [Array<Hash, Array>] JSON documents extracted from JSON script tags
           def script_documents(parsed_body)
-            parsed_body.css(JSON_SCRIPT_SELECTOR).filter_map { parse_json(_1.text) }
+            ::Html2rss::Html::Probe.scripts(parsed_body, ::Html2rss::Html::Probe::APPLICATION_JSON)
+                                   .filter_map { parse_json(_1.text) }
           end
 
           # @param parsed_body [Nokogiri::HTML::Document] parsed HTML document

data/lib/html2rss/auto_source/scraper/meta_oembed.rb

@@ -13,9 +13,6 @@ module Html2rss
 
         # Selector for OpenGraph meta tags.
         OG_META_SELECTOR = 'meta[property^="og:"], meta[property^="article:"], meta[name^="twitter:"]'
-        # Selector for oEmbed JSON link tag.
-        OEMBED_LINK_SELECTOR = 'link[rel="alternate"][type="application/json+oembed"][href]'
-
         # Mapping of meta property names to article attribute keys.
         META_MAP = {
           'og:title' => :title,
@@ -40,7 +37,11 @@ module Html2rss
             return false unless parsed_body
 
             !parsed_body.at_css('meta[property="og:title"]').nil? ||
-              !parsed_body.at_css(OEMBED_LINK_SELECTOR).nil?
+              ::Html2rss::Html::Probe.alternate_links(
+                parsed_body,
+                rel: 'alternate',
+                mime: ::Html2rss::Html::Probe::APPLICATION_JSON_OEMBED
+              ).any?
           end
         end
 
@@ -109,7 +110,7 @@ module Html2rss
 
         # @return [Hash{Symbol => Object}] oEmbed fields hash
         def fetch_oembed_data
-          return {} unless request_session && (link_node = parsed_body.at_css(OEMBED_LINK_SELECTOR))
+          return {} unless request_session && (link_node = oembed_link_node)
           return {} unless (oembed_url = resolve_url(link_node['href']))
 
           response = request_session.follow_up(url: oembed_url, relation: :auto_source, origin_url: url)
@@ -119,6 +120,15 @@ module Html2rss
           {}
         end
 
+        # @return [Nokogiri::XML::Element, nil] oEmbed descriptor link node
+        def oembed_link_node
+          ::Html2rss::Html::Probe.alternate_links(
+            parsed_body,
+            rel: 'alternate',
+            mime: ::Html2rss::Html::Probe::APPLICATION_JSON_OEMBED
+          ).first
+        end
+
         # @param response [Html2rss::RequestService::Response, nil] HTTP response
         # @return [Hash{Symbol => Object}] parsed oEmbed fields
         def parse_oembed_response(response)

data/lib/html2rss/auto_source/scraper/microdata.rb

@@ -39,15 +39,17 @@ module Html2rss
           # @param node [Nokogiri::XML::Element] itemscope candidate node
           # @return [String, nil] supported schema type name when present
           def supported_type_name(node)
-            normalized_types(node['itemtype']).find { SUPPORTED_TYPES.include?(_1) }
+            itemtype_type_names(node['itemtype']).find { SUPPORTED_TYPES.include?(_1) }
           end
 
           # @param itemtype [String, nil] raw itemtype attribute value
-          # @return [Array<String>] normalized schema type names
-          def normalized_types(itemtype)
+          # @return [Array<String>] canonical schema type names
+          def itemtype_type_names(itemtype)
             itemtype.to_s.split.filter_map do |value|
-              type = value.split('/').last.to_s.split('#').last.to_s
-              type unless type.empty?
+              short = value.split('/').last.to_s.split('#').last.to_s
+              next if short.empty?
+
+              Schema.canonicalize_type(short)
             end
           end
 
@@ -189,7 +191,7 @@ module Html2rss
             item = call(node)
             itemtype = node['itemtype']
             itemid = node['itemid']
-            item[:@type] = Microdata.normalized_types(itemtype).first if itemtype
+            item[:@type] = Microdata.itemtype_type_names(itemtype).first if itemtype
             item[:@id] = itemid if present?(itemid)
             item
           end

data/lib/html2rss/auto_source/scraper/native_feed.rb

@@ -0,0 +1,93 @@
+# frozen_string_literal: true
+
+module Html2rss
+  class AutoSource
+    module Scraper
+      ##
+      # Promotes a page's native RSS/Atom feed via {Syndication::Discovery} + {Syndication::Parser}.
+      class NativeFeed
+        include Enumerable
+
+        # @return [Symbol] scraper config key
+        def self.options_key = :native_feed
+
+        ##
+        # @param _opts [Hash] unused options
+        # @return [Integer] follow-up request slots (discovery probes + feed fetch share the budget)
+        def self.request_slots(_opts = {})
+          1
+        end
+
+        ##
+        # Shallow claim: head advertises a syndication alternate, or body looks feed-capable
+        # enough that path discovery is worth a follow-up slot.
+        #
+        # @param parsed_body [Nokogiri::HTML::Document, nil]
+        # @return [Boolean]
+        def self.articles?(parsed_body)
+          return false unless parsed_body.is_a?(Nokogiri::HTML::Document)
+
+          return true if ::Html2rss::Html::FeedLink.from_document(parsed_body).any?
+
+          parsed_body.css('head link[rel~="alternate"][href]').any? do |node|
+            href = node['href'].to_s
+            ::Html2rss::Html::Probe.mime_match?(
+              node['type'],
+              ::Html2rss::Html::Probe::APPLICATION_RSS_XML,
+              ::Html2rss::Html::Probe::APPLICATION_ATOM_XML
+            ) || href.match?(/rss|atom|feed|\.xml/i)
+          end
+        end
+
+        ##
+        # @param parsed_body [Nokogiri::HTML::Document]
+        # @param url [String, Html2rss::Url]
+        # @param request_session [Html2rss::RequestSession, nil]
+        # @param _opts [Hash]
+        # @option _opts [Object] :_reserved reserved for future scraper-specific options
+        def initialize(parsed_body, url:, request_session: nil, **_opts)
+          @parsed_body = parsed_body
+          @url = Html2rss::Url.from_absolute(url)
+          @request_session = request_session
+        end
+
+        ##
+        # @yieldparam article [Hash{Symbol => Object}]
+        # @return [Enumerator, void]
+        def each(&)
+          return enum_for(:each) unless block_given?
+
+          articles = fetch_articles
+          if articles.empty?
+            Log.info("#{self.class}: host=#{url.host} item_count=0 fallback=true")
+            return
+          end
+
+          Log.info("#{self.class}: host=#{url.host} item_count=#{articles.size} fallback=false")
+          articles.each(&)
+        end
+
+        private
+
+        attr_reader :parsed_body, :url, :request_session
+
+        def fetch_articles # rubocop:disable Metrics/MethodLength -- discovery + parse path
+          return [] unless request_session
+
+          response = Syndication::Discovery.best_feed_response(
+            page_url: url,
+            request_session:,
+            parsed_body:,
+            max_probes: self.class.request_slots
+          )
+          return [] unless response
+
+          Syndication::Parser.parse_response(response)
+        rescue Html2rss::Error, ArgumentError => error
+          Log.warn("#{self.class}: host=#{url.host} failed (#{error.class}: #{error.message})")
+          []
+        end
+      end
+    end
+  end
+end