html2rss 0.25.0 → 0.27.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 1b6b0ddc57b5cf372cc9bc6405bb21f7dc6d00aa526e234132974ccc6ef609f7
-  data.tar.gz: 59a60ef044a99686838b5498f0712a455a6c5a832c27619db0c99ba21742f6a8
+  metadata.gz: 286a89b05b23997b8cb5ef8b2930a59bd698bdeb07ba5007b3e30313b91324cd
+  data.tar.gz: 709147e30bb631551e8927e496fea5d76ef32373a76834b66adb54cbcdc8df16
 SHA512:
-  metadata.gz: 8994aa3b90d791ea8cf75df9b576669fffe505fe0895e207745ab0e591f187ab490382d00a95d49fc1073570f886d6940f27f1c1ca80ba7c57817740f48c4bc4
-  data.tar.gz: dc62483a0ade2b582fb3839778a3a518a7f1a08e781d2105c0e93c70aaa26313e153a4e30121170b8a59e8472517e6cb8d9958b9c06fa3a862358631f469c18a
+  metadata.gz: e9cabc478aa9d449697e2d8e99a5b91f6e2071d498eac34bdbc7828fffa06d2b02395d4f8e02853f0553ea97bdc35ca5a59c0b33fcc53b82ef809a247674826c
+  data.tar.gz: 32d343c2f3008917dbdbbb0859c29e710a89559e2ab079dcd3ba76672450ddaeb63572c8c71e0a1f5f7b327be7df667d785d2f7fb1069389e12a63093d83e3c9

data/README.md

@@ -13,7 +13,7 @@ Most people looking for a first working feed should start with `html2rss-web`, r
 Detailed usage guides, reference docs, and the feed directory live on the project website:
 
 - [Ruby gem documentation](https://html2rss.github.io/ruby-gem)
-- [Request strategies](https://html2rss.github.io/ruby-gem/reference/strategy) (`auto` = `faraday` → `botasaurus`; pin `browserless` explicitly)
+- [Request strategies](https://html2rss.github.io/ruby-gem/reference/strategy) (`auto` = `faraday` → `botasaurus`; or pin concrete strategies)
 - [Selectors & pagination](https://html2rss.github.io/ruby-gem/reference/selectors#paginated-feeds)
 - [Web application](https://html2rss.github.io/web-application)
 - [Feed directory](https://html2rss.github.io/feed-directory)
@@ -26,7 +26,7 @@ Cloud development: [Open in GitHub Codespaces](https://github.com/codespaces/new
 ## Architecture
 
 1. **Config** — loads and validates configuration (YAML/hash); schema via `html2rss schema` / `schema/html2rss-config.schema.json`
-2. **RequestService** — fetches pages (`faraday`, `botasaurus`, or explicit `browserless`)
+2. **RequestService** — fetches pages (`faraday`, `botasaurus`, or `local_file`)
 3. **Selectors** — extracts content via CSS selectors with extractors/post-processors
 4. **AutoSource** — auto-detects content (Schema.org, JSON state, semantic HTML, structural patterns)
 5. **FeedBuilder** — assembles Article objects and renders feeds (RSS 2.0 / JSON Feed 1.1)
@@ -35,6 +35,103 @@ Cloud development: [Open in GitHub Codespaces](https://github.com/codespaces/new
 Config -> Request -> Extraction -> Processing -> Building -> Output
 ```
 
+## Capture API
+
+The `Html2rss.capture` method analyzes any URL and produces a reusable feed config hash with an items selector and `enhance: true`. Use it to speed up writing feed configuration files.
+
+```ruby
+config = Html2rss.capture('https://example.com/articles')
+File.write('my-feed.yml', Html2rss::Config.to_yaml(config))
+```
+
+The CLI alias `html2rss capture` prints the generated config as YAML to stdout. See [`lib/html2rss/capture/README.md`](lib/html2rss/capture/README.md) for detailed documentation.
+
+## MCP Server
+
+html2rss ships with an [MCP](https://modelcontextprotocol.io/) server that exposes gem capabilities as AI-consumable tools, resources, and prompts:
+
+```bash
+# Start with stdio transport (default; for Cursor/Claude Desktop)
+html2rss mcp
+
+# Start with HTTP transport (binds 127.0.0.1 only — local use)
+html2rss mcp --transport http --port 8080
+```
+
+stdio uses stdout for JSON-RPC, so the daemon logs to **stderr**. It defaults to `LOG_LEVEL=info` (the gem library default stays `warn`) so a foreground watcher sees the start banner, each tool call, and pipeline fallbacks. Use `LOG_LEVEL=debug` for more detail or `LOG_LEVEL=warn` to quiet it.
+
+HTTP transport needs `rack`, `rackup`, and `webrick` (declared gem dependencies). It listens on `127.0.0.1` only; do not expose it on a public interface without your own auth and Host/Origin controls.
+
+**Strategy note:** MCP `scrape_url` / `capture_config` with `strategy: "auto"` run Faraday → Botasaurus AutoFallback. `inspect_url` uses Faraday when `auto` (cheap diagnostic); pin `botasaurus` when you need browser rendering for inspect.
+
+**Tool-call budget:** `scrape_url` is 1 call (auto already hops). Durable config is `capture_config` → `validate_config` → `apply_config`. Call `inspect_url` only when scrape/capture is weak or you need recon (final URL, status, https→http, native RSS/Atom).
+
+Cursor / Claude Desktop `mcp.json` must put Botasaurus on the **MCP process** (not only your shell):
+
+```json
+{
+  "mcpServers": {
+    "html2rss": {
+      "command": "mise",
+      "args": ["exec", "--", "html2rss", "mcp"],
+      "env": {
+        "BOTASAURUS_SCRAPER_URL": "http://127.0.0.1:4010"
+      }
+    }
+  }
+}
+```
+
+Read `html2rss://runtime` for a boolean `botasaurus_configured` (the URL is never returned). Every tool result is a JSON envelope (`ok`, `next_step`, `guidance`, `payload`) in both the text body and `structuredContent`. Follow `next_step` / `guidance`; do not parse scrape text as a raw item array.
+
+### Tools
+
+| Name              | When to use                                                                 |
+| ----------------- | --------------------------------------------------------------------------- |
+| `scrape_url`      | One-shot articles now (`payload.items`; empty is still success)             |
+| `inspect_url`     | Weak scrape/capture or recon (final_url, status, scheme_downgrade, feeds)   |
+| `capture_config`  | YAML draft in `payload.yaml`; strive `enhance: true`                        |
+| `validate_config` | Schema-check a `config` hash XOR `yaml` string (`isError` on failure)       |
+| `apply_config`    | RSS in `payload.rss`; `isError` when zero items; confirm `payload.item_count` |
+
+### Resources
+
+| URI                     | Description                                                     |
+| ----------------------- | --------------------------------------------------------------- |
+| `html2rss://schema`     | Full JSON Schema for feed configurations                        |
+| `html2rss://extractors` | Registered extractor **names** (options live in schema `$defs`) |
+| `html2rss://strategies` | Published MCP strategies (`auto`, `faraday`, `botasaurus`) |
+| `html2rss://runtime`    | `botasaurus_configured` boolean (never the scraper URL)         |
+
+### Prompts
+
+| Name                  | Description                                                          |
+| --------------------- | -------------------------------------------------------------------- |
+| `scrape-webpage`      | One `scrape_url` call; inspect only if weak or recon                 |
+| `capture-feed-config` | Capture YAML → validate → apply; catalog rewrite; strive enhance     |
+
+The MCP module (`Html2rss::MCP`) lazy-loads the `mcp` gem — no cost when the server is not running.
+
+## Botasaurus scrape API (Docker)
+
+Start the Botasaurus scrape API for JavaScript-rendered pages (this compose file is **not** the MCP server):
+
+```bash
+docker compose -f docker-compose.botasaurus.yml up -d
+```
+
+Set `BOTASAURUS_SCRAPER_URL` to `http://127.0.0.1:4010` and use strategy `botasaurus` in MCP tools, Capture, or the CLI.
+
+## Request Strategies
+
+| Strategy     | Description                                                                   |
+| ------------ | ----------------------------------------------------------------------------- |
+| `auto`       | Tries `faraday`, falls back to `botasaurus` (default in gem/CLI/MCP scrape)   |
+| `faraday`    | Plain HTTP requests via Faraday                                               |
+| `botasaurus` | Puppeteer-backed scraping for JavaScript pages                                |
+
+`inspect_url` keeps Faraday when `auto` for cheap diagnostics. Elsewhere, strategy can be set via CLI (`--strategy`), gem API keyword argument, or feed config `request.strategy`. See the [request strategies docs](https://html2rss.github.io/ruby-gem/reference/strategy) for more details.
+
 ## License
 
 This project is licensed under the MIT License — see the [LICENSE](LICENSE) file for details.

data/html2rss.gemspec

@@ -39,14 +39,17 @@ Gem::Specification.new do |spec|
   spec.add_dependency 'faraday-follow_redirects'
   spec.add_dependency 'faraday-gzip', '~> 3'
   spec.add_dependency 'kramdown'
+  spec.add_dependency 'mcp', '~> 1.2'
   spec.add_dependency 'mime-types', '> 3.0'
   spec.add_dependency 'nokogiri', '>= 1.10', '< 2.0'
-  spec.add_dependency 'puppeteer-ruby'
+  spec.add_dependency 'rack', '~> 3.0'
+  spec.add_dependency 'rackup', '~> 2.0'
   spec.add_dependency 'regexp_parser'
   spec.add_dependency 'reverse_markdown', '~> 3.0'
   spec.add_dependency 'rss'
   spec.add_dependency 'sanitize'
   spec.add_dependency 'thor'
   spec.add_dependency 'tzinfo'
+  spec.add_dependency 'webrick', '~> 1.9'
   spec.add_dependency 'zeitwerk'
 end

data/lib/html2rss/auto_source/README.md

@@ -0,0 +1,57 @@
+# AutoSource
+
+How html2rss builds feed items when a config has no (or incomplete) CSS selectors.
+
+## What and when
+
+`Html2rss.auto_source` / `auto_json_feed` (and any feed config with `auto_source:`) fetch a page once, then run an ordered set of scrapers against that response. Use it when you want “guess articles from this URL” without hand-writing selectors. Prefer explicit `selectors:` when you already know the list markup — that path stays on Nokogiri + `Html::Navigator`.
+
+Entry: `FeedPipeline` → `AutoSource#articles` → `Scraper.build_instance` → per-scraper extraction → `Cleanup`.
+
+## Live flow
+
+1. **Request** — `RequestSession` returns a `Response` with `body` (String) and `parsed_body` (Nokogiri HTML).
+2. **Scraper tiers** — Enabled scrapers that claim the page (shallow `articles?` or instance `extractable?`) run in `Scraper::SCRAPER_TIERS` order. Merge within a tier, then stop when enough articles survive Cleanup:
+
+   1. In-page structured: Schema, Microdata, Microformats2, JsonState, XhrArticles
+   2. Follow-up IO: WordPress API, Sitemap, MetaOembed
+   3. Heuristic: SemanticHtml
+   4. Heuristic: Html (skipped when earlier tiers already admitted at least one clean article)
+
+3. **Structured / API scrapers** — Schema, Microdata, MF2, JSON state, XHR JSON, oEmbed, WordPress REST, and Sitemap work on Nokogiri CSS/XPath or JSON/XML parsers. They do not use SST.
+4. **Heuristic scrapers** — `SemanticHtml` and `Html` normalize once into an `SST::Document`, then:
+
+   `SST::Normalizer` → `AutoSource::Segmenter` → `Scoring::Engine` → extractor / article materialization.
+
+5. **Cleanup** — Merge, dedupe, hard-exclude non-article destinations (via `PathClassifier` facts), drop junk titles, and trim to `limit`. Html is skipped when earlier tiers already admitted clean items.
+
+Segmenter strategies: `:semantic` (leaf containers + primary link), `:list` (repeated tag paths), `:cluster` (class/structure grids for anchorless cards). Scoring ranks and demotes; `LinkDestination::NoisePolicy` owns content-anchor eligibility. Cleanup owns feed-item admission.
+
+## Nokogiri vs SST boundaries
+
+| Surface | Owns DOM |
+| --- | --- |
+| `Response#parsed_body` | Single HTML parse for the page |
+| Schema / Microdata / MF2 / JsonState / XhrArticles / MetaOembed / app-shell detection | Nokogiri |
+| Sitemap detection (CSS/XPath) | Nokogiri; URL list parsing uses raw `response.body` (XML string) |
+| Selectors path / Sanitize transformers | Nokogiri (unchanged) |
+| `SST::Normalizer` | Sole Nokogiri consumer on the heuristic auto-source path |
+| Segmenter, Scoring, `Html::SstArticleExtractor`, heuristic chrome (`SST::Tags` / `SST::Text`) | SST only |
+
+Production heuristic scrapers should reuse one `SST::Document` memoized from `parsed_body` (or a shared Document passed in), not re-parse HTML strings.
+
+## Constraints
+
+- **`SST::Normalizer::MAX_NODES` (5_000)** — Beyond this, normalization degrades to a semantic-tag-only keep set and logs a warning.
+- **Top-K** — `Scoring::Engine::TOP_K` (99) caps ranked segments materialized into articles; list strategy also budgets `use_top_selectors`.
+- **Typed stages** — Pipeline stages take `SST::Document` / `Segment` / `RankedSegment`, not ad-hoc Hash bags. Internal scraper APIs may change; the public gem surface is `lib/html2rss.rb`.
+
+## Non-goals
+
+- Replacing Selectors, Schema, Microdata, MF2, JsonState, or XhrArticles with SST.
+- A second HTML parser beside Nokogiri.
+- Dual Response payload (Nokogiri + SST always).
+- App-shell classification on SST.
+- Rewriting Sanitize transformers off Nokogiri.
+
+See also {file:CONTEXT CONTEXT.md} for module ownership (chrome, scoring, clustering), {Html2rss::FeedPipeline} for the request-strategy fallback chain (unrelated to article scraping), and {Html2rss::Capture} to turn a listing URL into a durable items-selector config.

data/lib/html2rss/auto_source/cleanup.rb

@@ -6,84 +6,175 @@ module Html2rss
     # Cleanup is responsible for cleaning up the extracted articles.
     # :reek:MissingSafeMethod { enabled: false }
     # It applies various strategies to filter and refine the article list.
-    class Cleanup
+    # Sole producer of admission drop tallies for {Html2rss::Status}.
+    class Cleanup # rubocop:disable Metrics/ClassLength -- reject steps + tallies stay co-located
       # Default cleanup behavior for auto-sourced article lists.
       DEFAULT_CONFIG = {
-        keep_different_domain: false,
-        min_words_title: 3
+        keep_different_domain: false
       }.freeze
 
+      # Minimum alphanumeric word count for present titles.
+      MIN_WORDS = 3
+
       # Allowed URL schemes for article filtering.
       VALID_SCHEMES = %w[http https].to_set.freeze
 
+      # Photo-credit agencies (single list → junk title regexes).
+      CREDIT_AGENCIES = [
+        'AFP',
+        'Getty(?:\s+Images)?',
+        'Reuters',
+        'dpa',
+        'Imagn'
+      ].freeze
+      private_constant :CREDIT_AGENCIES
+
+      AGENCY_ALT = CREDIT_AGENCIES.join('|').freeze
+      private_constant :AGENCY_ALT
+
+      # Sole denylist for extracted titles. Order: higher-frequency reasons first.
+      JUNK_TITLE_RULES = [
+        [:credit, %r{\A(?:#{AGENCY_ALT})(?:\s*/\s*(?:#{AGENCY_ALT}))*\z}ix],
+        [:credit, /\A(?:Image|Photo|Credit)\s*[:|]?\s*(?:#{AGENCY_ALT})\b/ix],
+        [:credit, /\ACourtesy\b.+\b(?:via|pool|Handout|#{AGENCY_ALT})\b/ix],
+        [:credit, /\bHandout\b.+\b(?:#{AGENCY_ALT})\b|\b(?:#{AGENCY_ALT})\b.+\bHandout\b/ix],
+        [:credit, /\A(?:Live\s+Updates|Analysis)\s*[•·.:-]?\s*.*\b(?:#{AGENCY_ALT})\b/ix],
+        [:cms_token, /\A(?:lucy\.\w[\w.-]*|methode[-.][\w.-]+)\z/i],
+        [:json_blob, /\A\{\s*["']?text["']?\s*:/],
+        [:slug, /\A\p{Alnum}+(?:[-_]\p{Alnum}+){2,}\z/],
+        [:date_prefix, /\A\d{4}(?:[\s.-]+\d{1,2}){2}\b/],
+        [:titleized_path, /\A(?:\d+|\p{Lu}[\p{L}\p{M}]*)(?:\s+(?:\d+|\p{Lu}[\p{L}\p{M}]*))*\s+\d{6,}\z/],
+        [:video_chrome, /\AClipped\s+From\s+Video\b/i],
+        [:video_chrome, /\AVideo\s*[•·]/i],
+        [:template, /\ACreated\s+from\s+Template\s+ID\b/i],
+        [:template, /(\{\{[^}]+\}\}|%\{\w+\})/]
+      ].freeze
+      private_constant :JUNK_TITLE_RULES
+
+      # Admitted articles plus reason → count tallies for drops.
+      Result = Data.define(:articles, :drop_tallies)
+
       class << self
         # @param articles [Array<Article>] extracted article candidates
         # @param url [Html2rss::Url] feed source URL used for same-host filtering
         # @param keep_different_domain [Boolean] whether to keep off-domain entries
-        # @param min_words_title [Integer] minimum word count for title filtering
-        # @return [Array<Article>] cleaned article list
-        def call(articles, url:, keep_different_domain:, min_words_title:)
+        # @return [Result] cleaned articles and frozen drop tallies
+        def call(articles, url:, keep_different_domain: DEFAULT_CONFIG.fetch(:keep_different_domain)) # rubocop:disable Metrics/MethodLength -- ordered reject pipeline
           Log.debug "Cleanup: start with #{articles.size} articles"
+          tallies = Hash.new(0)
 
-          articles.select!(&:valid?)
-
-          deduplicate_by!(articles, :url)
-
-          keep_only_http_urls!(articles)
-          reject_different_domain!(articles, url) unless keep_different_domain
-          keep_only_with_min_words_title!(articles, min_words_title:)
+          reject_invalid!(articles, tallies)
+          deduplicate_by_url!(articles, tallies)
+          keep_only_http_urls!(articles, tallies)
+          reject_self_links!(articles, url, tallies)
+          reject_different_domain!(articles, url, tallies) unless keep_different_domain
+          reject_excluded_destinations!(articles, tallies)
+          reject_low_quality_titles!(articles, tallies)
 
           Log.debug "Cleanup: end with #{articles.size} articles"
-          articles
+          Result.new(articles:, drop_tallies: tallies.freeze)
         end
 
-        ##
-        # Deduplicates articles by a given key.
+        # First matching junk reason for a title, or nil when the title is acceptable.
         #
-        # @param articles [Array<Article>] The list of articles to process.
-        # @param key [Symbol] The key to deduplicate by.
-        # @return [Array<Article>] the mutated articles array
-        def deduplicate_by!(articles, key)
+        # @param title [String, nil] candidate title text
+        # @return [Symbol, nil]
+        def junk_reason(title)
+          return if title.nil?
+
+          normalized = normalize_title(title)
+          return if normalized.empty?
+
+          JUNK_TITLE_RULES.find { |_, pattern| pattern.match?(normalized) }&.first
+        end
+
+        private
+
+        def reject_invalid!(articles, tallies)
+          tally_reject!(articles, tallies, 'invalid') { |article| !article.valid? }
+        end
+
+        def deduplicate_by_url!(articles, tallies)
           seen = {}
-          articles.reject! do |article|
-            value = article.public_send(key)
-            value.nil? || seen.key?(value).tap { seen[value] = true }
+          tally_reject!(articles, tallies, 'duplicate_url') do |article|
+            identity = url_identity(article.url)
+            identity.nil? || seen.key?(identity).tap { seen[identity] = true }
           end
         end
 
-        ##
-        # Keeps only articles with HTTP or HTTPS URLs.
-        #
-        # @param articles [Array<Article>] The list of articles to process.
-        # @return [Array<Article>] the mutated articles array
-        def keep_only_http_urls!(articles)
-          articles.select! { |article| VALID_SCHEMES.include?(article.url&.scheme) }
+        def keep_only_http_urls!(articles, tallies)
+          tally_reject!(articles, tallies, 'bad_scheme') do |article|
+            !VALID_SCHEMES.include?(article.url&.scheme)
+          end
         end
 
-        ##
-        # Rejects articles that have a URL not on the same domain as the source.
-        #
-        # @param articles [Array<Article>] The list of articles to process.
-        # @param base_url [Html2rss::Url] The source URL to compare against.
-        # @return [Array<Article>] the mutated articles array
-        def reject_different_domain!(articles, base_url)
-          base_host = base_url.host
-          articles.select! { |article| article.url&.host == base_host }
+        def reject_self_links!(articles, base_url, tallies)
+          source_identity = url_identity(base_url)
+          tally_reject!(articles, tallies, 'self_link') do |article|
+            url_identity(article.url) == source_identity
+          end
         end
 
-        ##
-        # Keeps only articles with a title that is present and has at least `min_words_title` words.
-        #
-        # @param articles [Array<Article>] The list of articles to process.
-        # @param min_words_title [Integer] The minimum number of words in the title.
-        # @return [Array<Article>] the mutated articles array
-        def keep_only_with_min_words_title!(articles, min_words_title:)
-          articles.select! do |article|
-            article.title ? word_count_at_least?(article.title, min_words_title) : true
+        def reject_different_domain!(articles, base_url, tallies)
+          base_domain = base_url.domain
+          tally_reject!(articles, tallies, 'different_domain') do |article|
+            article.url&.domain != base_domain
           end
         end
 
-        private
+        # Hard-exclude non-article destination classes (commerce/affiliate/utility chrome).
+        # PathClassifier owns route facts; Cleanup owns feed-item admission.
+        def reject_excluded_destinations!(articles, tallies)
+          tally_reject!(articles, tallies, 'excluded_destination') do |article|
+            excluded_destination?(article.url)
+          end
+        end
+
+        def excluded_destination?(url)
+          return false unless url
+
+          facts = LinkDestination::DestinationFacts.build(url)
+          return true if facts.high_confidence_junk_path || facts.high_confidence_utility_destination
+
+          facts.utility_path && !facts.content_path && !facts.strong_post_suffix
+        end
+
+        # Keep missing titles (nil provenance). Drop present junk/unnatural titles —
+        # blanking them would hide bad extraction as "unknown" and inflate empty items.
+        def reject_low_quality_titles!(articles, tallies) # rubocop:disable Metrics/MethodLength -- junk vs word-count reasons
+          articles.reject! do |article|
+            title = article.title
+            next false if title.nil?
+
+            reason = junk_reason(title)
+            if reason
+              tallies[reason.to_s] += 1
+              next true
+            end
+
+            next false if word_count_at_least?(title, MIN_WORDS)
+
+            tallies['low_word_count'] += 1
+            true
+          end
+        end
+
+        def tally_reject!(articles, tallies, reason)
+          articles.reject! do |article|
+            next false unless yield(article)
+
+            tallies[reason] += 1
+            true
+          end
+        end
+
+        def url_identity(url)
+          url&.without_fragment&.to_s
+        end
+
+        def normalize_title(title)
+          title.to_s.strip.gsub(/\s+/, ' ')
+        end
 
         def word_count_at_least?(str, min_words)
           count = 0

data/lib/html2rss/auto_source/scraper/json_state.rb

@@ -27,6 +27,36 @@ module Html2rss
           def json_documents(parsed_body)
             DocumentScanner.json_documents(parsed_body)
           end
+
+          # Walks a JSON document tree and yields normalized article hashes.
+          # Shared with {XhrArticles} so XHR-captured JSON reuses one discovery algorithm.
+          #
+          # @param document [Hash, Array, Object] parsed JSON document node
+          # @param base_url [String, Html2rss::Url] base URL for relative link resolution
+          # @yield [Hash{Symbol => Object}, nil] normalized article hash
+          # @return [void]
+          def discover_articles(document, base_url:, &block)
+            case document
+            when Array then handle_array(document, base_url:, &block)
+            when Hash then document.each_value { discover_articles(_1, base_url:, &block) if traversable?(_1) }
+            end
+          end
+
+          private
+
+          def handle_array(array, base_url:, &block)
+            if CandidateDetector.array_of_articles?(array)
+              array.each do |entry|
+                yield(ArticleNormalizer.normalise(entry, base_url:))
+              end
+            else
+              array.each { discover_articles(_1, base_url:, &block) if traversable?(_1) }
+            end
+          end
+
+          def traversable?(value)
+            value.is_a?(Array) || value.is_a?(Hash)
+          end
         end
 
         # @param parsed_body [Nokogiri::HTML::Document, nil] parsed HTML document
@@ -51,7 +81,7 @@ module Html2rss
           return enum_for(:each) unless block_given?
 
           json_documents.each do |document|
-            discover_articles(document) do |article|
+            self.class.discover_articles(document, base_url: url) do |article|
               yield article if article
             end
           end
@@ -64,27 +94,6 @@ module Html2rss
         def json_documents
           self.class.json_documents(parsed_body)
         end
-
-        def discover_articles(document, &block)
-          case document
-          when Array then handle_array(document, &block)
-          when Hash then document.each_value { discover_articles(_1, &block) if traversable?(_1) }
-          end
-        end
-
-        def handle_array(array, &block)
-          if CandidateDetector.array_of_articles?(array)
-            array.each do |entry|
-              yield(ArticleNormalizer.normalise(entry, base_url: url))
-            end
-          else
-            array.each { discover_articles(_1, &block) if traversable?(_1) }
-          end
-        end
-
-        def traversable?(value)
-          value.is_a?(Array) || value.is_a?(Hash)
-        end
       end
     end
   end

data/lib/html2rss/auto_source/scraper/schema/item_list.rb

@@ -31,9 +31,8 @@ module Html2rss
             return unless object.is_a?(Hash)
             return unless emit?(object, from_list_item:)
 
-            article = Thing.new(object, url: base_url || '').call
-            titleize_list_item_stub!(article) if from_list_item
-            article
+            # Leave empty titles empty — do not invent from URL path (Cleanup allows nil).
+            Thing.new(object, url: base_url || '').call
           end
 
           # @param element [Object] raw list entry
@@ -56,17 +55,6 @@ module Html2rss
 
             Schema.normalize_types(object[:@type]).intersect?(Thing::SUPPORTED_TYPES)
           end
-
-          # URL-only ListItem stubs historically used a titleized path as title.
-          #
-          # @param article [Hash] scraped article hash
-          # @return [void]
-          def titleize_list_item_stub!(article)
-            return unless article[:title].to_s.empty?
-            return unless (article_url = article[:url])
-
-            article[:title] = article_url.titleized
-          end
         end
       end
     end

data/lib/html2rss/auto_source/scraper/xhr_articles.rb

@@ -0,0 +1,69 @@
+# frozen_string_literal: true
+
+require 'json'
+
+module Html2rss
+  class AutoSource
+    module Scraper
+      # Extracts articles from JSON XHR/fetch responses captured during a
+      # browser-tier scrape (see RequestService::Response#captured_responses).
+      class XhrArticles
+        include Enumerable
+
+        # @return [Symbol] scraper config key
+        def self.options_key = :xhr_articles
+
+        # @param _opts [Hash] unused scraper options
+        # @return [Integer] zero — no additional HTTP requests
+        def self.request_slots(_opts = {}) = 0
+
+        # Not detectable from HTML alone; instance {#extractable?} uses captures.
+        #
+        # @param _parsed_body [Nokogiri::HTML::Document, nil]
+        # @return [Boolean]
+        def self.articles?(_parsed_body) = false
+
+        # @param _parsed_body [Nokogiri::HTML::Document, nil] unused HTML document
+        # @param url [String, Html2rss::Url] page URL used to resolve relative links
+        # @param captured_responses [Array<Hash>] JSON bodies from Response#captured_responses
+        # @param _opts [Hash] scraper-specific options
+        # @option _opts [Object] :_reserved reserved for future scraper-specific options
+        def initialize(_parsed_body, url:, captured_responses: [], **_opts)
+          @url = url
+          @captured_responses = captured_responses
+        end
+
+        # @return [Boolean] true when any captured body contains article-like arrays
+        def extractable?
+          parsed_bodies.any? { |doc| JsonState::CandidateDetector.candidate_array?(doc) }
+        end
+
+        # @yield [Hash{Symbol => Object}] normalized article hash
+        # @return [Enumerator, void] article enumerator when no block is given
+        def each
+          return enum_for(:each) unless block_given?
+
+          parsed_bodies.each do |doc|
+            JsonState.discover_articles(doc, base_url: @url) { |article| yield article if article }
+          end
+        end
+
+        private
+
+        def parsed_bodies
+          @parsed_bodies ||= @captured_responses.filter_map { |captured| parse(captured) }
+                                                .select { |doc| JsonState::CandidateDetector.candidate_array?(doc) }
+        end
+
+        def parse(captured)
+          body = captured[:body] || captured['body']
+          return unless body.is_a?(String)
+
+          JSON.parse(body, symbolize_names: true)
+        rescue JSON::ParserError
+          nil
+        end
+      end
+    end
+  end
+end