html2rss 0.26.0 → 0.27.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 2df4782641361435095b1adbc805a0b5f8bbf47637450a271b212e0905e025a1
-  data.tar.gz: 38542dd16fd74c1983be1d20cfc49b64508ce70eeb501f0a312e0b9777c59f07
+  metadata.gz: 286a89b05b23997b8cb5ef8b2930a59bd698bdeb07ba5007b3e30313b91324cd
+  data.tar.gz: 709147e30bb631551e8927e496fea5d76ef32373a76834b66adb54cbcdc8df16
 SHA512:
-  metadata.gz: e72f53ac552c7a0c65917d9d736a42e318360f205b7f532d845f89c0ea6bd219c09da0c1784596027566af8946aa2a9de937d21b39899a8838bd9fd70dd163ee
-  data.tar.gz: 69b705d89b11160fe0c392fb2045e61db8dc25d63a7e81af4d91201fce0face5e6fe3496b4b154af9d71eb41a6969491897d48162df382cd30facc30b3ab8bdc
+  metadata.gz: e9cabc478aa9d449697e2d8e99a5b91f6e2071d498eac34bdbc7828fffa06d2b02395d4f8e02853f0553ea97bdc35ca5a59c0b33fcc53b82ef809a247674826c
+  data.tar.gz: 32d343c2f3008917dbdbbb0859c29e710a89559e2ab079dcd3ba76672450ddaeb63572c8c71e0a1f5f7b327be7df667d785d2f7fb1069389e12a63093d83e3c9

data/README.md

@@ -37,14 +37,14 @@ Config -> Request -> Extraction -> Processing -> Building -> Output
 
 ## Capture API
 
-The `Html2rss.capture` method analyzes any URL and produces a reusable feed config hash with derived CSS selectors. Use it to speed up writing feed configuration files.
+The `Html2rss.capture` method analyzes any URL and produces a reusable feed config hash with an items selector and `enhance: true`. Use it to speed up writing feed configuration files.
 
 ```ruby
 config = Html2rss.capture('https://example.com/articles')
-File.write('my-feed.yml', YAML.dump(Html2rss::HashUtil.deep_stringify_keys(config)))
+File.write('my-feed.yml', Html2rss::Config.to_yaml(config))
 ```
 
-The CLI alias `html2rss capture` prints the generated config as YAML to stdout. See [`docs/capture.md`](docs/capture.md) for detailed documentation.
+The CLI alias `html2rss capture` prints the generated config as YAML to stdout. See [`lib/html2rss/capture/README.md`](lib/html2rss/capture/README.md) for detailed documentation.
 
 ## MCP Server
 
@@ -58,19 +58,41 @@ html2rss mcp
 html2rss mcp --transport http --port 8080
 ```
 
+stdio uses stdout for JSON-RPC, so the daemon logs to **stderr**. It defaults to `LOG_LEVEL=info` (the gem library default stays `warn`) so a foreground watcher sees the start banner, each tool call, and pipeline fallbacks. Use `LOG_LEVEL=debug` for more detail or `LOG_LEVEL=warn` to quiet it.
+
 HTTP transport needs `rack`, `rackup`, and `webrick` (declared gem dependencies). It listens on `127.0.0.1` only; do not expose it on a public interface without your own auth and Host/Origin controls.
 
-**Strategy note:** MCP tool `strategy: "auto"` collapses to `faraday` (no FeedPipeline botasaurus fallback). If results are empty or JS-gated, retry with `strategy: "botasaurus"` and `BOTASAURUS_SCRAPER_URL` set.
+**Strategy note:** MCP `scrape_url` / `capture_config` with `strategy: "auto"` run Faraday → Botasaurus AutoFallback. `inspect_url` uses Faraday when `auto` (cheap diagnostic); pin `botasaurus` when you need browser rendering for inspect.
+
+**Tool-call budget:** `scrape_url` is 1 call (auto already hops). Durable config is `capture_config` → `validate_config` → `apply_config`. Call `inspect_url` only when scrape/capture is weak or you need recon (final URL, status, https→http, native RSS/Atom).
+
+Cursor / Claude Desktop `mcp.json` must put Botasaurus on the **MCP process** (not only your shell):
+
+```json
+{
+  "mcpServers": {
+    "html2rss": {
+      "command": "mise",
+      "args": ["exec", "--", "html2rss", "mcp"],
+      "env": {
+        "BOTASAURUS_SCRAPER_URL": "http://127.0.0.1:4010"
+      }
+    }
+  }
+}
+```
+
+Read `html2rss://runtime` for a boolean `botasaurus_configured` (the URL is never returned). Every tool result is a JSON envelope (`ok`, `next_step`, `guidance`, `payload`) in both the text body and `structuredContent`. Follow `next_step` / `guidance`; do not parse scrape text as a raw item array.
 
 ### Tools
 
-| Name              | When to use                                               |
-| ----------------- | --------------------------------------------------------- |
-| `scrape_url`      | One-shot articles now (no saved config)                   |
-| `inspect_url`     | Diagnose weak scrape/capture (scrapers/SST/segments)      |
-| `capture_config`  | Derive a durable feed config (+ quality `_meta`)          |
-| `validate_config` | Schema-check a config before apply (`isError` on failure) |
-| `apply_config`    | Run a validated config → RSS XML                          |
+| Name              | When to use                                                                 |
+| ----------------- | --------------------------------------------------------------------------- |
+| `scrape_url`      | One-shot articles now (`payload.items`; empty is still success)             |
+| `inspect_url`     | Weak scrape/capture or recon (final_url, status, scheme_downgrade, feeds)   |
+| `capture_config`  | YAML draft in `payload.yaml`; strive `enhance: true`                        |
+| `validate_config` | Schema-check a `config` hash XOR `yaml` string (`isError` on failure)       |
+| `apply_config`    | RSS in `payload.rss`; `isError` when zero items; confirm `payload.item_count` |
 
 ### Resources
 
@@ -78,14 +100,15 @@ HTTP transport needs `rack`, `rackup`, and `webrick` (declared gem dependencies)
 | ----------------------- | --------------------------------------------------------------- |
 | `html2rss://schema`     | Full JSON Schema for feed configurations                        |
 | `html2rss://extractors` | Registered extractor **names** (options live in schema `$defs`) |
-| `html2rss://strategies` | Registered request strategy names                               |
+| `html2rss://strategies` | Published MCP strategies (`auto`, `faraday`, `botasaurus`) |
+| `html2rss://runtime`    | `botasaurus_configured` boolean (never the scraper URL)         |
 
 ### Prompts
 
-| Name                  | Description                                             |
-| --------------------- | ------------------------------------------------------- |
-| `scrape-webpage`      | Guided scrape → inspect/retry with botasaurus if needed |
-| `capture-feed-config` | Guided capture → validate → optional apply              |
+| Name                  | Description                                                          |
+| --------------------- | -------------------------------------------------------------------- |
+| `scrape-webpage`      | One `scrape_url` call; inspect only if weak or recon                 |
+| `capture-feed-config` | Capture YAML → validate → apply; catalog rewrite; strive enhance     |
 
 The MCP module (`Html2rss::MCP`) lazy-loads the `mcp` gem — no cost when the server is not running.
 
@@ -103,11 +126,11 @@ Set `BOTASAURUS_SCRAPER_URL` to `http://127.0.0.1:4010` and use strategy `botasa
 
 | Strategy     | Description                                                                   |
 | ------------ | ----------------------------------------------------------------------------- |
-| `auto`       | Tries `faraday`, falls back to `botasaurus` (default in gem/CLI FeedPipeline) |
+| `auto`       | Tries `faraday`, falls back to `botasaurus` (default in gem/CLI/MCP scrape)   |
 | `faraday`    | Plain HTTP requests via Faraday                                               |
 | `botasaurus` | Puppeteer-backed scraping for JavaScript pages                                |
 
-MCP tools intentionally collapse `auto` → `faraday` (see MCP section above). Elsewhere, strategy can be set via CLI (`--strategy`), gem API keyword argument, or feed config `request.strategy`. See the [request strategies docs](https://html2rss.github.io/ruby-gem/reference/strategy) for more details.
+`inspect_url` keeps Faraday when `auto` for cheap diagnostics. Elsewhere, strategy can be set via CLI (`--strategy`), gem API keyword argument, or feed config `request.strategy`. See the [request strategies docs](https://html2rss.github.io/ruby-gem/reference/strategy) for more details.
 
 ## License
 

data/html2rss.gemspec

@@ -39,7 +39,7 @@ Gem::Specification.new do |spec|
   spec.add_dependency 'faraday-follow_redirects'
   spec.add_dependency 'faraday-gzip', '~> 3'
   spec.add_dependency 'kramdown'
-  spec.add_dependency 'mcp', '~> 1.0'
+  spec.add_dependency 'mcp', '~> 1.2'
   spec.add_dependency 'mime-types', '> 3.0'
   spec.add_dependency 'nokogiri', '>= 1.10', '< 2.0'
   spec.add_dependency 'rack', '~> 3.0'

data/lib/html2rss/auto_source/README.md

@@ -0,0 +1,57 @@
+# AutoSource
+
+How html2rss builds feed items when a config has no (or incomplete) CSS selectors.
+
+## What and when
+
+`Html2rss.auto_source` / `auto_json_feed` (and any feed config with `auto_source:`) fetch a page once, then run an ordered set of scrapers against that response. Use it when you want “guess articles from this URL” without hand-writing selectors. Prefer explicit `selectors:` when you already know the list markup — that path stays on Nokogiri + `Html::Navigator`.
+
+Entry: `FeedPipeline` → `AutoSource#articles` → `Scraper.build_instance` → per-scraper extraction → `Cleanup`.
+
+## Live flow
+
+1. **Request** — `RequestSession` returns a `Response` with `body` (String) and `parsed_body` (Nokogiri HTML).
+2. **Scraper tiers** — Enabled scrapers that claim the page (shallow `articles?` or instance `extractable?`) run in `Scraper::SCRAPER_TIERS` order. Merge within a tier, then stop when enough articles survive Cleanup:
+
+   1. In-page structured: Schema, Microdata, Microformats2, JsonState, XhrArticles
+   2. Follow-up IO: WordPress API, Sitemap, MetaOembed
+   3. Heuristic: SemanticHtml
+   4. Heuristic: Html (skipped when earlier tiers already admitted at least one clean article)
+
+3. **Structured / API scrapers** — Schema, Microdata, MF2, JSON state, XHR JSON, oEmbed, WordPress REST, and Sitemap work on Nokogiri CSS/XPath or JSON/XML parsers. They do not use SST.
+4. **Heuristic scrapers** — `SemanticHtml` and `Html` normalize once into an `SST::Document`, then:
+
+   `SST::Normalizer` → `AutoSource::Segmenter` → `Scoring::Engine` → extractor / article materialization.
+
+5. **Cleanup** — Merge, dedupe, hard-exclude non-article destinations (via `PathClassifier` facts), drop junk titles, and trim to `limit`. Html is skipped when earlier tiers already admitted clean items.
+
+Segmenter strategies: `:semantic` (leaf containers + primary link), `:list` (repeated tag paths), `:cluster` (class/structure grids for anchorless cards). Scoring ranks and demotes; `LinkDestination::NoisePolicy` owns content-anchor eligibility. Cleanup owns feed-item admission.
+
+## Nokogiri vs SST boundaries
+
+| Surface | Owns DOM |
+| --- | --- |
+| `Response#parsed_body` | Single HTML parse for the page |
+| Schema / Microdata / MF2 / JsonState / XhrArticles / MetaOembed / app-shell detection | Nokogiri |
+| Sitemap detection (CSS/XPath) | Nokogiri; URL list parsing uses raw `response.body` (XML string) |
+| Selectors path / Sanitize transformers | Nokogiri (unchanged) |
+| `SST::Normalizer` | Sole Nokogiri consumer on the heuristic auto-source path |
+| Segmenter, Scoring, `Html::SstArticleExtractor`, heuristic chrome (`SST::Tags` / `SST::Text`) | SST only |
+
+Production heuristic scrapers should reuse one `SST::Document` memoized from `parsed_body` (or a shared Document passed in), not re-parse HTML strings.
+
+## Constraints
+
+- **`SST::Normalizer::MAX_NODES` (5_000)** — Beyond this, normalization degrades to a semantic-tag-only keep set and logs a warning.
+- **Top-K** — `Scoring::Engine::TOP_K` (99) caps ranked segments materialized into articles; list strategy also budgets `use_top_selectors`.
+- **Typed stages** — Pipeline stages take `SST::Document` / `Segment` / `RankedSegment`, not ad-hoc Hash bags. Internal scraper APIs may change; the public gem surface is `lib/html2rss.rb`.
+
+## Non-goals
+
+- Replacing Selectors, Schema, Microdata, MF2, JsonState, or XhrArticles with SST.
+- A second HTML parser beside Nokogiri.
+- Dual Response payload (Nokogiri + SST always).
+- App-shell classification on SST.
+- Rewriting Sanitize transformers off Nokogiri.
+
+See also {file:CONTEXT CONTEXT.md} for module ownership (chrome, scoring, clustering), {Html2rss::FeedPipeline} for the request-strategy fallback chain (unrelated to article scraping), and {Html2rss::Capture} to turn a listing URL into a durable items-selector config.

data/lib/html2rss/auto_source/cleanup.rb

@@ -6,7 +6,8 @@ module Html2rss
     # Cleanup is responsible for cleaning up the extracted articles.
     # :reek:MissingSafeMethod { enabled: false }
     # It applies various strategies to filter and refine the article list.
-    class Cleanup
+    # Sole producer of admission drop tallies for {Html2rss::Status}.
+    class Cleanup # rubocop:disable Metrics/ClassLength -- reject steps + tallies stay co-located
       # Default cleanup behavior for auto-sourced article lists.
       DEFAULT_CONFIG = {
         keep_different_domain: false
@@ -18,97 +19,161 @@ module Html2rss
       # Allowed URL schemes for article filtering.
       VALID_SCHEMES = %w[http https].to_set.freeze
 
-      # Credit-agency-only or photo-credit titles (not headlines).
-      CREDIT_TITLE = %r{
-        \A(?:AFP|Getty(?:\s+Images)?|Reuters|dpa|Imagn)
-          (?:\s*/\s*(?:AFP|Getty(?:\s+Images)?|Reuters|dpa|Imagn))*\z
-        |
-        \A(?:Image|Photo|Credit)\s*[:|]?\s*
-          (?:AFP|Getty(?:\s+Images)?|Reuters|dpa|Imagn)\b
-      }ix
-
-      # Dotted / methode CMS tokens mistaken for titles.
-      CMS_TOKEN_TITLE = /\A(?:lucy\.\w[\w.-]*|methode[-.][\w.-]+)\z/i
-
-      # Raw URL slug / token clusters (hyphen or underscore, no natural phrasing).
-      SLUG_TITLE = /\A\p{Alnum}+(?:[-_]\p{Alnum}+){2,}\z/
-
-      # Date-prefix path tokens, raw or titleized ("2026 08 16 …", "2026-08-16-…").
-      DATE_PREFIX_TITLE = /\A\d{4}(?:[\s.-]+\d{1,2}){2}\b/
-
-      # Titleized path ending in a long numeric CMS id.
-      TITLEIZED_PATH_TITLE = /\A(?:\d+|\p{Lu}[\p{L}\p{M}]*)(?:\s+(?:\d+|\p{Lu}[\p{L}\p{M}]*))*\s+\d{6,}\z/
-
-      # Template / placeholder tokens mistaken for titles.
-      TEMPLATE_TITLE = /(\{\{[^}]+\}\}|%\{\w+\})/
+      # Photo-credit agencies (single list → junk title regexes).
+      CREDIT_AGENCIES = [
+        'AFP',
+        'Getty(?:\s+Images)?',
+        'Reuters',
+        'dpa',
+        'Imagn'
+      ].freeze
+      private_constant :CREDIT_AGENCIES
+
+      AGENCY_ALT = CREDIT_AGENCIES.join('|').freeze
+      private_constant :AGENCY_ALT
+
+      # Sole denylist for extracted titles. Order: higher-frequency reasons first.
+      JUNK_TITLE_RULES = [
+        [:credit, %r{\A(?:#{AGENCY_ALT})(?:\s*/\s*(?:#{AGENCY_ALT}))*\z}ix],
+        [:credit, /\A(?:Image|Photo|Credit)\s*[:|]?\s*(?:#{AGENCY_ALT})\b/ix],
+        [:credit, /\ACourtesy\b.+\b(?:via|pool|Handout|#{AGENCY_ALT})\b/ix],
+        [:credit, /\bHandout\b.+\b(?:#{AGENCY_ALT})\b|\b(?:#{AGENCY_ALT})\b.+\bHandout\b/ix],
+        [:credit, /\A(?:Live\s+Updates|Analysis)\s*[•·.:-]?\s*.*\b(?:#{AGENCY_ALT})\b/ix],
+        [:cms_token, /\A(?:lucy\.\w[\w.-]*|methode[-.][\w.-]+)\z/i],
+        [:json_blob, /\A\{\s*["']?text["']?\s*:/],
+        [:slug, /\A\p{Alnum}+(?:[-_]\p{Alnum}+){2,}\z/],
+        [:date_prefix, /\A\d{4}(?:[\s.-]+\d{1,2}){2}\b/],
+        [:titleized_path, /\A(?:\d+|\p{Lu}[\p{L}\p{M}]*)(?:\s+(?:\d+|\p{Lu}[\p{L}\p{M}]*))*\s+\d{6,}\z/],
+        [:video_chrome, /\AClipped\s+From\s+Video\b/i],
+        [:video_chrome, /\AVideo\s*[•·]/i],
+        [:template, /\ACreated\s+from\s+Template\s+ID\b/i],
+        [:template, /(\{\{[^}]+\}\}|%\{\w+\})/]
+      ].freeze
+      private_constant :JUNK_TITLE_RULES
+
+      # Admitted articles plus reason → count tallies for drops.
+      Result = Data.define(:articles, :drop_tallies)
 
       class << self
         # @param articles [Array<Article>] extracted article candidates
         # @param url [Html2rss::Url] feed source URL used for same-host filtering
         # @param keep_different_domain [Boolean] whether to keep off-domain entries
-        # @return [Array<Article>] cleaned article list
-        def call(articles, url:, keep_different_domain: DEFAULT_CONFIG.fetch(:keep_different_domain))
+        # @return [Result] cleaned articles and frozen drop tallies
+        def call(articles, url:, keep_different_domain: DEFAULT_CONFIG.fetch(:keep_different_domain)) # rubocop:disable Metrics/MethodLength -- ordered reject pipeline
           Log.debug "Cleanup: start with #{articles.size} articles"
+          tallies = Hash.new(0)
 
-          articles.select!(&:valid?)
-
-          deduplicate_by_url!(articles)
-          keep_only_http_urls!(articles)
-          reject_self_links!(articles, url)
-          reject_different_domain!(articles, url) unless keep_different_domain
-          reject_low_quality_titles!(articles)
+          reject_invalid!(articles, tallies)
+          deduplicate_by_url!(articles, tallies)
+          keep_only_http_urls!(articles, tallies)
+          reject_self_links!(articles, url, tallies)
+          reject_different_domain!(articles, url, tallies) unless keep_different_domain
+          reject_excluded_destinations!(articles, tallies)
+          reject_low_quality_titles!(articles, tallies)
 
           Log.debug "Cleanup: end with #{articles.size} articles"
-          articles
+          Result.new(articles:, drop_tallies: tallies.freeze)
+        end
+
+        # First matching junk reason for a title, or nil when the title is acceptable.
+        #
+        # @param title [String, nil] candidate title text
+        # @return [Symbol, nil]
+        def junk_reason(title)
+          return if title.nil?
+
+          normalized = normalize_title(title)
+          return if normalized.empty?
+
+          JUNK_TITLE_RULES.find { |_, pattern| pattern.match?(normalized) }&.first
         end
 
         private
 
-        def deduplicate_by_url!(articles)
+        def reject_invalid!(articles, tallies)
+          tally_reject!(articles, tallies, 'invalid') { |article| !article.valid? }
+        end
+
+        def deduplicate_by_url!(articles, tallies)
           seen = {}
-          articles.reject! do |article|
+          tally_reject!(articles, tallies, 'duplicate_url') do |article|
             identity = url_identity(article.url)
             identity.nil? || seen.key?(identity).tap { seen[identity] = true }
           end
         end
 
-        def keep_only_http_urls!(articles)
-          articles.select! { |article| VALID_SCHEMES.include?(article.url&.scheme) }
+        def keep_only_http_urls!(articles, tallies)
+          tally_reject!(articles, tallies, 'bad_scheme') do |article|
+            !VALID_SCHEMES.include?(article.url&.scheme)
+          end
         end
 
-        def reject_self_links!(articles, base_url)
+        def reject_self_links!(articles, base_url, tallies)
           source_identity = url_identity(base_url)
-          articles.reject! { |article| url_identity(article.url) == source_identity }
+          tally_reject!(articles, tallies, 'self_link') do |article|
+            url_identity(article.url) == source_identity
+          end
+        end
+
+        def reject_different_domain!(articles, base_url, tallies)
+          base_domain = base_url.domain
+          tally_reject!(articles, tallies, 'different_domain') do |article|
+            article.url&.domain != base_domain
+          end
         end
 
-        def reject_different_domain!(articles, base_url)
-          base_host = base_url.host
-          articles.select! { |article| article.url&.host == base_host }
+        # Hard-exclude non-article destination classes (commerce/affiliate/utility chrome).
+        # PathClassifier owns route facts; Cleanup owns feed-item admission.
+        def reject_excluded_destinations!(articles, tallies)
+          tally_reject!(articles, tallies, 'excluded_destination') do |article|
+            excluded_destination?(article.url)
+          end
+        end
+
+        def excluded_destination?(url)
+          return false unless url
+
+          facts = LinkDestination::DestinationFacts.build(url)
+          return true if facts.high_confidence_junk_path || facts.high_confidence_utility_destination
+
+          facts.utility_path && !facts.content_path && !facts.strong_post_suffix
         end
 
         # Keep missing titles (nil provenance). Drop present junk/unnatural titles —
         # blanking them would hide bad extraction as "unknown" and inflate empty items.
-        def reject_low_quality_titles!(articles)
-          articles.select! do |article|
+        def reject_low_quality_titles!(articles, tallies) # rubocop:disable Metrics/MethodLength -- junk vs word-count reasons
+          articles.reject! do |article|
             title = article.title
-            title.nil? || (word_count_at_least?(title, MIN_WORDS) && !junk_title?(title))
+            next false if title.nil?
+
+            reason = junk_reason(title)
+            if reason
+              tallies[reason.to_s] += 1
+              next true
+            end
+
+            next false if word_count_at_least?(title, MIN_WORDS)
+
+            tallies['low_word_count'] += 1
+            true
           end
         end
 
-        def url_identity(url)
-          url&.without_fragment&.to_s
+        def tally_reject!(articles, tallies, reason)
+          articles.reject! do |article|
+            next false unless yield(article)
+
+            tallies[reason] += 1
+            true
+          end
         end
 
-        def junk_title?(title)
-          CREDIT_TITLE.match?(title) || CMS_TOKEN_TITLE.match?(title) || unnatural_title?(title)
+        def url_identity(url)
+          url&.without_fragment&.to_s
         end
 
-        def unnatural_title?(title)
-          stripped = title.to_s.strip
-          SLUG_TITLE.match?(stripped) ||
-            DATE_PREFIX_TITLE.match?(stripped) ||
-            TITLEIZED_PATH_TITLE.match?(stripped) ||
-            TEMPLATE_TITLE.match?(stripped)
+        def normalize_title(title)
+          title.to_s.strip.gsub(/\s+/, ' ')
         end
 
         def word_count_at_least?(str, min_words)

data/lib/html2rss/auto_source/scraper.rb

@@ -16,6 +16,8 @@ module Html2rss
       APP_SHELL_ROOT_SELECTORS = '#app, #root, #__next, [data-reactroot], [ng-app], [id*="app-shell"]'
       # Maximum anchors tolerated before app-shell detection is considered unlikely.
       APP_SHELL_MAX_ANCHORS = 2
+      # Minimum same-page anchors suggesting a high-entropy homepage/hub surface.
+      HIGH_ENTROPY_MIN_ANCHORS = 20
       # Maximum visible text length tolerated for app-shell classification.
       APP_SHELL_MAX_VISIBLE_TEXT_LENGTH = 220
 
@@ -49,6 +51,9 @@ module Html2rss
           app_shell: 'app-shell surface detected (client-rendered page with little or no ' \
                      'server-rendered article HTML). Configure BOTASAURUS_SCRAPER_URL or target a direct ' \
                      'listing/update URL instead of a homepage or shell entrypoint.',
+          high_entropy_surface: 'high-entropy surface (e.g. news homepage) with few or no durable ' \
+                                'article destinations after admission. Target a section, topic, or ' \
+                                'listing/update URL instead of the site homepage.',
           unsupported_surface: 'unsupported extraction surface for auto mode. ' \
                                'Try a direct listing/changelog/category URL, ' \
                                'or use explicit selectors in a feed config.'
@@ -197,6 +202,7 @@ module Html2rss
       def self.classify_no_scraper_surface(parsed_body, body: nil)
         return :blocked_surface if blocked_surface?(parsed_body, body:)
         return :app_shell if app_shell_surface?(parsed_body)
+        return :high_entropy_surface if high_entropy_surface?(parsed_body)
 
         :unsupported_surface
       end
@@ -216,6 +222,13 @@ module Html2rss
       end
       private_class_method :app_shell_surface?
 
+      def self.high_entropy_surface?(parsed_body)
+        return false if parsed_body.nil?
+
+        parsed_body.css('body a[href]').size >= HIGH_ENTROPY_MIN_ANCHORS
+      end
+      private_class_method :high_entropy_surface?
+
       def self.sparse_anchor_surface?(parsed_body)
         parsed_body.css('body a[href]').size <= APP_SHELL_MAX_ANCHORS
       end

data/lib/html2rss/auto_source.rb

@@ -15,8 +15,8 @@ module Html2rss
   # @see Html2rss::AutoSource::Scraper::Schema
   # @see Html2rss::AutoSource::Scraper::SemanticHtml
   # @see Html2rss::AutoSource::Scraper::Html
-  # rubocop:disable Metrics/ClassLength -- defaults + tiered extract stay on the contributor entry type
-  class AutoSource
+  # {include:file:lib/html2rss/auto_source/README.md}
+  class AutoSource # rubocop:disable Metrics/ClassLength -- defaults + tiered extract stay on the contributor entry type
     # Default max articles to keep (also the short-circuit floor across scraper tiers).
     DEFAULT_LIMIT = 25
 
@@ -107,15 +107,26 @@ module Html2rss
     # Tiers: in-page structured → follow-up IO → SemanticHtml → Html.
     # SST is built only when a heuristic tier runs. Later tiers are skipped once
     # +limit+ articles with url+title remain after Cleanup; the result is capped to +limit+.
+    # Html is skipped when earlier tiers already admitted at least one clean article
+    # below +limit+ — quality over padding with weaker heuristic junk.
     #
     # @return [Array<Html2rss::Article>] extracted articles
     def articles
       @articles ||= extract_articles
     rescue Html2rss::AutoSource::Scraper::NoScraperFound => error
       Log.warn "#{self.class}: no scraper matched #{url} (#{error.message})"
+      @admission_drops = {}.freeze
       []
     end
 
+    ##
+    # Reason → count tallies from the final {Cleanup} pass (empty until {#articles} runs).
+    #
+    # @return [Hash{String => Integer}]
+    def admission_drops
+      @admission_drops || {}
+    end
+
     private
 
     attr_reader :url, :parsed_body, :body, :request_session, :captured_responses
@@ -130,6 +141,7 @@ module Html2rss
 
       Scraper::SCRAPER_TIERS.each do |tier|
         break if enough_articles?(articles)
+        break if skip_html_padding?(tier, articles)
 
         if Scraper.heuristic_tier?(tier)
           document ||= Scraper.normalize_sst(parsed_body)
@@ -160,16 +172,31 @@ module Html2rss
 
       raise Scraper.no_scraper_found_for(parsed_body, body:) unless matched
 
-      Cleanup.call(articles, url:, **cleanup_options).first(article_limit)
+      result = Cleanup.call(articles, url:, **cleanup_options)
+      @admission_drops = result.drop_tallies
+      result.articles.first(article_limit)
     end
     # rubocop:enable Metrics/AbcSize, Metrics/CyclomaticComplexity, Metrics/MethodLength, Metrics/PerceivedComplexity
 
     def enough_articles?(articles)
-      threshold = article_limit
-      return false if articles.size < threshold
+      return false if articles.size < article_limit
+
+      admitted_articles(articles).size >= article_limit
+    end
 
-      cleaned = Cleanup.call(articles.dup, url:, **cleanup_options)
-      cleaned.count { |article| article.url && !article.title.to_s.empty? } >= threshold
+    # Prefer fewer clean items over refilling +limit+ from the Html heuristic tier.
+    def skip_html_padding?(tier, articles)
+      html_only_tier?(tier) && admitted_articles(articles).any?
+    end
+
+    def html_only_tier?(tier)
+      tier == [Scraper::Html]
+    end
+
+    def admitted_articles(articles)
+      Cleanup.call(articles.dup, url:, **cleanup_options).articles.select do |article|
+        article.url && !article.title.to_s.empty?
+      end
     end
 
     def article_limit
@@ -193,5 +220,4 @@ module Html2rss
       @opts.fetch(:cleanup, {})
     end
   end
-  # rubocop:enable Metrics/ClassLength
 end

data/lib/html2rss/capture/README.md

@@ -0,0 +1,61 @@
+# Capture
+
+`Html2rss.capture` (and CLI `html2rss capture`) analyzes a URL through the feed pipeline and produces a reusable config with **items selector + `enhance: true` only** — no title/url/description attribute-selector soup. At feed-build time, `enhance: true` fills missing article fields via `Html::ArticleExtractor` on each matched item.
+
+## When to use it
+
+Point `capture` at a listing URL when you want a first-draft YAML config instead of hand-writing selectors. Treat the output as a draft: selector quality depends on page structure.
+
+## Gem API
+
+```ruby
+config = Html2rss.capture('https://example.com/articles')
+
+# {
+#   channel: { url: "...", title: "...", time_zone: "UTC" },
+#   selectors: {
+#     items: { selector: "div.post", enhance: true }
+#   }
+# }
+
+File.write('my-feed.yml', YAML.dump(Html2rss::HashUtil.deep_stringify_keys(config)))
+feed = Html2rss.feed(config)
+```
+
+`Capture.build` returns a `CaptureResult` with quality meta (`has_selectors`, `segment_strategy`, `admission_drops`, `selected_strategy`). `Html2rss.capture` returns only the config hash.
+
+### Options
+
+```ruby
+Html2rss.capture('https://spa-site.com', strategy: :botasaurus)
+Html2rss.capture('https://example.com', items_selector: '.article-card')
+Html2rss.capture('https://example.com', strategy: :local_file, local_file_path: './page.html')
+```
+
+`strategy: :auto` uses the same AutoFallback chain as scrape (`faraday` → `botasaurus`). When AutoFallback selects a concrete strategy (or you pin one), Capture **stamps** `strategy:` into the emitted config so later `Html2rss.feed(config)` replays the same transport.
+
+## CLI
+
+```bash
+html2rss capture https://example.com/articles
+html2rss capture https://example.com --strategy botasaurus
+html2rss capture https://example.com/articles > my-feed.yml
+html2rss capture https://example.com --input ./page.html
+html2rss capture https://example.com --explain   # quality JSON on stderr; YAML on stdout
+```
+
+## How it works
+
+1. **Request** — `FeedPipeline` (AutoFallback when `:auto`)
+2. **Discover** — AutoSource extracts admitted articles
+3. **Segment** — try SST Segmenter strategies `:list` → `:cluster` → `:semantic`
+4. **Gate** — emit items selector only when ≥ `MIN_SELECTOR_MATCHES` (2) articles match
+5. **Assemble** — `{ items: { selector:, enhance: true } }` plus channel
+
+## Constraints
+
+- Selector quality depends on page structure; treat output as a first draft.
+- When the quality gate fails, selectors are omitted (`has_selectors: false`) rather than inventing attribute selectors.
+- Botasaurus hops need `BOTASAURUS_SCRAPER_URL`.
+
+See also {Html2rss::AutoSource} for article discovery and {Html2rss::FeedPipeline} for the `:auto` request chain.