html2rss 0.33.0 → 0.34.0

data/lib/html2rss/auto_source/README.md

@@ -11,21 +11,25 @@ Entry: `FeedPipeline` → `AutoSource#articles` → `Scraper.build_instance` →
 ## Live flow
 
 1. **Request** — `RequestSession` returns a `Response` with `body` (String) and `parsed_body` (Nokogiri HTML, or parsed JSON for JSON media types). Direct syndication responses skip HTML scrapers and parse via `Syndication::Parser`.
-2. **JSON document body** — When the response is a JSON media type (`Response#json_response?`), AutoSource runs `JsonDocument` only (no HTML tiers). Scalar JSON is `unsupported_surface`. Same-domain Cleanup stays on by default (`keep_different_domain: false`). If the scraper yielded items and Cleanup kept none, `Log.warn` emits `auto_source.admission_emptied` with drop tallies; a `different_domain` tally hints at `auto_source.cleanup.keep_different_domain: true` / `--keep-different-domain`, which lets the remote source control off-domain link targets (no affiliation or safety check). Guidance stays in help/docs and that structured warning — stdout feed output stays composable.
-3. **Scraper tiers** (HTML only) — Enabled scrapers that claim the page (shallow `articles?` or instance `extractable?`) run in `Scraper::SCRAPER_TIERS` order. Merge within a tier, then stop when enough articles survive Cleanup:
+2. **JSON document body** — When the response is a JSON media type (`Response#json_response?`), AutoSource runs `JsonDocument` only (no HTML tiers). Scalar JSON is `unsupported_surface`. Same-domain Cleanup stays on by default (`keep_different_domain: false`). If the scraper yielded items and Cleanup kept none, `Log.warn` emits `auto_source.admission_emptied` with drop tallies; a `different_domain` tally hints at `auto_source.cleanup.keep_different_domain: true` / `--keep-different-domain`, which lets the remote source control off-domain link targets (no affiliation or safety check); a `sponsored` tally hints at `auto_source.cleanup.keep_sponsored: true` / `--keep-sponsored`. Guidance stays in help/docs and that structured warning — stdout feed output stays composable.
+3. **Scraper tiers** (HTML only) — Enabled scrapers that claim the page (shallow `articles?` or instance `extractable?`) run in `Scraper::SCRAPER_TIERS` order. Merge within a tier, then stop when `limit` counting articles survive Cleanup. Every admitted item counts except a media item (`Article#media?`: a Schema `VideoObject`, `AudioObject`, `PodcastEpisode` or `Clip`, standalone, in `@graph` or an array, or an ItemList element) on a page that is no media page, and a structured item the page does not show in its content (`Corroboration#shown?`: a `CORROBORATED_SCRAPERS` item kept by score, not rendered as a content anchor); both stay admitted but never stop the tiers, and the final cap to `limit` takes counting items first, then fills with the rest. A media page (`Corroboration#media_page?`: more than half of the items read off the page and shown in its content are media, each article identity counted once, such as a podcast's episode list) counts its media items and puts them first: media items, then the other counting items, each group in the document order of its content anchors (`Corroboration#page_order` over `PageLinks#content_position`, from the placement walk), not tier order. A page is judged a media page only after a heuristic tier has read its cards. A native feed's items come first and the page tiers still run to fill up to `limit`; a page copy of a feed article merges into the feed item (below):
 
-   0. Native feed: NativeFeed (head alternates + path discovery → RSS/Atom parse)
+   0. Native feed: NativeFeed (head alternates + path discovery → RSS/Atom/RDF parse); what the feed states about its channel (author, iTunes categories, explicit flag) is kept as `AutoSource#feed_channel` for `Channel` (`CONTEXT.md` § Channel), and where it was read (`NativeFeed#feed_url`) for Cleanup's domain rule. A link-less feed item takes its guid when that is an absolute http(s) URL, else its first audio or video file (`Syndication::Parser`)
    1. In-page structured: Schema, Microdata, Microformats2, JsonState, XhrArticles
    2. Follow-up IO: WordPress API, Sitemap, MetaOembed
    3. Heuristic: SemanticHtml
    4. Heuristic: Html (skipped when earlier tiers already admitted at least one clean article)
 
-4. **Structured / API scrapers** — Schema, Microdata, MF2, JSON state, XHR JSON, oEmbed, WordPress REST, and Sitemap work on Nokogiri CSS/XPath or JSON/XML parsers. They do not use SST. `JsonDocument` walks a JSON response body (JSON Feed mapping or `JsonState.discover_articles`).
-5. **Heuristic scrapers** — `SemanticHtml` and `Html` normalize once into an `SST::Document`, then:
+4. **Structured / API scrapers** — Schema, Microdata, MF2, JSON state, XHR JSON, oEmbed, WordPress REST, and Sitemap work on Nokogiri CSS/XPath or JSON/XML parsers. They do not extract from SST. `JsonDocument` walks a JSON response body (JSON Feed mapping or `JsonState.discover_articles`).
+5. **Heuristic scrapers** — `SemanticHtml` and `Html` share the page `SST::Document` (built lazily, at most once per page, by the first consumer: a heuristic tier or the page check), then:
 
    `SST::Normalizer` → `AutoSource::Segmenter` → `Scoring::Engine` → extractor / article materialization.
 
-6. **Cleanup** — Merge, dedupe, hard-exclude non-article destinations (via `PathClassifier` facts), drop junk titles, and trim to `limit`. Html is skipped when earlier tiers already admitted clean items.
+6. **Cleanup** — Page-check structured items, merge copies of one article (`Url#article_identity`) into one item (the first copy keeps every field it carries — a native feed item its date, guid, description, publisher title, declared media files and image, even when the page card's image is larger — and later copies fill only the fields it lacks, such as the page card's image when the feed item has none, or categories, via `Article#merge`), drop explicitly sponsored items of every source, drop items linking off the page's domain (unless `keep_different_domain`; a followed native feed's own items may also link to that feed's domain, such as a podcast platform's episode pages; an item whose URL is its own audio/video enclosure, a link-less episode served from a CDN, skips the rule), hard-exclude non-article destinations (via `PathClassifier` facts), drop untitled page-derived items (`blank_title`; publisher feeds may carry untitled posts), titles under three words (`WordCount`: unspaced Japanese or Chinese text counts one word per six characters) and junk titles, and trim to `limit`. Titles from the heuristic scrapers (`Scraper::HEURISTIC_SCRAPERS`) reach Cleanup composed by `Article::Title` (`Kicker: Headline` read from elements by `Html::TitlePieces`: kicker leaves in or beside the heading, the headline piece of an element-only heading or anchor or its `data-title`, and no date, time, duration, concealed, lead or trailing tag elements, nor teaser clock, label or type noise in en, de, fr, es, it, nl, pl, sv and ja; a Japanese leading `【…】`/`＜…＞` is dropped when it names a content type and is the kicker otherwise); every other scraper carries a publisher-declared headline (structured data, feeds, APIs) and keeps it verbatim apart from whitespace collapse and invisible characters (soft hyphens, zero-width characters). Cleanup rejects junk titles but never rewrites them. Html is skipped when earlier tiers already admitted clean items.
+
+   Page check: items from `Scraper::CORROBORATED_SCRAPERS` go through `Corroboration#verdict` before dedupe. `PageLinks` places each URL on the rendered page (content, peripheral, chrome, absent). Content keeps; chrome-only drops as `chrome_only`; unrendered drops as `not_on_page` when the page renders at least half of the item's scraper cohort (or the cohort has fewer than three items), otherwise it is scored with absence counting against it; peripheral items, and any item on a sparse page or a JSON response (no page), keep only on a private article-likeness score, else `unlikely`. Dropped items never count toward tier stop, so the heuristic tiers run and merge; `PageLinks` keys URLs by `Url#article_identity` (ignores scheme, fragment, trailing slash, and tracking query params; other params stay significant), the same key Cleanup dedupes by. A kept structured item whose URL differs from its rendered content anchor only by those takes the anchor's URL minus tracking params, never downgraded from `https` to `http`, so the item guid stays stable (`PageLinks#canonical_url` via `Corroboration#canonical_url`), so it merges with the heuristic copy; for a shared URL the kept structured copy wins dedupe.
+
+   Sponsored items: `PageLinks` marks a URL sponsored when every card linking it carries a `LinkDestination::Sponsorship` marker. A container marker (`rel="sponsored"`, a truthy `data-sponsored`/`data-advertorial`, a sponsored or native-advertising class token such as `card--sponsored` or `adv-native-strip`, or an ARIA ad label) marks every destination inside it; a label inside a card link marks that link's destination across its one-card unit; any other label (an ad-label class, or a standalone element whose whole text is an ad word such as "Anzeige" or a paid phrase such as "Sponsored by …") marks the headline destination of the `article`, `section`, `aside` or `li` it sits in; and a paid phrase outside every anchor binds at its first ancestor holding a destination: the one card it sits in alone (bare `div` cards included) is marked, and so is every card of a section whose header it is (a heading, a `header` element or a header box), never from plain text before the cards. Label words in headlines, anchor text or prose, ad words on topic chips, and unpaid attribution ("Presented by …"), never count. Cleanup drops such items from every source as `sponsored`, a native feed's paid post included; `auto_source.cleanup.keep_sponsored: true` keeps them, titled without the paid label. Rules: `CONTEXT.md` § Sponsored items.
 
 7. **Entry URL resolution** (pipeline, not this class) — when AutoFallback sees a weak homepage extract, `FeedResolution` may rewrite the scrape URL to a listing/feed before escalating strategies. See {Html2rss::FeedPipeline} and {Html2rss::FeedResolution}.
 
@@ -37,13 +41,13 @@ Segmenter strategies: `:semantic` (leaf containers + primary link), `:list` (car
 | --------------------------------------------------------------------------------------------- | ---------------------------------------------------------------- |
 | `Response#parsed_body`                                                                        | HTML document, or Hash/Array for JSON bodies                     |
 | `JsonDocument`                                                                                | Parsed JSON Hash/Array (no Nokogiri)                             |
-| Schema / Microdata / MF2 / JsonState / XhrArticles / MetaOembed / app-shell detection         | Nokogiri                                                         |
+| Schema / Microdata / MF2 / JsonState / XhrArticles / MetaOembed / app-shell detection         | Nokogiri (items page-checked against SST anchors via `PageLinks`) |
 | Sitemap detection (CSS/XPath)                                                                 | Nokogiri; URL list parsing uses raw `response.body` (XML string) |
 | Selectors path / Sanitize transformers                                                        | Nokogiri (unchanged)                                             |
 | `SST::Normalizer`                                                                             | Sole Nokogiri consumer on the heuristic auto-source path         |
 | Segmenter, Scoring, `Html::SstArticleExtractor`, heuristic chrome (`SST::Tags` / `SST::Text`) | SST only                                                         |
 
-Production heuristic scrapers should reuse one `SST::Document` memoized from `parsed_body` (or a shared Document passed in), not re-parse HTML strings.
+Production heuristic scrapers and the page check reuse one `SST::Document` and one page `Scoring::LinkResolver` memoized by `AutoSource`, not re-parse HTML strings.
 
 ## Constraints
 
@@ -51,6 +55,10 @@ Production heuristic scrapers should reuse one `SST::Document` memoized from `pa
 - **Top-K** — `Scoring::Engine::TOP_K` (99) caps ranked segments materialized into articles; list strategy also budgets `use_top_selectors`.
 - **Typed stages** — Pipeline stages take `SST::Document` / `Segment` / `RankedSegment`, not ad-hoc Hash bags. Internal scraper APIs may change; the public gem surface is `lib/html2rss.rb`.
 
+## Measuring changes
+
+Precision and recall changes are measured on the labeled corpus (`bin/auto-source-corpus`, ratchet spec, `make corpus-baseline`); see `CONTEXT.md` § Auto-source corpus. Wall/alloc budgets: `make perf-baseline` (auto-source), `make perf-baseline-selectors`, `make perf-baseline-feed` (native-feed parse, RSS/JSON Feed render, descriptions); `make perf-check` gates all three (`bin/lib/perf_baseline.rb` owns the 1.15x allocation rule). Revision-to-revision effect on saved real pages: `bin/compare-revisions`.
+
 ## Non-goals
 
 - Replacing Selectors, Schema, Microdata, MF2, JsonState, JsonDocument, or XhrArticles with SST.

data/lib/html2rss/auto_source/cleanup.rb

@@ -6,14 +6,16 @@ module Html2rss
     # Cleanup is responsible for cleaning up the extracted articles.
     # :reek:MissingSafeMethod { enabled: false }
     # It applies various strategies to filter and refine the article list.
-    # Sole producer of admission drop tallies for {Html2rss::Status}.
+    # Producer of the auto-source admission drop tallies for {Html2rss::Status}; {Html2rss::Selectors} tallies
+    # its own +missing_url+ drops.
     class Cleanup # rubocop:disable Metrics/ClassLength -- reject steps + tallies stay co-located
       # Default cleanup behavior for auto-sourced article lists.
       DEFAULT_CONFIG = {
-        keep_different_domain: false
+        keep_different_domain: false,
+        keep_sponsored: false
       }.freeze
 
-      # Minimum alphanumeric word count for present titles.
+      # Minimum word count ({WordCount}) for present titles.
       MIN_WORDS = 3
 
       # Allowed URL schemes for article filtering.
@@ -48,12 +50,14 @@ module Html2rss
         [:video_chrome, /\AVideo\s*[•·]/i],
         [:template, /\ACreated\s+from\s+Template\s+ID\b/i],
         [:template, /(\{\{[^}]+\}\}|%\{\w+\})/],
-        [:cta, /\Aread more\z/i],
-        [:cta, /\Alearn more\z/i],
         [:cta, /\Apdf\z/i]
       ].freeze
       private_constant :JUNK_TITLE_RULES
 
+      # Owner of the read-more vocabulary ("Read more", "More about X" in nine languages): a +:cta+ title.
+      TEXT_CLASSIFIER = LinkDestination::TextClassifier.new
+      private_constant :TEXT_CLASSIFIER
+
       # Minimum present-title length before short-title warnings.
       MIN_TITLE_LENGTH = 4
 
@@ -69,32 +73,56 @@ module Html2rss
         'auto_source.cleanup.keep_different_domain: true (or --keep-different-domain); ' \
         'allows published items to link to any domain supplied by the source'
 
+      # Config key / CLI flag named when sponsored drops emptied admission.
+      SPONSORED_HINT =
+        'auto_source.cleanup.keep_sponsored: true (or --keep-sponsored); ' \
+        'keeps items the page marks as sponsored'
+
+      ADMISSION_HINTS = { 'different_domain' => DIFFERENT_DOMAIN_HINT, 'sponsored' => SPONSORED_HINT }.freeze
+      private_constant :ADMISSION_HINTS
+
       class << self
-        # Mitigating config hint when admission drops emptied a non-empty capture.
+        # Mitigating config hints when admission drops emptied a non-empty capture.
         # Other drop reasons remain counts only.
         #
         # @param drop_tallies [Hash{String => Integer}]
         # @return [String, nil]
         def admission_hint(drop_tallies)
-          return unless drop_tallies.fetch('different_domain', 0).positive?
-
-          DIFFERENT_DOMAIN_HINT
+          hints = ADMISSION_HINTS.select { |reason, _| drop_tallies.fetch(reason, 0).positive? }.values
+          hints.join(' | ') unless hints.empty?
         end
 
         # @param articles [Array<Article>] extracted article candidates
         # @param url [Html2rss::Url] feed source URL used for same-host filtering
         # @param keep_different_domain [Boolean] when true, keep items whose URLs are on a
         #   different domain than +url+ (source-controlled off-domain links; default false)
+        # @param feed_url [Html2rss::Url, nil] the native feed the page advertises and that was read
+        #   ({Scraper::NativeFeed#feed_url}); that feed's own items may also link to its domain, or be their
+        #   own audio/video file on any domain
+        # @param keep_sponsored [Boolean] when true, keep items the page marks as sponsored
+        #   (default false)
+        # @param corroboration [Corroboration, nil] page check for structured-tier items, run
+        #   before dedupe so a dropped structured copy leaves the URL to a later tier
+        # @param page_links [#call, nil] lazy source of the page's {PageLinks}; called only when
+        #   a page-derived item is checked for sponsorship
         # @return [Result] cleaned articles and frozen drop tallies
-        def call(articles, url:, keep_different_domain: DEFAULT_CONFIG.fetch(:keep_different_domain)) # rubocop:disable Metrics/MethodLength -- ordered reject pipeline
+        # rubocop:disable-next Metrics/AbcSize, Metrics/MethodLength, Metrics/ParameterLists -- ordered reject pipeline
+        def call(articles, url:, keep_different_domain: DEFAULT_CONFIG.fetch(:keep_different_domain),
+                 keep_sponsored: DEFAULT_CONFIG.fetch(:keep_sponsored), corroboration: nil, page_links: nil,
+                 feed_url: nil)
           Log.debug "Cleanup: start with #{articles.size} articles"
           tallies = Hash.new(0)
 
           reject_invalid!(articles, tallies)
+          if corroboration
+            reject_uncorroborated!(articles, corroboration, tallies)
+            canonicalize_urls!(articles, corroboration)
+          end
           deduplicate_by_url!(articles, tallies)
+          reject_sponsored!(articles, page_links, tallies) unless keep_sponsored
           keep_only_http_urls!(articles, tallies)
           reject_self_links!(articles, url, tallies)
-          reject_different_domain!(articles, url, tallies) unless keep_different_domain
+          reject_different_domain!(articles, url, feed_url, tallies) unless keep_different_domain
           reject_excluded_destinations!(articles, tallies)
           reject_low_quality_titles!(articles, tallies)
 
@@ -112,7 +140,8 @@ module Html2rss
           normalized = normalize_title(title)
           return if normalized.empty?
 
-          JUNK_TITLE_RULES.find { |_, pattern| pattern.match?(normalized) }&.first
+          JUNK_TITLE_RULES.find { |_, pattern| pattern.match?(normalized) }&.first ||
+            (:cta if TEXT_CLASSIFIER.read_more?(normalized))
         end
 
         # @param items [Array<Html2rss::Article, RSS::Rss::Channel::Item>]
@@ -141,7 +170,7 @@ module Html2rss
         end
 
         def audit_url_diversity(items, metrics, violations)
-          unique_urls = items.filter_map { |item| url_identity(item_url(item)) }.uniq
+          unique_urls = items.filter_map { |item| item_url(item)&.article_identity }.uniq
           metrics[:unique_url_count] = unique_urls.size
           return [] unless items.size >= 2 && unique_urls.size < 2
 
@@ -171,7 +200,7 @@ module Html2rss
           if reason
             metrics[:junk_title_count] += 1
             violations[reason] += 1
-          elsif !word_count_at_least?(title, MIN_WORDS)
+          elsif !WordCount.at_least?(title, MIN_WORDS)
             metrics[:low_word_count] += 1
             violations[:low_word_count] += 1
           end
@@ -197,12 +226,48 @@ module Html2rss
           tally_reject!(articles, tallies, 'invalid') { |article| !article.valid? }
         end
 
+        # Tallies each non-keep verdict (+chrome_only+, +not_on_page+, +unlikely+) as its reason;
+        # the truthy tally count rejects the item.
+        def reject_uncorroborated!(articles, corroboration, tallies)
+          cohorts = articles.group_by(&:scraper)
+          articles.reject! do |article|
+            verdict = corroboration.verdict(article, cohort: cohorts.fetch(article.scraper))
+            next if verdict == :keep
+
+            tallies[verdict.to_s] += 1
+          end
+        end
+
+        # Kept structured items take the rendered anchor's URL, so dedupe merges them with the
+        # heuristic copy and the feed links the page's canonical form.
+        def canonicalize_urls!(articles, corroboration)
+          articles.map! do |article|
+            canonical = corroboration.canonical_url(article)
+            canonical ? article.with_url(canonical) : article
+          end
+        end
+
+        # One item per article identity: the first copy wins and later copies fill only the fields it
+        # lacks ({Article#merge}), so a native feed item keeps its date, guid and title and takes the
+        # page card's image. A read-more titled copy always loses to a copy with a real title: the
+        # link text is chrome, the other copy carries the headline.
         def deduplicate_by_url!(articles, tallies)
-          seen = {}
-          tally_reject!(articles, tallies, 'duplicate_url') do |article|
-            identity = url_identity(article.url)
-            identity.nil? || seen.key?(identity).tap { seen[identity] = true }
+          reject_read_more_copies!(articles, tallies)
+          kept = articles.each_with_object({}) do |article, by_identity|
+            identity = article.url.article_identity
+            by_identity[identity] = by_identity[identity]&.merge(article) || article
           end
+          folded = articles.size - kept.size
+          tallies['duplicate_url'] += folded if folded.positive?
+          articles.replace(kept.values)
+        end
+
+        # Items the page marks as sponsored ({PageLinks#sponsored?}), whatever their source: a paid
+        # post in the site's own feed is as sponsored as its card. Item text and URL paths are never read.
+        def reject_sponsored!(articles, page_links, tallies)
+          return unless page_links
+
+          tally_reject!(articles, tallies, 'sponsored') { |article| page_links.call.sponsored?(article.url) }
         end
 
         def keep_only_http_urls!(articles, tallies)
@@ -212,19 +277,32 @@ module Html2rss
         end
 
         def reject_self_links!(articles, base_url, tallies)
-          source_identity = url_identity(base_url)
+          source_identity = base_url.article_identity
           tally_reject!(articles, tallies, 'self_link') do |article|
-            url_identity(article.url) == source_identity
+            article.url.article_identity == source_identity
           end
         end
 
-        def reject_different_domain!(articles, base_url, tallies)
-          base_domain = base_url.domain
+        # Items link to the page's domain. A followed native feed's own items may also link to the feed's
+        # domain (a podcast platform's episode pages) or be their own audio/video file, which hosted podcasts
+        # serve from a third-party CDN; every other item keeps the rule.
+        def reject_different_domain!(articles, base_url, feed_url, tallies)
+          page_domain = base_url.domain
           tally_reject!(articles, tallies, 'different_domain') do |article|
-            article.url&.domain != base_domain
+            article.url.domain != page_domain && !followed_feed_exempt?(article, feed_url)
           end
         end
 
+        # A followed feed's item is exempt when it links to the feed's domain, or when its URL is its own
+        # audio/video file on any domain, whether the feed linked that file or the item is a link-less episode
+        # that took the file as its URL ({Syndication::Parser#article}).
+        def followed_feed_exempt?(article, feed_url)
+          return false unless feed_url && article.scraper == Scraper::NativeFeed
+
+          article.url.domain == feed_url.domain ||
+            article.enclosures.any? { |enclosure| enclosure.media? && enclosure.url == article.url }
+        end
+
         # Hard-exclude non-article destination classes (commerce/affiliate/utility chrome).
         # PathClassifier owns route facts; Cleanup owns feed-item admission.
         def reject_excluded_destinations!(articles, tallies)
@@ -240,26 +318,37 @@ module Html2rss
           facts.utility_path && !facts.content_path && !facts.strong_post_suffix
         end
 
-        # Keep missing titles (nil provenance). Drop present junk/unnatural titles —
-        # blanking them would hide bad extraction as "unknown" and inflate empty items.
-        def reject_low_quality_titles!(articles, tallies) # rubocop:disable Metrics/MethodLength -- junk vs word-count reasons
+        # Drops untitled page-derived items and junk/unnatural titles — blanking a bad title would
+        # hide bad extraction as "unknown" and admit empty items. A publisher-declared item may be
+        # untitled (microblog feed posts) and is kept. {Article} collapses a blank title to nil.
+        def reject_low_quality_titles!(articles, tallies)
           articles.reject! do |article|
-            title = article.title
-            next false if title.nil?
-
-            reason = junk_reason(title)
-            if reason
-              tallies[reason.to_s] += 1
-              next true
-            end
+            reason = title_reject_reason(article)
+            tallies[reason] += 1 if reason
+            reason
+          end
+        end
 
-            next false if word_count_at_least?(title, MIN_WORDS)
+        def title_reject_reason(article)
+          title = article.title
+          if title.nil?
+            'blank_title' if Scraper::PAGE_SCRAPERS.include?(article.scraper)
+          else
+            junk_reason(title)&.to_s || ('low_word_count' unless WordCount.at_least?(title, MIN_WORDS))
+          end
+        end
 
-            tallies['low_word_count'] += 1
-            true
+        def reject_read_more_copies!(articles, tallies)
+          titled = articles.reject { |article| read_more_title?(article) }.to_set { |a| a.url.article_identity }
+          tally_reject!(articles, tallies, 'duplicate_url') do |article|
+            read_more_title?(article) && titled.include?(article.url.article_identity)
           end
         end
 
+        def read_more_title?(article)
+          article.title && junk_reason(article.title) == :cta
+        end
+
         def tally_reject!(articles, tallies, reason)
           articles.reject! do |article|
             next false unless yield(article)
@@ -269,10 +358,6 @@ module Html2rss
           end
         end
 
-        def url_identity(url)
-          url&.without_fragment&.to_s
-        end
-
         def feed_item_as_article(item)
           case item
           when Article then item
@@ -291,15 +376,6 @@ module Html2rss
         def normalize_title(title)
           title.to_s.strip.gsub(/\s+/, ' ')
         end
-
-        def word_count_at_least?(str, min_words)
-          count = 0
-          str.to_s.scan(/\p{Alnum}+/) do
-            count += 1
-            return true if count >= min_words
-          end
-          false
-        end
       end
     end
   end

data/lib/html2rss/auto_source/corroboration.rb

@@ -0,0 +1,167 @@
+# frozen_string_literal: true
+
+module Html2rss
+  class AutoSource
+    ##
+    # Keep/drop verdict for structured-tier items, checked against the rendered page.
+    #
+    # Only {Scraper::CORROBORATED_SCRAPERS} items are checked; publisher-declared sources
+    # (native feed, CMS REST API, sitemap, oEmbed) are kept as-is. A URL rendered as a
+    # content anchor is kept. On a page that can corroborate, a URL rendered only in chrome
+    # is +:chrome_only+. A URL rendered nowhere is +:not_on_page+ when the page renders its
+    # cohort (at least half the cohort as content anchors, or a cohort too small to tell);
+    # an item of a cohort the page mostly does not render (an app shell, an aggregator linking
+    # through redirects) is scored with absence counting against it. A peripheral URL (aside,
+    # menu landmark), or any URL on a sparse page or without a page ({PageLinks#sparse?}),
+    # is decided by a private article-likeness score: +:keep+ or +:unlikely+.
+    #
+    # A kept item takes the rendered anchor's URL ({#canonical_url}); {Cleanup} applies it.
+    #
+    # {Cleanup} owns admission and tallies the drop verdicts.
+    class Corroboration
+      # Article-likeness score at or above which an uncorroborated item is kept.
+      LIKELY_SCORE = 2
+      # Fewest items in a scraper's cohort before its URL cohesion or render share counts.
+      MIN_COHORT_SIZE = 3
+
+      ##
+      # @yieldreturn [PageLinks] the page's links; called at most once, on the first checked item
+      def initialize(&page_links)
+        @page_links_source = page_links
+        @cohesion = {}.compare_by_identity
+        @rendered = {}.compare_by_identity
+      end
+
+      ##
+      # @param article [Html2rss::Article] item with a URL (Cleanup checks validity first)
+      # @param cohort [Array<Html2rss::Article>] the items produced by +article+'s scraper
+      # @return [Symbol] +:keep+, +:chrome_only+, +:not_on_page+, or +:unlikely+
+      def verdict(article, cohort:)
+        return :keep unless Scraper::CORROBORATED_SCRAPERS.include?(article.scraper)
+
+        case page_links.placement(article.url)
+        when :content then :keep
+        when :chrome then uncorroborated(:chrome_only, article, cohort)
+        when :absent then unrendered(article, cohort)
+        else scored(article, cohort)
+        end
+      end
+
+      ##
+      # Whether the page shows an item in its content. Only checked items can fail: one kept by
+      # score (rendered in chrome, a peripheral landmark, or nowhere) is not shown. Exempt items
+      # (publisher-declared, or read off the page by a heuristic tier) are taken as shown.
+      #
+      # @param article [Html2rss::Article] admitted item
+      # @return [Boolean]
+      def shown?(article)
+        !Scraper::CORROBORATED_SCRAPERS.include?(article.scraper) || page_links.placement(article.url) == :content
+      end
+
+      ##
+      # Whether the page is a media page (a podcast or video listing): more than half of the
+      # items read off the page ({Scraper::PAGE_SCRAPERS}) and shown in its content ({#shown?})
+      # are media items ({Html2rss::Article#media?}). Each article identity is one item: a
+      # structured media item and the card that renders it count once.
+      #
+      # @param articles [Array<Html2rss::Article>] admitted items
+      # @return [Boolean]
+      def media_page?(articles)
+        cards = articles.select { Scraper::PAGE_SCRAPERS.include?(_1.scraper) && shown?(_1) }
+                        .uniq { _1.url.article_identity }
+        cards.count(&:media?) * 2 > cards.size
+      end
+
+      ##
+      # A media page's order: its media items first, then the other items, each group in document
+      # order (the position of its content anchor), with items the page does not show in its
+      # content last in their given order.
+      #
+      # @param articles [Array<Html2rss::Article>]
+      # @return [Array<Html2rss::Article>]
+      def page_order(articles)
+        articles.partition(&:media?).flat_map do |group|
+          placed, unplaced = group.partition { page_links.content_position(_1.url) }
+          placed.sort_by { page_links.content_position(_1.url) } + unplaced
+        end
+      end
+
+      ##
+      # The page's own form of a checked item's URL, when it differs only by scheme, tracking
+      # params, trailing slash, or fragment from a rendered content anchor.
+      #
+      # @param article [Html2rss::Article]
+      # @return [Html2rss::Url, nil] nil when the item is exempt, not a content anchor, or already canonical
+      def canonical_url(article)
+        return unless Scraper::CORROBORATED_SCRAPERS.include?(article.scraper)
+
+        canonical = page_links.canonical_url(article.url)
+        canonical unless canonical.to_s == article.url.to_s
+      end
+
+      private
+
+      # A sparse page cannot show where a URL is missing from, so the score decides there.
+      def uncorroborated(reason, article, cohort)
+        page_links.sparse? ? scored(article, cohort) : reason
+      end
+
+      # Absence is evidence only against a cohort the page renders: a page showing most of a
+      # list left the rest out. A page that renders little of the cohort shows the list
+      # elsewhere (client-side, through redirects), so there absence only lowers the score.
+      def unrendered(article, cohort)
+        return uncorroborated(:not_on_page, article, cohort) if page_links.sparse? || rendered?(cohort)
+
+        scored(article, cohort, absent: true)
+      end
+
+      def scored(article, cohort, absent: false)
+        likely?(article, cohort, absent:) ? :keep : :unlikely
+      end
+
+      def page_links
+        @page_links ||= @page_links_source.call
+      end
+
+      def likely?(article, cohort, absent:)
+        score = [article.published_at, article.image, article.description].count(&:itself)
+        score += url_shape(article.url)
+        score += 1 if cohesive?(cohort)
+        score -= 1 if absent
+        score >= LIKELY_SCORE
+      end
+
+      # +1 for an article-shaped path, -1 for a section root, 0 otherwise. A shallow path
+      # ending in a post slug (+/harbor-ferry-route-returns+) is a root-level permalink, not a
+      # section root.
+      def url_shape(url)
+        facts = LinkDestination::DestinationFacts.build(url)
+        return 1 if facts.content_path || facts.strong_post_suffix
+        return 0 unless facts.shallow
+
+        facts.segments.last&.match?(LinkDestination::PathClassifier::POST_SLUG_SEGMENT) ? 0 : -1
+      end
+
+      # A cohort is cohesive when at least half its URLs share one parent path.
+      def cohesive?(cohort)
+        return false if cohort.size < MIN_COHORT_SIZE
+
+        @cohesion.fetch(cohort) do
+          modal = cohort.map { _1.url.path_segments[0...-1] }.tally.values.max
+          @cohesion[cohort] = modal * 2 >= cohort.size
+        end
+      end
+
+      # A cohort is rendered when at least half its URLs are content anchors, or when it is
+      # too small for its share to tell a rendered list from an unrendered one.
+      def rendered?(cohort)
+        return true if cohort.size < MIN_COHORT_SIZE
+
+        @rendered.fetch(cohort) do
+          content = cohort.count { page_links.placement(_1.url) == :content }
+          @rendered[cohort] = content * 2 >= cohort.size
+        end
+      end
+    end
+  end
+end

data/lib/html2rss/auto_source/page_links/anchor_claims.rb

@@ -0,0 +1,82 @@
+# frozen_string_literal: true
+
+module Html2rss
+  class AutoSource
+    class PageLinks
+      ##
+      # The anchors of one sponsorship walk, in document order, and which of them sit in a marked scope.
+      # Destinations resolve only when a scope asks for them. A claimed destination is sponsored unless
+      # some anchor to it stays unclaimed: a promo slot never hides the editorial card linking the story.
+      class AnchorClaims
+        # @param link_resolver [Scoring::LinkResolver]
+        def initialize(link_resolver)
+          @link_resolver = link_resolver
+          @anchors = []
+          @headline = []
+          @claimed = []
+          @keys = Set.new
+        end
+
+        # @return [Integer] anchors recorded so far; the next anchor's index
+        def size = @anchors.size
+
+        ##
+        # @param anchor [SST::Node]
+        # @param in_heading [Boolean] whether the anchor sits inside a heading
+        # @return [Integer] the anchor's index
+        def record(anchor, in_heading:)
+          @headline << in_heading
+          @anchors << anchor
+          @anchors.size - 1
+        end
+
+        # @param index [Integer]
+        # @return [Array, nil] article identity of the anchor's destination
+        def key_at(index) = key_of(@anchors.fetch(index))
+
+        # @param anchor [SST::Node]
+        # @return [Array, nil] article identity of the anchor's destination
+        def key_of(anchor) = @link_resolver.destination_facts(anchor)&.url&.article_identity
+
+        # @param range [Range] anchor indexes
+        # @return [Array<Array>] distinct destinations in +range+
+        def keys_in(range) = range.filter_map { key_at(_1) }.uniq
+
+        ##
+        # The one destination a card in +range+ stands for: its only one, else its only headline one (an
+        # anchor in or around a heading).
+        #
+        # @param range [Range] anchor indexes
+        # @return [Array, nil]
+        def dominant_key(range)
+          keys = keys_in(range)
+          return keys.first if keys.one?
+
+          headline_keys = keys_in(range.select { headline?(_1) })
+          headline_keys.first if headline_keys.one?
+        end
+
+        # Marks +keys+ sponsored and every anchor in +range+ as inside a marked scope.
+        #
+        # @param range [Range] anchor indexes
+        # @param keys [Array<Array>] destinations to mark; every one in +range+ by default
+        # @return [void]
+        def claim(range, keys = keys_in(range))
+          @keys.merge(keys)
+          @claimed.fill(true, range)
+        end
+
+        # @return [Set<Array>] claimed destinations, less those also linked outside every claimed range
+        def sponsored_keys
+          @anchors.each_index { @keys.delete(key_at(_1)) unless @claimed[_1] } unless @keys.empty?
+          @keys
+        end
+
+        private
+
+        def headline?(index) = @headline.fetch(index) || @anchors.fetch(index).children.any? { _1.find(&:heading?) }
+      end
+      private_constant :AnchorClaims
+    end
+  end
+end