html2rss 0.29.1 → 0.31.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 32c090eccd93210c9631778ff50ec7d32d36c7eccaa711cdd1a0dd5c3ea53616
-  data.tar.gz: 93e4d2f253a019d7de585c8acaaa8d06143accb4f3a43a4f8d13f9c96d838390
+  metadata.gz: e3450f1c6b664e4f9c7377220b65640a9e4bd640782d4f8831538c65a172cec3
+  data.tar.gz: 1f5543f86a3389a4f2abb9a76dec44ba41a4ffecadcf8d3d674ff084c459aaec
 SHA512:
-  metadata.gz: 93956c4dc3d35e06c81dff939d4ea5184da93834a0fb48f1b3dbd4dea21d615177ad45fa34e4f16abf396a184cc18acbe31c354df6764ba858ec340c78a3b08c
-  data.tar.gz: b906784ed2ae53b80aa2fc00fe813550befcfac723d97346ffb87cff7f01ca6b0f5db0a6e4a2430dd10485c20f6d4b251a54a1b9e0462998f2ab628c0123a809
+  metadata.gz: 6b2d0fc1afbd44a13405eb21ad89ea32dbdf8d3c0dd1dd0f6430f877398a2cb5275781de7d65386fbde71f43db34e35a54c1fb5d13a73a265c8a8a274f496f87
+  data.tar.gz: 6402bc203d89b23fc388872b310eb4c6aaa6b9423025a40ec6eaa606ecf5c78a64c09823a103b263344b72acae4fa385e11526f59571b699eaa54ec27379cf66

data/README.md

@@ -13,7 +13,7 @@ Most people looking for a first working feed should start with `html2rss-web`, r
 Detailed usage guides, reference docs, and the feed directory live on the project website:
 
 - [Ruby gem documentation](https://html2rss.github.io/ruby-gem)
-- [Request strategies](https://html2rss.github.io/ruby-gem/reference/strategy) (`auto` = `faraday` → `botasaurus`; or pin concrete strategies)
+- [Request strategies](https://html2rss.github.io/ruby-gem/reference/strategy) (`auto` = `default` → `botasaurus`; or pin concrete strategies)
 - [Selectors & pagination](https://html2rss.github.io/ruby-gem/reference/selectors#paginated-feeds)
 - [Web application](https://html2rss.github.io/web-application)
 - [Feed directory](https://html2rss.github.io/feed-directory)
@@ -26,7 +26,7 @@ Cloud development: [Open in GitHub Codespaces](https://github.com/codespaces/new
 ## Architecture
 
 1. **Config** — loads and validates configuration (YAML/hash); schema via `html2rss schema` / `schema/html2rss-config.schema.json`
-2. **RequestService** — fetches pages (`faraday`, `botasaurus`, or `local_file`)
+2. **RequestService** — fetches pages (`default` (HTTPX), `botasaurus`, or `local_file`)
 3. **Selectors** — extracts content via CSS selectors with extractors/post-processors
 4. **AutoSource** — auto-detects content (Schema.org, JSON state, semantic HTML, structural patterns)
 5. **FeedBuilder** — assembles Article objects and renders feeds (RSS 2.0 / JSON Feed 1.1)
@@ -82,7 +82,7 @@ Historic CLI aliases: `feed` → `apply`, `auto` → `scrape`.
 
 Inspect follows redirects and reports the landing URL in `final_url`. CLI text shows a `Final:` line **only when** the landing URL differs from what you typed — that line means the redirect succeeded, not that inspect stopped early.
 
-Cross-host redirects (e.g. `https://apex.example/` → `https://www.example/`) set `Host` per hop via Faraday/Net::HTTP; html2rss does not pin the entry hostname. When `final_url` differs and status is 4xx, retry on `final_url` or pass the site's canonical hostname. Details: [`lib/html2rss/page_recon/README.md`](lib/html2rss/page_recon/README.md).
+Cross-host redirects (e.g. `https://apex.example/` → `https://www.example/`) set `Host` per hop via HTTPX; html2rss does not pin the entry hostname. When `final_url` differs and status is 4xx, retry on `final_url` or pass the site's canonical hostname. Details: [`lib/html2rss/page_recon/README.md`](lib/html2rss/page_recon/README.md).
 
 ## Capture API
 
@@ -112,7 +112,7 @@ stdio uses stdout for JSON-RPC, so the daemon logs to **stderr**. It defaults to
 
 HTTP transport needs `rack`, `rackup`, and `webrick` (declared gem dependencies). It listens on `127.0.0.1` only; do not expose it on a public interface without your own auth and Host/Origin controls.
 
-**Strategy note:** MCP `scrape` / `capture` with `strategy: "auto"` run Faraday → Botasaurus AutoFallback. `inspect` uses Faraday when `auto` (cheap diagnostic); pin `botasaurus` when you need browser rendering for inspect.
+**Strategy note:** MCP `scrape` / `capture` with `strategy: "auto"` run default (HTTPX) → Botasaurus AutoFallback. `inspect` uses default (HTTPX) when `auto` (cheap diagnostic); pin `botasaurus` when you need browser rendering for inspect.
 
 **Tool-call budget:** `scrape` is 1 call (auto already hops). Durable config is `capture` → `test` → `apply` (or `validate` → `test` → `apply` when you already have YAML). Call `inspect` only when scrape/capture is weak or you need recon (final URL, status, https→http, native RSS/Atom).
 
@@ -157,7 +157,7 @@ Module guide: [`lib/html2rss/mcp/README.md`](lib/html2rss/mcp/README.md).
 | ----------------------- | --------------------------------------------------------------- |
 | `html2rss://schema`     | Full JSON Schema for feed configurations                        |
 | `html2rss://extractors` | Registered extractor **names** (options live in schema `$defs`) |
-| `html2rss://strategies` | Published MCP strategies (`auto`, `faraday`, `botasaurus`)      |
+| `html2rss://strategies` | Published MCP strategies (`auto`, `default`, `httpx`, `botasaurus`)      |
 | `html2rss://runtime`    | `version`, `mcp_contract_version`, `catalog_fingerprint`, `tools`, `botasaurus_configured` (never the scraper URL) |
 
 ### Prompts
@@ -183,11 +183,11 @@ Set `BOTASAURUS_SCRAPER_URL` to `http://127.0.0.1:4010` and use strategy `botasa
 
 | Strategy     | Description                                                                 |
 | ------------ | --------------------------------------------------------------------------- |
-| `auto`       | Tries `faraday`, falls back to `botasaurus` (default in gem/CLI/MCP scrape) |
-| `faraday`    | Plain HTTP requests via Faraday                                             |
+| `auto`       | Tries `default`, falls back to `botasaurus` (default in gem/CLI/MCP scrape) |
+| `default`    | Plain HTTP requests via HTTPX (alias: `httpx`; legacy: `faraday`)           |
 | `botasaurus` | Puppeteer-backed scraping for JavaScript pages                              |
 
-`inspect` keeps Faraday when `auto` for cheap diagnostics. Elsewhere, strategy can be set via CLI (`--strategy`), gem API keyword argument, or feed config `request.strategy`. See the [request strategies docs](https://html2rss.github.io/ruby-gem/reference/strategy) for more details.
+`inspect` keeps `default` when `auto` for cheap diagnostics. Elsewhere, strategy can be set via CLI (`--strategy`), gem API keyword argument, or feed config `strategy`. See the [request strategies docs](https://html2rss.github.io/ruby-gem/reference/strategy) for more details.
 
 ## License
 

data/html2rss.gemspec

@@ -35,9 +35,7 @@ Gem::Specification.new do |spec|
   spec.add_dependency 'addressable', '~> 2.7'
   spec.add_dependency 'brotli'
   spec.add_dependency 'dry-validation'
-  spec.add_dependency 'faraday', '> 2.0.1', '< 3.0'
-  spec.add_dependency 'faraday-follow_redirects'
-  spec.add_dependency 'faraday-gzip', '~> 3'
+  spec.add_dependency 'httpx', '~> 1.8'
   spec.add_dependency 'kramdown'
   spec.add_dependency 'mcp', '~> 1.2'
   spec.add_dependency 'mime-types', '> 3.0'

data/lib/html2rss/article/deduplicator.rb

@@ -1,7 +1,5 @@
 # frozen_string_literal: true
 
-require 'set' # rubocop:disable Lint/RedundantRequireStatement
-
 module Html2rss
   class Article
     ##
@@ -40,9 +38,9 @@ module Html2rss
       attr_reader :articles
 
       def deduplication_fingerprint_for(article)
-        return unless article.respond_to?(:deduplication_fingerprint)
-
         article.deduplication_fingerprint
+      rescue NoMethodError
+        nil
       end
     end
   end

data/lib/html2rss/article.rb

@@ -5,14 +5,9 @@ require 'zlib'
 module Html2rss
   ##
   # Article is a simple data object representing an article extracted from a page.
-  # It is enumerable and responds to all keys specified in PROVIDED_KEYS.
   #
   # Description and enclosure wire presentation live in {FeedBuilder::ItemPresentation}.
-  # rubocop:disable-next Metrics/ClassLength -- value object retains Marshal + defensive freeze helpers
   class Article
-    include Enumerable
-    include Comparable
-
     # Allowed article attributes accepted by the value object constructor.
     PROVIDED_KEYS = %i[id title description url image author guid published_at enclosures categories scraper].freeze
     # Separator used to build deterministic deduplication fingerprints.
@@ -23,26 +18,24 @@ module Html2rss
     # when attributes are lazily/conditionally accessed in different sequences.
     NOT_SET = Object.new.freeze
 
-    # @param options [Hash{Symbol => String}]
-    # @option options [String] :id stable article identifier
-    # @option options [String] :title article title
-    # @option options [String] :description raw extracted description/content (not feed-rendered HTML)
-    # @option options [String, Html2rss::Url] :url canonical article URL
-    # @option options [String, Html2rss::Url] :image image URL for description / JSON Feed +image+
-    # @option options [String] :author author name
-    # @option options [String] :guid explicit GUID override
-    # @option options [String, Time, DateTime] :published_at publication timestamp
-    # @option options [Array<Hash{Symbol => Object}>] :enclosures enclosure attribute hashes
-    # @option options [Array<String>] :categories category labels
-    # @option options [Class] :scraper scraper class that produced the article
-    def initialize(**options)
-      @to_h = options.each_with_object({}) { |(key, value), hash| hash[key] = freeze_option(value) }.freeze
+    # @param id [String, nil] stable article identifier
+    # @param title [String, nil] article title
+    # @param description [String, nil] raw extracted description/content (not feed-rendered HTML)
+    # @param url [String, Html2rss::Url, nil] canonical article URL
+    # @param image [String, Html2rss::Url, nil] image URL for description / JSON Feed +image+
+    # @param author [String, nil] author name
+    # @param guid [String, Array, nil] explicit GUID override
+    # @param published_at [String, Time, DateTime, nil] publication timestamp
+    # @param enclosures [Array<Hash{Symbol => Object}>, nil] enclosure attribute hashes
+    # @param categories [Array<String>, nil] category labels
+    # @param scraper [Class, nil] scraper class that produced the article
+    # rubocop:disable-next Metrics/ParameterLists -- eleven article keys stay explicit
+    def initialize(id: nil, title: nil, description: nil, url: nil, image: nil, author: nil, guid: nil,
+                   published_at: nil, enclosures: nil, categories: nil, scraper: nil)
+      @to_h = { id:, title:, description:, url:, image:, author:, guid:, published_at:, enclosures:, categories:,
+                scraper: }.compact.transform_values { freeze_option(_1) }.freeze
 
       @url = @image = @guid = @enclosures = @categories = @published_at = NOT_SET
-
-      return unless (unknown_keys = options.keys - PROVIDED_KEYS).any?
-
-      Log.warn "Article: unknown keys found: #{unknown_keys.join(', ')}"
     end
 
     # Checks if the article is valid based on the presence of URL, ID, and either title or description.
@@ -51,14 +44,6 @@ module Html2rss
       !url.to_s.empty? && (!title.to_s.empty? || !description.to_s.empty?) && !id.to_s.empty?
     end
 
-    # @yield [key, value]
-    # @return [Enumerator] if no block is given
-    def each
-      return enum_for(:each) unless block_given?
-
-      PROVIDED_KEYS.each { |key| yield(key, public_send(key)) }
-    end
-
     # @return [String, nil] stable article identifier
     def id = blank_string_to_nil(@to_h[:id])
 
@@ -139,14 +124,6 @@ module Html2rss
       @to_h[:scraper]
     end
 
-    # @param other [Object] value compared against this article
-    # @return [Integer, nil] comparison result for compatible Article values
-    def <=>(other)
-      return nil unless other.is_a?(Article)
-
-      0 if other.all? { |key, value| value == public_send(key) ? public_send(key) <=> value : false }
-    end
-
     private
 
     ##

data/lib/html2rss/auto_source/README.md

@@ -28,7 +28,7 @@ Entry: `FeedPipeline` → `AutoSource#articles` → `Scraper.build_instance` →
 
 6. **Entry URL resolution** (pipeline, not this class) — when AutoFallback sees a weak homepage extract, `FeedResolution` may rewrite the scrape URL to a listing/feed before escalating strategies. See {Html2rss::FeedPipeline} and {Html2rss::FeedResolution}.
 
-Segmenter strategies: `:semantic` (leaf containers + primary link), `:list` (repeated tag paths), `:cluster` (class/structure grids for anchorless cards). Scoring ranks and demotes; `LinkDestination::NoisePolicy` owns content-anchor eligibility. Cleanup owns feed-item admission.
+Segmenter strategies: `:semantic` (leaf containers + primary link), `:list` (cards grouped by parent; `main` is a filter), `:cluster` (class/structure grids for anchorless cards). Scoring ranks and demotes; `LinkDestination::NoisePolicy` owns content-anchor eligibility. Cleanup owns feed-item admission.
 
 ## Nokogiri vs SST boundaries
 

data/lib/html2rss/auto_source/cleanup.rb

@@ -17,7 +17,7 @@ module Html2rss
       MIN_WORDS = 3
 
       # Allowed URL schemes for article filtering.
-      VALID_SCHEMES = %w[http https].to_set.freeze
+      VALID_SCHEMES = Set['http', 'https'].freeze
 
       # Photo-credit agencies (single list → junk title regexes).
       CREDIT_AGENCIES = [
@@ -97,13 +97,14 @@ module Html2rss
           JUNK_TITLE_RULES.find { |_, pattern| pattern.match?(normalized) }&.first
         end
 
-        # @param items [Array] RSS or article-like objects with title and link/url
+        # @param items [Array<Html2rss::Article, RSS::Rss::Channel::Item>]
         # @return [AuditResult]
         def audit_feed_items(items)
-          metrics = initial_audit_metrics(items.size)
+          articles = items.map { |item| feed_item_as_article(item) }
+          metrics = initial_audit_metrics(articles.size)
           violations = Hash.new(0)
-          warnings = audit_url_diversity(items, metrics, violations)
-          audit_item_titles(items, metrics, violations)
+          warnings = audit_url_diversity(articles, metrics, violations)
+          audit_item_titles(articles, metrics, violations)
           warnings = finalize_audit_warnings(warnings, metrics)
           log_audit(metrics, warnings, violations)
           AuditResult.new(warnings:, metrics: metrics.freeze, violations: violations.freeze)
@@ -256,8 +257,16 @@ module Html2rss
           url&.without_fragment&.to_s
         end
 
+        def feed_item_as_article(item)
+          case item
+          when Article then item
+          else
+            Article.new(title: item.title, url: item.link)
+          end
+        end
+
         def item_url(item)
-          raw = item.respond_to?(:link) ? item.link : item.url
+          raw = item.url
           raw.nil? || raw.to_s.empty? ? nil : Html2rss::Url.from_absolute(raw.to_s)
         rescue ArgumentError
           nil

data/lib/html2rss/auto_source/scraper/html.rb

@@ -22,6 +22,11 @@ module Html2rss
         # @return [Symbol]
         def self.options_key = :html
 
+        ##
+        # @param _opts [Hash] unused options
+        # @return [Integer]
+        def self.request_slots(_opts = {}) = 0
+
         ##
         # @param parsed_body [Nokogiri::HTML::Document]
         # @return [Boolean]

data/lib/html2rss/auto_source/scraper/json_state.rb

@@ -14,6 +14,11 @@ module Html2rss
         # @return [Symbol] scraper config key
         def self.options_key = :json_state
 
+        ##
+        # @param _opts [Hash] unused options
+        # @return [Integer]
+        def self.request_slots(_opts = {}) = 0
+
         class << self
           # @param parsed_body [Nokogiri::HTML::Document, nil] parsed HTML document
           def articles?(parsed_body)

data/lib/html2rss/auto_source/scraper/meta_oembed.rb

@@ -8,7 +8,7 @@ module Html2rss
     module Scraper
       ##
       # Scrapes OpenGraph meta tags and oEmbed JSON descriptors from HTML pages.
-      class MetaOembed
+      class MetaOembed # rubocop:disable Metrics/ClassLength -- request_slots + extractable? stay on the scraper
         include Enumerable
 
         # Selector for OpenGraph meta tags.
@@ -30,6 +30,10 @@ module Html2rss
         # @return [Symbol] scraper config key
         def self.options_key = :meta_oembed
 
+        # @param _opts [Hash] unused options
+        # @return [Integer]
+        def self.request_slots(_opts = {}) = 0
+
         class << self
           # @param parsed_body [Nokogiri::HTML::Document, nil] parsed HTML document
           # @return [Boolean] whether OpenGraph tags or oEmbed link exist
@@ -57,6 +61,9 @@ module Html2rss
           @request_session = request_session
         end
 
+        # @return [Boolean]
+        def extractable? = self.class.articles?(parsed_body)
+
         ##
         # Yields normalized article hash extracted from OpenGraph meta tags and optional oEmbed data.
         #

data/lib/html2rss/auto_source/scraper/microdata.rb

@@ -17,6 +17,11 @@ module Html2rss
         # @return [Symbol] scraper config key
         def self.options_key = :microdata
 
+        ##
+        # @param _opts [Hash] unused options
+        # @return [Integer]
+        def self.request_slots(_opts = {}) = 0
+
         class << self
           # @param parsed_body [Nokogiri::HTML::Document, nil] parsed HTML document
           def articles?(parsed_body)
@@ -81,6 +86,9 @@ module Html2rss
           @url = url
         end
 
+        # @return [Boolean]
+        def extractable? = self.class.articles?(parsed_body)
+
         ##
         # Iterates over normalized article hashes extracted from supported Microdata roots.
         #

data/lib/html2rss/auto_source/scraper/microformats2.rb

@@ -18,6 +18,11 @@ module Html2rss
         # @return [Symbol] scraper config key
         def self.options_key = :microformats2
 
+        ##
+        # @param _opts [Hash] unused options
+        # @return [Integer]
+        def self.request_slots(_opts = {}) = 0
+
         class << self
           # @param parsed_body [Nokogiri::HTML::Document, nil] parsed HTML document
           # @return [Boolean] whether Microformats2 h-entry elements exist
@@ -38,6 +43,9 @@ module Html2rss
           @url = Html2rss::Url.from_absolute(url)
         end
 
+        # @return [Boolean]
+        def extractable? = self.class.articles?(parsed_body)
+
         ##
         # Yields normalized article hashes extracted from h-entry elements.
         #

data/lib/html2rss/auto_source/scraper/native_feed.rb

@@ -51,6 +51,9 @@ module Html2rss
           @request_session = request_session
         end
 
+        # @return [Boolean]
+        def extractable? = self.class.articles?(parsed_body)
+
         ##
         # @yieldparam article [Hash{Symbol => Object}]
         # @return [Enumerator, void]

data/lib/html2rss/auto_source/scraper/schema/thing.rb

@@ -19,9 +19,6 @@ module Html2rss
             ReviewNewsArticle SatiricalArticle ScholarlyArticle SocialMediaPosting TechArticle
           ].to_set.freeze
 
-          # Attributes exposed by `#call` in generated article hashes.
-          DEFAULT_ATTRIBUTES = %i[id title description author url image published_at categories].freeze
-
           # @param schema_object [Hash{Symbol => Object}] parsed schema.org object
           # @param url [String, Html2rss::Url, nil] base URL used for relative normalization
           def initialize(schema_object, url:)
@@ -29,8 +26,10 @@ module Html2rss
             @base_url = normalized_base_url(url)
           end
 
-          # @return [Hash] the scraped article hash with DEFAULT_ATTRIBUTES
-          def call = DEFAULT_ATTRIBUTES.to_h { [_1, public_send(_1)] }
+          # @return [Hash{Symbol => Object}] scraped article attributes
+          def call
+            { id:, title:, description:, author:, url:, image:, published_at:, categories: }
+          end
 
           # @return [String, nil] stable schema object identifier
           def id

data/lib/html2rss/auto_source/scraper/schema.rb

@@ -46,6 +46,11 @@ module Html2rss
         # @return [Symbol] scraper config key
         def self.options_key = :schema
 
+        ##
+        # @param _opts [Hash] unused options
+        # @return [Integer]
+        def self.request_slots(_opts = {}) = 0
+
         class << self
           # @param parsed_body [Nokogiri::HTML::Document] parsed HTML document
           # @return [Boolean] whether the page includes supported schema types
@@ -166,6 +171,9 @@ module Html2rss
           @opts = opts
         end
 
+        # @return [Boolean]
+        def extractable? = self.class.articles?(parsed_body)
+
         ##
         # @yield [Hash] Each scraped article_hash
         # @return [Array<Hash>] the scraped article_hashes

data/lib/html2rss/auto_source/scraper/semantic_html.rb

@@ -15,6 +15,11 @@ module Html2rss
         # @return [Symbol] config key used to enable or configure this scraper
         def self.options_key = :semantic_html
 
+        ##
+        # @param _opts [Hash] unused options
+        # @return [Integer]
+        def self.request_slots(_opts = {}) = 0
+
         # @param parsed_body [Nokogiri::HTML::Document] parsed HTML document
         # @return [Boolean] true when at least one semantic container has an eligible anchor
         def self.articles?(parsed_body)

data/lib/html2rss/auto_source/scraper/sitemap.rb

@@ -54,6 +54,9 @@ module Html2rss
           @max_age_days = opts.fetch(:max_age_days, Scraper::Sitemap::Parser::DEFAULT_MAX_AGE_DAYS)
         end
 
+        # @return [Boolean]
+        def extractable? = self.class.articles?(parsed_body)
+
         ##
         # Yields article hashes from the sitemap.
         #

data/lib/html2rss/auto_source/scraper/wordpress_api.rb

@@ -56,6 +56,9 @@ module Html2rss
           @page_scope = PageScope.from(parsed_body:, url: @url)
         end
 
+        # @return [Boolean]
+        def extractable? = self.class.articles?(parsed_body)
+
         ##
         # Yields article hashes from the WordPress posts API.
         #

data/lib/html2rss/auto_source/scraper.rb

@@ -36,11 +36,11 @@ module Html2rss
       SCRAPERS = SCRAPER_TIERS.flatten.freeze
 
       # Heuristic scrapers that share one memoized SST::Document per page.
-      HEURISTIC_SCRAPERS = [SemanticHtml, Html].freeze
+      HEURISTIC_SCRAPERS = Set[SemanticHtml, Html].freeze
       # Scrapers that accept a shared follow-up +request_session+.
-      REQUEST_SESSION_SCRAPERS = [NativeFeed, WordpressApi, Sitemap, MetaOembed].freeze
+      REQUEST_SESSION_SCRAPERS = Set[NativeFeed, WordpressApi, Sitemap, MetaOembed].freeze
       # Scrapers that consume browser-captured XHR/fetch JSON bodies.
-      CAPTURED_RESPONSE_SCRAPERS = [XhrArticles].freeze
+      CAPTURED_RESPONSE_SCRAPERS = Set[XhrArticles].freeze
 
       ##
       # Error raised when no suitable scraper is found.
@@ -110,7 +110,7 @@ module Html2rss
       # @param tier [Array<Class>]
       # @return [Boolean]
       def self.heuristic_tier?(tier)
-        tier.intersect?(HEURISTIC_SCRAPERS)
+        HEURISTIC_SCRAPERS.intersect?(tier)
       end
 
       ##
@@ -163,16 +163,6 @@ module Html2rss
         scraper.new(parsed_body, url:, **kwargs, **scraper_opts)
       end
 
-      ##
-      # @param instance [Object]
-      # @param parsed_body [Nokogiri::HTML::Document]
-      # @return [Boolean]
-      def self.extractable_instance?(instance, parsed_body)
-        return instance.extractable? if instance.respond_to?(:extractable?)
-
-        instance.class.articles?(parsed_body)
-      end
-
       ##
       # @param parsed_body [Nokogiri::HTML::Document]
       # @param body [String, nil] raw response body (preferred for blocked-surface checks)

data/lib/html2rss/auto_source/segment.rb

@@ -2,9 +2,14 @@
 
 module Html2rss
   class AutoSource
+    Segment = Data.define(:root_node, :primary_link, :strategy, :position)
+
     ##
     # One candidate content block discovered by {Segmenter}.
-    Segment = Data.define(:root_node, :primary_link, :strategy, :position) do
+    class Segment
+      # Allowed Segmenter strategies.
+      STRATEGIES = Set[:semantic, :list, :cluster].freeze
+
       ##
       # @param root_node [SST::Node]
       # @param primary_link [SST::Node, nil]
@@ -17,13 +22,11 @@ module Html2rss
         unless primary_link.nil? || primary_link.is_a?(SST::Node)
           raise ArgumentError, 'primary_link must be SST::Node or nil'
         end
-        raise ArgumentError, "unknown strategy: #{strategy.inspect}" unless Segment::STRATEGIES.include?(strategy)
+        raise ArgumentError, "unknown strategy: #{strategy.inspect}" unless STRATEGIES.include?(strategy)
         raise ArgumentError, 'position must be Integer' unless position.is_a?(Integer)
 
         new(root_node:, primary_link:, strategy:, position:)
       end
     end
-    # Allowed Segmenter strategies.
-    Segment::STRATEGIES = %i[semantic list cluster].to_set.freeze
   end
 end

data/lib/html2rss/auto_source/segmenter/cluster.rb

@@ -13,15 +13,18 @@ module Html2rss
         # @param segmenter [Segmenter]
         # @return [Array<Segment>]
         def call(segmenter)
+          primary = PrimaryLink.new(segmenter)
           nodes = best_group_nodes(segmenter)
-          nodes.each_with_index.map do |node, position|
-            Segment.build(root_node: node, primary_link: nil, strategy: :cluster, position:)
+          nodes.each_with_index.filter_map do |node, position|
+            link = primary.select(node)
+            next unless link || segmenter.permit_unanchored
+
+            Segment.build(root_node: node, primary_link: link, strategy: :cluster, position:)
           end
         end
 
         def best_group_nodes(segmenter)
-          class_groups = collect_class_groups(segmenter)
-          groups = class_groups.empty? ? collect_structure_groups(segmenter) : class_groups
+          groups = collect_groups(segmenter)
           return [] if groups.empty?
 
           resolver = OverlapResolver.new(index: segmenter.index)
@@ -33,33 +36,29 @@ module Html2rss
         module_function :best_group_nodes
         private_class_method :best_group_nodes
 
-        def collect_class_groups(segmenter) # rubocop:disable Metrics/AbcSize
-          groups = Hash.new { |h, k| h[k] = [] }
+        def collect_groups(segmenter) # rubocop:disable Metrics/AbcSize, Metrics/CyclomaticComplexity, Metrics/MethodLength
+          class_groups = Hash.new { |hash, key| hash[key] = [] }
+          structure_groups = Hash.new { |hash, key| hash[key] = [] }
+          min = segmenter.minimum_selector_frequency
+
           segmenter.index.each_node do |node|
             next if SST::Tags::CLUSTER_EXCLUDED_NAMES.include?(node.name)
             next if segmenter.index.ignored_chrome?(node)
 
             cls = normalize_class(node.attrs.class_names)
-            groups[cls] << node unless cls.empty?
-          end
-          groups.select { |_, nodes| nodes.size >= segmenter.minimum_selector_frequency }
-        end
-        module_function :collect_class_groups
-        private_class_method :collect_class_groups
-
-        def collect_structure_groups(segmenter) # rubocop:disable Metrics/AbcSize
-          groups = Hash.new { |h, k| h[k] = [] }
-          segmenter.index.each_node do |node|
-            next if SST::Tags::CLUSTER_EXCLUDED_NAMES.include?(node.name)
-            next if segmenter.index.ignored_chrome?(node)
+            class_groups[cls] << node unless cls.empty?
 
             sig = structure_signature(node)
-            groups[sig] << node unless sig.empty?
+            structure_groups[sig] << node unless sig.empty?
           end
-          groups.select { |_, nodes| nodes.size >= segmenter.minimum_selector_frequency }
+
+          frequent_class = class_groups.select { |_, nodes| nodes.size >= min }
+          return frequent_class unless frequent_class.empty?
+
+          structure_groups.select { |_, nodes| nodes.size >= min }
         end
-        module_function :collect_structure_groups
-        private_class_method :collect_structure_groups
+        module_function :collect_groups
+        private_class_method :collect_groups
 
         def normalize_class(class_names)
           return '' if class_names.empty?
@@ -73,7 +72,7 @@ module Html2rss
           kids = node.children
           return '' if kids.empty?
 
-          kids.map { |c| c.name.to_s }.join('>')
+          kids.map { |child| child.name.to_s }.join('>')
         end
         module_function :structure_signature
         private_class_method :structure_signature