html2rss 0.22.2 → 0.23.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: eb40d85981c1bc11576b0c9d7a3eae3948bd49239ad8b9a65bb6a4ea103b54f9
-  data.tar.gz: 6c4636413e1e995e98adec8d8cc742a5833208c2197a024abe90f4b81fce1774
+  metadata.gz: f53482ae1084593193664ab8301bd451f53f68b4afabc8f0a336d9ae6dcd357c
+  data.tar.gz: 65ef2c8261a216040484134f71de9b8579378d6f7aa20100d0c79cf684efcd65
 SHA512:
-  metadata.gz: 9134076ad05417b8b7176da26033e4d66137f862781378b44080d4ea4fe67e8932156c8eb7d22b58ad97f0d7f5e32102c96ef84741d681c2c7571b3add09afb0
-  data.tar.gz: 61958c85cb791adc78d6d18debcdbfb7a90948332696dac0a4ab4aee39f6b098c3366f948d8905d3d36e2052e03e75037006bd2e6cfa3420b9a11e46e3ebce18
+  metadata.gz: 18c6b5dfb6f23753b1c4885339b16505800b91235e65f397d7195d2301b7c32974c12be0087c35c2e63a5bd08185518ea49c6bd252cc68f22521814c6f6e74f8
+  data.tar.gz: 458fff5c3579ea95190e458a9cf5814418467fdfc37cedc1244faf4d7c7eb6582640b9d8465329f68eac3624c33c1691514da25c1ce1a7cb6ddb2c7999c38638

data/README.md

@@ -13,125 +13,28 @@ Most people looking for a first working feed should start with `html2rss-web`, r
 Detailed usage guides, reference docs, and the feed directory live on the project website:
 
 - [Ruby gem documentation](https://html2rss.github.io/ruby-gem)
+- [Request strategies](https://html2rss.github.io/ruby-gem/reference/strategy) (`auto`, `faraday`, `botasaurus`, `browserless`)
+- [Selectors & pagination](https://html2rss.github.io/ruby-gem/reference/selectors#paginated-feeds)
 - [Web application](https://html2rss.github.io/web-application)
 - [Feed directory](https://html2rss.github.io/feed-directory)
 - [Contributing guide](https://html2rss.github.io/get-involved/contributing)
 - [GitHub Discussions](https://github.com/orgs/html2rss/discussions)
 - [Sponsor on GitHub](https://github.com/sponsors/gildesmarais)
 
-### 💻 Try in Browser
+Cloud development: [Open in GitHub Codespaces](https://github.com/codespaces/new?repo=html2rss/html2rss) (also covered in the [installation guide](https://html2rss.github.io/ruby-gem/installation)).
 
-You can develop html2rss directly in your browser using GitHub Codespaces:
+## Architecture
 
-[![Open in GitHub Codespaces](https://github.com/codespaces/badge.svg)](https://github.com/codespaces/new?repo=html2rss/html2rss)
-
-The Codespace comes pre-configured with Ruby 3.4 (compatible with Ruby 4.0), all dependencies, and VS Code extensions ready to go!
-
-## 🤝 Contributing
-
-Please see the [contributing guide](https://html2rss.github.io/get-involved/contributing) for details on how to contribute.
-
-## 🏗️ Architecture
-
-### Core Components
-
-1. **Config** - Loads and validates configuration (YAML/hash)
-2. **RequestService** - Fetches pages using Faraday, Botasaurus, or Browserless
-3. **Selectors** - Extracts content via CSS selectors with extractors/post-processors
-4. **AutoSource** - Auto-detects content using Schema.org, JSON state blobs, semantic HTML, and structural patterns
-5. **RssBuilder** - Assembles Article objects and renders RSS 2.0
-
-### Data Flow
+1. **Config** — loads and validates configuration (YAML/hash); schema via `html2rss schema` / `schema/html2rss-config.schema.json`
+2. **RequestService** — fetches pages (`faraday`, `botasaurus`, or `browserless`)
+3. **Selectors** — extracts content via CSS selectors with extractors/post-processors
+4. **AutoSource** — auto-detects content (Schema.org, JSON state, semantic HTML, structural patterns)
+5. **FeedBuilder** — assembles Article objects and renders feeds (RSS 2.0 / JSON Feed 1.1)
 
 ```text
 Config -> Request -> Extraction -> Processing -> Building -> Output
 ```
 
-### Request Strategies
-
-- `auto` (default): pipeline fallback orchestration (`faraday` -> `botasaurus` -> `browserless`) based on extraction outcome and retry policy.
-- `faraday`: direct HTTP fetch.
-- `botasaurus`: delegates fetching to a Botasaurus scrape API. Requires `BOTASAURUS_SCRAPER_URL` (for example `http://localhost:4010`).
-- `browserless`: remote browser rendering via Browserless (`BROWSERLESS_IO_WEBSOCKET_URL` and token as needed).
-
-Auto fallback shares one request budget across all strategy attempts. For pagination-heavy or dynamic pages, increase `request.max_requests` (or `--max-requests`) when retries exhaust the budget.
-
-Auto fallback decisions are hidden at the default `LOG_LEVEL=warn`; run with `LOG_LEVEL=info` to include them in CLI output.
-
-Supported `request.botasaurus` options:
-
-- `navigation_mode` (`auto`, `get`, `google_get`, `google_get_bypass`; default `auto`)
-- `max_retries` (`0..3`; default `2`)
-- `wait_for_selector` (string)
-- `wait_timeout_seconds` (integer)
-- `block_images` (boolean)
-- `block_images_and_css` (boolean)
-- `wait_for_complete_page_load` (boolean)
-- `headless` (boolean, default `false`)
-- `proxy` (string)
-- `user_agent` (string)
-- `window_size` (two-item integer array, for example `[1920, 1080]`)
-- `lang` (string, for example `en-US`)
-
-Minimal YAML config example:
-
-```yaml
-channel:
-  url: https://example.com
-strategy: botasaurus
-auto_source: {}
-request:
-  botasaurus:
-    navigation_mode: auto
-    max_retries: 2
-    headless: false
-```
-
-Example request payload shape:
-
-```json
-{
-  "url": "https://example.com",
-  "navigation_mode": "auto",
-  "max_retries": 2,
-  "headless": false
-}
-```
-
-Example usage:
-
-```bash
-BOTASAURUS_SCRAPER_URL=http://localhost:4010 html2rss auto https://example.com --strategy botasaurus
-```
-
-Policy note: html2rss still enforces local request policy preflight and timeout budget. Botasaurus handles browser navigation/rendering internals, so some policy details are delegated to upstream execution.
-
-### Config schema workflow
-
-The config schema is generated from the runtime `dry-validation` contracts and exported for client-side tooling.
-
-- Ruby API: `Html2rss::Config.json_schema`
-- CLI: `html2rss schema`
-- CLI options:
-  - `html2rss schema --write tmp/html2rss-config.schema.json`
-  - `html2rss schema --no-pretty`
-- Runtime validation API: `Html2rss::Config.validate(config_hash)`
-- Runtime validation CLI: `html2rss validate config.yml`
-- Packaged JSON file: `schema/html2rss-config.schema.json`
-
-If you are an editor integration, automation script, or AI tool, prefer these stable discovery points:
-
-- call `html2rss schema` to read the current exported schema
-- read `schema/html2rss-config.schema.json` when working from the repository or installed gem
-- use `Html2rss::Config.schema_path` if you already have Ruby loaded
-- use `Html2rss::Config.validate` or `html2rss validate config.yml` when you need authoritative runtime validation of selector references
-
-Run `bundle exec rake config:schema` before committing to regenerate `schema/html2rss-config.schema.json` and keep the checked-in JSON Schema in sync with the validators. The exported schema covers client-side validation, while runtime validation remains authoritative for dynamic cross-field checks such as selector-key references.
-
-## 📄 License
-
-This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
-
-## 💖 Sponsoring
+## License
 
-If you find `html2rss` useful, please consider [sponsoring the project](https://github.com/sponsors/gildesmarais).
+This project is licensed under the MIT License — see the [LICENSE](LICENSE) file for details.

data/lib/html2rss/{articles → article}/deduplicator.rb

@@ -3,8 +3,7 @@
 require 'set' # rubocop:disable Lint/RedundantRequireStatement
 
 module Html2rss
-  # Shared helpers that operate on `RssBuilder::Article` collections.
-  module Articles
+  class Article
     ##
     # Deduplicates a list of articles while preserving their original order.
     #
@@ -15,7 +14,7 @@ module Html2rss
     # defaults to the article object's hash to preserve the original entry.
     class Deduplicator
       ##
-      # @param articles [Array<Html2rss::RssBuilder::Article>]
+      # @param articles [Array<Html2rss::Article>]
       # @raise [ArgumentError] if articles are not provided
       def initialize(articles)
         raise ArgumentError, 'articles must be provided' unless articles
@@ -26,7 +25,7 @@ module Html2rss
       ##
       # Returns the list of unique articles, preserving the order of the
       # original collection and keeping the first occurrence of a duplicate.
-      # @return [Array<Html2rss::RssBuilder::Article>]
+      # @return [Array<Html2rss::Article>]
       def call
         seen = Set.new
 

data/lib/html2rss/{rss_builder → article}/enclosure.rb

@@ -3,9 +3,9 @@
 require 'mime/types'
 
 module Html2rss
-  class RssBuilder
+  class Article
     ##
-    # Represents an enclosure for an RSS item.
+    # Represents an enclosure attached to an article (RSS / JSON Feed media).
     class Enclosure
       ##
       # Guesses the content type based on the file extension of the URL.
@@ -24,38 +24,22 @@ module Html2rss
         content_type.first&.to_s || 'application/octet-stream'
       end
 
-      # @param enclosure [Html2rss::RssBuilder::Enclosure, nil] built enclosure object for the current RSS item
-      # @param maker [RSS::Maker::RSS20::ItemsBase::ItemBase] RSS item builder
-      # @return [void]
-      def self.add(enclosure, maker)
-        return unless enclosure
-
-        maker.enclosure.tap do |enclosure_maker|
-          enclosure_maker.url = enclosure.url.to_s
-          enclosure_maker.type = enclosure.type
-          enclosure_maker.length = enclosure.bits_length
-        end
-      end
-
       # @param url [Html2rss::Url] absolute enclosure URL
       # @param type [String, nil] optional enclosure MIME type
-      # @param bits_length [Integer] enclosure byte length (historical name)
-      def initialize(url:, type: nil, bits_length: 0)
+      # @param bytes_length [Integer] enclosure length in bytes
+      def initialize(url:, type: nil, bytes_length: 0)
         raise ArgumentError, 'An Enclosure requires an absolute URL' if !url || !url.absolute?
 
         @url = url
         @type = type
-        @bits_length = bits_length
+        @bytes_length = bytes_length
       end
 
       # @return [String] explicit MIME type or one inferred from URL extension
       def type = @type || self.class.guess_content_type_from_url(url)
 
       # @return [Integer] enclosure length in bytes
-      def bytes_length = @bits_length
-
-      # @return [Integer] enclosure length in bytes (legacy reader name)
-      def bits_length = bytes_length
+      attr_reader :bytes_length
 
       # @return [Html2rss::Url] absolute enclosure URL
       attr_reader :url

data/lib/html2rss/article.rb

@@ -0,0 +1,206 @@
+# frozen_string_literal: true
+
+require 'zlib'
+require 'sanitize'
+require 'nokogiri'
+
+module Html2rss
+  ##
+  # Article is a simple data object representing an article extracted from a page.
+  # It is enumerable and responds to all keys specified in PROVIDED_KEYS.
+  # rubocop:disable Metrics/ClassLength
+  class Article
+    include Enumerable
+    include Comparable
+
+    # Allowed article attributes accepted by the value object constructor.
+    PROVIDED_KEYS = %i[id title description url image author guid published_at enclosures categories scraper].freeze
+    # Separator used to build deterministic deduplication fingerprints.
+    DEDUP_FINGERPRINT_SEPARATOR = '#!/'
+    # Sentinel object used to pre-initialize instance variables in the constructor.
+    # This ensures all Article instances share the exact same object shape (Ruby 3.3+ optimization),
+    # preventing performance warnings and slower instance variable access due to shape transitions
+    # when attributes are lazily/conditionally accessed in different sequences.
+    NOT_SET = Object.new.freeze
+
+    # @param options [Hash{Symbol => String}]
+    # @option options [String] :id stable article identifier
+    # @option options [String] :title article title
+    # @option options [String] :description article description/content
+    # @option options [String, Html2rss::Url] :url canonical article URL
+    # @option options [String, Html2rss::Url] :image image URL for fallback enclosure rendering
+    # @option options [String] :author author name
+    # @option options [String] :guid explicit GUID override
+    # @option options [String, Time, DateTime] :published_at publication timestamp
+    # @option options [Array<Hash{Symbol => Object}>] :enclosures enclosure attribute hashes
+    # @option options [Array<String>] :categories category labels
+    # @option options [Class] :scraper scraper class that produced the article
+    def initialize(**options)
+      @to_h = options.each_with_object({}) { |(k, v), h| h[k] = v.freeze if v }.freeze
+
+      @description = @url = @image = @guid = @enclosures = @enclosure = @categories = @published_at = NOT_SET
+
+      return unless (unknown_keys = options.keys - PROVIDED_KEYS).any?
+
+      Log.warn "Article: unknown keys found: #{unknown_keys.join(', ')}"
+    end
+
+    # Checks if the article is valid based on the presence of URL, ID, and either title or description.
+    # @return [Boolean] True if the article is valid, otherwise false.
+    def valid?
+      !url.to_s.empty? && (!title.to_s.empty? || !description.to_s.empty?) && !id.to_s.empty?
+    end
+
+    # @yield [key, value]
+    # @return [Enumerator] if no block is given
+    def each
+      return enum_for(:each) unless block_given?
+
+      PROVIDED_KEYS.each { |key| yield(key, public_send(key)) }
+    end
+
+    # @return [String, nil] stable article identifier
+    def id = blank_string_to_nil(@to_h[:id])
+
+    # @return [String, nil] article title
+    def title = blank_string_to_nil(@to_h[:title])
+
+    # @return [String] rendered article description
+    def description
+      return @description unless @description == NOT_SET
+
+      @description = Html::Rendering::DescriptionBuilder.new(
+        base: @to_h[:description],
+        title:,
+        url:,
+        enclosures:,
+        image:
+      ).call
+    end
+
+    # @return [Url, nil]
+    def url
+      return @url unless @url == NOT_SET
+
+      @url = Url.sanitize(@to_h[:url])
+    end
+
+    # @return [Url, nil]
+    def image
+      return @image unless @image == NOT_SET
+
+      @image = Url.sanitize(@to_h[:image])
+    end
+
+    # @return [String, nil]
+    def author = blank_string_to_nil(@to_h[:author])
+
+    # Generates a unique identifier based on the URL and ID using CRC32.
+    # @return [String]
+    def guid
+      return @guid unless @guid == NOT_SET
+
+      @guid = Zlib.crc32(fetch_guid).to_s(36).encode('utf-8')
+    end
+
+    ##
+    # Returns a deterministic fingerprint used to detect duplicate articles.
+    #
+    # @return [String, Integer]
+    def deduplication_fingerprint
+      dedup_from_url || dedup_from_id || dedup_from_guid || hash
+    end
+
+    # @return [Array<Html2rss::Article::Enclosure>] normalized enclosure objects
+    def enclosures
+      return @enclosures unless @enclosures == NOT_SET
+
+      @enclosures = Array(@to_h[:enclosures])
+                    .map { |enclosure| Enclosure.new(**enclosure) }
+    end
+
+    # @return [Html2rss::Article::Enclosure, nil]
+    def enclosure
+      return @enclosure unless @enclosure == NOT_SET
+
+      @enclosure = case (object = @to_h[:enclosures]&.first)
+                   when Hash
+                     Enclosure.new(**object)
+                   when nil
+                     Enclosure.new(url: image) if image
+                   else
+                     Log.warn "Article: unknown enclosure type: #{object.class}"
+                     nil
+                   end
+    end
+
+    # @return [Array<String>] normalized, unique category names
+    def categories
+      return @categories unless @categories == NOT_SET
+
+      @categories = @to_h[:categories].dup.to_a.tap do |categories|
+        categories.map! { |category| category.to_s.strip }
+        categories.reject!(&:empty?)
+        categories.uniq!
+      end
+    end
+
+    # Parses and returns the published_at time.
+    # @return [DateTime, nil]
+    def published_at
+      return @published_at unless @published_at == NOT_SET
+
+      string = @to_h[:published_at].to_s.strip
+      @published_at = string.empty? ? nil : DateTime.parse(string)
+    rescue ArgumentError
+      @published_at = nil
+    end
+
+    # @return [Class, nil] scraper class that produced this article
+    def scraper
+      @to_h[:scraper]
+    end
+
+    # @param other [Object] value compared against this article
+    # @return [Integer, nil] comparison result for compatible Article values
+    def <=>(other)
+      return nil unless other.is_a?(Article)
+
+      0 if other.all? { |key, value| value == public_send(key) ? public_send(key) <=> value : false }
+    end
+
+    private
+
+    def dedup_from_url
+      return unless (value = url)
+
+      [value.to_s, id].compact.join(DEDUP_FINGERPRINT_SEPARATOR)
+    end
+
+    def dedup_from_id
+      return if id.to_s.empty?
+
+      id
+    end
+
+    def dedup_from_guid
+      value = guid
+      return if value.to_s.empty?
+
+      [value, title, description].compact.join(DEDUP_FINGERPRINT_SEPARATOR)
+    end
+
+    def fetch_guid
+      guid = @to_h[:guid].map { |s| s.to_s.strip }.reject(&:empty?).join if @to_h[:guid].is_a?(Array)
+
+      guid || [url, id].join('#!/')
+    end
+
+    def blank_string_to_nil(value)
+      return if value.is_a?(String) && value.strip.empty?
+
+      value
+    end
+  end
+  # rubocop:enable Metrics/ClassLength
+end

data/lib/html2rss/auto_source/discovery/dom_clustering/group_scorer.rb

@@ -0,0 +1,80 @@
+# frozen_string_literal: true
+
+module Html2rss
+  class AutoSource
+    module Discovery
+      class DomClustering
+        ##
+        # Scores candidate DOM groups using heading, time, date, and word-count signals.
+        class GroupScorer
+          def initialize
+            @text_words = {}.compare_by_identity
+            @has_date = {}.compare_by_identity
+          end
+
+          # @param groups [Hash{String => Array<Nokogiri::XML::Node>}] candidate DOM groups
+          # @return [Array<Nokogiri::XML::Node>] nodes from the highest-scoring group
+          def select_best_group(groups)
+            best_nodes = []
+            best_score = -1
+
+            groups.each_value do |nodes|
+              score = score_group(nodes)
+              next if score.negative?
+
+              (best_nodes = nodes) && (best_score = score) if score > best_score
+            end
+
+            best_nodes
+          end
+
+          # @param nodes [Array<Nokogiri::XML::Node>]
+          # @return [Float] average visible word count across nodes
+          def avg_words(nodes)
+            nodes.sum { |n| text_words(n) } / nodes.size.to_f
+          end
+
+          private
+
+          def score_group(nodes)
+            avg_w = avg_words(nodes)
+            return -1 if avg_w < 5
+
+            score = nodes.size + (avg_w / 5.0)
+            score += 20 if nodes_heading?(nodes)
+            score += 20 if nodes_time?(nodes)
+            score += 40 if nodes_date?(nodes)
+            score
+          end
+
+          def nodes_heading?(nodes)
+            nodes.any? do |n|
+              n.at_css(Html2rss::Html::Navigator::HEADING_TAGS.join(',')) ||
+                n.at_css('.font-bold, .font-semibold')
+            end
+          end
+
+          def nodes_time?(nodes)
+            nodes.any? { |n| n.at_css('time, [datetime]') }
+          end
+
+          def nodes_date?(nodes)
+            nodes.any? { |n| date?(n) }
+          end
+
+          def text_words(node)
+            @text_words[node] ||= Html2rss::Html::Navigator.extract_visible_text(node).to_s.scan(/\p{Alnum}+/).size
+          end
+
+          def date?(node)
+            @has_date[node] ||= begin
+              text = Html2rss::Html::Navigator.extract_visible_text(node).to_s
+              text.match?(%r{\b\d{4}[-/]\d{2}[-/]\d{2}\b}) ||
+                text.match?(/\b(?:Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)\b/i)
+            end
+          end
+        end
+      end
+    end
+  end
+end

data/lib/html2rss/auto_source/discovery/dom_clustering/overlap_resolver.rb

@@ -0,0 +1,86 @@
+# frozen_string_literal: true
+
+module Html2rss
+  class AutoSource
+    module Discovery
+      class DomClustering
+        ##
+        # Filters layout wrapper groups and resolves 1-to-1 nested card wrappers.
+        class OverlapResolver
+          # @param layout_tags [Set<String>] tags treated as layout containers
+          # @param word_counter [#avg_words] averages visible word counts for nodes
+          def initialize(layout_tags:, word_counter:)
+            @layout_tags = layout_tags
+            @word_counter = word_counter
+          end
+
+          # Discard group A if any node of A contains > 1 node of another group B
+          #
+          # @param groups [Hash{String => Array<Nokogiri::XML::Node>}] candidate groups
+          # @return [Hash{String => Array<Nokogiri::XML::Node>}] groups that are not layout wrappers
+          def filter_containers(groups)
+            groups.reject do |cls_a, nodes_a|
+              groups.any? { |cls_b, nodes_b| cls_a != cls_b && container_of?(nodes_a, nodes_b) }
+            end
+          end
+
+          # If group A contains group B with the same size, keep the real content card.
+          #
+          # @param groups [Hash{String => Array<Nokogiri::XML::Node>}] candidate groups
+          # @return [Hash{String => Array<Nokogiri::XML::Node>}] groups after 1-to-1 resolution
+          def filter_1_to_1_overlap(groups)
+            discarded = Set.new
+            groups.each_key do |cls_a|
+              groups.each_key do |cls_b|
+                next if cls_a == cls_b || discarded.include?(cls_a) || discarded.include?(cls_b)
+
+                resolve_1_to_1_overlap(cls_a, cls_b, groups, discarded)
+              end
+            end
+
+            groups.except(*discarded)
+          end
+
+          private
+
+          attr_reader :layout_tags, :word_counter
+
+          # rubocop:disable Metrics/MethodLength
+          def container_of?(nodes_a, nodes_b)
+            return false unless layout_tags.include?(nodes_b.first.name)
+
+            nodes_a.any? do |node_a|
+              count = 0
+              nodes_b.each do |node_b|
+                next if node_a == node_b
+
+                if Html2rss::Html::Navigator.descendant_of?(node_b, node_a)
+                  count += 1
+                  break if count > 1
+                end
+              end
+              count > 1
+            end
+          end
+          # rubocop:enable Metrics/MethodLength
+
+          def resolve_1_to_1_overlap(cls_a, cls_b, groups, discarded)
+            nodes_a = groups[cls_a]
+            nodes_b = groups[cls_b]
+            return if nodes_a.size != nodes_b.size
+
+            nested = nodes_a.zip(nodes_b).all? { |a, b| a != b && Html2rss::Html::Navigator.descendant_of?(b, a) }
+            return unless nested
+
+            discarded << (keep_descendant?(nodes_a, nodes_b) ? cls_a : cls_b)
+          end
+
+          def keep_descendant?(nodes_a, nodes_b)
+            word_counter.avg_words(nodes_b) >= 0.8 * word_counter.avg_words(nodes_a) &&
+              layout_tags.include?(nodes_b.first.name)
+          end
+        end
+      end
+    end
+  end
+end