html2rss 0.24.0 → 0.26.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 8e71e2d9112602603e9cbdf23907d2ad6baabc633c22bd9305094c2ab405f9ad
-  data.tar.gz: 1239aecc1f56a8ab4f5f83990671d4655127ad481907552e89fcce30af7d25ef
+  metadata.gz: 2df4782641361435095b1adbc805a0b5f8bbf47637450a271b212e0905e025a1
+  data.tar.gz: 38542dd16fd74c1983be1d20cfc49b64508ce70eeb501f0a312e0b9777c59f07
 SHA512:
-  metadata.gz: fe0e1fe5a2b81886b0bda2f9087e12c42a5b309d105bcd7c1ea2268690740ee36390ad54bafb94757d758204968617569f981f489afe05572f41be28c1e4025a
-  data.tar.gz: cfad45944b508e5fafaae484cdb625e33ee43e336a2ab74b336f398c7eaab306432cf06d524ddb81b7ec5a002e77c065d7af491a6b245d0a2f5b2df6098aeda2
+  metadata.gz: e72f53ac552c7a0c65917d9d736a42e318360f205b7f532d845f89c0ea6bd219c09da0c1784596027566af8946aa2a9de937d21b39899a8838bd9fd70dd163ee
+  data.tar.gz: 69b705d89b11160fe0c392fb2045e61db8dc25d63a7e81af4d91201fce0face5e6fe3496b4b154af9d71eb41a6969491897d48162df382cd30facc30b3ab8bdc

data/README.md

@@ -13,7 +13,7 @@ Most people looking for a first working feed should start with `html2rss-web`, r
 Detailed usage guides, reference docs, and the feed directory live on the project website:
 
 - [Ruby gem documentation](https://html2rss.github.io/ruby-gem)
-- [Request strategies](https://html2rss.github.io/ruby-gem/reference/strategy) (`auto` = `faraday` → `botasaurus`; pin `browserless` explicitly)
+- [Request strategies](https://html2rss.github.io/ruby-gem/reference/strategy) (`auto` = `faraday` → `botasaurus`; or pin concrete strategies)
 - [Selectors & pagination](https://html2rss.github.io/ruby-gem/reference/selectors#paginated-feeds)
 - [Web application](https://html2rss.github.io/web-application)
 - [Feed directory](https://html2rss.github.io/feed-directory)
@@ -26,7 +26,7 @@ Cloud development: [Open in GitHub Codespaces](https://github.com/codespaces/new
 ## Architecture
 
 1. **Config** — loads and validates configuration (YAML/hash); schema via `html2rss schema` / `schema/html2rss-config.schema.json`
-2. **RequestService** — fetches pages (`faraday`, `botasaurus`, or explicit `browserless`)
+2. **RequestService** — fetches pages (`faraday`, `botasaurus`, or `local_file`)
 3. **Selectors** — extracts content via CSS selectors with extractors/post-processors
 4. **AutoSource** — auto-detects content (Schema.org, JSON state, semantic HTML, structural patterns)
 5. **FeedBuilder** — assembles Article objects and renders feeds (RSS 2.0 / JSON Feed 1.1)
@@ -35,6 +35,80 @@ Cloud development: [Open in GitHub Codespaces](https://github.com/codespaces/new
 Config -> Request -> Extraction -> Processing -> Building -> Output
 ```
 
+## Capture API
+
+The `Html2rss.capture` method analyzes any URL and produces a reusable feed config hash with derived CSS selectors. Use it to speed up writing feed configuration files.
+
+```ruby
+config = Html2rss.capture('https://example.com/articles')
+File.write('my-feed.yml', YAML.dump(Html2rss::HashUtil.deep_stringify_keys(config)))
+```
+
+The CLI alias `html2rss capture` prints the generated config as YAML to stdout. See [`docs/capture.md`](docs/capture.md) for detailed documentation.
+
+## MCP Server
+
+html2rss ships with an [MCP](https://modelcontextprotocol.io/) server that exposes gem capabilities as AI-consumable tools, resources, and prompts:
+
+```bash
+# Start with stdio transport (default; for Cursor/Claude Desktop)
+html2rss mcp
+
+# Start with HTTP transport (binds 127.0.0.1 only — local use)
+html2rss mcp --transport http --port 8080
+```
+
+HTTP transport needs `rack`, `rackup`, and `webrick` (declared gem dependencies). It listens on `127.0.0.1` only; do not expose it on a public interface without your own auth and Host/Origin controls.
+
+**Strategy note:** MCP tool `strategy: "auto"` collapses to `faraday` (no FeedPipeline botasaurus fallback). If results are empty or JS-gated, retry with `strategy: "botasaurus"` and `BOTASAURUS_SCRAPER_URL` set.
+
+### Tools
+
+| Name              | When to use                                               |
+| ----------------- | --------------------------------------------------------- |
+| `scrape_url`      | One-shot articles now (no saved config)                   |
+| `inspect_url`     | Diagnose weak scrape/capture (scrapers/SST/segments)      |
+| `capture_config`  | Derive a durable feed config (+ quality `_meta`)          |
+| `validate_config` | Schema-check a config before apply (`isError` on failure) |
+| `apply_config`    | Run a validated config → RSS XML                          |
+
+### Resources
+
+| URI                     | Description                                                     |
+| ----------------------- | --------------------------------------------------------------- |
+| `html2rss://schema`     | Full JSON Schema for feed configurations                        |
+| `html2rss://extractors` | Registered extractor **names** (options live in schema `$defs`) |
+| `html2rss://strategies` | Registered request strategy names                               |
+
+### Prompts
+
+| Name                  | Description                                             |
+| --------------------- | ------------------------------------------------------- |
+| `scrape-webpage`      | Guided scrape → inspect/retry with botasaurus if needed |
+| `capture-feed-config` | Guided capture → validate → optional apply              |
+
+The MCP module (`Html2rss::MCP`) lazy-loads the `mcp` gem — no cost when the server is not running.
+
+## Botasaurus scrape API (Docker)
+
+Start the Botasaurus scrape API for JavaScript-rendered pages (this compose file is **not** the MCP server):
+
+```bash
+docker compose -f docker-compose.botasaurus.yml up -d
+```
+
+Set `BOTASAURUS_SCRAPER_URL` to `http://127.0.0.1:4010` and use strategy `botasaurus` in MCP tools, Capture, or the CLI.
+
+## Request Strategies
+
+| Strategy     | Description                                                                   |
+| ------------ | ----------------------------------------------------------------------------- |
+| `auto`       | Tries `faraday`, falls back to `botasaurus` (default in gem/CLI FeedPipeline) |
+| `faraday`    | Plain HTTP requests via Faraday                                               |
+| `botasaurus` | Puppeteer-backed scraping for JavaScript pages                                |
+
+MCP tools intentionally collapse `auto` → `faraday` (see MCP section above). Elsewhere, strategy can be set via CLI (`--strategy`), gem API keyword argument, or feed config `request.strategy`. See the [request strategies docs](https://html2rss.github.io/ruby-gem/reference/strategy) for more details.
+
 ## License
 
 This project is licensed under the MIT License — see the [LICENSE](LICENSE) file for details.

data/html2rss.gemspec

@@ -39,14 +39,17 @@ Gem::Specification.new do |spec|
   spec.add_dependency 'faraday-follow_redirects'
   spec.add_dependency 'faraday-gzip', '~> 3'
   spec.add_dependency 'kramdown'
+  spec.add_dependency 'mcp', '~> 1.0'
   spec.add_dependency 'mime-types', '> 3.0'
   spec.add_dependency 'nokogiri', '>= 1.10', '< 2.0'
-  spec.add_dependency 'puppeteer-ruby'
+  spec.add_dependency 'rack', '~> 3.0'
+  spec.add_dependency 'rackup', '~> 2.0'
   spec.add_dependency 'regexp_parser'
   spec.add_dependency 'reverse_markdown', '~> 3.0'
   spec.add_dependency 'rss'
   spec.add_dependency 'sanitize'
   spec.add_dependency 'thor'
   spec.add_dependency 'tzinfo'
+  spec.add_dependency 'webrick', '~> 1.9'
   spec.add_dependency 'zeitwerk'
 end

data/lib/html2rss/auto_source/cleanup.rb

@@ -9,81 +9,107 @@ module Html2rss
     class Cleanup
       # Default cleanup behavior for auto-sourced article lists.
       DEFAULT_CONFIG = {
-        keep_different_domain: false,
-        min_words_title: 3
+        keep_different_domain: false
       }.freeze
 
+      # Minimum alphanumeric word count for present titles.
+      MIN_WORDS = 3
+
       # Allowed URL schemes for article filtering.
       VALID_SCHEMES = %w[http https].to_set.freeze
 
+      # Credit-agency-only or photo-credit titles (not headlines).
+      CREDIT_TITLE = %r{
+        \A(?:AFP|Getty(?:\s+Images)?|Reuters|dpa|Imagn)
+          (?:\s*/\s*(?:AFP|Getty(?:\s+Images)?|Reuters|dpa|Imagn))*\z
+        |
+        \A(?:Image|Photo|Credit)\s*[:|]?\s*
+          (?:AFP|Getty(?:\s+Images)?|Reuters|dpa|Imagn)\b
+      }ix
+
+      # Dotted / methode CMS tokens mistaken for titles.
+      CMS_TOKEN_TITLE = /\A(?:lucy\.\w[\w.-]*|methode[-.][\w.-]+)\z/i
+
+      # Raw URL slug / token clusters (hyphen or underscore, no natural phrasing).
+      SLUG_TITLE = /\A\p{Alnum}+(?:[-_]\p{Alnum}+){2,}\z/
+
+      # Date-prefix path tokens, raw or titleized ("2026 08 16 …", "2026-08-16-…").
+      DATE_PREFIX_TITLE = /\A\d{4}(?:[\s.-]+\d{1,2}){2}\b/
+
+      # Titleized path ending in a long numeric CMS id.
+      TITLEIZED_PATH_TITLE = /\A(?:\d+|\p{Lu}[\p{L}\p{M}]*)(?:\s+(?:\d+|\p{Lu}[\p{L}\p{M}]*))*\s+\d{6,}\z/
+
+      # Template / placeholder tokens mistaken for titles.
+      TEMPLATE_TITLE = /(\{\{[^}]+\}\}|%\{\w+\})/
+
       class << self
         # @param articles [Array<Article>] extracted article candidates
         # @param url [Html2rss::Url] feed source URL used for same-host filtering
         # @param keep_different_domain [Boolean] whether to keep off-domain entries
-        # @param min_words_title [Integer] minimum word count for title filtering
         # @return [Array<Article>] cleaned article list
-        def call(articles, url:, keep_different_domain:, min_words_title:)
+        def call(articles, url:, keep_different_domain: DEFAULT_CONFIG.fetch(:keep_different_domain))
           Log.debug "Cleanup: start with #{articles.size} articles"
 
           articles.select!(&:valid?)
 
-          deduplicate_by!(articles, :url)
-
+          deduplicate_by_url!(articles)
           keep_only_http_urls!(articles)
+          reject_self_links!(articles, url)
           reject_different_domain!(articles, url) unless keep_different_domain
-          keep_only_with_min_words_title!(articles, min_words_title:)
+          reject_low_quality_titles!(articles)
 
           Log.debug "Cleanup: end with #{articles.size} articles"
           articles
         end
 
-        ##
-        # Deduplicates articles by a given key.
-        #
-        # @param articles [Array<Article>] The list of articles to process.
-        # @param key [Symbol] The key to deduplicate by.
-        # @return [Array<Article>] the mutated articles array
-        def deduplicate_by!(articles, key)
+        private
+
+        def deduplicate_by_url!(articles)
           seen = {}
           articles.reject! do |article|
-            value = article.public_send(key)
-            value.nil? || seen.key?(value).tap { seen[value] = true }
+            identity = url_identity(article.url)
+            identity.nil? || seen.key?(identity).tap { seen[identity] = true }
           end
         end
 
-        ##
-        # Keeps only articles with HTTP or HTTPS URLs.
-        #
-        # @param articles [Array<Article>] The list of articles to process.
-        # @return [Array<Article>] the mutated articles array
         def keep_only_http_urls!(articles)
           articles.select! { |article| VALID_SCHEMES.include?(article.url&.scheme) }
         end
 
-        ##
-        # Rejects articles that have a URL not on the same domain as the source.
-        #
-        # @param articles [Array<Article>] The list of articles to process.
-        # @param base_url [Html2rss::Url] The source URL to compare against.
-        # @return [Array<Article>] the mutated articles array
+        def reject_self_links!(articles, base_url)
+          source_identity = url_identity(base_url)
+          articles.reject! { |article| url_identity(article.url) == source_identity }
+        end
+
         def reject_different_domain!(articles, base_url)
           base_host = base_url.host
           articles.select! { |article| article.url&.host == base_host }
         end
 
-        ##
-        # Keeps only articles with a title that is present and has at least `min_words_title` words.
-        #
-        # @param articles [Array<Article>] The list of articles to process.
-        # @param min_words_title [Integer] The minimum number of words in the title.
-        # @return [Array<Article>] the mutated articles array
-        def keep_only_with_min_words_title!(articles, min_words_title:)
+        # Keep missing titles (nil provenance). Drop present junk/unnatural titles —
+        # blanking them would hide bad extraction as "unknown" and inflate empty items.
+        def reject_low_quality_titles!(articles)
           articles.select! do |article|
-            article.title ? word_count_at_least?(article.title, min_words_title) : true
+            title = article.title
+            title.nil? || (word_count_at_least?(title, MIN_WORDS) && !junk_title?(title))
           end
         end
 
-        private
+        def url_identity(url)
+          url&.without_fragment&.to_s
+        end
+
+        def junk_title?(title)
+          CREDIT_TITLE.match?(title) || CMS_TOKEN_TITLE.match?(title) || unnatural_title?(title)
+        end
+
+        def unnatural_title?(title)
+          stripped = title.to_s.strip
+          SLUG_TITLE.match?(stripped) ||
+            DATE_PREFIX_TITLE.match?(stripped) ||
+            TITLEIZED_PATH_TITLE.match?(stripped) ||
+            TEMPLATE_TITLE.match?(stripped)
+        end
 
         def word_count_at_least?(str, min_words)
           count = 0

data/lib/html2rss/auto_source/scraper/html.rb

@@ -1,21 +1,11 @@
 # frozen_string_literal: true
 
-require 'nokogiri'
-
 module Html2rss
   class AutoSource
     module Scraper
       ##
-      # Scrapes article-like blocks from plain HTML by looking for repeated link
-      # structures when richer structured data is unavailable.
-      #
-      # The approach is intentionally heuristic:
-      # 1. collect repeated anchor paths
-      # 2. walk upward to a shared container shape
-      # 3. extract the best anchor found inside each container
-      #
-      # This scraper is broader and noisier than `SemanticHtml`, so it acts as a
-      # fallback for pages without stronger semantic signals.
+      # Fallback HTML list/cluster scraper via SST pipeline
+      # (Normalizer → Segmenter → Scoring::Engine → Html::SstArticleExtractor).
       class Html
         include Enumerable
 
@@ -25,53 +15,44 @@ module Html2rss
         DEFAULT_MINIMUM_SELECTOR_FREQUENCY = 2
         # Number of most frequent selectors kept for container extraction.
         DEFAULT_USE_TOP_SELECTORS = 5
+        # Maximum articles materialized after eligibility filtering.
+        TOP_K = Scoring::Engine::TOP_K
 
         ##
-        # @return [Symbol] config key used to enable or configure this scraper
+        # @return [Symbol]
         def self.options_key = :html
 
         ##
-        # Probes whether the document appears to contain repeated anchor
-        # structures that this fallback scraper can cluster into article-like
-        # containers.
-        #
-        # @param parsed_body [Nokogiri::HTML::Document] parsed HTML document
-        # @return [Boolean] true when the scraper can likely extract articles
+        # @param parsed_body [Nokogiri::HTML::Document]
+        # @return [Boolean]
         def self.articles?(parsed_body)
-          new(parsed_body, url: DETECTION_BASE_URL).any?
-        end
+          return false unless parsed_body
 
-        ##
-        # Simplify an XPath selector by removing the index notation.
-        # This keeps repeated anchor paths comparable across sibling blocks.
-        #
-        # @param xpath [String] original XPath
-        # @return [String] XPath without positional indexes
-        def self.simplify_xpath(xpath)
-          Discovery::ListCandidates.simplify_xpath(xpath)
+          new(parsed_body, url: DETECTION_BASE_URL).extractable?
+        rescue ArgumentError
+          false
         end
 
-        # @param parsed_body [Nokogiri::HTML::Document] The parsed HTML document.
-        # @param url [String] The base URL.
-        # @param extractor [Class] The extractor class to handle article extraction.
-        # @param opts [Hash] Additional options.
-        # @option opts [Integer] :minimum_selector_frequency minimum count before a selector is considered stable
-        # @option opts [Integer] :use_top_selectors number of top selectors to keep
-        def initialize(parsed_body, url:, extractor: Html2rss::Html::ArticleExtractor, **opts)
+        # @param parsed_body [Nokogiri::HTML::Document, nil] parsed HTML (when +document:+ omitted)
+        # @param url [String, Html2rss::Url]
+        # @param document [SST::Document, nil] memoized SST document from AutoSource
+        # @param link_resolver [Scoring::LinkResolver, nil] shared page-scoped resolver
+        # @param opts [Hash]
+        # @option opts [Boolean] :fallback_anchorless keep anchorless cluster cards
+        # @option opts [Integer] :minimum_selector_frequency list frequency floor
+        # @option opts [Integer] :use_top_selectors list selector budget
+        def initialize(parsed_body = nil, url:, document: nil, link_resolver: nil, **opts)
           @parsed_body = parsed_body
+          @provided_document = document
+          @provided_link_resolver = link_resolver
           @url = url
-          @extractor = extractor
           @opts = opts
           @fallback_anchorless = opts.fetch(:fallback_anchorless, false)
-          @link_heuristics = LinkHeuristics.new(url)
-          @ignored_cache = {}.compare_by_identity
         end
 
-        attr_reader :parsed_body
-
         ##
-        # @yieldparam [Hash] The scraped article hash
-        # @return [Enumerator] Enumerator for the scraped articles
+        # @yieldparam article [Html2rss::Article]
+        # @return [Enumerator]
         def each
           return enum_for(:each) unless block_given?
 
@@ -79,135 +60,77 @@ module Html2rss
         end
 
         ##
-        # @return [Boolean] true when the scraper can likely extract articles
+        # @return [Boolean]
         def extractable?
-          articles.any?
-        end
-
-        ##
-        # Decides whether a traversed node has reached a useful article-like
-        # boundary for the generic HTML scraper.
-        #
-        # The predicate prefers containers that add surrounding link context,
-        # which helps the scraper move from a leaf anchor toward a repeated
-        # teaser/card wrapper.
-        #
-        # @param node [Nokogiri::XML::Node] candidate boundary node
-        # @return [Boolean] true when the node is a good extraction boundary
-        def article_tag_condition?(node)
-          # Ignore tags that are below ignored DOM chrome.
-          return false if Html2rss::Html::Navigator.ignored_container_path?(node, @ignored_cache)
-          return true if %w[body html].include?(node.name)
-          return false unless (parent = node.parent)
-
-          anchor_count(parent) > anchor_count(node)
+          ranked_segments.any?
+        rescue ArgumentError
+          false
         end
 
         private
 
         def articles
           @articles ||= begin
-            extracted = each_article_tag.filter_map do |article_tag, selected_anchor|
-              extract_article(article_tag, selected_anchor:)
+            ranked = list_ranked
+            if ranked.empty? && @fallback_anchorless
+              materialize(cluster_ranked, fallback_anchorless: true)
+            else
+              materialize(ranked)
             end
-
-            extracted += find_anchorless_articles if @fallback_anchorless
-            extracted
           end
+        rescue ArgumentError
+          []
         end
 
-        def find_anchorless_articles
-          Discovery::DomClustering.call(
-            parsed_body,
-            minimum_selector_frequency:
-          ).map do |node|
-            @extractor.new(node, base_url: @url, selected_anchor: nil, fallback_anchorless: true).call
+        def ranked_segments
+          @ranked_segments ||= begin
+            ranked = list_ranked
+            ranked = cluster_ranked if @fallback_anchorless && ranked.empty?
+            ranked
           end
         end
 
-        ##
-        # @return [Integer]
-        def minimum_selector_frequency = @opts[:minimum_selector_frequency] || DEFAULT_MINIMUM_SELECTOR_FREQUENCY
-
-        ##
-        # @return [Boolean]
-        def use_top_selectors = @opts[:use_top_selectors] || DEFAULT_USE_TOP_SELECTORS
-
-        ##
-        # @param node [Nokogiri::XML::Node]
-        # @return [Integer]
-        def anchor_count(node)
-          (@anchor_counts ||= {}.compare_by_identity)[node] ||= node.name == 'a' ? 1 : node.css('a').size
-        end
-
-        ##
-        # @param node [Nokogiri::XML::Node]
-        # @return [Boolean]
-        def relevant_anchor?(node)
-          destination_facts = @link_heuristics.destination_facts(node)
-          return false unless destination_facts
-
-          !noise_anchor?(node, destination_facts:)
+        def list_ranked
+          @list_ranked ||= rank_strategy(:list, permit_unanchored: false)
         end
 
-        ##
-        # @yield [article_tag, selected_anchor]
-        # @yieldparam article_tag [Nokogiri::XML::Node]
-        # @yieldparam selected_anchor [Nokogiri::XML::Node]
-        # @return [Enumerator, nil]
-        def each_article_tag(&block)
-          return enum_for(:each_article_tag) unless block
-
-          anchor_filter = ->(node) { relevant_anchor?(node) }
-          boundary_condition = ->(node) { article_tag_condition?(node) }
-
-          list_candidates.each_article_tag(anchor_filter:, boundary_condition:, &block)
+        def cluster_ranked
+          @cluster_ranked ||= rank_strategy(:cluster, permit_unanchored: true)
         end
 
-        ##
-        # @param article_tag [Nokogiri::XML::Node]
-        # @param selected_anchor [Nokogiri::XML::Node, nil]
-        # @return [Hash, nil]
-        def extract_article(article_tag, selected_anchor: nil)
-          selected_anchor ||= preferred_anchor_for(article_tag)
-          return unless selected_anchor
-          return if noise_anchor?(
-            selected_anchor,
-            destination_facts: @link_heuristics.destination_facts(selected_anchor),
-            container: article_tag
+        def rank_strategy(strategy, permit_unanchored:)
+          segments = Segmenter.call(
+            document,
+            base_url: @url,
+            strategy:,
+            permit_unanchored:,
+            minimum_selector_frequency:,
+            use_top_selectors:,
+            link_resolver:
           )
-
-          @extractor.call(article_tag, base_url: @url, selected_anchor:)
+          engine.select_eligible(segments, limit: TOP_K)
         end
 
-        ##
-        # @param anchor [Nokogiri::XML::Node]
-        # @param destination_facts [LinkHeuristics::DestinationFacts, nil]
-        # @return [Boolean]
-        def noise_anchor?(anchor, destination_facts:, container: nil)
-          (@noise_anchors ||= {}.compare_by_identity)[anchor] ||= begin
-            text = Html2rss::Html::Navigator.extract_visible_text(anchor).to_s.strip
-            @link_heuristics.noise_anchor?(text:, destination_facts:, anchor:, container:)
+        def materialize(ranked, fallback_anchorless: false)
+          ranked.filter_map do |entry|
+            ::Html2rss::Html::SstArticleExtractor.call(entry, base_url: @url, scraper: self.class, fallback_anchorless:)
           end
         end
 
-        ##
-        # @param article_tag [Nokogiri::XML::Node]
-        # @return [Nokogiri::XML::Node, nil]
-        def preferred_anchor_for(article_tag)
-          article_tag.css(Html2rss::Html::Navigator::MAIN_ANCHOR_SELECTOR).find { relevant_anchor?(_1) } ||
-            Html2rss::Html::Navigator.main_anchor_for(article_tag)
+        def engine
+          @engine ||= Scoring::Engine.new(link_resolver:)
         end
 
-        ##
-        # @return [Discovery::ListCandidates]
-        def list_candidates
-          Discovery::ListCandidates.new(
-            parsed_body,
-            minimum_selector_frequency:,
-            use_top_selectors:
-          )
+        def link_resolver
+          @link_resolver ||= @provided_link_resolver || Scoring::LinkResolver.new(@url)
         end
+
+        def document
+          @document ||= @provided_document || SST::Normalizer.call(@parsed_body)
+        end
+
+        def minimum_selector_frequency = @opts[:minimum_selector_frequency] || DEFAULT_MINIMUM_SELECTOR_FREQUENCY
+        def use_top_selectors = @opts[:use_top_selectors] || DEFAULT_USE_TOP_SELECTORS
       end
     end
   end

data/lib/html2rss/auto_source/scraper/json_state/article_normalizer.rb

@@ -0,0 +1,116 @@
+# frozen_string_literal: true
+
+module Html2rss
+  class AutoSource
+    module Scraper
+      class JsonState
+        # Shapes raw entries into the structure required downstream.
+        module ArticleNormalizer
+          # Preferred keys when extracting title-like values from state payloads.
+          TITLE_KEYS = %i[title headline name text].freeze
+          # Preferred keys when extracting URL-like values from state payloads.
+          URL_KEYS = %i[url link href permalink slug path canonicalUrl shortUrl].freeze
+          # Preferred keys when extracting description-like values from state payloads.
+          DESCRIPTION_KEYS = %i[description summary excerpt dek subheading].freeze
+          # Preferred keys when extracting image-like values from state payloads.
+          IMAGE_KEYS = %i[image imageUrl thumbnailUrl thumbnail src featuredImage coverImage heroImage].freeze
+          # Preferred keys when extracting publication timestamps from state payloads.
+          PUBLISHED_AT_KEYS = %i[published_at publishedAt datePublished date publicationDate pubDate updatedAt
+                                 updated_at createdAt created_at].freeze
+          # Preferred keys when extracting category-like values from state payloads.
+          CATEGORY_KEYS = %i[categories tags section sections topic topics channel].freeze
+          # Preferred keys when extracting identifier-like values from state payloads.
+          ID_KEYS = %i[id guid uuid slug key].freeze
+
+          module_function
+
+          # rubocop:disable Metrics/MethodLength
+          # @param entry [Hash] raw article entry candidate
+          # @param base_url [String, Html2rss::Url] base URL for relative link resolution
+          # @return [Hash{Symbol => Object, nil}] normalized article hash for downstream extraction
+          def normalise(entry, base_url:)
+            return unless entry.is_a?(Hash)
+
+            title = string(ValueFinder.fetch(entry, TITLE_KEYS))
+            description = string(ValueFinder.fetch(entry, DESCRIPTION_KEYS))
+            article_url = resolve_link(entry, keys: URL_KEYS, base_url:,
+                                              log_key: 'JsonState: invalid URL encountered')
+            return unless article_url
+            return if title.nil? && description.nil?
+
+            {
+              title:,
+              description:,
+              url: article_url,
+              image: resolve_link(entry, keys: IMAGE_KEYS, base_url:,
+                                         log_key: 'JsonState: invalid image URL encountered'),
+              published_at: string(ValueFinder.fetch(entry, PUBLISHED_AT_KEYS)),
+              categories: categories(entry),
+              id: identifier(entry, article_url)
+            }.compact
+          end
+          # rubocop:enable Metrics/MethodLength
+
+          # @param value [Object] candidate scalar value
+          # @return [String, nil] normalized non-empty string value
+          def string(value)
+            trimmed = value.to_s.strip
+            trimmed unless trimmed.empty?
+          end
+
+          # @param entry [Hash] raw article entry candidate
+          # @param keys [Array<String>] preferred link keys
+          # @param base_url [String, Html2rss::Url] base URL for relative link resolution
+          # @param log_key [String] structured log message key
+          # @return [Html2rss::Url, nil] resolved absolute URL
+          def resolve_link(entry, keys:, base_url:, log_key:)
+            value = ValueFinder.fetch(entry, keys)
+            value = ValueFinder.fetch(value, keys) if value.is_a?(Hash)
+            string = string(value)
+            return unless string
+
+            Url.from_relative(string, base_url)
+          rescue ArgumentError
+            Log.debug(log_key, url: string)
+            nil
+          end
+
+          # rubocop:disable Metrics/MethodLength
+          # @param entry [Hash] raw article entry candidate
+          # @return [Array<String>, nil] normalized unique categories
+          def categories(entry)
+            raw = ValueFinder.fetch(entry, CATEGORY_KEYS)
+            names = case raw
+                    when Array then raw
+                    when Hash then raw.values
+                    when String then [raw]
+                    else []
+                    end
+
+            result = names.flat_map do |value|
+              case value
+              when Hash
+                string(ValueFinder.fetch(value, %i[name title label]))
+              else
+                string(value)
+              end
+            end.compact
+
+            result.uniq!
+            result unless result.empty?
+          end
+          # rubocop:enable Metrics/MethodLength
+
+          # @param entry [Hash] raw article entry candidate
+          # @param article_url [Html2rss::Url] resolved article URL
+          # @return [String] stable article identifier fallbacking to resolved URL
+          def identifier(entry, article_url)
+            value = ValueFinder.fetch(entry, ID_KEYS)
+            value = ValueFinder.fetch(value, ID_KEYS) if value.is_a?(Hash)
+            string(value) || article_url.to_s
+          end
+        end
+      end
+    end
+  end
+end