html2rss 0.31.0 → 0.32.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: e3450f1c6b664e4f9c7377220b65640a9e4bd640782d4f8831538c65a172cec3
-  data.tar.gz: 1f5543f86a3389a4f2abb9a76dec44ba41a4ffecadcf8d3d674ff084c459aaec
+  metadata.gz: d1d74d19e935fcf05c7bd0893f2113dfac1e6639de926ef8844159118ad13ff5
+  data.tar.gz: 3999a5e55251f993b6721d9d3f8c03b8c7588fa0834499a53c808789bb7a3ded
 SHA512:
-  metadata.gz: 6b2d0fc1afbd44a13405eb21ad89ea32dbdf8d3c0dd1dd0f6430f877398a2cb5275781de7d65386fbde71f43db34e35a54c1fb5d13a73a265c8a8a274f496f87
-  data.tar.gz: 6402bc203d89b23fc388872b310eb4c6aaa6b9423025a40ec6eaa606ecf5c78a64c09823a103b263344b72acae4fa385e11526f59571b699eaa54ec27379cf66
+  metadata.gz: 1c7bea03d8553b2e8ec76c8216b57e0f07b20157cb1c1710c31b4730281b410c95b61541fae56bc4d3853cedf5e62854e0255b920607e28cfa902e941e779953
+  data.tar.gz: 6911fdce7df3238bbe27b28b6a8f747077b12a5cd949b12e4cc0bddadff1433f0ef56662cc361f7d80c90924fe38813ac6584ef0ec08a6b39b0859bc4067d2d4

data/README.md

@@ -1,4 +1,4 @@
-![html2rss logo](https://github.com/html2rss/html2rss/raw/master/support/logo.png)
+![html2rss logo](https://html2rss.github.io/assets/logo.png)
 
 [![Gem Version](https://badge.fury.io/rb/html2rss.svg)](http://rubygems.org/gems/html2rss) [![Yard Docs](http://img.shields.io/badge/yard-docs-blue.svg)](https://www.rubydoc.info/gems/html2rss) ![Retro Badge: valid RSS](https://validator.w3.org/feed/images/valid-rss-rogers.png) [![CI](https://github.com/html2rss/html2rss/workflows/lint%20and%20test/badge.svg)](https://github.com/html2rss/html2rss/actions)
 
@@ -62,6 +62,9 @@ html2rss capture https://example.com/news | html2rss test -
 # One-shot articles now
 html2rss scrape https://example.com/news
 html2rss scrape https://example.com/news --format jsonfeed --explain
+# Off-domain item URLs are dropped by default. --keep-different-domain opts in
+# and lets the remote source control published links to unrelated domains.
+html2rss scrape https://example.com/app-list.json --keep-different-domain
 
 # Durable config workflow
 html2rss capture https://example.com/news --write feed.yml

data/lib/html2rss/auto_source/README.md

@@ -10,8 +10,9 @@ Entry: `FeedPipeline` → `AutoSource#articles` → `Scraper.build_instance` →
 
 ## Live flow
 
-1. **Request** — `RequestSession` returns a `Response` with `body` (String) and `parsed_body` (Nokogiri HTML). Direct syndication responses skip HTML scrapers and parse via `Syndication::Parser`.
-2. **Scraper tiers** — Enabled scrapers that claim the page (shallow `articles?` or instance `extractable?`) run in `Scraper::SCRAPER_TIERS` order. Merge within a tier, then stop when enough articles survive Cleanup:
+1. **Request** — `RequestSession` returns a `Response` with `body` (String) and `parsed_body` (Nokogiri HTML, or parsed JSON for JSON media types). Direct syndication responses skip HTML scrapers and parse via `Syndication::Parser`.
+2. **JSON document body** — When the response is a JSON media type (`Response#json_response?`), AutoSource runs `JsonDocument` only (no HTML tiers). Scalar JSON is `unsupported_surface`. Same-domain Cleanup stays on by default (`keep_different_domain: false`). If the scraper yielded items and Cleanup kept none, `Log.warn` emits `auto_source.admission_emptied` with drop tallies; a `different_domain` tally hints at `auto_source.cleanup.keep_different_domain: true` / `--keep-different-domain`, which lets the remote source control off-domain link targets (no affiliation or safety check). Guidance stays in help/docs and that structured warning — stdout feed output stays composable.
+3. **Scraper tiers** (HTML only) — Enabled scrapers that claim the page (shallow `articles?` or instance `extractable?`) run in `Scraper::SCRAPER_TIERS` order. Merge within a tier, then stop when enough articles survive Cleanup:
 
    0. Native feed: NativeFeed (head alternates + path discovery → RSS/Atom parse)
    1. In-page structured: Schema, Microdata, Microformats2, JsonState, XhrArticles
@@ -19,14 +20,14 @@ Entry: `FeedPipeline` → `AutoSource#articles` → `Scraper.build_instance` →
    3. Heuristic: SemanticHtml
    4. Heuristic: Html (skipped when earlier tiers already admitted at least one clean article)
 
-3. **Structured / API scrapers** — Schema, Microdata, MF2, JSON state, XHR JSON, oEmbed, WordPress REST, and Sitemap work on Nokogiri CSS/XPath or JSON/XML parsers. They do not use SST.
-4. **Heuristic scrapers** — `SemanticHtml` and `Html` normalize once into an `SST::Document`, then:
+4. **Structured / API scrapers** — Schema, Microdata, MF2, JSON state, XHR JSON, oEmbed, WordPress REST, and Sitemap work on Nokogiri CSS/XPath or JSON/XML parsers. They do not use SST. `JsonDocument` walks a JSON response body (JSON Feed mapping or `JsonState.discover_articles`).
+5. **Heuristic scrapers** — `SemanticHtml` and `Html` normalize once into an `SST::Document`, then:
 
    `SST::Normalizer` → `AutoSource::Segmenter` → `Scoring::Engine` → extractor / article materialization.
 
-5. **Cleanup** — Merge, dedupe, hard-exclude non-article destinations (via `PathClassifier` facts), drop junk titles, and trim to `limit`. Html is skipped when earlier tiers already admitted clean items.
+6. **Cleanup** — Merge, dedupe, hard-exclude non-article destinations (via `PathClassifier` facts), drop junk titles, and trim to `limit`. Html is skipped when earlier tiers already admitted clean items.
 
-6. **Entry URL resolution** (pipeline, not this class) — when AutoFallback sees a weak homepage extract, `FeedResolution` may rewrite the scrape URL to a listing/feed before escalating strategies. See {Html2rss::FeedPipeline} and {Html2rss::FeedResolution}.
+7. **Entry URL resolution** (pipeline, not this class) — when AutoFallback sees a weak homepage extract, `FeedResolution` may rewrite the scrape URL to a listing/feed before escalating strategies. See {Html2rss::FeedPipeline} and {Html2rss::FeedResolution}.
 
 Segmenter strategies: `:semantic` (leaf containers + primary link), `:list` (cards grouped by parent; `main` is a filter), `:cluster` (class/structure grids for anchorless cards). Scoring ranks and demotes; `LinkDestination::NoisePolicy` owns content-anchor eligibility. Cleanup owns feed-item admission.
 
@@ -34,7 +35,8 @@ Segmenter strategies: `:semantic` (leaf containers + primary link), `:list` (car
 
 | Surface                                                                                       | Owns DOM                                                         |
 | --------------------------------------------------------------------------------------------- | ---------------------------------------------------------------- |
-| `Response#parsed_body`                                                                        | Single HTML parse for the page                                   |
+| `Response#parsed_body`                                                                        | HTML document, or Hash/Array for JSON bodies                     |
+| `JsonDocument`                                                                                | Parsed JSON Hash/Array (no Nokogiri)                             |
 | Schema / Microdata / MF2 / JsonState / XhrArticles / MetaOembed / app-shell detection         | Nokogiri                                                         |
 | Sitemap detection (CSS/XPath)                                                                 | Nokogiri; URL list parsing uses raw `response.body` (XML string) |
 | Selectors path / Sanitize transformers                                                        | Nokogiri (unchanged)                                             |
@@ -51,7 +53,8 @@ Production heuristic scrapers should reuse one `SST::Document` memoized from `pa
 
 ## Non-goals
 
-- Replacing Selectors, Schema, Microdata, MF2, JsonState, or XhrArticles with SST.
+- Replacing Selectors, Schema, Microdata, MF2, JsonState, JsonDocument, or XhrArticles with SST.
+- Turning `keep_different_domain` on automatically for JSON documents (operators must opt in explicitly; the flag trusts the source for any published link domain).
 - A second HTML parser beside Nokogiri.
 - Dual Response payload (Nokogiri + SST always).
 - App-shell classification on SST.

data/lib/html2rss/auto_source/cleanup.rb

@@ -63,10 +63,28 @@ module Html2rss
       # Read-only ship-quality audit on RSS/article-like items (no mutation).
       AuditResult = Data.define(:warnings, :metrics, :violations)
 
+      # Config key / CLI flag named when different_domain drops emptied admission.
+      # Opt-in trusts the remote source for off-domain link targets (no affiliation check).
+      DIFFERENT_DOMAIN_HINT =
+        'auto_source.cleanup.keep_different_domain: true (or --keep-different-domain); ' \
+        'allows published items to link to any domain supplied by the source'
+
       class << self
+        # Mitigating config hint when admission drops emptied a non-empty capture.
+        # Other drop reasons remain counts only.
+        #
+        # @param drop_tallies [Hash{String => Integer}]
+        # @return [String, nil]
+        def admission_hint(drop_tallies)
+          return unless drop_tallies.fetch('different_domain', 0).positive?
+
+          DIFFERENT_DOMAIN_HINT
+        end
+
         # @param articles [Array<Article>] extracted article candidates
         # @param url [Html2rss::Url] feed source URL used for same-host filtering
-        # @param keep_different_domain [Boolean] whether to keep off-domain entries
+        # @param keep_different_domain [Boolean] when true, keep items whose URLs are on a
+        #   different domain than +url+ (source-controlled off-domain links; default false)
         # @return [Result] cleaned articles and frozen drop tallies
         def call(articles, url:, keep_different_domain: DEFAULT_CONFIG.fetch(:keep_different_domain)) # rubocop:disable Metrics/MethodLength -- ordered reject pipeline
           Log.debug "Cleanup: start with #{articles.size} articles"

data/lib/html2rss/auto_source/scraper/json_document.rb

@@ -0,0 +1,135 @@
+# frozen_string_literal: true
+
+module Html2rss
+  class AutoSource
+    module Scraper
+      # Extracts articles from a parsed JSON response body (Hash or Array).
+      # JSON Feed documents map item fields directly; every other document walks
+      # via {JsonState.discover_articles}. Does not scan HTML.
+      class JsonDocument
+        include Enumerable
+
+        # Exact JSON Feed 1.0 / 1.1 +version+ URLs (no substring host matching).
+        JSON_FEED_VERSIONS = Set[
+          'https://jsonfeed.org/version/1',
+          'https://jsonfeed.org/version/1.1'
+        ].freeze
+
+        # @return [Symbol] scraper config key
+        def self.options_key = :json_document
+
+        # @param _opts [Hash] unused scraper options
+        # @return [Integer] zero — no additional HTTP requests
+        def self.request_slots(_opts = {}) = 0
+
+        # Not detectable from HTML alone; instance {#extractable?} requires a
+        # Hash or Array body (wired by AutoSource for JSON responses).
+        #
+        # @param _parsed_body [Object] unused
+        # @return [Boolean]
+        def self.articles?(_parsed_body) = false
+
+        # @param parsed_body [Hash, Array, Object] parsed JSON document (or HTML when misrouted)
+        # @param url [String, Html2rss::Url] base URL for relative link resolution
+        # @param _opts [Hash] scraper-specific options
+        # @option _opts [Object] :_reserved reserved for future scraper-specific options
+        def initialize(parsed_body, url:, **_opts)
+          @parsed_body = parsed_body
+          @url = url
+        end
+
+        # @return [Boolean] true when +parsed_body+ is a Hash or Array
+        def extractable?
+          @parsed_body.is_a?(Hash) || @parsed_body.is_a?(Array)
+        end
+
+        # @yield [Hash{Symbol => Object}] normalized article hash
+        # @return [Enumerator, void] article enumerator when no block is given
+        def each(&block)
+          return enum_for(:each) unless block
+          return unless extractable?
+
+          if json_feed?
+            each_json_feed_item(&block)
+          else
+            JsonState.discover_articles(@parsed_body, base_url: @url) do |article|
+              yield(article) if article
+            end
+          end
+        end
+
+        private
+
+        def json_feed?
+          return false unless @parsed_body.is_a?(Hash)
+
+          JSON_FEED_VERSIONS.include?(@parsed_body[:version].to_s) && @parsed_body[:items].is_a?(Array)
+        end
+
+        def each_json_feed_item
+          @parsed_body[:items].each do |item|
+            article = map_json_feed_item(item)
+            yield article if article
+          end
+        end
+
+        def map_json_feed_item(item)
+          return unless item.is_a?(Hash)
+
+          article_url = resolve_json_feed_url(item)
+          return unless article_url
+
+          json_feed_article_hash(item, article_url)
+        end
+
+        def json_feed_article_hash(item, article_url)
+          {
+            title: string(item[:title]),
+            description: json_feed_description(item),
+            url: article_url,
+            image: resolve_optional_url(item[:image]),
+            published_at: string(item[:date_published]),
+            categories: json_feed_tags(item[:tags]),
+            id: string(item[:id]) || article_url.to_s
+          }.compact
+        end
+
+        def resolve_json_feed_url(item)
+          resolve_optional_url(item[:url]) || absolute_url(item[:id])
+        end
+
+        def resolve_optional_url(value)
+          str = string(value)
+          return unless str
+
+          Url.from_relative(str, @url)
+        rescue ArgumentError
+          nil
+        end
+
+        def absolute_url(value)
+          str = string(value)
+          return unless str
+
+          Url.from_absolute(str)
+        rescue ArgumentError
+          nil
+        end
+
+        def json_feed_description(item)
+          string(item[:content_text]) || string(item[:content_html])
+        end
+
+        def json_feed_tags(tags)
+          return unless tags.is_a?(Array)
+
+          names = tags.filter_map { |tag| string(tag) }
+          names.uniq!
+          names unless names.empty?
+        end
+
+        def string(value) = JsonState::ArticleNormalizer.string(value)
+      end
+    end
+  end
+end

data/lib/html2rss/auto_source/scraper/json_state/article_normalizer.rb

@@ -5,15 +5,15 @@ module Html2rss
     module Scraper
       class JsonState
         # Shapes raw entries into the structure required downstream.
-        module ArticleNormalizer
+        module ArticleNormalizer # rubocop:disable Metrics/ModuleLength -- field keys + URL/scalar unwrap stay co-located
           # Preferred keys when extracting title-like values from state payloads.
           TITLE_KEYS = %i[title headline name text].freeze
           # Preferred keys when extracting URL-like values from state payloads.
-          URL_KEYS = %i[url link href permalink slug path canonicalUrl shortUrl].freeze
+          URL_KEYS = %i[url link href permalink slug path canonicalUrl shortUrl alias self].freeze
           # Preferred keys when extracting description-like values from state payloads.
-          DESCRIPTION_KEYS = %i[description summary excerpt dek subheading].freeze
+          DESCRIPTION_KEYS = %i[description summary excerpt dek subheading abstract].freeze
           # Preferred keys when extracting image-like values from state payloads.
-          IMAGE_KEYS = %i[image imageUrl thumbnailUrl thumbnail src featuredImage coverImage heroImage].freeze
+          IMAGE_KEYS = %i[image imageUrl thumbnailUrl thumbnail src featuredImage coverImage heroImage icon].freeze
           # Preferred keys when extracting publication timestamps from state payloads.
           PUBLISHED_AT_KEYS = %i[published_at publishedAt datePublished date publicationDate pubDate updatedAt
                                  updated_at createdAt created_at].freeze
@@ -21,6 +21,12 @@ module Html2rss
           CATEGORY_KEYS = %i[categories tags section sections topic topics channel].freeze
           # Preferred keys when extracting identifier-like values from state payloads.
           ID_KEYS = %i[id guid uuid slug key].freeze
+          # Max times to unwrap a Hash URL value (e.g. path.alias, links.self.href).
+          MAX_URL_UNWRAP = 3
+          # Nested title/description keys (WordPress +rendered+, CMS +value+/+text+).
+          SCALAR_KEYS = %i[rendered value text].freeze
+          # Snake `_at`/`_on` or camel `At`/`On` suffix after a closed published-at miss.
+          DATE_KEY_SUFFIX = /_at\z|_on\z|[a-z]At\z|[a-z]On\z/
 
           module_function
 
@@ -44,37 +50,94 @@ module Html2rss
               url: article_url,
               image: resolve_link(entry, keys: IMAGE_KEYS, base_url:,
                                          log_key: 'JsonState: invalid image URL encountered'),
-              published_at: string(ValueFinder.fetch(entry, PUBLISHED_AT_KEYS)),
+              published_at: published_at(entry),
               categories: categories(entry),
               id: identifier(entry, article_url)
             }.compact
           end
           # rubocop:enable Metrics/MethodLength
 
-          # @param value [Object] candidate scalar value
-          # @return [String, nil] normalized non-empty string value
+          # @param value [Object] candidate scalar, or Hash wrapping +rendered+/+value+/+text+
+          # @return [String, nil] normalized non-empty string (HTML preserved for descriptions)
           def string(value)
+            value = unwrap_scalar(value)
+            return unless value.is_a?(String) || value.is_a?(Numeric)
+
             trimmed = value.to_s.strip
             trimmed unless trimmed.empty?
           end
 
+          # @param value [Object] candidate scalar or nested Hash
+          # @return [Object, nil] String/Numeric after at most {MAX_URL_UNWRAP} Hash unwraps
+          def unwrap_scalar(value)
+            depth = 0
+            while value.is_a?(Hash) && depth < MAX_URL_UNWRAP
+              nested = SCALAR_KEYS.find { |key| value.key?(key) }
+              break unless nested
+
+              value = value[nested]
+              depth += 1
+            end
+            value unless value.is_a?(Hash) || value.is_a?(Array)
+          end
+
           # @param entry [Hash] raw article entry candidate
-          # @param keys [Array<String>] preferred link keys
+          # @param keys [Array<Symbol>] preferred link keys
           # @param base_url [String, Html2rss::Url] base URL for relative link resolution
           # @param log_key [String] structured log message key
           # @return [Html2rss::Url, nil] resolved absolute URL
           def resolve_link(entry, keys:, base_url:, log_key:)
-            value = ValueFinder.fetch(entry, keys)
-            value = ValueFinder.fetch(value, keys) if value.is_a?(Hash)
+            value = unwrap_url(ValueFinder.fetch(entry, keys), keys) ||
+                    unwrap_url(ValueFinder.fetch(entry[:links], keys), keys)
             string = string(value)
             return unless string
 
             Url.from_relative(string, base_url)
           rescue ArgumentError
-            Log.debug(log_key, url: string)
+            Log.debug("#{log_key}: url=#{string}")
             nil
           end
 
+          # @param value [Object] candidate URL value, possibly a nested Hash
+          # @param keys [Array<Symbol>] preferred link keys
+          # @return [Object, nil] scalar URL candidate after at most {MAX_URL_UNWRAP} Hash unwraps
+          def unwrap_url(value, keys)
+            depth = 0
+            while value.is_a?(Hash) && depth < MAX_URL_UNWRAP
+              value = ValueFinder.fetch(value, keys)
+              depth += 1
+            end
+            value unless value.is_a?(Hash)
+          end
+
+          # @param entry [Hash] raw article entry candidate
+          # @return [String, nil] publication timestamp string or unix timestamp digits
+          def published_at(entry)
+            value = ValueFinder.fetch(entry, PUBLISHED_AT_KEYS)
+            string(value) || string(suffix_date_value(entry))
+          end
+
+          # @param hash [Hash, nil] entry or nested attributes/data hash
+          # @return [Object, nil] first string or numeric value whose key matches {DATE_KEY_SUFFIX}
+          def suffix_date_value(hash)
+            return unless hash.is_a?(Hash)
+
+            hash.each do |key, value|
+              next unless date_suffix_key?(key)
+              next unless value.is_a?(String) || value.is_a?(Numeric)
+
+              return value
+            end
+
+            suffix_date_value(hash[:attributes]) || suffix_date_value(hash[:data])
+          end
+
+          # @param key [Symbol, String] candidate published-at key
+          # @return [Boolean] whether the key ends in a date suffix (_at/_on/At/On)
+          def date_suffix_key?(key)
+            key.to_s.match?(DATE_KEY_SUFFIX)
+          end
+
           # rubocop:disable Metrics/MethodLength
           # @param entry [Hash] raw article entry candidate
           # @return [Array<String>, nil] normalized unique categories

data/lib/html2rss/auto_source/scraper/json_state/candidate_detector.rb

@@ -6,11 +6,6 @@ module Html2rss
       class JsonState
         # Identifies arrays that look like collections of article hashes.
         module CandidateDetector
-          # Preferred keys when extracting title-like values from state payloads.
-          TITLE_KEYS = %i[title headline name text].freeze
-          # Preferred keys when extracting URL-like values from state payloads.
-          URL_KEYS = %i[url link href permalink slug path canonicalUrl shortUrl].freeze
-
           module_function
 
           # @param document [Hash, Array, Object] candidate document node
@@ -48,13 +43,14 @@ module Html2rss
           # @param object [Hash] article candidate object
           # @return [Object, nil] detected title-like value
           def title_from(object)
-            ValueFinder.fetch(object, TITLE_KEYS)
+            ValueFinder.fetch(object, ArticleNormalizer::TITLE_KEYS)
           end
 
           # @param object [Hash] article candidate object
-          # @return [Object, nil] detected URL-like value
+          # @return [Object, nil] detected URL-like value (including links.self)
           def url_from(object)
-            ValueFinder.fetch(object, URL_KEYS)
+            ValueFinder.fetch(object, ArticleNormalizer::URL_KEYS) ||
+              ValueFinder.fetch(object[:links], ArticleNormalizer::URL_KEYS)
           end
         end
       end

data/lib/html2rss/auto_source/scraper.rb

@@ -32,8 +32,11 @@ module Html2rss
         [Html].freeze
       ].freeze
 
+      # JSON response body scrapers (routed by AutoSource before SCRAPER_TIERS).
+      DOCUMENT_SCRAPERS = [JsonDocument].freeze
+
       # Flat ordered list (request slot accounting, detection helpers).
-      SCRAPERS = SCRAPER_TIERS.flatten.freeze
+      SCRAPERS = (DOCUMENT_SCRAPERS + SCRAPER_TIERS.flatten).freeze
 
       # Heuristic scrapers that share one memoized SST::Document per page.
       HEURISTIC_SCRAPERS = Set[SemanticHtml, Html].freeze
@@ -51,8 +54,8 @@ module Html2rss
                            'Target a direct listing URL, configure BOTASAURUS_SCRAPER_URL, ' \
                            'or run from an environment that can complete anti-bot checks.',
           app_shell: 'app-shell surface detected (client-rendered page with little or no ' \
-                     'server-rendered article HTML). Configure BOTASAURUS_SCRAPER_URL or target a direct ' \
-                     'listing/update URL instead of a homepage or shell entrypoint.',
+                     'server-rendered article HTML). Target a direct listing/update URL instead of a ' \
+                     'homepage or shell entrypoint; a browser strategy does not add extractable markup here.',
           high_entropy_surface: 'high-entropy surface (e.g. news homepage) with few or no durable ' \
                                 'article destinations after admission. Target a section, topic, or ' \
                                 'listing/update URL instead of the site homepage.',
@@ -91,6 +94,7 @@ module Html2rss
       # @option opts [Hash] :microdata scraper toggle and configuration
       # @option opts [Hash] :microformats2 scraper toggle and configuration
       # @option opts [Hash] :json_state scraper toggle and configuration
+      # @option opts [Hash] :json_document scraper toggle and configuration
       # @option opts [Hash] :xhr_articles scraper toggle and configuration
       # @option opts [Hash] :meta_oembed scraper toggle and configuration
       # @option opts [Hash] :semantic_html scraper toggle and configuration
@@ -139,6 +143,7 @@ module Html2rss
       # @option opts [Hash] :microdata scraper toggle and configuration
       # @option opts [Hash] :microformats2 scraper toggle and configuration
       # @option opts [Hash] :json_state scraper toggle and configuration
+      # @option opts [Hash] :json_document scraper toggle and configuration
       # @option opts [Hash] :xhr_articles scraper toggle and configuration
       # @option opts [Hash] :meta_oembed scraper toggle and configuration
       # @option opts [Hash] :semantic_html scraper toggle and configuration

data/lib/html2rss/auto_source.rb

@@ -61,6 +61,9 @@ module Html2rss
         json_state: {
           enabled: true
         },
+        json_document: {
+          enabled: true
+        },
         xhr_articles: {
           enabled: true
         },
@@ -113,6 +116,7 @@ module Html2rss
       @parsed_body = response.parsed_body
       @body = response.body
       @url = response.url
+      @json_response = response.json_response?
       @captured_responses = response.captured_responses
       @opts = opts
       @request_session = request_session
@@ -131,7 +135,12 @@ module Html2rss
     def articles
       @articles ||= extract_articles
     rescue Html2rss::AutoSource::Scraper::NoScraperFound => error
-      Log.warn "#{self.class}: no scraper matched #{url} (#{error.message})"
+      Log.warn({
+        event_name: 'auto_source.no_scraper',
+        host: url.host,
+        category: error.category,
+        message: "#{self.class}: no scraper matched #{url} (#{error.message})"
+      }.to_json)
       @admission_drops = {}.freeze
       []
     end
@@ -150,6 +159,8 @@ module Html2rss
 
     # rubocop:disable-next Metrics/AbcSize, Metrics/CyclomaticComplexity, Metrics/MethodLength, Metrics/PerceivedComplexity
     def extract_articles
+      return extract_json_document_articles if json_response?
+
       articles = []
       matched = false
       document = nil
@@ -189,11 +200,55 @@ module Html2rss
 
       raise Scraper.no_scraper_found_for(parsed_body, body:) unless matched
 
+      finish_articles(articles)
+    end
+
+    def json_response?
+      @json_response
+    end
+
+    def extract_json_document_articles
+      instance = json_document_instance
+      raise Scraper::NoScraperFound.new(category: :unsupported_surface) unless instance&.extractable?
+
+      articles = run_scraper(instance)
+      raise Scraper::NoScraperFound.new(category: :unsupported_surface) if articles.empty?
+
+      finish_articles(articles)
+    end
+
+    def json_document_instance
+      Scraper.build_instance(
+        Scraper::JsonDocument,
+        parsed_body,
+        opts: @opts.fetch(:scraper, {}),
+        url:,
+        request_session:,
+        body:,
+        captured_responses:
+      )
+    end
+
+    def finish_articles(articles)
+      captured_count = articles.size
       result = Cleanup.call(articles, url:, **cleanup_options)
       @admission_drops = result.drop_tallies
+      log_admission_emptied(captured_count, result) if captured_count.positive? && result.articles.empty?
       result.articles.first(article_limit)
     end
 
+    def log_admission_emptied(captured_count, result)
+      payload = {
+        event_name: 'auto_source.admission_emptied',
+        host: url.host,
+        captured_count:,
+        drop_tallies: result.drop_tallies
+      }
+      hint = Cleanup.admission_hint(result.drop_tallies)
+      payload[:hint] = hint if hint
+      Log.warn(payload.to_json)
+    end
+
     def enough_articles?(articles)
       return false if articles.size < article_limit
 

data/lib/html2rss/cli.rb

@@ -213,6 +213,9 @@ module Html2rss
     desc 'scrape [URL]', 'Automatically source an RSS feed from a URL (one-shot auto-source)'
     method_option :items_selector, type: :string, desc: 'CSS selector hint for items'
     method_option :limit, type: :numeric, desc: 'Max articles (auto-source)'
+    method_option :keep_different_domain, type: :boolean, default: false,
+                                          desc: 'Allow published items to link to any domain ' \
+                                                'supplied by the source (no affiliation/safety check)'
     # @param url [String, nil]
     # @return [void]
     def scrape(url = nil)
@@ -430,7 +433,8 @@ module Html2rss
         max_redirects: options[:max_redirects],
         max_requests: options[:max_requests],
         local_file_path:,
-        limit: options[:limit]&.to_i
+        limit: options[:limit]&.to_i,
+        keep_different_domain: options[:keep_different_domain]
       )
     end
 

data/lib/html2rss/config/auto_source_contract.rb

@@ -36,6 +36,9 @@ module Html2rss
         optional(:json_state).hash do
           optional(:enabled).filled(:bool)
         end
+        optional(:json_document).hash do
+          optional(:enabled).filled(:bool)
+        end
         optional(:xhr_articles).hash do
           optional(:enabled).filled(:bool)
         end

data/lib/html2rss/config.rb

@@ -125,10 +125,18 @@ module Html2rss
       # @param items_selector [String, nil] optional selector hint for item extraction
       # @param request_controls [Html2rss::Config::RequestControls, nil] explicit request controls to write
       # @param limit [Integer, nil] max articles to keep in the auto-sourced feed
+      # @param keep_different_domain [Boolean, nil] when set, overrides cleanup.keep_different_domain
       # @return [Hash{Symbol => Object}] feed config hash ready for {from_hash}
-      def auto_source_config(url:, items_selector: nil, request_controls: nil, limit: nil)
+      # rubocop:disable-next Metrics/MethodLength -- keep_different_domain override stays co-located
+      def auto_source_config(url:, items_selector: nil, request_controls: nil, limit: nil,
+                             keep_different_domain: nil)
         auto_source = AutoSource::DEFAULT_CONFIG
         auto_source = auto_source.merge(limit:) unless limit.nil?
+        unless keep_different_domain.nil?
+          auto_source = auto_source.merge(
+            cleanup: auto_source[:cleanup].merge(keep_different_domain:)
+          )
+        end
 
         config = {
           channel: default_config[:channel].merge(url:),

data/lib/html2rss/error.rb

@@ -2,7 +2,13 @@
 
 module Html2rss
   # The Html2rss::Error base class.
-  class Error < StandardError; end
+  class Error < StandardError
+    # @return [Array<Hash{Symbol => Object}>, nil] tier attempt diagnostics when set by AutoFallback
+    attr_accessor :attempts
+
+    # @return [Hash{String => Object}, nil] allowlisted upstream transport telemetry when attached
+    attr_accessor :transport_meta
+  end
 
   # Raised when auto fallback exhausts all concrete tiers and extractors find no feed items.
   class NoFeedItemsExtracted < Error

data/lib/html2rss/feed_pipeline/auto_fallback.rb

@@ -1,5 +1,7 @@
 # frozen_string_literal: true
 
+require 'json'
+
 module Html2rss
   class FeedPipeline
     # Retries feed extraction across concrete request strategies for the :auto plan.
@@ -28,12 +30,13 @@ module Html2rss
       ##
       # Mutable run state for one auto-fallback chain: attempts plus selected result.
       class AttemptState
-        attr_reader :attempts, :result, :last_response, :entry_resolution
+        attr_reader :attempts, :result, :last_response, :last_error, :entry_resolution
 
         def initialize
           @attempts = []
           @result = nil
           @last_response = nil
+          @last_error = nil
           @entry_resolution = nil
           @resolution_tried = false
           @aborted = false
@@ -68,7 +71,11 @@ module Html2rss
         # @param error [Exception] caught error
         # @return [void]
         def record_error(strategy:, error:)
-          @attempts << { strategy:, items_count: nil, error_class: error.class.name }
+          attempt = { strategy:, items_count: nil, error_class: error.class.name }
+          meta = error.is_a?(Html2rss::Error) ? error.transport_meta : nil
+          attempt[:transport_meta] = meta if meta&.any?
+          @attempts << attempt
+          @last_error = error
         end
 
         # @param strategy [Symbol] strategy that returned a response
@@ -76,6 +83,7 @@ module Html2rss
         # @param transport_meta [Hash, nil] optional allowlisted upstream telemetry
         # @return [void]
         def record_items(strategy:, items_count:, transport_meta: nil)
+          @last_error = nil
           attempt = { strategy:, items_count:, error_class: nil }
           attempt[:transport_meta] = transport_meta if transport_meta && !transport_meta.empty?
           @attempts << attempt
@@ -134,7 +142,7 @@ module Html2rss
         state = run_attempts
         return state.result if state.succeeded?
 
-        finalize_failure(attempts: state.attempts, response: state.last_response)
+        finalize_failure(state)
       end
 
       private
@@ -228,9 +236,24 @@ module Html2rss
                  "budget_remaining=#{budget_remaining_label}")
       end
 
-      def finalize_failure(attempts:, response:)
-        surface_category = surface_category_for(response)
-        raise NoFeedItemsExtracted.new(attempts:, surface_category:)
+      def finalize_failure(state)
+        if raise_last_error?(state)
+          error = state.last_error
+          error.attempts = state.attempts if error.is_a?(Html2rss::Error)
+          raise error
+        end
+
+        raise NoFeedItemsExtracted.new(
+          attempts: state.attempts,
+          surface_category: surface_category_for(state.last_response)
+        )
+      end
+
+      def raise_last_error?(state)
+        error = state.last_error
+        return false if error.nil? || error.is_a?(RequestService::BotasaurusConfigurationError)
+
+        true
       end
 
       def surface_category_for(response)
@@ -248,8 +271,15 @@ module Html2rss
           Log.info("#{self.class}: auto fallback #{strategy} -> #{next_strategy} " \
                    "after timeout=#{error.class} #{detail}")
         else
-          Log.warn("#{self.class}: auto fallback #{strategy} -> #{next_strategy} " \
-                   "after error=#{error.class} #{detail}")
+          Log.warn({
+            event_name: 'auto_fallback.error',
+            host:,
+            strategy: strategy.to_s,
+            next_strategy: next_strategy.to_s,
+            error_class: error.class.name,
+            message: "#{self.class}: auto fallback #{strategy} -> #{next_strategy} " \
+                     "after error=#{error.class} #{detail}"
+          }.to_json)
         end
         Log.debug("#{self.class}: strategy=#{strategy} error=#{error.class}: #{error.message} #{detail}")
       end

data/lib/html2rss/feed_resolution.rb

@@ -1,5 +1,7 @@
 # frozen_string_literal: true
 
+require 'json'
+
 module Html2rss
   ##
   # Entry URL tournament for weak homepage/hub extracts.
@@ -120,12 +122,12 @@ module Html2rss
 
         nil
       rescue RequestService::RequestBudgetExceeded
-        Log.warn('FeedResolution: entry resolution skipped (request budget exhausted)')
+        log_budget_exhausted
         nil
       rescue *NON_FALLBACK_ERRORS
         raise
       rescue StandardError => error
-        Log.warn("FeedResolution: entry resolution retry failed (#{error.class})")
+        log_retry_failed(error)
         nil
       end
 
@@ -134,6 +136,32 @@ module Html2rss
       attr_reader :pipeline, :config, :response, :session, :strategy, :resources, :articles,
                   :scrape_target, :state, :budget, :assessment
 
+      def log_budget_exhausted
+        Log.warn({
+          event_name: 'feed_resolution.budget_exhausted',
+          host: channel_host,
+          message: 'FeedResolution: entry resolution skipped (request budget exhausted)'
+        }.to_json)
+      end
+
+      def log_retry_failed(error)
+        Log.warn({
+          event_name: 'feed_resolution.retry_failed',
+          host: channel_host,
+          error_class: error.class.name,
+          message: "FeedResolution: entry resolution retry failed (#{error.class.name})"
+        }.to_json)
+      end
+
+      # +Config#url+ is a String; parse once for structured logs.
+      #
+      # @return [String, nil]
+      def channel_host
+        Url.from_absolute(config.url.to_s).host
+      rescue ArgumentError
+        nil
+      end
+
       def eligible?
         return false if state.resolution_tried?
         return false if response.feed_response?

data/lib/html2rss/mcp/contract.rb

@@ -11,7 +11,7 @@ module Html2rss
       # Published MCP request strategies (excludes +local_file+).
       STRATEGIES = %w[auto default httpx botasaurus].freeze
       # Accepted migration strategies retained for backwards compatibility.
-      MIGRATION_STRATEGIES = %w[faraday].freeze
+      MIGRATION_STRATEGIES = %w[faraday browserless].freeze
       # Complete set of strategies accepted by {.assert_published_request!}.
       ALL_ACCEPTED_STRATEGIES = (STRATEGIES + MIGRATION_STRATEGIES).freeze
 

data/lib/html2rss/request_service/blocked_surface.rb

@@ -47,6 +47,17 @@ module Html2rss
           message: 'Blocked surface detected: Vercel Security Checkpoint. ' \
                    'This site is a JS-rendered SPA behind Vercel edge protection. ' \
                    'Configure BOTASAURUS_SCRAPER_URL or target a direct listing URL.'
+        },
+        {
+          key: :http_403_forbidden,
+          min_matches: 2,
+          patterns: [
+            %r{<title>\s*403\s+forbidden\s*</title>}i,
+            %r{<h1>\s*403\s+forbidden\s*</h1>}i
+          ],
+          message: 'Blocked surface detected: HTTP 403 Forbidden access block page. ' \
+                   'Target a direct listing URL, configure BOTASAURUS_SCRAPER_URL, ' \
+                   'or run from an environment that is not IP-blocked.'
         }
       ].freeze
 

data/lib/html2rss/request_service/botasaurus_contract.rb

@@ -198,6 +198,11 @@ module Html2rss
           "Botasaurus scrape failed (#{details.join(', ')})."
         end
 
+        # @return [Hash{String => Object}] allowlisted diagnostics plus error_category (frozen)
+        def transport_meta
+          diagnostics.merge('error_category' => error_category).freeze
+        end
+
         private
 
         attr_reader :payload, :transport_status

data/lib/html2rss/request_service/botasaurus_strategy.rb

@@ -49,14 +49,16 @@ module Html2rss
       end
 
       def raise_from_error!(error)
-        raise BlockedSurfaceDetected, "Blocked surface detected: #{error.challenge_message}" if error.challenge_block?
-
-        if error.timeout?
-          log_timeout!(reason: 'botasaurus_upstream')
-          raise RequestTimedOut.new(error.failure_message, timeout_phase: error.timeout_phase)
-        end
-
-        raise BotasaurusServiceError, error.failure_message
+        err = if error.challenge_block?
+                BlockedSurfaceDetected.new("Blocked surface detected: #{error.challenge_message}")
+              elsif error.timeout?
+                log_timeout!(reason: 'botasaurus_upstream')
+                RequestTimedOut.new(error.failure_message, timeout_phase: error.timeout_phase)
+              else
+                BotasaurusServiceError.new(error.failure_message)
+              end
+        err.transport_meta = error.transport_meta
+        raise err
       end
 
       def response_url(final_url)

data/lib/html2rss/request_service/httpx_strategy.rb

@@ -60,8 +60,18 @@ module Html2rss
 
       private
 
+      # Caps one HTTPX execute (including internal MAX_RETRIES) so a hanging
+      # :default tier cannot consume the full shared auto-fallback budget.
+      #
+      # @return [Float] monotonic deadline
       def request_deadline
-        monotonic_now + ctx.budget.effective_timeout_seconds(fallback: ctx.policy.total_timeout_seconds)
+        budget_timeout = ctx.budget.effective_timeout_seconds(fallback: ctx.policy.total_timeout_seconds)
+        monotonic_now + [budget_timeout, attempt_timeout_ceiling].min
+      end
+
+      # @return [Float] connect + read ceiling for a single attempt
+      def attempt_timeout_ceiling
+        (ctx.policy.connect_timeout_seconds + ctx.policy.read_timeout_seconds).to_f
       end
 
       def reset_redirect_tracking!

data/lib/html2rss/request_service/response.rb

@@ -71,8 +71,11 @@ module Html2rss
       # @return [String] normalized content type header value
       def content_type = header('content-type').to_s
 
-      # @return [Boolean] whether response content is JSON
-      def json_response? = content_type.include?('application/json')
+      # @return [Boolean] whether the media type is JSON (`*/json` or `*+json`)
+      def json_response?
+        media_type = content_type.split(';', 2).first.to_s.strip.downcase
+        media_type.end_with?('/json', '+json')
+      end
 
       # @return [Boolean] whether response content is HTML (header or sniffed body, never JSON)
       def html_response?

data/lib/html2rss/request_service.rb

@@ -61,9 +61,17 @@ module Html2rss
       httpx: HttpxStrategy,
       faraday: HttpxStrategy,
       botasaurus: BotasaurusStrategy,
+      browserless: BotasaurusStrategy,
       local_file: LocalFileStrategy
     }.freeze
 
+    # Deprecated strategy aliases and the preferred replacement name for warnings.
+    # @return [Hash{Symbol => Symbol}]
+    MIGRATION_REPLACEMENTS = {
+      faraday: :default,
+      browserless: :botasaurus
+    }.freeze
+
     # Canonical default strategy symbol.
     # @return [Symbol]
     DEFAULT_STRATEGY_NAME = :default
@@ -90,23 +98,49 @@ module Html2rss
       # @return [Response] the response from the executed strategy.
       # @raise [UnknownStrategy] if the strategy is not registered.
       def execute(ctx, strategy: default_strategy_name)
-        strategy_sym = strategy.to_sym
+        strategy_sym = resolve_strategy_name(strategy, relation: ctx.relation, host: ctx.url.host)
         strategy_class = STRATEGIES.fetch(strategy_sym) do
           raise UnknownStrategy,
                 "The strategy '#{strategy}' is not known. Available strategies: #{strategy_names.join(', ')}"
         end
 
-        warn_migration_strategy(strategy_sym) if strategy_sym == :faraday
         strategy_class.new(ctx).execute
       end
 
       private
 
-      def warn_migration_strategy(strategy)
-        message = "RequestService: strategy ':#{strategy}' is deprecated for migration and will be removed " \
-                  "in a future release. Use ':default' instead."
-        warn(message, category: :deprecated)
-        Log.warn(message)
+      ##
+      # Resolves deprecated migration aliases to their replacement strategy name.
+      # @param strategy [Symbol, String] requested strategy name
+      # @param relation [Symbol] request relation (+ :initial + or follow-up)
+      # @param host [String, nil] request host for structured deprecation telemetry
+      # @return [Symbol] canonical strategy name after migration remapping
+      def resolve_strategy_name(strategy, relation:, host:)
+        strategy_sym = strategy.to_sym
+        replacement = MIGRATION_REPLACEMENTS[strategy_sym]
+        return strategy_sym unless replacement
+
+        warn_migration_strategy(strategy_sym, replacement, relation:, host:)
+        replacement
+      end
+
+      ##
+      # Emits a structured +strategy_deprecated+ warning once per session (+ :initial + only).
+      # @param strategy [Symbol] deprecated strategy alias
+      # @param replacement [Symbol] preferred strategy name
+      # @param relation [Symbol] request relation
+      # @param host [String, nil] request host
+      # @return [void]
+      def warn_migration_strategy(strategy, replacement, relation:, host:)
+        return unless relation == :initial
+
+        Log.warn({
+          event_name: 'strategy_deprecated',
+          strategy: strategy.to_s,
+          replacement: replacement.to_s,
+          host:,
+          message: "#{name}: strategy=#{strategy.inspect} is deprecated; using #{replacement.inspect} instead."
+        }.to_json)
       end
     end
   end

data/lib/html2rss/version.rb

@@ -2,6 +2,6 @@
 
 module Html2rss
   # Current application version.
-  VERSION = '0.31.0'
+  VERSION = '0.32.1'
   public_constant :VERSION
 end

data/lib/html2rss.rb

@@ -134,6 +134,7 @@ module Html2rss # rubocop:disable Metrics/ModuleLength
   # @param max_requests [Integer] optional request budget override (default: 4 for sitemap sub-fetches)
   # @param local_file_path [String, nil] optional local HTML file path
   # @param limit [Integer, nil] max articles to keep (default: {AutoSource::DEFAULT_LIMIT})
+  # @param keep_different_domain [Boolean, nil] when set, overrides cleanup.keep_different_domain
   # @return [Html2rss::FeedResult]
   def self.scrape(url,
                   strategy: :auto,
@@ -141,9 +142,10 @@ module Html2rss # rubocop:disable Metrics/ModuleLength
                   max_redirects: nil,
                   max_requests: 4,
                   local_file_path: nil,
-                  limit: nil)
+                  limit: nil,
+                  keep_different_domain: nil)
     auto_feed_result(url, strategy:, items_selector:, max_redirects:, max_requests:,
-                          local_file_path:, limit:)
+                          local_file_path:, limit:, keep_different_domain:)
   end
 
   ##
@@ -246,6 +248,7 @@ module Html2rss # rubocop:disable Metrics/ModuleLength
   # @param max_requests [Integer] optional request budget override (default: 4 for sitemap sub-fetches)
   # @param local_file_path [String, nil] optional local HTML file path
   # @param limit [Integer, nil] max articles to keep (default: {AutoSource::DEFAULT_LIMIT})
+  # @param keep_different_domain [Boolean, nil] when set, overrides cleanup.keep_different_domain
   # @return [Html2rss::FeedResult]
   def self.auto_feed_result(url,
                             strategy: :auto,
@@ -253,9 +256,10 @@ module Html2rss # rubocop:disable Metrics/ModuleLength
                             max_redirects: nil,
                             max_requests: 4,
                             local_file_path: nil,
-                            limit: nil)
+                            limit: nil,
+                            keep_different_domain: nil)
     feed_result(build_auto_source_config(url:, strategy:, items_selector:, max_redirects:, max_requests:,
-                                         local_file_path:, limit:))
+                                         local_file_path:, limit:, keep_different_domain:))
   end
 
   ##
@@ -355,12 +359,13 @@ module Html2rss # rubocop:disable Metrics/ModuleLength
 
     # rubocop:disable-next Metrics/ParameterLists, Metrics/MethodLength
     def build_auto_source_config(url:, strategy:, items_selector:, max_redirects:, max_requests:,
-                                 local_file_path: nil, limit: nil)
+                                 local_file_path: nil, limit: nil, keep_different_domain: nil)
       config = Config.auto_source_config(
         url:,
         items_selector:,
         request_controls: Config::RequestControls.from_shortcut(strategy:, max_redirects:, max_requests:),
-        limit:
+        limit:,
+        keep_different_domain:
       )
       if local_file_path
         config[:request] ||= {}

data/schema/html2rss-config.schema.json

@@ -281,6 +281,18 @@
               },
               "required": []
             },
+            "json_document": {
+              "type": "object",
+              "properties": {
+                "enabled": {
+                  "type": "boolean",
+                  "not": {
+                    "type": "null"
+                  }
+                }
+              },
+              "required": []
+            },
             "xhr_articles": {
               "type": "object",
               "properties": {
@@ -402,6 +414,9 @@
           "json_state": {
             "enabled": true
           },
+          "json_document": {
+            "enabled": true
+          },
           "xhr_articles": {
             "enabled": true
           },

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: html2rss
 version: !ruby/object:Gem::Version
-  version: 0.31.0
+  version: 0.32.1
 platform: ruby
 authors:
 - Gil Desmarais
@@ -275,6 +275,7 @@ files:
 - lib/html2rss/auto_source/cleanup.rb
 - lib/html2rss/auto_source/scraper.rb
 - lib/html2rss/auto_source/scraper/html.rb
+- lib/html2rss/auto_source/scraper/json_document.rb
 - lib/html2rss/auto_source/scraper/json_state.rb
 - lib/html2rss/auto_source/scraper/json_state/article_normalizer.rb
 - lib/html2rss/auto_source/scraper/json_state/candidate_detector.rb
@@ -477,7 +478,7 @@ licenses:
 - MIT
 metadata:
   allowed_push_host: https://rubygems.org
-  changelog_uri: https://github.com/html2rss/html2rss/releases/tag/v0.31.0
+  changelog_uri: https://github.com/html2rss/html2rss/releases/tag/v0.32.1
   rubygems_mfa_required: 'true'
 rdoc_options: []
 require_paths: