html2rss 0.14.0 → 0.16.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: a2ce9bbe8640372b5e98672760d76aee5f6f23373dd4b22ca067d2cdaa6f2b15
-  data.tar.gz: ff280d9466ee6b15b1149f582dadf9b209f0e99e4fb02e6b82f91b25a7ca0b7a
+  metadata.gz: 3a33d49918c1e75268b3b314908305986dd863ccf31dbe1b6ace8202d3a652de
+  data.tar.gz: 7121a463570c62ffdddb85b9c0d7ba098bdc784ad6649b6bb34b232125a9bf49
 SHA512:
-  metadata.gz: d516b897253374425ccd3b26d21362df46c18c26694fe1a8aaddc06b956f93e36111d3310dc635b84d7626a2072014d27dacb075cd98d15a79d39aed40991bcb
-  data.tar.gz: 91ae4190d04967c1bc9d3f46b0a0bdbbd23f3dbda6559c3ff10391ab0d63d4b984a44789c8427dbace345af617a33e86c2f441c0a0d58dbcf2b734bd78b73b87
+  metadata.gz: e77faa7bd81f63894f0001157a81e858a67eae502f5a5f3ecd41790f48c630b3ab94cd281531cb0a8cf0a7693fa03c7d6176edde04d063804f69f15d4b7469f3
+  data.tar.gz: d84a132a76997336f840c7b5e9261ab304ecfb302628b10d839350fbe3de92919bdc3321b5f374231c703202560d7b392da51cfbcb66ae8e18debdcd69d9e3d0

data/README.md

@@ -10,21 +10,11 @@ With the _feed config_, you provide a URL to scrape and CSS selectors for extrac
 
 Support the development by sponsoring this project on GitHub. Thank you! 💓
 
-## Installation
-
-| Install | `gem install html2rss` |
-| ------- | ---------------------- |
-| Usage   | `html2rss help`        |
-
-You can also install it as a dependency in your Ruby project:
+## Generating a feed on the CLI
 
-|                      🤩 Like it? | Star it! ⭐️         |
-| -------------------------------: | -------------------- |
-| Add this line to your `Gemfile`: | `gem 'html2rss'`     |
-|                    Then execute: | `bundle`             |
-|                    In your code: | `require 'html2rss'` |
+[Install Ruby](https://www.ruby-lang.org/en/documentation/installation/) (latest version is recommended) on your machine and run `gem install html2rss` in your terminal.
 
-## Generating a feed on the CLI
+After the installation has finished, `html2rss help` will print usage information.
 
 ### using automatic generation
 
@@ -59,6 +49,14 @@ Build the feed from this config with: `html2rss feed ./my_config_file.yml`.
 
 ## Generating a feed with Ruby
 
+You can also install it as a dependency in your Ruby project:
+
+|                      🤩 Like it? | Star it! ⭐️         |
+| -------------------------------: | -------------------- |
+| Add this line to your `Gemfile`: | `gem 'html2rss'`     |
+|                    Then execute: | `bundle`             |
+|                    In your code: | `require 'html2rss'` |
+
 Here's a minimal working example using Ruby:
 
 ```ruby
@@ -117,7 +115,7 @@ channel:
 Command line usage example:
 
 ```sh
-bundle exec html2rss feed the_feed_config.yml id=42
+html2rss feed the_feed_config.yml id=42
 ```
 
 <details><summary>See a Ruby example</summary>
@@ -154,9 +152,9 @@ Your `selectors` hash can contain arbitrary named selectors, but only a few will
 | `comments`    | `comments`         | A URL.                                      |
 | `source`      | ~~source~~         | Not yet supported.                          |
 
-### The `selector` hash
+### Build RSS 2.0 item attributes by specifying selectors
 
-Every named selector in your `selectors` hash can have these attributes:
+Every named selector (i.e. `title`, `description`, see table above) in your `selectors` hash can have these attributes:
 
 | name           | value                                                    |
 | -------------- | -------------------------------------------------------- |
@@ -164,7 +162,7 @@ Every named selector in your `selectors` hash can have these attributes:
 | `extractor`    | Name of the extractor. See notes below.                  |
 | `post_process` | A hash or array of hashes. See notes below.              |
 
-## Using extractors
+#### Using extractors
 
 Extractors help with extracting the information from the selected HTML tag.
 
@@ -201,7 +199,7 @@ selectors:
 
 </details>
 
-## Using post processors
+### Using post processors
 
 Extracted information can be further manipulated with post processors.
 
@@ -218,7 +216,7 @@ Extracted information can be further manipulated with post processors.
 
 ⚠️ Always make use of the `sanitize_html` post processor for HTML content. _Never trust the internet!_ ⚠️
 
-### Chaining post processors
+#### Chaining post processors
 
 Pass an array to `post_process` to chain the post processors.
 
@@ -244,14 +242,14 @@ selectors:
 
 </details>
 
-### Post processor `gsub`
+##### Post processor `gsub`
 
 The post processor `gsub` makes use of Ruby's [`gsub`](https://apidock.com/ruby/String/gsub) method.
 
-| key           | type   | required | note                        |
-| ------------- | ------ | -------- | --------------------------- |
-| `pattern`     | String | yes      | Can be Regexp or String.    |
-| `replacement` | String | yes      | Can be a [backreference](). |
+| key           | type   | required | note                     |
+| ------------- | ------ | -------- | ------------------------ |
+| `pattern`     | String | yes      | Can be Regexp or String. |
+| `replacement` | String | yes      | Can be a backreference.  |
 
 <details><summary>See a Ruby example</summary>
 
@@ -283,7 +281,7 @@ selectors:
 
 </details>
 
-## Adding `<category>` tags to an item
+#### Adding `<category>` tags to an item
 
 The `categories` selector takes an array of selector names. Each value of those
 selectors will become a `<category>` on the RSS item.
@@ -326,7 +324,7 @@ selectors:
 
 </details>
 
-## Custom item GUID
+#### Custom item GUID
 
 By default, html2rss generates a GUID from the `title` or `description`.
 
@@ -371,7 +369,7 @@ selectors:
 
 </details>
 
-## Adding an `<enclosure>` tag to an item
+#### Adding an `<enclosure>` tag to an item
 
 An enclosure can be any file, e.g. a image, audio or video - think Podcast.
 
@@ -379,7 +377,7 @@ The `enclosure` selector needs to return a URL of the content to enclose. If the
 
 Since `html2rss` does no further inspection of the enclosure, its support comes with trade-offs:
 
-1. The content-type is guessed from the file extension of the URL.
+1. The content-type is guessed from the file extension of the URL, unless one is specified in `content_type`.
 2. If the content-type guessing fails, it will default to `application/octet-stream`.
 3. The content-length will always be undetermined and therefore stated as `0` bytes.
 
@@ -392,7 +390,12 @@ Read the [RSS 2.0 spec](http://www.rssboard.org/rss-profile#element-channel-item
 Html2rss.feed(
   channel: {},
   selectors: {
-    enclosure: { selector: 'audio', extractor: 'attribute', attribute: 'src' }
+    enclosure: {
+      selector: 'audio',
+      extractor: 'attribute',
+      attribute: 'src',
+      content_type: 'audio/mp3'
+    }
   }
 )
 ```
@@ -411,17 +414,16 @@ selectors:
     selector: "audio"
     extractor: "attribute"
     attribute: "src"
+    content_type: "audio/mp3"
 ```
 
 </details>
+
 ## Scraping and handling JSON responses
 
-By default, `html2rss` assumes the URL responds with HTML. However, it can also handle JSON responses. The JSON must return an Array or Hash.
+By default, `html2rss` assumes the URL responds with HTML. However, it can also handle JSON responses. The JSON response must be an Array or Hash.
 
-| key        | required | default | note                                                 |
-| ---------- | -------- | ------- | ---------------------------------------------------- |
-| `json`     | optional | false   | If set to `true`, the response is parsed as JSON.    |
-| `jsonpath` | optional | $       | Use [JSONPath syntax]() to select nodes of interest. |
+The JSON is converted to XML which you can query using CSS selectors.
 
 <details><summary>See a Ruby example</summary>
 
@@ -447,7 +449,73 @@ selectors:
 
 </details>
 
-## Set any HTTP header in the request
+## Customization of how requests to the channel URL are sent
+
+By default, html2rss issues a naiive HTTP request and extracts information from the response. That is performant and works for many websites.
+
+However, modern websites often do not render much HTML on the server, but evaluate JavaScript on the client to create the HTML. In such cases, the default strategy will not find the "juicy content".
+
+### Use Browserless.io
+
+You can use _Browserless.io_ to run a Chrome browser and return the website's source code after the website generated it.
+For this, you can either run your own Browserless.io instance (Docker image available -- [read their license](https://github.com/browserless/browserless/pkgs/container/chromium#licensing)!) or pay them for a hosted instance.
+
+To run a local Browserless.io instance, you can use the following Docker command:
+
+```sh
+docker run \
+  --rm \
+  -p 3000:3000 \
+  -e "CONCURRENT=10" \
+  -e "TOKEN=6R0W53R135510" \
+  ghcr.io/browserless/chromium
+```
+
+To make html2rss use your instance,
+
+1. specify the environment variables accordingly, and
+2. use the `browserless` strategy for those websites.
+
+When running locally with commands from above, you can skip setting the environment variables, as they are aligned with the default values.
+
+```sh
+BROWSERLESS_IO_WEBSOCKET_URL="ws://127.0.0.1:3000" BROWSERLESS_IO_API_TOKEN="6R0W53R135510" \
+  html2rss auto --strategy=browserless https://example.com
+```
+
+When using traditional feed configs, inside your channel config set `strategy: browserless`.
+
+<details><summary>See a YAML feed config example</summary>
+
+```yml
+channel:
+  url: https://www.imdb.com/user/ur67728460/ratings
+  time_zone: UTC
+  ttl: 1440
+  strategy: browserless
+  headers:
+    User-Agent: "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36"
+selectors:
+  items:
+    selector: "li.ipc-metadata-list-summary-item"
+  title:
+    selector: ".ipc-title__text"
+    post_process:
+      - name: gsub
+        pattern: "/^(\\d+.)\\s/"
+        replacement: ""
+      - name: template
+        string: "%{self} rated with: %{user_rating}"
+  link:
+    selector: "a.ipc-title-link-wrapper"
+    extractor: "href"
+  user_rating:
+    selector: "[data-testid='ratingGroup--other-user-rating'] > .ipc-rating-star--rating"
+```
+
+</details>
+
+### Set any HTTP header in the request
 
 To set HTTP request headers, you can add them to the channel's `headers` hash. This is useful for APIs that require an Authorization header.
 
@@ -595,24 +663,25 @@ Recommended further readings:
 - Fiddling with [`curl`](https://github.com/curl/curl) and [`pup`](https://github.com/ericchiang/pup) to find the selectors seems efficient (`curl URL | pup`).
 - [CSS selectors are versatile. Here's an overview.](https://www.w3.org/TR/selectors-4/#overview)
 
-### Contributing
+## Contributing
 
 Find ideas what to contribute in:
 
 1. <https://github.com/orgs/html2rss/discussions>
 2. the issues tracker: <https://github.com/html2rss/html2rss/issues>
 
-#### Development Helpers
-
-1. `bin/setup`: installs dependencies and sets up the development environment.
-2. `bin/guard`: automatically runs rspec, rubocop and reek when a file changes.
-3. for a modern Ruby development experience: install [`ruby-lsp`](https://github.com/Shopify/ruby-lsp) and integrate it to your IDE:
-   a. [Ruby in Visual Studio Code](https://code.visualstudio.com/docs/languages/ruby)
-
-#### How to submit changes
+To submit changes:
 
 1. Fork this repo ( <https://github.com/html2rss/html2rss/fork> )
 2. Create your feature branch (`git checkout -b my-new-feature`)
 3. Implement a commit your changes (`git commit -am 'feat: add XYZ'`)
 4. Push to the branch (`git push origin my-new-feature`)
 5. Create a new Pull Request using the Github web UI
+
+## Development Helpers
+
+1. `bin/setup`: installs dependencies and sets up the development environment.
+2. `bin/guard`: automatically runs rspec, rubocop and reek when a file changes.
+3. for a modern Ruby development experience: install [`ruby-lsp`](https://github.com/Shopify/ruby-lsp) and integrate it to your IDE.
+
+For example: [Ruby in Visual Studio Code](https://code.visualstudio.com/docs/languages/ruby).

data/html2rss.gemspec

@@ -39,11 +39,12 @@ Gem::Specification.new do |spec|
   spec.add_dependency 'mime-types', '> 3.0'
   spec.add_dependency 'nokogiri', '>= 1.10', '< 2.0'
   spec.add_dependency 'parallel'
+  spec.add_dependency 'puppeteer-ruby'
   spec.add_dependency 'regexp_parser'
-  spec.add_dependency 'reverse_markdown', '~> 2.0'
+  spec.add_dependency 'reverse_markdown', '~> 3.0'
   spec.add_dependency 'rss'
   spec.add_dependency 'sanitize', '~> 6.0'
   spec.add_dependency 'thor'
   spec.add_dependency 'tzinfo'
-  spec.add_dependency 'zeitwerk'
+  spec.add_dependency 'zeitwerk', '~> 2.6.0'
 end

data/lib/html2rss/auto_source/article.rb

@@ -2,6 +2,7 @@
 
 require 'zlib'
 require 'sanitize'
+require 'nokogiri'
 
 module Html2rss
   class AutoSource
@@ -14,6 +15,31 @@ module Html2rss
 
       PROVIDED_KEYS = %i[id title description url image guid published_at scraper].freeze
 
+      ##
+      # Removes the specified pattern from the beginning of the text
+      # within a given range if the pattern occurs before the range's end.
+      #
+      # @param text [String]
+      # @param pattern [String]
+      # @param end_of_range [Integer] - Optional, defaults to half the size of the text
+      # @return [String]
+      def self.remove_pattern_from_start(text, pattern, end_of_range: (text.size * 0.5).to_i)
+        return text unless text.is_a?(String) && pattern.is_a?(String)
+
+        index = text.index(pattern)
+        return text if index.nil? || index >= end_of_range
+
+        text.gsub(/^(.{0,#{end_of_range}})#{Regexp.escape(pattern)}/, '\1')
+      end
+
+      ##
+      # Checks if the text contains HTML tags.
+      # @param text [String]
+      # @return [Boolean]
+      def self.contains_html?(text)
+        Nokogiri::HTML.fragment(text).children.any?(&:element?)
+      end
+
       # @param options [Hash<Symbol, String>]
       def initialize(**options)
         @to_h = {}
@@ -50,9 +76,15 @@ module Html2rss
       def description
         return @description if defined?(@description)
 
-        return if url.to_s.empty? || @to_h[:description].to_s.empty?
+        return if (description = @to_h[:description]).to_s.empty?
+
+        @description = self.class.remove_pattern_from_start(description, title) if title
 
-        @description ||= Html2rss::AttributePostProcessors::SanitizeHtml.get(@to_h[:description], url)
+        if self.class.contains_html?(@description) && url
+          @description = Html2rss::AttributePostProcessors::SanitizeHtml.get(description, url)
+        else
+          @description
+        end
       end
 
       # @return [Addressable::URI, nil]
@@ -72,11 +104,11 @@ module Html2rss
       end
 
       # Parses and returns the published_at time.
-      # @return [Time, nil]
+      # @return [DateTime, nil]
       def published_at
-        return if (string = @to_h[:published_at].to_s).strip.empty?
+        return if (string = @to_h[:published_at].to_s.strip).empty?
 
-        @published_at ||= Time.parse(string)
+        @published_at ||= DateTime.parse(string)
       rescue ArgumentError
         nil
       end

data/lib/html2rss/auto_source/channel.rb

@@ -24,52 +24,45 @@ module Html2rss
       attr_writer :articles
       attr_reader :stylesheets
 
-      def url = extract_url
-      def title = extract_title
-      def language = extract_language
-      def description = extract_description
-      def image = extract_image
-      def ttl = extract_ttl
-      def last_build_date = headers['last-modified']
-
-      def generator
-        "html2rss V. #{::Html2rss::VERSION} (using auto_source scrapers: #{scraper_counts})"
+      def url = @url.normalize.to_s
+
+      def title
+        @title ||= if (title = parsed_body.at_css('head > title')&.text.to_s) && !title.empty?
+                     title.gsub(/\s+/, ' ').strip
+                   else
+                     Utils.titleized_channel_url(@url)
+                   end
       end
 
-      private
-
-      attr_reader :parsed_body, :headers
-
-      def extract_url
-        @url.normalize.to_s
-      end
-
-      def extract_title
-        parsed_body.at_css('head > title')&.text
-      end
+      def description = parsed_body.at_css('meta[name="description"]')&.[]('content')
+      def last_build_date = headers['last-modified']
 
-      def extract_language
+      def language
         return parsed_body['lang'] if parsed_body.name == 'html' && parsed_body['lang']
 
         parsed_body.at_css('[lang]')&.[]('lang')
       end
 
-      def extract_description
-        parsed_body.at_css('meta[name="description"]')&.[]('content') || ''
-      end
-
-      def extract_image
+      def image
         url = parsed_body.at_css('meta[property="og:image"]')&.[]('content')
         Html2rss::Utils.sanitize_url(url) if url
       end
 
-      def extract_ttl
+      def ttl
         ttl = headers['cache-control']&.match(/max-age=(\d+)/)&.[](1)
         return unless ttl
 
         ttl.to_i.fdiv(60).ceil
       end
 
+      def generator
+        "html2rss V. #{::Html2rss::VERSION} (using auto_source scrapers: #{scraper_counts})"
+      end
+
+      private
+
+      attr_reader :parsed_body, :headers
+
       def scraper_counts
         scraper_counts = +''
 

data/lib/html2rss/auto_source/cleanup.rb

@@ -13,10 +13,7 @@ module Html2rss
 
           articles.select!(&:valid?)
 
-          remove_short!(articles, :title)
-
           deduplicate_by!(articles, :url)
-          deduplicate_by!(articles, :title)
 
           keep_only_http_urls!(articles)
           reject_different_domain!(articles, url) unless keep_different_domain
@@ -27,19 +24,6 @@ module Html2rss
 
         private
 
-        ##
-        # Removes articles with short values for a given key.
-        #
-        # @param articles [Array<Article>] The list of articles to process.
-        # @param key [Symbol] The key to check for short values.
-        # @param min_words [Integer] The minimum number of words required.
-        def remove_short!(articles, key = :title, min_words: 2)
-          articles.reject! do |article|
-            value = article.public_send(key)
-            value.nil? || value.to_s.split.size < min_words
-          end
-        end
-
         ##
         # Deduplicates articles by a given key.
         #

data/lib/html2rss/auto_source/rss_builder.rb

@@ -60,7 +60,7 @@ module Html2rss
 
             item_maker.title = article.title
             item_maker.description = article.description
-            item_maker.pubDate = article.published_at
+            item_maker.pubDate = article.published_at&.rfc2822
             item_maker.link = article.url
           end
         end

data/lib/html2rss/auto_source/scraper/html.rb

@@ -0,0 +1,96 @@
+# frozen_string_literal: true
+
+require 'nokogiri'
+require 'set'
+
+module Html2rss
+  class AutoSource
+    module Scraper
+      ##
+      # Scrapes articles from HTML pages by
+      # finding similar structures around anchor tags in the parsed_body.
+      class Html
+        include Enumerable
+
+        TAGS_TO_IGNORE = /(nav|footer|header)/i
+
+        def self.articles?(parsed_body)
+          new(parsed_body, url: '').any?
+        end
+
+        def self.parent_until_condition(node, condition)
+          return nil if !node || node.document? || node.parent.name == 'html'
+          return node if condition.call(node)
+
+          parent_until_condition(node.parent, condition)
+        end
+
+        ##
+        # Simplify an XPath selector by removing the index notation.
+        def self.simplify_xpath(xpath)
+          xpath.gsub(/\[\d+\]/, '')
+        end
+
+        def initialize(parsed_body, url:)
+          @parsed_body = parsed_body
+          @url = url
+          @selectors = Hash.new(0)
+        end
+
+        attr_reader :parsed_body
+
+        ##
+        # @yieldparam [Hash] The scraped article hash
+        # @return [Enumerator] Enumerator for the scraped articles
+        def each
+          return enum_for(:each) unless block_given?
+
+          return if frequent_selectors.empty?
+
+          frequent_selectors.each do |selector|
+            parsed_body.xpath(selector).each do |selected_tag|
+              article_tag = self.class.parent_until_condition(selected_tag, method(:article_condition))
+              article_hash = SemanticHtml::Extractor.new(article_tag, url: @url).call
+
+              yield article_hash if article_hash
+            end
+          end
+        end
+
+        ##
+        # Find all the anchors in root.
+        # @param root [Nokogiri::XML::Node] The root node to search for anchors
+        # @return [Set<String>] The set of XPath selectors which exist at least min_frequency times
+        def frequent_selectors(root = @parsed_body.at_css('body'), min_frequency: 2)
+          @frequent_selectors ||= begin
+            root.traverse do |node|
+              next if !node.element? || node.name != 'a'
+
+              @selectors[self.class.simplify_xpath(node.path)] += 1
+            end
+
+            @selectors.keys
+                      .select { |selector| (@selectors[selector]).to_i >= min_frequency }
+                      .to_set
+          end
+        end
+
+        def article_condition(node)
+          # Ignore tags that are below a tag which is in TAGS_TO_IGNORE.
+          return false if node.path.match?(TAGS_TO_IGNORE)
+
+          # Ignore tags that are below a tag which has a class which matches TAGS_TO_IGNORE.
+          return false if self.class.parent_until_condition(node, proc do |current_node|
+            current_node.classes.any? { |klass| klass.match?(TAGS_TO_IGNORE) }
+          end)
+
+          return true if %w[body html].include?(node.name)
+
+          return true if node.parent.css('a').size > 1
+
+          false
+        end
+      end
+    end
+  end
+end

data/lib/html2rss/auto_source/scraper/schema/item_list.rb

@@ -0,0 +1,34 @@
+# frozen_string_literal: true
+
+module Html2rss
+  class AutoSource
+    module Scraper
+      class Schema
+        ##
+        # Handles schema.org ItemList objects, which contain
+        # 1. itemListElements, and/or
+        # 2. interesting attributes, i.e. description, url, image, itself.
+        #
+        # @see https://schema.org/ItemList
+        class ItemList < Thing
+          SUPPORTED_TYPES = Set['ItemList']
+
+          # @return [Array<Hash>] the scraped article hashes with DEFAULT_ATTRIBUTES
+          def call
+            hashes = [super]
+
+            return hashes if (elements = @schema_object[:itemListElement]).nil?
+
+            elements = [elements] unless elements.is_a?(Array)
+
+            elements.each do |schema_object|
+              hashes << ListItem.new(schema_object, url: @url).call
+            end
+
+            hashes
+          end
+        end
+      end
+    end
+  end
+end

data/lib/html2rss/auto_source/scraper/schema/list_item.rb

@@ -0,0 +1,25 @@
+# frozen_string_literal: true
+
+module Html2rss
+  class AutoSource
+    module Scraper
+      class Schema
+        ##
+        #
+        # @see https://schema.org/ListItem
+        class ListItem < Thing
+          def id =          (id = (schema_object.dig(:item, :@id) || super).to_s).empty? ? nil : id
+          def title =       schema_object.dig(:item, :name) || super || (url ? Utils.titleized_url(url) : nil)
+          def description = schema_object.dig(:item, :description) || super
+
+          # @return [Addressable::URI, nil]
+          def url
+            url = schema_object.dig(:item, :url) || super
+
+            Utils.build_absolute_url_from_relative(url, @url) if url
+          end
+        end
+      end
+    end
+  end
+end