coelacanth 0.3.10 → 0.4.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 62040d12a6682d20ad4ec71ec6ab5d45f00a4814f2c4cd8dad22bc555352daac
-  data.tar.gz: 400b7e76c8a0260abc9c713566323dac84cd50758cda8013e54747cd465f909a
+  metadata.gz: 9097f36247caad8f0764313b306398e3707290eca25d5cfffc05f61e97784884
+  data.tar.gz: 19e093800bcb9ae663e0f36a1ad18d15472ff2e76f34f6ddf65c396440aea2e0
 SHA512:
-  metadata.gz: 6ffc542b1004c3cc3b8170f46e6e2f465bd7012ecebd99e9495c88431169f73e34f4e4558fab6c65b30e71b5e10509b2d76cca576cbc44345e098fa174f65732
-  data.tar.gz: 05c11331d46384976a872638e1e601118e49de427eb4a989e76ee73b02773379e6a3172a0a264a015e5db910800cc8fe0188c5d9d178bb2b9522c84b6043c884
+  metadata.gz: 32c30afcf5316814e42f2ed2f5b94efe49daecdd78b7acdf1d10d64ed2c8253a86a0a0be3b66ed1824cfbefb32c5654f3abf0fb08c41411d736281f480375400
+  data.tar.gz: 4e6683d596d50535dd13df26d2c6c4339fe543013d37250c8e55f7605dc441aa3ed888fcf46f9550540583e3234bd52667cd28d1b9e098a18e3eb5faae354bed

data/.env.example

@@ -0,0 +1,5 @@
+# Copy this file to .env and fill in the values to configure Coelacanth.
+# Optional: only set when the remote browser requires authentication.
+COELACANTH_REMOTE_CLIENT_AUTHORIZATION=
+COELACANTH_REMOTE_CLIENT_USER_AGENT="Coelacanth Chrome Extension"
+COELACANTH_SCREENSHOT_ONE_API_KEY="your_screenshot_one_api_key_here"

data/CHANGELOG.md

@@ -4,13 +4,8 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
-## [v0.3.10] - 2025-06-19
-### :bug: Bug Fixes
-- [`f297847`](https://github.com/slidict/coelacanth/commit/f29784715aba9453599ad7fd6467d4d2d3e9e82c) - improve error handling in Ferrum client and bump version to 0.3.10 *(commit by [@yubele](https://github.com/yubele))*
-
+## [v0.4.1] - 2025-11-03
 ### :wrench: Chores
-- [`58d383f`](https://github.com/slidict/coelacanth/commit/58d383fdc34e220e584d92447e938078ed75a889) - **deps**: Bump base64 from 0.2.0 to 0.3.0 *(commit by [@dependabot[bot]](https://github.com/apps/dependabot))*
-- [`6f1b766`](https://github.com/slidict/coelacanth/commit/6f1b766ea8f65d2f9c0f1d8772cbd6f1c840a43b) - **deps**: Bump rake from 13.2.1 to 13.3.0 *(commit by [@dependabot[bot]](https://github.com/apps/dependabot))*
-- [`0c9a6b2`](https://github.com/slidict/coelacanth/commit/0c9a6b2b9f28f686665914054016c90c49407e69) - **deps**: Bump rubocop from 1.75.7 to 1.76.1 *(commit by [@dependabot[bot]](https://github.com/apps/dependabot))*
+- [`41e89b7`](https://github.com/slidict/coelacanth/commit/41e89b799573f6cfaf0a12e7abc5c260f1905aec) - Bump version from 0.4.0 to 0.4.1 *(commit by [@yubele](https://github.com/yubele))*
 
-[v0.3.10]: https://github.com/slidict/coelacanth/compare/v0.3.9...v0.3.10
+[v0.4.1]: https://github.com/slidict/coelacanth/compare/v0.4.0...v0.4.1

data/Gemfile

@@ -8,7 +8,7 @@ gemspec
 gem "ferrum", "~> 0.16"
 gem "rake", "~> 13.3"
 gem "rspec", "~> 3.0"
-gem "rubocop", "~> 1.76"
+gem "rubocop", "~> 1.81"
 gem "oga", "~> 3.4"
 gem "base64", "~> 0.3.0"
 

data/README.md

@@ -1,94 +1,188 @@
-# coelacanth
+# Coelacanth
 
 [![Gem Version](https://badge.fury.io/rb/coelacanth.svg)](https://badge.fury.io/rb/coelacanth)
 [![Build Status](https://github.com/slidict/coelacanth/actions/workflows/main.yml/badge.svg)](https://github.com/slidict/coelacanth/actions)
 [![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](https://opensource.org/licenses/MIT)
 
-`coelacanth` is a gem that allows you to parse and analyze web pages, extracting key statistics and information for further use within your projects.
+Coelacanth is a Ruby gem for extracting high-quality article content, metadata, and screenshots from arbitrary web pages. It is
+built to power content ingestion pipelines that have to withstand layout experiments, CMS redesigns, and inconsistent markup
+while remaining easy to extend.
+
+It is the successor to [`web_stat`](https://rubygems.org/gems/web_stat) and continues the same goal of reliable article
+extraction under the `slidict` umbrella. Compared to [`web_stat`](https://github.com/slidict/web_stat/) the gem has been
+re-architected with a modern extractor pipeline, built-in screenshot capture, and a clearer configuration story so you can drop
+it into contemporary ingestion stacks without bespoke glue code.
+
+## Table of contents
+- [Features](#features)
+- [Requirements](#requirements)
+- [Installation](#installation)
+- [Quick start](#quick-start)
+- [Extractor pipeline](#extractor-pipeline)
+- [Configuration](#configuration)
+- [Development workflow](#development-workflow)
+- [Testing](#testing)
+- [Contributing](#contributing)
+- [License](#license)
 
-## Installation
-
-Add this line to your application's Gemfile:
+## Features
+- **Layout-resilient extraction** – Multi-stage extractor falls back from structured metadata to heuristics and lightweight
+  machine learning so you continue to get clean article bodies even when markup drifts.
+- **UTF-8 normalization** – HTML responses are normalized into UTF-8 before parsing to play nicely with Japanese and other
+  multi-byte sources.
+- **Screenshot capture** – Fetches full-page PNGs via a configurable browser client so you can archive visual context alongside
+  the extracted text.
+- **Redirect resolution** – Follows HTTP redirects and long redirect chains to guarantee the extractor works on the final
+  landing page.
+- **Configurable HTTP headers** – Inject custom headers (user agent, authorization, etc.) into the remote browser session for
+  authenticated or geo-targeted crawling.
+
+### What's new compared to web_stat?
+
+- **Multi-stage pipeline** – `web_stat` relied on a single-pass heuristic extractor, whereas Coelacanth layers metadata,
+  heuristic, and optional ML probes that graduate based on confidence thresholds.
+- **First-class screenshots** – Capture full-page PNGs alongside the extracted text without writing a separate headless browser
+  integration.
+- **Environment-aware configuration** – Manage remote browser credentials, HTTP headers, and client selection through
+  `config/coelacanth.yml` instead of hand-tuned initializer code.
+- **Markdown-first output** – Get both Markdown and raw DOM representations from `Coelacanth.analyze` so you can publish the
+  same payload to static-site builders, CMS importers, or downstream summarizers.
+
+## Requirements
+- Ruby **3.4 or newer**
+- [Bundler](https://bundler.io/) for dependency management
+- A remote Chrome-compatible WebSocket endpoint when using the default Ferrum client (see [Configuration](#configuration))
 
+## Installation
+Add the gem to your application:
 
 ```ruby
-gem 'coelacanth'
+gem "coelacanth"
 ```
 
-And then execute:
+Install the dependencies:
 
 ```bash
-$ bundle install
+bundle install
 ```
 
-Or install it yourself as:
+Or install the gem directly:
 
 ```bash
-$ gem install coelacanth
+gem install coelacanth
 ```
 
-### Resolving UID Mismatch Between Docker and Host
+## Quick start
+```ruby
+require "coelacanth"
 
-To resolve issues related to the difference between Docker's UID and the host's UID, add the following line to your .bashrc or similar shell configuration file:
+result = Coelacanth.analyze("https://example.com/article")
 
-```bash
-export UID=${UID}
+result[:extraction] # => article metadata and body markdown
+result[:dom]        # => Oga DOM representation for downstream processing
+result[:screenshot] # => PNG screenshot as a binary string
 ```
 
-This will ensure that the environment variable UID is correctly set in your Docker containers, matching your host system's user ID.
+The returned hash includes:
+
+- `:extraction` – output from `Coelacanth::Extractor`, including title, Markdown body (`body_markdown` and
+  `body_markdown_list`), images, listings, published date, and the probe source and confidence score.
+- `:dom` – a parsed Oga DOM if you need to traverse the document manually.
+- `:screenshot` – raw PNG data that you can persist or feed to other systems.
+
+## Extractor pipeline
+Coelacanth ships with a multi-stage extractor that tries increasingly involved probes until one meets its confidence target:
+
+1. **MetadataProbe** (threshold `0.85`) pulls `schema.org` JSON-LD, Open Graph, Twitter Cards, or semantic containers such as
+   `<main>`/`<article>` when available.
+2. **HeuristicProbe** (threshold `0.75`) scores block-level nodes using text length, link density, punctuation density, DOM
+   depth, and sibling variance, then greedily attaches surrounding headers and media.
+3. **WeakMlProbe** (threshold `0.70`) optionally boosts accuracy with a lightweight classifier that combines heuristic features
+   with class and id tokens (e.g., `article-body`, `post`, `content`).
+4. **FallbackProbe** acts as a safety net by following AMP/print links or summarizing the whole document when the previous
+   probes fail.
+
+Markdown-based listings are generated from the extracted body so lists such as "Latest news" blocks can be stored alongside the
+article without scanning the rest of the page layout.
+
+## Configuration
+Runtime configuration is stored in `config/coelacanth.yml`. Environments inherit from the `development` section by default.
+
+```yaml
+development:
+  client: "ferrum" # Options: "ferrum", "screenshot_one"
+  remote_client:
+    ws_url: "ws://chrome:3000/chrome"
+    timeout: 10
+    headers:
+<% if (auth = ENV["COELACANTH_REMOTE_CLIENT_AUTHORIZATION"]).to_s.strip != "" %>
+      Authorization: "<%= auth %>"
+<% end %>
+      User-Agent: "<%= ENV.fetch("COELACANTH_REMOTE_CLIENT_USER_AGENT", "Coelacanth Chrome Extension") %>"
+  screenshot_one:
+    key: "<%= ENV.fetch("COELACANTH_SCREENSHOT_ONE_API_KEY", "your_screenshot_one_api_key_here") %>"
+```
 
-This explanation provides clear instructions on how to resolve the UID mismatch issue using the export command.
+- **Ferrum client** – Requires a running Chrome instance that exposes the DevTools protocol via WebSocket. Configure the URL,
+  timeout, and any headers to inject.
+- **ScreenshotOne client** – Supply an API key to offload screenshot capture to [ScreenshotOne](https://screenshotone.com/).
+- Configuration is environment-aware: set `RAILS_ENV`/`RACK_ENV` or use Rails' built-in environment handling when the gem is
+  used inside a Rails project.
 
-## Usage
-To use coelacanth, first require it.
+### Environment variables
 
-```ruby
-require 'coelacanth'
-```
+Configuration values that would otherwise contain credentials are loaded from environment variables. Set the following
+variables in your shell (or `dotenv` file) before running the gem:
 
-Then, you can easily parse and extract information from a web page like this:
+```bash
+# Optional: only set when the remote browser requires authentication.
+export COELACANTH_REMOTE_CLIENT_AUTHORIZATION="Bearer <token>"
 
-```ruby
-url = "https://example.com"
-stats = Coelacanth.analyze(url)
+export COELACANTH_REMOTE_CLIENT_USER_AGENT="Coelacanth Chrome Extension"
+export COELACANTH_SCREENSHOT_ONE_API_KEY="your_screenshot_one_api_key_here"
 ```
 
-- rspec
+If `COELACANTH_REMOTE_CLIENT_AUTHORIZATION` is omitted or left blank, the `Authorization` header is not injected into the
+remote browser session.
 
-```
-$ bundle exec rspec
-```
+When using Docker Compose, you can create a `.env` file or export the variables in your environment so the `app` service picks
+them up automatically.
 
-## Features
-- Get dom by oga
-- Get screenshot
+If you are working inside Docker, make sure the `UID` environment variable matches your host user by exporting it in your shell
+startup file:
 
-## Commit Message Guidelines
+```bash
+export UID=${UID}
+```
 
-To ensure consistency and facilitate automatic updates to the `CHANGELOG.md`, please follow the [Conventional Commits](https://www.conventionalcommits.org/) specification when creating commit messages. This helps maintain a clear and structured commit history.
+## Development workflow
+Clone the repository and install dependencies:
 
-When submitting a Pull Request (PR), make sure your commits adhere to these guidelines.
+```bash
+git clone https://github.com/slidict/coelacanth.git
+cd coelacanth
+bundle install
+```
 
-### Example of Conventional Commit Messages:
+You can open an interactive console with the gem loaded via:
 
-- `feat: add new feature for parsing web pages`
-- `fix: resolve issue with URL redirection`
-- `docs: update README with usage instructions`
-- `chore: update dependencies`
-- `build: update build configuration`
-- `ci: update CI pipeline`
-- `style: fix code style issues`
-- `refactor: refactor code for better readability`
-- `perf: improve performance of data processing`
-- `test: add new tests for URL parsing module`
+```bash
+bin/console
+```
+
+## Testing
+Run the test suite with RSpec:
 
-By following these guidelines, you help ensure that our project's commit history is easy to navigate and that versioning and release notes are generated correctly.
+```bash
+bundle exec rspec
+```
 
 ## Contributing
-Bug reports and pull requests are welcome on GitHub at https://github.com/slidict/coelacanth. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the Contributor Covenant code of conduct.
+Bug reports and pull requests are welcome on GitHub at
+[https://github.com/slidict/coelacanth](https://github.com/slidict/coelacanth). Please follow the
+[Conventional Commits](https://www.conventionalcommits.org/) specification so we can keep the changelog automation healthy.
 
-## License
-The gem is available as open-source under the terms of the MIT License.
+By participating in this project you agree to abide by the [Contributor Covenant](CODE_OF_CONDUCT.md).
 
-## Acknowledgments
-Special thanks to all the contributors and open-source projects that make this possible.
+## License
+Coelacanth is available as open source under the terms of the [MIT License](LICENSE.txt).

data/compose.yml

@@ -3,8 +3,11 @@ networks:
     driver: bridge
 services:
   app:
-    environment:
-      - UID=${UID}
+    environment:
+      - UID=${UID}
+      - COELACANTH_REMOTE_CLIENT_AUTHORIZATION=${COELACANTH_REMOTE_CLIENT_AUTHORIZATION:-}
+      - COELACANTH_REMOTE_CLIENT_USER_AGENT=${COELACANTH_REMOTE_CLIENT_USER_AGENT:-}
+      - COELACANTH_SCREENSHOT_ONE_API_KEY=${COELACANTH_SCREENSHOT_ONE_API_KEY:-}
     tty: true
     stdin_open: true
     build:

data/config/coelacanth.yml

@@ -4,10 +4,12 @@ development: &development
     ws_url: "ws://chrome:3000/chrome"
     timeout: 10 # seconds
     headers:
-      Authorization: "Bearer 1234567890"
-      User-Agent: "Coelacanth Chrome Extension"
+<% if (auth = ENV["COELACANTH_REMOTE_CLIENT_AUTHORIZATION"]).to_s.strip != "" %>
+      Authorization: "<%= auth %>"
+<% end %>
+      User-Agent: "<%= ENV.fetch("COELACANTH_REMOTE_CLIENT_USER_AGENT", "Coelacanth Chrome Extension") %>"
   screenshot_one:
-    key: "your_screenshot_one_api_key_here"
+    key: "<%= ENV.fetch("COELACANTH_SCREENSHOT_ONE_API_KEY", "your_screenshot_one_api_key_here") %>"
 test:
   <<: *development
 production:

data/lib/coelacanth/client/ferrum.rb

@@ -16,7 +16,7 @@ module Coelacanth::Client
       body = remote_client.body
       body
     rescue => e
-      raise "#{e.class}: #{e.message} RemoteClient: #{@remote_client.inspect}"
+      raise sanitized_remote_client_error(e)
     end
 
     def get_screenshot
@@ -26,11 +26,21 @@ module Coelacanth::Client
       File.read(tempfile.path)
     rescue => e
       tempfile.close
-      raise "#{e.class}: #{e.message} RemoteClient: #{@remote_client.inspect}"
+      raise sanitized_remote_client_error(e)
     end
 
     private
 
+    def sanitized_remote_client_error(error)
+      "#{error.class}: #{error.message} RemoteClient: #{sanitized_remote_client_identifier}"
+    end
+
+    def sanitized_remote_client_identifier
+      return "nil" unless @remote_client
+
+      "#{@remote_client.class.name}(object_id=#{@remote_client.object_id})"
+    end
+
     def remote_client
       return @remote_client if @remote_client
 

data/lib/coelacanth/client/screenshot_one.rb

@@ -1,19 +1,29 @@
 # frozen_string_literal: true
 
-require "open-uri"
 require "ferrum"
+require_relative "ferrum"
+require_relative "../http"
 
 module Coelacanth::Client
   # Coelacanth::Client
   class ScreenshotOne < Coelacanth::Client::Base
     def get_response
-      @origin_response = URI(@url).open
-      @status_code = @origin_response.status[0].to_i
-      body = @origin_response.read
-      body
-    rescue OpenURI::HTTPError => e
-      @status_code = e.io.status[0].to_i
-      raise e
+      uri = URI.parse(@url)
+      response = Coelacanth::HTTP.get_response(
+        uri,
+        open_timeout: Coelacanth::HTTP::DEFAULT_OPEN_TIMEOUT,
+        read_timeout: Coelacanth::HTTP::DEFAULT_READ_TIMEOUT
+      )
+      @origin_response = response
+      @status_code = response.code.to_i
+
+      return response.body if response.is_a?(Net::HTTPSuccess)
+
+      Coelacanth::HTTP.raise_http_error(uri, response)
+    rescue Coelacanth::TimeoutError
+      fallback_response = fallback_client.get_response
+      @status_code = fallback_client.instance_variable_get(:@status_code)
+      fallback_response
     end
 
     def get_screenshot
@@ -34,10 +44,22 @@ module Coelacanth::Client
       }
       uri.query = URI.encode_www_form(params)
 
-      response = Net::HTTP.get_response(uri)
+      response = Coelacanth::HTTP.get_response(
+        uri,
+        open_timeout: Coelacanth::HTTP::DEFAULT_OPEN_TIMEOUT,
+        read_timeout: 30
+      )
       raise "Failed to fetch screenshot: #{response.code}" unless response.is_a?(Net::HTTPSuccess)
 
       response.body
+    rescue Coelacanth::TimeoutError
+      fallback_client.get_screenshot
+    end
+
+    private
+
+    def fallback_client
+      @fallback_client ||= Coelacanth::Client::Ferrum.new(@url, @config)
     end
   end
 end

data/lib/coelacanth/configure.rb

@@ -15,7 +15,12 @@ module Coelacanth
     end
 
     def yaml
-      @yaml ||= YAML.unsafe_load(ERB.new(File.read(file)).result)[env]
+      @yaml ||= YAML.safe_load(
+        ERB.new(File.read(file)).result,
+        permitted_classes: [],
+        permitted_symbols: [],
+        aliases: true
+      )[env]
     end
 
     private

data/lib/coelacanth/dom.rb

@@ -1,12 +1,18 @@
 # frozen_string_literal: true
 
 require "oga"
+require_relative "http"
 
 module Coelacanth
   # Coelacanth::Dom
   class Dom
-    def oga(url)
-      Oga.parse_xml(Net::HTTP.get_response(URI.parse(url)).body)
+    def oga(url, html: nil)
+      html ||= begin
+        Coelacanth::HTTP.get_response(URI.parse(url)).body
+      rescue Coelacanth::TimeoutError
+        ""
+      end
+      Oga.parse_xml(html.to_s)
     end
   end
 end

data/lib/coelacanth/extractor/fallback_probe.rb

@@ -0,0 +1,34 @@
+# frozen_string_literal: true
+
+require "oga"
+
+require_relative "utilities"
+
+module Coelacanth
+  class Extractor
+    # Attempts final recovery strategies when all other probes fail.
+    class FallbackProbe
+      Result = Struct.new(
+        :title,
+        :node,
+        :published_at,
+        :byline,
+        :source_tag,
+        :confidence,
+        keyword_init: true
+      )
+
+      def call(doc:, url: nil)
+        body = doc.at_css("body") || doc
+        Result.new(
+          title: doc.at_css("title")&.text&.strip,
+          node: body,
+          published_at: nil,
+          byline: nil,
+          source_tag: :fallback,
+          confidence: 0.35
+        )
+      end
+    end
+  end
+end

data/lib/coelacanth/extractor/heuristic_probe.rb

@@ -0,0 +1,175 @@
+# frozen_string_literal: true
+
+require "oga"
+
+require_relative "utilities"
+
+module Coelacanth
+  class Extractor
+    # Scores DOM nodes based on simple heuristics to locate the primary article body.
+    class HeuristicProbe
+      Result = Struct.new(
+        :title,
+        :node,
+        :published_at,
+        :byline,
+        :source_tag,
+        :confidence,
+        keyword_init: true
+      )
+
+      BLOCK_SELECTOR = "article, main, section, div".freeze
+      TAG_WEIGHTS = Hash.new(0).merge(
+        "article" => 80,
+        "main" => 60,
+        "section" => 30,
+        "div" => 10
+      ).freeze
+      NEGATIVE_TOKENS = %w[nav footer header sidebar related share menu].freeze
+      POSITIVE_TOKENS = %w[content article body post entry text].freeze
+
+      def call(doc:, url: nil)
+        candidates = doc.css(BLOCK_SELECTOR).map do |node|
+          score_candidate(node)
+        end.compact
+
+        return if candidates.empty?
+
+        best = candidates.max_by { |candidate| candidate[:score] }
+        return if best[:score] < minimum_score
+
+        Result.new(
+          title: title_from_meta(doc),
+          node: expand(best[:node]),
+          published_at: published_at_from_meta(doc),
+          byline: byline_from_meta(doc),
+          source_tag: :heuristic,
+          confidence: confidence(best[:score])
+        )
+      end
+
+      private
+
+      def score_candidate(node)
+        text_length = Utilities.text_length(node)
+        return if text_length < 80
+
+        link_density = Utilities.link_density(node)
+        punct_density = Utilities.punctuation_density(node)
+        tag_weight = TAG_WEIGHTS[node.name]
+        class_weight = class_score(node)
+        depth_penalty = Utilities.depth(node) * 4
+        sibling_bonus = sibling_variance(node)
+
+        score = (
+          text_length * 0.35 +
+          punct_density * 280 -
+          link_density * 160 +
+          tag_weight +
+          class_weight +
+          sibling_bonus -
+          depth_penalty
+        )
+
+        { node: node, score: score }
+      end
+
+      def minimum_score
+        95
+      end
+
+      def class_score(node)
+        tokens = Utilities.class_id_tokens(node)
+        score = tokens.count { |token| POSITIVE_TOKENS.include?(token) } * 40
+        score -= tokens.count { |token| NEGATIVE_TOKENS.include?(token) } * 60
+        score
+      end
+
+      def sibling_variance(node)
+        parent = node.parent
+        return 0 unless parent
+
+        siblings = Utilities.element_children(parent)
+        return 0 if siblings.length < 2
+
+        lengths = siblings.map { |sibling| Utilities.text_length(sibling) }
+        mean = lengths.sum.to_f / lengths.length
+        variance = lengths.map { |length| (length - mean)**2 }.sum.to_f / lengths.length
+        Math.sqrt(variance) * 0.25
+      end
+
+      def expand(node)
+        return node unless node.parent
+
+        before = neighboring_nodes(node, -1).reverse
+        after = neighboring_nodes(node, 1)
+
+        wrap_fragment(before + [node] + after)
+      end
+
+      def neighboring_nodes(node, direction)
+        siblings = []
+        current = node
+        loop do
+          current = if direction.negative?
+                      Utilities.previous_element(current)
+                    else
+                      Utilities.next_element(current)
+                    end
+          break unless current
+
+          break unless include_in_expansion?(current)
+
+          siblings << current
+        end
+        siblings
+      end
+
+      def include_in_expansion?(node)
+        %w[h1 h2 h3 h4 h5 h6 img blockquote p ul ol figure].include?(node.name)
+      end
+
+      def wrap_fragment(nodes)
+        container = Oga::XML::Element.new(name: "article")
+        nodes.each { |node| container.children << node }
+        container
+      end
+
+      def confidence(score)
+        return 0.0 if score.to_f <= 0.0
+
+        value = 1.0 / (1.0 + Math.exp(-(score - 100) / 12.0))
+        value.clamp(0.0, 0.95)
+      end
+
+      def title_from_meta(doc)
+        Utilities.meta_content(
+          doc,
+          "meta[property='og:title']",
+          "meta[name='twitter:title']",
+          "meta[name='title']"
+        ) || doc.at_css("title")&.text&.strip
+      end
+
+      def published_at_from_meta(doc)
+        Utilities.parse_time(
+          Utilities.meta_content(
+            doc,
+            "meta[property='article:published_time']",
+            "meta[name='pubdate']",
+            "meta[name='publish_date']",
+            "meta[name='date']"
+          )
+        )
+      end
+
+      def byline_from_meta(doc)
+        Utilities.meta_content(
+          doc,
+          "meta[name='author']",
+          "meta[property='article:author']"
+        )
+      end
+    end
+  end
+end