coelacanth 0.3.9 → 0.4.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: add16c940fac675d2b5fa56866298362c0fa6a3114eb6073887e5689d4c85615
-  data.tar.gz: cd63d6e531d323c7c9d524f06d1c072331f1dad795afaf04b0440b0b2315960d
+  metadata.gz: 3f8a15ca435355182b77131ffa50b2c93b08693c3ccf0d26b5010e1f7f9df4fd
+  data.tar.gz: 34f198f28014ed558f14ef5599f9275579054a1ec9ebeb6b503c60555329bcf2
 SHA512:
-  metadata.gz: 5af74f841104e39c8d67e05f0ef0b0f91f9f2a989be871ea1f4d0d7477cf522b438caf50831617d7efae11b2b9388685616b7b68d21f55b363db1a7937d9f540
-  data.tar.gz: 4577b2d485b862175f3ae82de24d4e84a3efe8ffe860285a46926ec8fb305c3471e6637676ec209297d9bbdc220ee8f4d4a09782f68034d785f2537e213ec361
+  metadata.gz: aa7cdd1ac14d7104604114885facaffc6dc5e4d8b91911cb1ae26b34aa2550010bd1658f3c832a6e1dc4a0a059113317c7242bc187e02a396cc17398570407e0
+  data.tar.gz: 8f1c4c988ec19a862409e06fbe297529d058a78dc77f328959ab5274fdd372ef65d8e92b38ba76ef08b55ad7d5abeb75d585685f411e223ce9fc60515430ccc7

data/CHANGELOG.md

@@ -4,11 +4,19 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
-## [v0.3.9] - 2025-05-27
-### :bug: Bug Fixes
-- [`2fae729`](https://github.com/slidict/coelacanth/commit/2fae729f3daf5c21fc5125da9fd2ee93cd13b749) - gem-push.yml *(commit by [@yubele](https://github.com/yubele))*
+## [v0.4.0] - 2025-11-02
+### :sparkles: New Features
+- [`d13c21e`](https://github.com/slidict/coelacanth/commit/d13c21e106e7155ba7d545570341da37caa0b4b3) - support structural listings for digital.go.jp *(commit by [@yubele](https://github.com/yubele))*
+- [`19fb4d7`](https://github.com/slidict/coelacanth/commit/19fb4d7a29350bb063c6bb76a89c46d7d001fc3e) - expose body markdown blocks *(commit by [@yubele](https://github.com/yubele))*
 
 ### :wrench: Chores
-- [`82b605c`](https://github.com/slidict/coelacanth/commit/82b605c0e2f3ece1846b2d9c40350d257566b18d) - bumpversion *(commit by [@yubele](https://github.com/yubele))*
+- [`513d522`](https://github.com/slidict/coelacanth/commit/513d5226147b7be72c43817d0ca417a2ea355d31) - **deps**: Bump rubocop from 1.76.1 to 1.77.0 *(commit by [@dependabot[bot]](https://github.com/apps/dependabot))*
+- [`388658f`](https://github.com/slidict/coelacanth/commit/388658fcaa94f09123c67ab62dd69d207d03ee7a) - **deps**: Bump rubocop from 1.77.0 to 1.81.6 *(commit by [@dependabot[bot]](https://github.com/apps/dependabot))*
+- [`be62dbb`](https://github.com/slidict/coelacanth/commit/be62dbbf77fc9a324f29d684df167cbedcea7b8f) - translate listing samples to English *(commit by [@yubele](https://github.com/yubele))*
+- [`41bf358`](https://github.com/slidict/coelacanth/commit/41bf358eb955fb75b467719750a19f9d07a2cba1) - require ruby 3.4 or newer *(commit by [@yubele](https://github.com/yubele))*
+- [`cff304d`](https://github.com/slidict/coelacanth/commit/cff304d255952ad6be18e796c6649b72272bc23e) - **deps-dev**: Bump rexml in the bundler group across 1 directory *(commit by [@dependabot[bot]](https://github.com/apps/dependabot))*
+- [`962c267`](https://github.com/slidict/coelacanth/commit/962c267ab8f301e1375b67fc32a3cbc54effc232) - Bump version to 0.4.0 *(commit by [@yubele](https://github.com/yubele))*
+- [`bf3633c`](https://github.com/slidict/coelacanth/commit/bf3633c82a61684238485606300bfa0891d247eb) - Delete Gemfile.lock *(commit by [@yubele](https://github.com/yubele))*
+- [`62f2a0e`](https://github.com/slidict/coelacanth/commit/62f2a0ecb80fb5d77f704e6c48d1d8f4c7670818) - Add Gemfile.lock to .gitignore *(commit by [@yubele](https://github.com/yubele))*
 
-[v0.3.9]: https://github.com/slidict/coelacanth/compare/v0.3.8...v0.3.9
+[v0.4.0]: https://github.com/slidict/coelacanth/compare/v0.3.10...v0.4.0

data/Gemfile

@@ -6,11 +6,11 @@ source "https://rubygems.org"
 gemspec
 
 gem "ferrum", "~> 0.16"
-gem "rake", "~> 13.2"
+gem "rake", "~> 13.3"
 gem "rspec", "~> 3.0"
-gem "rubocop", "~> 1.75"
+gem "rubocop", "~> 1.81"
 gem "oga", "~> 3.4"
-gem "base64", "~> 0.2.0"
+gem "base64", "~> 0.3.0"
 
 group :development, :test do
   gem "webmock", "~> 3.25"

data/README.md

@@ -1,94 +1,167 @@
-# coelacanth
+# Coelacanth
 
 [![Gem Version](https://badge.fury.io/rb/coelacanth.svg)](https://badge.fury.io/rb/coelacanth)
 [![Build Status](https://github.com/slidict/coelacanth/actions/workflows/main.yml/badge.svg)](https://github.com/slidict/coelacanth/actions)
 [![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](https://opensource.org/licenses/MIT)
 
-`coelacanth` is a gem that allows you to parse and analyze web pages, extracting key statistics and information for further use within your projects.
+Coelacanth is a Ruby gem for extracting high-quality article content, metadata, and screenshots from arbitrary web pages. It is
+built to power content ingestion pipelines that have to withstand layout experiments, CMS redesigns, and inconsistent markup
+while remaining easy to extend.
+
+It is the successor to [`web_stat`](https://rubygems.org/gems/web_stat) and continues the same goal of reliable article
+extraction under the `slidict` umbrella. Compared to [`web_stat`](https://github.com/slidict/web_stat/) the gem has been
+re-architected with a modern extractor pipeline, built-in screenshot capture, and a clearer configuration story so you can drop
+it into contemporary ingestion stacks without bespoke glue code.
+
+## Table of contents
+- [Features](#features)
+- [Requirements](#requirements)
+- [Installation](#installation)
+- [Quick start](#quick-start)
+- [Extractor pipeline](#extractor-pipeline)
+- [Configuration](#configuration)
+- [Development workflow](#development-workflow)
+- [Testing](#testing)
+- [Contributing](#contributing)
+- [License](#license)
 
-## Installation
-
-Add this line to your application's Gemfile:
+## Features
+- **Layout-resilient extraction** – Multi-stage extractor falls back from structured metadata to heuristics and lightweight
+  machine learning so you continue to get clean article bodies even when markup drifts.
+- **UTF-8 normalization** – HTML responses are normalized into UTF-8 before parsing to play nicely with Japanese and other
+  multi-byte sources.
+- **Screenshot capture** – Fetches full-page PNGs via a configurable browser client so you can archive visual context alongside
+  the extracted text.
+- **Redirect resolution** – Follows HTTP redirects and long redirect chains to guarantee the extractor works on the final
+  landing page.
+- **Configurable HTTP headers** – Inject custom headers (user agent, authorization, etc.) into the remote browser session for
+  authenticated or geo-targeted crawling.
+
+### What's new compared to web_stat?
+
+- **Multi-stage pipeline** – `web_stat` relied on a single-pass heuristic extractor, whereas Coelacanth layers metadata,
+  heuristic, and optional ML probes that graduate based on confidence thresholds.
+- **First-class screenshots** – Capture full-page PNGs alongside the extracted text without writing a separate headless browser
+  integration.
+- **Environment-aware configuration** – Manage remote browser credentials, HTTP headers, and client selection through
+  `config/coelacanth.yml` instead of hand-tuned initializer code.
+- **Markdown-first output** – Get both Markdown and raw DOM representations from `Coelacanth.analyze` so you can publish the
+  same payload to static-site builders, CMS importers, or downstream summarizers.
+
+## Requirements
+- Ruby **3.4 or newer**
+- [Bundler](https://bundler.io/) for dependency management
+- A remote Chrome-compatible WebSocket endpoint when using the default Ferrum client (see [Configuration](#configuration))
 
+## Installation
+Add the gem to your application:
 
 ```ruby
-gem 'coelacanth'
+gem "coelacanth"
 ```
 
-And then execute:
+Install the dependencies:
 
 ```bash
-$ bundle install
+bundle install
 ```
 
-Or install it yourself as:
+Or install the gem directly:
 
 ```bash
-$ gem install coelacanth
+gem install coelacanth
 ```
 
-### Resolving UID Mismatch Between Docker and Host
+## Quick start
+```ruby
+require "coelacanth"
 
-To resolve issues related to the difference between Docker's UID and the host's UID, add the following line to your .bashrc or similar shell configuration file:
+result = Coelacanth.analyze("https://example.com/article")
 
-```bash
-export UID=${UID}
+result[:extraction] # => article metadata and body markdown
+result[:dom]        # => Oga DOM representation for downstream processing
+result[:screenshot] # => PNG screenshot as a binary string
 ```
 
-This will ensure that the environment variable UID is correctly set in your Docker containers, matching your host system's user ID.
+The returned hash includes:
+
+- `:extraction` – output from `Coelacanth::Extractor`, including title, Markdown body (`body_markdown` and
+  `body_markdown_list`), images, listings, published date, and the probe source and confidence score.
+- `:dom` – a parsed Oga DOM if you need to traverse the document manually.
+- `:screenshot` – raw PNG data that you can persist or feed to other systems.
+
+## Extractor pipeline
+Coelacanth ships with a multi-stage extractor that tries increasingly involved probes until one meets its confidence target:
+
+1. **MetadataProbe** (threshold `0.85`) pulls `schema.org` JSON-LD, Open Graph, Twitter Cards, or semantic containers such as
+   `<main>`/`<article>` when available.
+2. **HeuristicProbe** (threshold `0.75`) scores block-level nodes using text length, link density, punctuation density, DOM
+   depth, and sibling variance, then greedily attaches surrounding headers and media.
+3. **WeakMlProbe** (threshold `0.70`) optionally boosts accuracy with a lightweight classifier that combines heuristic features
+   with class and id tokens (e.g., `article-body`, `post`, `content`).
+4. **FallbackProbe** acts as a safety net by following AMP/print links or summarizing the whole document when the previous
+   probes fail.
+
+A `ListingCollector` scans nearby layout regions to surface structured listings (e.g., sidebar "latest news" blocks) so you can
+store related links alongside the article.
+
+## Configuration
+Runtime configuration is stored in `config/coelacanth.yml`. Environments inherit from the `development` section by default.
+
+```yaml
+development:
+  client: "ferrum" # Options: "ferrum", "screenshot_one"
+  remote_client:
+    ws_url: "ws://chrome:3000/chrome"
+    timeout: 10
+    headers:
+      Authorization: "Bearer 1234567890"
+      User-Agent: "Coelacanth Chrome Extension"
+  screenshot_one:
+    key: "your_screenshot_one_api_key_here"
+```
 
-This explanation provides clear instructions on how to resolve the UID mismatch issue using the export command.
+- **Ferrum client** – Requires a running Chrome instance that exposes the DevTools protocol via WebSocket. Configure the URL,
+  timeout, and any headers to inject.
+- **ScreenshotOne client** – Supply an API key to offload screenshot capture to [ScreenshotOne](https://screenshotone.com/).
+- Configuration is environment-aware: set `RAILS_ENV`/`RACK_ENV` or use Rails' built-in environment handling when the gem is
+  used inside a Rails project.
 
-## Usage
-To use coelacanth, first require it.
+If you are working inside Docker, make sure the `UID` environment variable matches your host user by exporting it in your shell
+startup file:
 
-```ruby
-require 'coelacanth'
+```bash
+export UID=${UID}
 ```
 
-Then, you can easily parse and extract information from a web page like this:
+## Development workflow
+Clone the repository and install dependencies:
 
-```ruby
-url = "https://example.com"
-stats = Coelacanth.analyze(url)
+```bash
+git clone https://github.com/slidict/coelacanth.git
+cd coelacanth
+bundle install
 ```
 
-- rspec
+You can open an interactive console with the gem loaded via:
 
+```bash
+bin/console
 ```
-$ bundle exec rspec
-```
-
-## Features
-- Get dom by oga
-- Get screenshot
-
-## Commit Message Guidelines
-
-To ensure consistency and facilitate automatic updates to the `CHANGELOG.md`, please follow the [Conventional Commits](https://www.conventionalcommits.org/) specification when creating commit messages. This helps maintain a clear and structured commit history.
 
-When submitting a Pull Request (PR), make sure your commits adhere to these guidelines.
+## Testing
+Run the test suite with RSpec:
 
-### Example of Conventional Commit Messages:
-
-- `feat: add new feature for parsing web pages`
-- `fix: resolve issue with URL redirection`
-- `docs: update README with usage instructions`
-- `chore: update dependencies`
-- `build: update build configuration`
-- `ci: update CI pipeline`
-- `style: fix code style issues`
-- `refactor: refactor code for better readability`
-- `perf: improve performance of data processing`
-- `test: add new tests for URL parsing module`
-
-By following these guidelines, you help ensure that our project's commit history is easy to navigate and that versioning and release notes are generated correctly.
+```bash
+bundle exec rspec
+```
 
 ## Contributing
-Bug reports and pull requests are welcome on GitHub at https://github.com/slidict/coelacanth. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the Contributor Covenant code of conduct.
+Bug reports and pull requests are welcome on GitHub at
+[https://github.com/slidict/coelacanth](https://github.com/slidict/coelacanth). Please follow the
+[Conventional Commits](https://www.conventionalcommits.org/) specification so we can keep the changelog automation healthy.
 
-## License
-The gem is available as open-source under the terms of the MIT License.
+By participating in this project you agree to abide by the [Contributor Covenant](CODE_OF_CONDUCT.md).
 
-## Acknowledgments
-Special thanks to all the contributors and open-source projects that make this possible.
+## License
+Coelacanth is available as open source under the terms of the [MIT License](LICENSE.txt).

data/lib/coelacanth/client/ferrum.rb

@@ -12,10 +12,11 @@ module Coelacanth::Client
 
     def get_response(url = nil)
       @status_code = remote_client.network.status
-      @origin_response = remote_client
-      body = remote_client.body
       remote_client.network.wait_for_idle! # might raise an error
+      body = remote_client.body
       body
+    rescue => e
+      raise "#{e.class}: #{e.message} RemoteClient: #{@remote_client.inspect}"
     end
 
     def get_screenshot
@@ -23,6 +24,9 @@ module Coelacanth::Client
       remote_client.network.wait_for_idle! # まずJSの完了を待つ
       remote_client.screenshot(path: tempfile.path, format: "png")
       File.read(tempfile.path)
+    rescue => e
+      tempfile.close
+      raise "#{e.class}: #{e.message} RemoteClient: #{@remote_client.inspect}"
     end
 
     private

data/lib/coelacanth/dom.rb

@@ -5,8 +5,9 @@ require "oga"
 module Coelacanth
   # Coelacanth::Dom
   class Dom
-    def oga(url)
-      Oga.parse_xml(Net::HTTP.get_response(URI.parse(url)).body)
+    def oga(url, html: nil)
+      html ||= Net::HTTP.get_response(URI.parse(url)).body
+      Oga.parse_xml(html)
     end
   end
 end

data/lib/coelacanth/extractor/fallback_probe.rb

@@ -0,0 +1,34 @@
+# frozen_string_literal: true
+
+require "oga"
+
+require_relative "utilities"
+
+module Coelacanth
+  class Extractor
+    # Attempts final recovery strategies when all other probes fail.
+    class FallbackProbe
+      Result = Struct.new(
+        :title,
+        :node,
+        :published_at,
+        :byline,
+        :source_tag,
+        :confidence,
+        keyword_init: true
+      )
+
+      def call(doc:, url: nil)
+        body = doc.at_css("body") || doc
+        Result.new(
+          title: doc.at_css("title")&.text&.strip,
+          node: body,
+          published_at: nil,
+          byline: nil,
+          source_tag: :fallback,
+          confidence: 0.35
+        )
+      end
+    end
+  end
+end

data/lib/coelacanth/extractor/heuristic_probe.rb

@@ -0,0 +1,175 @@
+# frozen_string_literal: true
+
+require "oga"
+
+require_relative "utilities"
+
+module Coelacanth
+  class Extractor
+    # Scores DOM nodes based on simple heuristics to locate the primary article body.
+    class HeuristicProbe
+      Result = Struct.new(
+        :title,
+        :node,
+        :published_at,
+        :byline,
+        :source_tag,
+        :confidence,
+        keyword_init: true
+      )
+
+      BLOCK_SELECTOR = "article, main, section, div".freeze
+      TAG_WEIGHTS = Hash.new(0).merge(
+        "article" => 80,
+        "main" => 60,
+        "section" => 30,
+        "div" => 10
+      ).freeze
+      NEGATIVE_TOKENS = %w[nav footer header sidebar related share menu].freeze
+      POSITIVE_TOKENS = %w[content article body post entry text].freeze
+
+      def call(doc:, url: nil)
+        candidates = doc.css(BLOCK_SELECTOR).map do |node|
+          score_candidate(node)
+        end.compact
+
+        return if candidates.empty?
+
+        best = candidates.max_by { |candidate| candidate[:score] }
+        return if best[:score] < minimum_score
+
+        Result.new(
+          title: title_from_meta(doc),
+          node: expand(best[:node]),
+          published_at: published_at_from_meta(doc),
+          byline: byline_from_meta(doc),
+          source_tag: :heuristic,
+          confidence: confidence(best[:score])
+        )
+      end
+
+      private
+
+      def score_candidate(node)
+        text_length = Utilities.text_length(node)
+        return if text_length < 80
+
+        link_density = Utilities.link_density(node)
+        punct_density = Utilities.punctuation_density(node)
+        tag_weight = TAG_WEIGHTS[node.name]
+        class_weight = class_score(node)
+        depth_penalty = Utilities.depth(node) * 4
+        sibling_bonus = sibling_variance(node)
+
+        score = (
+          text_length * 0.35 +
+          punct_density * 280 -
+          link_density * 160 +
+          tag_weight +
+          class_weight +
+          sibling_bonus -
+          depth_penalty
+        )
+
+        { node: node, score: score }
+      end
+
+      def minimum_score
+        95
+      end
+
+      def class_score(node)
+        tokens = Utilities.class_id_tokens(node)
+        score = tokens.count { |token| POSITIVE_TOKENS.include?(token) } * 40
+        score -= tokens.count { |token| NEGATIVE_TOKENS.include?(token) } * 60
+        score
+      end
+
+      def sibling_variance(node)
+        parent = node.parent
+        return 0 unless parent
+
+        siblings = Utilities.element_children(parent)
+        return 0 if siblings.length < 2
+
+        lengths = siblings.map { |sibling| Utilities.text_length(sibling) }
+        mean = lengths.sum.to_f / lengths.length
+        variance = lengths.map { |length| (length - mean)**2 }.sum.to_f / lengths.length
+        Math.sqrt(variance) * 0.25
+      end
+
+      def expand(node)
+        return node unless node.parent
+
+        before = neighboring_nodes(node, -1).reverse
+        after = neighboring_nodes(node, 1)
+
+        wrap_fragment(before + [node] + after)
+      end
+
+      def neighboring_nodes(node, direction)
+        siblings = []
+        current = node
+        loop do
+          current = if direction.negative?
+                      Utilities.previous_element(current)
+                    else
+                      Utilities.next_element(current)
+                    end
+          break unless current
+
+          break unless include_in_expansion?(current)
+
+          siblings << current
+        end
+        siblings
+      end
+
+      def include_in_expansion?(node)
+        %w[h1 h2 h3 h4 h5 h6 img blockquote p ul ol figure].include?(node.name)
+      end
+
+      def wrap_fragment(nodes)
+        container = Oga::XML::Element.new(name: "article")
+        nodes.each { |node| container.children << node }
+        container
+      end
+
+      def confidence(score)
+        return 0.0 if score.to_f <= 0.0
+
+        value = 1.0 / (1.0 + Math.exp(-(score - 100) / 12.0))
+        value.clamp(0.0, 0.95)
+      end
+
+      def title_from_meta(doc)
+        Utilities.meta_content(
+          doc,
+          "meta[property='og:title']",
+          "meta[name='twitter:title']",
+          "meta[name='title']"
+        ) || doc.at_css("title")&.text&.strip
+      end
+
+      def published_at_from_meta(doc)
+        Utilities.parse_time(
+          Utilities.meta_content(
+            doc,
+            "meta[property='article:published_time']",
+            "meta[name='pubdate']",
+            "meta[name='publish_date']",
+            "meta[name='date']"
+          )
+        )
+      end
+
+      def byline_from_meta(doc)
+        Utilities.meta_content(
+          doc,
+          "meta[name='author']",
+          "meta[property='article:author']"
+        )
+      end
+    end
+  end
+end

data/lib/coelacanth/extractor/image_collector.rb

@@ -0,0 +1,19 @@
+# frozen_string_literal: true
+
+module Coelacanth
+  class Extractor
+    # Collects image metadata from the extracted DOM node.
+    class ImageCollector
+      def call(node)
+        return [] unless node
+
+        node.css("img").map do |image|
+          {
+            src: image["src"].to_s.strip,
+            alt: image["alt"].to_s.strip
+          }
+        end.reject { |entry| entry[:src].empty? }
+      end
+    end
+  end
+end