gento 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: bb89ec6dceb5b98779d471fc330c3ac141fec9fe45bc870696f7b947496db6d2
+  data.tar.gz: dc3aee5c63e67925826b74f43d28144d96525f9bf6f7d3099ba83a7ef987cf99
+SHA512:
+  metadata.gz: 97e1e10f67231d1cdf2ec41ddd061fd6e2d052daab3c9acd7a9c5fa451ab9ea3452af191e919259bf341b546ab5520c6b3ee52de16866845ad7158dfe58a82b9
+  data.tar.gz: 9ffbe5105b485bc84592630b9b206f6167365e4e8f30b0dd3536c269873794bb08b52eaf3b273b14d6b2d6273bd46e296fbafd7ed085925a9a44e71476714367

data/CHANGELOG.md

@@ -0,0 +1,53 @@
+# Changelog
+
+## 0.1.0
+
+### Added
+
+- `Gento::Client#fetch`, which turns a slide URL into a `Deck` of
+  ordered page images.
+- Adapters for Speaker Deck, SlideShare, Docswell and public Google Slides.
+- A pure-Ruby HTML scanner, so the gem carries no native extensions and no
+  runtime dependencies.
+- A pluggable `Fetcher` seam, with a net/http implementation that guards
+  against redirects into private address ranges.
+- Charset detection, so Shift_JIS and EUC-JP decks decode correctly.
+- `gento` CLI, printing JSON or one image URL per line.
+
+- robots.txt, obeyed by default (RFC 9309). A `RobotsFetcher` wraps whatever
+  fetcher is in use, so every request an adapter makes is checked, not just
+  the deck page. `robots: false`, `--no-robots` and `GENTO_ROBOTS=off`
+  turn it off. A robots.txt that cannot be read is treated as a refusal
+  rather than as permission.
+
+- Proxy support in the net/http fetcher, honouring `HTTPS_PROXY` (which
+  net/http ignores on its own) and bypassing it for private addresses.
+
+- The demo's viewer: a lightbox on clicking a page, with arrow-key paging,
+  and a switch between the thumbnail grid and a continuous vertical read.
+  The choice is remembered. Built on `<dialog>` with no dependencies, and
+  layered over markup that still works with JavaScript off.
+- `GENTO_USER_AGENT`, so a deployed container can change how it
+  identifies itself without a rebuild.
+
+### Fixed
+
+- The tag scanner consumed element content, so a tag nested inside another of
+  the same name was invisible — which is most of the markup on a real page.
+- Typographic entities such as `&hellip;` and `&mdash;` were left undecoded.
+- Titles carrying the author's own line breaks are collapsed to single spaces.
+
+- Google Slides decks published to the web return the signed `viewpage` URLs
+  embedded in the document. `/export/png` answers 404 for those decks, so the
+  URLs it would have built were unusable. Ordinary shared decks keep using
+  `/export/png`, which does not expire.
+
+### Known gaps
+
+- SlideShare's bot protection decides on the caller's IP reputation, so from
+  a datacenter address it can refuse every request. Detected and reported as
+  itself; getting past it needs a JavaScript-capable `Fetcher`.
+- The signed `viewpage` URLs returned for published Google Slides decks
+  expire; how long they last has not been measured.
+- The robots.txt check does not follow redirects: the wrapped fetcher handles
+  those internally, so a redirect into a disallowed path is not caught.

data/LICENSE.txt

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 ngram
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,143 @@
+# gento
+
+スライド共有サービスの URL を渡すと、各ページの画像 URL を順番に並べて返す Ruby gem です。
+
+対応サービス: Speaker Deck ・ SlideShare ・ Docswell ・ Google スライド（公開設定のもの）
+
+ランタイム依存ゼロ・ネイティブ拡張ゼロで動きます。
+
+## インストール
+
+```console
+$ gem install gento
+```
+
+Gemfile なら:
+
+```ruby
+gem "gento"
+```
+
+## 使いかた
+
+```ruby
+require "gento"
+
+deck = Gento.fetch("https://speakerdeck.com/user/talk")
+
+deck.title       # => "快適なスライド閲覧生活を実現する Web サービスの開発"
+deck.author      # => "ngram"
+deck.page_count  # => 42
+deck.slides.map(&:url)
+# => ["https://files.speakerdeck.com/.../slide_0.jpg", ...]
+```
+
+`Deck` と `Slide` は不変オブジェクトで、`to_h` / `to_json` でそのまま出力できます。
+
+対応している URL かどうかは、取得する前に確認できます。
+
+```ruby
+Gento.supports?("https://speakerdeck.com/user/talk")  # => true
+```
+
+### コマンドライン
+
+```console
+$ gento https://speakerdeck.com/user/talk
+{
+  "provider": "speaker_deck",
+  "source_url": "https://speakerdeck.com/user/talk",
+  "title": "...",
+  "page_count": 42,
+  "slides": [ ... ]
+}
+
+$ gento --format urls https://speakerdeck.com/user/talk
+https://files.speakerdeck.com/presentations/.../slide_0.jpg
+...
+```
+
+### robots.txt
+
+**取得先の `robots.txt` を既定で参照し、拒否されているパスは取りに行きません。**
+デッキページだけでなく、アダプタが出すすべてのリクエスト（oEmbed、embed ビューなど）が対象です。
+
+```ruby
+Gento.fetch(url)                 # robots.txt に従う（既定）
+Gento.fetch(url, robots: false)  # 従わない（判断は利用者の責任）
+```
+
+```console
+$ gento --no-robots https://speakerdeck.com/user/talk
+```
+
+拒否された場合は `Gento::RobotsDisallowedError` になります。
+`robots.txt` 自体を読めなかったとき（5xx・429・接続失敗）も、RFC 9309 に従って拒否扱いです。
+
+詳細は [doc/robots.md](doc/robots.md) を参照してください。
+
+### HTTP クライアントの差し替え
+
+通信はすべて `Gento::Fetcher` を経由するので、独自の HTTP スタックを差し込めます。
+
+```ruby
+Gento.fetch(url, fetcher: MyFetcher.new)
+```
+
+詳細は [doc/design.md](doc/design.md) を参照してください。
+
+### エラー
+
+| 例外 | 起きるとき |
+| --- | --- |
+| `Gento::UnsupportedURLError` | 対応アダプタのない URL |
+| `Gento::RobotsDisallowedError` | `robots.txt` が許可していない（`FetchError` の一種） |
+| `Gento::FetchError` | 取得に失敗（`ResponseError` を含む） |
+| `Gento::ExtractionError` | 取得はできたがページを取り出せない |
+
+いずれも `Gento::Error` を継承しています。
+
+**SlideShare は接続元 IP によってボット判定で弾かれることがあり**、その場合は
+`ExtractionError` になります。データセンターの IP からは恒常的に弾かれます
+（[doc/verification.md](doc/verification.md#slideshare-のボット判定について)）。
+
+## 利用にあたっての注意
+
+**対象サービスの利用規約を確認し、遵守する責任は利用者にあります。**
+[doc/legal.md](doc/legal.md) を読んでから使ってください。
+
+## ドキュメント
+
+- [robots.txt の扱い](doc/robots.md) — 判定ルール、無効化、各サービスの実際の内容
+- [設計方針](doc/design.md) — 依存ゼロにした理由、Fetcher の差し替え、アダプタの足し方
+- [検証状況と既知の制限](doc/verification.md) — サービスごとの注意点、SlideShare のボット判定
+- [手元で試す](doc/development.md) — `main` を取得して gem・CLI・デモアプリを動かす
+- [デモ Web サービス](doc/demo.md) — `web/` の動かしかたと Cloudflare へのデプロイ
+- [利用にあたっての注意](doc/legal.md)
+- [RubyGems への公開手順](doc/publishing.md)
+
+## 開発
+
+```console
+$ bundle install
+$ bundle exec rspec      # gem         151 examples
+$ bundle exec rubocop
+$ (cd web && bundle exec rspec)                 # デモアプリ 27 examples
+$ (cd worker && npm test && npm run typecheck)  # Worker     15 tests
+```
+
+GitHub Codespaces なら `.devcontainer/` が上記をすべて用意します
+（[doc/codespaces.md](doc/codespaces.md)）。
+
+## 由来
+
+技術書典5 で頒布した『てっくやみなべ Vol.1』所収
+「快適なスライド閲覧⽣活を実現する Web サービスの開発」のサポートリポジトリを、
+現在も動く形で作り直したものです。当時の Lambda / PaaS 向けデプロイコードは、
+対象サービスが終了したため Cloudflare 向けに置き換えています。
+
+名前の gento は、スライドを映写して見せる「幻灯（げんとう）」から取っています。
+
+## ライセンス
+
+MIT License. [LICENSE.txt](LICENSE.txt) を参照してください。

data/exe/gento

@@ -0,0 +1,7 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+require "gento"
+require "gento/cli"
+
+exit Gento::CLI.start(ARGV)

data/lib/gento/adapters/base.rb

@@ -0,0 +1,121 @@
+# frozen_string_literal: true
+
+require "uri"
+
+module Gento
+  module Adapters
+    # Shared behaviour for the per-site adapters.
+    #
+    # An adapter answers two questions: "is this URL mine?" and "what are the
+    # pages of this deck?". Everything else — HTTP, value objects, JSON output
+    # — is the framework's job.
+    class Base
+      class << self
+        # Host suffixes this adapter claims, e.g. %w[speakerdeck.com].
+        def hosts
+          raise NotImplementedError, "#{self} must declare .hosts"
+        end
+
+        def provider
+          name.split("::").last
+              .gsub(/([a-z\d])([A-Z])/, '\1_\2')
+              .downcase
+        end
+
+        def handles?(url)
+          uri = coerce_uri(url)
+          return false unless uri&.host
+
+          host = uri.host.downcase.sub(/\Awww\./, "")
+          hosts.any? { |claimed| host == claimed || host.end_with?(".#{claimed}") }
+        end
+
+        def coerce_uri(url)
+          uri = url.is_a?(URI) ? url : URI.parse(url.to_s)
+          uri.is_a?(URI::HTTP) ? uri : nil
+        rescue URI::InvalidURIError
+          nil
+        end
+      end
+
+      attr_reader :fetcher
+
+      def initialize(fetcher:)
+        @fetcher = fetcher
+      end
+
+      # @return [Gento::Deck]
+      def fetch(url)
+        raise NotImplementedError, "#{self.class} must implement #fetch"
+      end
+
+      private
+
+      def provider
+        self.class.provider
+      end
+
+      def get(url, headers: {})
+        fetcher.get(url, headers: headers)
+      end
+
+      def absolute_url(candidate, base)
+        return nil if candidate.nil? || candidate.to_s.empty?
+
+        URI.join(base.to_s, candidate.to_s).to_s
+      rescue URI::Error
+        nil
+      end
+
+      # Title/author/description as advertised by OpenGraph, which every one of
+      # these sites emits and keeps stable for the sake of link previews.
+      def open_graph_metadata(html)
+        {
+          title: html.meta("og:title", "twitter:title") || html.title,
+          description: html.meta("og:description", "description"),
+          author: html.meta("og:author", "author", "article:author")
+        }
+      end
+
+      # The oEmbed endpoint a page advertises in its head, if any.
+      #
+      # Every site here publishes one, so discovering it beats hardcoding a
+      # URL per adapter: the site tells us where its own endpoint lives.
+      def oembed_endpoint(html)
+        html.tags("link")
+            .find { |attrs| attrs["type"].to_s.include?("json+oembed") }
+            &.fetch("href", nil)
+      end
+
+      # oEmbed metadata, or an empty hash.
+      #
+      # It is always a nice-to-have: cleaner titles and real author names than
+      # OpenGraph offers, but never something a fetch should fail over.
+      def fetch_oembed(html)
+        endpoint = oembed_endpoint(html)
+        return {} unless endpoint
+
+        get(endpoint, headers: { "accept" => "application/json" }).json
+      rescue FetchError, ExtractionError
+        {}
+      end
+
+      def build_slides(urls, width: nil, height: nil)
+        urls.each_with_index.map do |url, index|
+          Slide.new(number: index + 1, url: url, width: width, height: height)
+        end
+      end
+
+      # The same image is often referenced at several sizes through a query
+      # string (`?width=160`, a cache-busting timestamp). Dropping the query
+      # before deduplicating collapses those into one page.
+      def without_query(url)
+        url.to_s.split("?").first.to_s
+      end
+
+      def fail_extraction(message)
+        raise ExtractionError, "#{provider}: #{message}"
+      end
+    end
+  end
+end

data/lib/gento/adapters/docswell.rb

@@ -0,0 +1,58 @@
+# frozen_string_literal: true
+
+module Gento
+  module Adapters
+    # Docswell (https://www.docswell.com/s/<user>/<id>-<slug>).
+    #
+    # The deck page only renders the first handful of pages; the rest arrive
+    # by script. The embed view it links to carries the whole deck, so that is
+    # what this adapter reads. Each page appears there twice, full size and as
+    # a `?width=160` thumbnail, which dropping the query collapses.
+    class Docswell < Base
+      EMBED_URL = %r{\Ahttps://(?:www\.)?docswell\.com/slide/[A-Za-z0-9]+/embed}i
+      PAGE_IMAGE = %r{\Ahttps://[a-z0-9.-]*docswell\.com/page/[^/]+\.(?:jpe?g|png|webp)}i
+
+      def self.hosts
+        %w[docswell.com]
+      end
+
+      def fetch(url)
+        page = get(url).html
+
+        # The page links the embed both plainly and with a ?mode= variant.
+        embed_url = page.urls(EMBED_URL).map { |found| without_query(found) }.first
+        fail_extraction("could not find the embed view for #{url}") unless embed_url
+
+        images = slide_images(embed_url)
+        fail_extraction("found no slide images for #{url}") if images.empty?
+
+        build_deck(url, page, images)
+      end
+
+      private
+
+      # Page filenames are opaque ids, so document order is the only ordering
+      # available. The embed view lists the pages in slide order, which the
+      # deck's og:image corroborates: it is always the first page.
+      def slide_images(embed_url)
+        get(embed_url).html.urls(PAGE_IMAGE).map { |url| without_query(url) }.uniq
+      end
+
+      # The deck page carries no author metadata at all, so oEmbed is the only
+      # place a name comes from here.
+      def build_deck(url, page, images)
+        metadata = open_graph_metadata(page)
+        oembed = fetch_oembed(page)
+
+        Deck.new(
+          provider: provider,
+          source_url: url,
+          title: oembed["title"] || metadata[:title],
+          author: oembed["author_name"] || metadata[:author],
+          description: metadata[:description],
+          slides: build_slides(images)
+        )
+      end
+    end
+  end
+end

data/lib/gento/adapters/google_slides.rb

@@ -0,0 +1,90 @@
+# frozen_string_literal: true
+
+module Gento
+  module Adapters
+    # Google Slides, for decks shared publicly or published to the web.
+    #
+    # Google does not render page images into the deck page, so the job here
+    # is to find the ordered list of pages and name an image for each.
+    #
+    # The viewer at /embed is the obvious place to look and the wrong one: its
+    # payload mixes page ids with the ids of elements drawn on those pages,
+    # with nothing in the id itself to tell them apart. /htmlpresent instead
+    # references each page exactly once, in order, so that is what we read.
+    class GoogleSlides < Base
+      BASE_URL = "https://docs.google.com/presentation/d/%s"
+
+      # /presentation/d/<id>/... and the published /presentation/d/e/<id>/...
+      # Published ids contain hyphens, ordinary ones do not.
+      PRESENTATION_ID = %r{/presentation/d/(e/)?([A-Za-z0-9_-]{16,})}
+      PAGE_ID = /[?&]pageid=([A-Za-z0-9_]+)/
+      VIEWPAGE = %r{/viewpage\?}
+
+      def self.hosts
+        %w[docs.google.com]
+      end
+
+      def fetch(url)
+        published, id = presentation_id(url)
+        fail_extraction("could not find a presentation id in #{url}") unless id
+
+        page = get("#{deck_base(published, id)}/htmlpresent").html
+        images = published ? viewpage_images(page, url) : export_images(page, url, published, id)
+
+        Deck.new(
+          provider: provider,
+          source_url: url,
+          title: title_from(page),
+          slides: build_slides(images)
+        )
+      end
+
+      private
+
+      def deck_base(published, id)
+        format(BASE_URL, published ? "e/#{id}" : id)
+      end
+
+      # Ordinary decks: build export URLs from the page ids. These are stable,
+      # so a caller can keep them, and they redirect to a signed image.
+      def export_images(page, url, published, id)
+        base = deck_base(published, id)
+
+        page_ids(page, url).map { |page_id| "#{base}/export/png?id=#{id}&pageid=#{page_id}" }
+      end
+
+      # Published decks: /export/png answers 404 for them, whatever page id it
+      # is given. The only thing that serves their pages is the viewpage URL
+      # embedded in the document, which carries its own signature — so take it
+      # verbatim rather than trying to construct one.
+      #
+      # Those signatures expire, so unlike the export URLs these are worth
+      # fetching promptly rather than storing.
+      def viewpage_images(page, url)
+        images = page.urls(VIEWPAGE)
+        fail_extraction("found no slide pages for #{url}; is the deck still published?") if images.empty?
+
+        images
+      end
+
+      # Each page is referenced exactly once here, in order.
+      def page_ids(page, url)
+        ids = page.source.scan(PAGE_ID).flatten.uniq
+        fail_extraction("found no slide pages for #{url}; is the deck shared publicly?") if ids.empty?
+
+        ids
+      end
+
+      def presentation_id(url)
+        match = url.to_s.match(PRESENTATION_ID)
+        return [false, nil] unless match
+
+        [!match[1].nil?, match[2]]
+      end
+
+      def title_from(page)
+        page.title&.sub(/\s*-\s*Google (?:Slides|スライド)\s*\z/, "")
+      end
+    end
+  end
+end

data/lib/gento/adapters/slide_share.rb

@@ -0,0 +1,82 @@
+# frozen_string_literal: true
+
+module Gento
+  module Adapters
+    # SlideShare (https://www.slideshare.net/<user>/<slug>).
+    #
+    # SlideShare usually serves the deck to an ordinary HTTP client, but it
+    # sits behind bot protection that intermittently answers with a small
+    # JavaScript interstitial instead — most readily under bursty access.
+    #
+    # That interstitial has no slides in it, so without recognising it the
+    # adapter would report an empty deck and leave the caller guessing. Naming
+    # it costs one check and turns a mystery into something actionable: wait,
+    # or supply a JavaScript-capable Fetcher.
+    class SlideShare < Base
+      CHALLENGE = /Client Challenge|_fs-ch-/
+      # <slug>-<page>-<width>.jpg on the slide CDN.
+      SLIDE_IMAGE = %r{\Ahttps://[a-z0-9.-]*slidesharecdn\.com/\S+-(\d+)-(\d+)\.(?:jpe?g|png|webp)}i
+
+      def self.hosts
+        %w[slideshare.net]
+      end
+
+      def fetch(url)
+        page = get(url).html
+        fail_challenge(url) if challenge?(page)
+
+        images = slide_images(page)
+        fail_extraction("found no slide images at #{url}") if images.empty?
+
+        build_deck(url, page, images)
+      end
+
+      private
+
+      def challenge?(page)
+        page.title.to_s.match?(CHALLENGE) || page.source.match?(CHALLENGE)
+      end
+
+      def fail_challenge(url)
+        fail_extraction(
+          "#{url} returned SlideShare's JavaScript bot challenge instead of the deck. " \
+          "Fetching from SlideShare needs a JavaScript-capable Gento::Fetcher; " \
+          "the default net/http one cannot get past it."
+        )
+      end
+
+      # Pages are published at several widths, not always the same set for
+      # every page, so keep the widest copy of each.
+      def slide_images(page)
+        page.urls(SLIDE_IMAGE)
+            .group_by { |image| Integer(image[SLIDE_IMAGE, 1]) }
+            .sort_by(&:first)
+            .map { |_, group| group.max_by { |image| Integer(image[SLIDE_IMAGE, 2]) } }
+      end
+
+      def build_deck(url, page, images)
+        metadata = open_graph_metadata(page)
+        document = json_ld_document(page)
+
+        Deck.new(
+          provider: provider,
+          source_url: url,
+          title: metadata[:title],
+          author: author_from(document) || metadata[:author],
+          description: metadata[:description],
+          published_at: document["datePublished"],
+          slides: build_slides(images)
+        )
+      end
+
+      def json_ld_document(page)
+        page.json_ld.find { |node| node["@type"].to_s.match?(/Presentation|Article|CreativeWork/i) } || {}
+      end
+
+      def author_from(document)
+        author = document["author"]
+        author.is_a?(Hash) ? author["name"] : author
+      end
+    end
+  end
+end

data/lib/gento/adapters/speaker_deck.rb

@@ -0,0 +1,79 @@
+# frozen_string_literal: true
+
+module Gento
+  module Adapters
+    # Speaker Deck (https://speakerdeck.com/<user>/<slug>).
+    #
+    # The deck page carries every page image at full size, in <a href>
+    # elements. It also carries cover images for roughly thirty recommended
+    # decks, so matching on the CDN hostname alone would silently splice other
+    # people's slides into the result. Keying on this deck's own presentation
+    # id is what keeps them out.
+    class SpeakerDeck < Base
+      PLAYER_ID = %r{/player/([0-9a-f-]+)}i
+      DECK_ID = /\A[0-9a-f-]{8,}\z/i
+
+      def self.hosts
+        %w[speakerdeck.com]
+      end
+
+      def fetch(url)
+        page = get(url).html
+        oembed = fetch_oembed(page)
+
+        id = deck_id(page, oembed)
+        fail_extraction("could not find the presentation id for #{url}") unless id
+
+        images = slide_images(page, id)
+        fail_extraction("found no slide images at #{url}") if images.empty?
+
+        build_deck(url, page, oembed, images)
+      end
+
+      private
+
+      # Full-size pages only: the same presentation also publishes
+      # `preview_slide_N.jpg` thumbnails, which this pattern excludes.
+      def slide_images(page, id)
+        pattern = %r{
+          \Ahttps://files\.speakerdeck\.com/presentations/
+          #{Regexp.escape(id)}/slide_(\d+)\.[a-z]+
+        }xi
+
+        page.urls(pattern)
+            .map { |url| without_query(url) }
+            .uniq
+            .sort_by { |url| Integer(url[pattern, 1]) }
+      end
+
+      # oEmbed names the id inside a player URL; the page itself carries it
+      # bare, as the data-id of the embed placeholder.
+      def deck_id(page, oembed)
+        player_id(oembed["html"]) || embed_data_id(page) || player_id(page.source)
+      end
+
+      def player_id(text)
+        text.to_s[PLAYER_ID, 1]
+      end
+
+      def embed_data_id(page)
+        page.tags("div")
+            .filter_map { |attrs| attrs["data-id"] }
+            .find { |id| id.match?(DECK_ID) }
+      end
+
+      def build_deck(url, page, oembed, images)
+        metadata = open_graph_metadata(page)
+
+        Deck.new(
+          provider: provider,
+          source_url: url,
+          title: oembed["title"] || metadata[:title],
+          author: oembed["author_name"] || metadata[:author],
+          description: metadata[:description],
+          slides: build_slides(images)
+        )
+      end
+    end
+  end
+end