dubhe 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: c44a769b6d12785140253a7c9826cbc8c88e67a7f629abf6cbcdf27a048f9070
+  data.tar.gz: 1140b729850ab4896396f9dc759b2fd5863f6781d00d9989a9bc92251d52b898
+SHA512:
+  metadata.gz: fd7c98dcfc0feacc68d034a916724d5cde6854aa1f8f0425725e75e4d6413cf1ad30dc9cba2140fa7ec835519bf68680a10a94c31716601055859558b812487f
+  data.tar.gz: 1610be6a804566d45387b4566fb017fa563808acd3bb8a65f6aeaa89a68b4c53480b91a596ced2ba40fad9fd089aa2464c804a74fc7c63a2d5e699b29b187024

data/CHANGELOG.md

@@ -0,0 +1,5 @@
+# Changelog
+
+## 0.1.0 — 2026-09-21
+
+- Initial release.

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2026 ydah
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,58 @@
+# Dubhe
+
+Dubhe (α Ursae Majoris, from Arabic *dubb*, “bear”) is a feeds-first reader
+core for static documents, RSS/Atom/JSON Feed subscriptions, and local mail.
+It does not execute JavaScript, embed a WebView, or send mail.
+
+## Features
+
+- File-backed append-only entry/state indexes and content bodies
+- RSS, Atom, and JSON Feed subscriptions through Iklil
+- Conditional HTTP requests with ETag/Last-Modified and bounded redirects
+- Safe HTML policy through Jabbah: XSS removal and remote-image blocking
+- Feed discovery and OPML import/export
+- Local HTML/Markdown docs with root-confined path resolution
+- Ukdah-backed `.eml`/`.mbox` source API with CID image replacement
+- Small reader/history/search API and plain-text reader-mode rendering
+
+## Installation
+
+```ruby
+gem "dubhe"
+```
+
+```sh
+gem install dubhe
+```
+
+Dubhe requires Ruby 3.2 or newer. It uses Iklil, Jabbah, and Ukdah for
+parsing, while HTTP and persistence use Ruby's standard library.
+
+## Quick start
+
+```ruby
+require "dubhe"
+
+store = Dubhe::Store.new
+feeds = Dubhe::Sources::Feeds.new(store: store)
+feeds.add("https://example.test/feed.xml")
+feeds.refresh
+
+feeds.entries.each { |entry| puts entry.title }
+```
+
+Use `Dubhe::Policy` before displaying arbitrary HTML. Remote images are
+blocked by default and `Result#blocked_count` is intended for a visible notice
+in the UI. Dubhe is a reader core; a terminal or GPU view is an application
+concern.
+
+## Development
+
+```sh
+bundle install
+bundle exec rake test
+```
+
+## License
+
+Dubhe is released under the [MIT License](LICENSE.txt).

data/docs/adr/001-reader-core-boundaries.md

@@ -0,0 +1,25 @@
+# ADR 001: Keep the reader core source-oriented
+
+- Status: accepted
+- Date: 2026-09-21
+- Author: Yudai Takada
+
+## Context
+
+Docs, feeds, and mail share fetching, normalization, safe HTML handling,
+storage, and history. Their input parsers are separate libraries, while the
+reader needs one small source interface.
+
+## Decision
+
+Dubhe owns source orchestration, policies, file-backed storage, HTTP fetching,
+and reader state. Iklil owns feed parsing, Ukdah owns MIME parsing, and Jabbah
+owns HTML parsing/sanitization. The v1 renderer is intentionally plain and
+does not inspect author CSS. JavaScript and WebView embedding are out of scope.
+
+## Consequences
+
+The feed path is useful without a GUI and remains testable with deterministic
+fetchers. A future UI can consume `Entry`, `Body`, and `Source` without
+reimplementing security or persistence. A richer CSS renderer can be added
+later without changing source contracts.

data/docs/supported-elements.md

@@ -0,0 +1,9 @@
+# Reader-mode HTML elements
+
+The v1 plain renderer handles `h1`–`h6`, `p`, `br`, `hr`, `ul`, `ol`, `li`,
+`blockquote`, `pre`, `code`, `table`, `tr`, `td`, `th`, `img`, `a`, `em`,
+`strong`, `del`, `mark`, `sub`, `sup`, `figure`, and `figcaption`.
+
+Unknown elements contribute their children. `script`, `style`, `iframe`,
+`object`, `embed`, and `form` contribute nothing. Author `style` and `class`
+attributes are never used for layout or typography.

data/lib/dubhe/fetch.rb

@@ -0,0 +1,49 @@
+# frozen_string_literal: true
+
+require "net/http"
+require "uri"
+
+module Dubhe
+  module Fetch
+    Response = Data.define(:status, :headers, :body, :uri)
+
+    class Client
+      DEFAULT_USER_AGENT = "Dubhe/0.1 (+https://github.com/noxdea/dubhe)"
+      REDIRECTS = %w[301 302 303 307 308].freeze
+
+      def initialize(user_agent: DEFAULT_USER_AGENT, open_timeout: 10, read_timeout: 30, max_bytes: 10 * 1024 * 1024)
+        @user_agent = user_agent
+        @open_timeout = open_timeout
+        @read_timeout = read_timeout
+        @max_bytes = max_bytes
+      end
+
+      def get(url, headers: {}, redirects: 0)
+        uri = URI.parse(url.to_s)
+        raise Dubhe::Error, "only HTTP(S) URLs are supported" unless %w[http https].include?(uri.scheme)
+        raise Dubhe::Error, "too many redirects" if redirects > 5
+
+        request = Net::HTTP::Get.new(uri.request_uri)
+        request["User-Agent"] = @user_agent
+        request["Accept"] = "application/rss+xml, application/atom+xml, application/feed+json, application/json, text/xml, text/html;q=0.8"
+        headers.each { |key, value| request[key.to_s] = value.to_s unless value.nil? || value.to_s.empty? }
+        response = Net::HTTP.start(uri.host, uri.port, use_ssl: uri.scheme == "https",
+          open_timeout: @open_timeout, read_timeout: @read_timeout) do |http|
+          http.request(request)
+        end
+        if REDIRECTS.include?(response.code)
+          location = response["location"]
+          raise Dubhe::Error, "redirect without location" unless location
+
+          return get(URI.join(uri.to_s, location).to_s, headers: headers, redirects: redirects + 1)
+        end
+        body = response.body.to_s
+        raise Dubhe::Error, "response exceeds size limit" if body.bytesize > @max_bytes
+
+        Response.new(status: response.code.to_i, headers: response.each_header.to_h, body: body, uri: uri.to_s)
+      rescue SocketError, Timeout::Error, IOError, SystemCallError => error
+        raise Dubhe::Error, "fetch failed for #{url}: #{error.message}"
+      end
+    end
+  end
+end

data/lib/dubhe/models.rb

@@ -0,0 +1,15 @@
+# frozen_string_literal: true
+
+module Dubhe
+  Entry = Data.define(:id, :source_id, :title, :author, :published_at, :updated_at,
+    :url, :summary, :tags, :state, :parent_id)
+  Body = Data.define(:kind, :content, :base_url, :attachments, :headers)
+
+  Subscription = Data.define(:id, :title, :url, :site_url, :group, :etag, :last_modified,
+    :interval, :next_refresh_at, :failures)
+
+  class Entry
+    def read? = state == :read
+    def starred? = state == :starred
+  end
+end

data/lib/dubhe/policy.rb

@@ -0,0 +1,58 @@
+# frozen_string_literal: true
+
+require "pathname"
+require "uri"
+require "jabbah"
+
+module Dubhe
+  class Policy
+    class Denied < Dubhe::Error; end
+    Result = Data.define(:html, :document, :blocked_count)
+
+    attr_reader :profile
+
+    def initialize(profile: :feed)
+      @profile = profile
+    end
+
+    def apply(input, base_url: nil, allow_remote_images: false, on_blocked: nil)
+      urls = []
+      profile_options = Jabbah::Sanitize::PROFILES.fetch(profile.to_sym, Jabbah::Sanitize::DEFAULT).merge(
+        allow_remote_images: allow_remote_images)
+      document = Jabbah::Sanitize.clean(input, profile: profile_options, base_url: base_url,
+        on_blocked: lambda do |url|
+          urls << url
+          on_blocked&.call(url)
+        end,
+        allow: Jabbah::Sanitize::DEFAULT)
+      Result.new(html: document.to_html, document: document, blocked_count: urls.length)
+    end
+
+    def resolve_local(root, path)
+      root_path = Pathname.new(root.to_s).expand_path
+      raise Denied, "source root does not exist: #{root}" unless root_path.directory?
+
+      candidate = root_path.join(path.to_s).cleanpath
+      root_real = root_path.realpath
+      candidate_real = candidate.realpath
+      return candidate_real.to_s if candidate_real == root_real || candidate_real.to_s.start_with?("#{root_real}#{File::SEPARATOR}")
+
+      raise Denied, "path escapes source root: #{path}"
+    rescue Errno::ENOENT
+      # Check the nearest existing parent so a new path cannot escape via .. .
+      parent = candidate
+      parent = parent.parent until parent.exist? || parent.root?
+      parent_real = parent.realpath
+      return candidate.to_s if parent_real == root_real || parent_real.to_s.start_with?("#{root_real}#{File::SEPARATOR}")
+
+      raise Denied, "path escapes source root: #{path}"
+    end
+
+    def external_url?(url)
+      scheme = URI.parse(url.to_s).scheme.to_s.downcase
+      %w[http https mailto].include?(scheme)
+    rescue URI::InvalidURIError
+      false
+    end
+  end
+end

data/lib/dubhe/reader.rb

@@ -0,0 +1,85 @@
+# frozen_string_literal: true
+
+module Dubhe
+  class Reader
+    attr_reader :sources, :current_entry
+
+    def initialize(sources: [], store: Store.new)
+      @store = store
+      @sources = Array(sources)
+      @history = []
+      @history_index = -1
+      @current_entry = nil
+    end
+
+    def add_source(source)
+      @sources << source
+      source
+    end
+
+    def refresh
+      @sources.flat_map(&:refresh)
+    end
+
+    def entries(source_id: nil, filter: nil)
+      result = if source_id
+                 @sources.find { |source| source.id == source_id }&.entries(filter: filter) || []
+               else
+                 @sources.flat_map { |source| source.entries(filter: filter) }
+               end
+      result.sort_by { |entry| entry.updated_at || entry.published_at || Time.at(0) }.reverse
+    end
+
+    def open(entry)
+      source = @sources.find { |candidate| candidate.id == entry.source_id }
+      raise Error, "source not found: #{entry.source_id}" unless source
+
+      @current_entry = entry
+      @history = @history[0..@history_index] if @history_index >= 0
+      @history << entry
+      @history_index = @history.length - 1
+      source.fetch_body(entry)
+    end
+
+    def back
+      return nil if @history_index <= 0
+
+      @history_index -= 1
+      @current_entry = @history[@history_index]
+      body_for(@current_entry)
+    end
+
+    def forward
+      return nil if @history_index >= @history.length - 1
+
+      @history_index += 1
+      @current_entry = @history[@history_index]
+      body_for(@current_entry)
+    end
+
+    def mark_read(entry, read: true)
+      @store.update_state(entry.id, state: read ? :read : :unread)
+      entry
+    end
+
+    def star(entry, starred: true)
+      @store.update_state(entry.id, state: starred ? :starred : :read)
+      entry
+    end
+
+    def search(query, source_id: nil)
+      needle = query.to_s.downcase
+      entries(source_id: source_id).select do |entry|
+        body = @store.body(entry.id)&.content.to_s
+        [entry.title, entry.author, entry.summary, body].compact.join(" ").downcase.include?(needle)
+      end
+    end
+
+    private
+
+    def body_for(entry)
+      source = @sources.find { |candidate| candidate.id == entry.source_id }
+      source&.fetch_body(entry)
+    end
+  end
+end

data/lib/dubhe/render.rb

@@ -0,0 +1,59 @@
+# frozen_string_literal: true
+
+require "cgi/escape"
+
+module Dubhe
+  module Render
+    # EXTRACTION CANDIDATE: document renderer
+    # Canopus and hadar can share this once a second consumer needs it.
+    HANDLERS = {
+      "p" => :paragraph, "h1" => :heading, "h2" => :heading, "h3" => :heading,
+      "h4" => :heading, "h5" => :heading, "h6" => :heading, "li" => :list_item,
+      "blockquote" => :quote, "pre" => :code_block, "hr" => :rule,
+      "br" => :break, "table" => :table, "figure" => :figure
+    }.freeze
+    BLOCKS = (HANDLERS.keys + %w[div section article main header footer ul ol dl dt dd figcaption]).freeze
+
+    module_function
+
+    def call(node, theme: {})
+      root = node.respond_to?(:root) ? node.root : node
+      render_node(root, theme).strip
+    end
+
+    def render_node(node, theme)
+      return CGI.unescapeHTML(node.data.to_s) if node.text?
+      return "" unless node.element?
+      return "" if %w[script style iframe object embed form].include?(node.name)
+
+      content = node.children.map { |child| render_node(child, theme) }.join
+      case HANDLERS[node.name]
+      when :heading then "\n#{content.strip}\n"
+      when :paragraph then "\n#{content.strip}\n"
+      when :list_item then "\n• #{content.strip}"
+      when :quote then "\n#{content.lines.map { |line| "> #{line}" }.join}"
+      when :code_block then "\n```\n#{node.text}\n```\n"
+      when :rule then "\n────────\n"
+      when :break then "\n"
+      when :table then "\n#{render_table(node)}\n"
+      when :figure then "\n#{content.strip}\n"
+      else
+        BLOCKS.include?(node.name) ? "\n#{content}\n" : content
+      end
+    end
+
+    def render_table(node)
+      rows = node.search("tr").map do |row|
+        row.children.select { |child| child.element? && %w[td th].include?(child.name) }.map { |cell| cell.text.strip }
+      end
+      return "" if rows.empty?
+
+      width_count = rows.map(&:length).max.to_i
+      widths = (0...width_count).map { |index| rows.filter_map { |row| row[index]&.length }.max.to_i }
+      rows.map do |row|
+        values = (0...width_count).map { |index| row[index].to_s.ljust(widths[index]) }
+        "| #{values.join(" | ")} |"
+      end.join("\n")
+    end
+  end
+end

data/lib/dubhe/source.rb

@@ -0,0 +1,13 @@
+# frozen_string_literal: true
+
+module Dubhe
+  module Source
+    def id = raise NotImplementedError
+    def title = raise NotImplementedError
+    def icon = nil
+    def refresh = raise NotImplementedError
+    def entries(filter: nil) = raise NotImplementedError
+    def fetch_body(_entry) = raise NotImplementedError
+    def supports?(capability) = %i[refresh search unread].include?(capability)
+  end
+end

data/lib/dubhe/sources/docs.rb

@@ -0,0 +1,125 @@
+# frozen_string_literal: true
+
+require "digest"
+require "cgi/escape"
+require "pathname"
+require "uri"
+require "jabbah"
+
+module Dubhe
+  module Sources
+    class Docs
+      include Source
+
+      EXTENSIONS = %w[.html .htm .md .markdown].freeze
+
+      attr_reader :root
+
+      def initialize(root:, store: Store.new, policy: Policy.new(profile: :docs))
+        @root = File.expand_path(root.to_s)
+        @store = store
+        @policy = policy
+      end
+
+      def id = "docs:#{Digest::SHA256.hexdigest(@root)}"
+      def title = File.basename(@root)
+      def icon = "folder"
+
+      def refresh
+        entries = files.map do |path|
+          relative = Pathname.new(path).relative_path_from(Pathname.new(@root)).to_s
+          Entry.new(id: "#{id}:#{relative}", source_id: id, title: title_for(path), author: nil,
+            published_at: nil, updated_at: File.mtime(path), url: "file://#{path}", summary: nil,
+            tags: [], state: :unread, parent_id: parent_id(relative))
+        end
+        entries.each { |entry| @store.save_entry(entry); @store.save_body(entry.id, fetch_body(entry)) }
+        entries
+      end
+
+      def entries(filter: nil)
+        result = @store.entries(source_id: id)
+        result = refresh if result.empty?
+        filter ? result.select { |entry| filter.call(entry) } : result
+      end
+
+      def fetch_body(entry)
+        path = resolve(entry.url.to_s.sub("file://", ""))
+        if markdown?(path)
+          Body.new(kind: :html, content: markdown_to_html(File.read(path)), base_url: "file://#{path}", attachments: [], headers: {})
+        else
+          result = @policy.apply(File.binread(path), base_url: nil)
+          Body.new(kind: :html, content: result.html, base_url: "file://#{path}", attachments: [], headers: {"blocked_count" => result.blocked_count})
+        end
+      end
+
+      def resolve(path)
+        candidate = path.to_s.start_with?("file://") ? path.to_s.sub("file://", "") : path
+        @policy.resolve_local(@root, candidate)
+      end
+
+      def supports?(capability)
+        %i[refresh search tree unread].include?(capability)
+      end
+
+      private
+
+      def files
+        Dir.glob(File.join(@root, "**", "*")).select do |path|
+          File.file?(path) && EXTENSIONS.include?(File.extname(path).downcase) &&
+            path.split(File::SEPARATOR).none? { |part| part.start_with?(".") || %w[node_modules vendor].include?(part) }
+        end.sort
+      end
+
+      def markdown?(path)
+        %w[.md .markdown].include?(File.extname(path).downcase)
+      end
+
+      def title_for(path)
+        return File.basename(path, File.extname(path)) unless markdown?(path)
+
+        first_heading = File.foreach(path).find { |line| line.match?(/\A#\s+/) }
+        first_heading ? first_heading.sub(/\A#\s+/, "").strip : File.basename(path, File.extname(path))
+      end
+
+      def parent_id(relative)
+        directory = File.dirname(relative)
+        return nil if directory == "."
+
+        "#{id}:#{directory}"
+      end
+
+      def markdown_to_html(text)
+        lines = text.to_s.lines
+        output = []
+        paragraph = []
+        flush = lambda do
+          output << "<p>#{paragraph.join(" ").strip}</p>" unless paragraph.empty?
+          paragraph.clear
+        end
+        in_code = false
+        code = []
+        lines.each do |line|
+          if line.start_with?("```")
+            if in_code
+              output << "<pre><code>#{CGI.escapeHTML(code.join)}</code></pre>"
+              code.clear
+            end
+            in_code = !in_code
+          elsif in_code
+            code << line
+          elsif line.match?(/\A#+\s+/)
+            flush.call
+            level = line[/\A#+/].length
+            output << "<h#{level}>#{CGI.escapeHTML(line.sub(/\A#+\s+/, "").strip)}</h#{level}>"
+          elsif line.strip.empty?
+            flush.call
+          else
+            paragraph << CGI.escapeHTML(line.strip)
+          end
+        end
+        in_code ? output << "<pre><code>#{CGI.escapeHTML(code.join)}</code></pre>" : flush.call
+        output.join
+      end
+    end
+  end
+end

data/lib/dubhe/sources/feeds.rb

@@ -0,0 +1,206 @@
+# frozen_string_literal: true
+
+require "digest"
+require "fileutils"
+require "json"
+require "time"
+require "uri"
+require "iklil"
+require "jabbah"
+
+module Dubhe
+  module Sources
+    class Feeds
+      include Source
+
+      attr_reader :last_errors
+
+      def initialize(subscriptions: nil, store: Store.new, fetcher: Fetch::Client.new,
+                     policy: Policy.new(profile: :feed), interval: 3600, clock: -> { Time.now })
+        @store = store
+        @fetcher = fetcher
+        @policy = policy
+        @interval = interval
+        @clock = clock
+        @subscriptions_path = File.join(@store.root, "sources.jsonc")
+        values = subscriptions.nil? ? load_subscriptions : subscriptions
+        @subscriptions = Array(values).map { |value| normalize_subscription(value) }
+        @raw_entries = {}
+        @last_errors = {}
+      end
+
+      def id = "feeds"
+      def title = "Feeds"
+      def icon = "rss"
+
+      def add(url, title: nil, site_url: nil, group: nil)
+        normalized = url.to_s.strip
+        existing = @subscriptions.find { |subscription| subscription.url == normalized }
+        return existing if existing
+
+        subscription = Subscription.new(id: Digest::SHA256.hexdigest(normalized), title: title || normalized,
+          url: normalized, site_url: site_url, group: group, etag: nil, last_modified: nil,
+          interval: @interval, next_refresh_at: nil, failures: 0)
+        @subscriptions << subscription
+        persist_subscriptions
+        subscription
+      end
+
+      def remove(id_or_url)
+        before = @subscriptions.length
+        @subscriptions.reject! { |subscription| subscription.id == id_or_url.to_s || subscription.url == id_or_url.to_s }
+        changed = before != @subscriptions.length
+        persist_subscriptions if changed
+        changed
+      end
+
+      def subscriptions = @subscriptions.dup
+
+      def refresh(force: false)
+        refreshed = []
+        now = @clock.call
+        @subscriptions = @subscriptions.map do |subscription|
+          next subscription if !force && subscription.next_refresh_at && subscription.next_refresh_at > now
+
+          begin
+            headers = {"If-None-Match" => subscription.etag, "If-Modified-Since" => subscription.last_modified}
+            response = @fetcher.get(subscription.url, headers: headers)
+            if response.status == 304
+              refreshed.concat(@store.entries(source_id: id).select { |entry| entry.updated_at && entry.updated_at >= now - subscription.interval })
+              next schedule(subscription, now: now, failures: 0)
+            end
+            raise Error, "feed returned HTTP #{response.status}" unless response.status.between?(200, 299)
+
+            feed = Iklil.parse(response.body, base_url: subscription.url)
+            entries = feed.entries.map do |raw_entry|
+              entry = convert_entry(subscription, raw_entry)
+              @raw_entries[entry.id] = raw_entry
+              entry
+            end
+            entries.each do |entry|
+              @store.save_entry(entry)
+              @store.save_body(entry.id, fetch_body(entry))
+            end
+            refreshed.concat(entries)
+            schedule(subscription, now: now, failures: 0, title: feed.title, site_url: feed.site_url,
+              etag: response.headers["etag"], last_modified: response.headers["last-modified"])
+          rescue StandardError => error
+            @last_errors[subscription.id] = error
+            schedule(subscription, now: now, failures: subscription.failures + 1)
+          end
+        end
+        persist_subscriptions
+        refreshed
+      end
+
+      def entries(filter: nil)
+        result = @store.entries(source_id: id)
+        return result unless filter
+
+        result.select { |entry| filter.call(entry) }
+      end
+
+      def fetch_body(entry)
+        source = @raw_entries[entry.id]
+        if source.nil?
+          stored = @store.body(entry.id)
+          return stored if stored
+          return Body.new(kind: :text, content: entry.summary.to_s, base_url: entry.url, attachments: [], headers: {})
+        end
+        content = source.content.to_s
+        if source.html?
+          result = @policy.apply(content, base_url: entry.url)
+          extracted = Jabbah::Extract.article(result.document)
+          content = extracted ? extracted[:content].to_html : result.html
+          Body.new(kind: :html, content: content, base_url: entry.url, attachments: source.enclosures, headers: {"blocked_count" => result.blocked_count})
+        else
+          Body.new(kind: :text, content: content, base_url: entry.url, attachments: source.enclosures, headers: {})
+        end
+      end
+
+      def supports?(capability)
+        %i[refresh search unread].include?(capability)
+      end
+
+      def import_opml(bytes)
+        Iklil.parse_opml(bytes).each { |subscription| add(subscription.xml_url, title: subscription.title, site_url: subscription.html_url) }
+      end
+
+      def export_opml(title: "Dubhe subscriptions")
+        Iklil.render_opml(@subscriptions.map { |subscription| {title: subscription.title, xml_url: subscription.url,
+          html_url: subscription.site_url, category: subscription.group} }, title: title)
+      end
+
+      def self.discover(html, base_url: nil)
+        document = html.is_a?(Jabbah::Document) ? html : Jabbah.parse(html)
+        document.search("link").filter_map do |link|
+          rel = link["rel"].to_s.split.map(&:downcase)
+          type = link["type"].to_s.downcase
+          next unless rel.include?("alternate") && %w[application/rss+xml application/atom+xml application/feed+json application/json].include?(type)
+
+          href = link["href"]
+          next unless href && !href.empty?
+          base_url ? URI.join(base_url, href).to_s : href
+        rescue URI::InvalidURIError
+          nil
+        end.uniq
+      end
+
+      private
+
+      def normalize_subscription(value)
+        return value if value.is_a?(Subscription)
+
+        url = value[:url] || value["url"]
+        title = value[:title] || value["title"] || url
+        Subscription.new(id: Digest::SHA256.hexdigest(url.to_s), title: title, url: url.to_s,
+          site_url: value[:site_url] || value["site_url"], group: value[:group] || value["group"],
+          etag: value[:etag] || value["etag"], last_modified: value[:last_modified] || value["last_modified"],
+          interval: value[:interval] || value["interval"] || @interval,
+          next_refresh_at: parse_time(value[:next_refresh_at] || value["next_refresh_at"]),
+          failures: value[:failures] || value["failures"] || 0)
+      end
+
+      def load_subscriptions
+        return [] unless File.file?(@subscriptions_path)
+
+        records = JSON.parse(File.read(@subscriptions_path))
+        records.is_a?(Array) ? records.select { |value| value.is_a?(Hash) } : []
+      rescue JSON::ParserError, SystemCallError
+        []
+      end
+
+      def persist_subscriptions
+        FileUtils.mkdir_p(File.dirname(@subscriptions_path))
+        temporary = "#{@subscriptions_path}.tmp-#{Process.pid}"
+        records = @subscriptions.map { |subscription| subscription.to_h.merge(next_refresh_at: subscription.next_refresh_at&.iso8601) }
+        File.write(temporary, JSON.pretty_generate(records) << "\n")
+        File.rename(temporary, @subscriptions_path)
+      ensure
+        File.delete(temporary) if temporary && File.file?(temporary)
+      end
+
+      def parse_time(value)
+        return value if value.nil? || value.is_a?(Time)
+
+        Time.parse(value.to_s)
+      rescue ArgumentError
+        nil
+      end
+
+      def convert_entry(subscription, entry)
+        Entry.new(id: "#{subscription.id}:#{entry.id}", source_id: id, title: entry.title.to_s,
+          author: Array(entry.authors).join(", "), published_at: entry.published_at, updated_at: entry.updated_at,
+          url: entry.url, summary: entry.summary.to_s, tags: Array(entry.categories), state: :unread, parent_id: nil)
+      end
+
+      def schedule(subscription, now:, failures:, title: subscription.title, site_url: subscription.site_url,
+                   etag: subscription.etag, last_modified: subscription.last_modified)
+        delay = failures.zero? ? subscription.interval : [300 * (3**(failures - 1)), 21_600].min
+        Subscription.new(id: subscription.id, title: title || subscription.title, url: subscription.url,
+          site_url: site_url, group: subscription.group, etag: etag, last_modified: last_modified,
+          interval: subscription.interval, next_refresh_at: now + delay, failures: failures)
+      end
+    end
+  end
+end

data/lib/dubhe/sources/mail.rb

@@ -0,0 +1,78 @@
+# frozen_string_literal: true
+
+require "base64"
+require "digest"
+require "ukdah"
+
+module Dubhe
+  module Sources
+    class Mail
+      include Source
+
+      def initialize(path:, store: Store.new, policy: Policy.new(profile: :mail))
+        @path = File.expand_path(path.to_s)
+        @store = store
+        @policy = policy
+        @messages = {}
+      end
+
+      def id = "mail:#{Digest::SHA256.hexdigest(@path)}"
+      def title = File.basename(@path)
+      def icon = "mail"
+
+      def refresh
+        messages = if File.extname(@path).downcase == ".mbox"
+                     Ukdah::Mbox.each(File.binread(@path)).to_a
+                   else
+                     [Ukdah::Message.parse(File.binread(@path))]
+                   end
+        entries = messages.map do |message|
+          message_id = message.message_id || Digest::SHA256.hexdigest(message.raw.to_s)
+          entry = Entry.new(id: "#{id}:#{message_id}", source_id: id, title: message.subject.to_s,
+            author: message.from.map { |address| address.respond_to?(:email) ? address.email : address.to_s }.join(", "),
+            published_at: message.date, updated_at: message.date, url: nil, summary: message.header("x-snippet"),
+            tags: [], state: :unread, parent_id: nil)
+          @messages[entry.id] = message
+          @store.save_entry(entry)
+          @store.save_body(entry.id, fetch_body(entry))
+          entry
+        end
+        entries
+      end
+
+      def entries(filter: nil)
+        result = @store.entries(source_id: id)
+        result = refresh if result.empty?
+        filter ? result.select { |entry| filter.call(entry) } : result
+      end
+
+      def fetch_body(entry)
+        message = @messages[entry.id]
+        return @store.body(entry.id) unless message
+
+        if (part = message.html_part)
+          html = message.decoded(part)
+          message.inline_parts.each do |content_id, inline_part|
+            next if content_id.start_with?("<")
+
+            mime = inline_part.content_type.to_s
+            data = Base64.strict_encode64(inline_part.body.to_s)
+            html = html.gsub("cid:#{content_id}", "data:#{mime};base64,#{data}")
+          end
+          result = @policy.apply(html)
+          Body.new(kind: :html, content: result.html, base_url: nil,
+            attachments: message.attachments.map(&:filename).compact, headers: {"blocked_count" => result.blocked_count})
+        elsif (part = message.text_part)
+          Body.new(kind: :text, content: message.decoded(part), base_url: nil,
+            attachments: message.attachments.map(&:filename).compact, headers: {})
+        else
+          Body.new(kind: :text, content: "", base_url: nil, attachments: [], headers: {})
+        end
+      end
+
+      def supports?(capability)
+        %i[refresh search unread].include?(capability)
+      end
+    end
+  end
+end

data/lib/dubhe/store.rb

@@ -0,0 +1,143 @@
+# frozen_string_literal: true
+
+require "digest"
+require "fileutils"
+require "json"
+require "time"
+
+module Dubhe
+  class Store
+    attr_reader :root
+
+    def initialize(root: nil)
+      data_root = root || ENV["XDG_DATA_HOME"] || File.join(Dir.home, ".local", "share")
+      @root = root ? File.expand_path(root.to_s) : File.join(data_root.to_s, "dubhe")
+      @index_root = File.join(@root, "index")
+      @body_root = File.join(@root, "bodies")
+      FileUtils.mkdir_p([@index_root, @body_root])
+    end
+
+    def save_entry(entry)
+      append("entries.jsonl", serialize_entry(entry))
+      entry
+    end
+
+    def save_entries(entries)
+      Array(entries).each { |entry| save_entry(entry) }
+      entries
+    end
+
+    def entries(source_id: nil)
+      records = read_jsonl("entries.jsonl")
+      latest = {}
+      records.each { |record| latest[record["id"].to_s] = record }
+      states = state_records
+      latest.values.filter_map do |record|
+        next if source_id && record["source_id"] != source_id
+
+        deserialize_entry(record, states[record["id"].to_s])
+      end
+    end
+
+    def entry(id)
+      entries.find { |item| item.id == id.to_s }
+    end
+
+    def update_state(id, state: nil, starred: nil, tags: nil)
+      current = state_records[id.to_s] || {}
+      record = {"id" => id.to_s, "state" => (state || current["state"] || "unread"),
+        "starred" => starred.nil? ? current["starred"] : starred,
+        "tags" => tags.nil? ? current["tags"] : tags}
+      append("state.jsonl", record)
+      record
+    end
+
+    def save_body(id, body)
+      digest = Digest::SHA256.hexdigest(id.to_s)
+      directory = File.join(@body_root, digest[0, 2], digest[2, 2])
+      FileUtils.mkdir_p(directory)
+      path = File.join(directory, "#{digest}.#{extension(body.kind)}")
+      temporary = "#{path}.tmp-#{Process.pid}-#{rand(1_000_000)}"
+      File.binwrite(temporary, body.content.to_s)
+      File.rename(temporary, path)
+      path
+    ensure
+      File.delete(temporary) if temporary && File.file?(temporary)
+    end
+
+    def body(id)
+      digest = Digest::SHA256.hexdigest(id.to_s)
+      directory = File.join(@body_root, digest[0, 2], digest[2, 2])
+      path = %w[html text markdown].map { |kind| File.join(directory, "#{digest}.#{kind}") }.find { |candidate| File.file?(candidate) }
+      return nil unless path
+
+      kind = File.extname(path).delete_prefix(".").to_sym
+      Body.new(kind: kind, content: File.binread(path), base_url: nil, attachments: [], headers: {})
+    end
+
+    def compact!
+      rewrite("entries.jsonl", entries.map { |entry| serialize_entry(entry) })
+      rewrite("state.jsonl", state_records.values)
+      self
+    end
+
+    private
+
+    def append(name, record)
+      File.open(File.join(@index_root, name), "ab") { |file| file.write(JSON.generate(record) << "\n") }
+    end
+
+    def read_jsonl(name)
+      path = File.join(@index_root, name)
+      return [] unless File.file?(path)
+
+      File.foreach(path).filter_map do |line|
+        JSON.parse(line)
+      rescue JSON::ParserError
+        nil
+      end
+    end
+
+    def state_records
+      read_jsonl("state.jsonl").each_with_object({}) { |record, result| result[record["id"].to_s] = record }
+    end
+
+    def serialize_entry(entry)
+      {"id" => entry.id.to_s, "source_id" => entry.source_id.to_s, "title" => entry.title,
+        "author" => entry.author, "published_at" => encode_time(entry.published_at),
+        "updated_at" => encode_time(entry.updated_at), "url" => entry.url, "summary" => entry.summary,
+        "tags" => Array(entry.tags), "state" => entry.state.to_s, "parent_id" => entry.parent_id}
+    end
+
+    def deserialize_entry(record, state)
+      merged = record.merge(state || {})
+      Entry.new(id: record["id"], source_id: record["source_id"], title: record["title"],
+        author: record["author"], published_at: decode_time(record["published_at"]),
+        updated_at: decode_time(record["updated_at"]), url: record["url"], summary: record["summary"],
+        tags: Array(merged["tags"]), state: (merged["state"] || "unread").to_sym, parent_id: record["parent_id"])
+    end
+
+    def encode_time(value)
+      value&.iso8601
+    end
+
+    def decode_time(value)
+      Time.parse(value.to_s) if value && !value.to_s.empty?
+    rescue ArgumentError
+      nil
+    end
+
+    def extension(kind)
+      %i[html text markdown].include?(kind.to_sym) ? kind.to_s : "text"
+    end
+
+    def rewrite(name, records)
+      path = File.join(@index_root, name)
+      temporary = "#{path}.tmp-#{Process.pid}"
+      File.open(temporary, "wb") { |file| records.each { |record| file.write(JSON.generate(record) << "\n") } }
+      File.rename(temporary, path)
+    ensure
+      File.delete(temporary) if temporary && File.file?(temporary)
+    end
+  end
+end

data/lib/dubhe/version.rb

@@ -0,0 +1,5 @@
+# frozen_string_literal: true
+
+module Dubhe
+  VERSION = "0.1.0"
+end

data/lib/dubhe.rb

@@ -0,0 +1,18 @@
+# frozen_string_literal: true
+
+require_relative "dubhe/version"
+
+module Dubhe
+  class Error < StandardError; end
+end
+
+require_relative "dubhe/models"
+require_relative "dubhe/source"
+require_relative "dubhe/policy"
+require_relative "dubhe/store"
+require_relative "dubhe/fetch"
+require_relative "dubhe/render"
+require_relative "dubhe/reader"
+require_relative "dubhe/sources/feeds"
+require_relative "dubhe/sources/docs"
+require_relative "dubhe/sources/mail"

data/sig/dubhe.rbs

@@ -0,0 +1,39 @@
+module Dubhe
+  VERSION: String
+
+  class Error < StandardError
+  end
+
+  class Store
+    def initialize: (?root: String?) -> void
+    def save_entry: (untyped entry) -> untyped
+    def entries: (?source_id: String?) -> Array[untyped]
+    def entry: (String id) -> untyped
+    def update_state: (String id, ?state: Symbol?, ?starred: bool?, ?tags: Array[String]?) -> Hash[String, untyped]
+    def save_body: (String id, untyped body) -> String
+    def body: (String id) -> untyped
+  end
+
+  class Policy
+    class Denied < Error
+    end
+    Result: untyped
+    def initialize: (?profile: Symbol) -> void
+    def apply: (String input, ?base_url: String?, ?allow_remote_images: bool) -> untyped
+    def resolve_local: (String root, String path) -> String
+  end
+
+  module Render
+    def self.call: (untyped node, ?theme: Hash[untyped, untyped]) -> String
+  end
+
+  class Reader
+    def initialize: (?sources: Array[untyped], ?store: Store) -> void
+    def refresh: () -> Array[untyped]
+    def entries: (?source_id: String?, ?filter: Proc?) -> Array[untyped]
+    def open: (untyped entry) -> untyped
+    def back: () -> untyped
+    def forward: () -> untyped
+    def search: (String query, ?source_id: String?) -> Array[untyped]
+  end
+end

metadata

@@ -0,0 +1,148 @@
+--- !ruby/object:Gem::Specification
+name: dubhe
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+platform: ruby
+authors:
+- Yudai Takada
+bindir: bin
+cert_chain: []
+date: 1980-01-02 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: iklil
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.1'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.1'
+- !ruby/object:Gem::Dependency
+  name: jabbah
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.1'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.1'
+- !ruby/object:Gem::Dependency
+  name: base64
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.2'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.2'
+- !ruby/object:Gem::Dependency
+  name: cgi
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.4'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.4'
+- !ruby/object:Gem::Dependency
+  name: rexml
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '3.4'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '3.4'
+- !ruby/object:Gem::Dependency
+  name: ukdah
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.1'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.1'
+description: Dubhe provides file-backed reader storage, feed fetching, sanitization
+  policies, and source APIs for static documents and mail.
+email:
+- t.yudai92@gmail.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- CHANGELOG.md
+- LICENSE.txt
+- README.md
+- docs/adr/001-reader-core-boundaries.md
+- docs/supported-elements.md
+- lib/dubhe.rb
+- lib/dubhe/fetch.rb
+- lib/dubhe/models.rb
+- lib/dubhe/policy.rb
+- lib/dubhe/reader.rb
+- lib/dubhe/render.rb
+- lib/dubhe/source.rb
+- lib/dubhe/sources/docs.rb
+- lib/dubhe/sources/feeds.rb
+- lib/dubhe/sources/mail.rb
+- lib/dubhe/store.rb
+- lib/dubhe/version.rb
+- sig/dubhe.rbs
+homepage: https://github.com/noxdea/dubhe
+licenses:
+- MIT
+metadata:
+  allowed_push_host: https://rubygems.org
+  homepage_uri: https://github.com/noxdea/dubhe
+  source_code_uri: https://github.com/noxdea/dubhe/tree/main
+  changelog_uri: https://github.com/noxdea/dubhe/blob/main/CHANGELOG.md
+  rubygems_mfa_required: 'true'
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: 3.2.0
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubygems_version: 4.0.16
+specification_version: 4
+summary: A safe feeds-first document reader core
+test_files: []