pikuri-pdf 0.0.6

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 7d52809c4ac479bbf4823b0e47ae96ec955a56ba80616a2fdc495ca4e785eff7
+  data.tar.gz: '0178b0b772e9032fe1b952ae363827d46b7f73cf0b683c6b15d4368befe0cd85'
+SHA512:
+  metadata.gz: 7ef82b47e6c54b02957bfe0948f4cf4f949c3925ead0b994ff2a9260addd0562389b2ea3a2c83a1e053b461579795888f927a317cd7d2d10e9c43cb2901d59a4
+  data.tar.gz: 7fbcc9311f3d4ee942f26698c35e4306e19c3fde42d75569ee355d8610f14254ca9866a5d3a2550f6e6f1f83196828d69ac9dbcb378747f29f10ac3c1ff038d6

data/README.md

@@ -0,0 +1,78 @@
+# pikuri-pdf
+
+PDF text extraction for the
+[pikuri](https://codeberg.org/mvysny/pikuri) AI-assistant toolkit:
+in-process, pure Ruby, and **lazy** — paged reads parse only the
+pages the window needs, so showing the first page of a 500-page PDF
+never pays for the other 499.
+
+Provides:
+- `Pikuri::Extractors::PDF` — an extractor for the
+  `Pikuri::Extractor` registry, wrapping the pure-Ruby
+  [pdf-reader](https://github.com/yob/pdf-reader) gem. Once
+  registered, every pikuri surface that routes through the registry
+  picks PDFs up for free: the `read` tool pages through a local
+  `.pdf` with `--- Page N ---` markers, `web_scrape` extracts a
+  downloaded paper, the pikuri-vectordb indexer ingests a PDF
+  corpus.
+
+## Why a separate gem
+
+pikuri-core's pitch is a dependency tree you can audit in an
+evening. pdf-reader brings five transitive gems (Ascii85, afm,
+hashery, ruby-rc4, ttfunk) that serve nothing else in core — the
+largest single bite in that tree, for one file format. So PDF
+support is an opt-in sibling instead: install it when your agent
+needs PDFs, skip it (and its whole subtree) when it doesn't.
+
+Everything is pure Ruby, so the worst a malicious PDF can do to the
+parser is burn CPU and memory — there's no native code to corrupt.
+
+**This gem or [pikuri-extractors](../pikuri-extractors) — pick one
+per wiring.** pikuri-extractors' converter container also has a PDF
+arm (poppler's `pdftotext`, sandboxed, same `--- Page N ---`
+markers): on an agent that fetches untrusted documents from the
+web, parsing them in the networkless container is the stronger
+posture. This gem is the no-infrastructure wiring — in-process
+means no docker and no host CLIs, and it's what makes the lazy
+page-windowed reads possible (a subprocess converter must convert
+the whole document before emitting anything, and re-converts it on
+every paged read). The guide wires this gem in chapter 3 and
+supersedes it with pikuri-extractors in chapter 7's assistant.
+
+## Install
+
+```ruby
+# Gemfile
+gem 'pikuri-pdf'
+```
+
+## Usage
+
+Requiring the gem changes nothing — registration is an explicit
+opt-in your script makes, same philosophy as `c.add_extension`:
+
+```ruby
+require 'pikuri-core'
+require 'pikuri-pdf'
+
+Pikuri::Extractors::PDF.register
+
+# From here on, the registry handles PDFs everywhere:
+text = Pikuri::FileType.read_as_text(Pathname.new('paper.pdf'))
+```
+
+`register` inserts the extractor at the *front* of the registry: the
+`%PDF-` magic-byte sniff is the strongest signal there — it never
+misfires on text, and it must win over the HTML extractor's
+content-type match so a PDF served under a lying `Content-Type`
+header still extracts.
+
+## Limits
+
+Best-effort by design: pdf-reader produces clean text from PDFs
+generated from a digital source (LaTeX, Word export, ...) but
+nothing useful from scanned documents — those extract to the empty
+string, and the `read` tool words that as a scanned-image hint to
+the model. No OCR. Encrypted and XFA-form PDFs surface as
+`Error: ...` observations the model can react to.

data/lib/pikuri/extractors/pdf.rb

@@ -0,0 +1,134 @@
+# frozen_string_literal: true
+
+require 'pdf-reader'
+
+module Pikuri
+  module Extractors
+    # PDF → text extractor. Wraps the +pdf-reader+ gem: walk every
+    # page, emit a +"--- Page N ---"+ marker line followed by that
+    # page's extracted text, join the blocks with single newlines.
+    # The markers give every consumer page provenance — the Read
+    # tools tell the model to cite pages back to the user from them,
+    # +vectordb_search+ chunks carry them so a hit can say which page
+    # it came from, and what +vectordb_read+ shows matches what was
+    # indexed exactly. Pages with no extractable text contribute
+    # nothing (no marker either), so a fully scanned PDF extracts to
+    # the empty String — a deliberate silent skip callers detect by
+    # length if they care. No OCR in this path.
+    #
+    # == Why a separate gem
+    #
+    # This extractor lived in pikuri-core until pdf-reader's
+    # dependency tail (Ascii85, afm, hashery, ruby-rc4, ttfunk) became
+    # the largest single bite in the core's audit tree — five gems for
+    # one file format, serving nothing else in core. Splitting it out
+    # keeps the core minimal; hosts that want PDFs opt in with one
+    # {.register} call. Distinct from pikuri-extractors' sandboxed
+    # subprocess converters: this one is in-process and *lazy*
+    # ({.extract_lines} parses pages on demand), a property a
+    # subprocess converter structurally cannot have — see
+    # +Pikuri::Extractor+'s windowing yardoc.
+    #
+    # == Registration is explicit
+    #
+    # Requiring pikuri-pdf defines this module but registers nothing.
+    # A host script opts in with +Pikuri::Extractors::PDF.register+,
+    # which inserts it at the *front* of the registry — unlike
+    # pikuri-extractors' before-the-terminal insert — because the
+    # +%PDF-+ magic-byte sniff is the strongest signal in the
+    # registry: it must win over +HTML+'s content-type match so a PDF
+    # served under a lying header is still extracted, and it never
+    # misfires on text.
+    #
+    # Matched by the +%PDF-+ magic prefix *or* an +application/pdf+
+    # content-type.
+    #
+    # Best-effort by design: +pdf-reader+ produces clean text from
+    # PDFs generated from a digital source (LaTeX, Word export, ...)
+    # but nothing useful from scanned documents.
+    module PDF
+      # Insert this extractor at the front of
+      # +Pikuri::Extractor.registry+ (see "Registration is explicit"
+      # above for why the front). Idempotent.
+      #
+      # @return [Module] self, for one-line wiring in host scripts.
+      def self.register
+        registry = Pikuri::Extractor.registry
+        registry.unshift(self) unless registry.include?(self)
+        self
+      end
+
+      # @return [Symbol] {Pikuri::Extractor::Page#kind} tag.
+      def self.kind
+        :pdf
+      end
+
+      # @param sample [String] leading bytes of the content.
+      # @param content_type [String, nil] normalized content-type,
+      #   when the transport supplies one.
+      # @return [Boolean]
+      def self.matches?(sample:, content_type:)
+        content_type == 'application/pdf' || sample.start_with?(FileType::PDF_MAGIC)
+      end
+
+      # Render the PDF behind +io+ as plain text, one
+      # +"--- Page N ---"+-headed block per page that carries text.
+      # Defined as +extract_lines.to_a.join+ so the two duck-type
+      # shapes cannot drift apart.
+      #
+      # @param io [IO, StringIO] seekable IO positioned at the start
+      #   of the PDF bytes.
+      # @return [String] concatenated page blocks; possibly empty when
+      #   the PDF carries no extractable text (scanned image, empty
+      #   document).
+      # @raise [Pikuri::Extractor::Error] when +pdf-reader+ refuses
+      #   the document.
+      def self.extract(io)
+        extract_lines(io).to_a.join("\n")
+      end
+
+      # The lazy line stream behind {.extract}: a marker line per
+      # text-carrying page, then that page's lines. +pdf-reader+
+      # parses a page's content stream only when +Page#text+ is
+      # called, so a consumer that stops early (the
+      # +Pikuri::Extractor.extract_paged+ window) never pays for the
+      # pages past its window.
+      #
+      # +pdf-reader+ raises a handful of typed exceptions for
+      # documents it cannot parse — broken xrefs
+      # ({::PDF::Reader::MalformedPDFError}), invalid page references
+      # ({::PDF::Reader::InvalidPageError}), encrypted/XFA files
+      # ({::PDF::Reader::UnsupportedFeatureError}). All three describe
+      # a property of the document the LLM can react to ("try a
+      # different URL / file"), so they re-raise as
+      # {Pikuri::Extractor::Error} — from inside the enumerator, i.e.
+      # at consumption time, which for a broken xref means the first
+      # +next+. Genuine bugs in +pdf-reader+ itself surface as their
+      # own classes and crash loud.
+      #
+      # @param io [IO, StringIO] seekable IO positioned at the start
+      #   of the PDF bytes; must remain open while the enumerator is
+      #   consumed.
+      # @return [Enumerator<String>] chomped lines, produced
+      #   page-by-page.
+      # @raise [Pikuri::Extractor::Error] when +pdf-reader+ refuses
+      #   the document (raised on consumption).
+      def self.extract_lines(io)
+        Enumerator.new do |lines|
+          ::PDF::Reader.new(io).pages.each_with_index do |page, idx|
+            text = page.text.strip
+            next if text.empty?
+
+            lines << "--- Page #{idx + 1} ---"
+            text.split("\n").each { |line| lines << line }
+          end
+        rescue ::PDF::Reader::MalformedPDFError,
+               ::PDF::Reader::InvalidPageError,
+               ::PDF::Reader::UnsupportedFeatureError => e
+          raise Pikuri::Extractor::Error,
+                "PDF rendering failed: #{e.class.name.split('::').last}: #{e.message}"
+        end
+      end
+    end
+  end
+end

data/lib/pikuri-pdf.rb

@@ -0,0 +1,30 @@
+# frozen_string_literal: true
+
+require 'pikuri-core'
+
+# Entry file for the pikuri-pdf gem. Sets up a dedicated Zeitwerk
+# loader rooted at this gem's +lib/+, contributing to the shared
+# +Pikuri::+ namespace alongside pikuri-core. After +require
+# 'pikuri-pdf'+, +Pikuri::Extractors::PDF+ is defined — but *nothing
+# is registered*: extractors plug into +Pikuri::Extractor.registry+
+# only when the host script calls their +register+ explicitly, so a
+# +bin/pikuri-*+ picks which extractors it wires in (same opt-in
+# philosophy as +c.add_extension+, same shape as pikuri-extractors'
+# +DOCUMENTS.register+).
+#
+# The +Pikuri::Extractors+ namespace is cooperative: pikuri-extractors
+# contributes +Documents+ / +DOCUMENTS+, this gem contributes +PDF+.
+# Each gem's loader manages only its own files; the loader constant is
+# +PDF_LOADER+ (not +LOADER+) so both gems can be loaded together
+# without colliding in the shared namespace.
+module Pikuri
+  module Extractors
+    PDF_LOADER = Zeitwerk::Loader.new
+    PDF_LOADER.tag = 'pikuri-pdf'
+    PDF_LOADER.push_dir(File.expand_path('.', __dir__))
+    PDF_LOADER.ignore(__FILE__)
+    PDF_LOADER.inflector.inflect('pdf' => 'PDF')
+    PDF_LOADER.setup
+    PDF_LOADER.eager_load
+  end
+end

metadata

@@ -0,0 +1,94 @@
+--- !ruby/object:Gem::Specification
+name: pikuri-pdf
+version: !ruby/object:Gem::Version
+  version: 0.0.6
+platform: ruby
+authors:
+- Martin Vysny
+autorequire:
+bindir: bin
+cert_chain: []
+date: 2026-06-04 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: pikuri-core
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '='
+      - !ruby/object:Gem::Version
+        version: 0.0.6
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '='
+      - !ruby/object:Gem::Version
+        version: 0.0.6
+- !ruby/object:Gem::Dependency
+  name: pdf-reader
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.15'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.15'
+description: |
+  pikuri-pdf plugs PDF → text extraction into pikuri-core's
+  +Pikuri::Extractor+ registry. The bundled +Pikuri::Extractors::PDF+
+  extractor wraps the pure-Ruby pdf-reader gem and extracts lazily:
+  paged reads (the +read+ tool's windows) parse only the pages the
+  window needs, so the first page of a 500-page PDF never pays for
+  the other 499.
+
+  Shipped separately from pikuri-core so the core's dependency tree
+  stays minimal and auditable: pdf-reader and its transitive deps
+  (Ascii85, afm, hashery, ruby-rc4, ttfunk) ride along only for hosts
+  that opt into PDF support.
+
+  Registration is explicit — +Pikuri::Extractors::PDF.register+ — so
+  requiring the gem changes nothing by itself; the host script picks
+  which extractors it wires in. One registration extends the +read+
+  tool, +web_scrape+, and the pikuri-vectordb indexer simultaneously.
+email:
+- martin@vysny.me
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- README.md
+- lib/pikuri-pdf.rb
+- lib/pikuri/extractors/pdf.rb
+homepage: https://codeberg.org/mvysny/pikuri
+licenses:
+- MIT
+metadata:
+  source_code_uri: https://codeberg.org/mvysny/pikuri/src/branch/master
+  changelog_uri: https://codeberg.org/mvysny/pikuri/src/branch/master/CHANGELOG.md
+  bug_tracker_uri: https://codeberg.org/mvysny/pikuri/issues
+  rubygems_mfa_required: 'true'
+post_install_message:
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '3.3'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubygems_version: 3.5.22
+signing_key:
+specification_version: 4
+summary: In-process, lazily-paged PDF text extraction for pikuri.
+test_files: []