ucode 0.2.0 → 0.2.1

data/TODO.extract-code-chart/07-codechart-cli.md

@@ -0,0 +1,135 @@
+# TODO 07 — CodeChart CLI
+
+## Status
+
+Pending. Depends on TODO 06 (Writer), TODO 04 (Extractor),
+TODO 02 (block name resolver).
+
+## Goal
+
+`ucode code-chart fetch | extract | list` — the REQ (R4) commands.
+Thin Thor wrappers that delegate to the existing `CodeChart::*`
+modules. No orchestration logic in the CLI; every command is a
+single delegation.
+
+## Files
+
+- `lib/ucode/cli.rb` — add the `CodeChartCmd` Thor subcommand class.
+- `spec/ucode/cli_spec.rb` (extend existing) — verify the new
+  subcommand wires up.
+
+## Design
+
+### Subcommand shape
+
+```ruby
+class Cli < Thor
+  # …existing commands…
+
+  class CodeChartCmd < Thor
+    desc "fetch --block BLOCK", "Download the Code Charts PDF for a block"
+    option :block, type: :string, required: true,
+                  desc: "Block identifier (e.g. Sidetic)"
+    def fetch
+      puts JSON.pretty_generate(
+        Commands::FetchCommand.new.fetch_charts(
+          VersionResolver.resolve(nil),
+          block_first_cps: [block_first_cp!(options[:block])],
+        ),
+      )
+    end
+
+    desc "extract --block BLOCK --to DIR", "Extract per-codepoint SVG + provenance sidecars"
+    option :block, type: :string, required: true,
+                  desc: "Block identifier (e.g. Sidetic)"
+    option :to, type: :string, required: true,
+                desc: "Output directory"
+    def extract
+      # ...
+    end
+
+    desc "list", "List blocks that have Code Charts PDFs available locally"
+    def list
+      # ...
+    end
+  end
+
+  desc "code-chart", "Extract per-codepoint SVG glyphs from Unicode Code Charts PDFs"
+  subcommand "code-chart", CodeChartCmd
+end
+```
+
+### `extract` flow
+
+```ruby
+def extract
+  Ucode::Commands::FetchCommand.new.fetch_charts(
+    VersionResolver.resolve(nil),
+    block_first_cps: [block_first_cp!(options[:block])],
+  )
+  blocks_txt = Ucode::Cache.ucd_dir(VersionResolver.resolve(nil)).join("Blocks.txt")
+  block = Parsers::Blocks.find_by_name(blocks_txt, options[:block]) or
+    raise Thor::Error, "Unknown block: #{options[:block].inspect}"
+  pdf = Ucode::Glyphs::PdfFetcher.new(
+    VersionResolver.resolve(nil),
+    monolith_path: nil,
+    blocks: [block],
+  ).fetch(block_first_cp: block.range_first, force: false) or
+    raise Thor::Error, "PDF unavailable for block #{options[:block]}"
+
+  writer = Ucode::CodeChart::Writer.new(
+    output_root: Pathname.new(options[:to]),
+    pdf_path: pdf,
+    blocks_txt: blocks_txt,
+  )
+  summary = writer.write(block)
+  puts JSON.pretty_generate(summary.to_h.compact)
+end
+
+def block_first_cp!(block_id)
+  cache = Ucode::Cache.ucd_dir(VersionResolver.resolve(nil))
+  block = Ucode::Parsers::Blocks.find_by_name(cache.join("Blocks.txt"), block_id)
+  raise Thor::Error, "Unknown block: #{block_id.inspect}" unless block
+  block.range_first
+end
+```
+
+### Why `Ucode::Commands::FetchCommand.new.fetch_charts` for fetch
+
+`fetch_charts` is the existing CLI hook for "download a Code Charts
+PDF for these block first-cps." We just call it with the block's
+first cp. Reuse, don't reimplement.
+
+### Why resolve version once at the top of `extract`
+
+Per Candidate 4 of the architecture review (`refactor/build-context-resolve-version-once`,
+merged): every CLI method resolves the version once and threads it
+through. This CLI method does the same — one call per invocation.
+
+### Why no separate `Commands::CodeChartCommand` class
+
+Following the existing pattern (e.g. `Cli::Audit` calls
+`Commands::Audit::*Command` *only* when the logic is non-trivial).
+The CodeChart commands are trivial delegations — a one-liner each.
+The CLI methods call `CodeChart::Writer` and `CodeChart::Extractor`
+directly. Adding a `Commands::CodeChartCommand` class would be
+indirection without a payoff.
+
+If the extract logic grows (e.g. progress reporting, partial
+extraction), extract it into a Command class at that point.
+
+## Acceptance
+
+- `ucode code-chart fetch --block Sidetic` downloads the PDF.
+- `ucode code-chart extract --block Sidetic --to /tmp/s/` extracts
+  to the given directory.
+- `ucode code-chart list` prints available blocks.
+- Unknown block names produce a clean error, not a stack trace.
+- The CLI matches the REQ's signature exactly.
+
+## Out of scope
+
+- `--version` flag (the REQ doesn't specify, and existing commands
+  default to the configured version).
+- `--format svg|glif` (the REQ specifies SVG; `.glif` output is a
+  future extension).

data/TODO.extract-code-chart/08-specs.md

@@ -0,0 +1,87 @@
+# TODO 08 — CodeChart specs
+
+## Status
+
+Pending. Depends on TODOs 01–07.
+
+## Goal
+
+Comprehensive spec coverage for every new module. Per the project
+rules: real model instances (no doubles), tight focused tests,
+behavior assertions (not interaction counts).
+
+## Files
+
+Already enumerated per TODO. Final spec coverage:
+
+- `spec/ucode/fetch/code_charts_spec.rb` — happy path + HTTP 4xx +
+  wrong content-type + non-PDF body. (TODO 01)
+- `spec/ucode/parsers/blocks_spec.rb` (extend) — `find_by_name`
+  happy path + nil on miss. (TODO 02)
+- `spec/ucode/code_chart/extractor_spec.rb` — constructor invariants,
+  Resolver wiring, integration test against fixture PDF. (TODO 04)
+- `spec/ucode/code_chart/provenance_spec.rb` — value object
+  construction + `to_h` schema. (TODO 05)
+- `spec/ucode/code_chart/sidecar_spec.rb` — write sidecar, idempotent
+  re-write. (TODO 05)
+- `spec/ucode/code_chart/writer_spec.rb` — full lifecycle:
+  extract → write → summary. Idempotent re-run produces byte-identical
+  files. (TODO 06)
+- `spec/ucode/cli_spec.rb` (extend) — verify `ucode code-chart fetch`,
+  `extract`, `list` wire up. (TODO 07)
+
+## Design
+
+### Fixture strategy
+
+The existing `spec/fixtures/pdfs/basic_latin.pdf` is the only PDF
+fixture in the repo. It's tiny and validates the integration path.
+The Sidetic + Egyptian Ext-B PDFs are large (whole-block) and would
+inflate the repo. The integration spec uses `basic_latin.pdf` to
+exercise the full pipeline; per-codepoint assertions cover
+representative cases.
+
+If Sidetic-specific behavior must be tested, a smaller fixture PDF
+cropped to ~5 codepoints would be the right tool — out of scope for
+this TODO.
+
+### No doubles policy
+
+The project's `~/.claude/CLAUDE.md` rule: no doubles. All specs use
+real instances:
+- `Ucode::Models::Block.new(...)` for test blocks.
+- A temp directory + real `Blocks.txt` text for parser specs.
+- The real `Ucode::Glyphs::Resolver` for extractor specs.
+
+### Idempotency assertion pattern
+
+`Writer#write` idempotency is asserted via byte-equality:
+
+```ruby
+first_run  = writer.write(block)
+first_size = File.stat(svg_path).size
+sleep 0.01  # ensure mtime changes would be detectable
+second_run = writer.write(block)
+second_size = File.stat(svg_path).size
+expect(second_size).to eq(first_size)
+expect(File.read(svg_path)).to eq(expected_svg_bytes)
+```
+
+This is the existing pattern from `spec/ucode/repo/aggregate_writer_spec.rb`
+(idiempotency spec there). Reuse.
+
+## Acceptance
+
+- `bundle exec rspec spec/ucode/code_chart/ spec/ucode/fetch/code_charts_spec.rb spec/ucode/parsers/blocks_spec.rb`
+  passes 100%.
+- Coverage for the new files is ≥ 95% (per the project's per-file
+  floor of 30% + the overall 80% minimum).
+- No doubles are introduced (verify with `grep -r "double(" spec/ucode/code_chart/`).
+- The integration spec exercises both the Extractor and Writer
+  together end-to-end.
+
+## Out of scope
+
+- Performance benchmarks — separate concern.
+- Sidetic-specific fixtures — requires PDF curation beyond the
+  scope of this feature.

data/lib/ucode/cli.rb

@@ -3,6 +3,7 @@
 require "thor"
 
 require "ucode/commands"
+require "ucode/code_chart"
 require "ucode/version_resolver"
 
 module Ucode
@@ -157,6 +158,104 @@ module Ucode
     desc "site", "Generate the Vitepress site"
     subcommand "site", Site
 
+    # ─────────────── code-chart ───────────────
+    # Extract per-codepoint SVG glyphs from a Unicode Code Charts PDF.
+    # One folder per block under --to, with <U+XXXX>.svg + .json pairs.
+    class CodeChartCmd < Thor
+      desc "fetch --block BLOCK [VERSION]", "Download the Code Charts PDF for a block"
+      option :block, type: :string, required: true,
+                    desc: "Block identifier (e.g. Sidetic, Basic_Latin)"
+      def fetch(version = nil)
+        with_codechart_errors do
+          block_first_cp = resolve_block_first_cp!(options[:block], version)
+          result = Commands::FetchCommand.new.fetch_charts(
+            VersionResolver.resolve(version),
+            block_first_cps: [block_first_cp],
+          )
+          puts JSON.pretty_generate(result)
+        end
+      end
+
+      desc "extract --block BLOCK --to DIR [VERSION]",
+           "Extract per-codepoint SVG + provenance sidecars from a Code Charts PDF"
+      option :block, type: :string, required: true,
+                    desc: "Block identifier (e.g. Sidetic)"
+      option :to, type: :string, required: true,
+                  desc: "Output directory (will contain <block_id>/<U+XXXX>.svg + .json)"
+      def extract(version = nil)
+        with_codechart_errors do
+          version_str = VersionResolver.resolve(version)
+          block = resolve_block!(options[:block], version_str)
+          block_first_cp = block.range_first
+
+          # Download (idempotent — re-runs skip when the PDF is cached).
+          Commands::FetchCommand.new.fetch_charts(version_str, block_first_cps: [block_first_cp])
+
+          pdf = Ucode::Glyphs::PdfFetcher.new(version_str)
+            .fetch(block_first_cp: block_first_cp)
+          raise Ucode::CodeChartNotFoundError.new(
+            "Code Charts PDF unavailable for block #{block.id.inspect}",
+            context: { block_id: block.id, version: version_str },
+          ) unless pdf
+
+          writer = Ucode::CodeChart::Writer.new(
+            output_root: Pathname.new(options[:to]),
+            pdf_path: pdf,
+            ucd_version: version_str,
+          )
+          summary = writer.write(block)
+          puts JSON.pretty_generate(summary.to_h.compact)
+        end
+      end
+
+      desc "list", "List cached Code Charts PDFs under the version's cache"
+      def list
+        version = VersionResolver.resolve(nil)
+        pdfs_dir = Ucode::Cache.pdfs_dir(version)
+        files = pdfs_dir.exist? ? pdfs_dir.children.sort : []
+        if files.empty?
+          puts "(no cached Code Charts PDFs)"
+          return
+        end
+        files.each do |f|
+          puts f.basename.to_s
+        end
+      end
+
+      private
+
+      # Resolve a block name to its first codepoint via the cached
+      # Blocks.txt. Raises {Ucode::UnknownBlockError} on miss.
+      def resolve_block!(block_id, version)
+        blocks_txt = Ucode::Cache.ucd_dir(VersionResolver.resolve(version)).join("Blocks.txt")
+        Ucode::Parsers::Blocks.find_by_id!(blocks_txt, block_id)
+      end
+
+      def resolve_block_first_cp!(block_id, version)
+        resolve_block!(block_id, version).range_first
+      end
+
+      # Convert semantic Ucode errors into Thor errors so Thor's
+      # dispatch prints the message cleanly instead of a stack trace.
+      # Thor's `start` rescues only `Thor::Error`; without this bridge,
+      # any `Ucode::Error` subclass propagates as an uncaught exception.
+      def with_codechart_errors
+        yield
+      rescue Ucode::Error => e
+        raise Thor::Error, e.message
+      end
+    end
+
+    # Register the subcommand under the underscored method name
+    # (`code_chart`). Thor's `normalize_command_name` converts the
+    # user's hyphenated form (`code-chart`) to the underscored form
+    # before lookup, so `ucode code-chart <cmd>` dispatches correctly.
+    # `desc` first registers the method as a Thor command so the
+    # dispatch table has an entry; `subcommand` then attaches the
+    # CodeChartCmd class to it.
+    desc "code_chart <command>", "Extract SVG glyphs from Unicode Code Charts PDFs"
+    subcommand "code_chart", CodeChartCmd
+
     # ─────────────── lookup ───────────────
     class Lookup < Thor
       desc "block CODEPOINT", "Block name covering CODEPOINT (integer or 0xNNNN)"

data/lib/ucode/code_chart/extractor.rb

@@ -0,0 +1,122 @@
+# frozen_string_literal: true
+
+require "pathname"
+
+require "ucode/error"
+require "ucode/glyphs/embedded_fonts/catalog"
+require "ucode/glyphs/embedded_fonts/renderer"
+require "ucode/glyphs/embedded_fonts/source"
+require "ucode/glyphs/resolver"
+require "ucode/glyphs/sources/pillar1_embedded_tounicode"
+require "ucode/glyphs/sources/tier1_real_font"
+
+module Ucode
+  module CodeChart
+    # Walks every assigned codepoint in a block and returns one
+    # {Result} per codepoint that any tier produced a glyph for.
+    #
+    # This is **not** a new extraction pipeline — it composes the
+    # existing {Ucode::Glyphs::Resolver} with per-block inputs
+    # (the block's Code Charts PDF + optionally Tier 1 and Pillar 3
+    # sources). The Resolver owns tier selection; the Extractor owns
+    # inputs.
+    #
+    # The REQ (R2) describes extraction via "locate the grid cell
+    # whose margin label matches the codepoint" — that was the v0.1
+    # retired approach (cell-border compositing). The current path
+    # is the embedded-font walk (Pillar 1, via {EmbeddedFonts::Catalog})
+    # with Pillar 2 (positional correlation) and Pillar 3 (Last Resort
+    # placeholders) as fallbacks.
+    #
+    # ## Tier selection
+    #
+    # Pillar 1 is always configured (the embedded font walk over the
+    # block's PDF). Tier 1 (real-font cmap) and Pillar 3 (Last
+    # Resort) are optional — the caller injects pre-built sources.
+    # This avoids forcing the Extractor to construct Last Resort
+    # eagerly, which would fail in environments where the UFO is
+    # not checked out.
+    class Extractor
+      # Result of extracting one codepoint.
+      Result = Struct.new(:codepoint, :svg, :tier, :provenance, keyword_init: true)
+
+      # @param block [Ucode::Models::Block] block whose assigned
+      #   codepoints will be extracted
+      # @param pdf_path [Pathname, String] path to the per-block
+      #   Code Charts PDF (downloaded by the caller; the Extractor
+      #   doesn't fetch)
+      # @param cache_dir [Pathname, String, nil] directory for
+      #   cached extracted font streams. nil = default
+      #   (data/pdf-fonts/ relative to the gem root).
+      # @param tier1_sources [Array<Ucode::Glyphs::Source>, nil]
+      #   optional Tier 1 sources (real-font cmap). nil = no Tier 1
+      # @param pillar3_source [Ucode::Glyphs::Source, nil] optional
+      #   Pillar 3 (Last Resort) source. nil = no Pillar 3 fallback.
+      #   Callers that want Last Resort placeholders inject the
+      #   pre-built source here.
+      def initialize(block:, pdf_path:, cache_dir: nil,
+                     tier1_sources: nil, pillar3_source: nil)
+        @block = block
+        @pdf_path = Pathname.new(pdf_path)
+        @cache_dir = cache_dir && Pathname.new(cache_dir)
+        @tier1_sources = tier1_sources || []
+        @pillar3_source = pillar3_source
+      end
+
+      # @return [Array<Result>] one Result per codepoint that any
+      #   tier produced a glyph for. Codepoints no tier can serve
+      #   are silently skipped (no Result yielded).
+      def extract
+        resolver = build_resolver
+        results = []
+        each_codepoint do |cp|
+          resolver_result = resolver.resolve(cp)
+          next unless resolver_result&.svg
+
+          results << Result.new(
+            codepoint: cp,
+            svg: resolver_result.svg,
+            tier: resolver_result.tier,
+            provenance: resolver_result.provenance,
+          )
+        end
+        results
+      end
+
+      private
+
+      # Yields every codepoint in the block's range in ascending
+      # order. We yield the whole range because the Resolver's
+      # tiers handle unassigned codepoints — Pillar 3 (when
+      # configured) maps every codepoint via its Format 13 cmap,
+      # so unassigned slots get a placeholder. With no Pillar 3
+      # injected, only assigned codepoints (those the embedded
+      # font actually covers) yield Results; the rest are silently
+      # skipped, satisfying the REQ's "skip unassigned codepoints".
+      def each_codepoint
+        return enum_for(:each_codepoint) unless block_given?
+
+        (@block.range_first..@block.range_last).each do |cp|
+          yield cp
+        end
+      end
+
+      def build_resolver
+        sources = @tier1_sources.dup
+        sources.concat(embedded_pillar_sources)
+        sources << @pillar3_source if @pillar3_source
+        order = sources.map(&:tier).uniq
+        Glyphs::Resolver.new(sources: sources, order: order)
+      end
+
+      def embedded_pillar_sources
+        embedded_source = Glyphs::EmbeddedFonts::Source.new(
+          pdf: @pdf_path, cache_dir: @cache_dir,
+        )
+        catalog = Glyphs::EmbeddedFonts::Catalog.new(embedded_source)
+        renderer = Glyphs::EmbeddedFonts::Renderer.new(catalog)
+        [Glyphs::Sources::Pillar1EmbeddedTounicode.new(renderer: renderer)]
+      end
+    end
+  end
+end

data/lib/ucode/code_chart/provenance.rb

@@ -0,0 +1,81 @@
+# frozen_string_literal: true
+
+require "digest"
+require "time"
+
+require "ucode/version"
+
+module Ucode
+  module CodeChart
+    # Per-codepoint provenance value object — every field the REQ
+    # (R5) requires in the sidecar JSON next to each extracted SVG.
+    #
+    # Single source of truth for the provenance schema: the
+    # {Sidecar} writer reads this Struct, the Writer constructs it.
+    # Adding a field is one place to change.
+    #
+    # `extractor_version` reads from `Ucode::VERSION` at construction
+    # so the field stays in sync with the gem's version bump — single
+    # source of truth.
+    #
+    # `extracted_at` is the extraction event timestamp (UTC ISO8601),
+    # not the file-write timestamp.
+    Provenance = Struct.new(
+      :codepoint,
+      :block,
+      :source_pdf_url,
+      :source_pdf_sha256,
+      :ucd_version,
+      :extracted_at,
+      :extractor_version,
+      keyword_init: true,
+    )
+
+    # Computes the source PDF's URL from a block name and first
+    # codepoint. Mirrors the per-block URL convention in
+    # {Ucode::Fetch::CodeCharts}: 4-digit hex for BMP, 6-digit for
+    # supplementary planes.
+    #
+    # @param block_first_cp [Integer]
+    # @return [String]
+    def self.code_chart_url(block_first_cp)
+      width = block_first_cp > 0xFFFF ? 6 : 4
+      slug = block_first_cp.to_s(16).upcase.rjust(width, "0")
+      "#{Ucode.configuration.charts_base_url}/U#{slug}.pdf"
+    end
+
+    # Builds a Provenance from the inputs the {Writer} has on hand
+    # (block, codepoint, ucd_version, pdf_path). Computes the PDF
+    # hash + URL once. The `extracted_at` timestamp is fixed at
+    # call time so re-running the same block produces identical
+    # provenance JSON for unchanged codepoints.
+    #
+    # @param block [Ucode::Models::Block]
+    # @param codepoint [Integer]
+    # @param ucd_version [String]
+    # @param pdf_path [Pathname, String]
+    # @param now [Time, nil] override for tests
+    # @return [Provenance]
+    def self.build(block:, codepoint:, ucd_version:, pdf_path:, now: nil)
+      path = Pathname.new(pdf_path)
+      Provenance.new(
+        codepoint: format("U+%04X", codepoint),
+        block: block.id,
+        source_pdf_url: code_chart_url(block.range_first),
+        source_pdf_sha256: sha256_of(path),
+        ucd_version: ucd_version,
+        extracted_at: (now || Time.now.utc).iso8601,
+        extractor_version: Ucode::VERSION,
+      )
+    end
+
+    # @param path [Pathname]
+    # @return [String] hex digest, "" when the path doesn't exist
+    #   (callers can decide how to handle a missing hash)
+    def self.sha256_of(path)
+      return "" unless path.exist?
+
+      Digest::SHA256.file(path).hexdigest
+    end
+  end
+end

data/lib/ucode/code_chart/sidecar.rb

@@ -0,0 +1,52 @@
+# frozen_string_literal: true
+
+require "json"
+require "pathname"
+
+require "ucode/repo/atomic_writes"
+
+module Ucode
+  module CodeChart
+    # Writes a {Provenance} to disk as the sidecar JSON next to its
+    # corresponding SVG.
+    #
+    # Path: `<output_root>/<codepoint>.json` — colocated with the
+    # SVG so a downstream consumer can find both files by a single
+    # directory listing.
+    #
+    # Idempotent via {Ucode::Repo::AtomicWrites#write_atomic}: a
+    # re-write of byte-identical content is a no-op (no temp-file
+    # rename). Provenance JSON is canonical (sorted keys via Ruby's
+    # stdlib JSON), so the byte-equality test is sound.
+    class Sidecar
+      include Ucode::Repo::AtomicWrites
+
+      # @param output_root [Pathname, String] directory the SVG +
+      #   sidecar live in. Parent directories are created on demand.
+      def initialize(output_root:)
+        @output_root = Pathname.new(output_root)
+      end
+
+      # @param provenance [Ucode::CodeChart::Provenance]
+      # @return [Pathname] the written sidecar path
+      def write(provenance)
+        path = path_for(provenance)
+        payload = JSON.pretty_generate(provenance.to_h) + "\n"
+        write_atomic(path, payload)
+        path
+      end
+
+      # @param codepoint_id [String] e.g. "U+10920"
+      # @return [Pathname] the would-be path for a sidecar
+      def path_for_id(codepoint_id)
+        @output_root.join("#{codepoint_id}.json")
+      end
+
+      private
+
+      def path_for(provenance)
+        path_for_id(provenance.codepoint)
+      end
+    end
+  end
+end