ucode 0.2.0 → 0.2.2

data/TODO.extract-code-chart/05-provenance-and-sidecar.md

@@ -0,0 +1,147 @@
+# TODO 05 — CodeChart::Provenance and CodeChart::Sidecar
+
+## Status
+
+Pending. Depends on TODO 03 (namespace). Depends on TODO 04 (Extractor)
+because Sidecar consumes the Extractor's Result.
+
+## Goal
+
+`Provenance` is the value object carrying the metadata the REQ (R5)
+requires for each extracted SVG's sidecar JSON. `Sidecar` is the
+writer that serializes one Provenance to disk next to its SVG.
+
+## Files
+
+- `lib/ucode/code_chart/provenance.rb` — `Ucode::CodeChart::Provenance`
+  Struct with all REQ R5 fields plus the construction helper.
+- `lib/ucode/code_chart/sidecar.rb` — `Ucode::CodeChart::Sidecar`
+  class (writes a sidecar JSON next to an SVG, idempotent via the
+  existing `Ucode::Repo::AtomicWrites`).
+- `spec/ucode/code_chart/provenance_spec.rb`
+- `spec/ucode/code_chart/sidecar_spec.rb`
+
+## Design
+
+### Provenance value object
+
+The REQ (R5) lists these fields:
+
+```json
+{
+  "codepoint": "U+10920",
+  "block": "Sidetic",
+  "source_pdf_url": "https://www.unicode.org/charts/PDF/U-10920.pdf",
+  "source_pdf_sha256": "...",
+  "ucd_version": "17.0.0",
+  "extracted_at": "2026-06-30T12:00:00Z",
+  "extractor_version": "0.1.0"
+}
+```
+
+`Struct` is the right tool — single source of truth for the schema,
+keyword-init for clarity, immutable-by-convention. Mirror the
+existing `Ucode::Repo::BuildReportAccumulator` pattern.
+
+```ruby
+Provenance = Struct.new(
+  :codepoint,           # String "U+10920"
+  :block,               # String "Sidetic"
+  :source_pdf_url,      # String
+  :source_pdf_sha256,   # String (hex digest)
+  :ucd_version,         # String "17.0.0"
+  :extracted_at,        # String ISO8601 UTC
+  :extractor_version,   # String "0.2.0"
+  keyword_init: true,
+)
+```
+
+`extractor_version` reads from `Ucode::VERSION` so it stays in sync
+with the gem — single source of truth.
+
+`extracted_at` is set at construction (not at file write) so the
+field describes the extraction event, not the serialization event.
+
+### Provenance → Hash serialization
+
+`Provenance#to_h` returns the hash form. NO hand-rolled `to_json` /
+`from_json` per the global rule — `Provenance` is a value object, but
+its schema is simple enough that `to_h` + `JSON.pretty_generate` is
+the framework-driven approach (lutaml-model is overkill for a flat
+struct).
+
+Wait — re-reading the global rule: "ALL (de)serialization goes through
+the framework. In Coradoc and any project using `lutaml-model`." So
+the rule is for projects using lutaml-model, and ucode uses lutaml-model
+for UCD models. This Provenance struct is not a UCD model — it's a
+feature-local value object. JSON via `JSON.pretty_generate(provenance.to_h)`
+is acceptable and avoids ceremony.
+
+`to_h` produces a hash with the Struct's keyword keys. No
+indirection, no lutaml-model mapping for what is effectively a
+record.
+
+### Sidecar writer
+
+```ruby
+class Sidecar
+  include Ucode::Repo::AtomicWrites
+
+  def initialize(output_root:)
+    @output_root = Pathname.new(output_root)
+  end
+
+  # Writes <output_root>/<cp_id>.json next to the corresponding SVG.
+  # Idempotent: re-writing the same content is a no-op (byte-stable).
+  #
+  # @param provenance [Ucode::CodeChart::Provenance]
+  # @return [Pathname] the written path
+  def write(provenance)
+    path = path_for(provenance)
+    payload = JSON.pretty_generate(provenance.to_h)
+    write_atomic(path, payload + "\n")
+    path
+  end
+
+  private
+
+  def path_for(provenance)
+    @output_root.join("#{provenance.codepoint}.json")
+  end
+end
+```
+
+`Repo::AtomicWrites#write_atomic` is the project's single source of
+truth for idempotent file writes — bytes-identical re-writes are
+no-ops (the temp-file rename is skipped when content matches).
+Reuse, don't reimplement.
+
+### Why a separate Sidecar class
+
+A `Provenance.to_disk(path)` method would couple the value object
+to I/O. Keeping the writer separate lets:
+- Tests assert `Provenance#to_h` without touching disk.
+- The Writer (TODO 06) compose `Extractor` + `Sidecar` with explicit
+  dependency injection (seam for testing).
+- Future formats (e.g. a different sidecar schema) replace Sidecar
+  without touching Provenance.
+
+This is MECE: Provenance is data; Sidecar is I/O; Writer is
+orchestration.
+
+## Acceptance
+
+- `Provenance.new(codepoint: "U+10920", block: "Sidetic", ...)`
+  constructs without raising.
+- `Provenance#to_h` returns a Hash with exactly the REQ's fields.
+- `Sidecar#write(provenance)` writes `<codepoint>.json` next to
+  where the SVG lives; the JSON content matches `Provenance#to_h`.
+- Re-writing the same Provenance is a no-op (file unchanged).
+- Specs cover all five REQ fields plus the idempotency guarantee.
+
+## Out of scope
+
+- License attribution text — the REQ mentions `LICENSE-SOURCES.md`
+  obligations, but that's a fontist-side concern (downstream
+  essenfont build). Ucode emits provenance; the consumer stitches
+  attribution.

data/TODO.extract-code-chart/06-codechart-writer.md

@@ -0,0 +1,134 @@
+# TODO 06 — CodeChart::Writer
+
+## Status
+
+Pending. Depends on TODO 04 (Extractor) and TODO 05 (Provenance +
+Sidecar).
+
+## Goal
+
+`Ucode::CodeChart::Writer` is the single entry point for
+"extract every codepoint in block X and write SVG + sidecar JSON
+files under output_dir." It's the orchestration layer the CLI
+calls and the only thing that touches disk.
+
+## Files
+
+- `lib/ucode/code_chart/writer.rb` — `Ucode::CodeChart::Writer` class.
+- `spec/ucode/code_chart/writer_spec.rb`
+
+## Design
+
+### Class shape
+
+```ruby
+class Ucode::CodeChart::Writer
+  Summary = Struct.new(:block, :codepoints_total, :svgs_written,
+                       :sidecars_written, :pdf_sha256, keyword_init: true)
+
+  def initialize(output_root:, pdf_path:, cache_dir: nil,
+                 last_resort_root: nil, blocks_txt: nil)
+    @output_root = Pathname.new(output_root)
+    @pdf_path = Pathname.new(pdf_path)
+    @cache_dir = cache_dir
+    @last_resort_root = last_resort_root
+    @blocks_txt = blocks_txt || Ucode::Cache.ucd_dir(Ucode::VersionResolver.resolve(nil)).join("Blocks.txt")
+    @sidecar = Sidecar.new(output_root: @output_root)
+  end
+
+  # Extracts every codepoint in @block (a Models::Block) and writes
+  # SVG + sidecar JSON under @output_root. Returns a Summary.
+  #
+  # @param block [Ucode::Models::Block]
+  # @return [Summary]
+  def write(block)
+  end
+end
+```
+
+### Per-codepoint flow (single source of truth)
+
+```ruby
+def write(block)
+  output_root_for(block).mkpath
+  pdf_sha = sha256(@pdf_path)
+
+  extractor = Extractor.new(
+    block: block,
+    blocks_txt: @blocks_txt,
+    pdf_path: @pdf_path,
+    cache_dir: @cache_dir,
+    last_resort_root: @last_resort_root,
+  )
+  results = extractor.extract
+
+  svgs = 0
+  sidecars = 0
+  results.each do |result|
+    svg_path = output_root_for(block).join("#{cp_id(result.codepoint)}.svg")
+    File.write(svg_path, result.svg) unless svg_path.exist? && File.read(svg_path) == result.svg
+    svgs += 1 if svg_path.exist?
+
+    provenance = build_provenance(block, result.codepoint, pdf_sha)
+    @sidecar.write(provenance)
+    sidecars += 1
+  end
+
+  Summary.new(
+    block: block.id,
+    codepoints_total: results.size,
+    svgs_written: svgs,
+    sidecars_written: sidecars,
+    pdf_sha256: pdf_sha,
+  )
+end
+```
+
+### Why not use `Repo::AtomicWrites` for the SVGs
+
+The Sidecar uses `Repo::AtomicWrites` because JSON has a stable
+canonical form. SVG output from `EmbeddedFonts::Svg#to_s` is also
+byte-stable, but the writer pattern is simpler with `File.write` —
+the byte-equality check above guarantees idempotency at the I/O
+layer. Both paths reach the same outcome.
+
+If future output formats gain non-stable serialization (timestamps,
+random IDs), the SVG path will need `Repo::AtomicWrites` too. Until
+then, simpler is better.
+
+### Why compute `pdf_sha256` once
+
+Every Provenance in this block carries the same `source_pdf_sha256`.
+Computing it once avoids 32+ disk reads of the PDF per block
+extraction. The Writer is the single place that knows "one block,
+one PDF, one hash" — pushing the calculation into Sidecar or
+Provenance would require either (a) repeated computation per
+Provenance or (b) a parameter-passing thread through Extractor. Both
+violate locality.
+
+### Output layout
+
+`<output_root>/<block_id>/<U+XXXX>.svg` and `<U+XXXX>.json`.
+
+One folder per block keeps the `Writer`'s output self-contained and
+discoverable — a downstream consumer (fontisan) can iterate a block's
+folder without scanning the whole tree. This mirrors the existing
+`Ucode::Repo::Writers::BlocksWriter` output convention (one folder
+per block, index.json inside).
+
+## Acceptance
+
+- `Writer#write(block)` creates `<output_root>/<block_id>/` and
+  fills it with `<U+XXXX>.svg` + `<U+XXXX>.json` for every
+  extracted codepoint.
+- Re-running `Writer#write(block)` with no changes produces
+  byte-identical files (no rewrites).
+- `Summary#svgs_written` equals the number of extracted codepoints.
+- Specs cover the full lifecycle including idempotency.
+
+## Out of scope
+
+- Per-block write isolation — concurrent `Writer#write` calls for
+  different blocks are safe (different folders), but the Writer is
+  not thread-safe within a single block. That's a parallel-extraction
+  concern, not a per-block concern.

data/TODO.extract-code-chart/07-codechart-cli.md

@@ -0,0 +1,135 @@
+# TODO 07 — CodeChart CLI
+
+## Status
+
+Pending. Depends on TODO 06 (Writer), TODO 04 (Extractor),
+TODO 02 (block name resolver).
+
+## Goal
+
+`ucode code-chart fetch | extract | list` — the REQ (R4) commands.
+Thin Thor wrappers that delegate to the existing `CodeChart::*`
+modules. No orchestration logic in the CLI; every command is a
+single delegation.
+
+## Files
+
+- `lib/ucode/cli.rb` — add the `CodeChartCmd` Thor subcommand class.
+- `spec/ucode/cli_spec.rb` (extend existing) — verify the new
+  subcommand wires up.
+
+## Design
+
+### Subcommand shape
+
+```ruby
+class Cli < Thor
+  # …existing commands…
+
+  class CodeChartCmd < Thor
+    desc "fetch --block BLOCK", "Download the Code Charts PDF for a block"
+    option :block, type: :string, required: true,
+                  desc: "Block identifier (e.g. Sidetic)"
+    def fetch
+      puts JSON.pretty_generate(
+        Commands::FetchCommand.new.fetch_charts(
+          VersionResolver.resolve(nil),
+          block_first_cps: [block_first_cp!(options[:block])],
+        ),
+      )
+    end
+
+    desc "extract --block BLOCK --to DIR", "Extract per-codepoint SVG + provenance sidecars"
+    option :block, type: :string, required: true,
+                  desc: "Block identifier (e.g. Sidetic)"
+    option :to, type: :string, required: true,
+                desc: "Output directory"
+    def extract
+      # ...
+    end
+
+    desc "list", "List blocks that have Code Charts PDFs available locally"
+    def list
+      # ...
+    end
+  end
+
+  desc "code-chart", "Extract per-codepoint SVG glyphs from Unicode Code Charts PDFs"
+  subcommand "code-chart", CodeChartCmd
+end
+```
+
+### `extract` flow
+
+```ruby
+def extract
+  Ucode::Commands::FetchCommand.new.fetch_charts(
+    VersionResolver.resolve(nil),
+    block_first_cps: [block_first_cp!(options[:block])],
+  )
+  blocks_txt = Ucode::Cache.ucd_dir(VersionResolver.resolve(nil)).join("Blocks.txt")
+  block = Parsers::Blocks.find_by_name(blocks_txt, options[:block]) or
+    raise Thor::Error, "Unknown block: #{options[:block].inspect}"
+  pdf = Ucode::Glyphs::PdfFetcher.new(
+    VersionResolver.resolve(nil),
+    monolith_path: nil,
+    blocks: [block],
+  ).fetch(block_first_cp: block.range_first, force: false) or
+    raise Thor::Error, "PDF unavailable for block #{options[:block]}"
+
+  writer = Ucode::CodeChart::Writer.new(
+    output_root: Pathname.new(options[:to]),
+    pdf_path: pdf,
+    blocks_txt: blocks_txt,
+  )
+  summary = writer.write(block)
+  puts JSON.pretty_generate(summary.to_h.compact)
+end
+
+def block_first_cp!(block_id)
+  cache = Ucode::Cache.ucd_dir(VersionResolver.resolve(nil))
+  block = Ucode::Parsers::Blocks.find_by_name(cache.join("Blocks.txt"), block_id)
+  raise Thor::Error, "Unknown block: #{block_id.inspect}" unless block
+  block.range_first
+end
+```
+
+### Why `Ucode::Commands::FetchCommand.new.fetch_charts` for fetch
+
+`fetch_charts` is the existing CLI hook for "download a Code Charts
+PDF for these block first-cps." We just call it with the block's
+first cp. Reuse, don't reimplement.
+
+### Why resolve version once at the top of `extract`
+
+Per Candidate 4 of the architecture review (`refactor/build-context-resolve-version-once`,
+merged): every CLI method resolves the version once and threads it
+through. This CLI method does the same — one call per invocation.
+
+### Why no separate `Commands::CodeChartCommand` class
+
+Following the existing pattern (e.g. `Cli::Audit` calls
+`Commands::Audit::*Command` *only* when the logic is non-trivial).
+The CodeChart commands are trivial delegations — a one-liner each.
+The CLI methods call `CodeChart::Writer` and `CodeChart::Extractor`
+directly. Adding a `Commands::CodeChartCommand` class would be
+indirection without a payoff.
+
+If the extract logic grows (e.g. progress reporting, partial
+extraction), extract it into a Command class at that point.
+
+## Acceptance
+
+- `ucode code-chart fetch --block Sidetic` downloads the PDF.
+- `ucode code-chart extract --block Sidetic --to /tmp/s/` extracts
+  to the given directory.
+- `ucode code-chart list` prints available blocks.
+- Unknown block names produce a clean error, not a stack trace.
+- The CLI matches the REQ's signature exactly.
+
+## Out of scope
+
+- `--version` flag (the REQ doesn't specify, and existing commands
+  default to the configured version).
+- `--format svg|glif` (the REQ specifies SVG; `.glif` output is a
+  future extension).

data/TODO.extract-code-chart/08-specs.md

@@ -0,0 +1,87 @@
+# TODO 08 — CodeChart specs
+
+## Status
+
+Pending. Depends on TODOs 01–07.
+
+## Goal
+
+Comprehensive spec coverage for every new module. Per the project
+rules: real model instances (no doubles), tight focused tests,
+behavior assertions (not interaction counts).
+
+## Files
+
+Already enumerated per TODO. Final spec coverage:
+
+- `spec/ucode/fetch/code_charts_spec.rb` — happy path + HTTP 4xx +
+  wrong content-type + non-PDF body. (TODO 01)
+- `spec/ucode/parsers/blocks_spec.rb` (extend) — `find_by_name`
+  happy path + nil on miss. (TODO 02)
+- `spec/ucode/code_chart/extractor_spec.rb` — constructor invariants,
+  Resolver wiring, integration test against fixture PDF. (TODO 04)
+- `spec/ucode/code_chart/provenance_spec.rb` — value object
+  construction + `to_h` schema. (TODO 05)
+- `spec/ucode/code_chart/sidecar_spec.rb` — write sidecar, idempotent
+  re-write. (TODO 05)
+- `spec/ucode/code_chart/writer_spec.rb` — full lifecycle:
+  extract → write → summary. Idempotent re-run produces byte-identical
+  files. (TODO 06)
+- `spec/ucode/cli_spec.rb` (extend) — verify `ucode code-chart fetch`,
+  `extract`, `list` wire up. (TODO 07)
+
+## Design
+
+### Fixture strategy
+
+The existing `spec/fixtures/pdfs/basic_latin.pdf` is the only PDF
+fixture in the repo. It's tiny and validates the integration path.
+The Sidetic + Egyptian Ext-B PDFs are large (whole-block) and would
+inflate the repo. The integration spec uses `basic_latin.pdf` to
+exercise the full pipeline; per-codepoint assertions cover
+representative cases.
+
+If Sidetic-specific behavior must be tested, a smaller fixture PDF
+cropped to ~5 codepoints would be the right tool — out of scope for
+this TODO.
+
+### No doubles policy
+
+The project's `~/.claude/CLAUDE.md` rule: no doubles. All specs use
+real instances:
+- `Ucode::Models::Block.new(...)` for test blocks.
+- A temp directory + real `Blocks.txt` text for parser specs.
+- The real `Ucode::Glyphs::Resolver` for extractor specs.
+
+### Idempotency assertion pattern
+
+`Writer#write` idempotency is asserted via byte-equality:
+
+```ruby
+first_run  = writer.write(block)
+first_size = File.stat(svg_path).size
+sleep 0.01  # ensure mtime changes would be detectable
+second_run = writer.write(block)
+second_size = File.stat(svg_path).size
+expect(second_size).to eq(first_size)
+expect(File.read(svg_path)).to eq(expected_svg_bytes)
+```
+
+This is the existing pattern from `spec/ucode/repo/aggregate_writer_spec.rb`
+(idiempotency spec there). Reuse.
+
+## Acceptance
+
+- `bundle exec rspec spec/ucode/code_chart/ spec/ucode/fetch/code_charts_spec.rb spec/ucode/parsers/blocks_spec.rb`
+  passes 100%.
+- Coverage for the new files is ≥ 95% (per the project's per-file
+  floor of 30% + the overall 80% minimum).
+- No doubles are introduced (verify with `grep -r "double(" spec/ucode/code_chart/`).
+- The integration spec exercises both the Extractor and Writer
+  together end-to-end.
+
+## Out of scope
+
+- Performance benchmarks — separate concern.
+- Sidetic-specific fixtures — requires PDF curation beyond the
+  scope of this feature.

data/config/unicode17_universal_glyph_set.yml

@@ -1,6 +1,6 @@
 ---
 unicode_version: 17.0.0
-ucode_version: 0.1.1
+ucode_version: 0.2.1
 generated_at: '2026-06-28T00:00:00Z'
 default_sources:
 - kind: fontist

data/lib/ucode/audit/reference_factory.rb

@@ -63,4 +63,4 @@ module Ucode
       end
     end
   end
-end
+end

data/lib/ucode/cli.rb

@@ -3,6 +3,7 @@
 require "thor"
 
 require "ucode/commands"
+require "ucode/code_chart"
 require "ucode/version_resolver"
 
 module Ucode
@@ -157,6 +158,106 @@ module Ucode
     desc "site", "Generate the Vitepress site"
     subcommand "site", Site
 
+    # ─────────────── code-chart ───────────────
+    # Extract per-codepoint SVG glyphs from a Unicode Code Charts PDF.
+    # One folder per block under --to, with <U+XXXX>.svg + .json pairs.
+    class CodeChartCmd < Thor
+      desc "fetch --block BLOCK [VERSION]", "Download the Code Charts PDF for a block"
+      option :block, type: :string, required: true,
+                     desc: "Block identifier (e.g. Sidetic, Basic_Latin)"
+      def fetch(version = nil)
+        with_codechart_errors do
+          block_first_cp = resolve_block_first_cp!(options[:block], version)
+          result = Commands::FetchCommand.new.fetch_charts(
+            VersionResolver.resolve(version),
+            block_first_cps: [block_first_cp],
+          )
+          puts JSON.pretty_generate(result)
+        end
+      end
+
+      desc "extract --block BLOCK --to DIR [VERSION]",
+           "Extract per-codepoint SVG + provenance sidecars from a Code Charts PDF"
+      option :block, type: :string, required: true,
+                     desc: "Block identifier (e.g. Sidetic)"
+      option :to, type: :string, required: true,
+                  desc: "Output directory (will contain <block_id>/<U+XXXX>.svg + .json)"
+      def extract(version = nil)
+        with_codechart_errors do
+          version_str = VersionResolver.resolve(version)
+          block = resolve_block!(options[:block], version_str)
+          block_first_cp = block.range_first
+
+          # Download (idempotent — re-runs skip when the PDF is cached).
+          Commands::FetchCommand.new.fetch_charts(version_str, block_first_cps: [block_first_cp])
+
+          pdf = Ucode::Glyphs::PdfFetcher.new(version_str)
+            .fetch(block_first_cp: block_first_cp)
+          unless pdf
+            raise Ucode::CodeChartNotFoundError.new(
+              "Code Charts PDF unavailable for block #{block.id.inspect}",
+              context: { block_id: block.id, version: version_str },
+            )
+          end
+
+          writer = Ucode::CodeChart::Writer.new(
+            output_root: Pathname.new(options[:to]),
+            pdf_path: pdf,
+            ucd_version: version_str,
+          )
+          summary = writer.write(block)
+          puts JSON.pretty_generate(summary.to_h.compact)
+        end
+      end
+
+      desc "list", "List cached Code Charts PDFs under the version's cache"
+      def list
+        version = VersionResolver.resolve(nil)
+        pdfs_dir = Ucode::Cache.pdfs_dir(version)
+        files = pdfs_dir.exist? ? pdfs_dir.children.sort : []
+        if files.empty?
+          puts "(no cached Code Charts PDFs)"
+          return
+        end
+        files.each do |f|
+          puts f.basename
+        end
+      end
+
+      private
+
+      # Resolve a block name to its first codepoint via the cached
+      # Blocks.txt. Raises {Ucode::UnknownBlockError} on miss.
+      def resolve_block!(block_id, version)
+        blocks_txt = Ucode::Cache.ucd_dir(VersionResolver.resolve(version)).join("Blocks.txt")
+        Ucode::Parsers::Blocks.find_by_id!(blocks_txt, block_id)
+      end
+
+      def resolve_block_first_cp!(block_id, version)
+        resolve_block!(block_id, version).range_first
+      end
+
+      # Convert semantic Ucode errors into Thor errors so Thor's
+      # dispatch prints the message cleanly instead of a stack trace.
+      # Thor's `start` rescues only `Thor::Error`; without this bridge,
+      # any `Ucode::Error` subclass propagates as an uncaught exception.
+      def with_codechart_errors
+        yield
+      rescue Ucode::Error => e
+        raise Thor::Error, e.message
+      end
+    end
+
+    # Register the subcommand under the underscored method name
+    # (`code_chart`). Thor's `normalize_command_name` converts the
+    # user's hyphenated form (`code-chart`) to the underscored form
+    # before lookup, so `ucode code-chart <cmd>` dispatches correctly.
+    # `desc` first registers the method as a Thor command so the
+    # dispatch table has an entry; `subcommand` then attaches the
+    # CodeChartCmd class to it.
+    desc "code_chart <command>", "Extract SVG glyphs from Unicode Code Charts PDFs"
+    subcommand "code_chart", CodeChartCmd
+
     # ─────────────── lookup ───────────────
     class Lookup < Thor
       desc "block CODEPOINT", "Block name covering CODEPOINT (integer or 0xNNNN)"