ucode 0.1.0

data/TODO.new/02-audit-schema-design.md

@@ -0,0 +1,255 @@
+# 02 — Audit schema design
+
+## Goal
+
+Define the lutaml-model class hierarchy for the per-face font audit
+report. This is the in-memory shape; serialization to the directory
+tree is `03-directory-output-spec.md`.
+
+The schema is the contract: fontist.org codes against it, the HTML
+browser renders it, and the migration ports to it. Lock this before
+touching any extractor.
+
+## Source material
+
+Port from `fontisan/lib/fontisan/models/audit/` (15 files, ~750 lines
+total) with the adjustments below. Do not invent new fields without a
+documented consumer.
+
+## Top-level model
+
+```ruby
+# lib/ucode/models/audit/audit_report.rb
+class AuditReport < Lutaml::Model::Serializable
+  # --- Provenance ---
+  attribute :generated_at, :string
+  attribute :ucode_version, :string          # was: fontisan_version
+  attribute :source_file, :string
+  attribute :source_sha256, :string
+  attribute :source_format, :string
+
+  # --- Source layout ---
+  attribute :font_index, :integer
+  attribute :num_fonts_in_source, :integer
+
+  # --- Identity (name table) ---
+  attribute :family_name, :string
+  attribute :subfamily_name, :string
+  attribute :full_name, :string
+  attribute :postscript_name, :string
+  attribute :version, :string
+  attribute :font_revision, :float
+
+  # --- Style (OS/2 + head) ---
+  attribute :weight_class, :integer
+  attribute :width_class, :integer
+  attribute :italic, Lutaml::Model::Type::Boolean
+  attribute :bold, Lutaml::Model::Type::Boolean
+  attribute :panose, :string
+
+  # --- Coverage ---
+  attribute :total_codepoints, :integer
+  attribute :total_glyphs, :integer
+  attribute :cmap_subtables, :integer, collection: true
+  attribute :codepoint_ranges, CodepointRange, collection: true
+  attribute :codepoints, :string, collection: true  # "U+XXXX" form
+
+  # --- Aggregations (driven by ucode's own UCD, not ucd.all.flat.zip) ---
+  attribute :baseline, Baseline                 # see below — replaces ucd_version
+  attribute :blocks, BlockSummary, collection: true
+  attribute :scripts, ScriptSummary, collection: true  # was: unicode_scripts (string list)
+
+  # --- Optional deep tables (nil for Type 1) ---
+  attribute :licensing, Licensing
+  attribute :metrics, Metrics
+  attribute :hinting, Hinting
+  attribute :color_capabilities, ColorCapabilities
+  attribute :variation, VariationDetail
+  attribute :opentype_layout, OpenTypeLayout
+
+  # --- Audit signals ---
+  attribute :discrepancies, Discrepancy, collection: true  # NEW
+  attribute :warning, :string
+
+  key_value do
+    # ... one map line per attribute ...
+  end
+end
+```
+
+## New / changed sub-models vs fontisan
+
+### `Baseline` (NEW — replaces fontisan's bare `ucd_version` string)
+
+```ruby
+class Baseline < Lutaml::Model::Serializable
+  attribute :unicode_version, :string        # "17.0.0"
+  attribute :ucode_version, :string
+  attribute :fontisan_version, :string       # parser provenance
+  attribute :source, :string                 # "ucd-text + Unicode17Blocks overrides"
+  attribute :generated_at, :string
+end
+```
+
+### `BlockSummary` (replaces fontisan's `AuditBlock`)
+
+```ruby
+class BlockSummary < Lutaml::Model::Serializable
+  attribute :name, :string                   # original block name verbatim
+  attribute :first_cp, :integer
+  attribute :last_cp, :integer
+  attribute :range, :string                  # "U+0000–U+007F" (display form)
+  attribute :plane, :integer                 # 0-16
+  attribute :total_assigned, :integer        # ucode's curated count
+  attribute :covered_count, :integer
+  attribute :missing_count, :integer
+  attribute :coverage_percent, :float
+  attribute :status, :string                 # see enum below
+  attribute :missing_codepoints, :integer, collection: true  # always populated
+  attribute :covered_codepoints, :integer, collection: true  # verbose only
+end
+
+# status enum (string — no symbol serialization):
+#   COMPLETE              — covered_count == total_assigned
+#   PARTIAL               — 0 < covered_count < total_assigned
+#   UNCOVERED_ASSIGNED    — covered_count == 0 && total_assigned > 0
+#   NO_ASSIGNED_IN_BLOCK  — total_assigned == 0 (rare; PUA blocks)
+#   OUTSIDE_BASELINE      — block exists in font's cmap but not in baseline
+```
+
+### `ScriptSummary` (replaces fontisan's bare `unicode_scripts: String[]`)
+
+```ruby
+class ScriptSummary < Lutaml::Model::Serializable
+  attribute :script_code, :string            # "Latn", "Hani", ...
+  attribute :script_name, :string            # "Latin", "Han", ...
+  attribute :blocks_total, :integer
+  attribute :assigned_total, :integer
+  attribute :covered_total, :integer
+  attribute :coverage_percent, :float
+  attribute :status, :string                 # same enum as BlockSummary minus OUTSIDE_BASELINE
+end
+```
+
+### `Discrepancy` (NEW — cheap audit signal)
+
+```ruby
+class Discrepancy < Lutaml::Model::Serializable
+  attribute :kind, :string        # "os2_unicode_range_bit_without_cmap_codepoints"
+  attribute :detail, :string      # human-readable explanation
+  attribute :block_name, :string  # optional context
+  attribute :bit_position, :integer   # optional (OS/2 ulUnicodeRange bit)
+end
+```
+
+### Plane rollup
+
+```ruby
+class PlaneSummary < Lutaml::Model::Serializable
+  attribute :plane, :integer
+  attribute :blocks_total, :integer
+  attribute :assigned_total, :integer
+  attribute :covered_total, :integer
+  attribute :coverage_percent, :float
+end
+```
+
+Carried on the report as `attribute :plane_summaries, PlaneSummary, collection: true`.
+
+### Codepoint detail (verbose only — emitted to a separate file)
+
+```ruby
+class CodepointDetail < Lutaml::Model::Serializable
+  attribute :codepoint, :integer
+  attribute :name, :string
+  attribute :general_category, :string
+  attribute :script, :string
+  attribute :script_extensions, :string, collection: true
+  attribute :block_name, :string
+  attribute :age, :string
+  attribute :glyph_id, :integer               # GID in the audited font
+  attribute :glyph_svg_path, :string          # relative path under glyphs/, when emitted
+end
+```
+
+## Ported unchanged from fontisan
+
+These sub-models port across with namespace changes only (`Fontisan::`
+→ `Ucode::`):
+
+- `Licensing`
+- `Metrics`
+- `Hinting`
+- `ColorCapabilities`
+- `VariationDetail`
+- `OpenTypeLayout`
+- `CodepointRange`, `CodepointSetDiff`
+- `AuditAxis`, `NamedInstance`
+- `FsSelectionFlags`, `GaspRange`, `EmbeddingType`
+- `ScriptCoverageRow`, `ScriptFeatures`
+- `FieldChange` (for Differ), `DuplicateGroup` (for LibraryAuditor)
+- `LibrarySummary`
+- `AuditDiff` (for compare command)
+
+## What's dropped vs fontisan
+
+- **`language_coverage`** and `Models::Cldr::*` — CLDR is out of scope
+  (UCD Scripts.txt + ScriptExtensions.txt already define per-codepoint
+  script coverage; CLDR was overreach). See `00-README.md` decision.
+- **`cldr_version`** on AuditReport — same reason.
+- The `fontisan_version` field is renamed `ucode_version`. Fontisan is
+  now an internal parser, not the report's identity.
+
+## File layout
+
+```
+lib/ucode/models/audit/
+├── audit_report.rb          # top-level
+├── baseline.rb              # NEW
+├── block_summary.rb         # was AuditBlock
+├── script_summary.rb        # NEW (was: string list)
+├── plane_summary.rb         # NEW
+├── discrepancy.rb           # NEW
+├── codepoint_detail.rb      # NEW
+├── codepoint_range.rb
+├── codepoint_set_diff.rb
+├── audit_axis.rb
+├── named_instance.rb
+├── licensing.rb
+├── metrics.rb
+├── hinting.rb
+├── color_capabilities.rb
+├── variation_detail.rb
+├── opentype_layout.rb
+├── fs_selection_flags.rb
+├── gasp_range.rb
+├── embedding_type.rb
+├── script_coverage_row.rb
+├── script_features.rb
+├── field_change.rb
+├── duplicate_group.rb
+├── library_summary.rb
+└── audit_diff.rb
+```
+
+Plus the namespace hub `lib/ucode/models/audit.rb` declaring the
+autoloads (Ruby autoload — see project memory `feedback_require_relative.md`
+for the rule).
+
+## Acceptance
+
+- All ~26 model classes ported and spec'd with round-trip
+  `to_hash` / `from_hash` tests. No hand-rolled `to_h`.
+- No use of `double()` in any spec.
+- The `AuditReport` shape produces the JSON described in
+  `04-fontist-org-contract.md` when serialized via the directory emitter
+  in `13-directory-emitter.md`.
+- Rubocop clean on all new files.
+
+## References
+
+- Source: `fontisan/lib/fontisan/models/audit/`
+- Project memory: `lutaml_model_polymorphism_api.md`,
+  `feedback_lutaml_model_inheritance.md`
+- Contract: `TODO.new/04-fontist-org-contract.md`
+- Output: `TODO.new/03-directory-output-spec.md`

data/TODO.new/03-directory-output-spec.md

@@ -0,0 +1,203 @@
+# 03 — Directory output spec
+
+## Goal
+
+Lock the per-face output tree on disk. The HTML browser fetches chunks
+lazily from this layout; fontist.org consumes the same files. One face
+= one directory; the browser never loads the whole tree at once.
+
+## Why directory, not single file
+
+A Unicode-17-complete CJK font carries ~50,000 codepoints across ~10
+blocks. A single self-contained JSON would be tens of MB — too big for
+a browser to parse without jank, and too big for fontist.org to fetch
+just to render a coverage map. Splitting by concern lets the consumer
+fetch only what it needs.
+
+## Layout
+
+```
+output/font_audit/<label>/
+├── index.json                  # face metadata + totals + per-block stats only
+├── index.html                  # standalone browser (inlined CSS/JS, no chunks inlined)
+├── planes/
+│   ├── 0.json                  # BMP rollup
+│   ├── 2.json                  # CJK plane rollup
+│   └── ...                     # 17 files max
+├── blocks/
+│   ├── Basic_Latin.json        # per-block: stats + missing_codepoints (always)
+│   ├── CJK_Unified_Ideographs.json
+│   └── ...                     # one per touched block
+├── scripts/
+│   ├── Latin.json              # per-script rollup
+│   ├── Han.json
+│   └── ...
+├── codepoints/                 # verbose mode only (--verbose)
+│   ├── Basic_Latin.json        # per-block codepoint detail list
+│   ├── CJK_Unified_Ideographs.json
+│   └── ...                     # chunked per block; each file <1MB even for CJK
+└── glyphs/                     # opt-in (--with-glyphs); one SVG per codepoint
+    ├── U+0041.svg
+    ├── U+4E00.svg
+    └── ...
+```
+
+For a TTC collection, sibling faces share the source directory:
+
+```
+output/font_audit/<source_label>/
+├── index.json                  # collection-level summary (num_fonts_in_source, etc.)
+├── index.html                  # collection browser (lists faces)
+├── 00-<face_ps_name>/
+│   ├── index.json
+│   └── ... (per-face layout above)
+├── 01-<face_ps_name>/
+│   └── ...
+└── ...
+```
+
+Filename pattern for collection faces:
+`{font_index:02d}-{safe_filename(postscript_name)}` — same convention
+fontisan uses today. The `00`-prefix guarantees face-order sort and
+disambiguates broken fonts where two faces share a PostScript name.
+
+## Block filename encoding
+
+Block names use the original Unicode verbatim form (e.g.
+`Greek_And_Coptic`, `CJK_Ext_A`). They contain spaces and underscores
+but never slashes — safe as filenames as-is. **Do not slugify.**
+
+Replace only the characters filesystems reject: `/` → `_`. Unicode
+block names contain no `/` today, so this is a defensive no-op.
+
+## File contents
+
+### `index.json`
+
+Compact face metadata + rollups. Carries everything a renderer needs
+for the initial overview without expanding any block:
+
+```json
+{
+  "generated_at": "2026-06-27T12:00:00Z",
+  "ucode_version": "0.2.0",
+  "font": { ... AuditReport identity + style + coverage-totals ... },
+  "baseline": { "unicode_version": "17.0.0", ... },
+  "totals": {
+    "assigned_codepoints_total": 150000,
+    "covered_codepoints_total": 2857,
+    "blocks_touched": 24,
+    "blocks_complete": 12,
+    "blocks_partial": 12,
+    "scripts_touched": 5
+  },
+  "discrepancies": [ ... ],
+  "plane_summaries": [ ... ],
+  "block_summaries": [
+    {
+      "name": "Basic Latin",
+      "first_cp": 0, "last_cp": 127, "plane": 0,
+      "total_assigned": 128, "covered_count": 128,
+      "missing_count": 0, "coverage_percent": 100.0,
+      "status": "COMPLETE",
+      "missing_codepoints": []
+    },
+    ...
+  ],
+  "script_summaries": [ ... ]
+}
+```
+
+Per-block `missing_codepoints` is **always** embedded (decision in
+`00-README.md`). Per-block `covered_codepoints` is **never** in
+`index.json` — fetch `codepoints/<NAME>.json` for that.
+
+### `blocks/<NAME>.json`
+
+Single `BlockSummary` object (same shape as the entry in
+`block_summaries`) plus optional `codepoints` detail if emitted in
+verbose mode. Carries the full missing list. Cheap to fetch per-block
+on demand.
+
+### `planes/<N>.json` and `scripts/<CODE>.json`
+
+Rollup views. Useful for renderers that group by plane or script
+without iterating all blocks.
+
+### `codepoints/<NAME>.json` (verbose only)
+
+```json
+{
+  "block_name": "Basic Latin",
+  "codepoints": [
+    {
+      "codepoint": 65,
+      "name": "LATIN CAPITAL LETTER A",
+      "general_category": "Lu",
+      "script": "Latin",
+      "block_name": "Basic Latin",
+      "age": "1.1",
+      "glyph_id": 36,
+      "glyph_svg_path": "glyphs/U+0041.svg"
+    },
+    ...
+  ]
+}
+```
+
+Per-block chunking keeps each file under ~1MB even for CJK. The browser
+fetches this only when the user expands a block to see per-character
+detail.
+
+### `glyphs/U+XXXX.svg`
+
+Plain SVG file (one glyph outline). The browser fetches individually
+on click. Output via `fontisan` outline reading on the audited font
+(decision: render from audited font, not Code Charts).
+
+## Library mode
+
+```
+output/font_audit/
+├── index.json                  # library summary (font count, totals)
+├── index.html                  # library browser (cards of audited fonts)
+├── <font_label_1>/
+├── <font_label_2>/
+└── ...
+```
+
+`Ucode::Audit::LibraryAuditor` walks a directory, audits each font into
+its own subdirectory, then emits the library-level index pointing at
+each face's `index.json`.
+
+## Idempotency
+
+Every emitted file is content-hash compared (same pattern as
+`Ucode::Repo::AtomicWrites`). Re-running `ucode audit font <path>` on
+an unchanged source leaves existing files untouched. Re-running on a
+changed source rewrites only the affected chunks.
+
+Skip-newer check: if a chunk file's mtime is newer than the source
+font's mtime AND the baseline UCD's mtime, skip the rewrite entirely.
+This matches the canonical-dataset writer's idempotency rule from
+`CLAUDE.md`.
+
+## Acceptance
+
+- A `--verbose` audit of a 50k-codepoint CJK font produces an
+  `index.json` under 200KB and no per-chunk file over 1MB.
+- A non-verbose audit produces `index.json`, `planes/`, `blocks/`,
+  `scripts/` only — no `codepoints/` and no `glyphs/`.
+- A `--with-glyphs` audit additionally produces `glyphs/U+XXXX.svg` per
+  covered codepoint.
+- All filenames preserve original block names verbatim.
+- Re-running the same audit twice produces zero file writes on the
+  second run.
+
+## References
+
+- Schema: `TODO.new/02-audit-schema-design.md`
+- Contract: `TODO.new/04-fontist-org-contract.md`
+- Emitter impl: `TODO.new/13-directory-emitter.md`
+- Browser impl: `TODO.new/14-html-face-browser.md`
+- `Ucode::Repo::AtomicWrites` (existing pattern)

data/TODO.new/04-fontist-org-contract.md

@@ -0,0 +1,173 @@
+# 04 — fontist.org contract
+
+## Goal
+
+Pin the exact JSON contract `fontist.org` consumes. Both sides code
+against this doc. Any breaking change to the contract = minor version
+bump in ucode + a note here.
+
+## What fontist.org needs
+
+A coverage map per font shows, per Unicode block, how many codepoints
+the font covers vs how many are assigned. Renderer requirements:
+
+1. Face identity (name, foundry, version) — for the map's header.
+2. Per-block coverage stats — for the map's body.
+3. Per-block missing codepoints — for the "what's missing" drill-down.
+4. Plane rollup — for the map's overview band.
+5. Audit provenance — for the "data as of Unicode X.Y, generated at
+   <timestamp>" footer.
+
+fontist.org does **not** need:
+- Per-codepoint detail lists (verbose `codepoints/<NAME>.json`) — those
+  are for ucode's local browser.
+- Per-codepoint glyph SVGs (`glyphs/`) — fontist.org renders its own
+  glyphs from its own font copies.
+
+## Endpoint shape
+
+fontist.org fetches two URLs per audited font:
+
+### 1. `index.json` — the map
+
+Self-contained for the map view. Schema is the `AuditReport` shape
+minus the verbose-only fields, plus per-block `missing_codepoints`
+embedded directly.
+
+```json
+{
+  "generated_at": "2026-06-27T12:00:00Z",
+  "ucode_version": "0.2.0",
+  "baseline": {
+    "unicode_version": "17.0.0",
+    "ucode_version": "0.2.0",
+    "fontisan_version": "0.2.22",
+    "source": "ucd-text + Unicode17Blocks overrides",
+    "generated_at": "2026-06-27T12:00:00Z"
+  },
+  "font": {
+    "source_file": "Inter-Regular.ttf",
+    "source_sha256": "3b1a...",
+    "source_format": "ttf",
+    "font_index": null,
+    "num_fonts_in_source": 1,
+    "family_name": "Inter",
+    "subfamily_name": "Regular",
+    "full_name": "Inter Regular",
+    "postscript_name": "Inter-Regular",
+    "version": "Version 4.000;git-a52131595",
+    "font_revision": 4.0,
+    "weight_class": 400,
+    "width_class": 5,
+    "italic": false,
+    "bold": false,
+    "panose": "2 0 5 3 0 0 0 0 0 0",
+    "total_codepoints": 2857,
+    "total_glyphs": 1486,
+    "cmap_subtables": [4, 12, 14]
+  },
+  "totals": {
+    "assigned_codepoints_total": 150012,
+    "covered_codepoints_total": 2857,
+    "blocks_touched": 24,
+    "blocks_complete": 12,
+    "blocks_partial": 12,
+    "scripts_touched": 5,
+    "scripts_complete": 0
+  },
+  "plane_summaries": [
+    { "plane": 0, "blocks_total": 18, "assigned_total": 55000,
+      "covered_total": 2857, "coverage_percent": 5.19 },
+    ...
+  ],
+  "block_summaries": [
+    {
+      "name": "Basic Latin",
+      "first_cp": 0, "last_cp": 127, "plane": 0,
+      "total_assigned": 128, "covered_count": 128,
+      "missing_count": 0, "coverage_percent": 100.0,
+      "status": "COMPLETE",
+      "missing_codepoints": []
+    },
+    {
+      "name": "Greek and Coptic",
+      "first_cp": 880, "last_cp": 1023, "plane": 0,
+      "total_assigned": 135, "covered_count": 80,
+      "missing_count": 55, "coverage_percent": 59.26,
+      "status": "PARTIAL",
+      "missing_codepoints": [881, 883, 885, ...]
+    },
+    ...
+  ],
+  "script_summaries": [
+    { "script_code": "Latn", "script_name": "Latin",
+      "blocks_total": 4, "assigned_total": 1207,
+      "covered_total": 1307, "coverage_percent": 100.0,
+      "status": "COMPLETE" },
+    ...
+  ],
+  "discrepancies": [],
+  "warning": null
+}
+```
+
+### 2. `blocks/<NAME>.json` — on-demand block expansion
+
+If the renderer offers an "expand this block" interaction, it fetches
+the per-block file. Same shape as the entry in `block_summaries` — but
+already in its own file. Use this when iterating one block at a time
+without re-parsing `index.json`.
+
+```json
+{
+  "name": "CJK Unified Ideographs",
+  "first_cp": 19968, "last_cp": 40959, "plane": 0,
+  "total_assigned": 20992, "covered_count": 20950,
+  "missing_count": 42, "coverage_percent": 99.80,
+  "status": "PARTIAL",
+  "missing_codepoints": [19980, 19982, ...]
+}
+```
+
+## What fontist.org fetches but ignores
+
+These sections are in `index.json` but fontist.org does not render them.
+They exist for ucode's local browser and for archival consumers:
+
+- `font.italic`, `font.bold`, `font.panose`, `font.weight_class`,
+  `font.width_class` — style metadata.
+- `font.cmap_subtables` — internal parser provenance.
+- `font.total_glyphs` — distinct from `total_codepoints`.
+- `licensing`, `metrics`, `hinting`, `color_capabilities`, `variation`,
+  `opentype_layout` — full archival record fields. fontist.org may
+  surface these in a "details" tab; default behavior is to ignore.
+
+## Backwards-compatibility rules
+
+- **Field additions**: minor ucode version bump, no fontist.org change
+  required. Renderer ignores unknown fields.
+- **Field removals or renames**: major ucode version bump. Document in
+  this file with a "Migrating from X to Y" section. fontist.org must
+  update in lockstep.
+- **Status enum expansion** (e.g. adding a new value to
+  `block_summaries[].status`): minor bump. Renderer treats unknown
+  status as `PARTIAL`.
+
+## Acceptance
+
+- A fontist.org fetch of `index.json` is sufficient to render a
+  coverage map. No secondary fetch needed for the initial view.
+- A fontist.org fetch of `blocks/<NAME>.json` is sufficient to render
+  the per-block drill-down view.
+- Total payload for the initial view is under 500KB for fonts up to
+  ~30k codepoints; under 200KB for typical Latin-only fonts.
+- The contract is independently testable: a fixture `index.json` under
+  `spec/fixtures/audit/` exercises every documented field.
+
+## References
+
+- Schema source: `TODO.new/02-audit-schema-design.md`
+- Layout source: `TODO.new/03-directory-output-spec.md`
+- fontist.org repo: `/Users/mulgogi/src/fontist/fontist.org` (consumer)
+- Existing audit text renderer (fontisan):
+  `fontisan/lib/fontisan/formatters/audit_text_renderer.rb`