ucode 0.1.0 → 0.1.1

data/TODO.new/26-missing-glyph-reporter.md

@@ -0,0 +1,189 @@
+# 26 — Missing glyph reporter (drill-down view)
+
+## Goal
+
+Per-font drill-down view that renders the universal-set glyph SVG next
+to every missing codepoint. Turns "this font is missing U+10980" into
+"this font is missing U+10980, here's what it looks like."
+
+This is Part 3 of the user's three-part directive. Without it, the
+audit report (TODO 25) is a list of integers; with it, the user sees
+the actual glyph shape they're missing.
+
+## Why a separate TODO
+
+TODO 14 (HTML face browser) shows missing codepoints as chips. TODO 26
+adds a glyph rendering mode: each chip loads the universal-set SVG for
+that codepoint and shows it inline.
+
+Two different concerns:
+
+- TODO 14: the browser shell, navigation, sortable tables, plane band.
+- TODO 26: the glyph-rendering drill-down that the browser shell calls
+  into when the user clicks a chip.
+
+Building TODO 26 on top of TODO 14 keeps the glyph rendering isolated
+and reviewable.
+
+## Files to create / change
+
+- `lib/ucode/audit/browser/glyph_panel.rb` — new component: given a
+  codepoint + universal-set manifest path, returns the inline SVG
+  markup for the panel.
+- `lib/ucode/audit/browser/templates/glyph_panel.html.erb` — the panel
+  template (used both for inline expansion and standalone mode below).
+- `lib/ucode/audit/browser/missing_glyph_page.rb` — optional standalone
+  per-block "missing glyphs gallery" page: emits
+  `output/font_audit/<label>/missing/<BLOCK>.html` per touched block.
+- `lib/ucode/audit/browser/templates/missing_glyph_page.html.erb`.
+- `lib/ucode/audit/browser/face_page.rb` — update to accept a
+  `universal_set_root:` kwarg; when present, JS hooks become
+  glyph-aware.
+- `lib/ucode/audit/browser/templates/face.js` — add glyph-panel logic.
+- Specs:
+  - `spec/ucode/audit/browser/glyph_panel_spec.rb`
+  - `spec/ucode/audit/browser/missing_glyph_page_spec.rb`
+  - update `face_page_spec.rb` to cover the glyph-aware mode.
+
+## Glyph panel shape
+
+When the user clicks a codepoint chip (e.g. U+037D in Greek and Coptic
+block, marked missing), the panel expands inline:
+
+```
+┌──────────────────────────────────────────────────────┐
+│ U+037D  GREEK SMALL LETTER PAMPHYLIAN DIGAMMA        │
+│                                                      │
+│   [SVG:  ]                                           │
+│   [  SVG  ]   ← universal-set glyph rendered inline  │
+│   [     ]                                             │
+│                                                      │
+│ Source: tier-1:noto-sans                             │
+│ Unicode block: Greek and Coptic                      │
+│ Age: Unicode 5.1 (March 2008)                        │
+│ General category: Ll (Lowercase Letter)              │
+│                                                      │
+│ This font is missing this codepoint.                 │
+│ Universal glyph shown for reference.                 │
+└──────────────────────────────────────────────────────┘
+```
+
+The SVG comes from the universal-set directory
+(`output/universal_glyph_set/glyphs/U+037D.svg`), resolved via the
+manifest's per-codepoint entry.
+
+## Standalone missing-glyph gallery
+
+The standalone page (`missing_glyph_page.html.erb`) emits one HTML
+file per touched block, listing every missing codepoint in that block
+as a grid of glyph thumbnails:
+
+```
+output/font_audit/<label>/missing/
+├── Greek_and_Coptic.html       # ~55 missing glyphs as a grid
+├── Sidetic.html                # ~26 missing glyphs
+└── CJK_Unified_Ideographs.html # potentially thousands
+```
+
+The gallery is a static page — no JS needed. Each thumbnail links to
+the chip in the main `index.html` (so users can jump back to context).
+
+This page is what fontist.org can iframe or screenshot for the "what's
+missing" widget.
+
+## JS behavior (face.js additions)
+
+```js
+// when a codepoint chip is clicked:
+async function expandCodepoint(chip, codepoint) {
+  const panel = await renderPanel(codepoint);
+  chip.insertAdjacentElement('afterend', panel);
+}
+
+async function renderPanel(codepoint) {
+  // 1. fetch codepoints/<BLOCK>.json → get name, gc, age, etc.
+  // 2. fetch ../../../universal_glyph_set/glyphs/<U+XXXX>.svg
+  //    (path resolved from manifest field)
+  // 3. fetch ../../../universal_glyph_set/manifest.json → get source
+  // 4. build panel DOM, return
+}
+```
+
+If the universal set is not co-located with the audit output (e.g. the
+audit was generated on a different machine), the page shows a "glyph
+preview not available" message instead of the SVG. The page itself
+remains functional.
+
+## Universal-set path resolution
+
+`face_page.rb` records the universal-set path in the inlined JSON
+overview:
+
+```json
+{
+  ...
+  "universal_set": {
+    "available": true,
+    "manifest_path": "../../../universal_glyph_set/manifest.json",
+    "glyphs_dir": "../../../universal_glyph_set/glyphs/"
+  }
+}
+```
+
+The JS uses these paths at runtime. When `available: false`, the JS
+skips the SVG fetch and shows the text-only panel.
+
+## Standalone page generation
+
+The standalone missing-glyph page is opt-in:
+
+```bash
+bin/ucode audit font <path> --with-missing-glyph-pages
+```
+
+This flag implies `--with-glyphs` (TODO 14's verbose flag) and
+requires a universal-set manifest to be present. Output goes under
+`output/font_audit/<label>/missing/<BLOCK>.html` per touched block.
+
+## Performance considerations
+
+- A CJK font can be missing thousands of codepoints. The grid is
+  paginated client-side (50 per page); the static HTML emits only the
+  first page; subsequent pages are loaded via fetch from a parallel
+  JSON file (`missing/<BLOCK>.json`).
+- The SVG fetch is cached per session (Map); clicking the same
+  codepoint twice doesn't re-fetch.
+- For very large blocks (CJK), the standalone page emits at most 500
+  thumbnails; the rest are available via the JSON file.
+
+## Acceptance
+
+- Clicking a missing-codepoint chip on the face page opens a panel
+  that renders the universal-set glyph SVG inline.
+- The panel shows: codepoint id, name, age, gc, block, universal-set
+  source provenance, "this font is missing this codepoint" notice.
+- `--with-missing-glyph-pages` produces per-block standalone HTML
+  files at `output/font_audit/<label>/missing/<BLOCK>.html`.
+- When the universal set is not co-located, the panel shows a
+  text-only fallback without errors.
+- The standalone gallery page is self-contained (inlined CSS/JS);
+  works via `file://`.
+- Specs cover: panel rendering with SVG, panel rendering without SVG,
+  standalone gallery generation, pagination.
+- Rubocop clean.
+
+## Out of scope
+
+- The face browser shell itself — TODO 14.
+- The library browser (cross-font view) — TODO 15.
+- fontist.org consumer side — TODO 27.
+- The universal-set build — TODO 24.
+
+## References
+
+- Universal set build: `TODO.new/24-universal-glyph-set-build.md`
+- Font audit against universal set: `TODO.new/25-font-audit-against-universal-set.md`
+- HTML face browser: `TODO.new/14-html-face-browser.md`
+- fontist.org contract: `TODO.new/04-fontist-org-contract.md`
+- Existing face browser: `lib/ucode/audit/browser/face_page.rb`
+  (post-TODO 14)

data/TODO.new/27-fontist-org-consumer-integration.md

@@ -0,0 +1,200 @@
+# 27 — fontist.org consumer integration
+
+## Goal
+
+Wire fontist.org (the consumer side) to ingest ucode's per-font audit
+JSON + universal-set glyph references. Replaces the current
+fontisan-YAML consumer with the new ucode-JSON consumer defined in
+TODO 04.
+
+This is the "fully integrate with fontist.org" directive. Two repos
+are touched:
+
+- `fontist/ucode` (this repo) — produces the artifacts.
+- `fontist/fontist.org` (consumer) — fetches and renders them.
+
+The contract is locked in TODO 04; this TODO implements the consumer
+side and any producer-side emitter gaps the consumer surfaces.
+
+## What exists today (consumer side)
+
+fontist.org currently consumes (per
+`fontist.org/CLAUDE.md` and `fontist.org/coverage-architecture.md`):
+
+- `coverage/{formula_slug}/{PostScriptName}.yaml` — fontisan audit YAML.
+- `woff/{formula_slug}/{PostScriptName}.woff` — WOFF specimens.
+- `fonts.json`, `font-metadata.json` — fonts registry.
+- `unicode/blocks/*.json` — Unicode block reference data.
+- `unicode/indexes/*` — Unicode property indexes.
+
+The audit YAML is the fontisan AuditCommand output (legacy). The
+shape is documented in `fontist.org/coverage-architecture.md` §"Audit
+YAML Schema."
+
+## What changes
+
+fontist.org gains a parallel data feed for ucode audits:
+
+- `audit/{formula_slug}/{PostScriptName}/index.json` — ucode per-face
+  AuditReport (TODO 04 contract).
+- `audit/{formula_slug}/{PostScriptName}/blocks/<NAME>.json` — per-block
+  chunk.
+- `audit/{formula_slug}/{PostScriptName}/missing/<BLOCK>.html` —
+  optional, missing-glyph gallery (TODO 26).
+- `universal_glyph_set/manifest.json` — the universal-set manifest
+  (TODO 24), single global file.
+- `universal_glyph_set/glyphs/<U+XXXX>.svg` — universal-set glyphs.
+
+The legacy `coverage/` feed stays (fontisan still produces it during
+the migration window). fontist.org's renderer switches to `audit/`
+when present; falls back to `coverage/` when not.
+
+## fontist.org consumer work
+
+### Files to change in `fontist/fontist.org`
+
+- `scripts/fetch-data.sh` — add fetch of `audit/` from the archive;
+  add fetch of `universal_glyph_set/` (single zip, ~50 MB).
+- `src/lib/fonts/loader.ts` — load the new audit JSON shape; keep
+  legacy YAML loader as fallback.
+- `src/lib/unicode/data/loader.ts` — load the universal-set manifest;
+  expose `getUniversalGlyph(codepoint)` API.
+- `src/composables/useCoverage.ts` — switch from legacy YAML parsing
+  to the new JSON shape; preserve the existing API for component
+  compatibility.
+- `src/pages/FontBlockPage.vue` — render missing codepoints using the
+  universal-set glyphs (replace the current text-only chips with SVG
+  thumbnails when universal set is loaded).
+- `src/pages/FontDetailPage.vue` — use the new `index.json` shape;
+  surface tier breakdown ("X glyphs from Tier 1, Y from Pillar 3") in
+  the audit footer.
+- `src/components/UnicodeBrowser/` — add a `MissingGlyphGrid.vue`
+  component for the drill-down view.
+- `tests/audit-json-shape.test.ts` — independent contract test for
+  the new JSON shape (mirrors `spec/fixtures/audit/` in ucode).
+
+### Migration strategy
+
+1. **Phase A — parallel feed.** fontist.org fetches both `coverage/`
+   and `audit/`. Renderer uses `audit/` when present (a per-formula
+   flag controls rollout). This de-risks the migration.
+2. **Phase B — audit default.** Once all formulas have audit JSON,
+   flip the default. Legacy `coverage/` becomes backup-only.
+3. **Phase C — coverage decommission.** Stop fetching `coverage/`
+   when fontisan audit subsystem is removed (TODO 17-19).
+
+### SSG route additions
+
+fontist.org generates static HTML at build time. Per TODO 04, the
+audit is fetched per-font on the client (no per-codepoint static
+HTML). SSG routes:
+
+- `/fonts/<slug>/coverage/` — uses `audit/<slug>/<ps>/index.json`
+- `/fonts/<slug>/coverage/<ps>/` — per-face detail
+- `/fonts/<slug>/coverage/<ps>/<block-slug>/` — per-block drill-down
+  with missing-glyph grid
+
+~13,200 existing routes grow by ~3,000 audit routes. SSG build time
+budget: under 30 minutes (current: ~25 minutes).
+
+### Universal-set global artifact
+
+The universal-set directory is fetched once and shared across all
+font pages. It's served from `/universal-glyph/<U+XXXX>.svg` (no
+formula slug in the URL — it's not per-font).
+
+For SSG: don't pre-render every glyph; let the browser fetch on
+demand. The manifest is committed to fontist.org as
+`public/universal-glyph-manifest.json` (~5 MB).
+
+## ucode producer work
+
+The producer side is mostly TODO 04 + TODO 13 + TODO 24. What's left:
+
+### Audit JSON emitter gap
+
+TODO 13's emitter writes the per-face directory tree. We need a
+top-level emitter that walks a library and produces:
+
+```
+output/font_audit_release/
+├── audit/<formula_slug>/<ps>/...     # one face directory per TODO 03 layout
+├── universal_glyph_set/...           # the universal set (TODO 24)
+├── library.json                      # all formulas + faces index
+└── manifest.json                     # release manifest (versions, sha256s)
+```
+
+This is the artifact `fontist.org/scripts/fetch-data.sh` consumes.
+
+### Files to create in ucode
+
+- `lib/ucode/audit/release_emitter.rb` — walks a library audit, emits
+  the release tree.
+- `lib/ucode/commands/release.rb` — CLI: `bin/ucode release` produces
+  the release tree.
+- `spec/ucode/audit/release_emitter_spec.rb`
+
+### CI / publishing
+
+A new GHA workflow in `fontist-archive-private`:
+
+1. Matrix-audit every formula (existing pipeline).
+2. After all matrix jobs complete, a collector job:
+   - Downloads all per-formula audit outputs.
+   - Runs `bin/ucode release` to assemble the release tree.
+   - Uploads as a GitHub release artifact tagged
+     `audit-<unicode-version>-<date>`.
+3. `fontist.org/scripts/fetch-data.sh` fetches the latest such tag.
+
+## Acceptance
+
+### Producer (ucode)
+
+- `bin/ucode release` produces the release tree at
+  `output/font_audit_release/`.
+- The release tree contains all per-face audits + the universal-set +
+  a library.json + manifest.json.
+- manifest.json records: ucode version, unicode version, source-config
+  sha256, per-face count, per-block glyph count from universal set.
+- One smoke spec runs `release` against a small fixture library.
+
+### Consumer (fontist.org)
+
+- `scripts/fetch-data.sh` fetches `audit/` and `universal_glyph_set/`
+  without errors.
+- A test font page (e.g. `/fonts/manual/inter/coverage/Inter-Regular/`)
+  renders using the new JSON shape.
+- The block drill-down view shows missing glyphs as universal-set SVG
+  thumbnails.
+- The footer shows tier breakdown ("X glyphs via Tier 1, Y via Pillar
+  3") from the audit baseline.
+- Legacy `coverage/` fallback works when `audit/` is absent.
+- SSG build time stays under 30 minutes.
+- Contract test (`tests/audit-json-shape.test.ts`) passes against the
+  fixture from `ucode/spec/fixtures/audit/`.
+
+### Cross-repo
+
+- ucode's `spec/fixtures/audit/index.json` matches what fontist.org's
+  contract test expects.
+- fontist.org's coverage map renders correctly with both shape
+  versions for the migration window.
+
+## Out of scope
+
+- The producer-side audit subsystem itself — TODOs 06-16.
+- The universal-set build — TODO 24.
+- fontisan decommission — TODOs 17-19.
+- fontist.org's existing Unicode browser (per-codepoint detail) —
+  unchanged; it uses the legacy `unicode/blocks/*.json` data.
+
+## References
+
+- fontist.org contract: `TODO.new/04-fontist-org-contract.md`
+- Directory emitter: `TODO.new/13-directory-emitter.md`
+- HTML face browser: `TODO.new/14-html-face-browser.md`
+- Missing glyph reporter: `TODO.new/26-missing-glyph-reporter.md`
+- Universal-set build: `TODO.new/24-universal-glyph-set-build.md`
+- fontist.org consumer repo: `/Users/mulgogi/src/fontist/fontist.org`
+- fontist.org architecture: `fontist.org/CLAUDE.md`
+- Coverage architecture: `fontist.org/coverage-architecture.md`

data/TODO.new/28-implementation-order-update.md

@@ -0,0 +1,187 @@
+# 28 — Implementation order (universal set + audit + fontist.org)
+
+## Goal
+
+Update the canonical implementation order to include TODOs 23-31
+(universal glyph set, font audit against universal set, missing glyph
+reporter, fontist.org consumer integration, full Unicode 17 curation,
+specialist font acquisition, production build + validation). This
+file replaces TODO.new/22 as the authoritative sequencing reference
+for the new work; TODO 22 continues to govern TODOs 01-21.
+
+## Sequencing principles (carried from TODO 22)
+
+- **Schema and contract first.** Lock data shapes before porting code.
+- **Measure before optimizing.** Baseline audit (TODO 05) informs the
+  source config (TODO 23).
+- **One PR per TODO** unless tightly coupled.
+- **Migration order: port → wire → cleanup.**
+
+## Dependency graph (additions in bold)
+
+```
+01 pillar-terminology-alignment
+02 audit-schema-design
+03 directory-output-spec
+04 fontist-org-contract
+05 baseline-unicode17-coverage-audit ───┐
+                                        │
+06-16 audit migration track              │
+                                        │
+17-19 fontisan cleanup                   │
+                                        │
+20 canonical-resolver-4-tier ────────────┤
+                                        │
+21 canonical-unicode17-build            │
+                                        │
+**23 universal-glyph-set-source-map** ───┤
+                                        │
+**24 universal-glyph-set-build** ────────┤
+                                        │
+**25 font-audit-against-universal-set** ─┤  (shipped — PR #34)
+                                        │
+**29 universal-set-curation-uc17** ──────┤  (fills TODO 23's YAML)
+                                        │
+**30 tier1-font-acquisition** ───────────┤  (downloads specialist fonts)
+                                        │
+**31 universal-set-production-build** ───┤  (executes TODO 24 end-to-end)
+                                        │
+**26 missing-glyph-reporter** ───────────┤
+                                        │
+**27 fontist-org-consumer-integration** ─┘
+```
+
+## Phased rollout
+
+### Phase 1 — Audit migration (TODOs 01-16)
+
+Build the audit subsystem in ucode. Mode 2 (per-font audit) lands.
+fontist.org continues consuming legacy `coverage/` YAML during this
+phase.
+
+Status (as of this writing): PRs through TODO 12 merged; TODOs 13-16
+in flight.
+
+### Phase 2 — Canonical dataset (TODOs 20-21)
+
+Build the resolver (TODO 20) and run the canonical Unicode 17 build
+(TODO 21). Mode 1 (per-codepoint UCD dataset) lands. This work is
+independent of Phase 1; can run in parallel.
+
+### Phase 3 — Universal glyph set (TODOs 23-24) **new**
+
+Curate the per-block Tier 1 source map (TODO 23) from the baseline
+audit (TODO 05). Build the universal set as a standalone artifact
+(TODO 24).
+
+Dependencies:
+- TODO 05 must be complete (cmap-verified font recommendations).
+- TODO 20 must be merged (resolver mechanics).
+- TODO 21 must have produced a baseline run (validates the resolver).
+
+Output: `output/universal_glyph_set/` artifact.
+
+### Phase 4 — Universal-set-driven audit (TODOs 25-26) **new**
+
+Replace the cmap-vs-UCD audit (current) with cmap-vs-universal-set
+(TODO 25). Add the missing-glyph drill-down view (TODO 26).
+
+Dependencies:
+- TODO 24 must be complete (universal-set manifest exists).
+- TODO 13 must be merged (directory emitter, the wire format).
+- TODO 14 must be merged (face browser, the host of TODO 26 panels).
+
+Output: per-font audits carry provenance; missing-glyph galleries
+ship alongside.
+
+Status: TODO 25 shipped as PR #34.
+
+### Phase 4b — UC17 curation + production build (TODOs 29-31) **new**
+
+Fill the source config with concrete Tier 1 font recommendations for
+all ~340 Unicode 17 blocks (TODO 29). Acquire specialist fonts not in
+fontist's index (TODO 30). Execute the universal-set build end-to-end
+and validate coverage (TODO 31).
+
+Dependencies:
+- TODO 23 must be merged (source map mechanism — already shipped).
+- TODO 24 must be merged (build pipeline — already shipped).
+- TODO 05 baseline audit (font investigation analysis, encoded into
+  TODO 29).
+
+Output: populated `config/unicode17_universal_glyph_set.yml`,
+`data/fonts/<specialist>.ttf`, complete
+`output/universal_glyph_set/` artifact with 100% Tier 1 coverage
+outside documented residual gaps.
+
+### Phase 5 — fontist.org consumer (TODO 27) **new**
+
+Wire fontist.org to consume the new audit JSON + universal-set
+glyphs. Parallel-feed alongside legacy `coverage/` during migration.
+
+Dependencies:
+- TODO 04 contract locked (already done).
+- TODO 24 universal-set shipped.
+- TODO 25 audit JSON shape stable.
+- TODO 26 missing-glyph galleries available.
+- TODO 31 production build validates (universal-set is complete and
+  auditable).
+
+Output: fontist.org renders new audit data; legacy feed becomes
+backup-only.
+
+### Phase 6 — fontisan decommission (TODOs 17-19)
+
+Once fontist.org has fully migrated to the new feed, remove the
+audit subsystem from fontisan. Out of scope for this doc; tracked in
+TODO 22.
+
+## Cross-cutting concerns
+
+### Source-config stability
+
+TODO 23's YAML is the canonical Tier 1 font map. Any change there
+triggers:
+- TODO 24 rebuild (universal-set manifest delta).
+- TODO 25 re-audit (per-block coverage may shift).
+- TODO 27 re-release (fontist.org fetches new manifest).
+
+Recommendation: bump `ucode_version` field on every config edit;
+consumers detect drift via that field.
+
+### Performance
+
+- TODO 24 build: target under 4 hours for full Unicode 17.
+- TODO 25 re-audit: per-font is independent; can parallelize across CI
+  matrix (one job per formula).
+- TODO 27 SSG build: target under 30 minutes (current ~25).
+
+### Backwards compatibility
+
+- TODO 25 audit JSON: additive field (`missing_codepoint_provenance`).
+  Old consumers ignore the new field.
+- TODO 27 consumer: parallel-feed during migration; legacy fallback
+  when audit JSON is absent.
+
+## Acceptance
+
+- All Phase 3-5 TODOs ship as one PR each.
+- The sequencing above is reflected in the actual PR stack on GitHub.
+- TODO 22 is updated to cross-reference this file.
+
+## Out of scope
+
+- Anything already covered by TODO 22 (TODOs 01-21).
+- fontisan decommission (Phase 6, TODO 22).
+
+## References
+
+- Predecessor: `TODO.new/22-implementation-order.md`
+- Universal-set source map: `TODO.new/23-universal-glyph-set-source-map.md`
+- Universal-set build: `TODO.new/24-universal-glyph-set-build.md`
+- Font audit against universal set: `TODO.new/25-font-audit-against-universal-set.md`
+- Missing glyph reporter: `TODO.new/26-missing-glyph-reporter.md`
+- fontist.org consumer integration: `TODO.new/27-fontist-org-consumer-integration.md`
+- UC17 curation (Part 1): `TODO.new/29-universal-set-curation-uc17.md`
+- Tier 1 font acquisition: `TODO.new/30-tier1-font-acquisition.md`
+- Production build + validation: `TODO.new/31-universal-set-production-build.md`