ucode 0.1.0 → 0.1.1

data/TODO.new/35-universal-set-production-run.md

@@ -0,0 +1,160 @@
+# 35 — Universal set production run + glyph provenance (Part 1 close)
+
+## Goal
+
+Run `ucode universal-set build 17.0.0` end-to-end with the curated
+coverage matrix (TODO 32) and acquired fonts (TODO 33). Produce one
+SVG glyph per assigned Unicode 17 codepoint (~299,382 files), plus
+a manifest tracking which Tier 1 source produced each glyph.
+
+Output goes under `output/universal_glyph_set/`:
+
+```
+output/universal_glyph_set/
+  manifest.json                # version, counts, generated_at
+  entries/
+    U+0041.json                # { cp, source: { kind, label, tier }, sha256, ... }
+    U+0042.json
+    ...
+  glyphs/
+    U+0041.svg
+    U+0042.svg
+    ...
+```
+
+The manifest is the **glyph provenance** record — for every glyph,
+which font (or pillar 2 PDF, or Last Resort) produced it. This is
+what fontist.org's char page surfaces as "this glyph came from
+NotoSerifTaiYo, OFL, version 1.0".
+
+## Why a separate TODO
+
+TODO 31 built the production infrastructure (`universal-set build` +
+`validate` commands). It hasn't actually run against a complete
+font set — every prior attempt failed at pre-check because fonts
+were missing.
+
+With TODO 32 (policy) and TODO 33 (acquisition) done, the production
+run becomes possible. This TODO is the integration test: does the
+end-to-end pipeline produce a complete, validated, provenance-tracked
+universal set?
+
+## Scope
+
+### Phase A — Pre-check green
+
+1. Run `ucode universal-set pre-check 17.0.0`. Must report zero
+   missing fonts. If anything's still missing, bounce back to
+   TODO 33.
+
+2. Capture the pre-check report as a fixture under
+   `spec/fixtures/universal_set/pre_check_17.json`. Future
+   regressions are caught by diffing against this.
+
+### Phase B — Full build
+
+3. Run `ucode universal-set build 17.0.0 --to=./output/universal_glyph_set`.
+   Expected duration: ~30–60 minutes for 299,382 glyphs (dependent
+   on font cache warmth and pillar 2 PDF rendering for blocks that
+   need it).
+
+4. Capture build metrics:
+   - Total codepoints processed
+   - Per-tier breakdown (Tier 1 / Pillar 1 / Pillar 2 / Pillar 3)
+   - Per-block coverage %
+   - Wall-clock time
+
+5. Validate: `ucode universal-set validate ./output/universal_glyph_set`.
+   Must pass every check:
+   - manifest_loadable
+   - glyph_files_present (every codepoint has an SVG)
+   - totals_reconcile (manifest counts match file counts)
+   - provenance_complete (every entry has a source)
+   - structural_yaml_valid
+
+### Phase C — Provenance surfacing
+
+6. Extend the manifest schema to include per-entry provenance that
+   fontist.org can render. Each `entries/U+XXXX.json` carries:
+
+   ```json
+   {
+     "cp": 65,
+     "id": "U+0041",
+     "block_id": "Basic_Latin",
+     "source": {
+       "tier": 1,
+       "kind": "fontist",
+       "label": "noto-sans",
+       "version": "...",
+       "license": "OFL"
+     },
+     "sha256": "...",
+     "extracted_at": "2026-..."
+   }
+   ```
+
+7. Pillar 2 entries carry:
+   ```json
+   {
+     "source": {
+       "tier": 2,
+       "kind": "pdf_charts",
+       "pdf_url": "https://www.unicode.org/charts/PDF/U1E6C0.pdf",
+       "pdf_sha256": "...",
+       "cid": 41
+     }
+   }
+   ```
+
+8. Pillar 3 entries (Last Resort tofu) carry:
+   ```json
+   {
+     "source": {
+       "tier": 3,
+       "kind": "last_resort",
+       "label": "Last Resort Font"
+     }
+   }
+   ```
+
+### Phase D — HTML browser
+
+9. Generate `output/universal_glyph_set/index.html` — a static page
+   summarizing the build:
+
+   - Top-level stats (total codepoints, per-tier pie chart)
+   - Per-block table with coverage % and a sample of glyphs
+   - Click a glyph → see full provenance
+
+   Reuse the existing audit browser generator (`ucode audit browser`)
+   pattern. Output is dev-server-friendly — no JS build required.
+
+### Phase E — Idempotency
+
+10. Re-run the build without changing inputs. Every file must be
+    byte-identical (mtime unchanged). The existing `Idempotency`
+    module handles this; just verify it holds end-to-end.
+
+11. Re-run after touching one font (re-fetch Lentariso at the same
+    version). Only that font's glyphs should rewrite.
+
+## Acceptance
+
+- [ ] `output/universal_glyph_set/` exists with `manifest.json` +
+      `entries/` + `glyphs/`
+- [ ] 299,382+ glyph SVGs present (one per assigned codepoint)
+- [ ] 0 pillar 3 fallbacks for blocks with known Tier 1 sources
+- [ ] `universal-set validate` exits 0
+- [ ] HTML browser renders locally with no JS errors
+- [ ] Re-running build is a byte-identical no-op
+- [ ] Provenance JSON for U+0041 (Tier 1 noto-sans) and U+1E6C0
+      (Tier 1 NotoSerifTaiYo) and at least one pillar 2 entry
+
+## References
+
+- [TODO 24](24-universal-glyph-set-build.md) — build infrastructure
+- [TODO 31](31-universal-set-production-build.md) — production design
+- [TODO 32](32-uc17-coverage-matrix.md) — input policy
+- [TODO 33](33-specialist-font-acquisition-refresh.md) — input fonts
+- [TODO 38](38-fontist-org-glyph-consumer.md) — consumer wiring

data/TODO.new/36-per-font-coverage-audit.md

@@ -0,0 +1,145 @@
+# 36 — Per-font coverage audit against universal set (Part 2 master)
+
+## Goal
+
+Given the universal glyph set (TODO 35) as the **reference baseline**,
+audit ANY font's cmap coverage against it. For each font in our
+library (or any user-supplied font), produce:
+
+- Per-block coverage % (e.g. "Lentariso covers 100% of Sidetic,
+  0% of Basic Latin")
+- Per-block gap list (codepoints the font misses)
+- Per-block extras (codepoints the font covers that aren't assigned
+  in Unicode 17 — rare but possible for old font versions)
+- Overall coverage score weighted by Unicode 17 assigned count
+
+This is **Part 2** of the user's directive: use the universal set
+to highlight missing glyphs in specific fonts. The output drives
+font selection decisions ("which font should fontist.org use for
+this block?") and surfaces fonts that claim Unicode X.Y support
+but actually have cmap gaps.
+
+## Why a separate TODO
+
+TODO 25 built the `CoverageReference` infrastructure (universal set
+as the comparison baseline). TODO 26 built the missing-glyph
+reporter. Neither has been RUN against real fonts because the
+universal set wasn't built (TODO 35 unblocks that).
+
+With TODO 35 done, this TODO is the actual audit: walk each font
+in our library, compare cmap to universal set, emit per-font
+coverage reports.
+
+## Scope
+
+### Phase A — Audit library command
+
+1. Extend `ucode audit font` to accept an optional
+   `--reference-universal-set=<path>` flag. When provided, the
+   audit includes a `coverage` section comparing the font's cmap
+   to the universal set's per-block codepoint lists.
+
+2. The audit output gains a new section:
+   ```json
+   {
+     "coverage": {
+       "per_block": [
+         {
+           "block_id": "Sidetic",
+           "range": [10940, 1097F],
+           "assigned_count": 26,
+           "covered_count": 26,
+           "missing": [],
+           "extras": []
+         },
+         ...
+       ],
+       "overall": {
+         "total_assigned": 299382,
+         "total_covered": 145233,
+         "percentage": 48.5
+       }
+     }
+   }
+   ```
+
+3. Extend `ucode audit library` (walks a directory of fonts) to
+   produce a per-font summary table. Sortable by overall %, by
+   per-block coverage, or by font family.
+
+### Phase B — Reference baseline extraction
+
+4. Build a fast-loading reference structure from the universal set:
+
+   ```
+   output/universal_glyph_set/reference/
+     by_block.json     # { block_id → [cp_int, ...] }
+     all_cps.bin       # sorted array of cp_int, for fast bsearch
+   ```
+
+   The audit loads this once and compares each font's cmap against
+   it. Avoids re-reading 299k individual entry JSONs.
+
+5. The reference is generated as part of TODO 35's build step. This
+   TODO consumes it.
+
+### Phase C — Per-font gap browser
+
+6. Extend the HTML face browser (`ucode audit browser`) to surface
+   coverage gaps visually. For each font:
+
+   - Per-block table with coverage %, color-coded (green ≥95%,
+     yellow 50–95%, red <50%)
+   - Click a block → see the actual missing glyphs as a grid
+     (showing the universal set's glyph SVG for each missing cp,
+     so the user can see what the font is missing)
+
+7. Library-level summary page:
+   - Top-N fonts by overall coverage
+   - Heatmap: font × block, cell color = coverage %
+   - "Best font per block" table (which font has the highest
+     coverage for each block)
+
+### Phase D — Coverage regression detection
+
+8. When a font is updated (re-installed via fontist, or new version
+   fetched), re-run the audit and DIFF against the prior run.
+   Surface:
+   - Newly-covered codepoints (good)
+   - Newly-missing codepoints (regression — flag for review)
+
+9. CI mode: in the ucode release workflow, re-audit the universal
+   set's Tier 1 fonts against the latest universal set. Any
+   coverage regression blocks the release.
+
+### Phase E — Public coverage dashboard
+
+10. The HTML library browser can be published to
+    `fontist.org/unicode/coverage/` so users can search "which
+    font covers Cyrillic Extended-D?" and get an answer.
+
+    This is the fontist.org consumer integration for coverage
+    data — pairs with TODO 38 (glyph consumer).
+
+## Acceptance
+
+- [ ] `ucode audit font <path> --reference-universal-set=...` emits
+      a `coverage` section with per-block + overall stats
+- [ ] `ucode audit library <dir>` walks every font and produces a
+      sortable summary
+- [ ] HTML face browser shows per-block coverage with click-through
+      to missing-glyph grids
+- [ ] Library browser has a heatmap view
+- [ ] At least 10 fonts audited end-to-end as a smoke test:
+      Lentariso, Kedebideri, NotoSerifTaiYo, FSung-1, Noto Sans
+      CJK JP, Noto Sans Symbols, Noto Sans Symbols 2, Symbola,
+      Noto Music, Last Resort Font
+
+## References
+
+- [TODO 25](25-font-audit-against-universal-set.md) — CoverageReference
+- [TODO 26](26-missing-glyph-reporter.md) — gap reporter
+- [TODO 35](35-universal-set-production-run.md) — universal set (input)
+- [TODO 37](37-coverage-highlight-reporter.md) — visualizer detail
+- [TODO 38](38-fontist-org-glyph-consumer.md) — public dashboard
+- [TODO 40](40-archive-private-uses-ucode-audit.md) — bin/build uses this audit in CI

data/TODO.new/37-coverage-highlight-reporter.md

@@ -0,0 +1,125 @@
+# 37 — Coverage highlight reporter (missing-glyph visualizer)
+
+## Goal
+
+A focused visualizer that takes ONE font + the universal set and
+produces a per-block "missing glyph grid" — every codepoint the
+font doesn't cover, rendered as the universal set's reference glyph
+so the user can see at a glance what's missing.
+
+Pairs with TODO 36 (the audit data layer). TODO 36 produces the
+JSON-shaped gap lists; this TODO is the human-facing visualizer.
+
+## Why a separate TODO
+
+TODO 26 built a missing-glyph reporter, but its output is a flat
+list of codepoint ids. For a font like Noto Sans CJK JP missing
+200 codepoints across 5 CJK extensions, a flat list is useless —
+you can't see the patterns. This TODO is the visual layer that
+makes the data actionable.
+
+The audience is font maintainers ("what's my font missing?") and
+fontist.org ("which font should we use for this block?"). Both
+need to see the actual glyphs, not hex strings.
+
+## Scope
+
+### Phase A — Per-font highlight page
+
+1. New command: `ucode audit highlight <font-path>` — produces
+   `output/audit/highlight/<font-slug>/index.html`.
+
+2. Page structure:
+   - Header: font name, version, license, overall coverage %
+   - Per-block sections, sorted by missing count (most-missing
+     first):
+     ```
+     Block: CJK Unified Ideographs Extension J (U+31350..U+323AF)
+     Missing: 4123 of 4298 codepoints (95.7% missing)
+
+     [Grid of missing glyphs, each cell showing:
+       - The reference glyph SVG (from universal set)
+       - Codepoint id (U+31450)
+       - Codepoint name (CJK UNIFIED IDEOGRAPH-31450)
+     ]
+     ```
+
+3. Grid cell click → drill to per-codepoint page with full UCD
+   metadata (reuses the existing UnicodeCharPage shape).
+
+### Phase B — Comparison view
+
+4. New command: `ucode audit compare <left-font> <right-font>` —
+   side-by-side coverage diff:
+   - Left covers, right misses (red, right side)
+   - Left misses, right covers (red, left side)
+   - Both cover (no entry)
+   - Both miss (gray)
+
+5. Use case: "FSung-1 vs Noto Sans CJK JP for CJK Ext J — which
+   should we use as Tier 1?"
+
+### Phase C — Library heatmap
+
+6. Library-level heatmap page. Rows = fonts, columns = blocks,
+   cell color = coverage %.
+
+7. Filter controls:
+   - Show only blocks with assigned_count > N
+   - Show only fonts with overall coverage > X%
+   - Sort by family / version / coverage
+
+8. Cell click → drill to per-block per-font detail.
+
+### Phase D — Embed reference glyphs efficiently
+
+9. The highlight page embeds reference glyphs as inline SVG (not
+   `<img>` referencing SVG files — that's 200k HTTP requests for
+   a full CJK page). Inline SVG with `<symbol>` definitions:
+
+   ```html
+   <svg style="display:none">
+     <defs>
+       <symbol id="U+4E00" viewBox="0 0 1000 1000">
+         <path d="..."/>
+       </symbol>
+       <symbol id="U+4E8C" viewBox="0 0 1000 1000">
+         <path d="..."/>
+       </symbol>
+     </defs>
+   </svg>
+   <svg viewBox="0 0 1000 1000"><use href="#U+4E00"/></svg>
+   ```
+
+10. For large blocks (CJK 20k+ glyphs), partition the page by
+    block-range subsets so the browser doesn't choke. Pagination
+    or lazy-load via IntersectionObserver.
+
+### Phase E — Diff mode for font versions
+
+11. `ucode audit diff <font-v1> <font-v2>` — for the same font
+    family across versions, surface:
+    - Codepoints added in v2 (good — coverage improved)
+    - Codepoints removed in v2 (regression — flag for review)
+
+    Useful for tracking Noto Sans releases across Unicode versions.
+
+## Acceptance
+
+- [ ] `ucode audit highlight <font>` produces an HTML page with
+      per-block missing-glyph grids
+- [ ] `ucode audit compare <left> <right>` produces a side-by-side
+      diff page
+- [ ] Library heatmap renders with no perf issues for ≤50 fonts ×
+      ~340 blocks
+- [ ] Reference glyphs inlined as `<symbol>` defs (no per-glyph
+      HTTP requests)
+- [ ] CJK-scale block (20k+ glyphs) paginates or lazy-loads
+- [ ] Cell clicks navigate to per-codepoint pages (existing
+      UnicodeCharPage)
+
+## References
+
+- [TODO 26](26-missing-glyph-reporter.md) — flat-list predecessor
+- [TODO 36](36-per-font-coverage-audit.md) — audit data layer
+- `lib/ucode/audit/browser.rb` — existing HTML browser generator

data/TODO.new/38-fontist-org-glyph-consumer.md

@@ -0,0 +1,141 @@
+# 38 — fontist.org glyph consumer + provenance display
+
+## Goal
+
+Wire fontist.org's `UnicodeCharPage.vue` to consume the universal
+glyph set (TODO 35) — replace the current system-font-fallback
+glyph rendering with the actual SVG ucode extracted. Surface the
+font source (provenance) next to the glyph so users see "this glyph
+came from NotoSerifTaiYo, OFL".
+
+This closes the loop: ucode extracts → fontist.org displays.
+
+## Why a separate TODO
+
+Today the char page renders glyphs via `displayChar(cp, charData.c)`
+— browser-side font resolution. That means:
+
+- Tai Yo / Sidetic / Beria Erfe / Egyptian Hieroglyphs show tofu
+  on most systems (no system font has them)
+- The "which font is this from?" question has no answer
+- ucode's universal glyph set isn't consumed anywhere
+
+With the universal set built (TODO 35), fontist.org can fetch
+`/unicode/glyph/U+XXXX.svg` and render the actual extracted outline.
+Provenance comes from the per-codepoint JSON the universal set
+already emits.
+
+## Scope
+
+### Phase A — Sync glyphs into fontist.org's public/
+
+1. Extend `scripts/fetch-data.sh` to copy `unicode/` from
+   `fontist-archive-public` (per TODO 41 — the artifacts land there
+   via ucode's publish workflow, NOT via direct raw.githubusercontent.com
+   fetch):
+
+   ```bash
+   log "copying unicode/block-feed/ + universal-glyph-set/"
+   mkdir -p "$PUBLIC/unicode"
+   if [[ -d "$TMP/archive/unicode/block-feed" ]]; then
+     cp -r "$TMP/archive/unicode/block-feed/." "$PUBLIC/unicode/"
+   fi
+   if [[ -d "$TMP/archive/unicode/universal-glyph-set" ]]; then
+     mkdir -p "$PUBLIC/unicode/glyphs"
+     cp -r "$TMP/archive/unicode/universal-glyph-set/glyphs/." "$PUBLIC/unicode/glyphs/"
+     cp "$TMP/archive/unicode/universal-glyph-set/manifest.json" "$PUBLIC/unicode/manifest.json"
+   fi
+   ```
+
+2. Scale check: 299,382 SVG files at ~1KB each = ~300MB. Too big for
+   the fontist.org repo but fine for the public archive. For LOCAL
+   dev: full copy is OK. For deployment: rsync to CDN target (not
+   committed to git).
+
+3. For per-codepoint JSONs (1.2 GB): add `--with-codepoints` flag
+   (default OFF). When ON, download + extract the Release asset
+   (per TODO 41 §Phase A.3) to `public/codepoints/`.
+
+### Phase B — Char page glyph rendering
+
+4. Update `UnicodeCharPage.vue`:
+   - Replace `displayChar(cp, charData.c)` with `<img :src="`/unicode/glyph/U+${hex}.svg`">`
+     when the SVG exists; fall back to `displayChar` for missing
+     glyphs (unassigned codepoints not in universal set)
+   - Add a "Source" caption: `<small>Glyph from {{ source.label }} ({{ source.license }})</small>`
+
+5. Fetch the per-codepoint JSON (already wired in current PR #45)
+   to get `source.label`, `source.license`, `source.tier`. Show
+   tier as a badge: "Tier 1" / "Pillar 2" / "Last Resort".
+
+### Phase C — Per-block glyph grid
+
+6. On the block page (`UnicodeBlockPage.vue`), the existing char
+   grid currently uses `displayChar` for each cell. Replace with
+   inline SVG references via `<symbol>` defs (one def per glyph
+   on the page, cells `<use>` it). Pattern from TODO 37.
+
+7. For CJK-scale blocks (20k+ glyphs), lazy-load on scroll. The
+   existing block grid already paginates; just swap the rendering.
+
+### Phase D — Provenance badge component
+
+8. New Vue component `GlyphSourceBadge.vue`:
+   ```vue
+   <template>
+     <span class="gsb" :class="`gsb-${tier}`">
+       <span class="gsb-tier">{{ tierLabel }}</span>
+       <span class="gsb-label">{{ source.label }}</span>
+       <span class="gsb-license" v-if="source.license">{{ source.license }}</span>
+     </span>
+   </template>
+   ```
+
+9. Color coding:
+   - Tier 1 (real font): green
+   - Pillar 1 (PDF + ToUnicode): blue
+   - Pillar 2 (PDF correlation): yellow
+   - Pillar 3 (Last Resort): gray
+
+### Phase E — Per-block coverage indicator
+
+10. On the block list page (`/unicode`), each block entry shows
+    coverage stats from the universal set:
+    - "4123/4298 codepoints covered by Tier 1 (Noto Sans CJK JP)"
+    - Color-coded bar: green = full Tier 1, yellow = mixed, red =
+      pillar 3 (tofu)
+
+11. Click the bar → drill to the per-block highlight page (TODO 37).
+
+### Phase F — Glyph detail page
+
+12. New route `/unicode/glyph/:hex` — dedicated glyph detail:
+    - Large SVG render with zoom/pan
+    - Full outline path data (collapsible `<pre>`)
+    - Provenance chain (font → cmap → GID → glyf outline → SVG)
+    - Comparison: this glyph vs other Tier 1 fonts covering same cp
+
+    Useful for font designers checking extraction quality.
+
+## Acceptance
+
+- [ ] `scripts/fetch-data.sh` copies `unicode/` from fontist-archive-public
+      (block-feed + universal-glyph-set; per-cp JSONs via optional flag)
+- [ ] `UnicodeCharPage.vue` renders the universal-set SVG (not
+      system fallback) for codepoints in the universal set
+- [ ] Provenance badge shows next to every glyph
+- [ ] Block grid renders glyphs via inline SVG `<symbol>` defs
+      (no per-glyph HTTP requests)
+- [ ] Block list page shows per-block Tier 1 coverage %
+- [ ] `/unicode/glyph/:hex` route exists with full provenance view
+- [ ] Tai Yo / Sidetic / Egyptian Hieroglyphs render real glyphs
+      (no tofu) when sourced from universal set
+
+## References
+
+- [TODO 27](27-fontist-org-consumer-integration.md) — original consumer TODO
+- [TODO 35](35-universal-set-production-run.md) — universal set (input)
+- [TODO 37](37-coverage-highlight-reporter.md) — visualizer patterns
+- [TODO 41](41-ucode-unicode-archive-bridge.md) — publishing pipeline
+- `src/pages/UnicodeCharPage.vue` — current char page
+- `src/pages/UnicodeBlockPage.vue` — current block page