ucode 0.3.3 → 0.4.0

data/TODO.new/39-implementation-order-update-32-38.md

@@ -1,258 +0,0 @@
-# 39 — Implementation order update (TODOs 32–38)
-
-## Goal
-
-Sequence the remaining work for Part 1 (universal glyph set with
-full UC17 coverage) and Part 2 (per-font audit + highlight) so
-each PR is independently reviewable and the critical path is
-short.
-
-Extends [TODO 28](28-implementation-order-update.md) which sequenced
-TODOs 23–31.
-
-# 39 — Implementation order update (TODOs 32–41)
-
-## Goal
-
-Sequence the remaining work for Part 1 (universal glyph set with
-full UC17 coverage) and Part 2 (per-font audit + highlight) plus
-the pipeline wiring (TODOs 40–41) so each PR is independently
-reviewable and the critical path is short.
-
-Extends [TODO 28](28-implementation-order-update.md) which sequenced
-TODOs 23–31.
-
-## Critical path
-
-```
-                    ┌─────────────────────┐
-                    │  32 Coverage matrix │  ← policy only; no deps
-                    └──────────┬──────────┘
-                               │
-              ┌────────────────┼────────────────┐
-              ▼                                  ▼
-   ┌─────────────────────┐           ┌──────────────────────┐
-   │  33 Font acquisition │           │  34 Pillar 2         │  ← parallel
-   │  (URLs + formulas)  │           │  ContentStreamCorr.  │
-   └──────────┬──────────┘           └──────────┬───────────┘
-              │                                 │
-              └────────────┬────────────────────┘
-                           ▼
-              ┌─────────────────────────┐
-              │  35 Production run       │  ← end of Part 1
-              │  (universal set build)  │
-              └────────────┬────────────┘
-                           │
-              ┌────────────┴────────────┐
-              ▼                          ▼
-   ┌──────────────────────┐    ┌──────────────────────┐
-   │ 41 Unicode artifacts  │    │ 36 Per-font          │
-   │ → archive-public      │    │   coverage audit     │
-   │ bridge                │    └──────────┬───────────┘
-   └──────────┬───────────┘               │
-              │                            ▼
-              │                ┌──────────────────────┐
-              │                │ 37 Highlight reporter │
-              │                └──────────┬───────────┘
-              ▼                           │
-   ┌──────────────────────┐                │
-   │ 38 fontist.org glyph │←───────────────┘
-   │   consumer           │
-   └──────────────────────┘
-
-   ┌──────────────────────────────────────┐
-   │ 40 fontist-archive-private           │  ← can start any time;
-   │   bin/build uses ucode audit         │     independent of 32–38
-   └──────────────────────────────────────┘
-```
-
-## Phase 1 — Policy + acquisition (sequential, blocking)
-
-### Track A1 — Coverage matrix (TODO 32)
-
-**Branch**: `audit/coverage-matrix`
-**PR**: ucode/PR-XX
-
-YAML-only change to `config/unicode17_universal_glyph_set.yml`.
-Extends `Models::GlyphSourceMap` to accept `default_sources` at top
-level. Adds per-block specialists with full provenance/rationale.
-
-Acceptance: every block has a defined Tier 1 (or pillar 2 fallback
-policy). Reviewer can sign off without font availability.
-
-**Estimated**: 1–2 sessions. Mostly research + YAML writing.
-
-### Track A2 — Font acquisition refresh (TODO 33)
-
-**Branch**: `audit/font-acquisition-refresh`
-**PR**: ucode/PR-XX + 3+ fontist/formulas PRs
-
-Depends on A1 (uses the curated specialist list). Two halves:
-
-- **A2a — Direct URL fixes**: Lentariso, EgyptianText,
-  UniHieroglyphica, BabelStone, Symbola. Update
-  `specialist_fonts.yml`, verify sha256.
-- **A2b — fontist formula PRs**: open upstream PRs for Noto Sans
-  CJK JP, Noto Sans Symbols, Noto Sans Symbols 2, Noto Music,
-  Noto Sans Sharada, Noto Sans Sidetic, Noto Sans Tolong Siki,
-  Noto Sans Tangut, Noto Sans Arabic, Noto Sans Telugu, Noto Sans
-  Kannada.
-
-A2b blocks on external review (fontist maintainers). Until merged,
-ucode falls back to direct notofonts.github.io URLs (Phase C of
-TODO 33).
-
-**Estimated**: 2–3 sessions for A2a; A2b is async (external PRs).
-
-### Track A3 — Pillar 2 ContentStreamCorrelator (TODO 34)
-
-**Branch**: `audit/pillar2-correlator`
-**PR**: ucode/PR-XX
-
-Parallel to A1/A2 — no upstream deps. Generalizes `/tmp/correlate_v4.rb`
-into `Ucode::Glyphs::EmbeddedFonts::ContentStreamCorrelator`. Patches
-`Catalog#build_entry` to delegate when `tu_ref` is nil.
-
-**Estimated**: 2 sessions. Algorithm is proven; needs generalization
-+ tests on Sidetic and Beria Erfe PDFs.
-
-## Phase 2 — Production build (sequential, blocked by Phase 1)
-
-### Track B1 — Universal set production run (TODO 35)
-
-**Branch**: `audit/universal-set-production`
-**PR**: ucode/PR-XX (manifest + sample of glyphs as fixtures; full
-set is too big for git)
-
-Blocked by A1, A2, A3 (need fonts + pillar 2 fallback). Runs
-`ucode universal-set build 17.0.0` end-to-end. Emits manifest,
-entries, glyphs, HTML browser.
-
-**Estimated**: 1 session to run + validate + write summary doc.
-Wall-clock for build itself: 30–60 minutes.
-
-## Phase 3 — Pipeline wiring (parallel after Phase 2)
-
-### Track B2 — fontist-archive-private bin/build refactor (TODO 40)
-
-**Branch**: `audit/archive-private-uses-ucode` (in fontist-archive-private repo)
-**PR**: fontist-archive-private/PR-XX
-
-Independent of Phase 1/2 — can start any time. Swaps
-`Fontisan::Commands::AuditCommand` for `ucode audit font`, removes
-the UCD stub hack. After TODO 35 lands, adds the
-`--reference-universal-set` flag so audits include coverage
-comparison against the canonical glyphs.
-
-**Estimated**: 1 session for Phase A (swap invocation + remove stub).
-Phase B (universal-set reference) is a follow-up after TODO 41.
-
-### Track B3 — ucode Unicode artifacts → archive bridge (TODO 41)
-
-**Branch**: `audit/unicode-archive-bridge` (in ucode repo + fontist-archive-public repo)
-**PR**: ucode/PR-XX + fontist-archive-public/PR-XX
-
-Blocked by B1 (universal set must exist to be bridged). Adds the
-publish workflow in ucode, the `unicode/` directory in archive-public,
-and the fetch-data.sh updates in fontist.org.
-
-**Estimated**: 2 sessions. Workflow + sync scripts + verification.
-
-## Phase 4 — Consumer wiring (parallel after Phase 3)
-
-### Track C1 — Per-font coverage audit (TODO 36)
-
-**Branch**: `audit/per-font-coverage`
-**PR**: ucode/PR-XX
-
-Blocked by B1 (universal set is the reference). Extends
-`ucode audit font/library` with coverage section. Outputs JSON
-per-font/per-block coverage stats. Once B2 lands, this audit is
-also what `fontist-archive-private/bin/build` produces per formula.
-
-**Estimated**: 2 sessions.
-
-### Track C2 — Coverage highlight reporter (TODO 37)
-
-**Branch**: `audit/highlight-reporter`
-**PR**: ucode/PR-XX
-
-Blocked by C1 (consumes audit data). HTML visualizer with per-block
-missing-glyph grids, comparison view, library heatmap.
-
-**Estimated**: 2–3 sessions.
-
-### Track C3 — fontist.org glyph consumer (TODO 38)
-
-**Branch**: `feat/fontist-org-glyph-consumer` (in fontist/fontist.github.io repo)
-**PR**: fontist.github.io/PR-XX
-
-Blocked by B3 (universal set must be in fontist-archive-public
-under `unicode/`). Independent of C1/C2 (different consumer). Wires
-`UnicodeCharPage.vue` to render universal-set SVGs + provenance
-badge.
-
-**Estimated**: 2 sessions.
-
-## Sequencing rules
-
-1. **PR-per-TODO.** No bundled PRs unless tightly coupled (e.g.,
-   A2a + A2b could land together if A2b's fontist PRs are still
-   in review).
-
-2. **A3 + B2 can run in parallel with Phase 1/2.** Both are pure
-   code work and don't touch the curated config.
-
-3. **C1/C2/C3 all depend on B1 or B3 but not on each other.** They
-   can land in any order once their dependency merges.
-
-4. **External PRs (fontist/formulas) don't block ucode progress.**
-   Until they merge, ucode uses direct URLs as fallback. Once they
-   merge, ucode's config can switch back to `kind: fontist`.
-
-5. **Merging requires explicit user authorization per PR.** No
-   auto-merge.
-
-## Branch naming
-
-Following the convention in TODO.new/00-README.md:
-
-- ucode repo: `audit/<track-slug>` (e.g. `audit/coverage-matrix`)
-- fontist.org repo: `feat/<track-slug>` or `fix/<track-slug>` as
-  appropriate
-- fontist-archive-private repo: `audit/<track-slug>`
-- fontist-archive-public repo: `audit/<track-slug>`
-
-## What's NOT in this plan
-
-These items are out of scope for the current Part 1/Part 2 directive:
-
-- **CI for periodic re-build**: when Unicode versions update,
-  regenerate the set. Belongs in a separate TODO once the
-  infrastructure stabilizes.
-- **Real-time glyph extraction**: users extracting glyphs on
-  demand via ucode-as-a-service. Not in scope; the universal set
-  is pre-built.
-- **Color emoji extraction**: Noto Color Emoji uses CBDT/CBLC
-  bitmap tables, not vector outlines. Out of scope for vector
-  extraction; would need separate TODO.
-- **Glyph diffing across Unicode versions**: tracking how a
-  codepoint's official glyph changes between Unicode X.Y and X.Z.
-  Useful but separate.
-
-## Acceptance
-
-- [ ] Every TODO 32–41 lists its branch + PR-per-TODO commitment
-- [ ] Critical path is unambiguous: A1 → A2 → B1 → {B3, C1} → {C2, C3}
-- [ ] Parallel tracks (A3, B2) identified explicitly
-- [ ] External dependencies (fontist/formulas PRs) called out
-- [ ] Out-of-scope items listed so they don't creep in
-
-## References
-
-- [TODO 28](28-implementation-order-update.md) — prior sequencing (23–31)
-- [TODO 32](32-uc17-coverage-matrix.md) — Phase 1 start
-- [TODO 35](35-universal-set-production-run.md) — Phase 1 end
-- [TODO 36](36-per-font-coverage-audit.md) — Phase 2 start
-- [TODO 40](40-archive-private-uses-ucode-audit.md) — pipeline wiring
-- [TODO 41](41-ucode-unicode-archive-bridge.md) — publishing pipeline

data/TODO.new/40-archive-private-uses-ucode-audit.md

@@ -1,124 +0,0 @@
-# 40 — fontist-archive-private bin/build uses ucode audit
-
-## Goal
-
-Refactor `fontist-archive-private/bin/build` so it invokes
-`ucode audit font` instead of the dead `Fontisan::Commands::AuditCommand`
-path. The current script (last touched when fontisan still owned audits)
-has a UCD-stub hack at lines 13–21 that returns empty UCD data —
-exactly the functionality ucode provides natively via its own UCD
-parse + cache.
-
-This TODO is the engineering work to make the architecture doc
-(coverage-architecture.md §"Build Pipeline") match reality.
-
-## Why a separate TODO
-
-The audit migration (TODOs 06–12) ported fontisan's audit subsystem
-into ucode. The CLI command `ucode audit font <path>` produces the
-same shape of YAML that `Fontisan::Commands::AuditCommand` used to.
-But fontist-archive-private's `bin/build` was never updated to call
-the new tool — it still requires `fontisan` and stubs UCD out.
-
-Three problems with the current state:
-
-1. **Coverage aggregations are empty.** Every audit YAML currently
-   has `blocks: []`, `unicode_scripts: []` because the UCD stub
-   returns nil. Consumers (fontist.org's coverage browser) see
-   per-font cmap lists but no per-block fill ratios.
-
-2. **Two sources of truth for "what's in UCD."** fontisan used to
-   auto-download `ucd.all.flat.xml` (removed per CLAUDE.md); the
-   stub hack papers over the missing file. ucode has its own
-   authoritative UCD parse under `~/.cache/ucode/`.
-
-3. **No universal-set reference.** Even if the UCD stub were removed,
-   fontisan can't compare a font's cmap to the canonical universal
-   glyph set. ucode can (TODO 35 produces the set; TODO 36 adds the
-   comparison).
-
-## Scope
-
-### Phase A — Swap audit invocation
-
-1. Replace lines 100–115 of `bin/build`:
-
-   ```ruby
-   # OLD (broken — uses Fontisan::Commands::AuditCommand + UCD stub)
-   cmd = Fontisan::Commands::AuditCommand.new(face_path, font_index: font_index, no_codepoints: false)
-   report = cmd.run
-   File.write(audit_path, report.to_yaml)
-   ```
-
-   with:
-
-   ```ruby
-   # NEW — invoke ucode audit via CLI (shelling out keeps the build
-   # script decoupled from ucode's internal API)
-   system("ucode", "audit", "font", face_path,
-          "--font-index", font_index.to_s,
-          "--output", audit_path,
-          out: File::NULL, err: verbose ? $stderr : File::NULL)
-   ```
-
-   OR via the Ruby API if shell-out overhead becomes measurable:
-
-   ```ruby
-   Ucode::Commands::Audit::FontCommand.new.call(
-     path: face_path,
-     font_index: font_index,
-     output: audit_path,
-   )
-   ```
-
-2. **Remove the UCD stub hack** (lines 13–21 of `bin/build`). ucode
-   has its own UCD cache; no stub needed.
-
-3. **Update Gemfile** — add `ucode` gem, keep `fontisan` for
-   ConvertCommand (WOFF generation), keep `excavate` for archive
-   extraction.
-
-### Phase B — Universal-set reference (after TODO 35)
-
-4. When the universal set is published (TODO 35 + TODO 41 bridge),
-   pass `--reference-universal-set=<path>` to every `ucode audit font`
-   invocation. The audit YAML gains a `coverage` section comparing
-   the font's cmap to the canonical per-block codepoint lists.
-
-5. The universal set lives in fontist-archive-public under
-   `unicode/universal-glyph-set/` (per TODO 41). fontist-archive-private's
-   CI checks it out shallow before invoking bin/build, so the audit
-   can reference it.
-
-### Phase C — Cleanup
-
-6. Remove the `module Fontisan::Audit::Context` monkey-patch
-   entirely. Dead code once fontisan's AuditCommand is no longer
-   called.
-
-7. Update `coverage-architecture.md` examples to match the new
-   `ucode_version` field in the audit YAML schema (replaces
-   `fontisan_version` for the audit producer; fontisan_version may
-   still appear as the parser-layer version).
-
-8. Specs: add a test fixture — small formula YAML + small TTF →
-   assert bin/build produces an audit YAML with non-empty `blocks`
-   and the new `ucode_version` field.
-
-## Acceptance
-
-- [ ] `bin/build` invokes `ucode audit font` (not Fontisan)
-- [ ] UCD stub hack removed
-- [ ] Audit YAMLs include populated `blocks:` and `unicode_scripts:`
-      (not empty arrays)
-- [ ] Audit YAML carries `ucode_version` field
-- [ ] Universal-set comparison lands when TODO 35 + TODO 41 are done
-- [ ] At least one formula end-to-end (e.g. `google/abeezee`) produces
-      a complete audit YAML via the new path
-
-## References
-
-- `fontist-archive-private/bin/build` — current implementation
-- `fontist.org/coverage-architecture.md` — target architecture (updated)
-- [TODO 36](36-per-font-coverage-audit.md) — consumes the new audit data
-- [TODO 41](41-ucode-unicode-archive-bridge.md) — universal-set publishing

data/TODO.new/41-ucode-unicode-archive-bridge.md

@@ -1,160 +0,0 @@
-# 41 — ucode Unicode artifacts → fontist-archive-public bridge
-
-## Goal
-
-Publish ucode's Unicode-only artifacts (universal glyph set, block-feed,
-per-codepoint JSONs) into `fontist-archive-public/unicode/` so
-fontist.org has ONE source of truth for both per-font data AND
-per-codepoint data.
-
-Mirrors the existing fontisan pattern: ucode's CI builds artifacts in
-its own repo, then a sync workflow publishes them to
-fontist-archive-public.
-
-## Why a separate TODO
-
-Today ucode's output lives only in the ucode repo under `output/`
-(gitignored — too big to commit). PR #44 in fontist.org added a
-`fetch-data.sh --with-ucode` flag that pulls from
-`raw.githubusercontent.com/fontist/ucode/main/docs/public/` — direct
-fetch, bypassing the archive.
-
-Two problems with the direct-fetch approach:
-
-1. **Source/build separation is blurred.** The ucode repo would have
-   to commit built artifacts under `docs/public/` (1.2 GB if we ship
-   per-codepoint JSONs). Repo bloat, git history grows linearly with
-   Unicode versions.
-
-2. **Inconsistent with fontist-archive pattern.** Per-font data goes
-   private → public archive → site. Per-codepoint data should follow
-   the same shape.
-
-This TODO introduces `fontist-archive-public/unicode/` as the
-canonical public location for ucode's Unicode artifacts.
-
-## Scope
-
-### Phase A — ucode CI publishes to fontist-archive-public
-
-1. New GHA workflow in `fontist/ucode`: `.github/workflows/publish-unicode-archive.yml`
-   Triggers on:
-   - Push to main (after parse + universal-set build succeeds)
-   - Manual dispatch (regenerate without rebuilding UCD)
-
-2. The workflow runs:
-   - `ucode fetch ucd <version>` + `ucode fetch unihan <version>` +
-     `ucode fetch charts <version>` + `ucode fetch fonts`
-   - `ucode parse <version>` → produces `output/`
-   - `ucode block-feed --ucode-output=./output --target=./output/block-feed`
-   - `ucode universal-set build <version>` (TODO 35) → produces
-     `output/universal_glyph_set/`
-   - Sync into `fontist-archive-public` via git:
-
-   ```yaml
-   - name: Sync to fontist-archive-public
-     run: |
-       git clone --depth 1 https://${GH_TOKEN}@github.com/fontist/fontist-archive-public archive
-       rsync -a --delete output/block-feed/ archive/unicode/block-feed/
-       rsync -a --delete output/universal_glyph_set/ archive/unicode/universal-glyph-set/
-       # Per-codepoint JSONs are 1.2GB total — too big for git LFS.
-       # Either: (a) sample for production (Basic Latin + CJK subset);
-       # Or:     (b) push to a release artifact, not the repo itself.
-       # Decision: per-codepoint JSONs ship via GitHub Release assets
-       # attached to the workflow run, NOT committed to the repo.
-       cd archive
-       git config user.email "ucode-bot@fontist.org"
-       git config user.name "ucode-bot"
-       git commit -am "Sync Unicode data from ucode@${GITHUB_SHA}"
-       git push origin main
-   ```
-
-3. Per-codepoint JSONs (`output/blocks/<ID>/<U+XXXX>/index.json`) —
-   1.2 GB total, too big for the repo. Publish as a Release asset
-   `.tar.zst` per Unicode version. fontist.org's fetch-data.sh
-   downloads + extracts on demand.
-
-### Phase B — fontist-archive-public structure
-
-4. The public archive gains a new top-level `unicode/` directory:
-
-   ```
-   fontist-archive-public/
-   ├── coverage/                      # existing — per-font audit YAMLs
-   ├── fonts/                         # existing — WOFF specimens
-   ├── fonts.json                     # existing
-   ├── unicode/                       # NEW — ucode output
-   │   ├── block-feed/
-   │   │   ├── unicode-blocks.json
-   │   │   ├── unicode-version.json
-   │   │   └── unicode/blocks/<slug>.json
-   │   ├── universal-glyph-set/
-   │   │   ├── manifest.json
-   │   │   ├── entries/U+XXXX.json
-   │   │   └── glyphs/U+XXXX.svg
-   │   └── codepoints-{version}.tar.zst   # release asset link
-   ├── bin/sync-from-private
-   └── .github/workflows/sync.yml
-   ```
-
-5. Update `fontist-archive-public/bin/sync-from-private` to also
-   accept the ucode sync (or add a separate `sync-from-ucode` script).
-   The sync workflow in fontist-archive-public triggers on:
-   - fontist-archive-private pushes (existing — coverage/woff sync)
-   - ucode publish workflow run (new — unicode/ sync)
-
-### Phase C — fontist.org fetch-data.sh
-
-6. Extend `scripts/fetch-data.sh` to also copy `unicode/` from
-   `fontist-archive-public`:
-
-   ```bash
-   log "copying unicode/block-feed/"
-   mkdir -p "$PUBLIC/unicode"
-   if [[ -d "$TMP/archive/unicode/block-feed" ]]; then
-     cp -r "$TMP/archive/unicode/block-feed/." "$PUBLIC/unicode/"
-   fi
-
-   log "copying unicode/universal-glyph-set/"
-   if [[ -d "$TMP/archive/unicode/universal-glyph-set" ]]; then
-     cp -r "$TMP/archive/unicode/universal-glyph-set/." "$PUBLIC/unicode/glyphs/"
-   fi
-   ```
-
-7. The `--with-ucode` flag from PR #44 becomes a no-op (or redirects
-   to a warning to update fetch-data.sh). All ucode data flows through
-   the archive.
-
-8. For per-codepoint JSONs (1.2 GB tar.zst): add a `--with-codepoints`
-   flag to fetch-data.sh. Default OFF — production doesn't need all
-   299k JSONs; local dev can opt in. When ON, download the Release
-   asset, extract to `public/codepoints/`.
-
-### Phase D — Versioning
-
-9. `unicode/unicode-version.json` records the UCD version. fontist.org
-   reads this to display "Unicode 17.0.0 data, refreshed <date>".
-
-10. When a new Unicode version drops (UCD 18.0.0), ucode publishes a
-    NEW versioned directory:
-    `unicode/v18/block-feed/`, `unicode/v18/universal-glyph-set/`.
-    fontist.org can pin to a specific version.
-
-## Acceptance
-
-- [ ] ucode GHA workflow runs end-to-end on push to main
-- [ ] fontist-archive-public gains `unicode/block-feed/` and
-      `unicode/universal-glyph-set/`
-- [ ] Per-codepoint JSONs ship as a Release asset (not in-repo)
-- [ ] fontist.org `fetch-data.sh` copies `unicode/` from the archive
-      (no more direct raw.githubusercontent.com fetch)
-- [ ] `unicode-version.json` reflects the current UCD version
-- [ ] ucode repo stays lean (no built artifacts committed)
-
-## References
-
-- [TODO 35](35-universal-set-production-run.md) — universal-set build
-- [TODO 38](38-fontist-org-glyph-consumer.md) — consumer side
-- [TODO 40](40-archive-private-uses-ucode-audit.md) — per-font audit pipeline
-- `fontist.org/scripts/fetch-data.sh` — consumer (needs Phase C update)
-- `fontist.org/coverage-architecture.md` — updated architecture