ucode 0.1.0 → 0.1.1

data/TODO.new/31-universal-set-production-build.md

@@ -0,0 +1,205 @@
+# 31 — Universal set production build + coverage validation
+
+## Goal
+
+Execute the universal-set build (TODO 24) end-to-end against the
+curated source config (TODO 29) with the acquired fonts (TODO 30).
+Validate the output: every assigned Unicode 17 codepoint has a glyph,
+the manifest is complete, provenance is recorded, and per-tier
+coverage matches the curated expectations.
+
+This is the actual **production run**. It produces the artifact that
+fontist.org (TODO 27) and the missing-glyph reporter (TODO 26)
+consume.
+
+## Why a separate TODO
+
+TODO 24 built the **mechanics**. TODO 29 curated the **policy**. TODO
+30 fetched the **fonts**. TODO 31 is **execution + validation**.
+
+Splitting execution from mechanics lets us:
+
+- Catch curation gaps (a font that doesn't actually cover a block).
+- Catch resolver bugs (a Tier 1 font listed but never queried).
+- Catch pillar fallback regressions (pillar-2 should produce
+  identical results to correlate-v4, but only if the catalog wiring
+  is correct).
+- Produce an auditable coverage report alongside the manifest.
+
+## Pre-build validation
+
+Before running the build, assert:
+
+1. **Source config loads cleanly.** `SourceConfig.load(path)` returns
+   a `GlyphSourceMap` with no schema errors.
+2. **All fonts present.** Every `path:` entry in the YAML exists on
+   disk (or is fontist-discoverable). Missing fonts = list + abort.
+   Don't start a 4-hour build with known-missing inputs.
+3. **Coverage assertion runs.** TODO 29's `CoverageAssertion` runs;
+   gaps are listed but don't abort (expected for some blocks).
+
+If pre-build validation fails, abort with a typed
+`Ucode::Glyphs::UniversalSet::PreBuildError` listing each failure.
+
+## Build execution
+
+```bash
+bin/ucode universal-set build \
+  --version 17.0.0 \
+  --source-config config/unicode17_universal_glyph_set.yml \
+  --output output/universal_glyph_set \
+  --parallel 8
+```
+
+Expected runtime: ~3-4 hours for full Unicode 17 (150,000+ codepoints).
+CJK dominates the runtime (~45,000 ideographs via FSung).
+
+## Post-build validation
+
+After the build, validate:
+
+1. **Completeness.** Every assigned codepoint has a `glyphs/<U+XXXX>.svg`.
+2. **Manifest integrity.** `manifest.json` parses, has an entry for
+   every assigned codepoint, totals reconcile.
+3. **Provenance recorded.** Every entry has non-nil `tier` and
+   `source` fields.
+4. **No tofu leaks.** Count pillar-3 entries; investigate any that
+   aren't documented as expected gaps (unassigned, PUA,
+   noncharacter — Last Resort is correct for these).
+5. **Idempotency.** Re-running with no source changes produces zero
+   file writes.
+
+## Per-tier coverage report
+
+`reports/by_tier.json`:
+
+```json
+{
+  "tier-1": 148512,
+  "pillar-1": 800,
+  "pillar-2": 200,
+  "pillar-3": 1500,
+  "gaps": 0
+}
+```
+
+Target: tier-1 ≥ 95% of assigned codepoints. Tier-3 (Last Resort
+tofu) ≤ 1% of assigned codepoints (Last Resort is the correct tier
+for unassigned/PUA/noncharacter — those should be the only tier-3
+entries among assigned codepoints, and there should be none).
+
+## Per-block coverage report
+
+`reports/by_block.json`:
+
+```json
+{
+  "Sidetic": {
+    "assigned": 26, "tier-1": 26, "pillar-1": 0, "pillar-2": 0, "pillar-3": 0
+  },
+  "Beria_Erfe": {
+    "assigned": 50, "tier-1": 50, "pillar-1": 0, "pillar-2": 0, "pillar-3": 0
+  },
+  "Combining_Diacritical_Marks_Extended": {
+    "assigned": 90, "tier-1": 63, "pillar-1": 0, "pillar-2": 27, "pillar-3": 0
+  }
+}
+```
+
+Each block's per-tier breakdown makes it obvious where Tier 1 coverage
+is incomplete. In the example, Combining Diacritical Marks Extended
+has 27 codepoints that fell through to pillar-2 — the residual gap
+the curation (TODO 29) flagged.
+
+## Gap investigation
+
+`reports/gaps.json` lists every assigned codepoint that ended up at
+pillar-3 (tofu) — these are **bugs**:
+
+```json
+[
+  { "codepoint": 119808, "block": "Mathematical_Alphanumeric_Symbols",
+    "reason": "tier-1:noto-sans-math did not cover; pillar-2 catalog miss" }
+]
+```
+
+Each gap entry records the path through the resolver that led to tofu.
+Zero gaps = perfect coverage. Non-zero gaps = actionable curation
+follow-ups (typically: "add font X to block Y's source list").
+
+## CJK Ext J verification
+
+Special verification for the largest single block: CJK Unified
+Ideographs Extension J (4,298 codepoints). The build should produce:
+
+- `tier-1` count == 4,298 if FSung-* covers all of them.
+- `tier-1` + `pillar-1` count == 4,298 if FSung-* misses some that
+  Code Charts PDF covers.
+
+Either is acceptable. The `reports/by_block.json` row for Ext J
+documents which path actually fired.
+
+## Files to create
+
+- `lib/ucode/glyphs/universal_set/validator.rb` — post-build
+  validator. Reads manifest + glyphs dir, runs the 5 checks above.
+- `lib/ucode/glyphs/universal_set/coverage_report.rb` — emits
+  per-tier + per-block + gaps JSON reports.
+- `lib/ucode/glyphs/universal_set/pre_build_check.rb` — runs
+  pre-build validation (config + fonts + assertion).
+- `lib/ucode/commands/universal_set.rb` — autoload hub (extend if
+  present).
+- `lib/ucode/commands/universal_set/validate.rb` — CLI subcommand.
+- Specs:
+  - `spec/ucode/glyphs/universal_set/validator_spec.rb`
+  - `spec/ucode/glyphs/universal_set/coverage_report_spec.rb`
+  - `spec/ucode/glyphs/universal_set/pre_build_check_spec.rb`
+
+## CLI
+
+```bash
+bin/ucode universal-set build      # TODO 24, existing
+bin/ucode universal-set validate   # TODO 31, new — post-build validation
+bin/ucode universal-set report     # TODO 31, new — emit coverage reports
+bin/ucode universal-set pre-check  # TODO 31, new — pre-build validation
+```
+
+`build` runs `pre-check` automatically before starting; the standalone
+`pre-check` is for iterating on curation without burning a 4-hour
+build.
+
+## Acceptance
+
+- `bin/ucode universal-set build` completes against Unicode 17.0 in
+  under 4 hours.
+- `output/universal_glyph_set/manifest.json` shows
+  `codepoints_built == codepoints_assigned` (≥ 150,000).
+- `reports/gaps.json` is empty for assigned codepoints outside the
+  documented residual cases (Combining Diacritical Marks Extended
+  additions, Symbols Legacy Supp additions, Supp Arrows-C additions).
+- `reports/by_tier.json` shows tier-1 ≥ 95% (target: 100% for
+  assigned codepoints outside documented gaps).
+- Re-running with no source changes produces zero file writes.
+- The build correctly handles CJK Ext J: all 4,298 codepoints
+  resolved via FSung-* or noto-sans-cjk-jp fallback (no tofu leaks).
+- Residual gaps fall through to Pillar 2 cleanly; no crashes, no
+  silent skips.
+- `pre-check` aborts on missing font files with a clear list of
+  what's missing.
+- Rubocop clean.
+
+## Out of scope
+
+- Source config curation — TODO 29.
+- Font acquisition — TODO 30.
+- fontist.org consumer integration — TODO 27.
+- Site rendering of the universal set — TODO 26 / TODO 27.
+
+## References
+
+- Build mechanics: `TODO.new/24-universal-glyph-set-build.md`
+- Source config: `TODO.new/29-universal-set-curation-uc17.md`
+- Font acquisition: `TODO.new/30-tier1-font-acquisition.md`
+- Audit consumer: `TODO.new/25-font-audit-against-universal-set.md`
+- Existing builder: `lib/ucode/glyphs/universal_set/builder.rb`
+- Existing manifest model: `lib/ucode/models/universal_set_manifest.rb`

data/TODO.new/32-uc17-coverage-matrix.md

@@ -0,0 +1,165 @@
+# 32 — Universal glyph set: full UC17 coverage matrix (Part 1 master)
+
+## Goal
+
+Produce **one canonical Tier 1 font recommendation per Unicode 17 block**
+(~346 entries). This is the master output of Part 1 — the artifact that
+defines "full coverage" for ucode's universal glyph set. Once this
+matrix is encoded in `config/unicode17_universal_glyph_set.yml`, every
+downstream TODO (production build, per-font audit, missing-glyph
+reporter, fontist.org consumer) treats it as ground truth.
+
+The matrix does NOT require fonts to be installed or cmaps to be
+verified — that's TODO 35 (production build) and TODO 36 (per-font
+audit). This TODO is purely the **policy**: "for block X, use font Y
+(fallback chain Z)."
+
+## Why a separate TODO
+
+TODO 29 (UC17 curation) started this work but stopped at ~30 specialist
+blocks. The remaining ~315 blocks rely on a single `default_sources`
+entry pointing at `noto-sans` via fontist — which the fontist formula
+repo doesn't actually carry as a generic package. So the current config
+CLAIMS full coverage but the resolver can't materialize glyphs for most
+blocks.
+
+This TODO splits the policy work from the acquisition work:
+
+- **TODO 32 (this)**: decide the canonical font per block (policy)
+- **TODO 33**: fix the acquisition paths (URLs + fontist formulas)
+- **TODO 35**: build the universal set end-to-end (run the policy)
+
+Reviewers can sign off on the per-block choices here without waiting
+for font availability.
+
+## Coverage policy (the recommendation)
+
+### Tier 1 default — Noto Sans family
+
+Noto is the canonical Tier 1 source for ~250 of ~346 blocks. Where a
+dedicated Noto Sans <Script> variant exists, use it; otherwise fall
+back to `noto-sans` (Latin/core).
+
+| Script family | Tier 1 font | Blocks covered |
+|---|---|---|
+| Latin + extensions + IPA + Spacing Modifier + Combining Diacriticals | `noto-sans` | ~20 blocks |
+| Greek + Coptic | `noto-sans` | 2 |
+| Cyrillic (all extensions) | `noto-sans` | 4 |
+| Armenian, Hebrew | `noto-sans-armenian`, `noto-sans-hebrew` | 2 |
+| Arabic + extensions + Supplement | `noto-naskh-arabic` / `noto-sans-arabic` | 4 |
+| Syriac, Thaana, NKo, Samaritan, Mandaic | `noto-sans-<script>` | 5 |
+| Brahmic (Devanagari, Bengali, Gurmukhi, Gujarati, Oriya, Tamil, Telugu, Kannada, Malayalam, Sinhala) | `noto-sans-<script>` | 10 |
+| Tibetan, Myanmar, Georgian | `noto-sans-<script>` | 3+ |
+| Hangul Jamo + compatibility | `noto-sans-hangul` or `noto-sans-kr` | 5 |
+| Ethiopic + extensions | `noto-sans-ethiopic` | 3 |
+| Cherokee, Canadian Aboriginal | `noto-sans-cherokee`, `noto-sans-canadian-aboriginal` | 2 |
+| Khmer, Mongolian, Limbu, Tai Le, Tai Tham, Buginese | `noto-sans-<script>` | 6 |
+| Symbol blocks (Math, Arrows, Misc, Geometric, Dingbats) | `noto-sans-symbols`, `noto-sans-symbols-2`, `noto-sans-math` | ~10 |
+| Music | `noto-music` | 1 |
+
+### Tier 1 specialists (non-Noto)
+
+These blocks need fonts outside the Noto family. Each must be acquired
+via `ucode fetch fonts` (specialist manifest, TODO 30).
+
+| Block | Range | Tier 1 font | Provenance | Confidence |
+|---|---|---|---:|---|
+| Sidetic | U+10940–1095F | Lentariso ≥1.029 | github.com/Bry10022/Lentariso | HIGH |
+| Beria Erfe | U+16EA0–16EDF | Kedebideri 3.001 | software.sil.org/kedebideri | HIGH |
+| Tai Yo | U+1E6C0–1E6F3 | NotoSerifTaiYo | translationcommons.org | HIGH |
+| Tolong Siki | U+11DB0–11DEF | Noto Sans Tolong Siki | notofonts.github.io | HIGH |
+| Sharada Supplement | U+11B60–11B7F | Noto Sans Sharada | Google Fonts | HIGH |
+| Egyptian Hieroglyphs | U+13000–1342F | UniHieroglyphica v16 | suignard.com/Ptolemaic/ | HIGH |
+| Egyptian Hieroglyph Format Controls | U+13430–1345F | Egyptian Text | github.com/microsoft/font-tools | HIGH |
+| Egyptian Hieroglyphs Extended-A | U+13460–143FF | UniHieroglyphica v16 | suignard.com | HIGH |
+| Egyptian Hieroglyphs Extended-B (new UC17) | U+134A0.. | UniHieroglyphica v16 | suignard.com | HIGH |
+| CJK Unified Ideographs | U+4E00–9FFF | FSung-1.ttf (local) + Noto Sans CJK JP fallback | ~/Downloads/全宋體 | HIGH |
+| CJK Unified Ideographs Extension A | U+3400–4DBF | FSung + Noto Sans CJK JP | ~/Downloads/全宋體 | HIGH |
+| CJK Unified Ideographs Extension B–H | various | FSung-2.ttf..FSung-X.ttf | ~/Downloads/全宋體 | HIGH |
+| CJK Unified Ideographs Extension J (new UC17) | U+31350–323AF | FSung (latest) + Noto Sans CJK JP | ~/Downloads/全宋體 | HIGH |
+| Tangut + Components + Supplement | U+17000–187FF | Noto Sans Tangut | notofonts.github.io | HIGH |
+| Symbols for Legacy Computing Supplement | U+1CC00–1CCFF | BabelStone Pseudographica | babelstone.co.uk | MEDIUM |
+| Supplemental Arrows-C (new UC17) | U+1CF00–1CFCF | Symbola | dn-works.com / github.com/zhm/symbola mirror | MEDIUM |
+
+### Tier 1 emoji
+
+| Block | Range | Tier 1 font |
+|---|---|---|
+| Emoticons + Pictographs + Supplemental + Transport + Symbols & Pictographs Extended-A | various | Noto Emoji (monochrome; Noto Color Emoji for color rendering only) |
+| Variation Selectors | U+FE00–FE0F | Noto Sans (special handling — invisible format chars) |
+
+### Pillar 2 fallback (no Tier 1 available)
+
+Blocks with no redistributable Tier 1 font MUST go through pillar 2
+(content-stream correlation). TODO 34 builds this; TODO 32 just
+records the policy.
+
+| Block | Why pillar 2 | Pillar 2 PDF source |
+|---|---|---|
+| Sidetic (if Lentariso unavailable) | Private foundry | U10940.pdf |
+| Beria Erfe (if Kedebideri unavailable) | UFO source, complex extract | U16EA0.pdf |
+| Egyptian Hieroglyph Format Controls (gap) | Egyptian Text limitations | U13430.pdf |
+
+### Pillar 3 last resort (always-on fallback)
+
+When both Tier 1 and pillar 2 fail (or for unassigned/PUA ranges that
+still need a placeholder glyph), the resolver emits a Last Resort Font
+tofu box. This is encoded as the lowest-priority source on
+`default_sources`, not per-block.
+
+## Scope
+
+1. **YAML structure** — extend `Models::GlyphSourceMap` to accept a
+   `default_sources` block at the top level (currently forces ~315
+   repetitions of the same Noto Sans entry). See TODO 29 §"Architectural
+   improvements" for the shape.
+
+2. **Curate every block** — walk `output/blocks/index.json`, decide
+   Tier 1 for each. Output: ~340 distinct entries (or ~30 specialists +
+   `default_sources`).
+
+3. **Per-block rationale comment** — every non-default entry must
+   explain WHY (provenance URL, OFL check, known coverage gaps). This
+   becomes the documentation for the universal set; reviewers should
+   not need to chase external links to understand a choice.
+
+4. **Resolve the specialists named in TODO 29** that didn't have
+   concrete URLs:
+   - Lentariso: GitHub repo has no releases (the prior URL was 404).
+     Policy: vendor the TTFs from `TTFs/` folder of the repo ZIP
+     (downloadable via `git clone` or codeload ZIP).
+   - EgyptianText: Microsoft/font-tools has no releases. Policy: pull
+     from `EgyptianOpenType/` directory in the repo.
+   - UniHieroglyphica: canonical URL is `suignard.com/Ptolemaic/` (BBAW
+     page is authoritative), not the prior `/UniHieroglyphica/` path.
+   - Symbola: dn-works.com no longer hosts public downloads. Policy:
+     mirror via `github.com/zhm/symbola` (OFL, version-pinned).
+
+5. **Test fixtures** — for each curated specialist, capture a small
+   fixture (1–5 codepoint ids) and assert the source map returns the
+   expected font label. Tests run without the font installed.
+
+## Acceptance
+
+- [ ] `config/unicode17_universal_glyph_set.yml` lists every Unicode
+      17 block by id, with `sources:` per entry or implicit
+      `default_sources` fallback.
+- [ ] Each specialist entry carries `provenance`, `license`, `url`
+      (or `path` for local), and a rationale comment.
+- [ ] `Ucode::Models::GlyphSourceMap#sources_for(block_id)` returns
+      the right list for default AND specialist entries.
+- [ ] Every specialist URL is HTTP 200-verifiable (or marked
+      `local_only: true` for user-supplied fonts like FSung).
+- [ ] Curation specs cover at least: Basic_Latin (default),
+      Sidetic (specialist fontist), Tai Yo (specialist path),
+      CJK Unified Ideographs (specialist multi-source with fallback),
+      Egyptian Hieroglyphs (specialist path).
+
+## References
+
+- [TODO 23](23-universal-glyph-set-source-map.md) — source map mechanism
+- [TODO 29](29-universal-set-curation-uc17.md) — initial curation
+- [TODO 33](33-specialist-font-acquisition-refresh.md) — fix URLs
+- [TODO 35](35-universal-set-production-run.md) — build it
+- `docs/architecture.md` — 4-tier glyph strategy
+- BBAW Egyptological Unicode Fonts page — authoritative for Egyptian family

data/TODO.new/33-specialist-font-acquisition-refresh.md

@@ -0,0 +1,138 @@
+# 33 — Specialist font acquisition refresh
+
+## Goal
+
+Fix the broken acquisition paths that block the universal-set build
+(TODO 35) from completing. Five of the seven specialists in
+`config/specialist_fonts.yml` return HTTP 404/301 today:
+
+| Label | Current URL | Status | Working URL |
+|---|---|---|---|
+| Lentariso | `github.com/Bry10022/Lentariso/releases/download/1.033/Lentariso.otf` | 404 (no releases published) | Repo has no release artifacts; vendor `TTFs/*.ttf` from `archive/master.zip` |
+| NotoSerifTaiYo | `translationcommons.org/wp-content/uploads/2025/09/NotoSerifTaiYo.ttf` | 404 | Path changed; needs upstream contact or alternate mirror |
+| UniHieroglyphica | `suignard.com/UniHieroglyphica/UniHieroglyphica-16.0.zip` | 301 redirect | New path is `suignard.com/Ptolemaic/` per BBAW |
+| EgyptianText | `github.com/microsoft/font-tools/releases/download/v1.0/EgyptianText-Regular.ttf` | 404 (no releases) | Vendor from `EgyptianOpenType/` in the repo |
+| BabelStonePseudographica | `babelstone.co.uk/Fonts/Download/BabelStonePseudographica.zip` | 404 | Page exists; download path moved — needs page scrape |
+| Symbola | `dn-works.com/wp-content/uploads/2020/ufas/Symbola.zip` | 404 (site no longer hosts downloads) | Mirror at `github.com/zhm/symbola` (version-pinned) |
+
+Plus: `noto-sans`, `noto-sans-cjk-jp`, `noto-sans-arabic`, `noto-sans-telugu`,
+`noto-sans-kannada`, `noto-sans-symbols`, `noto-sans-symbols-2`, `noto-music`,
+`noto-sans-sharada`, `noto-sans-sidetic`, `noto-sans-tolong-siki`,
+`noto-sans-tangut` — none are resolvable via `fontist install` (not in
+the formulas repo).
+
+## Why a separate TODO
+
+The fontist formulas repo (`github.com/fontist/formulas`) doesn't carry
+most Noto variants as separate packages. ucode's pre-build check fails
+hard on the first missing font; without fixes here, TODO 35 cannot
+proceed.
+
+Two distinct fixes are needed:
+
+1. **Direct-fetch URLs** — for specialists with known canonical sources
+   not in fontist (Lentariso, EgyptianText, UniHieroglyphica,
+   NotoSerifTaiYo, BabelStone, Symbola). These go through
+   `ucode fetch fonts` via `config/specialist_fonts.yml`.
+
+2. **fontist formula PRs** — for Noto variants that SHOULD be in
+   fontist but aren't yet. Upstream PRs to
+   `github.com/fontist/formulas`. Until merged, ucode can fall back
+   to direct GitHub release URLs (notofonts.github.io publishes
+   release artifacts).
+
+## Scope
+
+### Phase A — Specialist URL refresh (this ucode repo)
+
+1. **Lentariso** — change `url:` from the dead release URL to the
+   codeload archive: `https://codeload.github.com/Bry10022/Lentariso/zip/refs/heads/master`,
+   set `extract: true`, `extract_member: TTFs/Lentariso-Regular.ttf`
+   (and Bold/Italic if needed). Set `extract_multi: true` if the
+   fetcher needs to pull multiple members.
+
+2. **EgyptianText** — `https://codeload.github.com/microsoft/font-tools/zip/refs/heads/main`,
+   `extract: true`, `extract_member: EgyptianOpenType/EgyptianText-Regular.ttf`.
+   License is MIT per the repo; confirm against the LICENSE file
+   before recording.
+
+3. **UniHieroglyphica** — change `url:` to the new path under
+   `suignard.com/Ptolemaic/`. The exact filename needs a HEAD request
+   to discover (likely `UniHieroglyphica-16.0.zip` or
+   `UniHieroglyphica.zip`). Record sha256 on first successful fetch.
+
+4. **BabelStonePseudographica** — fetch
+   `babelstone.co.uk/Fonts/Pseudographica.html`, parse for the actual
+   download link (likely `BabelStonePseudographica.ttf` direct, not
+   zip). Update URL accordingly.
+
+5. **Symbola** — change `url:` to
+   `https://raw.githubusercontent.com/zhm/symbola/master/fonts/Symbola.ttf`
+   (verified HTTP 200). License: OFL per the mirror; confirm upstream
+   license matches before recording.
+
+6. **NotoSerifTaiYo** — needs upstream contact (translationcommons.org
+   doesn't expose a current download). Options:
+   - Email the maintainers (out of scope for code)
+   - Mark `local_only: true` and document that the user must supply
+     the file
+   - Find a GitHub mirror with the font committed
+
+   **Recommendation:** mark `local_only: true` for now, document in
+   the entry's `provenance:` field. Pillar 2 (TODO 34) covers
+   U+1E6C0–U+1E6FF if the font isn't available.
+
+### Phase B — fontist formula PRs (external repo)
+
+For each missing Noto variant, open a PR against
+`github.com/fontist/formulas` adding a formula. Each formula is a
+small YAML carrying:
+
+- Font metadata (name, license, copyright)
+- One or more release URLs with sha256
+- Per-platform install paths
+
+Variants to upstream (in priority order):
+
+1. **noto-sans-cjk-jp** — covers the most codepoints; user-visible
+   block (CJK Unified Ideographs). Already documented at
+   `github.com/notofonts/noto-cjk`.
+2. **noto-sans-symbols** + **noto-sans-symbols-2** — cover ~10 symbol
+   blocks.
+3. **noto-music** — covers Musical Symbols block.
+4. **noto-sans-sharada**, **noto-sans-sidetic**, **noto-sans-tolong-siki** —
+   UC17 specialists.
+5. **noto-sans-arabic**, **noto-sans-telugu**, **noto-sans-kannada** —
+   scripts where ucode needs the variant.
+6. **noto-sans-tangut** — Tangut block.
+
+### Phase C — Local fallbacks (until Phase B merges)
+
+Until fontist/formulas merges the new formulas, ucode's fetcher
+subsystem can pull directly from `notofonts.github.io` release
+artifacts (e.g. `https://github.com/notofonts/notofonts.github.io/raw/main/fonts/NotoSansTolongSiki/hinted/ttf/NotoSansTolongSiki-Regular.ttf`).
+
+Extend `specialist_fonts.yml` to include these as fallback entries
+when `kind: fontist` resolution fails. The fetcher already supports
+`kind: path` for direct URLs; just add Noto variants as path-kind
+entries.
+
+## Acceptance
+
+- [ ] `config/specialist_fonts.yml` URLs all return HTTP 200 (or are
+      marked `local_only: true` with documented user-supplied path)
+- [ ] `ucode fetch fonts` succeeds for every entry (including the
+      previously-broken ones); sha256 recorded in the YAML
+- [ ] Universal-set pre-check (`ucode universal-set pre-check 17.0.0`)
+      reports zero `fontist`-kind missing fonts (path-kind allowed
+      for not-yet-upstreamed Noto variants)
+- [ ] At least 3 fontist/formulas PRs opened for the highest-priority
+      Noto variants (CJK JP, Symbols, Symbols 2)
+- [ ] Each PR carries the upstream license + sha256 in the formula YAML
+
+## References
+
+- [TODO 30](30-tier1-font-acquisition.md) — original acquisition design
+- [TODO 32](32-uc17-coverage-matrix.md) — what we need these fonts FOR
+- `config/specialist_fonts.yml` — current (broken) manifest
+- `lib/ucode/commands/fetch.rb` — fetcher implementation

data/TODO.new/34-pillar2-content-stream-correlator.md

@@ -0,0 +1,147 @@
+# 34 — Pillar 2 ContentStreamCorrelator (generalize correlate-v4)
+
+## Goal
+
+Promote pillar 2 (PDF content-stream positional correlation) from a
+throwaway proof-of-concept into a first-class fallback in the
+canonical 4-tier resolver. Today `lib/ucode/glyphs/embedded_fonts/catalog.rb`
+bails at line 226 when `tu_ref` (ToUnicode CMap) is nil; this TODO
+makes it delegate to a new `ContentStreamCorrelator` that recovers
+CID→codepoint mappings from chart geometry alone.
+
+Proven on Tai Yo (all 54 assigned codepoints correctly mapped without
+ToUnicode — see `/tmp/correlate_v4.rb`). This TODO generalizes that
+script and makes it the fallback for blocks where:
+
+- Tier 1 font is unavailable (Sidetic if Lentariso unavailable,
+  Beria Erfe if Kedebideri unavailable, Egyptian Hieroglyph Format
+  Controls gaps)
+- The Code Charts PDF embeds subsetted CIDFonts without ToUnicode
+  (common for private specimen fonts — Unicode Consortium uses 80+
+  such fonts that are not redistributable)
+
+## Why a separate TODO
+
+The Catalog is the entry point for pillar 1 extraction. When
+`tu_ref` is nil, today it returns nil, which means the resolver
+silently drops to pillar 3 (Last Resort tofu). For blocks like
+Egyptian Hieroglyphs (4k+ codepoints where source fonts are
+private), this would mean 4k tofu boxes instead of real outlines.
+
+Pillar 2 is the only path for these blocks. Generalizing the Tai Yo
+proof is the unlock.
+
+## Algorithm (extracted from correlate-v4)
+
+```ruby
+# 1. Render the chart page to SVG via mutool:
+#    `mutool draw -F svg <pdf> <page>` produces an SVG with:
+#      <defs><path id="font_N_M"/> for every CID M in specimen font N
+#      <use xlink:href="#font_N_M" transform="matrix(a,b,c,d,X,Y)"/>
+#      for every placement
+#
+# 2. Partition <use> elements by their font index (the N in font_N_M):
+#    - Labels: fonts that emit hex digits (typically font_3, font_8)
+#    - Specimens: the CIDFont carrying the actual glyph outlines
+#      (typically font_4 or font_6)
+#
+# 3. Cluster label uses by Y-row:
+#    yb = (y / 1.5).round * 1.5   # quantize to row height
+#    xb = (x / 50.0).round * 50.0 # quantize to column width
+#    clusters[[yb, xb]] << label
+#
+# 4. Per cluster, sort members by X and join decoded text:
+#    decode = ->(s) { s.gsub(/&#x([0-9a-fA-F]+);/) { [$1.to_i(16)].pack("U") } }
+#    cp_hex = members.sort_by { |m| m[:x] }.map { |m| decode.call(m[:text]) }.join
+#
+# 5. The rightmost cluster per Y-row is the specimen codepoint label.
+#    The rightmost <use> per Y-row in the specimen font is the
+#    specimen glyph placement. CID(M) ↔ codepoint established.
+#
+# 6. Lift <path id="font_<specimen_idx>_<CID>"> outline, normalize
+#    viewBox, emit glyph.svg.
+```
+
+The Y-quantization (1.5) and X-quantization (50.0) come from the
+Code Charts typesetting convention. They should be parameters, not
+constants — different charts may use different grid sizes. Empirical
+discovery: walk all labels, find the smallest Y-gap, use that as
+quantization base.
+
+## Combinator caveat (managed)
+
+Code Charts convention draws combining marks (Mn category) as
+"dotted-circle + mark" side-by-side. The dotted circle is a separate
+`<use>` element; it does NOT contaminate the mark's glyf outline.
+Verified clean on all 5 Tai Yo Mn codepoints.
+
+However, some foundries ship composite glyphs (mark + base in same
+glyf). For those we'd need a dotted-circle subtraction step:
+
+1. Detect U+25CC outline in the extracted path (signature: ring of
+   small dots)
+2. Remove its subpaths from the final glyph
+
+This is a follow-up if any block needs it. Initial implementation
+just extracts the outline as-is; compositing artifacts get flagged
+in the validator (TODO 35).
+
+## Scope
+
+1. **`Ucode::Glyphs::EmbeddedFonts::ContentStreamCorrelator`** —
+   new class next to `Catalog`. API:
+   ```ruby
+   correlator = ContentStreamCorrelator.new(pdf_page:, specimen_font_index:)
+   mapping = correlator.call  # { codepoint_int => cid_int }
+   ```
+
+2. **Patch `Catalog#build_entry`** — when `tu_ref` is nil, instead
+   of returning nil, delegate to ContentStreamCorrelator. Caller-
+   unchanged. Catalog callers see a populated entry regardless of
+   whether pillar 1 or pillar 2 produced the mapping.
+
+3. **Page-walk helper** — for a given block PDF, identify the
+   specimen font index automatically (currently hardcoded in
+   correlate-v4 as font_4). Heuristic: the font with the most
+   `<use>` placements AND the highest CID count in `<defs>` is the
+   specimen font.
+
+4. **Y-row quantization auto-discovery** — collect all label Y
+   positions, find the smallest non-trivial gap, use that as the
+   row-height quantization. Same for X-gap → column width.
+
+5. **Path lifting** — given the specimen font index and CID, find
+   `<path id="font_<idx>_<cid>">` in the SVG, extract its `d=`
+   attribute, normalize the viewBox (typical Code Charts cell is
+   ~1000×1000 user units).
+
+6. **mutool integration** — wrap the `mutool draw -F svg` shell
+   call. Cache the rendered SVG keyed by PDF path + page number
+   under `~/.cache/ucode/unicode/<version>/svg/<block_id>-<page>.svg`.
+
+7. **Specs** — fixture-based tests for:
+   - Tai Yo (proven baseline — must reproduce correlate-v4 output
+     exactly)
+   - Sidetic (no Tier 1 fallback available; pillar 2 mandatory)
+   - Beria Erfe (same)
+   - At least one block WITH ToUnicode to ensure pillar 1 still
+     works (regression guard)
+
+## Acceptance
+
+- [ ] `ContentStreamCorrelator` class exists with documented API
+- [ ] Catalog delegates to it when `tu_ref` is nil
+- [ ] Tai Yo test fixture reproduces the correlate-v4 mapping (54/54
+      codepoints correctly attributed)
+- [ ] Sidetic + Beria Erfe PDFs produce complete mappings via
+      pillar 2 (no tofu fallback)
+- [ ] Combinator cleanliness check: every Mn codepoint's extracted
+      glyph passes the "no U+25CC sub-path" heuristic
+- [ ] mutool SVG output is cached; re-runs are no-ops
+
+## References
+
+- `/tmp/correlate_v4.rb` — proven implementation (112 lines, Tai Yo)
+- `lib/ucode/glyphs/embedded_fonts/catalog.rb:226` — bail point
+- [TODO 20](20-canonical-resolver-4-tier.md) — original 4-tier design
+- [TODO 32](32-uc17-coverage-matrix.md) — pillar 2 fallback policy