ucode 0.1.0 → 0.1.1

data/TODO.full/07-fontisan-remove-ucd.md

@@ -0,0 +1,125 @@
+# 07 — fontisan: remove UCD/UCDXML subsystem
+
+## Goal
+
+Strip the UCD/UCDXML parsing subsystem out of `fontisan`. ucode now
+owns UCD parsing (`ucode parse`, `ucode fetch ucd`); fontisan's
+`lib/fontisan/ucd/`, `lib/fontisan/models/ucd/`, and `config/ucd.yml`
+are dead code that misleads consumers about who's the UCD authority.
+
+Pairs with TODO 06 (audit removal). Both land together as the 0.3.0
+release of fontisan.
+
+## Why now
+
+- ucode's UCD parse is the canonical path. It's faster, more complete
+  (parses NamesList, Unihan, all auxiliary + extracted files), and
+  ships its own UCD cache.
+- fontisan's UCD subsystem depended on `ucd.all.flat.xml` — which was
+  removed from the Unicode distribution in favor of UAX#44 text files.
+  The "real-shape-parsing" branch was an attempt to fix this; ucode
+  superseded it.
+- Leaving UCD in fontisan creates two sources of truth for "what does
+  U+XXXX mean" — exactly the anti-pattern the audit migration fought.
+
+## Scope
+
+### Code
+
+```
+lib/fontisan/ucd.rb                            # DELETE
+lib/fontisan/ucd/                              # DELETE entire directory
+  index_builder.rb
+  xml_parser.rb
+  text_file_parser.rb
+  ...
+lib/fontisan/models/ucd.rb                     # DELETE
+lib/fontisan/models/ucd/                       # DELETE entire directory
+  block.rb
+  script.rb
+  codepoint.rb
+  ...
+config/ucd.yml                                 # DELETE (was auto-downloaded UCD config)
+lib/fontisan/commands/ucdxml_command.rb        # DELETE (if exists)
+```
+
+### Tests
+
+```
+spec/fontisan/ucd/                             # DELETE
+spec/fontisan/models/ucd/                      # DELETE
+spec/fontisan/commands/ucdxml_command_spec.rb  # DELETE (if exists)
+spec/fixtures/ucd/                             # DELETE (test fixtures)
+spec/fixtures/ucd.all.flat.xml                 # DELETE
+spec/fixtures/ucd.all.flat.zip                 # DELETE
+```
+
+### CLI registration
+
+`lib/fontisan/cli.rb` — remove any UCD-related subcommands
+(`ucdxml` if present).
+
+### Documentation
+
+- `README.md` — remove UCD references
+- `docs/ucd.md` (if exists) — delete
+
+## Branch context
+
+The current `fix/ucdxml-real-shape-parsing` branch in fontisan was an
+attempt to revive the UCD-XML parser. That work is now superseded by
+ucode's UAX#44 text-file parsing (more reliable, doesn't depend on the
+removed `.flat.xml` artifact).
+
+This TODO **abandons** `fix/ucdxml-real-shape-parsing` and removes the
+subsystem entirely. The branch's exploration work is preserved in git
+history but not merged.
+
+## Migration path
+
+Consumers currently calling `Fontisan::UCD::IndexBuilder` etc. should
+switch to ucode's API:
+
+```ruby
+# Before (fontisan 0.2.x):
+index = Fontisan::UCD::IndexBuilder.new(version: "17.0.0").build
+index.block_for_cp(0x4E00)  # => "CJK_Unified_Ideographs"
+
+# After (ucode 0.1.1+):
+Ucode::Coordinator.new.indices_for(
+  ucd_dir: "/path/to/ucd",
+  unihan_dir: "/path/to/unihan"
+).blocks  # => sorted array of Block records
+```
+
+Document this in the CHANGELOG entry (combined with TODO 06's entry).
+
+## Version bump
+
+Same release as TODO 06: **0.3.0** (breaking change).
+
+## Acceptance
+
+- [ ] `lib/fontisan/ucd.rb` and `lib/fontisan/ucd/` deleted
+- [ ] `lib/fontisan/models/ucd.rb` and `lib/fontisan/models/ucd/` deleted
+- [ ] `config/ucd.yml` deleted
+- [ ] Any `ucdxml` CLI subcommand removed
+- [ ] All UCD-related specs + fixtures deleted
+- [ ] `bundle exec rspec` passes
+- [ ] `bundle exec rubocop` clean
+- [ ] CHANGELOG entry documents the removal + migration path
+- [ ] `fix/ucdxml-real-shape-parsing` branch marked as abandoned (closed
+      with "superseded by ucode" comment)
+
+## Dependencies / blockers
+
+- **TODO.new 10** — Aggregations-UCD rewrite (must be complete; verified
+  by ucode's specs passing with real Unicode 17 data).
+- **TODO 06** — companion audit removal; both ship in 0.3.0.
+
+## References
+
+- `lib/fontisan/ucd.rb` (slated for deletion)
+- [TODO.new/10](../TODO.new/10-aggregations-ucd-rewrite.md) — ucode UCD aggregations
+- [TODO 06](06-fontisan-remove-audit.md) — companion cleanup
+- `fix/ucdxml-real-shape-parsing` branch (to be abandoned)

data/TODO.full/08-archive-private-bin-build.md

@@ -0,0 +1,143 @@
+# 08 — fontist-archive-private bin/build uses ucode audit + fontisan convert
+
+## Goal
+
+Refactor `fontist-archive-private/bin/build` so it:
+
+1. Calls `ucode audit font` (instead of the dead `Fontisan::Commands::AuditCommand`)
+   for every matched font face.
+2. Calls `fontisan ConvertCommand` (unchanged) for WOFF specimen
+   generation (open-license only).
+3. Drops the UCD-stub hack (lines 13-21) — ucode has its own UCD.
+
+Pairs with TODO.new/40 (same refactor). This file is the production
+checklist + acceptance criteria.
+
+## Why now
+
+- TODO 06 + TODO 07 land together, removing fontisan's audit + UCD.
+- bin/build currently `require "fontisan"` and references
+  `Fontisan::Commands::AuditCommand` — will break loudly after 0.3.0.
+- The UCD stub hack returns empty aggregations — every audit YAML
+  currently has `blocks: []`, `unicode_scripts: []`. Consumers see
+  per-font cmap but no per-block fill ratios.
+
+## Scope
+
+### Phase A — Swap audit invocation
+
+1. **Replace the audit call** (lines ~100-115 of `bin/build`):
+
+   ```ruby
+   # OLD (broken — uses removed AuditCommand + UCD stub hack)
+   cmd = Fontisan::Commands::AuditCommand.new(
+     face_path, font_index: font_index, no_codepoints: false
+   )
+   report = cmd.run
+   report = report.is_a?(Array) ? report.first : report
+   File.write(audit_path, report.to_yaml)
+   ```
+
+   with:
+
+   ```ruby
+   # NEW — shell out to ucode (decouples bin/build from ucode's internal API)
+   args = ["ucode", "audit", "font", face_path,
+           "--font-index", font_index.to_s,
+           "--output", audit_path]
+   args += ["--reference-universal-set", universal_set_path] if universal_set_path
+   success = system(*args,
+                     out: verbose ? $stdout : File::NULL,
+                     err: verbose ? $stderr : File::NULL)
+   warn "WARN audit #{slug}: ucode exited #{$?.exitstatus}" if verbose && !success
+   ```
+
+2. **Keep the WOFF call** (lines ~125-140) — `fontisan ConvertCommand`
+   stays as-is.
+
+3. **Remove the UCD stub hack** (lines 13-21 of `bin/build`):
+
+   ```ruby
+   # DELETE THIS:
+   module Fontisan
+     module Audit
+       class Context
+         def ucd
+           @ucd ||= { version: nil, blocks_index: nil, scripts_index: nil,
+                      warning: "ucd_skipped" }
+         end
+       end
+     end
+   end
+   ```
+
+4. **Update Gemfile**:
+
+   ```ruby
+   # fontist-archive-private/Gemfile
+   source "https://rubygems.org"
+   gem "ucode", "~> 0.1"          # NEW (audit tool)
+   gem "fontisan", "~> 0.3"       # BUMPED (audit + UCD removed)
+   gem "excavate"                 # unchanged (archive extraction)
+   gem "rake"                     # dev
+   ```
+
+### Phase B — Universal-set reference (after TODO.new 35)
+
+5. Once the universal glyph set is published (TODO.new 35 + TODO.new 41),
+   pass `--reference-universal-set=<path>` so audits include per-block
+   coverage comparison against the canonical glyphs.
+
+6. The universal set lives in `fontist-archive-public/unicode/universal-glyph-set/`.
+   bin/build checks it out shallow before invoking audits:
+
+   ```ruby
+   # Pseudocode — runs once per CI workflow, not per formula
+   universal_set_path = sync_universal_set_from_public_archive
+   ```
+
+### Phase C — CI workflow updates
+
+7. `.github/workflows/build.yml`:
+   - Bump fontisan gem to 0.3.0+
+   - Add ucode gem (0.1.1+) install step
+   - Add a pre-step that fetches the universal set from
+     `fontist-archive-public/unicode/universal-glyph-set/`
+   - Otherwise matrix unchanged (ubuntu for google/sil/manual, macOS
+     for macos)
+
+### Phase D — Test fixtures + acceptance
+
+8. Add a fixture-driven test: small formula YAML + small TTF → run
+   bin/build locally → assert audit YAML has:
+   - Populated `blocks:` array (not empty)
+   - Populated `unicode_scripts:` array (not empty)
+   - `ucode_version:` field (not `fontisan_version:` for audit producer)
+   - `coverage:` section when universal-set reference is passed
+
+## Acceptance
+
+- [ ] `bin/build` invokes `ucode audit font` (not Fontisan)
+- [ ] UCD stub hack removed
+- [ ] Audit YAMLs include populated `blocks:` and `unicode_scripts:`
+- [ ] Audit YAML carries `ucode_version` field (replaces `fontisan_version`)
+- [ ] WOFF conversion still works (`Fontisan::ConvertCommand`)
+- [ ] Gemfile lists `ucode` and `fontisan` (0.3.0+) as runtime deps
+- [ ] At least one formula end-to-end produces a complete audit YAML
+- [ ] GHA workflow runs to completion on a small formula subset
+
+## Dependencies / blockers
+
+- **TODO 05** — ucode 0.1.1 published (bin/build will `gem install ucode`)
+- **TODO 06** + **TODO 07** — fontisan 0.3.0 published (bin/build can't
+  require both old and new fontisan)
+- **TODO.new 35** + **TODO.new 41** — universal-set production + archive
+  bridge (Phase B depends on these)
+
+## References
+
+- `fontist-archive-private/bin/build` — current implementation
+- [TODO 06](06-fontisan-remove-audit.md) — fontisan audit removal
+- [TODO 07](07-fontisan-remove-ucd.md) — fontisan UCD removal
+- [TODO.new 40](../TODO.new/40-archive-private-uses-ucode-audit.md) — earlier sketch
+- `fontist.org/coverage-architecture.md` — updated architecture doc

data/TODO.full/09-archive-public-structure.md

@@ -0,0 +1,164 @@
+# 09 — fontist-archive-public structure: coverage/ + woff/ + unicode/ + panglyph/
+
+## Goal
+
+Define the canonical directory structure of `fontist-archive-public`
+once all data streams are wired. Today it has `coverage/` (fontisan
+audit YAMLs) + `woff/` (open-license specimens) + `fonts.json`. We
+need to add `unicode/` (ucode's Unicode data) and `panglyph/` (the
+universal font).
+
+## Why a separate TODO
+
+Three independent pipelines now feed fontist-archive-public:
+
+1. **fontist-archive-private CI** (per-formula) → `coverage/` + `woff/`
+2. **ucode CI** (per-Unicode-version) → `unicode/block-feed/` +
+   `unicode/universal-glyph-set/` + `unicode/codepoints-{version}.tar.zst`
+3. **panglyph CI** (per-release) → `panglyph/v{X.Y.Z}/` + `manifest.json`
+
+Each is owned by a different repo and syncs on a different cadence.
+This TODO defines how they coexist in one archive without colliding.
+
+## Target structure
+
+```
+fontist-archive-public/
+├── README.md                            # canonical index
+│
+├── coverage/                            # ← from fontist-archive-private CI
+│   └── {formula_slug}/{PSName}.yaml     # per-face audit YAMLs
+│       google/abeezee/ABeeZee-Regular.yaml
+│       manual/inter/Inter-Bold.yaml
+│       macos/...
+│
+├── woff/                                # ← from fontist-archive-private CI (open-license only)
+│   └── {formula_slug}/{PSName}.woff2
+│       google/abeezee/ABeeZee-Regular.woff2
+│       (NO macos/ — proprietary)
+│
+├── fonts.json                           # font registry (canonical name → formula slugs)
+├── font-metadata.json                   # per-face metadata (weight, style, etc.)
+│
+├── unicode/                             # ← from ucode CI (TODO.new 41)
+│   ├── block-feed/                      # compact per-block Unicode data feed
+│   │   ├── unicode-blocks.json
+│   │   ├── unicode-version.json
+│   │   └── unicode/blocks/<slug>.json
+│   │
+│   ├── universal-glyph-set/             # one SVG per codepoint (TODO.new 35)
+│   │   ├── manifest.json                # version, counts, generatedAt
+│   │   ├── entries/U+XXXX.json          # per-glyph provenance
+│   │   └── glyphs/U+XXXX.svg
+│   │
+│   ├── codepoints-{version}.tar.zst     # per-codepoint detailed JSONs (TODO.new 41 §Phase A.3)
+│   └── codepoints-index.json            # quick lookup: cp_int → {block, name, ...}
+│
+├── panglyph/                            # ← from panglyph CI (TODO 04)
+│   ├── manifest.json                    # latest version + version index
+│   └── v17.0.0/                         # per-release directory
+│       ├── panglyph-unicode17.ttf
+│       ├── panglyph-unicode17.woff2
+│       ├── panglyph-unicode17.otf
+│       ├── coverage-report.json
+│       └── source-manifest.json         # OFL provenance per source font
+│
+├── bin/
+│   ├── sync-from-private                # existing: pulls coverage/ + woff/
+│   ├── sync-from-ucode                  # NEW: pulls unicode/
+│   └── sync-from-panglyph               # NEW: pulls panglyph/
+│
+└── .github/workflows/
+    ├── sync-private.yml                 # triggers on archive-private push
+    ├── sync-ucode.yml                   # triggers on ucode publish workflow
+    └── sync-panglyph.yml                # triggers on panglyph tag
+```
+
+## Three sync workflows
+
+### sync-private.yml (existing — minor update)
+
+Triggers on push to `fontist/fontist-archive-private` main.
+- Clones private shallow
+- Copies `coverage/` (ALL audit YAML — metadata is public)
+- Copies `woff/` (open-license only — checks each formula's license)
+- Updates `fonts.json` + `font-metadata.json`
+- Commits + pushes to public
+
+### sync-ucode.yml (NEW — TODO.new 41)
+
+Triggers on workflow_run of `fontist/ucode`'s `publish-unicode-archive.yml`.
+- Clones ucode's published artifacts (Release asset OR direct git push
+  from ucode CI — design decision in TODO.new 41)
+- Syncs `unicode/block-feed/`, `unicode/universal-glyph-set/`
+- Replaces `unicode/codepoints-{version}.tar.zst`
+- Updates `unicode/codepoints-index.json` (regenerated from per-cp JSONs)
+
+### sync-panglyph.yml (NEW — TODO 04)
+
+Triggers on tag push to `fontist/panglyph`.
+- Clones panglyph Release assets
+- Creates `panglyph/v{X.Y.Z}/` directory
+- Updates `panglyph/manifest.json` (latest version pointer)
+
+## Conflict resolution
+
+The three workflows can run concurrently. They write to disjoint
+directories (`coverage/`, `unicode/`, `panglyph/`), so git conflicts
+are unlikely. If two syncs race, the second one's commit fails cleanly
+(Git pre-receive hook rejects non-fast-forward) and re-runs on the
+next trigger.
+
+## Manifest of manifests
+
+Top-level `archive-public/manifest.json`:
+
+```json
+{
+  "updated_at": "2026-...",
+  "coverage": {
+    "total_formulas": 4283,
+    "total_faces": 12000,
+    "last_sync": "2026-..."
+  },
+  "woff": {
+    "total_faces": 9500,
+    "last_sync": "2026-..."
+  },
+  "unicode": {
+    "ucd_version": "17.0.0",
+    "block_count": 346,
+    "codepoint_count": 299382,
+    "universal_set_built_at": "2026-..."
+  },
+  "panglyph": {
+    "latest": "17.0.0",
+    "released_at": "2026-..."
+  }
+}
+```
+
+fontist.org's fetch-data.sh reads this to display "data refreshed X
+hours ago" on the site.
+
+## Acceptance
+
+- [ ] `unicode/` directory exists with block-feed + universal-glyph-set
+- [ ] `panglyph/` directory exists with at least one version
+- [ ] Three sync workflows exist + run independently
+- [ ] `archive-public/manifest.json` reflects the current state
+- [ ] No git conflicts when 2+ syncs run concurrently (disjoint paths)
+- [ ] fontist.org's fetch-data.sh can pull all four data streams
+
+## Dependencies / blockers
+
+- **TODO.new 41** — ucode → archive bridge (the `unicode/` sync)
+- **TODO 04** — panglyph publish (the `panglyph/` sync)
+- **TODO 08** — archive-private uses ucode (the `coverage/` sync content changes)
+
+## References
+
+- `fontist/fontist-archive-public` repo (current state)
+- `fontist.org/scripts/fetch-data.sh` — consumer of all four streams
+- [TODO.new 41](../TODO.new/41-ucode-unicode-archive-bridge.md) — ucode publishing
+- [TODO 04](04-panglyph-publish-pipeline.md) — panglyph publishing

data/TODO.full/10-fontist-org-woff-glyphs.md

@@ -0,0 +1,131 @@
+# 10 — fontist.org: per-font WOFF glyph rendering (open-license fonts)
+
+## Goal
+
+Render font specimens on fontist.org using actual WOFF files from
+`fontist-archive-public/woff/`. Currently the site has the WOFFs in
+`public/fonts/*.woff2` but the unicode browser uses system fallback
+fonts (via `displayChar(cp, category)`) which renders tofu for rare
+scripts.
+
+For open-license fonts, fontist.org should inject `@font-face` for the
+specific font slug and render chars using that font directly.
+
+## Why a separate TODO
+
+The fontist.org unicode browser today shows:
+- Block grid: chars rendered via system fallback
+- Char detail: same system fallback
+
+Users want to see "what does this codepoint look like in Inter / Noto
+Sans / FSung?" — not "what does my OS render it as?"
+
+For open-license fonts (Noto family, Google Fonts, OFL SIL fonts),
+fontist-archive-public/woff/ has the WOFF2 file. fontist.org can
+inject `@font-face` per-font and render codepoints using the active
+font.
+
+For proprietary fonts (Apple system fonts, Microsoft core fonts), the
+WOFFs aren't redistributable and the site can't serve them. Those
+fonts show coverage data only (TODO 11).
+
+## Scope
+
+### Phase A — Font injection helper
+
+1. New composable: `src/composables/useFontFace.ts`:
+
+   ```typescript
+   export function useFontFace(slug: string, familyName?: string) {
+     const css = `
+       @font-face {
+         font-family: '${familyName || slug}';
+         src: url('/fonts/${slug}.woff2') format('woff2');
+         font-display: swap;
+       }
+     `
+     // Inject into document head (deduped by slug)
+     // ...
+     return { fontFamily: `'${familyName || slug}'` }
+   }
+   ```
+
+2. Used by:
+   - `UnicodeBlockGrid.vue` when the user picks an active font
+   - `UnicodeCharPage.vue` to render the specimen glyph using the
+     active font
+   - `FontStylePage.vue` to render specimens in the font detail page
+
+3. **Active font state** — top-level Vue ref (Pinia store or
+   provide/inject). User selects via a font picker on the unicode
+   browser pages.
+
+### Phase B — Font picker UI
+
+4. New component: `src/components/FontPicker.vue`:
+   - Lists all open-license fonts in `public/fonts/`
+   - Search box + family grouping
+   - On select: sets active font + persists to localStorage
+
+5. Position on unicode browser:
+   - Top of `/unicode` page (sticky)
+   - Per-font pages (`/fonts/{slug}/unicode`) — auto-selects that font
+
+### Phase C — Block grid rendering with active font
+
+6. `UnicodeBlockGrid.vue` — accept a `fontSlug` prop. When set:
+   - Inject `@font-face` via useFontFace
+   - Apply `font-family: var(--active-font)` to each cell
+   - Cells with chars the active font doesn't cover get a visual
+     indicator (grayed-out / strikethrough)
+
+7. **Coverage overlay** — small indicator on each cell showing whether
+   the active font covers it (green dot = yes, gray dot = no). Source:
+   ucode audit data (TODO 11).
+
+### Phase D — Char detail rendering
+
+8. `UnicodeCharPage.vue` — when an active font is set:
+   - The large glyph at top renders in the active font (not system fallback)
+   - Show a small badge: "Rendered in {font name}" + a toggle to switch
+     to "system fallback" for comparison
+
+9. When no active font: keep current behavior (system fallback).
+
+### Phase E — Performance
+
+10. **Lazy font loading** — WOFF2 files can be large (Noto Sans CJK JP
+    is 18MB). Don't preload all fonts; only load the active font's WOFF.
+
+11. **Font preloading hints** for the most-commonly-picked fonts
+    (Noto Sans, Inter, etc.) via `<link rel="preload">` on the home
+    page.
+
+12. **CDN-friendly URLs** — `public/fonts/{slug}.woff2` should have
+    long cache headers + immutable filenames (sha-suffixed? TODO
+    separate).
+
+## Acceptance
+
+- [ ] `useFontFace` composable exists + injects `@font-face` cleanly
+- [ ] `FontPicker.vue` lists all open-license WOFFs in `public/fonts/`
+- [ ] Block grid renders cells in the active font
+- [ ] Char detail renders the large glyph in the active font
+- [ ] Per-cell coverage indicator (green = covered, gray = missing)
+- [ ] Lazy font loading (no preload of unused WOFFs)
+- [ ] Active font persists across page navigation (localStorage)
+
+## Dependencies / blockers
+
+- **fontist-archive-public** — must have `woff/` populated (TODO 09)
+- **TODO 11** — coverage data from ucode audit (for the green/gray
+  indicators)
+
+## References
+
+- `src/components/UnicodeBlockGrid.vue` — current grid component
+- `src/pages/UnicodeCharPage.vue` — current char page
+- `src/pages/FontStylePage.vue` — existing font detail page (may
+  already inject @font-face for specimens — reuse the pattern)
+- `public/fonts/*.woff2` — existing WOFF specimens
+- [TODO 11](11-fontist-org-audit-coverage.md) — coverage data layer