ucode 0.1.0 → 0.1.1

data/TODO.full/03-panglyph-font-builder.md

@@ -0,0 +1,201 @@
+# 03 — Panglyph font builder (outline extraction + assembly)
+
+## Goal
+
+Implement the actual font-build pipeline: read ucode's universal-set
+manifest, extract outlines from Tier 1 source fonts via fontisan,
+assemble them into a single OpenType/TrueType font, and write the
+result to disk in TTF / WOFF2 / OTF formats.
+
+This is the heart of panglyph. The CLI commands (`build`, `validate`,
+`publish`) are thin wrappers around these classes.
+
+## Architecture
+
+```
+ucode universal-set manifest
+       │
+       │ {cp: 65, source: {font: "noto-sans", gid: 36}}
+       │ {cp: 19968, source: {font: "FSung-1", gid: 18432}}
+       │ {cp: 128512, source: {font: "NotoSerifTaiYo", gid: 41}}
+       │ ...
+       ▼
+┌──────────────────────────────────────────────┐
+│  Builder                                     │
+│  - reads manifest                            │
+│  - groups codepoints by source font          │
+│  - for each source font:                     │
+│      OutlineExtractor.extract_many(font, cps)│
+│  - passes {cp → outline} to FontAssembler    │
+└──────────────────────────────────────────────┘
+       │
+       ▼
+┌──────────────────────────────────────────────┐
+│  FontAssembler                               │
+│  - opens an empty font skeleton              │
+│  - for each codepoint:                       │
+│      - allocates a GID in panglyph           │
+│      - writes the outline into glyf/CFF      │
+│      - sets cmap[cp] = GID                   │
+│  - finalizes: OS/2, name, head, hhea, hmtx   │
+│  - writes panglyph-unicode17.ttf             │
+└──────────────────────────────────────────────┘
+       │
+       ▼
+┌──────────────────────────────────────────────┐
+│  Woff2Writer (via fontisan ConvertCommand)   │
+│  - reads the TTF                             │
+│  - writes panglyph-unicode17.woff2           │
+└──────────────────────────────────────────────┘
+       │
+       ▼
+┌──────────────────────────────────────────────┐
+│  CoverageReport                              │
+│  - summarizes per-block source breakdown     │
+│  - writes coverage-report.json               │
+└──────────────────────────────────────────────┘
+```
+
+## Scope
+
+### Phase A — Outline extraction (`OutlineExtractor`)
+
+1. **Open source font** via `Fontisan::Font.open(path)`.
+2. **For each codepoint** in the source's responsibility list:
+   a. Look up GID via `font.cmap.unicode_map[cp]`
+   b. Get the outline:
+      - TrueType: `font.glyf.glyphs[gid]` → contour points
+      - CFF/OpenType: `font.cff.charstrings[gid]` → charstring program
+   c. Normalize to a uniform internal representation
+      (`Panglyph::Outline` struct with point arrays + flags).
+3. **Return** `{ cp => Outline }` hash.
+
+#### Edge cases
+
+- **Composite glyphs** (TrueType): if a glyph references other glyphs
+  via component references, recursively flatten into a single outline.
+  Track "this glyph was a composite" in the report.
+- **TTC collections**: pass `font_index:` to `Fontisan::Font.open`.
+- **Missing cmap entry**: the Tier 1 font claims to cover this cp but
+  its cmap doesn't actually map it. Log + skip; the universal set's
+  pre-check (TODO.new/35) catches this upstream.
+
+### Phase B — Font assembly (`FontAssembler`)
+
+4. **Initialize an empty font skeleton** with the standard required
+   tables: `cmap`, `head`, `hhea`, `hmtx`, `maxp`, `name`, `OS/2`,
+   `glyf`, `loca`, `post`. (For OTF: `CFF ` instead of `glyf`/`loca`.)
+5. **For each codepoint**:
+   a. Allocate next GID (`maxp.numGlyphs += 1`).
+   b. Write the outline to `glyf` (or `CFF `).
+   c. Add `cmap` subtable 4 entry: `unicode_map[cp] = gid`.
+   d. Add `hmtx` entry (advance width from source font; default 1000
+      if missing).
+   e. Add `name` entries — actually, only the font-family / version
+      name needs to be set; per-glyph names aren't required.
+6. **Finalize metadata**:
+   - `name`: family="panglyph Unicode 17", subfamily="Regular",
+     full="panglyph-unicode17", version="Version 17.0.0"
+   - `OS/2`: usWeightClass=400, fsSelection=REGULAR, ulUnicodeRange1..4
+     bit-packed for every range panglyph covers
+   - `head`: unitsPerEm=1000 (or 2048 for CJK-heavy), lowestRecPPEM=8
+   - `hhea`: numberOfHMetrics matches glyphs
+7. **Compute checksums + write final TTF**.
+
+#### fontisan extension required
+
+fontisan today only READS fonts + converts (WOFF). It needs new APIs:
+
+```ruby
+# lib/fontisan/font_writer.rb (NEW)
+class Fontisan::FontWriter
+  def initialize(format: :ttf)
+    @tables = {}
+    @format = format
+  end
+
+  def set_cmap(unicode_to_gid_map)
+    # ...
+  end
+
+  def add_glyph(gid, outline, advance_width: 1000, lsb: 0)
+    # ...
+  end
+
+  def set_name_records(records)
+    # ...
+  end
+
+  def write_to(path)
+    # computes checksums, writes head/glyf/loca/cmap/etc.
+  end
+end
+```
+
+This is a meaningful fontisan addition. Tracked separately as a
+fontisan TODO (see TODO.new/19 — fontisan docs update).
+
+### Phase C — WOFF2 conversion (`Woff2Writer`)
+
+8. Use `Fontisan::Commands::ConvertCommand.new(ttf_path, to: "woff2", output: ...)`.
+   No new code needed; just shell out from panglyph.
+
+### Phase D — Coverage report (`CoverageReport`)
+
+9. Walk the built font's cmap.
+10. Compare against ucode's universal-set codepoint list.
+11. Bucket per-block + per-source-tier:
+    ```json
+    {
+      "total_codepoints": 299382,
+      "covered": 297415,
+      "missing": 1967,
+      "by_block": [
+        { "block_id": "Egyptian_Hieroglyphs_Extended-B", "covered": 599, "missing": 1, "tier": 1 }
+      ],
+      "by_tier": [
+        { "tier": 1, "count": 295000 },
+        { "tier": 2, "count": 2415 },
+        { "tier": 3, "count": 1967 }
+      ]
+    }
+    ```
+
+## Performance considerations
+
+- 299,382 glyph extractions × ~1ms each = ~5 minutes of extraction.
+- Parallelize per-source-font: each source font's extractions are
+  independent, so use parallel gem or `Concurrent::Map`.
+- Memory: hold all outlines in memory at once (~1.2GB estimated). May
+  need to stream to disk for low-memory CI runners (use a temp sqlite
+  database keyed by GID).
+
+## Specs
+
+- **Builder** — fixture: 5 codepoints from 2 source fonts; assert
+  output font's cmap matches the manifest.
+- **OutlineExtractor** — fixture: a real font with known glyf layout;
+  assert extracted outline matches a known-good serialization.
+- **FontAssembler** — fixture: 3 outlines; assert output TTF parses
+  via `Fontisan::Font.open` and has the expected cmap.
+- **Woff2Writer** — fixture: a known TTF; assert WOFF2 output
+  byte-identical to a fontisan-direct conversion (sanity check).
+- **CoverageReport** — fixture: a built font with intentional gaps;
+  assert report correctly identifies missing codepoints.
+
+## Acceptance
+
+- [ ] `panglyph build 17.0.0` produces a valid TTF, WOFF2, and OTF
+- [ ] The built font's cmap contains every codepoint in the universal
+      set's manifest
+- [ ] `fontisan Font.open(panglyph.ttf)` succeeds (no corruption)
+- [ ] Coverage report JSON validates against schema
+- [ ] Build completes in <30 minutes on a single runner
+- [ ] Built font renders correctly in a browser (manual smoke test)
+
+## References
+
+- [TODO 01](01-panglyph-vision.md) — vision
+- [TODO 02](02-panglyph-repo-bootstrap.md) — repo skeleton
+- [TODO.new/24](../TODO.new/24-universal-glyph-set-build.md) — ucode's universal-set SVG output (panglyph input)
+- fontisan Font.open API (for reading source fonts)

data/TODO.full/04-panglyph-publish-pipeline.md

@@ -0,0 +1,126 @@
+# 04 — Panglyph publish pipeline (release to fontist-archive-public)
+
+## Goal
+
+When `panglyph build` completes, publish the resulting TTF/WOFF2/OTF
+artifacts to `fontist-archive-public/panglyph/`. This makes panglyph
+available to fontist.org and any other consumer that pulls from the
+public archive.
+
+## Why a separate TODO
+
+The build itself is local (TODO 03). Publishing involves:
+- Authenticating to fontist-archive-public via GHA bot token
+- Atomic sync (don't leave the archive half-updated)
+- Versioning (multiple panglyph versions can coexist)
+- Manifest update (so consumers can discover what's available)
+
+These concerns are distinct from the build, and any failure here
+shouldn't invalidate the build itself.
+
+## Scope
+
+### Phase A — `panglyph publish` command
+
+1. New CLI subcommand: `panglyph publish [VERSION]`.
+
+2. Behavior:
+   - Clone `fontist/fontist-archive-public` shallow into a temp dir.
+   - Copy artifacts:
+     ```
+     panglyph-unicode17-17.0.0.ttf       → archive-public/panglyph/v17.0.0/panglyph-unicode17.ttf
+     panglyph-unicode17-17.0.0.woff2     → archive-public/panglyph/v17.0.0/panglyph-unicode17.woff2
+     panglyph-unicode17-17.0.0.otf       → archive-public/panglyph/v17.0.0/panglyph-unicode17.otf
+     coverage-report.json                → archive-public/panglyph/v17.0.0/coverage-report.json
+     source-manifest.json                → archive-public/panglyph/v17.0.0/source-manifest.json
+     ```
+   - Update top-level `archive-public/panglyph/manifest.json`:
+     ```json
+     {
+       "latest": "17.0.0",
+       "versions": {
+         "17.0.0": {
+           "ucd_version": "17.0.0",
+           "panglyph_version": "17.0.0",
+           "released_at": "2026-...",
+           "coverage": { "covered": 297415, "total": 299382, "percentage": 99.3 },
+           "artifacts": {
+             "ttf": "v17.0.0/panglyph-unicode17.ttf",
+             "woff2": "v17.0.0/panglyph-unicode17.woff2",
+             "otf": "v17.0.0/panglyph-unicode17.otf"
+           },
+           "sha256": { "ttf": "...", "woff2": "...", "otf": "..." }
+         }
+       }
+     }
+     ```
+   - Commit + push via GHA bot identity:
+     ```
+     Author: fontist-bot <bot@fontist.org>
+     Message: panglyph: publish v17.0.0
+     ```
+
+3. **Atomicity**: write to a temp dir first, then `git mv` into place.
+   If any step fails, the archive is not left in a half-published state.
+
+### Phase B — Source manifest
+
+4. The build emits a `source-manifest.json` recording which Tier 1
+   source font contributed each glyph:
+
+   ```json
+   {
+     "sources": [
+       { "label": "noto-sans", "license": "OFL", "url": "...", "sha256": "...", "glyphs_contributed": 1107 },
+       { "label": "FSung-1", "license": "OFL", "path": "data/fonts/FSung-1.ttf", "sha256": "...", "glyphs_contributed": 20992 },
+       ...
+     ],
+     "total_sources": 17,
+     "total_glyphs": 297415
+   }
+   ```
+
+5. This manifest is the **provenance record** for the published font.
+   Anyone redistributing panglyph can verify OFL compliance.
+
+### Phase C — GitHub Release
+
+6. In addition to pushing to fontist-archive-public, the CI workflow
+   (TODO 02) creates a GitHub Release on the `panglyph` repo:
+   - Tag: `v17.0.0`
+   - Title: `panglyph Unicode 17.0.0`
+   - Body: copy of the coverage report + source manifest summary
+   - Assets: the TTF / WOFF2 / OTF files + JSON manifests
+
+7. Why both? `fontist-archive-public/panglyph/` is the canonical
+   machine-readable source. GitHub Releases is the human-discoverable
+   download page (people search GitHub for "panglyph" and find the
+   release directly).
+
+### Phase D — Idempotency + re-publishing
+
+8. If the same version is re-published (e.g. fixing a build bug):
+   - Detect existing version directory
+   - Compare new artifacts' sha256 to existing
+   - If identical, no-op (idempotent)
+   - If different, bump patch version (`17.0.0` → `17.0.1`) and publish
+     as new version. Old version stays for rollback.
+
+9. **NEVER overwrite a published version in-place.** Consumers may have
+   cached the old artifacts. Bumping patch is the only safe update.
+
+## Acceptance
+
+- [ ] `panglyph publish 17.0.0` updates fontist-archive-public/panglyph/
+- [ ] `manifest.json` reflects the new version
+- [ ] Source manifest records every contributing font + its sha256
+- [ ] GitHub Release exists on `fontist/panglyph` with the right tag
+- [ ] Re-publishing a built version is a no-op (idempotent)
+- [ ] Failed publish leaves archive-public unchanged (atomic)
+
+## References
+
+- [TODO 02](02-panglyph-repo-bootstrap.md) — CI workflow
+- [TODO 03](03-panglyph-font-builder.md) — build artifacts
+- [TODO.new/41](../TODO.new/41-ucode-unicode-archive-bridge.md) — archive bridge pattern
+- [TODO 09](09-archive-public-structure.md) — overall archive-public structure

data/TODO.full/05-ucode-0-1-1-release.md

@@ -0,0 +1,139 @@
+# 05 — ucode 0.1.1 patch release
+
+## Goal
+
+Cut the first public ucode gem release: **0.1.0 → 0.1.1**. Unblocks
+downstream consumers (fontist-archive-private, panglyph) that depend
+on a published gem.
+
+## Why 0.1.1 (not 0.1.0)
+
+0.1.0 was the initial commit (`Initial release: ucode 0.1.0`). The
+codebase has since gained:
+- BlockFeedEmitter (renamed from FontistConsumerEmitter)
+- 4-tier canonical resolver
+- Universal-set build infrastructure
+- Real UCD 17.0.0 parse pipeline
+- Audit subsystem ported from fontisan
+- Block-feed shape schema
+- Categorized Unihan data model
+
+A patch bump (0.1.1) signals "iteration on the initial release" without
+claiming API stability (which would warrant 0.2.0).
+
+## Scope
+
+### Phase A — Pre-release prep
+
+1. **Update `lib/ucode/version.rb`**:
+   ```ruby
+   module Ucode
+     VERSION = "0.1.1"
+   end
+   ```
+
+2. **Create `CHANGELOG.md`** (doesn't exist yet) with the 0.1.1 entry:
+   ```markdown
+   # Changelog
+
+   ## 0.1.1 — 2026-06-XX
+
+   ### Added
+   - BlockFeedEmitter: emits a compact per-block Unicode data feed
+     (renamed from FontistConsumerEmitter — the data is plain Unicode
+     data, not consumer-specific).
+   - Categorized Unihan model: 8 typed collections matching the Unihan
+     file structure (Dictionary Indices, Readings, Variants, etc.).
+   - Real-font Tier 1 source map for the universal glyph set (~17
+     specialists + Noto family default).
+   - Pillar 1 + Pillar 2 glyph extraction via 4-tier resolver.
+   - Per-codepoint properties from `extracted/` and `auxiliary/` UCD
+     files (display, segmentation, Indic, Hangul, Emoji, full binary
+     properties list).
+
+   ### Fixed
+   - NamedSequences parser field order (real UAX#44 is `Name; cps...`).
+   - BlockFeedEmitter canonical path (`blocks/<ID>/index.json` not
+     `blocks/<ID>.json`).
+   - fontist.org char page route params for combining/bidiclass.
+   - Vite dev server case-sensitive codepoints/ path (lowercase hex).
+   - Vue route-watcher for per-char data on navigation.
+
+   ### Removed
+   - All references to "fontist-consumer" naming from ucode (now
+     "block-feed"). The data emitted is Unicode data, not
+     consumer-specific.
+   ```
+
+3. **Update `ucode.gemspec`** if any metadata needs changing (likely
+   nothing — `spec.metadata` is already correct).
+
+4. **Run full test suite + rubocop** to make sure 0.1.1 actually ships
+   green:
+   ```bash
+   bundle exec rspec
+   bundle exec rubocop
+   ```
+
+### Phase B — Branch + PR
+
+5. Create release branch off `main`:
+   ```bash
+   git checkout main && git pull
+   git checkout -b release/0.1.1
+   ```
+
+6. Apply changes: version bump + CHANGELOG.
+
+7. Push + open PR:
+   ```bash
+   git push -u origin release/0.1.1
+   gh pr create --title "Release ucode 0.1.1" --body "..."
+   ```
+
+### Phase C — Merge + tag + publish
+
+8. **HARD GATE**: require explicit user authorization to merge + tag +
+   push to rubygems. Per global rules:
+   - NEVER push tags without user authorization
+   - NEVER merge to main without user authorization
+
+9. After user says "merge + tag + release":
+   ```bash
+   gh pr merge --merge release/0.1.1    # or rebase per user pref
+   git checkout main && git pull
+   git tag v0.1.1
+   git push origin v0.1.1
+   bundle exec rake release              # pushes to rubygems.org
+   ```
+
+10. Verify on RubyGems:
+    ```bash
+    gem search ucode --remote
+    # → ucode (0.1.1)
+    ```
+
+11. Create GitHub Release on the tag with the CHANGELOG entry as body.
+
+### Phase D — Downstream notifications
+
+12. Once 0.1.1 is on RubyGems, downstream consumers can update:
+    - `fontist/Gemfile` in fontist-archive-private: bump `ucode` to `~> 0.1`
+    - `panglyph.gemspec`: same
+    - `fontisan` Gemfile if it consumes ucode (it shouldn't after TODO 06/07)
+
+## Acceptance
+
+- [ ] `lib/ucode/version.rb` says `0.1.1`
+- [ ] `CHANGELOG.md` exists with the 0.1.1 entry
+- [ ] All specs pass; rubocop clean
+- [ ] PR opened against main, green CI
+- [ ] **WAIT for explicit user authorization before merge/tag/release**
+- [ ] After release: `gem install ucode` works; `ucode --version` prints `0.1.1`
+- [ ] GitHub Release exists at `fontist/ucode/releases/tag/v0.1.1`
+
+## References
+
+- `lib/ucode/version.rb` — version constant
+- `ucode.gemspec` — gem metadata
+- Global rule: NEVER push tags / merge to main without authorization

data/TODO.full/06-fontisan-remove-audit.md

@@ -0,0 +1,142 @@
+# 06 — fontisan: remove AuditCommand (and audit/ namespace)
+
+## Goal
+
+Strip the audit subsystem out of `fontisan`. ucode now owns font
+auditing (`ucode audit font`); fontisan's `AuditCommand`,
+`AuditLibraryCommand`, `AuditCompareCommand`, and the entire
+`lib/fontisan/audit/` and `lib/fontisan/models/audit/` namespaces
+are dead code that misleads consumers.
+
+The current `fontist-archive-private/bin/build` script still references
+`Fontisan::Commands::AuditCommand` — it'll break loudly once this lands,
+which is the point. TODO 08 wires bin/build to call `ucode audit font`
+instead.
+
+## Why now
+
+- ucode's audit is the canonical path (TODO.new 06-12 ported it).
+- fontisan's audit is unmaintained and uses a UCD-stub hack (lines
+  13-21 of `fontist-archive-private/bin/build`) that papers over the
+  fact that fontisan can't actually do UCD aggregation anymore.
+- Leaving it creates confusion: which is the source of truth?
+
+## Scope
+
+The audit removal touches these paths in `fontist/fontisan`:
+
+### Code
+
+```
+lib/fontisan/audit.rb                          # DELETE
+lib/fontisan/audit/                            # DELETE entire directory
+  cli/
+  formatters/
+  library_auditor.rb
+  differ.rb
+  face_card.rb
+  extractor.rb
+  ...
+lib/fontisan/models/audit.rb                   # DELETE
+lib/fontisan/models/audit/                     # DELETE entire directory
+lib/fontisan/formatters/audit_text_renderer.rb # DELETE
+lib/fontisan/formatters/audit_diff_text_renderer.rb # DELETE
+lib/fontisan/commands/audit_command.rb         # DELETE
+lib/fontisan/commands/audit_compare_command.rb # DELETE
+lib/fontisan/commands/audit_library_command.rb # DELETE
+```
+
+### Tests
+
+```
+spec/fontisan/audit/                           # DELETE
+spec/fontisan/models/audit/                    # DELETE
+spec/fontisan/commands/audit_command_spec.rb   # DELETE
+spec/fontisan/commands/audit_compare_command_spec.rb  # DELETE
+spec/fontisan/commands/audit_library_command_spec.rb  # DELETE
+spec/fontisan/cli/audit_cli_spec.rb            # DELETE
+spec/fontisan/formatters/audit_text_renderer_spec.rb  # DELETE
+spec/fontisan/formatters/audit_diff_text_renderer_spec.rb  # DELETE
+```
+
+### CLI registration
+
+`lib/fontisan/cli.rb` registers audit subcommands. Remove those
+registrations — keep `convert`, `info`, `dump-table`, `features`,
+`glyphs`, `export`.
+
+### Documentation
+
+- `README.md` — remove audit references; slim to "fontisan parses
+  fonts and converts between formats"
+- `docs/` — delete `audit.md`, `audit-format.md`, etc.
+
+## Migration path
+
+Consumers currently calling `Fontisan::Commands::AuditCommand.new(path).run`
+should switch to `ucode audit font <path>`. The output YAML shape is
+identical (ucode's audit was ported from fontisan's); only the tool
+name changes.
+
+Document this in the CHANGELOG entry:
+
+```markdown
+## 0.3.0 — 2026-XX-XX
+
+### Removed — Audit subsystem moved to ucode
+
+The audit pipeline (`AuditCommand`, `AuditLibraryCommand`,
+`AuditCompareCommand`, and all `lib/fontisan/audit/` support) has
+been removed. ucode now owns font auditing.
+
+Migration:
+
+  # Before (fontisan 0.2.x):
+  Fontisan::Commands::AuditCommand.new(path).run
+
+  # After (ucode 0.1.1+):
+  bundle exec ucode audit font <path>
+  # or via API:
+  Ucode::Commands::Audit::FontCommand.new.call(path: path)
+
+The audit YAML output shape is unchanged; existing consumers don't
+need to update their parsers.
+
+For the CI pipeline that runs audits per formula, see
+fontist-archive-private's `bin/build` (now calls `ucode audit font`).
+```
+
+## Version bump
+
+This is a breaking change → **minor version bump** (0.2.22 → 0.3.0).
+Per SemVer: removing public APIs is a breaking change in 0.x.
+
+## Acceptance
+
+- [ ] `lib/fontisan/audit.rb` and `lib/fontisan/audit/` are deleted
+- [ ] `lib/fontisan/models/audit.rb` and `lib/fontisan/models/audit/` deleted
+- [ ] `lib/fontisan/commands/audit_*.rb` deleted (3 files)
+- [ ] `lib/fontisan/formatters/audit_*.rb` deleted (2 files)
+- [ ] `lib/fontisan/cli.rb` no longer registers audit subcommands
+- [ ] All audit-related specs deleted
+- [ ] `bundle exec rspec` passes (existing fontisan specs unrelated to audit)
+- [ ] `bundle exec rubocop` clean
+- [ ] `fontisan --help` no longer shows audit subcommands
+- [ ] CHANGELOG entry documents the removal + migration path
+- [ ] Version bumped to 0.3.0
+- [ ] PR opened against main
+
+## Dependencies / blockers
+
+- **TODO.new 06-12** — these ported fontisan's audit subsystem into ucode.
+  Verify that port is complete before this removal lands (otherwise the
+  audit functionality disappears entirely).
+- **TODO 08** — `fontist-archive-private/bin/build` will break when this
+  merges. TODO 08 should land first OR in the same PR-bundle.
+
+## References
+
+- `lib/fontisan/audit.rb` (slated for deletion)
+- [TODO.new/06](../TODO.new/06-audit-namespace-skeleton.md) — ucode audit port (must be complete)
+- [TODO 07](07-fontisan-remove-ucd.md) — companion cleanup (UCD/UCDXML removal)
+- [TODO 08](08-archive-private-bin-build.md) — pipeline migration