ucode 0.1.0 → 0.1.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: a3b414e4e08c27e99b6d90ff8af3c62a8be0b4ac753c1f856bff3bf0558b7a73
-  data.tar.gz: 0afafb185a45a754b2fca7fad30a8b95f75c9f36861c20c6619f7e767af47d72
+  metadata.gz: 2bf2b84f7b6350471c31bd58b724534f5bde2a0c1e2bf44cfe41cd36a8ab6165
+  data.tar.gz: 7506b44a00233009e01dd5668d375082ec42f9ab2c6e3d0d319722949599496a
 SHA512:
-  metadata.gz: 1baf8cfab570ef9113c54ba978f191226bd025f01b3bc7a565b1a4cd2ca3f54030b74e046c254c6e94089f5cc3582b8d8244a852021c92154d4ffcd872b2d729
-  data.tar.gz: 332d8247601762a92319f31301b88b69af264f75bc09e01f52c14ecfd6d25969e1d4706c1030b36116b1941b1e65b1728ed44870804492ac00c4ae3e6941fccb
+  metadata.gz: 217b105ac13f10fcdaa51b39dbb23f8db3ba2b2026a0c9db4533df4ffc0434a06b1cbe89f078b6be2faa7603e0b8bf06b9a2e1ea5c775480cb213822ec8456e7
+  data.tar.gz: 1a0866921d181bb0b1cb8aa5113ea887a5a33eb1e0661d4d40eeec00e341d4b07cfb98f5ba9127b7bbcf834d79394c713a1f44a414b2759ba4bc08e96ef491db

data/CHANGELOG.md

@@ -0,0 +1,72 @@
+# Changelog
+
+All notable changes to ucode will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [0.1.1] — 2026-06-XX
+
+### Added
+
+- **BlockFeedEmitter**: emits a compact per-block Unicode data feed
+  (`unicode-blocks.json`, `unicode-version.json`, `unicode/blocks/<slug>.json`).
+  Renamed from `FontistConsumerEmitter` — the data is plain Unicode
+  data, not consumer-specific.
+- **Schema**: `schema/block-feed.output.schema.yml` documents the
+  canonical shape of the block-feed output as a YAML-encoded JSON
+  Schema. Acts as the contract between ucode (producer) and any
+  consumer of the feed.
+- **Categorized Unihan model**: 8 typed collections matching the
+  Unihan standard file structure (Dictionary Indices, Readings,
+  Variants, Numeric Values, Radical-Stroke Counts, Dictionary-Like
+  Data, IRG Sources, Other Mappings). Each category is a collection
+  of `UnihanField { name, values }` records.
+- **Real-font Tier 1 source map** for the universal glyph set (~17
+  specialists + Noto family default).
+- **Pillar 1 + Pillar 2 glyph extraction** via the 4-tier canonical
+  resolver.
+- **Per-codepoint properties from `extracted/` and `auxiliary/` UCD
+  files**: `display` (East Asian Width, Line Break Class, Vertical
+  Orientation), `break_segmentation` (Grapheme/Word/Sentence),
+  `indic` (Positional + Syllabic Category), `hangul` (Syllable Type),
+  `emoji` (6 booleans), full `binary_properties` list (now includes
+  PropList entries beyond DerivedCoreProperties).
+- **Audit subsystem** ported from fontisan: `ucode audit font`,
+  `ucode audit library`, `ucode audit compare`, `ucode audit browser`.
+- **Universal-set build infrastructure**: `ucode universal-set build`,
+  `pre-check`, `validate`, `report`.
+- **Block-feed CLI command**: `ucode block-feed` (renamed from
+  `ucode fontist-consumer`).
+
+### Fixed
+
+- `Parsers::NamedSequences` field order — real UAX#44 is
+  `Name; cp1 cp2 cp3 ...`, not the inverse.
+- `BlockFeedEmitter` canonical path — uses `blocks/<ID>/index.json`
+  (matches AggregateWriter output), not `blocks/<ID>.json`.
+- fontist.org `PropertyDetailPage.vue` route params — combining and
+  bidiclass routes were `:cc` / `:bc` but the page read `route.params.code`;
+  unified to `:code`.
+- Vite dev server case-sensitive `codepoints/` path — fetch now
+  lowercases the hex from the route URL.
+- Vue route-watcher for per-char data — top-level `await` only ran
+  once on initial mount; navigation between `/unicode/char/X` and
+  `/unicode/char/Y` left `charData` and `detail` refs stale.
+- `scrollBehavior` added to the router — page navigation now resets
+  scroll to top instead of preserving the prior page's position.
+
+### Removed
+
+- All references to "fontist-consumer" naming from ucode (now
+  "block-feed"). The data emitted is plain Unicode data, not
+  consumer-specific. Renames affect:
+  - `lib/ucode/repo/fontist_consumer_emitter.rb` → `block_feed_emitter.rb`
+  - `lib/ucode/commands/fontist_consumer.rb` → `block_feed.rb`
+  - `Repo::FontistConsumerEmitter` → `Repo::BlockFeedEmitter`
+  - `Commands::FontistConsumerCommand` → `Commands::BlockFeedCommand`
+  - CLI command `ucode fontist-consumer` → `ucode block-feed`
+
+## [0.1.0] — 2026-XX-XX
+
+Initial release.

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    ucode (0.1.0)
+    ucode (0.1.1)
       base64
       fontisan (~> 0.2)
       fontist (~> 3.0)
@@ -393,7 +393,7 @@ CHECKSUMS
   thor (1.5.0) sha256=e3a9e55fe857e44859ce104a84675ab6e8cd59c650a49106a05f55f136425e73
   track_open_instances (0.1.15) sha256=7f0e48821e6b4c881daaa40fb1583e308937c22a9c84883c150b399c3b5c3029
   tzinfo (2.0.6) sha256=8daf828cc77bcf7d63b0e3bdb6caa47e2272dcfaf4fbfe46f8c3a9df087a829b
-  ucode (0.1.0)
+  ucode (0.1.1)
   unibuf (0.1.2) sha256=2453cb9ff97b4a80ebb6ed4fec88669da847decf23f4f2b2d2490dcb270fc061
   unicode-display_width (3.2.0) sha256=0cdd96b5681a5949cdbc2c55e7b420facae74c4aaf9a9815eee1087cb1853c42
   unicode-emoji (4.2.0) sha256=519e69150f75652e40bf736106cfbc8f0f73aa3fb6a65afe62fefa7f80b0f80f

data/TODO.full/00-README.md

@@ -0,0 +1,116 @@
+# TODO.full — Master plan: panglyph + ucode release + fontisan cleanup + full pipeline
+
+This is the **shipping plan** for the full Fontist Unicode 17 stack:
+
+1. **ucode** — Unicode data + audit tool + universal glyph set (SVGs)
+2. **panglyph** (NEW) — assembles ucode's universal set into a single redistributable font
+3. **fontisan** — slim font parsing + WOFF conversion library (audit + UCD removed)
+4. **fontist-archive-private** — CI matrix runs ucode audit + fontisan convert per formula
+5. **fontist-archive-public** — public artifacts: `coverage/` + `woff/` + `unicode/` + `panglyph/`
+6. **fontist.org** — per-font unicode browser using WOFF (open-license) + ucode audit (all)
+
+The plan below covers everything that remains after TODO.new/ (which built the
+infrastructure). TODO.full/ is about wiring it all into a shippable product.
+
+## Directives from user
+
+- **D0** — Define + build "Fontist universal glyph set for Unicode 17" as a
+  single font, in a new repo `panglyph`. Uses fontisan to extract outlines
+  from source fonts, assembles into one redistributable font file.
+- **D1** — Publish ucode as a patch release (0.1.0 → 0.1.1).
+- **D2** — Clean up fontisan: remove `AuditCommand` and UCD/UCDXML
+  subsystems (now in ucode). Keep: font parsing primitives, ConvertCommand.
+- **D3** — Wire `fontist-archive-private` to use fontisan (WOFF) + ucode
+  (audit) for ALL fonts. Wire `fontist-archive-public` to host all artifacts.
+- **D4** — Update fontist.org to consume fontist-archive-public: render
+  per-font glyphs from WOFF (open-license), show per-font coverage from
+  ucode audit (all fonts).
+
+## File index
+
+### Foundation
+
+- [01 — Panglyph vision: what the universal font is, why it exists](01-panglyph-vision.md)
+- [02 — Panglyph repo bootstrap (gem skeleton, CLI, CI)](02-panglyph-repo-bootstrap.md)
+- [03 — Panglyph font builder (outline extract + assemble + write)](03-panglyph-font-builder.md)
+- [04 — Panglyph publish pipeline (release artifacts to fontist-archive-public)](04-panglyph-publish-pipeline.md)
+
+### Releases
+
+- [05 — ucode 0.1.1 patch release](05-ucode-0-1-1-release.md)
+
+### Cleanup
+
+- [06 — fontisan: remove AuditCommand (and audit/ namespace)](06-fontisan-remove-audit.md)
+- [07 — fontisan: remove UCD/UCDXML subsystem](07-fontisan-remove-ucd.md)
+
+### Pipeline
+
+- [08 — fontist-archive-private bin/build uses ucode audit + fontisan convert](08-archive-private-bin-build.md)
+- [09 — fontist-archive-public structure: unicode/ + panglyph/ + coverage/ + woff/](09-archive-public-structure.md)
+
+### Consumer
+
+- [10 — fontist.org: per-font WOFF glyph rendering (open-license)](10-fontist-org-woff-glyphs.md)
+- [11 — fontist.org: per-font ucode audit rendering (ALL fonts)](11-fontist-org-audit-coverage.md)
+
+### Sequencing
+
+- [12 — Implementation order (all directives)](12-implementation-order.md)
+
+## Critical path (high-level)
+
+```
+                ┌──────────────────────────────────┐
+                │  05 ucode 0.1.1 patch release    │  ← unblocks all downstream
+                └──────────────┬───────────────────┘
+                               │
+                ┌──────────────┴───────────────────┐
+                │                                  │
+                ▼                                  ▼
+   ┌────────────────────────┐         ┌────────────────────────┐
+   │  06 fontisan audit     │         │  01–04 panglyph        │
+   │     removal            │         │     (new repo)         │
+   │  07 fontisan UCD       │         └────────────┬───────────┘
+   │     removal            │                      │
+   └────────────┬───────────┘                      │
+                │                                  │
+                ▼                                  │
+   ┌────────────────────────┐                      │
+   │  08 archive-private    │                      │
+   │     bin/build refactor │                      │
+   └────────────┬───────────┘                      │
+                │                                  │
+                ▼                                  ▼
+   ┌────────────────────────┐         ┌────────────────────────┐
+   │  09 archive-public     │◄────────┤  04 panglyph publish   │
+   │     structure          │         │     to archive-public  │
+   └────────────┬───────────┘         └────────────────────────┘
+                │
+                ▼
+   ┌────────────────────────┐
+   │  10 fontist.org WOFF   │
+   │  11 fontist.org audit  │
+   └────────────────────────┘
+```
+
+## Repositories involved
+
+| Repo | Role | Branch / state |
+|---|---|---|
+| `fontist/ucode` | Unicode data + audit tool + universal glyph set | `fix/fontist-consumer-canonical-path` (PR #43) |
+| `fontist/panglyph` (NEW) | Universal font assembler | not yet created |
+| `fontist/fontisan` | Font parser + WOFF converter | `fix/ucdxml-real-shape-parsing` (cleanup target) |
+| `fontist/fontist-archive-private` | CI build env (per-formula) | main (uses old fontisan AuditCommand) |
+| `fontist/fontist-archive-public` | Public artifacts | main (no unicode/ or panglyph/ yet) |
+| `fontist/fontist.github.io` | Consumer site | `fix/unicode-char-page-fields` (PR #45) |
+
+## Conventions
+
+- **PR-per-TODO** unless tightly coupled.
+- **Merging requires explicit user authorization per PR.**
+- **Never push tags directly.** Tag + `rake release` only after explicit user sign-off.
+- **No AI attribution** in commits, PRs, or release notes.
+- **Original block names verbatim** (`CJK_Ext_A`, never slugified) in source data.
+- **Vector-only glyph extraction.** No OCR.
+- Branch naming: `<repo-scope>/<track-slug>` (e.g. `audit/remove-audit-command`).

data/TODO.full/01-panglyph-vision.md

@@ -0,0 +1,112 @@
+# 01 — Panglyph vision: the Fontist universal Unicode 17 font
+
+## What it is
+
+**panglyph** is a single redistributable font file that covers every
+assigned Unicode 17.0 codepoint (~299,382 glyphs). It's the **materialized
+form of ucode's universal glyph set**: where ucode produces one SVG per
+codepoint (sourced from Tier 1 fonts via fontisan), panglyph assembles
+those outlines into one OpenType/TrueType font that any application can
+install and use as a Unicode 17 fallback.
+
+Think "Noto Sans for everything" — except:
+- Sourced from many Tier 1 fonts (Noto family + FSung for CJK, Lentariso
+  for Sidetic, Kedebideri for Beria Erfe, NotoSerifTaiYo, UniHieroglyphica,
+  Egyptian Text, Symbola, BabelStone, etc.)
+- One font file, multiple script sources
+- Built reproducibly from ucode's per-block coverage matrix (TODO 32 in
+  TODO.new/)
+- Open License (OFL for the assembled font, matching source licenses)
+
+## Why it exists
+
+Today, no single font covers Unicode 17. Noto Sans comes closest but
+misses:
+- Rare UC17 additions (Sidetic, Beria Erfe, Tolong Siki, Tai Yo)
+- Egyptian Hieroglyphs Extended-B (needs UniHieroglyphica v16)
+- CJK Extension J (needs FSung)
+- Symbols for Legacy Computing Supplement (needs BabelStone)
+
+Users who want "Unicode 17 everywhere" must install 10+ fonts. panglyph
+collapses that to one.
+
+## Use cases
+
+1. **Browser fallback.** Browsers can be configured to use panglyph as
+   the last-resort font. Any codepoint not covered by the active font
+   gets panglyph's outline instead of tofu.
+2. **OS-level Unicode 17 coverage.** Install once, every app gets full
+   Unicode 17 rendering.
+3. **Print/PDF embedding.** Designers can embed a single font for any
+   Unicode 17 text.
+4. **Search/indexing tools.** Text extraction tools that need glyph
+   recognition for rare scripts get a uniform source.
+5. **Fontist.org specimen rendering.** When fontist.org shows a char
+   that the active font misses, fall back to panglyph instead of tofu.
+
+## What it is NOT
+
+- **Not a replacement for source fonts.** panglyph is a fallback. Active
+  fonts (the user's chosen Noto Sans, FSung, etc.) take priority.
+- **Not a font designer's tool.** It's a redistribution mechanism.
+- **Not a copy of Noto.** Different sources, different coverage policy.
+- **Not color emoji.** Vector outlines only (same as ucode's universal
+  glyph set). Color emoji would need a separate TODO.
+
+## Source policy
+
+panglyph is assembled from ucode's universal-set manifest
+(`output/universal_glyph_set/manifest.json`). For each codepoint:
+
+1. Look up the Tier 1 source font (per `config/unicode17_universal_glyph_set.yml`)
+2. Open the source font via fontisan
+3. Extract the glyf outline (or CFF charstring for OTF) for the codepoint's GID
+4. Copy the outline into panglyph's glyf table at the same GID
+
+Tier 2 (PDF-embedded extraction via correlate-v4 generalization) and
+Tier 3 (Last Resort tofu) are fallbacks when Tier 1 is unavailable.
+Tier 3 produces the recognizable "box with codepoint hex label" glyph
+familiar from Last Resort Font.
+
+## Licensing
+
+panglyph's assembled font is **OFL**. Every source font in the universal
+set must be OFL (or compatible — Apache, MIT, BSD, CC0, UFL, Bitstream,
+GUST, CC-BY). Specialist fonts with proprietary licenses cannot be
+included; their codepoints fall back to pillar 2 or pillar 3.
+
+This is enforced at ucode's universal-set pre-check (TODO 35 in TODO.new/).
+
+## Output formats
+
+| Format | Purpose |
+|---|---|
+| `panglyph-unicode17.ttf` | Installable system font (largest compatibility) |
+| `panglyph-unicode17.woff2` | Web font (smaller, used by fontist.org) |
+| `panglyph-unicode17.otf` | CFF-based variant (smaller for CJK-heavy ranges) |
+
+All three are produced by the build pipeline.
+
+## Versioning
+
+- **`panglyph-unicode17-17.0.0.ofl`** — pinned to UCD 17.0.0
+- **`panglyph-unicode17-17.0.1.ofl`** — patch release (e.g. fixed an
+  extraction bug); same Unicode data, regenerated glyphs
+- **`panglyph-unicode17-17.1.0.ofl`** — minor (new Tier 1 fonts added)
+- **`panglyph-unicode18-18.0.0.ofl`** — major (UCD 18 baseline)
+
+The first version tag is `v17.0.0` to match UCD.
+
+## Deliverables
+
+- One redistributable font file (TTF + WOFF2 + OTF)
+- A SHA-256 manifest of source contributions (provenance)
+- An OFL license file
+- A coverage report (per-block % sourced from Tier 1 / Pillar 2 / Pillar 3)
+
+## References
+
+- [TODO.new/32](../TODO.new/32-uc17-coverage-matrix.md) — Tier 1 source policy
+- [TODO.new/35](../TODO.new/35-universal-set-production-run.md) — universal-set SVGs (panglyph input)
+- [TODO.full/02](02-panglyph-repo-bootstrap.md) — repo skeleton
+- [TODO.full/03](03-panglyph-font-builder.md) — build implementation

data/TODO.full/02-panglyph-repo-bootstrap.md

@@ -0,0 +1,184 @@
+# 02 — Panglyph repo bootstrap
+
+## Goal
+
+Create the `fontist/panglyph` repository skeleton: gem structure, CLI
+entry point, CI workflow, README. No build logic yet (that's TODO 03) —
+just the scaffold that lets development proceed.
+
+## Why a separate repo
+
+panglyph consumes ucode's universal set as INPUT and produces a font
+as OUTPUT. The assembly logic (fontisan outline extraction + font
+writing + OFL packaging) is a distinct concern from ucode's Unicode
+data + audit work.
+
+Separation keeps:
+- ucode focused on Unicode data + audit
+- panglyph focused on font assembly
+- fontisan focused on parsing + conversion primitives
+
+## Repository structure
+
+```
+panglyph/
+├── README.md                          # what panglyph is, how to build
+├── LICENSE                            # OFL for the assembled font
+├── CHANGELOG.md                       # version history
+├── Gemfile                            # dev deps
+├── panglyph.gemspec                   # gem metadata
+├── lib/
+│   ├── panglyph.rb                    # top-level namespace + autoloads
+│   └── panglyph/
+│       ├── version.rb                 # VERSION = "17.0.0"
+│       ├── cli.rb                     # Thor CLI entry point
+│       ├── builder.rb                 # orchestrates the full build
+│       ├── outline_extractor.rb       # extracts glyf/CFF outline via fontisan
+│       ├── font_assembler.rb          # assembles outlines into a font
+│       ├── woff2_writer.rb            # converts TTF → WOFF2 (via fontisan)
+│       ├── manifest_reader.rb         # reads ucode's universal-set manifest
+│       ├── coverage_report.rb         # emits per-block source breakdown
+│       └── publisher.rb               # pushes artifacts to fontist-archive-public
+├── exe/
+│   └── panglyph                       # CLI executable
+├── spec/
+│   ├── spec_helper.rb
+│   └── panglyph/
+│       ├── builder_spec.rb
+│       ├── outline_extractor_spec.rb
+│       └── ...
+├── data/
+│   └── OFL.txt                        # OFL license template
+├── docs/
+│   ├── architecture.md                # build pipeline reference
+│   └── coverage-policy.md             # which fonts cover which blocks
+└── .github/
+    └── workflows/
+        └── build.yml                  # CI: build panglyph on tag push
+```
+
+## CLI surface
+
+```
+$ panglyph --help
+panglyph commands:
+  panglyph build [UCD_VERSION]      # Build panglyph-unicode<version>.<ext>
+  panglyph help [COMMAND]           # Describe subcommands
+  panglyph manifest [UCD_VERSION]   # Print source contributions manifest
+  panglyph publish [VERSION]        # Publish built artifacts to archive-public
+  panglyph validate [FONT_PATH]     # Verify the built font against the universal set
+  panglyph version                  # Print panglyph version
+```
+
+### `panglyph build`
+
+```
+$ panglyph build 17.0.0
+→ reads ucode's universal-set manifest (must already be built)
+→ for each codepoint: extracts outline from the Tier 1 source font
+→ assembles outlines into a TTF in memory
+→ writes panglyph-unicode17-17.0.0.ttf
+→ converts to panglyph-unicode17-17.0.0.woff2
+→ emits coverage-report.json (per-block source breakdown)
+```
+
+### `panglyph validate`
+
+```
+$ panglyph validate panglyph-unicode17-17.0.0.ttf
+→ cmap-walks the built font
+→ compares against ucode's universal-set codepoint list
+→ reports: 299382 codepoints, 297415 covered (99.3%), 1967 missing
+→ lists missing codepoints with their Tier 1 source (so the build can be fixed)
+```
+
+### `panglyph publish`
+
+```
+$ panglyph publish 17.0.0
+→ clones fontist-archive-public (shallow)
+→ copies panglyph-unicode17-17.0.0.{ttf,woff2} to archive-public/panglyph/
+→ updates archive-public/panglyph/manifest.json
+→ commits + pushes via GHA bot token
+```
+
+## Dependencies
+
+```ruby
+# panglyph.gemspec
+spec.add_dependency "fontisan", "~> 0.3"   # font parsing + writing primitives
+spec.add_dependency "ucode", "~> 0.1"      # universal-set manifest reader
+spec.add_dependency "thor", "~> 1.3"       # CLI
+spec.add_dependency "json", "~> 2.0"
+spec.add_dependency "rubyzip", "~> 2.3"    # OFL packaging
+```
+
+fontisan needs font-WRITING primitives added (it currently only reads +
+converts). TODO 03 lists what fontisan needs to expose.
+
+## CI workflow
+
+`.github/workflows/build.yml`:
+
+```yaml
+name: Build panglyph
+
+on:
+  push:
+    tags: ['v*']
+  workflow_dispatch:
+    inputs:
+      ucd_version:
+        description: 'UCD version to build (e.g. 17.0.0)'
+        required: true
+        default: '17.0.0'
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: ruby/setup-ruby@v1
+        with:
+          ruby-version: '3.4'
+          bundler-cache: true
+      - name: Fetch ucode universal set
+        run: |
+          git clone --depth 1 https://github.com/fontist/ucode ucode-repo
+          cd ucode-repo && bundle exec ucode universal-set build ${{ inputs.ucd_version }}
+          tar czf /tmp/universal-set.tar.gz output/universal_glyph_set/
+      - name: Build panglyph
+        run: |
+          bundle exec panglyph build ${{ inputs.ucd_version }} \
+            --universal-set=/tmp/universal-set.tar.gz
+      - name: Validate
+        run: bundle exec panglyph validate panglyph-unicode*.ttf
+      - name: Publish to fontist-archive-public
+        env:
+          GH_TOKEN: ${{ secrets.ARCHIVE_PUBLIC_BOT_TOKEN }}
+        run: bundle exec panglyph publish ${{ inputs.ucd_version }}
+      - uses: actions/upload-artifact@v4
+        with:
+          name: panglyph-${{ inputs.ucd_version }}
+          path: |
+            panglyph-unicode17-*.ttf
+            panglyph-unicode17-*.woff2
+            coverage-report.json
+```
+
+## Acceptance
+
+- [ ] `fontist/panglyph` repo exists
+- [ ] README.md explains what panglyph is + how to build (per TODO 01)
+- [ ] LICENSE is OFL
+- [ ] `bundle exec panglyph version` prints `17.0.0`
+- [ ] `bundle exec panglyph --help` lists build/manifest/publish/validate
+- [ ] CI workflow file exists and is syntactically valid
+- [ ] One trivial spec passes (`spec/panglyph/version_spec.rb`)
+- [ ] Repo is public
+
+## References
+
+- [TODO 01](01-panglyph-vision.md) — vision
+- [TODO 03](03-panglyph-font-builder.md) — build implementation
+- [TODO.new/35](../TODO.new/35-universal-set-production-run.md) — input format