ucode 0.1.0 → 0.1.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (174) hide show
  1. checksums.yaml +4 -4
  2. data/CHANGELOG.md +72 -0
  3. data/Gemfile.lock +2 -2
  4. data/TODO.full/00-README.md +116 -0
  5. data/TODO.full/01-panglyph-vision.md +112 -0
  6. data/TODO.full/02-panglyph-repo-bootstrap.md +184 -0
  7. data/TODO.full/03-panglyph-font-builder.md +201 -0
  8. data/TODO.full/04-panglyph-publish-pipeline.md +126 -0
  9. data/TODO.full/05-ucode-0-1-1-release.md +139 -0
  10. data/TODO.full/06-fontisan-remove-audit.md +142 -0
  11. data/TODO.full/07-fontisan-remove-ucd.md +125 -0
  12. data/TODO.full/08-archive-private-bin-build.md +143 -0
  13. data/TODO.full/09-archive-public-structure.md +164 -0
  14. data/TODO.full/10-fontist-org-woff-glyphs.md +131 -0
  15. data/TODO.full/11-fontist-org-audit-coverage.md +140 -0
  16. data/TODO.full/12-implementation-order.md +216 -0
  17. data/TODO.full/13-fontisan-font-writer-api.md +189 -0
  18. data/TODO.full/14-fontisan-table-writers.md +66 -0
  19. data/TODO.full/15-panglyph-builder-real.md +82 -0
  20. data/TODO.full/16-archive-public-sync-workflows.md +167 -0
  21. data/TODO.full/17-fontist-org-font-picker.md +73 -0
  22. data/TODO.full/18-comprehensive-spec-coverage.md +64 -0
  23. data/TODO.full/19-ucode-0-1-2-patch.md +32 -0
  24. data/TODO.full/20-fontisan-0-2-23-release.md +52 -0
  25. data/TODO.new/00-README.md +30 -0
  26. data/TODO.new/23-universal-glyph-set-source-map.md +312 -0
  27. data/TODO.new/24-universal-glyph-set-build.md +189 -0
  28. data/TODO.new/25-font-audit-against-universal-set.md +195 -0
  29. data/TODO.new/26-missing-glyph-reporter.md +189 -0
  30. data/TODO.new/27-fontist-org-consumer-integration.md +200 -0
  31. data/TODO.new/28-implementation-order-update.md +187 -0
  32. data/TODO.new/29-universal-set-curation-uc17.md +312 -0
  33. data/TODO.new/30-tier1-font-acquisition.md +241 -0
  34. data/TODO.new/31-universal-set-production-build.md +205 -0
  35. data/TODO.new/32-uc17-coverage-matrix.md +165 -0
  36. data/TODO.new/33-specialist-font-acquisition-refresh.md +138 -0
  37. data/TODO.new/34-pillar2-content-stream-correlator.md +147 -0
  38. data/TODO.new/35-universal-set-production-run.md +160 -0
  39. data/TODO.new/36-per-font-coverage-audit.md +145 -0
  40. data/TODO.new/37-coverage-highlight-reporter.md +125 -0
  41. data/TODO.new/38-fontist-org-glyph-consumer.md +141 -0
  42. data/TODO.new/39-implementation-order-update-32-38.md +258 -0
  43. data/TODO.new/40-archive-private-uses-ucode-audit.md +124 -0
  44. data/TODO.new/41-ucode-unicode-archive-bridge.md +160 -0
  45. data/config/specialist_fonts.yml +102 -0
  46. data/config/unicode17_tier1_fonts.yml +42 -0
  47. data/config/unicode17_universal_glyph_set.yml +293 -0
  48. data/lib/ucode/audit/block_aggregator.rb +57 -29
  49. data/lib/ucode/audit/browser/face_page.rb +128 -0
  50. data/lib/ucode/audit/browser/glyph_panel.rb +124 -0
  51. data/lib/ucode/audit/browser/library_page.rb +74 -0
  52. data/lib/ucode/audit/browser/missing_glyph_page.rb +87 -0
  53. data/lib/ucode/audit/browser/template.rb +47 -0
  54. data/lib/ucode/audit/browser/templates/face.css +200 -0
  55. data/lib/ucode/audit/browser/templates/face.html.erb +41 -0
  56. data/lib/ucode/audit/browser/templates/face.js +298 -0
  57. data/lib/ucode/audit/browser/templates/library.css +119 -0
  58. data/lib/ucode/audit/browser/templates/library.html.erb +42 -0
  59. data/lib/ucode/audit/browser/templates/library.js +99 -0
  60. data/lib/ucode/audit/browser/templates/missing_glyph_page.css +119 -0
  61. data/lib/ucode/audit/browser/templates/missing_glyph_page.html.erb +58 -0
  62. data/lib/ucode/audit/browser/templates/missing_glyph_page.js +2 -0
  63. data/lib/ucode/audit/browser.rb +32 -0
  64. data/lib/ucode/audit/context.rb +27 -1
  65. data/lib/ucode/audit/coverage_reference.rb +103 -0
  66. data/lib/ucode/audit/differ.rb +121 -0
  67. data/lib/ucode/audit/emitter/block_emitter.rb +52 -0
  68. data/lib/ucode/audit/emitter/codepoint_emitter.rb +87 -0
  69. data/lib/ucode/audit/emitter/collection_emitter.rb +80 -0
  70. data/lib/ucode/audit/emitter/face_directory.rb +212 -0
  71. data/lib/ucode/audit/emitter/glyph_emitter.rb +48 -0
  72. data/lib/ucode/audit/emitter/index_emitter.rb +149 -0
  73. data/lib/ucode/audit/emitter/library_emitter.rb +96 -0
  74. data/lib/ucode/audit/emitter/paths.rb +312 -0
  75. data/lib/ucode/audit/emitter/plane_emitter.rb +29 -0
  76. data/lib/ucode/audit/emitter/script_emitter.rb +29 -0
  77. data/lib/ucode/audit/emitter.rb +29 -0
  78. data/lib/ucode/audit/extractors/aggregations.rb +31 -2
  79. data/lib/ucode/audit/face_auditor.rb +86 -0
  80. data/lib/ucode/audit/formatters/audit_diff_text.rb +112 -0
  81. data/lib/ucode/audit/formatters/audit_text.rb +411 -0
  82. data/lib/ucode/audit/formatters/color.rb +48 -0
  83. data/lib/ucode/audit/formatters/library_summary_text.rb +98 -0
  84. data/lib/ucode/audit/formatters/text_formatter.rb +83 -0
  85. data/lib/ucode/audit/formatters.rb +23 -0
  86. data/lib/ucode/audit/library_aggregator.rb +86 -0
  87. data/lib/ucode/audit/library_auditor.rb +105 -0
  88. data/lib/ucode/audit/release/emitter.rb +152 -0
  89. data/lib/ucode/audit/release/face_card.rb +93 -0
  90. data/lib/ucode/audit/release/formula_audits.rb +50 -0
  91. data/lib/ucode/audit/release/library_index_builder.rb +78 -0
  92. data/lib/ucode/audit/release/manifest_builder.rb +127 -0
  93. data/lib/ucode/audit/release.rb +42 -0
  94. data/lib/ucode/audit/ucd_only_reference.rb +81 -0
  95. data/lib/ucode/audit/universal_set_reference.rb +136 -0
  96. data/lib/ucode/audit.rb +31 -0
  97. data/lib/ucode/cli.rb +339 -33
  98. data/lib/ucode/commands/audit/browser_command.rb +82 -0
  99. data/lib/ucode/commands/audit/collection_command.rb +103 -0
  100. data/lib/ucode/commands/audit/compare_command.rb +188 -0
  101. data/lib/ucode/commands/audit/font_command.rb +140 -0
  102. data/lib/ucode/commands/audit/library_command.rb +87 -0
  103. data/lib/ucode/commands/audit/reference_builder.rb +64 -0
  104. data/lib/ucode/commands/audit.rb +20 -0
  105. data/lib/ucode/commands/block_feed.rb +73 -0
  106. data/lib/ucode/commands/canonical_build.rb +138 -0
  107. data/lib/ucode/commands/fetch.rb +37 -1
  108. data/lib/ucode/commands/release.rb +115 -0
  109. data/lib/ucode/commands/universal_set.rb +211 -0
  110. data/lib/ucode/commands.rb +5 -0
  111. data/lib/ucode/coordinator/indices.rb +11 -0
  112. data/lib/ucode/coordinator.rb +138 -5
  113. data/lib/ucode/error.rb +30 -2
  114. data/lib/ucode/fetch/font_fetcher/result.rb +39 -0
  115. data/lib/ucode/fetch/font_fetcher.rb +16 -0
  116. data/lib/ucode/fetch/specialist_font_fetcher.rb +280 -0
  117. data/lib/ucode/fetch.rb +7 -3
  118. data/lib/ucode/glyphs/real_fonts/cmap_cache.rb +74 -0
  119. data/lib/ucode/glyphs/real_fonts.rb +1 -0
  120. data/lib/ucode/glyphs/resolver.rb +62 -0
  121. data/lib/ucode/glyphs/source.rb +48 -0
  122. data/lib/ucode/glyphs/source_builder.rb +61 -0
  123. data/lib/ucode/glyphs/source_config/coverage_assertion.rb +79 -0
  124. data/lib/ucode/glyphs/source_config/gap_report.rb +54 -0
  125. data/lib/ucode/glyphs/source_config.rb +104 -0
  126. data/lib/ucode/glyphs/sources/pillar1_embedded_tounicode.rb +63 -0
  127. data/lib/ucode/glyphs/sources/pillar3_last_resort.rb +51 -0
  128. data/lib/ucode/glyphs/sources/tier1_real_font.rb +104 -0
  129. data/lib/ucode/glyphs/sources.rb +20 -0
  130. data/lib/ucode/glyphs/universal_set/builder.rb +161 -0
  131. data/lib/ucode/glyphs/universal_set/coverage_report.rb +139 -0
  132. data/lib/ucode/glyphs/universal_set/idempotency.rb +86 -0
  133. data/lib/ucode/glyphs/universal_set/manifest_accumulator.rb +195 -0
  134. data/lib/ucode/glyphs/universal_set/manifest_writer.rb +61 -0
  135. data/lib/ucode/glyphs/universal_set/pre_build_check.rb +197 -0
  136. data/lib/ucode/glyphs/universal_set/validator.rb +204 -0
  137. data/lib/ucode/glyphs/universal_set.rb +45 -0
  138. data/lib/ucode/glyphs.rb +6 -0
  139. data/lib/ucode/models/audit/baseline.rb +6 -0
  140. data/lib/ucode/models/audit/block_summary.rb +7 -0
  141. data/lib/ucode/models/audit/codepoint_provenance.rb +39 -0
  142. data/lib/ucode/models/audit/release_face.rb +42 -0
  143. data/lib/ucode/models/audit/release_formula.rb +33 -0
  144. data/lib/ucode/models/audit/release_manifest.rb +43 -0
  145. data/lib/ucode/models/audit/release_universal_set.rb +37 -0
  146. data/lib/ucode/models/audit.rb +9 -0
  147. data/lib/ucode/models/block.rb +2 -0
  148. data/lib/ucode/models/build_report.rb +109 -0
  149. data/lib/ucode/models/codepoint/glyph.rb +42 -0
  150. data/lib/ucode/models/codepoint.rb +3 -0
  151. data/lib/ucode/models/glyph_source.rb +86 -0
  152. data/lib/ucode/models/glyph_source_map.rb +138 -0
  153. data/lib/ucode/models/specialist_font.rb +70 -0
  154. data/lib/ucode/models/specialist_font_manifest.rb +48 -0
  155. data/lib/ucode/models/unihan_entry.rb +81 -9
  156. data/lib/ucode/models/unihan_field.rb +21 -0
  157. data/lib/ucode/models/universal_set_entry.rb +47 -0
  158. data/lib/ucode/models/universal_set_manifest.rb +78 -0
  159. data/lib/ucode/models/validation_report.rb +99 -0
  160. data/lib/ucode/models.rb +9 -0
  161. data/lib/ucode/parsers/named_sequences.rb +5 -5
  162. data/lib/ucode/parsers/unihan.rb +50 -19
  163. data/lib/ucode/repo/aggregate_writer.rb +34 -2
  164. data/lib/ucode/repo/block_feed_emitter.rb +153 -0
  165. data/lib/ucode/repo/build_report_accumulator.rb +138 -0
  166. data/lib/ucode/repo/build_report_writer.rb +46 -0
  167. data/lib/ucode/repo/build_validator.rb +229 -0
  168. data/lib/ucode/repo/codepoint_writer.rb +50 -1
  169. data/lib/ucode/repo/paths.rb +8 -0
  170. data/lib/ucode/repo.rb +4 -0
  171. data/lib/ucode/version.rb +1 -1
  172. data/schema/block-feed.output.schema.yml +134 -0
  173. metadata +143 -2
  174. data/ucode.gemspec +0 -56
@@ -0,0 +1,140 @@
1
+ # 11 — fontist.org: per-font ucode audit rendering (ALL fonts)
2
+
3
+ ## Goal
4
+
5
+ Use ucode audit data from `fontist-archive-public/coverage/` to render
6
+ per-font coverage information on fontist.org. This works for ALL
7
+ fonts (open-license AND proprietary) because the audit YAML is
8
+ metadata only — no font file redistribution needed.
9
+
10
+ Pairs with TODO 10 (WOFF rendering for open-license). Together they
11
+ give users:
12
+ - Open-license fonts: actual glyphs + coverage stats
13
+ - Proprietary fonts: coverage stats only (no glyph rendering)
14
+
15
+ ## Why a separate TODO
16
+
17
+ Today fontist.org has `public/coverage/*.yaml` (audit data) but the
18
+ unicode browser doesn't surface it. Users can't ask:
19
+ - "Which fonts cover Sidetic?"
20
+ - "Does Inter have CJK Unified Ideographs?"
21
+ - "Show me glyphs in Noto Sans that Noto Serif is missing"
22
+
23
+ The audit data answers all of these. This TODO wires it into the UI.
24
+
25
+ ## Scope
26
+
27
+ ### Phase A — Audit data layer
28
+
29
+ 1. New module: `src/lib/fonts/coverage.ts`:
30
+ - Loads a specific font's audit YAML: `fetch('/coverage/{slug}/{PSName}.yaml')`
31
+ - Parses the YAML (using `js-yaml` — needs to add as dependency)
32
+ - Returns a typed `FontCoverage` object
33
+
34
+ 2. **Type definitions** in `src/lib/fonts/types.ts`:
35
+ ```typescript
36
+ export interface FontCoverage {
37
+ source_sha256: string
38
+ family_name: string
39
+ postscript_name: string
40
+ total_codepoints: number
41
+ total_glyphs: number
42
+ blocks: {
43
+ name: string
44
+ range: string
45
+ covered: number
46
+ total: number
47
+ fill_ratio: number
48
+ complete: boolean
49
+ }[]
50
+ unicode_scripts: string[]
51
+ // ... etc
52
+ }
53
+ ```
54
+
55
+ 3. **Per-block index** — pre-compute a reverse index:
56
+ `block_id → list of fonts that cover it (with fill ratio)`.
57
+ Updated when coverage/ syncs. Lives at
58
+ `public/coverage/by-block.json` (generated by
59
+ `fontist-archive-private/bin/sync-from-private`).
60
+
61
+ ### Phase B — Coverage badge on font pages
62
+
63
+ 4. On `/fonts/{slug}` (FontStylePage), show:
64
+ - Total codepoints covered
65
+ - Per-block coverage bars (top 10 blocks by fill ratio)
66
+ - "View in unicode browser" button (sets active font + navigates
67
+ to /unicode/block/{slug})
68
+
69
+ 5. On `/families/{family-slug}` (FontFamilyPage), show:
70
+ - Aggregate coverage across all styles in the family
71
+ - Per-style breakdown
72
+
73
+ ### Phase C — Per-font unicode browser
74
+
75
+ 6. When user selects a font in the FontPicker (TODO 10):
76
+ - Load that font's audit YAML
77
+ - For each block in `/unicode`:
78
+ - Show fill ratio (%)
79
+ - Show "View N covered chars" → navigates to block filtered
80
+ to chars the font covers
81
+
82
+ 7. In the block grid (`UnicodeBlockGrid.vue`):
83
+ - Cells with chars the active font covers: render in the WOFF (TODO 10)
84
+ - Cells with chars the active font misses: gray out + tooltip
85
+ "Not covered by {font}"
86
+
87
+ ### Phase D — "Best font per block" view
88
+
89
+ 8. New page: `/unicode/best-fonts/{block-slug}`:
90
+ - Lists fonts sorted by fill ratio for that block
91
+ - Top result is the "canonical Tier 1 font" (matches ucode's
92
+ universal-set manifest — TODO.new 32)
93
+
94
+ 9. Useful for users asking "which font should I install for
95
+ Devanagari?" — the answer is the top of this list.
96
+
97
+ ### Phase E — Coverage comparison
98
+
99
+ 10. `/compare` page — already exists for visual font comparison.
100
+ Extend to show coverage diff:
101
+ - Pick 2+ fonts
102
+ - Show side-by-side: codepoints one covers that the other misses
103
+ - Reuse TODO.new/37 (coverage highlight reporter) infrastructure
104
+
105
+ ### Phase F — Proprietary fonts
106
+
107
+ 11. For proprietary fonts (Apple, Microsoft — flagged in
108
+ `fonts.json` via license field):
109
+ - No WOFF specimen (TODO 10 doesn't apply)
110
+ - Audit YAML IS available (TODO 09)
111
+ - Pages show coverage stats only + a note "WOFF specimen not
112
+ available — font is not open-license"
113
+
114
+ ## Acceptance
115
+
116
+ - [ ] `src/lib/fonts/coverage.ts` parses audit YAMLs
117
+ - [ ] `FontStylePage` shows per-block coverage stats
118
+ - [ ] `FontPicker` selection drives coverage display on `/unicode`
119
+ - [ ] Block grid shows covered/missing cells per active font
120
+ - [ ] `/unicode/best-fonts/{block}` exists
121
+ - [ ] `/compare` page shows coverage diff
122
+ - [ ] Proprietary font pages show "coverage only" badge
123
+ - [ ] Coverage reverse-index (`coverage/by-block.json`) is generated
124
+ by sync-from-private
125
+
126
+ ## Dependencies / blockers
127
+
128
+ - **fontist-archive-public** — must have updated `coverage/` with ucode
129
+ audit shape (TODO 08 must land first)
130
+ - **TODO 09** — archive structure includes coverage/ sync
131
+ - **TODO 10** — FontPicker (Phase C uses it)
132
+
133
+ ## References
134
+
135
+ - `public/coverage/*.yaml` — audit data
136
+ - `src/pages/FontStylePage.vue` — font detail page
137
+ - `src/pages/UnicodeBlockPage.vue` — block page (per-block coverage view)
138
+ - [TODO 10](10-fontist-org-woff-glyphs.md) — WOFF rendering (pairs with this)
139
+ - [TODO.new/37](../TODO.new/37-coverage-highlight-reporter.md) — highlight reporter
140
+ - [TODO.new/36](../TODO.new/36-per-font-coverage-audit.md) — audit data layer
@@ -0,0 +1,216 @@
1
+ # 12 — Implementation order (all TODO.full directives)
2
+
3
+ ## Goal
4
+
5
+ Sequence all 11 TODOs (01-11) into a shipping plan with explicit
6
+ dependencies, parallel tracks, and PR boundaries. Each TODO is one PR
7
+ unless tightly coupled.
8
+
9
+ ## Critical path
10
+
11
+ ```
12
+ ┌────────────────────────┐
13
+ │ 05 ucode 0.1.1 release │ ← unblocks everything
14
+ └───────────┬────────────┘
15
+
16
+ ┌────────────────────┼─────────────────────┐
17
+ │ │ │
18
+ ▼ ▼ ▼
19
+ ┌────────────────────┐ ┌──────────────────┐ ┌──────────────────────┐
20
+ │ 06 fontisan audit │ │ 01-04 panglyph │ │ (TODO.new 32-41 │
21
+ │ removal │ │ bootstrap + │ │ still in flight) │
22
+ │ 07 fontisan UCD │ │ build + pub │ └──────────┬───────────┘
23
+ │ removal │ └────────┬─────────┘ │
24
+ │ (shipped as │ │ │
25
+ │ fontisan 0.3.0) │ │ │
26
+ └─────────┬──────────┘ │ │
27
+ │ │ │
28
+ ▼ ▼ ▼
29
+ ┌────────────────────┐ ┌──────────────────────────────────┐
30
+ │ 08 archive- │ │ 09 archive-public structure │
31
+ │ private │ │ (coverage + woff + unicode + │
32
+ │ bin/build │ │ panglyph all live here) │
33
+ │ refactor │ └─────────────┬────────────────────┘
34
+ └─────────┬──────────┘ │
35
+ │ │
36
+ └───────────┬───────────────┘
37
+
38
+
39
+ ┌─────────────────────────┐
40
+ │ 10 fontist.org WOFF │
41
+ │ rendering │
42
+ │ 11 fontist.org audit │
43
+ │ rendering │
44
+ └─────────────────────────┘
45
+ ```
46
+
47
+ ## Phase 1 — Foundation releases
48
+
49
+ ### Track F1 — ucode 0.1.1 patch (TODO 05)
50
+
51
+ **Repo**: `fontist/ucode`
52
+ **Branch**: `release/0.1.1`
53
+ **PR**: ucode/PR-XX
54
+
55
+ - Bump version, write CHANGELOG, run tests
56
+ - Open PR
57
+ - **HARD GATE**: wait for explicit user authorization before tag/release
58
+ - After merge: tag v0.1.1, `rake release` to rubygems
59
+
60
+ **Estimated**: 1 session to prep; release timing depends on user.
61
+
62
+ ### Track F2 — panglyph repo bootstrap (TODO 01, 02)
63
+
64
+ **Repo**: `fontist/panglyph` (NEW)
65
+ **Branch**: `main` (initial commit)
66
+ **PR**: N/A (initial repo creation)
67
+
68
+ - Create the repo on GitHub
69
+ - Bootstrap skeleton per TODO 02
70
+ - README + LICENSE (OFL) + minimal gemspec + CLI stub
71
+ - No build logic yet (TODO 03)
72
+
73
+ **Estimated**: 1 session. No external dependencies.
74
+
75
+ ### Track F3 — fontisan cleanup (TODO 06, 07)
76
+
77
+ **Repo**: `fontist/fontisan`
78
+ **Branch**: `audit/remove-audit-and-ucd` (off main, not off `fix/ucdxml-real-shape-parsing`)
79
+ **PR**: fontisan/PR-XX
80
+
81
+ - Delete audit + UCD subsystems
82
+ - Update README + CHANGELOG
83
+ - Bump version to 0.3.0
84
+ - Run specs to verify nothing else breaks
85
+
86
+ **Estimated**: 2 sessions. Should land AFTER ucode 0.1.1 (so consumers
87
+ have somewhere to migrate to).
88
+
89
+ ## Phase 2 — Pipeline wiring
90
+
91
+ ### Track P1 — fontist-archive-private refactor (TODO 08)
92
+
93
+ **Repo**: `fontist/fontist-archive-private`
94
+ **Branch**: `audit/use-ucode-and-fontisan-0-3`
95
+ **PR**: archive-private/PR-XX
96
+
97
+ - Blocked by F1 (ucode 0.1.1) + F3 (fontisan 0.3.0)
98
+ - Refactor `bin/build` to call `ucode audit font` + `fontisan convert`
99
+ - Remove UCD stub hack
100
+ - Update Gemfile + CI workflow
101
+
102
+ **Estimated**: 2 sessions.
103
+
104
+ ### Track P2 — panglyph build implementation (TODO 03)
105
+
106
+ **Repo**: `fontist/panglyph`
107
+ **Branch**: `feat/font-builder`
108
+ **PR**: panglyph/PR-XX
109
+
110
+ - Blocked by TODO.new 32-35 (universal set must exist as input)
111
+ - Implement OutlineExtractor + FontAssembler + Woff2Writer
112
+ - (May require extending fontisan with font-WRITING APIs — separate PR
113
+ to fontisan if so)
114
+
115
+ **Estimated**: 4-5 sessions. Largest piece of new code.
116
+
117
+ ### Track P3 — panglyph publish pipeline (TODO 04)
118
+
119
+ **Repo**: `fontist/panglyph`
120
+ **Branch**: `feat/publish-pipeline`
121
+ **PR**: panglyph/PR-XX
122
+
123
+ - Blocked by P2 (build must produce artifacts)
124
+ - Implement `panglyph publish` + CI workflow
125
+ - Updates fontist-archive-public/panglyph/
126
+
127
+ **Estimated**: 1 session.
128
+
129
+ ### Track P4 — fontist-archive-public structure (TODO 09)
130
+
131
+ **Repo**: `fontist/fontist-archive-public`
132
+ **Branch**: `audit/restructure-with-unicode-and-panglyph`
133
+ **PR**: archive-public/PR-XX
134
+
135
+ - Add `unicode/` directory (synced from ucode)
136
+ - Add `panglyph/` directory (synced from panglyph)
137
+ - Add three sync workflows (sync-private, sync-ucode, sync-panglyph)
138
+ - Update README to document structure
139
+
140
+ **Estimated**: 1-2 sessions.
141
+
142
+ ## Phase 3 — Consumer wiring
143
+
144
+ ### Track C1 — fontist.org WOFF rendering (TODO 10)
145
+
146
+ **Repo**: `fontist/fontist.github.io`
147
+ **Branch**: `feat/per-font-woff-rendering`
148
+ **PR**: fontist.github.io/PR-XX
149
+
150
+ - Blocked by P1 (WOFF specimens in archive-public) + P4 (structure)
151
+ - useFontFace composable + FontPicker + grid rendering
152
+
153
+ **Estimated**: 3 sessions.
154
+
155
+ ### Track C2 — fontist.org audit coverage (TODO 11)
156
+
157
+ **Repo**: `fontist/fontist.github.io`
158
+ **Branch**: `feat/per-font-audit-coverage`
159
+ **PR**: fontist.github.io/PR-XX
160
+
161
+ - Blocked by P1 (audit data in archive-public)
162
+ - Can run in parallel with C1
163
+ - Coverage data layer + per-block views + comparison
164
+
165
+ **Estimated**: 3 sessions.
166
+
167
+ ## Sequencing rules
168
+
169
+ 1. **PR-per-TODO.** No bundled PRs.
170
+ 2. **Hard gate on tag/release.** ucode 0.1.1, fontisan 0.3.0, panglyph
171
+ 17.0.0 tags all require explicit user authorization.
172
+ 3. **F3 (fontisan cleanup) must NOT land before F1 (ucode 0.1.1).**
173
+ Otherwise consumers have nowhere to migrate to.
174
+ 4. **P1 depends on F1 + F3.** Can't call `ucode audit font` if ucode
175
+ isn't published.
176
+ 5. **P2 + P3 (panglyph) can run in parallel with P1 (archive-private)
177
+ and P4 (archive-public).** Disjoint repos, no shared files.
178
+ 6. **C1 + C2 can run in parallel.** Different consumers of the same data.
179
+ 7. **External review doesn't block local progress.** If a PR is in
180
+ review, continue with the next TODO in the same track.
181
+
182
+ ## Branch naming
183
+
184
+ - ucode: `audit/<track-slug>` or `release/<version>`
185
+ - panglyph: `feat/<track-slug>` or `release/<version>`
186
+ - fontisan: `audit/<track-slug>`
187
+ - archive-private: `audit/<track-slug>`
188
+ - archive-public: `audit/<track-slug>`
189
+ - fontist.org: `feat/<track-slug>`
190
+
191
+ ## What's NOT in scope
192
+
193
+ - **Color emoji font** (Noto Color Emoji uses CBDT/CBLC bitmap tables
194
+ — panglyph would need a separate path). Separate TODO if needed.
195
+ - **Real-time glyph extraction service.** Users extracting glyphs on
196
+ demand. Out of scope; panglyph is pre-built.
197
+ - **Per-version diff visualizer.** Tracking how a codepoint's glyph
198
+ changes across Unicode versions. Useful but separate.
199
+ - **Mobile optimization of fontist.org.** Existing site is responsive;
200
+ no separate mobile work.
201
+
202
+ ## Acceptance
203
+
204
+ - [ ] Every TODO 01-11 has an assigned branch + PR-per-TODO
205
+ - [ ] Critical path is unambiguous
206
+ - [ ] Parallel tracks identified explicitly
207
+ - [ ] External dependencies (fontist/formulas, gem review) called out
208
+ - [ ] Out-of-scope items listed
209
+
210
+ ## References
211
+
212
+ - [TODO.new/39](../TODO.new/39-implementation-order-update-32-38.md) — prior sequencing
213
+ - [TODO 01](01-panglyph-vision.md) — panglyph vision
214
+ - [TODO 05](05-ucode-0-1-1-release.md) — ucode release (Phase 1)
215
+ - [TODO 08](08-archive-private-bin-build.md) — pipeline (Phase 2)
216
+ - [TODO 10](10-fontist-org-woff-glyphs.md) — consumer (Phase 3)
@@ -0,0 +1,189 @@
1
+ # 13 — fontisan FontWriter: clean API for writing fonts from scratch
2
+
3
+ ## Goal
4
+
5
+ Add a `Fontisan::FontWriter` class that lets callers assemble a new
6
+ font file (TTF/OTF) from scratch — cmap + glyf + name + metrics +
7
+ required housekeeping tables. This is the missing primitive panglyph
8
+ (TODO.full/03) needs to actually produce a font.
9
+
10
+ ## Why this is a separate TODO
11
+
12
+ fontisan today only **reads** fonts (cmap walk, glyf extraction) and
13
+ **converts** between formats (TTF → WOFF via re-serialization of
14
+ existing tables). Neither path can build a new font from scratch.
15
+
16
+ panglyph needs to:
17
+ 1. Read outlines from N source fonts (fontisan already can)
18
+ 2. Write a single merged font with those outlines + a fresh cmap
19
+
20
+ Step 2 is the gap. This TODO fills it with a clean, well-abstracted API.
21
+
22
+ ## Architectural principles
23
+
24
+ Following the project's quality bar:
25
+
26
+ ### OCP — one class per table
27
+
28
+ Each OpenType table is its own writer class, registered in a single
29
+ dispatch. Adding a new table = adding a new class, NOT modifying
30
+ existing code.
31
+
32
+ ```
33
+ Fontisan::FontWriter (orchestrator)
34
+ └── Tables::* (one per OpenType table)
35
+ ├── Head
36
+ ├── Cmap
37
+ ├── Name
38
+ ├── Hmtx
39
+ ├── Hhea
40
+ ├── Glyf
41
+ ├── Loca
42
+ ├── Maxp
43
+ ├── Os2
44
+ └── Post
45
+ ```
46
+
47
+ ### MECE — single responsibility per class
48
+
49
+ - `FontWriter` — orchestrates; holds the in-memory font model
50
+ - `Tables::Head` — knows head table layout (35 fields, byte order)
51
+ - `Tables::Cmap` — knows cmap subtable formats (4 for BMP, 12 for full)
52
+ - `Tables::Glyf` — knows TrueType outline serialization
53
+ - etc.
54
+
55
+ ### SSOT — table tags live in one place
56
+
57
+ `Fontisan::FontWriter::TABLE_TAGS` constant. Adding a new table =
58
+ adding to this constant; the orchestrator auto-discovers and calls
59
+ the right writer.
60
+
61
+ ### Model-driven — typed structures, not hashes
62
+
63
+ Outline points, name records, metrics — all typed structs/classes.
64
+ No `attribute :foo, :hash` anywhere (per global rule).
65
+
66
+ ## API design
67
+
68
+ ```ruby
69
+ # lib/fontisan/font_writer.rb
70
+
71
+ module Fontisan
72
+ class FontWriter
73
+ autoload :Tables, "fontisan/font_writer/tables"
74
+ autoload :FontModel, "fontisan/font_writer/font_model"
75
+ autoload :Outline, "fontisan/font_writer/outline"
76
+ autoload :NameRecord, "fontisan/font_writer/name_record"
77
+ autoload :Metrics, "fontisan/font_writer/metrics"
78
+
79
+ def initialize(format: :ttf)
80
+ @format = format
81
+ @model = FontModel.new
82
+ end
83
+
84
+ # @param unicode_map [Hash{Integer => Integer}] cp → gid
85
+ def set_cmap(unicode_map)
86
+ @model.cmap = unicode_map
87
+ end
88
+
89
+ # @param gid [Integer]
90
+ # @param outline [Outline] points + flags
91
+ # @param metrics [Metrics] advance width + lsb
92
+ def add_glyph(gid, outline:, metrics:)
93
+ @model.glyphs[gid] = GlyphEntry.new(outline: outline, metrics: metrics)
94
+ end
95
+
96
+ # @param records [Array<NameRecord>] language-tagged name records
97
+ def set_name_records(records)
98
+ @model.names = records
99
+ end
100
+
101
+ # @param version [String] e.g. "Version 17.0.0"
102
+ def set_version(version)
103
+ @model.font_version = version
104
+ end
105
+
106
+ # Writes the font to +path+ in the initialized format (ttf or otf).
107
+ # @return [Pathname]
108
+ def write_to(path)
109
+ Tables::Assembler.new(@model, format: @format).write(path)
110
+ end
111
+ end
112
+ end
113
+ ```
114
+
115
+ ## Typed data structures
116
+
117
+ ```ruby
118
+ # lib/fontisan/font_writer/outline.rb
119
+ class Fontisan::FontWriter::Outline < Struct.new(
120
+ :contours, :instructions, :is_composite, keyword_init: true
121
+ )
122
+ # contours: Array<Array<Point>>
123
+ # instructions: String (byte stream) or nil
124
+ # is_composite: bool — true if this glyph is a composite reference
125
+ end
126
+
127
+ # lib/fontisan/font_writer/point.rb
128
+ class Fontisan::FontWriter::Point < Struct.new(
129
+ :x, :y, :on_curve, keyword_init: true
130
+ )
131
+ end
132
+
133
+ # lib/fontisan/font_writer/name_record.rb
134
+ class Fontisan::FontWriter::NameRecord < Struct.new(
135
+ :name_id, :platform_id, :encoding_id, :language_id, :string, keyword_init: true
136
+ )
137
+ # Standard name_ids per OpenType spec (0=family, 1=subfamily, etc.)
138
+ end
139
+
140
+ # lib/fontisan/font_writer/metrics.rb
141
+ class Fontisan::FontWriter::Metrics < Struct.new(
142
+ :advance_width, :left_side_bearing, keyword_init: true
143
+ )
144
+ end
145
+
146
+ # lib/fontisan/font_writer/glyph_entry.rb
147
+ class Fontisan::FontWriter::GlyphEntry < Struct.new(
148
+ :outline, :metrics, keyword_init: true
149
+ )
150
+ end
151
+ ```
152
+
153
+ ## Scope
154
+
155
+ ### Phase A — Foundation (this TODO)
156
+
157
+ 1. Define `Fontisan::FontWriter` class with the API above.
158
+ 2. Define typed structs (Outline, Point, NameRecord, Metrics, GlyphEntry).
159
+ 3. Define `FontModel` — internal in-memory representation.
160
+ 4. Define `Tables::Assembler` — orchestrates table serialization.
161
+ 5. Implement `Tables::Head` (writes a fixed head table — simplest case).
162
+
163
+ ### Phase B — Real table writers (TODO.full/14)
164
+
165
+ 6. Implement `Tables::Cmap` (formats 4 + 12).
166
+ 7. Implement `Tables::Hmtx` + `Tables::Hhea` (metrics).
167
+ 8. Implement `Tables::Glyf` + `Tables::Loca` (TrueType outlines).
168
+ 9. Implement `Tables::Maxp`, `Tables::Os2`, `Tables::Post`, `Tables::Name`.
169
+
170
+ ### Phase C — Validation
171
+
172
+ 10. After writing, re-open with `Fontisan::Font.open` to verify the
173
+ output parses correctly (round-trip check).
174
+ 11. Validate against the OpenType spec checklist.
175
+
176
+ ## Acceptance
177
+
178
+ - [ ] `Fontisan::FontWriter` class exists with documented API
179
+ - [ ] All typed structs defined (no `:hash` attributes)
180
+ - [ ] `Tables::Head` writes a valid head table byte sequence
181
+ - [ ] Specs cover the public API + Head table serialization
182
+ - [ ] Uses Ruby autoload (no require_relative)
183
+ - [ ] No `send` to private methods; no `instance_variable_get/set`; no `respond_to?`
184
+
185
+ ## References
186
+
187
+ - [TODO.full/03](03-panglyph-font-builder.md) — consumer (panglyph)
188
+ - [TODO.full/14](14-fontisan-table-writers.md) — Phase B
189
+ - OpenType specification: https://learn.microsoft.com/en-us/typography/opentype/spec/
@@ -0,0 +1,66 @@
1
+ # 14 — fontisan table writers: real serialization for every required table
2
+
3
+ ## Goal
4
+
5
+ Implement the per-table writer classes referenced by TODO 13. After
6
+ this TODO, `Fontisan::FontWriter#write_to(path)` produces a valid
7
+ TTF file that opens in any font consumer.
8
+
9
+ ## Scope
10
+
11
+ One class per OpenType table, all under `Fontisan::FontWriter::Tables::*`:
12
+
13
+ ### `Tables::Head`
14
+ - 35 fixed fields; writes the canonical head structure
15
+ - Uses `BinData`-style binary packing (or `pack("...")` templates)
16
+
17
+ ### `Tables::Hhea` + `Tables::Hmtx`
18
+ - Hhea: 18 fixed fields
19
+ - Hmtx: per-glyph advanceWidth + lsb (long or short record based on glyph count)
20
+
21
+ ### `Tables::Maxp`
22
+ - TrueType: 16 fields; numGlyphs populated from FontModel
23
+ - CFF: 5 fields
24
+
25
+ ### `Tables::Name`
26
+ - Multi-record: name_id × platform_id × encoding_id × language_id
27
+ - Standard name_ids: 0=copyright, 1=family, 2=subfamily, 4=full name,
28
+ 5=version, 6=PostScript name
29
+
30
+ ### `Tables::OS2`
31
+ - 60+ fields (varies by version: v0=58, v5=68)
32
+ - Includes the 4 ulUnicodeRange bitfields (computed from FontModel.cmap)
33
+
34
+ ### `Tables::Post`
35
+ - Glyph names table; format 2 is most flexible
36
+ - Per-glyph PostScript name index
37
+
38
+ ### `Tables::Cmap`
39
+ - Subtable 4 (BMP, format 4) — segment mapping to delta
40
+ - Subtable 12 (full Unicode, format 12) — sparse groups
41
+ - Spliced into a single cmap table with proper platform/encoding headers
42
+
43
+ ### `Tables::Glyf` + `Tables::Loca`
44
+ - TrueType outline serialization: contour count → endpoints → flags → x-coords → y-coords → instructions
45
+ - Composite glyphs (referencing other glyphs by GID) supported
46
+ - Loca: offsets into glyf (short format if all glyphs fit in 2-byte words, long otherwise)
47
+
48
+ ### `Tables::Assembler`
49
+ - Computes table order (head + hhea + maxp + os2 + hmtx + cmap + post + loca + glyf + name)
50
+ - Writes offset table + table directory
51
+ - Aligns each table to 4-byte boundaries
52
+ - Computes checksums per table + the head checkSumAdjustment
53
+
54
+ ## Acceptance
55
+
56
+ - [ ] Each table class exists with `#bytes(model)` returning a String
57
+ - [ ] `Tables::Assembler#write(path)` produces a TTF that opens via `Fontisan::Font.open`
58
+ - [ ] All fields validate against the OpenType spec
59
+ - [ ] Composite glyphs serialize correctly
60
+ - [ ] Per-table specs cover edge cases (empty font, single glyph, full Unicode range)
61
+ - [ ] Uses Ruby autoload
62
+
63
+ ## References
64
+
65
+ - [TODO.full/13](13-fontisan-font-writer-api.md) — API design
66
+ - OpenType spec (per-table reference)
@@ -0,0 +1,82 @@
1
+ # 15 — panglyph Builder real implementation (uses FontWriter)
2
+
3
+ ## Goal
4
+
5
+ Wire panglyph's `Builder` to actually produce a TTF using
6
+ `Fontisan::FontWriter`. Replaces the stub from TODO.full/03 with a
7
+ real pipeline that:
8
+
9
+ 1. Reads ucode's universal-set manifest
10
+ 2. Opens each Tier 1 source font via `Fontisan::Font.open`
11
+ 3. Walks the cmap → glyf for each codepoint
12
+ 4. Assembles a single TTF via `Fontisan::FontWriter`
13
+ 5. Validates the output
14
+
15
+ ## Why this is a separate TODO
16
+
17
+ The skeleton classes exist (pushed in initial panglyph commit). They
18
+ warn "stub" because fontisan couldn't write fonts. TODO 13 + 14 add
19
+ that capability. This TODO makes panglyph USE it.
20
+
21
+ ## Scope
22
+
23
+ 1. Replace `OutlineExtractor#extract_many` stub with real fontisan calls:
24
+ ```ruby
25
+ font = Fontisan::Font.open(font_path, font_index:)
26
+ codepoints.each do |cp|
27
+ gid = font.cmap.unicode_map[cp]
28
+ next unless gid
29
+ outline_bytes = font.glyf.raw_bytes_for(gid)
30
+ advance_width = font.hmtx.advance_width(gid)
31
+ lsb = font.hmtx.lsb(gid)
32
+ extracted[cp] = Outline.new(
33
+ contours: parse_contours(outline_bytes),
34
+ instructions: parse_instructions(outline_bytes),
35
+ is_composite: font.glyf.composite?(gid),
36
+ )
37
+ metrics[cp] = Metrics.new(advance_width:, left_side_bearing: lsb)
38
+ end
39
+ ```
40
+
41
+ 2. Replace `FontAssembler#assemble` stub:
42
+ ```ruby
43
+ writer = Fontisan::FontWriter.new(format: :ttf)
44
+ writer.set_cmap(unicode_to_gid)
45
+ writer.set_name_records([
46
+ NameRecord.new(name_id: 1, platform_id: 3, encoding_id: 1, language_id: 0x409,
47
+ string: "panglyph Unicode #{major_version}"),
48
+ # ...
49
+ ])
50
+ writer.set_version("Version #{ucd_version}")
51
+ outlines.each do |cp, outline|
52
+ gid = unicode_to_gid[cp]
53
+ writer.add_glyph(gid, outline:, metrics: metrics[cp])
54
+ end
55
+ writer.write_to(ttf_path)
56
+ ```
57
+
58
+ 3. Real `CoverageReport` — walks the output TTF's cmap, compares to manifest.
59
+
60
+ 4. Real `Publisher` — syncs to fontist-archive-public via git.
61
+
62
+ ## Performance considerations
63
+
64
+ - 299,382 codepoints × ~1ms each = ~5 minutes for extraction
65
+ - Outline parsing dominates CPU; parallelize per source font
66
+ - Memory: hold all outlines in memory (~1.2GB estimated); use streaming
67
+ write to disk if low-memory CI runners complain
68
+
69
+ ## Acceptance
70
+
71
+ - [ ] `bundle exec panglyph build 17.0.0` produces a TTF
72
+ - [ ] Built TTF opens via `Fontisan::Font.open`
73
+ - [ ] Built TTF's cmap contains every codepoint in the manifest
74
+ - [ ] Built TTF renders in a browser (manual smoke test)
75
+ - [ ] Build completes in <30 minutes on a single runner
76
+ - [ ] No stubs remain in Builder / OutlineExtractor / FontAssembler
77
+
78
+ ## References
79
+
80
+ - [TODO.full/03](03-panglyph-font-builder.md) — original skeleton
81
+ - [TODO.full/13](13-fontisan-font-writer-api.md) — FontWriter API
82
+ - [TODO.full/14](14-fontisan-table-writers.md) — table writers