ucode 0.2.0 → 0.2.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/Gemfile +10 -2
- data/README.md +66 -20
- data/Rakefile +19 -8
- data/TODO.extract-code-chart/01-pdf-fetch-validation.md +80 -0
- data/TODO.extract-code-chart/02-block-name-resolver.md +68 -0
- data/TODO.extract-code-chart/03-codechart-namespace.md +82 -0
- data/TODO.extract-code-chart/04-codechart-extractor.md +154 -0
- data/TODO.extract-code-chart/05-provenance-and-sidecar.md +147 -0
- data/TODO.extract-code-chart/06-codechart-writer.md +134 -0
- data/TODO.extract-code-chart/07-codechart-cli.md +135 -0
- data/TODO.extract-code-chart/08-specs.md +87 -0
- data/config/unicode17_universal_glyph_set.yml +1 -1
- data/lib/ucode/audit/reference_factory.rb +1 -1
- data/lib/ucode/cli.rb +101 -0
- data/lib/ucode/code_chart/extractor.rb +120 -0
- data/lib/ucode/code_chart/provenance.rb +82 -0
- data/lib/ucode/code_chart/sidecar.rb +52 -0
- data/lib/ucode/code_chart/writer.rb +128 -0
- data/lib/ucode/code_chart.rb +39 -0
- data/lib/ucode/commands/fetch.rb +1 -1
- data/lib/ucode/commands/glyphs.rb +1 -1
- data/lib/ucode/commands/lookup.rb +1 -1
- data/lib/ucode/commands/parse.rb +1 -1
- data/lib/ucode/coordinator/indices.rb +2 -2
- data/lib/ucode/error.rb +11 -0
- data/lib/ucode/fetch/code_charts.rb +3 -4
- data/lib/ucode/fetch/http.rb +75 -7
- data/lib/ucode/glyphs/page_renderer.rb +15 -2
- data/lib/ucode/glyphs/pipeline.rb +1 -2
- data/lib/ucode/parsers/blocks.rb +34 -0
- data/lib/ucode/repo/aggregate_writer.rb +1 -1
- data/lib/ucode/repo/writers/blocks_writer.rb +13 -13
- data/lib/ucode/repo/writers/enums_writer.rb +2 -2
- data/lib/ucode/repo/writers/indexes_writer.rb +4 -4
- data/lib/ucode/repo/writers/manifest_writer.rb +4 -4
- data/lib/ucode/repo/writers/named_sequences_writer.rb +1 -1
- data/lib/ucode/repo/writers/planes_writer.rb +17 -17
- data/lib/ucode/repo/writers/relationships_writer.rb +1 -1
- data/lib/ucode/repo/writers/scripts_writer.rb +6 -6
- data/lib/ucode/repo/writers.rb +1 -1
- data/lib/ucode/version.rb +1 -1
- data/lib/ucode.rb +3 -0
- data/ucode.gemspec +6 -1
- metadata +19 -6
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: e2e60f073662cc78885f8ee6a0333a2307a8b894ddb9789b1e206f6ae63d25e3
|
|
4
|
+
data.tar.gz: 50b3984de26589d0aab193250e9c8dd3f56fd0d10a4a77a21ffdcb236c38c737
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 88d9ef3df0f99af9b3897cd429666092d48a373d9c60d3c83a6e06a59404e0523c736a8d971fbb6d6d29c9b4b80d9610cad54a64cda439326e19a80670d67ba9
|
|
7
|
+
data.tar.gz: 8146f6984defddb5b45204fe7a8852cd3051bf1474584eddc7489e5ec9fd4900d48452ac5462a1c6c2ac9b8213823ace68cebe6c5b5176dd0c25c922f8f1d4ba
|
data/Gemfile
CHANGED
|
@@ -4,8 +4,13 @@ source "https://rubygems.org"
|
|
|
4
4
|
|
|
5
5
|
gemspec
|
|
6
6
|
|
|
7
|
+
# Rake must be in the default group (not :development) because the
|
|
8
|
+
# GHA release workflow runs `bundle exec rake release` to publish the
|
|
9
|
+
# gem. The release runner installs with `--without development`, so
|
|
10
|
+
# gems in the :development group are excluded.
|
|
11
|
+
gem "rake"
|
|
12
|
+
|
|
7
13
|
group :development do
|
|
8
|
-
gem "rake"
|
|
9
14
|
gem "rspec"
|
|
10
15
|
gem "rubocop"
|
|
11
16
|
gem "rubocop-performance"
|
|
@@ -19,4 +24,7 @@ end
|
|
|
19
24
|
# local sibling checkout, set FONTISAN_PATH before running bundle.
|
|
20
25
|
# FONTISAN_PATH=../fontisan bundle install
|
|
21
26
|
gem "fontisan", path: ENV["FONTISAN_PATH"] if ENV["FONTISAN_PATH"]
|
|
22
|
-
|
|
27
|
+
# Pin fontisan to 0.2.22 — 0.2.23+ removed
|
|
28
|
+
# `Fontisan::Commands::AuditCommand` and 0.4.x removed the Audit
|
|
29
|
+
# subsystem entirely. See ucode.gemspec for the rationale.
|
|
30
|
+
gem "fontisan", "= 0.2.22" unless ENV["FONTISAN_PATH"]
|
data/README.md
CHANGED
|
@@ -2,14 +2,16 @@
|
|
|
2
2
|
|
|
3
3
|
`ucode` is a Ruby toolkit for the Unicode Character Database (UCD). It turns the
|
|
4
4
|
official UCD text files into a structured, browsable dataset: one JSON document
|
|
5
|
-
per assigned codepoint, plus a Vitepress site for navigation.
|
|
5
|
+
per assigned codepoint, plus a Vitepress site for navigation. It also extracts
|
|
6
|
+
per-codepoint SVG glyphs from the Unicode Code Charts PDFs and audits font
|
|
7
|
+
coverage against the Unicode baseline.
|
|
6
8
|
|
|
7
|
-
> **Status (v0.1).** The JSON dataset, lookup index,
|
|
8
|
-
>
|
|
9
|
-
>
|
|
10
|
-
>
|
|
9
|
+
> **Status (v0.2.1).** The JSON dataset, lookup index, Vitepress site, and
|
|
10
|
+
> 4-tier glyph extraction pipeline are production-ready. The `ucode code-chart`
|
|
11
|
+
> subcommand extracts standalone SVGs + provenance sidecars from per-block
|
|
12
|
+
> Code Charts PDFs. Font coverage auditing (`ucode audit`) is production-ready.
|
|
11
13
|
|
|
12
|
-
## What you get (v0.
|
|
14
|
+
## What you get (v0.2)
|
|
13
15
|
|
|
14
16
|
- **Per-codepoint JSON** at `output/blocks/<BLOCK>/<U+XXXX>/index.json` with
|
|
15
17
|
full UCD properties, the human-curated relationships from `NamesList.txt`
|
|
@@ -21,6 +23,14 @@ per assigned codepoint, plus a Vitepress site for navigation.
|
|
|
21
23
|
relationships, named sequences, manifest.
|
|
22
24
|
- **SQLite lookup index** for fast codepoint → block/script/char queries.
|
|
23
25
|
- **Vitepress site** at `site/` for browsing Plane → Block → Character.
|
|
26
|
+
- **4-tier glyph extraction** — per-codepoint `glyph.svg` sourced from real
|
|
27
|
+
fonts (Tier 1), PDF-embedded fonts (Pillars 1+2), or Last Resort UFO
|
|
28
|
+
(Pillar 3).
|
|
29
|
+
- **Per-block Code Chart extraction** — `ucode code-chart extract` produces
|
|
30
|
+
standalone SVG + provenance JSON for every codepoint in a block.
|
|
31
|
+
- **Font coverage audit** — `ucode audit` compares a font's cmap against the
|
|
32
|
+
Unicode baseline and reports per-block coverage, missing codepoints, and
|
|
33
|
+
optional HTML browsers.
|
|
24
34
|
|
|
25
35
|
## Install
|
|
26
36
|
|
|
@@ -31,7 +41,7 @@ gem install ucode
|
|
|
31
41
|
Or in a Gemfile:
|
|
32
42
|
|
|
33
43
|
```ruby
|
|
34
|
-
gem "ucode", "~> 0.
|
|
44
|
+
gem "ucode", "~> 0.2"
|
|
35
45
|
```
|
|
36
46
|
|
|
37
47
|
## Quick start
|
|
@@ -107,7 +117,39 @@ Then:
|
|
|
107
117
|
cd site && npm install && npm run dev
|
|
108
118
|
```
|
|
109
119
|
|
|
110
|
-
##
|
|
120
|
+
## Code Chart extraction
|
|
121
|
+
|
|
122
|
+
Extract per-codepoint SVG glyphs from a Unicode Code Charts PDF, with
|
|
123
|
+
provenance sidecar JSON:
|
|
124
|
+
|
|
125
|
+
```sh
|
|
126
|
+
# Download the Code Charts PDF for a block
|
|
127
|
+
ucode code-chart fetch --block Sidetic
|
|
128
|
+
|
|
129
|
+
# Extract every codepoint as SVG + provenance JSON
|
|
130
|
+
ucode code-chart extract --block Sidetic --to /tmp/sidetic/
|
|
131
|
+
|
|
132
|
+
# List cached Code Charts PDFs
|
|
133
|
+
ucode code-chart list
|
|
134
|
+
```
|
|
135
|
+
|
|
136
|
+
Output layout:
|
|
137
|
+
|
|
138
|
+
```
|
|
139
|
+
/tmp/sidetic/Sidetic/
|
|
140
|
+
U+10920.svg # vector glyph outline
|
|
141
|
+
U+10920.json # provenance: source PDF, sha256, version, timestamp
|
|
142
|
+
U+10921.svg
|
|
143
|
+
U+10921.json
|
|
144
|
+
...
|
|
145
|
+
```
|
|
146
|
+
|
|
147
|
+
The extractor uses the existing 4-tier glyph sourcing pipeline (Tier 1 →
|
|
148
|
+
Pillar 1 → Pillar 2 → Pillar 3). No new extraction logic — the
|
|
149
|
+
`ucode code-chart` subcommand is a thin CLI wrapper over `Ucode::CodeChart::Writer`,
|
|
150
|
+
which orchestrates `Ucode::Glyphs::Resolver` for each codepoint.
|
|
151
|
+
|
|
152
|
+
## Glyph extraction (4-tier pipeline)
|
|
111
153
|
|
|
112
154
|
The `ucode glyphs` command and the `--include-glyphs` flag on `ucode build`
|
|
113
155
|
are **opt-in and experimental in v0.1**. They emit per-codepoint `glyph.svg`
|
|
@@ -436,29 +478,33 @@ from the dataset.
|
|
|
436
478
|
|
|
437
479
|
## Architecture
|
|
438
480
|
|
|
439
|
-
|
|
481
|
+
Seven concerns, each isolated:
|
|
440
482
|
|
|
441
483
|
1. **`Ucode::Models`** — `lutaml-model` classes for every UCD aggregate.
|
|
442
484
|
2. **`Ucode::Parsers`** — one streaming parser per UCD text file.
|
|
443
485
|
3. **`Ucode::Coordinator`** — single-pass enrichment that merges indices
|
|
444
486
|
into each `CodePoint` as it streams.
|
|
445
|
-
4. **`Ucode::Repo`** — atomic, idempotent writers for the output tree
|
|
446
|
-
|
|
447
|
-
|
|
448
|
-
|
|
487
|
+
4. **`Ucode::Repo`** — atomic, idempotent writers for the output tree
|
|
488
|
+
(per-concern writers under `Repo::Writers::*`).
|
|
489
|
+
5. **`Ucode::Glyphs`** — 4-tier vector glyph extraction from Code Charts PDFs
|
|
490
|
+
(RealFonts, EmbeddedFonts, LastResort + Resolver).
|
|
491
|
+
6. **`Ucode::CodeChart`** — per-block SVG extraction + provenance sidecar
|
|
492
|
+
for the "Code Chart donor" use case.
|
|
493
|
+
7. **`Ucode::Site`** — Vitepress scaffold + config/page generator.
|
|
449
494
|
|
|
450
495
|
CLI is thin Thor dispatch over `Ucode::Commands::*`. Each command class
|
|
451
|
-
is a pure, in-process testable unit.
|
|
496
|
+
is a pure, in-process testable unit. Version resolution happens once
|
|
497
|
+
per CLI invocation and threads through to all sub-commands.
|
|
452
498
|
|
|
453
|
-
See `
|
|
454
|
-
`docs/
|
|
499
|
+
See `docs/architecture.md` for the canonical architecture reference and
|
|
500
|
+
`docs/adr/` for Architecture Decision Records.
|
|
455
501
|
|
|
456
502
|
## Authoritative source
|
|
457
503
|
|
|
458
|
-
ucode parses the **UCD text files** (
|
|
459
|
-
`
|
|
460
|
-
the human-curated relationship data
|
|
461
|
-
|
|
504
|
+
ucode parses the **UCD text files** (`UnicodeData.txt`, `NamesList.txt`,
|
|
505
|
+
`Blocks.txt`, etc.) per UAX #44. Never uses the flat XML dump — the text
|
|
506
|
+
files carry the human-curated relationship data that makes this project
|
|
507
|
+
valuable.
|
|
462
508
|
|
|
463
509
|
## License
|
|
464
510
|
|
data/Rakefile
CHANGED
|
@@ -4,15 +4,26 @@ require "rubygems"
|
|
|
4
4
|
require "rake"
|
|
5
5
|
require "bundler/gem_tasks"
|
|
6
6
|
|
|
7
|
-
|
|
8
|
-
|
|
7
|
+
begin
|
|
8
|
+
require "rspec/core/rake_task"
|
|
9
|
+
RSpec::Core::RakeTask.new(:spec)
|
|
10
|
+
rescue LoadError
|
|
11
|
+
# rspec is in the :development group; not available in the
|
|
12
|
+
# release runner (`bundle install --without development`).
|
|
13
|
+
end
|
|
9
14
|
|
|
10
|
-
|
|
11
|
-
|
|
15
|
+
begin
|
|
16
|
+
require "rubocop/rake_task"
|
|
17
|
+
RuboCop::RakeTask.new
|
|
18
|
+
rescue LoadError
|
|
19
|
+
end
|
|
12
20
|
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
|
|
21
|
+
begin
|
|
22
|
+
require "yard"
|
|
23
|
+
YARD::Rake::YardocTask.new do |t|
|
|
24
|
+
t.options = ["--output-dir", "docs/api"]
|
|
25
|
+
end
|
|
26
|
+
rescue LoadError
|
|
16
27
|
end
|
|
17
28
|
|
|
18
|
-
task default: %i[spec rubocop]
|
|
29
|
+
task default: %i[spec rubocop]
|
|
@@ -0,0 +1,80 @@
|
|
|
1
|
+
# TODO 01 — PDF fetch validation
|
|
2
|
+
|
|
3
|
+
## Status
|
|
4
|
+
|
|
5
|
+
Pending.
|
|
6
|
+
|
|
7
|
+
## Goal
|
|
8
|
+
|
|
9
|
+
Raise a typed `Ucode::CodeChartNotFoundError` when a Unicode Code
|
|
10
|
+
Charts PDF cannot be downloaded or fails content validation. The REQ
|
|
11
|
+
(R1) requires:
|
|
12
|
+
|
|
13
|
+
- HTTP 4xx / 5xx → `CodeChartNotFoundError`
|
|
14
|
+
- `Content-Type: application/pdf`
|
|
15
|
+
- First 4 bytes are `%PDF`
|
|
16
|
+
|
|
17
|
+
## Files
|
|
18
|
+
|
|
19
|
+
- `lib/ucode/error.rb` — add `Ucode::CodeChartNotFoundError` under the
|
|
20
|
+
`GlyphError` subtree.
|
|
21
|
+
- `lib/ucode.rb` — add an `autoload` for the new class so any rescue
|
|
22
|
+
clause triggers one load of `error.rb`.
|
|
23
|
+
- `lib/ucode/fetch/http.rb` — extend `Http.get` with an optional
|
|
24
|
+
`validate:` keyword. When `validate: :pdf`, after a successful
|
|
25
|
+
download, verify the `Content-Type` response header starts with
|
|
26
|
+
`application/pdf` and the first 4 bytes of the body are `%PDF`.
|
|
27
|
+
- `lib/ucode/fetch/code_charts.rb` — pass `validate: :pdf` to `Http.get`
|
|
28
|
+
for every chart PDF download.
|
|
29
|
+
- `spec/ucode/fetch/code_charts_spec.rb` (new) — cover the happy path
|
|
30
|
+
and the validation failure modes.
|
|
31
|
+
|
|
32
|
+
## Design
|
|
33
|
+
|
|
34
|
+
### Why a new error class
|
|
35
|
+
|
|
36
|
+
`FetchError` already covers transport failures, but it doesn't carry
|
|
37
|
+
"this URL produced an HTML error page / 404 / non-PDF body" semantics.
|
|
38
|
+
Splitting the type keeps existing `rescue Ucode::FetchError` callers
|
|
39
|
+
from accidentally swallowing the typed signal that "we expected a PDF
|
|
40
|
+
and didn't get one" — which is a different problem class from "the
|
|
41
|
+
network was down."
|
|
42
|
+
|
|
43
|
+
`CodeChartNotFoundError < Ucode::Error` (under `GlyphError`) reflects
|
|
44
|
+
the REQ's framing: the chart for the requested block is not
|
|
45
|
+
obtainable.
|
|
46
|
+
|
|
47
|
+
### Why `validate:` is optional on `Http.get`
|
|
48
|
+
|
|
49
|
+
`Http` is the single network boundary (per the comment at the top of
|
|
50
|
+
`http.rb`). All callers funnel through it. Adding an optional
|
|
51
|
+
keyword keeps the MECE pattern intact: non-PDF callers (UCD zip,
|
|
52
|
+
Unihan zip, font zip) pass nothing; the single PDF caller passes
|
|
53
|
+
`validate: :pdf`. No second network boundary is needed.
|
|
54
|
+
|
|
55
|
+
### Why no separate "magic bytes" check class
|
|
56
|
+
|
|
57
|
+
Magic-byte verification is 4 lines of code; extracting it into a
|
|
58
|
+
class would be ceremony. Inline check after `write_body`, raising
|
|
59
|
+
the typed error with the offending content-type or magic bytes in
|
|
60
|
+
the context payload.
|
|
61
|
+
|
|
62
|
+
## Acceptance
|
|
63
|
+
|
|
64
|
+
- `Http.get(url, dest:, validate: :pdf)` raises
|
|
65
|
+
`CodeChartNotFoundError` (a) when the response Content-Type is not
|
|
66
|
+
`application/pdf`, (b) when the first 4 bytes are not `%PDF`.
|
|
67
|
+
- `Fetch::CodeCharts.call(version, block_first_cps: [...])` raises
|
|
68
|
+
`CodeChartNotFoundError` when the unicode.org endpoint returns
|
|
69
|
+
4xx/5xx or non-PDF content.
|
|
70
|
+
- Existing callers of `Http.get` that don't pass `validate:` are
|
|
71
|
+
unchanged.
|
|
72
|
+
- Spec coverage: happy path, HTTP 404, wrong content-type, truncated
|
|
73
|
+
body missing the `%PDF` magic.
|
|
74
|
+
|
|
75
|
+
## Out of scope
|
|
76
|
+
|
|
77
|
+
- SHA-256 verification of the PDF — that's a downstream concern (the
|
|
78
|
+
Code Charts are not versioned by hash on unicode.org).
|
|
79
|
+
- Resumable / partial downloads — the existing `Http` writes a
|
|
80
|
+
`.part` then renames; that's sufficient.
|
|
@@ -0,0 +1,68 @@
|
|
|
1
|
+
# TODO 02 — Block name resolver
|
|
2
|
+
|
|
3
|
+
## Status
|
|
4
|
+
|
|
5
|
+
Pending. Depends on nothing.
|
|
6
|
+
|
|
7
|
+
## Goal
|
|
8
|
+
|
|
9
|
+
Add a class method `Ucode::Parsers::Blocks.find_by_name(name)` that
|
|
10
|
+
resolves a Unicode block identifier (e.g. `"Sidetic"`,
|
|
11
|
+
`"Egyptian_Hieroglyphs_Extended-B"`) to the `Ucode::Models::Block`
|
|
12
|
+
instance in a given version's cached `Blocks.txt`.
|
|
13
|
+
|
|
14
|
+
This is the CLI ergonomics glue: the REQ's `ucode code-chart extract
|
|
15
|
+
--block Sidetic` flow takes a human-readable name and needs to know
|
|
16
|
+
the block's range to know which `U+XXXX` codepoints to iterate.
|
|
17
|
+
|
|
18
|
+
## Files
|
|
19
|
+
|
|
20
|
+
- `lib/ucode/parsers/blocks.rb` — add `Blocks.find_by_name(path, name)`.
|
|
21
|
+
- `spec/ucode/parsers/blocks_spec.rb` — cover name lookup, missing-name,
|
|
22
|
+
case-sensitivity.
|
|
23
|
+
|
|
24
|
+
## Design
|
|
25
|
+
|
|
26
|
+
### Method shape
|
|
27
|
+
|
|
28
|
+
```ruby
|
|
29
|
+
# @param path [Pathname, String] path to a Blocks.txt
|
|
30
|
+
# @param name [String] block identifier (matches Models::Block#id)
|
|
31
|
+
# @return [Models::Block, nil] nil when no block matches
|
|
32
|
+
def find_by_name(path, name)
|
|
33
|
+
```
|
|
34
|
+
|
|
35
|
+
Returns nil for "not found" — callers (CLI, Extractor) decide whether
|
|
36
|
+
to raise. This matches `Models::Block` consumers that already expect
|
|
37
|
+
nilable lookups.
|
|
38
|
+
|
|
39
|
+
### Name matching rule
|
|
40
|
+
|
|
41
|
+
`Blocks.txt` uses `name` with whitespace collapsed to underscores
|
|
42
|
+
into `id`. `find_by_name` matches against `id` (the underscored
|
|
43
|
+
form). The REQ's example `--block Sidetic` shows that the caller
|
|
44
|
+
provides the underscored form already. This is consistent with the
|
|
45
|
+
existing `Parsers::Blocks` build logic (`name.gsub(/\s+/, "_")`).
|
|
46
|
+
|
|
47
|
+
### Why a separate method
|
|
48
|
+
|
|
49
|
+
`each_record` streams every block — the caller doesn't want to walk
|
|
50
|
+
~340 blocks for every name lookup. `find_by_name` short-circuits on
|
|
51
|
+
first match.
|
|
52
|
+
|
|
53
|
+
## Acceptance
|
|
54
|
+
|
|
55
|
+
- `find_by_name(path, "Basic_Latin")` returns the Basic Latin block.
|
|
56
|
+
- `find_by_name(path, "Nonexistent")` returns nil.
|
|
57
|
+
- Streaming still works for callers that need every block.
|
|
58
|
+
|
|
59
|
+
## Out of scope
|
|
60
|
+
|
|
61
|
+
- Fuzzy matching — exact match only. Callers validate the user's
|
|
62
|
+
input against `Parsers::Blocks.each_record(path).map(&:id)` to
|
|
63
|
+
surface "did you mean …?" suggestions if we ever want that; for
|
|
64
|
+
now, a clean `UnknownBlockError` at the call site is enough.
|
|
65
|
+
- Database-backed lookup — `Ucode::Database#block_ranges_by_name` is
|
|
66
|
+
a different concern (full UCD index). `find_by_name` operates on
|
|
67
|
+
the cached `Blocks.txt` directly because the CodeChart extractor
|
|
68
|
+
is meant to be runnable without a built database.
|
|
@@ -0,0 +1,82 @@
|
|
|
1
|
+
# TODO 03 — CodeChart namespace
|
|
2
|
+
|
|
3
|
+
## Status
|
|
4
|
+
|
|
5
|
+
Pending. Depends on TODO 02 (block name resolver) so the Extractor
|
|
6
|
+
can consume it; depends on TODO 01 (error class) so the namespace
|
|
7
|
+
can declare typed errors.
|
|
8
|
+
|
|
9
|
+
## Goal
|
|
10
|
+
|
|
11
|
+
Establish the `Ucode::CodeChart` namespace as the home for the
|
|
12
|
+
Code Charts per-codepoint extraction feature. The REQ calls this
|
|
13
|
+
namespace `Ucode::CodeChart::*`; we follow the REQ.
|
|
14
|
+
|
|
15
|
+
This is the autoload-hub file plus the autoload declaration in
|
|
16
|
+
`lib/ucode.rb`.
|
|
17
|
+
|
|
18
|
+
## Files
|
|
19
|
+
|
|
20
|
+
- `lib/ucode/code_chart.rb` — new autoload hub (defines
|
|
21
|
+
`Ucode::CodeChart` and declares child autoloads).
|
|
22
|
+
- `lib/ucode.rb` — add `autoload :CodeChart, "ucode/code_chart"` in
|
|
23
|
+
the namespace-hubs block.
|
|
24
|
+
|
|
25
|
+
## Design
|
|
26
|
+
|
|
27
|
+
### Autoload hub shape
|
|
28
|
+
|
|
29
|
+
```ruby
|
|
30
|
+
# lib/ucode/code_chart.rb
|
|
31
|
+
module Ucode
|
|
32
|
+
module CodeChart
|
|
33
|
+
autoload :Extractor, "ucode/code_chart/extractor"
|
|
34
|
+
autoload :Provenance, "ucode/code_chart/provenance"
|
|
35
|
+
autoload :Sidecar, "ucode/code_chart/sidecar"
|
|
36
|
+
autoload :Writer, "ucode/code_chart/writer"
|
|
37
|
+
end
|
|
38
|
+
end
|
|
39
|
+
```
|
|
40
|
+
|
|
41
|
+
Per the global rule (`~/.claude/CLAUDE.md`): declare autoloads in the
|
|
42
|
+
immediate parent namespace's file. `Ucode::CodeChart` is the immediate
|
|
43
|
+
parent of `Extractor`, `Provenance`, `Sidecar`, `Writer`; this file
|
|
44
|
+
is the immediate parent's file.
|
|
45
|
+
|
|
46
|
+
`Ucode` is the immediate parent of `CodeChart`; the autoload
|
|
47
|
+
declaration `autoload :CodeChart, "ucode/code_chart"` goes in
|
|
48
|
+
`lib/ucode.rb`.
|
|
49
|
+
|
|
50
|
+
### Why a new namespace (not under `Ucode::Glyphs`)
|
|
51
|
+
|
|
52
|
+
`Ucode::Glyphs::*` is the existing 4-tier sourcing pipeline
|
|
53
|
+
(`EmbeddedFonts`, `RealFonts`, `LastResort`, `Writer`). The REQ's
|
|
54
|
+
`CodeChart::*` is a feature-facing namespace that orchestrates the
|
|
55
|
+
glyphs pipeline for one specific use case (extracting from a per-block
|
|
56
|
+
PDF for the essenfont donor pipeline). Keeping the feature-facing
|
|
57
|
+
namespace separate from the implementation namespace:
|
|
58
|
+
|
|
59
|
+
- Lets callers say `Ucode::CodeChart.extract(block: "Sidetic")`
|
|
60
|
+
without first knowing about `Glyphs::EmbeddedFonts`.
|
|
61
|
+
- Makes it easy to swap the implementation later (different
|
|
62
|
+
resolution strategy, alternative PDF parser) without breaking the
|
|
63
|
+
public API.
|
|
64
|
+
- Keeps `Glyphs::` focused on tier mechanics, free of feature
|
|
65
|
+
ergonomics.
|
|
66
|
+
|
|
67
|
+
The REQ's namespace name is what we use.
|
|
68
|
+
|
|
69
|
+
## Acceptance
|
|
70
|
+
|
|
71
|
+
- `lib/ucode/code_chart.rb` exists with the autoload declarations.
|
|
72
|
+
- `lib/ucode.rb` has the new `autoload :CodeChart, "ucode/code_chart"`
|
|
73
|
+
in the namespace-hubs block.
|
|
74
|
+
- `Ucode::CodeChart` resolves to a module without loading any of its
|
|
75
|
+
children.
|
|
76
|
+
|
|
77
|
+
## Out of scope
|
|
78
|
+
|
|
79
|
+
- `Ucode::CodeChart::Command` (Thin wrapper). The CLI lives in
|
|
80
|
+
`lib/ucode/cli.rb` per the existing pattern; no separate
|
|
81
|
+
`Commands::CodeChartCommand` is introduced (single source of
|
|
82
|
+
truth for CLI dispatch).
|
|
@@ -0,0 +1,154 @@
|
|
|
1
|
+
# TODO 04 — CodeChart::Extractor
|
|
2
|
+
|
|
3
|
+
## Status
|
|
4
|
+
|
|
5
|
+
Pending. Depends on TODO 01 (error class), TODO 02 (block name
|
|
6
|
+
resolver), TODO 03 (namespace).
|
|
7
|
+
|
|
8
|
+
## Goal
|
|
9
|
+
|
|
10
|
+
`Ucode::CodeChart::Extractor` is the single entry point for
|
|
11
|
+
"extract every assigned codepoint in block X as a standalone SVG."
|
|
12
|
+
|
|
13
|
+
It orchestrates the existing 4-tier resolver (one source of truth for
|
|
14
|
+
"how do I get the SVG for a given codepoint") and returns a list of
|
|
15
|
+
extraction results — one per codepoint — that the downstream Writer
|
|
16
|
+
serializes to disk.
|
|
17
|
+
|
|
18
|
+
This is *not* a new extraction pipeline; it is the existing
|
|
19
|
+
`Ucode::Glyphs::Resolver` with per-block inputs pre-configured.
|
|
20
|
+
|
|
21
|
+
## Files
|
|
22
|
+
|
|
23
|
+
- `lib/ucode/code_chart/extractor.rb` — `Ucode::CodeChart::Extractor`
|
|
24
|
+
class.
|
|
25
|
+
- `spec/ucode/code_chart/extractor_spec.rb` — model/value-object
|
|
26
|
+
specs (constructor invariants, Resolver wiring) plus an integration
|
|
27
|
+
test against the fixture `spec/fixtures/pdfs/basic_latin.pdf`.
|
|
28
|
+
|
|
29
|
+
## Design
|
|
30
|
+
|
|
31
|
+
### Class shape
|
|
32
|
+
|
|
33
|
+
```ruby
|
|
34
|
+
class Ucode::CodeChart::Extractor
|
|
35
|
+
Result = Struct.new(:codepoint, :svg, :tier, :provenance, :base_font,
|
|
36
|
+
:gid, keyword_init: true)
|
|
37
|
+
|
|
38
|
+
def initialize(block:, blocks_txt:, pdf_fetcher: nil,
|
|
39
|
+
font_cache_dir: nil, last_resort_root: nil)
|
|
40
|
+
@block = block # Models::Block
|
|
41
|
+
@blocks_txt = blocks_txt # Pathname
|
|
42
|
+
@pdf_fetcher = pdf_fetcher # optional injectable
|
|
43
|
+
@font_cache_dir = font_cache_dir # default: data/pdf-fonts/
|
|
44
|
+
@last_resort_root = last_resort_root
|
|
45
|
+
end
|
|
46
|
+
|
|
47
|
+
# Walks every assigned codepoint in @block and returns one Result
|
|
48
|
+
# per codepoint. Codepoints with no glyph from any tier are
|
|
49
|
+
# silently skipped (no Result yielded) — the REQ's "skip
|
|
50
|
+
# unassigned codepoints with a warning" is satisfied by the
|
|
51
|
+
# Resolver returning nil for them.
|
|
52
|
+
#
|
|
53
|
+
# @return [Array<Result>]
|
|
54
|
+
def extract
|
|
55
|
+
end
|
|
56
|
+
end
|
|
57
|
+
```
|
|
58
|
+
|
|
59
|
+
### Wiring (single source of truth)
|
|
60
|
+
|
|
61
|
+
The Extractor does NOT implement tier selection. It builds a
|
|
62
|
+
`Ucode::Glyphs::Resolver` and calls `resolver.resolve(codepoint)` for
|
|
63
|
+
each cp. The Resolver's tier order is preserved (Pillar 1 → 2 → 3
|
|
64
|
+
for this feature; no Tier 1 because we're starting from the Code
|
|
65
|
+
Charts PDF, not a real-font source).
|
|
66
|
+
|
|
67
|
+
```ruby
|
|
68
|
+
def build_resolver
|
|
69
|
+
pdf = fetch_pdf!
|
|
70
|
+
embedded_source = Glyphs::EmbeddedFonts::Source.new(
|
|
71
|
+
pdf: pdf, cache_dir: @font_cache_dir,
|
|
72
|
+
)
|
|
73
|
+
catalog = Glyphs::EmbeddedFonts::Catalog.new(embedded_source)
|
|
74
|
+
pillar1 = Glyphs::Sources::Pillar1EmbeddedTounicode.new(
|
|
75
|
+
renderer: Glyphs::EmbeddedFonts::Renderer.new(catalog),
|
|
76
|
+
)
|
|
77
|
+
pillar3 = Glyphs::Sources::Pillar3LastResort.new(
|
|
78
|
+
renderer: Glyphs::LastResort::Renderer.new(
|
|
79
|
+
Glyphs::LastResort::Source.new(root: @last_resort_root),
|
|
80
|
+
),
|
|
81
|
+
)
|
|
82
|
+
Glyphs::Resolver.new(
|
|
83
|
+
sources: [pillar1, pillar3],
|
|
84
|
+
order: %i[pillar1 pillar3],
|
|
85
|
+
)
|
|
86
|
+
end
|
|
87
|
+
```
|
|
88
|
+
|
|
89
|
+
The tier ordering is documented inline: we skip Pillar 2 because for
|
|
90
|
+
the CodeChart use case the catalog's ToUnicode is the dominant path
|
|
91
|
+
and Pillar 2 (positional correlation) is reserved for fonts where
|
|
92
|
+
Pillar 1 fails. If a future use case needs Pillar 2, add it without
|
|
93
|
+
changing this constructor — that's the OCP payoff of consuming the
|
|
94
|
+
Resolver.
|
|
95
|
+
|
|
96
|
+
### Why no Tier 1
|
|
97
|
+
|
|
98
|
+
Tier 1 (real-font cmap) needs a configured `SourceConfig` mapping
|
|
99
|
+
block → font. The CodeChart use case is for blocks where no real
|
|
100
|
+
font exists (Sidetic, Egyptian Ext-B). Tier 1 wouldn't contribute
|
|
101
|
+
anything. The Extractor accepts a Tier 1 source in the future by
|
|
102
|
+
having callers pass a fully-built Resolver instead of constructing
|
|
103
|
+
one internally.
|
|
104
|
+
|
|
105
|
+
### Why PDF fetch is delegated to `PdfFetcher`
|
|
106
|
+
|
|
107
|
+
`Ucode::Glyphs::PdfFetcher` is the existing seam for resolving a
|
|
108
|
+
block to its PDF on disk (per-block cache + monolith fallback). It
|
|
109
|
+
already handles `force:` and the cache directory. The Extractor
|
|
110
|
+
constructs a `PdfFetcher` per call (cheap — it's just a path
|
|
111
|
+
resolver) and reuses it across codepoints.
|
|
112
|
+
|
|
113
|
+
### Per-codepoint loop
|
|
114
|
+
|
|
115
|
+
```ruby
|
|
116
|
+
def extract
|
|
117
|
+
resolver = build_resolver
|
|
118
|
+
@block.codepoint_ids.flat_map do |cp_id|
|
|
119
|
+
cp = Integer(cp_id.delete_prefix("U+"), 16)
|
|
120
|
+
resolver_result = resolver.resolve(cp)
|
|
121
|
+
next nil unless resolver_result&.svg
|
|
122
|
+
|
|
123
|
+
Result.new(
|
|
124
|
+
codepoint: cp,
|
|
125
|
+
svg: resolver_result.svg,
|
|
126
|
+
tier: resolver_result.tier,
|
|
127
|
+
provenance: resolver_result.provenance,
|
|
128
|
+
)
|
|
129
|
+
end.compact
|
|
130
|
+
end
|
|
131
|
+
```
|
|
132
|
+
|
|
133
|
+
The Resolver returns a `Sources::Result` (tier + codepoint + svg +
|
|
134
|
+
provenance). We adapt that to the Extractor's `Result` (with
|
|
135
|
+
codepoint + svg + tier + provenance), stripping the resolver-specific
|
|
136
|
+
shape at the boundary.
|
|
137
|
+
|
|
138
|
+
## Acceptance
|
|
139
|
+
|
|
140
|
+
- `Extractor.new(block: ..., blocks_txt: ...)` constructs without
|
|
141
|
+
raising when the block and PDF are present.
|
|
142
|
+
- `#extract` returns one Result per codepoint that any tier
|
|
143
|
+
produced a glyph for.
|
|
144
|
+
- `#extract` skips codepoints no tier could produce (returns no
|
|
145
|
+
Result, not a Result-with-nil).
|
|
146
|
+
- Integration test: against the fixture PDF, at least one codepoint's
|
|
147
|
+
Result has tier `:pillar1` and provenance `"pillar-1:embedded-tounicode"`.
|
|
148
|
+
|
|
149
|
+
## Out of scope
|
|
150
|
+
|
|
151
|
+
- Writing files — that's `CodeChart::Writer` (TODO 06).
|
|
152
|
+
- Provenance JSON — that's `CodeChart::Sidecar` (TODO 05).
|
|
153
|
+
- Tier 1 (real-font) source injection — not needed for the REQ's
|
|
154
|
+
blocks. Future extension point if a real-font fallback is desired.
|