ucode 0.3.3 → 0.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +45 -6
- data/config/unicode17_universal_glyph_set.yml +1 -1
- data/lib/ucode/cli.rb +201 -32
- data/lib/ucode/code_chart/batch_runner.rb +184 -0
- data/lib/ucode/code_chart/block_index.rb +89 -0
- data/lib/ucode/code_chart/coverage_gap_index.rb +115 -0
- data/lib/ucode/code_chart/extractor.rb +65 -13
- data/lib/ucode/code_chart/fetcher.rb +112 -0
- data/lib/ucode/code_chart/gap_analyzer/block_gap.rb +35 -0
- data/lib/ucode/code_chart/gap_analyzer/essenfont_manifest.rb +60 -0
- data/lib/ucode/code_chart/gap_analyzer/manifest.rb +39 -0
- data/lib/ucode/code_chart/gap_analyzer.rb +83 -0
- data/lib/ucode/code_chart/provenance.rb +99 -58
- data/lib/ucode/code_chart/sidecar.rb +3 -4
- data/lib/ucode/code_chart/verifier/builder.rb +49 -0
- data/lib/ucode/code_chart/verifier/mutool_strategy.rb +80 -0
- data/lib/ucode/code_chart/verifier/page_render_cache.rb +63 -0
- data/lib/ucode/code_chart/verifier/result.rb +27 -0
- data/lib/ucode/code_chart/verifier/resvg_strategy.rb +71 -0
- data/lib/ucode/code_chart/verifier/strategy.rb +73 -0
- data/lib/ucode/code_chart/verifier.rb +152 -0
- data/lib/ucode/code_chart/writer.rb +21 -14
- data/lib/ucode/code_chart.rb +6 -0
- data/lib/ucode/error.rb +16 -1
- data/lib/ucode/fetch/http.rb +23 -3
- data/lib/ucode/glyphs/embedded_fonts/catalog.rb +66 -3
- data/lib/ucode/glyphs/embedded_fonts/codepoint_mapper/correlator_strategy.rb +46 -0
- data/lib/ucode/glyphs/embedded_fonts/codepoint_mapper/strategy.rb +54 -0
- data/lib/ucode/glyphs/embedded_fonts/codepoint_mapper/tounicode_strategy.rb +41 -0
- data/lib/ucode/glyphs/embedded_fonts/codepoint_mapper/trace_strategy.rb +64 -0
- data/lib/ucode/glyphs/embedded_fonts/codepoint_mapper.rb +150 -90
- data/lib/ucode/glyphs/embedded_fonts/mutool/draw.rb +26 -0
- data/lib/ucode/glyphs/embedded_fonts/mutool/info.rb +23 -0
- data/lib/ucode/glyphs/embedded_fonts/mutool/show.rb +36 -0
- data/lib/ucode/glyphs/embedded_fonts/mutool/trace.rb +26 -0
- data/lib/ucode/glyphs/embedded_fonts/mutool.rb +30 -0
- data/lib/ucode/glyphs/embedded_fonts/page_trace_cache.rb +112 -0
- data/lib/ucode/glyphs/embedded_fonts/pdf_indexer.rb +50 -56
- data/lib/ucode/glyphs/embedded_fonts/{pdf_location.rb → pdf_source.rb} +1 -1
- data/lib/ucode/glyphs/embedded_fonts/renderer.rb +19 -2
- data/lib/ucode/glyphs/embedded_fonts/trace_runner.rb +12 -28
- data/lib/ucode/glyphs/embedded_fonts.rb +3 -1
- data/lib/ucode/glyphs/real_fonts/coverage_auditor.rb +0 -4
- data/lib/ucode/glyphs/real_fonts/font_coverage_report.rb +0 -2
- data/lib/ucode/glyphs/real_fonts/writer.rb +0 -2
- data/lib/ucode/glyphs/source.rb +11 -1
- data/lib/ucode/glyphs/sources/pillar1_embedded_tounicode.rb +11 -6
- data/lib/ucode/version.rb +1 -1
- data/lib/ucode.rb +2 -0
- metadata +28 -74
- data/TODO.extract-code-chart/01-pdf-fetch-validation.md +0 -80
- data/TODO.extract-code-chart/02-block-name-resolver.md +0 -68
- data/TODO.extract-code-chart/03-codechart-namespace.md +0 -82
- data/TODO.extract-code-chart/04-codechart-extractor.md +0 -154
- data/TODO.extract-code-chart/05-provenance-and-sidecar.md +0 -147
- data/TODO.extract-code-chart/06-codechart-writer.md +0 -134
- data/TODO.extract-code-chart/07-codechart-cli.md +0 -135
- data/TODO.extract-code-chart/08-specs.md +0 -87
- data/TODO.full/00-README.md +0 -116
- data/TODO.full/01-panglyph-vision.md +0 -112
- data/TODO.full/02-panglyph-repo-bootstrap.md +0 -184
- data/TODO.full/03-panglyph-font-builder.md +0 -201
- data/TODO.full/04-panglyph-publish-pipeline.md +0 -126
- data/TODO.full/05-ucode-0-1-1-release.md +0 -139
- data/TODO.full/06-fontisan-remove-audit.md +0 -142
- data/TODO.full/07-fontisan-remove-ucd.md +0 -125
- data/TODO.full/08-archive-private-bin-build.md +0 -143
- data/TODO.full/09-archive-public-structure.md +0 -164
- data/TODO.full/10-fontist-org-woff-glyphs.md +0 -131
- data/TODO.full/11-fontist-org-audit-coverage.md +0 -140
- data/TODO.full/12-implementation-order.md +0 -216
- data/TODO.full/13-fontisan-font-writer-api.md +0 -189
- data/TODO.full/14-fontisan-table-writers.md +0 -66
- data/TODO.full/15-panglyph-builder-real.md +0 -82
- data/TODO.full/16-archive-public-sync-workflows.md +0 -167
- data/TODO.full/17-fontist-org-font-picker.md +0 -73
- data/TODO.full/18-comprehensive-spec-coverage.md +0 -64
- data/TODO.full/19-ucode-0-1-2-patch.md +0 -32
- data/TODO.full/20-fontisan-0-2-23-release.md +0 -52
- data/TODO.new/00-README.md +0 -96
- data/TODO.new/01-pillar-terminology-alignment.md +0 -69
- data/TODO.new/02-audit-schema-design.md +0 -255
- data/TODO.new/03-directory-output-spec.md +0 -203
- data/TODO.new/04-fontist-org-contract.md +0 -173
- data/TODO.new/05-baseline-unicode17-coverage-audit.md +0 -144
- data/TODO.new/06-audit-namespace-skeleton.md +0 -105
- data/TODO.new/07-audit-models-port.md +0 -132
- data/TODO.new/08-extractors-cheap-port.md +0 -113
- data/TODO.new/09-extractors-expensive-port.md +0 -99
- data/TODO.new/10-aggregations-ucd-rewrite.md +0 -168
- data/TODO.new/11-differ-and-library-auditor-port.md +0 -102
- data/TODO.new/12-formatters-port.md +0 -115
- data/TODO.new/13-directory-emitter.md +0 -147
- data/TODO.new/14-html-face-browser.md +0 -144
- data/TODO.new/15-html-library-browser.md +0 -102
- data/TODO.new/16-cli-audit-subcommands.md +0 -142
- data/TODO.new/17-fontisan-cleanup-audit.md +0 -147
- data/TODO.new/18-fontisan-cleanup-ucd.md +0 -156
- data/TODO.new/19-fontisan-docs-update.md +0 -155
- data/TODO.new/20-canonical-resolver-4-tier.md +0 -182
- data/TODO.new/21-canonical-unicode17-build.md +0 -148
- data/TODO.new/22-implementation-order.md +0 -176
- data/TODO.new/23-universal-glyph-set-source-map.md +0 -312
- data/TODO.new/24-universal-glyph-set-build.md +0 -189
- data/TODO.new/25-font-audit-against-universal-set.md +0 -195
- data/TODO.new/26-missing-glyph-reporter.md +0 -189
- data/TODO.new/27-fontist-org-consumer-integration.md +0 -200
- data/TODO.new/28-implementation-order-update.md +0 -187
- data/TODO.new/29-universal-set-curation-uc17.md +0 -312
- data/TODO.new/30-tier1-font-acquisition.md +0 -241
- data/TODO.new/31-universal-set-production-build.md +0 -205
- data/TODO.new/32-uc17-coverage-matrix.md +0 -165
- data/TODO.new/33-specialist-font-acquisition-refresh.md +0 -138
- data/TODO.new/34-pillar2-content-stream-correlator.md +0 -147
- data/TODO.new/35-universal-set-production-run.md +0 -160
- data/TODO.new/36-per-font-coverage-audit.md +0 -145
- data/TODO.new/37-coverage-highlight-reporter.md +0 -125
- data/TODO.new/38-fontist-org-glyph-consumer.md +0 -141
- data/TODO.new/39-implementation-order-update-32-38.md +0 -258
- data/TODO.new/40-archive-private-uses-ucode-audit.md +0 -124
- data/TODO.new/41-ucode-unicode-archive-bridge.md +0 -160
|
@@ -1,23 +1,23 @@
|
|
|
1
1
|
# frozen_string_literal: true
|
|
2
2
|
|
|
3
|
-
require "open3"
|
|
4
3
|
require "pathname"
|
|
5
4
|
|
|
5
|
+
require "ucode/glyphs/embedded_fonts/mutool"
|
|
6
|
+
|
|
6
7
|
module Ucode
|
|
7
8
|
module Glyphs
|
|
8
9
|
module EmbeddedFonts
|
|
9
|
-
# Walks the Code Charts PDF once via `mutool info` + `mutool show`
|
|
10
|
-
# and builds an Array of {RawFontDescriptor} — one per Type0 font
|
|
11
|
-
# that has the required descendant CIDFont, FontDescriptor, and
|
|
12
|
-
# FontFile2/3 + Identity CIDToGIDMap.
|
|
13
|
-
#
|
|
14
|
-
# Pure subprocess + parsing concern. Does NOT resolve codepoint →
|
|
15
|
-
# GID (that's {CodepointMapper}'s job). The descriptor carries
|
|
16
|
-
# every ref the mapper needs to do its work.
|
|
17
10
|
class PdfIndexer
|
|
18
|
-
|
|
19
|
-
|
|
11
|
+
Type0Entry = Struct.new(:font_obj_id, :base_font, keyword_init: true)
|
|
12
|
+
|
|
13
|
+
# @param source [PdfSource]
|
|
14
|
+
# @param mutool_info [Mutool::Info] injectable for tests
|
|
15
|
+
# @param mutool_show [Mutool::Show] injectable for tests
|
|
16
|
+
def initialize(source:, mutool_info: Mutool::Info.new,
|
|
17
|
+
mutool_show: Mutool::Show.new)
|
|
20
18
|
@source = source
|
|
19
|
+
@mutool_info = mutool_info
|
|
20
|
+
@mutool_show = mutool_show
|
|
21
21
|
end
|
|
22
22
|
|
|
23
23
|
# @return [Array<RawFontDescriptor>]
|
|
@@ -68,11 +68,45 @@ module Ucode
|
|
|
68
68
|
# @param base_font [String] e.g. "GPJAHB+WolofGaraySansSerif"
|
|
69
69
|
# @return [Boolean] true if this font appears on any page
|
|
70
70
|
def font_appears?(base_font)
|
|
71
|
-
|
|
71
|
+
type0_base_fonts.include?(base_font)
|
|
72
72
|
end
|
|
73
73
|
|
|
74
74
|
private
|
|
75
75
|
|
|
76
|
+
# Single source of truth for the Type0 font list. Both
|
|
77
|
+
# `discover_type0_fonts` (keyed by obj_id) and
|
|
78
|
+
# `font_appears?` (membership check by name) derive their
|
|
79
|
+
# answer from this parse, so changes to `mutool info`'s output
|
|
80
|
+
# format break in exactly one place.
|
|
81
|
+
#
|
|
82
|
+
# @return [Array<Type0Entry>]
|
|
83
|
+
def type0_entries
|
|
84
|
+
@type0_entries ||= parse_type0_entries
|
|
85
|
+
end
|
|
86
|
+
|
|
87
|
+
# @return [Set<String>] every Type0 BaseFont name seen
|
|
88
|
+
def type0_base_fonts
|
|
89
|
+
@type0_base_fonts ||= type0_entries.each_with_object(Set.new) do |e, s|
|
|
90
|
+
s << e.base_font
|
|
91
|
+
end
|
|
92
|
+
end
|
|
93
|
+
|
|
94
|
+
def parse_type0_entries
|
|
95
|
+
seen = Set.new
|
|
96
|
+
mutool_info_text.each_line.filter_map do |line|
|
|
97
|
+
next unless line.include?("Type0")
|
|
98
|
+
|
|
99
|
+
m = line.match(/Type0\s+'([^']+)'\s+\S+\s+\((\d+)\s+0\s+R\)/)
|
|
100
|
+
next unless m
|
|
101
|
+
|
|
102
|
+
font_obj_id = m[2].to_i
|
|
103
|
+
next if seen.include?(font_obj_id)
|
|
104
|
+
|
|
105
|
+
seen << font_obj_id
|
|
106
|
+
Type0Entry.new(font_obj_id: font_obj_id, base_font: m[1])
|
|
107
|
+
end
|
|
108
|
+
end
|
|
109
|
+
|
|
76
110
|
def build_descriptors(type0_refs, type0_dicts, descendant_dicts, fontdesc_dicts)
|
|
77
111
|
type0_refs.filter_map do |font_obj_id, base_font|
|
|
78
112
|
build_descriptor(
|
|
@@ -133,35 +167,15 @@ module Ucode
|
|
|
133
167
|
# ---- mutool subprocess + dict parsing ----------------------------
|
|
134
168
|
|
|
135
169
|
def discover_type0_fonts
|
|
136
|
-
|
|
137
|
-
|
|
138
|
-
seen = Set.new
|
|
139
|
-
text.each_line do |line|
|
|
140
|
-
next unless line.include?("Type0")
|
|
141
|
-
|
|
142
|
-
m = line.match(/Type0\s+'([^']+)'\s+\S+\s+\((\d+)\s+0\s+R\)/)
|
|
143
|
-
next unless m
|
|
144
|
-
|
|
145
|
-
font_obj_id = m[2].to_i
|
|
146
|
-
next if seen.include?(font_obj_id)
|
|
147
|
-
|
|
148
|
-
seen << font_obj_id
|
|
149
|
-
result[font_obj_id] = m[1]
|
|
170
|
+
type0_entries.to_h do |e|
|
|
171
|
+
[e.font_obj_id, e.base_font]
|
|
150
172
|
end
|
|
151
|
-
result
|
|
152
173
|
end
|
|
153
174
|
|
|
154
175
|
def fetch_objects(obj_ids)
|
|
155
176
|
return {} if obj_ids.empty?
|
|
156
177
|
|
|
157
|
-
|
|
158
|
-
@source.pdf_to_s].concat(obj_ids.map(&:to_s))
|
|
159
|
-
out, err, status = Open3.capture3(*args)
|
|
160
|
-
unless status.success?
|
|
161
|
-
raise Ucode::EmbeddedFontsMissingError,
|
|
162
|
-
"mutool show failed: #{err.strip}"
|
|
163
|
-
end
|
|
164
|
-
|
|
178
|
+
out = @mutool_show.grep(@source.pdf_to_s, *obj_ids)
|
|
165
179
|
parse_grep_output(out)
|
|
166
180
|
end
|
|
167
181
|
|
|
@@ -208,27 +222,7 @@ module Ucode
|
|
|
208
222
|
end
|
|
209
223
|
|
|
210
224
|
def mutool_info_text
|
|
211
|
-
@mutool_info_text ||=
|
|
212
|
-
end
|
|
213
|
-
|
|
214
|
-
def run_mutool_info
|
|
215
|
-
out, err, status = Open3.capture3("mutool", "info", @source.pdf_to_s)
|
|
216
|
-
status.success? ? out + err : ""
|
|
217
|
-
end
|
|
218
|
-
|
|
219
|
-
def font_entries_cache
|
|
220
|
-
@font_entries_cache ||= begin
|
|
221
|
-
result = {}
|
|
222
|
-
mutool_info_text.each_line do |line|
|
|
223
|
-
next unless line.include?("Type0")
|
|
224
|
-
|
|
225
|
-
font_match = line.match(/Type0\s+'([^']+)'/)
|
|
226
|
-
next unless font_match
|
|
227
|
-
|
|
228
|
-
result[font_match[1]] = true
|
|
229
|
-
end
|
|
230
|
-
result
|
|
231
|
-
end
|
|
225
|
+
@mutool_info_text ||= @mutool_info.call(@source.pdf_to_s)
|
|
232
226
|
end
|
|
233
227
|
end
|
|
234
228
|
end
|
|
@@ -27,7 +27,7 @@ module Ucode
|
|
|
27
27
|
# the BaseFont (e.g. `CIAIIP+Uni2000Generalpunctuation.ttf`).
|
|
28
28
|
# Re-runs skip extraction when the cached file is newer than the
|
|
29
29
|
# PDF.
|
|
30
|
-
class
|
|
30
|
+
class PdfSource
|
|
31
31
|
attr_reader :pdf_path, :cache_dir
|
|
32
32
|
|
|
33
33
|
# @param pdf [String, Pathname, nil] path to a Code Charts PDF
|
|
@@ -12,7 +12,16 @@ module Ucode
|
|
|
12
12
|
# renderer.
|
|
13
13
|
class Renderer
|
|
14
14
|
# Result of rendering one codepoint.
|
|
15
|
-
|
|
15
|
+
#
|
|
16
|
+
# `source_page` + `source_cell` come from the Catalog's
|
|
17
|
+
# location index when available. Nil when the catalog hasn't
|
|
18
|
+
# traced the PDF (ToUnicode-only path) — downstream concerns
|
|
19
|
+
# treat that as "no positional data; skip cell-based diff".
|
|
20
|
+
Result = Struct.new(
|
|
21
|
+
:codepoint, :base_font, :gid, :svg,
|
|
22
|
+
:source_page, :source_cell,
|
|
23
|
+
keyword_init: true,
|
|
24
|
+
) do
|
|
16
25
|
def ok?
|
|
17
26
|
!svg.nil?
|
|
18
27
|
end
|
|
@@ -37,7 +46,15 @@ module Ucode
|
|
|
37
46
|
return nil if outline.nil? || outline.empty?
|
|
38
47
|
|
|
39
48
|
svg = Svg.new(outline, codepoint: codepoint, base_font: entry.base_font).to_s
|
|
40
|
-
|
|
49
|
+
location = @catalog.location_for(codepoint)
|
|
50
|
+
Result.new(
|
|
51
|
+
codepoint: codepoint,
|
|
52
|
+
base_font: entry.base_font,
|
|
53
|
+
gid: gid,
|
|
54
|
+
svg: svg,
|
|
55
|
+
source_page: location&.fetch(:page),
|
|
56
|
+
source_cell: location && { x: location[:x], y: location[:y] },
|
|
57
|
+
)
|
|
41
58
|
end
|
|
42
59
|
end
|
|
43
60
|
end
|
|
@@ -1,51 +1,35 @@
|
|
|
1
1
|
# frozen_string_literal: true
|
|
2
2
|
|
|
3
|
-
require "open3"
|
|
4
3
|
require "pathname"
|
|
5
4
|
|
|
6
|
-
require "ucode/
|
|
5
|
+
require "ucode/glyphs/embedded_fonts/mutool"
|
|
6
|
+
require "ucode/glyphs/embedded_fonts/trace_parser"
|
|
7
7
|
|
|
8
8
|
module Ucode
|
|
9
9
|
module Glyphs
|
|
10
10
|
module EmbeddedFonts
|
|
11
11
|
# Thin I/O wrapper around `mutool trace <pdf> <page>`.
|
|
12
12
|
#
|
|
13
|
-
#
|
|
14
|
-
#
|
|
15
|
-
#
|
|
16
|
-
#
|
|
17
|
-
# The only class in the trace pipeline that touches the
|
|
18
|
-
# filesystem / spawns subprocesses. Everything upstream
|
|
19
|
-
# (parser, correlator) is pure.
|
|
13
|
+
# Delegates the actual subprocess to {Mutool::Trace} and the
|
|
14
|
+
# XML parsing to {TraceParser}. Kept for backward compatibility
|
|
15
|
+
# with callers that pre-date the Mutool seam.
|
|
20
16
|
class TraceRunner
|
|
21
17
|
# @param pdf_path [Pathname, String]
|
|
22
|
-
|
|
18
|
+
# @param mutool [Mutool::Trace] injectable for tests
|
|
19
|
+
def initialize(pdf_path, mutool: Mutool::Trace.new)
|
|
23
20
|
@pdf_path = Pathname.new(pdf_path)
|
|
21
|
+
@mutool = mutool
|
|
24
22
|
end
|
|
25
23
|
|
|
26
24
|
# @param page_numbers [Array<Integer>] 1-based PDF page numbers
|
|
27
25
|
# @return [Array<TraceGlyph>]
|
|
28
26
|
def trace(page_numbers)
|
|
29
|
-
|
|
30
|
-
end
|
|
31
|
-
|
|
32
|
-
private
|
|
27
|
+
return [] if page_numbers.empty?
|
|
33
28
|
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
end
|
|
38
|
-
|
|
39
|
-
def run_mutool(page)
|
|
40
|
-
out, err, status = Open3.capture3(
|
|
41
|
-
"mutool", "trace", @pdf_path.to_s, page.to_s,
|
|
42
|
-
)
|
|
43
|
-
unless status.success?
|
|
44
|
-
raise Ucode::EmbeddedFontsMissingError,
|
|
45
|
-
"mutool trace failed: #{(out + err).strip}"
|
|
29
|
+
page_numbers.flat_map do |page|
|
|
30
|
+
xml = @mutool.call(@pdf_path, page)
|
|
31
|
+
TraceParser.parse(xml)
|
|
46
32
|
end
|
|
47
|
-
|
|
48
|
-
out + err
|
|
49
33
|
end
|
|
50
34
|
end
|
|
51
35
|
end
|
|
@@ -36,7 +36,7 @@ module Ucode
|
|
|
36
36
|
# `mutool info` (font enumeration) and `mutool show -b -o` (raw
|
|
37
37
|
# stream extraction).
|
|
38
38
|
module EmbeddedFonts
|
|
39
|
-
autoload :
|
|
39
|
+
autoload :PdfSource, "ucode/glyphs/embedded_fonts/pdf_source"
|
|
40
40
|
autoload :ToUnicode, "ucode/glyphs/embedded_fonts/tounicode"
|
|
41
41
|
autoload :FontEntry, "ucode/glyphs/embedded_fonts/font_entry"
|
|
42
42
|
autoload :RawFontDescriptor, "ucode/glyphs/embedded_fonts/raw_font_descriptor"
|
|
@@ -50,6 +50,8 @@ module Ucode
|
|
|
50
50
|
autoload :TraceParser, "ucode/glyphs/embedded_fonts/trace_parser"
|
|
51
51
|
autoload :TraceCorrelator, "ucode/glyphs/embedded_fonts/trace_correlator"
|
|
52
52
|
autoload :TraceRunner, "ucode/glyphs/embedded_fonts/trace_runner"
|
|
53
|
+
autoload :Mutool, "ucode/glyphs/embedded_fonts/mutool"
|
|
54
|
+
autoload :PageTraceCache, "ucode/glyphs/embedded_fonts/page_trace_cache"
|
|
53
55
|
autoload :Svg, "ucode/glyphs/embedded_fonts/svg"
|
|
54
56
|
autoload :Renderer, "ucode/glyphs/embedded_fonts/renderer"
|
|
55
57
|
autoload :Writer, "ucode/glyphs/embedded_fonts/writer"
|
data/lib/ucode/glyphs/source.rb
CHANGED
|
@@ -25,7 +25,17 @@ module Ucode
|
|
|
25
25
|
# One resolved glyph. Carries the SVG payload and enough
|
|
26
26
|
# provenance to debug "where did this glyph come from?" without
|
|
27
27
|
# holding a reference back to the source.
|
|
28
|
-
|
|
28
|
+
#
|
|
29
|
+
# The optional `base_font`, `gid`, `source_page`, `source_cell`
|
|
30
|
+
# fields carry the renderer's localization data so downstream
|
|
31
|
+
# concerns (Verifier pixel-diff against the source PDF cell,
|
|
32
|
+
# Provenance audit trail) don't need to re-derive it. Sources
|
|
33
|
+
# that don't know this data (e.g. LastResort) leave them nil.
|
|
34
|
+
Result = Struct.new(
|
|
35
|
+
:tier, :codepoint, :svg, :provenance,
|
|
36
|
+
:base_font, :gid, :source_page, :source_cell,
|
|
37
|
+
keyword_init: true,
|
|
38
|
+
)
|
|
29
39
|
|
|
30
40
|
# @return [Symbol] one of :tier1, :pillar1, :pillar2, :pillar3
|
|
31
41
|
def tier
|
|
@@ -1,8 +1,5 @@
|
|
|
1
1
|
# frozen_string_literal: true
|
|
2
2
|
|
|
3
|
-
require "ucode/glyphs/source"
|
|
4
|
-
require "ucode/glyphs/embedded_fonts/renderer"
|
|
5
|
-
|
|
6
3
|
module Ucode
|
|
7
4
|
module Glyphs
|
|
8
5
|
module Sources
|
|
@@ -31,7 +28,7 @@ module Ucode
|
|
|
31
28
|
# @param renderer [EmbeddedFonts::Renderer] the renderer to
|
|
32
29
|
# delegate to. Callers typically construct it with the
|
|
33
30
|
# {EmbeddedFonts::Catalog} built from the resolved Code
|
|
34
|
-
# Charts {EmbeddedFonts::
|
|
31
|
+
# Charts {EmbeddedFonts::PdfSource}. To enable pillar-2
|
|
35
32
|
# fallback, that Catalog must be constructed with
|
|
36
33
|
# +correlator_configs:+.
|
|
37
34
|
def initialize(renderer:)
|
|
@@ -54,8 +51,16 @@ module Ucode
|
|
|
54
51
|
result = @renderer.render(codepoint)
|
|
55
52
|
return nil unless result
|
|
56
53
|
|
|
57
|
-
Result.new(
|
|
58
|
-
|
|
54
|
+
Result.new(
|
|
55
|
+
tier: tier,
|
|
56
|
+
codepoint: codepoint,
|
|
57
|
+
svg: result.svg,
|
|
58
|
+
provenance: provenance,
|
|
59
|
+
base_font: result.base_font,
|
|
60
|
+
gid: result.gid,
|
|
61
|
+
source_page: result.source_page,
|
|
62
|
+
source_cell: result.source_cell,
|
|
63
|
+
)
|
|
59
64
|
end
|
|
60
65
|
end
|
|
61
66
|
end
|
data/lib/ucode/version.rb
CHANGED
data/lib/ucode.rb
CHANGED
|
@@ -34,7 +34,9 @@ module Ucode
|
|
|
34
34
|
autoload :GlyphError, "ucode/error"
|
|
35
35
|
autoload :LastResortMissingError, "ucode/error"
|
|
36
36
|
autoload :EmbeddedFontsMissingError, "ucode/error"
|
|
37
|
+
autoload :CodeChartError, "ucode/error"
|
|
37
38
|
autoload :CodeChartNotFoundError, "ucode/error"
|
|
39
|
+
autoload :CodeChartChecksumError, "ucode/error"
|
|
38
40
|
autoload :UnknownUnicodeVersionError, "ucode/error"
|
|
39
41
|
|
|
40
42
|
# Infrastructure
|
metadata
CHANGED
|
@@ -1,14 +1,14 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: ucode
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.
|
|
4
|
+
version: 0.5.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Ribose Inc.
|
|
8
8
|
autorequire:
|
|
9
9
|
bindir: exe
|
|
10
10
|
cert_chain: []
|
|
11
|
-
date: 2026-07-
|
|
11
|
+
date: 2026-07-21 00:00:00.000000000 Z
|
|
12
12
|
dependencies:
|
|
13
13
|
- !ruby/object:Gem::Dependency
|
|
14
14
|
name: base64
|
|
@@ -162,77 +162,6 @@ files:
|
|
|
162
162
|
- Gemfile
|
|
163
163
|
- README.md
|
|
164
164
|
- Rakefile
|
|
165
|
-
- TODO.extract-code-chart/01-pdf-fetch-validation.md
|
|
166
|
-
- TODO.extract-code-chart/02-block-name-resolver.md
|
|
167
|
-
- TODO.extract-code-chart/03-codechart-namespace.md
|
|
168
|
-
- TODO.extract-code-chart/04-codechart-extractor.md
|
|
169
|
-
- TODO.extract-code-chart/05-provenance-and-sidecar.md
|
|
170
|
-
- TODO.extract-code-chart/06-codechart-writer.md
|
|
171
|
-
- TODO.extract-code-chart/07-codechart-cli.md
|
|
172
|
-
- TODO.extract-code-chart/08-specs.md
|
|
173
|
-
- TODO.full/00-README.md
|
|
174
|
-
- TODO.full/01-panglyph-vision.md
|
|
175
|
-
- TODO.full/02-panglyph-repo-bootstrap.md
|
|
176
|
-
- TODO.full/03-panglyph-font-builder.md
|
|
177
|
-
- TODO.full/04-panglyph-publish-pipeline.md
|
|
178
|
-
- TODO.full/05-ucode-0-1-1-release.md
|
|
179
|
-
- TODO.full/06-fontisan-remove-audit.md
|
|
180
|
-
- TODO.full/07-fontisan-remove-ucd.md
|
|
181
|
-
- TODO.full/08-archive-private-bin-build.md
|
|
182
|
-
- TODO.full/09-archive-public-structure.md
|
|
183
|
-
- TODO.full/10-fontist-org-woff-glyphs.md
|
|
184
|
-
- TODO.full/11-fontist-org-audit-coverage.md
|
|
185
|
-
- TODO.full/12-implementation-order.md
|
|
186
|
-
- TODO.full/13-fontisan-font-writer-api.md
|
|
187
|
-
- TODO.full/14-fontisan-table-writers.md
|
|
188
|
-
- TODO.full/15-panglyph-builder-real.md
|
|
189
|
-
- TODO.full/16-archive-public-sync-workflows.md
|
|
190
|
-
- TODO.full/17-fontist-org-font-picker.md
|
|
191
|
-
- TODO.full/18-comprehensive-spec-coverage.md
|
|
192
|
-
- TODO.full/19-ucode-0-1-2-patch.md
|
|
193
|
-
- TODO.full/20-fontisan-0-2-23-release.md
|
|
194
|
-
- TODO.new/00-README.md
|
|
195
|
-
- TODO.new/01-pillar-terminology-alignment.md
|
|
196
|
-
- TODO.new/02-audit-schema-design.md
|
|
197
|
-
- TODO.new/03-directory-output-spec.md
|
|
198
|
-
- TODO.new/04-fontist-org-contract.md
|
|
199
|
-
- TODO.new/05-baseline-unicode17-coverage-audit.md
|
|
200
|
-
- TODO.new/06-audit-namespace-skeleton.md
|
|
201
|
-
- TODO.new/07-audit-models-port.md
|
|
202
|
-
- TODO.new/08-extractors-cheap-port.md
|
|
203
|
-
- TODO.new/09-extractors-expensive-port.md
|
|
204
|
-
- TODO.new/10-aggregations-ucd-rewrite.md
|
|
205
|
-
- TODO.new/11-differ-and-library-auditor-port.md
|
|
206
|
-
- TODO.new/12-formatters-port.md
|
|
207
|
-
- TODO.new/13-directory-emitter.md
|
|
208
|
-
- TODO.new/14-html-face-browser.md
|
|
209
|
-
- TODO.new/15-html-library-browser.md
|
|
210
|
-
- TODO.new/16-cli-audit-subcommands.md
|
|
211
|
-
- TODO.new/17-fontisan-cleanup-audit.md
|
|
212
|
-
- TODO.new/18-fontisan-cleanup-ucd.md
|
|
213
|
-
- TODO.new/19-fontisan-docs-update.md
|
|
214
|
-
- TODO.new/20-canonical-resolver-4-tier.md
|
|
215
|
-
- TODO.new/21-canonical-unicode17-build.md
|
|
216
|
-
- TODO.new/22-implementation-order.md
|
|
217
|
-
- TODO.new/23-universal-glyph-set-source-map.md
|
|
218
|
-
- TODO.new/24-universal-glyph-set-build.md
|
|
219
|
-
- TODO.new/25-font-audit-against-universal-set.md
|
|
220
|
-
- TODO.new/26-missing-glyph-reporter.md
|
|
221
|
-
- TODO.new/27-fontist-org-consumer-integration.md
|
|
222
|
-
- TODO.new/28-implementation-order-update.md
|
|
223
|
-
- TODO.new/29-universal-set-curation-uc17.md
|
|
224
|
-
- TODO.new/30-tier1-font-acquisition.md
|
|
225
|
-
- TODO.new/31-universal-set-production-build.md
|
|
226
|
-
- TODO.new/32-uc17-coverage-matrix.md
|
|
227
|
-
- TODO.new/33-specialist-font-acquisition-refresh.md
|
|
228
|
-
- TODO.new/34-pillar2-content-stream-correlator.md
|
|
229
|
-
- TODO.new/35-universal-set-production-run.md
|
|
230
|
-
- TODO.new/36-per-font-coverage-audit.md
|
|
231
|
-
- TODO.new/37-coverage-highlight-reporter.md
|
|
232
|
-
- TODO.new/38-fontist-org-glyph-consumer.md
|
|
233
|
-
- TODO.new/39-implementation-order-update-32-38.md
|
|
234
|
-
- TODO.new/40-archive-private-uses-ucode-audit.md
|
|
235
|
-
- TODO.new/41-ucode-unicode-archive-bridge.md
|
|
236
165
|
- UCODE_CHANGELOG.md
|
|
237
166
|
- config/specialist_fonts.yml
|
|
238
167
|
- config/unicode17_tier1_fonts.yml
|
|
@@ -310,9 +239,24 @@ files:
|
|
|
310
239
|
- lib/ucode/cache.rb
|
|
311
240
|
- lib/ucode/cli.rb
|
|
312
241
|
- lib/ucode/code_chart.rb
|
|
242
|
+
- lib/ucode/code_chart/batch_runner.rb
|
|
243
|
+
- lib/ucode/code_chart/block_index.rb
|
|
244
|
+
- lib/ucode/code_chart/coverage_gap_index.rb
|
|
313
245
|
- lib/ucode/code_chart/extractor.rb
|
|
246
|
+
- lib/ucode/code_chart/fetcher.rb
|
|
247
|
+
- lib/ucode/code_chart/gap_analyzer.rb
|
|
248
|
+
- lib/ucode/code_chart/gap_analyzer/block_gap.rb
|
|
249
|
+
- lib/ucode/code_chart/gap_analyzer/essenfont_manifest.rb
|
|
250
|
+
- lib/ucode/code_chart/gap_analyzer/manifest.rb
|
|
314
251
|
- lib/ucode/code_chart/provenance.rb
|
|
315
252
|
- lib/ucode/code_chart/sidecar.rb
|
|
253
|
+
- lib/ucode/code_chart/verifier.rb
|
|
254
|
+
- lib/ucode/code_chart/verifier/builder.rb
|
|
255
|
+
- lib/ucode/code_chart/verifier/mutool_strategy.rb
|
|
256
|
+
- lib/ucode/code_chart/verifier/page_render_cache.rb
|
|
257
|
+
- lib/ucode/code_chart/verifier/result.rb
|
|
258
|
+
- lib/ucode/code_chart/verifier/resvg_strategy.rb
|
|
259
|
+
- lib/ucode/code_chart/verifier/strategy.rb
|
|
316
260
|
- lib/ucode/code_chart/writer.rb
|
|
317
261
|
- lib/ucode/commands.rb
|
|
318
262
|
- lib/ucode/commands/audit.rb
|
|
@@ -363,10 +307,20 @@ files:
|
|
|
363
307
|
- lib/ucode/glyphs/embedded_fonts.rb
|
|
364
308
|
- lib/ucode/glyphs/embedded_fonts/catalog.rb
|
|
365
309
|
- lib/ucode/glyphs/embedded_fonts/codepoint_mapper.rb
|
|
310
|
+
- lib/ucode/glyphs/embedded_fonts/codepoint_mapper/correlator_strategy.rb
|
|
311
|
+
- lib/ucode/glyphs/embedded_fonts/codepoint_mapper/strategy.rb
|
|
312
|
+
- lib/ucode/glyphs/embedded_fonts/codepoint_mapper/tounicode_strategy.rb
|
|
313
|
+
- lib/ucode/glyphs/embedded_fonts/codepoint_mapper/trace_strategy.rb
|
|
366
314
|
- lib/ucode/glyphs/embedded_fonts/content_stream_correlator.rb
|
|
367
315
|
- lib/ucode/glyphs/embedded_fonts/font_entry.rb
|
|
316
|
+
- lib/ucode/glyphs/embedded_fonts/mutool.rb
|
|
317
|
+
- lib/ucode/glyphs/embedded_fonts/mutool/draw.rb
|
|
318
|
+
- lib/ucode/glyphs/embedded_fonts/mutool/info.rb
|
|
319
|
+
- lib/ucode/glyphs/embedded_fonts/mutool/show.rb
|
|
320
|
+
- lib/ucode/glyphs/embedded_fonts/mutool/trace.rb
|
|
321
|
+
- lib/ucode/glyphs/embedded_fonts/page_trace_cache.rb
|
|
368
322
|
- lib/ucode/glyphs/embedded_fonts/pdf_indexer.rb
|
|
369
|
-
- lib/ucode/glyphs/embedded_fonts/
|
|
323
|
+
- lib/ucode/glyphs/embedded_fonts/pdf_source.rb
|
|
370
324
|
- lib/ucode/glyphs/embedded_fonts/positional_matcher.rb
|
|
371
325
|
- lib/ucode/glyphs/embedded_fonts/raw_font_descriptor.rb
|
|
372
326
|
- lib/ucode/glyphs/embedded_fonts/renderer.rb
|
|
@@ -1,80 +0,0 @@
|
|
|
1
|
-
# TODO 01 — PDF fetch validation
|
|
2
|
-
|
|
3
|
-
## Status
|
|
4
|
-
|
|
5
|
-
Pending.
|
|
6
|
-
|
|
7
|
-
## Goal
|
|
8
|
-
|
|
9
|
-
Raise a typed `Ucode::CodeChartNotFoundError` when a Unicode Code
|
|
10
|
-
Charts PDF cannot be downloaded or fails content validation. The REQ
|
|
11
|
-
(R1) requires:
|
|
12
|
-
|
|
13
|
-
- HTTP 4xx / 5xx → `CodeChartNotFoundError`
|
|
14
|
-
- `Content-Type: application/pdf`
|
|
15
|
-
- First 4 bytes are `%PDF`
|
|
16
|
-
|
|
17
|
-
## Files
|
|
18
|
-
|
|
19
|
-
- `lib/ucode/error.rb` — add `Ucode::CodeChartNotFoundError` under the
|
|
20
|
-
`GlyphError` subtree.
|
|
21
|
-
- `lib/ucode.rb` — add an `autoload` for the new class so any rescue
|
|
22
|
-
clause triggers one load of `error.rb`.
|
|
23
|
-
- `lib/ucode/fetch/http.rb` — extend `Http.get` with an optional
|
|
24
|
-
`validate:` keyword. When `validate: :pdf`, after a successful
|
|
25
|
-
download, verify the `Content-Type` response header starts with
|
|
26
|
-
`application/pdf` and the first 4 bytes of the body are `%PDF`.
|
|
27
|
-
- `lib/ucode/fetch/code_charts.rb` — pass `validate: :pdf` to `Http.get`
|
|
28
|
-
for every chart PDF download.
|
|
29
|
-
- `spec/ucode/fetch/code_charts_spec.rb` (new) — cover the happy path
|
|
30
|
-
and the validation failure modes.
|
|
31
|
-
|
|
32
|
-
## Design
|
|
33
|
-
|
|
34
|
-
### Why a new error class
|
|
35
|
-
|
|
36
|
-
`FetchError` already covers transport failures, but it doesn't carry
|
|
37
|
-
"this URL produced an HTML error page / 404 / non-PDF body" semantics.
|
|
38
|
-
Splitting the type keeps existing `rescue Ucode::FetchError` callers
|
|
39
|
-
from accidentally swallowing the typed signal that "we expected a PDF
|
|
40
|
-
and didn't get one" — which is a different problem class from "the
|
|
41
|
-
network was down."
|
|
42
|
-
|
|
43
|
-
`CodeChartNotFoundError < Ucode::Error` (under `GlyphError`) reflects
|
|
44
|
-
the REQ's framing: the chart for the requested block is not
|
|
45
|
-
obtainable.
|
|
46
|
-
|
|
47
|
-
### Why `validate:` is optional on `Http.get`
|
|
48
|
-
|
|
49
|
-
`Http` is the single network boundary (per the comment at the top of
|
|
50
|
-
`http.rb`). All callers funnel through it. Adding an optional
|
|
51
|
-
keyword keeps the MECE pattern intact: non-PDF callers (UCD zip,
|
|
52
|
-
Unihan zip, font zip) pass nothing; the single PDF caller passes
|
|
53
|
-
`validate: :pdf`. No second network boundary is needed.
|
|
54
|
-
|
|
55
|
-
### Why no separate "magic bytes" check class
|
|
56
|
-
|
|
57
|
-
Magic-byte verification is 4 lines of code; extracting it into a
|
|
58
|
-
class would be ceremony. Inline check after `write_body`, raising
|
|
59
|
-
the typed error with the offending content-type or magic bytes in
|
|
60
|
-
the context payload.
|
|
61
|
-
|
|
62
|
-
## Acceptance
|
|
63
|
-
|
|
64
|
-
- `Http.get(url, dest:, validate: :pdf)` raises
|
|
65
|
-
`CodeChartNotFoundError` (a) when the response Content-Type is not
|
|
66
|
-
`application/pdf`, (b) when the first 4 bytes are not `%PDF`.
|
|
67
|
-
- `Fetch::CodeCharts.call(version, block_first_cps: [...])` raises
|
|
68
|
-
`CodeChartNotFoundError` when the unicode.org endpoint returns
|
|
69
|
-
4xx/5xx or non-PDF content.
|
|
70
|
-
- Existing callers of `Http.get` that don't pass `validate:` are
|
|
71
|
-
unchanged.
|
|
72
|
-
- Spec coverage: happy path, HTTP 404, wrong content-type, truncated
|
|
73
|
-
body missing the `%PDF` magic.
|
|
74
|
-
|
|
75
|
-
## Out of scope
|
|
76
|
-
|
|
77
|
-
- SHA-256 verification of the PDF — that's a downstream concern (the
|
|
78
|
-
Code Charts are not versioned by hash on unicode.org).
|
|
79
|
-
- Resumable / partial downloads — the existing `Http` writes a
|
|
80
|
-
`.part` then renames; that's sufficient.
|
|
@@ -1,68 +0,0 @@
|
|
|
1
|
-
# TODO 02 — Block name resolver
|
|
2
|
-
|
|
3
|
-
## Status
|
|
4
|
-
|
|
5
|
-
Pending. Depends on nothing.
|
|
6
|
-
|
|
7
|
-
## Goal
|
|
8
|
-
|
|
9
|
-
Add a class method `Ucode::Parsers::Blocks.find_by_name(name)` that
|
|
10
|
-
resolves a Unicode block identifier (e.g. `"Sidetic"`,
|
|
11
|
-
`"Egyptian_Hieroglyphs_Extended-B"`) to the `Ucode::Models::Block`
|
|
12
|
-
instance in a given version's cached `Blocks.txt`.
|
|
13
|
-
|
|
14
|
-
This is the CLI ergonomics glue: the REQ's `ucode code-chart extract
|
|
15
|
-
--block Sidetic` flow takes a human-readable name and needs to know
|
|
16
|
-
the block's range to know which `U+XXXX` codepoints to iterate.
|
|
17
|
-
|
|
18
|
-
## Files
|
|
19
|
-
|
|
20
|
-
- `lib/ucode/parsers/blocks.rb` — add `Blocks.find_by_name(path, name)`.
|
|
21
|
-
- `spec/ucode/parsers/blocks_spec.rb` — cover name lookup, missing-name,
|
|
22
|
-
case-sensitivity.
|
|
23
|
-
|
|
24
|
-
## Design
|
|
25
|
-
|
|
26
|
-
### Method shape
|
|
27
|
-
|
|
28
|
-
```ruby
|
|
29
|
-
# @param path [Pathname, String] path to a Blocks.txt
|
|
30
|
-
# @param name [String] block identifier (matches Models::Block#id)
|
|
31
|
-
# @return [Models::Block, nil] nil when no block matches
|
|
32
|
-
def find_by_name(path, name)
|
|
33
|
-
```
|
|
34
|
-
|
|
35
|
-
Returns nil for "not found" — callers (CLI, Extractor) decide whether
|
|
36
|
-
to raise. This matches `Models::Block` consumers that already expect
|
|
37
|
-
nilable lookups.
|
|
38
|
-
|
|
39
|
-
### Name matching rule
|
|
40
|
-
|
|
41
|
-
`Blocks.txt` uses `name` with whitespace collapsed to underscores
|
|
42
|
-
into `id`. `find_by_name` matches against `id` (the underscored
|
|
43
|
-
form). The REQ's example `--block Sidetic` shows that the caller
|
|
44
|
-
provides the underscored form already. This is consistent with the
|
|
45
|
-
existing `Parsers::Blocks` build logic (`name.gsub(/\s+/, "_")`).
|
|
46
|
-
|
|
47
|
-
### Why a separate method
|
|
48
|
-
|
|
49
|
-
`each_record` streams every block — the caller doesn't want to walk
|
|
50
|
-
~340 blocks for every name lookup. `find_by_name` short-circuits on
|
|
51
|
-
first match.
|
|
52
|
-
|
|
53
|
-
## Acceptance
|
|
54
|
-
|
|
55
|
-
- `find_by_name(path, "Basic_Latin")` returns the Basic Latin block.
|
|
56
|
-
- `find_by_name(path, "Nonexistent")` returns nil.
|
|
57
|
-
- Streaming still works for callers that need every block.
|
|
58
|
-
|
|
59
|
-
## Out of scope
|
|
60
|
-
|
|
61
|
-
- Fuzzy matching — exact match only. Callers validate the user's
|
|
62
|
-
input against `Parsers::Blocks.each_record(path).map(&:id)` to
|
|
63
|
-
surface "did you mean …?" suggestions if we ever want that; for
|
|
64
|
-
now, a clean `UnknownBlockError` at the call site is enough.
|
|
65
|
-
- Database-backed lookup — `Ucode::Database#block_ranges_by_name` is
|
|
66
|
-
a different concern (full UCD index). `find_by_name` operates on
|
|
67
|
-
the cached `Blocks.txt` directly because the CodeChart extractor
|
|
68
|
-
is meant to be runnable without a built database.
|