ucode 0.3.3 → 0.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/config/unicode17_universal_glyph_set.yml +1 -1
- data/lib/ucode/code_chart/extractor.rb +7 -3
- data/lib/ucode/code_chart/writer.rb +1 -1
- data/lib/ucode/error.rb +5 -0
- data/lib/ucode/glyphs/embedded_fonts/catalog.rb +20 -3
- data/lib/ucode/glyphs/embedded_fonts/codepoint_mapper/correlator_strategy.rb +46 -0
- data/lib/ucode/glyphs/embedded_fonts/codepoint_mapper/strategy.rb +54 -0
- data/lib/ucode/glyphs/embedded_fonts/codepoint_mapper/tounicode_strategy.rb +41 -0
- data/lib/ucode/glyphs/embedded_fonts/codepoint_mapper/trace_strategy.rb +64 -0
- data/lib/ucode/glyphs/embedded_fonts/codepoint_mapper.rb +143 -90
- data/lib/ucode/glyphs/embedded_fonts/mutool/draw.rb +26 -0
- data/lib/ucode/glyphs/embedded_fonts/mutool/info.rb +23 -0
- data/lib/ucode/glyphs/embedded_fonts/mutool/show.rb +36 -0
- data/lib/ucode/glyphs/embedded_fonts/mutool/trace.rb +26 -0
- data/lib/ucode/glyphs/embedded_fonts/mutool.rb +30 -0
- data/lib/ucode/glyphs/embedded_fonts/page_trace_cache.rb +90 -0
- data/lib/ucode/glyphs/embedded_fonts/pdf_indexer.rb +50 -56
- data/lib/ucode/glyphs/embedded_fonts/{pdf_location.rb → pdf_source.rb} +1 -1
- data/lib/ucode/glyphs/embedded_fonts/trace_runner.rb +12 -28
- data/lib/ucode/glyphs/embedded_fonts.rb +3 -1
- data/lib/ucode/glyphs/real_fonts/coverage_auditor.rb +0 -4
- data/lib/ucode/glyphs/real_fonts/font_coverage_report.rb +0 -2
- data/lib/ucode/glyphs/real_fonts/writer.rb +0 -2
- data/lib/ucode/glyphs/sources/pillar1_embedded_tounicode.rb +1 -1
- data/lib/ucode/version.rb +1 -1
- metadata +13 -74
- data/TODO.extract-code-chart/01-pdf-fetch-validation.md +0 -80
- data/TODO.extract-code-chart/02-block-name-resolver.md +0 -68
- data/TODO.extract-code-chart/03-codechart-namespace.md +0 -82
- data/TODO.extract-code-chart/04-codechart-extractor.md +0 -154
- data/TODO.extract-code-chart/05-provenance-and-sidecar.md +0 -147
- data/TODO.extract-code-chart/06-codechart-writer.md +0 -134
- data/TODO.extract-code-chart/07-codechart-cli.md +0 -135
- data/TODO.extract-code-chart/08-specs.md +0 -87
- data/TODO.full/00-README.md +0 -116
- data/TODO.full/01-panglyph-vision.md +0 -112
- data/TODO.full/02-panglyph-repo-bootstrap.md +0 -184
- data/TODO.full/03-panglyph-font-builder.md +0 -201
- data/TODO.full/04-panglyph-publish-pipeline.md +0 -126
- data/TODO.full/05-ucode-0-1-1-release.md +0 -139
- data/TODO.full/06-fontisan-remove-audit.md +0 -142
- data/TODO.full/07-fontisan-remove-ucd.md +0 -125
- data/TODO.full/08-archive-private-bin-build.md +0 -143
- data/TODO.full/09-archive-public-structure.md +0 -164
- data/TODO.full/10-fontist-org-woff-glyphs.md +0 -131
- data/TODO.full/11-fontist-org-audit-coverage.md +0 -140
- data/TODO.full/12-implementation-order.md +0 -216
- data/TODO.full/13-fontisan-font-writer-api.md +0 -189
- data/TODO.full/14-fontisan-table-writers.md +0 -66
- data/TODO.full/15-panglyph-builder-real.md +0 -82
- data/TODO.full/16-archive-public-sync-workflows.md +0 -167
- data/TODO.full/17-fontist-org-font-picker.md +0 -73
- data/TODO.full/18-comprehensive-spec-coverage.md +0 -64
- data/TODO.full/19-ucode-0-1-2-patch.md +0 -32
- data/TODO.full/20-fontisan-0-2-23-release.md +0 -52
- data/TODO.new/00-README.md +0 -96
- data/TODO.new/01-pillar-terminology-alignment.md +0 -69
- data/TODO.new/02-audit-schema-design.md +0 -255
- data/TODO.new/03-directory-output-spec.md +0 -203
- data/TODO.new/04-fontist-org-contract.md +0 -173
- data/TODO.new/05-baseline-unicode17-coverage-audit.md +0 -144
- data/TODO.new/06-audit-namespace-skeleton.md +0 -105
- data/TODO.new/07-audit-models-port.md +0 -132
- data/TODO.new/08-extractors-cheap-port.md +0 -113
- data/TODO.new/09-extractors-expensive-port.md +0 -99
- data/TODO.new/10-aggregations-ucd-rewrite.md +0 -168
- data/TODO.new/11-differ-and-library-auditor-port.md +0 -102
- data/TODO.new/12-formatters-port.md +0 -115
- data/TODO.new/13-directory-emitter.md +0 -147
- data/TODO.new/14-html-face-browser.md +0 -144
- data/TODO.new/15-html-library-browser.md +0 -102
- data/TODO.new/16-cli-audit-subcommands.md +0 -142
- data/TODO.new/17-fontisan-cleanup-audit.md +0 -147
- data/TODO.new/18-fontisan-cleanup-ucd.md +0 -156
- data/TODO.new/19-fontisan-docs-update.md +0 -155
- data/TODO.new/20-canonical-resolver-4-tier.md +0 -182
- data/TODO.new/21-canonical-unicode17-build.md +0 -148
- data/TODO.new/22-implementation-order.md +0 -176
- data/TODO.new/23-universal-glyph-set-source-map.md +0 -312
- data/TODO.new/24-universal-glyph-set-build.md +0 -189
- data/TODO.new/25-font-audit-against-universal-set.md +0 -195
- data/TODO.new/26-missing-glyph-reporter.md +0 -189
- data/TODO.new/27-fontist-org-consumer-integration.md +0 -200
- data/TODO.new/28-implementation-order-update.md +0 -187
- data/TODO.new/29-universal-set-curation-uc17.md +0 -312
- data/TODO.new/30-tier1-font-acquisition.md +0 -241
- data/TODO.new/31-universal-set-production-build.md +0 -205
- data/TODO.new/32-uc17-coverage-matrix.md +0 -165
- data/TODO.new/33-specialist-font-acquisition-refresh.md +0 -138
- data/TODO.new/34-pillar2-content-stream-correlator.md +0 -147
- data/TODO.new/35-universal-set-production-run.md +0 -160
- data/TODO.new/36-per-font-coverage-audit.md +0 -145
- data/TODO.new/37-coverage-highlight-reporter.md +0 -125
- data/TODO.new/38-fontist-org-glyph-consumer.md +0 -141
- data/TODO.new/39-implementation-order-update-32-38.md +0 -258
- data/TODO.new/40-archive-private-uses-ucode-audit.md +0 -124
- data/TODO.new/41-ucode-unicode-archive-bridge.md +0 -160
|
@@ -0,0 +1,36 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require "pathname"
|
|
4
|
+
require "tempfile"
|
|
5
|
+
|
|
6
|
+
require "ucode/glyphs/embedded_fonts/mutool"
|
|
7
|
+
|
|
8
|
+
module Ucode
|
|
9
|
+
module Glyphs
|
|
10
|
+
module EmbeddedFonts
|
|
11
|
+
module Mutool
|
|
12
|
+
class Show
|
|
13
|
+
def initialize(runner: SystemRunner.new)
|
|
14
|
+
@runner = runner
|
|
15
|
+
end
|
|
16
|
+
|
|
17
|
+
def grep(pdf, *obj_ids)
|
|
18
|
+
return "" if obj_ids.empty?
|
|
19
|
+
|
|
20
|
+
@runner.run("mutool", "show", "-g", pdf.to_s,
|
|
21
|
+
*obj_ids.map(&:to_s))
|
|
22
|
+
end
|
|
23
|
+
|
|
24
|
+
def stream(pdf, obj_id)
|
|
25
|
+
Tempfile.create("mutool-stream") do |tmp|
|
|
26
|
+
tmp.close
|
|
27
|
+
@runner.run("mutool", "show", "-o", tmp.path, "-b",
|
|
28
|
+
pdf.to_s, obj_id.to_s)
|
|
29
|
+
File.binread(tmp.path).force_encoding("UTF-8")
|
|
30
|
+
end
|
|
31
|
+
end
|
|
32
|
+
end
|
|
33
|
+
end
|
|
34
|
+
end
|
|
35
|
+
end
|
|
36
|
+
end
|
|
@@ -0,0 +1,26 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require "pathname"
|
|
4
|
+
|
|
5
|
+
require "ucode/glyphs/embedded_fonts/mutool"
|
|
6
|
+
|
|
7
|
+
module Ucode
|
|
8
|
+
module Glyphs
|
|
9
|
+
module EmbeddedFonts
|
|
10
|
+
module Mutool
|
|
11
|
+
class Trace
|
|
12
|
+
def initialize(runner: SystemRunner.new)
|
|
13
|
+
@runner = runner
|
|
14
|
+
end
|
|
15
|
+
|
|
16
|
+
def call(pdf, *pages)
|
|
17
|
+
return "" if pages.empty?
|
|
18
|
+
|
|
19
|
+
@runner.run("mutool", "trace", pdf.to_s,
|
|
20
|
+
*pages.map(&:to_s))
|
|
21
|
+
end
|
|
22
|
+
end
|
|
23
|
+
end
|
|
24
|
+
end
|
|
25
|
+
end
|
|
26
|
+
end
|
|
@@ -0,0 +1,30 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require "open3"
|
|
4
|
+
|
|
5
|
+
require "ucode/error"
|
|
6
|
+
|
|
7
|
+
module Ucode
|
|
8
|
+
module Glyphs
|
|
9
|
+
module EmbeddedFonts
|
|
10
|
+
module Mutool
|
|
11
|
+
autoload :Info, "ucode/glyphs/embedded_fonts/mutool/info"
|
|
12
|
+
autoload :Show, "ucode/glyphs/embedded_fonts/mutool/show"
|
|
13
|
+
autoload :Draw, "ucode/glyphs/embedded_fonts/mutool/draw"
|
|
14
|
+
autoload :Trace, "ucode/glyphs/embedded_fonts/mutool/trace"
|
|
15
|
+
|
|
16
|
+
class SystemRunner
|
|
17
|
+
def run(*argv)
|
|
18
|
+
out, err, status = Open3.capture3(*argv)
|
|
19
|
+
return out + err if status.success?
|
|
20
|
+
|
|
21
|
+
raise Ucode::MutoolError.new(
|
|
22
|
+
"mutool failed (exit #{status.exitstatus}): #{err.strip}",
|
|
23
|
+
context: { argv: argv },
|
|
24
|
+
)
|
|
25
|
+
end
|
|
26
|
+
end
|
|
27
|
+
end
|
|
28
|
+
end
|
|
29
|
+
end
|
|
30
|
+
end
|
|
@@ -0,0 +1,90 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require "pathname"
|
|
4
|
+
|
|
5
|
+
require "ucode/glyphs/embedded_fonts/mutool"
|
|
6
|
+
require "ucode/glyphs/embedded_fonts/trace_parser"
|
|
7
|
+
|
|
8
|
+
module Ucode
|
|
9
|
+
module Glyphs
|
|
10
|
+
module EmbeddedFonts
|
|
11
|
+
# Per-PDF trace cache. Runs `mutool trace` once per page
|
|
12
|
+
# (1..page_count), parses each result, and stores glyphs
|
|
13
|
+
# grouped by page so each TraceStrategy can correlate
|
|
14
|
+
# positionally without re-tracing.
|
|
15
|
+
#
|
|
16
|
+
# Replaces the O(F × P) explosion in the pre-cache TraceStrategy,
|
|
17
|
+
# which spawned mutool trace once per page per CID font. For
|
|
18
|
+
# Code Charts PDFs with F trace-needing fonts and P pages, that
|
|
19
|
+
# was F×P subprocess invocations; this cache makes it exactly P.
|
|
20
|
+
#
|
|
21
|
+
# Why per-page grouping: TraceCorrelator's algorithm clusters
|
|
22
|
+
# label glyphs by Y bucket, and Y positions are page-local.
|
|
23
|
+
#
|
|
24
|
+
# Lazy: glyphs are only fetched when a strategy first asks for
|
|
25
|
+
# them. PDFs where every font has /ToUnicode never trigger the
|
|
26
|
+
# trace at all.
|
|
27
|
+
class PageTraceCache
|
|
28
|
+
# @param pdf [Pathname, String]
|
|
29
|
+
# @param page_count [Integer] total pages in the PDF
|
|
30
|
+
# @param mutool [Mutool::Trace]
|
|
31
|
+
def initialize(pdf:, page_count:, mutool: Mutool::Trace.new)
|
|
32
|
+
@pdf = Pathname.new(pdf)
|
|
33
|
+
@page_count = page_count
|
|
34
|
+
@mutool = mutool
|
|
35
|
+
end
|
|
36
|
+
|
|
37
|
+
# @return [Array<Array<TraceGlyph>>] one Array per page,
|
|
38
|
+
# 1-indexed (index 0 is unused). Each inner array holds
|
|
39
|
+
# every glyph emitted by `mutool trace` on that page.
|
|
40
|
+
def glyphs_by_page
|
|
41
|
+
@glyphs_by_page ||= fetch_glyphs_by_page
|
|
42
|
+
end
|
|
43
|
+
|
|
44
|
+
# @param base_font [String] specimen font BaseFont name
|
|
45
|
+
# @yieldparam page [Integer] 1-based page number
|
|
46
|
+
# @yieldparam glyphs [Array<TraceGlyph>] every glyph on that
|
|
47
|
+
# page (all fonts — TraceCorrelator filters internally)
|
|
48
|
+
# @return [Boolean] true if at least one page references the
|
|
49
|
+
# font; false otherwise
|
|
50
|
+
# @return [Enumerator] if no block given
|
|
51
|
+
def each_page_for(base_font)
|
|
52
|
+
return enum_for(:each_page_for, base_font) unless block_given?
|
|
53
|
+
|
|
54
|
+
present_in_any = false
|
|
55
|
+
glyphs_by_page.each_with_index do |glyphs, idx|
|
|
56
|
+
next if idx.zero?
|
|
57
|
+
|
|
58
|
+
present = glyphs.any? { |g| g.font_name == base_font }
|
|
59
|
+
next unless present
|
|
60
|
+
|
|
61
|
+
present_in_any = true
|
|
62
|
+
yield idx, glyphs
|
|
63
|
+
end
|
|
64
|
+
present_in_any
|
|
65
|
+
end
|
|
66
|
+
|
|
67
|
+
# @param base_font [String]
|
|
68
|
+
# @return [Boolean] true if any page references this font
|
|
69
|
+
def references_font?(base_font)
|
|
70
|
+
glyphs_by_page.any? do |page_glyphs|
|
|
71
|
+
page_glyphs.any? { |g| g.font_name == base_font }
|
|
72
|
+
end
|
|
73
|
+
end
|
|
74
|
+
|
|
75
|
+
private
|
|
76
|
+
|
|
77
|
+
def fetch_glyphs_by_page
|
|
78
|
+
result = [[]] # index 0 unused (1-based pages)
|
|
79
|
+
return result unless @page_count.positive?
|
|
80
|
+
|
|
81
|
+
(1..@page_count).each do |page|
|
|
82
|
+
xml = @mutool.call(@pdf, page)
|
|
83
|
+
result << TraceParser.parse(xml)
|
|
84
|
+
end
|
|
85
|
+
result
|
|
86
|
+
end
|
|
87
|
+
end
|
|
88
|
+
end
|
|
89
|
+
end
|
|
90
|
+
end
|
|
@@ -1,23 +1,23 @@
|
|
|
1
1
|
# frozen_string_literal: true
|
|
2
2
|
|
|
3
|
-
require "open3"
|
|
4
3
|
require "pathname"
|
|
5
4
|
|
|
5
|
+
require "ucode/glyphs/embedded_fonts/mutool"
|
|
6
|
+
|
|
6
7
|
module Ucode
|
|
7
8
|
module Glyphs
|
|
8
9
|
module EmbeddedFonts
|
|
9
|
-
# Walks the Code Charts PDF once via `mutool info` + `mutool show`
|
|
10
|
-
# and builds an Array of {RawFontDescriptor} — one per Type0 font
|
|
11
|
-
# that has the required descendant CIDFont, FontDescriptor, and
|
|
12
|
-
# FontFile2/3 + Identity CIDToGIDMap.
|
|
13
|
-
#
|
|
14
|
-
# Pure subprocess + parsing concern. Does NOT resolve codepoint →
|
|
15
|
-
# GID (that's {CodepointMapper}'s job). The descriptor carries
|
|
16
|
-
# every ref the mapper needs to do its work.
|
|
17
10
|
class PdfIndexer
|
|
18
|
-
|
|
19
|
-
|
|
11
|
+
Type0Entry = Struct.new(:font_obj_id, :base_font, keyword_init: true)
|
|
12
|
+
|
|
13
|
+
# @param source [PdfSource]
|
|
14
|
+
# @param mutool_info [Mutool::Info] injectable for tests
|
|
15
|
+
# @param mutool_show [Mutool::Show] injectable for tests
|
|
16
|
+
def initialize(source:, mutool_info: Mutool::Info.new,
|
|
17
|
+
mutool_show: Mutool::Show.new)
|
|
20
18
|
@source = source
|
|
19
|
+
@mutool_info = mutool_info
|
|
20
|
+
@mutool_show = mutool_show
|
|
21
21
|
end
|
|
22
22
|
|
|
23
23
|
# @return [Array<RawFontDescriptor>]
|
|
@@ -68,11 +68,45 @@ module Ucode
|
|
|
68
68
|
# @param base_font [String] e.g. "GPJAHB+WolofGaraySansSerif"
|
|
69
69
|
# @return [Boolean] true if this font appears on any page
|
|
70
70
|
def font_appears?(base_font)
|
|
71
|
-
|
|
71
|
+
type0_base_fonts.include?(base_font)
|
|
72
72
|
end
|
|
73
73
|
|
|
74
74
|
private
|
|
75
75
|
|
|
76
|
+
# Single source of truth for the Type0 font list. Both
|
|
77
|
+
# `discover_type0_fonts` (keyed by obj_id) and
|
|
78
|
+
# `font_appears?` (membership check by name) derive their
|
|
79
|
+
# answer from this parse, so changes to `mutool info`'s output
|
|
80
|
+
# format break in exactly one place.
|
|
81
|
+
#
|
|
82
|
+
# @return [Array<Type0Entry>]
|
|
83
|
+
def type0_entries
|
|
84
|
+
@type0_entries ||= parse_type0_entries
|
|
85
|
+
end
|
|
86
|
+
|
|
87
|
+
# @return [Set<String>] every Type0 BaseFont name seen
|
|
88
|
+
def type0_base_fonts
|
|
89
|
+
@type0_base_fonts ||= type0_entries.each_with_object(Set.new) do |e, s|
|
|
90
|
+
s << e.base_font
|
|
91
|
+
end
|
|
92
|
+
end
|
|
93
|
+
|
|
94
|
+
def parse_type0_entries
|
|
95
|
+
seen = Set.new
|
|
96
|
+
mutool_info_text.each_line.filter_map do |line|
|
|
97
|
+
next unless line.include?("Type0")
|
|
98
|
+
|
|
99
|
+
m = line.match(/Type0\s+'([^']+)'\s+\S+\s+\((\d+)\s+0\s+R\)/)
|
|
100
|
+
next unless m
|
|
101
|
+
|
|
102
|
+
font_obj_id = m[2].to_i
|
|
103
|
+
next if seen.include?(font_obj_id)
|
|
104
|
+
|
|
105
|
+
seen << font_obj_id
|
|
106
|
+
Type0Entry.new(font_obj_id: font_obj_id, base_font: m[1])
|
|
107
|
+
end
|
|
108
|
+
end
|
|
109
|
+
|
|
76
110
|
def build_descriptors(type0_refs, type0_dicts, descendant_dicts, fontdesc_dicts)
|
|
77
111
|
type0_refs.filter_map do |font_obj_id, base_font|
|
|
78
112
|
build_descriptor(
|
|
@@ -133,35 +167,15 @@ module Ucode
|
|
|
133
167
|
# ---- mutool subprocess + dict parsing ----------------------------
|
|
134
168
|
|
|
135
169
|
def discover_type0_fonts
|
|
136
|
-
|
|
137
|
-
|
|
138
|
-
seen = Set.new
|
|
139
|
-
text.each_line do |line|
|
|
140
|
-
next unless line.include?("Type0")
|
|
141
|
-
|
|
142
|
-
m = line.match(/Type0\s+'([^']+)'\s+\S+\s+\((\d+)\s+0\s+R\)/)
|
|
143
|
-
next unless m
|
|
144
|
-
|
|
145
|
-
font_obj_id = m[2].to_i
|
|
146
|
-
next if seen.include?(font_obj_id)
|
|
147
|
-
|
|
148
|
-
seen << font_obj_id
|
|
149
|
-
result[font_obj_id] = m[1]
|
|
170
|
+
type0_entries.to_h do |e|
|
|
171
|
+
[e.font_obj_id, e.base_font]
|
|
150
172
|
end
|
|
151
|
-
result
|
|
152
173
|
end
|
|
153
174
|
|
|
154
175
|
def fetch_objects(obj_ids)
|
|
155
176
|
return {} if obj_ids.empty?
|
|
156
177
|
|
|
157
|
-
|
|
158
|
-
@source.pdf_to_s].concat(obj_ids.map(&:to_s))
|
|
159
|
-
out, err, status = Open3.capture3(*args)
|
|
160
|
-
unless status.success?
|
|
161
|
-
raise Ucode::EmbeddedFontsMissingError,
|
|
162
|
-
"mutool show failed: #{err.strip}"
|
|
163
|
-
end
|
|
164
|
-
|
|
178
|
+
out = @mutool_show.grep(@source.pdf_to_s, *obj_ids)
|
|
165
179
|
parse_grep_output(out)
|
|
166
180
|
end
|
|
167
181
|
|
|
@@ -208,27 +222,7 @@ module Ucode
|
|
|
208
222
|
end
|
|
209
223
|
|
|
210
224
|
def mutool_info_text
|
|
211
|
-
@mutool_info_text ||=
|
|
212
|
-
end
|
|
213
|
-
|
|
214
|
-
def run_mutool_info
|
|
215
|
-
out, err, status = Open3.capture3("mutool", "info", @source.pdf_to_s)
|
|
216
|
-
status.success? ? out + err : ""
|
|
217
|
-
end
|
|
218
|
-
|
|
219
|
-
def font_entries_cache
|
|
220
|
-
@font_entries_cache ||= begin
|
|
221
|
-
result = {}
|
|
222
|
-
mutool_info_text.each_line do |line|
|
|
223
|
-
next unless line.include?("Type0")
|
|
224
|
-
|
|
225
|
-
font_match = line.match(/Type0\s+'([^']+)'/)
|
|
226
|
-
next unless font_match
|
|
227
|
-
|
|
228
|
-
result[font_match[1]] = true
|
|
229
|
-
end
|
|
230
|
-
result
|
|
231
|
-
end
|
|
225
|
+
@mutool_info_text ||= @mutool_info.call(@source.pdf_to_s)
|
|
232
226
|
end
|
|
233
227
|
end
|
|
234
228
|
end
|
|
@@ -27,7 +27,7 @@ module Ucode
|
|
|
27
27
|
# the BaseFont (e.g. `CIAIIP+Uni2000Generalpunctuation.ttf`).
|
|
28
28
|
# Re-runs skip extraction when the cached file is newer than the
|
|
29
29
|
# PDF.
|
|
30
|
-
class
|
|
30
|
+
class PdfSource
|
|
31
31
|
attr_reader :pdf_path, :cache_dir
|
|
32
32
|
|
|
33
33
|
# @param pdf [String, Pathname, nil] path to a Code Charts PDF
|
|
@@ -1,51 +1,35 @@
|
|
|
1
1
|
# frozen_string_literal: true
|
|
2
2
|
|
|
3
|
-
require "open3"
|
|
4
3
|
require "pathname"
|
|
5
4
|
|
|
6
|
-
require "ucode/
|
|
5
|
+
require "ucode/glyphs/embedded_fonts/mutool"
|
|
6
|
+
require "ucode/glyphs/embedded_fonts/trace_parser"
|
|
7
7
|
|
|
8
8
|
module Ucode
|
|
9
9
|
module Glyphs
|
|
10
10
|
module EmbeddedFonts
|
|
11
11
|
# Thin I/O wrapper around `mutool trace <pdf> <page>`.
|
|
12
12
|
#
|
|
13
|
-
#
|
|
14
|
-
#
|
|
15
|
-
#
|
|
16
|
-
#
|
|
17
|
-
# The only class in the trace pipeline that touches the
|
|
18
|
-
# filesystem / spawns subprocesses. Everything upstream
|
|
19
|
-
# (parser, correlator) is pure.
|
|
13
|
+
# Delegates the actual subprocess to {Mutool::Trace} and the
|
|
14
|
+
# XML parsing to {TraceParser}. Kept for backward compatibility
|
|
15
|
+
# with callers that pre-date the Mutool seam.
|
|
20
16
|
class TraceRunner
|
|
21
17
|
# @param pdf_path [Pathname, String]
|
|
22
|
-
|
|
18
|
+
# @param mutool [Mutool::Trace] injectable for tests
|
|
19
|
+
def initialize(pdf_path, mutool: Mutool::Trace.new)
|
|
23
20
|
@pdf_path = Pathname.new(pdf_path)
|
|
21
|
+
@mutool = mutool
|
|
24
22
|
end
|
|
25
23
|
|
|
26
24
|
# @param page_numbers [Array<Integer>] 1-based PDF page numbers
|
|
27
25
|
# @return [Array<TraceGlyph>]
|
|
28
26
|
def trace(page_numbers)
|
|
29
|
-
|
|
30
|
-
end
|
|
31
|
-
|
|
32
|
-
private
|
|
27
|
+
return [] if page_numbers.empty?
|
|
33
28
|
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
end
|
|
38
|
-
|
|
39
|
-
def run_mutool(page)
|
|
40
|
-
out, err, status = Open3.capture3(
|
|
41
|
-
"mutool", "trace", @pdf_path.to_s, page.to_s,
|
|
42
|
-
)
|
|
43
|
-
unless status.success?
|
|
44
|
-
raise Ucode::EmbeddedFontsMissingError,
|
|
45
|
-
"mutool trace failed: #{(out + err).strip}"
|
|
29
|
+
page_numbers.flat_map do |page|
|
|
30
|
+
xml = @mutool.call(@pdf_path, page)
|
|
31
|
+
TraceParser.parse(xml)
|
|
46
32
|
end
|
|
47
|
-
|
|
48
|
-
out + err
|
|
49
33
|
end
|
|
50
34
|
end
|
|
51
35
|
end
|
|
@@ -36,7 +36,7 @@ module Ucode
|
|
|
36
36
|
# `mutool info` (font enumeration) and `mutool show -b -o` (raw
|
|
37
37
|
# stream extraction).
|
|
38
38
|
module EmbeddedFonts
|
|
39
|
-
autoload :
|
|
39
|
+
autoload :PdfSource, "ucode/glyphs/embedded_fonts/pdf_source"
|
|
40
40
|
autoload :ToUnicode, "ucode/glyphs/embedded_fonts/tounicode"
|
|
41
41
|
autoload :FontEntry, "ucode/glyphs/embedded_fonts/font_entry"
|
|
42
42
|
autoload :RawFontDescriptor, "ucode/glyphs/embedded_fonts/raw_font_descriptor"
|
|
@@ -50,6 +50,8 @@ module Ucode
|
|
|
50
50
|
autoload :TraceParser, "ucode/glyphs/embedded_fonts/trace_parser"
|
|
51
51
|
autoload :TraceCorrelator, "ucode/glyphs/embedded_fonts/trace_correlator"
|
|
52
52
|
autoload :TraceRunner, "ucode/glyphs/embedded_fonts/trace_runner"
|
|
53
|
+
autoload :Mutool, "ucode/glyphs/embedded_fonts/mutool"
|
|
54
|
+
autoload :PageTraceCache, "ucode/glyphs/embedded_fonts/page_trace_cache"
|
|
53
55
|
autoload :Svg, "ucode/glyphs/embedded_fonts/svg"
|
|
54
56
|
autoload :Renderer, "ucode/glyphs/embedded_fonts/renderer"
|
|
55
57
|
autoload :Writer, "ucode/glyphs/embedded_fonts/writer"
|
|
@@ -31,7 +31,7 @@ module Ucode
|
|
|
31
31
|
# @param renderer [EmbeddedFonts::Renderer] the renderer to
|
|
32
32
|
# delegate to. Callers typically construct it with the
|
|
33
33
|
# {EmbeddedFonts::Catalog} built from the resolved Code
|
|
34
|
-
# Charts {EmbeddedFonts::
|
|
34
|
+
# Charts {EmbeddedFonts::PdfSource}. To enable pillar-2
|
|
35
35
|
# fallback, that Catalog must be constructed with
|
|
36
36
|
# +correlator_configs:+.
|
|
37
37
|
def initialize(renderer:)
|
data/lib/ucode/version.rb
CHANGED
metadata
CHANGED
|
@@ -1,14 +1,14 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: ucode
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.
|
|
4
|
+
version: 0.4.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Ribose Inc.
|
|
8
8
|
autorequire:
|
|
9
9
|
bindir: exe
|
|
10
10
|
cert_chain: []
|
|
11
|
-
date: 2026-07-
|
|
11
|
+
date: 2026-07-08 00:00:00.000000000 Z
|
|
12
12
|
dependencies:
|
|
13
13
|
- !ruby/object:Gem::Dependency
|
|
14
14
|
name: base64
|
|
@@ -162,77 +162,6 @@ files:
|
|
|
162
162
|
- Gemfile
|
|
163
163
|
- README.md
|
|
164
164
|
- Rakefile
|
|
165
|
-
- TODO.extract-code-chart/01-pdf-fetch-validation.md
|
|
166
|
-
- TODO.extract-code-chart/02-block-name-resolver.md
|
|
167
|
-
- TODO.extract-code-chart/03-codechart-namespace.md
|
|
168
|
-
- TODO.extract-code-chart/04-codechart-extractor.md
|
|
169
|
-
- TODO.extract-code-chart/05-provenance-and-sidecar.md
|
|
170
|
-
- TODO.extract-code-chart/06-codechart-writer.md
|
|
171
|
-
- TODO.extract-code-chart/07-codechart-cli.md
|
|
172
|
-
- TODO.extract-code-chart/08-specs.md
|
|
173
|
-
- TODO.full/00-README.md
|
|
174
|
-
- TODO.full/01-panglyph-vision.md
|
|
175
|
-
- TODO.full/02-panglyph-repo-bootstrap.md
|
|
176
|
-
- TODO.full/03-panglyph-font-builder.md
|
|
177
|
-
- TODO.full/04-panglyph-publish-pipeline.md
|
|
178
|
-
- TODO.full/05-ucode-0-1-1-release.md
|
|
179
|
-
- TODO.full/06-fontisan-remove-audit.md
|
|
180
|
-
- TODO.full/07-fontisan-remove-ucd.md
|
|
181
|
-
- TODO.full/08-archive-private-bin-build.md
|
|
182
|
-
- TODO.full/09-archive-public-structure.md
|
|
183
|
-
- TODO.full/10-fontist-org-woff-glyphs.md
|
|
184
|
-
- TODO.full/11-fontist-org-audit-coverage.md
|
|
185
|
-
- TODO.full/12-implementation-order.md
|
|
186
|
-
- TODO.full/13-fontisan-font-writer-api.md
|
|
187
|
-
- TODO.full/14-fontisan-table-writers.md
|
|
188
|
-
- TODO.full/15-panglyph-builder-real.md
|
|
189
|
-
- TODO.full/16-archive-public-sync-workflows.md
|
|
190
|
-
- TODO.full/17-fontist-org-font-picker.md
|
|
191
|
-
- TODO.full/18-comprehensive-spec-coverage.md
|
|
192
|
-
- TODO.full/19-ucode-0-1-2-patch.md
|
|
193
|
-
- TODO.full/20-fontisan-0-2-23-release.md
|
|
194
|
-
- TODO.new/00-README.md
|
|
195
|
-
- TODO.new/01-pillar-terminology-alignment.md
|
|
196
|
-
- TODO.new/02-audit-schema-design.md
|
|
197
|
-
- TODO.new/03-directory-output-spec.md
|
|
198
|
-
- TODO.new/04-fontist-org-contract.md
|
|
199
|
-
- TODO.new/05-baseline-unicode17-coverage-audit.md
|
|
200
|
-
- TODO.new/06-audit-namespace-skeleton.md
|
|
201
|
-
- TODO.new/07-audit-models-port.md
|
|
202
|
-
- TODO.new/08-extractors-cheap-port.md
|
|
203
|
-
- TODO.new/09-extractors-expensive-port.md
|
|
204
|
-
- TODO.new/10-aggregations-ucd-rewrite.md
|
|
205
|
-
- TODO.new/11-differ-and-library-auditor-port.md
|
|
206
|
-
- TODO.new/12-formatters-port.md
|
|
207
|
-
- TODO.new/13-directory-emitter.md
|
|
208
|
-
- TODO.new/14-html-face-browser.md
|
|
209
|
-
- TODO.new/15-html-library-browser.md
|
|
210
|
-
- TODO.new/16-cli-audit-subcommands.md
|
|
211
|
-
- TODO.new/17-fontisan-cleanup-audit.md
|
|
212
|
-
- TODO.new/18-fontisan-cleanup-ucd.md
|
|
213
|
-
- TODO.new/19-fontisan-docs-update.md
|
|
214
|
-
- TODO.new/20-canonical-resolver-4-tier.md
|
|
215
|
-
- TODO.new/21-canonical-unicode17-build.md
|
|
216
|
-
- TODO.new/22-implementation-order.md
|
|
217
|
-
- TODO.new/23-universal-glyph-set-source-map.md
|
|
218
|
-
- TODO.new/24-universal-glyph-set-build.md
|
|
219
|
-
- TODO.new/25-font-audit-against-universal-set.md
|
|
220
|
-
- TODO.new/26-missing-glyph-reporter.md
|
|
221
|
-
- TODO.new/27-fontist-org-consumer-integration.md
|
|
222
|
-
- TODO.new/28-implementation-order-update.md
|
|
223
|
-
- TODO.new/29-universal-set-curation-uc17.md
|
|
224
|
-
- TODO.new/30-tier1-font-acquisition.md
|
|
225
|
-
- TODO.new/31-universal-set-production-build.md
|
|
226
|
-
- TODO.new/32-uc17-coverage-matrix.md
|
|
227
|
-
- TODO.new/33-specialist-font-acquisition-refresh.md
|
|
228
|
-
- TODO.new/34-pillar2-content-stream-correlator.md
|
|
229
|
-
- TODO.new/35-universal-set-production-run.md
|
|
230
|
-
- TODO.new/36-per-font-coverage-audit.md
|
|
231
|
-
- TODO.new/37-coverage-highlight-reporter.md
|
|
232
|
-
- TODO.new/38-fontist-org-glyph-consumer.md
|
|
233
|
-
- TODO.new/39-implementation-order-update-32-38.md
|
|
234
|
-
- TODO.new/40-archive-private-uses-ucode-audit.md
|
|
235
|
-
- TODO.new/41-ucode-unicode-archive-bridge.md
|
|
236
165
|
- UCODE_CHANGELOG.md
|
|
237
166
|
- config/specialist_fonts.yml
|
|
238
167
|
- config/unicode17_tier1_fonts.yml
|
|
@@ -363,10 +292,20 @@ files:
|
|
|
363
292
|
- lib/ucode/glyphs/embedded_fonts.rb
|
|
364
293
|
- lib/ucode/glyphs/embedded_fonts/catalog.rb
|
|
365
294
|
- lib/ucode/glyphs/embedded_fonts/codepoint_mapper.rb
|
|
295
|
+
- lib/ucode/glyphs/embedded_fonts/codepoint_mapper/correlator_strategy.rb
|
|
296
|
+
- lib/ucode/glyphs/embedded_fonts/codepoint_mapper/strategy.rb
|
|
297
|
+
- lib/ucode/glyphs/embedded_fonts/codepoint_mapper/tounicode_strategy.rb
|
|
298
|
+
- lib/ucode/glyphs/embedded_fonts/codepoint_mapper/trace_strategy.rb
|
|
366
299
|
- lib/ucode/glyphs/embedded_fonts/content_stream_correlator.rb
|
|
367
300
|
- lib/ucode/glyphs/embedded_fonts/font_entry.rb
|
|
301
|
+
- lib/ucode/glyphs/embedded_fonts/mutool.rb
|
|
302
|
+
- lib/ucode/glyphs/embedded_fonts/mutool/draw.rb
|
|
303
|
+
- lib/ucode/glyphs/embedded_fonts/mutool/info.rb
|
|
304
|
+
- lib/ucode/glyphs/embedded_fonts/mutool/show.rb
|
|
305
|
+
- lib/ucode/glyphs/embedded_fonts/mutool/trace.rb
|
|
306
|
+
- lib/ucode/glyphs/embedded_fonts/page_trace_cache.rb
|
|
368
307
|
- lib/ucode/glyphs/embedded_fonts/pdf_indexer.rb
|
|
369
|
-
- lib/ucode/glyphs/embedded_fonts/
|
|
308
|
+
- lib/ucode/glyphs/embedded_fonts/pdf_source.rb
|
|
370
309
|
- lib/ucode/glyphs/embedded_fonts/positional_matcher.rb
|
|
371
310
|
- lib/ucode/glyphs/embedded_fonts/raw_font_descriptor.rb
|
|
372
311
|
- lib/ucode/glyphs/embedded_fonts/renderer.rb
|
|
@@ -1,80 +0,0 @@
|
|
|
1
|
-
# TODO 01 — PDF fetch validation
|
|
2
|
-
|
|
3
|
-
## Status
|
|
4
|
-
|
|
5
|
-
Pending.
|
|
6
|
-
|
|
7
|
-
## Goal
|
|
8
|
-
|
|
9
|
-
Raise a typed `Ucode::CodeChartNotFoundError` when a Unicode Code
|
|
10
|
-
Charts PDF cannot be downloaded or fails content validation. The REQ
|
|
11
|
-
(R1) requires:
|
|
12
|
-
|
|
13
|
-
- HTTP 4xx / 5xx → `CodeChartNotFoundError`
|
|
14
|
-
- `Content-Type: application/pdf`
|
|
15
|
-
- First 4 bytes are `%PDF`
|
|
16
|
-
|
|
17
|
-
## Files
|
|
18
|
-
|
|
19
|
-
- `lib/ucode/error.rb` — add `Ucode::CodeChartNotFoundError` under the
|
|
20
|
-
`GlyphError` subtree.
|
|
21
|
-
- `lib/ucode.rb` — add an `autoload` for the new class so any rescue
|
|
22
|
-
clause triggers one load of `error.rb`.
|
|
23
|
-
- `lib/ucode/fetch/http.rb` — extend `Http.get` with an optional
|
|
24
|
-
`validate:` keyword. When `validate: :pdf`, after a successful
|
|
25
|
-
download, verify the `Content-Type` response header starts with
|
|
26
|
-
`application/pdf` and the first 4 bytes of the body are `%PDF`.
|
|
27
|
-
- `lib/ucode/fetch/code_charts.rb` — pass `validate: :pdf` to `Http.get`
|
|
28
|
-
for every chart PDF download.
|
|
29
|
-
- `spec/ucode/fetch/code_charts_spec.rb` (new) — cover the happy path
|
|
30
|
-
and the validation failure modes.
|
|
31
|
-
|
|
32
|
-
## Design
|
|
33
|
-
|
|
34
|
-
### Why a new error class
|
|
35
|
-
|
|
36
|
-
`FetchError` already covers transport failures, but it doesn't carry
|
|
37
|
-
"this URL produced an HTML error page / 404 / non-PDF body" semantics.
|
|
38
|
-
Splitting the type keeps existing `rescue Ucode::FetchError` callers
|
|
39
|
-
from accidentally swallowing the typed signal that "we expected a PDF
|
|
40
|
-
and didn't get one" — which is a different problem class from "the
|
|
41
|
-
network was down."
|
|
42
|
-
|
|
43
|
-
`CodeChartNotFoundError < Ucode::Error` (under `GlyphError`) reflects
|
|
44
|
-
the REQ's framing: the chart for the requested block is not
|
|
45
|
-
obtainable.
|
|
46
|
-
|
|
47
|
-
### Why `validate:` is optional on `Http.get`
|
|
48
|
-
|
|
49
|
-
`Http` is the single network boundary (per the comment at the top of
|
|
50
|
-
`http.rb`). All callers funnel through it. Adding an optional
|
|
51
|
-
keyword keeps the MECE pattern intact: non-PDF callers (UCD zip,
|
|
52
|
-
Unihan zip, font zip) pass nothing; the single PDF caller passes
|
|
53
|
-
`validate: :pdf`. No second network boundary is needed.
|
|
54
|
-
|
|
55
|
-
### Why no separate "magic bytes" check class
|
|
56
|
-
|
|
57
|
-
Magic-byte verification is 4 lines of code; extracting it into a
|
|
58
|
-
class would be ceremony. Inline check after `write_body`, raising
|
|
59
|
-
the typed error with the offending content-type or magic bytes in
|
|
60
|
-
the context payload.
|
|
61
|
-
|
|
62
|
-
## Acceptance
|
|
63
|
-
|
|
64
|
-
- `Http.get(url, dest:, validate: :pdf)` raises
|
|
65
|
-
`CodeChartNotFoundError` (a) when the response Content-Type is not
|
|
66
|
-
`application/pdf`, (b) when the first 4 bytes are not `%PDF`.
|
|
67
|
-
- `Fetch::CodeCharts.call(version, block_first_cps: [...])` raises
|
|
68
|
-
`CodeChartNotFoundError` when the unicode.org endpoint returns
|
|
69
|
-
4xx/5xx or non-PDF content.
|
|
70
|
-
- Existing callers of `Http.get` that don't pass `validate:` are
|
|
71
|
-
unchanged.
|
|
72
|
-
- Spec coverage: happy path, HTTP 404, wrong content-type, truncated
|
|
73
|
-
body missing the `%PDF` magic.
|
|
74
|
-
|
|
75
|
-
## Out of scope
|
|
76
|
-
|
|
77
|
-
- SHA-256 verification of the PDF — that's a downstream concern (the
|
|
78
|
-
Code Charts are not versioned by hash on unicode.org).
|
|
79
|
-
- Resumable / partial downloads — the existing `Http` writes a
|
|
80
|
-
`.part` then renames; that's sufficient.
|