ucode 0.3.3 → 0.5.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (122) hide show
  1. checksums.yaml +4 -4
  2. data/README.md +45 -6
  3. data/config/unicode17_universal_glyph_set.yml +1 -1
  4. data/lib/ucode/cli.rb +201 -32
  5. data/lib/ucode/code_chart/batch_runner.rb +184 -0
  6. data/lib/ucode/code_chart/block_index.rb +89 -0
  7. data/lib/ucode/code_chart/coverage_gap_index.rb +115 -0
  8. data/lib/ucode/code_chart/extractor.rb +65 -13
  9. data/lib/ucode/code_chart/fetcher.rb +112 -0
  10. data/lib/ucode/code_chart/gap_analyzer/block_gap.rb +35 -0
  11. data/lib/ucode/code_chart/gap_analyzer/essenfont_manifest.rb +60 -0
  12. data/lib/ucode/code_chart/gap_analyzer/manifest.rb +39 -0
  13. data/lib/ucode/code_chart/gap_analyzer.rb +83 -0
  14. data/lib/ucode/code_chart/provenance.rb +99 -58
  15. data/lib/ucode/code_chart/sidecar.rb +3 -4
  16. data/lib/ucode/code_chart/verifier/builder.rb +49 -0
  17. data/lib/ucode/code_chart/verifier/mutool_strategy.rb +80 -0
  18. data/lib/ucode/code_chart/verifier/page_render_cache.rb +63 -0
  19. data/lib/ucode/code_chart/verifier/result.rb +27 -0
  20. data/lib/ucode/code_chart/verifier/resvg_strategy.rb +71 -0
  21. data/lib/ucode/code_chart/verifier/strategy.rb +73 -0
  22. data/lib/ucode/code_chart/verifier.rb +152 -0
  23. data/lib/ucode/code_chart/writer.rb +21 -14
  24. data/lib/ucode/code_chart.rb +6 -0
  25. data/lib/ucode/error.rb +16 -1
  26. data/lib/ucode/fetch/http.rb +23 -3
  27. data/lib/ucode/glyphs/embedded_fonts/catalog.rb +66 -3
  28. data/lib/ucode/glyphs/embedded_fonts/codepoint_mapper/correlator_strategy.rb +46 -0
  29. data/lib/ucode/glyphs/embedded_fonts/codepoint_mapper/strategy.rb +54 -0
  30. data/lib/ucode/glyphs/embedded_fonts/codepoint_mapper/tounicode_strategy.rb +41 -0
  31. data/lib/ucode/glyphs/embedded_fonts/codepoint_mapper/trace_strategy.rb +64 -0
  32. data/lib/ucode/glyphs/embedded_fonts/codepoint_mapper.rb +150 -90
  33. data/lib/ucode/glyphs/embedded_fonts/mutool/draw.rb +26 -0
  34. data/lib/ucode/glyphs/embedded_fonts/mutool/info.rb +23 -0
  35. data/lib/ucode/glyphs/embedded_fonts/mutool/show.rb +36 -0
  36. data/lib/ucode/glyphs/embedded_fonts/mutool/trace.rb +26 -0
  37. data/lib/ucode/glyphs/embedded_fonts/mutool.rb +30 -0
  38. data/lib/ucode/glyphs/embedded_fonts/page_trace_cache.rb +112 -0
  39. data/lib/ucode/glyphs/embedded_fonts/pdf_indexer.rb +50 -56
  40. data/lib/ucode/glyphs/embedded_fonts/{pdf_location.rb → pdf_source.rb} +1 -1
  41. data/lib/ucode/glyphs/embedded_fonts/renderer.rb +19 -2
  42. data/lib/ucode/glyphs/embedded_fonts/trace_runner.rb +12 -28
  43. data/lib/ucode/glyphs/embedded_fonts.rb +3 -1
  44. data/lib/ucode/glyphs/real_fonts/coverage_auditor.rb +0 -4
  45. data/lib/ucode/glyphs/real_fonts/font_coverage_report.rb +0 -2
  46. data/lib/ucode/glyphs/real_fonts/writer.rb +0 -2
  47. data/lib/ucode/glyphs/source.rb +11 -1
  48. data/lib/ucode/glyphs/sources/pillar1_embedded_tounicode.rb +11 -6
  49. data/lib/ucode/version.rb +1 -1
  50. data/lib/ucode.rb +2 -0
  51. metadata +28 -74
  52. data/TODO.extract-code-chart/01-pdf-fetch-validation.md +0 -80
  53. data/TODO.extract-code-chart/02-block-name-resolver.md +0 -68
  54. data/TODO.extract-code-chart/03-codechart-namespace.md +0 -82
  55. data/TODO.extract-code-chart/04-codechart-extractor.md +0 -154
  56. data/TODO.extract-code-chart/05-provenance-and-sidecar.md +0 -147
  57. data/TODO.extract-code-chart/06-codechart-writer.md +0 -134
  58. data/TODO.extract-code-chart/07-codechart-cli.md +0 -135
  59. data/TODO.extract-code-chart/08-specs.md +0 -87
  60. data/TODO.full/00-README.md +0 -116
  61. data/TODO.full/01-panglyph-vision.md +0 -112
  62. data/TODO.full/02-panglyph-repo-bootstrap.md +0 -184
  63. data/TODO.full/03-panglyph-font-builder.md +0 -201
  64. data/TODO.full/04-panglyph-publish-pipeline.md +0 -126
  65. data/TODO.full/05-ucode-0-1-1-release.md +0 -139
  66. data/TODO.full/06-fontisan-remove-audit.md +0 -142
  67. data/TODO.full/07-fontisan-remove-ucd.md +0 -125
  68. data/TODO.full/08-archive-private-bin-build.md +0 -143
  69. data/TODO.full/09-archive-public-structure.md +0 -164
  70. data/TODO.full/10-fontist-org-woff-glyphs.md +0 -131
  71. data/TODO.full/11-fontist-org-audit-coverage.md +0 -140
  72. data/TODO.full/12-implementation-order.md +0 -216
  73. data/TODO.full/13-fontisan-font-writer-api.md +0 -189
  74. data/TODO.full/14-fontisan-table-writers.md +0 -66
  75. data/TODO.full/15-panglyph-builder-real.md +0 -82
  76. data/TODO.full/16-archive-public-sync-workflows.md +0 -167
  77. data/TODO.full/17-fontist-org-font-picker.md +0 -73
  78. data/TODO.full/18-comprehensive-spec-coverage.md +0 -64
  79. data/TODO.full/19-ucode-0-1-2-patch.md +0 -32
  80. data/TODO.full/20-fontisan-0-2-23-release.md +0 -52
  81. data/TODO.new/00-README.md +0 -96
  82. data/TODO.new/01-pillar-terminology-alignment.md +0 -69
  83. data/TODO.new/02-audit-schema-design.md +0 -255
  84. data/TODO.new/03-directory-output-spec.md +0 -203
  85. data/TODO.new/04-fontist-org-contract.md +0 -173
  86. data/TODO.new/05-baseline-unicode17-coverage-audit.md +0 -144
  87. data/TODO.new/06-audit-namespace-skeleton.md +0 -105
  88. data/TODO.new/07-audit-models-port.md +0 -132
  89. data/TODO.new/08-extractors-cheap-port.md +0 -113
  90. data/TODO.new/09-extractors-expensive-port.md +0 -99
  91. data/TODO.new/10-aggregations-ucd-rewrite.md +0 -168
  92. data/TODO.new/11-differ-and-library-auditor-port.md +0 -102
  93. data/TODO.new/12-formatters-port.md +0 -115
  94. data/TODO.new/13-directory-emitter.md +0 -147
  95. data/TODO.new/14-html-face-browser.md +0 -144
  96. data/TODO.new/15-html-library-browser.md +0 -102
  97. data/TODO.new/16-cli-audit-subcommands.md +0 -142
  98. data/TODO.new/17-fontisan-cleanup-audit.md +0 -147
  99. data/TODO.new/18-fontisan-cleanup-ucd.md +0 -156
  100. data/TODO.new/19-fontisan-docs-update.md +0 -155
  101. data/TODO.new/20-canonical-resolver-4-tier.md +0 -182
  102. data/TODO.new/21-canonical-unicode17-build.md +0 -148
  103. data/TODO.new/22-implementation-order.md +0 -176
  104. data/TODO.new/23-universal-glyph-set-source-map.md +0 -312
  105. data/TODO.new/24-universal-glyph-set-build.md +0 -189
  106. data/TODO.new/25-font-audit-against-universal-set.md +0 -195
  107. data/TODO.new/26-missing-glyph-reporter.md +0 -189
  108. data/TODO.new/27-fontist-org-consumer-integration.md +0 -200
  109. data/TODO.new/28-implementation-order-update.md +0 -187
  110. data/TODO.new/29-universal-set-curation-uc17.md +0 -312
  111. data/TODO.new/30-tier1-font-acquisition.md +0 -241
  112. data/TODO.new/31-universal-set-production-build.md +0 -205
  113. data/TODO.new/32-uc17-coverage-matrix.md +0 -165
  114. data/TODO.new/33-specialist-font-acquisition-refresh.md +0 -138
  115. data/TODO.new/34-pillar2-content-stream-correlator.md +0 -147
  116. data/TODO.new/35-universal-set-production-run.md +0 -160
  117. data/TODO.new/36-per-font-coverage-audit.md +0 -145
  118. data/TODO.new/37-coverage-highlight-reporter.md +0 -125
  119. data/TODO.new/38-fontist-org-glyph-consumer.md +0 -141
  120. data/TODO.new/39-implementation-order-update-32-38.md +0 -258
  121. data/TODO.new/40-archive-private-uses-ucode-audit.md +0 -124
  122. data/TODO.new/41-ucode-unicode-archive-bridge.md +0 -160
@@ -1,23 +1,23 @@
1
1
  # frozen_string_literal: true
2
2
 
3
- require "open3"
4
3
  require "pathname"
5
4
 
5
+ require "ucode/glyphs/embedded_fonts/mutool"
6
+
6
7
  module Ucode
7
8
  module Glyphs
8
9
  module EmbeddedFonts
9
- # Walks the Code Charts PDF once via `mutool info` + `mutool show`
10
- # and builds an Array of {RawFontDescriptor} — one per Type0 font
11
- # that has the required descendant CIDFont, FontDescriptor, and
12
- # FontFile2/3 + Identity CIDToGIDMap.
13
- #
14
- # Pure subprocess + parsing concern. Does NOT resolve codepoint →
15
- # GID (that's {CodepointMapper}'s job). The descriptor carries
16
- # every ref the mapper needs to do its work.
17
10
  class PdfIndexer
18
- # @param source [PdfLocation]
19
- def initialize(source:)
11
+ Type0Entry = Struct.new(:font_obj_id, :base_font, keyword_init: true)
12
+
13
+ # @param source [PdfSource]
14
+ # @param mutool_info [Mutool::Info] injectable for tests
15
+ # @param mutool_show [Mutool::Show] injectable for tests
16
+ def initialize(source:, mutool_info: Mutool::Info.new,
17
+ mutool_show: Mutool::Show.new)
20
18
  @source = source
19
+ @mutool_info = mutool_info
20
+ @mutool_show = mutool_show
21
21
  end
22
22
 
23
23
  # @return [Array<RawFontDescriptor>]
@@ -68,11 +68,45 @@ module Ucode
68
68
  # @param base_font [String] e.g. "GPJAHB+WolofGaraySansSerif"
69
69
  # @return [Boolean] true if this font appears on any page
70
70
  def font_appears?(base_font)
71
- font_entries_cache.key?(base_font)
71
+ type0_base_fonts.include?(base_font)
72
72
  end
73
73
 
74
74
  private
75
75
 
76
+ # Single source of truth for the Type0 font list. Both
77
+ # `discover_type0_fonts` (keyed by obj_id) and
78
+ # `font_appears?` (membership check by name) derive their
79
+ # answer from this parse, so changes to `mutool info`'s output
80
+ # format break in exactly one place.
81
+ #
82
+ # @return [Array<Type0Entry>]
83
+ def type0_entries
84
+ @type0_entries ||= parse_type0_entries
85
+ end
86
+
87
+ # @return [Set<String>] every Type0 BaseFont name seen
88
+ def type0_base_fonts
89
+ @type0_base_fonts ||= type0_entries.each_with_object(Set.new) do |e, s|
90
+ s << e.base_font
91
+ end
92
+ end
93
+
94
+ def parse_type0_entries
95
+ seen = Set.new
96
+ mutool_info_text.each_line.filter_map do |line|
97
+ next unless line.include?("Type0")
98
+
99
+ m = line.match(/Type0\s+'([^']+)'\s+\S+\s+\((\d+)\s+0\s+R\)/)
100
+ next unless m
101
+
102
+ font_obj_id = m[2].to_i
103
+ next if seen.include?(font_obj_id)
104
+
105
+ seen << font_obj_id
106
+ Type0Entry.new(font_obj_id: font_obj_id, base_font: m[1])
107
+ end
108
+ end
109
+
76
110
  def build_descriptors(type0_refs, type0_dicts, descendant_dicts, fontdesc_dicts)
77
111
  type0_refs.filter_map do |font_obj_id, base_font|
78
112
  build_descriptor(
@@ -133,35 +167,15 @@ module Ucode
133
167
  # ---- mutool subprocess + dict parsing ----------------------------
134
168
 
135
169
  def discover_type0_fonts
136
- text = mutool_info_text
137
- result = {}
138
- seen = Set.new
139
- text.each_line do |line|
140
- next unless line.include?("Type0")
141
-
142
- m = line.match(/Type0\s+'([^']+)'\s+\S+\s+\((\d+)\s+0\s+R\)/)
143
- next unless m
144
-
145
- font_obj_id = m[2].to_i
146
- next if seen.include?(font_obj_id)
147
-
148
- seen << font_obj_id
149
- result[font_obj_id] = m[1]
170
+ type0_entries.to_h do |e|
171
+ [e.font_obj_id, e.base_font]
150
172
  end
151
- result
152
173
  end
153
174
 
154
175
  def fetch_objects(obj_ids)
155
176
  return {} if obj_ids.empty?
156
177
 
157
- args = ["mutool", "show", "-g",
158
- @source.pdf_to_s].concat(obj_ids.map(&:to_s))
159
- out, err, status = Open3.capture3(*args)
160
- unless status.success?
161
- raise Ucode::EmbeddedFontsMissingError,
162
- "mutool show failed: #{err.strip}"
163
- end
164
-
178
+ out = @mutool_show.grep(@source.pdf_to_s, *obj_ids)
165
179
  parse_grep_output(out)
166
180
  end
167
181
 
@@ -208,27 +222,7 @@ module Ucode
208
222
  end
209
223
 
210
224
  def mutool_info_text
211
- @mutool_info_text ||= run_mutool_info
212
- end
213
-
214
- def run_mutool_info
215
- out, err, status = Open3.capture3("mutool", "info", @source.pdf_to_s)
216
- status.success? ? out + err : ""
217
- end
218
-
219
- def font_entries_cache
220
- @font_entries_cache ||= begin
221
- result = {}
222
- mutool_info_text.each_line do |line|
223
- next unless line.include?("Type0")
224
-
225
- font_match = line.match(/Type0\s+'([^']+)'/)
226
- next unless font_match
227
-
228
- result[font_match[1]] = true
229
- end
230
- result
231
- end
225
+ @mutool_info_text ||= @mutool_info.call(@source.pdf_to_s)
232
226
  end
233
227
  end
234
228
  end
@@ -27,7 +27,7 @@ module Ucode
27
27
  # the BaseFont (e.g. `CIAIIP+Uni2000Generalpunctuation.ttf`).
28
28
  # Re-runs skip extraction when the cached file is newer than the
29
29
  # PDF.
30
- class PdfLocation
30
+ class PdfSource
31
31
  attr_reader :pdf_path, :cache_dir
32
32
 
33
33
  # @param pdf [String, Pathname, nil] path to a Code Charts PDF
@@ -12,7 +12,16 @@ module Ucode
12
12
  # renderer.
13
13
  class Renderer
14
14
  # Result of rendering one codepoint.
15
- Result = Struct.new(:codepoint, :base_font, :gid, :svg, keyword_init: true) do
15
+ #
16
+ # `source_page` + `source_cell` come from the Catalog's
17
+ # location index when available. Nil when the catalog hasn't
18
+ # traced the PDF (ToUnicode-only path) — downstream concerns
19
+ # treat that as "no positional data; skip cell-based diff".
20
+ Result = Struct.new(
21
+ :codepoint, :base_font, :gid, :svg,
22
+ :source_page, :source_cell,
23
+ keyword_init: true,
24
+ ) do
16
25
  def ok?
17
26
  !svg.nil?
18
27
  end
@@ -37,7 +46,15 @@ module Ucode
37
46
  return nil if outline.nil? || outline.empty?
38
47
 
39
48
  svg = Svg.new(outline, codepoint: codepoint, base_font: entry.base_font).to_s
40
- Result.new(codepoint: codepoint, base_font: entry.base_font, gid: gid, svg: svg)
49
+ location = @catalog.location_for(codepoint)
50
+ Result.new(
51
+ codepoint: codepoint,
52
+ base_font: entry.base_font,
53
+ gid: gid,
54
+ svg: svg,
55
+ source_page: location&.fetch(:page),
56
+ source_cell: location && { x: location[:x], y: location[:y] },
57
+ )
41
58
  end
42
59
  end
43
60
  end
@@ -1,51 +1,35 @@
1
1
  # frozen_string_literal: true
2
2
 
3
- require "open3"
4
3
  require "pathname"
5
4
 
6
- require "ucode/error"
5
+ require "ucode/glyphs/embedded_fonts/mutool"
6
+ require "ucode/glyphs/embedded_fonts/trace_parser"
7
7
 
8
8
  module Ucode
9
9
  module Glyphs
10
10
  module EmbeddedFonts
11
11
  # Thin I/O wrapper around `mutool trace <pdf> <page>`.
12
12
  #
13
- # Runs mutool on the given pages, captures the XML output,
14
- # delegates parsing to {TraceParser}, and returns a flat
15
- # `Array<TraceGlyph>` across all pages.
16
- #
17
- # The only class in the trace pipeline that touches the
18
- # filesystem / spawns subprocesses. Everything upstream
19
- # (parser, correlator) is pure.
13
+ # Delegates the actual subprocess to {Mutool::Trace} and the
14
+ # XML parsing to {TraceParser}. Kept for backward compatibility
15
+ # with callers that pre-date the Mutool seam.
20
16
  class TraceRunner
21
17
  # @param pdf_path [Pathname, String]
22
- def initialize(pdf_path)
18
+ # @param mutool [Mutool::Trace] injectable for tests
19
+ def initialize(pdf_path, mutool: Mutool::Trace.new)
23
20
  @pdf_path = Pathname.new(pdf_path)
21
+ @mutool = mutool
24
22
  end
25
23
 
26
24
  # @param page_numbers [Array<Integer>] 1-based PDF page numbers
27
25
  # @return [Array<TraceGlyph>]
28
26
  def trace(page_numbers)
29
- page_numbers.flat_map { |page| trace_page(page) }
30
- end
31
-
32
- private
27
+ return [] if page_numbers.empty?
33
28
 
34
- def trace_page(page)
35
- xml = run_mutool(page)
36
- TraceParser.parse(xml)
37
- end
38
-
39
- def run_mutool(page)
40
- out, err, status = Open3.capture3(
41
- "mutool", "trace", @pdf_path.to_s, page.to_s,
42
- )
43
- unless status.success?
44
- raise Ucode::EmbeddedFontsMissingError,
45
- "mutool trace failed: #{(out + err).strip}"
29
+ page_numbers.flat_map do |page|
30
+ xml = @mutool.call(@pdf_path, page)
31
+ TraceParser.parse(xml)
46
32
  end
47
-
48
- out + err
49
33
  end
50
34
  end
51
35
  end
@@ -36,7 +36,7 @@ module Ucode
36
36
  # `mutool info` (font enumeration) and `mutool show -b -o` (raw
37
37
  # stream extraction).
38
38
  module EmbeddedFonts
39
- autoload :PdfLocation, "ucode/glyphs/embedded_fonts/pdf_location"
39
+ autoload :PdfSource, "ucode/glyphs/embedded_fonts/pdf_source"
40
40
  autoload :ToUnicode, "ucode/glyphs/embedded_fonts/tounicode"
41
41
  autoload :FontEntry, "ucode/glyphs/embedded_fonts/font_entry"
42
42
  autoload :RawFontDescriptor, "ucode/glyphs/embedded_fonts/raw_font_descriptor"
@@ -50,6 +50,8 @@ module Ucode
50
50
  autoload :TraceParser, "ucode/glyphs/embedded_fonts/trace_parser"
51
51
  autoload :TraceCorrelator, "ucode/glyphs/embedded_fonts/trace_correlator"
52
52
  autoload :TraceRunner, "ucode/glyphs/embedded_fonts/trace_runner"
53
+ autoload :Mutool, "ucode/glyphs/embedded_fonts/mutool"
54
+ autoload :PageTraceCache, "ucode/glyphs/embedded_fonts/page_trace_cache"
53
55
  autoload :Svg, "ucode/glyphs/embedded_fonts/svg"
54
56
  autoload :Renderer, "ucode/glyphs/embedded_fonts/renderer"
55
57
  autoload :Writer, "ucode/glyphs/embedded_fonts/writer"
@@ -5,10 +5,6 @@ require "time"
5
5
 
6
6
  require "fontisan"
7
7
 
8
- require_relative "block_coverage"
9
- require_relative "font_coverage_report"
10
- require_relative "unicode_17_blocks"
11
-
12
8
  module Ucode
13
9
  module Glyphs
14
10
  module RealFonts
@@ -2,8 +2,6 @@
2
2
 
3
3
  require "lutaml/model"
4
4
 
5
- require_relative "block_coverage"
6
-
7
5
  module Ucode
8
6
  module Glyphs
9
7
  module RealFonts
@@ -3,8 +3,6 @@
3
3
  require "fileutils"
4
4
  require "pathname"
5
5
 
6
- require_relative "font_coverage_report"
7
-
8
6
  module Ucode
9
7
  module Glyphs
10
8
  module RealFonts
@@ -25,7 +25,17 @@ module Ucode
25
25
  # One resolved glyph. Carries the SVG payload and enough
26
26
  # provenance to debug "where did this glyph come from?" without
27
27
  # holding a reference back to the source.
28
- Result = Struct.new(:tier, :codepoint, :svg, :provenance, keyword_init: true)
28
+ #
29
+ # The optional `base_font`, `gid`, `source_page`, `source_cell`
30
+ # fields carry the renderer's localization data so downstream
31
+ # concerns (Verifier pixel-diff against the source PDF cell,
32
+ # Provenance audit trail) don't need to re-derive it. Sources
33
+ # that don't know this data (e.g. LastResort) leave them nil.
34
+ Result = Struct.new(
35
+ :tier, :codepoint, :svg, :provenance,
36
+ :base_font, :gid, :source_page, :source_cell,
37
+ keyword_init: true,
38
+ )
29
39
 
30
40
  # @return [Symbol] one of :tier1, :pillar1, :pillar2, :pillar3
31
41
  def tier
@@ -1,8 +1,5 @@
1
1
  # frozen_string_literal: true
2
2
 
3
- require "ucode/glyphs/source"
4
- require "ucode/glyphs/embedded_fonts/renderer"
5
-
6
3
  module Ucode
7
4
  module Glyphs
8
5
  module Sources
@@ -31,7 +28,7 @@ module Ucode
31
28
  # @param renderer [EmbeddedFonts::Renderer] the renderer to
32
29
  # delegate to. Callers typically construct it with the
33
30
  # {EmbeddedFonts::Catalog} built from the resolved Code
34
- # Charts {EmbeddedFonts::PdfLocation}. To enable pillar-2
31
+ # Charts {EmbeddedFonts::PdfSource}. To enable pillar-2
35
32
  # fallback, that Catalog must be constructed with
36
33
  # +correlator_configs:+.
37
34
  def initialize(renderer:)
@@ -54,8 +51,16 @@ module Ucode
54
51
  result = @renderer.render(codepoint)
55
52
  return nil unless result
56
53
 
57
- Result.new(tier: tier, codepoint: codepoint,
58
- svg: result.svg, provenance: provenance)
54
+ Result.new(
55
+ tier: tier,
56
+ codepoint: codepoint,
57
+ svg: result.svg,
58
+ provenance: provenance,
59
+ base_font: result.base_font,
60
+ gid: result.gid,
61
+ source_page: result.source_page,
62
+ source_cell: result.source_cell,
63
+ )
59
64
  end
60
65
  end
61
66
  end
data/lib/ucode/version.rb CHANGED
@@ -1,5 +1,5 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  module Ucode
4
- VERSION = "0.3.3"
4
+ VERSION = "0.5.0"
5
5
  end
data/lib/ucode.rb CHANGED
@@ -34,7 +34,9 @@ module Ucode
34
34
  autoload :GlyphError, "ucode/error"
35
35
  autoload :LastResortMissingError, "ucode/error"
36
36
  autoload :EmbeddedFontsMissingError, "ucode/error"
37
+ autoload :CodeChartError, "ucode/error"
37
38
  autoload :CodeChartNotFoundError, "ucode/error"
39
+ autoload :CodeChartChecksumError, "ucode/error"
38
40
  autoload :UnknownUnicodeVersionError, "ucode/error"
39
41
 
40
42
  # Infrastructure
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: ucode
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.3.3
4
+ version: 0.5.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - Ribose Inc.
8
8
  autorequire:
9
9
  bindir: exe
10
10
  cert_chain: []
11
- date: 2026-07-03 00:00:00.000000000 Z
11
+ date: 2026-07-21 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: base64
@@ -162,77 +162,6 @@ files:
162
162
  - Gemfile
163
163
  - README.md
164
164
  - Rakefile
165
- - TODO.extract-code-chart/01-pdf-fetch-validation.md
166
- - TODO.extract-code-chart/02-block-name-resolver.md
167
- - TODO.extract-code-chart/03-codechart-namespace.md
168
- - TODO.extract-code-chart/04-codechart-extractor.md
169
- - TODO.extract-code-chart/05-provenance-and-sidecar.md
170
- - TODO.extract-code-chart/06-codechart-writer.md
171
- - TODO.extract-code-chart/07-codechart-cli.md
172
- - TODO.extract-code-chart/08-specs.md
173
- - TODO.full/00-README.md
174
- - TODO.full/01-panglyph-vision.md
175
- - TODO.full/02-panglyph-repo-bootstrap.md
176
- - TODO.full/03-panglyph-font-builder.md
177
- - TODO.full/04-panglyph-publish-pipeline.md
178
- - TODO.full/05-ucode-0-1-1-release.md
179
- - TODO.full/06-fontisan-remove-audit.md
180
- - TODO.full/07-fontisan-remove-ucd.md
181
- - TODO.full/08-archive-private-bin-build.md
182
- - TODO.full/09-archive-public-structure.md
183
- - TODO.full/10-fontist-org-woff-glyphs.md
184
- - TODO.full/11-fontist-org-audit-coverage.md
185
- - TODO.full/12-implementation-order.md
186
- - TODO.full/13-fontisan-font-writer-api.md
187
- - TODO.full/14-fontisan-table-writers.md
188
- - TODO.full/15-panglyph-builder-real.md
189
- - TODO.full/16-archive-public-sync-workflows.md
190
- - TODO.full/17-fontist-org-font-picker.md
191
- - TODO.full/18-comprehensive-spec-coverage.md
192
- - TODO.full/19-ucode-0-1-2-patch.md
193
- - TODO.full/20-fontisan-0-2-23-release.md
194
- - TODO.new/00-README.md
195
- - TODO.new/01-pillar-terminology-alignment.md
196
- - TODO.new/02-audit-schema-design.md
197
- - TODO.new/03-directory-output-spec.md
198
- - TODO.new/04-fontist-org-contract.md
199
- - TODO.new/05-baseline-unicode17-coverage-audit.md
200
- - TODO.new/06-audit-namespace-skeleton.md
201
- - TODO.new/07-audit-models-port.md
202
- - TODO.new/08-extractors-cheap-port.md
203
- - TODO.new/09-extractors-expensive-port.md
204
- - TODO.new/10-aggregations-ucd-rewrite.md
205
- - TODO.new/11-differ-and-library-auditor-port.md
206
- - TODO.new/12-formatters-port.md
207
- - TODO.new/13-directory-emitter.md
208
- - TODO.new/14-html-face-browser.md
209
- - TODO.new/15-html-library-browser.md
210
- - TODO.new/16-cli-audit-subcommands.md
211
- - TODO.new/17-fontisan-cleanup-audit.md
212
- - TODO.new/18-fontisan-cleanup-ucd.md
213
- - TODO.new/19-fontisan-docs-update.md
214
- - TODO.new/20-canonical-resolver-4-tier.md
215
- - TODO.new/21-canonical-unicode17-build.md
216
- - TODO.new/22-implementation-order.md
217
- - TODO.new/23-universal-glyph-set-source-map.md
218
- - TODO.new/24-universal-glyph-set-build.md
219
- - TODO.new/25-font-audit-against-universal-set.md
220
- - TODO.new/26-missing-glyph-reporter.md
221
- - TODO.new/27-fontist-org-consumer-integration.md
222
- - TODO.new/28-implementation-order-update.md
223
- - TODO.new/29-universal-set-curation-uc17.md
224
- - TODO.new/30-tier1-font-acquisition.md
225
- - TODO.new/31-universal-set-production-build.md
226
- - TODO.new/32-uc17-coverage-matrix.md
227
- - TODO.new/33-specialist-font-acquisition-refresh.md
228
- - TODO.new/34-pillar2-content-stream-correlator.md
229
- - TODO.new/35-universal-set-production-run.md
230
- - TODO.new/36-per-font-coverage-audit.md
231
- - TODO.new/37-coverage-highlight-reporter.md
232
- - TODO.new/38-fontist-org-glyph-consumer.md
233
- - TODO.new/39-implementation-order-update-32-38.md
234
- - TODO.new/40-archive-private-uses-ucode-audit.md
235
- - TODO.new/41-ucode-unicode-archive-bridge.md
236
165
  - UCODE_CHANGELOG.md
237
166
  - config/specialist_fonts.yml
238
167
  - config/unicode17_tier1_fonts.yml
@@ -310,9 +239,24 @@ files:
310
239
  - lib/ucode/cache.rb
311
240
  - lib/ucode/cli.rb
312
241
  - lib/ucode/code_chart.rb
242
+ - lib/ucode/code_chart/batch_runner.rb
243
+ - lib/ucode/code_chart/block_index.rb
244
+ - lib/ucode/code_chart/coverage_gap_index.rb
313
245
  - lib/ucode/code_chart/extractor.rb
246
+ - lib/ucode/code_chart/fetcher.rb
247
+ - lib/ucode/code_chart/gap_analyzer.rb
248
+ - lib/ucode/code_chart/gap_analyzer/block_gap.rb
249
+ - lib/ucode/code_chart/gap_analyzer/essenfont_manifest.rb
250
+ - lib/ucode/code_chart/gap_analyzer/manifest.rb
314
251
  - lib/ucode/code_chart/provenance.rb
315
252
  - lib/ucode/code_chart/sidecar.rb
253
+ - lib/ucode/code_chart/verifier.rb
254
+ - lib/ucode/code_chart/verifier/builder.rb
255
+ - lib/ucode/code_chart/verifier/mutool_strategy.rb
256
+ - lib/ucode/code_chart/verifier/page_render_cache.rb
257
+ - lib/ucode/code_chart/verifier/result.rb
258
+ - lib/ucode/code_chart/verifier/resvg_strategy.rb
259
+ - lib/ucode/code_chart/verifier/strategy.rb
316
260
  - lib/ucode/code_chart/writer.rb
317
261
  - lib/ucode/commands.rb
318
262
  - lib/ucode/commands/audit.rb
@@ -363,10 +307,20 @@ files:
363
307
  - lib/ucode/glyphs/embedded_fonts.rb
364
308
  - lib/ucode/glyphs/embedded_fonts/catalog.rb
365
309
  - lib/ucode/glyphs/embedded_fonts/codepoint_mapper.rb
310
+ - lib/ucode/glyphs/embedded_fonts/codepoint_mapper/correlator_strategy.rb
311
+ - lib/ucode/glyphs/embedded_fonts/codepoint_mapper/strategy.rb
312
+ - lib/ucode/glyphs/embedded_fonts/codepoint_mapper/tounicode_strategy.rb
313
+ - lib/ucode/glyphs/embedded_fonts/codepoint_mapper/trace_strategy.rb
366
314
  - lib/ucode/glyphs/embedded_fonts/content_stream_correlator.rb
367
315
  - lib/ucode/glyphs/embedded_fonts/font_entry.rb
316
+ - lib/ucode/glyphs/embedded_fonts/mutool.rb
317
+ - lib/ucode/glyphs/embedded_fonts/mutool/draw.rb
318
+ - lib/ucode/glyphs/embedded_fonts/mutool/info.rb
319
+ - lib/ucode/glyphs/embedded_fonts/mutool/show.rb
320
+ - lib/ucode/glyphs/embedded_fonts/mutool/trace.rb
321
+ - lib/ucode/glyphs/embedded_fonts/page_trace_cache.rb
368
322
  - lib/ucode/glyphs/embedded_fonts/pdf_indexer.rb
369
- - lib/ucode/glyphs/embedded_fonts/pdf_location.rb
323
+ - lib/ucode/glyphs/embedded_fonts/pdf_source.rb
370
324
  - lib/ucode/glyphs/embedded_fonts/positional_matcher.rb
371
325
  - lib/ucode/glyphs/embedded_fonts/raw_font_descriptor.rb
372
326
  - lib/ucode/glyphs/embedded_fonts/renderer.rb
@@ -1,80 +0,0 @@
1
- # TODO 01 — PDF fetch validation
2
-
3
- ## Status
4
-
5
- Pending.
6
-
7
- ## Goal
8
-
9
- Raise a typed `Ucode::CodeChartNotFoundError` when a Unicode Code
10
- Charts PDF cannot be downloaded or fails content validation. The REQ
11
- (R1) requires:
12
-
13
- - HTTP 4xx / 5xx → `CodeChartNotFoundError`
14
- - `Content-Type: application/pdf`
15
- - First 4 bytes are `%PDF`
16
-
17
- ## Files
18
-
19
- - `lib/ucode/error.rb` — add `Ucode::CodeChartNotFoundError` under the
20
- `GlyphError` subtree.
21
- - `lib/ucode.rb` — add an `autoload` for the new class so any rescue
22
- clause triggers one load of `error.rb`.
23
- - `lib/ucode/fetch/http.rb` — extend `Http.get` with an optional
24
- `validate:` keyword. When `validate: :pdf`, after a successful
25
- download, verify the `Content-Type` response header starts with
26
- `application/pdf` and the first 4 bytes of the body are `%PDF`.
27
- - `lib/ucode/fetch/code_charts.rb` — pass `validate: :pdf` to `Http.get`
28
- for every chart PDF download.
29
- - `spec/ucode/fetch/code_charts_spec.rb` (new) — cover the happy path
30
- and the validation failure modes.
31
-
32
- ## Design
33
-
34
- ### Why a new error class
35
-
36
- `FetchError` already covers transport failures, but it doesn't carry
37
- "this URL produced an HTML error page / 404 / non-PDF body" semantics.
38
- Splitting the type keeps existing `rescue Ucode::FetchError` callers
39
- from accidentally swallowing the typed signal that "we expected a PDF
40
- and didn't get one" — which is a different problem class from "the
41
- network was down."
42
-
43
- `CodeChartNotFoundError < Ucode::Error` (under `GlyphError`) reflects
44
- the REQ's framing: the chart for the requested block is not
45
- obtainable.
46
-
47
- ### Why `validate:` is optional on `Http.get`
48
-
49
- `Http` is the single network boundary (per the comment at the top of
50
- `http.rb`). All callers funnel through it. Adding an optional
51
- keyword keeps the MECE pattern intact: non-PDF callers (UCD zip,
52
- Unihan zip, font zip) pass nothing; the single PDF caller passes
53
- `validate: :pdf`. No second network boundary is needed.
54
-
55
- ### Why no separate "magic bytes" check class
56
-
57
- Magic-byte verification is 4 lines of code; extracting it into a
58
- class would be ceremony. Inline check after `write_body`, raising
59
- the typed error with the offending content-type or magic bytes in
60
- the context payload.
61
-
62
- ## Acceptance
63
-
64
- - `Http.get(url, dest:, validate: :pdf)` raises
65
- `CodeChartNotFoundError` (a) when the response Content-Type is not
66
- `application/pdf`, (b) when the first 4 bytes are not `%PDF`.
67
- - `Fetch::CodeCharts.call(version, block_first_cps: [...])` raises
68
- `CodeChartNotFoundError` when the unicode.org endpoint returns
69
- 4xx/5xx or non-PDF content.
70
- - Existing callers of `Http.get` that don't pass `validate:` are
71
- unchanged.
72
- - Spec coverage: happy path, HTTP 404, wrong content-type, truncated
73
- body missing the `%PDF` magic.
74
-
75
- ## Out of scope
76
-
77
- - SHA-256 verification of the PDF — that's a downstream concern (the
78
- Code Charts are not versioned by hash on unicode.org).
79
- - Resumable / partial downloads — the existing `Http` writes a
80
- `.part` then renames; that's sufficient.
@@ -1,68 +0,0 @@
1
- # TODO 02 — Block name resolver
2
-
3
- ## Status
4
-
5
- Pending. Depends on nothing.
6
-
7
- ## Goal
8
-
9
- Add a class method `Ucode::Parsers::Blocks.find_by_name(name)` that
10
- resolves a Unicode block identifier (e.g. `"Sidetic"`,
11
- `"Egyptian_Hieroglyphs_Extended-B"`) to the `Ucode::Models::Block`
12
- instance in a given version's cached `Blocks.txt`.
13
-
14
- This is the CLI ergonomics glue: the REQ's `ucode code-chart extract
15
- --block Sidetic` flow takes a human-readable name and needs to know
16
- the block's range to know which `U+XXXX` codepoints to iterate.
17
-
18
- ## Files
19
-
20
- - `lib/ucode/parsers/blocks.rb` — add `Blocks.find_by_name(path, name)`.
21
- - `spec/ucode/parsers/blocks_spec.rb` — cover name lookup, missing-name,
22
- case-sensitivity.
23
-
24
- ## Design
25
-
26
- ### Method shape
27
-
28
- ```ruby
29
- # @param path [Pathname, String] path to a Blocks.txt
30
- # @param name [String] block identifier (matches Models::Block#id)
31
- # @return [Models::Block, nil] nil when no block matches
32
- def find_by_name(path, name)
33
- ```
34
-
35
- Returns nil for "not found" — callers (CLI, Extractor) decide whether
36
- to raise. This matches `Models::Block` consumers that already expect
37
- nilable lookups.
38
-
39
- ### Name matching rule
40
-
41
- `Blocks.txt` uses `name` with whitespace collapsed to underscores
42
- into `id`. `find_by_name` matches against `id` (the underscored
43
- form). The REQ's example `--block Sidetic` shows that the caller
44
- provides the underscored form already. This is consistent with the
45
- existing `Parsers::Blocks` build logic (`name.gsub(/\s+/, "_")`).
46
-
47
- ### Why a separate method
48
-
49
- `each_record` streams every block — the caller doesn't want to walk
50
- ~340 blocks for every name lookup. `find_by_name` short-circuits on
51
- first match.
52
-
53
- ## Acceptance
54
-
55
- - `find_by_name(path, "Basic_Latin")` returns the Basic Latin block.
56
- - `find_by_name(path, "Nonexistent")` returns nil.
57
- - Streaming still works for callers that need every block.
58
-
59
- ## Out of scope
60
-
61
- - Fuzzy matching — exact match only. Callers validate the user's
62
- input against `Parsers::Blocks.each_record(path).map(&:id)` to
63
- surface "did you mean …?" suggestions if we ever want that; for
64
- now, a clean `UnknownBlockError` at the call site is enough.
65
- - Database-backed lookup — `Ucode::Database#block_ranges_by_name` is
66
- a different concern (full UCD index). `find_by_name` operates on
67
- the cached `Blocks.txt` directly because the CodeChart extractor
68
- is meant to be runnable without a built database.