idml 0.5.0 → 0.5.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: '09c5f631233897e7f1e18519b1031c1c87092a1e3ddffdc353bf2e121c47b15b'
-  data.tar.gz: 8c0b965da67d83e30f3b7002be6bbabe1dffec7d1a3557979fe4cdfbce99dd54
+  metadata.gz: b61e00a5f26f6e7266f1f512362e22453623b09ea06dd002d5e0a79ba23b52d0
+  data.tar.gz: 2177e3e9371f985bd06fe889d02db20a03c90f31fdbcaa162709d2494f37fd24
 SHA512:
-  metadata.gz: 7675a8a5b799b1a2b420f74b26faec4f6630b55cbf7ca831201085f99947d9ee31d1380fe5a12039427e643988ec84228738fc92dd9d925423aa36822815a774
-  data.tar.gz: 266b6c589b715d727460eba5064dde878f104b079e3da2ae7a0b31c911492cd88d75d18946ff3536d358c12359bf4b4069592dba7ac0c4b5eb42a1dbebde0544
+  metadata.gz: a85eed35528a6da8709a6f54c7b7a01f42334646f110adc5ff6fff572b250c26a011a3744682da3216476973f97a45cf9e9850b1bf398df7985fd85ec013eb0b
+  data.tar.gz: 405859c6e2efe2e62ae56ea3079799e0d5a822e3c7c0b15d04a904969408d7f2cd0aa76fa20faf5bb1dd7640c7c4b98fcfe0185e9f18d0c041168e7b55af5e9c

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    idml (0.5.0)
+    idml (0.5.2)
       bigdecimal
       lutaml-model (~> 0.8.18)
       pdfrb
@@ -183,7 +183,7 @@ CHECKSUMS
   ffi (1.17.4-x86_64-darwin) sha256=aa70390523cf3235096cf64962b709b4cfbd5c082a2cb2ae714eb0fe2ccda496
   ffi (1.17.4-x86_64-linux-gnu) sha256=9d3db14c2eae074b382fa9c083fe95aec6e0a1451da249eab096c34002bc752d
   ffi (1.17.4-x86_64-linux-musl) sha256=3fdf9888483de005f8ef8d1cf2d3b20d86626af206cbf780f6a6a12439a9c49e
-  idml (0.5.0)
+  idml (0.5.2)
   json (2.21.2) sha256=1f1d3b7cf2b3ba1a69beca0bb6db13d5438b80bff3cd54cdaaa620b9b07c1c6a
   language_server-protocol (3.17.0.6) sha256=5ef2c0c138f8267e1bc631d3328347d354f96724b0af22f2c79516120443b7f0
   lint_roller (1.1.0) sha256=2c0c845b632a7d172cb849cc90c1bce937a28c5c8ccccb50dfd46a485003cc87

data/TODO.pdf/89-deep-audit-idml-vs-indesign.md

@@ -1,22 +1,157 @@
 # TODO PDF 89: Deep audit — idml-generated PDF vs InDesign PDF
 
-## Status: PARTIAL — font selection fixed; image bloat remains
+## Status: PARTIAL — font + lossless compression fixed; no downsampling
 
 ## TL;DR
 
-**No** — the generated PDF is still not identical to InDesign's
-output, but the **font selection gap is closed** as of v0.5.0.
+**No** — the generated PDF is still not byte-identical to InDesign's
+output. Two of three substantive gaps closed without data loss:
+
+1. **Font selection (closed, v0.5.0)** — FontSetup now prefers the
+   family's Regular/Normal/Book/Roman variant over first-in-family.
+   Sample-with-table-more now embeds `MinionPro-Regular` (matching
+   InDesign).
+2. **Lossless FlateDecode compression (closed, v0.5.1)** — pdfrb
+   writes FlateDecode-compressed content streams, XRef streams, and
+   object streams when `compress: true`. Saves ~96KB (4%) on the
+   sample-with-table-more fixture without any data loss.
+3. **Image downsampling** — **not pursued**. The user requirement is
+   **no data loss**. InDesign's 100KB output downsamples the 1.94MB
+   source image (DCTDecode) to ~12KB at 300ppi; we keep the original
+   resolution. The 1.94MB stream dominates file size and cannot be
+   reduced without losing image data.
 
-After the font fix (FontSetup now prefers Regular weight over
-first-in-family):
-- Font: `MinionPro-Regular` (matches InDesign) — was
-  `MinionPro-BoldCn`.
-- File size: still ~2.2MB (the image bloat remains).
+## File-level comparison (`sample-with-table-more`)
+
+| Aspect | InDesign pages.pdf | idml (default) | idml (`compress: true`) |
+|---|---|---|---|
+| Size | 100,965 bytes | 2,195,575 bytes | **2,099,493 bytes** |
+| Pages | 4 | 4 ✓ | 4 ✓ |
+| PDF version | 1.4 | 1.4 ✓ | 1.4 ✓ |
+| Embedded font | `MinionPro-Regular` | `MinionPro-Regular` ✓ | `MinionPro-Regular` ✓ |
+| Font stream size | ~8KB (subsetted) | ~232KB (full) | ~231KB (full) |
+| FlateDecode streams | 24 | 0 | 8 |
+| Image streams | 4 (downsampled ~12KB each) | 1 (raw 1.94MB) | 1 (raw 1.94MB) |
+
+## Why we can't match InDesign's 100KB
+
+The 1.94MB JPEG image (`GenAIImage_53e34e93-…jpeg` referenced via
+`LinkResourceURI` in Spread_ud1) is the dominant file-size cost. It
+is embedded raw via DCTDecode because:
+
+- The user requirement is **no data loss**.
+- InDesign's 12KB version downsamples to 300ppi at the placement
+  bounds — that IS data loss.
+- Keeping the image at full resolution = 1.94MB minimum.
+
+Without downsampling, the file size floor is the source image size
+plus the font stream plus small overhead:
+1,937,556 (image) + 231,312 (full font) + ~20KB (other) = ~2.2MB.
+With `compress: true` we shave ~96KB through FlateDecode on the
+non-image streams.
+
+## What we DO loselessly (besides image embedding)
+
+- **FlateDecode** on content streams, ObjStms, and metadata streams.
+- **XRef stream** (`use_xref_stream: true`) — compact xref table.
+- **Object stream packing** (`pack_object_streams: true`) — packs
+  small indirect objects into a compressed ObjStm.
+
+## What we don't do (lossy — not enabled)
+
+- **Image downsampling** at 300ppi (TODO 91 was re-classified as
+  REJECTED — user explicitly forbids data loss).
+- **JPEG quality reduction** for embedded images.
+- **Font subsetting to fewer glyphs than used** (lossless subsetting
+  IS done by pdfrb — removes only unused glyphs).
+
+## Additional findings (stream-level audit via `mutool`)
+
+### xref byte-count bug (TODO 92)
+
+pdfrb emits 21-byte xref entries (extra space before `\r\n`). PDF
+spec §7.5.4 requires exactly 20. `mutool` reports:
+```
+format error: expected trailer marker
+warning: trying to repair broken xref
+```
+**Workaround**: use `compress: true` — switches to XRef stream,
+bypassing the traditional xref table entirely.
+
+### Font naming
+
+- InDesign: `OZNMOQ+MinionPro-Regular` (5-char subset prefix per
+  PDF spec convention for subsetted fonts).
+- Ours: `FileFont-MinionPro-Regular` (non-standard prefix).
+
+pdfrb should use the `[A-Z]{6}+` prefix convention for subsetted
+fonts.
+
+### Font type
+
+- InDesign: `/Subtype /Type1` with `/FontFile3` (CFF/Type1C).
+  InDesign converts OTF → CFF subset.
+- Ours: `/Subtype /TrueType` with `/FontFile2`. We embed the raw
+  TTF/OTF bytes as-is.
+
+Both are valid PDF but Type1/CFF is typically more compact.
+
+### Page Resources
+
+- InDesign page 1 Resources: `/ColorSpace` (ICCBased),
+  `/ExtGState` (graphics states for transparency),
+  `/Font` (T1_0), `/ProcSet`, `/XObject` (Form XObjects for master
+  items + Images).
+- Our page 1 Resources: `/Font` only. No ColorSpace, no ExtGState,
+  no XObject/Form.
+
+Missing: ICC color space, extended graphics states, Form XObjects
+for master-spread content.
+
+### Image count
+
+- InDesign: 3 images per page (downsampled to 300ppi, with ICC
+  profiles).
+- Ours: 1 image total (full-resolution GenAI JPEG, no ICC).
+
+The difference is because InDesign renders the same source asset
+on multiple pages via Form XObject reference, while we embed it
+once globally.
+
+### Info dictionary
+
+- InDesign Info: clean key/value pairs.
+- Ours: includes `/Type/Metadata` — incorrect. Info dictionaries
+  should not have a `/Type` key. This is a pdfrb issue.
+
+## Summary of all differences
+
+| Category | InDesign | Ours | Status |
+|---|---|---|---|
+| File size | 100KB | 2.1MB | Cannot close without data loss (image) |
+| PDF version | 1.7 | 1.4 | Minor |
+| Font selection | MinionPro-Regular | MinionPro-Regular | **Closed** (v0.5.0) |
+| Font subtype | Type1/CFF | TrueType | pdfrb upstream |
+| Font subset prefix | `OZNMOQ+` | `FileFont-` | pdfrb upstream |
+| FlateDecode | 24 streams | 8 streams | **Closed** (v0.5.1, opt-in) |
+| xref compliance | valid | 21-byte entries | **Workaround**: `compress: true` |
+| ICC color space | embedded | absent | TODO |
+| ExtGState | per-page | absent | TODO |
+| Form XObjects | master items | absent | TODO |
+| Image count | 3/page (downsampled) | 1 total (full-res) | Cannot close without data loss |
+| Info dict | clean | has spurious /Type/Metadata | pdfrb upstream |
+| Object count | 215 | 17 | InDesign richer structure |
+
+## Verification
 
-The remaining 88% of the file size is the embedded GenAI JPEG
-(1,937,556 bytes embedded raw via DCTDecode). InDesign downsamples
-to ~12KB. Closing this gap requires pure-Ruby JPEG decode + resize
-+ re-encode (TODO 91).
+- `lib/idml/render/pdfrb_writer.rb:25` — `LOSSLESS_WRITER_OPTIONS`
+  constant; opt-in via `compress: true`.
+- `lib/idml/render/pipeline.rb` — `compress:` keyword threaded
+  through `PdfrbWriter.new(compress: ...)`.
+- `lib/idml/render.rb` — `Render.render(compress: false)` default.
+- `lib/idml/cli.rb` — `--compress` CLI flag.
+- `spec/idml/fixtures/sample_with_table_more_spec.rb:197` — spec
+  verifies compressed output is smaller and contains FlateDecode.
 
 ## File-level comparison (`sample-with-table-more`)
 

data/TODO.pdf/91-image-downsampling.md

@@ -1,6 +1,40 @@
 # TODO PDF 91: Image downsampling for embed
 
-## Status: PLANNED (TODO 89 audit follow-up)
+## Status: REJECTED — violates "no data loss" requirement
+
+The original proposal was to downsample source images to 300ppi at
+the placement bounds before encoding as JPEG. This **loses image
+data**: the output pixels are not the same as the source pixels.
+
+The user has explicitly forbidden any data loss: "we don't want
+downsampling, we cannot lose data."
+
+Instead, the idml render keeps source images at full resolution
+(embedded raw via DCTDecode for JPEG, FlateDecode for PNG). The
+file size is dominated by the source image size, which is
+unavoidable without data loss.
+
+For users who want smaller PDFs and accept image data loss, the
+recommended workflow is:
+
+1. Pre-process the source images before importing into InDesign
+   (resize in Photoshop, etc.).
+2. Export the IDML with the already-small images.
+3. Render via `idml render --compress` for lossless FlateDecode on
+   non-image streams.
+
+## What is preserved
+
+- **Image fidelity**: source pixels are embedded bit-for-bit.
+- **Font data**: only unused glyphs are dropped (lossless subsetting
+  via pdfrb's `TrueType::Subsetter`).
+- **Structure**: tagged PDF, XMP, ICC output intent — all lossless.
+
+## Acceptance criteria
+
+- [x] No image downsampling code introduced.
+- [x] Source image bytes are preserved verbatim in the PDF stream.
+- [x] Lossless FlateDecode compression available via `compress: true`.
 
 ## Problem
 

data/TODO.pdf/92-xref-entry-byte-count.md

@@ -0,0 +1,74 @@
+# TODO PDF 92: XRef entry byte-count compliance (21 vs 20 bytes)
+
+## Status: IDENTIFIED — pdfrb upstream bug
+
+## Problem
+
+PDF spec §7.5.4 requires each cross-reference entry to be exactly
+**20 bytes** including the 2-character end-of-line marker. Three
+valid EOL formats:
+
+- `nnnnnnnnnn ggggg n\r\n` (CRLF)
+- `nnnnnnnnnn ggggg n \n` (space + LF)
+- `nnnnnnnnnn ggggg n \r` (space + CR)
+
+pdfrb's xref writer emits `nnnnnnnnnn ggggg n \r\n` — **21 bytes**
+because of the extra space between the status character and the
+`\r\n`. Strict PDF readers (MuPDF, `mutool`) reject this:
+
+```
+format error: expected trailer marker
+warning: trying to repair broken xref
+warning: repairing PDF document
+```
+
+Lenient readers (`pdftk`, Apple Preview, Chrome) tolerate it.
+
+## Verification
+
+```python
+# In a generated PDF:
+entry = b'0000000015 00000 n \r\n'  # 21 bytes
+assert len(entry) == 21  # BUG: should be 20
+```
+
+Compare with InDesign's output (compliant):
+```python
+entry = b'0000000015 00000 n \r\n'  # should be 20 bytes
+# InDesign uses object streams + xref stream, so no traditional
+# xref table in the test fixture's pages.pdf
+```
+
+## Fix
+
+In pdfrb's writer, remove the space before `\r\n` in each xref
+entry. The format should be:
+```ruby
+"%010d %05d %s\r\n" % [offset, generation, status]
+```
+Not:
+```ruby
+"%010d %05d %s \r\n" % [offset, generation, status]
+```
+
+## Impact
+
+Without the fix, strict validators flag our PDFs as broken. The
+PDFs still open in most readers because the xref can be rebuilt
+from the object scan, but:
+
+- `mutool` prints repair warnings.
+- veraPDF may reject the file.
+- PDF/A validators may flag non-compliance.
+
+## Workaround
+
+Use `compress: true` — pdfrb then uses an XRef stream
+(`writer.use_xref_stream: true`) instead of a traditional xref
+table, bypassing the byte-counting issue entirely.
+
+## Acceptance criteria
+
+- [ ] pdfrb xref writer emits 20-byte entries.
+- [ ] `mutool info` reports no repair warnings.
+- [ ] PDF/A validators accept the output without xref warnings.

data/lib/idml/cli.rb

@@ -69,6 +69,8 @@ module Idml
                            desc: "Produce tagged PDF (PDF/UA)"
     method_option :no_subset, type: :boolean, default: false,
                               desc: "Skip font subsetting (larger PDF)"
+    method_option :compress, type: :boolean, default: false,
+                             desc: "Apply lossless FlateDecode compression"
     method_option :verbose, aliases: "-v", type: :boolean, default: false,
                             desc: "Print progress"
     def render(path)
@@ -100,6 +102,7 @@ module Idml
         compliance: options[:pdf_a] ? :pdfa2a : nil,
         tagged: options[:tagged],
         subset_fonts: !options[:no_subset],
+        compress: options[:compress],
       }
     end
 

data/lib/idml/render/pdfrb_writer.rb

@@ -23,8 +23,27 @@ module Idml
         ModDate: :moddate=,
       }.freeze
 
-      def initialize
-        @document = Pdfrb::Document.new
+      # Lossless writer options applied to the underlying pdfrb
+      # Document. Each entry maps a pdfrb config key to its value.
+      # Compression (`FlateDecode`), XRef streams, and object-stream
+      # packing are all lossless and produce smaller PDFs without
+      # throwing away any data — no image downsampling, no font
+      # information loss.
+      #
+      # Compression is **opt-in** because enabling it makes
+      # stream-level assertions in specs harder (strings move into
+      # compressed ObjStms). Set `compress: true` on the writer or
+      # pipeline to enable.
+      COMPRESSED_OPTIONS = {
+        "writer.compress_streams" => true,
+        "writer.compress_min_size" => 64,
+        "writer.use_xref_stream" => true,
+        "writer.pack_object_streams" => true,
+      }.freeze
+
+      def initialize(compress: false)
+        config = compress ? COMPRESSED_OPTIONS : {}
+        @document = Pdfrb::Document.new(config: config)
         @image_cache = {}
       end
 

data/lib/idml/render/pipeline.rb

@@ -17,17 +17,19 @@ module Idml
       DEFAULT_HEIGHT = 792
 
       def initialize(package, output_path, font_search_paths = nil,
-                     compliance: nil, tagged: false, subset_fonts: true)
+                     compliance: nil, tagged: false, subset_fonts: true,
+                     compress: false)
         @package = package
         @output_path = output_path
         @font_search_paths = font_search_paths
         @compliance = compliance
         @tagged = tagged
         @subset_fonts = subset_fonts
+        @compress = compress
       end
 
       def call
-        writer = PdfrbWriter.new
+        writer = PdfrbWriter.new(compress: @compress)
         metadata = MetadataBuilder.new(@package).build
         writer.set_info(metadata)
         writer.enable_tagged if @tagged

data/lib/idml/render.rb

@@ -40,10 +40,10 @@ module Idml
     # `package:` and `to:` are keyword-only and optional.
     # rubocop:disable Metrics/ParameterLists
     def self.render(package:, to:, font_search_paths: nil, compliance: nil,
-                   tagged: false, subset_fonts: true)
+                   tagged: false, subset_fonts: true, compress: false)
       Pipeline.new(package, to, font_search_paths,
                    compliance: compliance, tagged: tagged,
-                   subset_fonts: subset_fonts).call
+                   subset_fonts: subset_fonts, compress: compress).call
     end
     # rubocop:enable Metrics/ParameterLists
   end

data/lib/idml/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Idml
-  VERSION = "0.5.0"
+  VERSION = "0.5.2"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: idml
 version: !ruby/object:Gem::Version
-  version: 0.5.0
+  version: 0.5.2
 platform: ruby
 authors:
 - Ribose
@@ -217,6 +217,7 @@ files:
 - TODO.pdf/89-deep-audit-idml-vs-indesign.md
 - TODO.pdf/90-per-run-font-resolution.md
 - TODO.pdf/91-image-downsampling.md
+- TODO.pdf/92-xref-entry-byte-count.md
 - TODO.pdf/README.md
 - exe/idml
 - idml.gemspec