idml 0.5.0 → 0.5.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: '09c5f631233897e7f1e18519b1031c1c87092a1e3ddffdc353bf2e121c47b15b'
-  data.tar.gz: 8c0b965da67d83e30f3b7002be6bbabe1dffec7d1a3557979fe4cdfbce99dd54
+  metadata.gz: d4b32bcaed6562461191096ac7bbcd0ff8a14c151056bdbf122c439a711dc869
+  data.tar.gz: 30cff3491f9588716cfb676b00509f728c7c2af21b0efe9cb5a6ac4f2b77fe0b
 SHA512:
-  metadata.gz: 7675a8a5b799b1a2b420f74b26faec4f6630b55cbf7ca831201085f99947d9ee31d1380fe5a12039427e643988ec84228738fc92dd9d925423aa36822815a774
-  data.tar.gz: 266b6c589b715d727460eba5064dde878f104b079e3da2ae7a0b31c911492cd88d75d18946ff3536d358c12359bf4b4069592dba7ac0c4b5eb42a1dbebde0544
+  metadata.gz: a97de15a173bca917a1e10aa4fc32eb553b2ee2bd88fc33f49953437513e923b90acdaa02a734b74bc3d51493ba464aadbfbb51f86ede65369c0cf2a7a779911
+  data.tar.gz: 55448f8fb8d465fe0fbc2d078b4530ef80843721055856717788d7a77f6984f5e20a44dcaa41a8e2741a66b17b616b174f322ed0a770b3236c8617b6545054c4

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    idml (0.5.0)
+    idml (0.5.1)
       bigdecimal
       lutaml-model (~> 0.8.18)
       pdfrb
@@ -183,7 +183,7 @@ CHECKSUMS
   ffi (1.17.4-x86_64-darwin) sha256=aa70390523cf3235096cf64962b709b4cfbd5c082a2cb2ae714eb0fe2ccda496
   ffi (1.17.4-x86_64-linux-gnu) sha256=9d3db14c2eae074b382fa9c083fe95aec6e0a1451da249eab096c34002bc752d
   ffi (1.17.4-x86_64-linux-musl) sha256=3fdf9888483de005f8ef8d1cf2d3b20d86626af206cbf780f6a6a12439a9c49e
-  idml (0.5.0)
+  idml (0.5.1)
   json (2.21.2) sha256=1f1d3b7cf2b3ba1a69beca0bb6db13d5438b80bff3cd54cdaaa620b9b07c1c6a
   language_server-protocol (3.17.0.6) sha256=5ef2c0c138f8267e1bc631d3328347d354f96724b0af22f2c79516120443b7f0
   lint_roller (1.1.0) sha256=2c0c845b632a7d172cb849cc90c1bce937a28c5c8ccccb50dfd46a485003cc87

data/TODO.pdf/89-deep-audit-idml-vs-indesign.md

@@ -1,22 +1,91 @@
 # TODO PDF 89: Deep audit — idml-generated PDF vs InDesign PDF
 
-## Status: PARTIAL — font selection fixed; image bloat remains
+## Status: PARTIAL — font + lossless compression fixed; no downsampling
 
 ## TL;DR
 
-**No** — the generated PDF is still not identical to InDesign's
-output, but the **font selection gap is closed** as of v0.5.0.
+**No** — the generated PDF is still not byte-identical to InDesign's
+output. Two of three substantive gaps closed without data loss:
+
+1. **Font selection (closed, v0.5.0)** — FontSetup now prefers the
+   family's Regular/Normal/Book/Roman variant over first-in-family.
+   Sample-with-table-more now embeds `MinionPro-Regular` (matching
+   InDesign).
+2. **Lossless FlateDecode compression (closed, v0.5.1)** — pdfrb
+   writes FlateDecode-compressed content streams, XRef streams, and
+   object streams when `compress: true`. Saves ~96KB (4%) on the
+   sample-with-table-more fixture without any data loss.
+3. **Image downsampling** — **not pursued**. The user requirement is
+   **no data loss**. InDesign's 100KB output downsamples the 1.94MB
+   source image (DCTDecode) to ~12KB at 300ppi; we keep the original
+   resolution. The 1.94MB stream dominates file size and cannot be
+   reduced without losing image data.
 
-After the font fix (FontSetup now prefers Regular weight over
-first-in-family):
-- Font: `MinionPro-Regular` (matches InDesign) — was
-  `MinionPro-BoldCn`.
-- File size: still ~2.2MB (the image bloat remains).
+## File-level comparison (`sample-with-table-more`)
+
+| Aspect | InDesign pages.pdf | idml (default) | idml (`compress: true`) |
+|---|---|---|---|
+| Size | 100,965 bytes | 2,195,575 bytes | **2,099,493 bytes** |
+| Pages | 4 | 4 ✓ | 4 ✓ |
+| PDF version | 1.4 | 1.4 ✓ | 1.4 ✓ |
+| Embedded font | `MinionPro-Regular` | `MinionPro-Regular` ✓ | `MinionPro-Regular` ✓ |
+| Font stream size | ~8KB (subsetted) | ~232KB (full) | ~231KB (full) |
+| FlateDecode streams | 24 | 0 | 8 |
+| Image streams | 4 (downsampled ~12KB each) | 1 (raw 1.94MB) | 1 (raw 1.94MB) |
+
+## Why we can't match InDesign's 100KB
+
+The 1.94MB JPEG image (`GenAIImage_53e34e93-…jpeg` referenced via
+`LinkResourceURI` in Spread_ud1) is the dominant file-size cost. It
+is embedded raw via DCTDecode because:
+
+- The user requirement is **no data loss**.
+- InDesign's 12KB version downsamples to 300ppi at the placement
+  bounds — that IS data loss.
+- Keeping the image at full resolution = 1.94MB minimum.
+
+Without downsampling, the file size floor is the source image size
+plus the font stream plus small overhead:
+1,937,556 (image) + 231,312 (full font) + ~20KB (other) = ~2.2MB.
+With `compress: true` we shave ~96KB through FlateDecode on the
+non-image streams.
+
+## What we DO loselessly (besides image embedding)
+
+- **FlateDecode** on content streams, ObjStms, and metadata streams.
+- **XRef stream** (`use_xref_stream: true`) — compact xref table.
+- **Object stream packing** (`pack_object_streams: true`) — packs
+  small indirect objects into a compressed ObjStm.
+
+## What we don't do (lossy — not enabled)
+
+- **Image downsampling** at 300ppi (TODO 91 was re-classified as
+  REJECTED — user explicitly forbids data loss).
+- **JPEG quality reduction** for embedded images.
+- **Font subsetting to fewer glyphs than used** (lossless subsetting
+  IS done by pdfrb — removes only unused glyphs).
+
+## Future audit work
+
+- **Stream-by-stream comparison** with `qpdf --qdf` to decompose both
+  PDFs into canonical form and diff.
+- **Font subsetting investigation** — pdfrb's `subset_fonts!` is
+  called but the font stream is still 232KB. The likely cause is
+  that no codepoints are recorded as "used" for the registered font
+  (the renderer uses a default font that doesn't get text drawn
+  through it). Future fix: register the font actually used to draw
+  text, so `used_codepoints` populates correctly.
+
+## Verification
 
-The remaining 88% of the file size is the embedded GenAI JPEG
-(1,937,556 bytes embedded raw via DCTDecode). InDesign downsamples
-to ~12KB. Closing this gap requires pure-Ruby JPEG decode + resize
-+ re-encode (TODO 91).
+- `lib/idml/render/pdfrb_writer.rb:25` — `LOSSLESS_WRITER_OPTIONS`
+  constant; opt-in via `compress: true`.
+- `lib/idml/render/pipeline.rb` — `compress:` keyword threaded
+  through `PdfrbWriter.new(compress: ...)`.
+- `lib/idml/render.rb` — `Render.render(compress: false)` default.
+- `lib/idml/cli.rb` — `--compress` CLI flag.
+- `spec/idml/fixtures/sample_with_table_more_spec.rb:197` — spec
+  verifies compressed output is smaller and contains FlateDecode.
 
 ## File-level comparison (`sample-with-table-more`)
 

data/TODO.pdf/91-image-downsampling.md

@@ -1,6 +1,40 @@
 # TODO PDF 91: Image downsampling for embed
 
-## Status: PLANNED (TODO 89 audit follow-up)
+## Status: REJECTED — violates "no data loss" requirement
+
+The original proposal was to downsample source images to 300ppi at
+the placement bounds before encoding as JPEG. This **loses image
+data**: the output pixels are not the same as the source pixels.
+
+The user has explicitly forbidden any data loss: "we don't want
+downsampling, we cannot lose data."
+
+Instead, the idml render keeps source images at full resolution
+(embedded raw via DCTDecode for JPEG, FlateDecode for PNG). The
+file size is dominated by the source image size, which is
+unavoidable without data loss.
+
+For users who want smaller PDFs and accept image data loss, the
+recommended workflow is:
+
+1. Pre-process the source images before importing into InDesign
+   (resize in Photoshop, etc.).
+2. Export the IDML with the already-small images.
+3. Render via `idml render --compress` for lossless FlateDecode on
+   non-image streams.
+
+## What is preserved
+
+- **Image fidelity**: source pixels are embedded bit-for-bit.
+- **Font data**: only unused glyphs are dropped (lossless subsetting
+  via pdfrb's `TrueType::Subsetter`).
+- **Structure**: tagged PDF, XMP, ICC output intent — all lossless.
+
+## Acceptance criteria
+
+- [x] No image downsampling code introduced.
+- [x] Source image bytes are preserved verbatim in the PDF stream.
+- [x] Lossless FlateDecode compression available via `compress: true`.
 
 ## Problem
 

data/lib/idml/cli.rb

@@ -69,6 +69,8 @@ module Idml
                            desc: "Produce tagged PDF (PDF/UA)"
     method_option :no_subset, type: :boolean, default: false,
                               desc: "Skip font subsetting (larger PDF)"
+    method_option :compress, type: :boolean, default: false,
+                             desc: "Apply lossless FlateDecode compression"
     method_option :verbose, aliases: "-v", type: :boolean, default: false,
                             desc: "Print progress"
     def render(path)
@@ -100,6 +102,7 @@ module Idml
         compliance: options[:pdf_a] ? :pdfa2a : nil,
         tagged: options[:tagged],
         subset_fonts: !options[:no_subset],
+        compress: options[:compress],
       }
     end
 

data/lib/idml/render/pdfrb_writer.rb

@@ -23,8 +23,27 @@ module Idml
         ModDate: :moddate=,
       }.freeze
 
-      def initialize
-        @document = Pdfrb::Document.new
+      # Lossless writer options applied to the underlying pdfrb
+      # Document. Each entry maps a pdfrb config key to its value.
+      # Compression (`FlateDecode`), XRef streams, and object-stream
+      # packing are all lossless and produce smaller PDFs without
+      # throwing away any data — no image downsampling, no font
+      # information loss.
+      #
+      # Compression is **opt-in** because enabling it makes
+      # stream-level assertions in specs harder (strings move into
+      # compressed ObjStms). Set `compress: true` on the writer or
+      # pipeline to enable.
+      COMPRESSED_OPTIONS = {
+        "writer.compress_streams" => true,
+        "writer.compress_min_size" => 64,
+        "writer.use_xref_stream" => true,
+        "writer.pack_object_streams" => true,
+      }.freeze
+
+      def initialize(compress: false)
+        config = compress ? COMPRESSED_OPTIONS : {}
+        @document = Pdfrb::Document.new(config: config)
         @image_cache = {}
       end
 

data/lib/idml/render/pipeline.rb

@@ -17,17 +17,19 @@ module Idml
       DEFAULT_HEIGHT = 792
 
       def initialize(package, output_path, font_search_paths = nil,
-                     compliance: nil, tagged: false, subset_fonts: true)
+                     compliance: nil, tagged: false, subset_fonts: true,
+                     compress: false)
         @package = package
         @output_path = output_path
         @font_search_paths = font_search_paths
         @compliance = compliance
         @tagged = tagged
         @subset_fonts = subset_fonts
+        @compress = compress
       end
 
       def call
-        writer = PdfrbWriter.new
+        writer = PdfrbWriter.new(compress: @compress)
         metadata = MetadataBuilder.new(@package).build
         writer.set_info(metadata)
         writer.enable_tagged if @tagged

data/lib/idml/render.rb

@@ -40,10 +40,10 @@ module Idml
     # `package:` and `to:` are keyword-only and optional.
     # rubocop:disable Metrics/ParameterLists
     def self.render(package:, to:, font_search_paths: nil, compliance: nil,
-                   tagged: false, subset_fonts: true)
+                   tagged: false, subset_fonts: true, compress: false)
       Pipeline.new(package, to, font_search_paths,
                    compliance: compliance, tagged: tagged,
-                   subset_fonts: subset_fonts).call
+                   subset_fonts: subset_fonts, compress: compress).call
     end
     # rubocop:enable Metrics/ParameterLists
   end

data/lib/idml/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Idml
-  VERSION = "0.5.0"
+  VERSION = "0.5.1"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: idml
 version: !ruby/object:Gem::Version
-  version: 0.5.0
+  version: 0.5.1
 platform: ruby
 authors:
 - Ribose