corp_pdf 1.0.5

data/docs/object_streams.md

@@ -0,0 +1,311 @@
+# PDF Object Streams Explained
+
+## Overview
+
+PDF Object Streams (also called "ObjStm") are a compression feature in PDFs that allows multiple PDF objects to be stored together in a single compressed stream. This reduces file size and improves performance. `CorpPdf` handles object streams transparently, so you don't need to worry about them when working with PDF objects—but understanding how they work helps explain how the library parses PDFs.
+
+## What Are Object Streams?
+
+Instead of storing objects individually in the PDF body:
+
+```
+5 0 obj
+<< /Type /Annot /Subtype /Widget >>
+endobj
+
+6 0 obj
+<< /Type /Annot /Subtype /Widget >>
+endobj
+
+7 0 obj
+<< /Type /Annot /Subtype /Widget >>
+endobj
+```
+
+Object streams allow multiple objects to be **packed together** in a single compressed stream:
+
+```
+10 0 obj
+<< /Type /ObjStm /N 3 /First 20 >>
+stream
+[compressed header + object bodies]
+endstream
+endobj
+```
+
+Where:
+- `/N 3` means there are 3 objects in this stream
+- `/First 20` means the object data starts at byte offset 20 (the first 20 bytes are the header)
+- The stream contains: header (object numbers + offsets) + object bodies
+
+## Object Stream Structure
+
+An object stream consists of two parts:
+
+### 1. Header Section (First N bytes)
+
+The header is a list of space-separated integers:
+```
+5 0 6 10 7 25
+```
+
+This means:
+- Object 5 starts at offset 0 (relative to the start of object data)
+- Object 6 starts at offset 10 (relative to the start of object data)
+- Object 7 starts at offset 25 (relative to the start of object data)
+
+Format: `obj_num offset obj_num offset ...` (pairs of object number and offset)
+
+### 2. Object Data Section (Starting at /First)
+
+The object data section contains the actual object bodies, concatenated together:
+```
+<< /Type /Annot /Subtype /Widget >><< /Type /Annot /Subtype /Widget >><< /Type /Annot /Subtype /Widget >>
+```
+
+The offsets in the header tell us where each object body starts within this data section.
+
+### Complete Example
+
+```
+10 0 obj
+<< /Type /ObjStm /N 3 /First 20 /Filter /FlateDecode >>
+stream
+[Compressed bytes containing:]
+Header: "5 0 6 10 7 25 "
+Data:   "<< /Type /Annot /Subtype /Widget >><< /Type /Annot /Subtype /Widget >><< /Type /Annot /Subtype /Widget >>"
+endstream
+endobj
+```
+
+After decompression:
+- Bytes 0-19: Header (`"5 0 6 10 7 25 "`)
+- Bytes 20+: Object data (`"<< /Type /Annot /Subtype /Widget >>..."`)
+- Object 5 body: bytes 20-29 (offset 0 + 20 = 20, next object at offset 10)
+- Object 6 body: bytes 30-59 (offset 10 + 20 = 30, next object at offset 25)
+- Object 7 body: bytes 45+ (offset 25 + 20 = 45)
+
+## How CorpPdf Parses Object Streams
+
+### Step 1: Cross-Reference Table Parsing
+
+When `ObjectResolver` parses the cross-reference table or xref stream, it identifies objects stored in object streams:
+
+```ruby
+# In parse_xref_stream_records
+when 2 then @entries[ref] ||= Entry.new(type: :in_objstm, objstm_num: f1, objstm_index: f2)
+```
+
+Where:
+- `f1` = object stream number (the container object)
+- `f2` = index within the object stream (0 = first object, 1 = second object, etc.)
+
+Example: If object 5 is at index 0 in object stream 10, then `Entry` stores:
+- `type: :in_objstm`
+- `objstm_num: 10` (the object stream container)
+- `objstm_index: 0` (first object in the stream)
+
+### Step 2: Lazy Loading
+
+When you request an object body that's in an object stream, `ObjectResolver` calls `load_objstm`:
+
+```ruby
+def load_objstm(container_ref)
+  return if @objstm_cache.key?(container_ref)  # Already cached
+  
+  # Get the object stream container's body
+  body = object_body(container_ref)
+  
+  # Extract dictionary to get /N and /First
+  dict_src = extract_dictionary(body)
+  n = DictScan.value_token_after("/N", dict_src).to_i
+  first = DictScan.value_token_after("/First", dict_src).to_i
+  
+  # Extract and decode stream data
+  raw = decode_stream_data(dict_src, extract_stream_body(body))
+  
+  # Parse the object stream
+  parsed = CorpPdf::ObjStm.parse(raw, n: n, first: first)
+  
+  # Cache the result
+  @objstm_cache[container_ref] = parsed
+end
+```
+
+### Step 3: Stream Decoding
+
+The `decode_stream_data` method handles:
+1. **Extracting stream body**: Removes `stream` and `endstream` keywords
+2. **Decompression**: If `/Filter /FlateDecode` is present, decompress using zlib
+3. **PNG Predictor**: If `/Predictor` is present (10-15), apply PNG predictor decoding
+
+Example:
+```ruby
+def decode_stream_data(dict_src, stream_chunk)
+  # Extract body between stream...endstream
+  body = extract_stream_body(stream_chunk)
+  
+  # Decompress if FlateDecode
+  data = if dict_src =~ %r{/Filter\s*/FlateDecode}
+           Zlib::Inflate.inflate(body)
+         else
+           body
+         end
+  
+  # Apply PNG predictor if present
+  if dict_src =~ %r{/Predictor\s+(\d+)}
+    # Decode PNG predictor (Sub, Up, Average, Paeth)
+    data = apply_png_predictor(data, columns)
+  end
+  
+  data
+end
+```
+
+### Step 4: Object Stream Parsing (`ObjStm.parse`)
+
+The `ObjStm.parse` method is the heart of object stream parsing:
+
+```ruby
+def self.parse(bytes, n:, first:)
+  # Extract header (first N bytes)
+  head = bytes[0...first]
+  
+  # Parse space-separated integers: obj_num offset obj_num offset ...
+  entries = head.strip.split(/\s+/).map!(&:to_i)
+  
+  # Extract each object body
+  refs = []
+  n.times do |i|
+    obj = entries[2 * i]        # Object number
+    off = entries[(2 * i) + 1]  # Offset in data section
+    
+    # Calculate next offset (or end of data)
+    next_off = i + 1 < n ? entries[(2 * (i + 1)) + 1] : (bytes.bytesize - first)
+    
+    # Extract object body: start at (first + off), length is (next_off - off)
+    body = bytes[first + off, next_off - off]
+    
+    refs << { ref: [obj, 0], body: body }
+  end
+  
+  refs
+end
+```
+
+**Step-by-step example:**
+
+Given:
+- `bytes` = decompressed stream data
+- `n = 3` (3 objects)
+- `first = 20` (data starts at byte 20)
+- Header: `"5 0 6 10 7 25 "` (12 bytes, padded to 20)
+
+Processing:
+1. Extract header: `bytes[0...20]` → `"5 0 6 10 7 25 "`
+2. Parse entries: `[5, 0, 6, 10, 7, 25]`
+3. For `i=0` (first object):
+   - `obj = entries[0]` = 5
+   - `off = entries[1]` = 0
+   - `next_off = entries[3]` = 10 (offset of next object)
+   - `body = bytes[20 + 0, 10 - 0]` = bytes[20...30]
+4. For `i=1` (second object):
+   - `obj = entries[2]` = 6
+   - `off = entries[3]` = 10
+   - `next_off = entries[5]` = 25
+   - `body = bytes[20 + 10, 25 - 10]` = bytes[30...45]
+5. For `i=2` (third object):
+   - `obj = entries[4]` = 7
+   - `off = entries[5]` = 25
+   - `next_off = bytes.bytesize - first` (end of data)
+   - `body = bytes[20 + 25, ...]` = bytes[45...end]
+
+Result:
+```ruby
+[
+  { ref: [5, 0], body: "<< /Type /Annot /Subtype /Widget >>" },
+  { ref: [6, 0], body: "<< /Type /Annot /Subtype /Widget >>" },
+  { ref: [7, 0], body: "<< /Type /Annot /Subtype /Widget >>" }
+]
+```
+
+### Step 5: Object Retrieval
+
+When `object_body(ref)` is called for an object in an object stream:
+
+```ruby
+def object_body(ref)
+  case (e = @entries[ref])&.type
+  when :in_file
+    # Regular object: extract from file
+    extract_from_file(e.offset)
+  when :in_objstm
+    # Object stream: load stream (if not cached), then get object by index
+    load_objstm([e.objstm_num, 0])
+    @objstm_cache[[e.objstm_num, 0]][e.objstm_index][:body]
+  end
+end
+```
+
+The index (`objstm_index`) tells us which object in the parsed array to return.
+
+## Why Object Streams Matter
+
+### Benefits
+
+1. **File Size**: Compressing multiple objects together is more efficient than compressing each individually
+2. **Performance**: Fewer objects to parse when opening the PDF
+3. **Common in Modern PDFs**: Most PDFs created by modern tools use object streams
+
+### Transparency in CorpPdf
+
+`CorpPdf` handles object streams automatically:
+- You don't need to know if an object is in a stream or not
+- `object_body(ref)` returns the object body the same way regardless
+- Object streams are cached after first load (no repeated parsing)
+- The same `DictScan` methods work on extracted object bodies
+
+## Cross-Reference Streams vs Object Streams
+
+**Important distinction:**
+
+1. **XRef Streams** (`/Type /XRef`): Used to find where objects are located in the PDF
+   - Contains byte offsets or references to object streams
+   - Replaces classic xref tables
+
+2. **Object Streams** (`/Type /ObjStm`): Used to store actual object bodies
+   - Contains compressed object dictionaries
+   - Referenced by xref streams or classic xref tables
+
+Both use the same stream format (compressed, potentially with PNG predictor), but serve different purposes.
+
+## PNG Predictor
+
+PNG Predictor is a compression technique that predicts values based on previous values to improve compression. `CorpPdf` supports all 5 PNG predictor types:
+
+1. **Type 0 (None)**: No prediction
+2. **Type 1 (Sub)**: Predict from left
+3. **Type 2 (Up)**: Predict from above
+4. **Type 3 (Average)**: Predict from average of left and above
+5. **Type 4 (Paeth)**: Predict using Paeth algorithm
+
+The `apply_png_predictor` method decodes predictor-encoded data row by row, using the `/Columns` parameter to determine row width.
+
+## Summary
+
+Object streams allow PDFs to store multiple objects in compressed streams. `CorpPdf` handles them by:
+
+1. **Identifying** objects in streams via xref parsing
+2. **Lazy loading** stream containers when needed
+3. **Decoding** compressed stream data (zlib + PNG predictor)
+4. **Parsing** the header to extract object offsets
+5. **Extracting** individual object bodies by offset
+6. **Caching** parsed streams for performance
+
+The parsing itself is straightforward:
+- Header is space-separated integers (object numbers and offsets)
+- Object data follows the header
+- Extract each object body using its offset
+
+Just like `DictScan`, object stream parsing is **text traversal**—once the stream is decompressed, it's just parsing space-separated numbers and extracting substrings by offset.
+

data/docs/pdf_structure.md

@@ -0,0 +1,251 @@
+# PDF File Structure
+
+## Overview
+
+PDF (Portable Document Format) files have a reputation for being complex binary formats, but at their core, they are **text-based files with a structured syntax**. Understanding this fundamental fact is key to understanding how PDF works.
+
+While PDFs can contain binary data (like compressed streams, images, and fonts), the **structure** of a PDF—its objects, dictionaries, arrays, and references—is defined using plain text syntax.
+
+## PDF File Anatomy
+
+A PDF file consists of several main parts:
+
+1. **Header**: `%PDF-1.4` (or similar version)
+2. **Body**: A collection of PDF objects (the actual content)
+3. **Cross-Reference Table (xref)**: Points to byte offsets of objects
+4. **Trailer**: Contains the root object reference and metadata
+5. **EOF Marker**: `%%EOF`
+
+### PDF Objects
+
+The body contains PDF objects. Each object has:
+- An object number and generation number (e.g., `5 0 obj`)
+- Content (dictionary, array, stream, etc.)
+- An `endobj` marker
+
+Example:
+```
+5 0 obj
+<< /Type /Page /Parent 3 0 R /MediaBox [0 0 612 792] >>
+endobj
+```
+
+## PDF Dictionaries
+
+**PDF dictionaries are the heart of PDF structure.** They're defined using angle brackets:
+
+```
+<< /Key1 value1 /Key2 value2 /Key3 value3 >>
+```
+
+Think of them like JSON objects or Ruby hashes, but with PDF-specific syntax:
+- Keys are PDF names (always start with `/`)
+- Values can be: strings, numbers, booleans, arrays, dictionaries, or object references
+- Whitespace is generally ignored (but required between tokens)
+
+### Dictionary Examples
+
+**Simple dictionary:**
+```
+<< /Type /Page /Width 612 /Height 792 >>
+```
+
+**Nested dictionary:**
+```
+<< 
+  /Type /Annot
+  /Subtype /Widget
+  /Rect [100 500 200 520]
+  /AP <<
+    /N <<
+      /Yes 10 0 R
+      /Off 11 0 R
+    >>
+  >>
+>>
+```
+
+**Dictionary with array:**
+```
+<< /Kids [5 0 R 6 0 R 7 0 R] >>
+```
+
+**Dictionary with string values:**
+```
+<< /Title (My Document) /Author (John Doe) >>
+```
+
+The parentheses `()` denote literal strings in PDF syntax. Hex strings use angle brackets: `<>`.
+
+## PDF Text-Based Syntax
+
+Despite being "binary" files, PDFs use text-based syntax for their structure. This means:
+
+1. **Dictionaries are text**: `<< ... >>` are just character sequences
+2. **Arrays are text**: `[ ... ]` are just character sequences  
+3. **References are text**: `5 0 R` means "object 5, generation 0"
+4. **Strings can be text or hex**: `(Hello)` or `<48656C6C6F>`
+
+### Why This Matters
+
+Because PDF dictionaries are just text with delimiters (`<<`, `>>`), we can parse them using **simple text traversal algorithms**—no complex parser generator, no AST construction, just:
+
+1. Find opening `<<`
+2. Track nesting depth by counting `<<` and `>>`
+3. When depth reaches zero, we've found a complete dictionary
+4. Repeat
+
+## PDF Object References
+
+PDFs use references to link objects together:
+```
+5 0 R
+```
+
+This means:
+- Object number: `5`
+- Generation number: `0` (usually 0 for non-incremental PDFs)
+- `R` means "reference"
+
+When you see `/Parent 5 0 R`, it means the `Parent` key references object 5.
+
+## PDF Arrays
+
+Arrays are space-separated lists in square brackets:
+```
+[0 0 612 792]
+```
+
+Can contain any PDF value type:
+```
+[5 0 R 6 0 R]
+[/Yes /Off]
+[(Hello) (World)]
+```
+
+## PDF Strings
+
+PDF strings come in two flavors:
+
+### Literal Strings (parentheses)
+```
+(Hello World)
+(Line 1\nLine 2)
+```
+
+Can contain escape sequences: `\n`, `\r`, `\t`, `\\(`, `\\)`, octal `\123`.
+
+### Hex Strings (angle brackets)
+```
+<48656C6C6F>
+<FEFF00480065006C006C006F>
+```
+
+The hex string `<FEFF...>` with BOM indicates UTF-16BE encoding.
+
+## PDF Names
+
+PDF names start with `/`:
+```
+/Type
+/Subtype
+/Widget
+```
+
+Names can contain most characters except special delimiters.
+
+## Stream Objects
+
+Some PDF objects contain **streams** (binary or text data):
+```
+10 0 obj
+<< /Length 100 /Filter /FlateDecode >>
+stream
+[compressed binary data here]
+endstream
+endobj
+```
+
+For parsing structure (dictionaries), we typically strip or ignore stream bodies because they can contain arbitrary binary data that would confuse text-based parsing.
+
+## Why CorpPdf Works
+
+`CorpPdf` works because **PDF dictionaries are just text patterns**. Despite looking complicated, the algorithms are straightforward:
+
+### Finding Dictionaries
+
+The `each_dictionary` method:
+1. Searches for `<<` (start of dictionary)
+2. Tracks nesting depth: `<<` increments, `>>` decrements
+3. When depth returns to 0, we've found a complete dictionary
+4. Yield it and continue searching
+
+This is **pure text traversal**—no PDF-specific knowledge beyond "dictionaries use `<<` and `>>`".
+
+### Extracting Values
+
+The `value_token_after` method:
+1. Finds a key (like `/V`)
+2. Skips whitespace
+3. Based on the next character, extracts the value:
+   - `(` → Extract literal string (handle escaping)
+   - `<` → Extract hex string or dictionary
+   - `[` → Extract array (match brackets)
+   - `/` → Extract name
+   - Otherwise → Extract atom (number, reference, etc.)
+
+Again, this is just **text pattern matching** with some bracket/depth tracking.
+
+### Why It Seems Complicated
+
+The complexity comes from:
+1. **Handling edge cases**: Escaped characters, nested structures, various value types
+2. **Preserving exact formatting**: When replacing values, we must maintain valid PDF syntax
+3. **Encoding/decoding**: PDF strings have special encoding rules (UTF-16BE BOM, escapes)
+4. **Safety checks**: Verifying dictionaries are still valid after modification
+
+But the **core concept** is simple: PDF dictionaries are text, so we can parse them with text traversal.
+
+## Example: Walking Through a PDF Dictionary
+
+Given this PDF dictionary text:
+```
+<< /Type /Annot /Subtype /Widget /V (Hello World) /Rect [100 500 200 520] >>
+```
+
+How `CorpPdf` would parse it:
+
+1. **`each_dictionary` finds it:**
+   - Finds `<<` at position 0
+   - Depth: 0 → 1 (after `<<`)
+   - Scans forward...
+   - Finds `>>` at position 64
+   - Depth: 1 → 0
+   - Yields: `"<< /Type /Annot /Subtype /Widget /V (Hello World) /Rect [100 500 200 520] >>"`
+
+2. **`value_token_after("/V", dict)` extracts value:**
+   - Finds `/V` (followed by space)
+   - Skips whitespace
+   - Next char is `(`, so extract literal string
+   - Scan forward, handle escaping, match closing `)`
+   - Returns: `"(Hello World)"`
+
+3. **`decode_pdf_string("(Hello World)")` decodes:**
+   - Starts with `(`, ends with `)`
+   - Extract inner: `"Hello World"`
+   - Unescape (no escapes here)
+   - Check for UTF-16BE BOM (none)
+   - Return: `"Hello World"`
+
+## Conclusion
+
+PDF files are **structured text files** with binary data embedded in streams. The structure itself—dictionaries, arrays, strings, references—is all text-based syntax. This is why `CorpPdf` can use simple text traversal to parse and modify PDF dictionaries without needing a full PDF parser.
+
+The apparent complexity in `CorpPdf` comes from:
+- Handling PDF's various value types
+- Proper encoding/decoding of strings
+- Careful preservation of structure during edits
+- Edge case handling (escaping, nesting, etc.)
+
+But the **fundamental approach** is elegantly simple: treat PDF dictionaries as text patterns and parse them with character-by-character traversal.
+

data/issues/README.md

@@ -0,0 +1,59 @@
+# Code Review Issues
+
+This folder contains documentation of code cleanup, refactoring opportunities, and improvement tasks found in the codebase.
+
+## Files
+
+- **[refactoring-opportunities.md](./refactoring-opportunities.md)** - Detailed list of code duplication and refactoring opportunities
+- **[memory-improvements.md](./memory-improvements.md)** - Memory usage issues and optimization opportunities for handling larger PDF documents
+
+## Summary
+
+### High Priority Issues
+1. **Widget Matching Logic** - Duplicated across 6+ locations
+2. **/Annots Array Manipulation** - Complex logic duplicated in 3 locations
+
+### Medium Priority Issues
+3. **Box Parsing Logic** - Repeated code blocks for 5 box types
+4. **Checkbox Appearance Creation** - Significant duplication in new code
+5. **PDF Metadata Formatting** - Could benefit from being shared utilities
+
+### Low Priority Issues
+6. Duplicated `next_fresh_object_number` implementation (may be intentional)
+7. Object reference extraction pattern duplication
+8. Unused method: `get_widget_rect_dimensions`
+9. Base64 decoding logic duplication
+
+### Completed ✅
+- **Page-Finding Logic** - Successfully refactored into `DictScan.is_page?` and unified page-finding methods
+
+## Quick Stats
+
+- **10 refactoring opportunities** identified (1 completed, 9 remaining)
+- **6+ locations** with widget matching duplication
+- **3 locations** with /Annots array manipulation duplication
+- **1 unused method** found
+- **2 new issues** identified in recent code additions
+
+## Memory & Performance
+
+### Memory Improvement Opportunities
+
+See **[memory-improvements.md](./memory-improvements.md)** for detailed analysis of memory usage and optimization strategies.
+
+**Key Issues:**
+- Duplicate PDF loading (2x memory usage)
+- Stream decompression cache retention
+- All-objects-in-memory operations
+- Multiple full PDF copies during write operations
+
+**Estimated Impact:** 50-90MB typical usage for 10MB PDF, can exceed 100-200MB+ for larger/complex PDFs (39+ pages).
+
+## Next Steps
+
+1. Review [refactoring-opportunities.md](./refactoring-opportunities.md) for detailed information
+2. Review [memory-improvements.md](./memory-improvements.md) for memory optimization strategies
+3. Prioritize improvements based on maintenance and performance needs
+4. Create test coverage before refactoring
+5. Implement improvements incrementally, starting with high-priority items
+