acro_that 0.1.0

data/docs/dict_scan_explained.md

@@ -0,0 +1,341 @@
+# DictScan Explained: Text Traversal in Action
+
+## The Big Picture
+
+`DictScan` is a module that appears complicated at first glance, but it's fundamentally just **text traversal**—walking through PDF files character by character to find and extract dictionary structures.
+
+This document explains how each function in `DictScan` works and why the text-traversal approach is both powerful and straightforward.
+
+## Core Principle
+
+**PDF dictionaries are text patterns.** They use `<<` and `>>` as delimiters, just like how programming languages use `{` and `}` or `[` and `]`. Once you recognize this, parsing becomes a matter of tracking depth and matching delimiters.
+
+## Function-by-Function Guide
+
+### `strip_stream_bodies(pdf)`
+
+**Purpose:** Remove binary stream data that would confuse text parsing.
+
+**How it works:**
+- Finds all `stream...endstream` blocks using regex
+- Replaces the binary content with a placeholder
+- Preserves the stream structure markers
+
+**Why:** Streams can contain arbitrary binary data (compressed images, fonts, etc.) that would break our text-based parsing. We strip them out since we're only interested in the dictionary structure.
+
+```ruby
+pdf.gsub(/stream\r?\n.*?endstream/mi) { "stream\nENDSTREAM_STRIPPED\nendstream" }
+```
+
+This is regex-based, but it's necessary preprocessing before we can safely do text traversal.
+
+---
+
+### `each_dictionary(str)`
+
+**Purpose:** Iterate through all dictionaries in a string.
+
+**Algorithm:**
+1. Find the first `<<` at position `i`
+2. Initialize depth counter to 0
+3. Scan forward:
+   - If we see `<<`, increment depth
+   - If we see `>>`, decrement depth
+   - If depth reaches 0, we've found a complete dictionary
+4. Yield the dictionary substring
+5. Continue from where we left off
+
+**Example:**
+```
+Input: "<< /A 1 >> << /B 2 >>"
+  i=0: find "<<"
+  depth=1, scan forward
+  see ">>", depth=0 → found "<< /A 1 >>"
+  yield and continue from i=11
+  i=11: find "<<"
+  depth=1, scan forward
+  see ">>", depth=0 → found "<< /B 2 >>"
+  yield and continue
+```
+
+**Why it works:** This is classic **bracket matching**. No PDF-specific knowledge needed—just counting delimiters.
+
+---
+
+### `unescape_literal(s)`
+
+**Purpose:** Decode PDF escape sequences in literal strings.
+
+**PDF escapes:**
+- `\n` → newline
+- `\r` → carriage return
+- `\t` → tab
+- `\b` → backspace
+- `\f` → form feed
+- `\\(` → literal `(`
+- `\\)` → literal `)`
+- `\123` → octal character (up to 3 digits)
+
+**Algorithm:** Character-by-character scan:
+1. If we see `\`, look ahead one character
+2. Map escape sequences to actual characters
+3. Handle octal sequences (1-3 digits)
+4. Otherwise, copy character as-is
+
+**Why it works:** This is standard escape sequence handling, identical to how many programming languages handle string literals.
+
+---
+
+### `decode_pdf_string(token)`
+
+**Purpose:** Decode a PDF string token into a Ruby string.
+
+**PDF string types:**
+1. **Literal:** `(Hello World)` or `(Hello\nWorld)`
+2. **Hex:** `<48656C6C6F>` or `<FEFF00480065006C006C006F>`
+
+**Algorithm:**
+1. Check if token starts with `(` → literal string
+   - Extract content between parentheses
+   - Unescape using `unescape_literal`
+   - Check for UTF-16BE BOM (`FE FF`)
+   - Decode accordingly
+2. Check if token starts with `<` → hex string
+   - Remove spaces, pad if odd length
+   - Convert hex to bytes
+   - Check for UTF-16BE BOM
+   - Decode accordingly
+3. Otherwise, return as-is (name, number, reference, etc.)
+
+**Why it works:** PDF strings have well-defined formats. We just pattern-match on the delimiters and decode accordingly.
+
+---
+
+### `encode_pdf_string(val)`
+
+**Purpose:** Encode a Ruby value into a PDF string token.
+
+**Handles:**
+- `true` → `"true"`
+- `false` → `"false"`
+- `Symbol` → `"/symbol_name"`
+- `String`:
+  - ASCII-only → literal string `(value)`
+  - Non-ASCII → hex string with UTF-16BE encoding
+
+**Why it works:** Reverse of `decode_pdf_string`—we know the target format and encode accordingly.
+
+---
+
+### `value_token_after(key, dict_src)`
+
+**Purpose:** Extract the value token that follows a key in a dictionary.
+
+**This is the heart of text traversal.** Here's how it works:
+
+1. **Find the key:**
+   ```ruby
+   match = dict_src.match(%r{#{Regexp.escape(key)}(?=[\s(<\[/])})
+   ```
+   Use regex to ensure the key is followed by a delimiter (whitespace, `(`, `<`, `[`, or `/`). This prevents partial matches.
+
+2. **Skip whitespace:**
+   ```ruby
+   i += 1 while i < dict_src.length && dict_src[i] =~ /\s/
+   ```
+
+3. **Switch on the next character:**
+   - **`(` → Literal string:**
+     - Track depth of parentheses
+     - Handle escaped characters (skip `\` and next char)
+     - Match closing `)` when depth returns to 0
+   - **`<` → Hex string or dictionary:**
+     - If `<<` → return `"<<"` (nested dictionary marker)
+     - Otherwise, find matching `>`
+   - **`[` → Array:**
+     - Track depth of brackets
+     - Match closing `]` when depth returns to 0
+   - **`/` → PDF name:**
+     - Extract until whitespace or delimiter
+   - **Otherwise → Atom:**
+     - Extract until whitespace or delimiter (number, reference, boolean, etc.)
+
+**Why it works:** PDF has well-defined token syntax. Each value type has distinct delimiters, so we can pattern-match on the first character and extract accordingly.
+
+**Example:**
+```
+Dict: "<< /V (Hello) /R [1 2 3] >>"
+value_token_after("/V", dict):
+  → Finds "/V" at position 3
+  → Skips space
+  → Next char is "("
+  → Extracts "(Hello)" using paren matching
+  → Returns "(Hello)"
+```
+
+---
+
+### `replace_key_value(dict_src, key, new_token)`
+
+**Purpose:** Replace a key's value in a dictionary string.
+
+**Algorithm:**
+1. Find the key using pattern matching
+2. Extract the existing value token using `value_token_after`
+3. Find exact byte positions:
+   - Key start/end
+   - Value start/end
+4. Replace using string slicing: `before + new_token + after`
+5. Verify dictionary is still valid (contains `<<` and `>>`)
+
+**Why it works:** We use **precise byte positions** rather than regex replacement. This:
+- Preserves exact formatting (whitespace, etc.)
+- Avoids regex edge cases
+- Is deterministic and safe
+
+**Example:**
+```
+Input:  "<< /V (Old) /X 1 >>"
+        before: "<< /V "
+        value:  "(Old)"
+        after:  " /X 1 >>"
+Output: "<< /V (New) /X 1 >>"
+```
+
+---
+
+### `upsert_key_value(dict_src, key, token)`
+
+**Purpose:** Insert a key-value pair if the key doesn't exist.
+
+**Algorithm:**
+- If key not found, insert after the opening `<<`
+- Uses simple string substitution: `"<<#{key} #{token}"`
+
+**Why it works:** Simple string manipulation when we know the key doesn't exist.
+
+---
+
+### `remove_ref_from_array(array_body, ref)` and `add_ref_to_array(array_body, ref)`
+
+**Purpose:** Manipulate object references in PDF arrays.
+
+**Algorithm:**
+- Use regex/gsub to find and replace reference patterns: `"5 0 R"`
+- Handle edge cases: empty arrays, spacing
+
+**Why it works:** Object references have a fixed format (`num gen R`), so we can pattern-match and replace.
+
+---
+
+## Why This Approach Works
+
+### 1. PDF Structure is Text-Based
+
+PDF dictionaries, arrays, strings, and references are all defined using text syntax. No binary parsing needed for structure.
+
+### 2. Delimiters Are Unique
+
+Each PDF value type has distinct delimiters:
+- Dictionaries: `<<` `>>`
+- Arrays: `[` `]`
+- Literal strings: `(` `)`
+- Hex strings: `<` `>`
+- Names: `/`
+- References: `R`
+
+We can pattern-match on these to extract values.
+
+### 3. Depth Tracking is Simple
+
+Nested structures (dictionaries, arrays, strings) can be parsed by tracking depth—increment on open, decrement on close. Standard algorithm from compiler theory.
+
+### 4. Position-Based Replacement is Safe
+
+When modifying dictionaries, we use exact byte positions rather than regex replacement. This:
+- Preserves formatting
+- Avoids edge cases
+- Is predictable
+
+### 5. No Full Parser Needed
+
+We don't need to:
+- Build an AST
+- Validate the entire PDF structure
+- Handle all PDF features
+
+We only need to:
+- Find dictionaries
+- Extract values
+- Replace values
+- Preserve structure
+
+This is a **minimal parser** that does exactly what we need.
+
+## Common Patterns
+
+### Pattern 1: Find and Extract
+
+```ruby
+# Find all dictionaries
+each_dictionary(pdf_text) do |dict|
+  # Extract a value
+  value_token = value_token_after("/V", dict)
+  value = decode_pdf_string(value_token)
+  puts value
+end
+```
+
+### Pattern 2: Find and Replace
+
+```ruby
+# Get dictionary
+dict = "<< /V (Old) >>"
+
+# Replace value
+new_dict = replace_key_value(dict, "/V", "(New)")
+
+# Result: "<< /V (New) >>"
+```
+
+### Pattern 3: Encode and Insert
+
+```ruby
+# Prepare new value
+token = encode_pdf_string("Hello")
+
+# Insert into dictionary
+dict = upsert_key_value(dict, "/V", token)
+```
+
+## Performance Considerations
+
+**Why this is fast:**
+- No AST construction
+- No full PDF parsing
+- Direct string manipulation
+- Minimal memory allocation
+
+**Trade-offs:**
+- Doesn't validate entire PDF structure
+- Assumes dictionaries are well-formed
+- Stream stripping is regex-based (could be optimized)
+
+## Conclusion
+
+`DictScan` appears complicated because it handles many edge cases and value types, but the **core approach is elegantly simple**:
+
+1. PDF dictionaries are text patterns
+2. Parse them with character-by-character traversal
+3. Track depth for nested structures
+4. Use precise positions for replacement
+
+No magic, no complex parsers—just careful text traversal with attention to PDF syntax rules.
+
+The complexity you see is:
+- **Edge case handling** (escaping, nesting, encoding)
+- **Safety checks** (verification, error handling)
+- **Support for multiple value types** (strings, arrays, dictionaries, references)
+
+But the **fundamental algorithm** is straightforward: find delimiters, track depth, extract substrings, replace substrings.
+

data/docs/object_streams.md

@@ -0,0 +1,311 @@
+# PDF Object Streams Explained
+
+## Overview
+
+PDF Object Streams (also called "ObjStm") are a compression feature in PDFs that allows multiple PDF objects to be stored together in a single compressed stream. This reduces file size and improves performance. `AcroThat` handles object streams transparently, so you don't need to worry about them when working with PDF objects—but understanding how they work helps explain how the library parses PDFs.
+
+## What Are Object Streams?
+
+Instead of storing objects individually in the PDF body:
+
+```
+5 0 obj
+<< /Type /Annot /Subtype /Widget >>
+endobj
+
+6 0 obj
+<< /Type /Annot /Subtype /Widget >>
+endobj
+
+7 0 obj
+<< /Type /Annot /Subtype /Widget >>
+endobj
+```
+
+Object streams allow multiple objects to be **packed together** in a single compressed stream:
+
+```
+10 0 obj
+<< /Type /ObjStm /N 3 /First 20 >>
+stream
+[compressed header + object bodies]
+endstream
+endobj
+```
+
+Where:
+- `/N 3` means there are 3 objects in this stream
+- `/First 20` means the object data starts at byte offset 20 (the first 20 bytes are the header)
+- The stream contains: header (object numbers + offsets) + object bodies
+
+## Object Stream Structure
+
+An object stream consists of two parts:
+
+### 1. Header Section (First N bytes)
+
+The header is a list of space-separated integers:
+```
+5 0 6 10 7 25
+```
+
+This means:
+- Object 5 starts at offset 0 (relative to the start of object data)
+- Object 6 starts at offset 10 (relative to the start of object data)
+- Object 7 starts at offset 25 (relative to the start of object data)
+
+Format: `obj_num offset obj_num offset ...` (pairs of object number and offset)
+
+### 2. Object Data Section (Starting at /First)
+
+The object data section contains the actual object bodies, concatenated together:
+```
+<< /Type /Annot /Subtype /Widget >><< /Type /Annot /Subtype /Widget >><< /Type /Annot /Subtype /Widget >>
+```
+
+The offsets in the header tell us where each object body starts within this data section.
+
+### Complete Example
+
+```
+10 0 obj
+<< /Type /ObjStm /N 3 /First 20 /Filter /FlateDecode >>
+stream
+[Compressed bytes containing:]
+Header: "5 0 6 10 7 25 "
+Data:   "<< /Type /Annot /Subtype /Widget >><< /Type /Annot /Subtype /Widget >><< /Type /Annot /Subtype /Widget >>"
+endstream
+endobj
+```
+
+After decompression:
+- Bytes 0-19: Header (`"5 0 6 10 7 25 "`)
+- Bytes 20+: Object data (`"<< /Type /Annot /Subtype /Widget >>..."`)
+- Object 5 body: bytes 20-29 (offset 0 + 20 = 20, next object at offset 10)
+- Object 6 body: bytes 30-59 (offset 10 + 20 = 30, next object at offset 25)
+- Object 7 body: bytes 45+ (offset 25 + 20 = 45)
+
+## How AcroThat Parses Object Streams
+
+### Step 1: Cross-Reference Table Parsing
+
+When `ObjectResolver` parses the cross-reference table or xref stream, it identifies objects stored in object streams:
+
+```ruby
+# In parse_xref_stream_records
+when 2 then @entries[ref] ||= Entry.new(type: :in_objstm, objstm_num: f1, objstm_index: f2)
+```
+
+Where:
+- `f1` = object stream number (the container object)
+- `f2` = index within the object stream (0 = first object, 1 = second object, etc.)
+
+Example: If object 5 is at index 0 in object stream 10, then `Entry` stores:
+- `type: :in_objstm`
+- `objstm_num: 10` (the object stream container)
+- `objstm_index: 0` (first object in the stream)
+
+### Step 2: Lazy Loading
+
+When you request an object body that's in an object stream, `ObjectResolver` calls `load_objstm`:
+
+```ruby
+def load_objstm(container_ref)
+  return if @objstm_cache.key?(container_ref)  # Already cached
+  
+  # Get the object stream container's body
+  body = object_body(container_ref)
+  
+  # Extract dictionary to get /N and /First
+  dict_src = extract_dictionary(body)
+  n = DictScan.value_token_after("/N", dict_src).to_i
+  first = DictScan.value_token_after("/First", dict_src).to_i
+  
+  # Extract and decode stream data
+  raw = decode_stream_data(dict_src, extract_stream_body(body))
+  
+  # Parse the object stream
+  parsed = AcroThat::ObjStm.parse(raw, n: n, first: first)
+  
+  # Cache the result
+  @objstm_cache[container_ref] = parsed
+end
+```
+
+### Step 3: Stream Decoding
+
+The `decode_stream_data` method handles:
+1. **Extracting stream body**: Removes `stream` and `endstream` keywords
+2. **Decompression**: If `/Filter /FlateDecode` is present, decompress using zlib
+3. **PNG Predictor**: If `/Predictor` is present (10-15), apply PNG predictor decoding
+
+Example:
+```ruby
+def decode_stream_data(dict_src, stream_chunk)
+  # Extract body between stream...endstream
+  body = extract_stream_body(stream_chunk)
+  
+  # Decompress if FlateDecode
+  data = if dict_src =~ %r{/Filter\s*/FlateDecode}
+           Zlib::Inflate.inflate(body)
+         else
+           body
+         end
+  
+  # Apply PNG predictor if present
+  if dict_src =~ %r{/Predictor\s+(\d+)}
+    # Decode PNG predictor (Sub, Up, Average, Paeth)
+    data = apply_png_predictor(data, columns)
+  end
+  
+  data
+end
+```
+
+### Step 4: Object Stream Parsing (`ObjStm.parse`)
+
+The `ObjStm.parse` method is the heart of object stream parsing:
+
+```ruby
+def self.parse(bytes, n:, first:)
+  # Extract header (first N bytes)
+  head = bytes[0...first]
+  
+  # Parse space-separated integers: obj_num offset obj_num offset ...
+  entries = head.strip.split(/\s+/).map!(&:to_i)
+  
+  # Extract each object body
+  refs = []
+  n.times do |i|
+    obj = entries[2 * i]        # Object number
+    off = entries[(2 * i) + 1]  # Offset in data section
+    
+    # Calculate next offset (or end of data)
+    next_off = i + 1 < n ? entries[(2 * (i + 1)) + 1] : (bytes.bytesize - first)
+    
+    # Extract object body: start at (first + off), length is (next_off - off)
+    body = bytes[first + off, next_off - off]
+    
+    refs << { ref: [obj, 0], body: body }
+  end
+  
+  refs
+end
+```
+
+**Step-by-step example:**
+
+Given:
+- `bytes` = decompressed stream data
+- `n = 3` (3 objects)
+- `first = 20` (data starts at byte 20)
+- Header: `"5 0 6 10 7 25 "` (12 bytes, padded to 20)
+
+Processing:
+1. Extract header: `bytes[0...20]` → `"5 0 6 10 7 25 "`
+2. Parse entries: `[5, 0, 6, 10, 7, 25]`
+3. For `i=0` (first object):
+   - `obj = entries[0]` = 5
+   - `off = entries[1]` = 0
+   - `next_off = entries[3]` = 10 (offset of next object)
+   - `body = bytes[20 + 0, 10 - 0]` = bytes[20...30]
+4. For `i=1` (second object):
+   - `obj = entries[2]` = 6
+   - `off = entries[3]` = 10
+   - `next_off = entries[5]` = 25
+   - `body = bytes[20 + 10, 25 - 10]` = bytes[30...45]
+5. For `i=2` (third object):
+   - `obj = entries[4]` = 7
+   - `off = entries[5]` = 25
+   - `next_off = bytes.bytesize - first` (end of data)
+   - `body = bytes[20 + 25, ...]` = bytes[45...end]
+
+Result:
+```ruby
+[
+  { ref: [5, 0], body: "<< /Type /Annot /Subtype /Widget >>" },
+  { ref: [6, 0], body: "<< /Type /Annot /Subtype /Widget >>" },
+  { ref: [7, 0], body: "<< /Type /Annot /Subtype /Widget >>" }
+]
+```
+
+### Step 5: Object Retrieval
+
+When `object_body(ref)` is called for an object in an object stream:
+
+```ruby
+def object_body(ref)
+  case (e = @entries[ref])&.type
+  when :in_file
+    # Regular object: extract from file
+    extract_from_file(e.offset)
+  when :in_objstm
+    # Object stream: load stream (if not cached), then get object by index
+    load_objstm([e.objstm_num, 0])
+    @objstm_cache[[e.objstm_num, 0]][e.objstm_index][:body]
+  end
+end
+```
+
+The index (`objstm_index`) tells us which object in the parsed array to return.
+
+## Why Object Streams Matter
+
+### Benefits
+
+1. **File Size**: Compressing multiple objects together is more efficient than compressing each individually
+2. **Performance**: Fewer objects to parse when opening the PDF
+3. **Common in Modern PDFs**: Most PDFs created by modern tools use object streams
+
+### Transparency in AcroThat
+
+`AcroThat` handles object streams automatically:
+- You don't need to know if an object is in a stream or not
+- `object_body(ref)` returns the object body the same way regardless
+- Object streams are cached after first load (no repeated parsing)
+- The same `DictScan` methods work on extracted object bodies
+
+## Cross-Reference Streams vs Object Streams
+
+**Important distinction:**
+
+1. **XRef Streams** (`/Type /XRef`): Used to find where objects are located in the PDF
+   - Contains byte offsets or references to object streams
+   - Replaces classic xref tables
+
+2. **Object Streams** (`/Type /ObjStm`): Used to store actual object bodies
+   - Contains compressed object dictionaries
+   - Referenced by xref streams or classic xref tables
+
+Both use the same stream format (compressed, potentially with PNG predictor), but serve different purposes.
+
+## PNG Predictor
+
+PNG Predictor is a compression technique that predicts values based on previous values to improve compression. `AcroThat` supports all 5 PNG predictor types:
+
+1. **Type 0 (None)**: No prediction
+2. **Type 1 (Sub)**: Predict from left
+3. **Type 2 (Up)**: Predict from above
+4. **Type 3 (Average)**: Predict from average of left and above
+5. **Type 4 (Paeth)**: Predict using Paeth algorithm
+
+The `apply_png_predictor` method decodes predictor-encoded data row by row, using the `/Columns` parameter to determine row width.
+
+## Summary
+
+Object streams allow PDFs to store multiple objects in compressed streams. `AcroThat` handles them by:
+
+1. **Identifying** objects in streams via xref parsing
+2. **Lazy loading** stream containers when needed
+3. **Decoding** compressed stream data (zlib + PNG predictor)
+4. **Parsing** the header to extract object offsets
+5. **Extracting** individual object bodies by offset
+6. **Caching** parsed streams for performance
+
+The parsing itself is straightforward:
+- Header is space-separated integers (object numbers and offsets)
+- Object data follows the header
+- Extract each object body using its offset
+
+Just like `DictScan`, object stream parsing is **text traversal**—once the stream is decompressed, it's just parsing space-separated numbers and extracting substrings by offset.
+