mercury-agent 0.5.7 → 0.5.10

package/examples/extensions/pdf/skill/forms.md

@@ -1,294 +1,353 @@
-**CRITICAL: You MUST complete these steps in order. Do not skip ahead to writing code.**
+# PDF Form Filling Guide
 
-If you need to fill out a PDF form, first check to see if the PDF has fillable form fields. Run this script from this file's directory:
- `python scripts/check_fillable_fields <file.pdf>`, and depending on the result go to either the "Fillable fields" or "Non-fillable fields" and follow those instructions.
+This guide covers filling PDF forms, both fillable (AcroForm) and non-fillable PDFs.
 
-# Fillable fields
-If the PDF has fillable form fields:
-- Run this script from this file's directory: `python scripts/extract_form_field_info.py <input.pdf> <field_info.json>`. It will create a JSON file with a list of fields in this format:
+## Overview
+
+There are two categories of PDF forms:
+
+1. **Fillable PDFs** (AcroForm/XFA) -- contain interactive form fields. Users can click and type into fields. These are straightforward to fill programmatically.
+2. **Non-fillable PDFs** -- just text and lines rendered as page content, with no interactive fields. Filling these requires overlaying text at precise coordinates using annotations or drawing.
+
+## Workflow Summary
+
+```
+Start
+|
++- Run check_fillable_fields.py
+|  |
+|  +- Has fields -> Fillable path
+|  |  +- extract_form_field_info.py  -> get field names and types
+|  |  +- check_bounding_boxes.py     -> visualize field positions
+|  |  +- fill_fillable_fields.py     -> fill from JSON mapping
+|  |  +- create_validation_image.py  -> verify the result visually
+|  |
+|  +- No fields -> Non-fillable path
+|     +- convert_pdf_to_images.py    -> render pages for visual inspection
+|     +- extract_form_structure.py   -> detect text labels and guess field positions
+|     +- fill_pdf_form_with_annotations.py -> overlay text using annotations
+|     +- create_validation_image.py  -> verify the result visually
+```
+
+---
+
+## Part 1: Fillable PDF Forms
+
+### Step 1: Detect fillable fields
+
+```bash
+python3 scripts/check_fillable_fields.py input.pdf
 ```
+
+This reports whether the PDF has AcroForm fields, lists their names, types, and current values.
+
+### Step 2: Extract field metadata
+
+```bash
+python3 scripts/extract_form_field_info.py input.pdf -o outbox/fields.json
+```
+
+Produces a JSON file with all field details:
+
+```json
 [
   {
-    "field_id": (unique ID for the field),
-    "page": (page number, 1-based),
-    "rect": ([left, bottom, right, top] bounding box in PDF coordinates, y=0 is the bottom of the page),
-    "type": ("text", "checkbox", "radio_group", or "choice"),
-  },
-  // Checkboxes have "checked_value" and "unchecked_value" properties:
-  {
-    "field_id": (unique ID for the field),
-    "page": (page number, 1-based),
-    "type": "checkbox",
-    "checked_value": (Set the field to this value to check the checkbox),
-    "unchecked_value": (Set the field to this value to uncheck the checkbox),
-  },
-  // Radio groups have a "radio_options" list with the possible choices.
-  {
-    "field_id": (unique ID for the field),
-    "page": (page number, 1-based),
-    "type": "radio_group",
-    "radio_options": [
-      {
-        "value": (set the field to this value to select this radio option),
-        "rect": (bounding box for the radio button for this option)
-      },
-      // Other radio options
-    ]
-  },
-  // Multiple choice fields have a "choice_options" list with the possible choices:
-  {
-    "field_id": (unique ID for the field),
-    "page": (page number, 1-based),
-    "type": "choice",
-    "choice_options": [
-      {
-        "value": (set the field to this value to select this option),
-        "text": (display text of the option)
-      },
-      // Other choice options
-    ],
+    "name": "first_name",
+    "type": "/Tx",
+    "value": "",
+    "rect": [72.0, 700.0, 250.0, 720.0],
+    "page": 0,
+    "flags": 0,
+    "read_only": false,
+    "required": false,
+    "options": null
   }
 ]
 ```
-- Convert the PDF to PNGs (one image for each page) with this script (run from this file's directory):
-`python scripts/convert_pdf_to_images.py <file.pdf> <output_directory>`
-Then analyze the images to determine the purpose of each form field (make sure to convert the bounding box PDF coordinates to image coordinates).
-- Create a `field_values.json` file in this format with the values to be entered for each field:
+
+Field types:
+- `/Tx` -- text field
+- `/Btn` -- button (checkbox, radio)
+- `/Ch` -- choice (dropdown, list)
+- `/Sig` -- signature
+
+### Step 3: Visualize field positions
+
+```bash
+python3 scripts/check_bounding_boxes.py input.pdf -o outbox/fields_overlay.png
 ```
-[
-  {
-    "field_id": "last_name", // Must match the field_id from `extract_form_field_info.py`
-    "description": "The user's last name",
-    "page": 1, // Must match the "page" value in field_info.json
-    "value": "Simpson"
-  },
-  {
-    "field_id": "Checkbox12",
-    "description": "Checkbox to be checked if the user is 18 or over",
-    "page": 1,
-    "value": "/On" // If this is a checkbox, use its "checked_value" value to check it. If it's a radio button group, use one of the "value" values in "radio_options".
-  },
-  // more fields
-]
+
+Renders each page with colored rectangles showing where fields are, labeled with field names. Use this to verify fields map to the right visual locations before filling.
+
+### Step 4: Fill the form
+
+Create a JSON file mapping field names to values:
+
+```json
+{
+  "first_name": "Alice",
+  "last_name": "Smith",
+  "email": "alice@example.com",
+  "agree_terms": true,
+  "country": "Canada"
+}
 ```
-- Run the `fill_fillable_fields.py` script from this file's directory to create a filled-in PDF:
-`python scripts/fill_fillable_fields.py <input pdf> <field_values.json> <output pdf>`
-This script will verify that the field IDs and values you provide are valid; if it prints error messages, correct the appropriate fields and try again.
 
-# Non-fillable fields
-If the PDF doesn't have fillable form fields, you'll add text annotations. First try to extract coordinates from the PDF structure (more accurate), then fall back to visual estimation if needed.
+Then fill:
 
-## Step 1: Try Structure Extraction First
+```bash
+python3 scripts/fill_fillable_fields.py input.pdf values.json -o outbox/filled.pdf
+```
 
-Run this script to extract text labels, lines, and checkboxes with their exact PDF coordinates:
-`python scripts/extract_form_structure.py <input.pdf> form_structure.json`
+For checkboxes, use `true`/`false`. For radio buttons, use the export value string. For dropdowns, use the option text.
 
-This creates a JSON file containing:
-- **labels**: Every text element with exact coordinates (x0, top, x1, bottom in PDF points)
-- **lines**: Horizontal lines that define row boundaries
-- **checkboxes**: Small square rectangles that are checkboxes (with center coordinates)
-- **row_boundaries**: Row top/bottom positions calculated from horizontal lines
+### Step 5: Validate the result
 
-**Check the results**: If `form_structure.json` has meaningful labels (text elements that correspond to form fields), use **Approach A: Structure-Based Coordinates**. If the PDF is scanned/image-based and has few or no labels, use **Approach B: Visual Estimation**.
+```bash
+python3 scripts/create_validation_image.py outbox/filled.pdf -o outbox/validation.png
+```
 
----
+Renders the filled PDF to an image. Visually inspect it to confirm all values appear correctly.
 
-## Approach A: Structure-Based Coordinates (Preferred)
+### Inline code (without scripts)
 
-Use this when `extract_form_structure.py` found text labels in the PDF.
+```python
+from pypdf import PdfReader, PdfWriter
 
-### A.1: Analyze the Structure
+reader = PdfReader("form.pdf")
+writer = PdfWriter()
+writer.append(reader)
 
-Read form_structure.json and identify:
+# Fill fields on page 0
+writer.update_page_form_field_values(
+    writer.pages[0],
+    {
+        "first_name": "Alice",
+        "last_name": "Smith",
+        "date": "2025-01-15",
+    },
+    auto_regenerate=False,
+)
 
-1. **Label groups**: Adjacent text elements that form a single label (e.g., "Last" + "Name")
-2. **Row structure**: Labels with similar `top` values are in the same row
-3. **Field columns**: Entry areas start after label ends (x0 = label.x1 + gap)
-4. **Checkboxes**: Use the checkbox coordinates directly from the structure
+writer.write("outbox/filled.pdf")
+writer.close()
+```
 
-**Coordinate system**: PDF coordinates where y=0 is at TOP of page, y increases downward.
+**Important**: Always set `auto_regenerate=False`. The default `True` can corrupt field appearances in some PDFs.
 
-### A.2: Check for Missing Elements
+### Handling checkboxes
 
-The structure extraction may not detect all form elements. Common cases:
-- **Circular checkboxes**: Only square rectangles are detected as checkboxes
-- **Complex graphics**: Decorative elements or non-standard form controls
-- **Faded or light-colored elements**: May not be extracted
+Checkboxes use specific values. To check a box, you need its "on" value:
 
-If you see form fields in the PDF images that aren't in form_structure.json, you'll need to use **visual analysis** for those specific fields (see "Hybrid Approach" below).
+```python
+from pypdf import PdfReader
 
-### A.3: Create fields.json with PDF Coordinates
+reader = PdfReader("form.pdf")
+fields = reader.get_fields()
 
-For each field, calculate entry coordinates from the extracted structure:
+for name, field in fields.items():
+    if field.get("/FT") == "/Btn":
+        # The /AP dict contains the appearance states
+        widget = field.get("/Kids", [field])
+        print(f"Checkbox '{name}': possible values = {field.get('/V')}")
+```
 
-**Text fields:**
-- entry x0 = label x1 + 5 (small gap after label)
-- entry x1 = next label's x0, or row boundary
-- entry top = same as label top
-- entry bottom = row boundary line below, or label bottom + row_height
+Typically checkboxes accept `/Yes` to check and `/Off` to uncheck:
 
-**Checkboxes:**
-- Use the checkbox rectangle coordinates directly from form_structure.json
-- entry_bounding_box = [checkbox.x0, checkbox.top, checkbox.x1, checkbox.bottom]
+```python
+writer.update_page_form_field_values(
+    writer.pages[0],
+    {"agree_terms": "/Yes"},
+    auto_regenerate=False,
+)
+```
 
-Create fields.json using `pdf_width` and `pdf_height` (signals PDF coordinates):
-```json
-{
-  "pages": [
-    {"page_number": 1, "pdf_width": 612, "pdf_height": 792}
-  ],
-  "form_fields": [
-    {
-      "page_number": 1,
-      "description": "Last name entry field",
-      "field_label": "Last Name",
-      "label_bounding_box": [43, 63, 87, 73],
-      "entry_bounding_box": [92, 63, 260, 79],
-      "entry_text": {"text": "Smith", "font_size": 10}
-    },
-    {
-      "page_number": 1,
-      "description": "US Citizen Yes checkbox",
-      "field_label": "Yes",
-      "label_bounding_box": [260, 200, 280, 210],
-      "entry_bounding_box": [285, 197, 292, 205],
-      "entry_text": {"text": "X"}
-    }
-  ]
-}
+### Flattening forms
+
+After filling, you may want to flatten the form so fields become static text (non-editable):
+
+```bash
+# Using qpdf (recommended)
+qpdf --flatten-annotations=all filled.pdf flattened.pdf
 ```
 
-**Important**: Use `pdf_width`/`pdf_height` and coordinates directly from form_structure.json.
+### Common pitfalls with fillable forms
+
+1. **Field names with dots**: Some PDFs use hierarchical names like `form1.page1.FirstName`. Always use the FULL qualified name from the extraction output.
 
-### A.4: Validate Bounding Boxes
+2. **Checkbox export values**: Checkboxes have an "export value" (often "Yes", "1", or "On"). Setting `true` maps to `/Yes`. Check the `extract_form_field_info.py` output to see the actual export value.
 
-Before filling, check your bounding boxes for errors:
-`python scripts/check_bounding_boxes.py fields.json`
+3. **Read-only fields**: Fields with the read-only flag (bit 1 of /Ff) cannot be filled. The script will warn about these.
 
-This checks for intersecting bounding boxes and entry boxes that are too small for the font size. Fix any reported errors before filling.
+4. **Appearance streams**: Some PDF viewers won't display values without proper appearance streams (/AP). If the filled PDF looks empty in a viewer, try flattening with qpdf.
+
+5. **Font embedding**: If the original form uses a custom font, filled values may render in a fallback font. For non-ASCII text, consider the reportlab overlay approach where you control the font.
 
 ---
 
-## Approach B: Visual Estimation (Fallback)
+## Part 2: Non-Fillable PDF Forms
+
+Non-fillable PDFs have no interactive fields. To "fill" them, you overlay text at specific coordinates using PDF annotations or by drawing on a separate layer and merging it.
+
+### Strategy
+
+1. Render the PDF to an image to see exactly where fields should be filled.
+2. Identify the coordinates of blank areas where values need to go.
+3. Use a reportlab overlay (default method, best compatibility) or FreeText annotations.
+
+### Step 1: Render to image for inspection
+
+```bash
+python3 scripts/convert_pdf_to_images.py input.pdf -o outbox/ --dpi 200
+```
+
+Open the resulting images to visually identify where to place text.
 
-Use this when the PDF is scanned/image-based and structure extraction found no usable text labels (e.g., all text shows as "(cid:X)" patterns).
+### Step 2: Extract form structure (heuristic)
 
-### B.1: Convert PDF to Images
+```bash
+python3 scripts/extract_form_structure.py input.pdf -o outbox/structure.json
+```
+
+This script uses pdfplumber to detect text labels and underlines/boxes, then heuristically guesses where fillable regions might be. The output is approximate -- always verify with the rendered image.
 
-`python scripts/convert_pdf_to_images.py <input.pdf> <images_dir/>`
+The output includes both pdfplumber coordinates (top-left origin) and PDF coordinates (bottom-left origin) for each detected region.
 
-### B.2: Initial Field Identification
+### Step 3: Define fill positions
 
-Examine each page image to identify form sections and get **rough estimates** of field locations:
-- Form field labels and their approximate positions
-- Entry areas (lines, boxes, or blank spaces for text input)
-- Checkboxes and their approximate locations
+Create a JSON file specifying what to write and where:
 
-For each field, note approximate pixel coordinates (they don't need to be precise yet).
+```json
+[
+  {
+    "page": 0,
+    "fields": [
+      {
+        "x": 200,
+        "y": 705,
+        "value": "Alice Smith",
+        "font_size": 11
+      },
+      {
+        "x": 200,
+        "y": 680,
+        "value": "alice@example.com",
+        "font_size": 11
+      }
+    ]
+  }
+]
+```
 
-### B.3: Zoom Refinement (CRITICAL for accuracy)
+Coordinates are in PDF points (1 point = 1/72 inch). Origin is bottom-left of the page.
 
-For each field, crop a region around the estimated position to refine coordinates precisely.
+### Step 4: Fill using overlay (default method)
 
-**Create a zoomed crop using ImageMagick:**
 ```bash
-magick <page_image> -crop <width>x<height>+<x>+<y> +repage <crop_output.png>
+python3 scripts/fill_pdf_form_with_annotations.py input.pdf fill_spec.json -o outbox/filled.pdf
 ```
 
-Where:
-- `<x>, <y>` = top-left corner of crop region (use your rough estimate minus padding)
-- `<width>, <height>` = size of crop region (field area plus ~50px padding on each side)
+This creates a transparent overlay with reportlab and merges it onto the original PDF. The original content is preserved underneath.
+
+To use FreeText annotations instead (original content completely untouched):
 
-**Example:** To refine a "Name" field estimated around (100, 150):
 ```bash
-magick images_dir/page_1.png -crop 300x80+50+120 +repage crops/name_field.png
+python3 scripts/fill_pdf_form_with_annotations.py input.pdf fill_spec.json -o outbox/filled.pdf --method annot
 ```
 
-(Note: if the `magick` command isn't available, try `convert` with the same arguments).
+### Alternative: manual overlay with reportlab
 
-**Examine the cropped image** to determine precise coordinates:
-1. Identify the exact pixel where the entry area begins (after the label)
-2. Identify where the entry area ends (before next field or edge)
-3. Identify the top and bottom of the entry line/box
+For full control over the overlay:
 
-**Convert crop coordinates back to full image coordinates:**
-- full_x = crop_x + crop_offset_x
-- full_y = crop_y + crop_offset_y
+```python
+import io
+from pypdf import PdfReader, PdfWriter
+from reportlab.pdfgen import canvas
+from reportlab.lib.pagesizes import letter
 
-Example: If the crop started at (50, 120) and the entry box starts at (52, 18) within the crop:
-- entry_x0 = 52 + 50 = 102
-- entry_top = 18 + 120 = 138
+# Create overlay
+packet = io.BytesIO()
+c = canvas.Canvas(packet, pagesize=letter)
+c.setFont("Helvetica", 11)
+c.drawString(200, 705, "Alice Smith")
+c.drawString(200, 680, "alice@example.com")
+c.save()
+packet.seek(0)
 
-**Repeat for each field**, grouping nearby fields into single crops when possible.
+# Merge overlay onto original
+original = PdfReader("input.pdf")
+overlay = PdfReader(packet)
 
-### B.4: Create fields.json with Refined Coordinates
+writer = PdfWriter()
+page = original.pages[0]
+page.merge_page(overlay.pages[0])
+writer.add_page(page)
 
-Create fields.json using `image_width` and `image_height` (signals image coordinates):
-```json
-{
-  "pages": [
-    {"page_number": 1, "image_width": 1700, "image_height": 2200}
-  ],
-  "form_fields": [
-    {
-      "page_number": 1,
-      "description": "Last name entry field",
-      "field_label": "Last Name",
-      "label_bounding_box": [120, 175, 242, 198],
-      "entry_bounding_box": [255, 175, 720, 218],
-      "entry_text": {"text": "Smith", "font_size": 10}
-    }
-  ]
-}
+# Copy remaining pages unchanged
+for p in original.pages[1:]:
+    writer.add_page(p)
+
+writer.write("outbox/filled.pdf")
+writer.close()
+```
+
+### Step 5: Validate
+
+```bash
+python3 scripts/create_validation_image.py outbox/filled.pdf -o outbox/validation.png
 ```
 
-**Important**: Use `image_width`/`image_height` and the refined pixel coordinates from the zoom analysis.
+Always visually inspect the result. Coordinates often need fine-tuning.
 
-### B.5: Validate Bounding Boxes
+### Common pitfalls with non-fillable forms
 
-Before filling, check your bounding boxes for errors:
-`python scripts/check_bounding_boxes.py fields.json`
+1. **Coordinate mismatch**: The most common error. PDF Y-axis is bottom-up (origin at bottom-left), but screen/image Y-axis is top-down. Always convert:
+   ```
+   pdf_x = pixel_x * 72 / dpi
+   pdf_y = page_height_points - (pixel_y * 72 / dpi)
+   ```
 
-This checks for intersecting bounding boxes and entry boxes that are too small for the font size. Fix any reported errors before filling.
+2. **Page rotation**: If a page is rotated (e.g., 90 degrees), the coordinate system rotates too. Check `page.mediabox` and `page.rotation` and adjust.
 
----
+3. **Different page sizes**: Not all pages in a PDF have the same dimensions. Always check each page's mediabox.
 
-## Hybrid Approach: Structure + Visual
+4. **Font metrics**: Characters have different widths. If you need to center text in a box, use `stringWidth()` from ReportLab:
+   ```python
+   from reportlab.pdfbase.pdfmetrics import stringWidth
+   w = stringWidth("Jane Doe", "Helvetica", 10)
+   x = box_x + (box_width - w) / 2  # centered x position
+   ```
 
-Use this when structure extraction works for most fields but misses some elements (e.g., circular checkboxes, unusual form controls).
+5. **Overlapping with existing content**: If the form has lines or boxes, your text might overlap. Adjust the Y coordinate by 2-3 points upward from the line.
 
-1. **Use Approach A** for fields that were detected in form_structure.json
-2. **Convert PDF to images** for visual analysis of missing fields
-3. **Use zoom refinement** (from Approach B) for the missing fields
-4. **Combine coordinates**: For fields from structure extraction, use `pdf_width`/`pdf_height`. For visually-estimated fields, you must convert image coordinates to PDF coordinates:
-   - pdf_x = image_x * (pdf_width / image_width)
-   - pdf_y = image_y * (pdf_height / image_height)
-5. **Use a single coordinate system** in fields.json - convert all to PDF coordinates with `pdf_width`/`pdf_height`
+6. **Checkbox simulation**: For non-fillable checkboxes, overlay "X" or a checkmark character at the checkbox coordinates.
 
 ---
 
-## Step 2: Validate Before Filling
+## Troubleshooting
 
-**Always validate bounding boxes before filling:**
-`python scripts/check_bounding_boxes.py fields.json`
+### Field values not visible after filling
 
-This checks for:
-- Intersecting bounding boxes (which would cause overlapping text)
-- Entry boxes that are too small for the specified font size
+- Set `auto_regenerate=False` when calling `update_page_form_field_values`.
+- Some PDF viewers cache field appearances. Try opening in a different viewer.
+- Flatten the form with qpdf to force rendering: `qpdf --flatten-annotations=all filled.pdf output.pdf`
 
-Fix any reported errors in fields.json before proceeding.
+### Wrong font or encoding in filled fields
 
-## Step 3: Fill the Form
+- pypdf uses the font already defined in the field. If the original font does not support your characters, the text may appear garbled.
+- For non-ASCII text, consider the reportlab overlay approach where you control the font.
 
-The fill script auto-detects the coordinate system and handles conversion:
-`python scripts/fill_pdf_form_with_annotations.py <input.pdf> fields.json <output.pdf>`
+### Coordinates are wrong for non-fillable forms
 
-## Step 4: Verify Output
+- PDF coordinate origin is bottom-left, not top-left.
+- To convert from pixel coordinates (in a rendered image) to PDF points:
+  ```
+  pdf_x = pixel_x * 72 / dpi
+  pdf_y = page_height_points - (pixel_y * 72 / dpi)
+  ```
+- Use `check_bounding_boxes.py` or render the page to an image and measure pixel positions.
 
-Convert the filled PDF to images and verify text placement:
-`python scripts/convert_pdf_to_images.py <output.pdf> <verify_images/>`
+### Large PDFs are slow
 
-If text is mispositioned:
-- **Approach A**: Check that you're using PDF coordinates from form_structure.json with `pdf_width`/`pdf_height`
-- **Approach B**: Check that image dimensions match and coordinates are accurate pixels
-- **Hybrid**: Ensure coordinate conversions are correct for visually-estimated fields
+- Use pypdfium2 instead of pdf2image for rendering -- it is significantly faster.
+- Process only the pages you need rather than the entire document.