@caoscompanybr/merlin 3.5.0

package/.merlin-core/skills/general/pdf/SKILL.md

@@ -0,0 +1,298 @@
+---
+name: pdf
+description: "Use this skill whenever the user wants to do anything with PDF files. This includes reading or extracting text/tables from PDFs, combining or merging multiple PDFs into one, splitting PDFs apart, rotating pages, adding watermarks, creating new PDFs, filling PDF forms, encrypting/decrypting PDFs, extracting images, and OCR on scanned PDFs to make them searchable. If the user mentions a .pdf file or asks to produce one, use this skill."
+license: MIT (Anthropic)
+compatibility: claude-code
+metadata:
+  source: anthropic/skills
+  imported: 2026-03-15
+  version: "3.0.0"
+  windows-setup: See references/windows-setup.md for Windows dependency installation
+  auto_activate: [pdf, pdf-generate, pdf-create, pdf-merge, pdf-extract]
+  file_triggers: [*.pdf]
+---
+
+# PDF Processing Guide
+
+## Overview
+
+This guide covers essential PDF processing operations using Python libraries and command-line tools. For advanced features, JavaScript libraries, and detailed examples, see `references/reference.md`. If you need to fill out a PDF form, read `references/forms.md` and follow its instructions.
+
+## Dependencies
+
+Some tools require system-level installation. See `references/windows-setup.md` for Windows setup instructions.
+
+| Tool        | Purpose                                          | Install                  |
+| ----------- | ------------------------------------------------ | ------------------------ |
+| pypdf       | Basic operations (merge, split, rotate, encrypt) | `pip install pypdf`      |
+| pdfplumber  | Text and table extraction                        | `pip install pdfplumber` |
+| reportlab   | Create PDFs from scratch                         | `pip install reportlab`  |
+| pypdfium2   | Fast rendering, image generation (PyMuPDF alt)   | `pip install pypdfium2`  |
+| pdftotext   | CLI text extraction                              | Requires poppler-utils   |
+| qpdf        | CLI merge/split/decrypt/optimize                 | System package           |
+| pytesseract | OCR for scanned PDFs                             | Requires Tesseract OCR   |
+
+## Quick Start
+
+```python
+from pypdf import PdfReader, PdfWriter
+
+# Read a PDF
+reader = PdfReader("document.pdf")
+print(f"Pages: {len(reader.pages)}")
+
+# Extract text
+text = ""
+for page in reader.pages:
+    text += page.extract_text()
+```
+
+## Python Libraries
+
+### pypdf — Basic Operations
+
+```python
+# Merge PDFs
+writer = PdfWriter()
+for pdf_file in ["doc1.pdf", "doc2.pdf", "doc3.pdf"]:
+    reader = PdfReader(pdf_file)
+    for page in reader.pages:
+        writer.add_page(page)
+with open("merged.pdf", "wb") as output:
+    writer.write(output)
+
+# Split PDF
+reader = PdfReader("input.pdf")
+for i, page in enumerate(reader.pages):
+    writer = PdfWriter()
+    writer.add_page(page)
+    with open(f"page_{i+1}.pdf", "wb") as output:
+        writer.write(output)
+
+# Extract Metadata
+reader = PdfReader("document.pdf")
+meta = reader.metadata
+print(f"Title: {meta.title}")
+print(f"Author: {meta.author}")
+
+# Rotate Pages
+reader = PdfReader("input.pdf")
+writer = PdfWriter()
+page = reader.pages[0]
+page.rotate(90)  # 90 degrees clockwise
+writer.add_page(page)
+with open("rotated.pdf", "wb") as output:
+    writer.write(output)
+
+# Password Protection
+writer.encrypt("userpassword", "ownerpassword")
+```
+
+### pdfplumber — Text and Table Extraction
+
+```python
+import pdfplumber
+
+with pdfplumber.open("document.pdf") as pdf:
+    for page in pdf.pages:
+        text = page.extract_text()
+        tables = page.extract_tables()
+```
+
+#### Advanced Table Extraction
+
+```python
+import pandas as pd
+import pdfplumber
+
+with pdfplumber.open("document.pdf") as pdf:
+    all_tables = []
+    for page in pdf.pages:
+        tables = page.extract_tables()
+        for table in tables:
+            if table:
+                df = pd.DataFrame(table[1:], columns=table[0])
+                all_tables.append(df)
+    if all_tables:
+        combined_df = pd.concat(all_tables, ignore_index=True)
+        combined_df.to_excel("extracted_tables.xlsx", index=False)
+```
+
+### reportlab — Create PDFs
+
+```python
+from reportlab.lib.pagesizes import letter
+from reportlab.pdfgen import canvas
+
+c = canvas.Canvas("hello.pdf", pagesize=letter)
+width, height = letter
+c.drawString(100, height - 100, "Hello World!")
+c.save()
+```
+
+#### Multi-Page Reports with Tables
+
+```python
+from reportlab.lib.pagesizes import letter
+from reportlab.platypus import SimpleDocTemplate, Paragraph, Spacer, PageBreak, Table, TableStyle
+from reportlab.lib.styles import getSampleStyleSheet
+from reportlab.lib import colors
+
+doc = SimpleDocTemplate("report.pdf", pagesize=letter)
+styles = getSampleStyleSheet()
+story = []
+
+# Title
+story.append(Paragraph("Report Title", styles['Title']))
+story.append(Spacer(1, 12))
+
+# Body text
+story.append(Paragraph("This is the body of the report. " * 20, styles['Normal']))
+
+# Table with styling
+data = [['Product', 'Q1', 'Q2', 'Q3'],
+        ['Widgets', '120', '135', '142'],
+        ['Gadgets', '85', '92', '98']]
+table = Table(data)
+table.setStyle(TableStyle([
+    ('BACKGROUND', (0, 0), (-1, 0), colors.grey),
+    ('TEXTCOLOR', (0, 0), (-1, 0), colors.whitesmoke),
+    ('ALIGN', (0, 0), (-1, -1), 'CENTER'),
+    ('GRID', (0, 0), (-1, -1), 1, colors.black),
+]))
+story.append(table)
+
+doc.build(story)
+```
+
+**IMPORTANT**: Never use Unicode subscript/superscript characters (₀₁₂₃, ⁰¹²³) in ReportLab PDFs. The built-in fonts do not include these glyphs, causing them to render as solid black boxes. Use `<sub>` and `<super>` tags in Paragraph objects instead:
+
+```python
+from reportlab.platypus import Paragraph
+from reportlab.lib.styles import getSampleStyleSheet
+styles = getSampleStyleSheet()
+
+chemical = Paragraph("H<sub>2</sub>O", styles['Normal'])
+squared = Paragraph("x<super>2</super> + y<super>2</super>", styles['Normal'])
+```
+
+## Command-Line Tools
+
+```bash
+# pdftotext (poppler-utils)
+pdftotext input.pdf output.txt           # Basic extraction
+pdftotext -layout input.pdf output.txt   # Preserve layout
+pdftotext -f 1 -l 5 input.pdf output.txt # Pages 1-5
+
+# qpdf
+qpdf --empty --pages file1.pdf file2.pdf -- merged.pdf      # Merge
+qpdf input.pdf --pages . 1-5 -- pages1-5.pdf                # Extract pages
+qpdf input.pdf output.pdf --rotate=+90:1                     # Rotate page 1
+qpdf --password=pass --decrypt encrypted.pdf decrypted.pdf   # Decrypt
+qpdf --linearize input.pdf optimized.pdf                     # Optimize for web
+
+# pdftk (if available)
+pdftk file1.pdf file2.pdf cat output merged.pdf
+
+# Extract images (poppler-utils)
+pdfimages -j input.pdf output_prefix
+```
+
+## Common Tasks
+
+### OCR Scanned PDFs
+
+```python
+import pytesseract
+from pdf2image import convert_from_path
+
+images = convert_from_path('scanned.pdf')
+text = ""
+for i, image in enumerate(images):
+    text += f"Page {i+1}:\n"
+    text += pytesseract.image_to_string(image)
+    text += "\n\n"
+```
+
+### Add Watermark
+
+```python
+from pypdf import PdfReader, PdfWriter
+
+watermark = PdfReader("watermark.pdf").pages[0]
+reader = PdfReader("document.pdf")
+writer = PdfWriter()
+for page in reader.pages:
+    page.merge_page(watermark)
+    writer.add_page(page)
+with open("watermarked.pdf", "wb") as output:
+    writer.write(output)
+```
+
+### Extract Images
+
+```bash
+# Using pdfimages (poppler-utils) — fastest method
+pdfimages -j input.pdf output_prefix
+# Extracts as output_prefix-000.jpg, output_prefix-001.jpg, etc.
+
+# List image info without extracting
+pdfimages -list input.pdf
+```
+
+### Password Protection
+
+```python
+from pypdf import PdfReader, PdfWriter
+
+reader = PdfReader("input.pdf")
+writer = PdfWriter()
+for page in reader.pages:
+    writer.add_page(page)
+writer.encrypt("userpassword", "ownerpassword")
+with open("encrypted.pdf", "wb") as output:
+    writer.write(output)
+```
+
+## PDF Form Filling
+
+For filling out PDF forms, read `references/forms.md` and follow its step-by-step instructions. The workflow:
+
+1. **Check for fillable fields** — `python scripts/check_fillable_fields.py <file.pdf>`
+2. **If fillable** — extract field info, analyze with images, create values JSON, fill with script
+3. **If not fillable** — extract structure or use visual estimation, create field coordinates, fill with annotations
+
+### Bundled Scripts
+
+| Script                                      | Purpose                                                 |
+| ------------------------------------------- | ------------------------------------------------------- |
+| `scripts/check_fillable_fields.py`          | Detect if PDF has fillable form fields                  |
+| `scripts/extract_form_field_info.py`        | Extract field IDs, types, and positions                 |
+| `scripts/extract_form_structure.py`         | Extract text labels, lines, checkboxes with coordinates |
+| `scripts/convert_pdf_to_images.py`          | Convert PDF pages to PNG images                         |
+| `scripts/check_bounding_boxes.py`           | Validate field coordinates before filling               |
+| `scripts/fill_fillable_fields.py`           | Fill PDFs with native fillable fields                   |
+| `scripts/fill_pdf_form_with_annotations.py` | Fill non-fillable PDFs with text annotations            |
+| `scripts/create_validation_image.py`        | Create validation overlay image                         |
+
+## Quick Reference
+
+| Task             | Best Tool               | Details                       |
+| ---------------- | ----------------------- | ----------------------------- |
+| Merge PDFs       | pypdf                   | `writer.add_page(page)`       |
+| Split PDFs       | pypdf                   | One page per file             |
+| Extract text     | pdfplumber              | `page.extract_text()`         |
+| Extract tables   | pdfplumber              | `page.extract_tables()`       |
+| Create PDFs      | reportlab               | Canvas or Platypus            |
+| CLI merge        | qpdf                    | `qpdf --empty --pages ...`    |
+| OCR scanned      | pytesseract + pdf2image | Convert to image first        |
+| Fill PDF forms   | bundled scripts         | See `references/forms.md`     |
+| Extract images   | pdfimages (poppler)     | `pdfimages -j input.pdf`      |
+| Render to images | pypdfium2               | See `references/reference.md` |
+| JS PDF creation  | pdf-lib                 | See `references/reference.md` |
+
+## Next Steps
+
+- For advanced pypdfium2, pdf-lib (JS), pdfjs-dist, and performance tips, see `references/reference.md`
+- For PDF form filling workflow, see `references/forms.md`
+- For Windows dependency setup, see `references/windows-setup.md`

package/.merlin-core/skills/general/pdf/references/forms.md

@@ -0,0 +1,312 @@
+**CRITICAL: You MUST complete these steps in order. Do not skip ahead to writing code.**
+
+If you need to fill out a PDF form, first check to see if the PDF has fillable form fields. Run this script from this file's directory:
+`python scripts/check_fillable_fields <file.pdf>`, and depending on the result go to either the "Fillable fields" or "Non-fillable fields" and follow those instructions.
+
+# Fillable fields
+
+If the PDF has fillable form fields:
+
+- Run this script from this file's directory: `python scripts/extract_form_field_info.py <input.pdf> <field_info.json>`. It will create a JSON file with a list of fields in this format:
+
+```
+[
+  {
+    "field_id": (unique ID for the field),
+    "page": (page number, 1-based),
+    "rect": ([left, bottom, right, top] bounding box in PDF coordinates, y=0 is the bottom of the page),
+    "type": ("text", "checkbox", "radio_group", or "choice"),
+  },
+  // Checkboxes have "checked_value" and "unchecked_value" properties:
+  {
+    "field_id": (unique ID for the field),
+    "page": (page number, 1-based),
+    "type": "checkbox",
+    "checked_value": (Set the field to this value to check the checkbox),
+    "unchecked_value": (Set the field to this value to uncheck the checkbox),
+  },
+  // Radio groups have a "radio_options" list with the possible choices.
+  {
+    "field_id": (unique ID for the field),
+    "page": (page number, 1-based),
+    "type": "radio_group",
+    "radio_options": [
+      {
+        "value": (set the field to this value to select this radio option),
+        "rect": (bounding box for the radio button for this option)
+      },
+      // Other radio options
+    ]
+  },
+  // Multiple choice fields have a "choice_options" list with the possible choices:
+  {
+    "field_id": (unique ID for the field),
+    "page": (page number, 1-based),
+    "type": "choice",
+    "choice_options": [
+      {
+        "value": (set the field to this value to select this option),
+        "text": (display text of the option)
+      },
+      // Other choice options
+    ],
+  }
+]
+```
+
+- Convert the PDF to PNGs (one image for each page) with this script (run from this file's directory):
+  `python scripts/convert_pdf_to_images.py <file.pdf> <output_directory>`
+  Then analyze the images to determine the purpose of each form field (make sure to convert the bounding box PDF coordinates to image coordinates).
+- Create a `field_values.json` file in this format with the values to be entered for each field:
+
+```
+[
+  {
+    "field_id": "last_name", // Must match the field_id from `extract_form_field_info.py`
+    "description": "The user's last name",
+    "page": 1, // Must match the "page" value in field_info.json
+    "value": "Simpson"
+  },
+  {
+    "field_id": "Checkbox12",
+    "description": "Checkbox to be checked if the user is 18 or over",
+    "page": 1,
+    "value": "/On" // If this is a checkbox, use its "checked_value" value to check it. If it's a radio button group, use one of the "value" values in "radio_options".
+  },
+  // more fields
+]
+```
+
+- Run the `fill_fillable_fields.py` script from this file's directory to create a filled-in PDF:
+  `python scripts/fill_fillable_fields.py <input pdf> <field_values.json> <output pdf>`
+  This script will verify that the field IDs and values you provide are valid; if it prints error messages, correct the appropriate fields and try again.
+
+# Non-fillable fields
+
+If the PDF doesn't have fillable form fields, you'll add text annotations. First try to extract coordinates from the PDF structure (more accurate), then fall back to visual estimation if needed.
+
+## Step 1: Try Structure Extraction First
+
+Run this script to extract text labels, lines, and checkboxes with their exact PDF coordinates:
+`python scripts/extract_form_structure.py <input.pdf> form_structure.json`
+
+This creates a JSON file containing:
+
+- **labels**: Every text element with exact coordinates (x0, top, x1, bottom in PDF points)
+- **lines**: Horizontal lines that define row boundaries
+- **checkboxes**: Small square rectangles that are checkboxes (with center coordinates)
+- **row_boundaries**: Row top/bottom positions calculated from horizontal lines
+
+**Check the results**: If `form_structure.json` has meaningful labels (text elements that correspond to form fields), use **Approach A: Structure-Based Coordinates**. If the PDF is scanned/image-based and has few or no labels, use **Approach B: Visual Estimation**.
+
+---
+
+## Approach A: Structure-Based Coordinates (Preferred)
+
+Use this when `extract_form_structure.py` found text labels in the PDF.
+
+### A.1: Analyze the Structure
+
+Read form_structure.json and identify:
+
+1. **Label groups**: Adjacent text elements that form a single label (e.g., "Last" + "Name")
+2. **Row structure**: Labels with similar `top` values are in the same row
+3. **Field columns**: Entry areas start after label ends (x0 = label.x1 + gap)
+4. **Checkboxes**: Use the checkbox coordinates directly from the structure
+
+**Coordinate system**: PDF coordinates where y=0 is at TOP of page, y increases downward.
+
+### A.2: Check for Missing Elements
+
+The structure extraction may not detect all form elements. Common cases:
+
+- **Circular checkboxes**: Only square rectangles are detected as checkboxes
+- **Complex graphics**: Decorative elements or non-standard form controls
+- **Faded or light-colored elements**: May not be extracted
+
+If you see form fields in the PDF images that aren't in form_structure.json, you'll need to use **visual analysis** for those specific fields (see "Hybrid Approach" below).
+
+### A.3: Create fields.json with PDF Coordinates
+
+For each field, calculate entry coordinates from the extracted structure:
+
+**Text fields:**
+
+- entry x0 = label x1 + 5 (small gap after label)
+- entry x1 = next label's x0, or row boundary
+- entry top = same as label top
+- entry bottom = row boundary line below, or label bottom + row_height
+
+**Checkboxes:**
+
+- Use the checkbox rectangle coordinates directly from form_structure.json
+- entry_bounding_box = [checkbox.x0, checkbox.top, checkbox.x1, checkbox.bottom]
+
+Create fields.json using `pdf_width` and `pdf_height` (signals PDF coordinates):
+
+```json
+{
+  "pages": [{ "page_number": 1, "pdf_width": 612, "pdf_height": 792 }],
+  "form_fields": [
+    {
+      "page_number": 1,
+      "description": "Last name entry field",
+      "field_label": "Last Name",
+      "label_bounding_box": [43, 63, 87, 73],
+      "entry_bounding_box": [92, 63, 260, 79],
+      "entry_text": { "text": "Smith", "font_size": 10 }
+    },
+    {
+      "page_number": 1,
+      "description": "US Citizen Yes checkbox",
+      "field_label": "Yes",
+      "label_bounding_box": [260, 200, 280, 210],
+      "entry_bounding_box": [285, 197, 292, 205],
+      "entry_text": { "text": "X" }
+    }
+  ]
+}
+```
+
+**Important**: Use `pdf_width`/`pdf_height` and coordinates directly from form_structure.json.
+
+### A.4: Validate Bounding Boxes
+
+Before filling, check your bounding boxes for errors:
+`python scripts/check_bounding_boxes.py fields.json`
+
+This checks for intersecting bounding boxes and entry boxes that are too small for the font size. Fix any reported errors before filling.
+
+---
+
+## Approach B: Visual Estimation (Fallback)
+
+Use this when the PDF is scanned/image-based and structure extraction found no usable text labels (e.g., all text shows as "(cid:X)" patterns).
+
+### B.1: Convert PDF to Images
+
+`python scripts/convert_pdf_to_images.py <input.pdf> <images_dir/>`
+
+### B.2: Initial Field Identification
+
+Examine each page image to identify form sections and get **rough estimates** of field locations:
+
+- Form field labels and their approximate positions
+- Entry areas (lines, boxes, or blank spaces for text input)
+- Checkboxes and their approximate locations
+
+For each field, note approximate pixel coordinates (they don't need to be precise yet).
+
+### B.3: Zoom Refinement (CRITICAL for accuracy)
+
+For each field, crop a region around the estimated position to refine coordinates precisely.
+
+**Create a zoomed crop using ImageMagick:**
+
+```bash
+magick <page_image> -crop <width>x<height>+<x>+<y> +repage <crop_output.png>
+```
+
+Where:
+
+- `<x>, <y>` = top-left corner of crop region (use your rough estimate minus padding)
+- `<width>, <height>` = size of crop region (field area plus ~50px padding on each side)
+
+**Example:** To refine a "Name" field estimated around (100, 150):
+
+```bash
+magick images_dir/page_1.png -crop 300x80+50+120 +repage crops/name_field.png
+```
+
+(Note: if the `magick` command isn't available, try `convert` with the same arguments).
+
+**Examine the cropped image** to determine precise coordinates:
+
+1. Identify the exact pixel where the entry area begins (after the label)
+2. Identify where the entry area ends (before next field or edge)
+3. Identify the top and bottom of the entry line/box
+
+**Convert crop coordinates back to full image coordinates:**
+
+- full_x = crop_x + crop_offset_x
+- full_y = crop_y + crop_offset_y
+
+Example: If the crop started at (50, 120) and the entry box starts at (52, 18) within the crop:
+
+- entry_x0 = 52 + 50 = 102
+- entry_top = 18 + 120 = 138
+
+**Repeat for each field**, grouping nearby fields into single crops when possible.
+
+### B.4: Create fields.json with Refined Coordinates
+
+Create fields.json using `image_width` and `image_height` (signals image coordinates):
+
+```json
+{
+  "pages": [{ "page_number": 1, "image_width": 1700, "image_height": 2200 }],
+  "form_fields": [
+    {
+      "page_number": 1,
+      "description": "Last name entry field",
+      "field_label": "Last Name",
+      "label_bounding_box": [120, 175, 242, 198],
+      "entry_bounding_box": [255, 175, 720, 218],
+      "entry_text": { "text": "Smith", "font_size": 10 }
+    }
+  ]
+}
+```
+
+**Important**: Use `image_width`/`image_height` and the refined pixel coordinates from the zoom analysis.
+
+### B.5: Validate Bounding Boxes
+
+Before filling, check your bounding boxes for errors:
+`python scripts/check_bounding_boxes.py fields.json`
+
+This checks for intersecting bounding boxes and entry boxes that are too small for the font size. Fix any reported errors before filling.
+
+---
+
+## Hybrid Approach: Structure + Visual
+
+Use this when structure extraction works for most fields but misses some elements (e.g., circular checkboxes, unusual form controls).
+
+1. **Use Approach A** for fields that were detected in form_structure.json
+2. **Convert PDF to images** for visual analysis of missing fields
+3. **Use zoom refinement** (from Approach B) for the missing fields
+4. **Combine coordinates**: For fields from structure extraction, use `pdf_width`/`pdf_height`. For visually-estimated fields, you must convert image coordinates to PDF coordinates:
+   - pdf_x = image_x \* (pdf_width / image_width)
+   - pdf_y = image_y \* (pdf_height / image_height)
+5. **Use a single coordinate system** in fields.json - convert all to PDF coordinates with `pdf_width`/`pdf_height`
+
+---
+
+## Step 2: Validate Before Filling
+
+**Always validate bounding boxes before filling:**
+`python scripts/check_bounding_boxes.py fields.json`
+
+This checks for:
+
+- Intersecting bounding boxes (which would cause overlapping text)
+- Entry boxes that are too small for the specified font size
+
+Fix any reported errors in fields.json before proceeding.
+
+## Step 3: Fill the Form
+
+The fill script auto-detects the coordinate system and handles conversion:
+`python scripts/fill_pdf_form_with_annotations.py <input.pdf> fields.json <output.pdf>`
+
+## Step 4: Verify Output
+
+Convert the filled PDF to images and verify text placement:
+`python scripts/convert_pdf_to_images.py <output.pdf> <verify_images/>`
+
+If text is mispositioned:
+
+- **Approach A**: Check that you're using PDF coordinates from form_structure.json with `pdf_width`/`pdf_height`
+- **Approach B**: Check that image dimensions match and coordinates are accurate pixels
+- **Hybrid**: Ensure coordinate conversions are correct for visually-estimated fields