@bastani/atomic 0.6.8 → 0.7.0-1

package/.agents/skills/pdf/SKILL.md

@@ -1,316 +0,0 @@
----
-name: pdf
-description: Use this skill whenever the user wants to do anything with PDF files. This includes reading or extracting text/tables from PDFs, combining or merging multiple PDFs into one, splitting PDFs apart, rotating pages, adding watermarks, creating new PDFs, filling PDF forms, encrypting/decrypting PDFs, extracting images, and OCR on scanned PDFs to make them searchable. If the user mentions a .pdf file or asks to produce one, use this skill.
-license: Proprietary. LICENSE.txt has complete terms
-metadata:
-  provider: atomic
----
-
-# PDF Processing Guide
-
-## Overview
-
-This guide covers essential PDF processing operations using Python libraries and command-line tools. For advanced features, JavaScript libraries, and detailed examples, see REFERENCE.md. If you need to fill out a PDF form, read FORMS.md and follow its instructions.
-
-## Quick Start
-
-```python
-from pypdf import PdfReader, PdfWriter
-
-# Read a PDF
-reader = PdfReader("document.pdf")
-print(f"Pages: {len(reader.pages)}")
-
-# Extract text
-text = ""
-for page in reader.pages:
-    text += page.extract_text()
-```
-
-## Python Libraries
-
-### pypdf - Basic Operations
-
-#### Merge PDFs
-```python
-from pypdf import PdfWriter, PdfReader
-
-writer = PdfWriter()
-for pdf_file in ["doc1.pdf", "doc2.pdf", "doc3.pdf"]:
-    reader = PdfReader(pdf_file)
-    for page in reader.pages:
-        writer.add_page(page)
-
-with open("merged.pdf", "wb") as output:
-    writer.write(output)
-```
-
-#### Split PDF
-```python
-reader = PdfReader("input.pdf")
-for i, page in enumerate(reader.pages):
-    writer = PdfWriter()
-    writer.add_page(page)
-    with open(f"page_{i+1}.pdf", "wb") as output:
-        writer.write(output)
-```
-
-#### Extract Metadata
-```python
-reader = PdfReader("document.pdf")
-meta = reader.metadata
-print(f"Title: {meta.title}")
-print(f"Author: {meta.author}")
-print(f"Subject: {meta.subject}")
-print(f"Creator: {meta.creator}")
-```
-
-#### Rotate Pages
-```python
-reader = PdfReader("input.pdf")
-writer = PdfWriter()
-
-page = reader.pages[0]
-page.rotate(90)  # Rotate 90 degrees clockwise
-writer.add_page(page)
-
-with open("rotated.pdf", "wb") as output:
-    writer.write(output)
-```
-
-### pdfplumber - Text and Table Extraction
-
-#### Extract Text with Layout
-```python
-import pdfplumber
-
-with pdfplumber.open("document.pdf") as pdf:
-    for page in pdf.pages:
-        text = page.extract_text()
-        print(text)
-```
-
-#### Extract Tables
-```python
-with pdfplumber.open("document.pdf") as pdf:
-    for i, page in enumerate(pdf.pages):
-        tables = page.extract_tables()
-        for j, table in enumerate(tables):
-            print(f"Table {j+1} on page {i+1}:")
-            for row in table:
-                print(row)
-```
-
-#### Advanced Table Extraction
-```python
-import pandas as pd
-
-with pdfplumber.open("document.pdf") as pdf:
-    all_tables = []
-    for page in pdf.pages:
-        tables = page.extract_tables()
-        for table in tables:
-            if table:  # Check if table is not empty
-                df = pd.DataFrame(table[1:], columns=table[0])
-                all_tables.append(df)
-
-# Combine all tables
-if all_tables:
-    combined_df = pd.concat(all_tables, ignore_index=True)
-    combined_df.to_excel("extracted_tables.xlsx", index=False)
-```
-
-### reportlab - Create PDFs
-
-#### Basic PDF Creation
-```python
-from reportlab.lib.pagesizes import letter
-from reportlab.pdfgen import canvas
-
-c = canvas.Canvas("hello.pdf", pagesize=letter)
-width, height = letter
-
-# Add text
-c.drawString(100, height - 100, "Hello World!")
-c.drawString(100, height - 120, "This is a PDF created with reportlab")
-
-# Add a line
-c.line(100, height - 140, 400, height - 140)
-
-# Save
-c.save()
-```
-
-#### Create PDF with Multiple Pages
-```python
-from reportlab.lib.pagesizes import letter
-from reportlab.platypus import SimpleDocTemplate, Paragraph, Spacer, PageBreak
-from reportlab.lib.styles import getSampleStyleSheet
-
-doc = SimpleDocTemplate("report.pdf", pagesize=letter)
-styles = getSampleStyleSheet()
-story = []
-
-# Add content
-title = Paragraph("Report Title", styles['Title'])
-story.append(title)
-story.append(Spacer(1, 12))
-
-body = Paragraph("This is the body of the report. " * 20, styles['Normal'])
-story.append(body)
-story.append(PageBreak())
-
-# Page 2
-story.append(Paragraph("Page 2", styles['Heading1']))
-story.append(Paragraph("Content for page 2", styles['Normal']))
-
-# Build PDF
-doc.build(story)
-```
-
-#### Subscripts and Superscripts
-
-**IMPORTANT**: Never use Unicode subscript/superscript characters (₀₁₂₃₄₅₆₇₈₉, ⁰¹²³⁴⁵⁶⁷⁸⁹) in ReportLab PDFs. The built-in fonts do not include these glyphs, causing them to render as solid black boxes.
-
-Instead, use ReportLab's XML markup tags in Paragraph objects:
-```python
-from reportlab.platypus import Paragraph
-from reportlab.lib.styles import getSampleStyleSheet
-
-styles = getSampleStyleSheet()
-
-# Subscripts: use <sub> tag
-chemical = Paragraph("H<sub>2</sub>O", styles['Normal'])
-
-# Superscripts: use <super> tag
-squared = Paragraph("x<super>2</super> + y<super>2</super>", styles['Normal'])
-```
-
-For canvas-drawn text (not Paragraph objects), manually adjust font the size and position rather than using Unicode subscripts/superscripts.
-
-## Command-Line Tools
-
-### pdftotext (poppler-utils)
-```bash
-# Extract text
-pdftotext input.pdf output.txt
-
-# Extract text preserving layout
-pdftotext -layout input.pdf output.txt
-
-# Extract specific pages
-pdftotext -f 1 -l 5 input.pdf output.txt  # Pages 1-5
-```
-
-### qpdf
-```bash
-# Merge PDFs
-qpdf --empty --pages file1.pdf file2.pdf -- merged.pdf
-
-# Split pages
-qpdf input.pdf --pages . 1-5 -- pages1-5.pdf
-qpdf input.pdf --pages . 6-10 -- pages6-10.pdf
-
-# Rotate pages
-qpdf input.pdf output.pdf --rotate=+90:1  # Rotate page 1 by 90 degrees
-
-# Remove password
-qpdf --password=mypassword --decrypt encrypted.pdf decrypted.pdf
-```
-
-### pdftk (if available)
-```bash
-# Merge
-pdftk file1.pdf file2.pdf cat output merged.pdf
-
-# Split
-pdftk input.pdf burst
-
-# Rotate
-pdftk input.pdf rotate 1east output rotated.pdf
-```
-
-## Common Tasks
-
-### Extract Text from Scanned PDFs
-```python
-# Requires: pip install pytesseract pdf2image
-import pytesseract
-from pdf2image import convert_from_path
-
-# Convert PDF to images
-images = convert_from_path('scanned.pdf')
-
-# OCR each page
-text = ""
-for i, image in enumerate(images):
-    text += f"Page {i+1}:\n"
-    text += pytesseract.image_to_string(image)
-    text += "\n\n"
-
-print(text)
-```
-
-### Add Watermark
-```python
-from pypdf import PdfReader, PdfWriter
-
-# Create watermark (or load existing)
-watermark = PdfReader("watermark.pdf").pages[0]
-
-# Apply to all pages
-reader = PdfReader("document.pdf")
-writer = PdfWriter()
-
-for page in reader.pages:
-    page.merge_page(watermark)
-    writer.add_page(page)
-
-with open("watermarked.pdf", "wb") as output:
-    writer.write(output)
-```
-
-### Extract Images
-```bash
-# Using pdfimages (poppler-utils)
-pdfimages -j input.pdf output_prefix
-
-# This extracts all images as output_prefix-000.jpg, output_prefix-001.jpg, etc.
-```
-
-### Password Protection
-```python
-from pypdf import PdfReader, PdfWriter
-
-reader = PdfReader("input.pdf")
-writer = PdfWriter()
-
-for page in reader.pages:
-    writer.add_page(page)
-
-# Add password
-writer.encrypt("userpassword", "ownerpassword")
-
-with open("encrypted.pdf", "wb") as output:
-    writer.write(output)
-```
-
-## Quick Reference
-
-| Task | Best Tool | Command/Code |
-|------|-----------|--------------|
-| Merge PDFs | pypdf | `writer.add_page(page)` |
-| Split PDFs | pypdf | One page per file |
-| Extract text | pdfplumber | `page.extract_text()` |
-| Extract tables | pdfplumber | `page.extract_tables()` |
-| Create PDFs | reportlab | Canvas or Platypus |
-| Command line merge | qpdf | `qpdf --empty --pages ...` |
-| OCR scanned PDFs | pytesseract | Convert to image first |
-| Fill PDF forms | pdf-lib or pypdf (see FORMS.md) | See FORMS.md |
-
-## Next Steps
-
-- For advanced pypdfium2 usage, see REFERENCE.md
-- For JavaScript libraries (pdf-lib), see REFERENCE.md
-- If you need to fill out a PDF form, follow the instructions in FORMS.md
-- For troubleshooting guides, see REFERENCE.md

package/.agents/skills/pdf/forms.md

@@ -1,294 +0,0 @@
-**CRITICAL: You MUST complete these steps in order. Do not skip ahead to writing code.**
-
-If you need to fill out a PDF form, first check to see if the PDF has fillable form fields. Run this script from this file's directory:
- `python scripts/check_fillable_fields <file.pdf>`, and depending on the result go to either the "Fillable fields" or "Non-fillable fields" and follow those instructions.
-
-# Fillable fields
-If the PDF has fillable form fields:
-- Run this script from this file's directory: `python scripts/extract_form_field_info.py <input.pdf> <field_info.json>`. It will create a JSON file with a list of fields in this format:
-```
-[
-  {
-    "field_id": (unique ID for the field),
-    "page": (page number, 1-based),
-    "rect": ([left, bottom, right, top] bounding box in PDF coordinates, y=0 is the bottom of the page),
-    "type": ("text", "checkbox", "radio_group", or "choice"),
-  },
-  // Checkboxes have "checked_value" and "unchecked_value" properties:
-  {
-    "field_id": (unique ID for the field),
-    "page": (page number, 1-based),
-    "type": "checkbox",
-    "checked_value": (Set the field to this value to check the checkbox),
-    "unchecked_value": (Set the field to this value to uncheck the checkbox),
-  },
-  // Radio groups have a "radio_options" list with the possible choices.
-  {
-    "field_id": (unique ID for the field),
-    "page": (page number, 1-based),
-    "type": "radio_group",
-    "radio_options": [
-      {
-        "value": (set the field to this value to select this radio option),
-        "rect": (bounding box for the radio button for this option)
-      },
-      // Other radio options
-    ]
-  },
-  // Multiple choice fields have a "choice_options" list with the possible choices:
-  {
-    "field_id": (unique ID for the field),
-    "page": (page number, 1-based),
-    "type": "choice",
-    "choice_options": [
-      {
-        "value": (set the field to this value to select this option),
-        "text": (display text of the option)
-      },
-      // Other choice options
-    ],
-  }
-]
-```
-- Convert the PDF to PNGs (one image for each page) with this script (run from this file's directory):
-`python scripts/convert_pdf_to_images.py <file.pdf> <output_directory>`
-Then analyze the images to determine the purpose of each form field (make sure to convert the bounding box PDF coordinates to image coordinates).
-- Create a `field_values.json` file in this format with the values to be entered for each field:
-```
-[
-  {
-    "field_id": "last_name", // Must match the field_id from `extract_form_field_info.py`
-    "description": "The user's last name",
-    "page": 1, // Must match the "page" value in field_info.json
-    "value": "Simpson"
-  },
-  {
-    "field_id": "Checkbox12",
-    "description": "Checkbox to be checked if the user is 18 or over",
-    "page": 1,
-    "value": "/On" // If this is a checkbox, use its "checked_value" value to check it. If it's a radio button group, use one of the "value" values in "radio_options".
-  },
-  // more fields
-]
-```
-- Run the `fill_fillable_fields.py` script from this file's directory to create a filled-in PDF:
-`python scripts/fill_fillable_fields.py <input pdf> <field_values.json> <output pdf>`
-This script will verify that the field IDs and values you provide are valid; if it prints error messages, correct the appropriate fields and try again.
-
-# Non-fillable fields
-If the PDF doesn't have fillable form fields, you'll add text annotations. First try to extract coordinates from the PDF structure (more accurate), then fall back to visual estimation if needed.
-
-## Step 1: Try Structure Extraction First
-
-Run this script to extract text labels, lines, and checkboxes with their exact PDF coordinates:
-`python scripts/extract_form_structure.py <input.pdf> form_structure.json`
-
-This creates a JSON file containing:
-- **labels**: Every text element with exact coordinates (x0, top, x1, bottom in PDF points)
-- **lines**: Horizontal lines that define row boundaries
-- **checkboxes**: Small square rectangles that are checkboxes (with center coordinates)
-- **row_boundaries**: Row top/bottom positions calculated from horizontal lines
-
-**Check the results**: If `form_structure.json` has meaningful labels (text elements that correspond to form fields), use **Approach A: Structure-Based Coordinates**. If the PDF is scanned/image-based and has few or no labels, use **Approach B: Visual Estimation**.
-
----
-
-## Approach A: Structure-Based Coordinates (Preferred)
-
-Use this when `extract_form_structure.py` found text labels in the PDF.
-
-### A.1: Analyze the Structure
-
-Read form_structure.json and identify:
-
-1. **Label groups**: Adjacent text elements that form a single label (e.g., "Last" + "Name")
-2. **Row structure**: Labels with similar `top` values are in the same row
-3. **Field columns**: Entry areas start after label ends (x0 = label.x1 + gap)
-4. **Checkboxes**: Use the checkbox coordinates directly from the structure
-
-**Coordinate system**: PDF coordinates where y=0 is at TOP of page, y increases downward.
-
-### A.2: Check for Missing Elements
-
-The structure extraction may not detect all form elements. Common cases:
-- **Circular checkboxes**: Only square rectangles are detected as checkboxes
-- **Complex graphics**: Decorative elements or non-standard form controls
-- **Faded or light-colored elements**: May not be extracted
-
-If you see form fields in the PDF images that aren't in form_structure.json, you'll need to use **visual analysis** for those specific fields (see "Hybrid Approach" below).
-
-### A.3: Create fields.json with PDF Coordinates
-
-For each field, calculate entry coordinates from the extracted structure:
-
-**Text fields:**
-- entry x0 = label x1 + 5 (small gap after label)
-- entry x1 = next label's x0, or row boundary
-- entry top = same as label top
-- entry bottom = row boundary line below, or label bottom + row_height
-
-**Checkboxes:**
-- Use the checkbox rectangle coordinates directly from form_structure.json
-- entry_bounding_box = [checkbox.x0, checkbox.top, checkbox.x1, checkbox.bottom]
-
-Create fields.json using `pdf_width` and `pdf_height` (signals PDF coordinates):
-```json
-{
-  "pages": [
-    {"page_number": 1, "pdf_width": 612, "pdf_height": 792}
-  ],
-  "form_fields": [
-    {
-      "page_number": 1,
-      "description": "Last name entry field",
-      "field_label": "Last Name",
-      "label_bounding_box": [43, 63, 87, 73],
-      "entry_bounding_box": [92, 63, 260, 79],
-      "entry_text": {"text": "Smith", "font_size": 10}
-    },
-    {
-      "page_number": 1,
-      "description": "US Citizen Yes checkbox",
-      "field_label": "Yes",
-      "label_bounding_box": [260, 200, 280, 210],
-      "entry_bounding_box": [285, 197, 292, 205],
-      "entry_text": {"text": "X"}
-    }
-  ]
-}
-```
-
-**Important**: Use `pdf_width`/`pdf_height` and coordinates directly from form_structure.json.
-
-### A.4: Validate Bounding Boxes
-
-Before filling, check your bounding boxes for errors:
-`python scripts/check_bounding_boxes.py fields.json`
-
-This checks for intersecting bounding boxes and entry boxes that are too small for the font size. Fix any reported errors before filling.
-
----
-
-## Approach B: Visual Estimation (Fallback)
-
-Use this when the PDF is scanned/image-based and structure extraction found no usable text labels (e.g., all text shows as "(cid:X)" patterns).
-
-### B.1: Convert PDF to Images
-
-`python scripts/convert_pdf_to_images.py <input.pdf> <images_dir/>`
-
-### B.2: Initial Field Identification
-
-Examine each page image to identify form sections and get **rough estimates** of field locations:
-- Form field labels and their approximate positions
-- Entry areas (lines, boxes, or blank spaces for text input)
-- Checkboxes and their approximate locations
-
-For each field, note approximate pixel coordinates (they don't need to be precise yet).
-
-### B.3: Zoom Refinement (CRITICAL for accuracy)
-
-For each field, crop a region around the estimated position to refine coordinates precisely.
-
-**Create a zoomed crop using ImageMagick:**
-```bash
-magick <page_image> -crop <width>x<height>+<x>+<y> +repage <crop_output.png>
-```
-
-Where:
-- `<x>, <y>` = top-left corner of crop region (use your rough estimate minus padding)
-- `<width>, <height>` = size of crop region (field area plus ~50px padding on each side)
-
-**Example:** To refine a "Name" field estimated around (100, 150):
-```bash
-magick images_dir/page_1.png -crop 300x80+50+120 +repage crops/name_field.png
-```
-
-(Note: if the `magick` command isn't available, try `convert` with the same arguments).
-
-**Examine the cropped image** to determine precise coordinates:
-1. Identify the exact pixel where the entry area begins (after the label)
-2. Identify where the entry area ends (before next field or edge)
-3. Identify the top and bottom of the entry line/box
-
-**Convert crop coordinates back to full image coordinates:**
-- full_x = crop_x + crop_offset_x
-- full_y = crop_y + crop_offset_y
-
-Example: If the crop started at (50, 120) and the entry box starts at (52, 18) within the crop:
-- entry_x0 = 52 + 50 = 102
-- entry_top = 18 + 120 = 138
-
-**Repeat for each field**, grouping nearby fields into single crops when possible.
-
-### B.4: Create fields.json with Refined Coordinates
-
-Create fields.json using `image_width` and `image_height` (signals image coordinates):
-```json
-{
-  "pages": [
-    {"page_number": 1, "image_width": 1700, "image_height": 2200}
-  ],
-  "form_fields": [
-    {
-      "page_number": 1,
-      "description": "Last name entry field",
-      "field_label": "Last Name",
-      "label_bounding_box": [120, 175, 242, 198],
-      "entry_bounding_box": [255, 175, 720, 218],
-      "entry_text": {"text": "Smith", "font_size": 10}
-    }
-  ]
-}
-```
-
-**Important**: Use `image_width`/`image_height` and the refined pixel coordinates from the zoom analysis.
-
-### B.5: Validate Bounding Boxes
-
-Before filling, check your bounding boxes for errors:
-`python scripts/check_bounding_boxes.py fields.json`
-
-This checks for intersecting bounding boxes and entry boxes that are too small for the font size. Fix any reported errors before filling.
-
----
-
-## Hybrid Approach: Structure + Visual
-
-Use this when structure extraction works for most fields but misses some elements (e.g., circular checkboxes, unusual form controls).
-
-1. **Use Approach A** for fields that were detected in form_structure.json
-2. **Convert PDF to images** for visual analysis of missing fields
-3. **Use zoom refinement** (from Approach B) for the missing fields
-4. **Combine coordinates**: For fields from structure extraction, use `pdf_width`/`pdf_height`. For visually-estimated fields, you must convert image coordinates to PDF coordinates:
-   - pdf_x = image_x * (pdf_width / image_width)
-   - pdf_y = image_y * (pdf_height / image_height)
-5. **Use a single coordinate system** in fields.json - convert all to PDF coordinates with `pdf_width`/`pdf_height`
-
----
-
-## Step 2: Validate Before Filling
-
-**Always validate bounding boxes before filling:**
-`python scripts/check_bounding_boxes.py fields.json`
-
-This checks for:
-- Intersecting bounding boxes (which would cause overlapping text)
-- Entry boxes that are too small for the specified font size
-
-Fix any reported errors in fields.json before proceeding.
-
-## Step 3: Fill the Form
-
-The fill script auto-detects the coordinate system and handles conversion:
-`python scripts/fill_pdf_form_with_annotations.py <input.pdf> fields.json <output.pdf>`
-
-## Step 4: Verify Output
-
-Convert the filled PDF to images and verify text placement:
-`python scripts/convert_pdf_to_images.py <output.pdf> <verify_images/>`
-
-If text is mispositioned:
-- **Approach A**: Check that you're using PDF coordinates from form_structure.json with `pdf_width`/`pdf_height`
-- **Approach B**: Check that image dimensions match and coordinates are accurate pixels
-- **Hybrid**: Ensure coordinate conversions are correct for visually-estimated fields