@brookmind/ai-toolkit 1.1.6 → 1.2.0

package/skills/pdf-processing-pro/FORMS.md

@@ -1,610 +0,0 @@
-# PDF Form Processing Guide
-
-Complete guide for processing PDF forms in production environments.
-
-## Table of contents
-
-- Form analysis and field detection
-- Form filling workflows
-- Validation strategies
-- Field types and handling
-- Multi-page forms
-- Flattening and finalization
-- Error handling patterns
-- Production examples
-
-## Form analysis
-
-### Analyze form structure
-
-Use `analyze_form.py` to extract complete form information:
-
-```bash
-python scripts/analyze_form.py application.pdf --output schema.json
-```
-
-Output format:
-
-```json
-{
-  "full_name": {
-    "type": "text",
-    "required": true,
-    "max_length": 100,
-    "x": 120.5,
-    "y": 450.2,
-    "width": 300,
-    "height": 20
-  },
-  "date_of_birth": {
-    "type": "text",
-    "required": true,
-    "format": "MM/DD/YYYY",
-    "x": 120.5,
-    "y": 400.8,
-    "width": 150,
-    "height": 20
-  },
-  "email_newsletter": {
-    "type": "checkbox",
-    "required": false,
-    "x": 120.5,
-    "y": 350.4,
-    "width": 15,
-    "height": 15
-  },
-  "preferred_contact": {
-    "type": "radio",
-    "required": true,
-    "options": ["email", "phone", "mail"],
-    "x": 120.5,
-    "y": 300.0,
-    "width": 200,
-    "height": 60
-  }
-}
-```
-
-### Programmatic analysis
-
-```python
-from pypdf import PdfReader
-
-reader = PdfReader("form.pdf")
-fields = reader.get_fields()
-
-for field_name, field_info in fields.items():
-    print(f"Field: {field_name}")
-    print(f"  Type: {field_info.get('/FT')}")
-    print(f"  Value: {field_info.get('/V')}")
-    print(f"  Flags: {field_info.get('/Ff', 0)}")
-    print()
-```
-
-## Form filling workflows
-
-### Basic workflow
-
-```bash
-# 1. Analyze form
-python scripts/analyze_form.py template.pdf --output schema.json
-
-# 2. Prepare data
-cat > data.json << EOF
-{
-  "full_name": "John Doe",
-  "date_of_birth": "01/15/1990",
-  "email": "john@example.com",
-  "email_newsletter": true,
-  "preferred_contact": "email"
-}
-EOF
-
-# 3. Validate data
-python scripts/validate_form.py data.json schema.json
-
-# 4. Fill form
-python scripts/fill_form.py template.pdf data.json filled.pdf
-
-# 5. Flatten (optional - makes fields non-editable)
-python scripts/flatten_form.py filled.pdf final.pdf
-```
-
-### Programmatic filling
-
-```python
-from pypdf import PdfReader, PdfWriter
-
-reader = PdfReader("template.pdf")
-writer = PdfWriter()
-
-# Clone all pages
-for page in reader.pages:
-    writer.add_page(page)
-
-# Fill form fields
-writer.update_page_form_field_values(
-    writer.pages[0],
-    {
-        "full_name": "John Doe",
-        "date_of_birth": "01/15/1990",
-        "email": "john@example.com",
-        "email_newsletter": "/Yes",  # Checkbox value
-        "preferred_contact": "/email"  # Radio value
-    }
-)
-
-# Save filled form
-with open("filled.pdf", "wb") as output:
-    writer.write(output)
-```
-
-## Field types and handling
-
-### Text fields
-
-```python
-# Simple text
-field_values["customer_name"] = "Jane Smith"
-
-# Formatted text (dates)
-field_values["date"] = "12/25/2024"
-
-# Numbers
-field_values["amount"] = "1234.56"
-
-# Multi-line text
-field_values["comments"] = "Line 1\nLine 2\nLine 3"
-```
-
-### Checkboxes
-
-Checkboxes typically use `/Yes` for checked, `/Off` for unchecked:
-
-```python
-# Check checkbox
-field_values["agree_to_terms"] = "/Yes"
-
-# Uncheck checkbox
-field_values["newsletter_opt_out"] = "/Off"
-```
-
-**Note**: Some PDFs use different values. Check with `analyze_form.py`:
-
-```json
-{
-  "some_checkbox": {
-    "type": "checkbox",
-    "on_value": "/On",   # ← Check this
-    "off_value": "/Off"
-  }
-}
-```
-
-### Radio buttons
-
-Radio buttons are mutually exclusive options:
-
-```python
-# Select one option from radio group
-field_values["preferred_contact"] = "/email"
-
-# Other options in same group
-# field_values["preferred_contact"] = "/phone"
-# field_values["preferred_contact"] = "/mail"
-```
-
-### Dropdown/List boxes
-
-```python
-# Single selection
-field_values["country"] = "United States"
-
-# List of available options in schema
-"country": {
-  "type": "dropdown",
-  "options": ["United States", "Canada", "Mexico", ...]
-}
-```
-
-## Validation strategies
-
-### Schema-based validation
-
-```python
-import json
-from jsonschema import validate, ValidationError
-
-# Load schema from analyze_form.py output
-with open("schema.json") as f:
-    schema = json.load(f)
-
-# Load form data
-with open("data.json") as f:
-    data = json.load(f)
-
-# Validate all fields
-errors = []
-
-for field_name, field_schema in schema.items():
-    value = data.get(field_name)
-
-    # Check required fields
-    if field_schema.get("required") and not value:
-        errors.append(f"Missing required field: {field_name}")
-
-    # Check field type
-    if value and field_schema.get("type") == "text":
-        if not isinstance(value, str):
-            errors.append(f"Field {field_name} must be string")
-
-    # Check max length
-    max_length = field_schema.get("max_length")
-    if value and max_length and len(str(value)) > max_length:
-        errors.append(f"Field {field_name} exceeds max length {max_length}")
-
-    # Check format (dates, emails, etc)
-    format_type = field_schema.get("format")
-    if value and format_type:
-        if not validate_format(value, format_type):
-            errors.append(f"Field {field_name} has invalid format")
-
-if errors:
-    print("Validation errors:")
-    for error in errors:
-        print(f"  - {error}")
-    exit(1)
-
-print("Validation passed")
-```
-
-### Format validation
-
-```python
-import re
-from datetime import datetime
-
-def validate_format(value, format_type):
-    """Validate field format."""
-
-    if format_type == "email":
-        pattern = r'^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$'
-        return re.match(pattern, value) is not None
-
-    elif format_type == "phone":
-        # US phone: (555) 123-4567 or 555-123-4567
-        pattern = r'^\(?\d{3}\)?[-.\s]?\d{3}[-.\s]?\d{4}$'
-        return re.match(pattern, value) is not None
-
-    elif format_type == "MM/DD/YYYY":
-        try:
-            datetime.strptime(value, "%m/%d/%Y")
-            return True
-        except ValueError:
-            return False
-
-    elif format_type == "SSN":
-        # XXX-XX-XXXX
-        pattern = r'^\d{3}-\d{2}-\d{4}$'
-        return re.match(pattern, value) is not None
-
-    elif format_type == "ZIP":
-        # XXXXX or XXXXX-XXXX
-        pattern = r'^\d{5}(-\d{4})?$'
-        return re.match(pattern, value) is not None
-
-    return True  # Unknown format, skip validation
-```
-
-## Multi-page forms
-
-### Handling multi-page forms
-
-```python
-from pypdf import PdfReader, PdfWriter
-
-reader = PdfReader("multi_page_form.pdf")
-writer = PdfWriter()
-
-# Clone all pages
-for page in reader.pages:
-    writer.add_page(page)
-
-# Fill fields on page 1
-writer.update_page_form_field_values(
-    writer.pages[0],
-    {
-        "name_page1": "John Doe",
-        "email_page1": "john@example.com"
-    }
-)
-
-# Fill fields on page 2
-writer.update_page_form_field_values(
-    writer.pages[1],
-    {
-        "address_page2": "123 Main St",
-        "city_page2": "Springfield"
-    }
-)
-
-# Fill fields on page 3
-writer.update_page_form_field_values(
-    writer.pages[2],
-    {
-        "signature_page3": "John Doe",
-        "date_page3": "12/25/2024"
-    }
-)
-
-with open("filled_multi_page.pdf", "wb") as output:
-    writer.write(output)
-```
-
-### Identifying page-specific fields
-
-```python
-# Analyze which fields are on which pages
-for page_num, page in enumerate(reader.pages, 1):
-    fields = page.get("/Annots", [])
-
-    if fields:
-        print(f"\nPage {page_num} fields:")
-        for field_ref in fields:
-            field = field_ref.get_object()
-            field_name = field.get("/T", "Unknown")
-            print(f"  - {field_name}")
-```
-
-## Flattening forms
-
-### Why flatten
-
-Flattening makes form fields non-editable, embedding values permanently:
-
-- **Security**: Prevent modifications
-- **Distribution**: Share read-only forms
-- **Printing**: Ensure correct appearance
-- **Archival**: Long-term storage
-
-### Flatten with pypdf
-
-```python
-from pypdf import PdfReader, PdfWriter
-
-reader = PdfReader("filled.pdf")
-writer = PdfWriter()
-
-# Add all pages
-for page in reader.pages:
-    writer.add_page(page)
-
-# Flatten all form fields
-writer.flatten_fields()
-
-# Save flattened PDF
-with open("flattened.pdf", "wb") as output:
-    writer.write(output)
-```
-
-### Using included script
-
-```bash
-python scripts/flatten_form.py filled.pdf flattened.pdf
-```
-
-## Error handling patterns
-
-### Robust form filling
-
-```python
-import logging
-from pathlib import Path
-from pypdf import PdfReader, PdfWriter
-from pypdf.errors import PdfReadError
-
-logging.basicConfig(level=logging.INFO)
-logger = logging.getLogger(__name__)
-
-def fill_form_safe(template_path, data, output_path):
-    """Fill form with comprehensive error handling."""
-
-    try:
-        # Validate inputs
-        template = Path(template_path)
-        if not template.exists():
-            raise FileNotFoundError(f"Template not found: {template_path}")
-
-        # Read template
-        logger.info(f"Reading template: {template_path}")
-        reader = PdfReader(template_path)
-
-        if not reader.pages:
-            raise ValueError("PDF has no pages")
-
-        # Check if form has fields
-        fields = reader.get_fields()
-        if not fields:
-            logger.warning("PDF has no form fields")
-            return False
-
-        # Create writer
-        writer = PdfWriter()
-        for page in reader.pages:
-            writer.add_page(page)
-
-        # Validate data against schema
-        missing_required = []
-        invalid_fields = []
-
-        for field_name, field_info in fields.items():
-            # Check required fields
-            is_required = field_info.get("/Ff", 0) & 2 == 2
-            if is_required and field_name not in data:
-                missing_required.append(field_name)
-
-            # Check invalid field names in data
-            if field_name in data:
-                value = data[field_name]
-                # Add type validation here if needed
-
-        if missing_required:
-            raise ValueError(f"Missing required fields: {missing_required}")
-
-        # Fill fields
-        logger.info("Filling form fields")
-        writer.update_page_form_field_values(
-            writer.pages[0],
-            data
-        )
-
-        # Write output
-        logger.info(f"Writing output: {output_path}")
-        with open(output_path, "wb") as output:
-            writer.write(output)
-
-        logger.info("Form filled successfully")
-        return True
-
-    except PdfReadError as e:
-        logger.error(f"PDF read error: {e}")
-        return False
-
-    except FileNotFoundError as e:
-        logger.error(f"File error: {e}")
-        return False
-
-    except ValueError as e:
-        logger.error(f"Validation error: {e}")
-        return False
-
-    except Exception as e:
-        logger.error(f"Unexpected error: {e}")
-        return False
-
-# Usage
-success = fill_form_safe(
-    "template.pdf",
-    {"name": "John", "email": "john@example.com"},
-    "filled.pdf"
-)
-
-if not success:
-    exit(1)
-```
-
-## Production examples
-
-### Example 1: Batch form processing
-
-```python
-import json
-import glob
-from pathlib import Path
-from fill_form_safe import fill_form_safe
-
-# Process multiple submissions
-submissions_dir = Path("submissions")
-template = "application_template.pdf"
-output_dir = Path("completed")
-output_dir.mkdir(exist_ok=True)
-
-for submission_file in submissions_dir.glob("*.json"):
-    print(f"Processing: {submission_file.name}")
-
-    # Load submission data
-    with open(submission_file) as f:
-        data = json.load(f)
-
-    # Fill form
-    applicant_id = data.get("id", "unknown")
-    output_file = output_dir / f"application_{applicant_id}.pdf"
-
-    success = fill_form_safe(template, data, output_file)
-
-    if success:
-        print(f"  ✓ Completed: {output_file}")
-    else:
-        print(f"  ✗ Failed: {submission_file.name}")
-```
-
-### Example 2: Form with conditional logic
-
-```python
-def prepare_form_data(raw_data):
-    """Prepare form data with conditional logic."""
-
-    form_data = {}
-
-    # Basic fields
-    form_data["full_name"] = raw_data["name"]
-    form_data["email"] = raw_data["email"]
-
-    # Conditional fields
-    if raw_data.get("is_student"):
-        form_data["student_id"] = raw_data["student_id"]
-        form_data["school_name"] = raw_data["school"]
-    else:
-        form_data["employer"] = raw_data.get("employer", "")
-
-    # Checkbox logic
-    form_data["newsletter"] = "/Yes" if raw_data.get("opt_in") else "/Off"
-
-    # Calculated fields
-    total = sum(raw_data.get("items", []))
-    form_data["total_amount"] = f"${total:.2f}"
-
-    return form_data
-
-# Usage
-raw_input = {
-    "name": "Jane Smith",
-    "email": "jane@example.com",
-    "is_student": True,
-    "student_id": "12345",
-    "school": "State University",
-    "opt_in": True,
-    "items": [10.00, 25.50, 15.75]
-}
-
-form_data = prepare_form_data(raw_input)
-fill_form_safe("template.pdf", form_data, "output.pdf")
-```
-
-## Best practices
-
-1. **Always analyze before filling**: Use `analyze_form.py` to understand structure
-2. **Validate early**: Check data before attempting to fill
-3. **Use logging**: Track operations for debugging
-4. **Handle errors gracefully**: Don't crash on invalid data
-5. **Test with samples**: Verify with small datasets first
-6. **Flatten when distributing**: Make read-only for recipients
-7. **Keep templates versioned**: Track form template changes
-8. **Document field mappings**: Maintain data-to-field documentation
-
-## Troubleshooting
-
-### Fields not filling
-
-1. Check field names match exactly (case-sensitive)
-2. Verify checkbox/radio values (`/Yes`, `/On`, etc.)
-3. Ensure PDF is not encrypted or protected
-4. Check if form uses XFA format (not supported by pypdf)
-
-### Encoding issues
-
-```python
-# Handle special characters
-field_values["name"] = "José García"  # UTF-8 encoded
-```
-
-### Large batch processing
-
-```python
-# Process in chunks to avoid memory issues
-chunk_size = 100
-
-for i in range(0, len(submissions), chunk_size):
-    chunk = submissions[i:i + chunk_size]
-    process_batch(chunk)
-```