flyfield 2025.8.29__tar.gz

flyfield-2025.8.29/LICENSE

@@ -0,0 +1,3 @@
+flyfield is licensed under the GNU General Public License version 3 or later (GPL-3.0-or-later).
+
+Full license text available at: https://www.gnu.org/licenses/gpl-3.0.html

flyfield-2025.8.29/MANIFEST.in

@@ -0,0 +1,3 @@
+include README.md
+include LICENSE
+recursive-include flyfield *.py

flyfield-2025.8.29/PKG-INFO

@@ -0,0 +1,164 @@
+Metadata-Version: 2.4
+Name: flyfield
+Version: 2025.8.29
+Summary: Tools for extracting, processing, and generating interactive fields for PDFs containing white box fields.
+Author-email: flywire <flywire0@gmail.com>
+License: GPL-3.0-or-later
+Keywords: pdf,form,fields,extraction,pymupdf,pypdfforms
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+)
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Topic :: Software Development :: Libraries
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: PyMuPDF>=1.18.0
+Requires-Dist: PyPDFForm>=0.3.3
+Dynamic: license-file
+
+# flyfield
+
+A tool to automatically detect white boxes in PDFs and convert them into interactive, automated form fields—aimed at users and developers looking to streamline PDF form workflows.
+
+---
+
+## Overview
+
+**flyfield** is a Python library and command-line tool designed to automate the conversion of white box placeholders within PDFs into fully interactive form fields. This enables users and developers to analyze form layouts, generate fillable fields, fill forms programmatically using CSV data, and capture filled data for further use.
+
+Powered by [PyMuPDF](https://pymupdf.readthedocs.io) and [PyPDFForm](https://pypdfform.readthedocs.io), flyfield provides a modular, extensible solution for reliable PDF form automation.
+
+---
+
+## Key Features
+
+- **White box detection:** Precisely extract white box regions as potential form fields from vector PDFs.
+- **Layout analysis:** Group extracted fields by page, line, and block with flexible gap detection.
+- **Form field generation:** Automatically produce Python scripts to create interactive PDF form fields aligned with detected boxes.
+- **Markup visualization:** Generate annotated PDFs marking detected fields for verification.
+- **Form filling and capture:** Fill PDF forms programmatically from CSV data and extract filled data into CSV format.
+- **CLI integration:** User-friendly command-line interface to orchestrate workflows from extraction to data capture.
+- **Open Source and Extensible:** Easily customize and extend for specific PDF processing needs.
+
+---
+
+## Installation
+
+For isolated installation, use [pipx](https://pipxproject.github.io/pipx/):
+
+```
+
+pipx install flyfield
+
+```
+
+Verify the installed version:
+
+```
+
+flyfield --version
+
+```
+
+Alternatively, install with `pip`:
+
+```
+
+pip install flyfield
+
+```
+
+---
+
+## Usage
+
+Execute commands against PDF files as needed:
+
+```
+
+flyfield --input-pdf myfile.pdf --markup
+
+```
+
+Common options:
+
+- `--markup` → Generate a marked-up PDF showing detected white boxes.
+- `--fields` → Generate and run scripts to add interactive form fields.
+- `--fill` → Fill form fields using CSV data.
+- `--capture` → Extract filled form data back to CSV.
+- `--input-csv` → Provide CSV input for field data, bypassing extraction.
+- `--debug` → Enable verbose debug output.
+
+Example workflow:
+
+```
+
+flyfield --input-pdf form_template.pdf --markup --fields
+flyfield --input-pdf form_template-fields.pdf --input-csv form_template.csv --fill
+flyfield --input-pdf form_template-filled.pdf --capture
+
+```
+
+---
+
+## For Developers
+
+Clone the repository and install development dependencies:
+
+```
+
+git clone https://github.com/flywire/flyfield.git
+cd flyfield
+pip install -e .[dev]
+
+```
+
+Run the test suite with coverage enabled:
+
+```
+
+tox
+
+```
+
+The project is modular with separate components, including:
+
+- `extract` for box detection
+- `layout` for field grouping and filtering
+- `markup_and_fields` for field generation and markup
+- `io_utils` for data input/output
+- `utils` for utility functions
+
+Get CLI help with:
+
+```
+
+python -m flyfield.cli --help
+
+```
+
+---
+
+## License
+
+Licensed under the **GNU General Public License v3.0 or later** (GPL-3.0-or-later). See [LICENSE](LICENSE) for details.
+
+---
+
+## Contributing
+
+Contributions are welcome! Please open issues to report bugs or request features, and submit pull requests with tests and documentation improvements.
+
+---
+
+## Acknowledgements
+
+- Built using [PyMuPDF](https://pymupdf.readthedocs.io) for PDF handling.
+- Uses [PyPDFForm](https://pypdfform.readthedocs.io) for interactive form creation.
+- Inspired by the need for robust automation of PDF workflows involving white boxed form fields.
+
+---

flyfield-2025.8.29/README.md

@@ -0,0 +1,142 @@
+# flyfield
+
+A tool to automatically detect white boxes in PDFs and convert them into interactive, automated form fields—aimed at users and developers looking to streamline PDF form workflows.
+
+---
+
+## Overview
+
+**flyfield** is a Python library and command-line tool designed to automate the conversion of white box placeholders within PDFs into fully interactive form fields. This enables users and developers to analyze form layouts, generate fillable fields, fill forms programmatically using CSV data, and capture filled data for further use.
+
+Powered by [PyMuPDF](https://pymupdf.readthedocs.io) and [PyPDFForm](https://pypdfform.readthedocs.io), flyfield provides a modular, extensible solution for reliable PDF form automation.
+
+---
+
+## Key Features
+
+- **White box detection:** Precisely extract white box regions as potential form fields from vector PDFs.
+- **Layout analysis:** Group extracted fields by page, line, and block with flexible gap detection.
+- **Form field generation:** Automatically produce Python scripts to create interactive PDF form fields aligned with detected boxes.
+- **Markup visualization:** Generate annotated PDFs marking detected fields for verification.
+- **Form filling and capture:** Fill PDF forms programmatically from CSV data and extract filled data into CSV format.
+- **CLI integration:** User-friendly command-line interface to orchestrate workflows from extraction to data capture.
+- **Open Source and Extensible:** Easily customize and extend for specific PDF processing needs.
+
+---
+
+## Installation
+
+For isolated installation, use [pipx](https://pipxproject.github.io/pipx/):
+
+```
+
+pipx install flyfield
+
+```
+
+Verify the installed version:
+
+```
+
+flyfield --version
+
+```
+
+Alternatively, install with `pip`:
+
+```
+
+pip install flyfield
+
+```
+
+---
+
+## Usage
+
+Execute commands against PDF files as needed:
+
+```
+
+flyfield --input-pdf myfile.pdf --markup
+
+```
+
+Common options:
+
+- `--markup` → Generate a marked-up PDF showing detected white boxes.
+- `--fields` → Generate and run scripts to add interactive form fields.
+- `--fill` → Fill form fields using CSV data.
+- `--capture` → Extract filled form data back to CSV.
+- `--input-csv` → Provide CSV input for field data, bypassing extraction.
+- `--debug` → Enable verbose debug output.
+
+Example workflow:
+
+```
+
+flyfield --input-pdf form_template.pdf --markup --fields
+flyfield --input-pdf form_template-fields.pdf --input-csv form_template.csv --fill
+flyfield --input-pdf form_template-filled.pdf --capture
+
+```
+
+---
+
+## For Developers
+
+Clone the repository and install development dependencies:
+
+```
+
+git clone https://github.com/flywire/flyfield.git
+cd flyfield
+pip install -e .[dev]
+
+```
+
+Run the test suite with coverage enabled:
+
+```
+
+tox
+
+```
+
+The project is modular with separate components, including:
+
+- `extract` for box detection
+- `layout` for field grouping and filtering
+- `markup_and_fields` for field generation and markup
+- `io_utils` for data input/output
+- `utils` for utility functions
+
+Get CLI help with:
+
+```
+
+python -m flyfield.cli --help
+
+```
+
+---
+
+## License
+
+Licensed under the **GNU General Public License v3.0 or later** (GPL-3.0-or-later). See [LICENSE](LICENSE) for details.
+
+---
+
+## Contributing
+
+Contributions are welcome! Please open issues to report bugs or request features, and submit pull requests with tests and documentation improvements.
+
+---
+
+## Acknowledgements
+
+- Built using [PyMuPDF](https://pymupdf.readthedocs.io) for PDF handling.
+- Uses [PyPDFForm](https://pypdfform.readthedocs.io) for interactive form creation.
+- Inspired by the need for robust automation of PDF workflows involving white boxed form fields.
+
+---

flyfield-2025.8.29/flyfield/__init__.py

@@ -0,0 +1,23 @@
+"""
+pdf_form_generator package
+
+This package provides modular components for:
+- PDF box extraction and layout processing
+- CSV input/output
+- PDF markup and form field generation/filling
+"""
+
+from .config import *
+from .utils import *
+from .extract import *
+from .io_utils import *
+from .markup_and_fields import *
+
+# Optionally define __all__ to specify what is exported on import *
+__all__ = [
+    "config",
+    "utils",
+    "extract",
+    "io_utils",
+    "markup_and_fields",
+]

flyfield-2025.8.29/flyfield/cli.py

@@ -0,0 +1,6 @@
+# flyfield/cli.py
+
+from .main import main
+
+if __name__ == "__main__":
+    main()

flyfield-2025.8.29/flyfield/config.py

@@ -0,0 +1,34 @@
+"""
+Configuration constants used throughout the project.
+"""
+
+# Default file paths
+DEFAULT_INPUT_PDF = "input.pdf"
+DEFAULT_CAPTURE_SUFFIX = "-capture.csv"
+DEFAULT_FIELD_GENERATOR_SUFFIX = "-field-generator.py"
+DEFAULT_FILLER_SUFFIX = "-filler.py"
+DEFAULT_MARKUP_SUFFIX = "-markup"
+DEFAULT_FIELDS_SUFFIX = "-fields"
+DEFAULT_FILL_SUFFIX = "-filled"
+
+# Color constants (normalized RGB)
+COLOR_WHITE = (1, 1, 1)
+COLOR_BLACK = (0, 0, 0)
+
+# Target color for extraction (normalized RGB)
+TARGET_COLOUR = (1, 1, 1)  # RGB normalized white for fill colour detection
+
+# Thresholds for layout grouping and gaps (arbitrary units, related to PDF coords)
+GAP_THRESHOLD = 3.0
+GAP = 1.9
+GAP_GROUP = 7.6
+
+# Constants for height filtering (units are PDF coordinate points)
+MIN_BOX_HEIGHT = 15.0
+MAX_BOX_HEIGHT = 30.0  # Filters out signature boxes
+
+# Field types considered numeric for special processing
+NUMERIC_FIELD_TYPES = ["Currency", "CurrencyDecimal", "DollarCents", "Dollars"]
+
+# Pages to test on, if set (list of page numbers)
+PDF_PAGES = None

flyfield-2025.8.29/flyfield/extract.py

@@ -0,0 +1,176 @@
+import logging
+from collections import defaultdict
+import fitz  # PyMuPDF
+from .config import (
+    TARGET_COLOUR,
+    MIN_BOX_HEIGHT,
+    MAX_BOX_HEIGHT,
+)
+from .utils import (
+    colour_match,
+    int_to_rgb,
+    allowed_text,
+)
+
+logger = logging.getLogger(__name__)
+
+
+def extract_boxes(pdf_path):
+    """
+    Extract filled rectangles (boxes) from each page of the PDF that match the TARGET_COLOUR.
+
+    Args:
+        pdf_path (str): Path to input PDF file.
+
+    Returns:
+        list of dict: Each dict contains box coordinates in PDF coordinate system (origin bottom-left)
+        and metadata such as page number and placeholder fields.
+    """
+    boxes = []
+    try:
+        with fitz.open(pdf_path) as doc:
+            for page_num in range(1, len(doc) + 1):
+                try:
+                    page = doc[page_num - 1]
+                except IndexError:
+                    logger.warning(f"Page {page_num} not found in document.")
+                    continue
+                page_height = page.rect.height
+                for drawing in page.get_drawings():
+                    rect = drawing.get("rect")
+                    fill_color = drawing.get("fill")
+                    if rect and colour_match(fill_color, target_color=TARGET_COLOUR):
+                        # Convert PyMuPDF page coordinates (origin top-left)
+                        # to PDF coordinate system (origin bottom-left)
+
+                        pdf_y0 = page_height - rect.y1
+                        pdf_y1 = page_height - rect.y0
+                        boxes.append(
+                            {
+                                "page_num": page_num,
+                                "x0": rect.x0,
+                                "y0": pdf_y0,
+                                "x1": rect.x1,
+                                "y1": pdf_y1,
+                                "left": round(rect.x0, 2),
+                                "bottom": round(pdf_y0, 2),
+                                "right": round(rect.x1, 2),
+                                "top": round(pdf_y1, 2),
+                                "chars": "",
+                                "field_type": None,
+                            }
+                        )
+    except Exception as e:
+        logger.error(f"Could not open PDF file {pdf_path}: {e}")
+    return boxes
+
+
+def filter_boxes(page, boxes):
+    """
+    Filter boxes on a PDF page by height and allowed text content.
+
+    Args:
+        page (fitz.Page): PyMuPDF page object.
+        boxes (list): List of box dicts extracted from the page.
+
+    Returns:
+        list: Filtered boxes meeting size and allowed text criteria.
+    """
+    filtered = []
+    page_height = page.rect.height
+    black = (0, 0, 0)  # RGB for black text matching
+
+    for box in boxes:
+        height = box.get("y1", 0) - box.get("y0", 0)
+        if height < MIN_BOX_HEIGHT or height > MAX_BOX_HEIGHT:
+            continue
+        # Convert box coordinates to PyMuPDF's coordinate system for clipping
+
+        pymupdf_y0 = page_height - box["y1"]
+        pymupdf_y1 = page_height - box["y0"]
+        clip_rect = fitz.Rect(box["x0"], pymupdf_y0, box["x1"], pymupdf_y1)
+
+        text_dict = page.get_text("dict", clip=clip_rect)
+
+        black_text_parts = []
+        non_black_text_parts = []
+
+        for block in text_dict.get("blocks", []):
+            for line in block.get("lines", []):
+                for span in line.get("spans", []):
+                    span_text = span.get("text", "").strip()
+                    if not span_text:
+                        continue
+                    span_color = span.get("color")
+                    rgb = None
+                    if span_color is not None:
+                        if isinstance(span_color, int):
+                            rgb = int_to_rgb(span_color)
+                        elif isinstance(span_color, str):
+                            try:
+                                rgb = fitz.utils.getColor(span_color)
+                            except Exception:
+                                rgb = None
+                    if rgb and colour_match(rgb, target_color=black):
+                        black_text_parts.append(span_text)
+                    else:
+                        non_black_text_parts.append(span_text)
+        fill_text = "".join(black_text_parts)
+        box_text = "".join(non_black_text_parts)
+
+        allowed, detected_field_type = allowed_text(
+            box_text, field_type=box.get("field_type")
+        )
+        if box_text and not allowed:
+            continue
+        box["field_type"] = detected_field_type
+        box["chars"] = box_text
+        box["fill"] = fill_text
+        filtered.append(box)
+    return filtered
+
+
+def remove_duplicates(boxes):
+    """
+    Remove duplicate boxes per page based on rounded coordinates.
+
+    Args:
+        boxes (list): List of box dicts.
+
+    Returns:
+        list: Boxes without duplicates.
+    """
+    page_groups = defaultdict(list)
+    for box in boxes:
+        page_groups[box["page_num"]].append(box)
+    cleaned = []
+    for page_num, page_boxes in page_groups.items():
+        seen = set()
+        for box in page_boxes:
+            key = (
+                round(box["x0"], 3),
+                round(box["y0"], 3),
+                round(box["x1"], 3),
+                round(box["y1"], 3),
+            )
+            if key not in seen:
+                seen.add(key)
+                cleaned.append(box)
+    return cleaned
+
+
+def sort_boxes(boxes, decimal_places=0):
+    """
+    Sort boxes by page number, top-to-bottom (descending), then left-to-right.
+
+    Args:
+        boxes (list): List of box dicts.
+        decimal_places (int): Precision for bottom coordinate rounding to group near boxes.
+
+    Returns:
+        list: Sorted boxes.
+    """
+    return sorted(
+        boxes,
+        key=lambda b: (b["page_num"], -round(b["bottom"], decimal_places), b["left"]),
+    )