natural-pdf 0.1.7__py3-none-any.whl → 0.1.9__py3-none-any.whl

docs/element-selection/index.md

@@ -1,229 +0,0 @@
-# Finding Elements with Selectors
-
-Natural PDF uses CSS-like selectors to find elements (text, lines, images, etc.) within a PDF page or document. This guide demonstrates how to use these selectors effectively.
-
-## Setup
-
-Let's load a sample PDF to work with. We'll use `01-practice.pdf` which has various elements.
-
-```python
-from natural_pdf import PDF
-
-# Load the PDF
-pdf = PDF("https://github.com/jsoma/natural-pdf/raw/refs/heads/main/pdfs/01-practice.pdf")
-
-# Select the first page
-page = pdf.pages[0]
-
-# Display the page
-page.show()
-```
-
-## Basic Element Finding
-
-The core methods are `find()` (returns the first match) and `find_all()` (returns all matches as an `ElementCollection`).
-
-The basic selector structure is `element_type[attribute_filter]:pseudo_class`.
-
-### Finding Text by Content
-
-```python
-# Find the first text element containing "Summary"
-summary_text = page.find('text:contains("Summary")')
-summary_text
-```
-
-```python
-# Find all text elements containing "Inadequate"
-contains_inadequate = page.find_all('text:contains("Inadequate")')
-len(contains_inadequate)
-```
-
-```python
-summary_text.highlight(label='summary')
-contains_inadequate.highlight(label="inadequate")
-page.to_image(width=700)
-```
-
-## Selecting by Element Type
-
-You can select specific types of elements found in PDFs.
-
-```python
-# Find all text elements
-all_text = page.find_all('text')
-len(all_text)
-```
-
-```python
-# Find all rectangle elements
-all_rects = page.find_all('rect')
-len(all_rects)
-```
-
-```python
-# Find all line elements
-all_lines = page.find_all('line')
-len(all_lines)
-```
-
-```python
-page.find_all('line').show()
-```
-
-## Filtering by Attributes
-
-Use square brackets `[]` to filter elements by their properties (attributes).
-
-### Common Attributes & Operators
-
-| Attribute     | Example Usage          | Operators | Notes |
-|---------------|------------------------|-----------|-------|
-| `size` (text) | `text[size>=12]`       | `>`, `<`, `>=`, `<=` | Font size in points |
-| `fontname`    | `text[fontname*=Bold]` | `=`, `*=`  | `*=` for contains substring |
-| `color` (text)| `text[color~=red]`     | `~=`      | Approx. match (name, rgb, hex) |
-| `width` (line)| `line[width>1]`        | `>`, `<`, `>=`, `<=` | Line thickness |
-| `source`      | `text[source=ocr]`     | `=`       | `pdf`, `ocr`, `detected` |
-| `type` (region)| `region[type=table]`  | `=`       | Layout analysis region type |
-
-```python
-# Find large text (size >= 11 points)
-page.find_all('text[size>=11]')
-```
-
-```python
-# Find text with 'Helvetica' in the font name
-page.find_all('text[fontname*=Helvetica]')
-```
-
-```python
-# Find red text (using approximate color match)
-# This PDF has text with color (0.8, 0.0, 0.0)
-red_text = page.find_all('text[color~=red]')
-```
-
-```python
-# Highlight the red text (ignoring existing highlights)
-red_text.show()
-```
-
-```python
-# Find thick lines (width >= 2)
-page.find_all('line[width>=2]')
-```
-
-## Using Pseudo-Classes
-
-Use colons `:` for special conditions (pseudo-classes).
-
-### Common Pseudo-Classes
-
-| Pseudo-Class          | Example Usage                           | Notes |
-|-----------------------|-----------------------------------------|-------|
-| `:contains('text')` | `text:contains('Report')`             | Finds elements containing specific text |
-| `:bold`               | `text:bold`                             | Finds text heuristically identified as bold |
-| `:italic`             | `text:italic`                           | Finds text heuristically identified as italic |
-| `:below(selector)`    | `text:below('line[width>=2]')`         | Finds elements physically below the reference element |
-| `:above(selector)`    | `text:above('text:contains("Summary")')`| Finds elements physically above the reference element |
-| `:left-of(selector)`  | `line:left-of('rect')`                 | Finds elements physically left of the reference element |
-| `:right-of(selector)` | `text:right-of('rect')`                | Finds elements physically right of the reference element |
-| `:near(selector)`     | `text:near('image')`                   | Finds elements physically near the reference element |
-
-*Note: Spatial pseudo-classes like `:below`, `:above` identify elements based on bounding box positions relative to the **first** element matched by the inner selector.*
-
-```python
-# Find bold text
-page.find_all('text:bold').show()
-```
-
-```python
-# Combine attribute and pseudo-class: bold text size >= 11
-page.find_all('text[size>=11]:bold')
-```
-
-### Spatial Pseudo-Classes Examples
-
-```python
-# Find the thick horizontal line first
-ref_line = page.find('line[width>=2]')
-
-# Find text elements strictly above that line
-text_above_line = page.find_all('text:above("line[width>=2]")')
-text_above_line
-```
-
-## Advanced Text Searching Options
-
-Pass options to `find()` or `find_all()` for more control over text matching.
-
-```python
-# Case-insensitive search for "summary"
-page.find_all('text:contains("summary")', case=False)
-```
-
-```python
-# Regular expression search for the inspection ID (e.g., INS-XXX...)
-# The ID is in the red text we found earlier
-page.find_all('text:contains("INS-\\w+")', regex=True)
-```
-
-```python
-# Combine regex and case-insensitivity
-page.find_all('text:contains("jungle health")', regex=True, case=False)
-```
-
-## Working with ElementCollections
-
-`find_all()` returns an `ElementCollection`, which is like a list but with extra PDF-specific methods.
-
-```python
-# Get all headings (using a selector for large, bold text)
-headings = page.find_all('text[size>=11]:bold')
-headings
-```
-
-```python
-# Get the first and last heading in reading order
-first = headings.first
-last = headings.last
-(first, last)
-```
-
-```python
-# Get the physically highest/lowest element in the collection
-highest = headings.highest()
-lowest = headings.lowest()
-(highest, lowest)
-```
-
-```python
-# Filter the collection further: headings containing "Service"
-service_headings = headings.find_all('text:contains("Service")')
-service_headings
-```
-
-```python
-# Extract text from all elements in the collection
-headings.extract_text()
-```
-
-*Remember: `.highest()`, `.lowest()`, `.leftmost()`, `.rightmost()` raise errors if the collection spans multiple pages.*
-
-## Font Variants
-
-Sometimes PDFs use font variants (prefixes like `AAAAAB+`) which can be useful for selection.
-
-```python
-# Find text elements with a specific font variant prefix (if any exist)
-# This example PDF doesn't use variants, but the selector works like this:
-page.find_all('text[font-variant=AAAAAB]')
-```
-
-## Next Steps
-
-Now that you can find elements, explore:
-
-- [Text Extraction](../text-extraction/index.ipynb): Get text content from found elements.
-- [Spatial Navigation](../pdf-navigation/index.ipynb): Use found elements as anchors to navigate (`.above()`, `.below()`, etc.).
-- [Working with Regions](../regions/index.ipynb): Define areas based on found elements.
-- [Visual Debugging](../visual-debugging/index.ipynb): Techniques for highlighting and visualizing elements.

docs/finetuning/index.md

@@ -1,176 +0,0 @@
-# OCR Fine-tuning
-
-While the built-in OCR engines (EasyOCR, PaddleOCR, Surya) offer good general performance, you might encounter situations where their accuracy isn't sufficient for your specific needs. This is often the case with:
-
-*   **Unique Fonts:** Documents using unusual or stylized fonts.
-*   **Specific Languages:** Languages or scripts not perfectly covered by the default models.
-*   **Low Quality Scans:** Noisy or degraded document images.
-*   **Specialized Layouts:** Text within complex tables, forms, or unusual arrangements.
-
-Fine-tuning allows you to adapt a pre-trained OCR recognition model to your specific data, significantly improving its accuracy on documents similar to those used for training.
-
-## Why Fine-tune?
-
--   **Higher Accuracy:** Achieve better text extraction results on your specific document types.
--   **Adaptability:** Train the model to recognize domain-specific terms, symbols, or layouts.
--   **Reduced Errors:** Minimize downstream errors in data extraction and processing pipelines.
-
-## Strategy: Detect + LLM Correct + Export
-
-Training an OCR model requires accurate ground truth: images of text snippets paired with their correct transcriptions. Manually creating this data is tedious. A powerful alternative leverages the strengths of different models:
-
-1.  **Detect Text Regions:** Use a robust local OCR engine (like Surya or PaddleOCR) primarily for its *detection* capabilities (`detect_only=True`). This identifies the *locations* of text on the page, even if the initial *recognition* isn't perfect. You can combine this with layout analysis or region selections (`.region()`, `.below()`, `.add_exclusion()`) to focus on the specific areas you care about.
-2.  **Correct with LLM:** For each detected text region, send the image snippet to a powerful Large Language Model (LLM) with multimodal capabilities (like GPT-4o, Claude 3.5 Sonnet/Haiku) using the `direct_ocr_llm` utility. The LLM performs high-accuracy OCR on the snippet, providing a "ground truth" transcription.
-3.  **Export for Fine-tuning:** Use the `PaddleOCRRecognitionExporter` to package the original image snippets (from step 1) along with their corresponding LLM-generated text labels (from step 2) into the specific format required by PaddleOCR for fine-tuning its *recognition* model.
-
-This approach combines the efficient spatial detection of local models with the superior text recognition of large generative models to create a high-quality fine-tuning dataset with minimal manual effort.
-
-## Example: Fine-tuning for Greek Spreadsheet Text
-
-Let's walk through an example of preparing data to fine-tune PaddleOCR for text from a scanned Greek spreadsheet, adapting the process described above.
-
-```python
-# --- 1. Setup and Load PDF ---
-from natural_pdf import PDF
-from natural_pdf.ocr.utils import direct_ocr_llm
-from natural_pdf.exporters import PaddleOCRRecognitionExporter
-import openai # Or your preferred LLM client library
-import os
-
-# Ensure your LLM API key is set (using environment variables is recommended)
-# os.environ["OPENAI_API_KEY"] = "sk-..." 
-# os.environ["ANTHROPIC_API_KEY"] = "sk-..." 
-
-# pdf_path = "path/to/your/document.pdf" 
-pdf_path = "pdfs/hidden/the-bad-one.pdf" # Replace with your PDF path
-pdf = PDF(pdf_path)
-
-# --- 2. (Optional) Exclude Irrelevant Areas ---
-# If the document has consistent headers, footers, or margins you want to ignore
-# Use exclusions *before* detection
-pdf.add_exclusion(lambda page: page.region(right=45)) # Exclude left margin/line numbers
-pdf.add_exclusion(lambda page: page.region(left=500)) # Exclude right margin
-
-# --- 3. Detect Text Regions ---
-# Use a good detection engine. Surya is often robust for line detection.
-# We only want the bounding boxes, not the initial (potentially inaccurate) OCR text.
-print("Detecting text regions...")
-# Process only a subset of pages for demonstration if needed
-for page in pdf.pages[:10]:
-    # Use a moderate resolution for detection; higher res used for LLM correction later
-    page.apply_ocr(engine='surya', resolution=120, detect_only=True) 
-print(f"Detection complete for {num_pages_to_process} pages.")
-
-# (Optional) Visualize detected boxes on a sample page
-# pdf.pages[9].find_all('text[source=ocr]').show() 
-
-# --- 4. Correct with LLM ---
-# Configure your LLM client (example using OpenAI client, adaptable for others)
-# For Anthropic: client = openai.OpenAI(base_url="https://api.anthropic.com/v1/", api_key=os.environ.get("ANTHROPIC_API_KEY"))
-client = openai.OpenAI(api_key=os.environ.get("OPENAI_API_KEY")) 
-
-# Craft a clear prompt for the LLM
-# Be as specific as possible! If it's in a specific language, what kinds
-# of characters, etc.
-prompt = """OCR this image patch. Return only the exact text content visible in the image. 
-Preserve original spelling, capitalization, punctuation, and symbols. 
-Do not add any explanatory text, translations, comments, or quotation marks around the result.
-The text is likely from a Greek document, potentially a spreadsheet, containing Modern Greek words or numbers."""
-
-# Define the correction function using direct_ocr_llm
-def correct_text_region(region):
-    # Use a high resolution for the LLM call for best accuracy
-    return direct_ocr_llm(
-        region, 
-        client, 
-        prompt=prompt, 
-        resolution=300, 
-        # model="claude-3-5-sonnet-20240620" # Example Anthropic model
-        model="gpt-4o-mini" # Example OpenAI model
-    )
-
-# Apply the correction function to the detected text regions
-print("Applying LLM correction to detected regions...")
-for page in pdf.pages[:num_pages_to_process]:
-    # This finds elements added by apply_ocr and passes their regions to 'correct_text_region'
-    # The returned text from the LLM replaces the original OCR text for these elements
-    # The source attribute is updated (e.g., to 'ocr-llm-corrected')
-    page.correct_ocr(correct_text_region) 
-print("LLM correction complete.")
-
-# --- 5. Export for PaddleOCR Fine-tuning ---
-print("Configuring exporter...")
-exporter = PaddleOCRRecognitionExporter(
-    # Select all of the non-blank OCR text
-    # Hopefully it's all been LLM-corrected! 
-    selector="text[source^=ocr][text!='']", 
-    resolution=300,     # Resolution for the exported image crops
-    padding=2,          # Add slight padding around text boxes
-    split_ratio=0.9,    # 90% for training, 10% for validation
-    random_seed=42,     # For reproducible train/val split
-    include_guide=True  # Include the Colab fine-tuning notebook
-)
-
-# Define the output directory
-output_directory = "./my_paddleocr_finetune_data"
-print(f"Exporting data to {output_directory}...")
-
-# Run the export process
-exporter.export(pdf, output_directory)
-
-print("Export complete.")
-print(f"Dataset ready for fine-tuning in: {output_directory}")
-print(f"Next step: Upload '{os.path.join(output_directory, 'fine_tune_paddleocr.ipynb')}' and the rest of the contents to Google Colab.")
-
-# --- Cleanup ---
-pdf.close() 
-```
-
-## Running the Fine-tuning
-
-The `PaddleOCRRecognitionExporter` automatically includes a Jupyter Notebook (`fine_tune_paddleocr.ipynb`) in the output directory. This notebook is pre-configured to guide you through the fine-tuning process on Google Colab (which offers free GPU access):
-
-1.  **Upload:** Upload the entire output directory (e.g., `my_paddleocr_finetune_data`) to your Google Drive or directly to your Colab instance.
-2.  **Open Notebook:** Open the `fine_tune_paddleocr.ipynb` notebook in Google Colab.
-3.  **Set Runtime:** Ensure the Colab runtime is set to use a GPU (Runtime -> Change runtime type -> GPU).
-4.  **Run Cells:** Execute the cells in the notebook sequentially. It will:
-    *   Install necessary libraries (PaddlePaddle, PaddleOCR).
-    *   Point the training configuration to your uploaded dataset (`images/`, `train.txt`, `val.txt`, `dict.txt`).
-    *   Download a pre-trained PaddleOCR model (usually a multilingual one).
-    *   Start the fine-tuning process using your data.
-    *   Save the fine-tuned model checkpoints.
-    *   Export the best model into an "inference format" suitable for use with `natural-pdf`.
-5.  **Download Model:** Download the resulting `inference_model` directory from Colab.
-
-## Using the Fine-tuned Model
-
-Once you have the `inference_model` directory, you can instruct `natural-pdf` to use it for OCR:
-
-```python
-from natural_pdf import PDF
-from natural_pdf.ocr import PaddleOCROptions
-
-# Path to the directory you downloaded from Colab
-finetuned_model_dir = "/path/to/your/downloaded/inference_model" 
-
-# Specify the path in PaddleOCROptions
-paddle_opts = PaddleOCROptions(
-    rec_model_dir=finetuned_model_dir,
-    rec_char_dict_path=os.path.join(finetuned_model_dir, 'your_dict.txt') # Or wherever your dict is
-    use_gpu=True # If using GPU locally
-)
-
-pdf = PDF("another-similar-document.pdf")
-page = pdf.pages[0]
-
-# Apply OCR using your fine-tuned model
-ocr_elements = page.apply_ocr(engine='paddle', options=paddle_opts)
-
-# Extract text using the improved results
-text = page.extract_text() 
-print(text)
-
-pdf.close()
-```
-
-By following this process, you can significantly enhance OCR performance on your specific documents using the power of fine-tuning. 

docs/index.md

@@ -1,170 +0,0 @@
-# Natural PDF
-
-A friendly library for working with PDFs, built on top of [pdfplumber](https://github.com/jsvine/pdfplumber).
-
-Natural PDF lets you find and extract content from PDFs using simple code that makes sense.
-
-- [Live demo here](https://colab.research.google.com/github/jsoma/natural-pdf/blob/main/notebooks/Examples.ipynb)
-
-<div style="max-width: 400px; margin: auto"><a href="assets/sample-screen.png"><img src="assets/sample-screen.png"></a></div>
-
-## Installation
-
-```
-pip install natural_pdf
-# All the extras
-pip install "natural_pdf[all]"
-```
-
-## Quick Example
-
-```python
-from natural_pdf import PDF
-
-pdf = PDF('document.pdf')
-page = pdf.pages[0]
-
-# Find the title and get content below it
-title = page.find('text:contains("Summary"):bold')
-content = title.below().extract_text()
-
-# Exclude everything above 'CONFIDENTIAL' and below last line on page
-page.add_exclusion(page.find('text:contains("CONFIDENTIAL")').above())
-page.add_exclusion(page.find_all('line')[-1].below())
-
-# Get the clean text without header/footer
-clean_text = page.extract_text()
-```
-
-## Key Features
-
-Here are a few highlights of what you can do:
-
-### Find Elements with Selectors
-
-Use CSS-like selectors to find text, shapes, and more.
-
-```python
-# Find bold text containing "Revenue"
-page.find('text:contains("Revenue"):bold').extract_text()
-
-# Find all large text
-page.find_all('text[size>=12]').extract_text()
-```
-
-[Learn more about selectors →](element-selection/index.ipynb)
-
-### Navigate Spatially
-
-Move around the page relative to elements, not just coordinates.
-
-```python
-# Extract text below a specific heading
-intro_text = page.find('text:contains("Introduction")').below().extract_text()
-
-# Extract text from one heading to the next
-methods_text = page.find('text:contains("Methods")').below(
-    until='text:contains("Results")'
-).extract_text()
-```
-
-[Explore more navigation methods →](pdf-navigation/index.ipynb)
-
-### Extract Clean Text
-
-Easily extract text content, automatically handling common page elements like headers and footers (if exclusions are set).
-
-```python
-# Extract all text from the page (respecting exclusions)
-page_text = page.extract_text()
-
-# Extract text from a specific region
-some_region = page.find(...)
-region_text = some_region.extract_text()
-```
-
-[Learn about text extraction →](text-extraction/index.ipynb)
-[Learn about exclusion zones →](regions/index.ipynb#exclusion-zones)
-
-### Apply OCR
-
-Extract text from scanned documents using various OCR engines.
-
-```python
-# Apply OCR using the default engine
-ocr_elements = page.apply_ocr()
-
-# Extract text (will use OCR results if available)
-text = page.extract_text()
-```
-
-[Explore OCR options →](ocr/index.md)
-
-### Analyze Document Layout
-
-Use AI models to detect document structures like titles, paragraphs, and tables.
-
-```python
-# Detect document structure
-page.analyze_layout()
-
-# Highlight titles and tables
-page.find_all('region[type=title]').highlight(color="purple")
-page.find_all('region[type=table]').highlight(color="blue")
-
-# Extract data from the first table
-table_data = page.find('region[type=table]').extract_table()
-```
-
-[Learn about layout models →](layout-analysis/index.ipynb)
-[Working with tables? →](tables/index.ipynb)
-
-### Document Question Answering
-
-Ask natural language questions directly to your documents.
-
-```python
-# Ask a question
-result = pdf.ask("What was the company's revenue in 2022?")
-if result.get("found", False):
-    print(f"Answer: {result['answer']}")
-```
-
-[Learn about Document QA →](document-qa/index.ipynb)
-
-### Visualize Your Work
-
-Debug and understand your extractions visually.
-
-```python
-# Highlight headings
-page.find_all('text[size>=14]').highlight(color="red", label="Headings")
-
-# Launch the interactive viewer (Jupyter)
-# Requires: pip install natural-pdf[interactive]
-page.viewer()
-
-# Or save an image
-# page.save_image("highlighted.png")
-```
-
-[See more visualization options →](visual-debugging/index.ipynb)
-
-## Documentation Topics
-
-Choose what you want to learn about:
-
-### Task-based Guides
-- [Getting Started](installation/index.md): Install the library and run your first extraction
-- [PDF Navigation](pdf-navigation/index.ipynb): Open PDFs and work with pages
-- [Element Selection](element-selection/index.ipynb): Find text and other elements using selectors
-- [Text Extraction](text-extraction/index.ipynb): Extract clean text from documents
-- [Regions](regions/index.ipynb): Work with specific areas of a page
-- [Visual Debugging](visual-debugging/index.ipynb): See what you're extracting
-- [OCR](ocr/index.md): Extract text from scanned documents
-- [Layout Analysis](layout-analysis/index.ipynb): Detect document structure
-- [Tables](tables/index.ipynb): Extract tabular data
-- [Document QA](document-qa/index.ipynb): Ask questions to your documents
-
-### Reference
-- [API Reference](api/index.md): Complete library reference

docs/installation/index.md

@@ -1,69 +0,0 @@
-# Getting Started with Natural PDF
-
-Let's get Natural PDF installed and run your first extraction.
-
-## Installation
-
-The base installation includes the core library and necessary AI dependencies (like PyTorch and Transformers):
-
-```bash
-pip install natural-pdf
-```
-
-### Optional Dependencies
-
-Natural PDF has modular dependencies for different features. Install them based on your needs:
-
-```bash
-# --- OCR Engines ---
-# Install support for EasyOCR
-pip install natural-pdf[easyocr]
-
-# Install support for PaddleOCR (requires paddlepaddle)
-pip install natural-pdf[paddle]
-
-# Install support for Surya OCR
-pip install natural-pdf[surya]
-
-# --- Layout Detection ---
-# Install support for YOLO layout model
-pip install natural-pdf[layout_yolo]
-
-# --- Interactive Widget ---
-# Install support for the interactive .viewer() widget in Jupyter
-pip install natural-pdf[interactive]
-
-# --- All Features ---
-# Install all optional dependencies
-pip install natural-pdf[all]
-```
-
-## Your First PDF Extraction
-
-Here's a quick example to make sure everything is working:
-
-```python
-from natural_pdf import PDF
-
-# Open a PDF
-pdf = PDF('your_document.pdf')
-
-# Get the first page
-page = pdf.pages[0]
-
-# Extract all text
-text = page.extract_text()
-print(text)
-
-# Find something specific
-title = page.find('text:bold')
-print(f"Found title: {title.text}")
-```
-
-## What's Next?
-
-Now that you have Natural PDF installed, you can:
-
-- Learn to [navigate PDFs](../pdf-navigation/index.ipynb)
-- Explore how to [select elements](../element-selection/index.ipynb)
-- See how to [extract text](../text-extraction/index.ipynb)