natural-pdf 0.1.7__py3-none-any.whl → 0.1.9__py3-none-any.whl

docs/layout-analysis/index.md

@@ -1,185 +0,0 @@
-# Document Layout Analysis
-
-Natural PDF can automatically detect the structure of a document (titles, paragraphs, tables, figures) using layout analysis models. This guide shows how to use this feature.
-
-## Setup
-
-We'll use a sample PDF that includes various layout elements.
-
-```python
-from natural_pdf import PDF
-
-pdf = PDF("https://github.com/jsoma/natural-pdf/raw/refs/heads/main/pdfs/01-practice.pdf")
-page = pdf.pages[0]
-
-page.to_image(width=700)
-```
-
-## Running Basic Layout Analysis
-
-Use the `analyze_layout()` method. By default, it uses the YOLO model.
-
-```python
-# Analyze the layout using the default engine (YOLO)
-# This adds 'region' elements to the page
-page.analyze_layout()
-```
-
-```python
-# Find all detected regions
-regions = page.find_all('region')
-len(regions) # Show how many regions were detected
-```
-
-```python
-first_region = regions[0]
-f"First region: type='{first_region.type}', confidence={first_region.confidence:.2f}"
-```
-
-## Visualizing Detected Layout
-
-Use `highlight()` or `show()` on the detected regions.
-
-```python
-# Highlight all detected regions, colored by type
-regions.highlight(group_by='type')
-page.to_image(width=700)
-```
-
-## Finding Specific Region Types
-
-Use attribute selectors to find regions of a specific type.
-
-```python
-# Find all detected titles
-titles = page.find_all('region[type=title]')
-titles
-```
-
-```python
-titles.show()
-```
-
-```python
-page.find_all('region[type=table]').show()
-```
-
-## Working with Layout Regions
-
-Detected regions are like any other `Region` object. You can extract text, find elements within them, etc.
-
-```python
-page.find('region[type=table]').extract_text(layout=True)
-```
-
-## Using Different Layout Models
-
-Natural PDF supports multiple engines (`yolo`, `paddle`, `tatr`). Specify the engine when calling `analyze_layout`.
-
-*Note: Using different engines requires installing the corresponding extras (e.g., `natural-pdf[layout_paddle]`).* `yolo` is the default.
-
-```python
-page.clear_detected_layout_regions()
-page.clear_highlights()
-
-page.analyze_layout(engine="paddle")
-page.find_all('region[model=paddle]').highlight(group_by='region_type')
-page.to_image(width=700)
-```
-
-```python
-# Analyze using Table Transformer (TATR) - specialized for tables
-page.clear_detected_layout_regions()
-page.clear_highlights()
-
-page.analyze_layout(engine="tatr")
-page.find_all('region[model=tatr]').highlight(group_by='region_type')
-page.to_image(width=700)
-```
-
-```python
-# Analyze using Table Transformer (TATR) - specialized for tables
-page.clear_detected_layout_regions()
-page.clear_highlights()
-
-page.analyze_layout(engine="docling")
-page.find_all('region[model=docling]').highlight(group_by='region_type')
-page.to_image(width=700)
-```
-
-```python
-# Analyze using Table Transformer (TATR) - specialized for tables
-page.clear_detected_layout_regions()
-page.clear_highlights()
-
-page.analyze_layout(engine="surya")
-page.find_all('region[model=surya]').highlight(group_by='region_type')
-page.to_image(width=700)
-```
-
-*Note: Calling `analyze_layout` multiple times (even with the same engine) can add duplicate regions. You might want to use `page.clear_detected_layout_regions()` first, or filter by model using `region[model=yolo]`.* 
-
-## Controlling Confidence Threshold
-
-Filter detections by their confidence score.
-
-```python
-# Re-run YOLO analysis (clearing previous results might be good practice)
-page.clear_detected_layout_regions()
-page.analyze_layout(engine="yolo")
-
-# Find only high-confidence regions (e.g., >= 0.8)
-high_conf_regions = page.find_all('region[confidence>=0.8]')
-len(high_conf_regions)
-```
-
-## Table Structure with TATR
-
-The TATR engine provides detailed table structure elements (`table`, `table-row`, `table-column`, `table-column-header`). This is very useful for precise table extraction.
-
-```python
-# Ensure TATR analysis has been run
-page.clear_detected_layout_regions()
-page.clear_highlights()
-
-page.analyze_layout(engine="tatr")
-page.find_all('region[model=tatr]').highlight(group_by='region_type')
-page.to_image(width=700)
-```
-
-```python
-# Find different structural elements from TATR
-tables = page.find_all('region[type=table][model=tatr]')
-rows = page.find_all('region[type=table-row][model=tatr]')
-cols = page.find_all('region[type=table-column][model=tatr]')
-hdrs = page.find_all('region[type=table-column-header][model=tatr]')
-
-f"Found: {len(tables)} tables, {len(rows)} rows, {len(cols)} columns, {len(hdrs)} headers (from TATR)"
-```
-
-### Enhanced Table Extraction with TATR
-
-When a `region[type=table]` comes from the TATR model, `extract_table()` can use the underlying row/column structure for more robust extraction.
-
-```python
-# Find the TATR table region again
-tatr_table = page.find('region[type=table][model=tatr]')
-
-# This extraction uses the detected rows/columns
-tatr_table.extract_table()
-```
-
-if you'd like the normal approach instead of the "intelligent" one, you can ask for pdfplumber.
-
-```python
-# This extraction uses the detected rows/columns
-tatr_table.extract_table(method='pdfplumber')
-```
-
-## Next Steps
-
-Layout analysis provides regions that you can use for:
-
-- [Table Extraction](../tables/index.ipynb): Especially powerful with TATR regions.
-- [Text Extraction](../text-extraction/index.ipynb): Extract text only from specific region types (e.g., paragraphs).
-- [Document QA](../document-qa/index.ipynb): Focus question answering on specific detected regions.

docs/ocr/index.md

@@ -1,209 +0,0 @@
-# OCR Integration
-
-Natural PDF includes OCR (Optical Character Recognition) to extract text from scanned documents or images embedded in PDFs.
-
-## OCR Engine Comparison
-
-Natural PDF supports multiple OCR engines:
-
-| Feature              | EasyOCR                            | PaddleOCR                                | Surya OCR                             |
-|----------------------|------------------------------------|------------------------------------------|---------------------------------------|
-| **Installation**     | `natural-pdf[easyocr]`             | `natural-pdf[paddle]`                    | `natural-pdf[surya]`                  |
-| **Primary Strength** | Good general performance, simpler  | Excellent Asian language, speed        | High accuracy, multilingual lines     |
-| **Speed**            | Moderate                           | Fast                                     | Moderate (GPU recommended)            |
-| **Memory Usage**     | Higher                             | Efficient                                | Higher (GPU recommended)            |
-| **Paragraph Detect** | Yes (via option)                   | No                                       | No (focuses on lines)                 |
-| **Handwritten**      | Better support                     | Limited                                  | Limited                               |
-| **Small Text**       | Moderate                           | Good                                     | Good                                  |
-| **When to Use**      | General documents, handwritten text| Asian languages, speed-critical tasks    | Highest accuracy needed, line-level   |
-
-## Basic OCR Usage
-
-Apply OCR directly to a page or region:
-
-```python
-from natural_pdf import PDF
-
-# Assume 'page' is a Page object from a PDF
-page = pdf.pages[0]
-
-# Apply OCR using the default engine (or specify one)
-ocr_elements = page.apply_ocr(languages=['en'])
-
-# Extract text (will use the results from apply_ocr if run previously)
-text = page.extract_text()
-print(text)
-```
-
-## Configuring OCR
-
-Specify the engine and basic options directly:
-
-## OCR Configuration
-
-```python
-# Use PaddleOCR for Chinese and English
-ocr_elements = page.apply_ocr(engine='paddle', languages=['zh-cn', 'en'])
-
-# Use EasyOCR with a lower confidence threshold
-ocr_elements = page.apply_ocr(engine='easyocr', languages=['en'], min_confidence=0.3)
-```
-
-For advanced, engine-specific settings, use the Options classes:
-
-```python
-from natural_pdf.ocr import PaddleOCROptions, EasyOCROptions, SuryaOCROptions
-
-# --- Configure PaddleOCR ---
-paddle_opts = PaddleOCROptions(
-    languages=['en', 'zh-cn'],
-    use_gpu=True,         # Explicitly enable GPU if available
-    use_angle_cls=False,  # Disable text direction classification (if text is upright)
-    det_db_thresh=0.25,   # Lower detection threshold (more boxes, potentially noisy)
-    rec_batch_num=16      # Increase recognition batch size for potential speedup on GPU
-    # rec_char_dict_path='/path/to/custom_dict.txt' # Optional: Path to a custom character dictionary
-    # See PaddleOCROptions documentation or source code for all parameters
- )
-ocr_elements = page.apply_ocr(engine='paddle', options=paddle_opts)
-
-# --- Configure EasyOCR ---
-easy_opts = EasyOCROptions(
-    languages=['en', 'fr'],
-    gpu=True,            # Explicitly enable GPU if available
-    paragraph=True,      # Group results into paragraphs (if structure is clear)
-    detail=1,            # Ensure bounding boxes are returned (required)
-    text_threshold=0.6,  # Confidence threshold for text detection (adjust based on tuning table)
-    link_threshold=0.4,  # Standard EasyOCR param, uncomment if confirmed in wrapper
-    low_text=0.4,        # Standard EasyOCR param, uncomment if confirmed in wrapper
-    batch_size=8         # Processing batch size (adjust based on memory)
-    # See EasyOCROptions documentation or source code for all parameters
- )
-ocr_elements = page.apply_ocr(engine='easyocr', options=easy_opts)
-
-# --- Configure Surya OCR ---
-# Surya focuses on line detection and recognition
-surya_opts = SuryaOCROptions(
-    languages=['en', 'de'], # Specify languages for recognition
-    # device='cuda',       # Use GPU ('cuda') or CPU ('cpu') <-- Set via env var TORCH_DEVICE
-    min_confidence=0.4   # Example: Adjust minimum confidence for results
-    # Core Surya options like device, batch size, and thresholds are typically
-    # set via environment variables (see note below).
-)
-ocr_elements = page.apply_ocr(engine='surya', options=surya_opts)
-```
-
-## Applying OCR Directly
-
-The `page.apply_ocr(...)` and `region.apply_ocr(...)` methods are the primary way to run OCR:
-
-```python
-# Apply OCR to a page and get the OCR elements
-ocr_elements = page.apply_ocr(engine='easyocr')
-print(f"Found {len(ocr_elements)} text elements via OCR")
-
-# Apply OCR to a specific region
-title = page.find('text:contains("Title")')
-content_region = title.below(height=300)
-region_ocr_elements = content_region.apply_ocr(engine='paddle', languages=['en'])
-```
-
-## OCR Engines
-
-Choose the engine best suited for your document and language requirements using the `engine` parameter in `apply_ocr`.
-
-## Finding and Working with OCR Text
-
-After applying OCR, work with the text just like regular text:
-
-```python
-# Find all OCR text elements
-ocr_text = page.find_all('text[source=ocr]')
-
-# Find high-confidence OCR text
-high_conf = page.find_all('text[source=ocr][confidence>=0.8]')
-
-# Extract text only from OCR elements
-ocr_text_content = page.find_all('text[source=ocr]').extract_text()
-
-# Filter OCR text by content
-names = page.find_all('text[source=ocr]:contains("Smith")', case=False)
-```
-
-## Visualizing OCR Results
-
-See OCR results to help debug issues:
-
-```python
-# Apply OCR 
-ocr_elements = page.apply_ocr()
-
-# Highlight all OCR elements
-for element in ocr_elements:
-    # Color based on confidence
-    if element.confidence >= 0.8:
-        color = "green"  # High confidence
-    elif element.confidence >= 0.5:
-        color = "yellow"  # Medium confidence
-    else:
-        color = "red"  # Low confidence
-        
-    element.highlight(color=color, label=f"OCR ({element.confidence:.2f})")
-
-# Get the visualization as an image
-image = page.to_image(labels=True)
-# Just return the image in a Jupyter cell
-image
-
-# Highlight only high-confidence elements
-high_conf = page.find_all('text[source=ocr][confidence>=0.8]')
-high_conf.highlight(color="green", label="High Confidence OCR")
-```
-
-## Detect + LLM OCR
-
-Sometimes you have a difficult piece of content where you need to use a local model to identify the content, then send it off in pieces to be identified by the LLM. You can do this with Natural PDF!
-
-```python
-from natural_pdf import PDF
-from natural_pdf.ocr.utils import direct_ocr_llm
-import openai
-
-pdf = PDF("needs-ocr.pdf")
-page = pdf.pages[0]
-
-# Detect
-page.apply_ocr('paddle', resolution=120, detect_only=True)
-
-# Build the framework
-client = openai.OpenAI(base_url="https://api.anthropic.com/v1/",  api_key='sk-XXXXX')
-prompt = """OCR this image. Return only the exact text from the image. Include misspellings,
-punctuation, etc. Do not surround it with quotation marks. Do not include translations or comments.
-The text is from a Greek spreadsheet, so most likely content is Modern Greek or numeric."""
-
-# This returns the cleaned-up text
-def correct(region):
-    return direct_ocr_llm(region, client, prompt=prompt, resolution=300, model="claude-3-5-haiku-20241022")
-
-# Run 'correct' on each text element
-page.correct_ocr(correct)
-
-# You're done!
-```
-
-## Debugging OCR
-
-```python
-from natural_pdf.utils.packaging import create_correction_task_package
-
-create_correction_task_package(pdf, "original.zip", overwrite=True)
-```
-
-This will at *some point* be official-ized, but for now you can look at `templates/spa` and see the correction package.
-
-## Next Steps
-
-With OCR capabilities, you can explore:
-
-- [Layout Analysis](../layout-analysis/index.ipynb) for automatically detecting document structure
-- [Document QA](../document-qa/index.ipynb) for asking questions about your documents
-- [Visual Debugging](../visual-debugging/index.ipynb) for visualizing OCR results

docs/pdf-navigation/index.ipynb

@@ -1,314 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "id": "bba1860e",
-   "metadata": {},
-   "source": [
-    "# PDF Navigation\n",
-    "\n",
-    "This guide covers the basics of working with PDFs in Natural PDF - opening documents, accessing pages, and navigating through content.\n",
-    "\n",
-    "## Opening a PDF\n",
-    "\n",
-    "The main entry point to Natural PDF is the `PDF` class:"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 1,
-   "id": "56d12ab5",
-   "metadata": {
-    "execution": {
-     "iopub.execute_input": "2025-04-03T14:50:38.434157Z",
-     "iopub.status.busy": "2025-04-03T14:50:38.433170Z",
-     "iopub.status.idle": "2025-04-03T14:50:49.768101Z",
-     "shell.execute_reply": "2025-04-03T14:50:49.767384Z"
-    }
-   },
-   "outputs": [],
-   "source": [
-    "from natural_pdf import PDF\n",
-    "\n",
-    "# Open a PDF file\n",
-    "pdf = PDF(\"https://github.com/jsoma/natural-pdf/raw/refs/heads/main/pdfs/0500000US42001.pdf\")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "c425482a",
-   "metadata": {},
-   "source": [
-    "## Accessing Pages\n",
-    "\n",
-    "Once you have a PDF object, you can access its pages:"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 2,
-   "id": "a3405aa9",
-   "metadata": {
-    "execution": {
-     "iopub.execute_input": "2025-04-03T14:50:49.770604Z",
-     "iopub.status.busy": "2025-04-03T14:50:49.770419Z",
-     "iopub.status.idle": "2025-04-03T14:50:50.700808Z",
-     "shell.execute_reply": "2025-04-03T14:50:50.699634Z"
-    }
-   },
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "This PDF has 153 pages\n",
-      "Page 1 has 985 characters\n",
-      "Page 2 has 778 characters\n",
-      "Page 3 has 522 characters\n",
-      "Page 4 has 984 characters\n",
-      "Page 5 has 778 characters\n",
-      "Page 6 has 523 characters\n"
-     ]
-    },
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Page 7 has 982 characters\n",
-      "Page 8 has 772 characters\n",
-      "Page 9 has 522 characters\n",
-      "Page 10 has 1008 characters\n"
-     ]
-    },
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Page 11 has 796 characters\n",
-      "Page 12 has 532 characters\n",
-      "Page 13 has 986 characters\n",
-      "Page 14 has 780 characters\n",
-      "Page 15 has 523 characters\n",
-      "Page 16 has 990 characters\n",
-      "Page 17 has 782 characters\n"
-     ]
-    },
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Page 18 has 520 characters\n",
-      "Page 19 has 1006 characters\n",
-      "Page 20 has 795 characters\n"
-     ]
-    }
-   ],
-   "source": [
-    "# Get the total number of pages\n",
-    "num_pages = len(pdf)\n",
-    "print(f\"This PDF has {num_pages} pages\")\n",
-    "\n",
-    "# Get a specific page (0-indexed)\n",
-    "first_page = pdf.pages[0]\n",
-    "last_page = pdf.pages[-1]\n",
-    "\n",
-    "# Iterate through the first 20 pages\n",
-    "for page in pdf.pages[:20]:\n",
-    "    print(f\"Page {page.number} has {len(page.extract_text())} characters\")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "2eca7327",
-   "metadata": {},
-   "source": [
-    "## Page Properties\n",
-    "\n",
-    "Each `Page` object has useful properties:"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 3,
-   "id": "348f28d7",
-   "metadata": {
-    "execution": {
-     "iopub.execute_input": "2025-04-03T14:50:50.713325Z",
-     "iopub.status.busy": "2025-04-03T14:50:50.711638Z",
-     "iopub.status.idle": "2025-04-03T14:50:50.738737Z",
-     "shell.execute_reply": "2025-04-03T14:50:50.726839Z"
-    }
-   },
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "612 792\n",
-      "20\n",
-      "19\n"
-     ]
-    }
-   ],
-   "source": [
-    "# Page dimensions in points (1/72 inch)\n",
-    "print(page.width, page.height)\n",
-    "\n",
-    "# Page number (1-indexed as shown in PDF viewers)\n",
-    "print(page.number)\n",
-    "\n",
-    "# Page index (0-indexed position in the PDF)\n",
-    "print(page.index)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "c7cf1839",
-   "metadata": {},
-   "source": [
-    "## Working Across Pages\n",
-    "\n",
-    "Natural PDF makes it easy to work with content across multiple pages:"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 4,
-   "id": "71a8f1ec",
-   "metadata": {
-    "execution": {
-     "iopub.execute_input": "2025-04-03T14:50:50.765495Z",
-     "iopub.status.busy": "2025-04-03T14:50:50.764444Z",
-     "iopub.status.idle": "2025-04-03T14:50:57.735494Z",
-     "shell.execute_reply": "2025-04-03T14:50:57.726489Z"
-    }
-   },
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "<natural_pdf.core.pdf.PDF at 0x1045224d0>"
-      ]
-     },
-     "execution_count": 4,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "# Extract text from all pages\n",
-    "all_text = pdf.extract_text()\n",
-    "\n",
-    "# Find elements across all pages\n",
-    "all_headings = pdf.find_all('text[size>=14]:bold')\n",
-    "\n",
-    "# Add exclusion zones to all pages (like headers/footers)\n",
-    "pdf.add_exclusion(\n",
-    "    lambda page: page.find('text:contains(\"CONFIDENTIAL\")').above() if page.find('text:contains(\"CONFIDENTIAL\")') else None,\n",
-    "    label=\"header\"\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "e18051a4",
-   "metadata": {},
-   "source": [
-    "## The Page Collection\n",
-    "\n",
-    "The `pdf.pages` object is a `PageCollection` that allows batch operations on pages:"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 5,
-   "id": "e5f1c662",
-   "metadata": {
-    "execution": {
-     "iopub.execute_input": "2025-04-03T14:50:57.752240Z",
-     "iopub.status.busy": "2025-04-03T14:50:57.751868Z",
-     "iopub.status.idle": "2025-04-03T14:50:57.770738Z",
-     "shell.execute_reply": "2025-04-03T14:50:57.759415Z"
-    }
-   },
-   "outputs": [],
-   "source": [
-    "# Extract text from specific pages\n",
-    "text = pdf.pages[2:5].extract_text()\n",
-    "\n",
-    "# Find elements across specific pages\n",
-    "elements = pdf.pages[2:5].find_all('text:contains(\"Annual Report\")')"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "9713e392",
-   "metadata": {},
-   "source": [
-    "## Document Sections Across Pages\n",
-    "\n",
-    "You can extract sections that span across multiple pages:"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 6,
-   "id": "d5b89a2b",
-   "metadata": {
-    "execution": {
-     "iopub.execute_input": "2025-04-03T14:50:57.782621Z",
-     "iopub.status.busy": "2025-04-03T14:50:57.781776Z",
-     "iopub.status.idle": "2025-04-03T14:50:57.811508Z",
-     "shell.execute_reply": "2025-04-03T14:50:57.805310Z"
-    }
-   },
-   "outputs": [],
-   "source": [
-    "# Get sections with headings as section starts\n",
-    "sections = pdf.pages.get_sections(\n",
-    "    start_elements='text[size>=14]:bold',\n",
-    "    new_section_on_page_break=False\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "f51594ce",
-   "metadata": {},
-   "source": [
-    "## Next Steps\n",
-    "\n",
-    "Now that you know how to navigate PDFs, you can:\n",
-    "\n",
-    "- [Find elements using selectors](../element-selection/index.ipynb)\n",
-    "- [Extract text from your documents](../text-extraction/index.ipynb)\n",
-    "- [Work with specific regions](../regions/index.ipynb)"
-   ]
-  }
- ],
- "metadata": {
-  "jupytext": {
-   "cell_metadata_filter": "-all",
-   "main_language": "python",
-   "notebook_metadata_filter": "-all",
-   "text_representation": {
-    "extension": ".md",
-    "format_name": "markdown"
-   }
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.10.13"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}