natural-pdf 0.2.16__tar.gz → 0.2.17__tar.gz

{natural_pdf-0.2.16 → natural_pdf-0.2.17}/.gitignore

@@ -7,6 +7,7 @@ bad-pdfs/
 bad-pdfs-analysis/**/*.pdf
 output
 output.pdf
+Demos.ipynb
 Untitled.ipynb
 conversation.md
 docs/tutorials/pdfs

{natural_pdf-0.2.16 → natural_pdf-0.2.17}/CLAUDE.md

@@ -61,6 +61,19 @@ Natural PDF is a Python library for intelligent PDF document processing that com
 - **Automatic conversion**: Elements from iterables are automatically converted to exclusion regions
 - **Backward compatibility**: Existing Region and callable exclusions continue to work unchanged
 
+### 6b. Multi-page Directional Navigation
+- **multipage parameter**: Directional methods now accept `multipage=True` to span pages
+  - `element.below(until="text:contains('End')", multipage=True)` searches across pages
+  - Returns `FlowRegion` when spanning multiple pages, `Region` when on single page
+  - Works with all directional methods: `.below()`, `.above()`, `.left()`, `.right()`
+- **Global auto_multipage option**: Set default behavior for all directional navigation
+  - `npdf.set_option('layout.auto_multipage', True)` enables multipage by default
+  - Individual calls can override with `multipage=False`
+- **Use cases**:
+  - Extract content between headers on different pages
+  - Find tables that span page boundaries
+  - Navigate document structure without manual page handling
+
 ### 7. Page Grouping with groupby()
 - **Simple grouping by selector text**: `pages.groupby('text[size=16]')` groups by header text
 - **Callable functions for complex logic**: `pages.groupby(lambda p: p.find('text:contains("CITY")').extract_text())`

{natural_pdf-0.2.16/natural_pdf.egg-info → natural_pdf-0.2.17}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: natural-pdf
-Version: 0.2.16
+Version: 0.2.17
 Summary: A more intuitive interface for working with PDFs
 Author-email: Jonathan Soma <jonathan.soma@gmail.com>
 License-Expression: MIT

{natural_pdf-0.2.16 → natural_pdf-0.2.17}/docs/element-selection/index.md

@@ -223,6 +223,116 @@ headings.extract_text()
 
 *Note: `.highest()`, `.lowest()`, etc. will complain if your collection spans multiple pages.*
 
+## Finding Elements with Statistical Properties
+
+Sometimes you need to find elements based on their extreme values - the leftmost text, the largest font, or the most common color. Natural PDF's aggregate selectors make this easy using statistical functions like `min()`, `max()`, and `avg()`.
+
+### Position-Based Selection
+
+```python
+# Find the leftmost text element on the page
+leftmost = page.find('text[x0=min()]')
+leftmost.show()
+```
+
+```python
+# Find the rightmost text (useful for page numbers)
+rightmost = page.find('text[x1=max()]')
+rightmost.show()
+```
+
+```python
+# Find text at the top and bottom of the page
+topmost = page.find('text[top=min()]')
+bottommost = page.find('text[bottom=max()]')
+```
+
+### Size and Dimension Selection
+
+```python
+# Find the largest text (often titles or headings)
+largest_text = page.find('text[size=max()]')
+print(f"Largest text: {largest_text.extract_text()} (size: {largest_text.size})")
+```
+
+```python
+# Find elements with average dimensions
+avg_width_text = page.find_all('text[width=avg()]')
+median_height_text = page.find_all('text[height=median()]')
+```
+
+### Finding Most Common Values
+
+The `mode()` function (or its alias `most_common()`) finds elements with the most frequently occurring value for any attribute:
+
+```python
+# Find text with the most common font size (body text)
+body_text = page.find_all('text[size=mode()]')
+print(f"Most common font size: {body_text.first.size if body_text else 'N/A'}")
+```
+
+```python
+# Find elements with the most common font name
+common_font = page.find_all('text[fontname=most_common()]')
+```
+
+### Color Proximity Matching
+
+For color attributes, you can find elements with colors closest to a target:
+
+```python
+# Find text closest to red
+red_text = page.find_all('text[color=closest("red")]')
+
+# Find rectangles with fill color closest to blue
+blue_rects = page.find_all('rect[fill=closest("#0000FF")]')
+
+# Works with any color format
+nearly_black = page.find_all('text[color=closest("rgb(10,10,10)")]')
+```
+
+### Combining Aggregate Conditions
+
+Multiple aggregate conditions create an intersection - elements must satisfy ALL conditions:
+
+```python
+# Find text that is both leftmost AND largest
+special_text = page.find('text[x0=min()][size=max()]')
+
+# Find the topmost element among large text
+topmost_large = page.find('text[size>12][top=min()]')
+```
+
+### Using Aggregates in Complex Selectors
+
+Aggregate functions work seamlessly with all Natural PDF features:
+
+```python
+# In OR selectors - find either the leftmost text OR the largest rectangle
+elements = page.find_all('text[x0=min()]|rect[width=max()]')
+
+# With spatial navigation
+element = page.find('text')
+# Navigate right until reaching the leftmost element
+right_region = element.right(until='text[x0=min()]')
+
+# With filters - leftmost among bold text
+leftmost_bold = page.find('text:bold[x0=min()]')
+```
+
+### Available Aggregate Functions
+
+| Function | Alias | Description | Works On |
+|----------|-------|-------------|----------|
+| `min()` | - | Minimum value | Numeric attributes |
+| `max()` | - | Maximum value | Numeric attributes |
+| `avg()` | `mean()` | Average/mean value | Numeric attributes |
+| `median()` | - | Median value | Numeric attributes |
+| `mode()` | `most_common()` | Most frequent value | Any attribute |
+| `closest(value)` | - | Closest match (colors only) | Color attributes |
+
+**Note**: Aggregates are calculated across all elements of the same type. For example, `text[x0=min()]` finds the minimum x0 among ALL text elements, not just those matching other filters.
+
 ## Dealing with Weird Font Names
 
 PDFs sometimes have bizarre font names that don't look like normal fonts. Don't worry - they're usually normal fonts with weird internal names.

natural_pdf-0.2.17/docs/guide_adjustment_stream.md

@@ -0,0 +1,90 @@
+# Guide Adjustment for Stream Extraction
+
+## Overview
+
+When using the `stream` extraction method (text-based edge detection) with explicit vertical guides, Natural PDF automatically adjusts guides that fall outside text bounds to ensure proper table extraction.
+
+## The Problem
+
+In pdfplumber's stream method, horizontal edges are only created where text exists. If vertical guides are placed outside the horizontal extent of text (e.g., at x=0 when text starts at x=51.6), these guides won't intersect with horizontal edges, causing missing columns in the extracted table.
+
+## The Solution
+
+Natural PDF automatically clips vertical guides to text bounds when:
+1. Using `method="stream"` or `horizontal_strategy="text"`
+2. Explicit vertical lines are provided
+3. Text elements exist in the region
+
+## Example
+
+```python
+from natural_pdf import PDF
+from natural_pdf.analyzers.guides import Guides
+
+# Load PDF and find headers
+pdf = PDF("document.pdf")
+page = pdf[0]
+headers = page.find_all("text[y<100]")  # Find header row
+
+# Create guides from headers
+guides = Guides(page)
+guides.vertical.from_headers(headers, margin=0)
+
+# Guides might include page boundaries (0, page.width)
+# which could be outside text bounds
+
+# Extract table - guides are automatically adjusted
+table = page.extract_table(method="stream", verticals=guides.vertical.data)
+
+# All columns including first and last are properly extracted
+```
+
+## How It Works
+
+1. **Detection**: When stream method is used with explicit vertical guides
+2. **Text Bounds**: The system finds all text elements and determines their bounding box
+3. **Adjustment**:
+   - Guides left of text bounds are moved to the left edge of text
+   - Guides right of text bounds are moved to the right edge of text
+   - Guides within text bounds remain unchanged
+4. **Extraction**: The adjusted guides are used for table extraction
+
+## When This Applies
+
+Guide adjustment happens when ALL of these conditions are met:
+- Extraction method is `pdfplumber` (or its aliases `stream`)
+- `horizontal_strategy` is `"text"` (text-based edge detection)
+- `vertical_strategy` is `"explicit"` (using provided guides)
+- `explicit_vertical_lines` are provided in table settings
+
+## Debugging
+
+Enable debug logging to see guide adjustments:
+
+```python
+import logging
+logging.basicConfig(level=logging.DEBUG)
+
+# Extract table - will show adjustment messages
+table = page.extract_table(method="stream", verticals=guides.vertical.data)
+```
+
+Example debug output:
+```
+Region (0, 0, 1224, 1584): Adjusted left guide from 0.0 to 51.6
+Region (0, 0, 1224, 1584): Adjusted right guide from 1224.0 to 1155.7
+Region (0, 0, 1224, 1584): Adjusted 26 guides for stream extraction. Text bounds: 51.6-1155.7
+```
+
+## Other Methods
+
+This adjustment only applies to stream/text-based extraction. When using:
+- `method="lattice"` (line-based): No adjustment, guides used as-is
+- `method="tatr"` or `method="text"`: Different extraction methods, guides not used
+
+## Best Practices
+
+1. **Use from_headers()**: This method creates appropriate guides for your content
+2. **Set margin=0**: For tables that span the full width of text
+3. **Verify with lattice first**: If your PDF has visible lines, lattice method may work better
+4. **Check text bounds**: Use `page.find_all("text").merge().bbox` to see text extent

natural_pdf-0.2.17/docs/guides_boundary_columns.md

@@ -0,0 +1,156 @@
+# Missing First/Last Columns in guides.extract_table()
+
+## Problem Description
+
+When using `guides.extract_table()`, the first and last columns may be missing from the extracted table. This happens because the `Guides.from_lines()` method by default does not include the page boundaries (x=0 and x=page.width) as vertical guides.
+
+### Example of the Issue
+
+```python
+# Default behavior - may miss boundary columns
+guides = Guides.from_lines(page)
+result = guides.extract_table()
+# First column "OFFICER" and last column may be missing
+```
+
+## Root Cause
+
+The `from_lines()` method detects lines in the PDF but doesn't automatically add guides at the page boundaries. If your table's first column starts at x=0 or the last column ends at x=page.width, and there are no explicit vertical lines at these positions, those columns won't have guides and will be excluded from extraction.
+
+## Solutions
+
+### Solution 1: Use the `outer` parameter (Recommended)
+
+The simplest fix is to use the `outer=True` parameter when creating guides:
+
+```python
+# Include outer boundaries when detecting lines
+guides = Guides.from_lines(page, outer=True)
+result = guides.extract_table()
+```
+
+### Solution 2: Use `include_outer_boundaries` in extract_table
+
+If you've already created guides, you can include boundaries during extraction:
+
+```python
+# Create guides normally
+guides = Guides.from_lines(page)
+
+# Include boundaries during extraction
+result = guides.extract_table(include_outer_boundaries=True)
+```
+
+### Solution 3: Manually add boundary guides
+
+For more control, you can manually add guides at the page boundaries:
+
+```python
+# Create guides
+guides = Guides.from_lines(page)
+
+# Add page boundaries
+guides.vertical.add([0, page.width])
+
+# Extract table
+result = guides.extract_table()
+```
+
+### Solution 4: Create guides from specific positions
+
+If you know the exact column positions:
+
+```python
+# Create guides with specific positions including boundaries
+guides = Guides(page)
+guides.vertical.add([0, 100, 200, 300, 400, page.width])
+guides.horizontal.from_lines(page)  # Get horizontal guides from lines
+
+result = guides.extract_table()
+```
+
+## Best Practices
+
+1. **Always use `outer=True`** when you expect table content at page boundaries:
+   ```python
+   guides = Guides.from_lines(page, outer=True)
+   ```
+
+2. **Check your guides** before extraction:
+   ```python
+   guides = Guides.from_lines(page)
+   print(f"Vertical guides: {guides.vertical.data}")
+   print(f"Page width: {page.width}")
+
+   # Check if boundaries are included
+   has_left = 0 in guides.vertical.data
+   has_right = page.width in guides.vertical.data
+   ```
+
+3. **Visualize guides** to debug issues:
+   ```python
+   # Show the page with guides overlaid
+   guides.show()
+   ```
+
+## Complete Example
+
+```python
+from natural_pdf import PDF
+from natural_pdf.analyzers import Guides
+
+# Load PDF
+pdf = PDF("document.pdf")
+page = pdf[0]
+
+# Method 1: Best practice - use outer=True
+guides = Guides.from_lines(page, outer=True)
+table = guides.extract_table()
+df = table.to_df()
+print(df)
+
+# Method 2: Alternative - use include_outer_boundaries
+guides = Guides.from_lines(page)
+table = guides.extract_table(include_outer_boundaries=True)
+df = table.to_df()
+print(df)
+
+# Method 3: Manual control
+guides = Guides.from_lines(page)
+if 0 not in guides.vertical.data:
+    guides.vertical.add([0])
+if page.width not in guides.vertical.data:
+    guides.vertical.add([page.width])
+table = guides.extract_table()
+df = table.to_df()
+print(df)
+```
+
+## When This Issue Occurs
+
+This issue typically occurs when:
+- Tables are designed with no margins (content starts at x=0)
+- Tables span the full page width
+- PDF generators don't include explicit border lines at page edges
+- Content is positioned exactly at page boundaries
+
+## Verification
+
+To verify if this is your issue:
+
+```python
+# Check text positions
+texts = page.find_all('text')
+min_x = min(t.x0 for t in texts)
+max_x = max(t.x1 for t in texts)
+
+print(f"Text spans from x={min_x} to x={max_x}")
+print(f"Page width: {page.width}")
+
+# Check guides
+guides = Guides.from_lines(page)
+print(f"First guide: {guides.vertical.data[0] if guides.vertical.data else 'None'}")
+print(f"Last guide: {guides.vertical.data[-1] if guides.vertical.data else 'None'}")
+
+# If min_x < first guide or max_x > last guide, you need boundaries
+```

{natural_pdf-0.2.16 → natural_pdf-0.2.17}/docs/quick-reference/index.md

@@ -74,6 +74,18 @@ page.find_all('text[source=pdf]')               # Original PDF text
 page.find_all('text[confidence>=0.8]')          # High-confidence OCR
 ```
 
+### Statistical Selectors (Aggregates)
+```py
+page.find('text[x0=min()]')                      # Leftmost text
+page.find('text[x1=max()]')                      # Rightmost text
+page.find('text[size=max()]')                    # Largest text
+page.find('text[width=avg()]')                   # Average width text
+page.find('text[height=median()]')               # Median height text
+page.find('text[fontname=mode()]')               # Most common font
+page.find('text[color=closest("red")]')          # Closest to red
+page.find('text[x0=min()][size=max()]')          # Leftmost AND largest
+```
+
 ## Essential Methods
 
 ### Finding Elements
@@ -86,10 +98,21 @@ element.previous()                              # Previous element
 
 ### Spatial Navigation
 ```py
-element.above(height=100)                       # Region above element
+# Smart defaults (new in 0.9.0)
+element.left()                                  # Default height='element' (matches element height)
+element.right()                                 # Default height='element' (matches element height)
+element.above()                                 # Default width='full' (full page width)
+element.below()                                 # Default width='full' (full page width)
+
+# Custom dimensions
+element.above(height=100)                       # Fixed height above
 element.below(until='line:horizontal')          # Below until boundary
-element.left(width=200)                         # Region to the left
-element.right()                                 # Region to the right
+element.left(width=200)                         # Fixed width to left
+element.right(height='full')                    # Full page height to right
+
+# Exclusion handling
+element.below(apply_exclusions=True)            # Skip exclusion zones
+element.expand('down', 50, apply_exclusions=True)  # Expand with exclusions
 ```
 
 ### Text Extraction
@@ -194,9 +217,16 @@ page.viewer()                                   # Launch interactive viewer (Jup
 
 ### Page-Level Exclusions
 ```py
-header = page.find('text:contains("CONFIDENTIAL")').above()
-page.add_exclusion(header)                      # Exclude from extraction
-page.clear_exclusions()                         # Remove exclusions
+# Smart exclusion behavior (new in 0.9.0)
+text_element = page.find('text:contains("CONFIDENTIAL")')
+page.add_exclusion(text_element)                # Excludes just the text bounding box
+
+# Traditional region exclusion
+header_region = page.find('text:contains("CONFIDENTIAL")').above()
+page.add_exclusion(header_region)               # Excludes entire region
+
+# Manage exclusions
+page.clear_exclusions()                         # Remove all exclusions
 text = page.extract_text(use_exclusions=False)  # Ignore exclusions
 ```
 
@@ -207,10 +237,27 @@ pdf.add_exclusion(
     lambda p: p.create_region(0, 0, p.width, p.height * 0.1),
     label="Header"
 )
+
+# Exclude specific text elements (new in 0.9.0)
+pdf.add_exclusion(
+    lambda p: p.find_all('text:contains("Header")'),  # Returns ElementCollection
+    label="Headers"
+)
 ```
 
 ## Configuration Options
 
+### Global Layout Settings
+```py
+import natural_pdf
+
+# Configure global directional offset (default: 5)
+natural_pdf.options.layout.directional_offset = 10  # Larger gap for directional methods
+
+# Reset to default
+natural_pdf.options.layout.directional_offset = 5
+```
+
 ### OCR Engines
 ```py
 from natural_pdf.ocr import EasyOCROptions, PaddleOCROptions
@@ -231,17 +278,17 @@ page.analyze_layout(engine='yolo', options=yolo_opts)
 
 ### Extract Inspection Report Data
 ```py
-# Find violation count
-violations = page.find('text:contains("Violation Count"):right(width=100)')
+# Find violation count (uses smart default height='element')
+violations = page.find('text:contains("Violation Count"):right()')
 
 # Get inspection number from the header box (regex search)
 inspection_num = page.find('text:contains("INS-[A-Z0-9]+")', regex=True)
 
-# Extract inspection date
+# Extract inspection date (custom width for wider field)
 inspection_date = page.find('text:contains("Date:"):right(width=150)')
 
-# Get site name (text to the right of "Site:")
-site_name = page.find('text:contains("Site:"):right(width=300)').extract_text()
+# Get site name (uses smart default height='element')
+site_name = page.find('text:contains("Site:"):right()').extract_text()
 ```
 
 ### Process Forms
@@ -250,9 +297,9 @@ site_name = page.find('text:contains("Site:"):right(width=300)').extract_text()
 page.add_exclusion(page.create_region(0, 0, page.width, 50))
 page.add_exclusion(page.create_region(0, page.height-50, page.width, page.height))
 
-# Extract form fields
+# Extract form fields (smart defaults + exclusion handling)
 fields = page.find_all('text:bold')
-values = [field.right(width=300).extract_text() for field in fields]
+values = [field.right(apply_exclusions=True).extract_text() for field in fields]
 ```
 
 ### Handle Scanned Documents

{natural_pdf-0.2.16 → natural_pdf-0.2.17}/docs/regions/index.md

@@ -43,7 +43,12 @@ mid_region.show(color="blue")
 
 ### Using Element Methods: `above()`, `below()`, `left()`, `right()`
 
-You can create regions relative to existing elements.
+You can create regions relative to existing elements. Natural PDF uses smart defaults for these directional methods:
+
+- **`.left()` and `.right()`**: Default to `height='element'` (matches the element's height)
+- **`.above()` and `.below()`**: Default to `width='full'` (full page width)
+
+These defaults match common use cases - when looking sideways you usually want the same height as your reference element, while looking up/down typically needs the full page width.
 
 ```python
 # Find a heading-like element
@@ -51,6 +56,7 @@ heading = page.find('text[size>=12]:bold')
 
 # Create a region below this heading element
 if heading:
+    # Uses default width='full' - extends across full page width
     region_below = heading.below()
 
     # Highlight the heading and the region below it
@@ -60,6 +66,24 @@ if heading:
         h.show()
 ```
 
+```python
+# Create regions to the left and right with smart defaults
+if heading:
+    # Default height='element' - matches heading height
+    region_left = heading.left()
+    region_right = heading.right()
+
+    # Or specify custom dimensions
+    region_left_tall = heading.left(height=200)  # 200px tall
+    region_right_full = heading.right(height='full')  # Full page height
+
+    with page.highlights() as h:
+        h.add(heading, color="red")
+        h.add(region_left, color="green", label="Left (element height)")
+        h.add(region_right, color="blue", label="Right (element height)")
+        h.show()
+```
+
 ```python
 # Create a region with height limit
 if heading:
@@ -212,6 +236,46 @@ with page.highlights() as h:
     h.show()
 ```
 
+### Global Offset Configuration
+
+You can configure global offsets that will be applied to all regions created with directional methods. This is useful for consistently adding padding or margins:
+
+```python
+from natural_pdf import PDF
+
+# Configure global offsets for all PDFs
+PDF.configure_offsets(
+    below_offset=5,     # Add 5px gap below elements
+    above_offset=5,     # Add 5px gap above elements
+    left_offset=2,      # Add 2px gap to the left
+    right_offset=2      # Add 2px gap to the right
+)
+
+# Now all directional methods will include these offsets
+heading = page.find('text:bold')
+if heading:
+    # This region will start 5px below the heading (not touching)
+    content_below = heading.below()
+
+    # This region will end 5px above the heading
+    content_above = heading.above(height=100)
+```
+
+```python
+# Reset to default offsets (all 0)
+PDF.configure_offsets(
+    below_offset=0,
+    above_offset=0,
+    left_offset=0,
+    right_offset=0
+)
+```
+
+These offsets are particularly useful when:
+- Extracting text that might be too close to headers/footers
+- Creating regions that need consistent spacing
+- Working with documents that have tight layouts
+
 ## Using Exclusion Zones with Regions
 
 Exclusion zones are regions that you want to ignore during operations like text extraction.