natural-pdf 0.1.7__py3-none-any.whl → 0.1.9__py3-none-any.whl

docs/tutorials/05-excluding-content.md

@@ -1,109 +0,0 @@
-# Excluding Content (Headers/Footers)
-
-Often, PDFs have repeating headers or footers on every page that you want to ignore when extracting the main content. `natural-pdf` allows you to define exclusion regions.
-
-We'll use a different PDF for this example, which has a distinct header and footer section: `0500000US42007.pdf`.
-
-```python
-#%pip install "natural-pdf[all]"
-```
-
-
-```python
-from natural_pdf import PDF
-
-pdf_url = "https://github.com/jsoma/natural-pdf/raw/refs/heads/main/pdfs/0500000US42007.pdf"
-
-# Load the PDF
-pdf = PDF(pdf_url)
-page = pdf.pages[0]
-
-# Let's see the bottom part of the text WITHOUT exclusions
-# It likely contains page numbers or other footer info.
-full_text_unfiltered = page.extract_text()
-
-# Show the last 200 characters (likely containing footer text)
-full_text_unfiltered[-200:]
-```
-
-## Approach 1: Excluding a Fixed Area
-
-A simple way to exclude headers or footers is to define a fixed region based on page coordinates. Let's exclude the bottom 200 pixels of the page.
-
-```python
-from natural_pdf import PDF
-
-pdf_url = "https://github.com/jsoma/natural-pdf/raw/refs/heads/main/pdfs/0500000US42007.pdf"
-pdf = PDF(pdf_url)
-
-# Define the exclusion region on every page using a lambda function
-footer_height = 200
-pdf.add_exclusion(
-    lambda page: page.region(top=page.height - footer_height),
-    label="Bottom 200pt Footer"
-)
-
-# Now extract text from the first page again, exclusions are active by default
-page = pdf.pages[0]
-
-# Visualize the excluded area
-footer_region_viz = page.region(top=page.height - footer_height)
-footer_region_viz.highlight(label="Excluded Footer Area")
-page.to_image()
-```
-
-```python
-filtered_text = page.extract_text() # use_exclusions=True is default
-
-# Show the last 200 chars with footer area excluded
-filtered_text[-200:]
-```
-
-This method is simple but might cut off content if the footer height varies or content extends lower on some pages.
-
-## Approach 2: Excluding Based on Elements
-
-A more robust way is to find specific elements that reliably mark the start of the footer (or end of the header) and exclude everything below (or above) them. In `Examples.md`, the footer was defined as everything below the last horizontal line.
-
-```python
-from natural_pdf import PDF
-
-pdf_url = "https://github.com/jsoma/natural-pdf/raw/refs/heads/main/pdfs/0500000US42007.pdf"
-pdf = PDF(pdf_url)
-page = pdf.pages[0] # Get page for finding elements
-
-# Find the last horizontal line on the first page
-# We'll use this logic to define our exclusion for all pages
-last_line = page.find_all('line')[-1]
-
-# Define the exclusion function using a lambda
-# This finds the last line on *each* page and excludes below it
-pdf.add_exclusion(
-    lambda p: p.find_all('line')[-1].below(),
-    label="Element-Based Footer"
-)
-
-# Extract text again, with the element-based exclusion active
-filtered_text_element = page.extract_text()
-
-# Show the last 200 chars with element-based footer exclusion
-"Element-Based Excluded (last 200 chars): " + filtered_text_element[-200:]
-
-# Visualize the element-based exclusion area
-page.clear_highlights()
-# Need to find the region again for visualization
-footer_boundary = page.find_all('line')[-1]
-footer_region_element = footer_boundary.below()
-footer_region_element.show(label="Excluded Footer Area (Element)")
-page.to_image()
-```
-
-This element-based approach is usually more reliable as it adapts to the content's position, but it depends on finding consistent boundary elements (like lines or specific text markers).
-
-<div class="admonition note">
-<p class="admonition-title">Applying Exclusions</p>
-
-    *   `pdf.add_exclusion(func)` applies the exclusion function (which takes a page and returns a region) to *all* pages in the PDF.
-    *   `page.add_exclusion(region)` adds an exclusion region only to that specific page.
-    *   `extract_text(use_exclusions=False)` can be used to temporarily disable exclusions.
-</div> 

docs/tutorials/06-document-qa.ipynb

@@ -1,332 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "id": "a0347143",
-   "metadata": {},
-   "source": [
-    "# Document Question Answering (QA)\n",
-    "\n",
-    "Sometimes, instead of searching for specific text patterns, you just want to ask the document a question directly. `natural-pdf` includes an extractive Question Answering feature.\n",
-    "\n",
-    "\"Extractive\" means it finds the literal answer text within the document, rather than generating a new answer or summarizing.\n",
-    "\n",
-    "Let's ask our `01-practice.pdf` a few questions."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 1,
-   "id": "883ee2f6",
-   "metadata": {
-    "execution": {
-     "iopub.execute_input": "2025-04-21T21:24:21.452596Z",
-     "iopub.status.busy": "2025-04-21T21:24:21.452452Z",
-     "iopub.status.idle": "2025-04-21T21:24:21.457320Z",
-     "shell.execute_reply": "2025-04-21T21:24:21.456901Z"
-    }
-   },
-   "outputs": [],
-   "source": [
-    "#%pip install \"natural-pdf[all]\""
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 2,
-   "id": "abecda7a",
-   "metadata": {
-    "execution": {
-     "iopub.execute_input": "2025-04-21T21:24:21.459009Z",
-     "iopub.status.busy": "2025-04-21T21:24:21.458879Z",
-     "iopub.status.idle": "2025-04-21T21:24:30.401038Z",
-     "shell.execute_reply": "2025-04-21T21:24:30.400660Z"
-    }
-   },
-   "outputs": [
-    {
-     "name": "stderr",
-     "output_type": "stream",
-     "text": [
-      "Device set to use cpu\n"
-     ]
-    },
-    {
-     "data": {
-      "text/plain": [
-       "{'answer': 'February 3, 1905',\n",
-       " 'confidence': 0.9979940056800842,\n",
-       " 'start': 6,\n",
-       " 'end': 6,\n",
-       " 'found': True,\n",
-       " 'page_num': 0,\n",
-       " 'source_elements': <ElementCollection[TextElement](count=1)>}"
-      ]
-     },
-     "execution_count": 2,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "from natural_pdf import PDF\n",
-    "\n",
-    "# Load the PDF and get the page\n",
-    "pdf = PDF(\"https://github.com/jsoma/natural-pdf/raw/refs/heads/main/pdfs/01-practice.pdf\")\n",
-    "page = pdf.pages[0]\n",
-    "\n",
-    "# Ask about the date\n",
-    "question_1 = \"What is the inspection date?\"\n",
-    "answer_1 = page.ask(question_1)\n",
-    "\n",
-    "# The result is a dictionary with the answer, confidence, etc.\n",
-    "answer_1"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 3,
-   "id": "d6b5a66f",
-   "metadata": {
-    "execution": {
-     "iopub.execute_input": "2025-04-21T21:24:30.403287Z",
-     "iopub.status.busy": "2025-04-21T21:24:30.402917Z",
-     "iopub.status.idle": "2025-04-21T21:24:31.285240Z",
-     "shell.execute_reply": "2025-04-21T21:24:31.284848Z"
-    }
-   },
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "{'answer': 'Jungle Health and Safety Inspection Service',\n",
-       " 'confidence': 0.9988948106765747,\n",
-       " 'start': 0,\n",
-       " 'end': 0,\n",
-       " 'found': True,\n",
-       " 'page_num': 0,\n",
-       " 'source_elements': <ElementCollection[TextElement](count=1)>}"
-      ]
-     },
-     "execution_count": 3,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "# Ask about the company name\n",
-    "question_2 = \"What company was inspected?\"\n",
-    "answer_2 = page.ask(question_2)\n",
-    "\n",
-    "# Display the answer dictionary\n",
-    "answer_2"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 4,
-   "id": "babaee28",
-   "metadata": {
-    "execution": {
-     "iopub.execute_input": "2025-04-21T21:24:31.286992Z",
-     "iopub.status.busy": "2025-04-21T21:24:31.286826Z",
-     "iopub.status.idle": "2025-04-21T21:24:32.069026Z",
-     "shell.execute_reply": "2025-04-21T21:24:32.068668Z"
-    }
-   },
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "{'answer': 'Inadequate Protective Equipment.',\n",
-       " 'confidence': 0.9997999668121338,\n",
-       " 'start': 26,\n",
-       " 'end': 26,\n",
-       " 'found': True,\n",
-       " 'page_num': 0,\n",
-       " 'source_elements': <ElementCollection[TextElement](count=1)>}"
-      ]
-     },
-     "execution_count": 4,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "# Ask about specific content from the table\n",
-    "question_3 = \"What is statute 5.8.3 about?\"\n",
-    "answer_3 = page.ask(question_3)\n",
-    "\n",
-    "# Display the answer\n",
-    "answer_3"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "cf24e07d",
-   "metadata": {},
-   "source": [
-    "The results include the extracted `answer`, a `confidence` score (useful for filtering uncertain answers), the `page_num`, and the `source_elements`.\n",
-    "\n",
-    "## Collecting Results into a DataFrame\n",
-    "\n",
-    "If you're asking multiple questions, it's often useful to collect the results into a pandas DataFrame for easier analysis."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 5,
-   "id": "00b777b5",
-   "metadata": {
-    "execution": {
-     "iopub.execute_input": "2025-04-21T21:24:32.070771Z",
-     "iopub.status.busy": "2025-04-21T21:24:32.070607Z",
-     "iopub.status.idle": "2025-04-21T21:24:35.309130Z",
-     "shell.execute_reply": "2025-04-21T21:24:35.308744Z"
-    }
-   },
-   "outputs": [
-    {
-     "data": {
-      "text/html": [
-       "<div>\n",
-       "<style scoped>\n",
-       "    .dataframe tbody tr th:only-of-type {\n",
-       "        vertical-align: middle;\n",
-       "    }\n",
-       "\n",
-       "    .dataframe tbody tr th {\n",
-       "        vertical-align: top;\n",
-       "    }\n",
-       "\n",
-       "    .dataframe thead th {\n",
-       "        text-align: right;\n",
-       "    }\n",
-       "</style>\n",
-       "<table border=\"1\" class=\"dataframe\">\n",
-       "  <thead>\n",
-       "    <tr style=\"text-align: right;\">\n",
-       "      <th></th>\n",
-       "      <th>question</th>\n",
-       "      <th>answer</th>\n",
-       "      <th>confidence</th>\n",
-       "    </tr>\n",
-       "  </thead>\n",
-       "  <tbody>\n",
-       "    <tr>\n",
-       "      <th>0</th>\n",
-       "      <td>What is the inspection date?</td>\n",
-       "      <td>February 3, 1905</td>\n",
-       "      <td>0.997994</td>\n",
-       "    </tr>\n",
-       "    <tr>\n",
-       "      <th>1</th>\n",
-       "      <td>What company was inspected?</td>\n",
-       "      <td>Jungle Health and Safety Inspection Service</td>\n",
-       "      <td>0.998895</td>\n",
-       "    </tr>\n",
-       "    <tr>\n",
-       "      <th>2</th>\n",
-       "      <td>What is statute 5.8.3 about?</td>\n",
-       "      <td>Inadequate Protective Equipment.</td>\n",
-       "      <td>0.999800</td>\n",
-       "    </tr>\n",
-       "    <tr>\n",
-       "      <th>3</th>\n",
-       "      <td>How many violations were there in total?</td>\n",
-       "      <td>4.12.7</td>\n",
-       "      <td>0.662557</td>\n",
-       "    </tr>\n",
-       "  </tbody>\n",
-       "</table>\n",
-       "</div>"
-      ],
-      "text/plain": [
-       "                                   question  \\\n",
-       "0              What is the inspection date?   \n",
-       "1               What company was inspected?   \n",
-       "2              What is statute 5.8.3 about?   \n",
-       "3  How many violations were there in total?   \n",
-       "\n",
-       "                                        answer  confidence  \n",
-       "0                             February 3, 1905    0.997994  \n",
-       "1  Jungle Health and Safety Inspection Service    0.998895  \n",
-       "2             Inadequate Protective Equipment.    0.999800  \n",
-       "3                                       4.12.7    0.662557  "
-      ]
-     },
-     "execution_count": 5,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "from natural_pdf import PDF\n",
-    "import pandas as pd\n",
-    "\n",
-    "pdf = PDF(\"https://github.com/jsoma/natural-pdf/raw/refs/heads/main/pdfs/01-practice.pdf\")\n",
-    "page = pdf.pages[0]\n",
-    "\n",
-    "# List of questions to ask\n",
-    "questions = [\n",
-    "    \"What is the inspection date?\",\n",
-    "    \"What company was inspected?\",\n",
-    "    \"What is statute 5.8.3 about?\",\n",
-    "    \"How many violations were there in total?\" # This might be less reliable\n",
-    "]\n",
-    "\n",
-    "# Collect answers for each question\n",
-    "results = []\n",
-    "for q in questions:\n",
-    "    answer_dict = page.ask(q)\n",
-    "    # Add the original question to the dictionary\n",
-    "    answer_dict['question'] = q\n",
-    "    results.append(answer_dict)\n",
-    "\n",
-    "# Convert the list of dictionaries to a DataFrame\n",
-    "# We select only the most relevant columns here\n",
-    "df_results = pd.DataFrame(results)[['question', 'answer', 'confidence']]\n",
-    "\n",
-    "# Display the DataFrame\n",
-    "df_results"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "381130c6",
-   "metadata": {},
-   "source": [
-    "This shows how you can iterate through questions, collect the answer dictionaries, and then create a structured DataFrame, making it easy to review questions, answers, and their confidence levels together.\n",
-    "\n",
-    "<div class=\"admonition note\">\n",
-    "<p class=\"admonition-title\">QA Model and Limitations</p>\n",
-    "\n",
-    "    *   The QA system relies on underlying transformer models. Performance and confidence scores vary.\n",
-    "    *   It works best for questions where the answer is explicitly stated. It cannot synthesize information or perform calculations (e.g., counting items might fail or return text containing a number rather than the count itself).\n",
-    "    *   You can potentially specify different QA models via the `model=` argument in `page.ask()` if others are configured.\n",
-    "</div> "
-   ]
-  }
- ],
- "metadata": {
-  "jupytext": {
-   "cell_metadata_filter": "-all",
-   "main_language": "python",
-   "notebook_metadata_filter": "-all"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.10.13"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}

docs/tutorials/06-document-qa.md

@@ -1,91 +0,0 @@
-# Document Question Answering (QA)
-
-Sometimes, instead of searching for specific text patterns, you just want to ask the document a question directly. `natural-pdf` includes an extractive Question Answering feature.
-
-"Extractive" means it finds the literal answer text within the document, rather than generating a new answer or summarizing.
-
-Let's ask our `01-practice.pdf` a few questions.
-
-```python
-#%pip install "natural-pdf[all]"
-```
-
-```python
-from natural_pdf import PDF
-
-# Load the PDF and get the page
-pdf = PDF("https://github.com/jsoma/natural-pdf/raw/refs/heads/main/pdfs/01-practice.pdf")
-page = pdf.pages[0]
-
-# Ask about the date
-question_1 = "What is the inspection date?"
-answer_1 = page.ask(question_1)
-
-# The result is a dictionary with the answer, confidence, etc.
-answer_1
-```
-
-```python
-# Ask about the company name
-question_2 = "What company was inspected?"
-answer_2 = page.ask(question_2)
-
-# Display the answer dictionary
-answer_2
-```
-
-```python
-# Ask about specific content from the table
-question_3 = "What is statute 5.8.3 about?"
-answer_3 = page.ask(question_3)
-
-# Display the answer
-answer_3
-```
-
-The results include the extracted `answer`, a `confidence` score (useful for filtering uncertain answers), the `page_num`, and the `source_elements`.
-
-## Collecting Results into a DataFrame
-
-If you're asking multiple questions, it's often useful to collect the results into a pandas DataFrame for easier analysis.
-
-```python
-from natural_pdf import PDF
-import pandas as pd
-
-pdf = PDF("https://github.com/jsoma/natural-pdf/raw/refs/heads/main/pdfs/01-practice.pdf")
-page = pdf.pages[0]
-
-# List of questions to ask
-questions = [
-    "What is the inspection date?",
-    "What company was inspected?",
-    "What is statute 5.8.3 about?",
-    "How many violations were there in total?" # This might be less reliable
-]
-
-# Collect answers for each question
-results = []
-for q in questions:
-    answer_dict = page.ask(q)
-    # Add the original question to the dictionary
-    answer_dict['question'] = q
-    results.append(answer_dict)
-
-# Convert the list of dictionaries to a DataFrame
-# We select only the most relevant columns here
-df_results = pd.DataFrame(results)[['question', 'answer', 'confidence']]
-
-# Display the DataFrame
-df_results
-```
-
-This shows how you can iterate through questions, collect the answer dictionaries, and then create a structured DataFrame, making it easy to review questions, answers, and their confidence levels together.
-
-<div class="admonition note">
-<p class="admonition-title">QA Model and Limitations</p>
-
-    *   The QA system relies on underlying transformer models. Performance and confidence scores vary.
-    *   It works best for questions where the answer is explicitly stated. It cannot synthesize information or perform calculations (e.g., counting items might fail or return text containing a number rather than the count itself).
-    *   You can potentially specify different QA models via the `model=` argument in `page.ask()` if others are configured.
-</div>