kreuzberg 1.0.0__tar.gz

kreuzberg-1.0.0/LICENSE

@@ -0,0 +1,7 @@
+Copyright 2025 Na'aman Hirschfeld
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

kreuzberg-1.0.0/PKG-INFO

@@ -0,0 +1,259 @@
+Metadata-Version: 2.2
+Name: kreuzberg
+Version: 1.0.0
+Summary: A text extraction library supporting PDFs, images, office documents and more
+Author-email: Na'aman Hirschfeld <nhirschfed@gmail.com>
+License: MIT
+Project-URL: homepage, https://github.com/Goldziher/kreuzberg
+Keywords: async,document-processing,docx,image-to-text,latex,markdown,ocr,odt,office-documents,pandoc,pdf,pdf-extraction,rag,tesseract,text-extraction,text-processing
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Classifier: Topic :: Text Processing :: General
+Classifier: Topic :: Utilities
+Classifier: Typing :: Typed
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: anyio>=4.8.0
+Requires-Dist: charset-normalizer>=3.4.1
+Requires-Dist: pypandoc>=1.15
+Requires-Dist: pypdfium2>=4.30.1
+Requires-Dist: pytesseract>=0.3.13
+Requires-Dist: typing-extensions>=4.12.2
+
+# Kreuzberg
+
+Kreuzberg is a library for simplified text extraction from PDF files. It's meant to offer simple, hassle free text
+extraction.
+
+Why?
+
+I am building, like many do now, a RAG focused service. I have text extraction needs.
+There are quite a lot of commercial options out there, and several open-source + paid options.
+But I wanted something simple, which does not require expansive round-trips to an external API.
+Furthermore, I wanted something that is easy to run locally and isn't very heavy / requires a GPU.
+
+Hence, this library.
+
+## Features
+
+- Extract text from PDFs, images, and office documents
+- Use modern Python with async (via `anyio`) and proper type hints
+- Extensive error handling for easy debugging
+
+## Installation
+
+1. Begin by installing the python package:
+
+   ```shell
+
+   pip install kreuzberg
+
+   ```
+
+2. Install the system dependencies:
+
+- [pandoc](https://pandoc.org/installing.html) (non-pdf text extraction, GPL v2.0 licensed but used via CLI only)
+- [tesseract-ocr](https://tesseract-ocr.github.io/) (for image/PDF OCR, Apache License)
+
+## Supported File Types
+
+Kreuzberg supports a wide range of file formats:
+
+### Document Formats
+
+- PDF (`.pdf`) - both searchable and scanned documents
+- Word Documents (`.docx`)
+- OpenDocument Text (`.odt`)
+- Rich Text Format (`.rtf`)
+
+### Image Formats
+
+- JPEG, JPG (`.jpg`, `.jpeg`, `.pjpeg`)
+- PNG (`.png`)
+- TIFF (`.tiff`, `.tif`)
+- BMP (`.bmp`)
+- GIF (`.gif`)
+- WebP (`.webp`)
+- JPEG 2000 (`.jp2`, `.jpx`, `.jpm`, `.mj2`)
+- Portable Anymap (`.pnm`)
+- Portable Bitmap (`.pbm`)
+- Portable Graymap (`.pgm`)
+- Portable Pixmap (`.ppm`)
+
+#### Text and Markup Formats
+
+- Plain Text (`.txt`)
+- Markdown (`.md`)
+- reStructuredText (`.rst`)
+- LaTeX (`.tex`)
+
+#### Data Formats
+
+- Comma-Separated Values (`.csv`)
+- Tab-Separated Values (`.tsv`)
+
+All formats support text extraction, with different processing methods:
+
+- PDFs are processed using pdfium2 for searchable PDFs and Tesseract OCR for scanned documents
+- Images are processed using Tesseract OCR
+- Office documents and other formats are processed using Pandoc
+- Plain text files are read directly with appropriate encoding detection
+
+## Usage
+
+Kreuzberg exports two async functions:
+
+- Extract text from a file (string path or `pathlib.Path`) using `extract_file()`
+- Extract text from a byte-string using `extract_bytes()`
+
+Note - both of these functions are async and therefore should be used in an async context.
+
+### Extract from File
+
+```python
+from pathlib import Path
+from kreuzberg import extract_file
+
+
+# Extract text from a PDF file
+async def extract_pdf():
+    result = await extract_file("document.pdf")
+    print(f"Extracted text: {result.content}")
+    print(f"Output mime type: {result.mime_type}")
+
+
+# Extract text from an image
+async def extract_image():
+    result = await extract_file("scan.png")
+    print(f"Extracted text: {result.content}")
+
+
+# or use Path
+
+async def extract_pdf():
+    result = await extract_file(Path("document.pdf"))
+    print(f"Extracted text: {result.content}")
+    print(f"Output mime type: {result.mime_type}")
+```
+
+### Extract from Bytes
+
+```python
+from kreuzberg import extract_bytes
+
+
+# Extract text from PDF bytes
+async def process_uploaded_pdf(pdf_content: bytes):
+    result = await extract_bytes(pdf_content, mime_type="application/pdf")
+    return result.content
+
+
+# Extract text from image bytes
+async def process_uploaded_image(image_content: bytes):
+    result = await extract_bytes(image_content, mime_type="image/jpeg")
+    return result.content
+```
+
+### Error Handling
+
+Kreuzberg raises two exception types:
+
+#### ValidationError
+
+Raised when there are issues with input validation:
+
+- Unsupported mime types
+- Non-existent files
+- Undetectable mime types
+
+#### ParsingError
+
+Raised when there are issues during the text extraction process:
+
+- PDF parsing failures
+- OCR errors
+- Pandoc conversion errors
+
+```python
+from kreuzberg import extract_file
+from kreuzberg.exceptions import ValidationError, ParsingError
+
+
+async def safe_extract():
+    try:
+        result = await extract_file("document.doc")
+        return result.content
+    except ValidationError as e:
+        print(f"Validation error: {e.message}")
+        print(f"Context: {e.context}")
+    except ParsingError as e:
+        print(f"Parsing error: {e.message}")
+        print(f"Context: {e.context}")  # Contains detailed error information
+```
+
+Both error types include helpful context information for debugging:
+
+```python
+try:
+    result = await extract_file("scanned.pdf")
+except ParsingError as e:
+# e.context might contain:
+# {
+#    "file_path": "scanned.pdf",
+#    "error": "Tesseract OCR failed: Unable to process image"
+# }
+```
+
+### ExtractionResult
+
+All extraction functions return an ExtractionResult named tuple containing:
+
+- content: The extracted text as a string
+- mime_type: The mime type of the output (either "text/plain" or, if pandoc is used- "text/markdown")
+
+```python
+from kreuzberg import ExtractionResult
+
+
+async def process_document(path: str) -> str:
+    result: ExtractionResult = await extract_file(path)
+    return result.content
+
+
+# or access the result as tuple
+
+async def process_document(path: str) -> str:
+    content, mime_type = await extract_file(path)
+    # do something with mime_type
+    return content
+```
+
+## Contribution
+
+This library is open to contribution. Feel free to open issues or submit PRs. Its better to discuss issues before
+submitting PRs to avoid disappointment.
+
+### Local Development
+
+1. Clone the repo
+2. Install the system dependencies
+3. Install the full dependencies with `uv sync`
+4. Install the pre-commit hooks with:
+   ```shell
+   pre-commit install && pre-commit install --hook-type commit-msg
+   ```
+5. Make your changes and submit a PR
+
+## License
+
+This library uses the MIT license.

kreuzberg-1.0.0/README.md

@@ -0,0 +1,227 @@
+# Kreuzberg
+
+Kreuzberg is a library for simplified text extraction from PDF files. It's meant to offer simple, hassle free text
+extraction.
+
+Why?
+
+I am building, like many do now, a RAG focused service. I have text extraction needs.
+There are quite a lot of commercial options out there, and several open-source + paid options.
+But I wanted something simple, which does not require expansive round-trips to an external API.
+Furthermore, I wanted something that is easy to run locally and isn't very heavy / requires a GPU.
+
+Hence, this library.
+
+## Features
+
+- Extract text from PDFs, images, and office documents
+- Use modern Python with async (via `anyio`) and proper type hints
+- Extensive error handling for easy debugging
+
+## Installation
+
+1. Begin by installing the python package:
+
+   ```shell
+
+   pip install kreuzberg
+
+   ```
+
+2. Install the system dependencies:
+
+- [pandoc](https://pandoc.org/installing.html) (non-pdf text extraction, GPL v2.0 licensed but used via CLI only)
+- [tesseract-ocr](https://tesseract-ocr.github.io/) (for image/PDF OCR, Apache License)
+
+## Supported File Types
+
+Kreuzberg supports a wide range of file formats:
+
+### Document Formats
+
+- PDF (`.pdf`) - both searchable and scanned documents
+- Word Documents (`.docx`)
+- OpenDocument Text (`.odt`)
+- Rich Text Format (`.rtf`)
+
+### Image Formats
+
+- JPEG, JPG (`.jpg`, `.jpeg`, `.pjpeg`)
+- PNG (`.png`)
+- TIFF (`.tiff`, `.tif`)
+- BMP (`.bmp`)
+- GIF (`.gif`)
+- WebP (`.webp`)
+- JPEG 2000 (`.jp2`, `.jpx`, `.jpm`, `.mj2`)
+- Portable Anymap (`.pnm`)
+- Portable Bitmap (`.pbm`)
+- Portable Graymap (`.pgm`)
+- Portable Pixmap (`.ppm`)
+
+#### Text and Markup Formats
+
+- Plain Text (`.txt`)
+- Markdown (`.md`)
+- reStructuredText (`.rst`)
+- LaTeX (`.tex`)
+
+#### Data Formats
+
+- Comma-Separated Values (`.csv`)
+- Tab-Separated Values (`.tsv`)
+
+All formats support text extraction, with different processing methods:
+
+- PDFs are processed using pdfium2 for searchable PDFs and Tesseract OCR for scanned documents
+- Images are processed using Tesseract OCR
+- Office documents and other formats are processed using Pandoc
+- Plain text files are read directly with appropriate encoding detection
+
+## Usage
+
+Kreuzberg exports two async functions:
+
+- Extract text from a file (string path or `pathlib.Path`) using `extract_file()`
+- Extract text from a byte-string using `extract_bytes()`
+
+Note - both of these functions are async and therefore should be used in an async context.
+
+### Extract from File
+
+```python
+from pathlib import Path
+from kreuzberg import extract_file
+
+
+# Extract text from a PDF file
+async def extract_pdf():
+    result = await extract_file("document.pdf")
+    print(f"Extracted text: {result.content}")
+    print(f"Output mime type: {result.mime_type}")
+
+
+# Extract text from an image
+async def extract_image():
+    result = await extract_file("scan.png")
+    print(f"Extracted text: {result.content}")
+
+
+# or use Path
+
+async def extract_pdf():
+    result = await extract_file(Path("document.pdf"))
+    print(f"Extracted text: {result.content}")
+    print(f"Output mime type: {result.mime_type}")
+```
+
+### Extract from Bytes
+
+```python
+from kreuzberg import extract_bytes
+
+
+# Extract text from PDF bytes
+async def process_uploaded_pdf(pdf_content: bytes):
+    result = await extract_bytes(pdf_content, mime_type="application/pdf")
+    return result.content
+
+
+# Extract text from image bytes
+async def process_uploaded_image(image_content: bytes):
+    result = await extract_bytes(image_content, mime_type="image/jpeg")
+    return result.content
+```
+
+### Error Handling
+
+Kreuzberg raises two exception types:
+
+#### ValidationError
+
+Raised when there are issues with input validation:
+
+- Unsupported mime types
+- Non-existent files
+- Undetectable mime types
+
+#### ParsingError
+
+Raised when there are issues during the text extraction process:
+
+- PDF parsing failures
+- OCR errors
+- Pandoc conversion errors
+
+```python
+from kreuzberg import extract_file
+from kreuzberg.exceptions import ValidationError, ParsingError
+
+
+async def safe_extract():
+    try:
+        result = await extract_file("document.doc")
+        return result.content
+    except ValidationError as e:
+        print(f"Validation error: {e.message}")
+        print(f"Context: {e.context}")
+    except ParsingError as e:
+        print(f"Parsing error: {e.message}")
+        print(f"Context: {e.context}")  # Contains detailed error information
+```
+
+Both error types include helpful context information for debugging:
+
+```python
+try:
+    result = await extract_file("scanned.pdf")
+except ParsingError as e:
+# e.context might contain:
+# {
+#    "file_path": "scanned.pdf",
+#    "error": "Tesseract OCR failed: Unable to process image"
+# }
+```
+
+### ExtractionResult
+
+All extraction functions return an ExtractionResult named tuple containing:
+
+- content: The extracted text as a string
+- mime_type: The mime type of the output (either "text/plain" or, if pandoc is used- "text/markdown")
+
+```python
+from kreuzberg import ExtractionResult
+
+
+async def process_document(path: str) -> str:
+    result: ExtractionResult = await extract_file(path)
+    return result.content
+
+
+# or access the result as tuple
+
+async def process_document(path: str) -> str:
+    content, mime_type = await extract_file(path)
+    # do something with mime_type
+    return content
+```
+
+## Contribution
+
+This library is open to contribution. Feel free to open issues or submit PRs. Its better to discuss issues before
+submitting PRs to avoid disappointment.
+
+### Local Development
+
+1. Clone the repo
+2. Install the system dependencies
+3. Install the full dependencies with `uv sync`
+4. Install the pre-commit hooks with:
+   ```shell
+   pre-commit install && pre-commit install --hook-type commit-msg
+   ```
+5. Make your changes and submit a PR
+
+## License
+
+This library uses the MIT license.

kreuzberg-1.0.0/kreuzberg/__init__.py

@@ -0,0 +1,11 @@
+from .exceptions import KreuzbergError, ParsingError, ValidationError
+from .extraction import ExtractionResult, extract_bytes, extract_file
+
+__all__ = [
+    "ExtractionResult",
+    "KreuzbergError",
+    "ParsingError",
+    "ValidationError",
+    "extract_bytes",
+    "extract_file",
+]

kreuzberg-1.0.0/kreuzberg/_extractors.py

@@ -0,0 +1,145 @@
+from __future__ import annotations
+
+from typing import TYPE_CHECKING, cast
+
+from charset_normalizer import detect
+from pypandoc import convert_file, convert_text
+from pypdfium2 import PdfDocument, PdfiumError
+from pytesseract import TesseractError, image_to_string
+
+from kreuzberg._mime_types import PANDOC_MIME_TYPE_EXT_MAP
+from kreuzberg._sync import run_sync
+from kreuzberg.exceptions import ParsingError
+
+if TYPE_CHECKING:
+    from pathlib import Path
+
+
+def _extract_pdf_with_tesseract(file_path: Path) -> str:
+    """Extract text from a scanned PDF file using pytesseract.
+
+    Args:
+        file_path: The path to the PDF file.
+
+    Raises:
+        ParsingError: If the text could not be extracted from the PDF file.
+
+    Returns:
+        The extracted text.
+    """
+    try:
+        # make it into an image here:
+        pdf = PdfDocument(str(file_path))
+        images = [page.render(scale=2.0).to_pil() for page in pdf]
+
+        text = "\n".join(image_to_string(img) for img in images)
+        return text.strip()
+    except (PdfiumError, TesseractError) as e:
+        raise ParsingError(
+            "Could not extract text from PDF file", context={"file_path": str(file_path), "error": str(e)}
+        ) from e
+
+
+def _extract_pdf_with_pdfium2(file_path: Path) -> str:
+    """Extract text from a searchable PDF file using pypdfium2.
+
+    Args:
+        file_path: The path to the PDF file.
+
+    Raises:
+        ParsingError: If the text could not be extracted from the PDF file.
+
+    Returns:
+        The extracted text.
+    """
+    try:
+        document = PdfDocument(file_path)
+        text = "\n".join(page.get_textpage().get_text_range() for page in document)
+        return text.strip()
+    except PdfiumError as e:
+        raise ParsingError(
+            "Could not extract text from PDF file", context={"file_path": str(file_path), "error": str(e)}
+        ) from e
+
+
+async def _extract_pdf_file(file_path: Path) -> str:
+    """Extract text from a PDF file.
+
+    Args:
+        file_path: The path to the PDF file.
+
+    Returns:
+        The extracted text.
+    """
+    if content := await run_sync(_extract_pdf_with_pdfium2, file_path):
+        return content
+
+    return await run_sync(_extract_pdf_with_tesseract, file_path)
+
+
+async def _extract_content_with_pandoc(file_data: bytes, mime_type: str, encoding: str | None = None) -> str:
+    """Extract text using pandoc.
+
+    Args:
+        file_data: The content of the file.
+        mime_type: The mime type of the file.
+        encoding: An optional encoding to use when decoding the string.
+
+    Raises:
+        ParsingError: If the text could not be extracted from the file using pandoc.
+
+    Returns:
+        The extracted text.
+    """
+    ext = PANDOC_MIME_TYPE_EXT_MAP[mime_type]
+    encoding = encoding or detect(file_data)["encoding"] or "utf-8"
+    try:
+        return cast(str, await run_sync(convert_text, file_data, to="md", format=ext, encoding=encoding))
+    except RuntimeError as e:
+        raise ParsingError(
+            f"Could not extract text from {PANDOC_MIME_TYPE_EXT_MAP[mime_type]} file contents",
+            context={"error": str(e)},
+        ) from e
+
+
+async def _extract_file_with_pandoc(file_path: Path | str, mime_type: str) -> str:
+    """Extract text using pandoc.
+
+    Args:
+        file_path: The path to the file.
+        mime_type: The mime type of the file.
+
+    Raises:
+        ParsingError: If the text could not be extracted from the file using pandoc.
+
+    Returns:
+        The extracted text.
+    """
+    ext = PANDOC_MIME_TYPE_EXT_MAP[mime_type]
+    try:
+        return cast(str, await run_sync(convert_file, file_path, to="md", format=ext))
+    except RuntimeError as e:
+        raise ParsingError(
+            f"Could not extract text from {PANDOC_MIME_TYPE_EXT_MAP[mime_type]} file",
+            context={"file_path": str(file_path), "error": str(e)},
+        ) from e
+
+
+async def _extract_image_with_tesseract(file_path: Path | str) -> str:
+    """Extract text from an image file.
+
+    Args:
+        file_path: The path to the image file.
+
+    Raises:
+        ParsingError: If the text could not be extracted from the image file.
+
+    Returns:
+        The extracted content.
+    """
+    try:
+        return cast(str, image_to_string(str(file_path)).strip())
+    except TesseractError as e:
+        raise ParsingError(
+            "Could not extract text from image file", context={"file_path": str(file_path), "error": str(e)}
+        ) from e