PyStreamPDF 2.1.2__tar.gz → 2.3.0__tar.gz

{pystreampdf-2.1.2 → pystreampdf-2.3.0}/Cargo.lock

@@ -575,7 +575,7 @@ dependencies = [
 
 [[package]]
 name = "pystreampdf"
-version = "2.1.2"
+version = "2.3.0"
 dependencies = [
  "chrono",
  "pyo3",
@@ -721,7 +721,7 @@ dependencies = [
 
 [[package]]
 name = "streampdf-core"
-version = "2.1.2"
+version = "2.3.0"
 dependencies = [
  "chrono",
  "pdfium-render",

{pystreampdf-2.1.2 → pystreampdf-2.3.0}/Cargo.toml

@@ -3,7 +3,7 @@ members = ["core", "python"]
 resolver = "2"
 
 [workspace.package]
-version = "2.1.2"
+version = "2.3.0"
 edition = "2021"
 authors = ["Georgi Mammen Mullassery <mullassery@gmail.com>"]
 license = "Proprietary"

pystreampdf-2.3.0/PKG-INFO

@@ -0,0 +1,271 @@
+Metadata-Version: 2.4
+Name: PyStreamPDF
+Version: 2.3.0
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: Other/Proprietary License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Rust
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Text Processing
+Requires-Dist: rich>=13.0
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: maturin>=1.0,<2.0 ; extra == 'dev'
+Requires-Dist: pytest>=7.0 ; extra == 'dev'
+Requires-Dist: pytest-timeout>=2.1 ; extra == 'dev'
+Requires-Dist: reportlab>=4.0 ; extra == 'dev'
+Requires-Dist: tiktoken>=0.5 ; extra == 'dev'
+Requires-Dist: tiktoken>=0.5 ; extra == 'tiktoken'
+Provides-Extra: dev
+Provides-Extra: tiktoken
+License-File: LICENSE
+Summary: Intelligence engine for PDFs: selective retrieval, structure analysis, and token-efficient RAG
+Keywords: pdf,retrieval,rag,ai,intelligence,search,indexing
+Author-email: Georgi Mammen Mullassery <mullassery@gmail.com>
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
+Project-URL: Issues, https://github.com/Mullassery/StreamPDF/issues
+Project-URL: Repository, https://github.com/Mullassery/StreamPDF
+
+# PyStreamPDF
+
+**Extract only what matters from PDFs, so you send less to LLMs.**
+
+Stop sending entire documents to LLMs. PyStreamPDF parses PDF structure and
+splits content into semantic, token-budget-aware chunks so you can send
+only the relevant parts of a document to a model. See "Token Savings"
+below for how much that saves in practice — it depends on your documents.
+
+[![PyPI](https://img.shields.io/pypi/v/pystreampdf)](https://pypi.org/project/pystreampdf)
+[![Python 3.9+](https://img.shields.io/badge/Python-3.9%2B-blue)](https://www.python.org)
+[![Tests: 557 Passing](https://img.shields.io/badge/tests-557%20passing-success)](./tests)
+[![License: Proprietary](https://img.shields.io/badge/License-Proprietary-blue.svg)](./LICENSE)
+
+---
+
+## 30-Second Start
+
+```python
+from pystreampdf import SemanticChunker, ElementType
+
+with open("financial_report.pdf", "rb") as f:
+    text = f.read().decode("latin-1", errors="ignore")  # or use a real PDF text extractor first
+
+chunker = SemanticChunker(target_tokens=500)
+chunks = chunker.chunk_content(
+    text, element_type=ElementType.TEXT, page_start=1, page_end=1
+)
+print(f"Split into {len(chunks)} semantic chunks")
+```
+
+There is no `Document`/`.extract_relevant()` class in this package —
+`SemanticChunker` (shown above) and `PDFCache` (see "Quick Start: Document
+Extraction" below) are the real, exported API. If you built against a
+`Document` class from an earlier version of these docs, it never actually
+existed in source; see Known Issues.
+
+---
+
+## Why PyStreamPDF?
+
+**The Problem:**
+- You send entire PDFs to LLMs (wasteful, expensive)
+- RAG systems retrieve too much content
+- Token costs skyrocket on large documents
+- No way to know which parts actually matter
+
+**The Solution:**
+- Semantic chunking with context awareness (real, in `SemanticChunker`)
+- Token-budget-aware caching (real, in `PDFCache` / `TokenBudgetConfig`)
+- Reduces the volume of text you send per query — see "Token Savings"
+  below for what's measured vs. illustrative
+
+---
+
+## Key Features
+
+- **Real encryption/permission detection:** Backed by PDFium's actual document security APIs — not a stub. Encrypted, password-protected, and permission-restricted PDFs are detected and reported accurately; opening with a wrong/missing password fails closed rather than silently succeeding.
+- **Honest failure signaling:** If a PDF can't be parsed (corrupt, truncated, unsupported), you get a real error — never fabricated placeholder content.
+- **Intelligent Extraction:** Find relevant sections automatically
+- **Semantic Chunking:** Context-aware splitting, not just word count
+- **Multi-Format:** Text, tables, images, charts, OCR
+- **Token Budgeting:** Allocate tokens by document type
+- **Smart Caching:** L1 memory + L2 disk (HMAC-signed on disk — no unverified deserialization)
+- **Metadata Preservation:** Keep tables, images, structure
+- **Production-Ready:** 557 passing tests (2 skipped when optional OCR system deps aren't installed)
+
+---
+
+## Real-World Use Cases
+
+See [examples/basic_parse.py](examples/basic_parse.py) for the Rust-backed
+`pystreampdf.open()` path (document/page/structure inspection — requires
+the compiled `_core` extension, see Known Issues) and
+[examples/token_budget_and_cache_example.py](examples/token_budget_and_cache_example.py)
+for the pure-Python `SemanticChunker` + `PDFCache` + `TokenBudgetConfig`
+path shown above. Both are real, runnable scripts — copying them is more
+reliable than a README snippet for an evolving API.
+
+---
+
+## Token Savings
+
+**A note on how token counts are computed:** by default, token counts use a
+`len(text) / 4` heuristic (a common rule of thumb, but approximate — it can
+be off by a meaningful margin, especially for code or non-English text).
+Install the optional `tiktoken` extra for exact BPE token counts:
+
+```bash
+pip install "pystreampdf[tiktoken]"
+```
+
+When `tiktoken` is installed, `pystreampdf.tokenizer.is_exact()` returns
+`True` and counts use the real `cl100k_base` encoding; otherwise it falls
+back to the heuristic. Check `pystreampdf.tokenizer.TOKENIZER_MODE` to see
+which mode produced a given number.
+
+Actual token savings depend entirely on your documents and how narrowly
+you scope `extract_relevant`-style queries — there's no committed
+benchmark result in this repo backing a specific percentage (the earlier
+"70%"/"10-50x" style claims in this README and in `__init__.py`'s
+docstring weren't measured against anything checked in). Run
+[examples/token_budget_and_cache_example.py](examples/token_budget_and_cache_example.py)
+against your own documents to get a real number for your use case.
+
+---
+
+## Security Notes
+
+- **Encryption & permissions**: `pystreampdf._core.PyPdfDocument.is_encrypted()`,
+  `.permissions()`, and `.open_with_password()` are backed by PDFium's real
+  document security APIs. A wrong or missing password on an encrypted PDF
+  raises an error — it never falls back to an unauthenticated parse.
+- **Cache integrity**: the on-disk (L2) cache is HMAC-SHA256 signed using a
+  key generated on first use and stored with owner-only (0600) permissions.
+  A tampered or foreign cache file fails signature verification and is
+  discarded before any deserialization occurs.
+- **MCP/DAB connector**: binds to `127.0.0.1` with no cross-origin access
+  and read-only permissions by default. Wider exposure requires an explicit
+  `allow_remote=True` opt-in — review the security implications first, since
+  the connector has no authentication of its own.
+
+---
+
+## Installation
+
+```bash
+pip install pystreampdf
+
+# With exact token counting (tiktoken)
+pip install "pystreampdf[tiktoken]"
+```
+
+---
+
+## Documentation
+
+- [Quick Start](docs/QUICKSTART.md) — Process your first PDF
+- [Extraction Strategies](docs/EXTRACTION.md) — Different approaches for different documents
+- [Token Budgeting](docs/TOKEN_BUDGETS.md) — Control context allocation
+- [Token Budget Multipliers](docs/TOKEN_BUDGET_MULTIPLIERS.md) — Comprehensive token budget guide with examples
+- [Examples](examples/) — Real-world RAG optimization
+
+## Quick Start: Document Extraction
+
+```python
+from pystreampdf import SemanticChunker, PDFCache, TokenBudgetConfig, BudgetRule
+
+# Setup budget config
+budget_config = TokenBudgetConfig(
+    base_budget=800,
+    rules=[BudgetRule("complex", 1.1, match_fields=["filename"])]
+)
+
+# Initialize cache with budget management
+cache = PDFCache(
+    memory_limit_mb=500,
+    disk_cache_dir="./pdf_cache",
+    token_budget_config=budget_config
+)
+
+# Process PDF with intelligent chunking
+def extract_pdf(pdf_path):
+    chunks, preview, title, pages = cache.get_or_process(
+        pdf_path,
+        process_fn=extract_chunks
+    )
+    return chunks
+
+# Use semantic chunker directly
+chunker = SemanticChunker(target_tokens=500)
+chunks = chunker.chunk_content(text, element_type=ElementType.TEXT, page_start=1, page_end=1)
+```
+
+## Budget Configuration
+
+### Default Settings
+- **Minimum budget**: 500 tokens (fixed)
+- **Maximum budget**: 1000 tokens (fixed)
+- **Default base**: 1000 tokens
+- **Multiplier range**: 0.5 - 1.5 (common)
+
+### Dynamic Adjustment via Multipliers
+Use keyword-based rules to scale within the 500-1000 range without changing hard limits. See examples for [financial](docs/TOKEN_BUDGET_MULTIPLIERS.md#high-complexity-documents), [legal](docs/TOKEN_BUDGET_MULTIPLIERS.md#high-complexity-documents), and [summary](docs/TOKEN_BUDGET_MULTIPLIERS.md#low-complexity-documents) documents.
+
+## MCP Integration (optional, local by default)
+
+PyStreamPDF ships an optional MCP/DAB connector (`pystreampdf._mcp_connector`)
+exposing document-processing tools (extract text/tables/images, OCR,
+structure detection, metadata, chunking, citations, validation). It's
+opt-in, binds to `127.0.0.1` only, and requires `allow_remote=True` to be
+exposed beyond localhost. See [examples/mcp_pystreampdf.py](examples/mcp_pystreampdf.py).
+
+## Known Issues
+
+- Earlier versions of this README documented a `Document` class with
+  `.extract_relevant()` and a `.token_savings` attribute — verified
+  against source: this class has never existed in this package. Fixed
+  in this pass to describe the real, exported API (`SemanticChunker`,
+  `PDFCache`, `TokenBudgetConfig`, and the Rust-backed `pystreampdf.open()`).
+- The published PyPI wheel is the only wheel on PyPI — macOS arm64 only,
+  no sdist. On any other platform or Python version, `pip install` will
+  fail without a local Rust toolchain to build from source.
+- ~~The published version (2.2.1) is ahead of what's tagged in this repo's
+  Cargo.toml/__init__.py (2.2.0)~~ Resolved: 2.3.0 (this release) is
+  unambiguously ahead of both 2.2.0 and 2.2.1.
+- The Rust-backed `pystreampdf.open()` / `pystreampdf.load_index()` API
+  (used in `examples/basic_parse.py`) silently becomes `None` if the
+  compiled `_core` extension isn't available (e.g. a from-source install
+  without `maturin develop`) — falling back to `SemanticChunker`/`PDFCache`
+  in that case, per the code above, not a hard error.
+- The "Tests & Build" GitHub Actions workflow was red for several weeks
+  due to two CI infrastructure bugs (not the test suite itself), both
+  now fixed and confirmed green in CI:
+  [run 32611116541](https://github.com/Mullassery/PyStreamPDF/actions/runs/32611116541).
+  First, `dtolnay/rust-toolchain@v1` required an explicit `toolchain`
+  input the workflow didn't provide, failing both jobs before any tests
+  ran — fixed by pinning to `dtolnay/rust-toolchain@stable`. Second, once
+  that was fixed, `maturin develop` failed in the Python-test job because
+  `actions/setup-python` doesn't provide an active virtualenv, which
+  maturin requires — fixed by creating and activating a `.venv` before
+  the build step. Actual CI output as of 2.2.0: `pytest` — 536 passed, 2
+  skipped (Python 3.10, 3.11, and 3.12, each identical); `cargo test` — 23
+  passed, 0 failed. (The Rust suite has 23 tests, not 536 — an earlier
+  version of this section conflated the pytest count with the Rust one.)
+  As of 2.3.0: `pytest` — 557 passed, 2 skipped (21 new tests added for
+  `tests/test_mcp_tools.py`).
+
+## License
+
+Proprietary License — Free to use with explicit attribution. See [LICENSE](LICENSE).
+
+---
+
+**PyStreamPDF v2.3.0** | Intelligent PDF processing for AI | Python 3.9+ | 557 passing tests
+

pystreampdf-2.3.0/README.md

@@ -0,0 +1,235 @@
+# PyStreamPDF
+
+**Extract only what matters from PDFs, so you send less to LLMs.**
+
+Stop sending entire documents to LLMs. PyStreamPDF parses PDF structure and
+splits content into semantic, token-budget-aware chunks so you can send
+only the relevant parts of a document to a model. See "Token Savings"
+below for how much that saves in practice — it depends on your documents.
+
+[![PyPI](https://img.shields.io/pypi/v/pystreampdf)](https://pypi.org/project/pystreampdf)
+[![Python 3.9+](https://img.shields.io/badge/Python-3.9%2B-blue)](https://www.python.org)
+[![Tests: 557 Passing](https://img.shields.io/badge/tests-557%20passing-success)](./tests)
+[![License: Proprietary](https://img.shields.io/badge/License-Proprietary-blue.svg)](./LICENSE)
+
+---
+
+## 30-Second Start
+
+```python
+from pystreampdf import SemanticChunker, ElementType
+
+with open("financial_report.pdf", "rb") as f:
+    text = f.read().decode("latin-1", errors="ignore")  # or use a real PDF text extractor first
+
+chunker = SemanticChunker(target_tokens=500)
+chunks = chunker.chunk_content(
+    text, element_type=ElementType.TEXT, page_start=1, page_end=1
+)
+print(f"Split into {len(chunks)} semantic chunks")
+```
+
+There is no `Document`/`.extract_relevant()` class in this package —
+`SemanticChunker` (shown above) and `PDFCache` (see "Quick Start: Document
+Extraction" below) are the real, exported API. If you built against a
+`Document` class from an earlier version of these docs, it never actually
+existed in source; see Known Issues.
+
+---
+
+## Why PyStreamPDF?
+
+**The Problem:**
+- You send entire PDFs to LLMs (wasteful, expensive)
+- RAG systems retrieve too much content
+- Token costs skyrocket on large documents
+- No way to know which parts actually matter
+
+**The Solution:**
+- Semantic chunking with context awareness (real, in `SemanticChunker`)
+- Token-budget-aware caching (real, in `PDFCache` / `TokenBudgetConfig`)
+- Reduces the volume of text you send per query — see "Token Savings"
+  below for what's measured vs. illustrative
+
+---
+
+## Key Features
+
+- **Real encryption/permission detection:** Backed by PDFium's actual document security APIs — not a stub. Encrypted, password-protected, and permission-restricted PDFs are detected and reported accurately; opening with a wrong/missing password fails closed rather than silently succeeding.
+- **Honest failure signaling:** If a PDF can't be parsed (corrupt, truncated, unsupported), you get a real error — never fabricated placeholder content.
+- **Intelligent Extraction:** Find relevant sections automatically
+- **Semantic Chunking:** Context-aware splitting, not just word count
+- **Multi-Format:** Text, tables, images, charts, OCR
+- **Token Budgeting:** Allocate tokens by document type
+- **Smart Caching:** L1 memory + L2 disk (HMAC-signed on disk — no unverified deserialization)
+- **Metadata Preservation:** Keep tables, images, structure
+- **Production-Ready:** 557 passing tests (2 skipped when optional OCR system deps aren't installed)
+
+---
+
+## Real-World Use Cases
+
+See [examples/basic_parse.py](examples/basic_parse.py) for the Rust-backed
+`pystreampdf.open()` path (document/page/structure inspection — requires
+the compiled `_core` extension, see Known Issues) and
+[examples/token_budget_and_cache_example.py](examples/token_budget_and_cache_example.py)
+for the pure-Python `SemanticChunker` + `PDFCache` + `TokenBudgetConfig`
+path shown above. Both are real, runnable scripts — copying them is more
+reliable than a README snippet for an evolving API.
+
+---
+
+## Token Savings
+
+**A note on how token counts are computed:** by default, token counts use a
+`len(text) / 4` heuristic (a common rule of thumb, but approximate — it can
+be off by a meaningful margin, especially for code or non-English text).
+Install the optional `tiktoken` extra for exact BPE token counts:
+
+```bash
+pip install "pystreampdf[tiktoken]"
+```
+
+When `tiktoken` is installed, `pystreampdf.tokenizer.is_exact()` returns
+`True` and counts use the real `cl100k_base` encoding; otherwise it falls
+back to the heuristic. Check `pystreampdf.tokenizer.TOKENIZER_MODE` to see
+which mode produced a given number.
+
+Actual token savings depend entirely on your documents and how narrowly
+you scope `extract_relevant`-style queries — there's no committed
+benchmark result in this repo backing a specific percentage (the earlier
+"70%"/"10-50x" style claims in this README and in `__init__.py`'s
+docstring weren't measured against anything checked in). Run
+[examples/token_budget_and_cache_example.py](examples/token_budget_and_cache_example.py)
+against your own documents to get a real number for your use case.
+
+---
+
+## Security Notes
+
+- **Encryption & permissions**: `pystreampdf._core.PyPdfDocument.is_encrypted()`,
+  `.permissions()`, and `.open_with_password()` are backed by PDFium's real
+  document security APIs. A wrong or missing password on an encrypted PDF
+  raises an error — it never falls back to an unauthenticated parse.
+- **Cache integrity**: the on-disk (L2) cache is HMAC-SHA256 signed using a
+  key generated on first use and stored with owner-only (0600) permissions.
+  A tampered or foreign cache file fails signature verification and is
+  discarded before any deserialization occurs.
+- **MCP/DAB connector**: binds to `127.0.0.1` with no cross-origin access
+  and read-only permissions by default. Wider exposure requires an explicit
+  `allow_remote=True` opt-in — review the security implications first, since
+  the connector has no authentication of its own.
+
+---
+
+## Installation
+
+```bash
+pip install pystreampdf
+
+# With exact token counting (tiktoken)
+pip install "pystreampdf[tiktoken]"
+```
+
+---
+
+## Documentation
+
+- [Quick Start](docs/QUICKSTART.md) — Process your first PDF
+- [Extraction Strategies](docs/EXTRACTION.md) — Different approaches for different documents
+- [Token Budgeting](docs/TOKEN_BUDGETS.md) — Control context allocation
+- [Token Budget Multipliers](docs/TOKEN_BUDGET_MULTIPLIERS.md) — Comprehensive token budget guide with examples
+- [Examples](examples/) — Real-world RAG optimization
+
+## Quick Start: Document Extraction
+
+```python
+from pystreampdf import SemanticChunker, PDFCache, TokenBudgetConfig, BudgetRule
+
+# Setup budget config
+budget_config = TokenBudgetConfig(
+    base_budget=800,
+    rules=[BudgetRule("complex", 1.1, match_fields=["filename"])]
+)
+
+# Initialize cache with budget management
+cache = PDFCache(
+    memory_limit_mb=500,
+    disk_cache_dir="./pdf_cache",
+    token_budget_config=budget_config
+)
+
+# Process PDF with intelligent chunking
+def extract_pdf(pdf_path):
+    chunks, preview, title, pages = cache.get_or_process(
+        pdf_path,
+        process_fn=extract_chunks
+    )
+    return chunks
+
+# Use semantic chunker directly
+chunker = SemanticChunker(target_tokens=500)
+chunks = chunker.chunk_content(text, element_type=ElementType.TEXT, page_start=1, page_end=1)
+```
+
+## Budget Configuration
+
+### Default Settings
+- **Minimum budget**: 500 tokens (fixed)
+- **Maximum budget**: 1000 tokens (fixed)
+- **Default base**: 1000 tokens
+- **Multiplier range**: 0.5 - 1.5 (common)
+
+### Dynamic Adjustment via Multipliers
+Use keyword-based rules to scale within the 500-1000 range without changing hard limits. See examples for [financial](docs/TOKEN_BUDGET_MULTIPLIERS.md#high-complexity-documents), [legal](docs/TOKEN_BUDGET_MULTIPLIERS.md#high-complexity-documents), and [summary](docs/TOKEN_BUDGET_MULTIPLIERS.md#low-complexity-documents) documents.
+
+## MCP Integration (optional, local by default)
+
+PyStreamPDF ships an optional MCP/DAB connector (`pystreampdf._mcp_connector`)
+exposing document-processing tools (extract text/tables/images, OCR,
+structure detection, metadata, chunking, citations, validation). It's
+opt-in, binds to `127.0.0.1` only, and requires `allow_remote=True` to be
+exposed beyond localhost. See [examples/mcp_pystreampdf.py](examples/mcp_pystreampdf.py).
+
+## Known Issues
+
+- Earlier versions of this README documented a `Document` class with
+  `.extract_relevant()` and a `.token_savings` attribute — verified
+  against source: this class has never existed in this package. Fixed
+  in this pass to describe the real, exported API (`SemanticChunker`,
+  `PDFCache`, `TokenBudgetConfig`, and the Rust-backed `pystreampdf.open()`).
+- The published PyPI wheel is the only wheel on PyPI — macOS arm64 only,
+  no sdist. On any other platform or Python version, `pip install` will
+  fail without a local Rust toolchain to build from source.
+- ~~The published version (2.2.1) is ahead of what's tagged in this repo's
+  Cargo.toml/__init__.py (2.2.0)~~ Resolved: 2.3.0 (this release) is
+  unambiguously ahead of both 2.2.0 and 2.2.1.
+- The Rust-backed `pystreampdf.open()` / `pystreampdf.load_index()` API
+  (used in `examples/basic_parse.py`) silently becomes `None` if the
+  compiled `_core` extension isn't available (e.g. a from-source install
+  without `maturin develop`) — falling back to `SemanticChunker`/`PDFCache`
+  in that case, per the code above, not a hard error.
+- The "Tests & Build" GitHub Actions workflow was red for several weeks
+  due to two CI infrastructure bugs (not the test suite itself), both
+  now fixed and confirmed green in CI:
+  [run 32611116541](https://github.com/Mullassery/PyStreamPDF/actions/runs/32611116541).
+  First, `dtolnay/rust-toolchain@v1` required an explicit `toolchain`
+  input the workflow didn't provide, failing both jobs before any tests
+  ran — fixed by pinning to `dtolnay/rust-toolchain@stable`. Second, once
+  that was fixed, `maturin develop` failed in the Python-test job because
+  `actions/setup-python` doesn't provide an active virtualenv, which
+  maturin requires — fixed by creating and activating a `.venv` before
+  the build step. Actual CI output as of 2.2.0: `pytest` — 536 passed, 2
+  skipped (Python 3.10, 3.11, and 3.12, each identical); `cargo test` — 23
+  passed, 0 failed. (The Rust suite has 23 tests, not 536 — an earlier
+  version of this section conflated the pytest count with the Rust one.)
+  As of 2.3.0: `pytest` — 557 passed, 2 skipped (21 new tests added for
+  `tests/test_mcp_tools.py`).
+
+## License
+
+Proprietary License — Free to use with explicit attribution. See [LICENSE](LICENSE).
+
+---
+
+**PyStreamPDF v2.3.0** | Intelligent PDF processing for AI | Python 3.9+ | 557 passing tests