dot-parser 2.1.0__tar.gz → 2.1.2__tar.gz

{dot_parser-2.1.0 → dot_parser-2.1.2}/CHANGELOG.md

@@ -7,6 +7,23 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [2.1.2] - 2026-10-27
+
+### Fixed
+
+- `chunk()` keeps every chunk within `max_tokens`. A paragraph larger than
+  the limit on its own (a long table, text without blank lines) is now split
+  instead of kept whole, and an undersized chunk is absorbed into its neighbour
+  only when the result still fits: a long run of heading-only sections (a table
+  of contents, an index) no longer piles up into a single chunk.
+
+## [2.1.1] - 2026-09-30
+
+### Changed
+
+- `Mistral` now defaults to `mistral-ocr-4-1`; `mistral-ocr-4-0` retires on
+  2026-09-30. Pricing is unchanged.
+
 ## [2.1.0] - 2026-09-30
 
 ### Added
@@ -83,6 +100,8 @@ this release.
 - `parse_with_images(backend=...)` accepts DOCX- and PPTX-capable backends in
   its type signature, matching what the function already supported at runtime.
 
-[Unreleased]: https://gitlab.com/deepika6190303/deepika-open-toolbox/dot-parser/-/compare/v2.1.0...main
+[Unreleased]: https://gitlab.com/deepika6190303/deepika-open-toolbox/dot-parser/-/compare/v2.1.2...main
+[2.1.2]: https://gitlab.com/deepika6190303/deepika-open-toolbox/dot-parser/-/compare/v2.1.1...v2.1.2
+[2.1.1]: https://gitlab.com/deepika6190303/deepika-open-toolbox/dot-parser/-/compare/v2.1.0...v2.1.1
 [2.1.0]: https://gitlab.com/deepika6190303/deepika-open-toolbox/dot-parser/-/compare/v2.0.0...v2.1.0
 [2.0.0]: https://gitlab.com/deepika6190303/deepika-open-toolbox/dot-parser/-/tags/v2.0.0

{dot_parser-2.1.0 → dot_parser-2.1.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: dot-parser
-Version: 2.1.0
+Version: 2.1.2
 Summary: Document-to-markdown parser and chunker for RAG pipelines
 Project-URL: Homepage, https://gitlab.com/deepika6190303/deepika-open-toolbox/dot-parser
 Project-URL: Repository, https://gitlab.com/deepika6190303/deepika-open-toolbox/dot-parser
@@ -154,7 +154,7 @@ described either by a VLM or by OCR annotations from the same call. See
 
 - **`Pymu(ocr_fallback=True)`** -- pymupdf4llm, default. Fast and CPU-only. The built-in OCR fallback kicks in when an OCR engine (tesseract, rapidocr, paddleocr) is available. Set `ocr_fallback=False` to force the no-OCR code path (matches a runtime image without OCR shipped).
 - **`Docling(force_full_page_ocr=True)`** -- IBM Docling, layout-aware ML pipeline + Tesseract OCR. Robust on image-only and broken-encoding PDFs. Install with `pip install dot-parser[docling]`.
-- **`Mistral(api_key=None, model="mistral-ocr-4-0")`** -- Mistral OCR cloud API. Reads `MISTRAL_API_KEY` from env. Implements `parse_pdfs` via the Batch API. Install with `pip install dot-parser[mistral]`.
+- **`Mistral(api_key=None, model="mistral-ocr-4-1")`** -- Mistral OCR cloud API. Reads `MISTRAL_API_KEY` from env. Implements `parse_pdfs` via the Batch API. Install with `pip install dot-parser[mistral]`.
 - **`Llama(api_key=None, tier="cost_effective")`** -- LlamaCloud parsing API. Reads `LLAMA_CLOUD_API_KEY` from env. Tiers: `fast`, `cost_effective`, `agentic`, `agentic_plus`. Install with `pip install dot-parser[llama]`.
 
 #### Mistral OCR enrichments

{dot_parser-2.1.0 → dot_parser-2.1.2}/README.md

@@ -116,7 +116,7 @@ described either by a VLM or by OCR annotations from the same call. See
 
 - **`Pymu(ocr_fallback=True)`** -- pymupdf4llm, default. Fast and CPU-only. The built-in OCR fallback kicks in when an OCR engine (tesseract, rapidocr, paddleocr) is available. Set `ocr_fallback=False` to force the no-OCR code path (matches a runtime image without OCR shipped).
 - **`Docling(force_full_page_ocr=True)`** -- IBM Docling, layout-aware ML pipeline + Tesseract OCR. Robust on image-only and broken-encoding PDFs. Install with `pip install dot-parser[docling]`.
-- **`Mistral(api_key=None, model="mistral-ocr-4-0")`** -- Mistral OCR cloud API. Reads `MISTRAL_API_KEY` from env. Implements `parse_pdfs` via the Batch API. Install with `pip install dot-parser[mistral]`.
+- **`Mistral(api_key=None, model="mistral-ocr-4-1")`** -- Mistral OCR cloud API. Reads `MISTRAL_API_KEY` from env. Implements `parse_pdfs` via the Batch API. Install with `pip install dot-parser[mistral]`.
 - **`Llama(api_key=None, tier="cost_effective")`** -- LlamaCloud parsing API. Reads `LLAMA_CLOUD_API_KEY` from env. Tiers: `fast`, `cost_effective`, `agentic`, `agentic_plus`. Install with `pip install dot-parser[llama]`.
 
 #### Mistral OCR enrichments

{dot_parser-2.1.0 → dot_parser-2.1.2}/benchmark/README.md

@@ -72,5 +72,5 @@ dropped equations, wrong table cells), so we keep it purely manual.
 | `pymu` | $0.00 | pymupdf4llm with OCR fallback **disabled** — reproduces le-lab production where no OCR engine is shipped in the runtime image |
 | `pymu_ocr` | $0.00 | pymupdf4llm with built-in OCR fallback **enabled** — uses tesseract (or rapidocr/paddleocr) automatically when text extraction looks suspect |
 | `docling` | $0.00 | layout-aware ML pipeline + Tesseract OCR with `force_full_page_ocr=True` |
-| `mistral` | $4.00 | `mistral-ocr-4-0` cloud API. With `--mistral-batch`: $2.00 per 1k pages (50% off) via the Mistral Batch API. |
+| `mistral` | $4.00 | `mistral-ocr-4-1` cloud API. With `--mistral-batch`: $2.00 per 1k pages (50% off) via the Mistral Batch API. |
 | `llama` | $3.00 | LlamaCloud `cost_effective` tier |

{dot_parser-2.1.0 → dot_parser-2.1.2}/docs/IMAGE_EXTRACTION.md

@@ -30,7 +30,7 @@ parse_with_images("doc.docx", backend=Mistral())
 
 The `.docx` is uploaded to Mistral and `ocr.process` is called with `document_url`
 plus `bbox_annotation_format`: a single call returns markdown, images and
-annotations. Model: `mistral-ocr-latest`.
+annotations. Model: `mistral-ocr-4-1`.
 
 - **+** One API call, no per-image round trip.
 - **+** Handles natively the formats markitdown cannot pass through (the document is rasterised, so SmartArt / EMF / SVG render correctly).
@@ -68,7 +68,7 @@ parse_with_images("doc.pdf")  # backend=Mistral() implied
 parse_with_images("doc.pdf", backend=Mistral())  # explicit, identical
 ```
 
-Same `bbox_annotation_format` flow as DOCX path B. Model: `mistral-ocr-latest`.
+Same `bbox_annotation_format` flow as DOCX path B. Model: `mistral-ocr-4-1`.
 
 - **+** One call returns markdown, images and annotations; robust on scanned or broken-encoding PDFs; exact image positions.
 - **−** Billed per page; fixed annotation schema (same caveat as DOCX path B).

{dot_parser-2.1.0 → dot_parser-2.1.2}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "dot-parser"
-version = "2.1.0"
+version = "2.1.2"
 description = "Document-to-markdown parser and chunker for RAG pipelines"
 readme = "README.md"
 authors = [{ name = "Kannon For Deep Tech", email = "louis.letarnec@deepika.ai" }]

{dot_parser-2.1.0 → dot_parser-2.1.2}/src/dot_parser/backends/mistral.py

@@ -217,7 +217,7 @@ class Mistral:
         self,
         *,
         api_key: str | None = None,
-        model: str = "mistral-ocr-4-0",
+        model: str = "mistral-ocr-4-1",
         language: str | None = None,
         extract_headers_footers: bool = False,
         confidence_scores: Literal["page", "word"] | None = None,

{dot_parser-2.1.0 → dot_parser-2.1.2}/src/dot_parser/chunking.py

@@ -3,6 +3,7 @@
 
 import re
 import statistics
+from typing import cast
 
 import semchunk
 
@@ -63,17 +64,16 @@ def _split_oversized(section: dict, max_tokens: int) -> list[dict]:
     body = section["body"]
     path = section["section_path"]
 
-    full_text = f"{heading}\n\n{body}" if heading else body
-    if count_tokens(full_text) <= max_tokens:
+    prefix = f"{heading}\n\n" if heading else ""
+    if count_tokens(prefix + body) <= max_tokens:
         return [section]
 
     # Try paragraph splitting first
     paragraphs = re.split(r"\n\n+", body)
     if len(paragraphs) > 1:
-        sub_chunks = _group_paragraphs(paragraphs, max_tokens, heading)
+        sub_chunks = _group_paragraphs(paragraphs, max_tokens, prefix)
     else:
-        # Semantic splitting via semchunk
-        sub_chunks = semchunk.chunk(body, chunk_size=max_tokens, token_counter=count_tokens)
+        sub_chunks = _split_text(body, max_tokens, prefix)
 
     results = []
     for i, text in enumerate(sub_chunks):
@@ -96,31 +96,37 @@ def _split_oversized(section: dict, max_tokens: int) -> list[dict]:
     return results
 
 
-def _group_paragraphs(paragraphs: list[str], max_tokens: int, heading: str | None) -> list[str]:
-    """Group paragraphs into chunks that fit within max_tokens."""
+def _split_text(text: str, max_tokens: int, prefix: str) -> list[str]:
+    """Split text semantically (semchunk) into pieces that fit within max_tokens
+    even with `prefix` (the heading the first piece carries) before them.
+    semchunk returns a bare list unless asked for offsets."""
+    pieces = semchunk.chunk(
+        text, chunk_size=max_tokens, token_counter=lambda t: count_tokens(prefix + t)
+    )
+    return cast("list[str]", pieces)
+
+
+def _group_paragraphs(paragraphs: list[str], max_tokens: int, prefix: str) -> list[str]:
+    """Group paragraphs into chunks that fit within max_tokens, the first one
+    with `prefix` (its heading) before it. A paragraph too large on its own is
+    split semantically first."""
     groups: list[str] = []
     current: list[str] = []
-    current_tokens = 0
 
-    # Account for heading in first group
-    heading_tokens = count_tokens(heading + "\n\n") if heading else 0
-    first_group = True
+    def fits(texts: list[str]) -> bool:
+        lead = "" if groups else prefix
+        return count_tokens(lead + "\n\n".join(texts)) <= max_tokens
 
     for para in paragraphs:
         para = para.strip()
         if not para:
             continue
-        para_tokens = count_tokens(para)
-        overhead = heading_tokens if first_group and not current else 0
-
-        if current and current_tokens + overhead + para_tokens > max_tokens:
-            groups.append("\n\n".join(current))
-            current = [para]
-            current_tokens = para_tokens
-            first_group = False
-        else:
-            current.append(para)
-            current_tokens += para_tokens
+        pieces = [para] if fits([para]) else _split_text(para, max_tokens, prefix)
+        for piece in pieces:
+            if current and not fits([*current, piece]):
+                groups.append("\n\n".join(current))
+                current = []
+            current.append(piece)
 
     if current:
         groups.append("\n\n".join(current))
@@ -168,15 +174,19 @@ def _merge_small(sections: list[dict], max_tokens: int) -> list[dict]:
 
 
 def _absorb_undersized(sections: list[dict], max_tokens: int) -> list[dict]:
-    """Absorb chunks smaller than 15% of max_tokens into neighbors."""
+    """Absorb chunks smaller than 15% of max_tokens into neighbors, when the
+    result still fits within max_tokens."""
     threshold = max_tokens * 0.15
     result: list[dict] = []
 
+    def fits(first: str, second: str) -> bool:
+        return count_tokens(first + "\n\n" + second) <= max_tokens
+
     for section in sections:
         text = _section_text(section)
         tokens = count_tokens(text)
 
-        if tokens < threshold and result:
+        if tokens < threshold and result and fits(_section_text(result[-1]), text):
             # Append to previous
             prev = result[-1]
             prev_text = _section_text(prev)
@@ -192,7 +202,7 @@ def _absorb_undersized(sections: list[dict], max_tokens: int) -> list[dict]:
     # Check if first chunk is undersized and can be prepended to next
     if len(result) >= 2:
         first_text = _section_text(result[0])
-        if count_tokens(first_text) < threshold:
+        if count_tokens(first_text) < threshold and fits(first_text, _section_text(result[1])):
             second = result[1]
             second_text = _section_text(second)
             result[1] = {

{dot_parser-2.1.0 → dot_parser-2.1.2}/tests/test_chunking.py

@@ -1,7 +1,10 @@
 # SPDX-FileCopyrightText: Kannon For Deep Tech
 # SPDX-License-Identifier: AGPL-3.0-or-later
 
+import pytest
+
 from dot_parser import Chunk, chunk
+from dot_parser.tokens import count_tokens
 
 
 class TestChunkBasic:
@@ -62,6 +65,28 @@ class TestSplitOversized:
             assert c.section_path == ["# Big"]
 
 
+class TestMaxTokensBound:
+    @pytest.mark.parametrize("merge", [True, False])
+    def test_an_oversized_paragraph_among_others_is_split(self, merge):
+        md = "# Big\n\nIntro.\n\n" + "word " * 2000 + "\n\nOutro."
+        chunks = chunk(md, max_tokens=64, merge=merge)
+        assert all(count_tokens(c.content) <= 64 for c in chunks)
+        assert chunks[0].heading == "# Big"
+
+    @pytest.mark.parametrize("merge", [True, False])
+    def test_a_table_without_blank_lines_is_split(self, merge):
+        table = "\n".join(f"| row {i} | 0.{i:04d} | 12.5 |" for i in range(500))
+        md = f"# Results\n\nThe table below.\n\n{table}\n\nEnd."
+        chunks = chunk(md, max_tokens=64, merge=merge)
+        assert all(count_tokens(c.content) <= 64 for c in chunks)
+
+    def test_a_run_of_heading_only_sections_stays_within_the_bound(self):
+        md = "# Index\n\n" + "\n\n".join(f"## Entry {i}" for i in range(300))
+        chunks = chunk(md, max_tokens=64)
+        assert all(count_tokens(c.content) <= 64 for c in chunks)
+        assert "## Entry 299" in chunks[-1].content
+
+
 class TestMerge:
     def test_small_sections_merged(self):
         md = "# A\n\n## B\n\nShort.\n\n## C\n\nAlso short."

{dot_parser-2.1.0 → dot_parser-2.1.2}/tests/test_images.py

@@ -129,7 +129,7 @@ def _make_mistral_with_fake_client(pages):
     constructing the instance manually.
     """
     instance = Mistral.__new__(Mistral)
-    instance._model = "mistral-ocr-4-0"
+    instance._model = "mistral-ocr-4-1"
     instance._language = None
 
     uploaded = SimpleNamespace(id="file-xyz")
@@ -261,7 +261,7 @@ class TestMistralParsePdfWithImages:
         ]
 
         instance = Mistral.__new__(Mistral)
-        instance._model = "mistral-ocr-4-0"
+        instance._model = "mistral-ocr-4-1"
         instance._language = None
         uploaded = SimpleNamespace(id="file-xyz")
         signed = SimpleNamespace(url="https://signed.example/doc.pdf")
@@ -297,7 +297,7 @@ class TestMistralParsePdfWithImages:
     def test_retry_failure_propagates(self):
         """If the no-annotation retry also fails, the exception escapes."""
         instance = Mistral.__new__(Mistral)
-        instance._model = "mistral-ocr-4-0"
+        instance._model = "mistral-ocr-4-1"
         instance._language = None
         ocr_calls: list[dict] = []
 

{dot_parser-2.1.0 → dot_parser-2.1.2}/tests/test_mistral_batch_logging.py

@@ -63,7 +63,7 @@ def _backend(job, downloads: dict[str, bytes]) -> Mistral:
     no ``mistralai`` import / real key is needed."""
     m = Mistral.__new__(Mistral)
     m._client = _FakeClient(job, downloads)  # ty: ignore[invalid-assignment]
-    m._model = "mistral-ocr-4-0"
+    m._model = "mistral-ocr-4-1"
     return m
 
 

{dot_parser-2.1.0 → dot_parser-2.1.2}/uv.lock

@@ -654,7 +654,7 @@ standard = [
 
 [[package]]
 name = "dot-parser"
-version = "2.1.0"
+version = "2.1.2"
 source = { editable = "." }
 dependencies = [
     { name = "beautifulsoup4" },