dot-parser 2.0.0__tar.gz → 2.1.1__tar.gz

dot_parser-2.1.1/CHANGELOG.md

@@ -0,0 +1,96 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+## [2.1.1] - 2026-09-30
+
+### Changed
+
+- `Mistral` now defaults to `mistral-ocr-4-1`; `mistral-ocr-4-0` retires on
+  2026-09-30. Pricing is unchanged.
+
+## [2.1.0] - 2026-09-30
+
+### Added
+
+- `extract_docx_toc()` returns the table of contents stored in a DOCX as
+  `TocEntry(level, text)` items (`TocEntry` is exported), whether Word wrapped
+  it in a content control or inserted it as a bare field.
+- `min_image_size=N` on `parse_with_images()`, `parse_docx_with_images()` and
+  `parse_pptx_with_images()` drops images whose shorter side is under `N`
+  pixels, with their anchors: letters or words pasted as pictures, icons,
+  separator lines. On the markitdown + VLM path they are dropped before
+  interpretation, so they cost no VLM call. `0` (default) keeps every image;
+  images whose size can't be read (EMF, WMF, SVG) are always kept.
+
+### Changed
+
+- DOCX conversion (`parse()` and the markitdown + VLM image path) now runs
+  through one converter, so a DOCX yields the same text with or without image
+  extraction. It keeps markitdown's pipeline and fixes, in the Markdown it
+  produces:
+  - headings in custom styles derived from `Heading N` or carrying an outline
+    level (`Style Titre 2`, `Annexe`...) are emitted as headings instead of
+    body text;
+  - headings carry Word's automatic numbers (`# 1. Introduction`,
+    `## 2.3 Objet`), recomputed from `numbering.xml`; a number already typed
+    in the heading is not doubled;
+  - `Title` and `Subtitle` paragraphs (cover page) are rendered bold; like
+    Word's own TOC, they are not treated as headings;
+  - bold inside a heading is dropped (`## Zoom`, not `## **Zoom**`);
+  - the stored table of contents and the lists of figures and tables are
+    left out: they repeat headings and captions with page numbers, and
+    `extract_docx_toc()` exposes the TOC;
+  - tables use their first row as the header instead of a blank one, escape
+    `|` in cells, stay rectangular across merged cells (a vertically merged
+    value is repeated in each row it spans) and flatten nested tables into
+    their parent cell.
+- `parse()` drops embedded images from a DOCX instead of emitting
+  `![alt](data:image/png;base64...)` anchors with a truncated payload.
+- `parse()` raises `ParseError` on a corrupted DOCX instead of returning its
+  raw bytes as text.
+- `mammoth` and `beautifulsoup4` are declared as direct dependencies.
+
+## [2.0.0] - 2026-07-30
+
+First public release.
+
+### Added
+
+- `parse()` converts PDF, DOCX, PPTX, HTML, XLSX, CSV, Markdown and plain text
+  into clean Markdown.
+- Selectable PDF backends: `Pymu` (default, local and fast), `Docling` (local
+  layout-aware ML pipeline), `Mistral` (cloud OCR) and `Llama` (LlamaCloud).
+- `parse_with_images()` extracts images alongside the Markdown, with anchors
+  kept at their position in the text. Images are described either by a VLM or
+  by Mistral OCR annotations in the same call.
+- `parse_pdfs()` parses many PDFs in one batch job through the Mistral Batch
+  API, falling back to a per-file loop on backends without batch support.
+- `chunk()` splits Markdown into chunks carrying their heading hierarchy in
+  `section_path`.
+- Cost helpers: `cost_per_1k_pages()` and `estimate_cost()`, usable without
+  instantiating a backend.
+- Backend protocols (`Backend`, `ImageBackend`, `BatchBackend`,
+  `DocxImageBackend`, `PptxImageBackend`) so third-party backends can plug in.
+
+### Changed
+
+These matter only if you were installing the package straight from git before
+this release.
+
+- `LlamaTier` is a `Literal` of the four accepted tiers rather than a bare
+  `str`, and is defined once alongside the pricing table.
+- `zip()` calls over sequences that must line up now use `strict=True`: a length
+  mismatch raises instead of silently truncating.
+- `parse_with_images(backend=...)` accepts DOCX- and PPTX-capable backends in
+  its type signature, matching what the function already supported at runtime.
+
+[Unreleased]: https://gitlab.com/deepika6190303/deepika-open-toolbox/dot-parser/-/compare/v2.1.1...main
+[2.1.1]: https://gitlab.com/deepika6190303/deepika-open-toolbox/dot-parser/-/compare/v2.1.0...v2.1.1
+[2.1.0]: https://gitlab.com/deepika6190303/deepika-open-toolbox/dot-parser/-/compare/v2.0.0...v2.1.0
+[2.0.0]: https://gitlab.com/deepika6190303/deepika-open-toolbox/dot-parser/-/tags/v2.0.0

{dot_parser-2.0.0 → dot_parser-2.1.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: dot-parser
-Version: 2.0.0
+Version: 2.1.1
 Summary: Document-to-markdown parser and chunker for RAG pipelines
 Project-URL: Homepage, https://gitlab.com/deepika6190303/deepika-open-toolbox/dot-parser
 Project-URL: Repository, https://gitlab.com/deepika6190303/deepika-open-toolbox/dot-parser
@@ -15,6 +15,8 @@ Classifier: Programming Language :: Python :: 3
 Classifier: Programming Language :: Python :: 3.12
 Classifier: Programming Language :: Python :: 3.13
 Requires-Python: <3.14,>=3.12
+Requires-Dist: beautifulsoup4>=4.12
+Requires-Dist: mammoth>=1.11
 Requires-Dist: markitdown[all]>=0.1
 Requires-Dist: pillow>=10.0
 Requires-Dist: pydantic>=2
@@ -152,7 +154,7 @@ described either by a VLM or by OCR annotations from the same call. See
 
 - **`Pymu(ocr_fallback=True)`** -- pymupdf4llm, default. Fast and CPU-only. The built-in OCR fallback kicks in when an OCR engine (tesseract, rapidocr, paddleocr) is available. Set `ocr_fallback=False` to force the no-OCR code path (matches a runtime image without OCR shipped).
 - **`Docling(force_full_page_ocr=True)`** -- IBM Docling, layout-aware ML pipeline + Tesseract OCR. Robust on image-only and broken-encoding PDFs. Install with `pip install dot-parser[docling]`.
-- **`Mistral(api_key=None, model="mistral-ocr-4-0")`** -- Mistral OCR cloud API. Reads `MISTRAL_API_KEY` from env. Implements `parse_pdfs` via the Batch API. Install with `pip install dot-parser[mistral]`.
+- **`Mistral(api_key=None, model="mistral-ocr-4-1")`** -- Mistral OCR cloud API. Reads `MISTRAL_API_KEY` from env. Implements `parse_pdfs` via the Batch API. Install with `pip install dot-parser[mistral]`.
 - **`Llama(api_key=None, tier="cost_effective")`** -- LlamaCloud parsing API. Reads `LLAMA_CLOUD_API_KEY` from env. Tiers: `fast`, `cost_effective`, `agentic`, `agentic_plus`. Install with `pip install dot-parser[llama]`.
 
 #### Mistral OCR enrichments
@@ -235,7 +237,7 @@ package is covered, anything underscore-prefixed is internal and may change in
 any release. Public names are never removed without a deprecation period.
 
 ```toml
-dependencies = ["dot-parser>=1.0,<2"]
+dependencies = ["dot-parser>=2.0,<3"]
 ```
 
 See [docs/VERSIONING.md](docs/VERSIONING.md) for the full policy.

{dot_parser-2.0.0 → dot_parser-2.1.1}/README.md

@@ -116,7 +116,7 @@ described either by a VLM or by OCR annotations from the same call. See
 
 - **`Pymu(ocr_fallback=True)`** -- pymupdf4llm, default. Fast and CPU-only. The built-in OCR fallback kicks in when an OCR engine (tesseract, rapidocr, paddleocr) is available. Set `ocr_fallback=False` to force the no-OCR code path (matches a runtime image without OCR shipped).
 - **`Docling(force_full_page_ocr=True)`** -- IBM Docling, layout-aware ML pipeline + Tesseract OCR. Robust on image-only and broken-encoding PDFs. Install with `pip install dot-parser[docling]`.
-- **`Mistral(api_key=None, model="mistral-ocr-4-0")`** -- Mistral OCR cloud API. Reads `MISTRAL_API_KEY` from env. Implements `parse_pdfs` via the Batch API. Install with `pip install dot-parser[mistral]`.
+- **`Mistral(api_key=None, model="mistral-ocr-4-1")`** -- Mistral OCR cloud API. Reads `MISTRAL_API_KEY` from env. Implements `parse_pdfs` via the Batch API. Install with `pip install dot-parser[mistral]`.
 - **`Llama(api_key=None, tier="cost_effective")`** -- LlamaCloud parsing API. Reads `LLAMA_CLOUD_API_KEY` from env. Tiers: `fast`, `cost_effective`, `agentic`, `agentic_plus`. Install with `pip install dot-parser[llama]`.
 
 #### Mistral OCR enrichments
@@ -199,7 +199,7 @@ package is covered, anything underscore-prefixed is internal and may change in
 any release. Public names are never removed without a deprecation period.
 
 ```toml
-dependencies = ["dot-parser>=1.0,<2"]
+dependencies = ["dot-parser>=2.0,<3"]
 ```
 
 See [docs/VERSIONING.md](docs/VERSIONING.md) for the full policy.

{dot_parser-2.0.0 → dot_parser-2.1.1}/benchmark/README.md

@@ -72,5 +72,5 @@ dropped equations, wrong table cells), so we keep it purely manual.
 | `pymu` | $0.00 | pymupdf4llm with OCR fallback **disabled** — reproduces le-lab production where no OCR engine is shipped in the runtime image |
 | `pymu_ocr` | $0.00 | pymupdf4llm with built-in OCR fallback **enabled** — uses tesseract (or rapidocr/paddleocr) automatically when text extraction looks suspect |
 | `docling` | $0.00 | layout-aware ML pipeline + Tesseract OCR with `force_full_page_ocr=True` |
-| `mistral` | $4.00 | `mistral-ocr-4-0` cloud API. With `--mistral-batch`: $2.00 per 1k pages (50% off) via the Mistral Batch API. |
+| `mistral` | $4.00 | `mistral-ocr-4-1` cloud API. With `--mistral-batch`: $2.00 per 1k pages (50% off) via the Mistral Batch API. |
 | `llama` | $3.00 | LlamaCloud `cost_effective` tier |

{dot_parser-2.0.0 → dot_parser-2.1.1}/docs/IMAGE_EXTRACTION.md

@@ -30,7 +30,7 @@ parse_with_images("doc.docx", backend=Mistral())
 
 The `.docx` is uploaded to Mistral and `ocr.process` is called with `document_url`
 plus `bbox_annotation_format`: a single call returns markdown, images and
-annotations. Model: `mistral-ocr-latest`.
+annotations. Model: `mistral-ocr-4-1`.
 
 - **+** One API call, no per-image round trip.
 - **+** Handles natively the formats markitdown cannot pass through (the document is rasterised, so SmartArt / EMF / SVG render correctly).
@@ -68,7 +68,7 @@ parse_with_images("doc.pdf")  # backend=Mistral() implied
 parse_with_images("doc.pdf", backend=Mistral())  # explicit, identical
 ```
 
-Same `bbox_annotation_format` flow as DOCX path B. Model: `mistral-ocr-latest`.
+Same `bbox_annotation_format` flow as DOCX path B. Model: `mistral-ocr-4-1`.
 
 - **+** One call returns markdown, images and annotations; robust on scanned or broken-encoding PDFs; exact image positions.
 - **−** Billed per page; fixed annotation schema (same caveat as DOCX path B).

{dot_parser-2.0.0 → dot_parser-2.1.1}/docs/VERSIONING.md

@@ -1,7 +1,7 @@
 # Versioning and stability
 
 `dot-parser` follows [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
-This page says what that actually covers, so `>=1.0,<2` means something concrete.
+This page says what that actually covers, so `>=2.0,<3` means something concrete.
 
 ## What is public
 
@@ -94,11 +94,11 @@ manual. Use them to validate a release without committing to it.
 ### From PyPI — the supported way
 
 ```toml
-dependencies = ["dot-parser>=1.0,<2"]
+dependencies = ["dot-parser>=2.0,<3"]
 ```
 
 The upper bound is what makes the guarantees above useful: you get every fix and
-feature of the 1.x line, and never an unannounced break.
+feature of the 2.x line, and never an unannounced break.
 
 ### From git — development only
 

{dot_parser-2.0.0 → dot_parser-2.1.1}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "dot-parser"
-version = "2.0.0"
+version = "2.1.1"
 description = "Document-to-markdown parser and chunker for RAG pipelines"
 readme = "README.md"
 authors = [{ name = "Kannon For Deep Tech", email = "louis.letarnec@deepika.ai" }]
@@ -12,6 +12,10 @@ requires-python = ">=3.12,<3.14"
 dependencies = [
     "pymupdf4llm>=1.27.2.2",
     "markitdown[all]>=0.1",
+    # Imported directly by the DOCX converter (docx_markdown.py); markitdown
+    # pulls both in too, but only through its optional extras.
+    "mammoth>=1.11",
+    "beautifulsoup4>=4.12",
     "semchunk>=3.0",
     "python-docx>=1.2.0",
     "pillow>=10.0",

{dot_parser-2.0.0 → dot_parser-2.1.1}/src/dot_parser/__init__.py

@@ -5,6 +5,7 @@ from importlib.metadata import version
 
 from dot_parser.backends import Backend, Docling, ImageBackend, Llama, Mistral, Pymu
 from dot_parser.chunking import chunk
+from dot_parser.docx_markdown import extract_docx_toc
 from dot_parser.images import (
     VLM,
     ExtractedImage,
@@ -12,7 +13,7 @@ from dot_parser.images import (
     PageInfo,
     ParseResult,
 )
-from dot_parser.models import Chunk, ParseError
+from dot_parser.models import Chunk, ParseError, TocEntry
 from dot_parser.parsers import (
     interpret_images,
     parse,
@@ -39,9 +40,11 @@ __all__ = [
     "ParseError",
     "ParseResult",
     "Pymu",
+    "TocEntry",
     "chunk",
     "cost_per_1k_pages",
     "estimate_cost",
+    "extract_docx_toc",
     "interpret_images",
     "parse",
     "parse_pdfs",

{dot_parser-2.0.0 → dot_parser-2.1.1}/src/dot_parser/backends/mistral.py

@@ -217,7 +217,7 @@ class Mistral:
         self,
         *,
         api_key: str | None = None,
-        model: str = "mistral-ocr-4-0",
+        model: str = "mistral-ocr-4-1",
         language: str | None = None,
         extract_headers_footers: bool = False,
         confidence_scores: Literal["page", "word"] | None = None,

{dot_parser-2.0.0 → dot_parser-2.1.1}/src/dot_parser/docx_images.py

@@ -34,7 +34,7 @@ import tempfile
 import time
 from pathlib import Path
 
-from dot_parser.image_utils import dedupe_titles
+from dot_parser.image_utils import dedupe_titles, drop_small_images
 from dot_parser.images import (
     VLM,
     ExtractedImage,
@@ -523,7 +523,15 @@ def _office_to_markdown_via_markitdown(file_path: str) -> str:
     first comma (replacing the base64 payload with ``...``) to keep
     markdown small for the LLM-summarisation use case it was designed
     for. We need the full payload to extract the image bytes.
+
+    DOCX goes through :func:`dot_parser.docx_markdown.docx_to_markdown`,
+    the same converter :func:`dot_parser.parse` uses, so the text is
+    identical with or without image extraction.
     """
+    if Path(file_path).suffix.lower() == ".docx":
+        from dot_parser.docx_markdown import docx_to_markdown
+
+        return docx_to_markdown(file_path, keep_data_uris=True)
     try:
         import markitdown
     except ImportError as e:
@@ -540,6 +548,7 @@ def _parse_office_with_images(
     suffix: str,
     include_tables: bool,
     deadline: float | None = None,
+    min_image_size: int = 0,
 ) -> ParseResult:
     """Shared markitdown+VLM pipeline for DOCX and PPTX sources."""
     if isinstance(source, bytes):
@@ -555,6 +564,8 @@ def _parse_office_with_images(
         raw_md = _office_to_markdown_via_markitdown(str(source))
 
     markdown, images = _extract_data_url_images(raw_md)
+    # Before interpretation, so dropped images cost no VLM call.
+    markdown, images = drop_small_images(markdown, images, min_image_size)
     if not include_tables:
         markdown = strip_markdown_tables(markdown)
     if images:
@@ -574,6 +585,7 @@ def parse_docx_with_images(
     *,
     include_tables: bool = True,
     deadline: float | None = None,
+    min_image_size: int = 0,
 ) -> ParseResult:
     """Parse a DOCX into markdown + interpreted images.
 
@@ -599,6 +611,7 @@ def parse_docx_with_images(
         suffix=".docx",
         include_tables=include_tables,
         deadline=deadline,
+        min_image_size=min_image_size,
     )
 
 
@@ -608,6 +621,7 @@ def parse_pptx_with_images(
     *,
     include_tables: bool = True,
     deadline: float | None = None,
+    min_image_size: int = 0,
 ) -> ParseResult:
     """Parse a PPTX into markdown + interpreted embedded pictures.
 
@@ -630,4 +644,5 @@ def parse_pptx_with_images(
         suffix=".pptx",
         include_tables=include_tables,
         deadline=deadline,
+        min_image_size=min_image_size,
     )