rakam-systems-documents 0.1.0__tar.gz

rakam_systems_documents-0.1.0/PKG-INFO

@@ -0,0 +1,68 @@
+Metadata-Version: 2.3
+Name: rakam-systems-documents
+Version: 0.1.0
+Summary: Generic document preparation: any source -> PreparedContent (markdown + structured rows + provenance)
+Author: Roman Grebnev
+License: MIT
+Requires-Dist: pydantic>=2.10.6
+Requires-Dist: pymupdf4llm>=0.0.17,<0.0.18
+Requires-Dist: pymupdf>=1.24.0
+Requires-Dist: openpyxl>=3.1.0
+Requires-Dist: httpx>=0.27.0
+Requires-Dist: docling==2.62.0 ; extra == 'docling'
+Requires-Python: >=3.10
+Provides-Extra: docling
+Description-Content-Type: text/markdown
+
+# rakam-systems-documents
+
+Generic **document preparation**: turn any source document into clean, structured,
+provenance-tagged content the rest of the stack can reason over.
+
+```python
+from rakam_systems_documents import prepare, MistralOCRProvider
+
+pc = prepare(raw_bytes, mime="application/pdf", filename="invoice.pdf",
+             ocr=MistralOCRProvider())      # ocr optional; native PDFs skip it
+pc.markdown      # what a model reads
+pc.rows          # structured rows (tabular sources), header-keyed + provenance
+pc.provenance    # page/sheet/row refs
+pc.provider      # pdf_text | mistral_ocr | docling_ocr | tabular | text | none
+```
+
+## What it is (and isn't)
+
+**One job:** `bytes -> PreparedContent`. Format dispatch, parse/decode, hybrid-OCR
+routing, provenance normalization, truncation. A pure function — the only I/O is the
+injected OCR provider.
+
+**Not** its job: storage, chunking, embedding, indexing, async/status, or any business
+meaning (extraction, matching, classification). Those belong to the caller — the library
+gives every caller one document→content primitive so none reimplements parsing/OCR.
+
+## Formats
+
+| Source | Handling |
+|---|---|
+| Native PDF | `pymupdf4llm` → markdown (text layer, exact, no egress) |
+| Scanned PDF / image | **hybrid gate** → OCR provider (see below) |
+| Excel / CSV | `openpyxl` / `csv` → markdown table **+ structured rows + (sheet,row) provenance** |
+| Email / text | rfc822 + multi-encoding decode (binary rejected) |
+
+## OCR is pluggable
+
+Native-text PDFs never hit OCR. Scanned pages / images route to an `OCRProvider`:
+
+- **`MistralOCRProvider`** (default) — the Mistral hosted OCR API (`/v1/ocr`). Needs `MISTRAL_API_KEY`.
+- **`DoclingOCRProvider`** — on-prem, no egress. Requires the optional extra:
+  `pip install rakam-systems-documents[docling]`.
+
+A missing/unavailable engine degrades gracefully (native text still works; a scan yields
+`provider="none"` + a help hint, never an exception).
+
+## Install
+
+```
+pip install rakam-systems-documents            # Mistral OCR path
+pip install rakam-systems-documents[docling]    # + on-prem OCR
+```

rakam_systems_documents-0.1.0/README.md

@@ -0,0 +1,52 @@
+# rakam-systems-documents
+
+Generic **document preparation**: turn any source document into clean, structured,
+provenance-tagged content the rest of the stack can reason over.
+
+```python
+from rakam_systems_documents import prepare, MistralOCRProvider
+
+pc = prepare(raw_bytes, mime="application/pdf", filename="invoice.pdf",
+             ocr=MistralOCRProvider())      # ocr optional; native PDFs skip it
+pc.markdown      # what a model reads
+pc.rows          # structured rows (tabular sources), header-keyed + provenance
+pc.provenance    # page/sheet/row refs
+pc.provider      # pdf_text | mistral_ocr | docling_ocr | tabular | text | none
+```
+
+## What it is (and isn't)
+
+**One job:** `bytes -> PreparedContent`. Format dispatch, parse/decode, hybrid-OCR
+routing, provenance normalization, truncation. A pure function — the only I/O is the
+injected OCR provider.
+
+**Not** its job: storage, chunking, embedding, indexing, async/status, or any business
+meaning (extraction, matching, classification). Those belong to the caller — the library
+gives every caller one document→content primitive so none reimplements parsing/OCR.
+
+## Formats
+
+| Source | Handling |
+|---|---|
+| Native PDF | `pymupdf4llm` → markdown (text layer, exact, no egress) |
+| Scanned PDF / image | **hybrid gate** → OCR provider (see below) |
+| Excel / CSV | `openpyxl` / `csv` → markdown table **+ structured rows + (sheet,row) provenance** |
+| Email / text | rfc822 + multi-encoding decode (binary rejected) |
+
+## OCR is pluggable
+
+Native-text PDFs never hit OCR. Scanned pages / images route to an `OCRProvider`:
+
+- **`MistralOCRProvider`** (default) — the Mistral hosted OCR API (`/v1/ocr`). Needs `MISTRAL_API_KEY`.
+- **`DoclingOCRProvider`** — on-prem, no egress. Requires the optional extra:
+  `pip install rakam-systems-documents[docling]`.
+
+A missing/unavailable engine degrades gracefully (native text still works; a scan yields
+`provider="none"` + a help hint, never an exception).
+
+## Install
+
+```
+pip install rakam-systems-documents            # Mistral OCR path
+pip install rakam-systems-documents[docling]    # + on-prem OCR
+```

rakam_systems_documents-0.1.0/pyproject.toml

@@ -0,0 +1,52 @@
+[build-system]
+requires = ["uv_build >= 0.7.19, <0.9.0"]
+build-backend = "uv_build"
+
+[project]
+name = "rakam-systems-documents"
+version = "0.1.0"
+description = "Generic document preparation: any source -> PreparedContent (markdown + structured rows + provenance)"
+readme = "README.md"
+requires-python = ">=3.10"
+license = { text = "MIT" }
+authors = [
+    { name = "Roman Grebnev" }
+]
+dependencies = [
+    "pydantic>=2.10.6",
+    # Cap below 0.0.18: that release pulls in `pymupdf-layout` -> `onnxruntime`,
+    # a heavy ML dep whose py<3.11 pin (onnxruntime 1.24.x) has no cp310 wheels —
+    # it breaks the 3.10 build and defeats this package's light, no-egress goal.
+    "pymupdf4llm>=0.0.17,<0.0.18",
+    "pymupdf>=1.24.0",
+    "openpyxl>=3.1.0",
+    "httpx>=0.27.0",
+]
+
+[project.optional-dependencies]
+# On-prem, no-egress OCR. Optional — the default (hosted) OCR path needs none of this.
+docling = ["docling==2.62.0"]
+
+[dependency-groups]
+dev = [
+    "build>=1.2.2.post1",
+    "pytest>=8.3.5",
+    "pytest-cov>=5.0.0",
+    "twine>=6.1.0",
+]
+
+[tool.isort]
+profile = "black"
+line_length = 88
+
+[tool.black]
+line-length = 88
+target-version = ["py311"]
+
+[tool.uv.build-backend]
+module-root = "src"
+exclude = ["**/tests/**"]
+
+[tool.pytest.ini_options]
+testpaths = ["src/rakam_systems_documents/tests"]
+python_files = "test_*.py"

rakam_systems_documents-0.1.0/src/rakam_systems_documents/__init__.py

@@ -0,0 +1,24 @@
+"""rakam-systems-documents — generic document preparation.
+
+``prepare(bytes, mime, filename) -> PreparedContent``: any source (native/scanned
+PDF, image, Excel/CSV, email/text) to markdown + structured rows + provenance.
+A pure, side-effect-free primitive; storage / chunking / embedding / extraction
+belong to the caller.
+"""
+from __future__ import annotations
+
+from .prepare import prepare
+from .providers import DoclingOCRProvider, MistralOCRProvider, OCRProvider
+from .schema import PreparedContent, SourceRef, TableRow
+
+__version__ = "0.1.0"
+
+__all__ = [
+    "prepare",
+    "PreparedContent",
+    "TableRow",
+    "SourceRef",
+    "OCRProvider",
+    "MistralOCRProvider",
+    "DoclingOCRProvider",
+]

rakam_systems_documents-0.1.0/src/rakam_systems_documents/prepare.py

@@ -0,0 +1,184 @@
+"""The single entry point: ``prepare(bytes, mime, filename) -> PreparedContent``.
+
+A pure orchestrator — format dispatch + parse/decode + hybrid-OCR routing +
+provenance normalization + truncation. No storage, no chunking, no embedding, no
+async, no business meaning. The only I/O is the injected OCR provider.
+
+Parsing mirrors the ``rakam_systems`` vectorstore loaders (``pymupdf4llm`` for
+PDFs, ``openpyxl`` for spreadsheets) without importing that heavy package, so
+this stays a light dependency.
+"""
+from __future__ import annotations
+
+import base64  # noqa: F401  (kept for symmetry with providers; harmless)
+import csv
+import io
+
+from .providers import OCRProvider
+from .schema import PreparedContent, SourceRef, TableRow
+
+# Below this many stripped characters a PDF page is treated as "no text layer"
+# (i.e. scanned) and routed to OCR.
+_PDF_TEXT_MIN = 20
+
+
+def prepare(
+    content: bytes,
+    mime: str,
+    filename: str,
+    ocr: OCRProvider | None = None,
+    max_chars: int = 8000,
+) -> PreparedContent:
+    """Turn one document into ``PreparedContent``. Never raises — an unreadable
+    file yields ``provider="none"`` + a ``meta['help']`` hint."""
+    ext = filename.rsplit(".", 1)[-1].lower() if "." in filename else ""
+    try:
+        if mime == "application/pdf" or ext == "pdf" or content[:5] == b"%PDF-":
+            pc = _prepare_pdf(content, mime, ocr)
+        elif mime.startswith("image/") or ext in ("png", "jpg", "jpeg", "tiff", "webp"):
+            pc = _prepare_image(content, mime, ocr)
+        elif ext in ("xlsx", "xlsm") or "spreadsheetml" in mime or "ms-excel" in mime:
+            pc = _prepare_xlsx(content)
+        elif ext == "csv" or mime == "text/csv":
+            pc = _prepare_csv(content)
+        elif ext == "eml" or mime == "message/rfc822":
+            pc = _prepare_email(content)
+        else:
+            text = _decode_text(content)
+            pc = PreparedContent(
+                markdown=text or "",
+                provider="text" if text else "none",
+                meta={} if text else {"help": "could not read this file (binary?)"},
+            )
+    except Exception as exc:  # noqa: BLE001 — best-effort: a bad file must never raise
+        pc = PreparedContent(provider="none", meta={"help": f"processing failed: {exc}"})
+
+    if len(pc.markdown) > max_chars:
+        pc.markdown = pc.markdown[:max_chars]
+        pc.truncated = True
+    return pc
+
+
+# ── PDF (hybrid: native text first, OCR for scans) ─────────────────────────
+def _pdf_text_len(content: bytes) -> tuple[int, int]:
+    import fitz
+
+    with fitz.open(stream=content, filetype="pdf") as doc:
+        total = sum(len(page.get_text().strip()) for page in doc)
+        return total, doc.page_count
+
+
+def _prepare_pdf(content: bytes, mime: str, ocr: OCRProvider | None) -> PreparedContent:
+    text_len, pages = _pdf_text_len(content)
+    if text_len >= _PDF_TEXT_MIN:
+        import fitz
+        import pymupdf4llm
+
+        with fitz.open(stream=content, filetype="pdf") as doc:
+            markdown = pymupdf4llm.to_markdown(doc)
+        return PreparedContent(
+            markdown=markdown,
+            provider="pdf_text",
+            provenance=[SourceRef(page=i + 1) for i in range(pages)],
+            meta={"page_count": pages},
+        )
+    if ocr and ocr.available():
+        markdown, provenance = ocr.ocr(content, mime)
+        return PreparedContent(
+            markdown=markdown,
+            provider=f"{ocr.name}_ocr",
+            provenance=provenance,
+            meta={"page_count": len(provenance) or pages},
+        )
+    return PreparedContent(
+        provider="none",
+        meta={"page_count": pages, "help": "no text layer and OCR unavailable"},
+    )
+
+
+def _prepare_image(content: bytes, mime: str, ocr: OCRProvider | None) -> PreparedContent:
+    if ocr and ocr.available():
+        markdown, provenance = ocr.ocr(content, mime)
+        return PreparedContent(markdown=markdown, provider=f"{ocr.name}_ocr", provenance=provenance)
+    return PreparedContent(provider="none", meta={"help": "image needs OCR; provider unavailable"})
+
+
+# ── tabular (markdown table + structured rows + provenance) ────────────────
+def _rows_to_markdown(headers: list[str], data_rows: list[list[str]]) -> str:
+    lines = ["| " + " | ".join(headers) + " |", "| " + " | ".join("---" for _ in headers) + " |"]
+    lines += ["| " + " | ".join(r) + " |" for r in data_rows]
+    return "\n".join(lines)
+
+
+def _prepare_xlsx(content: bytes) -> PreparedContent:
+    import openpyxl
+
+    wb = openpyxl.load_workbook(io.BytesIO(content), read_only=True, data_only=True)
+    sheet_names = [ws.title for ws in wb.worksheets]
+    md_blocks: list[str] = []
+    rows_out: list[TableRow] = []
+    for ws in wb.worksheets:
+        headers: list[str] | None = None
+        data_rows: list[list[str]] = []
+        for r_idx, row in enumerate(ws.iter_rows(values_only=True), start=1):
+            vals = ["" if c is None else str(c) for c in row]
+            if headers is None:
+                headers = vals
+                continue
+            data_rows.append(vals)
+            cells = {(headers[i] if i < len(headers) else f"col{i}"): vals[i] for i in range(len(vals))}
+            rows_out.append(TableRow(cells=cells, source=SourceRef(sheet=ws.title, row=r_idx)))
+        if headers:
+            md_blocks.append(f"### {ws.title}\n" + _rows_to_markdown(headers, data_rows))
+    wb.close()
+    return PreparedContent(
+        markdown="\n\n".join(md_blocks),
+        rows=rows_out,
+        provider="tabular",
+        meta={"sheet_names": sheet_names},
+    )
+
+
+def _prepare_csv(content: bytes) -> PreparedContent:
+    text = _decode_text(content) or ""
+    reader = list(csv.reader(io.StringIO(text)))
+    if not reader:
+        return PreparedContent(provider="text")
+    headers, data = reader[0], reader[1:]
+    rows_out = [
+        TableRow(
+            cells={(headers[i] if i < len(headers) else f"col{i}"): vals[i] for i in range(len(vals))},
+            source=SourceRef(row=r_idx),
+        )
+        for r_idx, vals in enumerate(data, start=2)
+    ]
+    return PreparedContent(
+        markdown=_rows_to_markdown(headers, data),
+        rows=rows_out,
+        provider="tabular",
+        meta={"row_count": len(rows_out)},
+    )
+
+
+# ── email / text ───────────────────────────────────────────────────────────
+def _decode_text(content: bytes) -> str | None:
+    if b"\x00" in content[:4096]:  # reject binary masquerading as text
+        return None
+    for encoding in ("utf-8-sig", "utf-8", "latin-1"):
+        try:
+            return content.decode(encoding)
+        except UnicodeDecodeError:
+            continue
+    return None
+
+
+def _prepare_email(content: bytes) -> PreparedContent:
+    import email
+    from email import policy
+
+    msg = email.message_from_bytes(content, policy=policy.default)
+    body = msg.get_body(preferencelist=("plain", "html"))
+    text = body.get_content() if body else ""
+    subject = msg.get("subject", "")
+    markdown = f"# {subject}\n\n**From:** {msg.get('from', '')}\n\n{text}".strip()
+    return PreparedContent(markdown=markdown, provider="text", meta={"subject": subject})

rakam_systems_documents-0.1.0/src/rakam_systems_documents/providers/__init__.py

@@ -0,0 +1,6 @@
+"""OCR providers — the pluggable, engine-agnostic seam for scanned documents."""
+from __future__ import annotations
+
+from .ocr import DoclingOCRProvider, MistralOCRProvider, OCRProvider
+
+__all__ = ["OCRProvider", "MistralOCRProvider", "DoclingOCRProvider"]

rakam_systems_documents-0.1.0/src/rakam_systems_documents/providers/ocr.py

@@ -0,0 +1,96 @@
+"""OCR for scanned pages / images — behind one interface so the engine is a
+config choice, never a fork. Mistral is the default (hosted API); Docling is the
+on-prem, no-egress swap (optional extra ``rakam-systems-documents[docling]``).
+"""
+from __future__ import annotations
+
+import base64
+import os
+from typing import Protocol, runtime_checkable
+
+from ..schema import SourceRef
+
+
+@runtime_checkable
+class OCRProvider(Protocol):
+    """An OCR engine. ``ocr`` returns ``(markdown, provenance)`` and must be
+    best-effort — the caller treats an empty return as "unreadable", never an
+    exception path."""
+
+    name: str
+
+    def available(self) -> bool: ...
+
+    def ocr(self, content: bytes, mime: str) -> tuple[str, list[SourceRef]]: ...
+
+
+class MistralOCRProvider:
+    """Default engine — the Mistral hosted OCR API (``/v1/ocr``)."""
+
+    name = "mistral"
+
+    def __init__(
+        self,
+        api_key: str | None = None,
+        model: str = "mistral-ocr-latest",
+        timeout: float = 90.0,
+    ) -> None:
+        self.api_key = api_key or os.getenv("MISTRAL_API_KEY")
+        self.model = os.getenv("OCR_MODEL", model)
+        self.timeout = timeout
+
+    def available(self) -> bool:
+        return bool(self.api_key)
+
+    def ocr(self, content: bytes, mime: str) -> tuple[str, list[SourceRef]]:
+        import httpx
+
+        is_pdf = mime == "application/pdf" or content[:5] == b"%PDF-"
+        field = "document_url" if is_pdf else "image_url"
+        data_url = f"data:{mime};base64,{base64.b64encode(content).decode()}"
+        payload = {"model": self.model, "document": {"type": field, field: data_url}}
+        resp = httpx.post(
+            "https://api.mistral.ai/v1/ocr",
+            json=payload,
+            headers={"Authorization": f"Bearer {self.api_key}"},
+            timeout=self.timeout,
+        )
+        resp.raise_for_status()
+        pages = resp.json().get("pages", [])
+        markdown = "\n\n".join(p.get("markdown", "") for p in pages)
+        return markdown, [SourceRef(page=i + 1) for i in range(len(pages))]
+
+
+class DoclingOCRProvider:
+    """On-prem, no-egress OCR via Docling. Requires the ``docling`` extra; when
+    absent, ``available()`` is False and the caller falls back to text-only."""
+
+    name = "docling"
+
+    def available(self) -> bool:
+        try:
+            import docling  # noqa: F401
+
+            return True
+        except ImportError:
+            return False
+
+    def ocr(self, content: bytes, mime: str) -> tuple[str, list[SourceRef]]:
+        import os as _os
+        import tempfile
+
+        from docling.document_converter import DocumentConverter
+
+        is_pdf = mime == "application/pdf" or content[:5] == b"%PDF-"
+        suffix = ".pdf" if is_pdf else ".png"
+        with tempfile.NamedTemporaryFile(suffix=suffix, delete=False) as tf:
+            tf.write(content)
+            path = tf.name
+        try:
+            document = DocumentConverter().convert(path).document
+            markdown = document.export_to_markdown()
+        finally:
+            _os.unlink(path)
+        # Docling exposes page count; provenance stays page-level (best-effort).
+        pages = getattr(document, "num_pages", None) or 1
+        return markdown, [SourceRef(page=i + 1) for i in range(pages)]

rakam_systems_documents-0.1.0/src/rakam_systems_documents/schema.py

@@ -0,0 +1,41 @@
+"""The neutral currency the prepare core moves: ``PreparedContent``.
+
+Downstream-agnostic on purpose — it names *content* (markdown, structured rows,
+provenance), never business meaning. Extraction / matching / classification are
+the consumer's job, not this package's.
+"""
+from __future__ import annotations
+
+from pydantic import BaseModel, Field
+
+
+class SourceRef(BaseModel):
+    """Where a fragment came from. Best-effort per provider: exact for native
+    PDFs / spreadsheets, page-level for scan-OCR, empty when unknown."""
+
+    page: int | None = None      # PDF / scan (1-indexed)
+    sheet: str | None = None     # spreadsheet
+    row: int | None = None       # spreadsheet / table (1-indexed)
+
+
+class TableRow(BaseModel):
+    """One row of a tabular source, header-keyed when a header is detected."""
+
+    cells: dict[str, str]
+    source: SourceRef = Field(default_factory=SourceRef)
+
+
+class PreparedContent(BaseModel):
+    """The result of preparing one document.
+
+    ``markdown`` is what a model reads; ``rows`` carries structured tabular data
+    for precise downstream mapping; ``provenance`` traces prose/scan fragments to
+    their source. ``meta`` holds light, non-business signals only
+    (page_count / sheet_names / encoding / help)."""
+
+    markdown: str = ""
+    rows: list[TableRow] = Field(default_factory=list)
+    provenance: list[SourceRef] = Field(default_factory=list)
+    provider: str = "none"        # pdf_text | mistral_ocr | docling_ocr | tabular | text | none
+    truncated: bool = False
+    meta: dict = Field(default_factory=dict)

rakam_systems_documents-0.1.0/src/rakam_systems_documents/tests/test_prepare.py

@@ -0,0 +1,146 @@
+"""Behavioural tests for prepare() — one per dispatch/route, using in-test
+sample bytes (native PDF, scanned PDF, xlsx, csv, email, binary).
+"""
+from __future__ import annotations
+
+import io
+from email.message import EmailMessage
+
+import fitz  # pymupdf
+import openpyxl
+import pytest
+
+from rakam_systems_documents import PreparedContent, SourceRef, prepare
+
+PDF = "application/pdf"
+XLSX = "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet"
+
+
+# ── sample builders ─────────────────────────────────────────────────────────
+def _native_pdf() -> bytes:
+    doc = fitz.open()
+    doc.new_page().insert_text((72, 72), "OFFRE FOURNISSEUR ACME\nTube acier E24 50x50\n1200 ml", fontsize=11)
+    return doc.tobytes()
+
+
+def _scanned_pdf() -> bytes:
+    src = fitz.open()
+    src.new_page().insert_text((72, 72), "FACTURE scan\nTotal 5040 EUR", fontsize=14)
+    pix = src[0].get_pixmap(dpi=120)
+    out = fitz.open()
+    page = out.new_page(width=src[0].rect.width, height=src[0].rect.height)
+    page.insert_image(page.rect, pixmap=pix)  # image only → no text layer
+    return out.tobytes()
+
+
+def _catalogue_xlsx() -> bytes:
+    wb = openpyxl.Workbook()
+    ws = wb.active
+    ws.title = "Catalogue"
+    ws.append(["Code", "Designation", "Grade"])
+    ws.append(["ST-E24-5050", "Tube acier 50x50x3", "E24"])
+    ws.append(["ST-S235-6060", "Tube acier 60x60x4", "S235"])
+    buf = io.BytesIO()
+    wb.save(buf)
+    return buf.getvalue()
+
+
+def _email() -> bytes:
+    msg = EmailMessage()
+    msg["Subject"] = "Demande de prix"
+    msg["From"] = "achat@client.example"
+    msg.set_content("Merci de chiffrer 1200 ml de tube acier E24.")
+    return bytes(msg)
+
+
+class _MockOCR:
+    name = "mistral"
+
+    def available(self) -> bool:
+        return True
+
+    def ocr(self, content: bytes, mime: str):
+        return "# (OCR) FACTURE\n\nTotal 5040 EUR", [SourceRef(page=1)]
+
+
+# ── tests ────────────────────────────────────────────────────────────────────
+def test_native_pdf_uses_text_layer_not_ocr():
+    called = {"ocr": False}
+
+    class _Spy(_MockOCR):
+        def ocr(self, content, mime):
+            called["ocr"] = True
+            return "should-not-be-used", []
+
+    pc = prepare(_native_pdf(), PDF, "offer.pdf", ocr=_Spy())
+    assert pc.provider == "pdf_text"
+    assert called["ocr"] is False
+    assert "ACME" in pc.markdown
+    assert pc.provenance and pc.provenance[0].page == 1
+
+
+def test_scanned_pdf_routes_to_ocr():
+    pc = prepare(_scanned_pdf(), PDF, "invoice.pdf", ocr=_MockOCR())
+    assert pc.provider == "mistral_ocr"
+    assert "5040" in pc.markdown
+
+
+def test_scanned_pdf_without_ocr_degrades_gracefully():
+    pc = prepare(_scanned_pdf(), PDF, "invoice.pdf", ocr=None)
+    assert pc.provider == "none"
+    assert "help" in pc.meta
+    assert pc.markdown == ""
+
+
+def test_xlsx_emits_markdown_table_and_structured_rows_with_provenance():
+    pc = prepare(_catalogue_xlsx(), XLSX, "catalogue.xlsx")
+    assert pc.provider == "tabular"
+    assert "| Code | Designation | Grade |" in pc.markdown
+    assert len(pc.rows) == 2
+    first = pc.rows[0]
+    assert first.cells["Code"] == "ST-E24-5050"
+    assert first.cells["Grade"] == "E24"
+    assert first.source.sheet == "Catalogue"
+    assert first.source.row == 2
+
+
+def test_csv_emits_rows_with_row_provenance():
+    csv_bytes = b"produit,quantite\nTube E24,1200\nEPI,30\n"
+    pc = prepare(csv_bytes, "text/csv", "req.csv")
+    assert pc.provider == "tabular"
+    assert len(pc.rows) == 2
+    assert pc.rows[0].cells == {"produit": "Tube E24", "quantite": "1200"}
+    assert pc.rows[0].source.row == 2
+
+
+def test_email_decodes_body_and_keeps_subject():
+    pc = prepare(_email(), "message/rfc822", "req.eml")
+    assert pc.provider == "text"
+    assert pc.meta["subject"] == "Demande de prix"
+    assert "tube acier E24" in pc.markdown
+
+
+def test_binary_masquerading_as_text_is_rejected():
+    pc = prepare(b"\x89PNG\r\n\x00\x00garbage", "text/plain", "fake.txt")
+    assert pc.provider == "none"
+    assert "help" in pc.meta
+
+
+def test_plain_text_multi_encoding():
+    # latin-1-encodable accents (em-dash is not latin-1) — exercises the fallback decode
+    pc = prepare("café déjà vu à Genève".encode("latin-1"), "text/plain", "note.txt")
+    assert pc.provider == "text"
+    assert "café" in pc.markdown
+
+
+def test_markdown_truncated_to_max_chars():
+    big = ("# title\n" + "x" * 20000).encode()
+    pc = prepare(big, "text/markdown", "big.md", max_chars=100)
+    assert pc.truncated is True
+    assert len(pc.markdown) == 100
+
+
+def test_prepare_never_raises_returns_prepared_content():
+    pc = prepare(b"", "application/octet-stream", "empty.bin")
+    assert isinstance(pc, PreparedContent)
+    assert pc.provider == "none"