docmd-cli 0.1.1__tar.gz → 0.1.2__tar.gz

{docmd_cli-0.1.1 → docmd_cli-0.1.2}/ARCHITECTURE.md

@@ -3,6 +3,31 @@
 ## The pitch (keep this pinned above your desk)
 We are not competing on conversion quality. We are selling **"hit an endpoint, get clean Markdown back"** — no Python env, no 8GB+ RAM, no GPU, no dependency hell. The open-source core proves the engine works and builds trust. The hosted API sells convenience.
 
+## Real dependency gap found by testing (2026-09-18)
+The pitch above says "no dependency hell." That held for text-layer PDFs, DOCX, and
+PPTX when actually tested against real downloaded documents (an arXiv paper, an IMF
+report). It did **not** hold for OCR: Marker's current recognition model (used for
+both scanned-PDF OCR and equation recognition) is a VLM served through either `vllm`
+or llama.cpp's `llama-server` binary - no plain-transformers/CPU fallback exists in
+this version. Neither ships via pip, so `pip install docmd-cli[full]` alone cannot
+actually OCR a scanned document or handle a PDF with equations, despite the README
+previously implying "OCR - bundled by Marker, free" meant it worked out of the box.
+
+Confirmed via a real test: converting a genuinely scanned PDF (downloaded from
+archive.org) raised a missing-binary error on this dev machine (no Homebrew). Fixed
+two ways: (1) a real fix - download a prebuilt `llama-server` binary directly from
+llama.cpp's GitHub releases (no package manager needed) and set `LLAMA_CPP_BINARY`,
+which then produced correct real OCR output; (2) a docmd fix - a dedicated
+`MissingSystemDependencyError` with install instructions instead of a raw stack trace
+when the binary is missing (see `docmd/errors.py`).
+
+**Implications to carry into Stage 2/3 (`api/`)**: the hosted API's Docker image
+does not currently install `llama-server` either - meaning a customer uploading a
+scanned PDF or an equation-heavy document to the live API would hit this same error
+today. This needs a fix in `docmd-api`'s Dockerfile before OCR can be honestly
+advertised as working end-to-end there. Not yet fixed as of this note - flagged here
+so it isn't lost.
+
 ## Positioning risk (checked 2026-09-18)
 Marker's *code* is Apache-2.0 (no restriction). Marker's *model weights* use a modified
 Open RAIL-M license: free for research, personal use, and organizations under $5M in
@@ -128,7 +153,9 @@ Treat this folder as the actual differentiator. A thin wrapper is a weekend proj
 
 - `pip install docmd-cli` -> core PDF conversion (Marker bundles OCR here already; there is
   no separate lean/no-OCR base install, since Marker's own base dependencies include
-  `surya-ocr`)
+  `surya-ocr`) - but actually *running* OCR (or equation recognition) on top of that
+  also needs the `llama-server` native binary, which no `pip install` variant can
+  provide - see "Real dependency gap found by testing" above
 - `pip install docmd-cli[full]` -> adds DOCX/PPTX/EPUB/XLSX support via Marker's `full` extra
 
 ## License clarity

{docmd_cli-0.1.1 → docmd_cli-0.1.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: docmd-cli
-Version: 0.1.1
+Version: 0.1.2
 Summary: Convert PDFs, DOCX, and PPTX to clean, structure-preserving Markdown.
 Project-URL: Homepage, https://github.com/taherzribi/docmd
 Project-URL: Issues, https://github.com/taherzribi/docmd/issues
@@ -67,7 +67,7 @@ core, usable standalone today.
 | Input | Output |
 |---|---|
 | PDF (text-based) | Markdown with preserved headings, lists, tables |
-| PDF (scanned) | Markdown via OCR — bundled by Marker, free |
+| PDF (scanned) | Markdown via OCR — bundled by Marker, free, but needs [one extra native binary](#ocr-and-equations-need-one-native-binary) |
 | DOCX | Markdown with formatting preserved (`pip install docmd-cli[full]`) |
 | PPTX | Markdown, one section per slide (`pip install docmd-cli[full]`) |
 
@@ -116,6 +116,35 @@ sudo apt-get install libpango-1.0-0 libpangocairo-1.0-0 libgdk-pixbuf-2.0-0 libf
 
 PDF conversion (the base install) does not need this.
 
+## OCR and equations need one native binary
+
+Found by testing docmd against a real scanned document, not documented anywhere
+upstream at the time of writing: Marker's current OCR and equation-recognition model
+is a vision-language model served through either `vllm` (GPU/Linux-oriented) or
+[llama.cpp](https://github.com/ggml-org/llama.cpp)'s `llama-server` binary - there is
+no plain-CPU/transformers fallback. `pip install docmd-cli` cannot provide either one,
+since neither ships as a normal Python wheel.
+
+This only matters for **scanned PDFs** (no embedded text layer) and PDFs with
+**equations** - a normal text-layer PDF never touches this code path, and everything
+else in this README works with just `pip install`.
+
+```bash
+# macOS / Linux with Homebrew
+brew install llama.cpp
+
+# No Homebrew: download a prebuilt binary directly, no package manager needed
+# (pick the archive matching your OS/arch from the releases page)
+curl -LO https://github.com/ggml-org/llama.cpp/releases/latest/download/llama-<version>-bin-macos-arm64.tar.gz
+tar xzf llama-<version>-bin-macos-arm64.tar.gz
+export LLAMA_CPP_BINARY=$PWD/llama-<version>/llama-server
+```
+
+Without it, converting a scanned PDF or one with equations raises a
+`MissingSystemDependencyError` with these same instructions - not a raw stack trace.
+First real OCR run also downloads the model's GGUF weights from Hugging Face
+(a few GB), separate from the PyTorch weights Marker already downloaded.
+
 ## Image handling
 
 Images default to a text placeholder (`*[... omitted]*`) - no binary data, nothing to

{docmd_cli-0.1.1 → docmd_cli-0.1.2}/README.md

@@ -40,7 +40,7 @@ core, usable standalone today.
 | Input | Output |
 |---|---|
 | PDF (text-based) | Markdown with preserved headings, lists, tables |
-| PDF (scanned) | Markdown via OCR — bundled by Marker, free |
+| PDF (scanned) | Markdown via OCR — bundled by Marker, free, but needs [one extra native binary](#ocr-and-equations-need-one-native-binary) |
 | DOCX | Markdown with formatting preserved (`pip install docmd-cli[full]`) |
 | PPTX | Markdown, one section per slide (`pip install docmd-cli[full]`) |
 
@@ -89,6 +89,35 @@ sudo apt-get install libpango-1.0-0 libpangocairo-1.0-0 libgdk-pixbuf-2.0-0 libf
 
 PDF conversion (the base install) does not need this.
 
+## OCR and equations need one native binary
+
+Found by testing docmd against a real scanned document, not documented anywhere
+upstream at the time of writing: Marker's current OCR and equation-recognition model
+is a vision-language model served through either `vllm` (GPU/Linux-oriented) or
+[llama.cpp](https://github.com/ggml-org/llama.cpp)'s `llama-server` binary - there is
+no plain-CPU/transformers fallback. `pip install docmd-cli` cannot provide either one,
+since neither ships as a normal Python wheel.
+
+This only matters for **scanned PDFs** (no embedded text layer) and PDFs with
+**equations** - a normal text-layer PDF never touches this code path, and everything
+else in this README works with just `pip install`.
+
+```bash
+# macOS / Linux with Homebrew
+brew install llama.cpp
+
+# No Homebrew: download a prebuilt binary directly, no package manager needed
+# (pick the archive matching your OS/arch from the releases page)
+curl -LO https://github.com/ggml-org/llama.cpp/releases/latest/download/llama-<version>-bin-macos-arm64.tar.gz
+tar xzf llama-<version>-bin-macos-arm64.tar.gz
+export LLAMA_CPP_BINARY=$PWD/llama-<version>/llama-server
+```
+
+Without it, converting a scanned PDF or one with equations raises a
+`MissingSystemDependencyError` with these same instructions - not a raw stack trace.
+First real OCR run also downloads the model's GGUF weights from Hugging Face
+(a few GB), separate from the PyTorch weights Marker already downloaded.
+
 ## Image handling
 
 Images default to a text placeholder (`*[... omitted]*`) - no binary data, nothing to

{docmd_cli-0.1.1 → docmd_cli-0.1.2}/docmd/__init__.py

@@ -18,7 +18,7 @@ from docmd.postprocess.image_handling import apply_image_handling
 from docmd.postprocess.table_cleanup import clean_tables
 
 __all__ = ["convert", "convert_document", "ConvertConfig", "ConversionResult"]
-__version__ = "0.1.1"
+__version__ = "0.1.2"
 
 
 def convert_document(

{docmd_cli-0.1.1 → docmd_cli-0.1.2}/docmd/converters/marker_converter.py

@@ -13,7 +13,12 @@ from typing import Any
 
 from docmd.config import ConvertConfig
 from docmd.converters.base import ConversionResult
-from docmd.errors import ConversionError, EncryptedDocumentError, MissingExtraError
+from docmd.errors import (
+    ConversionError,
+    EncryptedDocumentError,
+    MissingExtraError,
+    MissingSystemDependencyError,
+)
 
 # Marker loads its models (a few hundred MB to ~1GB of weights, downloaded
 # from Hugging Face on first use) lazily and caches them at module scope, so
@@ -21,6 +26,11 @@ from docmd.errors import ConversionError, EncryptedDocumentError, MissingExtraEr
 _model_dict: dict[str, Any] | None = None
 
 _PASSWORD_HINTS = ("password", "encrypt")
+# Raised as surya.inference.backends.spawn.SpawnError when OCR/equation
+# recognition needs the llama-server binary and it isn't on PATH - matched
+# by message/type name rather than importing surya's internal exception
+# class, so this doesn't break if that module path moves.
+_MISSING_BINARY_HINTS = ("llama-server", "spawnerror")
 
 
 def _get_model_dict() -> dict[str, Any]:
@@ -39,6 +49,18 @@ def _build_config_dict(config: ConvertConfig):
         "output_format": "markdown",
         "force_ocr": config.force_ocr,
         "use_llm": config.use_llm,
+        # Marker's own default (4) spins up a multiprocessing.ProcessPoolExecutor
+        # for page-text extraction on any sufficiently multi-page PDF. On
+        # spawn-based platforms (macOS, Windows) that crashes with
+        # "An attempt has been made to start a new process before the
+        # current process has finished its bootstrapping phase" whenever the
+        # caller isn't wrapped in `if __name__ == "__main__":` - an easy trap
+        # for a library used from a plain script, a notebook, or a web
+        # server's request handler. Marker's own bundled server.py sets this
+        # to 1 for exactly this reason; docmd does the same as a library
+        # default, trading a bit of extraction parallelism for not crashing
+        # on arbitrary callers.
+        "pdftext_workers": 1,
     }
     if config.image_mode == "skip":
         options["disable_image_extraction"] = True
@@ -71,8 +93,11 @@ class MarkerConverter:
             rendered = converter(filepath)
         except Exception as exc:
             message = str(exc).lower()
+            exc_type_name = type(exc).__name__.lower()
             if any(hint in message for hint in _PASSWORD_HINTS):
                 raise EncryptedDocumentError() from exc
+            if any(hint in message or hint in exc_type_name for hint in _MISSING_BINARY_HINTS):
+                raise MissingSystemDependencyError(str(exc)) from exc
             raise ConversionError(
                 f"Marker failed to convert '{filepath}': {exc}", cause=exc
             ) from exc

{docmd_cli-0.1.1 → docmd_cli-0.1.2}/docmd/errors.py

@@ -35,6 +35,30 @@ class MissingExtraError(DocmdError):
         )
 
 
+class MissingSystemDependencyError(DocmdError):
+    """Raised when Marker's OCR/equation recognition needs a native binary
+    that isn't installed - not a pip package, so `pip install docmd[full]`
+    can't provide it. See ARCHITECTURE.md "OCR and equation recognition need
+    a native binary" for why this exists."""
+
+    def __init__(self, detail: str) -> None:
+        super().__init__(
+            "OCR/equation recognition needs the `llama-server` binary from "
+            "llama.cpp, which pip cannot install. This only matters for "
+            "scanned PDFs or documents with equations - a normal text-layer "
+            "PDF doesn't hit this path.\n\n"
+            "Install it with:\n"
+            "  macOS:  brew install llama.cpp\n"
+            "  Linux:  brew install llama.cpp, or download a release from\n"
+            "          https://github.com/ggml-org/llama.cpp/releases\n"
+            "No Homebrew? Download the right archive from that releases "
+            "page directly (llama-<version>-bin-macos-arm64.tar.gz etc.), "
+            "then either put `llama-server` on your PATH or set "
+            "LLAMA_CPP_BINARY to its full path.\n\n"
+            f"Underlying error: {detail}"
+        )
+
+
 class EncryptedDocumentError(DocmdError):
     """Raised when the input is a password-protected / encrypted document."""
 

{docmd_cli-0.1.1 → docmd_cli-0.1.2}/docmd/postprocess/image_handling.py

@@ -18,7 +18,7 @@ import re
 from pathlib import Path
 from typing import Any
 
-_IMAGE_RE = re.compile(r"!\[([^\]]*)\]\(([^)]+)\)")
+_IMAGE_RE = re.compile(r"!\[([^\]]*)\]\(([^)]*)\)")
 
 
 def apply_image_handling(
@@ -32,6 +32,16 @@ def apply_image_handling(
     def _replace(match: re.Match[str]) -> str:
         nonlocal counter
         alt_text, ref = match.group(1), match.group(2)
+
+        if not ref.strip():
+            # `![]()` with no reference at all - real Marker output on some
+            # scanned pages (an OCR'd region with no recoverable image data).
+            # No mode should keep this: a placeholder/skip has nothing to
+            # describe, and alt-text mode would otherwise emit a link to
+            # nowhere (an empty href renders as a broken image for no
+            # legitimate reason, since there was never a file to begin with).
+            return ""
+
         filename = Path(ref).name
 
         if mode == "skip":

{docmd_cli-0.1.1 → docmd_cli-0.1.2}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "docmd-cli"
-version = "0.1.1"
+version = "0.1.2"
 description = "Convert PDFs, DOCX, and PPTX to clean, structure-preserving Markdown."
 readme = "README.md"
 license = { text = "MIT" }

docmd_cli-0.1.2/tests/test_marker_converter_errors.py

@@ -0,0 +1,49 @@
+"""Unit tests for MarkerConverter's exception classification - mocked so
+these run fast and don't depend on whether llama-server happens to be
+installed on the machine running the tests. See
+docmd/converters/marker_converter.py for the real (slow, environment-
+dependent) version of this failure, found via real OCR testing on a
+genuinely scanned PDF from archive.org.
+"""
+
+from __future__ import annotations
+
+import docmd.converters.marker_converter as marker_converter_module
+from docmd.config import ConvertConfig
+from docmd.converters.marker_converter import MarkerConverter
+from docmd.errors import MissingSystemDependencyError
+
+
+class _FakeSpawnError(Exception):
+    """Stands in for surya.inference.backends.spawn.SpawnError without
+    importing surya's internal exception class directly."""
+
+
+class _FakePdfConverterMissingBinary:
+    def __init__(self, **_kwargs) -> None:
+        pass
+
+    def __call__(self, _filepath: str):
+        raise _FakeSpawnError(
+            "llama-server binary not found. Install with:\n"
+            "  macOS:  brew install llama.cpp"
+        )
+
+
+def test_missing_llama_server_binary_raises_clear_error(monkeypatch, tmp_path):
+    monkeypatch.setattr(marker_converter_module, "_get_model_dict", lambda: {})
+    monkeypatch.setattr(
+        "marker.converters.pdf.PdfConverter", _FakePdfConverterMissingBinary
+    )
+
+    dummy_pdf = tmp_path / "dummy.pdf"
+    dummy_pdf.write_bytes(b"%PDF-1.4\n%%EOF")
+
+    try:
+        MarkerConverter().convert(str(dummy_pdf), ConvertConfig())
+        raise AssertionError("expected MissingSystemDependencyError")
+    except MissingSystemDependencyError as exc:
+        message = str(exc)
+        assert "llama.cpp" in message
+        assert "brew install llama.cpp" in message
+        assert "LLAMA_CPP_BINARY" in message

{docmd_cli-0.1.1 → docmd_cli-0.1.2}/tests/test_postprocess.py

@@ -9,6 +9,30 @@ def test_heading_normalize_fixes_level_skip():
     assert out == "# Title\n## Subsection\n"
 
 
+def test_heading_normalize_fixes_level_skip_then_leaves_recovered_depth_alone():
+    """The exact pattern found in a real 27-page IMF report while stress-
+    testing against downloaded (not synthetic) documents: an H1 jumps
+    straight to H4 (skipping H2/H3), gets clamped to H2 - but a *later*
+    heading that's also originally H4 is left alone, because an H3 heading
+    in between legitimately re-establishes the depth. Never reproduced
+    synthetically (see skip_headings.pdf in git history, removed after
+    Marker's own layout model wouldn't cooperate) - this encodes the real
+    structure without redistributing the copyrighted source PDF."""
+    md = (
+        "# Summary of the Economy Classification\n"
+        "#### General Features and Composition of Groups\n"
+        "### Advanced Economies\n"
+        "#### Emerging Market and Developing Economies\n"
+    )
+    out = normalize_headings(md)
+    assert out == (
+        "# Summary of the Economy Classification\n"
+        "## General Features and Composition of Groups\n"
+        "### Advanced Economies\n"
+        "#### Emerging Market and Developing Economies\n"
+    )
+
+
 def test_heading_normalize_drops_empty_heading():
     md = "# Title\n##\nBody text.\n"
     out = normalize_headings(md)
@@ -98,3 +122,16 @@ def test_image_handling_alt_text_mode_fills_empty_alt():
     md = "![](img.png)"
     out = apply_image_handling(md, images={}, mode="alt-text")
     assert out == "![Image 1](img.png)"
+
+
+def test_image_handling_drops_fully_empty_image_reference():
+    """Real Marker output on a scanned page: `![]()` with no href at all,
+    directly adjacent to a real image reference with no separator between
+    them - found while OCR-testing a genuinely scanned PDF. The old regex
+    required at least one character inside the parens, so `![]()` never
+    matched and leaked through every mode unprocessed."""
+    md = "![]()![](real.jpeg)"
+
+    assert apply_image_handling(md, images={}, mode="placeholder") == "*[image omitted]*"
+    assert apply_image_handling(md, images={}, mode="skip") == ""
+    assert apply_image_handling(md, images={}, mode="alt-text") == "![Image 1](real.jpeg)"

{docmd_cli-0.1.1 → docmd_cli-0.1.2}/tests/test_postprocess_more_integration.py

@@ -16,18 +16,32 @@ FIXTURES = Path(__file__).parent / "fixtures"
 
 def test_inconsistent_heading_levels_get_normalized_to_match():
     """running_header.pdf has 4 structurally identical section headings
-    (same paragraph style). Raw Marker assigns them *different* levels -
-    Section 1 comes out H3, Sections 2-4 come out H2, a genuine model
-    inconsistency, not a deliberately constructed skip. Confirm that first,
-    then confirm docmd normalizes them to be consistent."""
+    (same paragraph style). On the machine this was first written on (macOS,
+    MPS), raw Marker assigned them *different* levels - Section 1 came back
+    H3, Sections 2-4 came back H2 - a genuine model inconsistency, not a
+    deliberately constructed skip.
+
+    That specific raw output isn't portable, though: CI (Linux, CPU) ran the
+    exact same PDF through the exact same pinned Marker version and got all
+    four sections back as H2 from the start - a real platform-dependent
+    difference in the underlying ML model's inference, not something our
+    code controls. Hard-pinning the raw "bug" broke CI twice for that
+    reason. So this only *asserts* the thing docmd actually guarantees -
+    consistent heading levels across all four sections, however Marker
+    happened to render them - and reports (without failing) whether this
+    particular run's raw output shows the original inconsistency."""
     raw = MarkerConverter().convert(str(FIXTURES / "running_header.pdf"), ConvertConfig())
-    assert "### **Section 1: Findings**" in raw.markdown
-    assert "## **Section 2: Findings**" in raw.markdown
+    if "### **Section 1: Findings**" in raw.markdown:
+        print("raw Marker output reproduced the H3-vs-H2 inconsistency on this run")
+    else:
+        print("raw Marker output did not reproduce the inconsistency on this run/platform")
 
     result = convert_document(str(FIXTURES / "running_header.pdf"))
-    for n in range(1, 5):
-        assert f"## **Section {n}: Findings**" in result.markdown
-        assert f"### **Section {n}: Findings**" not in result.markdown
+    levels = {
+        n: "###" if f"### **Section {n}: Findings**" in result.markdown else "##"
+        for n in range(1, 5)
+    }
+    assert len(set(levels.values())) == 1, f"inconsistent levels in docmd output: {levels}"
 
 
 def test_running_header_does_not_leak_into_output():