docmd-cli 0.1.1__tar.gz → 0.1.3__tar.gz

{docmd_cli-0.1.1 → docmd_cli-0.1.3}/ARCHITECTURE.md

@@ -1,8 +1,39 @@
 # Architecture: Doc-to-Markdown API
 
+See [CONTRACT.md](CONTRACT.md) for what docmd actually guarantees about its output,
+independent of Marker or any future backend - the formalized version of a testing
+philosophy this project arrived at the hard way, via two real CI failures caused by
+tests that pinned Marker's exact raw output instead of docmd's own guarantees
+(commits `066fd1b` and part of `c7225c5`).
+
 ## The pitch (keep this pinned above your desk)
 We are not competing on conversion quality. We are selling **"hit an endpoint, get clean Markdown back"** — no Python env, no 8GB+ RAM, no GPU, no dependency hell. The open-source core proves the engine works and builds trust. The hosted API sells convenience.
 
+## Real dependency gap found by testing (2026-09-18)
+The pitch above says "no dependency hell." That held for text-layer PDFs, DOCX, and
+PPTX when actually tested against real downloaded documents (an arXiv paper, an IMF
+report). It did **not** hold for OCR: Marker's current recognition model (used for
+both scanned-PDF OCR and equation recognition) is a VLM served through either `vllm`
+or llama.cpp's `llama-server` binary - no plain-transformers/CPU fallback exists in
+this version. Neither ships via pip, so `pip install docmd-cli[full]` alone cannot
+actually OCR a scanned document or handle a PDF with equations, despite the README
+previously implying "OCR - bundled by Marker, free" meant it worked out of the box.
+
+Confirmed via a real test: converting a genuinely scanned PDF (downloaded from
+archive.org) raised a missing-binary error on this dev machine (no Homebrew). Fixed
+two ways: (1) a real fix - download a prebuilt `llama-server` binary directly from
+llama.cpp's GitHub releases (no package manager needed) and set `LLAMA_CPP_BINARY`,
+which then produced correct real OCR output; (2) a docmd fix - a dedicated
+`MissingSystemDependencyError` with install instructions instead of a raw stack trace
+when the binary is missing (see `docmd/errors.py`).
+
+**Implications to carry into Stage 2/3 (`api/`)**: the hosted API's Docker image
+does not currently install `llama-server` either - meaning a customer uploading a
+scanned PDF or an equation-heavy document to the live API would hit this same error
+today. This needs a fix in `docmd-api`'s Dockerfile before OCR can be honestly
+advertised as working end-to-end there. Not yet fixed as of this note - flagged here
+so it isn't lost.
+
 ## Positioning risk (checked 2026-09-18)
 Marker's *code* is Apache-2.0 (no restriction). Marker's *model weights* use a modified
 Open RAIL-M license: free for research, personal use, and organizations under $5M in
@@ -128,16 +159,22 @@ Treat this folder as the actual differentiator. A thin wrapper is a weekend proj
 
 - `pip install docmd-cli` -> core PDF conversion (Marker bundles OCR here already; there is
   no separate lean/no-OCR base install, since Marker's own base dependencies include
-  `surya-ocr`)
+  `surya-ocr`) - but actually *running* OCR (or equation recognition) on top of that
+  also needs the `llama-server` native binary, which no `pip install` variant can
+  provide - see "Real dependency gap found by testing" above
 - `pip install docmd-cli[full]` -> adds DOCX/PPTX/EPUB/XLSX support via Marker's `full` extra
 
 ## License clarity
 
-MIT for the `docmd` wrapper is fine and expected for an OSS trust play. Marker's code is
-Apache-2.0. Marker's *model weights* are under a modified Open RAIL-M license: free for
-research, personal use, and organizations under $5M funding/revenue; beyond that, a paid
-license from Datalab is required (see datalab.to/pricing). Re-check this before scaling
-the hosted API past that threshold.
+Full detail moved to [docs/licensing.md](docs/licensing.md) so it's maintained in one
+place instead of drifting between README and this file. Short version: MIT for the
+`docmd` wrapper is fine and expected; Marker's code (Apache-2.0) and model weights
+(modified Open RAIL-M, $5M funding/revenue threshold) are licensed separately, and the
+weights license is the one that actually constrains a commercial hosted API at scale.
+Re-check `docs/licensing.md` before scaling `api/` past that threshold, and get written
+clarification from Datalab on hosted-API competitiveness before a commercial launch -
+see "Positioning risk" above for why Datalab's own competing hosted API makes this
+worth resolving explicitly, not assuming.
 
 ## What "done" looks like for Stage 1 (this repo, today)
 

docmd_cli-0.1.3/CONTRACT.md

@@ -0,0 +1,110 @@
+# The docmd contract
+
+This is what docmd guarantees about its output, independent of which extraction
+backend is running underneath (currently [Marker](https://github.com/datalab-to/marker),
+pinned to `marker-pdf==2.0.0`) or which platform it runs on. Everything here is
+enforced by a test that fails if the guarantee breaks - the file is named next to
+each one.
+
+The reason this document exists: a test suite that asserts *the backend's exact raw
+output* is testing the backend, not docmd, and breaks the moment the backend changes
+- confirmed the hard way twice in this project's history (`066fd1b`, `c7225c5`), where
+a test pinned an exact heading level Marker happened to produce on one platform, and
+CI failed on a different platform running the identical, pinned backend version. Both
+fixes replaced "assert the backend did X" with "assert docmd guarantees Y" - that
+distinction is the entire point of this document.
+
+## What's guaranteed
+
+**Heading hierarchy** (`docmd/postprocess/heading_normalize.py`,
+`tests/test_postprocess.py`, `tests/test_postprocess_more_integration.py`)
+- No heading level skips more than one deeper than the deepest level seen so far.
+  `H1` followed directly by `H4` becomes `H1` followed by `H2`; a later, legitimately
+  deep heading elsewhere in the same document is left alone once the hierarchy has
+  caught up. Confirmed against six independent real documents (an IMF report, four
+  arXiv papers/patents, one hand-built fixture) - this is the single most common
+  real-world defect found in this project's testing.
+- No orphaned `#` with empty text.
+- No heading whose text exactly repeats the immediately preceding heading (a Marker
+  artifact from running page headers).
+- A document's first heading is always promoted to H1 if nothing shallower precedes it.
+
+**Table structure** (`docmd/postprocess/table_cleanup.py`)
+- Every row in a rendered table has the same column count as its header, padded or
+  truncated as needed.
+- A missing or malformed separator row is synthesized.
+- A stray row that is itself separator-shaped (all dashes) is dropped rather than
+  rendered as literal data - found via a real two-column paper where a numbered
+  equation came back as a one-row table with two separator rows.
+- A table split into two blocks by a page break, with a repeated identical header, is
+  merged into one continuous table.
+- **Not guaranteed**: correct table content when the backend's own reading order is
+  wrong. docmd normalizes structure; it doesn't re-derive reading order the backend
+  got wrong (confirmed real gap: a patent's front-page bibliographic table).
+
+**Image references** (`docmd/postprocess/image_handling.py`)
+- No malformed Markdown image syntax (`![]()`  with an empty or partial reference)
+  ever appears in output, in any mode.
+- Three modes, each with a specific, tested behavior: `placeholder` (default, no
+  binary data referenced at all), `alt-text` (real links, non-empty alt text, and the
+  image file is actually written to `output_dir` if given), `skip` (removed entirely).
+
+**Errors** (`docmd/errors.py`)
+- A small, fixed set of `DocmdError` subclasses for known failure categories:
+  `UnsupportedFormatError`, `MissingExtraError`, `MissingSystemDependencyError`,
+  `EncryptedDocumentError`, `ConversionError`. Each carries an actionable message,
+  not a raw backend stack trace, for the failure modes docmd knows about.
+
+**Backend variation**
+- Platform or backend-version differences in raw extraction output never change
+  docmd's own guarantees above. They may change the *content* extracted (a backend
+  bug is still a backend bug), but never the *structural* invariants this document
+  lists.
+
+**Provenance** (`docmd/converters/base.py:ConversionResult.provenance`)
+- Every `convert_document()` call returns `docmd_version`, `backend`,
+  `backend_version`, `ocr_used`, and `conversion_duration_ms` - same shape regardless
+  of which backend ran. This records *what ran*, not that the *output* is
+  reproducible - see OCR determinism below, where it isn't.
+
+## What's explicitly not guaranteed (yet)
+
+Found by real-world testing, not fixed:
+
+- **Form/checkbox structure.** Adjacent checkbox option labels (e.g. a tax form's
+  filing-status options) are not separated or structured - they extract as one
+  run-on phrase, identical to raw backend output.
+- **RTL bidirectional punctuation.** Citation brackets and similar LTR punctuation
+  embedded in right-to-left script can render reversed (`]1[` instead of `[1]`).
+- **Multi-column reading order in dense bibliographic/legal layouts.** Proven to
+  work correctly on a two-column academic paper; proven to fail on a patent's
+  front-page citation block. No known rule yet for which case a given document falls
+  into.
+- **OCR reproducibility.** Running the identical file through the identical code path
+  twice, in the same process, produced different text - confirmed directly, not
+  inferred (28,916 vs 27,973 characters on a real degraded scan; individual word
+  choices and even a poem's line-break structure differed between the two runs).
+  Root cause investigated and is architectural, not a misconfiguration: OCR/equation
+  recognition already requests greedy decoding (`temperature=0.0`, confirmed in
+  `surya/inference/backends/openai_client.py`), but `llama-server` runs with
+  `--parallel 8` by default (`surya/inference/backends/llamacpp.py`), and
+  floating-point matrix multiplication under concurrent batched inference is not
+  strictly order-independent - a well-documented property of essentially every
+  production LLM-serving stack (vLLM, llama.cpp, TensorRT-LLM), not specific to this
+  one. A small numerical difference from batch composition can flip an argmax choice
+  at a near-tied token, and the autoregressive generation diverges from there.
+  `SURYA_INFERENCE_PARALLEL=1` (env var, not currently exposed through docmd's own
+  config) would remove the batching-composition variable and likely reduce this a
+  lot, at a real throughput cost - untested, and even then, multi-threaded CPU matrix
+  math without an explicit `--threads 1` could still leave some residual variance.
+  Does not affect the plain `pdftext` text-extraction path (no VLM inference
+  involved) - only pages that actually go through OCR or equation recognition.
+
+## Writing a new test against this contract
+
+Ask: does this assertion describe something docmd promises, or something the
+backend happened to do on one machine? If the latter, either don't assert it, or -
+if it's worth pinning as a canary for upstream behavior changing silently - say so
+explicitly in the test's docstring and keep it separate from the real assertion, the
+way `test_marker_really_does_split_a_page_spanning_table` and
+`test_running_header_does_not_leak_into_output` do.

{docmd_cli-0.1.1 → docmd_cli-0.1.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: docmd-cli
-Version: 0.1.1
+Version: 0.1.3
 Summary: Convert PDFs, DOCX, and PPTX to clean, structure-preserving Markdown.
 Project-URL: Homepage, https://github.com/taherzribi/docmd
 Project-URL: Issues, https://github.com/taherzribi/docmd/issues
@@ -67,7 +67,7 @@ core, usable standalone today.
 | Input | Output |
 |---|---|
 | PDF (text-based) | Markdown with preserved headings, lists, tables |
-| PDF (scanned) | Markdown via OCR — bundled by Marker, free |
+| PDF (scanned) | Markdown via OCR — bundled by Marker, free, but needs [one extra native binary](#ocr-and-equations-need-one-native-binary) |
 | DOCX | Markdown with formatting preserved (`pip install docmd-cli[full]`) |
 | PPTX | Markdown, one section per slide (`pip install docmd-cli[full]`) |
 
@@ -116,6 +116,42 @@ sudo apt-get install libpango-1.0-0 libpangocairo-1.0-0 libgdk-pixbuf-2.0-0 libf
 
 PDF conversion (the base install) does not need this.
 
+## OCR and equations need one native binary
+
+Found by testing docmd against a real scanned document, not documented anywhere
+upstream at the time of writing: Marker's current OCR and equation-recognition model
+is a vision-language model served through either `vllm` (GPU/Linux-oriented) or
+[llama.cpp](https://github.com/ggml-org/llama.cpp)'s `llama-server` binary - there is
+no plain-CPU/transformers fallback. `pip install docmd-cli` cannot provide either one,
+since neither ships as a normal Python wheel.
+
+This only matters for **scanned PDFs** (no embedded text layer) and PDFs with
+**equations** - a normal text-layer PDF never touches this code path, and everything
+else in this README works with just `pip install`.
+
+```bash
+# macOS / Linux with Homebrew
+brew install llama.cpp
+
+# No Homebrew: download a prebuilt binary directly, no package manager needed
+# (pick the archive matching your OS/arch from the releases page)
+curl -LO https://github.com/ggml-org/llama.cpp/releases/latest/download/llama-<version>-bin-macos-arm64.tar.gz
+tar xzf llama-<version>-bin-macos-arm64.tar.gz
+export LLAMA_CPP_BINARY=$PWD/llama-<version>/llama-server
+```
+
+Without it, converting a scanned PDF or one with equations raises a
+`MissingSystemDependencyError` with these same instructions - not a raw stack trace.
+First real OCR run also downloads the model's GGUF weights from Hugging Face
+(a few GB), separate from the PyTorch weights Marker already downloaded.
+
+**OCR output is not guaranteed reproducible run to run** - confirmed directly:
+the same scanned file through the same code path twice produced different text.
+This is architectural (concurrent batched inference under the hood), not a docmd
+bug and not fixable by a config flag alone - see [CONTRACT.md](CONTRACT.md) for the
+full investigation. Doesn't affect plain text-layer PDFs, only pages that actually
+go through OCR or equation recognition.
+
 ## Image handling
 
 Images default to a text placeholder (`*[... omitted]*`) - no binary data, nothing to
@@ -147,24 +183,41 @@ docmd convert report.pdf -o output/report.md --image-mode alt-text
 
 `image_mode="skip"` drops images entirely - no placeholder, no files.
 
+## Provenance
+
+Every conversion carries tracking info in `result.provenance` - useful for "this
+converted differently yesterday" debugging:
+
+```python
+from docmd import convert_document
+
+result = convert_document("report.pdf")
+print(result.provenance)
+# {'docmd_version': '0.1.3', 'backend': 'marker', 'backend_version': '2.0.0',
+#  'ocr_used': False, 'conversion_duration_ms': 489}
+```
+
+Same shape regardless of which backend actually ran - `convert()` (the plain
+string-returning function) doesn't expose this; use `convert_document()` for it.
+
 ## How it works
 
 `docmd` wraps [Marker](https://github.com/datalab-to/marker) with sane defaults and a
 clean output format, then runs its own post-processing pass
 (`docmd/postprocess/`) to fix table structure and normalize heading levels — see
 [ARCHITECTURE.md](ARCHITECTURE.md) for why this is the actual differentiation, not
-just a thin wrapper.
+just a thin wrapper, and [CONTRACT.md](CONTRACT.md) for exactly what's guaranteed
+about the output (and what isn't, yet) — independent of Marker or any future backend.
 
 ## License
 
 The `docmd` wrapper code is MIT — see [LICENSE](LICENSE).
 
-`docmd` depends on [Marker](https://github.com/datalab-to/marker), whose *code* is
-Apache-2.0 and whose *model weights* are licensed under a modified Open RAIL-M license:
-free for research, personal use, and organizations under $5M in funding or revenue.
-Commercial use beyond that threshold requires a license from
-[Datalab](https://www.datalab.to/pricing). This applies to you if you deploy `docmd`
-commercially at scale — check Marker's current license terms directly before doing so.
+`docmd` depends on [Marker](https://github.com/datalab-to/marker), whose code and
+model weights carry *separate* licenses — the weights license has a revenue/funding
+threshold that matters if you deploy commercially at scale. See
+[docs/licensing.md](docs/licensing.md) for the precise terms; don't rely on this
+sentence alone.
 
 ## Roadmap
 

{docmd_cli-0.1.1 → docmd_cli-0.1.3}/README.md

@@ -40,7 +40,7 @@ core, usable standalone today.
 | Input | Output |
 |---|---|
 | PDF (text-based) | Markdown with preserved headings, lists, tables |
-| PDF (scanned) | Markdown via OCR — bundled by Marker, free |
+| PDF (scanned) | Markdown via OCR — bundled by Marker, free, but needs [one extra native binary](#ocr-and-equations-need-one-native-binary) |
 | DOCX | Markdown with formatting preserved (`pip install docmd-cli[full]`) |
 | PPTX | Markdown, one section per slide (`pip install docmd-cli[full]`) |
 
@@ -89,6 +89,42 @@ sudo apt-get install libpango-1.0-0 libpangocairo-1.0-0 libgdk-pixbuf-2.0-0 libf
 
 PDF conversion (the base install) does not need this.
 
+## OCR and equations need one native binary
+
+Found by testing docmd against a real scanned document, not documented anywhere
+upstream at the time of writing: Marker's current OCR and equation-recognition model
+is a vision-language model served through either `vllm` (GPU/Linux-oriented) or
+[llama.cpp](https://github.com/ggml-org/llama.cpp)'s `llama-server` binary - there is
+no plain-CPU/transformers fallback. `pip install docmd-cli` cannot provide either one,
+since neither ships as a normal Python wheel.
+
+This only matters for **scanned PDFs** (no embedded text layer) and PDFs with
+**equations** - a normal text-layer PDF never touches this code path, and everything
+else in this README works with just `pip install`.
+
+```bash
+# macOS / Linux with Homebrew
+brew install llama.cpp
+
+# No Homebrew: download a prebuilt binary directly, no package manager needed
+# (pick the archive matching your OS/arch from the releases page)
+curl -LO https://github.com/ggml-org/llama.cpp/releases/latest/download/llama-<version>-bin-macos-arm64.tar.gz
+tar xzf llama-<version>-bin-macos-arm64.tar.gz
+export LLAMA_CPP_BINARY=$PWD/llama-<version>/llama-server
+```
+
+Without it, converting a scanned PDF or one with equations raises a
+`MissingSystemDependencyError` with these same instructions - not a raw stack trace.
+First real OCR run also downloads the model's GGUF weights from Hugging Face
+(a few GB), separate from the PyTorch weights Marker already downloaded.
+
+**OCR output is not guaranteed reproducible run to run** - confirmed directly:
+the same scanned file through the same code path twice produced different text.
+This is architectural (concurrent batched inference under the hood), not a docmd
+bug and not fixable by a config flag alone - see [CONTRACT.md](CONTRACT.md) for the
+full investigation. Doesn't affect plain text-layer PDFs, only pages that actually
+go through OCR or equation recognition.
+
 ## Image handling
 
 Images default to a text placeholder (`*[... omitted]*`) - no binary data, nothing to
@@ -120,24 +156,41 @@ docmd convert report.pdf -o output/report.md --image-mode alt-text
 
 `image_mode="skip"` drops images entirely - no placeholder, no files.
 
+## Provenance
+
+Every conversion carries tracking info in `result.provenance` - useful for "this
+converted differently yesterday" debugging:
+
+```python
+from docmd import convert_document
+
+result = convert_document("report.pdf")
+print(result.provenance)
+# {'docmd_version': '0.1.3', 'backend': 'marker', 'backend_version': '2.0.0',
+#  'ocr_used': False, 'conversion_duration_ms': 489}
+```
+
+Same shape regardless of which backend actually ran - `convert()` (the plain
+string-returning function) doesn't expose this; use `convert_document()` for it.
+
 ## How it works
 
 `docmd` wraps [Marker](https://github.com/datalab-to/marker) with sane defaults and a
 clean output format, then runs its own post-processing pass
 (`docmd/postprocess/`) to fix table structure and normalize heading levels — see
 [ARCHITECTURE.md](ARCHITECTURE.md) for why this is the actual differentiation, not
-just a thin wrapper.
+just a thin wrapper, and [CONTRACT.md](CONTRACT.md) for exactly what's guaranteed
+about the output (and what isn't, yet) — independent of Marker or any future backend.
 
 ## License
 
 The `docmd` wrapper code is MIT — see [LICENSE](LICENSE).
 
-`docmd` depends on [Marker](https://github.com/datalab-to/marker), whose *code* is
-Apache-2.0 and whose *model weights* are licensed under a modified Open RAIL-M license:
-free for research, personal use, and organizations under $5M in funding or revenue.
-Commercial use beyond that threshold requires a license from
-[Datalab](https://www.datalab.to/pricing). This applies to you if you deploy `docmd`
-commercially at scale — check Marker's current license terms directly before doing so.
+`docmd` depends on [Marker](https://github.com/datalab-to/marker), whose code and
+model weights carry *separate* licenses — the weights license has a revenue/funding
+threshold that matters if you deploy commercially at scale. See
+[docs/licensing.md](docs/licensing.md) for the precise terms; don't rely on this
+sentence alone.
 
 ## Roadmap
 

{docmd_cli-0.1.1 → docmd_cli-0.1.3}/docmd/__init__.py

@@ -18,7 +18,7 @@ from docmd.postprocess.image_handling import apply_image_handling
 from docmd.postprocess.table_cleanup import clean_tables
 
 __all__ = ["convert", "convert_document", "ConvertConfig", "ConversionResult"]
-__version__ = "0.1.1"
+__version__ = "0.1.3"
 
 
 def convert_document(
@@ -92,4 +92,5 @@ def _convert_path(
         page_count=result.page_count,
         images=result.images,
         metadata=result.metadata,
+        provenance=result.provenance,
     )

{docmd_cli-0.1.1 → docmd_cli-0.1.3}/docmd/converters/base.py

@@ -26,6 +26,15 @@ class ConversionResult:
     """Maps an image filename referenced in `markdown` (e.g.
     '_page_0_Figure_1.jpeg') to a PIL.Image.Image instance."""
     metadata: dict[str, Any] = field(default_factory=dict)
+    """Whatever the backend itself returned - opaque, backend-specific, not
+    a contract. For Marker this is its own metadata dict (page_stats,
+    table_of_contents, ...). A different backend would put different keys
+    here; don't build stable behavior on top of this dict's shape."""
+    provenance: dict[str, Any] = field(default_factory=dict)
+    """docmd's own tracking info, same shape regardless of which backend
+    ran: docmd_version, backend, backend_version, ocr_used,
+    conversion_duration_ms. For debugging "this converted differently
+    yesterday" - see ARCHITECTURE.md."""
 
 
 class Converter(Protocol):

docmd_cli-0.1.3/docmd/converters/marker_converter.py

@@ -0,0 +1,164 @@
+"""Wraps `marker-pdf` (https://github.com/datalab-to/marker) as a docmd Converter.
+
+Uses Marker's Python API directly (not its CLI/server) so we control input and
+output cleanly. `PdfConverter` is Marker's general-purpose converter despite
+the name: it dispatches to the right internal provider (PDF, DOCX, PPTX, ...)
+based on the file's actual content, via `provider_from_filepath`.
+"""
+
+from __future__ import annotations
+
+import importlib.metadata
+import time
+from pathlib import Path
+from typing import Any
+
+from docmd.config import ConvertConfig
+from docmd.converters.base import ConversionResult
+from docmd.errors import (
+    ConversionError,
+    EncryptedDocumentError,
+    MissingExtraError,
+    MissingSystemDependencyError,
+)
+
+# Marker loads its models (a few hundred MB to ~1GB of weights, downloaded
+# from Hugging Face on first use) lazily and caches them at module scope, so
+# repeated conversions within one process don't reload them.
+_model_dict: dict[str, Any] | None = None
+
+_PASSWORD_HINTS = ("password", "encrypt")
+# Raised as surya.inference.backends.spawn.SpawnError when OCR/equation
+# recognition needs the llama-server binary and it isn't on PATH - matched
+# by message/type name rather than importing surya's internal exception
+# class, so this doesn't break if that module path moves.
+_MISSING_BINARY_HINTS = ("llama-server", "spawnerror")
+
+
+def _get_model_dict() -> dict[str, Any]:
+    global _model_dict
+    if _model_dict is None:
+        from marker.models import create_model_dict
+
+        _model_dict = create_model_dict()
+    return _model_dict
+
+
+_docmd_version_cache: str | None = None
+_marker_version_cache: str | None = None
+
+
+def _docmd_version() -> str:
+    # Read from installed package metadata rather than `from docmd import
+    # __version__` - that would import docmd/__init__.py, which imports
+    # this module transitively (via the converter registry), a circular
+    # import.
+    global _docmd_version_cache
+    if _docmd_version_cache is None:
+        try:
+            _docmd_version_cache = importlib.metadata.version("docmd-cli")
+        except importlib.metadata.PackageNotFoundError:
+            _docmd_version_cache = "unknown"
+    return _docmd_version_cache
+
+
+def _marker_version() -> str:
+    global _marker_version_cache
+    if _marker_version_cache is None:
+        try:
+            _marker_version_cache = importlib.metadata.version("marker-pdf")
+        except importlib.metadata.PackageNotFoundError:
+            _marker_version_cache = "unknown"
+    return _marker_version_cache
+
+
+def _build_config_dict(config: ConvertConfig):
+    from marker.config.parser import ConfigParser
+
+    options: dict[str, Any] = {
+        "output_format": "markdown",
+        "force_ocr": config.force_ocr,
+        "use_llm": config.use_llm,
+        # Marker's own default (4) spins up a multiprocessing.ProcessPoolExecutor
+        # for page-text extraction on any sufficiently multi-page PDF. On
+        # spawn-based platforms (macOS, Windows) that crashes with
+        # "An attempt has been made to start a new process before the
+        # current process has finished its bootstrapping phase" whenever the
+        # caller isn't wrapped in `if __name__ == "__main__":` - an easy trap
+        # for a library used from a plain script, a notebook, or a web
+        # server's request handler. Marker's own bundled server.py sets this
+        # to 1 for exactly this reason; docmd does the same as a library
+        # default, trading a bit of extraction parallelism for not crashing
+        # on arbitrary callers.
+        "pdftext_workers": 1,
+    }
+    if config.image_mode == "skip":
+        options["disable_image_extraction"] = True
+
+    return ConfigParser(options)
+
+
+class MarkerConverter:
+    """Converter implementation backed by Marker."""
+
+    def convert(self, filepath: str | Path, config: ConvertConfig) -> ConversionResult:
+        filepath = str(filepath)
+        try:
+            from marker.converters.pdf import PdfConverter
+            from marker.output import text_from_rendered
+        except ImportError as exc:
+            raise MissingExtraError(Path(filepath).suffix) from exc
+
+        config_parser = _build_config_dict(config)
+        config_dict = config_parser.generate_config_dict()
+
+        start = time.monotonic()
+        try:
+            converter = PdfConverter(
+                config=config_dict,
+                artifact_dict=_get_model_dict(),
+                processor_list=config_parser.get_processors(),
+                renderer=config_parser.get_renderer(),
+                llm_service=config_parser.get_llm_service(),
+            )
+            rendered = converter(filepath)
+        except Exception as exc:
+            message = str(exc).lower()
+            exc_type_name = type(exc).__name__.lower()
+            if any(hint in message for hint in _PASSWORD_HINTS):
+                raise EncryptedDocumentError() from exc
+            if any(hint in message or hint in exc_type_name for hint in _MISSING_BINARY_HINTS):
+                raise MissingSystemDependencyError(str(exc)) from exc
+            raise ConversionError(
+                f"Marker failed to convert '{filepath}': {exc}", cause=exc
+            ) from exc
+
+        duration_ms = int((time.monotonic() - start) * 1000)
+        markdown, _, images = text_from_rendered(rendered)
+        metadata = dict(getattr(rendered, "metadata", {}) or {})
+        page_stats = metadata.get("page_stats", [])
+        page_count = len(page_stats) or 1
+
+        # A page whose text_extraction_method isn't "pdftext" went through
+        # OCR/vision-based recognition rather than reading an embedded text
+        # layer - independent of force_ocr, since Marker also falls back to
+        # this per-page for a PDF with no usable text layer at all.
+        ocr_used = config.force_ocr or any(
+            page.get("text_extraction_method") != "pdftext" for page in page_stats
+        )
+
+        provenance = {
+            "docmd_version": _docmd_version(),
+            "backend": "marker",
+            "backend_version": _marker_version(),
+            "ocr_used": ocr_used,
+            "conversion_duration_ms": duration_ms,
+        }
+
+        return ConversionResult(
+            markdown=markdown,
+            page_count=page_count,
+            images=images,
+            metadata=metadata,
+            provenance=provenance,
+        )

{docmd_cli-0.1.1 → docmd_cli-0.1.3}/docmd/errors.py

@@ -35,6 +35,30 @@ class MissingExtraError(DocmdError):
         )
 
 
+class MissingSystemDependencyError(DocmdError):
+    """Raised when Marker's OCR/equation recognition needs a native binary
+    that isn't installed - not a pip package, so `pip install docmd[full]`
+    can't provide it. See ARCHITECTURE.md "OCR and equation recognition need
+    a native binary" for why this exists."""
+
+    def __init__(self, detail: str) -> None:
+        super().__init__(
+            "OCR/equation recognition needs the `llama-server` binary from "
+            "llama.cpp, which pip cannot install. This only matters for "
+            "scanned PDFs or documents with equations - a normal text-layer "
+            "PDF doesn't hit this path.\n\n"
+            "Install it with:\n"
+            "  macOS:  brew install llama.cpp\n"
+            "  Linux:  brew install llama.cpp, or download a release from\n"
+            "          https://github.com/ggml-org/llama.cpp/releases\n"
+            "No Homebrew? Download the right archive from that releases "
+            "page directly (llama-<version>-bin-macos-arm64.tar.gz etc.), "
+            "then either put `llama-server` on your PATH or set "
+            "LLAMA_CPP_BINARY to its full path.\n\n"
+            f"Underlying error: {detail}"
+        )
+
+
 class EncryptedDocumentError(DocmdError):
     """Raised when the input is a password-protected / encrypted document."""
 

{docmd_cli-0.1.1 → docmd_cli-0.1.3}/docmd/postprocess/image_handling.py

@@ -18,7 +18,7 @@ import re
 from pathlib import Path
 from typing import Any
 
-_IMAGE_RE = re.compile(r"!\[([^\]]*)\]\(([^)]+)\)")
+_IMAGE_RE = re.compile(r"!\[([^\]]*)\]\(([^)]*)\)")
 
 
 def apply_image_handling(
@@ -32,6 +32,16 @@ def apply_image_handling(
     def _replace(match: re.Match[str]) -> str:
         nonlocal counter
         alt_text, ref = match.group(1), match.group(2)
+
+        if not ref.strip():
+            # `![]()` with no reference at all - real Marker output on some
+            # scanned pages (an OCR'd region with no recoverable image data).
+            # No mode should keep this: a placeholder/skip has nothing to
+            # describe, and alt-text mode would otherwise emit a link to
+            # nowhere (an empty href renders as a broken image for no
+            # legitimate reason, since there was never a file to begin with).
+            return ""
+
         filename = Path(ref).name
 
         if mode == "skip":

{docmd_cli-0.1.1 → docmd_cli-0.1.3}/docmd/postprocess/table_cleanup.py

@@ -98,7 +98,14 @@ def clean_tables(markdown: str) -> str:
         block_end = j
         while block_end < len(lines) and _is_row(lines[block_end]):
             block_end += 1
-        data_rows = [_split_cells(l) for l in lines[j:block_end]]
+        # A stray extra separator row (found via a real two-column paper: a
+        # single-row "table" - actually a numbered equation - with two
+        # separator rows back to back and no real data) would otherwise get
+        # treated as a literal data row of dashes. Drop any row that's
+        # itself separator-shaped rather than rendering it as content.
+        data_rows = [
+            _split_cells(l) for l in lines[j:block_end] if not _is_separator_row(l)
+        ]
 
         # Look ahead past blank lines / Marker's page separator for a
         # continuation: another table block whose header repeats this one.
@@ -120,7 +127,11 @@ def clean_tables(markdown: str) -> str:
             next_block_end = p
             while next_block_end < len(lines) and _is_row(lines[next_block_end]):
                 next_block_end += 1
-            data_rows.extend(_split_cells(l) for l in lines[p:next_block_end])
+            data_rows.extend(
+                _split_cells(l)
+                for l in lines[p:next_block_end]
+                if not _is_separator_row(l)
+            )
             k = next_block_end
 
         out.extend(_normalize_block(header, data_rows))

docmd_cli-0.1.3/docs/licensing.md

@@ -0,0 +1,50 @@
+# Licensing
+
+Three separate things, three separate licenses. Conflating them is the exact
+overclaim this document exists to prevent.
+
+## docmd itself
+
+MIT - see [LICENSE](../LICENSE). No restrictions on commercial use, modification, or
+redistribution of docmd's own wrapper/post-processing code.
+
+## Marker's source code
+
+docmd depends on [Marker](https://github.com/datalab-to/marker), pinned to
+`marker-pdf==2.0.0`. Marker's code is Apache-2.0. No restriction relevant here.
+
+## Marker's model weights
+
+This is the one that actually constrains you, and it's separate from the code
+license above - a common point of confusion, and the reason this file exists rather
+than a single blanket "Marker is Apache-2.0" statement.
+
+Marker's model weights (the actual OCR/layout/recognition models, downloaded
+separately from the code) are licensed under a **modified Open RAIL-M license**:
+
+- Free for research, personal use, and organizations under **$5M in funding or
+  revenue**.
+- Beyond that threshold, a commercial license from [Datalab](https://www.datalab.to/pricing)
+  (Marker's maintainer) is required.
+
+**This applies to you if** you deploy docmd commercially at a scale that crosses that
+threshold - not at small/bootstrapped scale, but worth planning around before it
+matters. Check Marker's current license terms directly before relying on this
+summary; license terms can change between versions, and the terms above reflect
+`marker-pdf==2.0.0` specifically, checked 2026-09-18.
+
+**If you're building a hosted/SaaS product on top of docmd**: Datalab runs its own
+hosted document-conversion API. Before launching a commercial hosted API built on
+Marker, get written clarification from Datalab on whether your product would be
+considered competitive with theirs under their weights license - don't assume the
+$5M threshold is the only relevant term. This is a business/legal step, not
+something resolvable by reading the license text alone.
+
+## Other dependencies
+
+docmd's `[full]` extra and the OCR/equation path pull in several other open-source
+projects (`weasyprint`, `python-docx`, `python-pptx`, `llama.cpp`, and Marker's own
+transitive dependencies). None of these are known to impose restrictions beyond their
+standard OSS licenses (BSD/MIT/Apache-family) as of this writing, but this file
+doesn't audit them individually - check `pyproject.toml` for the current pinned
+versions if you need to.

{docmd_cli-0.1.1 → docmd_cli-0.1.3}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "docmd-cli"
-version = "0.1.1"
+version = "0.1.3"
 description = "Convert PDFs, DOCX, and PPTX to clean, structure-preserving Markdown."
 readme = "README.md"
 license = { text = "MIT" }

{docmd_cli-0.1.1 → docmd_cli-0.1.3}/tests/test_converters.py

@@ -32,6 +32,21 @@ def test_convert_pdf_produces_structured_markdown():
     assert "|" in result.markdown  # a table survived
 
 
+def test_convert_document_includes_provenance():
+    """docmd's own tracking info (docmd_version, backend, backend_version,
+    ocr_used, conversion_duration_ms), same shape regardless of which
+    backend ran - see docmd/converters/base.py:ConversionResult.provenance.
+    For debugging "this converted differently yesterday"."""
+    result = convert_document(str(FIXTURES / "sample.pdf"))
+    prov = result.provenance
+    assert prov["backend"] == "marker"
+    assert prov["docmd_version"] != "unknown"
+    assert prov["backend_version"] != "unknown"
+    assert prov["ocr_used"] is False  # sample.pdf has a real text layer
+    assert isinstance(prov["conversion_duration_ms"], int)
+    assert prov["conversion_duration_ms"] > 0
+
+
 @pytest.mark.skipif(
     not _HAS_WEASYPRINT_DEPS,
     reason="weasyprint's native deps (Pango/GObject/Cairo) aren't installed "

docmd_cli-0.1.3/tests/test_marker_converter_errors.py

@@ -0,0 +1,49 @@
+"""Unit tests for MarkerConverter's exception classification - mocked so
+these run fast and don't depend on whether llama-server happens to be
+installed on the machine running the tests. See
+docmd/converters/marker_converter.py for the real (slow, environment-
+dependent) version of this failure, found via real OCR testing on a
+genuinely scanned PDF from archive.org.
+"""
+
+from __future__ import annotations
+
+import docmd.converters.marker_converter as marker_converter_module
+from docmd.config import ConvertConfig
+from docmd.converters.marker_converter import MarkerConverter
+from docmd.errors import MissingSystemDependencyError
+
+
+class _FakeSpawnError(Exception):
+    """Stands in for surya.inference.backends.spawn.SpawnError without
+    importing surya's internal exception class directly."""
+
+
+class _FakePdfConverterMissingBinary:
+    def __init__(self, **_kwargs) -> None:
+        pass
+
+    def __call__(self, _filepath: str):
+        raise _FakeSpawnError(
+            "llama-server binary not found. Install with:\n"
+            "  macOS:  brew install llama.cpp"
+        )
+
+
+def test_missing_llama_server_binary_raises_clear_error(monkeypatch, tmp_path):
+    monkeypatch.setattr(marker_converter_module, "_get_model_dict", lambda: {})
+    monkeypatch.setattr(
+        "marker.converters.pdf.PdfConverter", _FakePdfConverterMissingBinary
+    )
+
+    dummy_pdf = tmp_path / "dummy.pdf"
+    dummy_pdf.write_bytes(b"%PDF-1.4\n%%EOF")
+
+    try:
+        MarkerConverter().convert(str(dummy_pdf), ConvertConfig())
+        raise AssertionError("expected MissingSystemDependencyError")
+    except MissingSystemDependencyError as exc:
+        message = str(exc)
+        assert "llama.cpp" in message
+        assert "brew install llama.cpp" in message
+        assert "LLAMA_CPP_BINARY" in message

{docmd_cli-0.1.1 → docmd_cli-0.1.3}/tests/test_postprocess.py

@@ -9,6 +9,30 @@ def test_heading_normalize_fixes_level_skip():
     assert out == "# Title\n## Subsection\n"
 
 
+def test_heading_normalize_fixes_level_skip_then_leaves_recovered_depth_alone():
+    """The exact pattern found in a real 27-page IMF report while stress-
+    testing against downloaded (not synthetic) documents: an H1 jumps
+    straight to H4 (skipping H2/H3), gets clamped to H2 - but a *later*
+    heading that's also originally H4 is left alone, because an H3 heading
+    in between legitimately re-establishes the depth. Never reproduced
+    synthetically (see skip_headings.pdf in git history, removed after
+    Marker's own layout model wouldn't cooperate) - this encodes the real
+    structure without redistributing the copyrighted source PDF."""
+    md = (
+        "# Summary of the Economy Classification\n"
+        "#### General Features and Composition of Groups\n"
+        "### Advanced Economies\n"
+        "#### Emerging Market and Developing Economies\n"
+    )
+    out = normalize_headings(md)
+    assert out == (
+        "# Summary of the Economy Classification\n"
+        "## General Features and Composition of Groups\n"
+        "### Advanced Economies\n"
+        "#### Emerging Market and Developing Economies\n"
+    )
+
+
 def test_heading_normalize_drops_empty_heading():
     md = "# Title\n##\nBody text.\n"
     out = normalize_headings(md)
@@ -98,3 +122,36 @@ def test_image_handling_alt_text_mode_fills_empty_alt():
     md = "![](img.png)"
     out = apply_image_handling(md, images={}, mode="alt-text")
     assert out == "![Image 1](img.png)"
+
+
+def test_image_handling_drops_fully_empty_image_reference():
+    """Real Marker output on a scanned page: `![]()` with no href at all,
+    directly adjacent to a real image reference with no separator between
+    them - found while OCR-testing a genuinely scanned PDF. The old regex
+    required at least one character inside the parens, so `![]()` never
+    matched and leaked through every mode unprocessed."""
+    md = "![]()![](real.jpeg)"
+
+    assert apply_image_handling(md, images={}, mode="placeholder") == "*[image omitted]*"
+    assert apply_image_handling(md, images={}, mode="skip") == ""
+    assert apply_image_handling(md, images={}, mode="alt-text") == "![Image 1](real.jpeg)"
+
+
+def test_clean_tables_drops_stray_duplicate_separator_row():
+    """Real Marker output found stress-testing a real two-column paper
+    ("Attention Is All You Need"): a single-row "table" - actually a
+    numbered equation rendered as a 1-cell table - came out with two
+    separator rows back to back and no real data row. The old code treated
+    the second separator as a literal data row and printed a row of dashes
+    into the output."""
+    md = (
+        "| Attention(Q, K, V) = softmax(...)V | (1) |\n"
+        "|----------------------------------------|-----|\n"
+        "|----------------------------------------|-----|\n"
+    )
+    out = clean_tables(md)
+    lines = [l for l in out.splitlines() if l.strip()]
+    assert lines == [
+        "| Attention(Q, K, V) = softmax(...)V | (1) |",
+        "| --- | --- |",
+    ]

{docmd_cli-0.1.1 → docmd_cli-0.1.3}/tests/test_postprocess_more_integration.py

@@ -16,18 +16,32 @@ FIXTURES = Path(__file__).parent / "fixtures"
 
 def test_inconsistent_heading_levels_get_normalized_to_match():
     """running_header.pdf has 4 structurally identical section headings
-    (same paragraph style). Raw Marker assigns them *different* levels -
-    Section 1 comes out H3, Sections 2-4 come out H2, a genuine model
-    inconsistency, not a deliberately constructed skip. Confirm that first,
-    then confirm docmd normalizes them to be consistent."""
+    (same paragraph style). On the machine this was first written on (macOS,
+    MPS), raw Marker assigned them *different* levels - Section 1 came back
+    H3, Sections 2-4 came back H2 - a genuine model inconsistency, not a
+    deliberately constructed skip.
+
+    That specific raw output isn't portable, though: CI (Linux, CPU) ran the
+    exact same PDF through the exact same pinned Marker version and got all
+    four sections back as H2 from the start - a real platform-dependent
+    difference in the underlying ML model's inference, not something our
+    code controls. Hard-pinning the raw "bug" broke CI twice for that
+    reason. So this only *asserts* the thing docmd actually guarantees -
+    consistent heading levels across all four sections, however Marker
+    happened to render them - and reports (without failing) whether this
+    particular run's raw output shows the original inconsistency."""
     raw = MarkerConverter().convert(str(FIXTURES / "running_header.pdf"), ConvertConfig())
-    assert "### **Section 1: Findings**" in raw.markdown
-    assert "## **Section 2: Findings**" in raw.markdown
+    if "### **Section 1: Findings**" in raw.markdown:
+        print("raw Marker output reproduced the H3-vs-H2 inconsistency on this run")
+    else:
+        print("raw Marker output did not reproduce the inconsistency on this run/platform")
 
     result = convert_document(str(FIXTURES / "running_header.pdf"))
-    for n in range(1, 5):
-        assert f"## **Section {n}: Findings**" in result.markdown
-        assert f"### **Section {n}: Findings**" not in result.markdown
+    levels = {
+        n: "###" if f"### **Section {n}: Findings**" in result.markdown else "##"
+        for n in range(1, 5)
+    }
+    assert len(set(levels.values())) == 1, f"inconsistent levels in docmd output: {levels}"
 
 
 def test_running_header_does_not_leak_into_output():
@@ -43,19 +57,25 @@ def test_running_header_does_not_leak_into_output():
 
 
 def test_merged_header_cells_survive_cleanly():
-    """merged_cells.pdf has a spanned header cell (colspan). Honest negative
-    result: Marker already renders this as a clean, consistent 5-column
-    table (blank cells for the spanned columns) with no ragged rows -
-    nothing for table_cleanup.py to fix here. Pinned so this doesn't
-    silently regress, and so nobody assumes this case needs a fix it
-    doesn't need."""
+    """merged_cells.pdf has a spanned header cell (colspan). On the machine
+    this was first written on, raw Marker already renders it as a clean,
+    consistent 5-column table with no ragged rows - an honest negative
+    result, nothing for table_cleanup.py to fix.
+
+    Same lesson as test_inconsistent_heading_levels_get_normalized_to_match:
+    the exact raw structure is backend inference output, not something
+    docmd controls, so it's reported (via print) rather than asserted. What
+    docmd actually guarantees - a well-formed, consistent table in the
+    processed output, with the real data intact - is what's asserted."""
     raw = MarkerConverter().convert(str(FIXTURES / "merged_cells.pdf"), ConvertConfig())
-    table_lines = [l for l in raw.markdown.splitlines() if l.strip().startswith("|")]
-    # spanned quarter-label row, column-label row, separator, 2 data rows.
-    assert len(table_lines) == 5
-    assert all(line.count("|") == 6 for line in table_lines)  # 5 columns, consistent
+    raw_table_lines = [l for l in raw.markdown.splitlines() if l.strip().startswith("|")]
+    raw_col_counts = {line.count("|") for line in raw_table_lines}
+    print(f"raw table: {len(raw_table_lines)} rows, column counts seen: {raw_col_counts}")
 
     result = convert_document(str(FIXTURES / "merged_cells.pdf"))
+    proc_table_lines = [l for l in result.markdown.splitlines() if l.strip().startswith("|")]
+    proc_col_counts = {line.count("|") for line in proc_table_lines}
+    assert len(proc_col_counts) == 1, f"docmd output has inconsistent column counts: {proc_col_counts}"
     assert "West" in result.markdown and "$1.2M" in result.markdown
 
 

docmd_cli-0.1.1/docmd/converters/marker_converter.py

@@ -1,89 +0,0 @@
-"""Wraps `marker-pdf` (https://github.com/datalab-to/marker) as a docmd Converter.
-
-Uses Marker's Python API directly (not its CLI/server) so we control input and
-output cleanly. `PdfConverter` is Marker's general-purpose converter despite
-the name: it dispatches to the right internal provider (PDF, DOCX, PPTX, ...)
-based on the file's actual content, via `provider_from_filepath`.
-"""
-
-from __future__ import annotations
-
-from pathlib import Path
-from typing import Any
-
-from docmd.config import ConvertConfig
-from docmd.converters.base import ConversionResult
-from docmd.errors import ConversionError, EncryptedDocumentError, MissingExtraError
-
-# Marker loads its models (a few hundred MB to ~1GB of weights, downloaded
-# from Hugging Face on first use) lazily and caches them at module scope, so
-# repeated conversions within one process don't reload them.
-_model_dict: dict[str, Any] | None = None
-
-_PASSWORD_HINTS = ("password", "encrypt")
-
-
-def _get_model_dict() -> dict[str, Any]:
-    global _model_dict
-    if _model_dict is None:
-        from marker.models import create_model_dict
-
-        _model_dict = create_model_dict()
-    return _model_dict
-
-
-def _build_config_dict(config: ConvertConfig):
-    from marker.config.parser import ConfigParser
-
-    options: dict[str, Any] = {
-        "output_format": "markdown",
-        "force_ocr": config.force_ocr,
-        "use_llm": config.use_llm,
-    }
-    if config.image_mode == "skip":
-        options["disable_image_extraction"] = True
-
-    return ConfigParser(options)
-
-
-class MarkerConverter:
-    """Converter implementation backed by Marker."""
-
-    def convert(self, filepath: str | Path, config: ConvertConfig) -> ConversionResult:
-        filepath = str(filepath)
-        try:
-            from marker.converters.pdf import PdfConverter
-            from marker.output import text_from_rendered
-        except ImportError as exc:
-            raise MissingExtraError(Path(filepath).suffix) from exc
-
-        config_parser = _build_config_dict(config)
-        config_dict = config_parser.generate_config_dict()
-
-        try:
-            converter = PdfConverter(
-                config=config_dict,
-                artifact_dict=_get_model_dict(),
-                processor_list=config_parser.get_processors(),
-                renderer=config_parser.get_renderer(),
-                llm_service=config_parser.get_llm_service(),
-            )
-            rendered = converter(filepath)
-        except Exception as exc:
-            message = str(exc).lower()
-            if any(hint in message for hint in _PASSWORD_HINTS):
-                raise EncryptedDocumentError() from exc
-            raise ConversionError(
-                f"Marker failed to convert '{filepath}': {exc}", cause=exc
-            ) from exc
-
-        markdown, _, images = text_from_rendered(rendered)
-        metadata = dict(getattr(rendered, "metadata", {}) or {})
-        page_count = len(metadata.get("page_stats", [])) or 1
-
-        return ConversionResult(
-            markdown=markdown,
-            page_count=page_count,
-            images=images,
-            metadata=metadata,
-        )