docmd-cli 0.1.2__tar.gz → 0.1.3__tar.gz

{docmd_cli-0.1.2 → docmd_cli-0.1.3}/ARCHITECTURE.md

@@ -1,5 +1,11 @@
 # Architecture: Doc-to-Markdown API
 
+See [CONTRACT.md](CONTRACT.md) for what docmd actually guarantees about its output,
+independent of Marker or any future backend - the formalized version of a testing
+philosophy this project arrived at the hard way, via two real CI failures caused by
+tests that pinned Marker's exact raw output instead of docmd's own guarantees
+(commits `066fd1b` and part of `c7225c5`).
+
 ## The pitch (keep this pinned above your desk)
 We are not competing on conversion quality. We are selling **"hit an endpoint, get clean Markdown back"** — no Python env, no 8GB+ RAM, no GPU, no dependency hell. The open-source core proves the engine works and builds trust. The hosted API sells convenience.
 
@@ -160,11 +166,15 @@ Treat this folder as the actual differentiator. A thin wrapper is a weekend proj
 
 ## License clarity
 
-MIT for the `docmd` wrapper is fine and expected for an OSS trust play. Marker's code is
-Apache-2.0. Marker's *model weights* are under a modified Open RAIL-M license: free for
-research, personal use, and organizations under $5M funding/revenue; beyond that, a paid
-license from Datalab is required (see datalab.to/pricing). Re-check this before scaling
-the hosted API past that threshold.
+Full detail moved to [docs/licensing.md](docs/licensing.md) so it's maintained in one
+place instead of drifting between README and this file. Short version: MIT for the
+`docmd` wrapper is fine and expected; Marker's code (Apache-2.0) and model weights
+(modified Open RAIL-M, $5M funding/revenue threshold) are licensed separately, and the
+weights license is the one that actually constrains a commercial hosted API at scale.
+Re-check `docs/licensing.md` before scaling `api/` past that threshold, and get written
+clarification from Datalab on hosted-API competitiveness before a commercial launch -
+see "Positioning risk" above for why Datalab's own competing hosted API makes this
+worth resolving explicitly, not assuming.
 
 ## What "done" looks like for Stage 1 (this repo, today)
 

docmd_cli-0.1.3/CONTRACT.md

@@ -0,0 +1,110 @@
+# The docmd contract
+
+This is what docmd guarantees about its output, independent of which extraction
+backend is running underneath (currently [Marker](https://github.com/datalab-to/marker),
+pinned to `marker-pdf==2.0.0`) or which platform it runs on. Everything here is
+enforced by a test that fails if the guarantee breaks - the file is named next to
+each one.
+
+The reason this document exists: a test suite that asserts *the backend's exact raw
+output* is testing the backend, not docmd, and breaks the moment the backend changes
+- confirmed the hard way twice in this project's history (`066fd1b`, `c7225c5`), where
+a test pinned an exact heading level Marker happened to produce on one platform, and
+CI failed on a different platform running the identical, pinned backend version. Both
+fixes replaced "assert the backend did X" with "assert docmd guarantees Y" - that
+distinction is the entire point of this document.
+
+## What's guaranteed
+
+**Heading hierarchy** (`docmd/postprocess/heading_normalize.py`,
+`tests/test_postprocess.py`, `tests/test_postprocess_more_integration.py`)
+- No heading level skips more than one deeper than the deepest level seen so far.
+  `H1` followed directly by `H4` becomes `H1` followed by `H2`; a later, legitimately
+  deep heading elsewhere in the same document is left alone once the hierarchy has
+  caught up. Confirmed against six independent real documents (an IMF report, four
+  arXiv papers/patents, one hand-built fixture) - this is the single most common
+  real-world defect found in this project's testing.
+- No orphaned `#` with empty text.
+- No heading whose text exactly repeats the immediately preceding heading (a Marker
+  artifact from running page headers).
+- A document's first heading is always promoted to H1 if nothing shallower precedes it.
+
+**Table structure** (`docmd/postprocess/table_cleanup.py`)
+- Every row in a rendered table has the same column count as its header, padded or
+  truncated as needed.
+- A missing or malformed separator row is synthesized.
+- A stray row that is itself separator-shaped (all dashes) is dropped rather than
+  rendered as literal data - found via a real two-column paper where a numbered
+  equation came back as a one-row table with two separator rows.
+- A table split into two blocks by a page break, with a repeated identical header, is
+  merged into one continuous table.
+- **Not guaranteed**: correct table content when the backend's own reading order is
+  wrong. docmd normalizes structure; it doesn't re-derive reading order the backend
+  got wrong (confirmed real gap: a patent's front-page bibliographic table).
+
+**Image references** (`docmd/postprocess/image_handling.py`)
+- No malformed Markdown image syntax (`![]()`  with an empty or partial reference)
+  ever appears in output, in any mode.
+- Three modes, each with a specific, tested behavior: `placeholder` (default, no
+  binary data referenced at all), `alt-text` (real links, non-empty alt text, and the
+  image file is actually written to `output_dir` if given), `skip` (removed entirely).
+
+**Errors** (`docmd/errors.py`)
+- A small, fixed set of `DocmdError` subclasses for known failure categories:
+  `UnsupportedFormatError`, `MissingExtraError`, `MissingSystemDependencyError`,
+  `EncryptedDocumentError`, `ConversionError`. Each carries an actionable message,
+  not a raw backend stack trace, for the failure modes docmd knows about.
+
+**Backend variation**
+- Platform or backend-version differences in raw extraction output never change
+  docmd's own guarantees above. They may change the *content* extracted (a backend
+  bug is still a backend bug), but never the *structural* invariants this document
+  lists.
+
+**Provenance** (`docmd/converters/base.py:ConversionResult.provenance`)
+- Every `convert_document()` call returns `docmd_version`, `backend`,
+  `backend_version`, `ocr_used`, and `conversion_duration_ms` - same shape regardless
+  of which backend ran. This records *what ran*, not that the *output* is
+  reproducible - see OCR determinism below, where it isn't.
+
+## What's explicitly not guaranteed (yet)
+
+Found by real-world testing, not fixed:
+
+- **Form/checkbox structure.** Adjacent checkbox option labels (e.g. a tax form's
+  filing-status options) are not separated or structured - they extract as one
+  run-on phrase, identical to raw backend output.
+- **RTL bidirectional punctuation.** Citation brackets and similar LTR punctuation
+  embedded in right-to-left script can render reversed (`]1[` instead of `[1]`).
+- **Multi-column reading order in dense bibliographic/legal layouts.** Proven to
+  work correctly on a two-column academic paper; proven to fail on a patent's
+  front-page citation block. No known rule yet for which case a given document falls
+  into.
+- **OCR reproducibility.** Running the identical file through the identical code path
+  twice, in the same process, produced different text - confirmed directly, not
+  inferred (28,916 vs 27,973 characters on a real degraded scan; individual word
+  choices and even a poem's line-break structure differed between the two runs).
+  Root cause investigated and is architectural, not a misconfiguration: OCR/equation
+  recognition already requests greedy decoding (`temperature=0.0`, confirmed in
+  `surya/inference/backends/openai_client.py`), but `llama-server` runs with
+  `--parallel 8` by default (`surya/inference/backends/llamacpp.py`), and
+  floating-point matrix multiplication under concurrent batched inference is not
+  strictly order-independent - a well-documented property of essentially every
+  production LLM-serving stack (vLLM, llama.cpp, TensorRT-LLM), not specific to this
+  one. A small numerical difference from batch composition can flip an argmax choice
+  at a near-tied token, and the autoregressive generation diverges from there.
+  `SURYA_INFERENCE_PARALLEL=1` (env var, not currently exposed through docmd's own
+  config) would remove the batching-composition variable and likely reduce this a
+  lot, at a real throughput cost - untested, and even then, multi-threaded CPU matrix
+  math without an explicit `--threads 1` could still leave some residual variance.
+  Does not affect the plain `pdftext` text-extraction path (no VLM inference
+  involved) - only pages that actually go through OCR or equation recognition.
+
+## Writing a new test against this contract
+
+Ask: does this assertion describe something docmd promises, or something the
+backend happened to do on one machine? If the latter, either don't assert it, or -
+if it's worth pinning as a canary for upstream behavior changing silently - say so
+explicitly in the test's docstring and keep it separate from the real assertion, the
+way `test_marker_really_does_split_a_page_spanning_table` and
+`test_running_header_does_not_leak_into_output` do.

{docmd_cli-0.1.2 → docmd_cli-0.1.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: docmd-cli
-Version: 0.1.2
+Version: 0.1.3
 Summary: Convert PDFs, DOCX, and PPTX to clean, structure-preserving Markdown.
 Project-URL: Homepage, https://github.com/taherzribi/docmd
 Project-URL: Issues, https://github.com/taherzribi/docmd/issues
@@ -145,6 +145,13 @@ Without it, converting a scanned PDF or one with equations raises a
 First real OCR run also downloads the model's GGUF weights from Hugging Face
 (a few GB), separate from the PyTorch weights Marker already downloaded.
 
+**OCR output is not guaranteed reproducible run to run** - confirmed directly:
+the same scanned file through the same code path twice produced different text.
+This is architectural (concurrent batched inference under the hood), not a docmd
+bug and not fixable by a config flag alone - see [CONTRACT.md](CONTRACT.md) for the
+full investigation. Doesn't affect plain text-layer PDFs, only pages that actually
+go through OCR or equation recognition.
+
 ## Image handling
 
 Images default to a text placeholder (`*[... omitted]*`) - no binary data, nothing to
@@ -176,24 +183,41 @@ docmd convert report.pdf -o output/report.md --image-mode alt-text
 
 `image_mode="skip"` drops images entirely - no placeholder, no files.
 
+## Provenance
+
+Every conversion carries tracking info in `result.provenance` - useful for "this
+converted differently yesterday" debugging:
+
+```python
+from docmd import convert_document
+
+result = convert_document("report.pdf")
+print(result.provenance)
+# {'docmd_version': '0.1.3', 'backend': 'marker', 'backend_version': '2.0.0',
+#  'ocr_used': False, 'conversion_duration_ms': 489}
+```
+
+Same shape regardless of which backend actually ran - `convert()` (the plain
+string-returning function) doesn't expose this; use `convert_document()` for it.
+
 ## How it works
 
 `docmd` wraps [Marker](https://github.com/datalab-to/marker) with sane defaults and a
 clean output format, then runs its own post-processing pass
 (`docmd/postprocess/`) to fix table structure and normalize heading levels — see
 [ARCHITECTURE.md](ARCHITECTURE.md) for why this is the actual differentiation, not
-just a thin wrapper.
+just a thin wrapper, and [CONTRACT.md](CONTRACT.md) for exactly what's guaranteed
+about the output (and what isn't, yet) — independent of Marker or any future backend.
 
 ## License
 
 The `docmd` wrapper code is MIT — see [LICENSE](LICENSE).
 
-`docmd` depends on [Marker](https://github.com/datalab-to/marker), whose *code* is
-Apache-2.0 and whose *model weights* are licensed under a modified Open RAIL-M license:
-free for research, personal use, and organizations under $5M in funding or revenue.
-Commercial use beyond that threshold requires a license from
-[Datalab](https://www.datalab.to/pricing). This applies to you if you deploy `docmd`
-commercially at scale — check Marker's current license terms directly before doing so.
+`docmd` depends on [Marker](https://github.com/datalab-to/marker), whose code and
+model weights carry *separate* licenses — the weights license has a revenue/funding
+threshold that matters if you deploy commercially at scale. See
+[docs/licensing.md](docs/licensing.md) for the precise terms; don't rely on this
+sentence alone.
 
 ## Roadmap
 

{docmd_cli-0.1.2 → docmd_cli-0.1.3}/README.md

@@ -118,6 +118,13 @@ Without it, converting a scanned PDF or one with equations raises a
 First real OCR run also downloads the model's GGUF weights from Hugging Face
 (a few GB), separate from the PyTorch weights Marker already downloaded.
 
+**OCR output is not guaranteed reproducible run to run** - confirmed directly:
+the same scanned file through the same code path twice produced different text.
+This is architectural (concurrent batched inference under the hood), not a docmd
+bug and not fixable by a config flag alone - see [CONTRACT.md](CONTRACT.md) for the
+full investigation. Doesn't affect plain text-layer PDFs, only pages that actually
+go through OCR or equation recognition.
+
 ## Image handling
 
 Images default to a text placeholder (`*[... omitted]*`) - no binary data, nothing to
@@ -149,24 +156,41 @@ docmd convert report.pdf -o output/report.md --image-mode alt-text
 
 `image_mode="skip"` drops images entirely - no placeholder, no files.
 
+## Provenance
+
+Every conversion carries tracking info in `result.provenance` - useful for "this
+converted differently yesterday" debugging:
+
+```python
+from docmd import convert_document
+
+result = convert_document("report.pdf")
+print(result.provenance)
+# {'docmd_version': '0.1.3', 'backend': 'marker', 'backend_version': '2.0.0',
+#  'ocr_used': False, 'conversion_duration_ms': 489}
+```
+
+Same shape regardless of which backend actually ran - `convert()` (the plain
+string-returning function) doesn't expose this; use `convert_document()` for it.
+
 ## How it works
 
 `docmd` wraps [Marker](https://github.com/datalab-to/marker) with sane defaults and a
 clean output format, then runs its own post-processing pass
 (`docmd/postprocess/`) to fix table structure and normalize heading levels — see
 [ARCHITECTURE.md](ARCHITECTURE.md) for why this is the actual differentiation, not
-just a thin wrapper.
+just a thin wrapper, and [CONTRACT.md](CONTRACT.md) for exactly what's guaranteed
+about the output (and what isn't, yet) — independent of Marker or any future backend.
 
 ## License
 
 The `docmd` wrapper code is MIT — see [LICENSE](LICENSE).
 
-`docmd` depends on [Marker](https://github.com/datalab-to/marker), whose *code* is
-Apache-2.0 and whose *model weights* are licensed under a modified Open RAIL-M license:
-free for research, personal use, and organizations under $5M in funding or revenue.
-Commercial use beyond that threshold requires a license from
-[Datalab](https://www.datalab.to/pricing). This applies to you if you deploy `docmd`
-commercially at scale — check Marker's current license terms directly before doing so.
+`docmd` depends on [Marker](https://github.com/datalab-to/marker), whose code and
+model weights carry *separate* licenses — the weights license has a revenue/funding
+threshold that matters if you deploy commercially at scale. See
+[docs/licensing.md](docs/licensing.md) for the precise terms; don't rely on this
+sentence alone.
 
 ## Roadmap
 

{docmd_cli-0.1.2 → docmd_cli-0.1.3}/docmd/__init__.py

@@ -18,7 +18,7 @@ from docmd.postprocess.image_handling import apply_image_handling
 from docmd.postprocess.table_cleanup import clean_tables
 
 __all__ = ["convert", "convert_document", "ConvertConfig", "ConversionResult"]
-__version__ = "0.1.2"
+__version__ = "0.1.3"
 
 
 def convert_document(
@@ -92,4 +92,5 @@ def _convert_path(
         page_count=result.page_count,
         images=result.images,
         metadata=result.metadata,
+        provenance=result.provenance,
     )

{docmd_cli-0.1.2 → docmd_cli-0.1.3}/docmd/converters/base.py

@@ -26,6 +26,15 @@ class ConversionResult:
     """Maps an image filename referenced in `markdown` (e.g.
     '_page_0_Figure_1.jpeg') to a PIL.Image.Image instance."""
     metadata: dict[str, Any] = field(default_factory=dict)
+    """Whatever the backend itself returned - opaque, backend-specific, not
+    a contract. For Marker this is its own metadata dict (page_stats,
+    table_of_contents, ...). A different backend would put different keys
+    here; don't build stable behavior on top of this dict's shape."""
+    provenance: dict[str, Any] = field(default_factory=dict)
+    """docmd's own tracking info, same shape regardless of which backend
+    ran: docmd_version, backend, backend_version, ocr_used,
+    conversion_duration_ms. For debugging "this converted differently
+    yesterday" - see ARCHITECTURE.md."""
 
 
 class Converter(Protocol):

{docmd_cli-0.1.2 → docmd_cli-0.1.3}/docmd/converters/marker_converter.py

@@ -8,6 +8,8 @@ based on the file's actual content, via `provider_from_filepath`.
 
 from __future__ import annotations
 
+import importlib.metadata
+import time
 from pathlib import Path
 from typing import Any
 
@@ -42,6 +44,34 @@ def _get_model_dict() -> dict[str, Any]:
     return _model_dict
 
 
+_docmd_version_cache: str | None = None
+_marker_version_cache: str | None = None
+
+
+def _docmd_version() -> str:
+    # Read from installed package metadata rather than `from docmd import
+    # __version__` - that would import docmd/__init__.py, which imports
+    # this module transitively (via the converter registry), a circular
+    # import.
+    global _docmd_version_cache
+    if _docmd_version_cache is None:
+        try:
+            _docmd_version_cache = importlib.metadata.version("docmd-cli")
+        except importlib.metadata.PackageNotFoundError:
+            _docmd_version_cache = "unknown"
+    return _docmd_version_cache
+
+
+def _marker_version() -> str:
+    global _marker_version_cache
+    if _marker_version_cache is None:
+        try:
+            _marker_version_cache = importlib.metadata.version("marker-pdf")
+        except importlib.metadata.PackageNotFoundError:
+            _marker_version_cache = "unknown"
+    return _marker_version_cache
+
+
 def _build_config_dict(config: ConvertConfig):
     from marker.config.parser import ConfigParser
 
@@ -82,6 +112,7 @@ class MarkerConverter:
         config_parser = _build_config_dict(config)
         config_dict = config_parser.generate_config_dict()
 
+        start = time.monotonic()
         try:
             converter = PdfConverter(
                 config=config_dict,
@@ -102,13 +133,32 @@ class MarkerConverter:
                 f"Marker failed to convert '{filepath}': {exc}", cause=exc
             ) from exc
 
+        duration_ms = int((time.monotonic() - start) * 1000)
         markdown, _, images = text_from_rendered(rendered)
         metadata = dict(getattr(rendered, "metadata", {}) or {})
-        page_count = len(metadata.get("page_stats", [])) or 1
+        page_stats = metadata.get("page_stats", [])
+        page_count = len(page_stats) or 1
+
+        # A page whose text_extraction_method isn't "pdftext" went through
+        # OCR/vision-based recognition rather than reading an embedded text
+        # layer - independent of force_ocr, since Marker also falls back to
+        # this per-page for a PDF with no usable text layer at all.
+        ocr_used = config.force_ocr or any(
+            page.get("text_extraction_method") != "pdftext" for page in page_stats
+        )
+
+        provenance = {
+            "docmd_version": _docmd_version(),
+            "backend": "marker",
+            "backend_version": _marker_version(),
+            "ocr_used": ocr_used,
+            "conversion_duration_ms": duration_ms,
+        }
 
         return ConversionResult(
             markdown=markdown,
             page_count=page_count,
             images=images,
             metadata=metadata,
+            provenance=provenance,
         )

{docmd_cli-0.1.2 → docmd_cli-0.1.3}/docmd/postprocess/table_cleanup.py

@@ -98,7 +98,14 @@ def clean_tables(markdown: str) -> str:
         block_end = j
         while block_end < len(lines) and _is_row(lines[block_end]):
             block_end += 1
-        data_rows = [_split_cells(l) for l in lines[j:block_end]]
+        # A stray extra separator row (found via a real two-column paper: a
+        # single-row "table" - actually a numbered equation - with two
+        # separator rows back to back and no real data) would otherwise get
+        # treated as a literal data row of dashes. Drop any row that's
+        # itself separator-shaped rather than rendering it as content.
+        data_rows = [
+            _split_cells(l) for l in lines[j:block_end] if not _is_separator_row(l)
+        ]
 
         # Look ahead past blank lines / Marker's page separator for a
         # continuation: another table block whose header repeats this one.
@@ -120,7 +127,11 @@ def clean_tables(markdown: str) -> str:
             next_block_end = p
             while next_block_end < len(lines) and _is_row(lines[next_block_end]):
                 next_block_end += 1
-            data_rows.extend(_split_cells(l) for l in lines[p:next_block_end])
+            data_rows.extend(
+                _split_cells(l)
+                for l in lines[p:next_block_end]
+                if not _is_separator_row(l)
+            )
             k = next_block_end
 
         out.extend(_normalize_block(header, data_rows))

docmd_cli-0.1.3/docs/licensing.md

@@ -0,0 +1,50 @@
+# Licensing
+
+Three separate things, three separate licenses. Conflating them is the exact
+overclaim this document exists to prevent.
+
+## docmd itself
+
+MIT - see [LICENSE](../LICENSE). No restrictions on commercial use, modification, or
+redistribution of docmd's own wrapper/post-processing code.
+
+## Marker's source code
+
+docmd depends on [Marker](https://github.com/datalab-to/marker), pinned to
+`marker-pdf==2.0.0`. Marker's code is Apache-2.0. No restriction relevant here.
+
+## Marker's model weights
+
+This is the one that actually constrains you, and it's separate from the code
+license above - a common point of confusion, and the reason this file exists rather
+than a single blanket "Marker is Apache-2.0" statement.
+
+Marker's model weights (the actual OCR/layout/recognition models, downloaded
+separately from the code) are licensed under a **modified Open RAIL-M license**:
+
+- Free for research, personal use, and organizations under **$5M in funding or
+  revenue**.
+- Beyond that threshold, a commercial license from [Datalab](https://www.datalab.to/pricing)
+  (Marker's maintainer) is required.
+
+**This applies to you if** you deploy docmd commercially at a scale that crosses that
+threshold - not at small/bootstrapped scale, but worth planning around before it
+matters. Check Marker's current license terms directly before relying on this
+summary; license terms can change between versions, and the terms above reflect
+`marker-pdf==2.0.0` specifically, checked 2026-09-18.
+
+**If you're building a hosted/SaaS product on top of docmd**: Datalab runs its own
+hosted document-conversion API. Before launching a commercial hosted API built on
+Marker, get written clarification from Datalab on whether your product would be
+considered competitive with theirs under their weights license - don't assume the
+$5M threshold is the only relevant term. This is a business/legal step, not
+something resolvable by reading the license text alone.
+
+## Other dependencies
+
+docmd's `[full]` extra and the OCR/equation path pull in several other open-source
+projects (`weasyprint`, `python-docx`, `python-pptx`, `llama.cpp`, and Marker's own
+transitive dependencies). None of these are known to impose restrictions beyond their
+standard OSS licenses (BSD/MIT/Apache-family) as of this writing, but this file
+doesn't audit them individually - check `pyproject.toml` for the current pinned
+versions if you need to.

{docmd_cli-0.1.2 → docmd_cli-0.1.3}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "docmd-cli"
-version = "0.1.2"
+version = "0.1.3"
 description = "Convert PDFs, DOCX, and PPTX to clean, structure-preserving Markdown."
 readme = "README.md"
 license = { text = "MIT" }

{docmd_cli-0.1.2 → docmd_cli-0.1.3}/tests/test_converters.py

@@ -32,6 +32,21 @@ def test_convert_pdf_produces_structured_markdown():
     assert "|" in result.markdown  # a table survived
 
 
+def test_convert_document_includes_provenance():
+    """docmd's own tracking info (docmd_version, backend, backend_version,
+    ocr_used, conversion_duration_ms), same shape regardless of which
+    backend ran - see docmd/converters/base.py:ConversionResult.provenance.
+    For debugging "this converted differently yesterday"."""
+    result = convert_document(str(FIXTURES / "sample.pdf"))
+    prov = result.provenance
+    assert prov["backend"] == "marker"
+    assert prov["docmd_version"] != "unknown"
+    assert prov["backend_version"] != "unknown"
+    assert prov["ocr_used"] is False  # sample.pdf has a real text layer
+    assert isinstance(prov["conversion_duration_ms"], int)
+    assert prov["conversion_duration_ms"] > 0
+
+
 @pytest.mark.skipif(
     not _HAS_WEASYPRINT_DEPS,
     reason="weasyprint's native deps (Pango/GObject/Cairo) aren't installed "

{docmd_cli-0.1.2 → docmd_cli-0.1.3}/tests/test_postprocess.py

@@ -135,3 +135,23 @@ def test_image_handling_drops_fully_empty_image_reference():
     assert apply_image_handling(md, images={}, mode="placeholder") == "*[image omitted]*"
     assert apply_image_handling(md, images={}, mode="skip") == ""
     assert apply_image_handling(md, images={}, mode="alt-text") == "![Image 1](real.jpeg)"
+
+
+def test_clean_tables_drops_stray_duplicate_separator_row():
+    """Real Marker output found stress-testing a real two-column paper
+    ("Attention Is All You Need"): a single-row "table" - actually a
+    numbered equation rendered as a 1-cell table - came out with two
+    separator rows back to back and no real data row. The old code treated
+    the second separator as a literal data row and printed a row of dashes
+    into the output."""
+    md = (
+        "| Attention(Q, K, V) = softmax(...)V | (1) |\n"
+        "|----------------------------------------|-----|\n"
+        "|----------------------------------------|-----|\n"
+    )
+    out = clean_tables(md)
+    lines = [l for l in out.splitlines() if l.strip()]
+    assert lines == [
+        "| Attention(Q, K, V) = softmax(...)V | (1) |",
+        "| --- | --- |",
+    ]

{docmd_cli-0.1.2 → docmd_cli-0.1.3}/tests/test_postprocess_more_integration.py

@@ -57,19 +57,25 @@ def test_running_header_does_not_leak_into_output():
 
 
 def test_merged_header_cells_survive_cleanly():
-    """merged_cells.pdf has a spanned header cell (colspan). Honest negative
-    result: Marker already renders this as a clean, consistent 5-column
-    table (blank cells for the spanned columns) with no ragged rows -
-    nothing for table_cleanup.py to fix here. Pinned so this doesn't
-    silently regress, and so nobody assumes this case needs a fix it
-    doesn't need."""
+    """merged_cells.pdf has a spanned header cell (colspan). On the machine
+    this was first written on, raw Marker already renders it as a clean,
+    consistent 5-column table with no ragged rows - an honest negative
+    result, nothing for table_cleanup.py to fix.
+
+    Same lesson as test_inconsistent_heading_levels_get_normalized_to_match:
+    the exact raw structure is backend inference output, not something
+    docmd controls, so it's reported (via print) rather than asserted. What
+    docmd actually guarantees - a well-formed, consistent table in the
+    processed output, with the real data intact - is what's asserted."""
     raw = MarkerConverter().convert(str(FIXTURES / "merged_cells.pdf"), ConvertConfig())
-    table_lines = [l for l in raw.markdown.splitlines() if l.strip().startswith("|")]
-    # spanned quarter-label row, column-label row, separator, 2 data rows.
-    assert len(table_lines) == 5
-    assert all(line.count("|") == 6 for line in table_lines)  # 5 columns, consistent
+    raw_table_lines = [l for l in raw.markdown.splitlines() if l.strip().startswith("|")]
+    raw_col_counts = {line.count("|") for line in raw_table_lines}
+    print(f"raw table: {len(raw_table_lines)} rows, column counts seen: {raw_col_counts}")
 
     result = convert_document(str(FIXTURES / "merged_cells.pdf"))
+    proc_table_lines = [l for l in result.markdown.splitlines() if l.strip().startswith("|")]
+    proc_col_counts = {line.count("|") for line in proc_table_lines}
+    assert len(proc_col_counts) == 1, f"docmd output has inconsistent column counts: {proc_col_counts}"
     assert "West" in result.markdown and "$1.2M" in result.markdown