pdf-rtldoc 1.0.6__py3-none-any.whl

pdf_rtldoc-1.0.6.dist-info/METADATA

@@ -0,0 +1,163 @@
+Metadata-Version: 2.4
+Name: pdf-rtldoc
+Version: 1.0.6
+Summary: Geometry-first PDF parser for RTL and complex layouts (Arabic textbooks, financial tables, multi-column docs)
+License: MIT
+Project-URL: Homepage, https://github.com/umerjavaidkh/rtldoc
+Project-URL: Repository, https://github.com/umerjavaidkh/rtldoc
+Project-URL: Issues, https://github.com/umerjavaidkh/rtldoc/issues
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: pymupdf>=1.24
+Requires-Dist: numpy
+Provides-Extra: cv
+Requires-Dist: opencv-python-headless; extra == "cv"
+Provides-Extra: dev
+Requires-Dist: arabic-reshaper; extra == "dev"
+Requires-Dist: python-bidi; extra == "dev"
+Dynamic: license-file
+
+# rtldoc
+
+**A geometry-first PDF parser for RTL & complex layouts.** It extracts text,
+reading order, and tables from born-digital PDFs using the glyph coordinates
+already in the file — **no OCR, no GPU, no API**. Built for the pages that
+quietly break Docling, Marker, and VLM parsers: Arabic/RTL, multi-column,
+and vector tables.
+
+---
+
+## Why it's better where it matters
+
+- **RTL / Arabic done right.** Reading order and bidi are rebuilt from glyph
+  *positions*, not from a reading-order model trained on English. It fixes the
+  presentation-form and lam-alef bugs that silently corrupt ~⅓ of Arabic words
+  in every general parser — encoding bugs no layout model can fine-tune away.
+- **Tables without a model.** Reads table structure from the PDF's own vector
+  rules at 100% (vs ~85–95% from a CNN), and recovers *borderless* tables by
+  column alignment.
+- **Deterministic & auditable.** Every output traces to a rule you can point
+  at. CPU-only, **6–60 pages/sec**, $0/page — a VLM is 100–1000× the cost and
+  can't be audited.
+- **Page visual summary, no vision model.** Opt-in (`--visual`) geometry pass
+  that turns a page's own vector drawings into a structured summary: image
+  size/color stats, table dimensions, and — for vector flowcharts/diagrams —
+  the actual boxes, their text labels, and which ones a connecting line joins.
+  Built entirely from the PDF's own drawing commands (PyMuPDF `get_drawings()`),
+  not a screenshot or a guess. HTML output renders any detected diagram as a
+  real chart via Mermaid.js, fed the extracted nodes/edges directly.
+
+## Proven at scale
+
+Tested on **119 PDFs / 13,557 pages** it never saw during development — an
+Arabic teacher's guide, two SEC 10-Ks, 96+ arXiv papers (15 fields), 5 OpenStax
+physics/chemistry/calculus textbooks (figures, geometry, exercises), the
+3,130-page PostgreSQL 18 manual (deeply-nested reference tables, code blocks),
+and a growing set of real-world forms, reports, and scanned documents.
+The checks are *property-based and label-free*, so they scale to any corpus:
+
+| Property (must hold on every page) | Result |
+|---|---:|
+| crashes | **0** |
+| encoding leaks (presentation forms in output) | **0** |
+| malformed tables | **0** |
+| non-deterministic pages | **0** |
+| text coverage vs the PDF's own glyph stream | **~99%** |
+
+**Table quality, scored with TEDS** (the PubTabNet/OmniDocBench standard) on a
+borderless financial statement — where the whole point is a hard table:
+
+| rtldoc | pdfplumber | naive `get_text` |
+|---:|---:|---:|
+| **0.942** | 0.061 | 0.000 |
+
+Everything is reproducible in [`eval/`](eval/) (harnesses, arXiv manifest,
+saved reports).
+
+## Use it
+
+```bash
+pip install pdf-rtldoc
+```
+
+(Published on PyPI as `pdf-rtldoc` -- the plain `rtldoc` name was already
+taken by an unrelated project. The installed command, and the module you
+import in Python, are both still `rtldoc`.)
+
+Or install straight from a specific release without going through PyPI:
+
+```bash
+pip install "git+https://github.com/umerjavaidkh/rtldoc.git@v1.0.6"
+```
+
+Then:
+
+```bash
+rtldoc parse book.pdf --md out/ --json out.json
+rtldoc parse book.pdf --html out_html/   # real <table>/<figure>, RTL-aware dir=
+rtldoc parse book.pdf --html out_html/ --visual  # + diagram/image/table visual summary
+rtldoc audit book.pdf                    # flags low-confidence pages for review
+```
+
+Runtime deps are just PyMuPDF + numpy. (From a clone: `pip install -e .`.)
+
+Zero-setup via Docker (399 MB, no compiler/GPU):
+
+```bash
+docker build -t rtldoc . && docker run --rm -v "$PWD:/d" rtldoc parse /d/book.pdf --md /d/out
+```
+
+Output: per-page Markdown (tables as GFM, images extracted + auto-captioned)
+plus structured JSON, or a self-contained HTML page per PDF page.
+
+## Honest limits
+
+- Borderless-table *grid geometry* is approximate on the hardest wide,
+  multi-level-header tables (occasional row/column structure mismatches
+  — that's the 0.942 TEDS on our graded case, not 1.0).
+- A page dominated by a figure/table spanning the full content width can
+  still be under-counted as fewer columns than it visually has — the
+  whitespace-gutter detector requires the gap to stay empty across most
+  of the page's height, and a full-width element defeats that locally.
+- A table cell whose own text wraps onto a later line, where that line's
+  content coincidentally re-aligns with an earlier row's columns, can
+  occasionally attach to the wrong row (`_merge_wrapped_label_rows` — a
+  narrow, tracked edge case, not a general table-detection failure).
+- Scanned / no-text-layer pages now OCR via Tesseract (needs the `tesseract`
+  binary on PATH — `brew install tesseract` / `apt install tesseract-ocr`;
+  no Python package required). Word-level positioning, not this repo's
+  glyph-exact reading order; a page with no tesseract installed just gets
+  no blocks, as before.
+- Best semantic typing needs a one-time per-publisher style map (~20 min).
+- Diagram detection reconstructs simple box-and-arrow flowcharts reliably;
+  dense multi-level diagrams (deep tree/org-chart hierarchies with many
+  branches) get correct node/box detection but not yet reliable connection
+  tracing — a harder, separate problem noted for future work.
+
+## Roadmap
+
+- Reliable connection tracing for dense/branching diagrams (see above).
+- A page-level chart/figure classification pass, so bar charts, legends,
+  and gridlines are recognized and set aside before table/diagram
+  detection runs, rather than relying on those detectors' own guards to
+  reject them case by case.
+- A cell-level golden regression corpus (`eval/golden/` + `eval/regression.py`)
+  now exists and grows with each table-detection fix; still short of full
+  coverage across document types.
+
+## Publishing a release (maintainers)
+
+Bump `version` in `pyproject.toml`, commit, tag (`git tag -a vX.Y.Z`), push
+the tag, then publish a GitHub Release from it. `.github/workflows/publish.yml`
+builds and uploads to PyPI automatically when the release is published, via
+PyPI's Trusted Publisher (OIDC) mechanism — no API token stored anywhere.
+
+One-time setup (already done for `pdf-rtldoc`): on pypi.org, under the
+project's *Publishing* settings, add a trusted publisher with owner
+`umerjavaidkh`, repository `rtldoc`, workflow filename `publish.yml`, and
+environment name `pypi`.
+
+---
+
+*Design rationale, the full bug log, and methodology: **[DESIGN.md](DESIGN.md)**.*

pdf_rtldoc-1.0.6.dist-info/RECORD

@@ -0,0 +1,16 @@
+pdf_rtldoc-1.0.6.dist-info/licenses/LICENSE,sha256=xNgR7dC-qNVH0g0hABLrZoCct0E67zBtkG8eMVnFTF8,1068
+rtldoc/__init__.py,sha256=AhTCif3PKa_5RBv1CwqDaXpStwMXEVGMcgYrrkyBQwc,70
+rtldoc/arabic.py,sha256=qKDQ_2qDMyeFSF5bWQHqrJ_2l1V3E6pxfpVBbZNHe9I,8401
+rtldoc/cli.py,sha256=xhxORmoaaS3tw-_HmakUz2Zt6Qp6uHXQJbj-hn6AoP0,3575
+rtldoc/cvfallback.py,sha256=DT4Vu5zrd16Mzl1enLeGWHbapSJOQOEKX6iwTL4hGkE,4781
+rtldoc/geobidi.py,sha256=QlFM8Ej-k9AM0ugXx2uPSbWNwSQLDmG_jF73KtVcT9I,10921
+rtldoc/layout.py,sha256=khLpBDaVkpHj8cVFHDw7VdjuU0LINhjqGu-3CQ3PUoU,109664
+rtldoc/ocr.py,sha256=8XmmDz9i1vgvRCJ-m0tzZZPAnyxxMYSFG_xiVqdv59g,4398
+rtldoc/pipeline.py,sha256=7Ty4C16_vIVbWRO34bgiqKkQZq6ENKphS_6w_NV81Is,39087
+rtldoc/primitives.py,sha256=ETs6GK-Pa_5a_wxG0cAhs9ZoDzpTKRSTbeRItEzv4eg,28233
+rtldoc/visual.py,sha256=KmyxdWHGf0NwCkJiFB0YFm5u2TCcinBkUP6M5KgSgoA,24006
+pdf_rtldoc-1.0.6.dist-info/METADATA,sha256=xaYxzIXdea4D-mBx57J1YxhgUdgjvEcKeins5nIBrkA,7143
+pdf_rtldoc-1.0.6.dist-info/WHEEL,sha256=K260EYznzXsJYBQGqmI8VTxEdiZYNvDZwW9cBh9-_MA,91
+pdf_rtldoc-1.0.6.dist-info/entry_points.txt,sha256=pMfV5CHW0E6pF5b_m_p-KhE6w_gAj-8acLyLiujgVYM,43
+pdf_rtldoc-1.0.6.dist-info/top_level.txt,sha256=6mnTflvaxkcMy5mrITEFB6zrlPgz-eFjnh9M_P8CIUs,7
+pdf_rtldoc-1.0.6.dist-info/RECORD,,

pdf_rtldoc-1.0.6.dist-info/WHEEL

@@ -0,0 +1,5 @@
+Wheel-Version: 1.0
+Generator: setuptools (83.0.0)
+Root-Is-Purelib: true
+Tag: py3-none-any
+

pdf_rtldoc-1.0.6.dist-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+rtldoc = rtldoc.cli:main

pdf_rtldoc-1.0.6.dist-info/licenses/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Umer Javaid
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

pdf_rtldoc-1.0.6.dist-info/top_level.txt

@@ -0,0 +1 @@
+rtldoc

rtldoc/__init__.py

@@ -0,0 +1 @@
+"""rtldoc — a geometry-first parser for RTL complex-layout PDFs."""

rtldoc/arabic.py

@@ -0,0 +1,204 @@
+"""
+Arabic text repair for PDF-extracted strings.
+
+Three failure modes are handled, in this order:
+  1. Presentation forms  (U+FB50-FDFF, U+FE70-FEFF) leaking into the text layer
+     because the PDF's ToUnicode map points at shaped glyphs, not base letters.
+  2. Visual-order strings (logically reversed) produced by extractors that do no
+     bidi reordering, or partially-reversed strings caused by ligature glyphs
+     (lam-alef) that survive reversal as a unit.
+  3. Cosmetic noise: kashida/tatweel, non-breaking marks, digit systems.
+
+Nothing here is lossy by default. Diacritics (harakat) are PRESERVED because
+vowelled text is the whole point of a primary-school Arabic corpus.
+"""
+
+from __future__ import annotations
+
+import re
+import unicodedata
+from dataclasses import dataclass
+
+# --------------------------------------------------------------------------
+# character classes
+# --------------------------------------------------------------------------
+
+PRESENTATION_RANGES = ((0xFB50, 0xFDFF), (0xFE70, 0xFEFF))
+ARABIC_BLOCK = re.compile(r"[\u0600-\u06FF\u0750-\u077F\uFB50-\uFDFF\uFE70-\uFEFF]")
+HARAKAT = re.compile(r"[\u064B-\u0652\u0670\u0640]")   # incl. tatweel
+TATWEEL = "\u0640"
+INVISIBLES = dict.fromkeys(map(ord, "\u200b\u200c\u200d\u200e\u200f\u202a\u202b\u202c\u2066\u2067\u2068\u2069"), None)
+
+ARABIC_INDIC = {ord(c): ord(d) for c, d in zip("٠١٢٣٤٥٦٧٨٩", "0123456789")}
+EASTERN_INDIC = {ord(c): ord(d) for c, d in zip("۰۱۲۳۴۵۶۷۸۹", "0123456789")}
+
+# High-frequency Arabic function words. Used as a language-model-free
+# orientation test: a correctly-ordered string contains many of these,
+# a reversed one contains almost none.
+STOPWORDS = frozenset(
+    (
+        "في من على إلى عن أن إن التي الذي هذا هذه ذلك مع كان قد هو هي هم "
+        "بعد قبل بين كل عند ثم أو حتى لكن كما حيث منذ نحو لدى وفي ومن والتي "
+        "الهدف إجابات النص التلاميذ القراءة السؤال إجابة الصف الدرس"
+    ).split()
+)
+_WORD = re.compile(r"[\u0600-\u06FF\u0750-\u077F]+")
+_STRIP_HARAKAT = re.compile(r"[\u064B-\u0652\u0670\u0640]")
+AL = "\u0627\u0644"          # definite article, word-INITIAL in logical order
+AL_REV = "\u0644\u0627"      # what it looks like at word-END when reversed
+
+
+# --------------------------------------------------------------------------
+# primitives
+# --------------------------------------------------------------------------
+
+def is_arabic(text: str, threshold: float = 0.25) -> bool:
+    """True if a meaningful fraction of the letters are Arabic script."""
+    letters = [c for c in text if c.isalpha()]
+    if not letters:
+        return False
+    hits = sum(1 for c in letters if ARABIC_BLOCK.match(c))
+    return hits / len(letters) >= threshold
+
+
+def deshape(text: str) -> str:
+    """Map Arabic presentation forms back to base letters, char by char.
+
+    We deliberately do NOT run NFKC over the whole string: NFKC would also
+    fold full-width Latin, superscripts and some math symbols that carry
+    meaning in a textbook. We only touch the two presentation blocks.
+    """
+    out = []
+    for ch in text:
+        cp = ord(ch)
+        if any(lo <= cp <= hi for lo, hi in PRESENTATION_RANGES):
+            decomposed = unicodedata.normalize("NFKC", ch)
+            out.append(decomposed)
+        else:
+            out.append(ch)
+    return "".join(out)
+
+
+def _score_orientation(text: str) -> int:
+    """How 'logically-ordered' is this Arabic string?
+
+    Two independent signals, both computed on whole tokens (substring matching
+    is worthless here -- 'لا' and 'ما' occur inside half the words in the
+    language and fire constantly on reversed text):
+
+      * exact function-word hits, weighted x3
+      * definite-article position: 'ال' opens roughly a third of tokens in
+        written Arabic. Reversed, it closes them instead. The difference
+        between prefix-count and suffix-count is a very strong, very cheap
+        orientation signal that needs no dictionary.
+    """
+    tokens = [_STRIP_HARAKAT.sub("", t) for t in _WORD.findall(text)]
+    if not tokens:
+        return 0
+    stops = sum(1 for t in tokens if t in STOPWORDS)
+    prefix = sum(1 for t in tokens if len(t) > 3 and t.startswith(AL))
+    suffix = sum(1 for t in tokens if len(t) > 3 and t.endswith(AL_REV))
+    return 3 * stops + (prefix - suffix)
+
+
+MIRROR = str.maketrans("()[]{}<>«»", ")(][}{><»«")
+
+
+def _reverse_preserving_runs(text: str) -> str:
+    """Reverse the string, but keep Latin/digit runs internally intact and
+    mirror directional punctuation.
+
+    Two things a naive `text[::-1]` gets wrong:
+      * an embedded '2021' or 'PDF' was already laid out LTR inside the RTL
+        flow, so reversing char-by-char corrupts it into '1202'
+      * '(' in visual order is the *closing* paren in logical order
+    """
+    tokens = re.findall(r"[A-Za-z0-9][A-Za-z0-9.,:/\-]*|.", text, re.S)
+    return "".join(reversed(tokens)).translate(MIRROR)
+
+
+def fix_order(text: str) -> tuple[str, bool]:
+    """Return (text_in_logical_order, was_reversed).
+
+    Uses a stopword-frequency vote rather than trusting the extractor. This
+    catches the nasty middle case: strings that are *mostly* reordered but
+    where lam-alef ligatures stayed put, which no bidi library will detect
+    for you because the string is already 'valid' Unicode.
+    """
+    if not is_arabic(text):
+        return text, False
+    forward = _score_orientation(text)
+    flipped = _reverse_preserving_runs(text)
+    backward = _score_orientation(flipped)
+    if backward > forward:
+        return flipped, True
+    return text, False
+
+
+_LEAD_PUNCT = re.compile(r"^([.،؛:!؟?]+)\s*(.+)$", re.S)
+_LEAD_ORDINAL = re.compile(r"^([.\u0660-\u06690-9]{1,3})\s*[.\u061B]\s*", re.S)
+
+
+def _repair_punctuation(text: str) -> str:
+    """Move sentence-final punctuation that the reversal stranded at the head.
+
+    Full stops, commas and question marks are Unicode-neutral characters: they
+    take direction from context, so in a visual-order stream they sit at the
+    physical left edge and land at string-start after reversal.
+    """
+    t = text.strip()
+    m = _LEAD_PUNCT.match(t)
+    if m and is_arabic(m.group(2)):
+        t = f"{m.group(2).strip()}{m.group(1)}"
+    # list ordinals: '.. في العبارة1' -> '1. في العبارة'
+    m2 = re.match(r"^\.\s*(.*?)([\u0660-\u06690-9]{1,2})\.?$", t, re.S)
+    if m2 and is_arabic(m2.group(1)):
+        t = f"{m2.group(2)}. {m2.group(1).strip()}."
+    return t
+
+
+@dataclass
+class NormalizeOptions:
+    strip_tatweel: bool = True
+    strip_harakat: bool = False        # keep vowels: this is a teaching corpus
+    western_digits: bool = False       # keep ٢٢ as ٢٢ unless asked
+    collapse_space: bool = True
+
+
+def normalize(text: str, opts: NormalizeOptions | None = None) -> tuple[str, dict]:
+    """Full repair pipeline. Returns (clean_text, diagnostics)."""
+    opts = opts or NormalizeOptions()
+    diag = {"had_presentation_forms": False, "was_reversed": False}
+    text = text.translate(INVISIBLES)
+
+    has_pf = any(any(lo <= ord(c) <= hi for lo, hi in PRESENTATION_RANGES) for c in text)
+    diag["had_presentation_forms"] = has_pf
+
+    # ORDER MATTERS. Reverse BEFORE deshaping.
+    #
+    # A lam-alef ligature is a single codepoint (U+FEFB..U+FEFC) in the
+    # content stream. If we deshape first it becomes two codepoints, and the
+    # subsequent reversal flips them: لا -> ال. Every word containing 'لا'
+    # silently corrupts, which is the failure behind PyMuPDF #2199 and the
+    # reason so much Arabic PDF text "looks almost right". Reversing while
+    # the ligature is still atomic keeps it intact.
+    probe = deshape(text) if has_pf else text          # score on readable text
+    _, reversed_ = fix_order(probe)
+    if reversed_:
+        text = _reverse_preserving_runs(text)
+    diag["was_reversed"] = reversed_
+
+    if has_pf:
+        text = deshape(text)
+    text = _repair_punctuation(text)
+
+    if opts.strip_tatweel:
+        text = text.replace(TATWEEL, "")
+    if opts.strip_harakat:
+        text = HARAKAT.sub("", text)
+    if opts.western_digits:
+        text = text.translate(ARABIC_INDIC).translate(EASTERN_INDIC)
+    if opts.collapse_space:
+        text = re.sub(r"[ \t\u00a0]+", " ", text).strip()
+
+    return text, diag

rtldoc/cli.py

@@ -0,0 +1,75 @@
+"""rtldoc CLI:  python -m rtldoc.cli parse book.pdf --pages 1-5 --md out/"""
+import argparse, json, os, sys
+import fitz
+from . import pipeline, primitives, arabic
+
+def _pages(spec, n):
+    if not spec: return list(range(n))
+    out=[]
+    for part in spec.split(","):
+        if "-" in part:
+            a,b=part.split("-"); out += list(range(int(a)-1,int(b)))
+        else: out.append(int(part)-1)
+    return [p for p in out if 0 <= p < n]
+
+def main(argv=None):
+    ap = argparse.ArgumentParser(prog="rtldoc")
+    sub = ap.add_subparsers(dest="cmd", required=True)
+
+    p = sub.add_parser("parse"); p.add_argument("pdf"); p.add_argument("--pages")
+    p.add_argument("--style-map"); p.add_argument("--json"); p.add_argument("--md")
+    p.add_argument("--html", help="write one HTML page per PDF page, plus styles.css and index.html")
+    p.add_argument("--no-geo", action="store_true")
+    p.add_argument("--strip-harakat", action="store_true")
+    p.add_argument("--visual", action="store_true",
+                   help="add a geometry-only visual summary (images/diagrams/tables/layout) per page, no vision model")
+
+    s = sub.add_parser("styles", help="style census -> label once per book series")
+    s.add_argument("pdf"); s.add_argument("--pages"); s.add_argument("--out")
+
+    a = sub.add_parser("audit"); a.add_argument("pdf"); a.add_argument("--pages")
+
+    args = ap.parse_args(argv)
+    doc = fitz.open(args.pdf); idx = _pages(getattr(args,"pages",None), doc.page_count)
+
+    if args.cmd == "styles":
+        prof = primitives.style_profile([primitives.extract_page(doc[i]) for i in idx])
+        payload = {e["style"]: "" for e in prof}
+        print(json.dumps(prof, ensure_ascii=False, indent=2)[:4000])
+        if args.out:
+            open(args.out,"w").write(json.dumps(payload, ensure_ascii=False, indent=2))
+            print(f"\n-> blank style map written to {args.out}; fill in roles and pass with --style-map", file=sys.stderr)
+        return
+
+    style_map = json.load(open(args.style_map)) if getattr(args,"style_map",None) else None
+    opts = arabic.NormalizeOptions(strip_harakat=getattr(args,"strip_harakat",False))
+    results = [pipeline.parse_page(doc[i], style_map, opts,
+                                   geometry_bidi=not getattr(args,"no_geo",False),
+                                   visual_summary=getattr(args,"visual",False)) for i in idx]
+
+    if args.cmd == "audit":
+        print(json.dumps(pipeline.audit(results), ensure_ascii=False, indent=2)); return
+
+    img_dir = args.html or args.md or (os.path.dirname(args.json) if args.json else None)
+    if img_dir:
+        n = pipeline.save_images(doc, results, os.path.join(img_dir, "images"))
+        if n:
+            print(f"images -> {os.path.join(img_dir, 'images')}/ ({n})", file=sys.stderr)
+
+    if args.json:
+        os.makedirs(os.path.dirname(args.json) or ".", exist_ok=True)
+        open(args.json,"w").write(pipeline.to_json(results)); print(f"json -> {args.json}")
+    if args.md:
+        os.makedirs(args.md, exist_ok=True)
+        for r in results:
+            open(os.path.join(args.md, f"page_{r.page:04d}.md"),"w").write(pipeline.to_markdown(r))
+        print(f"markdown -> {args.md}/")
+    if args.html:
+        title = os.path.splitext(os.path.basename(args.pdf))[0]
+        pipeline.save_html(results, args.html, title_prefix=title)
+        print(f"html -> {args.html}/ ({len(results)} pages, index.html)")
+    if not args.json and not args.md and not args.html:
+        for r in results: print(pipeline.to_markdown(r))
+
+if __name__ == "__main__":
+    main()

rtldoc/cvfallback.py

@@ -0,0 +1,111 @@
+"""
+Layer 2b -- pixel fallback.
+
+Used only when a page has no usable vector fills (flattened PDF, scanned page,
+or a designer who used placed images instead of frames). Classical CV is
+enough here and costs ~15 ms/page against ~400 ms for a layout transformer:
+
+  * colour-plane segmentation finds tinted panels
+  * morphological run-length smearing finds text blocks
+  * Hough/morphology finds rules and table lines
+
+Coordinates are returned in PDF points so they drop straight into the same
+Region list as the vector path. That interchangeability is the point: the
+downstream reading-order and typing stages never learn which branch ran.
+"""
+
+from __future__ import annotations
+
+import cv2
+import numpy as np
+
+from .primitives import Rect
+
+
+def render(page, dpi: int = 200) -> tuple[np.ndarray, float]:
+    """Rasterise a PyMuPDF page. Returns (BGR image, points-per-pixel scale)."""
+    zoom = dpi / 72.0
+    import fitz
+    pix = page.get_pixmap(matrix=fitz.Matrix(zoom, zoom), alpha=False)
+    img = np.frombuffer(pix.samples, dtype=np.uint8).reshape(pix.h, pix.w, 3)
+    return cv2.cvtColor(img, cv2.COLOR_RGB2BGR), 1.0 / zoom
+
+
+def tinted_panels(img: np.ndarray, scale: float, min_area_pt: float = 2000.0,
+                  sat_min: int = 18, val_min: int = 120) -> list[Rect]:
+    """Find flat, low-saturation colour blocks -- the classic textbook callout.
+
+    Deliberately excludes near-white (val high + sat ~0) and photographs
+    (high local variance) so we get containers, not content.
+    """
+    hsv = cv2.cvtColor(img, cv2.COLOR_BGR2HSV)
+    mask = ((hsv[..., 1] > sat_min) & (hsv[..., 2] > val_min)).astype(np.uint8) * 255
+    mask = cv2.morphologyEx(mask, cv2.MORPH_CLOSE, np.ones((9, 9), np.uint8))
+
+    boxes = []
+    cnts, _ = cv2.findContours(mask, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
+    for c in cnts:
+        x, y, w, h = cv2.boundingRect(c)
+        if w * h * scale * scale < min_area_pt:
+            continue
+        # flatness test: a tint is uniform, a photo is not
+        patch = img[y:y + h, x:x + w]
+        if patch.size == 0 or float(patch.std()) > 34:
+            continue
+        boxes.append((x * scale, y * scale, (x + w) * scale, (y + h) * scale))
+    return boxes
+
+
+def text_blocks(img: np.ndarray, scale: float, smear: tuple[int, int] = (25, 5)) -> list[Rect]:
+    """Run-length smearing: dilate horizontally to fuse glyphs into lines,
+    then vertically to fuse lines into paragraphs."""
+    gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
+    bin_ = cv2.adaptiveThreshold(gray, 255, cv2.ADAPTIVE_THRESH_GAUSSIAN_C,
+                                 cv2.THRESH_BINARY_INV, 35, 15)
+    horiz = cv2.dilate(bin_, np.ones((1, smear[0]), np.uint8), iterations=1)
+    block = cv2.dilate(horiz, np.ones((smear[1], 1), np.uint8), iterations=2)
+    block = cv2.morphologyEx(block, cv2.MORPH_CLOSE, np.ones((5, 5), np.uint8))
+
+    out = []
+    cnts, _ = cv2.findContours(block, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
+    for c in cnts:
+        x, y, w, h = cv2.boundingRect(c)
+        if w < 20 or h < 8:
+            continue
+        out.append((x * scale, y * scale, (x + w) * scale, (y + h) * scale))
+    return out
+
+
+def rules(img: np.ndarray, scale: float, min_len_frac: float = 0.15) -> list[Rect]:
+    """Long thin horizontal/vertical strokes: separators and table grids."""
+    gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
+    bin_ = cv2.threshold(gray, 0, 255, cv2.THRESH_BINARY_INV | cv2.THRESH_OTSU)[1]
+    h, w = bin_.shape
+    found = []
+    for kernel, minlen in (
+        (np.ones((1, max(int(w * min_len_frac), 10)), np.uint8), None),
+        (np.ones((max(int(h * min_len_frac), 10), 1), np.uint8), None),
+    ):
+        det = cv2.morphologyEx(bin_, cv2.MORPH_OPEN, kernel, iterations=1)
+        cnts, _ = cv2.findContours(det, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
+        for c in cnts:
+            x, y, cw, ch = cv2.boundingRect(c)
+            found.append((x * scale, y * scale, (x + cw) * scale, (y + ch) * scale))
+    return found
+
+
+def deskew(img: np.ndarray, max_angle: float = 6.0) -> tuple[np.ndarray, float]:
+    """Estimate and correct small rotations. Only relevant for scans."""
+    gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
+    bin_ = cv2.threshold(gray, 0, 255, cv2.THRESH_BINARY_INV | cv2.THRESH_OTSU)[1]
+    coords = np.column_stack(np.where(bin_ > 0))
+    if coords.shape[0] < 100:
+        return img, 0.0
+    angle = cv2.minAreaRect(coords.astype(np.float32))[-1]
+    angle = angle + 90 if angle < -45 else angle
+    if abs(angle) > max_angle:
+        return img, 0.0
+    h, w = img.shape[:2]
+    m = cv2.getRotationMatrix2D((w / 2, h / 2), angle, 1.0)
+    return cv2.warpAffine(img, m, (w, h), flags=cv2.INTER_CUBIC,
+                          borderMode=cv2.BORDER_REPLICATE), float(angle)