verifydoc 0.2.0__tar.gz → 0.3.0__tar.gz

{verifydoc-0.2.0 → verifydoc-0.3.0}/CHANGELOG.md

@@ -1,5 +1,28 @@
 # Changelog
 
+## v0.3.0 — 2026-07-21
+
+First **real-model** results (issue #3).
+
+- **RapidOCR adapter** (PP-OCRv6 via ONNX Runtime) — architecture-independent,
+  the robust real-OCR default; runs where paddle's GPU wheels don't.
+- **PaddleOCR adapter** validated end-to-end (PP-OCRv5 pipeline) on a real GPU.
+- **Real results on CORD + FUNSD** for both extractors, committed under
+  `paper/generated/` with an honest reading (`REAL_MODELS_RESULTS.md`):
+  grounding/learned signals rank errors (AUROC 0.74–0.89) while
+  verbalized/consensus don't; grounding-conditioned gap +0.83/+0.84. The
+  floor field-finder's low recall forces conformal abstention at 2–5% — the
+  documented motivation for the selective framing.
+- **Harness**: `extractor:` dispatch via the adapter registry; CORD/FUNSD
+  image export (`with_images`); schema `x-aliases` for on-page label variants.
+- **Learned combiner** (`verifydoc/confidence/learned.py`) as a harness row.
+- **Grounder ~1000× faster** (exact-match fast path, token-anchored fuzzy
+  scan, token-overlap for paragraph values) — FUNSD grounding hours → seconds.
+- **GPU runbook** (`docs/REAL_MODELS.md`) with the hard-won version pins
+  (`paddlex==3.1.0`, `langchain<0.2`, Blackwell caveat, dots.ocr→vllm).
+- dots.ocr: adapter stub validated to load/generate with `transformers==4.51.3`;
+  full run deferred to its vllm serving path (issue #3).
+
 ## v0.2.0 — 2026-07-21
 
 Real data, faster grounding, learned fusion.

{verifydoc-0.2.0 → verifydoc-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: verifydoc
-Version: 0.2.0
+Version: 0.3.0
 Summary: A trust layer for document -> structured-JSON extraction: calibrated per-field confidence, source grounding, and abstention on top of any extractor.
 Project-URL: Homepage, https://github.com/bhaskargurram-ai/verifydoc
 Project-URL: Repository, https://github.com/bhaskargurram-ai/verifydoc

{verifydoc-0.2.0 → verifydoc-0.3.0}/benchmark/datasets/cord.py

@@ -37,20 +37,52 @@ CORD_SCHEMA_RAW: dict = {
         "sub_total": {
             "type": "object",
             "properties": {
-                "subtotal_price": {"type": "number", "x-numeric-tol": 0.01},
-                "tax_price": {"type": "number", "x-numeric-tol": 0.01},
-                "discount_price": {"type": "number", "x-numeric-tol": 0.01},
-                "service_price": {"type": "number", "x-numeric-tol": 0.01},
+                "subtotal_price": {
+                    "type": "number",
+                    "x-numeric-tol": 0.01,
+                    "x-aliases": ["subtotal", "sub total", "sub-total"],
+                },
+                "tax_price": {
+                    "type": "number",
+                    "x-numeric-tol": 0.01,
+                    "x-aliases": ["tax", "ppn", "pb1"],
+                },
+                "discount_price": {
+                    "type": "number",
+                    "x-numeric-tol": 0.01,
+                    "x-aliases": ["discount", "diskon"],
+                },
+                "service_price": {
+                    "type": "number",
+                    "x-numeric-tol": 0.01,
+                    "x-aliases": ["service", "service charge"],
+                },
             },
         },
         "total": {
             "type": "object",
             "properties": {
-                "total_price": {"type": "number", "x-numeric-tol": 0.01},
-                "cashprice": {"type": "number", "x-numeric-tol": 0.01},
-                "changeprice": {"type": "number", "x-numeric-tol": 0.01},
-                "creditcardprice": {"type": "number", "x-numeric-tol": 0.01},
-                "menuqty_cnt": {"type": "string"},
+                "total_price": {
+                    "type": "number",
+                    "x-numeric-tol": 0.01,
+                    "x-aliases": ["total", "grand total", "amount due"],
+                },
+                "cashprice": {
+                    "type": "number",
+                    "x-numeric-tol": 0.01,
+                    "x-aliases": ["cash", "tunai"],
+                },
+                "changeprice": {
+                    "type": "number",
+                    "x-numeric-tol": 0.01,
+                    "x-aliases": ["change", "kembali", "kembalian"],
+                },
+                "creditcardprice": {
+                    "type": "number",
+                    "x-numeric-tol": 0.01,
+                    "x-aliases": ["credit card", "debit", "card"],
+                },
+                "menuqty_cnt": {"type": "string", "x-aliases": ["qty", "items", "item qty"]},
             },
         },
     },
@@ -110,11 +142,18 @@ def load(
     split: str = "validation",
     limit: int | None = 100,
     cache_dir: str | Path = "data",
+    with_images: bool = False,
 ) -> list[BenchDocument]:  # pragma: no cover - network on first call
-    """Load CORD v2 (streamed; JSON-cached locally after the first download)."""
+    """Load CORD v2 (streamed; JSON-cached locally after the first download).
+
+    ``with_images=True`` additionally exports each receipt PNG (needed by
+    image-reading OCR adapters) and sets ``page.image_path``.
+    """
     schema = Schema.from_json_schema(CORD_SCHEMA_RAW, name="cord-receipt")
     cache = Path(cache_dir) / f"cord_{split}_{limit or 'all'}.json"
-    if cache.exists():
+    image_dir = Path(cache_dir) / "cord_images" / split
+    need_images = with_images and not image_dir.exists()
+    if cache.exists() and not need_images:
         rows = json.loads(cache.read_text(encoding="utf-8"))
     else:
         try:
@@ -123,11 +162,15 @@ def load(
             raise ImportError("CORD loader requires: pip install 'verifydoc[data]'") from exc
         stream = load_dataset("naver-clova-ix/cord-v2", split=split, streaming=True)
         rows = []
+        if with_images:
+            image_dir.mkdir(parents=True, exist_ok=True)
         for i, row in enumerate(stream):
             if limit is not None and i >= limit:
                 break
             gt = json.loads(row["ground_truth"])
             width, height = row["image"].size
+            if with_images:
+                row["image"].save(image_dir / f"{i:05d}.png")
             rows.append(
                 {
                     "gt_parse": gt["gt_parse"],
@@ -144,6 +187,9 @@ def load(
         doc = document_from_valid_lines(
             f"cord-{split}-{i:05d}", row["valid_line"], row["width"], row["height"]
         )
+        image_path = image_dir / f"{i:05d}.png"
+        if with_images and image_path.exists():
+            doc.pages[0].image_path = str(image_path)
         golds = golds_from_gt_parse(row["gt_parse"], schema, doc)
         if golds and doc.pages[0].words:
             out.append(BenchDocument(doc=doc, schema=schema, golds=golds))

{verifydoc-0.2.0 → verifydoc-0.3.0}/benchmark/datasets/funsd.py

@@ -160,5 +160,6 @@ def load(
             width, height = img.size
         item = bench_from_annotation(f"funsd-{split}-{ann_path.stem}", annotation, width, height)
         if item is not None:
+            item.doc.pages[0].image_path = str(img_path)  # for image-reading OCR adapters
             out.append(item)
     return out

verifydoc-0.3.0/configs/cord-paddleocr.yaml

@@ -0,0 +1,15 @@
+# Pinned real-model experiment (issue #3): PaddleOCR on CORD receipts.
+# Needs a GPU + `pip install paddlepaddle-gpu paddleocr`; CORD images are
+# exported automatically (with_images) — see docs/REAL_MODELS.md.
+dataset: cord
+split: validation
+limit: 100
+extractor: paddleocr-vl
+seed: 7
+k: 1   # the OCR pipeline is deterministic: consensus over k>1 is degenerate;
+       # the live signals here are token-prob (rec scores) and grounding.
+calibration_fraction: 0.5
+alphas: [0.02, 0.05]
+n_boot: 300
+# record at run time for the paper table header:
+#   paddleocr==<version>, GPU model, driver version

verifydoc-0.3.0/configs/cord-rapidocr.yaml

@@ -0,0 +1,11 @@
+# Real-model experiment: RapidOCR (PP-OCR models via ONNX) on CORD receipts.
+# Architecture-independent (CPU or GPU) — install: pip install rapidocr onnxruntime
+dataset: cord
+split: validation
+limit: 100
+extractor: rapidocr
+seed: 7
+k: 1   # deterministic OCR: token-prob (rec scores) + grounding are the live signals
+calibration_fraction: 0.5
+alphas: [0.02, 0.05]
+n_boot: 300

verifydoc-0.3.0/configs/funsd-paddleocr.yaml

@@ -0,0 +1,9 @@
+# Real-model experiment: PaddleOCR on FUNSD scanned forms (noisy, low-res).
+dataset: funsd
+split: testing
+extractor: paddleocr-vl
+seed: 7
+k: 1   # deterministic OCR: token-prob + grounding are the live signals
+calibration_fraction: 0.5
+alphas: [0.02, 0.05]
+n_boot: 300

verifydoc-0.3.0/configs/funsd-rapidocr.yaml

@@ -0,0 +1,9 @@
+# Real-model experiment: RapidOCR (PP-OCR models via ONNX) on FUNSD forms.
+dataset: funsd
+split: testing
+extractor: rapidocr
+seed: 7
+k: 1
+calibration_fraction: 0.5
+alphas: [0.02, 0.05]
+n_boot: 300

verifydoc-0.3.0/docs/REAL_MODELS.md

@@ -0,0 +1,77 @@
+# Running real extractors (GPU runbook)
+
+The same harness, with a real extractor instead of the simulated one
+(issue #3). Reproduces `paper/generated/{cord,funsd}-{rapidocr,paddleocr}/`.
+
+## Hard-won environment notes (read first)
+
+These were discovered running on RunPod; they save hours:
+
+- **PaddleOCR needs a GPU architecture paddle supports.** PaddlePaddle's
+  precompiled wheels have **no kernels for Blackwell (sm_120, e.g. RTX PRO
+  6000 / 5090)** — you get `RuntimeError: Unsupported GPU architecture`. Use
+  Ampere (A100 sm_80, A5000/A6000 sm_86) or Ada (RTX 4090 / L4 / L40 sm_89).
+- **Pin `paddlex==3.1.0`** to match `paddleocr==3.1.0`, or the constructor
+  dies with `PaddlePredictorOption.__init__() takes 1 positional argument`.
+- **Pin `langchain<0.2`** (paddlex imports the removed `langchain.docstore`).
+- **RapidOCR is architecture-independent** (ONNX Runtime) — it runs anywhere,
+  CPU or GPU, and is the most robust real-OCR option. Prefer it when the GPU
+  is exotic.
+- **dots.ocr** loads with `transformers==4.51.3` + `qwen_vl_utils`, but raw
+  `transformers.generate` fights its custom model; its intended runtime is
+  **vllm serving** — wire the adapter to a vllm endpoint (issue #3).
+
+## Setup
+
+```bash
+git clone https://github.com/bhaskargurram-ai/verifydoc && cd verifydoc
+python3.11 -m venv .venv && source .venv/bin/activate
+pip install -e ".[dev,data]"
+
+# Option A — RapidOCR (recommended, architecture-independent)
+pip install rapidocr onnxruntime          # or onnxruntime-gpu for speed
+
+# Option B — PaddleOCR (needs a paddle-supported GPU)
+pip install "paddlepaddle-gpu==3.1.0" -i https://www.paddlepaddle.org.cn/packages/stable/cu126/
+pip install "paddleocr==3.1.0" "paddlex==3.1.0" "langchain<0.2" "langchain-community<0.2"
+```
+
+## Smoke test one receipt
+
+```bash
+python - <<'EOF'
+from benchmark.datasets import cord
+from verifydoc.adapters import get_adapter
+
+item = cord.load(split="validation", limit=1, with_images=True)[0]
+adapter = get_adapter("rapidocr")          # or "paddleocr-vl"
+for p in adapter.extract(item.doc, item.schema):
+    print(p.path, p.value, p.meta.get("token_logprobs"))
+EOF
+```
+
+## Full run
+
+```bash
+python scripts/run_benchmark.py --config configs/cord-rapidocr.yaml   --out paper/generated/cord-rapidocr
+python scripts/run_benchmark.py --config configs/funsd-rapidocr.yaml  --out paper/generated/funsd-rapidocr
+python scripts/run_benchmark.py --config configs/cord-paddleocr.yaml  --out paper/generated/cord-paddleocr   # paddle GPU
+python scripts/run_benchmark.py --config configs/funsd-paddleocr.yaml --out paper/generated/funsd-paddleocr
+```
+
+The harness dispatches `extractor:` through the adapter registry
+(`mock` | `text-search` | `rapidocr` | `paddleocr-vl` | `dots-ocr` |
+`docling` | `api-vlm`) and `dataset:` (`synthetic` | `cord` | `funsd`).
+CORD/FUNSD images are exported automatically (`with_images`).
+
+## What to report
+
+Pin model versions in the table header (`rapidocr==x`, `paddleocr==3.1.0`,
+GPU + driver). Results feed the paper via `make results`; see
+`paper/generated/REAL_MODELS_RESULTS.md` for the current numbers and reading.
+
+## Cloud
+
+Any Ampere/Ada single-GPU pod (RTX 4090 ≈ $0.4–0.7/hr) runs everything except
+Blackwell-only cards for paddle. The full sweep is an hours-not-days job.
+```

verifydoc-0.3.0/paper/generated/REAL_MODELS_RESULTS.md

@@ -0,0 +1,60 @@
+# Real-model results (issue #3)
+
+Results from **real OCR extractors** on real documents, produced by the same
+harness as the synthetic slice. Runs are pinned in `configs/*-<model>.yaml`
+and reproduced with `scripts/run_benchmark.py`.
+
+## Summary — the thesis holds on real data
+
+Across both real extractors and both datasets, the ranking of confidence
+signals is consistent and matches the paper's claim:
+
+| Signal | ranks errors? | CORD AUROC (RapidOCR / PaddleOCR) |
+|---|---|---|
+| **learned combiner** | best | **0.89 / 0.84** |
+| **grounding** | strong | 0.82 / 0.74 |
+| token-prob (rec. score) | moderate | 0.69 / 0.68 |
+| verbalized / consensus | uninformative (k=1) | 0.50 / 0.50 |
+
+**Grounding-conditioned correctness** (the core hypothesis) is large on real
+data: grounded fields are 84–85% correct vs ~1% for ungrounded
+(gap +0.83 RapidOCR, +0.84 PaddleOCR; box acc @IoU 0.5 = 0.75 / 0.78).
+
+**Honest limitation:** with the `text-search` field-finder over raw OCR (the
+floor baseline, no structured-output model), field recall is low (~0.19) and
+the base error rate is high — so conformal abstention at a 2–5% budget forces
+full review (Coverage@2% ≈ 0). This is the motivation for the whole selective
+framing: you report *selective risk*, not headline accuracy, and you need a
+stronger extractor to raise the STP rate. The synthetic slice (strong
+simulated extractor) shows the other end (Coverage@2% ≈ 1.0).
+
+## RapidOCR (PP-OCRv6 via ONNX Runtime)
+
+- Config: `configs/cord-rapidocr.yaml`, `configs/funsd-rapidocr.yaml`.
+- CORD: F1 0.28, grounding AUROC 0.82, learned 0.89, grounding gap +0.83.
+- Tables: `paper/generated/{cord,funsd}-rapidocr/`.
+
+## PaddleOCR (PP-OCRv5 pipeline, RTX 4090)
+
+- Config: `configs/cord-paddleocr.yaml`, `configs/funsd-paddleocr.yaml`.
+- The `paddleocr-vl` adapter runs the PP-OCR **detection+recognition
+  pipeline** (not the separate PaddleOCR-VL VLM); recognition scores feed the
+  token-prob signal.
+- CORD: F1 0.28, grounding AUROC 0.74, learned 0.84, coverage@2% 0.026,
+  grounding gap +0.84, box acc@0.5 0.78.
+- Tables: `paper/generated/{cord,funsd}-paddleocr/`.
+
+## dots.ocr (VLM) — deferred to vllm serving
+
+Status: **partially validated, not run at scale.** The model loads and
+generates via `transformers==4.51.3` + `qwen_vl_utils.process_vision_info`,
+but raw `transformers.generate` fights the model's custom code across
+versions (`mm_token_type_ids` handling), and the intended runtime per the
+model authors is **vllm serving**. A `DotsOCRAdapter` stub exists
+(`verifydoc/adapters/dots_ocr.py`); wiring it to a vllm endpoint is the clean
+follow-up (tracked in issue #3). It is a *third data point* (VLM vs. the two
+OCR pipelines), not load-bearing for the thesis, which the two OCR extractors
+above already establish on real data.
+
+*Every number here is regenerated by the pinned configs; re-verify model
+versions before the paper.*

verifydoc-0.3.0/paper/generated/cord-paddleocr/calibration.md

@@ -0,0 +1,34 @@
+# Calibration: signal x calibrator (test split)
+
+| signal | calibrator | ece | adaptive_ece | brier | nll |
+|---|---|---|---|---|---|
+| token_prob | raw | 0.2641 | 0.2508 | 0.2469 | 1.0664 |
+| token_prob | temperature | 0.1199 | 0.1648 | 0.2002 | 0.5915 |
+| token_prob | platt | 0.1629 | 0.1577 | 0.1959 | 0.5934 |
+| token_prob | isotonic | 0.1344 | 0.1459 | 0.1943 | 0.9008 |
+| token_prob | histogram | 0.0371 | 0.0933 | 0.1891 | 0.7167 |
+| verbalized | raw | 0.1753 | 0.1948 | 0.2500 | 0.6931 |
+| verbalized | temperature | 0.1753 | 0.1948 | 0.2500 | 0.6931 |
+| verbalized | platt | 0.0225 | 0.1411 | 0.2198 | 0.6315 |
+| verbalized | isotonic | 0.0225 | 0.1411 | 0.2198 | 0.6315 |
+| verbalized | histogram | 0.0225 | 0.1411 | 0.2198 | 0.6315 |
+| grounding | raw | 0.2607 | 0.2607 | 0.2447 | 4.7090 |
+| grounding | temperature | 0.0801 | 0.1365 | 0.1690 | 0.5177 |
+| grounding | platt | 0.0194 | 0.0752 | 0.1376 | 0.4406 |
+| grounding | isotonic | 0.0238 | 0.0781 | 0.1374 | 0.4405 |
+| grounding | histogram | 0.0299 | 0.0874 | 0.1429 | 0.6120 |
+| consensus | raw | 0.3247 | 0.3247 | 0.3247 | 8.9711 |
+| consensus | temperature | 0.0225 | 0.1411 | 0.2198 | 0.6315 |
+| consensus | platt | 0.0225 | 0.1411 | 0.2198 | 0.6315 |
+| consensus | isotonic | 0.0225 | 0.1411 | 0.2198 | 0.6315 |
+| consensus | histogram | 0.0225 | 0.1411 | 0.2198 | 0.6315 |
+| combined | raw | 0.2991 | 0.2994 | 0.2847 | 2.6709 |
+| combined | temperature | 0.2396 | 0.2244 | 0.2113 | 0.6508 |
+| combined | platt | 0.1842 | 0.1667 | 0.1810 | 0.8118 |
+| combined | isotonic | 0.1101 | 0.1140 | 0.1544 | 0.4963 |
+| combined | histogram | 0.0353 | 0.1508 | 0.1943 | 0.7352 |
+| learned | raw | 0.1902 | 0.1884 | 0.1639 | 0.5090 |
+| learned | temperature | 0.1429 | 0.1464 | 0.1558 | 0.4855 |
+| learned | platt | 0.1069 | 0.0850 | 0.1306 | 0.4522 |
+| learned | isotonic | 0.0567 | 0.0681 | 0.1148 | 0.3896 |
+| learned | histogram | 0.0745 | 0.0985 | 0.1346 | 0.7547 |

verifydoc-0.3.0/paper/generated/cord-paddleocr/conformal.md

@@ -0,0 +1,16 @@
+# Conformal abstention: guarantee vs forced review
+
+| signal | alpha | threshold | test_coverage | achieved_risk | guarantee_held |
+|---|---|---|---|---|---|
+| token_prob | 0.0200 | inf | 0.0000 | 0.0000 | True |
+| token_prob | 0.0500 | inf | 0.0000 | 0.0000 | True |
+| verbalized | 0.0200 | inf | 0.0000 | 0.0000 | True |
+| verbalized | 0.0500 | inf | 0.0000 | 0.0000 | True |
+| grounding | 0.0200 | inf | 0.0000 | 0.0000 | True |
+| grounding | 0.0500 | inf | 0.0000 | 0.0000 | True |
+| consensus | 0.0200 | inf | 0.0000 | 0.0000 | True |
+| consensus | 0.0500 | inf | 0.0000 | 0.0000 | True |
+| combined | 0.0200 | inf | 0.0000 | 0.0000 | True |
+| combined | 0.0500 | inf | 0.0000 | 0.0000 | True |
+| learned | 0.0200 | inf | 0.0000 | 0.0000 | True |
+| learned | 0.0500 | inf | 0.0000 | 0.0000 | True |

verifydoc-0.3.0/paper/generated/cord-paddleocr/extraction.md

@@ -0,0 +1,5 @@
+# Extraction quality (single-run baseline)
+
+| precision | recall | f1 | exact_match | omission_rate | hallucination_rate |
+|---|---|---|---|---|---|
+| 0.6379 | 0.1893 | 0.2787 | 0.1929 | 0.7216 | 0.0048 |

verifydoc-0.3.0/paper/generated/cord-paddleocr/grounding.md

@@ -0,0 +1,5 @@
+# Grounding quality (consensus predictions)
+
+| box_acc@0.5 | box_acc@0.7 | mean_iou | acc_grounded | acc_ungrounded | grounding_gap |
+|---|---|---|---|---|---|
+| 0.7804 | 0.7804 | 0.8278 | 0.8528 | 0.0154 | 0.8374 |

verifydoc-0.3.0/paper/generated/cord-paddleocr/selective.md

@@ -0,0 +1,10 @@
+# Selective prediction by signal (raw confidence, test split)
+
+| signal | e_aurc | auroc | coverage@2% | coverage@5% | e_aurc_ci |
+|---|---|---|---|---|---|
+| token_prob | 0.2160 | 0.6835 | 0.0000 | 0.0000 | [0.1188, 0.2996] |
+| verbalized | 0.3159 | 0.5000 | 0.0000 | 0.0000 | [0.1855, 0.3562] |
+| grounding | 0.1931 | 0.7449 | 0.0000 | 0.0000 | [0.0805, 0.2193] |
+| consensus | 0.3159 | 0.5000 | 0.0000 | 0.0000 | [0.1855, 0.3562] |
+| combined | 0.2731 | 0.6693 | 0.0000 | 0.0000 | [0.1698, 0.3466] |
+| learned | 0.0969 | 0.8405 | 0.0260 | 0.0260 | [0.0369, 0.1664] |

verifydoc-0.3.0/paper/generated/cord-rapidocr/calibration.md

@@ -0,0 +1,34 @@
+# Calibration: signal x calibrator (test split)
+
+| signal | calibrator | ece | adaptive_ece | brier | nll |
+|---|---|---|---|---|---|
+| token_prob | raw | 0.3068 | 0.3043 | 0.2877 | 1.8498 |
+| token_prob | temperature | 0.0963 | 0.1537 | 0.2082 | 0.6063 |
+| token_prob | platt | 0.0980 | 0.1249 | 0.1963 | 0.5813 |
+| token_prob | isotonic | 0.1067 | 0.1488 | 0.2006 | 1.2121 |
+| token_prob | histogram | 0.0687 | 0.1328 | 0.2006 | 0.8964 |
+| verbalized | raw | 0.1415 | 0.1981 | 0.2500 | 0.6931 |
+| verbalized | temperature | 0.1415 | 0.1981 | 0.2500 | 0.6931 |
+| verbalized | platt | 0.0311 | 0.1195 | 0.2309 | 0.6546 |
+| verbalized | isotonic | 0.0311 | 0.1195 | 0.2309 | 0.6546 |
+| verbalized | histogram | 0.0311 | 0.1195 | 0.2309 | 0.6546 |
+| grounding | raw | 0.2638 | 0.2638 | 0.2423 | 3.7864 |
+| grounding | temperature | 0.1265 | 0.1707 | 0.1596 | 0.4947 |
+| grounding | platt | 0.0425 | 0.1013 | 0.1136 | 0.3811 |
+| grounding | isotonic | 0.0267 | 0.0908 | 0.1118 | 0.3739 |
+| grounding | histogram | 0.0414 | 0.1034 | 0.1175 | 0.5380 |
+| consensus | raw | 0.3585 | 0.3585 | 0.3585 | 9.9055 |
+| consensus | temperature | 0.0311 | 0.1195 | 0.2309 | 0.6546 |
+| consensus | platt | 0.0311 | 0.1195 | 0.2309 | 0.6546 |
+| consensus | isotonic | 0.0311 | 0.1195 | 0.2309 | 0.6546 |
+| consensus | histogram | 0.0311 | 0.1195 | 0.2309 | 0.6546 |
+| combined | raw | 0.3252 | 0.3252 | 0.3029 | 2.8066 |
+| combined | temperature | 0.2896 | 0.2621 | 0.1982 | 0.5909 |
+| combined | platt | 0.2292 | 0.2108 | 0.1630 | 0.5564 |
+| combined | isotonic | 0.0306 | 0.0972 | 0.1127 | 0.3756 |
+| combined | histogram | 0.0338 | 0.1407 | 0.1746 | 0.6830 |
+| learned | raw | 0.2562 | 0.2204 | 0.1554 | 0.4886 |
+| learned | temperature | 0.1776 | 0.1630 | 0.1419 | 0.4470 |
+| learned | platt | 0.1659 | 0.1440 | 0.1164 | 0.4084 |
+| learned | isotonic | 0.0761 | 0.0919 | 0.0822 | 0.6131 |
+| learned | histogram | 0.0861 | 0.0889 | 0.0833 | 0.6082 |

verifydoc-0.3.0/paper/generated/cord-rapidocr/conformal.md

@@ -0,0 +1,16 @@
+# Conformal abstention: guarantee vs forced review
+
+| signal | alpha | threshold | test_coverage | achieved_risk | guarantee_held |
+|---|---|---|---|---|---|
+| token_prob | 0.0200 | inf | 0.0000 | 0.0000 | True |
+| token_prob | 0.0500 | inf | 0.0000 | 0.0000 | True |
+| verbalized | 0.0200 | inf | 0.0000 | 0.0000 | True |
+| verbalized | 0.0500 | inf | 0.0000 | 0.0000 | True |
+| grounding | 0.0200 | inf | 0.0000 | 0.0000 | True |
+| grounding | 0.0500 | inf | 0.0000 | 0.0000 | True |
+| consensus | 0.0200 | inf | 0.0000 | 0.0000 | True |
+| consensus | 0.0500 | inf | 0.0000 | 0.0000 | True |
+| combined | 0.0200 | inf | 0.0000 | 0.0000 | True |
+| combined | 0.0500 | inf | 0.0000 | 0.0000 | True |
+| learned | 0.0200 | inf | 0.0000 | 0.0000 | True |
+| learned | 0.0500 | inf | 0.0000 | 0.0000 | True |

verifydoc-0.3.0/paper/generated/cord-rapidocr/extraction.md

@@ -0,0 +1,5 @@
+# Extraction quality (single-run baseline)
+
+| precision | recall | f1 | exact_match | omission_rate | hallucination_rate |
+|---|---|---|---|---|---|
+| 0.6175 | 0.1865 | 0.2756 | 0.1869 | 0.7041 | 0.0048 |

verifydoc-0.3.0/paper/generated/cord-rapidocr/grounding.md

@@ -0,0 +1,5 @@
+# Grounding quality (consensus predictions)
+
+| box_acc@0.5 | box_acc@0.7 | mean_iou | acc_grounded | acc_ungrounded | grounding_gap |
+|---|---|---|---|---|---|
+| 0.7460 | 0.7460 | 0.8061 | 0.8405 | 0.0127 | 0.8279 |

verifydoc-0.3.0/paper/generated/cord-rapidocr/selective.md

@@ -0,0 +1,10 @@
+# Selective prediction by signal (raw confidence, test split)
+
+| signal | e_aurc | auroc | coverage@2% | coverage@5% | e_aurc_ci |
+|---|---|---|---|---|---|
+| token_prob | 0.1851 | 0.6872 | 0.0000 | 0.0000 | [0.1110, 0.2675] |
+| verbalized | 0.2826 | 0.5000 | 0.0000 | 0.0000 | [0.2037, 0.3743] |
+| grounding | 0.1228 | 0.8181 | 0.0000 | 0.0000 | [0.0579, 0.1817] |
+| consensus | 0.2826 | 0.5000 | 0.0000 | 0.0000 | [0.2037, 0.3743] |
+| combined | 0.2491 | 0.7367 | 0.0000 | 0.0000 | [0.1543, 0.3299] |
+| learned | 0.0797 | 0.8870 | 0.0000 | 0.0000 | [0.0289, 0.1519] |

verifydoc-0.3.0/paper/generated/funsd-paddleocr/calibration.md

@@ -0,0 +1,34 @@
+# Calibration: signal x calibrator (test split)
+
+| signal | calibrator | ece | adaptive_ece | brier | nll |
+|---|---|---|---|---|---|
+| token_prob | raw | 0.3567 | 0.3884 | 0.4337 | 1.7918 |
+| token_prob | temperature | 0.0507 | 0.1525 | 0.2526 | 0.6983 |
+| token_prob | platt | 0.0936 | 0.1533 | 0.2468 | 0.6870 |
+| token_prob | isotonic | 0.0542 | 0.1321 | 0.2511 | 0.6953 |
+| token_prob | histogram | 0.0535 | 0.1139 | 0.2421 | 0.6779 |
+| verbalized | raw | 0.0397 | 0.1349 | 0.2500 | 0.6931 |
+| verbalized | temperature | 0.0397 | 0.1349 | 0.2500 | 0.6931 |
+| verbalized | platt | 0.0553 | 0.1400 | 0.2515 | 0.6961 |
+| verbalized | isotonic | 0.0553 | 0.1400 | 0.2515 | 0.6961 |
+| verbalized | histogram | 0.0553 | 0.1400 | 0.2515 | 0.6961 |
+| grounding | raw | 0.3242 | 0.3187 | 0.3254 | 4.8572 |
+| grounding | temperature | 0.1053 | 0.2078 | 0.2415 | 0.6762 |
+| grounding | platt | 0.1270 | 0.1991 | 0.2367 | 0.6660 |
+| grounding | isotonic | 0.0890 | 0.1651 | 0.2258 | 0.6445 |
+| grounding | histogram | 0.0973 | 0.1664 | 0.2321 | 0.6582 |
+| consensus | raw | 0.5397 | 0.5397 | 0.5397 | 14.9120 |
+| consensus | temperature | 0.1130 | 0.1634 | 0.2612 | 0.7157 |
+| consensus | platt | 0.0553 | 0.1400 | 0.2515 | 0.6961 |
+| consensus | isotonic | 0.0553 | 0.1400 | 0.2515 | 0.6961 |
+| consensus | histogram | 0.0553 | 0.1400 | 0.2515 | 0.6961 |
+| combined | raw | 0.4449 | 0.4449 | 0.4245 | 3.3265 |
+| combined | temperature | 0.1461 | 0.2009 | 0.2500 | 0.6936 |
+| combined | platt | 0.0742 | 0.1862 | 0.2378 | 0.6701 |
+| combined | isotonic | 0.0773 | 0.1414 | 0.2302 | 0.6548 |
+| combined | histogram | 0.0665 | 0.1188 | 0.2297 | 0.6521 |
+| learned | raw | 0.0639 | 0.1586 | 0.2327 | 0.6574 |
+| learned | temperature | 0.0941 | 0.1580 | 0.2338 | 0.6595 |
+| learned | platt | 0.0969 | 0.1579 | 0.2338 | 0.6594 |
+| learned | isotonic | 0.1163 | 0.1404 | 0.2397 | 0.6746 |
+| learned | histogram | 0.1248 | 0.1409 | 0.2494 | 0.9558 |

verifydoc-0.3.0/paper/generated/funsd-paddleocr/conformal.md

@@ -0,0 +1,16 @@
+# Conformal abstention: guarantee vs forced review
+
+| signal | alpha | threshold | test_coverage | achieved_risk | guarantee_held |
+|---|---|---|---|---|---|
+| token_prob | 0.0200 | inf | 0.0000 | 0.0000 | True |
+| token_prob | 0.0500 | inf | 0.0000 | 0.0000 | True |
+| verbalized | 0.0200 | inf | 0.0000 | 0.0000 | True |
+| verbalized | 0.0500 | inf | 0.0000 | 0.0000 | True |
+| grounding | 0.0200 | inf | 0.0000 | 0.0000 | True |
+| grounding | 0.0500 | inf | 0.0000 | 0.0000 | True |
+| consensus | 0.0200 | inf | 0.0000 | 0.0000 | True |
+| consensus | 0.0500 | inf | 0.0000 | 0.0000 | True |
+| combined | 0.0200 | inf | 0.0000 | 0.0000 | True |
+| combined | 0.0500 | inf | 0.0000 | 0.0000 | True |
+| learned | 0.0200 | inf | 0.0000 | 0.0000 | True |
+| learned | 0.0500 | inf | 0.0000 | 0.0000 | True |

verifydoc-0.3.0/paper/generated/funsd-paddleocr/extraction.md

@@ -0,0 +1,5 @@
+# Extraction quality (single-run baseline)
+
+| precision | recall | f1 | exact_match | omission_rate | hallucination_rate |
+|---|---|---|---|---|---|
+| 0.4692 | 0.2934 | 0.3480 | 0.1282 | 0.4293 | 0.0000 |

verifydoc-0.3.0/paper/generated/funsd-paddleocr/grounding.md

@@ -0,0 +1,5 @@
+# Grounding quality (consensus predictions)
+
+| box_acc@0.5 | box_acc@0.7 | mean_iou | acc_grounded | acc_ungrounded | grounding_gap |
+|---|---|---|---|---|---|
+| 0.4006 | 0.3691 | 0.4741 | 0.9843 | 0.1474 | 0.8369 |

verifydoc-0.3.0/paper/generated/funsd-paddleocr/selective.md

@@ -0,0 +1,10 @@
+# Selective prediction by signal (raw confidence, test split)
+
+| signal | e_aurc | auroc | coverage@2% | coverage@5% | e_aurc_ci |
+|---|---|---|---|---|---|
+| token_prob | 0.3802 | 0.4278 | 0.0000 | 0.0000 | [0.3116, 0.4448] |
+| verbalized | 0.3382 | 0.5000 | 0.0000 | 0.0000 | [0.2907, 0.4224] |
+| grounding | 0.2169 | 0.6805 | 0.0000 | 0.0000 | [0.1879, 0.3290] |
+| consensus | 0.3382 | 0.5000 | 0.0000 | 0.0000 | [0.2907, 0.4224] |
+| combined | 0.2234 | 0.6900 | 0.0000 | 0.0000 | [0.1804, 0.3074] |
+| learned | 0.2306 | 0.6640 | 0.0000 | 0.0000 | [0.1858, 0.3105] |

verifydoc-0.3.0/paper/generated/funsd-rapidocr/calibration.md

@@ -0,0 +1,34 @@
+# Calibration: signal x calibrator (test split)
+
+| signal | calibrator | ece | adaptive_ece | brier | nll |
+|---|---|---|---|---|---|
+| token_prob | raw | 0.3033 | 0.3343 | 0.3992 | 2.0715 |
+| token_prob | temperature | 0.0273 | 0.1592 | 0.2516 | 0.6964 |
+| token_prob | platt | 0.0533 | 0.1493 | 0.2517 | 0.6967 |
+| token_prob | isotonic | 0.0324 | 0.1438 | 0.2463 | 0.6855 |
+| token_prob | histogram | 0.0534 | 0.1583 | 0.2518 | 1.2098 |
+| verbalized | raw | 0.0128 | 0.1410 | 0.2500 | 0.6931 |
+| verbalized | temperature | 0.0128 | 0.1410 | 0.2500 | 0.6931 |
+| verbalized | platt | 0.0273 | 0.1381 | 0.2506 | 0.6943 |
+| verbalized | isotonic | 0.0273 | 0.1381 | 0.2506 | 0.6943 |
+| verbalized | histogram | 0.0273 | 0.1381 | 0.2506 | 0.6943 |
+| grounding | raw | 0.3358 | 0.3486 | 0.3474 | 5.1829 |
+| grounding | temperature | 0.0678 | 0.1763 | 0.2413 | 0.6756 |
+| grounding | platt | 0.0651 | 0.1634 | 0.2365 | 0.6657 |
+| grounding | isotonic | 0.0880 | 0.1705 | 0.2410 | 0.6765 |
+| grounding | histogram | 0.0811 | 0.1487 | 0.2337 | 0.6575 |
+| consensus | raw | 0.5128 | 0.5128 | 0.5128 | 14.1698 |
+| consensus | temperature | 0.0861 | 0.1449 | 0.2572 | 0.7078 |
+| consensus | platt | 0.0273 | 0.1381 | 0.2506 | 0.6943 |
+| consensus | isotonic | 0.0273 | 0.1381 | 0.2506 | 0.6943 |
+| consensus | histogram | 0.0273 | 0.1381 | 0.2506 | 0.6943 |
+| combined | raw | 0.4375 | 0.4375 | 0.4268 | 4.2464 |
+| combined | temperature | 0.1284 | 0.1852 | 0.2471 | 0.6873 |
+| combined | platt | 0.0452 | 0.1585 | 0.2393 | 0.6718 |
+| combined | isotonic | 0.0956 | 0.1432 | 0.2468 | 0.6887 |
+| combined | histogram | 0.0714 | 0.1240 | 0.2380 | 0.6700 |
+| learned | raw | 0.0408 | 0.0998 | 0.2374 | 0.6676 |
+| learned | temperature | 0.0766 | 0.1001 | 0.2384 | 0.6697 |
+| learned | platt | 0.0755 | 0.0993 | 0.2383 | 0.6695 |
+| learned | isotonic | 0.0768 | 0.1010 | 0.2393 | 0.6719 |
+| learned | histogram | 0.0710 | 0.1285 | 0.2425 | 0.6802 |

verifydoc-0.3.0/paper/generated/funsd-rapidocr/conformal.md

@@ -0,0 +1,16 @@
+# Conformal abstention: guarantee vs forced review
+
+| signal | alpha | threshold | test_coverage | achieved_risk | guarantee_held |
+|---|---|---|---|---|---|
+| token_prob | 0.0200 | inf | 0.0000 | 0.0000 | True |
+| token_prob | 0.0500 | inf | 0.0000 | 0.0000 | True |
+| verbalized | 0.0200 | inf | 0.0000 | 0.0000 | True |
+| verbalized | 0.0500 | inf | 0.0000 | 0.0000 | True |
+| grounding | 0.0200 | inf | 0.0000 | 0.0000 | True |
+| grounding | 0.0500 | inf | 0.0000 | 0.0000 | True |
+| consensus | 0.0200 | inf | 0.0000 | 0.0000 | True |
+| consensus | 0.0500 | inf | 0.0000 | 0.0000 | True |
+| combined | 0.0200 | inf | 0.0000 | 0.0000 | True |
+| combined | 0.0500 | inf | 0.0000 | 0.0000 | True |
+| learned | 0.0200 | inf | 0.0000 | 0.0000 | True |
+| learned | 0.0500 | inf | 0.0000 | 0.0000 | True |

verifydoc-0.3.0/paper/generated/funsd-rapidocr/extraction.md

@@ -0,0 +1,5 @@
+# Extraction quality (single-run baseline)
+
+| precision | recall | f1 | exact_match | omission_rate | hallucination_rate |
+|---|---|---|---|---|---|
+| 0.4683 | 0.3110 | 0.3627 | 0.1617 | 0.4054 | 0.0000 |

verifydoc-0.3.0/paper/generated/funsd-rapidocr/grounding.md

@@ -0,0 +1,5 @@
+# Grounding quality (consensus predictions)
+
+| box_acc@0.5 | box_acc@0.7 | mean_iou | acc_grounded | acc_ungrounded | grounding_gap |
+|---|---|---|---|---|---|
+| 0.4114 | 0.3604 | 0.4596 | 0.9927 | 0.1531 | 0.8396 |