verifydoc 0.1.0__tar.gz → 0.2.0__tar.gz

{verifydoc-0.1.0 → verifydoc-0.2.0}/CHANGELOG.md

@@ -1,5 +1,26 @@
 # Changelog
 
+## v0.2.0 — 2026-07-21
+
+Real data, faster grounding, learned fusion.
+
+- **CORD v2 slice**: real receipts with real text layers (word quads) and
+  located gold boxes; Indonesian thousands-comma prices score numerically.
+- **FUNSD slice**: 199 scanned forms; question→answer links become gold
+  fields with exact annotated answer boxes.
+- **Harness**: `dataset:` (synthetic | cord | funsd) and `extractor:` (any
+  adapter registry name) dispatch; `make results` regenerates all slices.
+- **Learned combiner**: logistic fusion over signal values + missing
+  indicators, fit on the calibration split; reported as a `learned` row in
+  every table (ablation vs the transparent weighted mean).
+- **Grounder performance (~1000×)**: exact-match fast path, token-anchored
+  fuzzy scan with length pruning, token-overlap scoring for paragraph-length
+  values. FUNSD run: hours → 8 seconds.
+- **Fix**: text-search label matching uses word boundaries ("total" no longer
+  fires inside "Subtotal") — caught by the harness itself.
+- Demo GIF rendered from a real pipeline run; GPU runbook
+  (docs/REAL_MODELS.md) + pinned real-model config; paper skeleton.
+
 ## v0.1.0 — 2026-07-21
 
 First public release.

{verifydoc-0.1.0 → verifydoc-0.2.0}/Makefile

@@ -23,8 +23,11 @@ cov:
 	pytest --cov=verifydoc --cov-report=term-missing
 
 # Regenerate every table/figure in paper/ from pinned configs + seeds.
+# (cord/funsd download their source data once into data/, then cache)
 results:
 	$(PY) scripts/run_benchmark.py --config configs/demo.yaml --out paper/generated
+	$(PY) scripts/run_benchmark.py --config configs/cord.yaml --out paper/generated/cord
+	$(PY) scripts/run_benchmark.py --config configs/funsd.yaml --out paper/generated/funsd
 
 demo:
 	streamlit run ui/streamlit_app.py

{verifydoc-0.1.0 → verifydoc-0.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: verifydoc
-Version: 0.1.0
+Version: 0.2.0
 Summary: A trust layer for document -> structured-JSON extraction: calibrated per-field confidence, source grounding, and abstention on top of any extractor.
 Project-URL: Homepage, https://github.com/bhaskargurram-ai/verifydoc
 Project-URL: Repository, https://github.com/bhaskargurram-ai/verifydoc
@@ -253,6 +253,10 @@ Description-Content-Type: text/markdown
 [![Python 3.11+](https://img.shields.io/badge/python-3.11+-blue.svg)](pyproject.toml)
 [![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
 
+![VerifyDoc demo: a silently-wrong total is caught by grounding and routed to review](docs/demo.gif)
+
+*Above: a real pipeline run (`scripts/make_demo_gif.py`). The extractor returned `$1,432.50`; the page says `$1,234.50`. Grounding support drops to 0.78, the field misses the accept threshold, and the reviewer is pointed at the exact source region. The other three fields are auto-accepted.*
+
 ## The problem
 
 Modern document parsers read pages at 96%+ benchmark accuracy — and still emit **fluent, plausible, silently-wrong values** (`$42.50` → `$45.20`) with **no reliable per-field signal telling you which values to trust**. Commercial APIs (Box, Azure, Textract) sell field-level confidence as a closed feature. No popular open-source parser leads with it. ([full USP audit](docs/USP.md))

{verifydoc-0.1.0 → verifydoc-0.2.0}/README.md

@@ -7,6 +7,10 @@
 [![Python 3.11+](https://img.shields.io/badge/python-3.11+-blue.svg)](pyproject.toml)
 [![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
 
+![VerifyDoc demo: a silently-wrong total is caught by grounding and routed to review](docs/demo.gif)
+
+*Above: a real pipeline run (`scripts/make_demo_gif.py`). The extractor returned `$1,432.50`; the page says `$1,234.50`. Grounding support drops to 0.78, the field misses the accept threshold, and the reviewer is pointed at the exact source region. The other three fields are auto-accepted.*
+
 ## The problem
 
 Modern document parsers read pages at 96%+ benchmark accuracy — and still emit **fluent, plausible, silently-wrong values** (`$42.50` → `$45.20`) with **no reliable per-field signal telling you which values to trust**. Commercial APIs (Box, Azure, Textract) sell field-level confidence as a closed feature. No popular open-source parser leads with it. ([full USP audit](docs/USP.md))

{verifydoc-0.1.0 → verifydoc-0.2.0}/benchmark/card.md

@@ -10,8 +10,9 @@ extraction quality, calibration, selective risk, and grounding
 | Slice | Source | License | Status | Gold boxes |
 |---|---|---|---|---|
 | `synthetic` | generated in-repo (`benchmark/datasets/synthetic.py`) | Apache-2.0 | shipped, CI-run | yes (from layout) |
-| `cord` | naver-clova-ix/cord-v2 (HF hub) | CC-BY-4.0 (verify at release) | loader shipped | value-line boxes planned |
-| `funsd` / `sroie` / `docile` / `xfund` | official downloads | varies — some research-only | planned | planned |
+| `cord` | naver-clova-ix/cord-v2 (HF hub) | CC-BY-4.0 (verify at release) | shipped — real text layers, harness-run | yes (located word quads) |
+| `funsd` | guillaumejaume.github.io/FUNSD | research use | shipped — real forms, harness-run | yes (annotated answer boxes) |
+| `sroie` / `docile` / `xfund` | official downloads | varies — some research-only | planned | planned |
 
 ## Redistribution policy (PROJECT.md §5.H)
 

verifydoc-0.2.0/benchmark/datasets/cord.py

@@ -0,0 +1,150 @@
+"""CORD v2 loader (receipts, CC-BY-4.0) -> VerifyDocBench items with REAL
+text layers and gold boxes.
+
+Pages are built from CORD's ``valid_line`` word quads (normalized to image
+size), gold values from ``gt_parse`` (flattened to leaf paths with our path
+convention, so arrays like ``menu[0].price`` score natively), and gold boxes
+are located on the real page. Downloads stream from the HF hub
+(pip install 'verifydoc[data]'); results are cached as JSON under ``data/``
+so a run downloads once. Never imported by unit tests (golden rule #5) —
+pure helpers below are tested with fixture rows instead.
+"""
+
+from __future__ import annotations
+
+import json
+from pathlib import Path
+from typing import Any
+
+from benchmark.datasets.synthetic import BenchDocument
+from verifydoc.grounding.attach import _locate
+from verifydoc.types import Document, FieldGold, Page, Schema, Word, flatten_json
+
+CORD_SCHEMA_RAW: dict = {
+    "type": "object",
+    "properties": {
+        "menu": {
+            "type": "array",
+            "items": {
+                "type": "object",
+                "properties": {
+                    "nm": {"type": "string", "x-scoring": "semantic"},
+                    "cnt": {"type": "string"},
+                    "price": {"type": "number", "x-numeric-tol": 0.01},
+                },
+            },
+        },
+        "sub_total": {
+            "type": "object",
+            "properties": {
+                "subtotal_price": {"type": "number", "x-numeric-tol": 0.01},
+                "tax_price": {"type": "number", "x-numeric-tol": 0.01},
+                "discount_price": {"type": "number", "x-numeric-tol": 0.01},
+                "service_price": {"type": "number", "x-numeric-tol": 0.01},
+            },
+        },
+        "total": {
+            "type": "object",
+            "properties": {
+                "total_price": {"type": "number", "x-numeric-tol": 0.01},
+                "cashprice": {"type": "number", "x-numeric-tol": 0.01},
+                "changeprice": {"type": "number", "x-numeric-tol": 0.01},
+                "creditcardprice": {"type": "number", "x-numeric-tol": 0.01},
+                "menuqty_cnt": {"type": "string"},
+            },
+        },
+    },
+}
+
+
+def document_from_valid_lines(
+    doc_id: str, valid_lines: list[dict[str, Any]], width: int, height: int
+) -> Document:
+    """Build a real text-layer Document from CORD ``valid_line`` word quads."""
+    words: list[Word] = []
+    lines: list[str] = []
+    for line in valid_lines:
+        line_words = line.get("words") or []
+        texts = []
+        for w in line_words:
+            quad = w["quad"]
+            xs = [quad["x1"], quad["x2"], quad["x3"], quad["x4"]]
+            ys = [quad["y1"], quad["y2"], quad["y3"], quad["y4"]]
+            bbox = (
+                max(0.0, min(xs) / width),
+                max(0.0, min(ys) / height),
+                min(1.0, max(xs) / width),
+                min(1.0, max(ys) / height),
+            )
+            if bbox[2] > bbox[0] and bbox[3] > bbox[1]:
+                words.append(Word(text=w["text"], bbox=bbox))
+                texts.append(w["text"])
+        if texts:
+            lines.append(" ".join(texts))
+    page = Page(
+        page=0, width=float(width), height=float(height), text="\n".join(lines), words=words
+    )
+    return Document(doc_id=doc_id, pages=[page])
+
+
+def golds_from_gt_parse(gt_parse: dict[str, Any], schema: Schema, doc: Document) -> list[FieldGold]:
+    """Flatten gt_parse into scored gold fields; locate gold boxes on the page."""
+    golds = []
+    for path, value in flatten_json(gt_parse).items():
+        if value is None or str(value).strip() == "":
+            continue
+        leaf = schema.leaf(path)
+        golds.append(
+            FieldGold(
+                path=path,
+                value=value,
+                scoring=leaf.scoring if leaf else "exact",
+                numeric_tol=leaf.numeric_tol if leaf else 1e-6,
+                gold_box=_locate(str(value), doc, min_support=0.7),
+            )
+        )
+    return golds
+
+
+def load(
+    split: str = "validation",
+    limit: int | None = 100,
+    cache_dir: str | Path = "data",
+) -> list[BenchDocument]:  # pragma: no cover - network on first call
+    """Load CORD v2 (streamed; JSON-cached locally after the first download)."""
+    schema = Schema.from_json_schema(CORD_SCHEMA_RAW, name="cord-receipt")
+    cache = Path(cache_dir) / f"cord_{split}_{limit or 'all'}.json"
+    if cache.exists():
+        rows = json.loads(cache.read_text(encoding="utf-8"))
+    else:
+        try:
+            from datasets import load_dataset
+        except ImportError as exc:
+            raise ImportError("CORD loader requires: pip install 'verifydoc[data]'") from exc
+        stream = load_dataset("naver-clova-ix/cord-v2", split=split, streaming=True)
+        rows = []
+        for i, row in enumerate(stream):
+            if limit is not None and i >= limit:
+                break
+            gt = json.loads(row["ground_truth"])
+            width, height = row["image"].size
+            rows.append(
+                {
+                    "gt_parse": gt["gt_parse"],
+                    "valid_line": gt["valid_line"],
+                    "width": width,
+                    "height": height,
+                }
+            )
+        cache.parent.mkdir(parents=True, exist_ok=True)
+        cache.write_text(json.dumps(rows), encoding="utf-8")
+
+    out = []
+    for i, row in enumerate(rows):
+        doc = document_from_valid_lines(
+            f"cord-{split}-{i:05d}", row["valid_line"], row["width"], row["height"]
+        )
+        golds = golds_from_gt_parse(row["gt_parse"], schema, doc)
+        if golds and doc.pages[0].words:
+            out.append(BenchDocument(doc=doc, schema=schema, golds=golds))
+    return out

verifydoc-0.2.0/benchmark/datasets/funsd.py

@@ -0,0 +1,164 @@
+"""FUNSD loader (199 scanned forms, noisy real-world KIE) -> VerifyDocBench.
+
+Question->answer links become gold fields: the path is a slug of the question
+text, the value is the linked answer text, and the gold box is the exact
+union of the answer's word boxes (no locating needed — FUNSD annotates them).
+Download (~16 MB zip) happens once into ``data/``; unit tests use fixture
+annotations only (golden rule #5).
+
+License note (benchmark/card.md): FUNSD permits research use; we ship this
+loader + our added splits, never the images.
+"""
+
+from __future__ import annotations
+
+import json
+import re
+from pathlib import Path
+from typing import Any
+
+from benchmark.datasets.synthetic import BenchDocument
+from verifydoc.eval.extraction import parse_number
+from verifydoc.types import Document, FieldGold, Grounding, Page, Schema, SchemaLeaf, Word
+
+FUNSD_URL = "https://guillaumejaume.github.io/FUNSD/dataset.zip"
+
+_SLUG_RE = re.compile(r"[^a-z0-9]+")
+
+
+def slugify(question: str) -> str:
+    """Question text -> a stable field path ('DATE:' -> 'date')."""
+    slug = _SLUG_RE.sub("_", question.casefold()).strip("_")
+    return slug or "field"
+
+
+def _norm_box(box: list[int], width: int, height: int) -> tuple[float, float, float, float] | None:
+    x0, y0, x1, y1 = box
+    bbox = (
+        max(0.0, x0 / width),
+        max(0.0, y0 / height),
+        min(1.0, x1 / width),
+        min(1.0, y1 / height),
+    )
+    if bbox[2] <= bbox[0] or bbox[3] <= bbox[1]:
+        return None
+    return bbox
+
+
+def bench_from_annotation(
+    doc_id: str, annotation: dict[str, Any], width: int, height: int
+) -> BenchDocument | None:
+    """Build one benchmark item from a FUNSD annotation dict."""
+    entries = {e["id"]: e for e in annotation.get("form", [])}
+
+    words: list[Word] = []
+    lines: list[str] = []
+    for entry in annotation.get("form", []):
+        texts = []
+        for w in entry.get("words", []):
+            bbox = _norm_box(w["box"], width, height)
+            if bbox is not None and w["text"].strip():
+                words.append(Word(text=w["text"], bbox=bbox))
+                texts.append(w["text"])
+        if texts:
+            lines.append(" ".join(texts))
+    doc = Document(
+        doc_id=doc_id,
+        pages=[
+            Page(
+                page=0, width=float(width), height=float(height), text="\n".join(lines), words=words
+            )
+        ],
+    )
+
+    # question -> linked answers, in annotation order
+    answers_by_question: dict[int, list[int]] = {}
+    for entry in annotation.get("form", []):
+        for a, b in entry.get("linking", []):
+            qa = (a, b)
+            if (
+                entries.get(qa[0], {}).get("label") == "question"
+                and entries.get(qa[1], {}).get("label") == "answer"
+            ):
+                answers_by_question.setdefault(qa[0], [])
+                if qa[1] not in answers_by_question[qa[0]]:
+                    answers_by_question[qa[0]].append(qa[1])
+
+    golds: list[FieldGold] = []
+    leaves: list[SchemaLeaf] = []
+    seen_paths: set[str] = set()
+    for q_id, a_ids in answers_by_question.items():
+        question = entries[q_id]["text"].strip()
+        # DECISION: multiple linked answers concatenate in annotation order.
+        value = " ".join(entries[a]["text"].strip() for a in a_ids if entries[a]["text"].strip())
+        if not question or not value:
+            continue
+        path = slugify(question)
+        n = 2
+        while path in seen_paths:
+            path = f"{slugify(question)}_{n}"
+            n += 1
+        seen_paths.add(path)
+
+        answer_boxes = [
+            _norm_box(w["box"], width, height) for a in a_ids for w in entries[a].get("words", [])
+        ]
+        answer_boxes = [b for b in answer_boxes if b is not None]
+        gold_box = (
+            Grounding(
+                page=0,
+                bbox=(
+                    min(b[0] for b in answer_boxes),
+                    min(b[1] for b in answer_boxes),
+                    max(b[2] for b in answer_boxes),
+                    max(b[3] for b in answer_boxes),
+                ),
+                support=1.0,
+            )
+            if answer_boxes
+            else None
+        )
+        scoring = "numeric" if parse_number(value) is not None else "semantic"
+        golds.append(
+            FieldGold(path=path, value=value, scoring=scoring, numeric_tol=1e-6, gold_box=gold_box)
+        )
+        leaves.append(SchemaLeaf(path=path, type="string", scoring=scoring))
+
+    if not golds or not words:
+        return None
+    schema = Schema(name=f"funsd-{doc_id}", leaves=leaves)
+    return BenchDocument(doc=doc, schema=schema, golds=golds)
+
+
+def load(
+    split: str = "testing", limit: int | None = None, cache_dir: str | Path = "data"
+) -> list[BenchDocument]:  # pragma: no cover - network on first call
+    """Load FUNSD (downloads + unzips once into ``cache_dir``)."""
+    from PIL import Image
+
+    root = Path(cache_dir) / "funsd"
+    if not root.exists():
+        import io
+        import urllib.request
+        import zipfile
+
+        Path(cache_dir).mkdir(parents=True, exist_ok=True)
+        with urllib.request.urlopen(FUNSD_URL) as resp:
+            payload = resp.read()
+        with zipfile.ZipFile(io.BytesIO(payload)) as zf:
+            zf.extractall(root)
+
+    ann_dir = root / "dataset" / f"{split}_data" / "annotations"
+    img_dir = root / "dataset" / f"{split}_data" / "images"
+    out: list[BenchDocument] = []
+    for i, ann_path in enumerate(sorted(ann_dir.glob("*.json"))):
+        if limit is not None and i >= limit:
+            break
+        annotation = json.loads(ann_path.read_text(encoding="utf-8"))
+        img_path = img_dir / f"{ann_path.stem}.png"
+        with Image.open(img_path) as img:
+            width, height = img.size
+        item = bench_from_annotation(f"funsd-{split}-{ann_path.stem}", annotation, width, height)
+        if item is not None:
+            out.append(item)
+    return out

verifydoc-0.2.0/configs/cord-paddleocr.yaml

@@ -0,0 +1,14 @@
+# Pinned real-model experiment (issue #3): PaddleOCR-VL on CORD receipts.
+# Needs a 24 GB GPU + `pip install paddlepaddle-gpu paddleocr` and CORD
+# images exported to disk — see docs/REAL_MODELS.md.
+dataset: cord
+split: validation
+limit: 100
+extractor: paddleocr-vl   # any name from the adapter registry
+seed: 7
+k: 5
+calibration_fraction: 0.5
+alphas: [0.02, 0.05]
+n_boot: 300
+# record at run time for the paper table header:
+#   paddleocr==<version>, GPU model, driver version

verifydoc-0.2.0/configs/cord.yaml

@@ -0,0 +1,14 @@
+# CORD v2 slice: REAL receipts, REAL text layers + gold boxes; extractor
+# errors simulated at seeded rates (real-model configs land with issue #3).
+# First run downloads ~100 receipts from the HF hub, then caches to data/.
+dataset: cord
+split: validation
+limit: 100
+seed: 7
+k: 5
+error_rate: 0.18
+omit_rate: 0.04
+hallucinate_rate: 0.06
+calibration_fraction: 0.5
+alphas: [0.02, 0.05]
+n_boot: 300

verifydoc-0.2.0/configs/funsd.yaml

@@ -0,0 +1,12 @@
+# FUNSD slice: real scanned forms, question->answer gold fields with exact
+# annotated answer boxes; extractor errors simulated at seeded rates.
+dataset: funsd
+split: testing        # 50 forms; use `training` (149) for scale
+seed: 7
+k: 5
+error_rate: 0.18
+omit_rate: 0.04
+hallucinate_rate: 0.06
+calibration_fraction: 0.5
+alphas: [0.02, 0.05]
+n_boot: 300

verifydoc-0.2.0/docs/REAL_MODELS.md

@@ -0,0 +1,61 @@
+# Running real extractors (GPU runbook)
+
+The shipped adapters run PaddleOCR-VL and dots.ocr on a single 24 GB GPU.
+This is the paper's headline experiment (issue #3): the same harness, with a
+real extractor instead of the simulated one.
+
+## One-time setup (Linux + NVIDIA, 24 GB)
+
+```bash
+git clone https://github.com/bhaskargurram-ai/verifydoc && cd verifydoc
+python3.11 -m venv .venv && source .venv/bin/activate
+pip install -e ".[dev,pdf,data]"
+
+# PaddleOCR-VL
+pip install paddlepaddle-gpu paddleocr
+
+# dots.ocr
+pip install torch transformers accelerate
+```
+
+## Smoke test one receipt
+
+```bash
+python - <<'EOF'
+from benchmark.datasets import cord
+from verifydoc.adapters import get_adapter
+
+item = cord.load(split="validation", limit=1)[0]
+adapter = get_adapter("paddleocr-vl")
+for p in adapter.extract(item.doc, item.schema):
+    print(p.path, p.value, p.meta.get("token_logprobs"))
+EOF
+```
+
+Note: OCR adapters read `page.image_path` — for CORD, export images first
+(`row["image"].save(...)`) and set `image_path` on each page; a
+`--with-images` flag on the CORD loader is the intended patch (part of
+issue #3).
+
+## Full run
+
+```bash
+python scripts/run_benchmark.py --config configs/cord-paddleocr.yaml \
+  --out paper/generated/cord-paddleocr
+```
+
+`configs/cord-paddleocr.yaml` is checked in as the pinned experiment config;
+the harness dispatches `extractor:` through the adapter registry
+(`mock` | `text-search` | `paddleocr-vl` | `dots-ocr` | `docling` | `api-vlm`).
+
+## What to report
+
+The identical tables the simulated run produces (`calibration.md`,
+`selective.md`, `conformal.md`, `grounding.md`) — swap them into the paper
+via `make results`. Pin model versions in the config (`paddleocr==x.y.z`)
+and record GPU + driver in the table header for reproducibility.
+
+## Cloud options
+
+Any of: RunPod / Lambda / Vast single RTX 4090 (24 GB) instance ≈ $0.4–0.8/hr;
+the full 100-receipt sweep with k=5 sampling is an hours-not-days job.

{verifydoc-0.1.0 → verifydoc-0.2.0}/paper/generated/calibration.md

@@ -12,11 +12,11 @@
 | verbalized | platt | 0.0430 | 0.0750 | 0.1047 | 0.3673 |
 | verbalized | isotonic | 0.0419 | 0.0720 | 0.1050 | 0.3673 |
 | verbalized | histogram | 0.0440 | 0.0973 | 0.1044 | 0.3657 |
-| grounding | raw | 0.0805 | 0.0762 | 0.0598 | 0.1535 |
-| grounding | temperature | 0.0563 | 0.0603 | 0.0402 | 0.1108 |
-| grounding | platt | 0.0001 | 0.0000 | 0.0000 | 0.0001 |
-| grounding | isotonic | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
-| grounding | histogram | 0.0000 | 0.0000 | 0.0048 | 0.0157 |
+| grounding | raw | 0.0856 | 0.0711 | 0.0655 | 0.3134 |
+| grounding | temperature | 0.0675 | 0.0605 | 0.0439 | 0.1371 |
+| grounding | platt | 0.0513 | 0.0146 | 0.0230 | 0.1106 |
+| grounding | isotonic | 0.0017 | 0.0064 | 0.0056 | 0.0237 |
+| grounding | histogram | 0.0023 | 0.0023 | 0.0090 | 0.0277 |
 | consensus | raw | 0.1705 | 0.1705 | 0.0605 | 0.2222 |
 | consensus | temperature | 0.0608 | 0.0608 | 0.0329 | 0.1070 |
 | consensus | platt | 0.0111 | 0.0111 | 0.0113 | 0.0607 |
@@ -27,3 +27,8 @@
 | combined | platt | 0.0109 | 0.0109 | 0.0105 | 0.0503 |
 | combined | isotonic | 0.0107 | 0.0107 | 0.0101 | 0.1693 |
 | combined | histogram | 0.0113 | 0.0113 | 0.0112 | 0.1825 |
+| learned | raw | 0.0006 | 0.0199 | 0.0110 | 0.0553 |
+| learned | temperature | 0.0026 | 0.0183 | 0.0111 | 0.0553 |
+| learned | platt | 0.0111 | 0.0111 | 0.0109 | 0.0574 |
+| learned | isotonic | 0.0114 | 0.0114 | 0.0113 | 0.1927 |
+| learned | histogram | 0.0002 | 0.0195 | 0.0112 | 0.0622 |

{verifydoc-0.1.0 → verifydoc-0.2.0}/paper/generated/conformal.md

@@ -6,9 +6,11 @@
 | token_prob | 0.0500 | 0.9230 | 0.8779 | 0.0199 | True |
 | verbalized | 0.0200 | inf | 0.0000 | 0.0000 | True |
 | verbalized | 0.0500 | inf | 0.0000 | 0.0000 | True |
-| grounding | 0.0200 | 0.8667 | 0.8837 | 0.0000 | True |
-| grounding | 0.0500 | 0.8462 | 0.9128 | 0.0318 | True |
+| grounding | 0.0200 | 0.8667 | 0.8779 | 0.0000 | True |
+| grounding | 0.0500 | 0.8462 | 0.9070 | 0.0321 | True |
 | consensus | 0.0200 | 0.2000 | 1.0000 | 0.0114 | True |
 | consensus | 0.0500 | 0.2000 | 1.0000 | 0.0114 | True |
 | combined | 0.0200 | 0.4761 | 1.0000 | 0.0114 | True |
 | combined | 0.0500 | 0.4761 | 1.0000 | 0.0114 | True |
+| learned | 0.0200 | 0.9741 | 1.0000 | 0.0114 | True |
+| learned | 0.0500 | 0.9741 | 1.0000 | 0.0114 | True |

verifydoc-0.2.0/paper/generated/cord/calibration.md

@@ -0,0 +1,34 @@
+# Calibration: signal x calibrator (test split)
+
+| signal | calibrator | ece | adaptive_ece | brier | nll |
+|---|---|---|---|---|---|
+| token_prob | raw | 0.0957 | 0.1080 | 0.1206 | 0.4056 |
+| token_prob | temperature | 0.1088 | 0.1175 | 0.1085 | 0.3563 |
+| token_prob | platt | 0.0537 | 0.0560 | 0.0842 | 0.3137 |
+| token_prob | isotonic | 0.0471 | 0.0536 | 0.0847 | 0.3316 |
+| token_prob | histogram | 0.0535 | 0.0674 | 0.0887 | 0.3061 |
+| verbalized | raw | 0.0610 | 0.0687 | 0.1246 | 0.4348 |
+| verbalized | temperature | 0.0201 | 0.0556 | 0.1222 | 0.4106 |
+| verbalized | platt | 0.0089 | 0.0414 | 0.1210 | 0.4067 |
+| verbalized | isotonic | 0.0128 | 0.0490 | 0.1211 | 0.4470 |
+| verbalized | histogram | 0.0101 | 0.0542 | 0.1214 | 0.4083 |
+| grounding | raw | 0.0882 | 0.0816 | 0.0761 | 1.0679 |
+| grounding | temperature | 0.0663 | 0.0761 | 0.0668 | 0.2475 |
+| grounding | platt | 0.0429 | 0.0470 | 0.0545 | 0.2359 |
+| grounding | isotonic | 0.0281 | 0.0468 | 0.0485 | 0.2854 |
+| grounding | histogram | 0.0292 | 0.0482 | 0.0494 | 0.3670 |
+| consensus | raw | 0.1541 | 0.1573 | 0.0875 | 0.5650 |
+| consensus | temperature | 0.0892 | 0.0947 | 0.0751 | 0.5385 |
+| consensus | platt | 0.0093 | 0.0461 | 0.0467 | 0.1955 |
+| consensus | isotonic | 0.0271 | 0.0475 | 0.0478 | 0.2442 |
+| consensus | histogram | 0.0271 | 0.0453 | 0.0478 | 0.2444 |
+| combined | raw | 0.0845 | 0.0724 | 0.0516 | 0.2203 |
+| combined | temperature | 0.0517 | 0.0360 | 0.0446 | 0.2210 |
+| combined | platt | 0.0323 | 0.0367 | 0.0451 | 0.2023 |
+| combined | isotonic | 0.0232 | 0.0376 | 0.0390 | 0.2944 |
+| combined | histogram | 0.0212 | 0.0387 | 0.0373 | 0.2452 |
+| learned | raw | 0.0131 | 0.0256 | 0.0424 | 0.1744 |
+| learned | temperature | 0.0250 | 0.0311 | 0.0431 | 0.1796 |
+| learned | platt | 0.0269 | 0.0309 | 0.0368 | 0.1869 |
+| learned | isotonic | 0.0200 | 0.0427 | 0.0362 | 0.2394 |
+| learned | histogram | 0.0211 | 0.0446 | 0.0371 | 0.2058 |

verifydoc-0.2.0/paper/generated/cord/conformal.md

@@ -0,0 +1,16 @@
+# Conformal abstention: guarantee vs forced review
+
+| signal | alpha | threshold | test_coverage | achieved_risk | guarantee_held |
+|---|---|---|---|---|---|
+| token_prob | 0.0200 | 0.9308 | 0.8165 | 0.0486 | False |
+| token_prob | 0.0500 | 0.9170 | 0.8678 | 0.0705 | False |
+| verbalized | 0.0200 | inf | 0.0000 | 0.0000 | True |
+| verbalized | 0.0500 | inf | 0.0000 | 0.0000 | True |
+| grounding | 0.0200 | 0.9048 | 0.8612 | 0.0365 | False |
+| grounding | 0.0500 | 0.7143 | 0.8992 | 0.0570 | False |
+| consensus | 0.0200 | inf | 0.0000 | 0.0000 | True |
+| consensus | 0.0500 | 0.2000 | 1.0000 | 0.0494 | True |
+| combined | 0.0200 | 0.6793 | 0.9793 | 0.0358 | False |
+| combined | 0.0500 | 0.4801 | 0.9984 | 0.0494 | True |
+| learned | 0.0200 | 0.9319 | 0.9793 | 0.0341 | False |
+| learned | 0.0500 | 0.6041 | 0.9968 | 0.0479 | True |

verifydoc-0.2.0/paper/generated/cord/extraction.md

@@ -0,0 +1,5 @@
+# Extraction quality (single-run baseline)
+
+| precision | recall | f1 | exact_match | omission_rate | hallucination_rate |
+|---|---|---|---|---|---|
+| 0.8455 | 0.8163 | 0.8296 | 0.8123 | 0.0408 | 0.0056 |

verifydoc-0.2.0/paper/generated/cord/grounding.md

@@ -0,0 +1,5 @@
+# Grounding quality (consensus predictions)
+
+| box_acc@0.5 | box_acc@0.7 | mean_iou | acc_grounded | acc_ungrounded | grounding_gap |
+|---|---|---|---|---|---|
+| 0.9967 | 0.9967 | 0.9975 | 0.9571 | 0.2500 | 0.7071 |

verifydoc-0.2.0/paper/generated/cord/selective.md

@@ -0,0 +1,10 @@
+# Selective prediction by signal (raw confidence, test split)
+
+| signal | e_aurc | auroc | coverage@2% | coverage@5% | e_aurc_ci |
+|---|---|---|---|---|---|
+| token_prob | 0.0414 | 0.8343 | 0.0182 | 0.8165 | [0.0237, 0.0621] |
+| verbalized | 0.1183 | 0.5280 | 0.0000 | 0.0331 | [0.0864, 0.1527] |
+| grounding | 0.0497 | 0.8752 | 0.0000 | 0.8893 | [0.0161, 0.0530] |
+| consensus | 0.0426 | 0.6095 | 0.0000 | 1.0000 | [0.0177, 0.0529] |
+| combined | 0.0340 | 0.6304 | 0.1624 | 1.0000 | [0.0154, 0.0572] |
+| learned | 0.0352 | 0.6320 | 0.0892 | 1.0000 | [0.0174, 0.0571] |