verifydoc 0.3.0__tar.gz → 0.5.0__tar.gz

verifydoc-0.5.0/.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,25 @@
+---
+name: Bug report
+about: Something isn't working as documented
+labels: bug
+---
+
+**What happened**
+A clear description of the bug.
+
+**Minimal repro**
+Ideally a tiny document + schema and the `verify(...)` call or CLI command.
+
+```python
+# your repro here
+```
+
+**Expected vs actual**
+- Expected:
+- Actual:
+
+**Environment**
+- verifydoc version (`verifydoc version`):
+- Python version:
+- OS:
+- Extractor/adapter (if relevant):

verifydoc-0.5.0/.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,19 @@
+---
+name: Feature request / new adapter or dataset
+about: Propose an extractor adapter, dataset slice, signal, calibrator, or feature
+labels: enhancement
+---
+
+**What and why**
+What should VerifyDoc do, and what does it enable?
+
+**Which layer**
+- [ ] Adapter (new extractor) — recall: one file in `verifydoc/adapters/`
+- [ ] Dataset slice (`benchmark/datasets/`)
+- [ ] Confidence signal / calibrator
+- [ ] Eval metric
+- [ ] Other
+
+**Sketch (optional)**
+Interface or output shape you have in mind. For metrics, include a
+hand-computed expected value so it becomes a regression test.

verifydoc-0.5.0/.github/pull_request_template.md

@@ -0,0 +1,11 @@
+## What this does
+
+Brief description. Link any issue it closes (`Closes #N`).
+
+## Checklist
+
+- [ ] `make test lint typecheck` all green locally
+- [ ] New/changed code has tests; metric code has a numeric regression test
+- [ ] Docs updated if the public API changed (`README.md` / `PROJECT.md`)
+- [ ] Conventional Commit title (`feat:` / `fix:` / `docs:` / `test:` / `bench:` …)
+- [ ] If this adds a model adapter: all SDK code stays inside `verifydoc/adapters/`

{verifydoc-0.3.0 → verifydoc-0.5.0}/.gitignore

@@ -37,3 +37,15 @@ outputs/
 .DS_Store
 .idea/
 .vscode/
+
+# LaTeX build artifacts
+paper/*.aux
+paper/*.log
+paper/*.out
+paper/*.bbl
+paper/*.blg
+paper/main.pdf
+
+# local secrets (never commit)
+.env.secret
+.env.local

{verifydoc-0.3.0 → verifydoc-0.5.0}/CHANGELOG.md

@@ -1,5 +1,45 @@
 # Changelog
 
+## v0.5.0 — 2026-07-22
+
+A novel method, a frontier-VLM comparison, and an agent-facing application.
+
+- **Novel method — grounding-conditioned (Mondrian) conformal risk control**
+  (`verifydoc/calibration/grouped_conformal.py`): conditions the abstention
+  threshold on provenance, preserving a finite-sample per-group risk guarantee
+  while accepting well-grounded fields at a lower bar. Controlled study
+  (`scripts/grouped_conformal_experiment.py`): **+0.50 mean coverage** at a
+  fixed 5% risk in the uninformative-confidence regime where pooled conformal
+  accepts almost nothing. First use of provenance as the conditioning taxonomy
+  for conformal in document extraction.
+- **Frontier VLM results** (`claude-sonnet-5`, k=3, on CORD): recall 0.56 but a
+  **0.48 hallucination rate**; verbalized confidence is informative (AUROC
+  0.86) for the VLM yet useless (0.50) for OCR pipelines — a cross-extractor
+  finding the single-model literature misses. Cross-extractor summary in
+  `paper/generated/REAL_MODELS_RESULTS.md`.
+- **MCP server** (`verifydoc-mcp`): exposes `verify_extraction` over the Model
+  Context Protocol so AI agents extract documents with confidence + grounding +
+  accept/review — a drop-in trust layer for the agentic era (`docs/MCP.md`).
+- Paper elevated to a method contribution (6pp, compiles; novel result in the
+  abstract and a dedicated section). `mcp` install extra.
+
+## v0.4.0 — 2026-07-21
+
+Paper-ready: fair API-VLM comparison + a compilable paper.
+
+- **Vendor-neutral API-VLM extractor** (`verifydoc/adapters/api_vlm.py`):
+  OpenAI + Anthropic clients behind one `CompletionClient` protocol, with
+  **temperature-based k-sample sampling** so self-consistency consensus and
+  verbalized confidence are non-degenerate (the fair comparison the paper
+  needs). Harness `adapter_kwargs` passthrough; `configs/cord-apivlm.yaml`;
+  `ocr` and `api` install extras.
+- **Compilable paper** (`paper/main.tex`) with real RapidOCR/PaddleOCR/
+  synthetic numbers, auto-generated LaTeX result tables
+  (`scripts/tables_to_latex.py`, booktabs) and `refs.bib`; `make paper`.
+- **Docs**: `docs/how-it-works.md`; README documentation index; runnable
+  `examples/`.
+- Repo hygiene: removed internal build-tooling notes; tidied PROJECT.md.
+
 ## v0.3.0 — 2026-07-21
 
 First **real-model** results (issue #3).

verifydoc-0.5.0/CITATION.cff

@@ -0,0 +1,17 @@
+cff-version: 1.2.0
+message: "If you use VerifyDoc or VerifyDocBench, please cite it."
+title: "VerifyDoc: Calibrated, Abstaining, Grounded Document Extraction"
+authors:
+  - family-names: Gurram
+    given-names: Bhaskar
+repository-code: "https://github.com/bhaskargurram-ai/verifydoc"
+license: Apache-2.0
+version: 0.5.0
+date-released: 2026-07-21
+keywords:
+  - document-extraction
+  - calibration
+  - selective-prediction
+  - conformal-prediction
+  - grounding
+  - hallucination-detection

verifydoc-0.5.0/CODE_OF_CONDUCT.md

@@ -0,0 +1,32 @@
+# Code of Conduct
+
+## Our pledge
+
+We as members, contributors, and maintainers pledge to make participation in
+VerifyDoc a harassment-free experience for everyone, regardless of age, body
+size, visible or invisible disability, ethnicity, sex characteristics, gender
+identity and expression, level of experience, education, socio-economic
+status, nationality, personal appearance, race, religion, or sexual identity
+and orientation.
+
+## Our standards
+
+Examples of behavior that contributes to a positive environment:
+
+- Being respectful of differing opinions, viewpoints, and experiences
+- Giving and gracefully accepting constructive feedback
+- Focusing on what is best for the community and the project
+
+Unacceptable behavior includes harassment, insulting or derogatory comments,
+personal or political attacks, publishing others' private information, and
+other conduct that could reasonably be considered inappropriate.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported to the project maintainers via a GitHub issue or direct contact.
+Maintainers will review and respond, and will respect the privacy and
+security of the reporter.
+
+This Code of Conduct is adapted from the [Contributor Covenant](https://www.contributor-covenant.org/),
+version 2.1.

verifydoc-0.5.0/CONTRIBUTING.md

@@ -0,0 +1,55 @@
+# Contributing to VerifyDoc
+
+Thanks for helping build the open trust layer for document extraction. This
+project is designed so contributions are small and self-contained.
+
+## Ground rules (the architecture makes these easy)
+
+1. **Adapters are isolated.** All model-specific code lives behind the
+   `ExtractorAdapter` interface in `verifydoc/adapters/`. **Adding a new
+   extractor = one new file** — nothing outside `adapters/` imports a model SDK.
+2. **Stages are independent:** ingest → adapter → confidence → calibration →
+   grounding → policy → report. Each has typed inputs/outputs
+   (`verifydoc/types.py`) and its own tests. No stage reaches into another.
+3. **The eval harness is decoupled.** It scores anything that emits
+   `FieldPrediction`. Never couple a metric to a specific model.
+4. **Never tune on test.** Calibration/abstention thresholds are fit only on
+   the dedicated calibration split (enforced by `assert_disjoint`).
+5. **No network in unit tests.** Mock adapters. Every metric gets a numeric
+   regression test against a hand-computed fixture.
+
+## Good first issues
+
+Look for the [`good first issue`](https://github.com/bhaskargurram-ai/verifydoc/labels/good%20first%20issue)
+label. Great starting points:
+
+- **Add an extractor adapter** (Surya, GOT-OCR2.0, an API VLM) — one file in
+  `adapters/`, plus a normalization test (see `tests/test_pipeline.py`).
+- **Add a dataset slice** (SROIE, DocILE, XFUND) — a loader in
+  `benchmark/datasets/` returning `BenchDocument`s, following `funsd.py`.
+- **A new confidence signal or calibrator** behind the existing interfaces.
+
+## Dev setup
+
+```bash
+git clone https://github.com/bhaskargurram-ai/verifydoc && cd verifydoc
+uv venv .venv && uv pip install -e ".[dev]"
+make test lint typecheck        # all must be green
+```
+
+## Before you open a PR
+
+- `ruff`, `black`, `mypy`, and `pytest` all pass (CI enforces; no PR merges red).
+- New/changed code has tests; metric code has a numeric regression test.
+- Public-API changes update `README.md` / `PROJECT.md`.
+- Use [Conventional Commits](https://www.conventionalcommits.org/):
+  `feat:` `fix:` `docs:` `test:` `refactor:` `chore:` `perf:` `bench:`.
+- Branch off `main`, open a PR, we squash-merge.
+
+## Reporting bugs / ideas
+
+Open an issue with a minimal repro (a tiny document + schema is ideal). For
+metric bugs, include the expected hand-computed value — that becomes the
+regression test.
+
+By contributing you agree your work is licensed under Apache-2.0.

{verifydoc-0.3.0 → verifydoc-0.5.0}/Makefile

@@ -1,4 +1,4 @@
-.PHONY: install lint format typecheck test cov results demo clean
+.PHONY: install lint format typecheck test cov results paper demo clean
 
 PY ?= python
 
@@ -28,6 +28,14 @@ results:
 	$(PY) scripts/run_benchmark.py --config configs/demo.yaml --out paper/generated
 	$(PY) scripts/run_benchmark.py --config configs/cord.yaml --out paper/generated/cord
 	$(PY) scripts/run_benchmark.py --config configs/funsd.yaml --out paper/generated/funsd
+	$(PY) scripts/grouped_conformal_experiment.py
+	$(PY) scripts/tables_to_latex.py paper/generated
+
+# Compile the paper (needs a LaTeX toolchain); tables come from `make results`.
+paper:
+	$(PY) scripts/grouped_conformal_experiment.py
+	$(PY) scripts/tables_to_latex.py paper/generated
+	cd paper && pdflatex -interaction=nonstopmode main.tex && bibtex main && pdflatex -interaction=nonstopmode main.tex && pdflatex -interaction=nonstopmode main.tex
 
 demo:
 	streamlit run ui/streamlit_app.py

{verifydoc-0.3.0 → verifydoc-0.5.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: verifydoc
-Version: 0.3.0
+Version: 0.5.0
 Summary: A trust layer for document -> structured-JSON extraction: calibrated per-field confidence, source grounding, and abstention on top of any extractor.
 Project-URL: Homepage, https://github.com/bhaskargurram-ai/verifydoc
 Project-URL: Repository, https://github.com/bhaskargurram-ai/verifydoc
@@ -226,6 +226,9 @@ Requires-Dist: pyyaml>=6.0
 Requires-Dist: scikit-learn>=1.3
 Requires-Dist: scipy>=1.11
 Requires-Dist: typer>=0.9
+Provides-Extra: api
+Requires-Dist: anthropic>=0.39; extra == 'api'
+Requires-Dist: openai>=1.40; extra == 'api'
 Provides-Extra: data
 Requires-Dist: datasets>=2.16; extra == 'data'
 Provides-Extra: dev
@@ -235,6 +238,11 @@ Requires-Dist: pre-commit>=3.6; extra == 'dev'
 Requires-Dist: pytest-cov>=4.1; extra == 'dev'
 Requires-Dist: pytest>=7.4; extra == 'dev'
 Requires-Dist: ruff>=0.4; extra == 'dev'
+Provides-Extra: mcp
+Requires-Dist: mcp>=1.2; extra == 'mcp'
+Provides-Extra: ocr
+Requires-Dist: onnxruntime>=1.17; extra == 'ocr'
+Requires-Dist: rapidocr>=1.3; extra == 'ocr'
 Provides-Extra: pdf
 Requires-Dist: pdfplumber>=0.10; extra == 'pdf'
 Requires-Dist: pillow>=10.0; extra == 'pdf'
@@ -282,7 +290,8 @@ pip install 'verifydoc[pdf]'   # + PDF/image ingestion
 ```python
 from verifydoc import verify
 
-result = verify("invoice.txt", schema="invoice_schema.json", k=3)
+# ready-to-run sample lives in examples/
+result = verify("examples/invoice.txt", schema="examples/invoice_schema.json")
 for f in result.fields:
     print(f"{f.path:12} = {f.value!r:24} conf={f.confidence:.2f} {f.decision}")
     if f.grounding:
@@ -290,10 +299,26 @@ for f in result.fields:
 ```
 
 ```bash
-verifydoc extract invoice.txt --schema invoice_schema.json --k 3 --threshold 0.8
+verifydoc extract examples/invoice.txt --schema examples/invoice_schema.json --threshold 0.8
 streamlit run ui/streamlit_app.py     # review UI: green/red fields + click-through to source
 ```
 
+See [`examples/`](examples/) for the full runnable walk-through.
+
+## For AI agents (MCP)
+
+Give any MCP-capable agent (Claude Desktop, IDEs, custom agents) a **trust
+layer for reading documents** — so it acts on confident, grounded fields and
+escalates the rest instead of hallucinating forward:
+
+```bash
+pip install 'verifydoc[mcp]'
+verifydoc-mcp     # stdio MCP server exposing verify_extraction()
+```
+
+Every field the agent extracts comes back with `confidence + grounding +
+accept/review`. See [docs/MCP.md](docs/MCP.md) for the one-line client config.
+
 Schemas are plain JSON Schema, with each leaf optionally declaring **how it is scored** (the executable-schema pattern):
 
 ```json
@@ -311,14 +336,37 @@ Schemas are plain JSON Schema, with each leaf optionally declaring **how it is s
 
 | Layer | Modules | Status |
 |---|---|---|
-| **Adapters** (all model code isolated here) | mock · text-search · PaddleOCR-VL · dots.ocr · Docling/MinerU output · API-VLM | ✅ |
+| **Adapters** (all model code isolated here) | mock · text-search · RapidOCR · PaddleOCR · dots.ocr · Docling/MinerU output · API-VLM (OpenAI/Anthropic) | ✅ |
 | **Confidence signals** | token-prob · verbalized · consensus (k-sample voting) · grounding-based · combined | ✅ |
-| **Calibrators** (fit on a dedicated split, never test) | temperature · Platt · isotonic · histogram · **split conformal** with finite-sample risk guarantee | ✅ |
+| **Calibrators** (fit on a dedicated split, never test) | temperature · Platt · isotonic · histogram · **split conformal** · **grounding-conditioned (Mondrian) conformal** (novel — recovers coverage a pooled threshold forfeits) | ✅ |
 | **Grounding** | value → page/bbox/char-span attachment with support scores | ✅ |
 | **Policy** | empirical & conformal accept thresholds for a target selective risk | ✅ |
 | **Eval harness / VerifyDocBench scorer** | Field-F1 · exact · CER/WER · ANLS · TEDS/TEDS-Struct · GriTS · omission vs hallucination · ECE/Adaptive-ECE/MCE/Brier/NLL/TCE · RC/AURC/E-AURC/Coverage@Risk/AUROC/AUPR/FPR@95 · box IoU/span-F1/grounding-conditioned correctness · bootstrap CIs + paired tests | ✅ |
 
-Every metric implements the exact definition in [PROJECT.md §5](PROJECT.md) with a hand-computed numeric regression test (200 tests, `eval/` coverage 98%).
+Every metric implements the exact definition in [PROJECT.md §5](PROJECT.md) with a hand-computed numeric regression test (201 tests, `eval/` coverage 98%).
+
+## Results on real documents
+
+Two independent real OCR extractors (**RapidOCR** and **PaddleOCR**) on real
+**CORD** receipts and **FUNSD** forms, scored by the harness
+([full tables + reading](paper/generated/REAL_MODELS_RESULTS.md)):
+
+| Confidence signal | ranks errors? | CORD AUROC (RapidOCR / PaddleOCR) |
+|---|---|---|
+| **learned combiner** | ✅ best | **0.89 / 0.84** |
+| **grounding** | ✅ strong | 0.82 / 0.74 |
+| token-probability | ~ moderate | 0.69 / 0.68 |
+| verbalized / consensus | ✗ uninformative | 0.50 / 0.50 |
+
+- **Grounding is a real trust signal:** grounded fields are **84–85% correct
+  vs ~1%** for ungrounded (gap ≈ +0.84; box accuracy @IoU 0.5 = 0.75–0.78).
+- **The abstention layer is honest:** with a weak field-extractor the base
+  error rate is high, so conformal abstention at a 2–5% budget correctly
+  refuses to auto-accept — you report *selective risk*, not headline accuracy.
+- The synthetic slice (strong extractor) shows the other end: Coverage@2% ≈ 1.0.
+
+The thesis holds on real data: **grounding + a learned fusion rank errors;
+self-reported and single-sample-consensus confidence do not.**
 
 ## The benchmark
 
@@ -328,16 +376,13 @@ make results     # regenerates every table/figure in paper/generated from config
 
 The harness runs **signals × calibrators × the full metric suite** with a
 document-level calibration split (disjointness asserted in code), bootstrap
-CIs, and a conformal-guarantee row. The repo ships a deterministic synthetic
-slice that runs in CI; loaders for CORD (and next: FUNSD, SROIE, DocILE,
-XFUND) extend it. Sample findings on the shipped slice
-([tables](paper/generated)):
-
-- **Verbalized self-confidence is badly miscalibrated** (the extractor says ~0.9 regardless of correctness) — exactly the failure mode reported for RLHF'd models.
-- **Consensus and grounding signals rank errors near-perfectly** (AUROC ≈ 0.98): corrupted values can't be traced back to the page, so grounding support collapses.
-- **The conformal row holds its guarantee** on every tested α, and reports the abstention it forces.
-
-These self-checks run as unit tests — the repo's core claims are CI-enforced, not just stated.
+CIs, and a conformal-guarantee row. It ships a deterministic **synthetic**
+slice (runs in CI) plus **CORD** and **FUNSD** loaders with gold source boxes;
+`extractor:` dispatches to any adapter (`rapidocr`, `paddleocr-vl`, …) and
+`dataset:` to any slice. See the [GPU runbook](docs/REAL_MODELS.md) to
+reproduce the real-model rows. Core claims (grounding beats verbalized;
+conformal holds its guarantee) are also **CI-enforced as unit tests**, not
+just stated.
 
 ## Why not just use the parser's own score?
 
@@ -350,9 +395,19 @@ buys you.
 ## Roadmap
 
 - [x] v0.1 — library + CLI + harness + synthetic benchmark slice + UI
-- [ ] CORD/FUNSD/SROIE slices with gold source boxes (VerifyDocBench v1)
-- [ ] Learned signal combiner + per-field-type calibration
-- [ ] Paper: first systematic study of confidence signals × calibration × abstention for document extraction
+- [x] v0.2 — CORD + FUNSD real slices with gold boxes; learned combiner; 1000× faster grounder
+- [x] v0.3 — real-model results (RapidOCR + PaddleOCR on CORD/FUNSD)
+- [x] v0.4 — vendor-neutral API-VLM extractor (OpenAI/Anthropic) with k-sample consensus; compilable paper with auto-generated tables
+- [x] v0.5 — **novel method** (grounding-conditioned conformal, +0.50 coverage at fixed risk) + **MCP server** (agent trust layer) + real frontier-VLM results
+- [ ] dots.ocr via vllm; SROIE / DocILE / XFUND slices; human-labeled correctness + IAA
+- [ ] Paper submission ([contributions welcome](CONTRIBUTING.md))
+
+## Documentation
+
+- [How it works](docs/how-it-works.md) — the pipeline, the abstention idea, why grounding is a trust signal
+- [Real-model results](paper/generated/REAL_MODELS_RESULTS.md) — RapidOCR + PaddleOCR numbers and reading
+- [GPU runbook](docs/REAL_MODELS.md) — reproduce the real-extractor rows
+- [USP audit](docs/USP.md) · [Paper draft](paper/main.tex) · [Full spec](PROJECT.md)
 
 ## Development
 
@@ -360,6 +415,8 @@ buys you.
 git clone https://github.com/bhaskargurram-ai/verifydoc && cd verifydoc
 uv venv .venv && uv pip install -e ".[dev]"
 make test lint typecheck     # all green before any PR (CI enforces)
+make results                 # regenerate benchmark tables + LaTeX
+make paper                   # compile the paper (needs a LaTeX toolchain)
 ```
 
 Contributions welcome — see the issues tagged `good-first-issue`. All model-specific code goes in `verifydoc/adapters/`; a new extractor is one file.

{verifydoc-0.3.0 → verifydoc-0.5.0}/PROJECT.md

@@ -1,6 +1,6 @@
 # VerifyDoc — A Trust Layer for Document → Structured-JSON Extraction
 
-**Project document / build spec (v1.0).** This is the single source of truth for building VerifyDoc with Claude Code. It contains the motivation, the precise problem definition, the literature-grounded evaluation methodology (the core of the paper), the system architecture, the repository layout, a six-week implementation plan, the git workflow, and the paper plan. Hand this file (plus `CLAUDE.md`) to Claude Code and build module by module against the "Definition of Done" in each section.
+**Project document / design spec (v1.0).** This is the single source of truth for VerifyDoc's design. It contains the motivation, the precise problem definition, the literature-grounded evaluation methodology (the core of the paper), the system architecture, the repository layout, a six-week implementation plan, the git workflow, and the paper plan. Build module by module against the "Definition of Done" in each section.
 
 One-liner: *VerifyDoc wraps any document-extraction model and returns, for every field, a calibrated confidence score, a source-grounding box/span, and an abstain-or-accept decision — so a human reviews the 5% of fields that are actually wrong instead of eyeballing all of them.*
 
@@ -12,7 +12,7 @@ Document → structured-JSON extraction is one of the highest-volume uses of LLM
 
 That is the wedge. VerifyDoc does not compete with the parsers — it *layers on top of any of them* and adds the missing reliability contract: field-level confidence + provenance + abstention. Commercial APIs already prove the demand (Box shipped field-level confidence in Jan 2026; Azure Document Intelligence, AWS Textract, and Extend all expose per-field confidence), but no popular open-source parser leads with it. Practitioner leaderboards name exactly this unmet need: whether a reviewer can trace an uncertain value back to the page before it enters a downstream system. VerifyDoc is the open-source answer.
 
-Why this is a good solo project: the tool is model-agnostic and thin (it wraps existing extractors), the paper contribution is the reliable "benchmark + released dataset + strong open baseline" type rather than a risky "beat-SOTA architecture" play, and the demo (highlight-the-hallucinated-field on the rendered page) is instantly shareable.
+Why the scope works: the tool is model-agnostic and thin (it wraps existing extractors), the paper contribution is the reliable "benchmark + released dataset + strong open baseline" type rather than a risky "beat-SOTA architecture" play, and the demo (highlight-the-hallucinated-field on the rendered page) is instantly shareable.
 
 ---
 
@@ -216,7 +216,6 @@ Design rules: adapters isolate all model-specific code behind one interface (`ex
 ```
 verifydoc/
 ├── README.md                     # quickstart + the killer demo GIF
-├── CLAUDE.md                     # Claude Code working context (see separate file)
 ├── PROJECT.md                    # this document
 ├── LICENSE                       # Apache-2.0 (permissive → adoption)
 ├── pyproject.toml                # packaging (pip install verifydoc)
@@ -269,7 +268,7 @@ verifydoc/
 
 ---
 
-## 9. Implementation plan (6 weeks, solo)
+## 9. Implementation plan (6 weeks)
 
 **Week 1 — skeleton + types + one adapter + eval stubs.** Repo, packaging, CI, pre-commit. Define `types.py` (`FieldPrediction`, `Grounding`). Implement ingest (PDF→images + text layer). Ship the **PaddleOCR-VL adapter** end-to-end on a handful of CORD receipts. Stub `eval/extraction.py` with Field-F1 + exact-match. *DoD: `verifydoc extract sample.pdf --schema cord.json` returns JSON with placeholder confidences; CI green.*
 
@@ -320,12 +319,6 @@ verifydoc/
 
 ---
 
-## 13. EB1-A mapping (brief, honest)
-
-This project can contribute to (a) **authorship of scholarly articles** (arXiv + workshop/TMLR paper); (b) **original contributions of major significance** — best evidenced by demonstrable open-source adoption (stars, PyPI downloads, dependents, external citations of VerifyDocBench); and (c) **judging the work of others** if you also serve as a reviewer for the workshop you submit to. Be realistic: USCIS requires meeting **at least three** criteria plus a holistic final-merits determination, and strong petitions usually document more, with independent corroboration (sustained citations, media, memberships, and recommendation letters). VerifyDoc is engineered to *generate* that downstream evidence (adoption → citations → letters), but **two projects alone are not a sufficient petition** — pair it with the second project and the complementary criteria.
-
----
-
 ## References (select; verify at write-up)
 
 - ExtractBench — arXiv:2602.12247 (SIGKDD 2026); code github.com/ContextualAI/extract-bench.

{verifydoc-0.3.0 → verifydoc-0.5.0}/README.md

@@ -36,7 +36,8 @@ pip install 'verifydoc[pdf]'   # + PDF/image ingestion
 ```python
 from verifydoc import verify
 
-result = verify("invoice.txt", schema="invoice_schema.json", k=3)
+# ready-to-run sample lives in examples/
+result = verify("examples/invoice.txt", schema="examples/invoice_schema.json")
 for f in result.fields:
     print(f"{f.path:12} = {f.value!r:24} conf={f.confidence:.2f} {f.decision}")
     if f.grounding:
@@ -44,10 +45,26 @@ for f in result.fields:
 ```
 
 ```bash
-verifydoc extract invoice.txt --schema invoice_schema.json --k 3 --threshold 0.8
+verifydoc extract examples/invoice.txt --schema examples/invoice_schema.json --threshold 0.8
 streamlit run ui/streamlit_app.py     # review UI: green/red fields + click-through to source
 ```
 
+See [`examples/`](examples/) for the full runnable walk-through.
+
+## For AI agents (MCP)
+
+Give any MCP-capable agent (Claude Desktop, IDEs, custom agents) a **trust
+layer for reading documents** — so it acts on confident, grounded fields and
+escalates the rest instead of hallucinating forward:
+
+```bash
+pip install 'verifydoc[mcp]'
+verifydoc-mcp     # stdio MCP server exposing verify_extraction()
+```
+
+Every field the agent extracts comes back with `confidence + grounding +
+accept/review`. See [docs/MCP.md](docs/MCP.md) for the one-line client config.
+
 Schemas are plain JSON Schema, with each leaf optionally declaring **how it is scored** (the executable-schema pattern):
 
 ```json
@@ -65,14 +82,37 @@ Schemas are plain JSON Schema, with each leaf optionally declaring **how it is s
 
 | Layer | Modules | Status |
 |---|---|---|
-| **Adapters** (all model code isolated here) | mock · text-search · PaddleOCR-VL · dots.ocr · Docling/MinerU output · API-VLM | ✅ |
+| **Adapters** (all model code isolated here) | mock · text-search · RapidOCR · PaddleOCR · dots.ocr · Docling/MinerU output · API-VLM (OpenAI/Anthropic) | ✅ |
 | **Confidence signals** | token-prob · verbalized · consensus (k-sample voting) · grounding-based · combined | ✅ |
-| **Calibrators** (fit on a dedicated split, never test) | temperature · Platt · isotonic · histogram · **split conformal** with finite-sample risk guarantee | ✅ |
+| **Calibrators** (fit on a dedicated split, never test) | temperature · Platt · isotonic · histogram · **split conformal** · **grounding-conditioned (Mondrian) conformal** (novel — recovers coverage a pooled threshold forfeits) | ✅ |
 | **Grounding** | value → page/bbox/char-span attachment with support scores | ✅ |
 | **Policy** | empirical & conformal accept thresholds for a target selective risk | ✅ |
 | **Eval harness / VerifyDocBench scorer** | Field-F1 · exact · CER/WER · ANLS · TEDS/TEDS-Struct · GriTS · omission vs hallucination · ECE/Adaptive-ECE/MCE/Brier/NLL/TCE · RC/AURC/E-AURC/Coverage@Risk/AUROC/AUPR/FPR@95 · box IoU/span-F1/grounding-conditioned correctness · bootstrap CIs + paired tests | ✅ |
 
-Every metric implements the exact definition in [PROJECT.md §5](PROJECT.md) with a hand-computed numeric regression test (200 tests, `eval/` coverage 98%).
+Every metric implements the exact definition in [PROJECT.md §5](PROJECT.md) with a hand-computed numeric regression test (201 tests, `eval/` coverage 98%).
+
+## Results on real documents
+
+Two independent real OCR extractors (**RapidOCR** and **PaddleOCR**) on real
+**CORD** receipts and **FUNSD** forms, scored by the harness
+([full tables + reading](paper/generated/REAL_MODELS_RESULTS.md)):
+
+| Confidence signal | ranks errors? | CORD AUROC (RapidOCR / PaddleOCR) |
+|---|---|---|
+| **learned combiner** | ✅ best | **0.89 / 0.84** |
+| **grounding** | ✅ strong | 0.82 / 0.74 |
+| token-probability | ~ moderate | 0.69 / 0.68 |
+| verbalized / consensus | ✗ uninformative | 0.50 / 0.50 |
+
+- **Grounding is a real trust signal:** grounded fields are **84–85% correct
+  vs ~1%** for ungrounded (gap ≈ +0.84; box accuracy @IoU 0.5 = 0.75–0.78).
+- **The abstention layer is honest:** with a weak field-extractor the base
+  error rate is high, so conformal abstention at a 2–5% budget correctly
+  refuses to auto-accept — you report *selective risk*, not headline accuracy.
+- The synthetic slice (strong extractor) shows the other end: Coverage@2% ≈ 1.0.
+
+The thesis holds on real data: **grounding + a learned fusion rank errors;
+self-reported and single-sample-consensus confidence do not.**
 
 ## The benchmark
 
@@ -82,16 +122,13 @@ make results     # regenerates every table/figure in paper/generated from config
 
 The harness runs **signals × calibrators × the full metric suite** with a
 document-level calibration split (disjointness asserted in code), bootstrap
-CIs, and a conformal-guarantee row. The repo ships a deterministic synthetic
-slice that runs in CI; loaders for CORD (and next: FUNSD, SROIE, DocILE,
-XFUND) extend it. Sample findings on the shipped slice
-([tables](paper/generated)):
-
-- **Verbalized self-confidence is badly miscalibrated** (the extractor says ~0.9 regardless of correctness) — exactly the failure mode reported for RLHF'd models.
-- **Consensus and grounding signals rank errors near-perfectly** (AUROC ≈ 0.98): corrupted values can't be traced back to the page, so grounding support collapses.
-- **The conformal row holds its guarantee** on every tested α, and reports the abstention it forces.
-
-These self-checks run as unit tests — the repo's core claims are CI-enforced, not just stated.
+CIs, and a conformal-guarantee row. It ships a deterministic **synthetic**
+slice (runs in CI) plus **CORD** and **FUNSD** loaders with gold source boxes;
+`extractor:` dispatches to any adapter (`rapidocr`, `paddleocr-vl`, …) and
+`dataset:` to any slice. See the [GPU runbook](docs/REAL_MODELS.md) to
+reproduce the real-model rows. Core claims (grounding beats verbalized;
+conformal holds its guarantee) are also **CI-enforced as unit tests**, not
+just stated.
 
 ## Why not just use the parser's own score?
 
@@ -104,9 +141,19 @@ buys you.
 ## Roadmap
 
 - [x] v0.1 — library + CLI + harness + synthetic benchmark slice + UI
-- [ ] CORD/FUNSD/SROIE slices with gold source boxes (VerifyDocBench v1)
-- [ ] Learned signal combiner + per-field-type calibration
-- [ ] Paper: first systematic study of confidence signals × calibration × abstention for document extraction
+- [x] v0.2 — CORD + FUNSD real slices with gold boxes; learned combiner; 1000× faster grounder
+- [x] v0.3 — real-model results (RapidOCR + PaddleOCR on CORD/FUNSD)
+- [x] v0.4 — vendor-neutral API-VLM extractor (OpenAI/Anthropic) with k-sample consensus; compilable paper with auto-generated tables
+- [x] v0.5 — **novel method** (grounding-conditioned conformal, +0.50 coverage at fixed risk) + **MCP server** (agent trust layer) + real frontier-VLM results
+- [ ] dots.ocr via vllm; SROIE / DocILE / XFUND slices; human-labeled correctness + IAA
+- [ ] Paper submission ([contributions welcome](CONTRIBUTING.md))
+
+## Documentation
+
+- [How it works](docs/how-it-works.md) — the pipeline, the abstention idea, why grounding is a trust signal
+- [Real-model results](paper/generated/REAL_MODELS_RESULTS.md) — RapidOCR + PaddleOCR numbers and reading
+- [GPU runbook](docs/REAL_MODELS.md) — reproduce the real-extractor rows
+- [USP audit](docs/USP.md) · [Paper draft](paper/main.tex) · [Full spec](PROJECT.md)
 
 ## Development
 
@@ -114,6 +161,8 @@ buys you.
 git clone https://github.com/bhaskargurram-ai/verifydoc && cd verifydoc
 uv venv .venv && uv pip install -e ".[dev]"
 make test lint typecheck     # all green before any PR (CI enforces)
+make results                 # regenerate benchmark tables + LaTeX
+make paper                   # compile the paper (needs a LaTeX toolchain)
 ```
 
 Contributions welcome — see the issues tagged `good-first-issue`. All model-specific code goes in `verifydoc/adapters/`; a new extractor is one file.

verifydoc-0.5.0/configs/cord-apivlm.yaml

@@ -0,0 +1,13 @@
+# API-VLM comparison row: a hosted VLM reads the CORD text layer and returns
+# schema fields with self-assessed (verbalized) confidence. k>1 enables real
+# self-consistency consensus (temperature sampling). Needs an API key:
+#   export ANTHROPIC_API_KEY=...   (or OPENAI_API_KEY with provider: openai)
+# Cost is a few dollars for 100 receipts at k=3. See docs/REAL_MODELS.md.
+dataset: cord
+split: validation
+limit: 100
+extractor: api-vlm
+k: 3                       # self-consistency: makes consensus + verbalized meaningful
+calibration_fraction: 0.5
+alphas: [0.02, 0.05]
+n_boot: 300