evalproof 0.2.0__tar.gz

evalproof-0.2.0/.gitignore

@@ -0,0 +1,16 @@
+.venv/
+__pycache__/
+*.pyc
+.pytest_cache/
+.mypy_cache/
+.ruff_cache/
+.coverage
+coverage.xml
+htmlcov/
+dist/
+build/
+*.egg-info/
+.evalproof/
+runs/
+!examples/**/runs/
+.DS_Store

evalproof-0.2.0/CHANGELOG.md

@@ -0,0 +1,99 @@
+# Changelog
+
+All notable changes are listed here. The project follows [Semantic Versioning](https://semver.org/).
+
+## [0.2.0] — 2026-10-16
+
+The first release on PyPI: `pip install evalproof`.
+
+### Added
+
+- A supported Python API: `from evalproof import compare_runs, decide, calibrate, ...`.
+  The names in `evalproof.__all__` are the public surface; other modules are internal.
+- Python 3.14 support, tested in CI alongside 3.11 to 3.13.
+- Releases publish to PyPI from GitHub Actions with trusted publishing, so no token is
+  stored anywhere. Each file on PyPI carries a PEP 740 attestation, and the GitHub release
+  carries the same files with a build-provenance attestation.
+- `SECURITY.md` with private vulnerability reporting and the trust boundaries, backed by a
+  test that runs both hosted providers with a fake key and finds it in no file written.
+- `CODE_OF_CONDUCT.md` (Contributor Covenant 2.1), issue and pull-request templates,
+  `CITATION.cff`, `CODEOWNERS`, Dependabot for actions and Python dependencies with a
+  three-day cooldown, and a pre-commit configuration.
+
+### Changed
+
+- The version is written once, in `src/evalproof/_version.py`; the build reads it.
+- The README published to PyPI rewrites relative links to GitHub and replaces the Mermaid
+  diagram with a link, so the PyPI page has no broken links. CI checks this on every
+  commit, along with `twine check --strict` and a clean install of the built wheel.
+- CI pins every action to a commit SHA, checks out without persisted credentials, sets
+  timeouts and cancels superseded pull-request runs, and audits the workflows with zizmor.
+  The composite action pins `setup-uv` the same way.
+- Package metadata: documentation, source, changelog and live-demo links; classifiers for
+  typing, Python 3.14 and beta status.
+
+## [0.1.2] — 2026-10-15
+
+### Added
+
+- `app/`: an interactive Streamlit app, live at https://evalproof.streamlit.app.
+  *Check a drop* starts from four scenarios (the recorded run, the same rates on 4× the
+  cases, a small but certain drop, a twenty-case suite) or from sliders, and shows the
+  gate's verdict, every paired case as a grid, the coin-flip distribution behind the exact
+  McNemar p-value, and the suite size at which the same drop would become significant.
+  *The recorded run* replays the committed `examples/support-qa` runs: select any of the
+  twelve changed answers to read both versions side by side, add a latency budget, and
+  calibrate the judge on a gold sample of chosen size. It runs the library from `src/`
+  through the same `compare_runs` and `decide` calls as the CLI; CI installs the app's own
+  requirements and drives every scenario.
+
+### Fixed
+
+- `evalproof compare` raised OverflowError when a metric had more than about 1,030
+  discordant pairs: the exact McNemar test summed `comb(n, i) * 0.5**n` in floating
+  point, and `comb(n, i)` outgrows the largest float. The binomial tail is now summed
+  as integers and divided once, so it stays exact at any size.
+
+## [0.1.1] — 2026-10-13
+
+### Fixed
+
+- The Wilson interval now returns exactly 0 at k=0 and exactly 1 at k=n. Computing
+  the boundaries by subtraction left platform-dependent rounding (2.8e-17 on x86_64
+  Linux, 0.0 on arm64 macOS).
+
+### Changed
+
+- The three-day `exclude-newer` resolution guard is declared in `[tool.uv]`, so
+  `uv.lock` means the same thing on every machine and in CI.
+- CI pins uv to the version that wrote `uv.lock`.
+
+## [0.1.0] — 2026-10-13
+
+First release.
+
+### Added
+
+- Suites in YAML: a JSONL golden set, a system under test, scorers, an optional judge,
+  calibration settings and a gate.
+- Systems under test: Ollama, Anthropic and OpenAI models with a prompt template; Python
+  functions; HTTP endpoints. System failures are recorded per case, never fatal.
+- Scorers: exact match, contains, regex with accept/reject answer keys, a JSON Schema
+  subset, numeric tolerance, recall@k, MRR, nDCG, and an LLM judge with a strict JSON
+  verdict, one parse retry and an explicit undecided state.
+- Pairwise judging with order swap; position-inconsistent verdicts become ties.
+- Judge calibration: confusion matrix, TPR, FPR, Cohen's κ, and prevalence-corrected pass
+  rates by adjusted classify-and-count (bootstrap interval) and prediction-powered
+  inference (normal interval).
+- Paired comparison of two runs: exact McNemar for pass/fail, paired bootstrap for scores.
+- A CI gate that fails only on a drop that is both beyond `max_drop` and significant at
+  `alpha`, plus p95-latency and cost budgets, with distinct exit codes.
+- Content-addressed response cache; runs record the Ollama version and model digests.
+- Markdown reports for GitHub step summaries, JSON reports, and a composite GitHub Action.
+- Two examples: an offline ticket router (CI's gate contract) and a support assistant for
+  a fictional product judged by a local model, with measurements.
+
+[0.2.0]: https://github.com/technobasant/evalproof/compare/v0.1.2...v0.2.0
+[0.1.2]: https://github.com/technobasant/evalproof/compare/v0.1.1...v0.1.2
+[0.1.1]: https://github.com/technobasant/evalproof/compare/v0.1.0...v0.1.1
+[0.1.0]: https://github.com/technobasant/evalproof/releases/tag/v0.1.0

evalproof-0.2.0/CITATION.cff

@@ -0,0 +1,25 @@
+cff-version: 1.2.0
+message: "If you use evalproof in research or a published evaluation, please cite it."
+title: "evalproof: LLM evaluation as measurement"
+abstract: >-
+  Calibrates an LLM judge against a small gold-labelled sample, corrects its pass rate with
+  adjusted classify-and-count and prediction-powered inference, compares runs on paired
+  cases with the exact McNemar test and a paired bootstrap, and gates CI only on
+  regressions that are both larger than a tolerance and statistically significant.
+type: software
+authors:
+  - family-names: Bhattarai
+    given-names: Basant
+    website: "https://basantbhattarai.com.np"
+version: 0.2.0
+date-released: "2026-10-16"
+license: Apache-2.0
+repository-code: "https://github.com/technobasant/evalproof"
+url: "https://pypi.org/project/evalproof/"
+keywords:
+  - llm-evaluation
+  - llm-as-a-judge
+  - calibration
+  - prediction-powered inference
+  - mcnemar test
+  - continuous integration

evalproof-0.2.0/LICENSE

@@ -0,0 +1,202 @@
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

evalproof-0.2.0/NOTICE

@@ -0,0 +1,4 @@
+evalproof
+Copyright 2026 Basant Bhattarai
+
+This product is licensed under the Apache License, Version 2.0 (see LICENSE).

evalproof-0.2.0/PKG-INFO

@@ -0,0 +1,279 @@
+Metadata-Version: 2.5
+Name: evalproof
+Version: 0.2.0
+Summary: Treat LLM evaluation as measurement: calibrate the judge, correct the pass rate, gate CI on significant regressions.
+Project-URL: Homepage, https://github.com/technobasant/evalproof
+Project-URL: Documentation, https://github.com/technobasant/evalproof#readme
+Project-URL: Source, https://github.com/technobasant/evalproof
+Project-URL: Issues, https://github.com/technobasant/evalproof/issues
+Project-URL: Changelog, https://github.com/technobasant/evalproof/blob/main/CHANGELOG.md
+Project-URL: Live demo, https://evalproof.streamlit.app
+Project-URL: Write-up, https://basantbhattarai.com.np/writing/eval-score-drop-twelve-answers/
+Author: Basant Bhattarai
+License-Expression: Apache-2.0
+License-File: LICENSE
+License-File: NOTICE
+Keywords: calibration,ci,evaluation,llm,llm-as-a-judge,statistics
+Classifier: Development Status :: 4 - Beta
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Software Development :: Quality Assurance
+Classifier: Topic :: Software Development :: Testing
+Classifier: Typing :: Typed
+Requires-Python: >=3.11
+Requires-Dist: httpx>=0.27
+Requires-Dist: numpy>=1.26
+Requires-Dist: pydantic>=2.7
+Requires-Dist: pyyaml>=6.0
+Description-Content-Type: text/markdown
+
+# evalproof
+
+[![PyPI](https://img.shields.io/pypi/v/evalproof)](https://pypi.org/project/evalproof/)
+[![Python](https://img.shields.io/pypi/pyversions/evalproof)](https://pypi.org/project/evalproof/)
+[![ci](https://github.com/technobasant/evalproof/actions/workflows/ci.yml/badge.svg)](https://github.com/technobasant/evalproof/actions/workflows/ci.yml)
+[![licence](https://img.shields.io/badge/licence-Apache--2.0-blue)](https://github.com/technobasant/evalproof/blob/main/LICENSE)
+[![Open in Streamlit](https://static.streamlit.io/badges/streamlit_badge_black_white.svg)](https://evalproof.streamlit.app)
+
+**Treat LLM evaluation as measurement.** Calibrate the judge, correct its pass rate, and
+gate CI on regressions that are real rather than noise.
+
+**Try it in the browser: [evalproof.streamlit.app](https://evalproof.streamlit.app).** Check whether
+a drop between two runs is real, or replay the run below and read the twelve answers that
+changed. It runs this repository's code; after a quiet spell it takes about half a minute to wake.
+
+An LLM judge is a classifier with error rates. If it passes bad answers at rate FPR and
+good answers at rate TPR, its pass rate converges to `FPR + (TPR − FPR) · p`, not to the
+true rate `p`. That is classify-and-count, and it is biased the way any classifier's count
+is — the effect is worked through with a seeded simulation in
+[*The classifier was 95% accurate. The trend it counted was a third too flat*](https://basantbhattarai.com.np/writing/count-prevalence-not-classifications/)
+([lab](https://github.com/technobasant/prevalence-shift)).
+
+evalproof:
+
+1. **measures the judge** against a small gold-labelled sample — confusion matrix, TPR,
+   FPR, Cohen's κ;
+2. **corrects its pass rate** with adjusted classify-and-count and prediction-powered
+   inference, each with a confidence interval;
+3. **compares runs on paired cases** — exact McNemar for pass/fail, paired bootstrap for
+   scores;
+4. **gates CI** only when a drop is both larger than your tolerance *and* statistically
+   significant, plus latency and cost budgets, with distinct exit codes.
+
+It is local-first: Ollama is the default provider, Anthropic and OpenAI are optional, and
+every model call is cached by content so re-runs are free and reproducible.
+
+## Install
+
+```bash
+pip install evalproof          # or: uv add evalproof
+uv tool install evalproof      # just the CLI
+```
+
+Python 3.11 to 3.14. `compare`, `calibrate` and `report` work on run files and need no
+model. `run` needs a system to evaluate: a Python function, an HTTP endpoint, a local
+Ollama model, or an Anthropic or OpenAI key.
+
+## What it found on a real run
+
+A local `qwen3:1.7b` support assistant for a fictional product, judged by `qwen3:8b`, 83
+questions, one Apple M5. Full method and every table in [MEASUREMENTS.md](https://github.com/technobasant/evalproof/blob/main/MEASUREMENTS.md).
+
+| Question | Measured |
+|---|---|
+| How good is the 8B judge? | TPR 92.6%, FPR 6.7%, Cohen's κ 0.78 against the answer key |
+| Did swapping the grounded prompt for a "friendly" one regress quality? | −4.8 pts on the judge, 8 cases worse vs 4 better, McNemar p = 0.39 — **gate warns, does not fail**; a delta-only gate would have failed it |
+| What did regress? | p95 latency, 1,469 → 4,422 ms; mean answer 19 → 77 tokens. A 2,500 ms budget fails the change with exit 3 |
+| Does correcting the judge's rate help? | On average yes (offset 4.8 → ~3.8 pts), but with 30 labels the interval doubles; at 83 cases the judge's bias is smaller than its own noise |
+| Was the gold trustworthy? | No: of 13 judge–key disagreements the key was wrong 7 times, the judge 4. Most of the "bias" corrected was the gold's |
+| Is pairwise judging safer? | No: 42% of verdicts flipped with the order of the two answers, and the judge preferred the candidate 42 to 6 — mostly where both answers were right and the candidate's was five times longer |
+| Is it reproducible? | 20/20 identical outputs and verdicts with the cache off; a warm-cache re-run took 0.05 s instead of 363 s |
+
+## Five-minute quickstart
+
+The offline example lives in the repository:
+
+```bash
+git clone https://github.com/technobasant/evalproof && cd evalproof
+uv sync
+
+# Offline: no model, no network. Four versions of a ticket router.
+for v in baseline regressed noisy refactored; do
+  uv run evalproof run examples/deterministic/suite.yaml \
+    --system "python:router:${v}_route" --out "runs/$v.json" --quiet
+done
+uv run evalproof compare runs/baseline.json runs/regressed.json \
+  --suite examples/deterministic/suite.yaml --gate; echo "exit $?"   # 1: real regression
+uv run evalproof compare runs/baseline.json runs/noisy.json \
+  --suite examples/deterministic/suite.yaml --gate; echo "exit $?"   # 0: 2 of 60 is noise
+```
+
+With Ollama installed, the LLM example is
+[`examples/support-qa`](https://github.com/technobasant/evalproof/blob/main/examples/support-qa) (`ollama pull qwen3:1.7b` and
+`ollama pull qwen3:8b`; about 6–10 minutes per run on an Apple M5).
+
+The browser app in [`app/`](https://github.com/technobasant/evalproof/blob/main/app) runs locally too:
+`uv run --with-requirements app/requirements.txt streamlit run app/streamlit_app.py`.
+
+## Concepts
+
+**Suite** — one YAML file: a JSONL golden set, the system under test, scorers, an optional
+judge, calibration settings and a gate.
+
+```yaml
+name: support-qa
+dataset: cases.jsonl              # {"id", "input", "reference", ...extra fields}
+system:
+  target: ollama:qwen3:1.7b       # or anthropic:…, openai:…, python:module:func, https://…
+  system_prompt_file: prompt.md
+  prompt_template: "{question}"
+  params: {temperature: 0, seed: 0, think: false}
+judge:
+  target: ollama:qwen3:8b
+scorers:
+  - {name: answer_key, type: regex, params: {any_of_field: accept, none_of_field: reject}}
+  - {name: judge, type: llm_judge, params: {rubric_file: rubric.md}}
+calibration: {judge_scorer: judge, gold_scorer: answer_key, gold_sample: 30}
+gate: {metric: judge, max_drop: 0.03, alpha: 0.05, min_cases: 40, max_p95_latency_ms: 2500}
+```
+
+**Run** — `evalproof run suite.yaml` writes one self-describing JSON file: every case's
+output, latency, tokens and scores, plus the dataset hash, seed, Ollama version and model
+digests.
+
+**Scorers** — `exact_match`, `contains`, `regex` (accept/reject answer keys),
+`json_schema`, `numeric_tolerance`, `recall_at_k`, `mrr`, `ndcg`, and `llm_judge`: a strict
+JSON verdict, one retry on an unparseable reply, and an explicit *undecided* state that is
+counted, never scored as a pass. `evalproof pairwise` judges two runs head to head in both
+orders and reports order-dependent verdicts as ties.
+
+**Calibrate** — `evalproof calibrate run.json --judge-scorer judge --labels gold.jsonl`
+(human labels) or `--gold-scorer answer_key --gold-sample 30`:
+
+| Estimate | What it is |
+|---|---|
+| raw judge pass rate | classify-and-count, Wilson interval |
+| gold labels only | the labelled sample alone |
+| ACC | `(q − FPR) / (TPR − FPR)`, bootstrap interval over both the judged set and the gold sample |
+| PPI | judge mean on unlabelled cases + mean (label − judge) on the gold sample, normal interval |
+
+**Compare and gate** — `evalproof compare baseline.json candidate.json --suite suite.yaml --gate`
+
+| Exit | Meaning |
+|---|---|
+| 0 | pass — possibly with warnings (a big drop that is not significant; a significant drop within tolerance) |
+| 1 | quality regression: drop ≥ `max_drop` **and** p < `alpha` |
+| 3 | budget breach: candidate p95 latency or cost over the limit |
+| 4 | cannot gate: fewer than `min_cases` paired cases, or the runs used different data |
+| 2 | usage or configuration error |
+
+## From Python
+
+The CLI is a thin layer over the same functions:
+
+```python
+from evalproof import GateConfig, calibrate, compare_runs, decide, load_run
+
+baseline = load_run("runs/baseline.json")
+candidate = load_run("runs/candidate.json")
+
+cmp = compare_runs(baseline, candidate)  # paired cases: exact McNemar, bootstrap intervals
+decision = decide(cmp, GateConfig(metric="judge", max_drop=0.03))
+print(decision.status, decision.exit_code, decision.warnings)
+
+report = calibrate(baseline, "judge", gold_scorer="answer_key", gold_sample=30)
+for est in report.estimates:  # raw judge rate, gold only, ACC, PPI
+    print(est.method, est.point, est.low, est.high)
+```
+
+The names in `evalproof.__all__` are the supported API. Other modules are internal and may
+change between minor versions; see the [changelog](https://github.com/technobasant/evalproof/blob/main/CHANGELOG.md).
+
+## Architecture
+
+The architecture diagram is in the [README on GitHub](https://github.com/technobasant/evalproof#architecture).
+
+## Decisions
+
+| Decision | Alternative | Why | ADR |
+|---|---|---|---|
+| Treat the judge as a classifier to be calibrated | Trust the judge's pass rate | Its count is biased by its own error rates, and the bias moves with the true rate | [0001](https://github.com/technobasant/evalproof/blob/main/docs/adr/0001-judge-is-a-classifier.md) |
+| Local Ollama by default, HTTP-only providers, content-addressed cache | Hosted APIs and vendor SDKs | Reproducible, free to re-run, tied to a model digest; no SDK dependency | [0002](https://github.com/technobasant/evalproof/blob/main/docs/adr/0002-local-first-providers.md) |
+| Gate on "large AND significant", on paired cases | "Fail if the score dropped by X" | On a few dozen cases, X-point drops are mostly noise, and a noisy gate gets switched off | [0003](https://github.com/technobasant/evalproof/blob/main/docs/adr/0003-gate-on-significance-not-delta.md) |
+| Gate one metric; report the rest | Gate every metric | Testing many metrics at α = 0.05 each inflates false failures | 0003 |
+| Pointwise verdicts for gating; pairwise as a diagnostic | Pairwise preference for gating | Measured: 42% of pairwise verdicts flipped with answer order, with a strong length preference | [MEASUREMENTS §6](https://github.com/technobasant/evalproof/blob/main/MEASUREMENTS.md) |
+| numpy and the standard library for statistics | SciPy, statsmodels | Small enough to read and to test against hand-worked values | — |
+
+## In CI
+
+The composite action runs `compare --gate`, appends the report to the job summary, and
+fails the job on exit codes 1, 3 or 4:
+
+```yaml
+- uses: technobasant/evalproof@v0.2.0
+  with:
+    baseline: eval/baseline.json     # produced on main, e.g. as a build artifact
+    candidate: eval/candidate.json   # produced on this branch
+    suite: eval/suite.yaml
+```
+
+This repository's own CI runs the offline example on every commit and asserts that the
+gate returns 1 for the real regression and 0 for the noisy and refactored candidates.
+
+## How it relates to other tools
+
+[promptfoo](https://www.promptfoo.dev/docs/intro/) (open-source, local evals of prompts,
+models and RAG pipelines with deterministic and model-graded assertions, a GitHub Action,
+and red teaming), [DeepEval](https://deepeval.com/docs/getting-started) (pytest-style
+LLM tests with research-backed metrics such as G-Eval), [Ragas](https://docs.ragas.io/en/stable/)
+(LLM-driven metrics and experiments for LLM and RAG applications) and
+[Inspect](https://inspect.aisi.org.uk/) (the UK AI Security Institute's framework for
+model and agent evaluations, with sandboxed tools and a log viewer) are broad evaluation
+frameworks.
+
+evalproof is deliberately narrow: it is the measurement layer — how much a judge's verdict
+is worth, and whether a difference between two runs is real. Its run file is plain,
+documented JSON, so it can sit beside an existing harness: produce the per-case verdicts
+however you like, then calibrate and gate here.
+
+## Limits
+
+* The measurements cover one machine, two small local models, one fictional domain and
+  83 cases. They show the method; they are not a benchmark of those models.
+* Calibration is only as good as its gold labels — [MEASUREMENTS §4](https://github.com/technobasant/evalproof/blob/main/MEASUREMENTS.md)
+  shows a regex answer key being wrong more often than the judge. Use human labels when
+  the decision matters.
+* ACC needs both classes in the gold sample and a judge better than chance; PPI needs the
+  gold sample to be random. evalproof reports when either does not hold rather than
+  printing a number.
+* The percentile-bootstrap p-value for continuous scores is an approximation; pass/fail
+  metrics use the exact McNemar test.
+* Cost tracking needs a price table in the suite for hosted models.
+
+## Roadmap
+
+* Stratified gold sampling (by tag) and per-tag calibration.
+* Multiple-comparison correction when more than one metric is gated.
+* A reasoning judge (`think: true`) measured against the same answer key.
+* Importers for other harnesses' per-case results.
+
+## Project
+
+[Changelog](https://github.com/technobasant/evalproof/blob/main/CHANGELOG.md) · [Contributing](https://github.com/technobasant/evalproof/blob/main/CONTRIBUTING.md) ·
+[Security policy](https://github.com/technobasant/evalproof/blob/main/SECURITY.md) · [Code of conduct](https://github.com/technobasant/evalproof/blob/main/CODE_OF_CONDUCT.md) ·
+[How to cite](https://github.com/technobasant/evalproof/blob/main/CITATION.cff). Releases are published to PyPI from GitHub Actions with
+[trusted publishing](https://docs.pypi.org/trusted-publishers/), and each file carries a
+signed provenance attestation.
+
+## Licence
+
+Apache-2.0. See [LICENSE](https://github.com/technobasant/evalproof/blob/main/LICENSE) and [NOTICE](https://github.com/technobasant/evalproof/blob/main/NOTICE).