evalproof 0.2.1__tar.gz → 0.3.0__tar.gz

{evalproof-0.2.1 → evalproof-0.3.0}/CHANGELOG.md

@@ -2,6 +2,81 @@
 
 All notable changes are listed here. The project follows [Semantic Versioning](https://semver.org/).
 
+## [Unreleased]
+
+## [0.3.0] — 2026-11-12
+
+### Added
+
+- `evalproof import` and `evalproof.import_run`: per-case results from promptfoo (JSON and
+  JSONL), Inspect (`.json` and `.eval` logs), DeepEval, lm-evaluation-harness
+  (`samples_*.jsonl`) and generic long or wide `jsonl` and `csv` files become a run that
+  `compare` and the gate read. Case ids are stable across runs, system errors score as
+  failures, repeats collapse by `--epochs`, and NaN or infinite scores are refused with the
+  case and metric named. Raises `evalproof.ImportFormatError` (a `ValueError`; exit 2 on the
+  command line).
+- Gating several metrics: `gate.metrics` in the suite or `--metric` repeated on
+  `compare`, with Holm-adjusted p-values (`evalproof.holm`). The decision carries
+  `adjusted_p`, the report gains a Holm-adjusted p column, and a single metric behaves as
+  before. ADR 0003 has the reasoning.
+- The composite action's `metric` input takes several names, separated by newlines or commas.
+- `examples/retrieval`: an offline retrieval example for recall@k, MRR and nDCG, with a
+  regression that recall@5 cannot see and MRR and nDCG@5 can. Contributed by
+  @HarshRajSinghania in #5. CI asserts its gate exits 1 on the regression and 0 on no change.
+- MEASUREMENTS §8: the support-qa judge with thinking on (#3), with its runs, suite and
+  analysis in `measure.py`. Against labels audited case by case it passes no wrong answer
+  and κ rises, at 6.7× the judge time; the gate's decision does not change.
+- A provider profile table (`providers/profiles.py`) behind one OpenAI-compatible adapter:
+  `openai`, `azure`, `gemini`, `mistral`, `groq`, `together`, `fireworks`, `deepseek`, `xai`,
+  `openrouter`, `cerebras`, the local servers `vllm`, `lmstudio` and `llamacpp`, and
+  `openai-compatible` for any other server (set `base_url`). Each profile carries its base
+  URL, key variable, auth header, token-limit field and a link to the provider's docs.
+  `OpenAIProvider` stays importable.
+- `TargetConfig.api_key_env` names the environment variable that holds the key, and
+  `TargetConfig.headers` adds request headers to every HTTP provider; suite headers win over
+  a provider's own, and neither is part of the cache key or written to a run. A named key
+  variable that is not set is an error rather than a 401.
+- Anthropic: a `refusal` stop reason is recorded as a case error naming its category, and so
+  is `max_tokens` reached before any text, with a hint to raise `params.max_tokens`. The
+  OpenAI-compatible adapter does the same for `finish_reason` `length` with no text, and
+  for `content_filter`, and reads replies that arrive as a list of typed content parts.
+- A test pins the cache key of an existing Ollama call, so a change to it cannot slip in.
+
+### Changed
+
+- Hosted providers (everything except Ollama and the local servers) send `temperature` and
+  `seed` only when the suite sets them. Before, `openai` sent `temperature: 0` and
+  `seed: 0` on every call, which reasoning models reject; now an unset value is left to the
+  provider's default. A hosted run that relied on the implicit `temperature: 0` should set
+  it in `params`. Ollama and the local servers still send both, so local runs repeat as
+  before and keep their cache keys; `GenerationParams` defaults are unchanged. A hosted call
+  that now leaves a value out gets a new cache key, so a reply cached at the old implicit
+  `temperature: 0` is not reused for it. To point `openai` at an older local server through
+  `base_url`, use `openai-compatible` or a local profile instead, which send both values.
+- `openai` sends the token limit as `max_completion_tokens`, which OpenAI's reasoning models
+  require (they reject `max_tokens`), and `mistral` sends its seed as `random_seed`. The
+  default limit is still 512.
+- The llama.cpp profile defaults to port 9931, the one its server documents now; for an older
+  build on 8080, set `base_url`.
+
+### Fixed
+
+- Anthropic calls to current Claude models failed with HTTP 400, because `temperature` was
+  sent on every request. It goes out only when the suite sets it, and no seed is sent, since
+  the Messages API has none.
+- A gate built without validation and given no metric passed by gating nothing; it now
+  cannot gate (exit 4), and `compare --metric ""` is a usage error (exit 2).
+- Found in review before release, in the new importers: two promptfoo prompts whose first 60
+  characters matched merged into one system (the label is no longer truncated, and two
+  prompts that share a label are an error), promptfoo output with repeated tests but no
+  `repeatIndex` now warns that it reads as separate cases, Inspect message ids no longer
+  change the dataset hash, a `null` promptfoo `testIdx` no longer crashes, and under
+  `--epochs any` or `mean` a case with one good repeat is no longer marked as an error.
+- `examples/support-qa/cases.jsonl`: the q21 and q22 references named Enterprise, which the
+  product facts do not give SSO or an admin console (#7). The original set is kept as
+  `cases.v1.jsonl` so the recorded runs stay reproducible; the runs judged on the revised set
+  are in `runs/v2/`, and MEASUREMENTS §8 reports both.
+
 ## [0.2.1] — 2026-10-16
 
 ### Fixed
@@ -26,7 +101,7 @@ The first release on PyPI: `pip install evalproof`.
   stored anywhere. Each file on PyPI carries a PEP 740 attestation, and the GitHub release
   carries the same files with a build-provenance attestation.
 - `SECURITY.md` with private vulnerability reporting and the trust boundaries, backed by a
-  test that runs both hosted providers with a fake key and finds it in no file written.
+  test that runs both hosted providers with a placeholder key and finds it in no file written.
 - `CODE_OF_CONDUCT.md` (Contributor Covenant 2.1), issue and pull-request templates,
   `CITATION.cff`, `CODEOWNERS`, Dependabot for actions and Python dependencies with a
   three-day cooldown, and a pre-commit configuration.
@@ -101,9 +176,11 @@ First release.
   `alpha`, plus p95-latency and cost budgets, with distinct exit codes.
 - Content-addressed response cache; runs record the Ollama version and model digests.
 - Markdown reports for GitHub step summaries, JSON reports, and a composite GitHub Action.
-- Two examples: an offline ticket router (CI's gate contract) and a support assistant for
-  a fictional product judged by a local model, with measurements.
+- Two examples: an offline ticket router (CI's gate contract) and a product support
+  assistant judged by a local model, with measurements.
 
+[Unreleased]: https://github.com/technobasant/evalproof/compare/v0.3.0...HEAD
+[0.3.0]: https://github.com/technobasant/evalproof/compare/v0.2.1...v0.3.0
 [0.2.1]: https://github.com/technobasant/evalproof/compare/v0.2.0...v0.2.1
 [0.2.0]: https://github.com/technobasant/evalproof/compare/v0.1.2...v0.2.0
 [0.1.2]: https://github.com/technobasant/evalproof/compare/v0.1.1...v0.1.2

{evalproof-0.2.1 → evalproof-0.3.0}/CITATION.cff

@@ -11,8 +11,8 @@ authors:
   - family-names: Bhattarai
     given-names: Basant
     website: "https://basantbhattarai.com.np"
-version: 0.2.1
-date-released: "2026-10-16"
+version: 0.3.0
+date-released: "2026-11-12"
 license: Apache-2.0
 repository-code: "https://github.com/technobasant/evalproof"
 url: "https://pypi.org/project/evalproof/"

{evalproof-0.2.1 → evalproof-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: evalproof
-Version: 0.2.1
+Version: 0.3.0
 Summary: Treat LLM evaluation as measurement: calibrate the judge, correct the pass rate, gate CI on significant regressions.
 Project-URL: Homepage, https://github.com/technobasant/evalproof
 Project-URL: Documentation, https://github.com/technobasant/evalproof#readme
@@ -67,8 +67,9 @@ evalproof:
 4. **gates CI** only when a drop is both larger than your tolerance *and* statistically
    significant, plus latency and cost budgets, with distinct exit codes.
 
-It is local-first: Ollama is the default provider, Anthropic and OpenAI are optional, and
-every model call is cached by content so re-runs are free and reproducible.
+It is local-first: Ollama is the default, every major hosted provider is one `target:` line
+away ([Providers](#providers)), and every model call is cached by content so re-runs are free
+and reproducible.
 
 ## Install
 
@@ -78,12 +79,12 @@ uv tool install evalproof      # just the CLI
 ```
 
 Python 3.11 to 3.14. `compare`, `calibrate` and `report` work on run files and need no
-model. `run` needs a system to evaluate: a Python function, an HTTP endpoint, a local
-Ollama model, or an Anthropic or OpenAI key.
+model. `run` needs a system to evaluate: a Python function, an HTTP endpoint, a local model
+(Ollama, vLLM, LM Studio, llama.cpp), or a key for a hosted provider.
 
 ## What it found on a real run
 
-A local `qwen3:1.7b` support assistant for a fictional product, judged by `qwen3:8b`, 83
+A local `qwen3:1.7b` product support assistant, judged by `qwen3:8b`, 83
 questions, one Apple M5. Full method and every table in [MEASUREMENTS.md](https://github.com/technobasant/evalproof/blob/main/MEASUREMENTS.md).
 
 | Question | Measured |
@@ -94,7 +95,8 @@ questions, one Apple M5. Full method and every table in [MEASUREMENTS.md](https:
 | Does correcting the judge's rate help? | On average yes (offset 4.8 → ~3.8 pts), but with 30 labels the interval doubles; at 83 cases the judge's bias is smaller than its own noise |
 | Was the gold trustworthy? | No: of 13 judge–key disagreements the key was wrong 7 times, the judge 4. Most of the "bias" corrected was the gold's |
 | Is pairwise judging safer? | No: 42% of verdicts flipped with the order of the two answers, and the judge preferred the candidate 42 to 6 — mostly where both answers were right and the candidate's was five times longer |
-| Is it reproducible? | 20/20 identical outputs and verdicts with the cache off; a warm-cache re-run took 0.05 s instead of 363 s |
+| Does a reasoning judge help? | Against audited labels, yes: thinking on passed no wrong answer (three without it) and κ rose from 0.86–0.88 to 0.90–0.94, at 6.7× the judge time. Against the raw answer key it looked worse, because the key is wrong on exactly those cases. The gate's decision did not change |
+| Is it reproducible? | 20/20 identical outputs and verdicts with the cache off; a warm-cache re-run took 0.05 s instead of 363 s; a re-judge three days later matched 83/83 |
 
 ## Five-minute quickstart
 
@@ -115,6 +117,11 @@ uv run evalproof compare runs/baseline.json runs/noisy.json \
   --suite examples/deterministic/suite.yaml --gate; echo "exit $?"   # 0: 2 of 60 is noise
 ```
 
+[`examples/retrieval`](https://github.com/technobasant/evalproof/blob/main/examples/retrieval) runs the same gate on a retrieval change: recall@5
+does not move when the top hit drops to rank 5, while MRR and nDCG@5 do, and the gate fails
+it. It was contributed by [@HarshRajSinghania](https://github.com/HarshRajSinghania), and CI
+checks its exit codes too.
+
 With Ollama installed, the LLM example is
 [`examples/support-qa`](https://github.com/technobasant/evalproof/blob/main/examples/support-qa) (`ollama pull qwen3:1.7b` and
 `ollama pull qwen3:8b`; about 6–10 minutes per run on an Apple M5).
@@ -122,6 +129,63 @@ With Ollama installed, the LLM example is
 The browser app in [`app/`](https://github.com/technobasant/evalproof/blob/main/app) runs locally too:
 `uv run --with-requirements app/requirements.txt streamlit run app/streamlit_app.py`.
 
+## Providers
+
+The target is `<provider>:<model>`; the model name is everything after the first colon, so
+`openrouter:anthropic/claude-sonnet-4.5` and `ollama:qwen3:8b` both work. Two native adapters
+(`ollama`, `anthropic`) and one OpenAI-compatible adapter driven by a table of provider
+profiles ([`providers/profiles.py`](https://github.com/technobasant/evalproof/blob/main/src/evalproof/providers/profiles.py), each row cites the
+provider's documentation). Plain HTTP throughout, no vendor SDK.
+
+| Provider | Example target | Key variable | Where |
+|---|---|---|---|
+| `ollama` | `ollama:qwen3:8b` | none | local |
+| `vllm` | `vllm:meta-llama/Llama-3.1-8B-Instruct` | `VLLM_API_KEY`, optional | local |
+| `lmstudio` | `lmstudio:qwen3-8b` | optional, via `api_key_env` | local |
+| `llamacpp` | `llamacpp:qwen3-8b` | `LLAMA_API_KEY`, optional | local |
+| `anthropic` | `anthropic:claude-sonnet-4-5` | `ANTHROPIC_API_KEY` | hosted |
+| `openai` | `openai:gpt-5` | `OPENAI_API_KEY` | hosted |
+| `azure` | `azure:<deployment>` | `AZURE_OPENAI_API_KEY` | hosted, needs `base_url` |
+| `gemini` | `gemini:gemini-2.5-flash` | `GEMINI_API_KEY` | hosted |
+| `mistral` | `mistral:mistral-large-latest` | `MISTRAL_API_KEY` | hosted |
+| `groq` | `groq:llama-3.3-70b-versatile` | `GROQ_API_KEY` | hosted |
+| `together` | `together:meta-llama/Llama-3.3-70B-Instruct-Turbo` | `TOGETHER_API_KEY` | hosted |
+| `fireworks` | `fireworks:accounts/fireworks/models/llama-v3p3-70b-instruct` | `FIREWORKS_API_KEY` | hosted |
+| `deepseek` | `deepseek:deepseek-chat` | `DEEPSEEK_API_KEY` | hosted |
+| `xai` | `xai:grok-4` | `XAI_API_KEY` | hosted |
+| `openrouter` | `openrouter:anthropic/claude-sonnet-4.5` | `OPENROUTER_API_KEY` | hosted |
+| `cerebras` | `cerebras:gpt-oss-120b` | `CEREBRAS_API_KEY` | hosted |
+| `openai-compatible` | `openai-compatible:<model>` | optional, via `api_key_env` | any server, needs `base_url` |
+
+```yaml
+system:
+  target: openai-compatible:gpt-4o    # a gateway, LiteLLM proxy, TGI, SGLang, ...
+  base_url: https://llm-gateway.internal/v1
+  api_key_env: TEAM_GATEWAY_KEY       # the variable to read; the key itself never goes in the suite
+  headers: {x-team: search}           # extra HTTP headers on every call, never secrets
+  params: {max_tokens: 2048}
+```
+
+* `base_url` overrides any profile's default (a regional endpoint, a proxy, an older
+  `llama-server`, which listened on port 8080 before October 2026 and now defaults to 9931).
+* `api_key_env` replaces the profile's variable for any provider; a variable named there
+  must be set.
+* `headers` merge into every request and win over the adapter's own, for example
+  `anthropic-version`. They are not part of the cache key, and a suite file is usually
+  committed, so keep keys in the environment.
+* Sampling: local servers (`ollama`, `vllm`, `lmstudio`, `llamacpp`) receive `temperature` and
+  `seed` on every call, defaults included, so a local run repeats. Hosted providers receive
+  them only when `params` sets them, because reasoning models reject a non-default
+  temperature; Anthropic never receives a seed, and since its current models also reject
+  `temperature`, it gets one only when the suite asks. `params.extra` adds any other body
+  field, such as `{thinking: {type: adaptive}, output_config: {effort: low}}`.
+* Thinking models spend output tokens on thinking inside `max_tokens`, which defaults to 512
+  (changing it would change every cache key). Set `params.max_tokens` higher for them; a reply
+  cut off before its first word, a refusal, or a content-filter block is recorded as that
+  case's error rather than as an empty answer.
+* AWS Bedrock and Google Vertex AI sign requests with cloud credentials, which evalproof does
+  not implement. Use a gateway's OpenAI-compatible endpoint through `openai-compatible`.
+
 ## Concepts
 
 **Suite** — one YAML file: a JSONL golden set, the system under test, scorers, an optional
@@ -131,7 +195,7 @@ judge, calibration settings and a gate.
 name: support-qa
 dataset: cases.jsonl              # {"id", "input", "reference", ...extra fields}
 system:
-  target: ollama:qwen3:1.7b       # or anthropic:…, openai:…, python:module:func, https://…
+  target: ollama:qwen3:1.7b       # or groq:…, anthropic:…, python:module:func, https://…
   system_prompt_file: prompt.md
   prompt_template: "{question}"
   params: {temperature: 0, seed: 0, think: false}
@@ -174,6 +238,56 @@ orders and reports order-dependent verdicts as ties.
 | 4 | cannot gate: fewer than `min_cases` paired cases, or the runs used different data |
 | 2 | usage or configuration error |
 
+The gate checks one metric by default. To gate several, repeat `--metric` or list them in
+the suite; each metric's p-value is then Holm-adjusted for the number of metrics gated, so
+the chance of failing a build on noise stays at `alpha` for the set as a whole, and the
+report shows the raw and the adjusted p-value side by side. Metrics that are not gated are
+still reported.
+
+```yaml
+gate:
+  metrics: [judge, answer_key]   # one metric: `metric: judge`
+  max_drop: 0.03
+  alpha: 0.05
+  min_cases: 40
+```
+
+```bash
+evalproof compare baseline.json candidate.json --gate --metric judge --metric answer_key
+```
+
+## Import results from other tools
+
+You do not have to run your evals with evalproof to gate them. `evalproof import` turns the
+per-case results another tool wrote into a run file that `compare` and the gate read:
+
+```bash
+evalproof import promptfoo output.json --system openai:gpt-5 --out runs/candidate.json
+evalproof import inspect logs/task.eval --out runs/candidate.json
+evalproof import deepeval .deepeval/.latest_test_run.json --out runs/candidate.json
+evalproof import lm-eval results/gpt2/samples_hellaswag_2026-11-12T10-20-30.jsonl --out runs/gpt2.json
+evalproof import csv results.csv --system a --out runs/a.json       # id,metric,passed,score,...
+evalproof import jsonl results.jsonl --wide --id-field case --out runs/a.json  # one column per metric
+```
+
+Import each side with the same importer, then gate as usual, for example
+`evalproof compare runs/baseline.json runs/candidate.json --gate --metric pass`. What to know:
+
+* Case ids are stable between runs of the same dataset (they come from the test's inputs, or
+  the tool's own sample id, never from its position), which is what lets two files pair.
+  The dataset hash covers the ids and inputs as the tool recorded them, so runs imported by
+  different importers refuse to gate (exit 4) instead of pairing on a coincidence.
+* A system error counts as a failure on every metric, as in a native run. Repeats of one case
+  (epochs, repeated tests) collapse by `--epochs`: `mean` (the default), `first`, `all`
+  or `any`. Under `mean`, `first` and `any`, a case with at least one repeat that ran is not
+  counted as a system error; its errored repeats still score as failures.
+* A NaN or infinite score is refused at import, naming the case and metric, because it would
+  make the bootstrap report a significant difference that is not there.
+* promptfoo files can hold several providers and `--system` picks one. Inspect `.eval` logs
+  compressed with zstd need Python 3.14 or newer; otherwise run
+  `inspect log convert --to json <file>` first. DeepEval's conversational cases are skipped
+  with a warning.
+
 ## From Python
 
 The CLI is a thin layer over the same functions:
@@ -207,7 +321,7 @@ The architecture diagram is in the [README on GitHub](https://github.com/technob
 | Treat the judge as a classifier to be calibrated | Trust the judge's pass rate | Its count is biased by its own error rates, and the bias moves with the true rate | [0001](https://github.com/technobasant/evalproof/blob/main/docs/adr/0001-judge-is-a-classifier.md) |
 | Local Ollama by default, HTTP-only providers, content-addressed cache | Hosted APIs and vendor SDKs | Reproducible, free to re-run, tied to a model digest; no SDK dependency | [0002](https://github.com/technobasant/evalproof/blob/main/docs/adr/0002-local-first-providers.md) |
 | Gate on "large AND significant", on paired cases | "Fail if the score dropped by X" | On a few dozen cases, X-point drops are mostly noise, and a noisy gate gets switched off | [0003](https://github.com/technobasant/evalproof/blob/main/docs/adr/0003-gate-on-significance-not-delta.md) |
-| Gate one metric; report the rest | Gate every metric | Testing many metrics at α = 0.05 each inflates false failures | 0003 |
+| One metric by default; several with Holm correction | Gate every metric at α each | Testing many metrics at α = 0.05 each inflates false failures | 0003 |
 | Pointwise verdicts for gating; pairwise as a diagnostic | Pairwise preference for gating | Measured: 42% of pairwise verdicts flipped with answer order, with a strong length preference | [MEASUREMENTS §6](https://github.com/technobasant/evalproof/blob/main/MEASUREMENTS.md) |
 | numpy and the standard library for statistics | SciPy, statsmodels | Small enough to read and to test against hand-worked values | — |
 
@@ -217,11 +331,12 @@ The composite action runs `compare --gate`, appends the report to the job summar
 fails the job on exit codes 1, 3 or 4:
 
 ```yaml
-- uses: technobasant/evalproof@v0.2.0
+- uses: technobasant/evalproof@v0.3.0
   with:
     baseline: eval/baseline.json     # produced on main, e.g. as a build artifact
     candidate: eval/candidate.json   # produced on this branch
     suite: eval/suite.yaml
+    # metric: "judge, answer_key"    # optional: gate several metrics, newline- or comma-separated
 ```
 
 This repository's own CI runs the offline example on every commit and asserts that the
@@ -241,11 +356,12 @@ frameworks.
 evalproof is deliberately narrow: it is the measurement layer — how much a judge's verdict
 is worth, and whether a difference between two runs is real. Its run file is plain,
 documented JSON, so it can sit beside an existing harness: produce the per-case verdicts
-however you like, then calibrate and gate here.
+however you like, then calibrate and gate here. For promptfoo, Inspect, DeepEval and
+lm-eval you do not even need to do that: evalproof [imports their results](#import-results-from-other-tools).
 
 ## Limits
 
-* The measurements cover one machine, two small local models, one fictional domain and
+* The measurements cover one machine, two small local models, one product domain and
   83 cases. They show the method; they are not a benchmark of those models.
 * Calibration is only as good as its gold labels — [MEASUREMENTS §4](https://github.com/technobasant/evalproof/blob/main/MEASUREMENTS.md)
   shows a regex answer key being wrong more often than the judge. Use human labels when
@@ -260,9 +376,6 @@ however you like, then calibrate and gate here.
 ## Roadmap
 
 * Stratified gold sampling (by tag) and per-tag calibration.
-* Multiple-comparison correction when more than one metric is gated.
-* A reasoning judge (`think: true`) measured against the same answer key.
-* Importers for other harnesses' per-case results.
 
 ## Project
 

{evalproof-0.2.1 → evalproof-0.3.0}/README.md

@@ -31,8 +31,9 @@ evalproof:
 4. **gates CI** only when a drop is both larger than your tolerance *and* statistically
    significant, plus latency and cost budgets, with distinct exit codes.
 
-It is local-first: Ollama is the default provider, Anthropic and OpenAI are optional, and
-every model call is cached by content so re-runs are free and reproducible.
+It is local-first: Ollama is the default, every major hosted provider is one `target:` line
+away ([Providers](#providers)), and every model call is cached by content so re-runs are free
+and reproducible.
 
 ## Install
 
@@ -42,12 +43,12 @@ uv tool install evalproof      # just the CLI
 ```
 
 Python 3.11 to 3.14. `compare`, `calibrate` and `report` work on run files and need no
-model. `run` needs a system to evaluate: a Python function, an HTTP endpoint, a local
-Ollama model, or an Anthropic or OpenAI key.
+model. `run` needs a system to evaluate: a Python function, an HTTP endpoint, a local model
+(Ollama, vLLM, LM Studio, llama.cpp), or a key for a hosted provider.
 
 ## What it found on a real run
 
-A local `qwen3:1.7b` support assistant for a fictional product, judged by `qwen3:8b`, 83
+A local `qwen3:1.7b` product support assistant, judged by `qwen3:8b`, 83
 questions, one Apple M5. Full method and every table in [MEASUREMENTS.md](MEASUREMENTS.md).
 
 | Question | Measured |
@@ -58,7 +59,8 @@ questions, one Apple M5. Full method and every table in [MEASUREMENTS.md](MEASUR
 | Does correcting the judge's rate help? | On average yes (offset 4.8 → ~3.8 pts), but with 30 labels the interval doubles; at 83 cases the judge's bias is smaller than its own noise |
 | Was the gold trustworthy? | No: of 13 judge–key disagreements the key was wrong 7 times, the judge 4. Most of the "bias" corrected was the gold's |
 | Is pairwise judging safer? | No: 42% of verdicts flipped with the order of the two answers, and the judge preferred the candidate 42 to 6 — mostly where both answers were right and the candidate's was five times longer |
-| Is it reproducible? | 20/20 identical outputs and verdicts with the cache off; a warm-cache re-run took 0.05 s instead of 363 s |
+| Does a reasoning judge help? | Against audited labels, yes: thinking on passed no wrong answer (three without it) and κ rose from 0.86–0.88 to 0.90–0.94, at 6.7× the judge time. Against the raw answer key it looked worse, because the key is wrong on exactly those cases. The gate's decision did not change |
+| Is it reproducible? | 20/20 identical outputs and verdicts with the cache off; a warm-cache re-run took 0.05 s instead of 363 s; a re-judge three days later matched 83/83 |
 
 ## Five-minute quickstart
 
@@ -79,6 +81,11 @@ uv run evalproof compare runs/baseline.json runs/noisy.json \
   --suite examples/deterministic/suite.yaml --gate; echo "exit $?"   # 0: 2 of 60 is noise
 ```
 
+[`examples/retrieval`](examples/retrieval) runs the same gate on a retrieval change: recall@5
+does not move when the top hit drops to rank 5, while MRR and nDCG@5 do, and the gate fails
+it. It was contributed by [@HarshRajSinghania](https://github.com/HarshRajSinghania), and CI
+checks its exit codes too.
+
 With Ollama installed, the LLM example is
 [`examples/support-qa`](examples/support-qa) (`ollama pull qwen3:1.7b` and
 `ollama pull qwen3:8b`; about 6–10 minutes per run on an Apple M5).
@@ -86,6 +93,63 @@ With Ollama installed, the LLM example is
 The browser app in [`app/`](app) runs locally too:
 `uv run --with-requirements app/requirements.txt streamlit run app/streamlit_app.py`.
 
+## Providers
+
+The target is `<provider>:<model>`; the model name is everything after the first colon, so
+`openrouter:anthropic/claude-sonnet-4.5` and `ollama:qwen3:8b` both work. Two native adapters
+(`ollama`, `anthropic`) and one OpenAI-compatible adapter driven by a table of provider
+profiles ([`providers/profiles.py`](src/evalproof/providers/profiles.py), each row cites the
+provider's documentation). Plain HTTP throughout, no vendor SDK.
+
+| Provider | Example target | Key variable | Where |
+|---|---|---|---|
+| `ollama` | `ollama:qwen3:8b` | none | local |
+| `vllm` | `vllm:meta-llama/Llama-3.1-8B-Instruct` | `VLLM_API_KEY`, optional | local |
+| `lmstudio` | `lmstudio:qwen3-8b` | optional, via `api_key_env` | local |
+| `llamacpp` | `llamacpp:qwen3-8b` | `LLAMA_API_KEY`, optional | local |
+| `anthropic` | `anthropic:claude-sonnet-4-5` | `ANTHROPIC_API_KEY` | hosted |
+| `openai` | `openai:gpt-5` | `OPENAI_API_KEY` | hosted |
+| `azure` | `azure:<deployment>` | `AZURE_OPENAI_API_KEY` | hosted, needs `base_url` |
+| `gemini` | `gemini:gemini-2.5-flash` | `GEMINI_API_KEY` | hosted |
+| `mistral` | `mistral:mistral-large-latest` | `MISTRAL_API_KEY` | hosted |
+| `groq` | `groq:llama-3.3-70b-versatile` | `GROQ_API_KEY` | hosted |
+| `together` | `together:meta-llama/Llama-3.3-70B-Instruct-Turbo` | `TOGETHER_API_KEY` | hosted |
+| `fireworks` | `fireworks:accounts/fireworks/models/llama-v3p3-70b-instruct` | `FIREWORKS_API_KEY` | hosted |
+| `deepseek` | `deepseek:deepseek-chat` | `DEEPSEEK_API_KEY` | hosted |
+| `xai` | `xai:grok-4` | `XAI_API_KEY` | hosted |
+| `openrouter` | `openrouter:anthropic/claude-sonnet-4.5` | `OPENROUTER_API_KEY` | hosted |
+| `cerebras` | `cerebras:gpt-oss-120b` | `CEREBRAS_API_KEY` | hosted |
+| `openai-compatible` | `openai-compatible:<model>` | optional, via `api_key_env` | any server, needs `base_url` |
+
+```yaml
+system:
+  target: openai-compatible:gpt-4o    # a gateway, LiteLLM proxy, TGI, SGLang, ...
+  base_url: https://llm-gateway.internal/v1
+  api_key_env: TEAM_GATEWAY_KEY       # the variable to read; the key itself never goes in the suite
+  headers: {x-team: search}           # extra HTTP headers on every call, never secrets
+  params: {max_tokens: 2048}
+```
+
+* `base_url` overrides any profile's default (a regional endpoint, a proxy, an older
+  `llama-server`, which listened on port 8080 before October 2026 and now defaults to 9931).
+* `api_key_env` replaces the profile's variable for any provider; a variable named there
+  must be set.
+* `headers` merge into every request and win over the adapter's own, for example
+  `anthropic-version`. They are not part of the cache key, and a suite file is usually
+  committed, so keep keys in the environment.
+* Sampling: local servers (`ollama`, `vllm`, `lmstudio`, `llamacpp`) receive `temperature` and
+  `seed` on every call, defaults included, so a local run repeats. Hosted providers receive
+  them only when `params` sets them, because reasoning models reject a non-default
+  temperature; Anthropic never receives a seed, and since its current models also reject
+  `temperature`, it gets one only when the suite asks. `params.extra` adds any other body
+  field, such as `{thinking: {type: adaptive}, output_config: {effort: low}}`.
+* Thinking models spend output tokens on thinking inside `max_tokens`, which defaults to 512
+  (changing it would change every cache key). Set `params.max_tokens` higher for them; a reply
+  cut off before its first word, a refusal, or a content-filter block is recorded as that
+  case's error rather than as an empty answer.
+* AWS Bedrock and Google Vertex AI sign requests with cloud credentials, which evalproof does
+  not implement. Use a gateway's OpenAI-compatible endpoint through `openai-compatible`.
+
 ## Concepts
 
 **Suite** — one YAML file: a JSONL golden set, the system under test, scorers, an optional
@@ -95,7 +159,7 @@ judge, calibration settings and a gate.
 name: support-qa
 dataset: cases.jsonl              # {"id", "input", "reference", ...extra fields}
 system:
-  target: ollama:qwen3:1.7b       # or anthropic:…, openai:…, python:module:func, https://…
+  target: ollama:qwen3:1.7b       # or groq:…, anthropic:…, python:module:func, https://…
   system_prompt_file: prompt.md
   prompt_template: "{question}"
   params: {temperature: 0, seed: 0, think: false}
@@ -138,6 +202,56 @@ orders and reports order-dependent verdicts as ties.
 | 4 | cannot gate: fewer than `min_cases` paired cases, or the runs used different data |
 | 2 | usage or configuration error |
 
+The gate checks one metric by default. To gate several, repeat `--metric` or list them in
+the suite; each metric's p-value is then Holm-adjusted for the number of metrics gated, so
+the chance of failing a build on noise stays at `alpha` for the set as a whole, and the
+report shows the raw and the adjusted p-value side by side. Metrics that are not gated are
+still reported.
+
+```yaml
+gate:
+  metrics: [judge, answer_key]   # one metric: `metric: judge`
+  max_drop: 0.03
+  alpha: 0.05
+  min_cases: 40
+```
+
+```bash
+evalproof compare baseline.json candidate.json --gate --metric judge --metric answer_key
+```
+
+## Import results from other tools
+
+You do not have to run your evals with evalproof to gate them. `evalproof import` turns the
+per-case results another tool wrote into a run file that `compare` and the gate read:
+
+```bash
+evalproof import promptfoo output.json --system openai:gpt-5 --out runs/candidate.json
+evalproof import inspect logs/task.eval --out runs/candidate.json
+evalproof import deepeval .deepeval/.latest_test_run.json --out runs/candidate.json
+evalproof import lm-eval results/gpt2/samples_hellaswag_2026-11-12T10-20-30.jsonl --out runs/gpt2.json
+evalproof import csv results.csv --system a --out runs/a.json       # id,metric,passed,score,...
+evalproof import jsonl results.jsonl --wide --id-field case --out runs/a.json  # one column per metric
+```
+
+Import each side with the same importer, then gate as usual, for example
+`evalproof compare runs/baseline.json runs/candidate.json --gate --metric pass`. What to know:
+
+* Case ids are stable between runs of the same dataset (they come from the test's inputs, or
+  the tool's own sample id, never from its position), which is what lets two files pair.
+  The dataset hash covers the ids and inputs as the tool recorded them, so runs imported by
+  different importers refuse to gate (exit 4) instead of pairing on a coincidence.
+* A system error counts as a failure on every metric, as in a native run. Repeats of one case
+  (epochs, repeated tests) collapse by `--epochs`: `mean` (the default), `first`, `all`
+  or `any`. Under `mean`, `first` and `any`, a case with at least one repeat that ran is not
+  counted as a system error; its errored repeats still score as failures.
+* A NaN or infinite score is refused at import, naming the case and metric, because it would
+  make the bootstrap report a significant difference that is not there.
+* promptfoo files can hold several providers and `--system` picks one. Inspect `.eval` logs
+  compressed with zstd need Python 3.14 or newer; otherwise run
+  `inspect log convert --to json <file>` first. DeepEval's conversational cases are skipped
+  with a warning.
+
 ## From Python
 
 The CLI is a thin layer over the same functions:
@@ -167,7 +281,7 @@ flowchart LR
   suite[suite.yaml] --> runner
   data[(cases.jsonl)] --> runner
   subgraph runner[evalproof run]
-    sys[system under test<br/>ollama · anthropic · openai<br/>python · http] --> scorers
+    sys[system under test<br/>ollama · anthropic · openai-compatible<br/>python · http] --> scorers
     scorers[scorers<br/>deterministic · retrieval · llm_judge]
     cache[(content-addressed<br/>cache)]
     sys <--> cache
@@ -189,7 +303,7 @@ flowchart LR
 | Treat the judge as a classifier to be calibrated | Trust the judge's pass rate | Its count is biased by its own error rates, and the bias moves with the true rate | [0001](docs/adr/0001-judge-is-a-classifier.md) |
 | Local Ollama by default, HTTP-only providers, content-addressed cache | Hosted APIs and vendor SDKs | Reproducible, free to re-run, tied to a model digest; no SDK dependency | [0002](docs/adr/0002-local-first-providers.md) |
 | Gate on "large AND significant", on paired cases | "Fail if the score dropped by X" | On a few dozen cases, X-point drops are mostly noise, and a noisy gate gets switched off | [0003](docs/adr/0003-gate-on-significance-not-delta.md) |
-| Gate one metric; report the rest | Gate every metric | Testing many metrics at α = 0.05 each inflates false failures | 0003 |
+| One metric by default; several with Holm correction | Gate every metric at α each | Testing many metrics at α = 0.05 each inflates false failures | 0003 |
 | Pointwise verdicts for gating; pairwise as a diagnostic | Pairwise preference for gating | Measured: 42% of pairwise verdicts flipped with answer order, with a strong length preference | [MEASUREMENTS §6](MEASUREMENTS.md) |
 | numpy and the standard library for statistics | SciPy, statsmodels | Small enough to read and to test against hand-worked values | — |
 
@@ -199,11 +313,12 @@ The composite action runs `compare --gate`, appends the report to the job summar
 fails the job on exit codes 1, 3 or 4:
 
 ```yaml
-- uses: technobasant/evalproof@v0.2.0
+- uses: technobasant/evalproof@v0.3.0
   with:
     baseline: eval/baseline.json     # produced on main, e.g. as a build artifact
     candidate: eval/candidate.json   # produced on this branch
     suite: eval/suite.yaml
+    # metric: "judge, answer_key"    # optional: gate several metrics, newline- or comma-separated
 ```
 
 This repository's own CI runs the offline example on every commit and asserts that the
@@ -223,11 +338,12 @@ frameworks.
 evalproof is deliberately narrow: it is the measurement layer — how much a judge's verdict
 is worth, and whether a difference between two runs is real. Its run file is plain,
 documented JSON, so it can sit beside an existing harness: produce the per-case verdicts
-however you like, then calibrate and gate here.
+however you like, then calibrate and gate here. For promptfoo, Inspect, DeepEval and
+lm-eval you do not even need to do that: evalproof [imports their results](#import-results-from-other-tools).
 
 ## Limits
 
-* The measurements cover one machine, two small local models, one fictional domain and
+* The measurements cover one machine, two small local models, one product domain and
   83 cases. They show the method; they are not a benchmark of those models.
 * Calibration is only as good as its gold labels — [MEASUREMENTS §4](MEASUREMENTS.md)
   shows a regex answer key being wrong more often than the judge. Use human labels when
@@ -242,9 +358,6 @@ however you like, then calibrate and gate here.
 ## Roadmap
 
 * Stratified gold sampling (by tag) and per-tag calibration.
-* Multiple-comparison correction when more than one metric is gated.
-* A reasoning judge (`think: true`) measured against the same answer key.
-* Importers for other harnesses' per-case results.
 
 ## Project
 

{evalproof-0.2.1 → evalproof-0.3.0}/src/evalproof/__init__.py

@@ -23,10 +23,11 @@ from evalproof.gate import (
     GateDecision,
     decide,
 )
+from evalproof.importers import ImportFormatError, import_run
 from evalproof.models import Case, CaseResult, Run, RunInfo, Score, SystemOutput
 from evalproof.report import calibration_markdown, comparison_markdown, run_markdown
 from evalproof.runner import load_run, run_suite, save_run
-from evalproof.stats import mcnemar_exact, paired_bootstrap, wilson
+from evalproof.stats import holm, mcnemar_exact, paired_bootstrap, wilson
 
 __all__ = [
     "EXIT_BUDGET",
@@ -44,6 +45,7 @@ __all__ = [
     "Estimate",
     "GateConfig",
     "GateDecision",
+    "ImportFormatError",
     "MetricComparison",
     "Run",
     "RunInfo",
@@ -57,6 +59,8 @@ __all__ = [
     "compare_runs",
     "comparison_markdown",
     "decide",
+    "holm",
+    "import_run",
     "load_config",
     "load_dataset",
     "load_run",

{evalproof-0.2.1 → evalproof-0.3.0}/src/evalproof/_version.py

@@ -1,3 +1,3 @@
 """The package version: the single place it is written. pyproject.toml reads it at build time."""
 
-__version__ = "0.2.1"
+__version__ = "0.3.0"