evalproof 0.3.0__tar.gz → 0.4.0__tar.gz

{evalproof-0.3.0 → evalproof-0.4.0}/CHANGELOG.md

@@ -4,6 +4,71 @@ All notable changes are listed here. The project follows [Semantic Versioning](h
 
 ## [Unreleased]
 
+## [0.4.0] — 2026-11-13
+
+### Added
+
+- `claude-code:<model>` and `codex:<model>` providers: run your own logged-in Claude Code or
+  Codex CLI as a judge or a system under test, with no API key. Each call runs in a fresh
+  empty directory with the agent's tools, settings, MCP servers, skills and memory switched
+  off and no permission-bypass flag; API-key variables are removed from the child's
+  environment unless `api_key_env` names one or `EVALPROOF_CLI_AUTH=api-key` keeps them.
+  Neither CLI takes a temperature, seed or token limit, so those params are ignored (Codex
+  maps `think` to its reasoning effort). Claude Code runs with no tools; Codex still offers a
+  JavaScript isolate, sub-agents and a question to the user, so any item in its output other
+  than a message, reasoning, an error or a plan fails the call as an isolation break, and a
+  codex too old for the flags evalproof sets gets a message to upgrade. ADR 0004 has the
+  reasoning.
+- `EVALPROOF_CLI_AUTH` (`login`, the default, or `api-key`) chooses whether a CLI provider
+  uses the subscription login or its own API key variable; `evalproof doctor` prints the mode
+  in effect and warns when `api-key` has no key to use. `api_key_env` still wins.
+- `evalproof doctor [suite.yaml]`: reports each CLI's path, version, login mode and the
+  isolation flags it will use, without a model call, and exits 1 when a target of the suite
+  is not usable. Without a suite it surveys the installed CLIs and exits 1 only when none is
+  usable. It prints no account details.
+- Run facts for CLI targets in `runner.environment`: `{judge,system}_cli`,
+  `{judge,system}_cli_version`, `{judge,system}_isolation` and `{judge,system}_model_resolved`
+  (the model ids that answered, sorted, cached replies included). The response cache keys on
+  the CLI fingerprint, so a CLI upgrade is a cache miss.
+- `compare` reports `same_judge`, `judge_changes` and `system_changes`, and the gate refuses
+  to decide (exit 4) when the judge changed between the runs. `--allow-judge-change` or
+  `gate.allow_judge_change: true` accepts the change and keeps a warning.
+- Experimental `opencode:<provider>/<model>`, `cursor:<model>` and `copilot:<model>`
+  providers, on the same footing as the CLI providers above (fresh empty directory, the CLI's
+  key variable removed unless `api_key_env` names one, bounded runtime, CLI fingerprint with the
+  isolation tags `opencode/0`, `cursor/0`, `copilot/0`). Each prints one warning on stderr the
+  first time it is used. Only OpenCode was exercised for real, and only against a local ollama
+  model; Cursor (not logged in) and Copilot (not installed) follow their documentation, and the
+  README's "Experimental providers" says what was and was not verified. `evalproof doctor`
+  without a suite now also lists whichever of these CLIs are on `PATH`.
+- `evalproof.providers.TransientError`, a `ProviderError` that `with_retries` retries; the CLI
+  providers raise it for rate limits and server errors.
+- `tests/test_live_agent_cli.py`: one real call per installed CLI, skipped unless
+  `EVALPROOF_LIVE=1`.
+- `evalproof init [--dir PATH] [--judge TARGET] [--ci keyless|api-key|none] [--no-skill]
+  [--force]`: writes `evals/suite.yaml`, three sample cases in `evals/cases.jsonl`, an
+  `evals/README.md`, a GitHub Actions workflow at `.github/workflows/evalproof.yml` and an
+  agent skill at `.agents/skills/evalproof/SKILL.md` and `.claude/skills/evalproof/SKILL.md`.
+  The judge is the first of `claude`, `codex`, `ollama` found on `PATH`, else a hosted
+  Anthropic model. The keyless workflow makes no model calls and gates the committed
+  `evals/runs/latest.json` against the base branch; the api-key workflow runs the suite in CI
+  (with `EVALPROOF_CLI_AUTH: api-key` for a CLI judge, so no second suite file is written)
+  and pins the judge CLI to the version found at init time. Existing files are kept unless
+  `--force` is passed, nothing is written outside `--dir`, and no model is called.
+
+### Changed
+
+- A gate that passed in 0.3.0 can now exit 4 when the judge's target, CLI version, isolation
+  or resolved model differs between two runs that both carry these keys. Runs written before
+  this release carry none and compare as before.
+- A changed system target or CLI version between two runs now adds a warning, so the status
+  is `warn` instead of `pass` (the exit code stays 0).
+- `runner.environment` records `judge_target` and `system_target` for every model target (not
+  only CLI ones), and the system target it records is the one in effect, with any override
+  applied.
+- The gate's "too few paired cases" reason now says how to fix it (add cases, or lower
+  `gate.min_cases`).
+
 ## [0.3.0] — 2026-11-12
 
 ### Added
@@ -179,7 +244,8 @@ First release.
 - Two examples: an offline ticket router (CI's gate contract) and a product support
   assistant judged by a local model, with measurements.
 
-[Unreleased]: https://github.com/technobasant/evalproof/compare/v0.3.0...HEAD
+[Unreleased]: https://github.com/technobasant/evalproof/compare/v0.4.0...HEAD
+[0.4.0]: https://github.com/technobasant/evalproof/compare/v0.3.0...v0.4.0
 [0.3.0]: https://github.com/technobasant/evalproof/compare/v0.2.1...v0.3.0
 [0.2.1]: https://github.com/technobasant/evalproof/compare/v0.2.0...v0.2.1
 [0.2.0]: https://github.com/technobasant/evalproof/compare/v0.1.2...v0.2.0

{evalproof-0.3.0 → evalproof-0.4.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: evalproof
-Version: 0.3.0
+Version: 0.4.0
 Summary: Treat LLM evaluation as measurement: calibrate the judge, correct the pass rate, gate CI on significant regressions.
 Project-URL: Homepage, https://github.com/technobasant/evalproof
 Project-URL: Documentation, https://github.com/technobasant/evalproof#readme
@@ -100,6 +100,23 @@ questions, one Apple M5. Full method and every table in [MEASUREMENTS.md](https:
 
 ## Five-minute quickstart
 
+In your own repository, one command writes a starter suite, a CI workflow and an agent skill,
+and a second runs it:
+
+```bash
+uvx evalproof init
+uvx evalproof run evals/suite.yaml --out evals/runs/latest.json
+```
+
+`init` picks the judge from what is on your `PATH`: a logged-in `claude`, then `codex`, then
+Ollama, then a hosted Anthropic model with `ANTHROPIC_API_KEY`. It writes `evals/` (the
+suite, three sample cases, a README), `.github/workflows/evalproof.yml` (`--ci keyless`, the
+default, makes no model calls in CI; `--ci api-key` runs the suite there; `--ci none` writes
+no workflow) and a skill for coding agents at `.agents/skills/evalproof/SKILL.md` and
+`.claude/skills/evalproof/SKILL.md` (`--no-skill` skips it). It prints one line per file,
+never overwrites one without `--force`, and makes no model call. Replace the sample cases
+with your own: the gate needs 20 before it will decide.
+
 The offline example lives in the repository:
 
 ```bash
@@ -135,7 +152,11 @@ The target is `<provider>:<model>`; the model name is everything after the first
 `openrouter:anthropic/claude-sonnet-4.5` and `ollama:qwen3:8b` both work. Two native adapters
 (`ollama`, `anthropic`) and one OpenAI-compatible adapter driven by a table of provider
 profiles ([`providers/profiles.py`](https://github.com/technobasant/evalproof/blob/main/src/evalproof/providers/profiles.py), each row cites the
-provider's documentation). Plain HTTP throughout, no vendor SDK.
+provider's documentation). Plain HTTP throughout, no vendor SDK. Two more targets run a coding
+agent you already have logged in, `claude-code` and `codex`: see
+[Use your Claude Code or Codex login](#use-your-claude-code-or-codex-login). Three experimental
+ones run OpenCode, Cursor and GitHub Copilot the same way: see
+[Experimental providers](#experimental-providers).
 
 | Provider | Example target | Key variable | Where |
 |---|---|---|---|
@@ -156,6 +177,11 @@ provider's documentation). Plain HTTP throughout, no vendor SDK.
 | `openrouter` | `openrouter:anthropic/claude-sonnet-4.5` | `OPENROUTER_API_KEY` | hosted |
 | `cerebras` | `cerebras:gpt-oss-120b` | `CEREBRAS_API_KEY` | hosted |
 | `openai-compatible` | `openai-compatible:<model>` | optional, via `api_key_env` | any server, needs `base_url` |
+| `claude-code` | `claude-code:haiku` | none: your Claude Code login | local CLI, see below |
+| `codex` | `codex:gpt-6-luna` | none: your ChatGPT login | local CLI, see below |
+| `opencode` | `opencode:ollama/qwen3:8b` | none: OpenCode's own login, per model provider | local CLI, experimental |
+| `cursor` | `cursor:auto` | none: your Cursor login | local CLI, experimental |
+| `copilot` | `copilot:gpt-5` | none: your GitHub Copilot login | local CLI, experimental |
 
 ```yaml
 system:
@@ -186,6 +212,134 @@ system:
 * AWS Bedrock and Google Vertex AI sign requests with cloud credentials, which evalproof does
   not implement. Use a gateway's OpenAI-compatible endpoint through `openai-compatible`.
 
+## Use your Claude Code or Codex login
+
+If Claude Code or Codex is already logged in on your machine, either can be the judge or the
+system under test, with no API key:
+
+```yaml
+judge:
+  target: claude-code:haiku        # an alias (haiku, sonnet, opus) or a full model id
+system:
+  target: codex:gpt-6-luna
+concurrency: 4
+```
+
+Run `evalproof doctor suite.yaml` first. It prints each CLI's path, version, login mode and
+the isolation flags it will pass, and exits 1 if a target of the suite is not usable. Without a
+suite it surveys every CLI it knows that is installed, and exits 1 only when none is usable. It
+makes no model call.
+
+evalproof runs your own `claude` or `codex` binary as a subprocess, sends the prompt on stdin
+and reads the JSON it prints. It never reads a token file or the keychain. The environment
+variable `EVALPROOF_CLI_AUTH` says which credentials the CLI uses:
+
+* `login` (the default, also when it is unset or empty) removes the CLI's key variables from
+  the call (`ANTHROPIC_API_KEY` and `ANTHROPIC_AUTH_TOKEN` for Claude Code, `OPENAI_API_KEY`
+  and `CODEX_API_KEY` for Codex), so the login is what gets used.
+* `api-key` leaves them in place, so the CLI uses its own key. This is the route for CI.
+* Any other value is an error.
+
+`api_key_env` in the suite works in either mode and wins: it passes that variable to the CLI as
+its own key variable. The mode is not part of the run's fingerprint or the cache key, because
+it is the same model either way. `evalproof doctor` prints the mode in effect and warns when
+`api-key` is chosen but the CLI's key variable is not set.
+
+* **Isolation is a requirement, not an option.** A coding agent is not a model endpoint. Left
+  alone it loads the project's `CLAUDE.md` or `AGENTS.md`, your memory, plugins, MCP servers
+  and skills into the prompt, and it can run commands. A judge that has read your project's
+  instructions is not a judge. So every call runs in a fresh empty directory, with the
+  agent's own system prompt replaced by yours (or one neutral line), its tools, settings, MCP
+  servers and skills switched off, and no permission-bypass flag. `evalproof doctor` prints
+  the exact flags. The two CLIs do not isolate equally well:
+  * Claude Code is run with **no tools at all** (`--tools ''`).
+  * Codex runs in a read-only sandbox with its shell, plugins, apps, hooks, multi-agent mode,
+    sleep and image tools, bundled skills and your user config and rules switched off, but
+    codex-cli 0.167.2 has no flag that removes every tool. Still offered are a JavaScript
+    isolate (`exec`: no Node, no file system, no network, with a few built-in helpers such as
+    the current time, MCP resource listing and `apply_patch`), sub-agents that get the same
+    restricted tools, and a question to the user, which cannot be answered in exec mode. **Any
+    tool use fails the call**: if the output holds an item that is not a message, reasoning,
+    an error or a plan, evalproof raises `codex used a tool (<type>); isolation broken` and
+    discards the reply. A codex too old to know a flag evalproof sets exits with `Unknown
+    feature flag`, and evalproof says to upgrade it (it was tested with codex-cli 0.167.2).
+* **No temperature, seed or token limit.** Neither CLI exposes one, so `temperature`, `seed`
+  and `max_tokens` are ignored. `think` maps to Codex's reasoning effort (`false` is `low`,
+  `true` is `medium`; `params.extra.reasoning_effort` sets it directly). A run is therefore
+  not repeatable the way a seeded local run is, so the run records the CLI, its version, the
+  isolation scheme and the model id that actually answered, and the cache keys on the CLI
+  version. To see how much a judge wobbles, score the same outputs twice with `--no-cache`, or
+  [calibrate](#concepts) it against labels.
+* **A changed judge is not comparable.** `compare` and the gate refuse (exit 4) when the judge's
+  target, CLI version, isolation or resolved model differs between the two runs, because the
+  scores then come from a different instrument. Pass `--allow-judge-change` or set
+  `gate.allow_judge_change: true` to accept it. A changed system target or CLI version only
+  warns, since that is usually the change under test. Runs written before 0.4.0 carry none of
+  these keys and compare as before.
+* **Latency and cost.** A call takes about 1.4 s on `claude-code:haiku` and about 4 s on
+  Codex, mostly start-up, so set `concurrency`. There is no API bill; the calls count against
+  your subscription's usage limits.
+* **Terms.** OpenAI documents Codex CLI use on a ChatGPT plan, including automation.
+  Anthropic's terms allow a user to run the unmodified Claude Code binary on their own
+  subscription; its plan limits assume ordinary individual use, and it forbids third-party
+  products from routing other users' requests through subscription credentials. evalproof
+  never touches credentials: it runs your own `claude` binary, on your machine, for your own
+  evals. For CI or team-wide use, use an API key (`EVALPROOF_CLI_AUTH=api-key`, or
+  `api_key_env`). See Anthropic's
+  [legal and compliance page](https://code.claude.com/docs/en/legal-and-compliance). Nothing
+  uses a CLI unless a suite names a `claude-code:` or `codex:` target (or an experimental one,
+  below).
+
+### Experimental providers
+
+`opencode:`, `cursor:` and `copilot:` follow the same rules as the two above: your own logged-in
+binary, run as a subprocess in a fresh empty directory, its API-key variable removed from the
+child's environment unless `api_key_env` names one, a bounded runtime, and the CLI, its version
+and an isolation tag (`opencode/0`, `cursor/0`, `copilot/0`) in the run and in the cache key.
+They are experimental because their isolation is less proven than Claude Code's and Codex's.
+Each prints `evalproof: <name> provider is experimental` on stderr the first time it is used in
+a process. Like the other two they ignore `temperature`, `seed` and `max_tokens`, and
+`evalproof doctor` without a suite lists each of them whose binary is on `PATH`. What was
+checked, and what was not:
+
+* **`opencode:<provider>/<model>`** runs `opencode run --standalone --archived --format json`
+  with a custom agent that has every tool off and every permission denied, your system text as
+  that agent's prompt, and project config, the global config directory and the Claude Code
+  compatibility files switched off. The prompt is on stdin. *Verified* only against local
+  ollama models (`qwen3:8b`, a handful of calls): no tool event, a canary `AGENTS.md` in the
+  project not read, a canary `AGENTS.md` in the global config directory read by default and not
+  read with these settings, the parser and the error path. *Not verified:* any hosted model;
+  whether `~/.claude/CLAUDE.md` is read (that test was inconclusive); whether the empty
+  `OPENCODE_CONFIG_DIR` hides a login stored there; the token counts (input adds cache reads
+  and writes, output adds reasoning, summed per field name). OpenCode has no single login, so
+  `doctor` prints `auth: not checked`. Only `OPENCODE_API_KEY` is removed from the environment:
+  the key of the model's own provider (`ANTHROPIC_API_KEY` for `opencode:anthropic/...`) is
+  left, because naming that provider in the model id is choosing its key, and removing it would
+  break CI. `--archived` keeps a session out of OpenCode's lists but does not delete it, so
+  sessions accumulate in OpenCode's data directory (each run is a private `--standalone`
+  server in a temporary project); prune them yourself.
+* **`cursor:<model>`** runs `cursor-agent -p --mode ask --trust` with an empty `--workspace`.
+  *Not verified:* Cursor was not logged in on the test machine, so no call succeeded. The flags
+  come from the CLI's help and documentation, and the parser from its source; it is lenient (a
+  JSON result object, an array of events, terminal noise around it) and an unexpected shape
+  fails with a message that says what was expected. The empty workspace is the only isolation:
+  no flag ignores rules files, user-level rules were not checked, and whether ask mode really
+  has no tools is untested. There is no system-prompt option, so your system text goes in front
+  of the prompt under a heading, and the prompt is an argument (nothing says the CLI reads
+  stdin), so it is visible in `ps` while the call runs and a prompt over 100,000 bytes is
+  refused. The key variable is `CURSOR_API_KEY`, passed through the environment, never as
+  `--api-key`. `doctor` reads `cursor-agent about` for the login state (`status` hung) and
+  prints no account.
+* **`copilot:<model>`** runs `copilot -s --no-auto-update --no-ask-user --available-tools= -p`.
+  *Not verified:* Copilot CLI was not installed on the test machine; every flag comes from
+  GitHub's documentation. `--available-tools=` is an allow-list with nothing in it, the most
+  restrictive documented tool option; whether the CLI accepts an empty list was not tried. No
+  `--allow-*` flag is ever passed, so a tool that needs approval cannot get it. The reply is
+  plain text, so token counts are 0. `COPILOT_HOME` is not redirected, so user-level Copilot
+  instructions may still be read. System text and prompt travel as for Cursor. Only
+  `COPILOT_GITHUB_TOKEN` is removed; `GH_TOKEN` and `GITHUB_TOKEN` are left because `gh` and CI
+  set them for other reasons. `doctor` prints `auth: not checked`.
+
 ## Concepts
 
 **Suite** — one YAML file: a JSONL golden set, the system under test, scorers, an optional
@@ -210,7 +364,8 @@ gate: {metric: judge, max_drop: 0.03, alpha: 0.05, min_cases: 40, max_p95_latenc
 
 **Run** — `evalproof run suite.yaml` writes one self-describing JSON file: every case's
 output, latency, tokens and scores, plus the dataset hash, seed, Ollama version and model
-digests.
+digests, and for a `claude-code` or `codex` target the CLI, its version and the model that
+answered.
 
 **Scorers** — `exact_match`, `contains`, `regex` (accept/reject answer keys),
 `json_schema`, `numeric_tolerance`, `recall_at_k`, `mrr`, `ndcg`, and `llm_judge`: a strict
@@ -235,7 +390,7 @@ orders and reports order-dependent verdicts as ties.
 | 0 | pass — possibly with warnings (a big drop that is not significant; a significant drop within tolerance) |
 | 1 | quality regression: drop ≥ `max_drop` **and** p < `alpha` |
 | 3 | budget breach: candidate p95 latency or cost over the limit |
-| 4 | cannot gate: fewer than `min_cases` paired cases, or the runs used different data |
+| 4 | cannot gate: fewer than `min_cases` paired cases, the runs used different data, or the judge changed (unless `--allow-judge-change`) |
 | 2 | usage or configuration error |
 
 The gate checks one metric by default. To gate several, repeat `--metric` or list them in
@@ -322,6 +477,7 @@ The architecture diagram is in the [README on GitHub](https://github.com/technob
 | Local Ollama by default, HTTP-only providers, content-addressed cache | Hosted APIs and vendor SDKs | Reproducible, free to re-run, tied to a model digest; no SDK dependency | [0002](https://github.com/technobasant/evalproof/blob/main/docs/adr/0002-local-first-providers.md) |
 | Gate on "large AND significant", on paired cases | "Fail if the score dropped by X" | On a few dozen cases, X-point drops are mostly noise, and a noisy gate gets switched off | [0003](https://github.com/technobasant/evalproof/blob/main/docs/adr/0003-gate-on-significance-not-delta.md) |
 | One metric by default; several with Holm correction | Gate every metric at α each | Testing many metrics at α = 0.05 each inflates false failures | 0003 |
+| Run a coding-agent CLI as a provider only in isolation, and gate on its fingerprint | Pass the CLI's defaults through, or ignore its version | A CLI that reads your project's instructions is not a judge, and a silent upgrade changes the instrument | [0004](https://github.com/technobasant/evalproof/blob/main/docs/adr/0004-agent-clis-as-providers.md) |
 | Pointwise verdicts for gating; pairwise as a diagnostic | Pairwise preference for gating | Measured: 42% of pairwise verdicts flipped with answer order, with a strong length preference | [MEASUREMENTS §6](https://github.com/technobasant/evalproof/blob/main/MEASUREMENTS.md) |
 | numpy and the standard library for statistics | SciPy, statsmodels | Small enough to read and to test against hand-worked values | — |
 
@@ -331,7 +487,7 @@ The composite action runs `compare --gate`, appends the report to the job summar
 fails the job on exit codes 1, 3 or 4:
 
 ```yaml
-- uses: technobasant/evalproof@v0.3.0
+- uses: technobasant/evalproof@v0.4.0
   with:
     baseline: eval/baseline.json     # produced on main, e.g. as a build artifact
     candidate: eval/candidate.json   # produced on this branch
@@ -342,6 +498,14 @@ fails the job on exit codes 1, 3 or 4:
 This repository's own CI runs the offline example on every commit and asserts that the
 gate returns 1 for the real regression and 0 for the noisy and refactored candidates.
 
+If the judge is a Claude Code or Codex CLI, `evalproof init --ci api-key` writes a workflow that
+installs the CLI at the version found at init time and runs `evals/suite.yaml` as it is. The run
+step sets `EVALPROOF_CLI_AUTH: api-key` and the repository secret (`ANTHROPIC_API_KEY` for
+Claude Code, `OPENAI_API_KEY` for Codex), so the CLI uses that key instead of a login. The
+suite names no key and there is no second suite file; a suite with an `api_key_env` still uses
+that variable. Keep the pinned CLI version equal to the one in your baseline run, because the
+gate exits 4 when the judge's CLI version differs.
+
 ## How it relates to other tools
 
 [promptfoo](https://www.promptfoo.dev/docs/intro/) (open-source, local evals of prompts,

{evalproof-0.3.0 → evalproof-0.4.0}/README.md

@@ -64,6 +64,23 @@ questions, one Apple M5. Full method and every table in [MEASUREMENTS.md](MEASUR
 
 ## Five-minute quickstart
 
+In your own repository, one command writes a starter suite, a CI workflow and an agent skill,
+and a second runs it:
+
+```bash
+uvx evalproof init
+uvx evalproof run evals/suite.yaml --out evals/runs/latest.json
+```
+
+`init` picks the judge from what is on your `PATH`: a logged-in `claude`, then `codex`, then
+Ollama, then a hosted Anthropic model with `ANTHROPIC_API_KEY`. It writes `evals/` (the
+suite, three sample cases, a README), `.github/workflows/evalproof.yml` (`--ci keyless`, the
+default, makes no model calls in CI; `--ci api-key` runs the suite there; `--ci none` writes
+no workflow) and a skill for coding agents at `.agents/skills/evalproof/SKILL.md` and
+`.claude/skills/evalproof/SKILL.md` (`--no-skill` skips it). It prints one line per file,
+never overwrites one without `--force`, and makes no model call. Replace the sample cases
+with your own: the gate needs 20 before it will decide.
+
 The offline example lives in the repository:
 
 ```bash
@@ -99,7 +116,11 @@ The target is `<provider>:<model>`; the model name is everything after the first
 `openrouter:anthropic/claude-sonnet-4.5` and `ollama:qwen3:8b` both work. Two native adapters
 (`ollama`, `anthropic`) and one OpenAI-compatible adapter driven by a table of provider
 profiles ([`providers/profiles.py`](src/evalproof/providers/profiles.py), each row cites the
-provider's documentation). Plain HTTP throughout, no vendor SDK.
+provider's documentation). Plain HTTP throughout, no vendor SDK. Two more targets run a coding
+agent you already have logged in, `claude-code` and `codex`: see
+[Use your Claude Code or Codex login](#use-your-claude-code-or-codex-login). Three experimental
+ones run OpenCode, Cursor and GitHub Copilot the same way: see
+[Experimental providers](#experimental-providers).
 
 | Provider | Example target | Key variable | Where |
 |---|---|---|---|
@@ -120,6 +141,11 @@ provider's documentation). Plain HTTP throughout, no vendor SDK.
 | `openrouter` | `openrouter:anthropic/claude-sonnet-4.5` | `OPENROUTER_API_KEY` | hosted |
 | `cerebras` | `cerebras:gpt-oss-120b` | `CEREBRAS_API_KEY` | hosted |
 | `openai-compatible` | `openai-compatible:<model>` | optional, via `api_key_env` | any server, needs `base_url` |
+| `claude-code` | `claude-code:haiku` | none: your Claude Code login | local CLI, see below |
+| `codex` | `codex:gpt-6-luna` | none: your ChatGPT login | local CLI, see below |
+| `opencode` | `opencode:ollama/qwen3:8b` | none: OpenCode's own login, per model provider | local CLI, experimental |
+| `cursor` | `cursor:auto` | none: your Cursor login | local CLI, experimental |
+| `copilot` | `copilot:gpt-5` | none: your GitHub Copilot login | local CLI, experimental |
 
 ```yaml
 system:
@@ -150,6 +176,134 @@ system:
 * AWS Bedrock and Google Vertex AI sign requests with cloud credentials, which evalproof does
   not implement. Use a gateway's OpenAI-compatible endpoint through `openai-compatible`.
 
+## Use your Claude Code or Codex login
+
+If Claude Code or Codex is already logged in on your machine, either can be the judge or the
+system under test, with no API key:
+
+```yaml
+judge:
+  target: claude-code:haiku        # an alias (haiku, sonnet, opus) or a full model id
+system:
+  target: codex:gpt-6-luna
+concurrency: 4
+```
+
+Run `evalproof doctor suite.yaml` first. It prints each CLI's path, version, login mode and
+the isolation flags it will pass, and exits 1 if a target of the suite is not usable. Without a
+suite it surveys every CLI it knows that is installed, and exits 1 only when none is usable. It
+makes no model call.
+
+evalproof runs your own `claude` or `codex` binary as a subprocess, sends the prompt on stdin
+and reads the JSON it prints. It never reads a token file or the keychain. The environment
+variable `EVALPROOF_CLI_AUTH` says which credentials the CLI uses:
+
+* `login` (the default, also when it is unset or empty) removes the CLI's key variables from
+  the call (`ANTHROPIC_API_KEY` and `ANTHROPIC_AUTH_TOKEN` for Claude Code, `OPENAI_API_KEY`
+  and `CODEX_API_KEY` for Codex), so the login is what gets used.
+* `api-key` leaves them in place, so the CLI uses its own key. This is the route for CI.
+* Any other value is an error.
+
+`api_key_env` in the suite works in either mode and wins: it passes that variable to the CLI as
+its own key variable. The mode is not part of the run's fingerprint or the cache key, because
+it is the same model either way. `evalproof doctor` prints the mode in effect and warns when
+`api-key` is chosen but the CLI's key variable is not set.
+
+* **Isolation is a requirement, not an option.** A coding agent is not a model endpoint. Left
+  alone it loads the project's `CLAUDE.md` or `AGENTS.md`, your memory, plugins, MCP servers
+  and skills into the prompt, and it can run commands. A judge that has read your project's
+  instructions is not a judge. So every call runs in a fresh empty directory, with the
+  agent's own system prompt replaced by yours (or one neutral line), its tools, settings, MCP
+  servers and skills switched off, and no permission-bypass flag. `evalproof doctor` prints
+  the exact flags. The two CLIs do not isolate equally well:
+  * Claude Code is run with **no tools at all** (`--tools ''`).
+  * Codex runs in a read-only sandbox with its shell, plugins, apps, hooks, multi-agent mode,
+    sleep and image tools, bundled skills and your user config and rules switched off, but
+    codex-cli 0.167.2 has no flag that removes every tool. Still offered are a JavaScript
+    isolate (`exec`: no Node, no file system, no network, with a few built-in helpers such as
+    the current time, MCP resource listing and `apply_patch`), sub-agents that get the same
+    restricted tools, and a question to the user, which cannot be answered in exec mode. **Any
+    tool use fails the call**: if the output holds an item that is not a message, reasoning,
+    an error or a plan, evalproof raises `codex used a tool (<type>); isolation broken` and
+    discards the reply. A codex too old to know a flag evalproof sets exits with `Unknown
+    feature flag`, and evalproof says to upgrade it (it was tested with codex-cli 0.167.2).
+* **No temperature, seed or token limit.** Neither CLI exposes one, so `temperature`, `seed`
+  and `max_tokens` are ignored. `think` maps to Codex's reasoning effort (`false` is `low`,
+  `true` is `medium`; `params.extra.reasoning_effort` sets it directly). A run is therefore
+  not repeatable the way a seeded local run is, so the run records the CLI, its version, the
+  isolation scheme and the model id that actually answered, and the cache keys on the CLI
+  version. To see how much a judge wobbles, score the same outputs twice with `--no-cache`, or
+  [calibrate](#concepts) it against labels.
+* **A changed judge is not comparable.** `compare` and the gate refuse (exit 4) when the judge's
+  target, CLI version, isolation or resolved model differs between the two runs, because the
+  scores then come from a different instrument. Pass `--allow-judge-change` or set
+  `gate.allow_judge_change: true` to accept it. A changed system target or CLI version only
+  warns, since that is usually the change under test. Runs written before 0.4.0 carry none of
+  these keys and compare as before.
+* **Latency and cost.** A call takes about 1.4 s on `claude-code:haiku` and about 4 s on
+  Codex, mostly start-up, so set `concurrency`. There is no API bill; the calls count against
+  your subscription's usage limits.
+* **Terms.** OpenAI documents Codex CLI use on a ChatGPT plan, including automation.
+  Anthropic's terms allow a user to run the unmodified Claude Code binary on their own
+  subscription; its plan limits assume ordinary individual use, and it forbids third-party
+  products from routing other users' requests through subscription credentials. evalproof
+  never touches credentials: it runs your own `claude` binary, on your machine, for your own
+  evals. For CI or team-wide use, use an API key (`EVALPROOF_CLI_AUTH=api-key`, or
+  `api_key_env`). See Anthropic's
+  [legal and compliance page](https://code.claude.com/docs/en/legal-and-compliance). Nothing
+  uses a CLI unless a suite names a `claude-code:` or `codex:` target (or an experimental one,
+  below).
+
+### Experimental providers
+
+`opencode:`, `cursor:` and `copilot:` follow the same rules as the two above: your own logged-in
+binary, run as a subprocess in a fresh empty directory, its API-key variable removed from the
+child's environment unless `api_key_env` names one, a bounded runtime, and the CLI, its version
+and an isolation tag (`opencode/0`, `cursor/0`, `copilot/0`) in the run and in the cache key.
+They are experimental because their isolation is less proven than Claude Code's and Codex's.
+Each prints `evalproof: <name> provider is experimental` on stderr the first time it is used in
+a process. Like the other two they ignore `temperature`, `seed` and `max_tokens`, and
+`evalproof doctor` without a suite lists each of them whose binary is on `PATH`. What was
+checked, and what was not:
+
+* **`opencode:<provider>/<model>`** runs `opencode run --standalone --archived --format json`
+  with a custom agent that has every tool off and every permission denied, your system text as
+  that agent's prompt, and project config, the global config directory and the Claude Code
+  compatibility files switched off. The prompt is on stdin. *Verified* only against local
+  ollama models (`qwen3:8b`, a handful of calls): no tool event, a canary `AGENTS.md` in the
+  project not read, a canary `AGENTS.md` in the global config directory read by default and not
+  read with these settings, the parser and the error path. *Not verified:* any hosted model;
+  whether `~/.claude/CLAUDE.md` is read (that test was inconclusive); whether the empty
+  `OPENCODE_CONFIG_DIR` hides a login stored there; the token counts (input adds cache reads
+  and writes, output adds reasoning, summed per field name). OpenCode has no single login, so
+  `doctor` prints `auth: not checked`. Only `OPENCODE_API_KEY` is removed from the environment:
+  the key of the model's own provider (`ANTHROPIC_API_KEY` for `opencode:anthropic/...`) is
+  left, because naming that provider in the model id is choosing its key, and removing it would
+  break CI. `--archived` keeps a session out of OpenCode's lists but does not delete it, so
+  sessions accumulate in OpenCode's data directory (each run is a private `--standalone`
+  server in a temporary project); prune them yourself.
+* **`cursor:<model>`** runs `cursor-agent -p --mode ask --trust` with an empty `--workspace`.
+  *Not verified:* Cursor was not logged in on the test machine, so no call succeeded. The flags
+  come from the CLI's help and documentation, and the parser from its source; it is lenient (a
+  JSON result object, an array of events, terminal noise around it) and an unexpected shape
+  fails with a message that says what was expected. The empty workspace is the only isolation:
+  no flag ignores rules files, user-level rules were not checked, and whether ask mode really
+  has no tools is untested. There is no system-prompt option, so your system text goes in front
+  of the prompt under a heading, and the prompt is an argument (nothing says the CLI reads
+  stdin), so it is visible in `ps` while the call runs and a prompt over 100,000 bytes is
+  refused. The key variable is `CURSOR_API_KEY`, passed through the environment, never as
+  `--api-key`. `doctor` reads `cursor-agent about` for the login state (`status` hung) and
+  prints no account.
+* **`copilot:<model>`** runs `copilot -s --no-auto-update --no-ask-user --available-tools= -p`.
+  *Not verified:* Copilot CLI was not installed on the test machine; every flag comes from
+  GitHub's documentation. `--available-tools=` is an allow-list with nothing in it, the most
+  restrictive documented tool option; whether the CLI accepts an empty list was not tried. No
+  `--allow-*` flag is ever passed, so a tool that needs approval cannot get it. The reply is
+  plain text, so token counts are 0. `COPILOT_HOME` is not redirected, so user-level Copilot
+  instructions may still be read. System text and prompt travel as for Cursor. Only
+  `COPILOT_GITHUB_TOKEN` is removed; `GH_TOKEN` and `GITHUB_TOKEN` are left because `gh` and CI
+  set them for other reasons. `doctor` prints `auth: not checked`.
+
 ## Concepts
 
 **Suite** — one YAML file: a JSONL golden set, the system under test, scorers, an optional
@@ -174,7 +328,8 @@ gate: {metric: judge, max_drop: 0.03, alpha: 0.05, min_cases: 40, max_p95_latenc
 
 **Run** — `evalproof run suite.yaml` writes one self-describing JSON file: every case's
 output, latency, tokens and scores, plus the dataset hash, seed, Ollama version and model
-digests.
+digests, and for a `claude-code` or `codex` target the CLI, its version and the model that
+answered.
 
 **Scorers** — `exact_match`, `contains`, `regex` (accept/reject answer keys),
 `json_schema`, `numeric_tolerance`, `recall_at_k`, `mrr`, `ndcg`, and `llm_judge`: a strict
@@ -199,7 +354,7 @@ orders and reports order-dependent verdicts as ties.
 | 0 | pass — possibly with warnings (a big drop that is not significant; a significant drop within tolerance) |
 | 1 | quality regression: drop ≥ `max_drop` **and** p < `alpha` |
 | 3 | budget breach: candidate p95 latency or cost over the limit |
-| 4 | cannot gate: fewer than `min_cases` paired cases, or the runs used different data |
+| 4 | cannot gate: fewer than `min_cases` paired cases, the runs used different data, or the judge changed (unless `--allow-judge-change`) |
 | 2 | usage or configuration error |
 
 The gate checks one metric by default. To gate several, repeat `--metric` or list them in
@@ -304,6 +459,7 @@ flowchart LR
 | Local Ollama by default, HTTP-only providers, content-addressed cache | Hosted APIs and vendor SDKs | Reproducible, free to re-run, tied to a model digest; no SDK dependency | [0002](docs/adr/0002-local-first-providers.md) |
 | Gate on "large AND significant", on paired cases | "Fail if the score dropped by X" | On a few dozen cases, X-point drops are mostly noise, and a noisy gate gets switched off | [0003](docs/adr/0003-gate-on-significance-not-delta.md) |
 | One metric by default; several with Holm correction | Gate every metric at α each | Testing many metrics at α = 0.05 each inflates false failures | 0003 |
+| Run a coding-agent CLI as a provider only in isolation, and gate on its fingerprint | Pass the CLI's defaults through, or ignore its version | A CLI that reads your project's instructions is not a judge, and a silent upgrade changes the instrument | [0004](docs/adr/0004-agent-clis-as-providers.md) |
 | Pointwise verdicts for gating; pairwise as a diagnostic | Pairwise preference for gating | Measured: 42% of pairwise verdicts flipped with answer order, with a strong length preference | [MEASUREMENTS §6](MEASUREMENTS.md) |
 | numpy and the standard library for statistics | SciPy, statsmodels | Small enough to read and to test against hand-worked values | — |
 
@@ -313,7 +469,7 @@ The composite action runs `compare --gate`, appends the report to the job summar
 fails the job on exit codes 1, 3 or 4:
 
 ```yaml
-- uses: technobasant/evalproof@v0.3.0
+- uses: technobasant/evalproof@v0.4.0
   with:
     baseline: eval/baseline.json     # produced on main, e.g. as a build artifact
     candidate: eval/candidate.json   # produced on this branch
@@ -324,6 +480,14 @@ fails the job on exit codes 1, 3 or 4:
 This repository's own CI runs the offline example on every commit and asserts that the
 gate returns 1 for the real regression and 0 for the noisy and refactored candidates.
 
+If the judge is a Claude Code or Codex CLI, `evalproof init --ci api-key` writes a workflow that
+installs the CLI at the version found at init time and runs `evals/suite.yaml` as it is. The run
+step sets `EVALPROOF_CLI_AUTH: api-key` and the repository secret (`ANTHROPIC_API_KEY` for
+Claude Code, `OPENAI_API_KEY` for Codex), so the CLI uses that key instead of a login. The
+suite names no key and there is no second suite file; a suite with an `api_key_env` still uses
+that variable. Keep the pinned CLI version equal to the one in your baseline run, because the
+gate exits 4 when the judge's CLI version differs.
+
 ## How it relates to other tools
 
 [promptfoo](https://www.promptfoo.dev/docs/intro/) (open-source, local evals of prompts,

{evalproof-0.3.0 → evalproof-0.4.0}/src/evalproof/_version.py

@@ -1,3 +1,3 @@
 """The package version: the single place it is written. pyproject.toml reads it at build time."""
 
-__version__ = "0.3.0"
+__version__ = "0.4.0"