evalproof 0.3.0__tar.gz → 0.4.1__tar.gz

{evalproof-0.3.0 → evalproof-0.4.1}/CHANGELOG.md

@@ -4,6 +4,114 @@ All notable changes are listed here. The project follows [Semantic Versioning](h
 
 ## [Unreleased]
 
+## [0.4.1] — 2026-11-13
+
+### Fixed
+
+- `cursor:` ask mode is not tool-free: it has a read tool that runs without approval and reads
+  files in the workspace and outside it. The workspace now carries a `.cursor/cli.json` that
+  allows nothing and denies reads (verified inside and outside the workspace), writes and shell
+  commands (ask mode has no write or shell tool; those entries are a second guard), and any
+  `tool_call` event in the output fails the call as an isolation break.
+- `cursor:` wrote every call's chat to `~/.cursor/chats` and read the user's
+  `~/.cursor/cli-config.json`. `CURSOR_CONFIG_DIR` and `CURSOR_DATA_DIR` now point into the
+  call's temp root, which keeps the login (verified) and starts from a config that allows
+  nothing.
+- `copilot:` kept every call's session in `~/.copilot` and read the user's Copilot config.
+  `COPILOT_HOME` now points into the call's temp root, which keeps the login (verified).
+- `copilot:` passed `--available-tools=`, which copilot 1.0.110 ignores, so the model could use
+  its tools (it read a file). It now passes an allow-list with one unknown name, which disables
+  every tool (verified), and any tool event or tool request in the output fails the call.
+
+### Changed
+
+- `cursor:` uses `--output-format stream-json` (the json result joined the model's preamble
+  with the answer).
+- `copilot:` takes the prompt on stdin, prints JSONL, runs with `--no-custom-instructions` and
+  `--disable-builtin-mcps`, reports real token counts from `--usage-output-file`, and records
+  the model that answered each call: with `copilot:auto` Copilot routes every call, so the run
+  artifact shows the routed model, not `auto`.
+- Isolation tags are now `cursor/2` and `copilot/2`, so earlier cached replies for these two
+  are not reused.
+- `evalproof doctor` warns that `cursor-agent` sends the names and descriptions of the user's
+  skills (`~/.claude/skills`, `~/.cursor/skills` and others) in every prompt, which no option
+  turns off, and that `copilot:auto` as the judge makes compare and the gate refuse runs as a
+  judge change (Copilot picks the model per call).
+
+### Documentation
+
+- The experimental providers section now says what was verified and what was not: Cursor
+  reads instruction files from the directories above the workspace; Cursor and Copilot keep
+  their own system prompt; Copilot's `--no-custom-instructions` was verified with canary
+  files; the Copilot tool guard has not fired live; named-model availability and the
+  premium-request rate depend on plan and policy; with tools off Copilot may write a made-up
+  tool call as text.
+
+## [0.4.0] — 2026-11-13
+
+### Added
+
+- `claude-code:<model>` and `codex:<model>` providers: run your own logged-in Claude Code or
+  Codex CLI as a judge or a system under test, with no API key. Each call runs in a fresh
+  empty directory with the agent's tools, settings, MCP servers, skills and memory switched
+  off and no permission-bypass flag; API-key variables are removed from the child's
+  environment unless `api_key_env` names one or `EVALPROOF_CLI_AUTH=api-key` keeps them.
+  Neither CLI takes a temperature, seed or token limit, so those params are ignored (Codex
+  maps `think` to its reasoning effort). Claude Code runs with no tools; Codex still offers a
+  JavaScript isolate, sub-agents and a question to the user, so any item in its output other
+  than a message, reasoning, an error or a plan fails the call as an isolation break, and a
+  codex too old for the flags evalproof sets gets a message to upgrade. ADR 0004 has the
+  reasoning.
+- `EVALPROOF_CLI_AUTH` (`login`, the default, or `api-key`) chooses whether a CLI provider
+  uses the subscription login or its own API key variable; `evalproof doctor` prints the mode
+  in effect and warns when `api-key` has no key to use. `api_key_env` still wins.
+- `evalproof doctor [suite.yaml]`: reports each CLI's path, version, login mode and the
+  isolation flags it will use, without a model call, and exits 1 when a target of the suite
+  is not usable. Without a suite it surveys the installed CLIs and exits 1 only when none is
+  usable. It prints no account details.
+- Run facts for CLI targets in `runner.environment`: `{judge,system}_cli`,
+  `{judge,system}_cli_version`, `{judge,system}_isolation` and `{judge,system}_model_resolved`
+  (the model ids that answered, sorted, cached replies included). The response cache keys on
+  the CLI fingerprint, so a CLI upgrade is a cache miss.
+- `compare` reports `same_judge`, `judge_changes` and `system_changes`, and the gate refuses
+  to decide (exit 4) when the judge changed between the runs. `--allow-judge-change` or
+  `gate.allow_judge_change: true` accepts the change and keeps a warning.
+- Experimental `opencode:<provider>/<model>`, `cursor:<model>` and `copilot:<model>`
+  providers, on the same footing as the CLI providers above (fresh empty directory, the CLI's
+  key variable removed unless `api_key_env` names one, bounded runtime, CLI fingerprint with the
+  isolation tags `opencode/0`, `cursor/0`, `copilot/0`). Each prints one warning on stderr the
+  first time it is used. Only OpenCode was exercised for real, and only against a local ollama
+  model; Cursor (not logged in) and Copilot (not installed) follow their documentation, and the
+  README's "Experimental providers" says what was and was not verified. `evalproof doctor`
+  without a suite now also lists whichever of these CLIs are on `PATH`.
+- `evalproof.providers.TransientError`, a `ProviderError` that `with_retries` retries; the CLI
+  providers raise it for rate limits and server errors.
+- `tests/test_live_agent_cli.py`: one real call per installed CLI, skipped unless
+  `EVALPROOF_LIVE=1`.
+- `evalproof init [--dir PATH] [--judge TARGET] [--ci keyless|api-key|none] [--no-skill]
+  [--force]`: writes `evals/suite.yaml`, three sample cases in `evals/cases.jsonl`, an
+  `evals/README.md`, a GitHub Actions workflow at `.github/workflows/evalproof.yml` and an
+  agent skill at `.agents/skills/evalproof/SKILL.md` and `.claude/skills/evalproof/SKILL.md`.
+  The judge is the first of `claude`, `codex`, `ollama` found on `PATH`, else a hosted
+  Anthropic model. The keyless workflow makes no model calls and gates the committed
+  `evals/runs/latest.json` against the base branch; the api-key workflow runs the suite in CI
+  (with `EVALPROOF_CLI_AUTH: api-key` for a CLI judge, so no second suite file is written)
+  and pins the judge CLI to the version found at init time. Existing files are kept unless
+  `--force` is passed, nothing is written outside `--dir`, and no model is called.
+
+### Changed
+
+- A gate that passed in 0.3.0 can now exit 4 when the judge's target, CLI version, isolation
+  or resolved model differs between two runs that both carry these keys. Runs written before
+  this release carry none and compare as before.
+- A changed system target or CLI version between two runs now adds a warning, so the status
+  is `warn` instead of `pass` (the exit code stays 0).
+- `runner.environment` records `judge_target` and `system_target` for every model target (not
+  only CLI ones), and the system target it records is the one in effect, with any override
+  applied.
+- The gate's "too few paired cases" reason now says how to fix it (add cases, or lower
+  `gate.min_cases`).
+
 ## [0.3.0] — 2026-11-12
 
 ### Added
@@ -179,7 +287,9 @@ First release.
 - Two examples: an offline ticket router (CI's gate contract) and a product support
   assistant judged by a local model, with measurements.
 
-[Unreleased]: https://github.com/technobasant/evalproof/compare/v0.3.0...HEAD
+[Unreleased]: https://github.com/technobasant/evalproof/compare/v0.4.1...HEAD
+[0.4.1]: https://github.com/technobasant/evalproof/compare/v0.4.0...v0.4.1
+[0.4.0]: https://github.com/technobasant/evalproof/compare/v0.3.0...v0.4.0
 [0.3.0]: https://github.com/technobasant/evalproof/compare/v0.2.1...v0.3.0
 [0.2.1]: https://github.com/technobasant/evalproof/compare/v0.2.0...v0.2.1
 [0.2.0]: https://github.com/technobasant/evalproof/compare/v0.1.2...v0.2.0

{evalproof-0.3.0 → evalproof-0.4.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: evalproof
-Version: 0.3.0
+Version: 0.4.1
 Summary: Treat LLM evaluation as measurement: calibrate the judge, correct the pass rate, gate CI on significant regressions.
 Project-URL: Homepage, https://github.com/technobasant/evalproof
 Project-URL: Documentation, https://github.com/technobasant/evalproof#readme
@@ -100,6 +100,23 @@ questions, one Apple M5. Full method and every table in [MEASUREMENTS.md](https:
 
 ## Five-minute quickstart
 
+In your own repository, one command writes a starter suite, a CI workflow and an agent skill,
+and a second runs it:
+
+```bash
+uvx evalproof init
+uvx evalproof run evals/suite.yaml --out evals/runs/latest.json
+```
+
+`init` picks the judge from what is on your `PATH`: a logged-in `claude`, then `codex`, then
+Ollama, then a hosted Anthropic model with `ANTHROPIC_API_KEY`. It writes `evals/` (the
+suite, three sample cases, a README), `.github/workflows/evalproof.yml` (`--ci keyless`, the
+default, makes no model calls in CI; `--ci api-key` runs the suite there; `--ci none` writes
+no workflow) and a skill for coding agents at `.agents/skills/evalproof/SKILL.md` and
+`.claude/skills/evalproof/SKILL.md` (`--no-skill` skips it). It prints one line per file,
+never overwrites one without `--force`, and makes no model call. Replace the sample cases
+with your own: the gate needs 20 before it will decide.
+
 The offline example lives in the repository:
 
 ```bash
@@ -135,7 +152,11 @@ The target is `<provider>:<model>`; the model name is everything after the first
 `openrouter:anthropic/claude-sonnet-4.5` and `ollama:qwen3:8b` both work. Two native adapters
 (`ollama`, `anthropic`) and one OpenAI-compatible adapter driven by a table of provider
 profiles ([`providers/profiles.py`](https://github.com/technobasant/evalproof/blob/main/src/evalproof/providers/profiles.py), each row cites the
-provider's documentation). Plain HTTP throughout, no vendor SDK.
+provider's documentation). Plain HTTP throughout, no vendor SDK. Two more targets run a coding
+agent you already have logged in, `claude-code` and `codex`: see
+[Use your Claude Code or Codex login](#use-your-claude-code-or-codex-login). Three experimental
+ones run OpenCode, Cursor and GitHub Copilot the same way: see
+[Experimental providers](#experimental-providers).
 
 | Provider | Example target | Key variable | Where |
 |---|---|---|---|
@@ -156,6 +177,11 @@ provider's documentation). Plain HTTP throughout, no vendor SDK.
 | `openrouter` | `openrouter:anthropic/claude-sonnet-4.5` | `OPENROUTER_API_KEY` | hosted |
 | `cerebras` | `cerebras:gpt-oss-120b` | `CEREBRAS_API_KEY` | hosted |
 | `openai-compatible` | `openai-compatible:<model>` | optional, via `api_key_env` | any server, needs `base_url` |
+| `claude-code` | `claude-code:haiku` | none: your Claude Code login | local CLI, see below |
+| `codex` | `codex:gpt-6-luna` | none: your ChatGPT login | local CLI, see below |
+| `opencode` | `opencode:ollama/qwen3:8b` | none: OpenCode's own login, per model provider | local CLI, experimental |
+| `cursor` | `cursor:auto` | none: your Cursor login | local CLI, experimental |
+| `copilot` | `copilot:auto` | none: your GitHub Copilot login | local CLI, experimental |
 
 ```yaml
 system:
@@ -186,6 +212,187 @@ system:
 * AWS Bedrock and Google Vertex AI sign requests with cloud credentials, which evalproof does
   not implement. Use a gateway's OpenAI-compatible endpoint through `openai-compatible`.
 
+## Use your Claude Code or Codex login
+
+If Claude Code or Codex is already logged in on your machine, either can be the judge or the
+system under test, with no API key:
+
+```yaml
+judge:
+  target: claude-code:haiku        # an alias (haiku, sonnet, opus) or a full model id
+system:
+  target: codex:gpt-6-luna
+concurrency: 4
+```
+
+Run `evalproof doctor suite.yaml` first. It prints each CLI's path, version, login mode and
+the isolation flags it will pass, and exits 1 if a target of the suite is not usable. Without a
+suite it surveys every CLI it knows that is installed, and exits 1 only when none is usable. It
+makes no model call.
+
+evalproof runs your own `claude` or `codex` binary as a subprocess, sends the prompt on stdin
+and reads the JSON it prints. It never reads a token file or the keychain. The environment
+variable `EVALPROOF_CLI_AUTH` says which credentials the CLI uses:
+
+* `login` (the default, also when it is unset or empty) removes the CLI's key variables from
+  the call (`ANTHROPIC_API_KEY` and `ANTHROPIC_AUTH_TOKEN` for Claude Code, `OPENAI_API_KEY`
+  and `CODEX_API_KEY` for Codex), so the login is what gets used.
+* `api-key` leaves them in place, so the CLI uses its own key. This is the route for CI.
+* Any other value is an error.
+
+`api_key_env` in the suite works in either mode and wins: it passes that variable to the CLI as
+its own key variable. The mode is not part of the run's fingerprint or the cache key, because
+it is the same model either way. `evalproof doctor` prints the mode in effect and warns when
+`api-key` is chosen but the CLI's key variable is not set.
+
+* **Isolation is a requirement, not an option.** A coding agent is not a model endpoint. Left
+  alone it loads the project's `CLAUDE.md` or `AGENTS.md`, your memory, plugins, MCP servers
+  and skills into the prompt, and it can run commands. A judge that has read your project's
+  instructions is not a judge. So every call runs in a fresh empty directory, with the
+  agent's own system prompt replaced by yours (or one neutral line), its tools, settings, MCP
+  servers and skills switched off, and no permission-bypass flag. `evalproof doctor` prints
+  the exact flags. The two CLIs do not isolate equally well:
+  * Claude Code is run with **no tools at all** (`--tools ''`).
+  * Codex runs in a read-only sandbox with its shell, plugins, apps, hooks, multi-agent mode,
+    sleep and image tools, bundled skills and your user config and rules switched off, but
+    codex-cli 0.167.2 has no flag that removes every tool. Still offered are a JavaScript
+    isolate (`exec`: no Node, no file system, no network, with a few built-in helpers such as
+    the current time, MCP resource listing and `apply_patch`), sub-agents that get the same
+    restricted tools, and a question to the user, which cannot be answered in exec mode. **Any
+    tool use fails the call**: if the output holds an item that is not a message, reasoning,
+    an error or a plan, evalproof raises `codex used a tool (<type>); isolation broken` and
+    discards the reply. A codex too old to know a flag evalproof sets exits with `Unknown
+    feature flag`, and evalproof says to upgrade it (it was tested with codex-cli 0.167.2).
+* **No temperature, seed or token limit.** Neither CLI exposes one, so `temperature`, `seed`
+  and `max_tokens` are ignored. `think` maps to Codex's reasoning effort (`false` is `low`,
+  `true` is `medium`; `params.extra.reasoning_effort` sets it directly). A run is therefore
+  not repeatable the way a seeded local run is, so the run records the CLI, its version, the
+  isolation scheme and the model id that actually answered, and the cache keys on the CLI
+  version. To see how much a judge wobbles, score the same outputs twice with `--no-cache`, or
+  [calibrate](#concepts) it against labels.
+* **A changed judge is not comparable.** `compare` and the gate refuse (exit 4) when the judge's
+  target, CLI version, isolation or resolved model differs between the two runs, because the
+  scores then come from a different instrument. Pass `--allow-judge-change` or set
+  `gate.allow_judge_change: true` to accept it. A changed system target or CLI version only
+  warns, since that is usually the change under test. Runs written before 0.4.0 carry none of
+  these keys and compare as before.
+* **Latency and cost.** A call takes about 1.4 s on `claude-code:haiku` and about 4 s on
+  Codex, mostly start-up, so set `concurrency`. There is no API bill; the calls count against
+  your subscription's usage limits.
+* **Terms.** OpenAI documents Codex CLI use on a ChatGPT plan, including automation.
+  Anthropic's terms allow a user to run the unmodified Claude Code binary on their own
+  subscription; its plan limits assume ordinary individual use, and it forbids third-party
+  products from routing other users' requests through subscription credentials. evalproof
+  never touches credentials: it runs your own `claude` binary, on your machine, for your own
+  evals. For CI or team-wide use, use an API key (`EVALPROOF_CLI_AUTH=api-key`, or
+  `api_key_env`). See Anthropic's
+  [legal and compliance page](https://code.claude.com/docs/en/legal-and-compliance). Nothing
+  uses a CLI unless a suite names a `claude-code:` or `codex:` target (or an experimental one,
+  below).
+
+### Experimental providers
+
+`opencode:`, `cursor:` and `copilot:` follow the same rules as the two above: your own logged-in
+binary, run as a subprocess in a fresh directory in the system temp directory (empty but for
+Cursor's permission file), its API-key variable removed from the child's environment unless
+`api_key_env` names one, a bounded runtime, and the CLI, its version and an isolation tag
+(`opencode/0`, `cursor/2`, `copilot/2`) in the run and in the cache key. They are experimental
+because their isolation is less proven than Claude Code's and Codex's. Each prints
+`evalproof: <name> provider is experimental` on stderr the first time it is used in a process.
+Like the other two they ignore `temperature`, `seed` and `max_tokens`, and `evalproof doctor`
+without a suite lists each of them whose binary is on `PATH`. One difference from the two
+above: Cursor and Copilot have no system-prompt option, so they keep their own agent system
+prompt and your system text goes in front of the prompt under a heading, as an instruction in
+the user turn. What was checked, and what was not:
+
+* **`opencode:<provider>/<model>`** runs `opencode run --standalone --archived --format json`
+  with a custom agent that has every tool off and every permission denied, your system text as
+  that agent's prompt, and project config, the global config directory and the Claude Code
+  compatibility files switched off. The prompt is on stdin. *Verified* only against local
+  ollama models (`qwen3:8b`, a handful of calls): no tool event, a canary `AGENTS.md` in the
+  project not read, a canary `AGENTS.md` in the global config directory read by default and not
+  read with these settings, the parser and the error path. *Not verified:* any hosted model;
+  whether `~/.claude/CLAUDE.md` is read (that test was inconclusive); whether the empty
+  `OPENCODE_CONFIG_DIR` hides a login stored there; the token counts (input adds cache reads
+  and writes, output adds reasoning, summed per field name). OpenCode has no single login, so
+  `doctor` prints `auth: not checked`. Only `OPENCODE_API_KEY` is removed from the environment:
+  the key of the model's own provider (`ANTHROPIC_API_KEY` for `opencode:anthropic/...`) is
+  left, because naming that provider in the model id is choosing its key, and removing it would
+  break CI. `--archived` keeps a session out of OpenCode's lists but does not delete it, so
+  sessions accumulate in OpenCode's data directory (each run is a private `--standalone`
+  server in a temporary project); prune them yourself.
+* **`cursor:<model>`** runs `cursor-agent -p --output-format stream-json --mode ask --trust`
+  with a `--workspace` that holds only a `.cursor/cli.json` permission file (nothing allowed;
+  reads, writes and shell commands denied), and with `CURSOR_CONFIG_DIR` and `CURSOR_DATA_DIR`
+  pointed at empty directories in the call's temp root. *Verified live* on 2026-11-13 with
+  cursor-agent 2026.11.11:
+  * Ask mode has no write or shell tool, but it is **not** tool-free: it has a read tool that
+    runs without approval and reads files inside the workspace and outside it by absolute path.
+    The permission file made every read fail with "Permission denied", inside and outside, but
+    the model still tries, so the stream carries a `tool_call` event; evalproof fails the call
+    on any such event (`cursor used a tool (readToolCall); isolation broken`) and discards the
+    reply. The write and shell entries in the deny list are a second guard for later versions;
+    only reads were tested.
+  * Cursor reads instruction files (`AGENTS.md`, `CLAUDE.md`, `.cursor/rules`) from the
+    workspace **and from the directories above it**. The workspace is empty and sits in the
+    system temp directory, so nothing is found there unless `TMPDIR` points inside a project.
+    `~/.claude/CLAUDE.md` was not read.
+  * **Your skills list is sent.** Cursor puts the names and descriptions of the skills under
+    your home directory (`~/.claude/skills`, `~/.cursor/skills`, `~/.codex/skills`,
+    `~/.agents/skills`, plugin skills) and its own subagents in every prompt. The paths are
+    fixed and no setting turns them off; moving `HOME` logs the CLI out. They count as input
+    tokens on every call, and `doctor` warns about it. Verified with a login; with
+    `CURSOR_API_KEY` it was not tested.
+  * The redirected config and data directories keep the login (it is in the system keychain),
+    keep each call's chat out of `~/.cursor/chats` and `~/.cursor/projects`, and mean your own
+    `~/.cursor/cli-config.json` (and its allow list) is not read. A fresh config directory gets
+    a default allow list with `Shell(ls)`, so evalproof writes one that allows nothing.
+
+  `stream-json` rather than `json` because the json result joins the model's preamble text
+  ("I'll read the file now.") with the answer and carries no tool events. Token counts come from
+  the result event (output includes reasoning). The model recorded is the one you asked for;
+  the result carries no model id. The prompt is an argument (nothing says the CLI reads stdin),
+  so it is visible in `ps` while the call runs and a prompt over 100,000 bytes is refused. The
+  key variable is `CURSOR_API_KEY`, passed through the environment, never as `--api-key`.
+  `doctor` reads `cursor-agent about` for the login state (`status` hung) and prints no account.
+* **`copilot:<model>`** runs `copilot --no-auto-update --no-ask-user --no-custom-instructions
+  --disable-builtin-mcps --available-tools=evalproof-no-tools --output-format json
+  --usage-output-file <tmp>`, with the prompt on stdin and `COPILOT_HOME` pointed at an empty
+  directory in the call's temp root. *Verified live* on 2026-11-13 with copilot 1.0.110:
+  * An empty allow-list (`--available-tools=`) is **ignored**: the model called its `view` tool
+    and read a file. An allow-list of one name that is no tool disables every tool
+    (`session.info: Disabled tools: bash, create, edit, glob, grep, ...`), and that is what
+    evalproof passes. Any `tool.execution*` event or tool request in the output also fails the
+    call (`copilot used a tool (view); isolation broken`); with tools off it never fired live,
+    so that guard is tested only against canned output. No `--allow-*` flag is ever passed.
+    With no tools the model sometimes writes a made-up tool call and its "result" as plain
+    text; that is reply text, not a tool run, and is scored as such.
+  * `--no-custom-instructions` works: without it a canary `AGENTS.md`, `CLAUDE.md` and
+    `.github/copilot-instructions.md` in the working directory were all read; with it none was,
+    nor an `AGENTS.md` in the directory above. No skills or MCP servers were offered.
+  * Copilot keeps its own coding-agent system prompt and adds the date and a note that it is
+    not sandboxed; your system text is an instruction in the user turn.
+  * `COPILOT_HOME` in the temp root keeps the login (it is in the system keychain) and keeps
+    each call's session, logs and config out of `~/.copilot` (verified: no new file there after
+    a call). Your own Copilot config, plugins and MCP servers live in that directory, so they
+    are not used either; that follows from the redirect and was not tested separately.
+  * Models and cost: on the test account named models (`gpt-6-luna`, `claude-haiku-5.5`) were
+    rejected with "is not available", and only `copilot:auto` worked; which models you can name
+    depends on your plan and your organisation's policy. `auto` routes each call to a model of
+    Copilot's choosing (`gpt-6-luna` on one call, `mai-code-1.1-flash` on another), so the run
+    artifact records every model that answered, not `auto`, and a suite run on `auto` can mix
+    models. `auto` used one premium request per call there; the rate depends on the model and
+    the plan.
+  * **`copilot:auto` makes a poor judge.** The judge's resolved model is part of what
+    [`compare` and the gate check](#use-your-claude-code-or-codex-login): two runs on `auto` often saw
+    different models, so they are refused as a judge change (exit 4) unless you allow it.
+    `doctor` warns about it. Name a model for the judge if your plan allows one.
+
+  Token counts are read from the usage file (input already includes cache reads and writes,
+  output includes reasoning); a missing file means 0, never a failure. Only
+  `COPILOT_GITHUB_TOKEN` is removed; `GH_TOKEN` and `GITHUB_TOKEN` are left because `gh` and CI
+  set them for other reasons. `doctor` prints `auth: not checked`.
+
 ## Concepts
 
 **Suite** — one YAML file: a JSONL golden set, the system under test, scorers, an optional
@@ -210,7 +417,8 @@ gate: {metric: judge, max_drop: 0.03, alpha: 0.05, min_cases: 40, max_p95_latenc
 
 **Run** — `evalproof run suite.yaml` writes one self-describing JSON file: every case's
 output, latency, tokens and scores, plus the dataset hash, seed, Ollama version and model
-digests.
+digests, and for a `claude-code` or `codex` target the CLI, its version and the model that
+answered.
 
 **Scorers** — `exact_match`, `contains`, `regex` (accept/reject answer keys),
 `json_schema`, `numeric_tolerance`, `recall_at_k`, `mrr`, `ndcg`, and `llm_judge`: a strict
@@ -235,7 +443,7 @@ orders and reports order-dependent verdicts as ties.
 | 0 | pass — possibly with warnings (a big drop that is not significant; a significant drop within tolerance) |
 | 1 | quality regression: drop ≥ `max_drop` **and** p < `alpha` |
 | 3 | budget breach: candidate p95 latency or cost over the limit |
-| 4 | cannot gate: fewer than `min_cases` paired cases, or the runs used different data |
+| 4 | cannot gate: fewer than `min_cases` paired cases, the runs used different data, or the judge changed (unless `--allow-judge-change`) |
 | 2 | usage or configuration error |
 
 The gate checks one metric by default. To gate several, repeat `--metric` or list them in
@@ -322,6 +530,7 @@ The architecture diagram is in the [README on GitHub](https://github.com/technob
 | Local Ollama by default, HTTP-only providers, content-addressed cache | Hosted APIs and vendor SDKs | Reproducible, free to re-run, tied to a model digest; no SDK dependency | [0002](https://github.com/technobasant/evalproof/blob/main/docs/adr/0002-local-first-providers.md) |
 | Gate on "large AND significant", on paired cases | "Fail if the score dropped by X" | On a few dozen cases, X-point drops are mostly noise, and a noisy gate gets switched off | [0003](https://github.com/technobasant/evalproof/blob/main/docs/adr/0003-gate-on-significance-not-delta.md) |
 | One metric by default; several with Holm correction | Gate every metric at α each | Testing many metrics at α = 0.05 each inflates false failures | 0003 |
+| Run a coding-agent CLI as a provider only in isolation, and gate on its fingerprint | Pass the CLI's defaults through, or ignore its version | A CLI that reads your project's instructions is not a judge, and a silent upgrade changes the instrument | [0004](https://github.com/technobasant/evalproof/blob/main/docs/adr/0004-agent-clis-as-providers.md) |
 | Pointwise verdicts for gating; pairwise as a diagnostic | Pairwise preference for gating | Measured: 42% of pairwise verdicts flipped with answer order, with a strong length preference | [MEASUREMENTS §6](https://github.com/technobasant/evalproof/blob/main/MEASUREMENTS.md) |
 | numpy and the standard library for statistics | SciPy, statsmodels | Small enough to read and to test against hand-worked values | — |
 
@@ -331,7 +540,7 @@ The composite action runs `compare --gate`, appends the report to the job summar
 fails the job on exit codes 1, 3 or 4:
 
 ```yaml
-- uses: technobasant/evalproof@v0.3.0
+- uses: technobasant/evalproof@v0.4.1
   with:
     baseline: eval/baseline.json     # produced on main, e.g. as a build artifact
     candidate: eval/candidate.json   # produced on this branch
@@ -342,6 +551,14 @@ fails the job on exit codes 1, 3 or 4:
 This repository's own CI runs the offline example on every commit and asserts that the
 gate returns 1 for the real regression and 0 for the noisy and refactored candidates.
 
+If the judge is a Claude Code or Codex CLI, `evalproof init --ci api-key` writes a workflow that
+installs the CLI at the version found at init time and runs `evals/suite.yaml` as it is. The run
+step sets `EVALPROOF_CLI_AUTH: api-key` and the repository secret (`ANTHROPIC_API_KEY` for
+Claude Code, `OPENAI_API_KEY` for Codex), so the CLI uses that key instead of a login. The
+suite names no key and there is no second suite file; a suite with an `api_key_env` still uses
+that variable. Keep the pinned CLI version equal to the one in your baseline run, because the
+gate exits 4 when the judge's CLI version differs.
+
 ## How it relates to other tools
 
 [promptfoo](https://www.promptfoo.dev/docs/intro/) (open-source, local evals of prompts,