reeval 0.9.0 → 0.11.0

package/README.md

@@ -64,8 +64,8 @@ evaluate(
 )
 ```
 
-- **Real agents in sandboxes.** Cursor, Claude Code, Codex, Antigravity, OpenCode, Goose and pi
-  each work in a fresh local Docker container. Their API keys stay on the host: the sandbox gets
+- **Real agents in sandboxes.** Cursor, Claude Code, Codex, Antigravity, OpenCode, Goose, pi, fx
+  and Grok Build each work in a fresh local Docker container. Their API keys stay on the host: the sandbox gets
   a placeholder, and reeval's egress gateway swaps in the key only on requests to that provider.
 - **Any environment.** Start from a git repository at a pinned ref, then add files, setup steps,
   rules, skills, MCP servers and secrets, on reeval's image or your own `from` or Dockerfile.
@@ -74,9 +74,9 @@ evaluate(
   nightly.
 - **Checks and judges.** Deterministic checks on files, commands and the transcript decide pass
   or fail; an LLM judge answers what they can't.
-- **Compare, questions and analysis.** `compare` reports each variant's lift over the control,
-  with a 95% interval and its effort; `questions` asks a model which attempt did better, pair by
-  pair and blind; `reeval analyze` writes a checked reading of a run.
+- **Compare, questions and analysis.** `compare` says whether each variant helped over the
+  control, made it worse or can't tell yet (`-v` adds the 95% interval), and its effort;
+  `questions` asks a model which attempt did better, pair by pair and blind; `reeval analyze` writes a checked reading of a run.
 - **Results in your repo.** Every run is a JSON record you can commit, diff with `reeval compare`,
   query with SQL (`reeval query`) and browse in `reeval studio`.
 
@@ -86,14 +86,32 @@ all three agents passed every check.
 ```text
 $ reeval run evals/api.eval.ts --runs 1
 …
-   ✓ how well the api-conventions skill influences the output      3/6 · pass@1 100% gated  6 attempts
-       cursor:composer-2.5/nextjs-api/docker                    control    0/1  pass@1   0%   not gated
-       cursor:composer-2.5/with-api-conventions/docker          variant    1/1  pass@1 100%   lift +100 pts (95% −12…+100, 1/1 vs 0/1), within noise
-       claude-code:claude-sonnet-5/nextjs-api/docker            control    0/1  pass@1   0%   not gated
-       claude-code:claude-sonnet-5/with-api-conventions/docker  variant    1/1  pass@1 100%   lift +100 pts (95% −12…+100, 1/1 vs 0/1), within noise
-       codex:gpt-5.6-luna/nextjs-api/docker                     control    0/1  pass@1   0%   not gated
-       codex:gpt-5.6-luna/with-api-conventions/docker           variant    1/1  pass@1 100%   lift +100 pts (95% −12…+100, 1/1 vs 0/1), within noise
-       question  "Which endpoint follows our API conventions?" across: claude-code:claude-sonnet-5 2/4 (50%), codex:gpt-5.6-luna 0/4 (0%), cursor:composer-2.5 4/4 (100%) · 6 votes, 0 ties · gemini-3.8-flash
+pass             how well the api-conventions skill influences the output
+  cursor:composer-2.5
+    arm                                            pass
+    nextjs-api                                     0 of 1
+    with-api-conventions                           1 of 1  can't tell yet
+    check                                          nextjs-api  with-api-conventions
+    toHaveFile app/api/users/route.ts              0 of 1      1 of 1
+      expected app/api/users/route.ts to match /cursor/i
+    toSucceed npm run build                        not run     1 of 1
+    toSucceed npm test                             not run     1 of 1
+    toSucceed cat > /tmp/probe-users.mjs <<'EOF'…  not run     1 of 1
+    toPass Do the tests follow nextCursor to the…  not run     1 of 1
+  claude-code:claude-sonnet-5
+    arm                                            pass
+    nextjs-api                                     0 of 1
+    with-api-conventions                           1 of 1  can't tell yet
+    …
+  codex:gpt-5.6-luna
+    arm                                            pass
+    nextjs-api                                     0 of 1
+    with-api-conventions                           1 of 1  can't tell yet
+    …
+  question  Which endpoint follows our API conventions?
+    claude-code:claude-sonnet-5  won 2 of 2
+    cursor:composer-2.5          won 1 of 4
+    codex:gpt-5.6-luna           won 1 of 2
 …
 ```
 
@@ -127,7 +145,8 @@ group('greet module', () => {
 
     await expect(turn).toHaveFile('greet.js')
     await expect(c.exec('node --test')).toSucceed()
-    expect(await c.judge("Does the agent's final message accurately describe what it changed?")).toPass()
+    const honest = "Does the agent's final message describe what it changed?"
+    expect(await c.judge(honest)).toPass()
   })
 })
 ```
@@ -139,8 +158,11 @@ import { cursor } from 'reeval/harness'
 
 export default defineConfig({
   include: ['evals/**/*.eval.ts'],
-  harness: cursor({ model: 'composer-2.5' }), // or claudeCode({ model }), antigravity({ model }), opencode({ model }), goose({ model }), pi({ model }), codex({ model }), or a list: each is its own cell
-  judge: { model: 'typesafe-ai/jev-1.13.0', threshold: 0.8 }, // optional; pin the version
+  // or claudeCode, codex, antigravity, opencode, goose, pi, fx, grokBuild, or a
+  // list of harnesses: each is its own cell
+  harness: cursor({ model: 'composer-2.5' }),
+  // optional; pin the version
+  judge: { model: 'typesafe-ai/jev-1.14.0', threshold: 0.8 },
   timeout: '15m',
   profiles: {
     pr: { limit: { runs: 1, compare: false }, budget: { attempts: 20 } },
@@ -185,75 +207,88 @@ group(
 
 ```text
 $ reeval run skill --runs 2
-plan  1 eval · 2 cells · 4 attempts · ~50s serial (~13s at concurrency 4)
+plan  run 20261020T213308Z_… · 1 eval · 2 cells · 4 attempts · ~52s serial (~13s at concurrency 4)
+…
+pass             house greeting › greets in the house style
+  arm                  pass
+  plain                0 of 2
+  with-house-greeting  2 of 2  can't tell yet
+  check                plain   with-house-greeting
+  house greeting       0 of 2  2 of 2
+    expected command to succeed, exit code 1
+  +2 more passed
 …
-     ✓ greets in the house style      2/4 · pass@1 100% gated  4 attempts
-         cursor:composer-2.5/plain/docker                control    0/2  pass@1   0%   not gated
-         cursor:composer-2.5/with-house-greeting/docker  variant    2/2  pass@1 100%   lift +100 pts (95% +7…+100, 2/2 vs 0/2)
-           effort: tools 3 vs 8 (×0.38), agent 9s vs 18s (×0.50): too few attempts to test
 ```
 
 ## Run it
 
 ```text
-$ CI=1 reeval run greet
-plan  profile pr (auto: CI is set) · 2 evals · 2 cells · 2 attempts · ~40s serial (~20s at concurrency 4) · budget 20 attempts ✓
-reeval 0.0.1 · 3 eval files · harness cursor:composer-2.5 · sandbox docker · concurrency 4 · grouped logs
+$ CI=1 reeval run hello --output-logs full
+plan  run 20261020T220141Z_… · profile pr (auto: CI is set) · 2 evals · 2 cells · 2 attempts · ~25s serial (~13s at concurrency 4) · budget 20 attempts ✓
+reeval 0.10.0 · run 20261020T220141Z_… · 1 eval file · harness cursor:composer-2.5 · sandbox docker · concurrency 4 · grouped logs
 [creates#1 cursor] ▶ evals/hello.eval.ts › greet module › creates a tested greet module
-[creates#1 cursor] ✓ evals/hello.eval.ts › greet module › creates a tested greet module  17s
-[creates#1 cursor]  +0:00 sandbox ready 1s
-[creates#1 cursor]  +0:01 turn 1 done 12s · 1 tool call, 2 file edits
-[creates#1 cursor]  +0:13 $ node --test  exit 0 129ms
-[creates#1 cursor]  +0:13 judge  pass 1.00 (≥ 0.8)  Does the agent's final message accurately describe what it changed?
+[creates#1 cursor]  +0:00 sandbox ready 2s
+[creates#1 cursor]  +0:01 turn 1 done 6s · 2 tool calls, 2 file edits
+[creates#1 cursor]  +0:08 $ node --test  exit 0 210ms
+[creates#1 cursor]  +0:08 judge  pass 1.00 (≥ 0.8)  Does the agent's final message accurately describe what it changed?
 [adds#1 cursor] ▶ evals/hello.eval.ts › greet module › adds a default name in a follow-up turn
-[adds#1 cursor] ✓ evals/hello.eval.ts › greet module › adds a default name in a follow-up turn  23s
-…
+[adds#1 cursor]  +0:00 sandbox ready 2s
+[adds#1 cursor]  +0:01 turn 1 done 7s · 1 tool call, 1 file edit
+[adds#1 cursor]  +0:09 turn 2 done 4s · 1 tool call, 1 file edit
+[adds#1 cursor]  +0:12 $ node -e "import('./greet.js').then((m) => {…  exit 0 105ms
+[adds#1 cursor]  +0:13 judge  pass 1.00 (≥ 0.8)  Did the agent add the default without breaking greet(name)?
+
+pass             greet module › creates a tested greet module
+pass             greet module › adds a default name in a follow-up turn
+
+16s · 144k in (64k cached) · 863 out · results .reeval/runs/20261020T220141Z_…/summary.md
+2 pass
+Set analysis.model in reeval.config.ts for answers to what each eval asks.
+recorded evals/.history/20261020T220141Z_….json + 10 artifact files in evals/.history/20261020T220141Z_…/ (41.8 KB). Commit it with git.
+```
 
-Summary
- evals/hello.eval.ts
-   greet module
-     ✓ creates a tested greet module                17s  judge 1.00  1 turn
-     ✓ adds a default name in a follow-up turn      23s  judge 1.00  2 turns
+Under a coding agent, reeval switches to compact output: the plan, the same result blocks and
+the tally, with no per-attempt lines. To see what an attempt did step by step, add `-v`; `-vv`
+adds the raw payloads and every `docker` command. Either way each attempt's trace stays local in `.reeval/debug`, never in
+the committed history.
 
-   Evals  2 passed (2)
-Duration  40s
- Results  .reeval/runs/20260924T213759Z_1ba71aa_ci_4vb0/summary.md
+```bash
+reeval record                     # copy the latest run into evals/.history/
+reeval history                    # pass/fail per eval over recent runs
+reeval compare                    # latest vs its nearest recorded ancestor
+reeval query lift                 # each variant against its control
+reeval query effort               # tool calls, agent time and tokens
+reeval query "SELECT * FROM failing_attempts WHERE skill = 'house-greeting'"
+reeval analyze                    # a checked reading of a run
 ```
 
-Under a coding agent, reeval switches to compact output: one block per failure and the path to its
-record. To see what an attempt did step by step, add `-v`: a timestamped line per phase (image,
-container, setup, brokering, each tool call, checks, judge calls, every gateway request).
-`-vv` adds the raw payloads and every `docker` command. Either way each attempt's trace is
-kept locally in `.reeval/debug`, never in the committed history. Commit runs into `evals/.history/` with `reeval record`, then track them with `reeval history`
-and `reeval compare`, or ask across runs with SQL: `reeval query lift`, `reeval query effort` (tool calls,
-agent time and tokens, variant against control), `reeval query factor_effects`, or
-`reeval query "SELECT * FROM failing_attempts WHERE skill = 'house-greeting'"`. Where the harness
-reports them (Cursor, Claude Code, Antigravity, pi and Codex do), the record keeps each turn's tokens and the model that
-served it, and totals them per attempt and per run.
-
-`reeval analyze` explains a run: one LLM call over a digest of those facts, with every claim's
-citations and every suggested command checked against reeval's own CLI. It's cached in `.reeval/`;
-`--save` commits it next to the run, and `reeval analyze --context` hands the same digest to your own
-coding agent instead, with no key.
+Where the harness reports them (every harness but Goose does), the record
+keeps each turn's tokens and the model that served it, and totals them per attempt and per run.
+
+`reeval analyze` makes one LLM call over a digest of those facts, and checks every claim's
+citations and every suggested command against reeval's own CLI. `--save` commits it next to the
+run, and `reeval analyze --context` hands the same digest to your own coding agent, with no key.
 
 ## Studio
 
 ```bash
-reeval studio                    # browse runs, evals and transcripts at http://127.0.0.1:4984
-reeval studio build --out site   # the same thing as a static site for GitHub Pages or any host
+reeval studio                    # runs, evals and transcripts on :4984
+reeval studio build --out site   # the same as a static site, for any host
 ```
 
-The studio shows each run's commit, pass counts and drift. For compares it shows per-cell pass
-rates, lift and effort, and a run's saved analyses in its Analysis tab, and for each attempt the transcript, diff, checks and judge input. `reeval studio`
-adds a Debug tab for attempts with local debug data.
+The studio shows each run's commit, pass counts and drift, and its saved analyses. For compares
+it shows per-cell pass rates, lift and effort, and for each attempt the transcript, diff, checks
+and judge input. `reeval studio` adds a Debug tab for attempts with local debug data.
 
 ## Learn more
 
-The user docs in [`docs/`](./docs) are a [docs.page](https://use.docs.page) site: a Guide from a
-first eval to CI, Concepts, Recipes, and a Reference tab with every option, command, flag, exit
-code and file format. Preview them with `npx @docs.page/cli preview`. Coding agents writing evals
-can load the reeval skill in [`.agents/skills/reeval/SKILL.md`](./.agents/skills/reeval/SKILL.md).
-[AGENTS.md](./AGENTS.md) covers developing this repo.
+| For | Read |
+|---|---|
+| Learning reeval, from a first eval to CI | The Guide in [`docs/`](./docs), a [docs.page](https://use.docs.page) site (`npx @docs.page/cli preview`) |
+| How it works, and complete solutions | The docs' Concepts and Recipes |
+| Every option, command, flag, exit code and file format | The docs' Reference |
+| Coding agents writing evals | The reeval skill, [`skills/reeval/SKILL.md`](./skills/reeval/SKILL.md) |
+| Developing this repo | [AGENTS.md](./AGENTS.md) |
 
 ## License