@cursor/july 0.1.107 → 0.1.109

package/dist/docs/evals.md

@@ -1,30 +1,63 @@
 # Evals
 
-An eval is a repeatable check that runs your agent against a fixed input
-and gates the recorded trajectory: the run completed, the right tool
-ran, the reply has the right shape. Evals are how you know a prompt
-tweak helped, a refactor didn't regress the agent, and last month's fix
-is still holding.
+An eval sends a fixed message to your agent and asserts over the
+trajectory it records: the turn completed, the right tool ran with the
+right input, the reply has the right shape. Evals are how you know a
+prompt tweak helped, a refactor didn't regress the agent, and last
+month's fix still holds.
 
-Evals exercise the same surface your users hit. The runner starts (or
-targets) a real agent server, drives sessions over the public API, and
-grades what comes back. A passing eval means the agent started,
-accepted a message, and did what you asserted.
+Nothing is mocked. The runner starts (or targets) a real agent server,
+drives sessions over the public API, and grades the events it gets
+back. The model runs and server tools execute, so
+[keep side effects out of eval sessions](#keep-side-effects-out-of-eval-sessions)
+before you point an eval at an agent that posts anywhere.
 
-## Define evals with `defineEval`
+## Evals, hooks, or hillclimbing?
 
-The Agent SDK discovers evals under the project-root `evals/` directory,
-in `.eval.ts` or `.eval.js` files. That's a sibling of `agent/`, never
-inside it (`agent/evals/` is silently ignored). TypeScript is the normal
-authoring format.
+All three read the same session event stream. Pick by the question you
+are asking.
 
-The file path is the eval's identity, so you don't author an id.
-Directories group related evals: `evals/builds/api.eval.ts` becomes id
-`builds/api`. An `index` filename collapses to its directory, so
-`evals/builds/index.eval.ts` becomes `builds`.
+| You want to | Use |
+| --- | --- |
+| Gate one fixed input's behavior, locally and in CI | Evals (this page) |
+| Observe every live session: metrics, audit, alerts | [Hooks](/docs/reference/hooks.md) |
+| Improve an agent one measured round at a time | [Hillclimbing](/docs/hillclimbing.md); each kept win lands an eval |
+
+[Hooks, channel events, or evals?](/docs/reference/hooks.md#hooks-channel-events-or-evals)
+has the side-by-side table.
+
+### When not to write an eval
+
+- Test a server tool's own logic with
+  `agent-sdk call <tool> --dir . --input '{...}'` or a unit test. No
+  model turn, no credential.
+- Explore a prompt with `agent-sdk run --dir . --message "..."` and
+  read the trajectory. Write the eval once you know which decision to
+  gate.
+- Stop a bad turn while it runs with
+  [`needsApproval`](/docs/reference/tools.md#gate-a-tool-on-human-approval)
+  on the tool or [`defineResult`](/docs/reference/result.md). Evals grade
+  after the fact.
+
+## Write your first eval
+
+Evals live under the project-root `evals/` directory, a sibling of
+`agent/`. `agent/evals/` is silently ignored. Discovery loads every
+`.eval.ts` (or `.eval.js`) file under `evals/`, plus one config file.
+
+```text
+my-agent/
+  agent/
+    agent.ts
+    tools/inspect_pr.ts
+  evals/
+    evals.config.ts      # required to run: maxConcurrency
+    readiness.eval.ts    # id: readiness
+    prs.eval.ts          # cases: prs/checkout, prs/search
+```
 
-An eval is a single `async test(t)`. You drive the agent with `t` and
-assert on the run with the same `t`:
+An eval is a single `async test(t)`. You drive the agent with `t.send`
+and assert on the recorded run with the same `t`:
 
 ```ts
 // evals/readiness.eval.ts
@@ -46,12 +79,49 @@ export default defineEval({
 });
 ```
 
-One file can also hold several datapoints through `cases` (provide
-either `test` or `cases`, not both). Each case id becomes
-`<fileId>/<case.id>`:
+```ts
+// evals/evals.config.ts
+import { defineEvalConfig } from "@cursor/july/evals";
+
+export default defineEvalConfig({ maxConcurrency: 20 });
+```
+
+Run it under Node 22.13 or newer (never Bun) with a Cursor credential
+in place; see [Credentials](#credentials):
+
+```bash
+agent-sdk eval --dir . --list
+agent-sdk eval --dir . readiness
+```
+
+```text
+PASS  readiness (14.2s) — Inspects a PR without approving it.
+  ✓ succeeded
+  ✓ calledTool(inspect_pr)
+  ✓ notCalledTool(approve_pr)
+  ✓ check(includes)
+
+1 passed, 0 failed, 1 total
+artifacts: <project state directory>/evals/2026-09-11T15-02-11-402Z
+```
+
+Every local run writes each case's assertions, inputs, tool calls, and
+`t.log` lines under that artifacts directory. Open
+`evals/<case-id>.json` there when a case fails; see
+[Where results land](#where-results-land).
+
+## Name cases by path
+
+The file path is the eval's identity, so you don't author an id.
+`evals/builds/api.eval.ts` becomes `builds/api`. An `index` filename
+collapses to its directory: `evals/builds/index.eval.ts` becomes
+`builds`.
+
+One file can hold several datapoints through `cases`. Provide either
+`test` or `cases`, not both. Each case id becomes `<fileId>/<case.id>`:
 
 ```ts
-// evals/prs.eval.ts → prs/checkout, prs/search
+// evals/prs.eval.ts: prs/checkout, prs/search
 export default defineEval({
   tags: ["smoke", "prs"],
   cases: [
@@ -80,93 +150,75 @@ export default defineEval({
 });
 ```
 
-Case ids must be single path segments, unique within the file.
-Each case can set its own `description`, `tags`, `timeoutMs`, and
-`iterations`. A case-level value replaces the file-level value for that
-datapoint.
+Case ids are single path segments, unique within the file. A case can
+set its own `description`, `tags`, `timeoutMs`, `iterations`, `judge`,
+`reporters`, and `metadata`. A case-level value replaces the file-level
+one for that datapoint, except `metadata`, which merges with case keys
+winning, and `reporters`, which adds to the file's list. `metadata` is
+free-form data carried onto the result and every reporter.
+
+A file may instead export an array of `defineEval` calls to fan out
+over a dataset. Ids are then the file id plus a zero-padded index
+(`sql/0000`, `sql/0001`, ...); see [Load a dataset](#load-a-dataset).
+Prefer `cases` when datapoints are hand-written and deserve stable
+names.
 
 ### Iterations
 
-`iterations` (file or case, default `1`) runs a datapoint repeatedly.
-Discovery expands `iterations: 3` on case `nyc` to runnable ids
-`weather/nyc/1`, `weather/nyc/2`, `weather/nyc/3` (filter prefix
-`weather/nyc` still selects all three). Each expanded case exposes
-`t.iteration` / `t.iterations` on the test context. Cap is 100.
+`iterations` (file or case, default `1`, cap `100`) runs a datapoint
+repeatedly. Discovery expands `iterations: 3` on case `nyc` to runnable
+ids `weather/nyc/1`, `weather/nyc/2`, and `weather/nyc/3`. The filter
+`weather/nyc` still selects all three. Each expanded case exposes
+`t.iteration` and `t.iterations`.
 
-`maxConcurrency` counts **authored datapoints**, not expanded
-iterations: siblings `…/1`…`…/n` share one concurrency slot and run
-sequentially. A suite with 11 cases × 3 iterations and
-`maxConcurrency: 20` therefore has at most 11 cases in flight, not 33.
+`maxConcurrency` counts authored datapoints, not expanded iterations.
+Iterations of one datapoint share a concurrency slot and run in
+sequence, so a suite of 11 cases with 3 iterations each and
+`maxConcurrency: 20` has at most 11 cases in flight.
 
-## Configure eval runs
+## Drive the agent with `t.send`
 
-Each project with evals needs `evals/evals.config.ts` or
-`evals/evals.config.js`, and it must set `maxConcurrency`. Each case
-issues real model-provider requests, so concurrency is capped hard at
-200. Existing projects use 20. Discovery with `eval --list` works
-without this file, but running a case does not.
+`t.send(message, options?)` runs one turn and waits for it to settle:
+complete, park on an approval request, or fail. Several sends in one
+case share the session, which is how you write multi-turn evals.
 
-```ts
-import { defineEvalConfig } from "@cursor/july/evals";
+Each send resolves to a turn result: `message` (the assistant text),
+`sessionId`, `events`, `toolCalls` (tool names in order), `ok`, and
+`index`. The turn carries the same assertion vocabulary as `t`, scoped
+to that turn, so you can grade an intermediate turn before the next
+send overwrites `t.reply`. `turn.expectOk()` throws when the turn
+failed, for later steps that depend on it.
 
-export default defineEvalConfig({
-  maxConcurrency: 20, // required
-  // timeoutMs: 180_000, // optional project-wide default
-  // judge: { model: "..." }, // default judge model for t.judge.*
-  // reporters: [], // destinations that observe every case
-  // maxPlaygroundRuns: 50, // playground history only (default 20)
+Read the whole case with `t.reply` (last assistant text), `t.events`
+(every event so far), `t.turns` (settled turns, oldest first), and
+`t.sessionId`. `t.signal` aborts when the case hits its timeout; pass
+it to your own async work.
+
+Three options apply on the first send only, because they shape session
+creation:
+
+| Option | Effect |
+| --- | --- |
+| `workspaceFiles` | `{ path: contents }` seeded into the local session workspace. Prefer this over machine-local paths |
+| `workspaceDir` | Absolute harness cwd for the local runtime |
+| `cloud` | Per-session cloud options merged over the agent's static `cloud` config. Pin a fixture repo here for cloud evals instead of on the agent's default `cloud.repos`. On the cloud runtime, seeded files reach the agent as described under [Where does a turn run?](/docs/concepts.md#where-does-a-turn-run) |
+
+```ts
+await t.send("Review pr/diff.patch and post findings.", {
+  workspaceFiles: {
+    "pr/diff.patch": [
+      "diff --git a/app/routes/search.ts b/app/routes/search.ts",
+      "+res.send(`<h1>Results for ${req.query.q}</h1>`);",
+    ].join("\n"),
+  },
 });
 ```
 
-The timeout order is case or file `timeoutMs`, CLI `--timeout-ms`,
-project config `timeoutMs`, then the 180-second runner default.
+## Assert over the trajectory
 
-The optional fields:
-
-| Option | Default | Meaning |
-| --- | --- | --- |
-| `timeoutMs` | `180_000` | Project-wide per-case timeout |
-| `judge` | unset | Default judge model for `t.judge.*`; see [Judge free-form output](#judge-free-form-output) |
-| `reporters` | unset | Destinations that observe every case; `--skip-report` suppresses them |
-| `maxPlaygroundRuns` | `20` | Max batches in the playground / `/v1/dev/evals*` history (not CLI `eval`). Hard-capped at 500. |
-
-Reporters come from `@cursor/july/evals/reporters`: `JUnit` writes a
-JUnit XML file for CI, `Artifacts` writes per-case files, and
-`combineReporters` merges several into one (`renderJUnitXml` renders
-the XML for a custom destination). A file or case can add its own
-`reporters` on top of the config list.
-
-Playground batches survive restarts whenever `agent/storage.ts` exists
-with an `evals` table or a KV core providing `delete` and `list` (the
-table is derived over the core); see
-[Storage](/docs/storage.md#eval-and-a-b-tables). Without storage they live
-in process memory and disappear when `serve` exits. Navigating away
-and back still works while the process is up.
-
-## Drive and assert with `t`
-
-`t` is both the driver and the assertion surface. You write ordinary
-control flow, sending turns and asserting inline.
-
-Drive the agent with `t.send(message, options?)`. It runs one turn and
-waits for the session to park or fail. Multiple sends in one case share
-the session, which is how you write multi-turn evals.
-
-Each `t.send` resolves to a turn result with `message`,
-`sessionId`, `events`, `toolCalls`, `ok`, and `index`. The turn carries
-the same assertion vocabulary as `t`, scoped to that turn, so you can
-grade an intermediate turn before the next send overwrites `t.reply`.
-`turn.expectOk()` throws when the turn failed, for later
-steps that depend on it.
-
-Read the full case state with `t.reply` (the last assistant text),
-`t.events` (session events captured so far), `t.turns` (settled
-turns, oldest first), and `t.sessionId`. `t.signal` aborts when the
-case hits its timeout; pass it to your own async work. A thrown
-[turn result](/docs/reference/result.md) `commit` fails the turn, so
-`t.succeeded()` fails too.
-
-Assert with the gates:
+Assertions record; they never throw. One run reports every failure
+instead of dying on the first. Assertions on `t` read the whole run.
+Assertions on a turn read only that turn.
 
 | Gate | Checks |
 | --- | --- |
@@ -175,157 +227,483 @@ Assert with the gates:
 | `t.messageIncludes(token)` | the joined assistant text matches a string or `RegExp` |
 | `t.calledTool(name, matcher?)` | a matching call to `name` happened |
 | `t.notCalledTool(name)` | no request for `name`, in any lifecycle state |
-| `t.loadedSkill(name)` | the agent opened the skill's `SKILL.md` (read, grep, or shell `cat`) |
-| `t.toolOrder(names)` | tool requests appear in this relative order (extra calls allowed) |
+| `t.loadedSkill(name)` | the agent opened `skills/<name>/SKILL.md` (read, grep, or shell `cat`) |
+| `t.toolOrder(names)` | tool requests appear in this relative order; extra calls allowed |
 | `t.usedNoTools()` | no tool calls at all |
 | `t.maxToolCalls(max)` | at most `max` tool calls |
 | `t.noFailedActions()` | no tool call reported an error |
 | `t.calledSubagent(name, matcher?)` | a matching subagent delegation happened |
 | `t.taggedArtifact(kind?, predicate?)` | at least one [artifact](/docs/reference/artifacts.md) was tagged |
-| `t.event(type, matcher?)` | at least one matching event of `type` occurred |
-| `t.notEvent(type, matcher?)` | no matching event of `type` occurred |
+| `t.event(type, matcher?)` | at least one matching [event](/docs/reference/sessions.md#which-events-can-i-stream) of `type` |
+| `t.notEvent(type, matcher?)` | no matching event of `type` |
 | `t.eventOrder(matchers)` | matching event groups occur in this relative order |
 | `t.eventsSatisfy(label, predicate)` | your predicate over the typed event stream |
-| `t.check(value, expectation)` | any value, against a builder |
-| `t.score(name, value)` | records a 0–1 score you computed; soft until you add a bar |
-| `t.requireToolCall(name, matcher?)` | gates on a matching call and returns it, so later code can read its input and output |
-| `t.requireInputRequest(filter?)` | gates on exactly one pending approval request and returns it |
+| `t.check(value, expectation)` | any value, against a [builder](#grade-values-with-expectation-builders) |
+| `t.score(name, value)` | a 0-1 score you computed; soft until you add a bar |
+
+Three more assertions gate and return the matched fact. They stop the
+test body when nothing matches, without a duplicate execution error.
+`t.requireToolCall(name, matcher?)` returns the call so later code can
+read its `input` and `output`. `t.requireInputRequest(filter?)` returns
+the single pending approval request. `await t.require(value, expectation)`
+does the same for a value check.
+
+A case with no assertions passes when at least one turn completed. Add
+`t.succeeded()` and behavior gates anyway. They make the contract
+visible in review.
+
+### What good cases assert
+
+Gate decisions and shape, not prose. Model wording varies run to run.
+Tool choice, tool avoidance, and output structure are the stable
+contract.
+
+1. `t.succeeded()`: always, first.
+2. The tool decision: `calledTool` for the intended path,
+   `notCalledTool` for the likely wrong alternative. The pair is
+   stronger than either alone.
+3. Output shape: a regex for the contract (`/ready|blocked/i`, a JSON
+   marker, a findings-block fence), never exact sentences.
+4. For structured output, parse `t.reply` and check fields with
+   `matches` or `satisfies` instead of substring-matching JSON.
+
+The common failure modes: asserting exact phrasing, packing more than
+about five gates into one case (split it), and cases that depend on
+live external state that drifts (pin the input).
 
-Every gate returns a handle: `.soft()` demotes it to tracked-only,
-`.atLeast(0.7)` adds a soft score bar, and `.gate(0.8)` promotes a
-scored assertion into a hard gate.
+### Narrow tool assertions with matchers
 
 With no matcher, `calledTool` is request-based: a requested call counts
-even when its result has not arrived. Pass
-`t.calledTool("inspect_pr", { status: "completed" })` to require the
-call to return. `input`, `output`, and `count` matcher fields accept a
-literal, a `RegExp`, or a predicate.
-
-The expectation builders are `includes(string | RegExp)`,
-`equals(value)`, `matches(schema)`, `similarity(expected)`, and
-`satisfies(predicate, label)`. `includes` stringifies its input,
-`equals` compares values deeply, `matches` validates against a Standard
-Schema (or anything with `safeParse`, like Zod), `similarity` scores
-normalized text similarity, and `satisfies` runs your predicate. The
-plain function `normalizedSimilarity(actual, expected)` returns the
-same 0–1 score for use with `t.score`.
-
-A few more context members shape a case: `t.require(value, expectation)`
-records a gate and stops the test body when it fails, without a
-duplicate execution error. `t.skip(reason)` ends the case as skipped
-(reported separately, never changes the exit code; call it before
-sending messages). `t.metric(name, value)` records a structured score
-for the playground case card. `t.log(message)` records a debug line for
-the CLI and playground result.
-
-Three `t.send` options apply on session create (first `t.send` only):
-
-- `workspaceFiles`: `{ path: contents }`, seeded into the local session
-  workspace. Prefer this over machine-local paths.
-- `workspaceDir`: absolute harness cwd (local runtime).
-- `cloud`: per-session cloud options merged over the agent's static
-  `cloud` config (repos / env / …). Use a pinned `repos` override to
-  attach a fixture repo for cloud evals without putting it on the
-  agent's default `cloud.repos`. Cloud ignores `workspaceFiles` seeds.
+even before its result arrives. A matcher narrows it:
 
 ```ts
-const toolResults = t.events.filter((e) => e.type === "action.result");
+t.calledTool("inspect_pr", { status: "completed" });
+t.calledTool("apply_agents", { input: { verdict: "update" } });
+t.calledTool("bash", { input: { command: /^gh pr view/ }, count: 1 });
+t.calledTool("read_file", {
+  output: (value) => String(value).includes("TODO"),
+});
+```
+
+`input`, `output`, and `count` accept a literal, a `RegExp`, or a
+predicate. Object literals partial-deep-match, so `{ verdict: "update" }`
+matches arguments that also carry other keys. `status` is one of
+`completed`, `failed`, `pending`, or `rejected` (a human denied the
+approval). `calledSubagent` takes `{ output, status, count, callId }`.
+`event`, `notEvent`, and `eventOrder` take `{ data, count }`.
+
+### Grade values with expectation builders
+
+`t.check(value, expectation)` grades any value: `t.reply`, a parsed
+JSON field, a tool's output.
+
+| Builder | Checks | Severity |
+| --- | --- | --- |
+| `includes(string \| RegExp)` | substring or match; structured values are stringified first | gate |
+| `equals(value)` | deep equality | gate |
+| `matches(schema)` | a Standard Schema (Zod, Valibot, ...) or anything with `safeParse` | gate |
+| `similarity(expected)` | normalized text similarity, 0-1 | soft |
+| `satisfies(predicate, label)` | your predicate; `label` is the failure detail | gate |
+
+```ts
+import { matches, satisfies } from "@cursor/july/evals";
+import { z } from "zod";
+
+const verdict = JSON.parse(t.reply ?? "{}");
 t.check(
-  toolResults.length,
-  satisfies((n) => (n as number) <= 4, "at most 4 tool calls")
+  verdict,
+  matches(z.object({ ready: z.boolean(), blockers: z.array(z.string()) }))
+);
+t.check(
+  verdict.blockers.length,
+  satisfies((n) => (n as number) <= 3, "at most 3 blockers")
 );
 ```
 
-A case with no explicit gates falls back to whether at least one turn
-completed successfully. Add `t.succeeded()` and behavior-specific gates
-anyway. They make the contract visible during review.
+`normalizedSimilarity(actual, expected)` returns the same 0-1 score as
+`similarity`, for use with `t.score`.
+
+### Record without gating
 
-### Judge free-form output
+- `t.metric(name, value)` records a structured score or label. It shows
+  on the CLI result, the playground case card, JUnit output, and
+  artifacts.
+- `t.log(message)` records a debug line, streamed under `--verbose`.
+- `t.skip(reason)` ends the case as skipped. Skipped cases report
+  separately and never change the exit code. Call it before sending
+  messages.
 
-When wording matters and no regex captures it, `t.judge` grades the
-reply with an LLM. The built-in graders are `factuality(expected)`,
-`summarizes(expected)`, `closedQA(criteria)`, and `sql(expected)`. Each
-scores `t.reply` by default; pass `{ on }` to grade another value.
+## Gates, soft scores, and verdicts
+
+Every assertion returns a handle, so severity rides on the assertion
+instead of a separate thresholds map:
+
+```ts
+t.succeeded(); // gate (default)
+t.calledTool("get_weather").soft(); // tracked, never fails
+t.check(t.reply, similarity("Sunny, 72°F")).atLeast(0.8); // soft, with a bar
+t.judge.closedQA("cites a source").gate(0.8); // promoted to a gate
+```
+
+- `.gate(threshold?)` is hard. A miss fails the case and `eval` exits 1.
+- `.soft(threshold?)` is tracked. With no threshold it never fails.
+- `.atLeast(threshold)` is soft with a bar. A miss marks the case
+  `scored`.
+
+Each case ends with one verdict:
+
+| Verdict | Meaning | Exit code |
+| --- | --- | --- |
+| `passed` | every gate passed and no soft bar was missed | 0 |
+| `failed` | a gate failed, or the test body threw | 1 |
+| `scored` | only soft bars were missed | 0, or 1 under `--strict` |
+| `skipped` | `t.skip(reason)`, or a judge with no credentials | 0 |
+
+The CLI prints soft misses as `~` and gate misses as `✗`. Start a new
+benchmark with `t.score("recall", recall)` and `.atLeast()` so its
+number reports for a while without blocking merges. Add `--strict`
+once the bars are trustworthy.
+
+## Judge free-form output
+
+When wording matters and no regex captures it, `t.judge` grades with an
+LLM. The graders are `factuality(expected)`, `summarizes(expected)`,
+`closedQA(criteria)`, and `sql(expected)`. Each scores `t.reply` by
+default; pass `{ on }` to grade another value.
 
 ```ts
-t.judge.factuality("It is 54°F in NYC right now.").atLeast(0.7);
+const summary = await t.send("Why did CI fail on PR 42?");
+t.judge.factuality("The lint step failed on src/sidebar.ts.").atLeast(0.7);
+t.judge.closedQA("names the failing step", { on: summary.message }).gate(1);
 ```
 
 Judge assertions are soft by default, so a judge never fails a build
-until you give it a bar with `.atLeast(0.7)` or promote it with
-`.gate(0.8)`. The judge model comes from `defineEvalConfig({ judge })`,
+until you give it a bar with `.atLeast()` or promote it with `.gate()`.
+The recorded detail names the choice the judge made and its rationale.
+
+The judge model comes from `defineEvalConfig({ judge })`,
 `defineEval({ judge })`, a case-level `judge`, or a per-call
-`{ model }` override; the nearest one wins. For a domain-specific judge
-whose verdict is not a single score, `t.judge.model(prompt)` sends a
-raw prompt to the same model and returns the reply. You then record the
-parsed result with `t.score` or `t.check`.
+`{ model }`. The nearest one wins. A judge call with no model
+configured fails the case. A judge that cannot reach a model (no
+credential) ends the case as `skipped`, unless a deterministic gate
+already failed.
+
+For a domain-specific judge whose verdict is not a single score,
+`t.judge.model(prompt)` sends a raw prompt to the same model and
+returns the reply. Record the parsed result with `t.score` or
+`t.check`. Anything derived from the agent under test is untrusted
+input to your prompt: wrap it with `fenceUntrusted` and include
+`EVAL_JUDGE_INJECTION_GUARD`, as the built-in graders do.
 
-## Run evals from the CLI
+```ts
+import {
+  EVAL_JUDGE_INJECTION_GUARD,
+  fenceUntrusted,
+} from "@cursor/july/evals";
+
+const gold = ["XSS in search.ts", "open redirect in login.ts"];
+const reply = await t.judge.model(
+  [
+    "For each GOLD finding, answer whether SUBMISSION reports it.",
+    "Reply with one line per finding: <index> YES|NO.",
+    EVAL_JUDGE_INJECTION_GUARD,
+    fenceUntrusted("GOLD", gold.map((g, i) => `${i + 1}. ${g}`).join("\n")),
+    fenceUntrusted("SUBMISSION", t.reply ?? ""),
+  ].join("\n\n")
+);
+const hits = reply.match(/\bYES\b/g)?.length ?? 0;
+t.score("recall", hits / gold.length).atLeast(0.5);
+```
+
+## Keep side effects out of eval sessions
+
+Eval sessions run the real agent, tools included. A reviewer that
+comments on GitHub or posts to Slack will do so from an eval unless
+the tool checks the session's purpose. Eval sessions carry
+`purpose: "eval"`; live traffic carries `"live"`. Branch on it in the
+tool, hook, or result handler that actuates:
+
+```ts
+// agent/tools/post_findings.ts
+async execute({ findings }, ctx) {
+  if (ctx.session.purpose === "eval") {
+    return { posted: false, reason: "eval", count: findings.length };
+  }
+  // post the review
+}
+```
+
+Return a shaped result instead of throwing, so the eval can still
+assert `t.calledTool("post_findings", { input: ... })` on the decision.
+The same check belongs in [hooks](/docs/reference/hooks.md) that meter or
+page and in [`defineResult`](/docs/reference/result.md) commits.
+
+## Worked examples
+
+### Multi-turn: grade each turn
+
+```ts
+// evals/intro.eval.ts
+import { defineEval, includes, satisfies } from "@cursor/july/evals";
+
+export default defineEval({
+  description: "Introduces itself once; a repeat mention gets a short ack.",
+  async test(t) {
+    const intro = await t.send("Meet Jenny! @Jenny introduce yourself.");
+    intro.expectOk();
+    t.check(intro.message, includes(/jenny/i));
+
+    const repeat = await t.send("Meet, @Jenny!");
+    t.succeeded();
+    repeat.usedNoTools();
+    t.check(
+      repeat.message,
+      satisfies((r) => (r as string).trim().length <= 280, "short ack")
+    );
+    t.check(
+      repeat.message,
+      satisfies((r) => !/what i can do/i.test(r as string), "no second intro")
+    );
+  },
+});
+```
 
-The `eval` command discovers, filters, and runs cases.
+`t.succeeded()` grades the whole session. `repeat.usedNoTools()` and the
+checks on `repeat.message` read only the second turn, even though
+`t.reply` now holds its text.
 
-Run the CLI under Node 22.13 or newer. Do not use Bun. Its HTTP/2 client
-breaks tool-result streams and causes eval turns to fail.
+### Approvals: assert the parked decision
+
+For a tool with `needsApproval`, the turn parks instead of finishing.
+Gate on `t.parked()` and on the arguments the model chose:
+
+```ts
+// evals/agents.eval.ts (one case; RULE and SLACK are fixture strings)
+{
+  id: "update-rule",
+  description: "A repeated billing rule parks the AGENTS.md write.",
+  async test(t) {
+    await t.send("Weekly AGENTS.md review. Read week/ and call apply_agents once.", {
+      workspaceFiles: {
+        "week/prs.md": RULE,
+        "week/slack.md": SLACK,
+        "week/tree/AGENTS.md.txt": "# API\n\nKeep handlers thin.\n",
+      },
+    });
+    t.parked();
+    t.calledTool("apply_agents", { input: { verdict: "update" } });
+  },
+},
+```
+
+`t.parked()` and `t.succeeded()` are exclusive: a parked run is a clean
+stop on an unanswered approval, not a completed one. Pair the parked
+case with a sibling that expects `verdict: "skip"` and `t.succeeded()`,
+so both branches stay pinned.
+
+## Pin fixtures
+
+A fixed input is what makes an eval repeatable. Pick the fixture by the
+surface under test.
+
+| Agent surface | Fixture |
+| --- | --- |
+| Chat or domain assistant | One canonical prompt string, chosen once and frozen |
+| Tool-heavy | Run `agent-sdk call <tool>` first to pin what the tool returns, then freeze the prompt that triggers it |
+| GitHub webhook | `agent-sdk github replay <pr> --events '*' --dry-run --out fixtures/github` snapshots real payloads for offline replay ([GitHub guide](/docs/guides/github.md#test-with-github-replay)) |
+| PR reviewer with host preparation | Diff, metadata, and gold labels pinned to commit SHAs; keep any live PR matrix small |
+| Workspace-dependent | `workspaceFiles` on the first `t.send`, never developer-machine paths |
+
+Tag the fast, reliably passing core `smoke` and run `--tag smoke` in
+the inner loop. Leave slow or drift-prone cases untagged for explicit
+runs.
+
+### Materialize API-backed fixtures
+
+An input that only points at external data (a pull request URL, a
+snapshot id, a pair of commit SHAs) is not self-contained. Fetch it once
+and commit the rendered fixture before you expand the suite:
+
+1. Save the diff, metadata, and labels under `fixtures/` at pinned
+   revisions.
+2. Seed those files with `workspaceFiles`, or read them from the
+   fixture directory.
+3. Assert decisions and output shape against the saved evidence.
+4. Keep a small `smoke` subset for any remaining live checks.
+
+`maxConcurrency` limits parallel datapoints, not the model or API
+fan-out inside one datapoint. Materialized fixtures keep a large suite
+from exhausting provider and GitHub rate limits.
+
+### Load a dataset
+
+Read committed fixtures with `loadJson`, `loadJsonl`, and `loadYaml`
+from `@cursor/july/evals/loaders`. Relative paths resolve against the
+project root the runner discovered, not the cwd the CLI ran from. Eval
+files are ES modules, so top-level `await` can load a dataset and fan
+one file out over it:
+
+```ts
+// evals/sql.eval.ts: sql/0000, sql/0001, ...
+import { defineEval, equals } from "@cursor/july/evals";
+import { loadYaml } from "@cursor/july/evals/loaders";
+
+const rows = await loadYaml<{ task: string; prompt: string; sql: string }[]>(
+  "evals/data/cases.yaml"
+);
+
+export default rows.map((row) =>
+  defineEval({
+    description: row.task,
+    async test(t) {
+      await t.send(row.prompt);
+      t.succeeded();
+      t.check(t.reply, equals(row.sql));
+    },
+  })
+);
+```
+
+## Configure eval runs
+
+`evals/evals.config.ts` (or `.js`) holds project-wide defaults. It must
+set `maxConcurrency`. Each case issues real model requests, so
+concurrency is hard-capped at 200; the templates use 10.
+`eval --list` works without the file. Running a case does not.
+
+```ts
+import { defineEvalConfig } from "@cursor/july/evals";
+
+export default defineEvalConfig({
+  maxConcurrency: 20,
+  timeoutMs: 180_000,
+  judge: { model: "gpt-5.4-mini" },
+});
+```
+
+| Option | Default | Meaning |
+| --- | --- | --- |
+| `maxConcurrency` | required | Datapoints in flight at once, 1-200 |
+| `timeoutMs` | `180_000` | Per-case timeout. Precedence: case or file `timeoutMs`, then `--timeout-ms`, then this value |
+| `judge` | unset | Default judge model for `t.judge.*` |
+| `reporters` | unset | Destinations that observe every case; `--skip-report` suppresses them |
+| `maxPlaygroundRuns` | `20` | Batches kept in playground history (cap 500). Counts `--prod` and `--url` batches, not the default local run |
+
+Reporters ship results somewhere; the runner still does the grading.
+`JUnit({ filePath, suiteName? })` writes JUnit XML and
+`Artifacts({ dir })` writes per-case files, both from
+`@cursor/july/evals/reporters`. A custom reporter is an object with any
+of `onRunStart`, `onEvalComplete`, and `onRunComplete`. A reporter that
+throws is logged and never fails the run. CI usually attaches the
+built-in two with `--junit` and `--artifacts` instead of `reporters`,
+so output paths stay with the pipeline, not the eval author.
+
+Playground batches survive restarts when the project has
+[storage](/docs/storage.md#eval-table). Otherwise they live in process
+memory until `serve` exits.
+
+## Run evals from the CLI
 
 ```bash
-agent-sdk eval --dir . --list                         # discover only
-agent-sdk eval --dir .                                # run all
-agent-sdk eval --dir . builds/checkout                # one datapoint
-agent-sdk eval --dir . builds search                  # several ids or prefixes
-agent-sdk eval --dir . --tag smoke --tag pull-request # any matching tag
-agent-sdk eval --dir . --json --no-stream             # machine-readable results
-agent-sdk eval --dir . --verbose                      # logs + reply snippets
+agent-sdk eval --dir . --list                          # discover only
+agent-sdk eval --dir .                                 # run all
+agent-sdk eval --dir . builds/checkout                 # one datapoint
+agent-sdk eval --dir . builds search                   # several ids or prefixes
+agent-sdk eval --dir . --tag smoke --tag pull-request  # any matching tag
+agent-sdk eval --dir . --verbose                       # t.log lines + reply snippets
+agent-sdk eval --dir . --timeout-ms 300000             # override the per-case timeout
 ```
 
 Id filters use OR semantics. Each filter selects an exact id and its
-descendants. For example, `builds` selects `builds`,
-`builds/checkout`, and every other case below that path. Repeated tags
-also use OR semantics. When you provide both ids and tags, a case must
-match both groups.
-
-`eval` boots an ephemeral server on port 0 with a temp state root
-outside the project, so cases don't inherit ambient monorepo rules and
-don't write into the project state directory. Point `--url` at a running server to eval
-a live agent instead:
+descendants: `builds` selects `builds`, `builds/checkout`, and every
+other case below that path. Repeated tags also use OR. With both ids
+and tags, a case must match both groups.
+
+By default `eval` boots a throwaway server with its own state root, so
+cases don't inherit your checkout's `AGENTS.md` and session state stays
+out of the project. Artifacts still land in the project state
+directory; see [Where results land](#where-results-land). `--slug`
+picks the target in a multi-agent directory.
+
+`--url` runs the batch on a running server instead, the same way
+`--prod` does: that server discovers its own `evals/`, results land in
+its playground history, and the local-only flags (`--junit`,
+`--artifacts`, `--max-concurrency`, `--skip-report`) do not apply. See
+[Run evals in the playground or on a deployment](#run-evals-in-the-playground-or-on-a-deployment).
 
 ```bash
-agent-sdk eval --dir . \
-  --url http://127.0.0.1:3000/weather-agent \
+agent-sdk eval --url http://127.0.0.1:3000/weather-agent \
   --bearer-token "$AGENT_TOKEN"
 ```
 
-The eval definitions still come from `--dir`; `--url` only changes the
-agent that receives the turns. For a locally mounted multi-agent
-directory, `--slug weather-agent` chooses the target. Use
-`--state-root` to keep ephemeral session state at a chosen path,
-`--timeout-ms` to override the project timeout, and `--no-stream` to
-keep live progress off stderr. A TTY streams turn progress by default.
-`--verbose` still writes `t.log` lines to stderr and adds reply snippets
-to text results.
+See [CLI: eval](/docs/reference/cli.md#eval) for every flag.
+
+### Credentials
 
-Model turns need a Cursor credential from `agent-sdk login` or
-`CURSOR_API_KEY`.
+Model turns need a
+[Cursor credential](/docs/reference/cli.md#environment-variables):
+`CURSOR_API_KEY`, `CURSOR_SERVICE_ACCOUNT_KEY`, or a saved
+`agent-sdk login`. The judge uses the same one. `eval --list` needs
+none.
 
-See [CLI: eval](/docs/reference/cli.md#eval) for flags and exit codes.
+### Where results land
 
-### JSON results
+Every local run writes artifacts to a timestamped directory under
+`evals/` in the project state directory, whatever `--state-root` says.
+`--artifacts <dir>` chooses the path and `--no-artifacts` skips them.
+The directory holds `summary.json`,
+`results.jsonl`, and `evals/<case-id>.json` with every assertion, the
+inputs, tool calls with arguments and output, the final text, and
+`t.log` lines. Start there when a case fails. `--out <file>` also
+writes the full results JSON to a path of your choice.
 
-Use `--json --no-stream` in scripts and CI. The top-level result carries
-the totals and one result per case:
+The artifact does not include the session's event stream. Pass
+`--state-root <path>` to keep the ephemeral server's
+[session data](/docs/reference/sessions.md#where-does-the-agent-sdk-store-session-data)
+on disk when you need the raw events.
+
+## Run evals in CI
+
+Run the suite non-interactively, write JUnit for the CI annotations,
+and fail the job on a red gate:
+
+```bash
+# CURSOR_API_KEY comes from the CI secret store
+agent-sdk eval --dir . --json --no-stream \
+  --junit reports/evals.xml \
+  --artifacts reports/evals \
+  > reports/evals.json
+```
+
+The exit code follows the
+[verdict table](#gates-soft-scores-and-verdicts); `2` means nothing
+matched the selection. `--max-concurrency` overrides the project
+setting, for example to run lower on a shared runner.
+
+The JSON on stdout carries the totals and one result per case:
 
 ```json
 {
   "ok": true,
   "passed": 1,
   "failed": 0,
+  "scored": 0,
+  "skipped": 0,
+  "strict": false,
+  "artifactsDir": "/work/my-agent/reports/evals",
   "results": [
     {
       "id": "readiness",
+      "verdict": "passed",
       "ok": true,
-      "assertions": [{ "name": "succeeded", "passed": true }],
+      "assertions": [
+        { "name": "succeeded", "passed": true },
+        { "name": "calledTool(inspect_pr)", "passed": true }
+      ],
       "sessionId": "ses_123",
-      "inputs": ["Is checkout pull request 42 ready to approve?"],
+      "inputs": ["Is https://github.com/acme/checkout/pull/42 ready to approve?"],
       "toolCalls": [{ "toolName": "inspect_pr", "isError": false }],
+      "metrics": {},
       "logs": [],
       "durationMs": 12340
     }
@@ -333,126 +711,64 @@ the totals and one result per case:
 }
 ```
 
-Each case result can also include `description`, `finalText`, `tools`,
-`error`, and tool arguments or output. This shape lets CI report the
-failed assertion without parsing terminal text.
+Each result can also include `description`, `finalText`, `tools`,
+`error`, `skipReason`, `metadata`, `tags`, and tool `args` and `output`.
+A soft miss shows as `"severity": "soft"` with `score` and `threshold`
+on the assertion. This shape lets CI report the failed assertion
+without parsing terminal text.
+
+Keep CI green without weakening gates:
+
+- Run `--tag smoke` on every push and the full suite on a schedule.
+- For probabilistic behavior, use `iterations` and a soft bar instead
+  of one hard gate.
 
-## Run evals in the playground
+## Run evals in the playground or on a deployment
 
-Start the server, open the playground, and choose **Evals**. You can run
-every case or one case, watch progress, and open the resulting session
-trace. The Evals tab works on a normal `serve`.
+Start the server, open the playground, and choose **Evals**. Run every
+case or one case, watch progress, and open the resulting session trace.
+Playground runs target the live server instead of an ephemeral one, so
+their sessions appear in the session list. One batch runs at a time.
 
 ```bash
 agent-sdk serve --dir .
 ```
 
-Playground runs target the live server instead of an ephemeral one.
-Their sessions appear in the session list. One eval batch can run at a
-time. Persistence follows the rule under
-[Configure eval runs](#configure-eval-runs). See
-[Playground eval routes](/docs/reference/http-api.md#playground-eval-routes).
-The start request returns `202` while cases run in the background.
-Poll until the snapshot status becomes `completed`, `failed`, or `cancelled`.
-Configuration errors appear on a failed snapshot.
-
-On `--prod` / `--url`, the CLI prints the Eval ID as soon as the batch is
-accepted (and a Playground deep link with `?view=evals&evalRunId=…`):
+`--prod` (or `--url`) starts the same server-side batch on the team's
+hosted deployment (or the server you name), so results land in that
+server's playground history:
 
 ```bash
 agent-sdk eval --prod --slug vulnerability-scanner --tag deepsec
-# Eval ID: evalrun_…
-# Cancel:  agent-sdk eval cancel evalrun_… --prod --slug vulnerability-scanner
-# Playground: https://…/playground?view=evals&evalRunId=evalrun_…
-
-agent-sdk eval cancel evalrun_… --prod --slug vulnerability-scanner
-agent-sdk eval status evalrun_… --prod --slug vulnerability-scanner
+# Eval ID: <evalId>
+# Playground: <deployment>/playground?view=evals&evalRunId=<evalId>
+agent-sdk eval status <evalId> --prod --slug vulnerability-scanner
+agent-sdk eval cancel <evalId> --prod --slug vulnerability-scanner
 ```
 
-## What good cases assert
-
-Gate decisions and shape, not prose. Model wording varies run to run.
-Tool choice, tool avoidance, and output structure are the stable
-contract.
-
-1. `t.succeeded()`: always, first.
-2. The tool decision: `calledTool` for the intended path,
-   `notCalledTool` for the likely wrong alternative. The pair is
-   stronger than either alone.
-3. Output shape: a regex for the contract (`/ready|blocked/i`, a JSON
-   marker, a findings-block fence), never exact sentences.
-4. For structured output, parse `t.reply` and check fields with
-   `satisfies` instead of substring-matching JSON.
-
-The common failure modes: asserting exact phrasing, packing more than
-about five gates into one case (split it), and cases that depend on live
-external state that drifts (pin the input; see fixtures).
-
-## Pick fixtures by agent type
-
-The right fixture depends on the surface under test.
-
-| Agent surface | Fixture |
-| --- | --- |
-| Chat / domain assistant | A canonical prompt string, chosen once and frozen |
-| Tool-heavy | Run `agent-sdk call <tool>` first to pin what the tool returns, then freeze the prompt that triggers it |
-| GitHub webhook | `agent-sdk github replay <pr> --events '*' --dry-run --out fixtures/github` snapshots real payloads for offline replay ([GitHub guide](/docs/guides/github.md)) |
-| PR reviewer with host preparation | Diff, metadata, and gold labels pinned to commit SHAs; keep any live PR matrix small |
-| Workspace-dependent | `workspaceFiles` in `t.send` options, never developer-machine paths |
-
-Tag the fast, reliably passing core `smoke` and run `--tag smoke` in the
-inner loop. Leave slow or flaky-prone cases untagged for explicit runs.
-
-### Materialize API-backed fixtures
-
-An input that only points at external data, such as a pull request URL,
-snapshot id, or pair of commit SHAs, is not self-contained. Fetch it
-once and commit the rendered fixture before you expand the suite.
-
-1. Save the diff, metadata, and labels under `fixtures/` at pinned
-   revisions.
-2. Seed those files with `workspaceFiles`, or read them from the fixture
-   directory.
-3. Assert decisions and output shape against the saved evidence.
-4. Keep a small `smoke` subset for any remaining live pipeline checks.
-
-Read committed fixtures with `@cursor/july/evals/loaders`: `loadJson`,
-`loadJsonl`, and `loadYaml` resolve relative paths against the project
-root the runner discovered, not the cwd the CLI was invoked from
-(`resolveFixturePath` and `evalFixtureRoot` expose the same
-resolution for other file formats).
-
-`maxConcurrency` limits parallel datapoints. It does not limit model or
-API fan-out inside one datapoint. Materialized fixtures prevent a large
-suite from exhausting provider and GitHub rate limits. The
-[evals skill](/docs/skills/evals.md) has the full fixture workflow.
+The CLI prints the Eval ID as soon as the batch is accepted. Pass
+`--no-wait` to return right away and poll with `eval status` later; it
+exits `3` while the batch is still running. Hosted history follows
+`maxPlaygroundRuns` and the persistence rule under
+[Configure eval runs](#configure-eval-runs). The HTTP surface is under
+[Playground eval routes](/docs/reference/http-api.md#playground-eval-routes).
 
 ## Keep improvements with regression evals
 
-Every [hillclimb](/docs/hillclimbing.md) round that keeps a change must land
-an eval that would have failed before the change. If you can't express
-the improvement as a gate (a `calledTool` shift, a bounded
-`action.result` count, an output-shape regex), the improvement is
-unverified, and it'll regress silently.
-
-The rule cuts the other way too: never weaken an existing gate to make a
-round pass. That's the freeze line moving, and it turns your regression
-suite into a list of checks that no longer protect anything.
-
-## Compare variants on live traffic
+Every [hillclimb](/docs/hillclimbing.md) round that keeps a change must
+land an eval that would have failed before the change. If you can't
+express the improvement as a gate (a `calledTool` shift, a bounded
+`maxToolCalls`, an output-shape check), the improvement is unverified,
+and it'll regress silently.
 
-Use `defineAB` to compare variant metrics on live sessions. It is not a
-test runner and has no `agent-sdk ab` command. Keep `defineEval` as the
-regression ratchet. Eval sessions do not enroll or change live metrics.
-See [Live A/B metrics](/docs/ab.md) for assignment, behavior, collection,
-and inspection.
+The rule cuts the other way too: never weaken an existing gate to make
+a round pass. That's the freeze line moving, and it turns your
+regression suite into a list of checks that no longer protect anything.
 
 ## What's next
 
 Continue with these pages:
 
-- [Live A/B metrics](/docs/ab.md): sticky variants and cumulative metrics
-  on live sessions
 - [Hillclimbing](/docs/hillclimbing.md): the loop evals make trustworthy
 - [Building agents with agents](/docs/building-with-agents.md): have a
   coding agent write the first suite