@cursor/july 0.1.111 → 0.1.112

package/dist/docs/evals.md

@@ -1,778 +1,251 @@
 # Evals
 
-An eval sends a fixed message to your agent and asserts over the
-trajectory it records: the turn completed, the right tool ran with the
-right input, the reply has the right shape. Evals are how you know a
-prompt tweak helped, a refactor didn't regress the agent, and last
-month's fix still holds.
-
-Nothing is mocked. The runner starts (or targets) a real agent server,
-drives sessions over the public API, and grades the events it gets
-back. The model runs and server tools execute, so
-[keep side effects out of eval sessions](#keep-side-effects-out-of-eval-sessions)
-before you point an eval at an agent that posts anywhere.
+An eval sends a fixed input to the real agent and checks the resulting
+trajectory: whether the turn succeeded, which tools ran, and what shape
+the answer took. Use evals to protect behavior you already understand,
+not to discover what the prompt should do.
 
-## Evals, hooks, or hillclimbing?
+Evals run real model turns and real tools. Guard external writes before
+running a suite against an agent that can post, merge, or deploy.
 
-All three read the same session event stream. Pick by the question you
-are asking.
-
-| You want to | Use |
-| --- | --- |
-| Gate one fixed input's behavior, locally and in CI | Evals (this page) |
-| Observe every live session: metrics, audit, alerts | [Hooks](/docs/reference/hooks.md) |
-| Improve an agent one measured round at a time | [Hillclimbing](/docs/hillclimbing.md); each kept win lands an eval |
-
-[Hooks, channel events, or evals?](/docs/reference/hooks.md#hooks-channel-events-or-evals)
-has the side-by-side table.
-
-### When not to write an eval
-
-- Test a server tool's own logic with
-  `agent-sdk call <tool> --dir . --input '{...}'` or a unit test. No
-  model turn, no credential.
-- Explore a prompt with `agent-sdk run --dir . --message "..."` and
-  read the trajectory. Write the eval once you know which decision to
-  gate.
-- Stop a bad turn while it runs with
-  [`needsApproval`](/docs/reference/tools.md#gate-a-tool-on-human-approval)
-  on the tool or `defineResult`. Evals grade
-  after the fact.
-
-## Write your first eval
-
-Evals live under the project-root `evals/` directory, a sibling of
-`agent/`. `agent/evals/` is silently ignored. Discovery loads every
-`.eval.ts` (or `.eval.js`) file under `evals/`, plus one config file.
-
-```text
-my-agent/
-  agent/
-    agent.ts
-    tools/inspect_pr.ts
-  evals/
-    evals.config.ts      # required to run: maxConcurrency
-    readiness.eval.ts    # id: readiness
-    prs.eval.ts          # cases: prs/checkout, prs/search
-```
+## Protect a tool decision
 
-An eval is a single `async test(t)`. You drive the agent with `t.send`
-and assert on the recorded run with the same `t`:
+This smoke case asks for a PR verdict, requires the read-only inspection
+tool, and fails if the agent tries to approve. The CLI reports all three
+decisions together instead of stopping at the first miss.
 
 ```ts
 // evals/readiness.eval.ts
 import { defineEval, includes } from "@cursor/july/evals";
 
 export default defineEval({
-  description: "Inspects a PR without approving it.",
   tags: ["smoke"],
-  timeoutMs: 120_000,
   async test(t) {
     await t.send(
       "Is https://github.com/acme/checkout/pull/42 ready to approve?"
     );
+
     t.succeeded();
     t.calledTool("inspect_pr");
     t.notCalledTool("approve_pr");
-    t.check(t.reply, includes(/ready|approve/i));
+    t.check(t.reply, includes(/ready|blocked|approve/i));
   },
 });
 ```
 
+Evals live in the project-root `evals/` directory, beside `agent/`.
+Add the required concurrency config once:
+
 ```ts
 // evals/evals.config.ts
 import { defineEvalConfig } from "@cursor/july/evals";
 
-export default defineEvalConfig({ maxConcurrency: 20 });
+export default defineEvalConfig({
+  maxConcurrency: 20,
+});
 ```
 
-Run it under Node 22.13 or newer (never Bun) with a Cursor credential
-in place; see [Credentials](#credentials):
-
 ```bash
 agent-sdk eval --dir . --list
-agent-sdk eval --dir . readiness
-```
-
-```text
-PASS  readiness (14.2s) — Inspects a PR without approving it.
-  ✓ succeeded
-  ✓ calledTool(inspect_pr)
-  ✓ notCalledTool(approve_pr)
-  ✓ check(includes)
-
-1 passed, 0 failed, 1 total
-artifacts: <project state directory>/evals/2026-09-11T15-02-11-402Z
-```
-
-Every local run writes each case's assertions, inputs, tool calls, and
-`t.log` lines under that artifacts directory. Open
-`evals/<case-id>.json` there when a case fails; see
-[Where results land](#where-results-land).
-
-## Name cases by path
-
-The file path is the eval's identity, so you don't author an id.
-`evals/builds/api.eval.ts` becomes `builds/api`. An `index` filename
-collapses to its directory: `evals/builds/index.eval.ts` becomes
-`builds`.
-
-One file can hold several datapoints through `cases`. Provide either
-`test` or `cases`, not both. Each case id becomes `<fileId>/<case.id>`:
-
-```ts
-// evals/prs.eval.ts: prs/checkout, prs/search
-export default defineEval({
-  tags: ["smoke", "prs"],
-  cases: [
-    {
-      id: "checkout",
-      description: "Checkout PR readiness.",
-      async test(t) {
-        await t.send(
-          "Is https://github.com/acme/checkout/pull/42 ready to approve?"
-        );
-        t.succeeded();
-        t.calledTool("inspect_pr");
-      },
-    },
-    {
-      id: "search",
-      async test(t) {
-        await t.send(
-          "Check https://github.com/acme/search/pull/7 before approval."
-        );
-        t.succeeded();
-        t.calledTool("inspect_pr");
-      },
-    },
-  ],
-});
-```
-
-Case ids are single path segments, unique within the file. A case can
-set its own `description`, `tags`, `timeoutMs`, `iterations`, `judge`,
-`reporters`, and `metadata`. A case-level value replaces the file-level
-one for that datapoint, except `metadata`, which merges with case keys
-winning, and `reporters`, which adds to the file's list. `metadata` is
-free-form data carried onto the result and every reporter.
-
-A file may instead export an array of `defineEval` calls to fan out
-over a dataset. Ids are then the file id plus a zero-padded index
-(`sql/0000`, `sql/0001`, ...); see [Load a dataset](#load-a-dataset).
-Prefer `cases` when datapoints are hand-written and deserve stable
-names.
-
-### Iterations
-
-`iterations` (file or case, default `1`, cap `100`) runs a datapoint
-repeatedly. Discovery expands `iterations: 3` on case `nyc` to runnable
-ids `weather/nyc/1`, `weather/nyc/2`, and `weather/nyc/3`. The filter
-`weather/nyc` still selects all three. Each expanded case exposes
-`t.iteration` and `t.iterations`.
-
-`maxConcurrency` counts authored datapoints, not expanded iterations.
-Iterations of one datapoint share a concurrency slot and run in
-sequence, so a suite of 11 cases with 3 iterations each and
-`maxConcurrency: 20` has at most 11 cases in flight.
-
-## Drive the agent with `t.send`
-
-`t.send(message, options?)` runs one turn and waits for it to settle:
-complete, park on an approval request, or fail. Several sends in one
-case share the session, which is how you write multi-turn evals.
-
-Each send resolves to a turn result: `message` (the assistant text),
-`sessionId`, `events`, `toolCalls` (tool names in order), `ok`, and
-`index`. The turn carries the same assertion vocabulary as `t`, scoped
-to that turn, so you can grade an intermediate turn before the next
-send overwrites `t.reply`. `turn.expectOk()` throws when the turn
-failed, for later steps that depend on it.
-
-Read the whole case with `t.reply` (last assistant text), `t.events`
-(every event so far), `t.turns` (settled turns, oldest first), and
-`t.sessionId`. `t.signal` aborts when the case hits its timeout; pass
-it to your own async work.
-
-Three options apply on the first send only, because they shape session
-creation:
-
-| Option | Effect |
-| --- | --- |
-| `workspaceFiles` | `{ path: contents }` seeded into the local session workspace. Prefer this over machine-local paths |
-| `workspaceDir` | Absolute harness cwd for the local runtime |
-| `cloud` | Per-session cloud options merged over the agent's static `cloud` config. Pin a fixture repo here for cloud evals instead of on the agent's default `cloud.repos`. On the cloud runtime, seeded files reach the agent as described under [Choose a runtime](/docs/reference/agent-config.md#choose-a-runtime) |
-
-```ts
-await t.send("Review pr/diff.patch and post findings.", {
-  workspaceFiles: {
-    "pr/diff.patch": [
-      "diff --git a/app/routes/search.ts b/app/routes/search.ts",
-      "+res.send(`<h1>Results for ${req.query.q}</h1>`);",
-    ].join("\n"),
-  },
-});
-```
-
-## Assert over the trajectory
-
-Assertions record; they never throw. One run reports every failure
-instead of dying on the first. Assertions on `t` read the whole run.
-Assertions on a turn read only that turn.
-
-| Gate | Checks |
-| --- | --- |
-| `t.succeeded()` | the run did not fail and is not parked on an unanswered approval |
-| `t.parked()` | the run cleanly parked on an unanswered approval request |
-| `t.messageIncludes(token)` | the joined assistant text matches a string or `RegExp` |
-| `t.calledTool(name, matcher?)` | a matching call to `name` happened |
-| `t.notCalledTool(name)` | no request for `name`, in any lifecycle state |
-| `t.loadedSkill(name)` | the agent opened `skills/<name>/SKILL.md` (read, grep, or shell `cat`) |
-| `t.toolOrder(names)` | tool requests appear in this relative order; extra calls allowed |
-| `t.usedNoTools()` | no tool calls at all |
-| `t.maxToolCalls(max)` | at most `max` tool calls |
-| `t.noFailedActions()` | no tool call reported an error |
-| `t.calledSubagent(name, matcher?)` | a matching subagent delegation happened |
-| `t.taggedArtifact(kind?, predicate?)` | at least one [artifact](/docs/reference/artifacts.md) was tagged |
-| `t.event(type, matcher?)` | at least one matching [event](/docs/reference/sessions.md#which-events-can-i-stream) of `type` |
-| `t.notEvent(type, matcher?)` | no matching event of `type` |
-| `t.eventOrder(matchers)` | matching event groups occur in this relative order |
-| `t.eventsSatisfy(label, predicate)` | your predicate over the typed event stream |
-| `t.check(value, expectation)` | any value, against a [builder](#grade-values-with-expectation-builders) |
-| `t.score(name, value)` | a 0-1 score you computed; soft until you add a bar |
-
-Three more assertions gate and return the matched fact. They stop the
-test body when nothing matches, without a duplicate execution error.
-`t.requireToolCall(name, matcher?)` returns the call so later code can
-read its `input` and `output`. `t.requireInputRequest(filter?)` returns
-the single pending approval request. `await t.require(value, expectation)`
-does the same for a value check.
-
-A case with no assertions passes when at least one turn completed. Add
-`t.succeeded()` and behavior gates anyway. They make the contract
-visible in review.
-
-### What good cases assert
-
-Gate decisions and shape, not prose. Model wording varies run to run.
-Tool choice, tool avoidance, and output structure are the stable
-contract.
-
-1. `t.succeeded()`: always, first.
-2. The tool decision: `calledTool` for the intended path,
-   `notCalledTool` for the likely wrong alternative. The pair is
-   stronger than either alone.
-3. Output shape: a regex for the contract (`/ready|blocked/i`, a JSON
-   marker, a findings-block fence), never exact sentences.
-4. For structured output, parse `t.reply` and check fields with
-   `matches` or `satisfies` instead of substring-matching JSON.
-
-The common failure modes: asserting exact phrasing, packing more than
-about five gates into one case (split it), and cases that depend on
-live external state that drifts (pin the input).
-
-### Narrow tool assertions with matchers
-
-With no matcher, `calledTool` is request-based: a requested call counts
-even before its result arrives. A matcher narrows it:
-
-```ts
-t.calledTool("inspect_pr", { status: "completed" });
-t.calledTool("apply_agents", { input: { verdict: "update" } });
-t.calledTool("bash", { input: { command: /^gh pr view/ }, count: 1 });
-t.calledTool("read_file", {
-  output: (value) => String(value).includes("TODO"),
-});
+agent-sdk eval --dir . --tag smoke
 ```
 
-`input`, `output`, and `count` accept a literal, a `RegExp`, or a
-predicate. Object literals partial-deep-match, so `{ verdict: "update" }`
-matches arguments that also carry other keys. `status` is one of
-`completed`, `failed`, `pending`, or `rejected` (a human denied the
-approval). `calledSubagent` takes `{ output, status, count, callId }`.
-`event`, `notEvent`, and `eventOrder` take `{ data, count }`.
-
-### Grade values with expectation builders
+## Check answer shape or quality
 
-`t.check(value, expectation)` grades any value: `t.reply`, a parsed
-JSON field, a tool's output.
-
-| Builder | Checks | Severity |
-| --- | --- | --- |
-| `includes(string \| RegExp)` | substring or match; structured values are stringified first | gate |
-| `equals(value)` | deep equality | gate |
-| `matches(schema)` | a Standard Schema (Zod, Valibot, ...) or anything with `safeParse` | gate |
-| `similarity(expected)` | normalized text similarity, 0-1 | soft |
-| `satisfies(predicate, label)` | your predicate; `label` is the failure detail | gate |
+Prefer a deterministic shape check when the output has a contract. The
+failure tells you which field was wrong, and the case remains stable
+when the model changes its wording.
 
 ```ts
-import { matches, satisfies } from "@cursor/july/evals";
+import { matches } from "@cursor/july/evals";
 import { z } from "zod";
 
 const verdict = JSON.parse(t.reply ?? "{}");
 t.check(
   verdict,
-  matches(z.object({ ready: z.boolean(), blockers: z.array(z.string()) }))
-);
-t.check(
-  verdict.blockers.length,
-  satisfies((n) => (n as number) <= 3, "at most 3 blockers")
+  matches(
+    z.object({
+      ready: z.boolean(),
+      blockers: z.array(z.string()),
+    })
+  )
 );
 ```
 
-`normalizedSimilarity(actual, expected)` returns the same 0-1 score as
-`similarity`, for use with `t.score`.
-
-### Record without gating
-
-- `t.metric(name, value)` records a structured score or label. It shows
-  on the CLI result, the playground case card, JUnit output, and
-  artifacts.
-- `t.log(message)` records a debug line, streamed under `--verbose`.
-- `t.skip(reason)` ends the case as skipped. Skipped cases report
-  separately and never change the exit code. Call it before sending
-  messages.
-
-## Gates, soft scores, and verdicts
-
-Every assertion returns a handle, so severity rides on the assertion
-instead of a separate thresholds map:
+Use a judge when correctness depends on meaning that a schema, regex, or
+known value cannot capture:
 
 ```ts
-t.succeeded(); // gate (default)
-t.calledTool("get_weather").soft(); // tracked, never fails
-t.check(t.reply, similarity("Sunny, 72°F")).atLeast(0.8); // soft, with a bar
-t.judge.closedQA("cites a source").gate(0.8); // promoted to a gate
+t.judge
+  .factuality("The lint step failed on src/sidebar.ts.")
+  .atLeast(0.8);
 ```
 
-- `.gate(threshold?)` is hard. A miss fails the case and `eval` exits 1.
-- `.soft(threshold?)` is tracked. With no threshold it never fails.
-- `.atLeast(threshold)` is soft with a bar. A miss marks the case
-  `scored`.
-
-Each case ends with one verdict:
+Judge checks are tracked scores until you give them a hard gate.
+Configure the judge model in `evals.config.ts`. The
+[Evals reference](/docs/reference/evals.md#judges) owns grader options and
+severity rules.
 
-| Verdict | Meaning | Exit code |
-| --- | --- | --- |
-| `passed` | every gate passed and no soft bar was missed | 0 |
-| `failed` | a gate failed, or the test body threw | 1 |
-| `scored` | only soft bars were missed | 0, or 1 under `--strict` |
-| `skipped` | `t.skip(reason)`, or a judge with no credentials | 0 |
+## Test a conversation or approval
 
-The CLI prints soft misses as `~` and gate misses as `✗`. Start a new
-benchmark with `t.score("recall", recall)` and `.atLeast()` so its
-number reports for a while without blocking merges. Add `--strict`
-once the bars are trustworthy.
-
-## Judge free-form output
-
-When wording matters and no regex captures it, `t.judge` grades with an
-LLM. The graders are `factuality(expected)`, `summarizes(expected)`,
-`closedQA(criteria)`, and `sql(expected)`. Each scores `t.reply` by
-default; pass `{ on }` to grade another value.
+Several `t.send` calls in one case share a session. Assertions on a
+returned turn inspect only that turn, while assertions on `t` inspect
+the whole conversation.
 
 ```ts
-const summary = await t.send("Why did CI fail on PR 42?");
-t.judge.factuality("The lint step failed on src/sidebar.ts.").atLeast(0.7);
-t.judge.closedQA("names the failing step", { on: summary.message }).gate(1);
-```
-
-Judge assertions are soft by default, so a judge never fails a build
-until you give it a bar with `.atLeast()` or promote it with `.gate()`.
-The recorded detail names the choice the judge made and its rationale.
-
-The judge model comes from `defineEvalConfig({ judge })`,
-`defineEval({ judge })`, a case-level `judge`, or a per-call
-`{ model }`. The nearest one wins. A judge call with no model
-configured fails the case. A judge that cannot reach a model (no
-credential) ends the case as `skipped`, unless a deterministic gate
-already failed.
+const first = await t.send("Inspect PR 42.");
+first.calledTool("inspect_pr");
 
-For a domain-specific judge whose verdict is not a single score,
-`t.judge.model(prompt)` sends a raw prompt to the same model and
-returns the reply. Record the parsed result with `t.score` or
-`t.check`. Anything derived from the agent under test is untrusted
-input to your prompt: wrap it with `fenceUntrusted` and include
-`EVAL_JUDGE_INJECTION_GUARD`, as the built-in graders do.
-
-```ts
-import {
-  EVAL_JUDGE_INJECTION_GUARD,
-  fenceUntrusted,
-} from "@cursor/july/evals";
-
-const gold = ["XSS in search.ts", "open redirect in login.ts"];
-const reply = await t.judge.model(
-  [
-    "For each GOLD finding, answer whether SUBMISSION reports it.",
-    "Reply with one line per finding: <index> YES|NO.",
-    EVAL_JUDGE_INJECTION_GUARD,
-    fenceUntrusted("GOLD", gold.map((g, i) => `${i + 1}. ${g}`).join("\n")),
-    fenceUntrusted("SUBMISSION", t.reply ?? ""),
-  ].join("\n\n")
-);
-const hits = reply.match(/\bYES\b/g)?.length ?? 0;
-t.score("recall", hits / gold.length).atLeast(0.5);
+const followUp = await t.send("Now summarize only the blockers.");
+followUp.usedNoTools();
+t.check(followUp.message, includes(/blocker/i));
+t.succeeded();
 ```
 
-## Keep side effects out of eval sessions
-
-Eval sessions run the real agent, tools included. A reviewer that
-comments on GitHub or posts to Slack will do so from an eval unless
-the tool checks the session's purpose. Eval sessions carry
-`purpose: "eval"`; live traffic carries `"live"`. Branch on it in the
-tool, hook, or result handler that actuates:
+For a tool that requires approval, the expected result is a parked turn
+instead of a completed one:
 
 ```ts
-// agent/tools/post_findings.ts
-async execute({ findings }, ctx) {
-  if (ctx.session.purpose === "eval") {
-    return { posted: false, reason: "eval", count: findings.length };
-  }
-  // post the review
-}
-```
-
-Return a shaped result instead of throwing, so the eval can still
-assert `t.calledTool("post_findings", { input: ... })` on the decision.
-The same check belongs in [hooks](/docs/reference/hooks.md) that meter or
-page and in `defineResult` commits.
-
-## Worked examples
-
-### Multi-turn: grade each turn
-
-```ts
-// evals/intro.eval.ts
-import { defineEval, includes, satisfies } from "@cursor/july/evals";
-
-export default defineEval({
-  description: "Introduces itself once; a repeat mention gets a short ack.",
-  async test(t) {
-    const intro = await t.send("Meet Jenny! @Jenny introduce yourself.");
-    intro.expectOk();
-    t.check(intro.message, includes(/jenny/i));
-
-    const repeat = await t.send("Meet, @Jenny!");
-    t.succeeded();
-    repeat.usedNoTools();
-    t.check(
-      repeat.message,
-      satisfies((r) => (r as string).trim().length <= 280, "short ack")
-    );
-    t.check(
-      repeat.message,
-      satisfies((r) => !/what i can do/i.test(r as string), "no second intro")
-    );
-  },
+await t.send("Apply the approved policy update.");
+t.parked();
+t.calledTool("apply_policy", {
+  input: { verdict: "update" },
 });
 ```
 
-`t.succeeded()` grades the whole session. `repeat.usedNoTools()` and the
-checks on `repeat.message` read only the second turn, even though
-`t.reply` now holds its text.
+Pair an approval case with a safe branch that calls
+`t.notCalledTool("apply_policy")`, so both sides of the decision stay
+protected.
 
-### Approvals: assert the parked decision
+## Pin realistic inputs
 
-For a tool with `needsApproval`, the turn parks instead of finishing.
-Gate on `t.parked()` and on the arguments the model chose:
+A useful eval changes only when the agent changes. Freeze canonical chat
+prompts, seed workspace evidence directly, and save webhook payloads
+instead of depending on a developer's checkout or live external state.
 
 ```ts
-// evals/agents.eval.ts (one case; RULE and SLACK are fixture strings)
-{
-  id: "update-rule",
-  description: "A repeated billing rule parks the AGENTS.md write.",
-  async test(t) {
-    await t.send("Weekly AGENTS.md review. Read week/ and call apply_agents once.", {
-      workspaceFiles: {
-        "week/prs.md": RULE,
-        "week/slack.md": SLACK,
-        "week/tree/AGENTS.md.txt": "# API\n\nKeep handlers thin.\n",
-      },
-    });
-    t.parked();
-    t.calledTool("apply_agents", { input: { verdict: "update" } });
+await t.send("Review pr/diff.patch and report blockers.", {
+  workspaceFiles: {
+    "pr/diff.patch": [
+      "diff --git a/app/routes/search.ts b/app/routes/search.ts",
+      "+res.send(`<h1>Results for ${req.query.q}</h1>`);",
+    ].join("\n"),
   },
-},
-```
-
-`t.parked()` and `t.succeeded()` are exclusive: a parked run is a clean
-stop on an unanswered approval, not a completed one. Pair the parked
-case with a sibling that expects `verdict: "skip"` and `t.succeeded()`,
-so both branches stay pinned.
-
-## Pin fixtures
-
-A fixed input is what makes an eval repeatable. Pick the fixture by the
-surface under test.
-
-| Agent surface | Fixture |
-| --- | --- |
-| Chat or domain assistant | One canonical prompt string, chosen once and frozen |
-| Tool-heavy | Run `agent-sdk call <tool>` first to pin what the tool returns, then freeze the prompt that triggers it |
-| GitHub webhook | `agent-sdk github replay <pr> --events '*' --dry-run --out fixtures/github` snapshots real payloads for offline replay ([GitHub guide](/docs/guides/github.md#test-with-github-replay)) |
-| PR reviewer with host preparation | Diff, metadata, and gold labels pinned to commit SHAs; keep any live PR matrix small |
-| Workspace-dependent | `workspaceFiles` on the first `t.send`, never developer-machine paths |
-
-Tag the fast, reliably passing core `smoke` and run `--tag smoke` in
-the inner loop. Leave slow or drift-prone cases untagged for explicit
-runs.
-
-### Materialize API-backed fixtures
-
-An input that only points at external data (a pull request URL, a
-snapshot id, a pair of commit SHAs) is not self-contained. Fetch it once
-and commit the rendered fixture before you expand the suite:
-
-1. Save the diff, metadata, and labels under `fixtures/` at pinned
-   revisions.
-2. Seed those files with `workspaceFiles`, or read them from the
-   fixture directory.
-3. Assert decisions and output shape against the saved evidence.
-4. Keep a small `smoke` subset for any remaining live checks.
-
-`maxConcurrency` limits parallel datapoints, not the model or API
-fan-out inside one datapoint. Materialized fixtures keep a large suite
-from exhausting provider and GitHub rate limits.
-
-### Load a dataset
-
-Read committed fixtures with `loadJson`, `loadJsonl`, and `loadYaml`
-from `@cursor/july/evals/loaders`. Relative paths resolve against the
-project root the runner discovered, not the cwd the CLI ran from. Eval
-files are ES modules, so top-level `await` can load a dataset and fan
-one file out over it:
-
-```ts
-// evals/sql.eval.ts: sql/0000, sql/0001, ...
-import { defineEval, equals } from "@cursor/july/evals";
-import { loadYaml } from "@cursor/july/evals/loaders";
-
-const rows = await loadYaml<{ task: string; prompt: string; sql: string }[]>(
-  "evals/data/cases.yaml"
-);
-
-export default rows.map((row) =>
-  defineEval({
-    description: row.task,
-    async test(t) {
-      await t.send(row.prompt);
-      t.succeeded();
-      t.check(t.reply, equals(row.sql));
-    },
-  })
-);
-```
-
-## Configure eval runs
-
-`evals/evals.config.ts` (or `.js`) holds project-wide defaults. It must
-set `maxConcurrency`. Each case issues real model requests, so
-concurrency is hard-capped at 200; the templates use 10.
-`eval --list` works without the file. Running a case does not.
-
-```ts
-import { defineEvalConfig } from "@cursor/july/evals";
-
-export default defineEvalConfig({
-  maxConcurrency: 20,
-  timeoutMs: 180_000,
-  judge: { model: "gpt-5.4-mini" },
 });
 ```
 
-| Option | Default | Meaning |
-| --- | --- | --- |
-| `maxConcurrency` | required | Datapoints in flight at once, 1-200 |
-| `timeoutMs` | `180_000` | Per-case timeout. Precedence: case or file `timeoutMs`, then `--timeout-ms`, then this value |
-| `judge` | unset | Default judge model for `t.judge.*` |
-| `reporters` | unset | Destinations that observe every case; `--skip-report` suppresses them |
-| `maxPlaygroundRuns` | `20` | Batches kept in playground history (cap 500). Counts `--prod` and `--url` batches, not the default local run |
-
-Reporters ship results somewhere; the runner still does the grading.
-`JUnit({ filePath, suiteName? })` writes JUnit XML and
-`Artifacts({ dir })` writes per-case files, both from
-`@cursor/july/evals/reporters`. A custom reporter is an object with any
-of `onRunStart`, `onEvalComplete`, and `onRunComplete`. A reporter that
-throws is logged and never fails the run. CI usually attaches the
-built-in two with `--junit` and `--artifacts` instead of `reporters`,
-so output paths stay with the pipeline, not the eval author.
-
-Playground batches survive restarts when the project configures
-durable storage. Otherwise they live in process
-memory until `serve` exits.
-
-## Run evals from the CLI
+For GitHub agents, snapshot a PR into committed fixtures:
 
 ```bash
-agent-sdk eval --dir . --list                          # discover only
-agent-sdk eval --dir .                                 # run all
-agent-sdk eval --dir . builds/checkout                 # one datapoint
-agent-sdk eval --dir . builds search                   # several ids or prefixes
-agent-sdk eval --dir . --tag smoke --tag pull-request  # any matching tag
-agent-sdk eval --dir . --verbose                       # t.log lines + reply snippets
-agent-sdk eval --dir . --timeout-ms 300000             # override the per-case timeout
+agent-sdk github replay acme/checkout#42 \
+  --events '*' \
+  --dry-run \
+  --out fixtures/github
 ```
 
-Id filters use OR semantics. Each filter selects an exact id and its
-descendants: `builds` selects `builds`, `builds/checkout`, and every
-other case below that path. Repeated tags also use OR. With both ids
-and tags, a case must match both groups.
+Keep a small, fast `smoke` set for every change. Put larger datasets and
+drift-prone live cases behind an explicit selection. See
+[Datasets and fixtures](/docs/reference/evals.md#datasets-and-fixtures) for
+loaders and case expansion.
 
-By default `eval` boots a throwaway server with its own state root, so
-cases don't inherit your checkout's `AGENTS.md` and session state stays
-out of the project. Artifacts still land in the project state
-directory; see [Where results land](#where-results-land). `--slug`
-picks the target in a multi-agent directory.
+## Run and debug a regression suite
 
-`--url` runs the batch on a running server instead, the same way
-`--prod` does: that server discovers its own `evals/`, results land in
-its playground history, and the local-only flags (`--junit`,
-`--artifacts`, `--max-concurrency`, `--skip-report`) do not apply. See
-[Run evals in the playground or on a deployment](#run-evals-in-the-playground-or-on-a-deployment).
+Run one case while iterating, then the smoke suite before review. A
+failure artifact records the input, assertions, tool calls, and final
+text, making it the first place to look before changing the prompt.
 
 ```bash
-agent-sdk eval --url http://127.0.0.1:3000/weather-agent \
-  --bearer-token "$AGENT_TOKEN"
+agent-sdk eval --dir . readiness
+agent-sdk eval --dir . --tag smoke
+agent-sdk eval --dir . --json
 ```
 
-See [CLI: eval](/docs/reference/cli.md#eval) for every flag.
-
-### Credentials
-
-Model turns need a
-[Cursor credential](/docs/reference/cli.md#environment-variables):
-`CURSOR_API_KEY`, `CURSOR_SERVICE_ACCOUNT_KEY`, or a saved
-`agent-sdk login`. The judge uses the same one. `eval --list` needs
-none.
+By default, results land under `evals/<stamp>/` in the project state
+directory. Open `evals/<case-id>.json` inside that run directory for the
+failed case. This artifact location is independent of `--state-root`;
+pass `--artifacts <dir>` to choose another destination.
 
-### Where results land
-
-Every local run writes artifacts to a timestamped directory under
-`evals/` in the project state directory, whatever `--state-root` says.
-`--artifacts <dir>` chooses the path and `--no-artifacts` skips them.
-The directory holds `summary.json`,
-`results.jsonl`, and `evals/<case-id>.json` with every assertion, the
-inputs, tool calls with arguments and output, the final text, and
-`t.log` lines. Start there when a case fails. `--out <file>` also
-writes the full results JSON to a path of your choice.
-
-The artifact does not include the session's event stream. Pass
-`--state-root <path>` to keep the ephemeral server's
-[session data](/docs/reference/sessions.md#where-does-the-agent-sdk-store-session-data)
-on disk when you need the raw events.
+Model turns need a credential. Resolution checks `CURSOR_API_KEY`,
+`CURSOR_API_KEY_FILE`, `CURSOR_SERVICE_ACCOUNT_KEY`, then a saved
+`agent-sdk login`. `eval --list` only discovers cases and needs none.
 
 ## Run evals in CI
 
-Run the suite non-interactively, write JUnit for the CI annotations,
-and fail the job on a red gate:
+Write machine-readable output and JUnit annotations, then let a failed
+gate fail the job:
 
 ```bash
-# CURSOR_API_KEY comes from the CI secret store
+# CURSOR_API_KEY comes from the CI secret store.
 agent-sdk eval --dir . --json --no-stream \
   --junit reports/evals.xml \
   --artifacts reports/evals \
   > reports/evals.json
 ```
 
-The exit code follows the
-[verdict table](#gates-soft-scores-and-verdicts); `2` means nothing
-matched the selection. `--max-concurrency` overrides the project
-setting, for example to run lower on a shared runner.
-
-The JSON on stdout carries the totals and one result per case:
-
-```json
-{
-  "ok": true,
-  "passed": 1,
-  "failed": 0,
-  "scored": 0,
-  "skipped": 0,
-  "strict": false,
-  "artifactsDir": "/work/my-agent/reports/evals",
-  "results": [
-    {
-      "id": "readiness",
-      "verdict": "passed",
-      "ok": true,
-      "assertions": [
-        { "name": "succeeded", "passed": true },
-        { "name": "calledTool(inspect_pr)", "passed": true }
-      ],
-      "sessionId": "ses_123",
-      "inputs": ["Is https://github.com/acme/checkout/pull/42 ready to approve?"],
-      "toolCalls": [{ "toolName": "inspect_pr", "isError": false }],
-      "metrics": {},
-      "logs": [],
-      "durationMs": 12340
-    }
-  ]
-}
-```
+Run `--tag smoke` on pull requests and the full suite on a schedule when
+cost or latency makes every-push coverage impractical. Use soft scores
+for a new probabilistic benchmark until its threshold is trustworthy.
 
-Each result can also include `description`, `finalText`, `tools`,
-`error`, `skipReason`, `metadata`, `tags`, and tool `args` and `output`.
-A soft miss shows as `"severity": "soft"` with `score` and `threshold`
-on the assertion. This shape lets CI report the failed assertion
-without parsing terminal text.
+The [CLI reference](/docs/reference/cli.md#eval) covers selectors, hosted
+runs, exit codes, and JSON output.
 
-Keep CI green without weakening gates:
-
-- Run `--tag smoke` on every push and the full suite on a schedule.
-- For probabilistic behavior, use `iterations` and a soft bar instead
-  of one hard gate.
+## Keep side effects out of eval sessions
 
-## Run evals in the playground or on a deployment
+Eval sessions carry `purpose: "eval"`. Check it at the deterministic
+write boundary and return the decision without performing the external
+action:
 
-Start the server, open the playground, and choose **Evals**. Run every
-case or one case, watch progress, and open the resulting session trace.
-Playground runs target the live server instead of an ephemeral one, so
-their sessions appear in the session list. One batch runs at a time.
+```ts
+async execute({ findings }, ctx) {
+  if (ctx.session.purpose === "eval") {
+    return {
+      posted: false,
+      reason: "eval",
+      count: findings.length,
+    };
+  }
 
-```bash
-agent-sdk serve --dir .
+  return await postFindings(findings);
+}
 ```
 
-`--prod` (or `--url`) starts the same server-side batch on the team's
-hosted deployment (or the server you name), so results land in that
-server's playground history:
+This preserves the tool call in the trajectory, so the eval can still
+assert that the agent chose to post.
 
-```bash
-agent-sdk eval --prod --slug vulnerability-scanner --tag deepsec
-# Eval ID: <evalId>
-# Playground: <deployment>/playground?view=evals&evalRunId=<evalId>
-agent-sdk eval status <evalId> --prod --slug vulnerability-scanner
-agent-sdk eval cancel <evalId> --prod --slug vulnerability-scanner
-```
+## Evals, hooks, or hillclimbing?
 
-The CLI prints the Eval ID as soon as the batch is accepted. Pass
-`--no-wait` to return right away and poll with `eval status` later; it
-exits `3` while the batch is still running. Hosted history follows
-`maxPlaygroundRuns` and the persistence rule under
-[Configure eval runs](#configure-eval-runs). The HTTP surface is under
-[Playground eval routes](/docs/reference/http-api.md#playground-eval-routes).
+Use an eval to gate one fixed input. Use a
+[hook](/docs/reference/hooks.md) to observe every live session for metrics,
+auditing, or alerts. Use [hillclimbing](/docs/hillclimbing.md) to decide
+which source change improves a fixed set of inputs; every kept
+hillclimb change should add an eval.
 
-## Keep improvements with regression evals
+## Gates and scores
 
-Every [hillclimb](/docs/hillclimbing.md) round that keeps a change must
-land an eval that would have failed before the change. If you can't
-express the improvement as a gate (a `calledTool` shift, a bounded
-`maxToolCalls`, an output-shape check), the improvement is unverified,
-and it'll regress silently.
+`t.succeeded()`, tool assertions, and deterministic checks are hard
+gates by default. A failed gate fails the case. `.soft()` records a
+measurement without blocking, while `.atLeast(threshold)` records a
+score and marks a miss as `scored`.
+
+Start with decisions and output shape: the intended tool, the tempting
+wrong tool, and one structural check. Exact prose is rarely a stable
+contract.
+
+## Keep improvements with regression evals
 
-The rule cuts the other way too: never weaken an existing gate to make
-a round pass. That's the freeze line moving, and it turns your
-regression suite into a list of checks that no longer protect anything.
+Every kept improvement needs an eval that would have failed before the
+change. A tool-choice gate, bounded `maxToolCalls`, or output-shape check
+turns the improvement into a durable contract.
 
-## What's next
+Never weaken an existing gate to make a new implementation pass. That
+moves the freeze line instead of proving the change.
 
-Continue with these pages:
+## Related
 
-- [Hillclimbing](/docs/hillclimbing.md): the loop evals make trustworthy
-- [Building agents with agents](/docs/building-with-agents.md): have a
-  coding agent write the first suite
-- [GitHub guide](/docs/guides/github.md): deterministic webhook fixtures
-  with `github replay`
-- [Sessions and streaming](/docs/reference/sessions.md): the events
-  `t.events` contains
+- [Evals reference](/docs/reference/evals.md): complete authoring and runner
+  contracts
+- [`skills/evals/SKILL.md`](https://github.com/cursor/cursor/blob/main/packages/agent-serve/skills/evals/SKILL.md): author and seed
+  cases with a coding agent
+- [Hillclimbing](/docs/hillclimbing.md): measure a change and lock the win
+- [Sessions](/docs/reference/sessions.md): events available to trajectory
+  assertions