@cursor/july 0.1.91 → 0.1.93

package/dist/docs/evals.md

@@ -0,0 +1,460 @@
+# Evals
+
+An eval is a repeatable check that runs your agent against a fixed input
+and gates the recorded trajectory: the run completed, the right tool
+ran, the reply has the right shape. Evals are how you know a prompt
+tweak helped, a refactor didn't regress the agent, and last month's fix
+is still holding.
+
+Evals exercise the same surface your users hit. The runner starts (or
+targets) a real agent server, drives sessions over the public API, and
+grades what comes back. A passing eval means the agent started,
+accepted a message, and did what you asserted.
+
+## Define evals with `defineEval`
+
+The Agent SDK discovers evals under the project-root `evals/` directory,
+in `.eval.ts` or `.eval.js` files. That's a sibling of `agent/`, never
+inside it (`agent/evals/` is silently ignored). TypeScript is the normal
+authoring format.
+
+The file path is the eval's identity, so you don't author an id.
+Directories group related evals: `evals/builds/api.eval.ts` becomes id
+`builds/api`. An `index` filename collapses to its directory, so
+`evals/builds/index.eval.ts` becomes `builds`.
+
+An eval is a single `async test(t)`. You drive the agent with `t` and
+assert on the run with the same `t`:
+
+```ts
+// evals/readiness.eval.ts
+import { defineEval, includes } from "@cursor/july/evals";
+
+export default defineEval({
+  description: "Inspects a PR without approving it.",
+  tags: ["smoke"],
+  timeoutMs: 120_000,
+  async test(t) {
+    await t.send(
+      "Is https://github.com/acme/checkout/pull/42 ready to approve?"
+    );
+    t.succeeded();
+    t.calledTool("inspect_pr");
+    t.notCalledTool("approve_pr");
+    t.check(t.reply, includes(/ready|approve/i));
+  },
+});
+```
+
+One file can also hold several datapoints through `cases` (provide
+either `test` or `cases`, not both). Each case id becomes
+`<fileId>/<case.id>`:
+
+```ts
+// evals/prs.eval.ts → prs/checkout, prs/search
+export default defineEval({
+  tags: ["smoke", "prs"],
+  cases: [
+    {
+      id: "checkout",
+      description: "Checkout PR readiness.",
+      async test(t) {
+        await t.send(
+          "Is https://github.com/acme/checkout/pull/42 ready to approve?"
+        );
+        t.succeeded();
+        t.calledTool("inspect_pr");
+      },
+    },
+    {
+      id: "search",
+      async test(t) {
+        await t.send(
+          "Check https://github.com/acme/search/pull/7 before approval."
+        );
+        t.succeeded();
+        t.calledTool("inspect_pr");
+      },
+    },
+  ],
+});
+```
+
+Case ids must be single path segments, unique within the file.
+Each case can set its own `description`, `tags`, `timeoutMs`, and
+`iterations`. A case-level value replaces the file-level value for that
+datapoint.
+
+### Iterations
+
+`iterations` (file or case, default `1`) runs a datapoint repeatedly.
+Discovery expands `iterations: 3` on case `nyc` to runnable ids
+`weather/nyc/1`, `weather/nyc/2`, `weather/nyc/3` (filter prefix
+`weather/nyc` still selects all three). Each expanded case exposes
+`t.iteration` / `t.iterations` on the test context. Cap is 100.
+
+`maxConcurrency` counts **authored datapoints**, not expanded
+iterations: siblings `…/1`…`…/n` share one concurrency slot and run
+sequentially. A suite with 11 cases × 3 iterations and
+`maxConcurrency: 20` therefore has at most 11 cases in flight, not 33.
+
+## Configure eval runs
+
+Each project with evals needs `evals/evals.config.ts` or
+`evals/evals.config.js`, and it must set `maxConcurrency`. Each case
+issues real model-provider requests, so concurrency is capped hard at
+200. Existing projects use 20. Discovery with `eval --list` works
+without this file, but running a case does not.
+
+```ts
+import { defineEvalConfig } from "@cursor/july/evals";
+
+export default defineEvalConfig({
+  maxConcurrency: 20, // required
+  // timeoutMs: 180_000, // optional project-wide default
+  // judge: { model: "..." }, // default judge model for t.judge.*
+  // reporters: [], // destinations that observe every case
+  // maxPlaygroundRuns: 50, // playground history only (default 20)
+});
+```
+
+The timeout order is case or file `timeoutMs`, CLI `--timeout-ms`,
+project config `timeoutMs`, then the 180-second runner default.
+
+The optional fields:
+
+| Option | Default | Meaning |
+| --- | --- | --- |
+| `timeoutMs` | `180_000` | Project-wide per-case timeout |
+| `judge` | unset | Default judge model for `t.judge.*`; see [Judge free-form output](#judge-free-form-output) |
+| `reporters` | unset | Destinations that observe every case; `--skip-report` suppresses them |
+| `maxPlaygroundRuns` | `20` | Max batches in the playground / `/v1/dev/evals*` history (not CLI `eval`). Hard-capped at 500. |
+
+Reporters come from `@cursor/july/evals/reporters`: `JUnit` writes a
+JUnit XML file for CI, `Artifacts` writes per-case files, and
+`combineReporters` merges several into one (`renderJUnitXml` renders
+the XML for a custom destination). A file or case can add its own
+`reporters` on top of the config list.
+
+Playground batches survive restarts whenever `agent/storage.ts` exists
+with an `evals` table or a KV core providing `delete` and `list` (the
+table is derived over the core); see
+[Storage](/docs/storage.md#eval-and-a-b-tables). Without storage they live
+in process memory and disappear when `serve` exits. Navigating away
+and back still works while the process is up.
+
+## Drive and assert with `t`
+
+`t` is both the driver and the assertion surface. You write ordinary
+control flow, sending turns and asserting inline.
+
+Drive the agent with `t.send(message, options?)`. It runs one turn and
+waits for the session to park or fail. Multiple sends in one case share
+the session, which is how you write multi-turn evals.
+
+Each `t.send` resolves to a turn result with `message`, `sessionId`,
+`events`, `toolCalls`, `ok`, and `index`. The turn carries the same
+assertion vocabulary as `t`, scoped to that turn, so you can grade an
+intermediate turn before the next send overwrites `t.reply`.
+`turn.expectOk()` throws when the turn failed, for later steps that
+depend on it.
+
+Read the full case state with `t.reply` (the last assistant text),
+`t.events` (session events captured so far), `t.turns` (settled
+turns, oldest first), and `t.sessionId`. `t.signal` aborts when the
+case hits its timeout; pass it to your own async work.
+
+Assert with the gates:
+
+| Gate | Checks |
+| --- | --- |
+| `t.succeeded()` | the run did not fail and is not parked on an unanswered approval |
+| `t.parked()` | the run cleanly parked on an unanswered approval request |
+| `t.messageIncludes(token)` | the joined assistant text matches a string or `RegExp` |
+| `t.calledTool(name, matcher?)` | a matching call to `name` happened |
+| `t.notCalledTool(name)` | no request for `name`, in any lifecycle state |
+| `t.loadedSkill(name)` | the agent opened the skill's `SKILL.md` (read, grep, or shell `cat`) |
+| `t.toolOrder(names)` | tool requests appear in this relative order (extra calls allowed) |
+| `t.usedNoTools()` | no tool calls at all |
+| `t.maxToolCalls(max)` | at most `max` tool calls |
+| `t.noFailedActions()` | no tool call reported an error |
+| `t.calledSubagent(name, matcher?)` | a matching subagent delegation happened |
+| `t.taggedArtifact(kind?, predicate?)` | at least one [artifact](/docs/reference/artifacts.md) was tagged |
+| `t.event(type, matcher?)` | at least one matching event of `type` occurred |
+| `t.notEvent(type, matcher?)` | no matching event of `type` occurred |
+| `t.eventOrder(matchers)` | matching event groups occur in this relative order |
+| `t.eventsSatisfy(label, predicate)` | your predicate over the typed event stream |
+| `t.check(value, expectation)` | any value, against a builder |
+| `t.score(name, value)` | records a 0–1 score you computed; soft until you add a bar |
+| `t.requireToolCall(name, matcher?)` | gates on a matching call and returns it, so later code can read its input and output |
+| `t.requireInputRequest(filter?)` | gates on exactly one pending approval request and returns it |
+
+Every gate returns a handle: `.soft()` demotes it to tracked-only,
+`.atLeast(0.7)` adds a soft score bar, and `.gate(0.8)` promotes a
+scored assertion into a hard gate.
+
+With no matcher, `calledTool` is request-based: a requested call counts
+even when its result has not arrived. Pass
+`t.calledTool("inspect_pr", { status: "completed" })` to require the
+call to return. `input`, `output`, and `count` matcher fields accept a
+literal, a `RegExp`, or a predicate.
+
+The expectation builders are `includes(string | RegExp)`,
+`equals(value)`, `matches(schema)`, `similarity(expected)`, and
+`satisfies(predicate, label)`. `includes` stringifies its input,
+`equals` compares values deeply, `matches` validates against a Standard
+Schema (or anything with `safeParse`, like Zod), `similarity` scores
+normalized text similarity, and `satisfies` runs your predicate. The
+plain function `normalizedSimilarity(actual, expected)` returns the
+same 0–1 score for use with `t.score`.
+
+A few more context members shape a case: `t.require(value, expectation)`
+records a gate and stops the test body when it fails, without a
+duplicate execution error. `t.skip(reason)` ends the case as skipped
+(reported separately, never changes the exit code; call it before
+sending messages). `t.metric(name, value)` records a structured score
+for the playground case card. `t.log(message)` records a debug line for
+the CLI and playground result.
+
+Three `t.send` options apply on session create (first `t.send` only):
+
+- `workspaceFiles`: `{ path: contents }`, seeded into the local session
+  workspace. Prefer this over machine-local paths.
+- `workspaceDir`: absolute harness cwd (local runtime).
+- `cloud`: per-session cloud options merged over the agent's static
+  `cloud` config (repos / env / …). Use a pinned `repos` override to
+  attach a fixture repo for cloud evals without putting it on the
+  agent's default `cloud.repos`. Cloud ignores `workspaceFiles` seeds.
+
+```ts
+const toolResults = t.events.filter((e) => e.type === "action.result");
+t.check(
+  toolResults.length,
+  satisfies((n) => (n as number) <= 4, "at most 4 tool calls")
+);
+```
+
+A case with no explicit gates falls back to whether at least one turn
+completed successfully. Add `t.succeeded()` and behavior-specific gates
+anyway. They make the contract visible during review.
+
+### Judge free-form output
+
+When wording matters and no regex captures it, `t.judge` grades the
+reply with an LLM. The built-in graders are `factuality(expected)`,
+`summarizes(expected)`, `closedQA(criteria)`, and `sql(expected)`. Each
+scores `t.reply` by default; pass `{ on }` to grade another value.
+
+```ts
+t.judge.factuality("It is 54°F in NYC right now.").atLeast(0.7);
+```
+
+Judge assertions are soft by default, so a judge never fails a build
+until you give it a bar with `.atLeast(0.7)` or promote it with
+`.gate(0.8)`. The judge model comes from `defineEvalConfig({ judge })`,
+`defineEval({ judge })`, a case-level `judge`, or a per-call
+`{ model }` override; the nearest one wins. For a domain-specific judge
+whose verdict is not a single score, `t.judge.model(prompt)` sends a
+raw prompt to the same model and returns the reply. You then record the
+parsed result with `t.score` or `t.check`.
+
+## Run evals from the CLI
+
+The `eval` command discovers, filters, and runs cases.
+
+Run the CLI under Node 22.13 or newer. Do not use Bun. Its HTTP/2 client
+breaks tool-result streams and causes eval turns to fail.
+
+```bash
+agent-sdk eval --dir . --list                         # discover only
+agent-sdk eval --dir .                                # run all
+agent-sdk eval --dir . builds/checkout                # one datapoint
+agent-sdk eval --dir . builds search                  # several ids or prefixes
+agent-sdk eval --dir . --tag smoke --tag pull-request # any matching tag
+agent-sdk eval --dir . --json --no-stream             # machine-readable results
+agent-sdk eval --dir . --verbose                      # logs + reply snippets
+```
+
+Id filters use OR semantics. Each filter selects an exact id and its
+descendants. For example, `builds` selects `builds`,
+`builds/checkout`, and every other case below that path. Repeated tags
+also use OR semantics. When you provide both ids and tags, a case must
+match both groups.
+
+`eval` boots an ephemeral server on port 0 with a temp state root
+outside the project, so cases don't inherit ambient monorepo rules and
+don't write into the project state directory. Point `--url` at a running server to eval
+a live agent instead:
+
+```bash
+agent-sdk eval --dir . \
+  --url http://127.0.0.1:3000/weather-agent \
+  --bearer-token "$AGENT_TOKEN"
+```
+
+The eval definitions still come from `--dir`; `--url` only changes the
+agent that receives the turns. For a locally mounted multi-agent
+directory, `--slug weather-agent` chooses the target. Use
+`--state-root` to keep ephemeral session state at a chosen path,
+`--timeout-ms` to override the project timeout, and `--no-stream` to
+keep live progress off stderr. A TTY streams turn progress by default.
+`--verbose` still writes `t.log` lines to stderr and adds reply snippets
+to text results.
+
+Model turns need a Cursor credential from `agent-sdk login` or
+`CURSOR_API_KEY`.
+
+See [CLI: eval](/docs/reference/cli.md#eval) for flags and exit codes.
+
+### JSON results
+
+Use `--json --no-stream` in scripts and CI. The top-level result carries
+the totals and one result per case:
+
+```json
+{
+  "ok": true,
+  "passed": 1,
+  "failed": 0,
+  "results": [
+    {
+      "id": "readiness",
+      "ok": true,
+      "assertions": [{ "name": "succeeded", "passed": true }],
+      "sessionId": "ses_123",
+      "inputs": ["Is checkout pull request 42 ready to approve?"],
+      "toolCalls": [{ "toolName": "inspect_pr", "isError": false }],
+      "logs": [],
+      "durationMs": 12340
+    }
+  ]
+}
+```
+
+Each case result can also include `description`, `finalText`, `tools`,
+`error`, and tool arguments or output. This shape lets CI report the
+failed assertion without parsing terminal text.
+
+## Run evals in the playground
+
+Start the server, open the playground, and choose **Evals**. You can run
+every case or one case, watch progress, and open the resulting session
+trace. The Evals tab works on a normal `serve`.
+
+```bash
+agent-sdk serve --dir .
+```
+
+Playground runs target the live server instead of an ephemeral one.
+Their sessions appear in the session list. One eval batch can run at a
+time. Persistence follows the rule under
+[Configure eval runs](#configure-eval-runs). See
+[Playground eval routes](/docs/reference/http-api.md#playground-eval-routes).
+The start request returns `202` while cases run in the background.
+Poll until the snapshot status becomes `completed`, `failed`, or `cancelled`.
+Configuration errors appear on a failed snapshot.
+
+On `--prod` / `--url`, the CLI prints the Eval ID as soon as the batch is
+accepted (and a Playground deep link with `?view=evals&evalRunId=…`):
+
+```bash
+agent-sdk eval --prod --slug vulnerability-scanner --tag deepsec
+# Eval ID: evalrun_…
+# Cancel:  agent-sdk eval cancel evalrun_… --prod --slug vulnerability-scanner
+# Playground: https://…/playground?view=evals&evalRunId=evalrun_…
+
+agent-sdk eval cancel evalrun_… --prod --slug vulnerability-scanner
+agent-sdk eval status evalrun_… --prod --slug vulnerability-scanner
+```
+
+## What good cases assert
+
+Gate decisions and shape, not prose. Model wording varies run to run.
+Tool choice, tool avoidance, and output structure are the stable
+contract.
+
+1. `t.succeeded()`: always, first.
+2. The tool decision: `calledTool` for the intended path,
+   `notCalledTool` for the likely wrong alternative. The pair is
+   stronger than either alone.
+3. Output shape: a regex for the contract (`/ready|blocked/i`, a JSON
+   marker, a findings-block fence), never exact sentences.
+4. For structured output, parse `t.reply` and check fields with
+   `satisfies` instead of substring-matching JSON.
+
+The common failure modes: asserting exact phrasing, packing more than
+about five gates into one case (split it), and cases that depend on live
+external state that drifts (pin the input; see fixtures).
+
+## Pick fixtures by agent type
+
+The right fixture depends on the surface under test.
+
+| Agent surface | Fixture |
+| --- | --- |
+| Chat / domain assistant | A canonical prompt string, chosen once and frozen |
+| Tool-heavy | Run `agent-sdk call <tool>` first to pin what the tool returns, then freeze the prompt that triggers it |
+| GitHub webhook | `agent-sdk github replay <pr> --events '*' --dry-run --out fixtures/github` snapshots real payloads for offline replay ([GitHub guide](/docs/guides/github.md)) |
+| PR reviewer with host preparation | Diff, metadata, and gold labels pinned to commit SHAs; keep any live PR matrix small |
+| Workspace-dependent | `workspaceFiles` in `t.send` options, never developer-machine paths |
+
+Tag the fast, reliably passing core `smoke` and run `--tag smoke` in the
+inner loop. Leave slow or flaky-prone cases untagged for explicit runs.
+
+### Materialize API-backed fixtures
+
+An input that only points at external data, such as a pull request URL,
+snapshot id, or pair of commit SHAs, is not self-contained. Fetch it
+once and commit the rendered fixture before you expand the suite.
+
+1. Save the diff, metadata, and labels under `fixtures/` at pinned
+   revisions.
+2. Seed those files with `workspaceFiles`, or read them from the fixture
+   directory.
+3. Assert decisions and output shape against the saved evidence.
+4. Keep a small `smoke` subset for any remaining live pipeline checks.
+
+Read committed fixtures with `@cursor/july/evals/loaders`: `loadJson`,
+`loadJsonl`, and `loadYaml` resolve relative paths against the project
+root the runner discovered, not the cwd the CLI was invoked from
+(`resolveFixturePath` and `evalFixtureRoot` expose the same
+resolution for other file formats).
+
+`maxConcurrency` limits parallel datapoints. It does not limit model or
+API fan-out inside one datapoint. Materialized fixtures prevent a large
+suite from exhausting provider and GitHub rate limits. The
+[evals skill](https://github.com/cursor/cursor/blob/main/packages/agent-serve/skills/evals/SKILL.md) has the full fixture workflow.
+
+## Keep improvements with regression evals
+
+Every [hillclimb](/docs/hillclimbing.md) round that keeps a change must land
+an eval that would have failed before the change. If you can't express
+the improvement as a gate (a `calledTool` shift, a bounded
+`action.result` count, an output-shape regex), the improvement is
+unverified, and it'll regress silently.
+
+The rule cuts the other way too: never weaken an existing gate to make a
+round pass. That's the freeze line moving, and it turns your regression
+suite into a list of checks that no longer protect anything.
+
+## Compare variants on live traffic
+
+Use `defineAB` to compare variant metrics on live sessions. It is not a
+test runner and has no `agent-sdk ab` command. Keep `defineEval` as the
+regression ratchet. Eval sessions do not enroll or change live metrics.
+See [Live A/B metrics](/docs/ab.md) for assignment, behavior, collection,
+and inspection.
+
+## What's next
+
+Continue with these pages:
+
+- [Live A/B metrics](/docs/ab.md): sticky variants and cumulative metrics
+  on live sessions
+- [Hillclimbing](/docs/hillclimbing.md): the loop evals make trustworthy
+- [Building agents with agents](/docs/building-with-agents.md): have a
+  coding agent write the first suite
+- [GitHub guide](/docs/guides/github.md): deterministic webhook fixtures
+  with `github replay`
+- [Sessions and streaming](/docs/reference/sessions.md): the events
+  `t.events` contains