@cursor/july 0.1.109 → 0.1.112

package/docs/reference/evals.md

@@ -0,0 +1,298 @@
+---
+title: "Evals"
+description: "Reference for eval discovery, cases, assertions, judges, configuration, fixtures, reporters, and runner output."
+---
+
+# Evals reference
+
+This page is the complete authoring and runner contract for
+`@cursor/july/evals`. Start with the [Evals guide](../evals.md) for the
+workflow and first regression case.
+
+## Discovery and case IDs
+
+Eval files live under the project-root `evals/` directory and end in
+`.eval.ts` or `.eval.js`. The path under that directory becomes the
+eval ID:
+
+- `evals/readiness.eval.ts` becomes `readiness`.
+- `evals/builds/api.eval.ts` becomes `builds/api`.
+- `evals/builds/index.eval.ts` becomes `builds`.
+
+Export one `defineEval` with either `test` or `cases`, not both. A case
+ID appends one path segment to the file ID:
+
+```ts
+export default defineEval({
+  tags: ["smoke"],
+  cases: [
+    {
+      id: "checkout",
+      async test(t) {
+        await t.send("Review checkout PR 42.");
+        t.succeeded();
+      },
+    },
+    {
+      id: "search",
+      async test(t) {
+        await t.send("Review search PR 7.");
+        t.succeeded();
+      },
+    },
+  ],
+});
+```
+
+Those cases are `prs/checkout` and `prs/search` when the file is
+`evals/prs.eval.ts`. Case IDs must be unique single path segments.
+
+A file can also export an array of `defineEval` calls. The runner names
+them with zero-padded indexes such as `sql/0000`. Use named `cases` for
+handwritten scenarios and arrays for loaded datasets.
+
+`iterations` repeats one datapoint from 1 to 100 times. For
+`iterations: 3`, `weather/nyc` expands to `weather/nyc/1`,
+`weather/nyc/2`, and `weather/nyc/3`; selecting `weather/nyc` runs all
+three.
+
+## Configuration
+
+Every running suite needs `evals/evals.config.ts` with
+`maxConcurrency`:
+
+```ts
+import { defineEvalConfig } from "@cursor/july/evals";
+
+export default defineEvalConfig({
+  maxConcurrency: 20,
+  timeoutMs: 180_000,
+  judge: { model: "gpt-5.4-mini" },
+});
+```
+
+| Option | Contract |
+| --- | --- |
+| `maxConcurrency` | Required; concurrent authored datapoints, from 1 to 200 |
+| `timeoutMs` | Per-case timeout; case/file, CLI, then config precedence |
+| `judge` | Default model for `t.judge` |
+| `reporters` | Objects notified as cases and runs complete |
+| `maxPlaygroundRuns` | Number of server-side batches kept in playground history |
+
+A case can override `description`, `tags`, `timeoutMs`, `iterations`,
+`judge`, `reporters`, and `metadata`. Case metadata merges over
+file-level metadata; case reporters add to the file's reporters.
+
+## Drive turns with `t.send`
+
+`await t.send(message, options?)` runs one turn and waits until it
+finishes, fails, or parks for approval. Several sends in one test share
+the session.
+
+The returned turn exposes its assistant `message`, `sessionId`,
+`events`, ordered `toolCalls`, `ok`, and turn `index`. Assertions on the
+turn inspect only that turn; assertions on `t` inspect the whole run.
+Useful run values include `t.reply`, `t.events`, `t.turns`,
+`t.sessionId`, and the timeout `t.signal`.
+
+These options apply on the first send because they shape the session:
+
+| Option | Contract |
+| --- | --- |
+| `workspaceFiles` | Relative path-to-content map seeded into the session workspace |
+| `workspaceDir` | Absolute local harness working directory |
+| `cloud` | Cloud session options merged over the agent's static cloud config |
+
+Use `turn.expectOk()` when later test steps depend on that turn
+succeeding.
+
+## Trajectory assertions
+
+Assertions record failures and let the test continue, so one case
+reports every violated contract.
+
+| Assertion | Checks |
+| --- | --- |
+| `t.succeeded()` | Run finished without failure or an unanswered approval |
+| `t.parked()` | Run stopped on an unanswered approval |
+| `t.calledTool(name, matcher?)` | Matching tool request occurred |
+| `t.notCalledTool(name)` | No request for that tool occurred |
+| `t.loadedSkill(name)` | Agent opened a named skill |
+| `t.toolOrder(names)` | Tool requests appeared in relative order |
+| `t.usedNoTools()` | No tool was requested |
+| `t.maxToolCalls(max)` | Tool-call count stayed under the bound |
+| `t.noFailedActions()` | No tool result failed |
+| `t.calledSubagent(name, matcher?)` | Matching subagent call occurred |
+| `t.taggedArtifact(kind?, predicate?)` | Matching durable artifact was tagged |
+| `t.event(type, matcher?)` | Matching session event occurred |
+| `t.notEvent(type, matcher?)` | Matching session event did not occur |
+| `t.eventOrder(matchers)` | Event groups appeared in relative order |
+| `t.eventsSatisfy(label, predicate)` | Predicate accepted the event stream |
+| `t.check(value, expectation)` | Value met an expectation builder |
+| `t.score(name, value)` | Recorded a numeric score from 0 to 1 |
+
+`t.requireToolCall`, `t.requireInputRequest`, and `t.require` return the
+matched value and stop the remaining test body when no match exists.
+
+## Tool and event matchers
+
+Tool matchers can narrow by `input`, `output`, `status`, and `count`.
+Values accept literals, regular expressions, or predicates. Object
+literals partial-deep-match:
+
+```ts
+t.calledTool("inspect_pr", {
+  input: { verdict: "review" },
+  status: "completed",
+  count: 1,
+});
+```
+
+Tool status is `completed`, `failed`, `pending`, or `rejected`.
+Subagent matchers accept `output`, `status`, `count`, and `callId`.
+Event matchers accept `data` and `count`.
+
+## Expectation builders
+
+Import builders from `@cursor/july/evals`:
+
+| Builder | Default severity |
+| --- | --- |
+| `includes(string | RegExp)` | Hard gate |
+| `equals(value)` | Hard gate |
+| `matches(schema)` | Hard gate |
+| `satisfies(predicate, label)` | Hard gate |
+| `similarity(expected)` | Soft score |
+
+`matches` accepts Standard Schema implementations such as Zod.
+`normalizedSimilarity(value, expected)` returns the same 0-to-1 value
+used by `similarity`.
+
+## Severity and verdicts
+
+Every assertion returns a severity handle:
+
+```ts
+t.succeeded();
+t.calledTool("search").soft();
+t.check(t.reply, similarity("Expected answer")).atLeast(0.8);
+t.judge.closedQA("names the failing step").gate(1);
+```
+
+- `.gate(threshold?)` fails the case when it misses.
+- `.soft(threshold?)` records a value without failing.
+- `.atLeast(threshold)` records a soft bar; a miss produces a `scored`
+  verdict.
+
+Case verdicts are `passed`, `failed`, `scored`, or `skipped`. The CLI
+exits 1 for a failed gate. A scored result exits 0 unless `--strict` is
+set. Exit 2 means the selection matched no cases.
+
+## Judges
+
+Use a judge for meaning that deterministic checks cannot express:
+
+```ts
+t.judge.factuality(expected).atLeast(0.8);
+t.judge.summarizes(expected).atLeast(0.8);
+t.judge.closedQA("names the root cause").gate(1);
+t.judge.sql(expected).atLeast(0.9);
+```
+
+Each grades `t.reply` by default; pass `{ on }` to grade another value.
+Judge configuration can live in the project config, eval, case, or
+call, with the nearest value winning.
+
+`t.judge.model(prompt)` returns a raw model reply for custom grading.
+Treat agent output as untrusted input: wrap it with `fenceUntrusted` and
+include `EVAL_JUDGE_INJECTION_GUARD` in the judge prompt.
+
+## Datasets and fixtures
+
+Load committed JSON, JSONL, or YAML relative to the project root:
+
+```ts
+import { defineEval, includes } from "@cursor/july/evals";
+import { loadYaml } from "@cursor/july/evals/loaders";
+
+const rows = await loadYaml<
+  Array<{ task: string; prompt: string; expected: string }>
+>("evals/data/cases.yaml");
+
+export default rows.map(row =>
+  defineEval({
+    description: row.task,
+    async test(t) {
+      await t.send(row.prompt);
+      t.succeeded();
+      t.check(t.reply, includes(row.expected));
+    },
+  })
+);
+```
+
+Materialize API-backed evidence before running a large suite. Commit
+the exact payload, diff, or metadata revision and seed it with
+`workspaceFiles`.
+
+## Reporters and results
+
+Built-in reporters include `JUnit({ filePath, suiteName? })` and
+`Artifacts({ dir })`. Custom reporters can expose `onRunStart`,
+`onEvalComplete`, and `onRunComplete`. Reporter errors are logged and
+do not change the eval verdict.
+
+JSON output contains run totals plus one result per case. A result can
+include the case ID, verdict, assertions, session ID, inputs, tool
+calls, metrics, logs, duration, metadata, tags, final text, and error or
+skip details.
+
+## CLI selection and artifacts
+
+```bash
+agent-sdk eval --dir . --list
+agent-sdk eval --dir .
+agent-sdk eval --dir . builds/checkout
+agent-sdk eval --dir . --tag smoke
+agent-sdk eval --dir . --verbose
+```
+
+ID filters match an exact ID and its descendants. Repeated IDs use OR;
+repeated tags use OR. When both are present, a case must match both
+groups.
+
+The local runner starts an ephemeral server. Default artifacts go under
+`evals/<stamp>/` in the project state directory and include
+`summary.json`, `results.jsonl`, and `evals/<case-id>.json`.
+`--artifacts <dir>` chooses another destination; `--no-artifacts`
+disables them. This location is independent of `--state-root`.
+
+Model and judge turns resolve credentials in this order:
+`CURSOR_API_KEY`, `CURSOR_API_KEY_FILE`,
+`CURSOR_SERVICE_ACCOUNT_KEY`, then `agent-sdk login`. Listing cases
+needs no credential.
+
+## Playground and hosted runs
+
+The playground **Evals** view starts batches on its running server and
+keeps their sessions in the normal session list. `--url` targets a
+named running server; `--prod --slug <slug>` targets a hosted
+deployment.
+
+```bash
+agent-sdk eval --prod --slug vulnerability-scanner --tag smoke
+agent-sdk eval status <eval-id> --prod --slug vulnerability-scanner
+agent-sdk eval cancel <eval-id> --prod --slug vulnerability-scanner
+```
+
+Server-side batches use the target server's discovered evals and store
+results in its playground history. Local-only reporter and concurrency
+flags do not apply to `--url` or `--prod` runs.
+
+## Related
+
+- [Evals guide](../evals.md): author and run a regression workflow
+- [CLI reference](./cli.md#eval): every runner flag
+- [Sessions](./sessions.md): event vocabulary used by trajectory checks
+- [Artifacts](./artifacts.md): durable outputs asserted by
+  `taggedArtifact`