eve 0.6.0-beta.18 → 0.6.0-beta.20

package/CHANGELOG.md

@@ -1,5 +1,28 @@
 # eve
 
+## 0.6.0-beta.20
+
+### Minor Changes
+
+- 407c6e2: The dev TUI startup header is now a single `▲ eve <agent name>` brand line plus one rotating tip ("Use /channels to add more ways to reach your agent.", "Use /deploy to see your agent go live.", "Type /help to see every command."). The config rows (Model, Instructions, Tools, Skills, Subagents, Server) and the key-hints line are gone. The model moved to the status line, the empty input row shows a dim `Type to chat · / for commands` placeholder, discovery error/warning counts still render, and `eve info` keeps the full configuration detail.
+- 407c6e2: The dev TUI footer now ends with a persistent status line: the agent's model, the session's token flow (`⁕ ↑ 394.4K ↓ 4.3K`, input up, output down, `⁕ ↑ 0 ↓ 0` at startup), the linked Vercel project and team, and a `deploy pending` marker that appears when `/channels` adds a channel and clears when `/deploy` ships it. The Vercel segment stays hidden until the directory is linked. Token usage moved here from the working row, and `--context-size` appends a context-fill percentage.
+- 1d65cd6: Reworked the eval authoring API around a single imperative `test(t)` function. An eval now drives the agent and asserts on what it produced in one place — `async test(t) { await t.send(...); t.completed(); t.calledTool(...); t.check(t.reply, includes(...)); t.judge.autoevals.closedQA(...).atLeast(0.6); }` — replacing the separate `run`/`input`, `checks`, `scores`, `expected`, and `thresholds` fields. Assertions carry their own severity: run-level checks and `eve/evals/expect` value assertions (`includes`/`equals`/`matches`/`similarity`) are hard gates by default, while `t.judge.autoevals.*` and `.soft(...)` assertions are tracked and only fail under `--strict`. LLM-as-judge moved under `t.judge.autoevals.*`, and the judge model is now configured via `judge: { model }` (optional) on `defineEvalConfig`/`defineEval` instead of `model`. The `eve/evals/checks` and `eve/evals/scores` entry points are removed in favor of `t`'s built-in vocabulary and `eve/evals/expect`.
+- a33fa66: The dev TUI now shows dev-server rebuilds as one status row that updates in place: `tui/setup-panel.ts changed · rebuilding…`, then `· rebuilt`. Only the latest rebuild shows, paths shrink to their last two components, and the transcript no longer stacks a full log line per rebuild.
+- a33fa66: The dev TUI now starts with stdout and stderr logs hidden. Use `/loglevel <all|stderr|none>` to change what the transcript shows. Logs stay buffered either way, so `/loglevel all` brings back earlier output in its original order.
+
+### Patch Changes
+
+- 407c6e2: The dev TUI's bottom question panel (setup flows, select/text/acknowledge questions) opens with a stronger `▔` full-width rule and carries a consistent one-space left margin. Gutter glyphs sit at column 1 and text aligns at column 3 instead of touching the terminal edge.
+- e1cd134: `eve eval --url` now authenticates to remote targets with the same Vercel OIDC headers as the dev client, including the trusted OIDC IDP token that bypasses Deployment Protection — so evals can run against protected deployments without configuring a `VERCEL_AUTOMATION_BYPASS_SECRET`.
+- 9a35ea7: Fix `eve init` to launch pnpm-managed projects through `pnpm exec` and generate Web Chat configuration that matches the installed Eve Next.js API.
+- e1cd134: Workflow runs only route to `deploymentId: "latest"` on Vercel production. Preview and CLI deployments carry no git branch reference, so latest resolution failed with HTTP 400 ("Source deployment has no git branch") and every turn errored; they now pin workflow runs to their own immutable deployment.
+
+## 0.6.0-beta.19
+
+### Minor Changes
+
+- 9061941: The dev TUI's `/model` now opens a configure menu uniting the model and provider setup, and the separate `/vercel` command is removed: "Change model" runs the searchable AI Gateway catalog picker, and "Configure provider" (bold yellow with "Required to enable the agent" until a Vercel link or gateway credential is detected) runs the provider questions `/vercel` used to ask. Each action returns to the menu, which shows each row's current value on its own line — e.g. "AI Gateway (Linked to my-project)" — and keeps the latest outcome visible beneath the options ("✓ Model changed to …"); Esc leaves. Error messages and the startup attention line now point at `/model`, and `/model <provider/model-id>` still applies a model directly.
+
 ## 0.6.0-beta.18
 
 ### Minor Changes

package/dist/docs/public/advanced/dev-tui.md

@@ -9,24 +9,20 @@ description: "Drive an Eve agent locally in an interactive terminal UI: chat, st
 eve dev
 ```
 
-On startup the TUI prints a header for the connected agent — the model, instructions prompt, and the tools, skills, and subagents it has available:
+On startup the TUI prints a brand line with your agent's name, plus a rotating tip (local sessions only):
 
 ```text
- ▲ Weather Agent
- · Model         openai/gpt-5
- · Instructions  agent/instructions.md
- · Tools         get_weather, get_forecast, geocode
- · Subagents     researcher
- · Server        http://localhost:3000
-
- Type to chat  ·  ↑ history  ·  /new reset session  ·  /model /vercel /channels /deploy  ·  /exit quit  ·  Ctrl+C interrupt
+ ▲ eve weather-agent
+ Use /channels to add more ways to reach your agent.
 ```
 
-From there the conversation streams straight into your terminal's normal scrollback — your prompts, the agent's replies, reasoning, tool calls, nested subagents, connection-authorization prompts, and any captured `stdout`/`stderr` — so you keep native scrolling, copy/paste, and a transcript that persists after you exit. Each turn is rendered without boxes: a colored gutter glyph marks who's speaking, tool calls collapse to a one-line summary (`✓ get_weather  city="SF" → 73°F`), and a subagent's work is indented beneath its `◆` header. A sticky line at the bottom shows the input prompt or the live status (spinner, token usage). Press `Enter` to send; `Ctrl+C` interrupts a running turn or quits at the prompt. Slash commands: `/new` starts a fresh session and `/exit` quits.
+If agent discovery reported problems, an error/warning count renders between the two lines. Instructions, tools, skills, and subagents are one `eve info` away, and `/help` lists every command.
 
-When `eve dev` runs the server locally, three more slash commands manage the project without leaving the session. `/vercel` opens with two questions — which model provider to use (picking something other than AI Gateway shows wiring instructions for your own provider and stops there) and how to connect to AI Gateway (paste your own `AI_GATEWAY_API_KEY`, saved straight to `.env.local`, or connect via a project) — and the project path then walks the same Vercel team/project pickers as setup (picking again re-links) and pulls the project's environment so an AI Gateway credential lands in `.env.local` — the dev server reloads env files automatically, no restart needed. `/channels` shows the agent's channel list — already-registered channels render as locked rows — and adds the one you pick, including the Slack Connect provisioning, then installs the dependencies the scaffold added so the dev server can load the new channels right away; after each addition the list repaints with the new channel locked, until Done (or Esc) leaves the flow. `/deploy` ships the agent to Vercel production, linking first when the directory is unlinked. Each command echoes as an invocation line, asks through a bordered panel that takes the input area's place — one question at a time, clearly separate from the chat transcript — and finishes with a one-line `⎿` result; loading states stay on the ephemeral status line instead of piling into the transcript. These commands are not available when connected to a remote server with `--url`, and when a turn fails because AI Gateway authentication is missing or stale, the error points you at `/vercel` directly. The TUI also checks at startup: a missing model-provider setup surfaces as an attention line — `⚠ 1 setup issue: model provider not linked · /vercel` — so the fix is visible before the first message fails, and each command's outcome hangs under it with the `⎿` connector. Local sessions also get `/model`: bare `/model` opens the same searchable model picker setup uses (the AI Gateway catalog, pre-selected on the model the runtime is serving), `/model <provider/model-id>` applies one directly, and the change is written into your agent's authored source — the command reports success only after the reloaded runtime confirms the new id.
+From there the conversation streams straight into your terminal's normal scrollback — your prompts, the agent's replies, reasoning, tool calls, nested subagents, connection-authorization prompts, and any captured `stdout`/`stderr` — so you keep native scrolling, copy/paste, and a transcript that persists after you exit. Each turn is rendered without boxes: a colored gutter glyph marks who's speaking, tool calls collapse to a one-line summary (`✓ get_weather  city="SF" → 73°F`), and a subagent's work is indented beneath its `◆` header. A sticky footer keeps you oriented. The input prompt shows a dim `Type to chat · / for commands` placeholder while empty, and beneath it a persistent status line shows the model, the session's token flow (`⁕ ↑ 394.4K ↓ 4.3K`), the linked Vercel project and team (`▲ my-agent (acme)`), and a yellow `deploy pending` marker once a channel added this session still needs `/deploy`. The Vercel segment stays hidden until the directory is linked. Press `Enter` to send; `Ctrl+C` interrupts a running turn or quits at the prompt. Slash commands: `/new` starts a fresh session and `/exit` quits.
 
-The prompt input behaves like a shell line editor: `↑`/`↓` cycle through the messages you've sent this session, `←`/`→`, Home/End, and `Ctrl+A`/`Ctrl+E` move the caret, and `Ctrl+U`/`Ctrl+K`/`Ctrl+W` kill the line, the rest of the line, or the previous word. If a turn fails terminally — the server session dies or the connection drops — the TUI starts a fresh session and notes it inline so you can keep going (server-side context resets with the old session). Errors render compactly, with docs links highlighted, and a code bug escaping your agent's own code shows its stack trace dim beneath the error headline. Captured server `stdout`/`stderr` renders as dim, indented log runs behind a `│` rule — consecutive lines from the same source share one label, and nothing is ever hidden.
+When `eve dev` runs the server locally, three more slash commands manage the project without leaving the session. Bare `/model` opens a two-row configure menu that loops until Esc. "Change model" runs the same searchable model picker setup uses (the AI Gateway catalog, pre-selected on the model the runtime is serving); a model change is written into your agent's authored source, and the command reports success only after Eve confirms the new id (`/model <provider/model-id>` applies one directly, skipping the menu). The provider row opens the provider questions: which model provider to use (picking something other than AI Gateway shows wiring instructions for your own provider and stops there, leaving any existing setup untouched) and how to connect to AI Gateway — paste your own `AI_GATEWAY_API_KEY`, saved straight to `.env.local`, or connect via a project, which walks the same Vercel team/project pickers as setup (picking again re-links) and pulls the project's environment so an AI Gateway credential lands in `.env.local`; the dev server reloads env files automatically, no restart needed. The row demands attention (a bold yellow "Configure provider" with "Required to enable the agent") until a link or gateway credential is detected, then names the connection (e.g. "AI Gateway (Linked to my-project in my-team)") after, and each action's latest outcome stays visible beneath the menu (e.g. "✓ Model changed to openai/gpt-5.5"). `/channels` shows the agent's channel list — already-registered channels render as locked rows — and adds the one you pick, including the Slack Connect provisioning, then installs the dependencies the scaffold added so the dev server can load the new channels right away; after each addition the list repaints with the new channel locked, until Done (or Esc) leaves the flow. `/deploy` ships the agent to Vercel production, linking first when the directory is unlinked. Each command echoes as an invocation line, asks through a bordered panel that takes the input area's place — one question at a time, clearly separate from the chat transcript — and finishes with a one-line `⎿` result; loading states stay on the ephemeral status line instead of piling into the transcript. These commands are not available when connected to a remote server with `--url`, and when a turn fails because AI Gateway authentication is missing or stale, the error points you at `/model` directly. The TUI also checks at startup: a missing model-provider setup surfaces as an attention line — `⚠ 1 setup issue: model provider not linked · /model` — so the fix is visible before the first message fails, and each command's outcome hangs under it with the `⎿` connector.
+
+The prompt input behaves like a shell line editor: `↑`/`↓` cycle through the messages you've sent this session, `←`/`→`, Home/End, and `Ctrl+A`/`Ctrl+E` move the caret, and `Ctrl+U`/`Ctrl+K`/`Ctrl+W` kill the line, the rest of the line, or the previous word. If a turn fails terminally — the server session dies or the connection drops — the TUI starts a fresh session and notes it inline so you can keep going (server-side context resets with the old session). Errors render compactly, with docs links highlighted, and a code bug escaping your agent's own code shows its stack trace dim beneath the error headline. Captured server `stdout`/`stderr` renders as dim, indented log runs behind a `│` rule — consecutive lines from the same source share one label. Dev-server rebuilds condense further, into one status row that updates in place: `tui/setup-panel.ts changed · rebuilding…`, then `· rebuilt`. Only the latest rebuild shows, and paths shrink to their last two components. The TUI hides logs by default. `/loglevel <all|stderr|none>` switches what the transcript shows, and because logs stay buffered either way, the switch is retroactive: `/loglevel all` brings back everything captured so far, in its original order. Bare `/loglevel` reports the current mode; the `--logs` flag sets the starting one.
 
 The agent will sometimes need something from you, and the TUI asks inline. Tool approvals are a `y`/`n`. Option questions let you pick with `↑`/`↓` and `Enter`, or you can type a freeform answer. If a tool needs an authorized [connection](../connections), the URL shows up right in the transcript, and the turn picks back up once you've finished the flow.
 

package/dist/docs/public/advanced/security-model.md

@@ -42,7 +42,7 @@ A [channel](../channels/overview) is your agent's front door, which makes authen
   claims. A body field is attacker-controlled; treating it as identity is
   cross-user impersonation.
 
-The `support-fixture` dashboard channel is a concrete custom-channel fixture for these rules: it authenticates the raw body with an HMAC, compares signatures in constant time, and trusts the body-supplied principal only after the signature verifies.
+A custom channel that accepts dashboard-style webhooks should follow the same shape: authenticate the raw body with an HMAC, compare signatures in constant time, and trust any body-supplied principal only after the signature verifies.
 
 ## Authored markdown is data
 

package/dist/docs/public/evals/assertions.mdx

@@ -0,0 +1,108 @@
+---
+title: "Assertions"
+description: "Run-level methods, t.check value assertions, the matcher mini-language, and gate vs soft severity."
+---
+
+Assertions are how an eval grades what its `test(t)` function produced. Each one **records** a result onto `t` and returns a chainable handle — the runner reads the recorded results to compute the verdict, so a single run reports every failing assertion rather than dying on the first. There are two deterministic surfaces: run-level methods on `t`, and `t.check` for grading a specific value. For model-graded assertions, see [Judge](./judge).
+
+## Run-level assertions
+
+Run-level assertions read the whole run, so they take no value. They are methods on `t` and gate by default.
+
+| Assertion                                           | Asserts                                                                           |
+| --------------------------------------------------- | --------------------------------------------------------------------------------- |
+| `t.completed()`                                     | The run did not fail and did not park on unanswered HITL input                    |
+| `t.didNotFail()`                                    | No terminal failure and no `turn.failed`/`step.failed` events (parked runs pass)  |
+| `t.waiting()`                                       | The run parked on HITL input (for approval-shaped evals)                          |
+| `t.messageIncludes(token)`                          | Joined assistant text contains `token` (string or RegExp)                         |
+| `t.outputEquals(value)` / `t.outputMatches(schema)` | Deep equality / Standard Schema (e.g. Zod) validation of the parsed output        |
+| `t.calledTool(name, opts?)`                         | A matching tool call happened (`input`, `output`, `isError`, `times` constraints) |
+| `t.notCalledTool(name)`                             | No call to `name`                                                                 |
+| `t.toolOrder([...names])`                           | Tool names appear in order (other calls may interleave)                           |
+| `t.usedNoTools()`                                   | No tool calls at all                                                              |
+| `t.maxToolCalls(n)`                                 | At most `n` tool calls                                                            |
+| `t.noFailedActions()`                               | No tool, subagent, or skill action reported a failure                             |
+| `t.calledSubagent(name, opts?)`                     | A subagent delegation happened (`remoteUrl`, `output` constraints)                |
+| `t.event(predicate, label)`                         | Escape hatch: any predicate over the typed event stream                           |
+
+`t.completed()` subsumes `t.didNotFail()` — reach for `completed` unless you specifically want to allow a parked run.
+
+```ts
+await t.send("What is the weather in Brooklyn?");
+t.completed();
+t.calledTool("get_weather");
+t.usedNoTools(); // mutually exclusive with the line above — pick the one you mean
+```
+
+## Value assertions with `t.check`
+
+`t.check(value, assertion)` grades an explicit value against a builder from `eve/evals/expect`. The value can be `t.reply`, a turn's `.message`, parsed JSON, or any local you computed:
+
+```ts
+import { includes, equals, matches, similarity } from "eve/evals/expect";
+
+t.check(t.reply, includes("sunny")); // substring (gate)
+t.check(parsed, equals({ city: "Brooklyn" })); // deep structural equality (gate)
+t.check(parsed, matches(WeatherSchema)); // Standard Schema, e.g. Zod (gate)
+t.check(t.reply, similarity("Sunny, 72F")); // fuzzy 0–1 Levenshtein (soft)
+```
+
+| Builder                | Scores                                           | Default |
+| ---------------------- | ------------------------------------------------ | ------- |
+| `includes(substring)`  | value (coerced to string) contains `substring`   | gate    |
+| `equals(value)`        | deep structural equality                         | gate    |
+| `matches(schema)`      | validates against a Standard Schema              | gate    |
+| `similarity(expected)` | normalized Levenshtein similarity, 1 = identical | soft    |
+
+Pick the cheapest builder that captures what "correct" means. When exact match is too strict but a judge model is overkill, `similarity` is the middle ground; for nuanced grading, reach for the [judge](./judge).
+
+## The matcher mini-language
+
+`t.calledTool` and `t.calledSubagent` take a matcher object — `{ input, output, isError, times }` for tools, `{ remoteUrl, output }` for subagents. Each field accepts a literal (objects partial-deep-match), a RegExp, or a function. A matcher function receives the value and returns either a boolean (acts as a predicate) or an expected value to compare against (handy for runner-assigned values like environment-provided URLs):
+
+```ts
+t.calledTool("bash", { input: { command: /^pwd/ }, isError: false, times: 1 });
+
+t.calledTool("echo", { output: (value) => String(value).includes(marker) });
+
+t.calledSubagent("weather", {
+  remoteUrl: () => process.env.WEATHER_AGENT_URL!,
+  output: /72F/,
+});
+```
+
+## Run state and derived facts
+
+A turn that leaves the session open for a next message is the normal end state of a successful turn. Parking on unanswered HITL input is tracked separately — that is what `t.completed()` and `t.waiting()` key off.
+
+Beyond the raw `t.events` stream, the runner derives typed facts the assertions read: tool calls (name, input, output, error state), subagent calls, and HITL input requests. The built-in assertions cover almost everything; when you need to read the stream directly, `t.event(predicate, label)` is the escape hatch:
+
+```ts
+t.event(
+  (events) =>
+    events.some((e) => e.type === "message.completed" && e.data.message?.includes(marker)),
+  "assistant reply includes the marker",
+);
+```
+
+## Severity
+
+Every assertion returns a chainable handle. Severity rides on the assertion — there is no separate thresholds map to keep in sync.
+
+- `.gate(threshold?)` — hard. A miss marks the eval `failed` and `eve eval` exits non-zero.
+- `.soft(threshold?)` — tracked data. A below-threshold miss marks the eval `scored`, fatal only under `--strict`. With no threshold, it is tracked-only and never fails.
+- `.atLeast(threshold)` — soft with a bar (equivalent to `.soft(threshold)`).
+
+The defaults are chosen so you rarely set severity. Run-level methods and `includes`/`equals`/`matches` are gates; `similarity` and every `t.judge.*` assertion are soft. Annotate only when you deviate:
+
+```ts
+t.calledTool("get_weather").soft(); // record the tool call as a metric, don't gate
+t.check(t.reply, similarity("Sunny")).atLeast(0.8); // gate the fuzzy match under --strict
+t.check(t.reply, includes("error")).soft(); // track without failing the build
+```
+
+## What to read next
+
+- [Judge](./judge): LLM-graded assertions with thresholds
+- [Cases](./cases): where assertions attach
+- [Running evals](./running): how verdicts map to exit codes

package/dist/docs/public/evals/cases.mdx

@@ -1,34 +1,38 @@
 ---
 title: "Cases"
-description: "Author prompt evals, script multi-turn evals with run(ctx), and fan one file out over a dataset."
+description: "Author single-turn and multi-turn evals with test(t), and fan one file out over a dataset."
 ---
 
-Each eval file is one graded case: the runner executes it against the target, captures every event, and applies [checks](./checks) and [scores](./scores) to the result. An eval is either a prompt eval (`input`) or a scripted eval (`run`).
+Each eval file is one graded case. The runner executes its `test(t)` function against the target, captures every event, and computes a verdict from the [assertions](./assertions) you recorded. Every eval — single-turn, multi-turn, HITL, or dataset-driven — is the same shape: one `async test(t)` function that drives the agent and asserts inline.
 
-## Prompt evals
+## Single-turn evals
 
-Prompt evals pair an `input` with an optional `expected`. `input` can be a string or an object (objects are `JSON.stringify`d); the runner sends it as a single user turn. `expected` is optional, which is handy when you only care about behavior:
+The common case sends one turn and asserts on the reply. `t.send(input)` resolves once the turn settles; `t.reply` is the last assistant message:
 
 ```ts title="evals/weather/brooklyn-forecast.eval.ts"
 import { defineEval } from "eve/evals";
-import { Checks } from "eve/evals/checks";
+import { includes } from "eve/evals/expect";
 
 export default defineEval({
-  input: "What is the weather in Brooklyn?",
-  expected: "Sunny",
-  checks: [Checks.didNotFail()],
-  scores: [],
+  async test(t) {
+    await t.send("What is the weather in Brooklyn?");
+    t.completed();
+    t.check(t.reply, includes("Sunny"));
+  },
 });
 ```
 
+Some evals only care about behavior, not text — assert on the run and skip the content check entirely:
+
 ```ts title="evals/weather/no-tools-for-greetings.eval.ts"
 import { defineEval } from "eve/evals";
-import { Checks } from "eve/evals/checks";
 
 export default defineEval({
-  input: "Hello!",
-  checks: [Checks.didNotFail(), Checks.toolNotCalled("get_weather")],
-  scores: [],
+  async test(t) {
+    await t.send("Hello!");
+    t.completed();
+    t.notCalledTool("get_weather");
+  },
 });
 ```
 
@@ -45,70 +49,95 @@ evals/
 └── smoke.eval.ts
 ```
 
-## Datasets: exporting an array
+## Multi-turn evals
 
-To fan one file out over a dataset, default-export an array of `defineEval(...)` values. Eval modules are ESM, so top-level `await` can load anything. Ids derive from the file name plus a zero-padded index (`sql/0000`, `sql/0001`, …, in array order). The loaders (`loadJson`, `loadYaml` from `eve/evals/loaders`) parse fixture files relative to the app root:
+Drive several turns in sequence — branching, HITL approvals, structured output, attachments, multiple sessions. Because assertions live in the function, an intermediate value is just a local variable: judge a draft before the next turn overwrites it, then keep going.
 
-```ts title="evals/sql.eval.ts"
+```ts title="evals/draft-then-send.eval.ts"
 import { defineEval } from "eve/evals";
-import { loadYaml } from "eve/evals/loaders";
-import { Text } from "eve/evals/scores";
+import { includes } from "eve/evals/expect";
 
-const doc = await loadYaml("evals/data/cases.yaml");
-const rows = doc.evals as readonly { task: string; prompt: string; sql: string }[];
+export default defineEval({
+  async test(t) {
+    const draft = await t.send("Draft the follow-up email.");
+    t.check(draft.message, includes("Best regards"));
+    t.judge.autoevals.closedQA("professional tone", { on: draft.message }).atLeast(0.6);
 
-export default rows.map((row) =>
-  defineEval({
-    description: row.task,
-    input: row.prompt,
-    expected: row.sql,
-    scores: [Text.exact()],
-  }),
-);
+    await t.send("Now send it.");
+    t.calledTool("send_email");
+  },
+});
 ```
 
-The loaders are meant for fixtures, not runtime agent code.
-
-## Scripted evals
+Bespoke preconditions that no built-in assertion expresses are plain `throw`s — a thrown error marks the eval `failed` with the message in the result:
 
-Scripted evals define `run(ctx)` instead of `input`. Use them for smoke-style behavior: multi-turn branching, HITL approvals, structured output, attachments, and multiple independent sessions.
-
-```ts title="evals/approve-tool.eval.ts"
+```ts title="evals/session-continuity.eval.ts"
 import { defineEval } from "eve/evals";
-import { Checks } from "eve/evals/checks";
+import { includes } from "eve/evals/expect";
 
 export default defineEval({
-  async run({ session }) {
-    await session.send("run `pwd`");
-    session.expectInputRequests({ toolName: "bash" });
-    const turn = await session.respondAll("approve");
-    return turn.message;
+  requires: ["mockModels"],
+  async test(t) {
+    await t.send("My favorite word is marigold.");
+    const firstSessionId = t.sessionId;
+
+    const second = await t.send("Thanks for remembering.");
+    second.expectOk();
+    if (t.sessionId !== firstSessionId) {
+      throw new Error(`Expected one session; got ${firstSessionId} then ${t.sessionId}.`);
+    }
+
+    t.completed();
+    t.check(second.message, includes("Thanks for remembering."));
   },
-  checks: [Checks.didNotFail(), Checks.toolCalled("bash", { input: { command: /pwd/ } })],
-  scores: [],
 });
 ```
 
-The return value of `run` becomes the output that scorers grade (set `parseOutput` to transform the raw result instead). Throwing marks the eval failed with the error message in the result.
-
-## The session API
+## The drive API
 
-`ctx.session` is the primary `EveEvalSession`; `ctx.newSession()` creates another independent session against the same target:
+`t` drives the primary session; `t.newSession()` returns an independent `EveEvalSession` against the same target, whose events feed the same run-level assertions.
 
-- `session.send(input)` sends a turn and waits for it to settle. It accepts the same input as `ClientSession.send()` — a string or a structured message.
-- `session.sendFile(text, path, mediaType?)` attaches a local file as a data URL.
-- `session.expectInputRequests(filter?)` asserts the previous turn parked on HITL input and returns the pending requests.
-- `session.respondAll(optionId)` answers every pending input request with the same option and sends the responses as the next turn.
-- `session.events` is the full typed event stream captured so far.
+- `t.send(input)` sends a turn and waits for it to settle. It accepts the same input as `ClientSession.send()` — a string or a structured message — and resolves to a turn carrying `.message` and `.expectOk()`.
+- `t.sendFile(text, path, mediaType?)` attaches a local file as a data URL.
+- `t.expectInputRequests(filter?)` asserts the previous turn parked on HITL input and returns the pending requests.
+- `t.respond(...responses)` answers specific pending input requests and sends them as the next turn.
+- `t.respondAll(optionId)` answers every pending input request with the same option and sends the responses as the next turn.
+- `t.reply` is the last assistant message (or `null`); `t.sessionId` is the current session id; `t.events` is the full typed event stream captured so far.
 
-Each `send` resolves to an `EveEvalTurn` carrying the turn's `message`, `events`, and status. `turn.expectOk()` throws only when the turn ended failed — a session left open for a next message is the normal end state of a successful turn.
+Each `send` (and `respond`/`respondAll`) resolves to a turn whose `expectOk()` throws only when the turn ended failed — a session left open for a next message is the normal end state of a successful turn.
 
-Events from every eval session are captured in the result and artifacts. `ctx.log(message)` records debug lines into the eval artifact; `--verbose` also streams them to stdout as evals run.
+Events from every session are captured in the result and artifacts. `t.log(message)` records debug lines into the eval artifact; `--verbose` also streams them to stdout as evals run. `t.signal` is an `AbortSignal` that fires on timeout.
 
 For driving sessions created outside the eval — by a channel webhook or a schedule — see [Targets and requirements](./targets).
 
+## Datasets: exporting an array
+
+To fan one file out over a dataset, default-export an array of `defineEval(...)` values. Eval modules are ESM, so top-level `await` can load anything. Ids derive from the file name plus a zero-padded index (`sql/0000`, `sql/0001`, …, in array order). The loaders (`loadJson`, `loadYaml` from `eve/evals/loaders`) parse fixture files relative to the app root:
+
+```ts title="evals/sql.eval.ts"
+import { defineEval } from "eve/evals";
+import { loadYaml } from "eve/evals/loaders";
+import { equals } from "eve/evals/expect";
+
+const doc = await loadYaml("evals/data/cases.yaml");
+const rows = doc.evals as readonly { task: string; prompt: string; sql: string }[];
+
+export default rows.map((row) =>
+  defineEval({
+    description: row.task,
+    async test(t) {
+      await t.send(row.prompt);
+      t.completed();
+      t.check(t.reply, equals(row.sql));
+    },
+  }),
+);
+```
+
+The loaders are meant for fixtures, not runtime agent code.
+
 ## What to read next
 
-- [Checks](./checks): assert on what the eval did
-- [Scores](./scores): grade how well it did it
+- [Assertions](./assertions): assert on what the eval did
+- [Judge](./judge): grade quality with an LLM judge
 - [TypeScript client](../client/messages): the send/turn protocol eval sessions build on

package/dist/docs/public/evals/judge.mdx

@@ -0,0 +1,94 @@
+---
+title: "Judge"
+description: "Grade evals with an LLM judge via t.judge.autoevals, set thresholds on the assertion, and configure the judge model."
+---
+
+When no deterministic [assertion](./assertions) captures what "good" means — factual correctness, summary quality, free-form criteria — grade the run with an LLM judge. The `t.judge.*` assertions are the only model-backed ones, and they use a judge model that is resolved separately from the agent under test: Eve only uses it for scoring, never to swap out the agent.
+
+```ts
+import { defineEval } from "eve/evals";
+
+export default defineEval({
+  async test(t) {
+    await t.send("Explain quantum tunneling to a 10-year-old.");
+    t.completed();
+    t.judge.autoevals.closedQA("uses no math beyond arithmetic").atLeast(0.8);
+  },
+});
+```
+
+## The graders
+
+The judges live under `t.judge.autoevals` — the namespace names the [Braintrust autoevals](https://github.com/braintrustdata/autoevals) grader family, so the factuality and closedQA semantics are autoevals', not Eve-invented. Each grades `t.reply` by default and is soft by default (tracked, no gate):
+
+| Grader                                   | Grades                                                                                 |
+| ---------------------------------------- | -------------------------------------------------------------------------------------- |
+| `t.judge.autoevals.factuality(expected)` | Factual consistency of the reply against an expected answer (A–E buckets)              |
+| `t.judge.autoevals.summarizes(expected)` | How well the reply summarizes the expected text                                        |
+| `t.judge.autoevals.closedQA(criteria)`   | Whether the reply satisfies a free-form yes/no criterion (no expected answer to match) |
+| `t.judge.autoevals.sql(expected)`        | Semantic equivalence of two SQL statements                                             |
+
+The reference or criteria is the positional argument. An options object follows:
+
+- `on` — the value to grade, defaulting to `t.reply`. Pass an intermediate draft or parsed value to grade it instead.
+- `model` / `modelOptions` — a per-call judge override (see below).
+
+```ts
+const draft = await t.send("Draft the welcome email.");
+t.judge.autoevals.closedQA("professional tone", { on: draft.message }).atLeast(0.6);
+```
+
+## Soft scoring and thresholds
+
+Judge assertions are soft, so the threshold rides on the assertion handle — there is no separate thresholds map:
+
+- **No threshold** — tracked-only. The score lands in reports and artifacts and never fails the eval. Use it to watch a metric without gating on it.
+- `.atLeast(threshold)` — a soft bar. A below-threshold score marks the eval `scored`, fatal only under `eve eval --strict`.
+- `.gate(threshold)` — promote a judge to a hard gate that fails the eval outright.
+
+```ts
+t.judge.autoevals.closedQA("cites a source"); // tracked, never fails
+t.judge.autoevals.closedQA("cites a source").atLeast(0.6); // soft, fails under --strict below 0.6
+t.judge.autoevals.factuality(reference).gate(0.8); // hard gate at 0.8
+```
+
+A judge runs once per assertion and burns tokens, so reach for one only when nothing deterministic will do. Several slow judge calls in one eval can fan out with `await Promise.all([...])`.
+
+## Configuring the judge model
+
+The judge model is resolved once when the runner builds `t`. It is **never** the model under test. Three levels resolve innermost-wins:
+
+1. **Per-call** — `t.judge.autoevals.closedQA("…", { model, modelOptions })`.
+2. **Per-eval** — `defineEval({ judge: { model, modelOptions }, test })`.
+3. **Project default** — `defineEvalConfig({ judge: { model, modelOptions } })` in `evals.config.ts`.
+
+```ts title="evals/evals.config.ts"
+import { defineEvalConfig } from "eve/evals";
+
+export default defineEvalConfig({
+  judge: { model: "openai/gpt-5.4-mini" }, // the default judge for every eval in this tree
+});
+```
+
+```ts title="evals/quantum.eval.ts"
+import { defineEval } from "eve/evals";
+
+export default defineEval({
+  judge: { model: "anthropic/claude-opus-4.8" }, // a stronger judge for this eval
+  async test(t) {
+    await t.send("Explain quantum tunneling to a 10-year-old.");
+    t.judge.autoevals.factuality(reference).atLeast(0.7);
+    t.judge.autoevals.closedQA("is concise", { model: "anthropic/claude-haiku-4-5" }); // cheaper, per-call
+  },
+});
+```
+
+`judge` in `evals.config.ts` is optional — a tree of fully deterministic evals can omit it. But calling `t.judge.*` with no judge model resolved is a fail-fast error at eval definition time.
+
+A **string model id** (e.g. `"anthropic/claude-opus-4.8"`) routes through the Vercel AI Gateway and needs `AI_GATEWAY_API_KEY` or `VERCEL_OIDC_TOKEN` in the environment; an **AI SDK `LanguageModel` instance** is used directly. With a model configured but no credentials, a judge-backed eval **skips visibly** like other real-model legs, so mock-model fixture runs stay green. For provider-specific judge settings, use `modelOptions.providerOptions`.
+
+## What to read next
+
+- [Assertions](./assertions): deterministic run-level and value assertions
+- [Reporters](./reporters): ship judged scores to Braintrust experiments
+- [Targets and requirements](./targets): gating judge-backed evals on credentials

package/dist/docs/public/evals/meta.json

@@ -1,4 +1,4 @@
 {
   "title": "Evals",
-  "pages": ["overview", "cases", "checks", "scores", "targets", "reporters", "running"]
+  "pages": ["overview", "cases", "assertions", "judge", "targets", "reporters", "running"]
 }