eve 0.10.0 → 0.11.1

package/docs/connections.mdx

@@ -1,15 +1,15 @@
 ---
 title: "Connections"
-description: "Expose external MCP and OpenAPI servers to the model: with connection tokens the model never sees."
+description: "Expose external MCP and OpenAPI servers to the model, with connection tokens the model never sees."
 ---
 
-A connection wires an agent into an external server you don't author: an MCP server (Linear, GitHub, a warehouse) or any HTTP API with an OpenAPI document. Eve handles the parts you'd otherwise hand-roll: it discovers the remote tools, surfaces them to the model, and brokers auth.
+A connection wires an agent into an external server you don't author, either an MCP server (Linear, GitHub, a warehouse) or any HTTP API with an OpenAPI document. Eve handles the parts you'd otherwise hand-roll, discovering the remote tools, surfacing them to the model, and brokering auth.
 
 Connections live under `agent/connections/`. The runtime name comes from the filename, so `agent/connections/linear.ts` registers as `"linear"`. The model never sees a connection's URL or credentials. It discovers tools through the built-in `connection__search` and calls them by their qualified name, `connection__<connection>__<tool>` (e.g. `connection__linear__list_issues`).
 
 ## MCP connections
 
-`defineMcpClientConnection` points at an MCP server. All you have to supply is a `url` and a `description`:
+`defineMcpClientConnection` points at an MCP server. Supply a `url` and a `description`:
 
 ```ts title="agent/connections/linear.ts"
 import { defineMcpClientConnection } from "eve/connections";
@@ -23,11 +23,11 @@ export default defineMcpClientConnection({
 });
 ```
 
-The `url` must speak Streamable HTTP or SSE. Write the `description` for the model, not for yourself. It shows up in `connection__search`, and the model leans on it to decide which connection to query.
+The `url` must speak Streamable HTTP or SSE. Write the `description` for the model, not for yourself. It shows up in `connection__search`, and the model uses it to decide which connection to query.
 
 ### Static-token auth
 
-`getToken` returns a `TokenResult` (`{ token, expiresAt? }`), and Eve sends it as `Authorization: Bearer <token>` on every request. Because it runs on each connection attempt, you can mint a fresh token from wherever you keep secrets: an env var, a secrets manager, an internal vault, or your own OAuth exchange. If the token has a known TTL, set `expiresAt` (milliseconds since epoch) and Eve refreshes ahead of time rather than waiting for a `401`.
+`getToken` returns a `TokenResult` (`{ token, expiresAt? }`), and Eve sends it as `Authorization: Bearer <token>` on every request. Because it runs on each connection attempt, you can mint a fresh token from wherever you keep secrets, including an env var, a secrets manager, an internal vault, or your own OAuth exchange. If the token has a known TTL, set `expiresAt` (milliseconds since epoch) and Eve refreshes ahead of time rather than waiting for a `401`.
 
 When `getToken` is the only auth, `principalType` defaults to `"app"`: one shared credential keyed across all sessions. Switch to `principalType: "user"` when each end-user carries their own token.
 
@@ -35,7 +35,7 @@ Eve resolves and caches connection tokens per step; they never land in conversat
 
 ### No auth
 
-Drop `auth` entirely for servers that need no token, like a localhost server during development or a public one:
+Drop `auth` entirely for servers that need no token, such as a localhost server during development or a public one:
 
 ```ts
 export default defineMcpClientConnection({
@@ -44,9 +44,11 @@ export default defineMcpClientConnection({
 });
 ```
 
+We recommend using no-auth connections only for services that are intentionally public, local-only, or otherwise protected outside Eve. Do not use no-auth connections for sensitive third-party services.
+
 ### Headers
 
-Reach for `headers` when the server wants a non-Bearer scheme (an API-key header) or some extra configuration. Headers stack on top of `auth`:
+Use `headers` when the server wants a non-Bearer scheme (an API-key header) or extra configuration. Headers stack on top of `auth`:
 
 ```ts
 export default defineMcpClientConnection({
@@ -58,7 +60,7 @@ export default defineMcpClientConnection({
 
 ### Tool filters
 
-To narrow which remote tools the model sees, set exactly one of `tools.allow` or `tools.block`. Filtered-out tools just don't appear in `connection__search`:
+To narrow which remote tools the model sees, set exactly one of `tools.allow` or `tools.block`. Filtered-out tools do not appear in `connection__search`:
 
 ```ts
 export default defineMcpClientConnection({
@@ -84,11 +86,13 @@ export default defineMcpClientConnection({
 });
 ```
 
-`never()` lets every call through, `once()` asks for approval the first time in a session, and `always()` asks every time. The pause and resume here is the same human-in-the-loop flow covered in [Tools](./tools).
+`never()` lets every call through, `once()` asks for approval the first time in a session, and `always()` asks every time. The pause and resume is the same human-in-the-loop flow covered in [Tools](./tools).
+
+For connection tools that can create, modify, delete, transmit, purchase, message, or access sensitive data, use approval, tool allow-lists, or other safeguards appropriate to the action.
 
 ## OpenAPI connections
 
-`defineOpenAPIConnection` turns any OpenAPI 3.x or Swagger 2.0 document into connection tools, one per operation. Pass an HTTPS URL Eve fetches at runtime, or an inline parsed object:
+`defineOpenAPIConnection` turns any OpenAPI 3.x document into connection tools, one per operation. Pass an HTTPS URL Eve fetches at runtime, or an inline parsed object:
 
 ```ts title="agent/connections/petstore.ts"
 import { defineOpenAPIConnection } from "eve/connections";
@@ -104,10 +108,10 @@ Each operation becomes `connection__<connection>__<operationId>` (e.g. `connecti
 
 `auth`, `headers`, and `approval` work exactly as they do for MCP. There are two fields specific to OpenAPI:
 
-| Field        | Purpose                                                                                                                                             |
-| ------------ | --------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `baseUrl`    | Base URL operation paths resolve against. Optional; defaults to the document's first usable `servers` entry or Swagger `schemes`/`host`/`basePath`. |
-| `operations` | Filter keyed on `operationId` (`allow` or `block`). Mirrors `tools` on MCP connections, but names operations not tools.                             |
+| Field        | Purpose                                                                                                                 |
+| ------------ | ----------------------------------------------------------------------------------------------------------------------- |
+| `baseUrl`    | Base URL operation paths resolve against. Optional; defaults to the document's first usable `servers` entry.            |
+| `operations` | Filter keyed on `operationId` (`allow` or `block`). Mirrors `tools` on MCP connections, but names operations not tools. |
 
 ## Interactive OAuth via Vercel Connect
 
@@ -128,7 +132,7 @@ export default defineMcpClientConnection({
 
 ## Self-hosted interactive OAuth
 
-Running your own OAuth? Use `defineInteractiveAuthorization` from `eve/connections`, which takes a three-method form. Eve mints a callback URL, parks the turn on a framework-owned webhook, and resumes once the token comes back. No Vercel Connect required. Interactive auth is always `principalType: "user"`, and the factory pins that for you.
+To run your own OAuth, use `defineInteractiveAuthorization` from `eve/connections`, which takes a three-method form and needs no Vercel Connect. Eve mints a callback URL, parks (durably suspends) the turn on a framework-owned webhook, and resumes once the token comes back. Interactive auth is always `principalType: "user"`, and the factory pins that for you.
 
 ```ts title="agent/connections/linear.ts"
 import {
@@ -168,9 +172,18 @@ export default defineMcpClientConnection({
 });
 ```
 
-`getToken` runs before every tool call. `startAuthorization` and `completeAuthorization` are both-or-neither: provide one without the other and you get a definition error. The `challenge` rides along verbatim on the `authorization.required` event. Set `url` for redirect/device flows, `userCode` for a device code, `instructions` as the call to action when there's no URL, and `displayName` for the human-readable provider name channels show on the sign-in affordance (e.g. "Salesforce"). Drop `resume` when the provider keeps flow state server-side, so nothing has to cross the step boundary.
+`getToken` runs before every tool call. `startAuthorization` and `completeAuthorization` are both-or-neither: provide one without the other and you get a definition error. The `challenge` rides along verbatim on the `authorization.required` event. Its fields:
+
+| Field          | Purpose                                                                                   |
+| -------------- | ----------------------------------------------------------------------------------------- |
+| `url`          | The authorize URL for redirect or device flows.                                           |
+| `userCode`     | The device code, for device flows.                                                        |
+| `instructions` | The call to action when there's no URL.                                                   |
+| `displayName`  | Human-readable provider name channels show on the sign-in affordance (e.g. "Salesforce"). |
 
-`displayName` is presentation-only — the connection's path-derived name still keys the authorization scope, token cache, and callback URL. You can also set `displayName` on the `auth` definition itself (e.g. `auth: { ...connect("sfdc"), displayName: "Salesforce" }`); that definition-level value wins over one the strategy stamps on the challenge, and channels fall back to title-casing the connection name when neither is set.
+Drop `resume` when the provider keeps flow state server-side, so nothing has to cross the step boundary.
+
+`displayName` is presentation-only. The connection's path-derived name still keys the authorization scope, token cache, and callback URL. You can also set `displayName` on the `auth` definition itself (e.g. `auth: { ...connect("sfdc"), displayName: "Salesforce" }`); that definition-level value wins over one the strategy stamps on the challenge, and channels fall back to title-casing the connection name when neither is set.
 
 ### Signaling authorization state
 
@@ -192,24 +205,31 @@ To narrow a caught error, use `isConnectionAuthorizationRequiredError(err)` and
 
 ### Handling a revoked token mid-call
 
-`getToken` only runs _before_ a tool call, so a grant revoked while a tool is mid-flight first surfaces as a downstream `401` inside your `execute`. A plain throw there is just a tool error — the model sees a failure and the cached bearer sticks around. Instead, map a provider `401` to `ctx.requireAuth()` (or rethrow `ConnectionAuthorizationRequiredError`). Eve then evicts the rejected token from its per-step cache and re-runs the consent flow with a fresh one, exactly as it does for a connection whose server rejects the bearer.
+`getToken` only runs _before_ a tool call, so a grant revoked while a tool is mid-flight first surfaces as a downstream `401` inside your `execute`. A plain throw there is only a tool error, so the model sees a failure and the cached bearer sticks around. Instead, map a provider `401` to `ctx.requireAuth()` (or rethrow `ConnectionAuthorizationRequiredError`). Eve then evicts the rejected token from its per-step cache and re-runs the consent flow with a fresh one, exactly as it does for a connection whose server rejects the bearer.
 
 ```ts title="agent/tools/list_issues.ts"
-async execute(_input, ctx) {
-  const { token } = await ctx.getToken();
-  const res = await fetch("https://api.linear.app/graphql", {
-    headers: { authorization: `Bearer ${token}` },
-  });
-  // The grant was revoked since getToken ran: re-challenge instead of
-  // returning a dead-token error to the model.
-  if (res.status === 401) ctx.requireAuth();
-  return await res.json();
-}
+import { defineTool } from "eve/tools";
+import { z } from "zod";
+
+export default defineTool({
+  description: "List open Linear issues.",
+  inputSchema: z.object({}),
+  async execute(_input, ctx) {
+    const { token } = await ctx.getToken();
+    const res = await fetch("https://api.linear.app/graphql", {
+      headers: { authorization: `Bearer ${token}` },
+    });
+    // The grant was revoked since getToken ran: re-challenge instead of
+    // returning a dead-token error to the model.
+    if (res.status === 401) ctx.requireAuth();
+    return await res.json();
+  },
+});
 ```
 
 ### Authorization and approval together
 
-A tool can require both sign-in (`auth`) and a human approval. Today the model's approval gate runs before the tool's `execute`, so the order the user sees is **approve, then sign in**. Eve records the approval on session state the moment it's granted, and that record survives the sign-in park — so when the turn resumes after authorization, the tool is not put through approval a second time. You get one approval and one sign-in, never a double prompt.
+A tool can require both sign-in (`auth`) and a human approval. The model's approval gate runs before the tool's `execute`, so the order the user sees is **approve, then sign in**. Eve records the approval on session state the moment it's granted, and that record survives the sign-in park, so when the turn resumes after authorization the tool is not put through approval again. You get one approval and one sign-in, never a double prompt.
 
 ## What to read next
 

package/docs/evals/assertions.mdx

@@ -3,37 +3,38 @@ title: "Assertions"
 description: "Run-level methods, t.check value assertions, the matcher mini-language, and gate vs soft severity."
 ---
 
-Assertions are how an eval grades what its `test(t)` function produced. Each one **records** a result onto `t` and returns a chainable handle — the runner reads the recorded results to compute the verdict, so a single run reports every failing assertion rather than dying on the first. There are two deterministic surfaces: run-level methods on `t`, and `t.check` for grading a specific value. For model-graded assertions, see [Judge](./judge).
+Assertions are how an eval grades what its `test(t)` function produced. Each one **records** a result onto `t` and returns a chainable handle. The runner reads the recorded results to compute the verdict, so a single run reports every failing assertion rather than dying on the first. There are two deterministic surfaces: run-level methods on `t`, and `t.check` for grading a specific value. For model-graded assertions, see [Judge](./judge).
 
 ## Run-level assertions
 
-Run-level assertions read the whole run, so they take no value. They are methods on `t` and gate by default.
-
-| Assertion                                           | Asserts                                                                           |
-| --------------------------------------------------- | --------------------------------------------------------------------------------- |
-| `t.completed()`                                     | The run did not fail and did not park on unanswered HITL input                    |
-| `t.didNotFail()`                                    | No terminal failure and no `turn.failed`/`step.failed` events (parked runs pass)  |
-| `t.waiting()`                                       | The run parked on HITL input (for approval-shaped evals)                          |
-| `t.messageIncludes(token)`                          | Joined assistant text contains `token` (string or RegExp)                         |
-| `t.outputEquals(value)` / `t.outputMatches(schema)` | Deep equality / Standard Schema (e.g. Zod) validation of the parsed output        |
-| `t.calledTool(name, opts?)`                         | A matching tool call happened (`input`, `output`, `isError`, `times` constraints) |
-| `t.notCalledTool(name)`                             | No call to `name`                                                                 |
-| `t.toolOrder([...names])`                           | Tool names appear in order (other calls may interleave)                           |
-| `t.usedNoTools()`                                   | No tool calls at all                                                              |
-| `t.maxToolCalls(n)`                                 | At most `n` tool calls                                                            |
-| `t.noFailedActions()`                               | No tool, subagent, or skill action reported a failure                             |
-| `t.calledSubagent(name, opts?)`                     | A subagent delegation happened (`remoteUrl`, `output` constraints)                |
-| `t.event(predicate, label)`                         | Escape hatch: any predicate over the typed event stream                           |
-
-`t.completed()` subsumes `t.didNotFail()` — reach for `completed` unless you specifically want to allow a parked run.
+Run-level assertions read the whole run, so they take no value. They are methods on `t` and gate by default. Several key off whether a run **parked**: paused on an unanswered human-in-the-loop (HITL) input request, waiting for an approval or answer before it can continue.
+
+| Assertion                                           | Asserts                                                                                 |
+| --------------------------------------------------- | --------------------------------------------------------------------------------------- |
+| `t.completed()`                                     | The run did not fail and did not park on unanswered HITL input                          |
+| `t.didNotFail()`                                    | No terminal failure and no `turn.failed`/`step.failed` events (parked runs pass)        |
+| `t.waiting()`                                       | The run parked on HITL input (for approval-shaped evals)                                |
+| `t.messageIncludes(token)`                          | Joined assistant text contains `token` (string or RegExp)                               |
+| `t.outputEquals(value)` / `t.outputMatches(schema)` | Deep equality or Standard Schema (e.g. Zod) validation of the agent's structured output |
+| `t.calledTool(name, opts?)`                         | A matching tool call happened (`input`, `output`, `isError`, `times` constraints)       |
+| `t.notCalledTool(name)`                             | No call to `name`                                                                       |
+| `t.toolOrder([...names])`                           | Tool names appear in order (other calls may interleave)                                 |
+| `t.usedNoTools()`                                   | No tool calls at all                                                                    |
+| `t.maxToolCalls(n)`                                 | At most `n` tool calls                                                                  |
+| `t.noFailedActions()`                               | No tool, subagent, or skill action reported a failure                                   |
+| `t.calledSubagent(name, opts?)`                     | A subagent delegation happened (`remoteUrl`, `output` constraints)                      |
+| `t.event(predicate, label)`                         | Escape hatch: any predicate over the typed event stream                                 |
+
+`t.completed()` subsumes `t.didNotFail()`, so reach for `completed` unless you specifically want to allow a parked run. The structured output that `t.outputEquals` and `t.outputMatches` read is the agent's structured output (see the [output schema guide](../guides/client/output-schema)).
 
 ```ts
 await t.send("What is the weather in Brooklyn?");
 t.completed();
 t.calledTool("get_weather");
-t.usedNoTools(); // mutually exclusive with the line above — pick the one you mean
 ```
 
+`t.calledTool` and `t.usedNoTools` are mutually exclusive; assert one or the other, never both in the same run.
+
 ## Value assertions with `t.check`
 
 `t.check(value, assertion)` grades an explicit value against a builder from `eve/evals/expect`. The value can be `t.reply`, a turn's `.message`, parsed JSON, or any local you computed:
@@ -54,11 +55,11 @@ t.check(t.reply, similarity("Sunny, 72F")); // fuzzy 0–1 Levenshtein (soft)
 | `matches(schema)`      | validates against a Standard Schema              | gate    |
 | `similarity(expected)` | normalized Levenshtein similarity, 1 = identical | soft    |
 
-Pick the cheapest builder that captures what "correct" means. When exact match is too strict but a judge model is overkill, `similarity` is the middle ground; for nuanced grading, reach for the [judge](./judge).
+Pick the cheapest builder that captures what "correct" means. When exact match is too strict but a judge model is overkill, `similarity` is the middle ground. For nuanced grading, reach for the [judge](./judge).
 
 ## The matcher mini-language
 
-`t.calledTool` and `t.calledSubagent` take a matcher object — `{ input, output, isError, times }` for tools, `{ remoteUrl, output }` for subagents. Each field accepts a literal (objects partial-deep-match), a RegExp, or a function. A matcher function receives the value and returns either a boolean (acts as a predicate) or an expected value to compare against (handy for runner-assigned values like environment-provided URLs):
+`t.calledTool` and `t.calledSubagent` take a matcher object: `{ input, output, isError, times }` for tools, `{ remoteUrl, output }` for subagents. Each field accepts a literal (objects partial-deep-match), a RegExp, or a function. A matcher function receives the value and returns either a boolean (acts as a predicate) or an expected value to compare against (handy for runner-assigned values like environment-provided URLs):
 
 ```ts
 t.calledTool("bash", { input: { command: /^pwd/ }, isError: false, times: 1 });
@@ -73,9 +74,9 @@ t.calledSubagent("weather", {
 
 ## Run state and derived facts
 
-A turn that leaves the session open for a next message is the normal end state of a successful turn. Parking on unanswered HITL input is tracked separately — that is what `t.completed()` and `t.waiting()` key off.
+Beyond the raw `t.events` stream, the runner derives typed facts the assertions read: tool calls (name, input, output, error state), subagent calls, and HITL input requests. A turn that leaves the session open for a next message is the normal end state of a successful turn; parking on unanswered HITL input is tracked separately, and that is what `t.completed()` and `t.waiting()` key off.
 
-Beyond the raw `t.events` stream, the runner derives typed facts the assertions read: tool calls (name, input, output, error state), subagent calls, and HITL input requests. The built-in assertions cover almost everything; when you need to read the stream directly, `t.event(predicate, label)` is the escape hatch:
+The built-in assertions cover almost everything. When you need to read the stream directly, `t.event(predicate, label)` is the escape hatch:
 
 ```ts
 t.event(
@@ -87,11 +88,11 @@ t.event(
 
 ## Severity
 
-Every assertion returns a chainable handle. Severity rides on the assertion — there is no separate thresholds map to keep in sync.
+Every assertion returns a chainable handle. Severity rides on the assertion, so there is no separate thresholds map to keep in sync.
 
-- `.gate(threshold?)` — hard. A miss marks the eval `failed` and `eve eval` exits non-zero.
-- `.soft(threshold?)` — tracked data. A below-threshold miss marks the eval `scored`, fatal only under `--strict`. With no threshold, it is tracked-only and never fails.
-- `.atLeast(threshold)` — soft with a bar (equivalent to `.soft(threshold)`).
+- `.gate(threshold?)` is hard. A miss marks the eval `failed` and `eve eval` exits non-zero.
+- `.soft(threshold?)` is tracked data. A below-threshold miss marks the eval `scored`, fatal only under `--strict`. With no threshold, it is tracked-only and never fails.
+- `.atLeast(threshold)` is soft with a bar (equivalent to `.soft(threshold)`).
 
 The defaults are chosen so you rarely set severity. Run-level methods and `includes`/`equals`/`matches` are gates; `similarity` and every `t.judge.*` assertion are soft. Annotate only when you deviate:
 

package/docs/evals/cases.mdx

@@ -3,11 +3,11 @@ title: "Cases"
 description: "Author single-turn and multi-turn evals with test(t), and fan one file out over a dataset."
 ---
 
-Each eval file is one graded case. The runner executes its `test(t)` function against the target, captures every event, and computes a verdict from the [assertions](./assertions) you recorded. Every eval — single-turn, multi-turn, HITL, or dataset-driven — is the same shape: one `async test(t)` function that drives the agent and asserts inline.
+Each eval file is one graded case by default, and a single file can fan out over a dataset by default-exporting an array (covered below). The runner executes each `test(t)` function against the target, captures every event, and computes a verdict from the [assertions](./assertions) you recorded. Every eval shares one shape, whether single-turn, multi-turn, human-in-the-loop (HITL), or dataset-driven: one `async test(t)` function that drives the agent and asserts inline.
 
 ## Single-turn evals
 
-The common case sends one turn and asserts on the reply. `t.send(input)` resolves once the turn settles; `t.reply` is the last assistant message:
+The common case sends one turn and asserts on the reply. `t.send(input)` resolves once the turn settles, and `t.reply` is the last assistant message:
 
 ```ts title="evals/weather/brooklyn-forecast.eval.ts"
 import { defineEval } from "eve/evals";
@@ -22,7 +22,7 @@ export default defineEval({
 });
 ```
 
-Some evals only care about behavior, not text — assert on the run and skip the content check entirely:
+Some evals only care about behavior, not text. Assert on the run and skip the content check entirely:
 
 ```ts title="evals/weather/no-tools-for-greetings.eval.ts"
 import { defineEval } from "eve/evals";
@@ -43,7 +43,7 @@ Identity is the file path, so directories are the grouping mechanism. `evals/wea
 ```text
 evals/
 ├── weather/
-│   ├── shared.ts                    # helpers — not an eval
+│   ├── shared.ts                    # helpers, not an eval
 │   ├── brooklyn-forecast.eval.ts
 │   └── no-tools-for-greetings.eval.ts
 └── smoke.eval.ts
@@ -51,7 +51,7 @@ evals/
 
 ## Multi-turn evals
 
-Drive several turns in sequence — branching, HITL approvals, structured output, attachments, multiple sessions. Because assertions live in the function, an intermediate value is just a local variable: judge a draft before the next turn overwrites it, then keep going.
+Drive several turns in sequence for branching, HITL approvals, structured output, attachments, or multiple sessions. Because assertions live in the function, an intermediate value is a local variable. Judge a draft before the next turn overwrites it, then keep going.
 
 ```ts title="evals/draft-then-send.eval.ts"
 import { defineEval } from "eve/evals";
@@ -69,7 +69,7 @@ export default defineEval({
 });
 ```
 
-Bespoke preconditions that no built-in assertion expresses are plain `throw`s — a thrown error marks the eval `failed` with the message in the result:
+For a precondition no built-in assertion expresses, `throw`. A thrown error marks the eval `failed` with the message in the result:
 
 ```ts title="evals/session-continuity.eval.ts"
 import { defineEval } from "eve/evals";
@@ -96,22 +96,22 @@ export default defineEval({
 
 `t` drives the primary session; `t.newSession()` returns an independent `EveEvalSession` against the same target, whose events feed the same run-level assertions.
 
-- `t.send(input)` sends a turn and waits for it to settle. It accepts the same input as `ClientSession.send()` — a string or a structured message — and resolves to a turn carrying `.message` and `.expectOk()`.
+- `t.send(input)` sends a turn and waits for it to settle. It accepts the same input as `ClientSession.send()` (a string or a structured message) and resolves to a turn carrying `.message` and `.expectOk()`.
 - `t.sendFile(text, path, mediaType?)` attaches a local file as a data URL.
 - `t.expectInputRequests(filter?)` asserts the previous turn parked on HITL input and returns the pending requests.
 - `t.respond(...responses)` answers specific pending input requests and sends them as the next turn.
 - `t.respondAll(optionId)` answers every pending input request with the same option and sends the responses as the next turn.
 - `t.reply` is the last assistant message (or `null`); `t.sessionId` is the current session id; `t.events` is the full typed event stream captured so far.
 
-Each `send` (and `respond`/`respondAll`) resolves to a turn whose `expectOk()` throws only when the turn ended failed — a session left open for a next message is the normal end state of a successful turn.
+Each `send` (and `respond`/`respondAll`) resolves to a turn whose `expectOk()` throws only when the turn ended failed. A session left open for a next message is the normal end state of a successful turn.
 
 Events from every session are captured in the result and artifacts. `t.log(message)` records debug lines into the eval artifact; `--verbose` also streams them to stdout as evals run. `t.signal` is an `AbortSignal` that fires on timeout.
 
-For driving sessions created outside the eval — by a channel webhook or a schedule — see [Targets](./targets).
+For driving sessions created outside the eval, by a channel webhook or a schedule, see [Targets](./targets).
 
 ## Datasets: exporting an array
 
-To fan one file out over a dataset, default-export an array of `defineEval(...)` values. Eval modules are ESM, so top-level `await` can load anything. Ids derive from the file name plus a zero-padded index (`sql/0000`, `sql/0001`, …, in array order). The loaders (`loadJson`, `loadYaml` from `eve/evals/loaders`) parse fixture files relative to the app root:
+To fan one file out over a dataset, default-export an array of `defineEval(...)` values. Eval modules are ESM, so top-level `await` can load anything. Ids derive from the file name plus a zero-padded index in array order (`sql/0000`, `sql/0001`, and so on). The loaders (`loadJson`, `loadYaml` from `eve/evals/loaders`) parse fixture files relative to the app root:
 
 ```ts title="evals/sql.eval.ts"
 import { defineEval } from "eve/evals";

package/docs/evals/judge.mdx

@@ -3,7 +3,7 @@ title: "Judge"
 description: "Grade evals with an LLM judge via t.judge.autoevals, set thresholds on the assertion, and configure the judge model."
 ---
 
-When no deterministic [assertion](./assertions) captures what "good" means — factual correctness, summary quality, free-form criteria — grade the run with an LLM judge. The `t.judge.*` assertions are the only model-backed ones, and they use a judge model that is resolved separately from the agent under test: Eve only uses it for scoring, never to swap out the agent.
+When no deterministic [assertion](./assertions) captures what "good" means (factual correctness, summary quality, free-form criteria), grade the run with an LLM judge. The `t.judge.*` assertions are the only model-backed ones, and they use a judge model that is resolved separately from the agent under test. Eve only uses it for scoring, never to swap out the agent.
 
 ```ts
 import { defineEval } from "eve/evals";
@@ -19,7 +19,7 @@ export default defineEval({
 
 ## The graders
 
-The judges live under `t.judge.autoevals` — the namespace names the [Braintrust autoevals](https://github.com/braintrustdata/autoevals) grader family, so the factuality and closedQA semantics are autoevals', not Eve-invented. Each grades `t.reply` by default and is soft by default (tracked, no gate):
+The judges live under `t.judge.autoevals`. The namespace names the [Braintrust autoevals](https://github.com/braintrustdata/autoevals) grader family, so the factuality and closedQA semantics are autoevals', not Eve-invented. Each grader scores `t.reply` by default and is soft by default (tracked, no gate):
 
 | Grader                                   | Grades                                                                                 |
 | ---------------------------------------- | -------------------------------------------------------------------------------------- |
@@ -30,8 +30,8 @@ The judges live under `t.judge.autoevals` — the namespace names the [Braintrus
 
 The reference or criteria is the positional argument. An options object follows:
 
-- `on` — the value to grade, defaulting to `t.reply`. Pass an intermediate draft or parsed value to grade it instead.
-- `model` / `modelOptions` — a per-call judge override (see below).
+- `on` is the value to grade, defaulting to `t.reply`. Pass an intermediate draft or parsed value to grade it instead.
+- `model` and `modelOptions` are a per-call judge override (see below).
 
 ```ts
 const draft = await t.send("Draft the welcome email.");
@@ -40,11 +40,11 @@ t.judge.autoevals.closedQA("professional tone", { on: draft.message }).atLeast(0
 
 ## Soft scoring and thresholds
 
-Judge assertions are soft, so the threshold rides on the assertion handle — there is no separate thresholds map:
+Judge assertions are soft, so the threshold rides on the assertion handle. There is no separate thresholds map:
 
-- **No threshold** — tracked-only. The score lands in reports and artifacts and never fails the eval. Use it to watch a metric without gating on it.
-- `.atLeast(threshold)` — a soft bar. A below-threshold score marks the eval `scored`, fatal only under `eve eval --strict`.
-- `.gate(threshold)` — promote a judge to a hard gate that fails the eval outright.
+- **No threshold** is tracked-only. The score lands in reports and artifacts and never fails the eval. Use it to watch a metric without gating on it.
+- `.atLeast(threshold)` is a soft bar. A below-threshold score marks the eval `scored`, fatal only under `eve eval --strict`.
+- `.gate(threshold)` promotes a judge to a hard gate that fails the eval outright.
 
 ```ts
 t.judge.autoevals.closedQA("cites a source"); // tracked, never fails
@@ -58,9 +58,9 @@ A judge runs once per assertion and burns tokens, so reach for one only when not
 
 The judge model is resolved once when the runner builds `t`. It is **never** the model under test. Three levels resolve innermost-wins:
 
-1. **Per-call** — `t.judge.autoevals.closedQA("…", { model, modelOptions })`.
-2. **Per-eval** — `defineEval({ judge: { model, modelOptions }, test })`.
-3. **Project default** — `defineEvalConfig({ judge: { model, modelOptions } })` in `evals.config.ts`.
+1. **Per-call**: `t.judge.autoevals.closedQA("…", { model, modelOptions })`.
+2. **Per-eval**: `defineEval({ judge: { model, modelOptions }, test })`.
+3. **Project default**: `defineEvalConfig({ judge: { model, modelOptions } })` in `evals.config.ts`.
 
 ```ts title="evals/evals.config.ts"
 import { defineEvalConfig } from "eve/evals";
@@ -78,14 +78,14 @@ export default defineEval({
   async test(t) {
     await t.send("Explain quantum tunneling to a 10-year-old.");
     t.judge.autoevals.factuality(reference).atLeast(0.7);
-    t.judge.autoevals.closedQA("is concise", { model: "anthropic/claude-haiku-4-5" }); // cheaper, per-call
+    t.judge.autoevals.closedQA("is concise", { model: "anthropic/claude-haiku-4.5" }); // cheaper, per-call
   },
 });
 ```
 
-`judge` in `evals.config.ts` is optional — a tree of fully deterministic evals can omit it. But calling `t.judge.*` with no judge model resolved is a fail-fast error at eval definition time.
+`judge` in `evals.config.ts` is optional, and a tree of fully deterministic evals can omit it. Calling `t.judge.*` with no judge model resolved records a failed gate: the runner scores the assertion after the `test` function runs, the missing model throws, and the eval fails with that message.
 
-A **string model id** (e.g. `"anthropic/claude-opus-4.8"`) routes through the Vercel AI Gateway and needs `AI_GATEWAY_API_KEY` or `VERCEL_OIDC_TOKEN` in the environment; an **AI SDK `LanguageModel` instance** is used directly. With a model configured but no credentials, a judge-backed eval **skips visibly** like other real-model legs, so mock-model fixture runs stay green. For provider-specific judge settings, use `modelOptions.providerOptions`.
+A **string model id** (e.g. `"anthropic/claude-opus-4.8"`) routes through the Vercel AI Gateway and needs `AI_GATEWAY_API_KEY` or `VERCEL_OIDC_TOKEN` in the environment. An **AI SDK `LanguageModel` instance** is used directly. With a model configured but no credentials, a judge-backed eval **skips visibly** rather than failing, so the run reports the skip instead of a spurious error. For provider-specific judge settings, use `modelOptions.providerOptions`.
 
 ## What to read next
 

package/docs/evals/overview.mdx

@@ -3,13 +3,13 @@ title: "Overview"
 description: "Define repeatable scored checks for an Eve agent with defineEval and run them with eve eval."
 ---
 
-An eval is a scored check that runs your agent against real sessions and grades the result. Use it to catch regressions when you change a prompt or a tool: drive the agent through one or more turns, then assert on what it did — the run completed, the right tool ran, the reply contains the right text — and optionally ship the results to Braintrust.
+An eval is a scored check that runs your agent against real sessions and grades the result, catching regressions when you change a prompt or a tool. Drive the agent through one or more turns, assert on what it did (the run completed, the right tool ran, the reply contains the right text), and optionally ship the results to Braintrust.
 
-Evals exercise the same HTTP surface your users hit. The runner boots (or targets) a real agent server, drives sessions through the [TypeScript client](../guides/client/overview) protocol, and grades what comes back — so a passing eval means the agent actually booted, accepted a request, and produced the result you asserted.
+Evals exercise the same HTTP surface your users hit. The runner boots (or targets) a real agent server, drives sessions through the [TypeScript client](../guides/client/overview) protocol, and grades what comes back, so a passing eval means the agent booted, accepted a request, and produced the result you asserted.
 
 ## `defineEval`
 
-Eve discovers evals under the app-root `evals/` directory, in `.eval.ts` files. Each file is exactly one eval — one graded case. The file path is the eval's identity, so you don't author an `id` or `name`; directories group related evals (`evals/weather/brooklyn-forecast.eval.ts` → id `weather/brooklyn-forecast`).
+Eve discovers evals under the app-root `evals/` directory, in `.eval.ts` files. Each file is one eval by default. A file can also default-export an array to fan out over a dataset (see [Cases](./cases)). The file path is the eval's identity, so you don't author an `id` or `name`. Directories group related evals (`evals/weather/brooklyn-forecast.eval.ts` becomes id `weather/brooklyn-forecast`).
 
 ```text
 my-agent/
@@ -40,7 +40,7 @@ export default defineEval({
 });
 ```
 
-`test` is the only required field. The rest are optional: `description`, `judge`, `tags`, `metadata`, `timeoutMs`, `reporters`. The init template adds `evals/**/*.ts` to `tsconfig.json`, so your eval code type-checks alongside the app.
+`test` is the only required field. The rest are optional: `description`, `judge`, `tags`, `metadata`, `timeoutMs`, and `reporters`. The init template adds `evals/**/*.ts` to `tsconfig.json`, so your eval code type-checks alongside the app.
 
 ## `evals.config.ts`
 
@@ -56,11 +56,11 @@ export default defineEvalConfig({
 });
 ```
 
-Everything is optional. `judge` sets the default model for [LLM-as-judge](./judge) assertions (`t.judge.*`) — only evals that use them need it, so a tree of fully deterministic evals can omit it entirely. `reporters`, `maxConcurrency`, and `timeoutMs` round out the defaults. Config `reporters` observe every eval in the run — set one `Braintrust()` here instead of adding it to each eval. CLI flags (`--max-concurrency`, `--timeout`) and per-eval values take precedence over the config defaults.
+Everything is optional. `judge` sets the default model for [LLM-as-judge](./judge) assertions (`t.judge.*`); a tree of fully deterministic evals can omit it. `reporters`, `maxConcurrency`, and `timeoutMs` round out the defaults. Config `reporters` observe every eval in the run, so set one `Braintrust()` here instead of adding it to each eval. CLI flags (`--max-concurrency`, `--timeout`) and per-eval values take precedence over the config defaults.
 
 ## The `t` context
 
-`t` is both the driver and the assertion surface. There are no separate `input`, `run`, `checks`, or `scores` fields — you write ordinary control flow, sending turns and asserting inline.
+`t` is both the driver and the assertion surface. There are no separate `input`, `run`, `checks`, or `scores` fields. You write ordinary control flow, sending turns and asserting inline.
 
 - **Drive** the agent: `t.send(...)`, `t.respond(...)`, `t.respondAll(...)`, `t.sendFile(...)`, `t.expectInputRequests(...)`, `t.newSession()`. Read what came back with `t.reply` (the last assistant message), `t.sessionId`, and `t.events`. See [Cases](./cases).
 - **Assert** with three surfaces, covered next.
@@ -69,16 +69,16 @@ Everything is optional. `judge` sets the default model for [LLM-as-judge](./judg
 
 Each surface matches a genuinely different kind of judgment:
 
-- **Run-level methods** read the whole run — `t.completed()`, `t.calledTool("get_weather")`, `t.usedNoTools()`, `t.toolOrder([...])`. They take no value because they observe the run itself. See [Assertions](./assertions).
-- **`t.check(value, assertion)`** grades an explicit value with a deterministic builder from `eve/evals/expect` — `t.check(t.reply, includes("sunny"))`. Grade `t.reply`, an intermediate draft, parsed JSON — anything. See [Assertions](./assertions).
-- **`t.judge.autoevals.*`** is the LLM-as-judge surface — `t.judge.autoevals.closedQA("cites a source")`. It grades `t.reply` by default and uses the configured judge model, never the agent under test. See [Judge](./judge).
+- **Run-level methods** read the whole run, like `t.completed()`, `t.calledTool("get_weather")`, `t.usedNoTools()`, and `t.toolOrder([...])`. They take no value because they observe the run itself. See [Assertions](./assertions).
+- **`t.check(value, assertion)`** grades an explicit value with a deterministic builder from `eve/evals/expect`, such as `t.check(t.reply, includes("sunny"))`. Grade `t.reply`, an intermediate draft, parsed JSON, or anything else. See [Assertions](./assertions).
+- **`t.judge.autoevals.*`** is the LLM-as-judge surface, like `t.judge.autoevals.closedQA("cites a source")`. It grades `t.reply` by default and uses the configured judge model, never the agent under test. See [Judge](./judge).
 
 ## Gate vs soft
 
-Every assertion returns a chainable handle, so severity rides on the assertion itself — there is no separate thresholds map:
+Every assertion returns a chainable handle, so severity rides on the assertion itself. There is no separate thresholds map.
 
 - **Gates** are hard. A failed gate marks the eval `failed` and `eve eval` exits non-zero. Run-level methods, `includes`, `equals`, and `matches` are gates by default.
-- **Soft** assertions are tracked data. They land in reports and artifacts, and a below-threshold soft assertion marks the eval `scored` — visible but not fatal, unless you pass `--strict`. `similarity` and every `t.judge.*` assertion are soft by default. A soft assertion with no threshold is tracked-only and never fails.
+- **Soft** assertions are tracked data. They land in reports and artifacts, and a below-threshold soft assertion marks the eval `scored` (visible but not fatal, unless you pass `--strict`). `similarity` and every `t.judge.*` assertion are soft by default. A soft assertion with no threshold is tracked-only and never fails.
 
 Override per assertion: `.gate(threshold?)` promotes to a hard gate, `.soft(threshold?)` demotes to tracked, and `.atLeast(threshold)` is a soft assertion with a bar.
 
@@ -89,7 +89,7 @@ t.judge.autoevals.closedQA("cites a source"); // soft, tracked (no threshold)
 t.judge.autoevals.factuality(reference).atLeast(0.7); // soft, gated under --strict at 0.7
 ```
 
-## Run it
+## Run evals with eve eval
 
 ```bash
 eve eval                       # run all discovered evals against a local dev server
@@ -101,7 +101,9 @@ Exit code `0` means every eval passed its gates. See [Running evals](./running)
 
 ## A good baseline
 
-Most apps do fine with a few small smoke evals. Assert behavior with `t.completed()` plus one or two content checks, keep dataset fixtures in `evals/data/`, and only reach for a judge or Braintrust once you actually need fuzzy grading or shared result review. In CI, run `eve eval --strict` so soft threshold misses fail the build too.
+Most apps do fine with a few small smoke evals. Assert behavior with `t.completed()` plus one or two content checks, keep dataset fixtures in `evals/data/`, and reach for a judge or Braintrust only when you need fuzzy grading or shared result review. In CI, run `eve eval --strict` so soft threshold misses fail the build too.
+
+## What to read next
 
 The rest of this section covers each piece:
 
@@ -111,8 +113,4 @@ The rest of this section covers each piece:
 - [Targets](./targets): local vs remote targets for the same eval files
 - [Reporters](./reporters): Braintrust experiments and JUnit XML
 - [Running evals](./running): the `eve eval` CLI, exit codes, and artifacts
-
-## What to read next
-
-- [Cases](./cases): author your first evals
 - [Tools](../tools): the surface most evals assert on

package/docs/evals/reporters.mdx

@@ -1,11 +1,13 @@
 ---
 title: "Reporters"
-description: "Ship eval results to Braintrust experiments or JUnit XML — Eve runs and scores everything itself."
+description: "Ship eval results to Braintrust experiments or JUnit XML. Eve runs and scores everything itself."
 ---
 
-Eve runs and grades everything itself; reporters just ship the results out. The CLI prints a console summary by default — one line per eval, failed assertions with their messages — and reporters from `eve/evals/reporters` add destinations on top.
+Eve runs and grades everything itself; reporters ship the results out. The CLI prints a console summary by default (one line per eval, with failed assertions and their messages), and reporters from `eve/evals/reporters` add destinations on top.
 
-Reporters attach in two places. Declare them in `evals.config.ts` to observe **every** eval in the run — the usual choice for a shared destination like one Braintrust experiment, so you don't repeat the reporter in each file. Or list them on an individual eval's `reporters` to scope a destination to that eval (or to a group of evals that share one instance).
+You are responsible for ensuring any observability or eval provider is approved for the data exported to it.
+
+Reporters attach in two places. Declare them in `evals.config.ts` to observe **every** eval in the run, the usual choice for a shared destination like one Braintrust experiment, so you don't repeat the reporter in each file. Or list them on an individual eval's `reporters` to scope a destination to that eval (or to a group of evals that share one instance).
 
 ## Braintrust
 
@@ -38,13 +40,13 @@ export default defineEval({
 
 The reporter config takes an optional `projectName` and `experimentName`, plus a base experiment (by name or id) to diff against. Gate assertions log as binary scores under a `gate:` prefix so experiments diff gate regressions the same way they diff soft-score regressions. Eval `metadata` rides along to reporters.
 
-A reporter instance observes the evals that reference it: share one instance across several evals — the config, a `shared.ts` export, or every entry of a dataset array — and their results land in a single experiment. Listing the same config reporter on an eval too does not double-report it.
+A reporter instance observes the evals that reference it. Share one instance across several evals (the config, a `shared.ts` export, or every entry of a dataset array) and their results land in a single experiment. Listing the same config reporter on an eval too does not double-report it.
 
-Braintrust needs its SDK installed in the app and credentials in the environment. Pass `--skip-report` to run the eval without shipping results (this also suppresses config reporters) — useful locally when iterating.
+Braintrust needs its SDK installed in the app and credentials in the environment: install the `braintrust` package (`npm install braintrust`) and set `BRAINTRUST_API_KEY`. Pass `--skip-report` to run the eval without shipping results, which also suppresses config reporters and is useful locally when iterating.
 
 ## JUnit
 
-`JUnit({ filePath })` writes JUnit XML for CI annotations. The `--junit <path>` CLI flag does the same thing without touching the eval file, which is usually the better fit — CI owns the output path, not the eval:
+`JUnit({ filePath })` writes JUnit XML for CI annotations. The `--junit <path>` CLI flag does the same thing without touching the eval file, usually the better fit because CI owns the output path, not the eval:
 
 ```bash
 eve eval --strict --junit .eve/junit.xml
@@ -54,7 +56,17 @@ Each eval becomes one `<testcase>` named by its path-derived id; failed gates an
 
 ## Custom reporters
 
-A reporter implements the `EvalReporter` interface from `eve/evals/reporters` and receives the same structured results the built-ins do. Reach for one only when a destination isn't covered — the per-run artifacts under `.eve/evals/` already capture everything for ad-hoc inspection.
+A reporter implements the `EvalReporter` interface from `eve/evals/reporters` and receives the same structured results the built-ins do. The runner calls three lifecycle methods, each of which may return a promise for async work like a remote upload:
+
+```ts
+interface EvalReporter {
+  onRunStart(evaluations: readonly EveEval[], target: EveEvalTarget): void | Promise<void>;
+  onEvalComplete(result: EveEvalResult): void | Promise<void>;
+  onRunComplete(summary: EveEvalRunSummary): void | Promise<void>;
+}
+```
+
+`onRunStart` fires once before any eval runs, `onEvalComplete` fires after each observed eval with its checks, scores, and verdict, and `onRunComplete` fires once with the aggregated summary. Reach for a custom reporter only when a destination isn't covered. The per-run artifacts under `.eve/evals/` already capture everything for ad-hoc inspection.
 
 ## What to read next
 

package/docs/evals/running.mdx

@@ -15,7 +15,7 @@ eve eval --timeout 60000       # per-eval timeout in milliseconds
 eve eval --max-concurrency 4   # cap concurrent eval executions (default 8)
 eve eval --junit .eve/junit.xml  # write JUnit XML
 eve eval --list                # print discovered evals without running
-eve eval --verbose             # stream per-eval ctx.log lines to stdout
+eve eval --verbose             # stream per-eval t.log lines to stdout
 eve eval --json                # machine-readable output
 eve eval --skip-report         # skip config and eval-defined reporters (e.g. Braintrust)
 ```
@@ -27,7 +27,7 @@ Positional ids match exactly or by directory prefix: `eve eval weather` runs `ev
 | Code | Means                                                                           |
 | ---- | ------------------------------------------------------------------------------- |
 | `0`  | Every eval passed its gates (and soft thresholds, under `--strict`)             |
-| `1`  | Any eval failed — a failed gate, an execution error, or a strict threshold miss |
+| `1`  | Any eval failed (a failed gate, an execution error, or a strict threshold miss) |
 | `2`  | Configuration error                                                             |
 
 ## Artifacts