eve 0.10.0 → 0.11.0

package/docs/evals/assertions.mdx

@@ -3,37 +3,38 @@ title: "Assertions"
 description: "Run-level methods, t.check value assertions, the matcher mini-language, and gate vs soft severity."
 ---
 
-Assertions are how an eval grades what its `test(t)` function produced. Each one **records** a result onto `t` and returns a chainable handle — the runner reads the recorded results to compute the verdict, so a single run reports every failing assertion rather than dying on the first. There are two deterministic surfaces: run-level methods on `t`, and `t.check` for grading a specific value. For model-graded assertions, see [Judge](./judge).
+Assertions are how an eval grades what its `test(t)` function produced. Each one **records** a result onto `t` and returns a chainable handle. The runner reads the recorded results to compute the verdict, so a single run reports every failing assertion rather than dying on the first. There are two deterministic surfaces: run-level methods on `t`, and `t.check` for grading a specific value. For model-graded assertions, see [Judge](./judge).
 
 ## Run-level assertions
 
-Run-level assertions read the whole run, so they take no value. They are methods on `t` and gate by default.
-
-| Assertion                                           | Asserts                                                                           |
-| --------------------------------------------------- | --------------------------------------------------------------------------------- |
-| `t.completed()`                                     | The run did not fail and did not park on unanswered HITL input                    |
-| `t.didNotFail()`                                    | No terminal failure and no `turn.failed`/`step.failed` events (parked runs pass)  |
-| `t.waiting()`                                       | The run parked on HITL input (for approval-shaped evals)                          |
-| `t.messageIncludes(token)`                          | Joined assistant text contains `token` (string or RegExp)                         |
-| `t.outputEquals(value)` / `t.outputMatches(schema)` | Deep equality / Standard Schema (e.g. Zod) validation of the parsed output        |
-| `t.calledTool(name, opts?)`                         | A matching tool call happened (`input`, `output`, `isError`, `times` constraints) |
-| `t.notCalledTool(name)`                             | No call to `name`                                                                 |
-| `t.toolOrder([...names])`                           | Tool names appear in order (other calls may interleave)                           |
-| `t.usedNoTools()`                                   | No tool calls at all                                                              |
-| `t.maxToolCalls(n)`                                 | At most `n` tool calls                                                            |
-| `t.noFailedActions()`                               | No tool, subagent, or skill action reported a failure                             |
-| `t.calledSubagent(name, opts?)`                     | A subagent delegation happened (`remoteUrl`, `output` constraints)                |
-| `t.event(predicate, label)`                         | Escape hatch: any predicate over the typed event stream                           |
-
-`t.completed()` subsumes `t.didNotFail()` — reach for `completed` unless you specifically want to allow a parked run.
+Run-level assertions read the whole run, so they take no value. They are methods on `t` and gate by default. Several key off whether a run **parked**: paused on an unanswered human-in-the-loop (HITL) input request, waiting for an approval or answer before it can continue.
+
+| Assertion                                           | Asserts                                                                                 |
+| --------------------------------------------------- | --------------------------------------------------------------------------------------- |
+| `t.completed()`                                     | The run did not fail and did not park on unanswered HITL input                          |
+| `t.didNotFail()`                                    | No terminal failure and no `turn.failed`/`step.failed` events (parked runs pass)        |
+| `t.waiting()`                                       | The run parked on HITL input (for approval-shaped evals)                                |
+| `t.messageIncludes(token)`                          | Joined assistant text contains `token` (string or RegExp)                               |
+| `t.outputEquals(value)` / `t.outputMatches(schema)` | Deep equality or Standard Schema (e.g. Zod) validation of the agent's structured output |
+| `t.calledTool(name, opts?)`                         | A matching tool call happened (`input`, `output`, `isError`, `times` constraints)       |
+| `t.notCalledTool(name)`                             | No call to `name`                                                                       |
+| `t.toolOrder([...names])`                           | Tool names appear in order (other calls may interleave)                                 |
+| `t.usedNoTools()`                                   | No tool calls at all                                                                    |
+| `t.maxToolCalls(n)`                                 | At most `n` tool calls                                                                  |
+| `t.noFailedActions()`                               | No tool, subagent, or skill action reported a failure                                   |
+| `t.calledSubagent(name, opts?)`                     | A subagent delegation happened (`remoteUrl`, `output` constraints)                      |
+| `t.event(predicate, label)`                         | Escape hatch: any predicate over the typed event stream                                 |
+
+`t.completed()` subsumes `t.didNotFail()`, so reach for `completed` unless you specifically want to allow a parked run. The structured output that `t.outputEquals` and `t.outputMatches` read is the agent's structured output (see the [output schema guide](../guides/client/output-schema)).
 
 ```ts
 await t.send("What is the weather in Brooklyn?");
 t.completed();
 t.calledTool("get_weather");
-t.usedNoTools(); // mutually exclusive with the line above — pick the one you mean
 ```
 
+`t.calledTool` and `t.usedNoTools` are mutually exclusive; assert one or the other, never both in the same run.
+
 ## Value assertions with `t.check`
 
 `t.check(value, assertion)` grades an explicit value against a builder from `eve/evals/expect`. The value can be `t.reply`, a turn's `.message`, parsed JSON, or any local you computed:
@@ -54,11 +55,11 @@ t.check(t.reply, similarity("Sunny, 72F")); // fuzzy 0–1 Levenshtein (soft)
 | `matches(schema)`      | validates against a Standard Schema              | gate    |
 | `similarity(expected)` | normalized Levenshtein similarity, 1 = identical | soft    |
 
-Pick the cheapest builder that captures what "correct" means. When exact match is too strict but a judge model is overkill, `similarity` is the middle ground; for nuanced grading, reach for the [judge](./judge).
+Pick the cheapest builder that captures what "correct" means. When exact match is too strict but a judge model is overkill, `similarity` is the middle ground. For nuanced grading, reach for the [judge](./judge).
 
 ## The matcher mini-language
 
-`t.calledTool` and `t.calledSubagent` take a matcher object — `{ input, output, isError, times }` for tools, `{ remoteUrl, output }` for subagents. Each field accepts a literal (objects partial-deep-match), a RegExp, or a function. A matcher function receives the value and returns either a boolean (acts as a predicate) or an expected value to compare against (handy for runner-assigned values like environment-provided URLs):
+`t.calledTool` and `t.calledSubagent` take a matcher object: `{ input, output, isError, times }` for tools, `{ remoteUrl, output }` for subagents. Each field accepts a literal (objects partial-deep-match), a RegExp, or a function. A matcher function receives the value and returns either a boolean (acts as a predicate) or an expected value to compare against (handy for runner-assigned values like environment-provided URLs):
 
 ```ts
 t.calledTool("bash", { input: { command: /^pwd/ }, isError: false, times: 1 });
@@ -73,9 +74,9 @@ t.calledSubagent("weather", {
 
 ## Run state and derived facts
 
-A turn that leaves the session open for a next message is the normal end state of a successful turn. Parking on unanswered HITL input is tracked separately — that is what `t.completed()` and `t.waiting()` key off.
+Beyond the raw `t.events` stream, the runner derives typed facts the assertions read: tool calls (name, input, output, error state), subagent calls, and HITL input requests. A turn that leaves the session open for a next message is the normal end state of a successful turn; parking on unanswered HITL input is tracked separately, and that is what `t.completed()` and `t.waiting()` key off.
 
-Beyond the raw `t.events` stream, the runner derives typed facts the assertions read: tool calls (name, input, output, error state), subagent calls, and HITL input requests. The built-in assertions cover almost everything; when you need to read the stream directly, `t.event(predicate, label)` is the escape hatch:
+The built-in assertions cover almost everything. When you need to read the stream directly, `t.event(predicate, label)` is the escape hatch:
 
 ```ts
 t.event(
@@ -87,11 +88,11 @@ t.event(
 
 ## Severity
 
-Every assertion returns a chainable handle. Severity rides on the assertion — there is no separate thresholds map to keep in sync.
+Every assertion returns a chainable handle. Severity rides on the assertion, so there is no separate thresholds map to keep in sync.
 
-- `.gate(threshold?)` — hard. A miss marks the eval `failed` and `eve eval` exits non-zero.
-- `.soft(threshold?)` — tracked data. A below-threshold miss marks the eval `scored`, fatal only under `--strict`. With no threshold, it is tracked-only and never fails.
-- `.atLeast(threshold)` — soft with a bar (equivalent to `.soft(threshold)`).
+- `.gate(threshold?)` is hard. A miss marks the eval `failed` and `eve eval` exits non-zero.
+- `.soft(threshold?)` is tracked data. A below-threshold miss marks the eval `scored`, fatal only under `--strict`. With no threshold, it is tracked-only and never fails.
+- `.atLeast(threshold)` is soft with a bar (equivalent to `.soft(threshold)`).
 
 The defaults are chosen so you rarely set severity. Run-level methods and `includes`/`equals`/`matches` are gates; `similarity` and every `t.judge.*` assertion are soft. Annotate only when you deviate:
 

package/docs/evals/cases.mdx

@@ -3,11 +3,11 @@ title: "Cases"
 description: "Author single-turn and multi-turn evals with test(t), and fan one file out over a dataset."
 ---
 
-Each eval file is one graded case. The runner executes its `test(t)` function against the target, captures every event, and computes a verdict from the [assertions](./assertions) you recorded. Every eval — single-turn, multi-turn, HITL, or dataset-driven — is the same shape: one `async test(t)` function that drives the agent and asserts inline.
+Each eval file is one graded case by default, and a single file can fan out over a dataset by default-exporting an array (covered below). The runner executes each `test(t)` function against the target, captures every event, and computes a verdict from the [assertions](./assertions) you recorded. Every eval shares one shape, whether single-turn, multi-turn, human-in-the-loop (HITL), or dataset-driven: one `async test(t)` function that drives the agent and asserts inline.
 
 ## Single-turn evals
 
-The common case sends one turn and asserts on the reply. `t.send(input)` resolves once the turn settles; `t.reply` is the last assistant message:
+The common case sends one turn and asserts on the reply. `t.send(input)` resolves once the turn settles, and `t.reply` is the last assistant message:
 
 ```ts title="evals/weather/brooklyn-forecast.eval.ts"
 import { defineEval } from "eve/evals";
@@ -22,7 +22,7 @@ export default defineEval({
 });
 ```
 
-Some evals only care about behavior, not text — assert on the run and skip the content check entirely:
+Some evals only care about behavior, not text. Assert on the run and skip the content check entirely:
 
 ```ts title="evals/weather/no-tools-for-greetings.eval.ts"
 import { defineEval } from "eve/evals";
@@ -43,7 +43,7 @@ Identity is the file path, so directories are the grouping mechanism. `evals/wea
 ```text
 evals/
 ├── weather/
-│   ├── shared.ts                    # helpers — not an eval
+│   ├── shared.ts                    # helpers, not an eval
 │   ├── brooklyn-forecast.eval.ts
 │   └── no-tools-for-greetings.eval.ts
 └── smoke.eval.ts
@@ -51,7 +51,7 @@ evals/
 
 ## Multi-turn evals
 
-Drive several turns in sequence — branching, HITL approvals, structured output, attachments, multiple sessions. Because assertions live in the function, an intermediate value is just a local variable: judge a draft before the next turn overwrites it, then keep going.
+Drive several turns in sequence for branching, HITL approvals, structured output, attachments, or multiple sessions. Because assertions live in the function, an intermediate value is a local variable. Judge a draft before the next turn overwrites it, then keep going.
 
 ```ts title="evals/draft-then-send.eval.ts"
 import { defineEval } from "eve/evals";
@@ -69,7 +69,7 @@ export default defineEval({
 });
 ```
 
-Bespoke preconditions that no built-in assertion expresses are plain `throw`s — a thrown error marks the eval `failed` with the message in the result:
+For a precondition no built-in assertion expresses, `throw`. A thrown error marks the eval `failed` with the message in the result:
 
 ```ts title="evals/session-continuity.eval.ts"
 import { defineEval } from "eve/evals";
@@ -96,22 +96,22 @@ export default defineEval({
 
 `t` drives the primary session; `t.newSession()` returns an independent `EveEvalSession` against the same target, whose events feed the same run-level assertions.
 
-- `t.send(input)` sends a turn and waits for it to settle. It accepts the same input as `ClientSession.send()` — a string or a structured message — and resolves to a turn carrying `.message` and `.expectOk()`.
+- `t.send(input)` sends a turn and waits for it to settle. It accepts the same input as `ClientSession.send()` (a string or a structured message) and resolves to a turn carrying `.message` and `.expectOk()`.
 - `t.sendFile(text, path, mediaType?)` attaches a local file as a data URL.
 - `t.expectInputRequests(filter?)` asserts the previous turn parked on HITL input and returns the pending requests.
 - `t.respond(...responses)` answers specific pending input requests and sends them as the next turn.
 - `t.respondAll(optionId)` answers every pending input request with the same option and sends the responses as the next turn.
 - `t.reply` is the last assistant message (or `null`); `t.sessionId` is the current session id; `t.events` is the full typed event stream captured so far.
 
-Each `send` (and `respond`/`respondAll`) resolves to a turn whose `expectOk()` throws only when the turn ended failed — a session left open for a next message is the normal end state of a successful turn.
+Each `send` (and `respond`/`respondAll`) resolves to a turn whose `expectOk()` throws only when the turn ended failed. A session left open for a next message is the normal end state of a successful turn.
 
 Events from every session are captured in the result and artifacts. `t.log(message)` records debug lines into the eval artifact; `--verbose` also streams them to stdout as evals run. `t.signal` is an `AbortSignal` that fires on timeout.
 
-For driving sessions created outside the eval — by a channel webhook or a schedule — see [Targets](./targets).
+For driving sessions created outside the eval, by a channel webhook or a schedule, see [Targets](./targets).
 
 ## Datasets: exporting an array
 
-To fan one file out over a dataset, default-export an array of `defineEval(...)` values. Eval modules are ESM, so top-level `await` can load anything. Ids derive from the file name plus a zero-padded index (`sql/0000`, `sql/0001`, …, in array order). The loaders (`loadJson`, `loadYaml` from `eve/evals/loaders`) parse fixture files relative to the app root:
+To fan one file out over a dataset, default-export an array of `defineEval(...)` values. Eval modules are ESM, so top-level `await` can load anything. Ids derive from the file name plus a zero-padded index in array order (`sql/0000`, `sql/0001`, and so on). The loaders (`loadJson`, `loadYaml` from `eve/evals/loaders`) parse fixture files relative to the app root:
 
 ```ts title="evals/sql.eval.ts"
 import { defineEval } from "eve/evals";

package/docs/evals/judge.mdx

@@ -3,7 +3,7 @@ title: "Judge"
 description: "Grade evals with an LLM judge via t.judge.autoevals, set thresholds on the assertion, and configure the judge model."
 ---
 
-When no deterministic [assertion](./assertions) captures what "good" means — factual correctness, summary quality, free-form criteria — grade the run with an LLM judge. The `t.judge.*` assertions are the only model-backed ones, and they use a judge model that is resolved separately from the agent under test: Eve only uses it for scoring, never to swap out the agent.
+When no deterministic [assertion](./assertions) captures what "good" means (factual correctness, summary quality, free-form criteria), grade the run with an LLM judge. The `t.judge.*` assertions are the only model-backed ones, and they use a judge model that is resolved separately from the agent under test. Eve only uses it for scoring, never to swap out the agent.
 
 ```ts
 import { defineEval } from "eve/evals";
@@ -19,7 +19,7 @@ export default defineEval({
 
 ## The graders
 
-The judges live under `t.judge.autoevals` — the namespace names the [Braintrust autoevals](https://github.com/braintrustdata/autoevals) grader family, so the factuality and closedQA semantics are autoevals', not Eve-invented. Each grades `t.reply` by default and is soft by default (tracked, no gate):
+The judges live under `t.judge.autoevals`. The namespace names the [Braintrust autoevals](https://github.com/braintrustdata/autoevals) grader family, so the factuality and closedQA semantics are autoevals', not Eve-invented. Each grader scores `t.reply` by default and is soft by default (tracked, no gate):
 
 | Grader                                   | Grades                                                                                 |
 | ---------------------------------------- | -------------------------------------------------------------------------------------- |
@@ -30,8 +30,8 @@ The judges live under `t.judge.autoevals` — the namespace names the [Braintrus
 
 The reference or criteria is the positional argument. An options object follows:
 
-- `on` — the value to grade, defaulting to `t.reply`. Pass an intermediate draft or parsed value to grade it instead.
-- `model` / `modelOptions` — a per-call judge override (see below).
+- `on` is the value to grade, defaulting to `t.reply`. Pass an intermediate draft or parsed value to grade it instead.
+- `model` and `modelOptions` are a per-call judge override (see below).
 
 ```ts
 const draft = await t.send("Draft the welcome email.");
@@ -40,11 +40,11 @@ t.judge.autoevals.closedQA("professional tone", { on: draft.message }).atLeast(0
 
 ## Soft scoring and thresholds
 
-Judge assertions are soft, so the threshold rides on the assertion handle — there is no separate thresholds map:
+Judge assertions are soft, so the threshold rides on the assertion handle. There is no separate thresholds map:
 
-- **No threshold** — tracked-only. The score lands in reports and artifacts and never fails the eval. Use it to watch a metric without gating on it.
-- `.atLeast(threshold)` — a soft bar. A below-threshold score marks the eval `scored`, fatal only under `eve eval --strict`.
-- `.gate(threshold)` — promote a judge to a hard gate that fails the eval outright.
+- **No threshold** is tracked-only. The score lands in reports and artifacts and never fails the eval. Use it to watch a metric without gating on it.
+- `.atLeast(threshold)` is a soft bar. A below-threshold score marks the eval `scored`, fatal only under `eve eval --strict`.
+- `.gate(threshold)` promotes a judge to a hard gate that fails the eval outright.
 
 ```ts
 t.judge.autoevals.closedQA("cites a source"); // tracked, never fails
@@ -58,9 +58,9 @@ A judge runs once per assertion and burns tokens, so reach for one only when not
 
 The judge model is resolved once when the runner builds `t`. It is **never** the model under test. Three levels resolve innermost-wins:
 
-1. **Per-call** — `t.judge.autoevals.closedQA("…", { model, modelOptions })`.
-2. **Per-eval** — `defineEval({ judge: { model, modelOptions }, test })`.
-3. **Project default** — `defineEvalConfig({ judge: { model, modelOptions } })` in `evals.config.ts`.
+1. **Per-call**: `t.judge.autoevals.closedQA("…", { model, modelOptions })`.
+2. **Per-eval**: `defineEval({ judge: { model, modelOptions }, test })`.
+3. **Project default**: `defineEvalConfig({ judge: { model, modelOptions } })` in `evals.config.ts`.
 
 ```ts title="evals/evals.config.ts"
 import { defineEvalConfig } from "eve/evals";
@@ -78,14 +78,14 @@ export default defineEval({
   async test(t) {
     await t.send("Explain quantum tunneling to a 10-year-old.");
     t.judge.autoevals.factuality(reference).atLeast(0.7);
-    t.judge.autoevals.closedQA("is concise", { model: "anthropic/claude-haiku-4-5" }); // cheaper, per-call
+    t.judge.autoevals.closedQA("is concise", { model: "anthropic/claude-haiku-4.5" }); // cheaper, per-call
   },
 });
 ```
 
-`judge` in `evals.config.ts` is optional — a tree of fully deterministic evals can omit it. But calling `t.judge.*` with no judge model resolved is a fail-fast error at eval definition time.
+`judge` in `evals.config.ts` is optional, and a tree of fully deterministic evals can omit it. Calling `t.judge.*` with no judge model resolved records a failed gate: the runner scores the assertion after the `test` function runs, the missing model throws, and the eval fails with that message.
 
-A **string model id** (e.g. `"anthropic/claude-opus-4.8"`) routes through the Vercel AI Gateway and needs `AI_GATEWAY_API_KEY` or `VERCEL_OIDC_TOKEN` in the environment; an **AI SDK `LanguageModel` instance** is used directly. With a model configured but no credentials, a judge-backed eval **skips visibly** like other real-model legs, so mock-model fixture runs stay green. For provider-specific judge settings, use `modelOptions.providerOptions`.
+A **string model id** (e.g. `"anthropic/claude-opus-4.8"`) routes through the Vercel AI Gateway and needs `AI_GATEWAY_API_KEY` or `VERCEL_OIDC_TOKEN` in the environment. An **AI SDK `LanguageModel` instance** is used directly. With a model configured but no credentials, a judge-backed eval **skips visibly** rather than failing, so the run reports the skip instead of a spurious error. For provider-specific judge settings, use `modelOptions.providerOptions`.
 
 ## What to read next
 

package/docs/evals/overview.mdx

@@ -3,13 +3,13 @@ title: "Overview"
 description: "Define repeatable scored checks for an Eve agent with defineEval and run them with eve eval."
 ---
 
-An eval is a scored check that runs your agent against real sessions and grades the result. Use it to catch regressions when you change a prompt or a tool: drive the agent through one or more turns, then assert on what it did — the run completed, the right tool ran, the reply contains the right text — and optionally ship the results to Braintrust.
+An eval is a scored check that runs your agent against real sessions and grades the result, catching regressions when you change a prompt or a tool. Drive the agent through one or more turns, assert on what it did (the run completed, the right tool ran, the reply contains the right text), and optionally ship the results to Braintrust.
 
-Evals exercise the same HTTP surface your users hit. The runner boots (or targets) a real agent server, drives sessions through the [TypeScript client](../guides/client/overview) protocol, and grades what comes back — so a passing eval means the agent actually booted, accepted a request, and produced the result you asserted.
+Evals exercise the same HTTP surface your users hit. The runner boots (or targets) a real agent server, drives sessions through the [TypeScript client](../guides/client/overview) protocol, and grades what comes back, so a passing eval means the agent booted, accepted a request, and produced the result you asserted.
 
 ## `defineEval`
 
-Eve discovers evals under the app-root `evals/` directory, in `.eval.ts` files. Each file is exactly one eval — one graded case. The file path is the eval's identity, so you don't author an `id` or `name`; directories group related evals (`evals/weather/brooklyn-forecast.eval.ts` → id `weather/brooklyn-forecast`).
+Eve discovers evals under the app-root `evals/` directory, in `.eval.ts` files. Each file is one eval by default. A file can also default-export an array to fan out over a dataset (see [Cases](./cases)). The file path is the eval's identity, so you don't author an `id` or `name`. Directories group related evals (`evals/weather/brooklyn-forecast.eval.ts` becomes id `weather/brooklyn-forecast`).
 
 ```text
 my-agent/
@@ -40,7 +40,7 @@ export default defineEval({
 });
 ```
 
-`test` is the only required field. The rest are optional: `description`, `judge`, `tags`, `metadata`, `timeoutMs`, `reporters`. The init template adds `evals/**/*.ts` to `tsconfig.json`, so your eval code type-checks alongside the app.
+`test` is the only required field. The rest are optional: `description`, `judge`, `tags`, `metadata`, `timeoutMs`, and `reporters`. The init template adds `evals/**/*.ts` to `tsconfig.json`, so your eval code type-checks alongside the app.
 
 ## `evals.config.ts`
 
@@ -56,11 +56,11 @@ export default defineEvalConfig({
 });
 ```
 
-Everything is optional. `judge` sets the default model for [LLM-as-judge](./judge) assertions (`t.judge.*`) — only evals that use them need it, so a tree of fully deterministic evals can omit it entirely. `reporters`, `maxConcurrency`, and `timeoutMs` round out the defaults. Config `reporters` observe every eval in the run — set one `Braintrust()` here instead of adding it to each eval. CLI flags (`--max-concurrency`, `--timeout`) and per-eval values take precedence over the config defaults.
+Everything is optional. `judge` sets the default model for [LLM-as-judge](./judge) assertions (`t.judge.*`); a tree of fully deterministic evals can omit it. `reporters`, `maxConcurrency`, and `timeoutMs` round out the defaults. Config `reporters` observe every eval in the run, so set one `Braintrust()` here instead of adding it to each eval. CLI flags (`--max-concurrency`, `--timeout`) and per-eval values take precedence over the config defaults.
 
 ## The `t` context
 
-`t` is both the driver and the assertion surface. There are no separate `input`, `run`, `checks`, or `scores` fields — you write ordinary control flow, sending turns and asserting inline.
+`t` is both the driver and the assertion surface. There are no separate `input`, `run`, `checks`, or `scores` fields. You write ordinary control flow, sending turns and asserting inline.
 
 - **Drive** the agent: `t.send(...)`, `t.respond(...)`, `t.respondAll(...)`, `t.sendFile(...)`, `t.expectInputRequests(...)`, `t.newSession()`. Read what came back with `t.reply` (the last assistant message), `t.sessionId`, and `t.events`. See [Cases](./cases).
 - **Assert** with three surfaces, covered next.
@@ -69,16 +69,16 @@ Everything is optional. `judge` sets the default model for [LLM-as-judge](./judg
 
 Each surface matches a genuinely different kind of judgment:
 
-- **Run-level methods** read the whole run — `t.completed()`, `t.calledTool("get_weather")`, `t.usedNoTools()`, `t.toolOrder([...])`. They take no value because they observe the run itself. See [Assertions](./assertions).
-- **`t.check(value, assertion)`** grades an explicit value with a deterministic builder from `eve/evals/expect` — `t.check(t.reply, includes("sunny"))`. Grade `t.reply`, an intermediate draft, parsed JSON — anything. See [Assertions](./assertions).
-- **`t.judge.autoevals.*`** is the LLM-as-judge surface — `t.judge.autoevals.closedQA("cites a source")`. It grades `t.reply` by default and uses the configured judge model, never the agent under test. See [Judge](./judge).
+- **Run-level methods** read the whole run, like `t.completed()`, `t.calledTool("get_weather")`, `t.usedNoTools()`, and `t.toolOrder([...])`. They take no value because they observe the run itself. See [Assertions](./assertions).
+- **`t.check(value, assertion)`** grades an explicit value with a deterministic builder from `eve/evals/expect`, such as `t.check(t.reply, includes("sunny"))`. Grade `t.reply`, an intermediate draft, parsed JSON, or anything else. See [Assertions](./assertions).
+- **`t.judge.autoevals.*`** is the LLM-as-judge surface, like `t.judge.autoevals.closedQA("cites a source")`. It grades `t.reply` by default and uses the configured judge model, never the agent under test. See [Judge](./judge).
 
 ## Gate vs soft
 
-Every assertion returns a chainable handle, so severity rides on the assertion itself — there is no separate thresholds map:
+Every assertion returns a chainable handle, so severity rides on the assertion itself. There is no separate thresholds map.
 
 - **Gates** are hard. A failed gate marks the eval `failed` and `eve eval` exits non-zero. Run-level methods, `includes`, `equals`, and `matches` are gates by default.
-- **Soft** assertions are tracked data. They land in reports and artifacts, and a below-threshold soft assertion marks the eval `scored` — visible but not fatal, unless you pass `--strict`. `similarity` and every `t.judge.*` assertion are soft by default. A soft assertion with no threshold is tracked-only and never fails.
+- **Soft** assertions are tracked data. They land in reports and artifacts, and a below-threshold soft assertion marks the eval `scored` (visible but not fatal, unless you pass `--strict`). `similarity` and every `t.judge.*` assertion are soft by default. A soft assertion with no threshold is tracked-only and never fails.
 
 Override per assertion: `.gate(threshold?)` promotes to a hard gate, `.soft(threshold?)` demotes to tracked, and `.atLeast(threshold)` is a soft assertion with a bar.
 
@@ -89,7 +89,7 @@ t.judge.autoevals.closedQA("cites a source"); // soft, tracked (no threshold)
 t.judge.autoevals.factuality(reference).atLeast(0.7); // soft, gated under --strict at 0.7
 ```
 
-## Run it
+## Run evals with eve eval
 
 ```bash
 eve eval                       # run all discovered evals against a local dev server
@@ -101,7 +101,9 @@ Exit code `0` means every eval passed its gates. See [Running evals](./running)
 
 ## A good baseline
 
-Most apps do fine with a few small smoke evals. Assert behavior with `t.completed()` plus one or two content checks, keep dataset fixtures in `evals/data/`, and only reach for a judge or Braintrust once you actually need fuzzy grading or shared result review. In CI, run `eve eval --strict` so soft threshold misses fail the build too.
+Most apps do fine with a few small smoke evals. Assert behavior with `t.completed()` plus one or two content checks, keep dataset fixtures in `evals/data/`, and reach for a judge or Braintrust only when you need fuzzy grading or shared result review. In CI, run `eve eval --strict` so soft threshold misses fail the build too.
+
+## What to read next
 
 The rest of this section covers each piece:
 
@@ -111,8 +113,4 @@ The rest of this section covers each piece:
 - [Targets](./targets): local vs remote targets for the same eval files
 - [Reporters](./reporters): Braintrust experiments and JUnit XML
 - [Running evals](./running): the `eve eval` CLI, exit codes, and artifacts
-
-## What to read next
-
-- [Cases](./cases): author your first evals
 - [Tools](../tools): the surface most evals assert on

package/docs/evals/reporters.mdx

@@ -1,11 +1,13 @@
 ---
 title: "Reporters"
-description: "Ship eval results to Braintrust experiments or JUnit XML — Eve runs and scores everything itself."
+description: "Ship eval results to Braintrust experiments or JUnit XML. Eve runs and scores everything itself."
 ---
 
-Eve runs and grades everything itself; reporters just ship the results out. The CLI prints a console summary by default — one line per eval, failed assertions with their messages — and reporters from `eve/evals/reporters` add destinations on top.
+Eve runs and grades everything itself; reporters ship the results out. The CLI prints a console summary by default (one line per eval, with failed assertions and their messages), and reporters from `eve/evals/reporters` add destinations on top.
 
-Reporters attach in two places. Declare them in `evals.config.ts` to observe **every** eval in the run — the usual choice for a shared destination like one Braintrust experiment, so you don't repeat the reporter in each file. Or list them on an individual eval's `reporters` to scope a destination to that eval (or to a group of evals that share one instance).
+You are responsible for ensuring any observability or eval provider is approved for the data exported to it.
+
+Reporters attach in two places. Declare them in `evals.config.ts` to observe **every** eval in the run, the usual choice for a shared destination like one Braintrust experiment, so you don't repeat the reporter in each file. Or list them on an individual eval's `reporters` to scope a destination to that eval (or to a group of evals that share one instance).
 
 ## Braintrust
 
@@ -38,13 +40,13 @@ export default defineEval({
 
 The reporter config takes an optional `projectName` and `experimentName`, plus a base experiment (by name or id) to diff against. Gate assertions log as binary scores under a `gate:` prefix so experiments diff gate regressions the same way they diff soft-score regressions. Eval `metadata` rides along to reporters.
 
-A reporter instance observes the evals that reference it: share one instance across several evals — the config, a `shared.ts` export, or every entry of a dataset array — and their results land in a single experiment. Listing the same config reporter on an eval too does not double-report it.
+A reporter instance observes the evals that reference it. Share one instance across several evals (the config, a `shared.ts` export, or every entry of a dataset array) and their results land in a single experiment. Listing the same config reporter on an eval too does not double-report it.
 
-Braintrust needs its SDK installed in the app and credentials in the environment. Pass `--skip-report` to run the eval without shipping results (this also suppresses config reporters) — useful locally when iterating.
+Braintrust needs its SDK installed in the app and credentials in the environment: install the `braintrust` package (`npm install braintrust`) and set `BRAINTRUST_API_KEY`. Pass `--skip-report` to run the eval without shipping results, which also suppresses config reporters and is useful locally when iterating.
 
 ## JUnit
 
-`JUnit({ filePath })` writes JUnit XML for CI annotations. The `--junit <path>` CLI flag does the same thing without touching the eval file, which is usually the better fit — CI owns the output path, not the eval:
+`JUnit({ filePath })` writes JUnit XML for CI annotations. The `--junit <path>` CLI flag does the same thing without touching the eval file, usually the better fit because CI owns the output path, not the eval:
 
 ```bash
 eve eval --strict --junit .eve/junit.xml
@@ -54,7 +56,17 @@ Each eval becomes one `<testcase>` named by its path-derived id; failed gates an
 
 ## Custom reporters
 
-A reporter implements the `EvalReporter` interface from `eve/evals/reporters` and receives the same structured results the built-ins do. Reach for one only when a destination isn't covered — the per-run artifacts under `.eve/evals/` already capture everything for ad-hoc inspection.
+A reporter implements the `EvalReporter` interface from `eve/evals/reporters` and receives the same structured results the built-ins do. The runner calls three lifecycle methods, each of which may return a promise for async work like a remote upload:
+
+```ts
+interface EvalReporter {
+  onRunStart(evaluations: readonly EveEval[], target: EveEvalTarget): void | Promise<void>;
+  onEvalComplete(result: EveEvalResult): void | Promise<void>;
+  onRunComplete(summary: EveEvalRunSummary): void | Promise<void>;
+}
+```
+
+`onRunStart` fires once before any eval runs, `onEvalComplete` fires after each observed eval with its checks, scores, and verdict, and `onRunComplete` fires once with the aggregated summary. Reach for a custom reporter only when a destination isn't covered. The per-run artifacts under `.eve/evals/` already capture everything for ad-hoc inspection.
 
 ## What to read next
 

package/docs/evals/running.mdx

@@ -15,7 +15,7 @@ eve eval --timeout 60000       # per-eval timeout in milliseconds
 eve eval --max-concurrency 4   # cap concurrent eval executions (default 8)
 eve eval --junit .eve/junit.xml  # write JUnit XML
 eve eval --list                # print discovered evals without running
-eve eval --verbose             # stream per-eval ctx.log lines to stdout
+eve eval --verbose             # stream per-eval t.log lines to stdout
 eve eval --json                # machine-readable output
 eve eval --skip-report         # skip config and eval-defined reporters (e.g. Braintrust)
 ```
@@ -27,7 +27,7 @@ Positional ids match exactly or by directory prefix: `eve eval weather` runs `ev
 | Code | Means                                                                           |
 | ---- | ------------------------------------------------------------------------------- |
 | `0`  | Every eval passed its gates (and soft thresholds, under `--strict`)             |
-| `1`  | Any eval failed — a failed gate, an execution error, or a strict threshold miss |
+| `1`  | Any eval failed (a failed gate, an execution error, or a strict threshold miss) |
 | `2`  | Configuration error                                                             |
 
 ## Artifacts

package/docs/evals/targets.mdx

@@ -3,7 +3,7 @@ title: "Targets"
 description: "Point evals at a local dev server or a deployment with the same eval files."
 ---
 
-An eval target is always an HTTP URL. `eve eval` starts a local dev server, while `eve eval --url <url>` runs against an existing server or deployment — the same eval files work for both, which is what makes evals usable as end-to-end tests in CI.
+An eval target is always an HTTP URL. `eve eval` starts a local dev server, while `eve eval --url <url>` runs against an existing server or deployment. The same eval files work for both, which is what makes evals usable as end-to-end tests in CI.
 
 The runner polls `/eve/v1/health`, verifies `/eve/v1/info`, and exposes the live target as `t.target` inside the `test` function.
 
@@ -22,11 +22,22 @@ export default defineEval({
 });
 ```
 
-- `t.target.fetch(path, init)` performs an authenticated fetch against the target — useful for channel and webhook ingress.
+- `t.target.fetch(path, init)` performs an authenticated fetch against the target, useful for channel and webhook ingress. See [Authentication](#authentication) for how the runner authenticates.
 - `t.target.dispatchSchedule(id)` triggers a [schedule](../schedules) through the dev-only schedule route and returns the session ids it created. It works only against a target with dev routes enabled (the local `eve eval` dev server, or a deployment running in development mode), and throws otherwise.
-- `t.target.attachSession(sessionId, { startIndex? })` consumes one turn from a session created outside the eval — by a channel or a schedule — so its events feed the run-level assertions.
+- `t.target.attachSession(sessionId, { startIndex? })` consumes one turn from a session created outside the eval, by a channel or a schedule, so its events feed the run-level assertions. `startIndex` skips events before that position, so a session already partway through its stream resumes from where you left off rather than replaying from the start.
 
-Sessions attached this way are full `EveEvalSession`s: you can keep driving them with `send` and read their event streams. The run-level assertions on `t` (`t.completed()`, `t.calledTool(...)`) read the whole run, attached sessions included.
+Sessions attached this way are full `EveEvalSession`s: you can keep driving them with `send` and read their event streams. The run-level assertions on `t` (`t.completed()`, `t.calledTool(...)`) read the whole run, including attached sessions.
+
+## Authentication
+
+Local targets send no auth: `eve eval` owns the dev server it boots. A remote `--url` target connects with the same credentials as every other development client, resolved in this order:
+
+- A Vercel OIDC trusted-IDP token, sent as a per-request header. It bypasses Deployment Protection without a per-project secret, so a CI job with a pulled OIDC token reaches a protected preview deployment without extra setup.
+- An `x-vercel-protection-bypass` header, added when `VERCEL_AUTOMATION_BYPASS_SECRET` is set.
+- A bearer token resolved from the same OIDC cascade.
+- `EVE_EVAL_AUTH_TOKEN`, which overrides the bearer with a static token for targets whose auth is not OIDC-based.
+
+`t.target.fetch(path, init)` carries these same credentials, so channel and webhook ingress you exercise through it authenticates the same way the session protocol does.
 
 ## What to read next