eve 0.61.1 → 0.63.0

package/docs/evals/assertions.mdx

@@ -83,7 +83,11 @@ Pick the cheapest builder that captures what "correct" means. When exact match i
 
 ## The matcher mini-language
 
-`t.calledTool` and `t.calledSubagent` take matcher objects. Tools accept `{ input, output, status, count }`; subagents accept `{ callId, childSessionId, remoteUrl, output, status, count }`. Calls match `status: "completed"` by default; use `"pending"`, `"failed"`, or `"rejected"` explicitly for lifecycle checks. A numeric `count` requires an exact number of calls matching every supplied constraint. Use a predicate for ranges or other custom count requirements; it receives the observed number of matching calls.
+`t.calledTool` and `t.calledSubagent` take matcher objects. Tools accept `{ input, output, status, count }`; subagents accept `{ callId, childSessionId, remoteUrl, output, status, count }`. Calls match `status: "completed"` by default. Tool lifecycle checks also accept `"pending"`, `"failed"`, and `"rejected"`. Subagent checks use the shared task status values: `"working"`, `"input_required"`, `"completed"`, `"failed"`, and `"cancelled"`. A numeric `count` requires an exact number of calls matching every supplied constraint. Use a predicate for ranges or other custom count requirements; it receives the observed number of matching calls.
+
+A background subagent that has only returned a working task receipt remains `working`; use `t.calledSubagent(name, { status: "working" })` to assert that delegation. Both blocking and background calls become `completed` when their actual successful result is observed through `subagent.completed`, even when the child session stays available for another call. A rejected dispatch is `failed`; an explicit cancelled child outcome is `cancelled`.
+
+The runner derives these values from captured events. It does not read stored task state, and the current stream does not expose every task transition. A proxied input request alone does not update the subagent call to `input_required`; assert that request with `requireInputRequest`. Check task notifications for background failure and cancellation when no child outcome event is present.
 
 Matcher values accept a literal (objects partial-deep-match), a RegExp, or a predicate function that returns a boolean:
 
@@ -162,7 +166,7 @@ Every assertion returns a chainable handle. Severity rides on the assertion, so
 - `.atLeast(threshold)` is soft with a bar (equivalent to `.soft(threshold)`).
 - `.label(name)` adds a stable name when an eval records several assertions from the same family.
 
-The defaults are chosen so you rarely set severity. Run-level methods and `includes`/`equals`/`matches` are gates; `similarity` and every `t.judge.*` assertion are soft. Annotate only when you deviate:
+The defaults are chosen so you rarely set severity. Run-level methods and `includes`/`equals`/`matches` are gates; `similarity` and every `t.judge(...)` assertion are soft. Annotate only when you deviate:
 
 ```ts
 t.calledTool("get_weather").soft(); // record the tool call as a metric, don't gate

package/docs/evals/cases.mdx

@@ -70,7 +70,7 @@ export default defineEval({
   async test(t) {
     const draft = await t.send("Draft the follow-up email.");
     t.check(draft.message, includes("Best regards"));
-    t.judge.autoevals.closedQA("professional tone", { on: draft.message }).atLeast(0.6);
+    t.judge("professional tone", { on: draft.message }).atLeast(0.6);
 
     await draft.session.send("Now send it.");
     t.calledTool("send_email");
@@ -94,11 +94,9 @@ export default defineEval({
     t.succeeded();
     second.messageIncludes("marigold");
 
-    t.judge.autoevals
-      .closedQA("The assistant remembers the user's favorite word across turns", {
-        on: first.session.transcript,
-      })
-      .atLeast(0.8);
+    t.judge("The assistant remembers the user's favorite word across turns", {
+      on: first.session.transcript,
+    }).atLeast(0.8);
   },
 });
 ```

package/docs/evals/judge.mdx

@@ -1,9 +1,9 @@
 ---
 title: "Judge"
-description: "Grade evals with an LLM judge via t.judge.autoevals, set thresholds on the assertion, and configure the judge model."
+description: "Grade evals with evaluation models using criteria, typed questions, or batches, and set thresholds on each assertion."
 ---
 
-When no deterministic [assertion](./assertions) captures what "good" means (factual correctness, summary quality, free-form criteria), grade the run with an LLM judge. The `t.judge.*` assertions are the only model-backed ones, and they use a judge model that is resolved separately from the agent under test. eve only uses it for scoring, never to swap out the agent.
+Use `t.judge(...)` when a deterministic [assertion](./assertions) cannot capture what a good response looks like. It calls [`evaluate` from `eve/ai`](../guides/evaluate#evaluate-inside-a-tool) and defaults to `typesafe-ai/jev`. The judge is separate from the agent under test and is used only for scoring.
 
 ```ts
 import { defineEval } from "eve/evals";
@@ -12,110 +12,153 @@ export default defineEval({
   async test(t) {
     await t.send("Explain quantum tunneling to a 10-year-old.");
     t.succeeded();
-    t.judge.autoevals.closedQA("uses no math beyond arithmetic").atLeast(0.8);
+    t.judge("The response uses no math beyond arithmetic.").atLeast(0.8);
   },
 });
 ```
 
-## The graders
+No judge configuration is required. Configure credentials for the evaluator as described under [Configuring the judge model](#configuring-the-judge-model).
 
-The judges live under `t.judge.autoevals`. The namespace names the [Braintrust autoevals](https://github.com/braintrustdata/autoevals) grader family, so the factuality and closedQA semantics are autoevals', not eve-invented. Each grader scores the most recently settled turn’s assistant message by default and is soft by default (tracked, no gate):
+## The graders
 
-| Grader                                   | Grades                                                                                 |
-| ---------------------------------------- | -------------------------------------------------------------------------------------- |
-| `t.judge.autoevals.factuality(expected)` | Factual consistency of the reply against an expected answer (A–E buckets)              |
-| `t.judge.autoevals.summarizes(expected)` | How well the reply summarizes the expected text                                        |
-| `t.judge.autoevals.closedQA(criteria)`   | Whether the reply satisfies a free-form yes/no criterion (no expected answer to match) |
-| `t.judge.autoevals.sql(expected)`        | Semantic equivalence of two SQL statements                                             |
+`t.judge` accepts a criteria string, one typed question, or a batch of named questions. A string becomes a boolean question asking whether the response meets those criteria. A single question returns an assertion handle; a batch returns one handle per question.
 
-Judges receive the latest message text from the session whose turn most recently settled as the input prompt. Approval responses and stream reads preserve that prompt. For multipart messages, including `session.sendFile()`, the prompt contains the text parts.
+| Question                     | Assertion score                                                 |
+| ---------------------------- | --------------------------------------------------------------- |
+| Criteria string or `boolean` | The returned probability of true, from 0 to 1                   |
+| `score`                      | The returned rubric position divided by the highest level index |
+| `choice`                     | 1 when the selected option equals `expected`; otherwise 0       |
 
-A failed `closedQA` assertion prints the input and judge explanation:
+Boolean scores are model estimates, not guaranteed calibrated confidence. A rubric needs at least two levels, ordered from worst to best. The SDK can return a fractional position on that rubric; eve normalizes it to the 0–1 scale used by assertion thresholds.
 
-```text
-✗ judge.autoevals.closedQA [citation] (0% < 80%): prompt: "Name the source."
-  criteria: "cites a source"
-  response: "It is widely believed."
-  rationale: "The response does not cite a source."
-  choice: "N"
+```ts
+t.judge({
+  type: "score",
+  instructions: "Grade the response's clarity.",
+  criteria: ["Unclear", "Mostly clear", "Clear and concise"],
+})
+  .label("clarity")
+  .atLeast(0.75);
 ```
 
-The reference or criteria is the positional argument. An options object follows:
+Choice questions require an `expected` option key. eve uses that expectation locally for scoring; it does not send it to the evaluator. Instructions and rubric descriptions accept strings, JSON objects, or JSON arrays. Descriptions can also be `null`.
+
+## Choose the value to grade
 
-- `on` is the value to grade, defaulting to the most recently settled turn’s assistant message. Pass an intermediate draft or parsed value to grade it instead.
-- `model` and `modelOptions` are a per-call judge override (see below).
+By default, the evaluator receives `{ input, output }`: the latest prompt and assistant message from the session whose turn most recently settled. Approval responses and stream reads preserve that prompt. For multipart messages, including `session.sendFile()`, the prompt contains the text parts.
+
+Pass `on` to replace the output. JSON values stay structured rather than being converted to strings:
 
 ```ts
 const draft = await t.send("Draft the welcome email.");
-t.judge.autoevals.closedQA("professional tone", { on: draft.message }).atLeast(0.6);
+t.judge("The response has a professional tone.", { on: draft.message }).atLeast(0.6);
 ```
 
-For a multi-turn eval, pass `session.transcript` to grade a session's complete observed conversation instead of only its final reply:
+For a multi-turn eval, pass `session.transcript`:
 
 ```ts
 const { session } = await t.send("My favorite word is marigold. Remember it.");
 await session.send("What is my favorite word?");
+t.judge("The assistant remembers the user's favorite word across turns.", {
+  on: session.transcript,
+}).atLeast(0.8);
+```
+
+The transcript contains user and assistant messages in turn order, excluding reasoning, tool calls, and tool results. See [Multi-turn evals](./cases#multi-turn-evals).
 
-t.judge.autoevals
-  .closedQA("The assistant remembers the user's favorite word across turns", {
-    on: session.transcript,
-  })
-  .atLeast(0.8);
+## Evaluate several questions together
+
+A batch makes one evaluation request against one shared state. Pass `state` to replace the default `{ input, output }` completely. Batch calls use `state` instead of `on`.
+
+```ts
+const judgments = t.judge({
+  state: { response: turn.message ?? "", reference },
+  questions: {
+    accurate: {
+      type: "boolean",
+      instructions: "Is the response consistent with the reference?",
+    },
+    outcome: {
+      type: "choice",
+      instructions: "Classify the response.",
+      criteria: {
+        answered: "Answers the request",
+        declined: "Declines the request",
+      },
+      expected: "answered",
+    },
+  },
+});
+judgments.accurate.atLeast(0.9);
+judgments.outcome.gate();
 ```
 
-`session.transcript` contains the session's user and assistant messages in turn order. It excludes reasoning, tool calls, and tool results. See [Multi-turn evals](./cases#multi-turn-evals) for the transcript format and independent sessions.
+Keys identify questions and their assertion handles; instructions must explain what to judge. Every question sees the same state. Use separate calls for unrelated inputs. A batch must contain at least one question, and a provider or validation failure fails every assertion in the batch.
 
 ## Soft scoring and thresholds
 
-Judge assertions are soft, so the threshold rides on the assertion handle. There is no separate thresholds map:
+Judges are soft by default:
 
-- **No threshold** is tracked-only. The score lands in reports and artifacts and never fails the eval. Use it to watch a metric without gating on it.
-- `.atLeast(threshold)` is a soft bar. A below-threshold score marks the eval `scored`, fatal only under `eve eval --strict`.
-- `.gate(threshold)` promotes a judge to a hard gate that fails the eval outright.
+- Without a threshold, the score is tracked in reports and artifacts without failing the eval.
+- `.atLeast(threshold)` sets a soft bar. A lower score marks the eval `scored`, fatal only under `eve eval --strict`.
+- `.gate(threshold)` makes a lower score fail the eval outright. An omitted gate threshold is 1.
 
 ```ts
-t.judge.autoevals.closedQA("cites a source"); // tracked, never fails
-t.judge.autoevals.closedQA("cites a source").label("citation").atLeast(0.6);
-t.judge.autoevals.factuality(reference).gate(0.8); // hard gate at 0.8
+t.judge("The response cites a source.");
+t.judge("The response cites a source.").label("citation").atLeast(0.8);
+t.judge("The response cites a source.").gate(0.9);
 ```
 
-A judge runs once per assertion and burns tokens, so reach for one only when nothing deterministic will do. Judge calls start when recorded, and the runner waits for all of them during finalization; assertion handles themselves are intentionally not awaitable.
+A boolean probability need not reach exactly 1 even for a good response, so choose an explicit gate threshold for boolean judgments. Choice scores are binary.
+
+Calls capture their state and questions and start immediately. The runner waits for them during finalization; handles are not awaitable. Each standalone call makes its own request, while a batch shares one request. Eval timeouts cancel pending judgments and stop waiting even if a provider ignores cancellation.
 
 ## Configuring the judge model
 
-The judge model is resolved once when the runner builds `t`. It is **never** the model under test. Three levels resolve innermost-wins:
+Model settings resolve in this order:
 
-1. **Per-call**: `t.judge.autoevals.closedQA("…", { model, modelOptions })`.
-2. **Per-eval**: `defineEval({ judge: { model, modelOptions }, test })`.
-3. **Project default**: `defineEvalConfig({ judge: { model, modelOptions } })` in `evals.config.ts`.
+1. Per-call: `t.judge("…", { model, modelOptions })`.
+2. Per-eval: `defineEval({ judge: { model, modelOptions }, test })`.
+3. Project default: `defineEvalConfig({ judge: { model, modelOptions } })`.
+4. The default model used by `evaluate`, currently `typesafe-ai/jev`.
+
+A per-eval `judge` replaces the project's judge configuration. Per-call fields override the resolved configuration. `judge` and its `model` are optional; deterministic evals make no evaluation requests.
 
 ```ts title="evals/evals.config.ts"
 import { defineEvalConfig } from "eve/evals";
 
 export default defineEvalConfig({
-  judge: { model: "openai/gpt-5.6-luna" }, // the default judge for every eval in this tree
+  judge: { model: "typesafe-ai/jev" },
 });
 ```
 
-```ts title="evals/quantum.eval.ts"
-import { defineEval } from "eve/evals";
+String IDs resolve through the AI SDK default provider, or Vercel AI Gateway when none is configured. Gateway uses `AI_GATEWAY_API_KEY` or Vercel OIDC credentials; eve's existing local Gateway connection is also honored by `evaluate`. An explicit provider evaluation model uses that provider's credentials:
 
-export default defineEval({
-  judge: { model: "anthropic/claude-opus-4.8" }, // a stronger judge for this eval
-  async test(t) {
-    await t.send("Explain quantum tunneling to a 10-year-old.");
-    t.judge.autoevals.factuality(reference).atLeast(0.7);
-    t.judge.autoevals.closedQA("is concise", { model: "anthropic/claude-haiku-4.5" }); // cheaper, per-call
-  },
+```ts
+import { openai } from "@ai-sdk/openai";
+
+t.judge("The response addresses the request.", {
+  model: openai.evaluationModel("gpt-5.6-luna"),
+  modelOptions: { providerOptions: { openai: { reasoningEffort: "high" } } },
 });
 ```
 
-`judge` in `evals.config.ts` is optional, and a tree of fully deterministic evals can omit it. Calling `t.judge.*` with no judge model resolved records a failed gate: the runner scores the assertion after the `test` function runs, the missing model throws, and the eval fails with that message.
+Use the provider's `evaluationModel` factory, not a language model instance. AI SDK supplies evaluation adapters for supported language models. Provider support and the evaluation contract are experimental; see [AI SDK evaluation](https://ai-sdk.dev/docs/ai-sdk-core/evaluation).
+
+String model IDs resolve through the default provider's evaluation API. Gateway accepts native evaluation models such as `typesafe-ai/jev`; passing a language model ID such as `openai/gpt-5.6-luna` does not automatically select the OpenAI adapter. Use an evaluation-model instance as shown above for language models.
+
+Missing credentials, invalid answers, unsupported question types, and provider errors become failed gates even for tracked-only judgments. They do not silently skip the eval or select another model. Validation, authentication, and retries use the shared `evaluate` implementation.
+
+## Diagnostics and migration
+
+Reports use `judge.boolean`, `judge.score`, and `judge.choice`; batch assertion names also include their question key. Diagnostics include the captured state, question, and raw answer. Assertion metadata retains the normalized score, model identity, token usage, warnings, and available provider and response metadata. Usage for a batch is marked as shared across its assertions, not a separate charge for each question. Evaluation models do not promise a prose rationale.
+
+The autoevals graders have been removed. Replace `t.judge.autoevals.closedQA(criteria, options)` with `t.judge(criteria, options)`. Scores now represent a probability rather than a binary yes/no answer, so review thresholds against your examples. For factuality, summaries, or SQL equivalence, write an explicit question and include the reference in shared state. No old grader buckets are preserved.
 
-A **string model id** (e.g. `"anthropic/claude-opus-4.8"`) routes through the Vercel AI Gateway and needs `AI_GATEWAY_API_KEY` or `VERCEL_OIDC_TOKEN` in the environment. An **AI SDK `LanguageModel` instance** is used directly. With a model configured but no credentials, a judge-backed eval **skips visibly** rather than failing, so the run reports the skip instead of a spurious error. For provider-specific judge settings, use `modelOptions.providerOptions`.
+Change configured language model instances to evaluation model instances. `modelOptions.providerOptions`, assertion labels, and threshold methods remain available. [Braintrust reporting](./reporters) continues to accept judge scores independently of the grader implementation.
 
 ## What to read next
 
 - [Assertions](./assertions): deterministic run-level and value assertions
-- [Reporters](./reporters): ship judged scores to Braintrust experiments
-- [Targets](./targets): local vs remote targets for judge-backed evals
+- [Reporters](./reporters): publish scores and inspect results
+- [Targets](./targets): local and remote eval targets

package/docs/evals/overview.mdx

@@ -51,12 +51,12 @@ import { defineEvalConfig } from "eve/evals";
 import { Braintrust } from "eve/evals/reporters";
 
 export default defineEvalConfig({
-  judge: { model: "openai/gpt-5.6-luna" },
+  judge: { model: "typesafe-ai/jev" },
   reporters: [Braintrust({ projectName: "my-agent" })],
 });
 ```
 
-Everything is optional. `judge` sets the default model for [LLM-as-judge](./judge) assertions (`t.judge.*`); a tree of fully deterministic evals can omit it. `reporters`, `maxConcurrency`, and `timeoutMs` round out the defaults. Config `reporters` observe every eval in the run, so set one `Braintrust()` here instead of adding it to each eval. CLI flags (`--max-concurrency`, `--timeout`) and per-eval values take precedence over the config defaults.
+Everything is optional. `judge` sets the default model for [LLM-as-judge](./judge) assertions (`t.judge(...)`); omit it to use the default evaluator, `typesafe-ai/jev`. Deterministic evals make no judge requests. `reporters`, `maxConcurrency`, and `timeoutMs` round out the defaults. Config `reporters` observe every eval in the run, so set one `Braintrust()` here instead of adding it to each eval. CLI flags (`--max-concurrency`, `--timeout`) and per-eval values take precedence over the config defaults.
 
 ## Deterministic fixture models
 
@@ -109,22 +109,24 @@ Each surface matches a genuinely different kind of judgment:
 
 - **Scoped methods** read the final whole run on `t`, snapshot one independent session when invoked there, or inspect one immutable `EveEvalTurn`. See [Assertions](./assertions).
 - **`t.check(value, assertion)`** grades an explicit value with a deterministic builder from `eve/evals/expect`, such as `t.check(turn.message, includes("sunny"))`. Grade `turn.message`, an intermediate draft, parsed JSON, or anything else. See [Assertions](./assertions).
-- **`t.judge.autoevals.*`** is the LLM-as-judge surface, like `t.judge.autoevals.closedQA("cites a source")`. It grades the most recently settled turn’s assistant message by default; pass `{ on: turn.session.transcript }` to grade a multi-turn conversation. The judge uses the configured judge model, never the agent under test. See [Judge](./judge).
+- **`t.judge(...)`** is the LLM-as-judge surface, like `t.judge("cites a source")`. It grades the most recently settled turn’s assistant message by default; pass `{ on: turn.session.transcript }` to grade a multi-turn conversation. The judge uses the configured judge model, never the agent under test. See [Judge](./judge).
 
 ## Gate vs soft
 
 Every assertion returns a chainable handle, so severity rides on the assertion itself. There is no separate thresholds map.
 
 - **Gates** are hard. A failed gate marks the eval `failed` and `eve eval` exits non-zero. Run-level methods, `includes`, `equals`, and `matches` are gates by default.
-- **Soft** assertions are tracked data. They land in reports and artifacts, and a below-threshold soft assertion marks the eval `scored` (visible but not fatal, unless you pass `--strict`). `similarity` and every `t.judge.*` assertion are soft by default. A soft assertion with no threshold is tracked-only and never fails.
+- **Soft** assertions are tracked data. They land in reports and artifacts, and a below-threshold soft assertion marks the eval `scored` (visible but not fatal, unless you pass `--strict`). `similarity` and every `t.judge(...)` assertion are soft by default. A soft assertion with no threshold is tracked-only and never fails.
 
 Override per assertion: `.gate(threshold?)` promotes to a hard gate, `.soft(threshold?)` demotes to tracked, and `.atLeast(threshold)` is a soft assertion with a bar.
 
 ```ts
 t.succeeded(); // gate
 t.calledTool("get_weather").soft(); // record as a metric, don't gate
-t.judge.autoevals.closedQA("cites a source"); // soft, tracked (no threshold)
-t.judge.autoevals.factuality(reference).atLeast(0.7); // soft, gated under --strict at 0.7
+t.judge("cites a source"); // soft, tracked (no threshold)
+t.judge("The response is consistent with output.reference.", {
+  on: { response: turn.message ?? "", reference },
+}).atLeast(0.7); // soft, gated under --strict at 0.7
 ```
 
 Use `await t.require(value, assertion)` for a gate that must pass before the script can safely continue. Use `t.skip(reason)` as the first operation for an intentionally unsupported target capability.

package/docs/evals/reporters.mdx

@@ -18,7 +18,7 @@ import { defineEvalConfig } from "eve/evals";
 import { Braintrust } from "eve/evals/reporters";
 
 export default defineEvalConfig({
-  judge: { model: "openai/gpt-5.6-luna" },
+  judge: { model: "typesafe-ai/jev" },
   reporters: [Braintrust({ projectName: "weather-agent" })],
 });
 ```

package/docs/guides/dev-tui.md

@@ -17,19 +17,19 @@ The transcript remains in your terminal scrollback after you exit. Run `/help` i
 
 | Command     | Description                                                                                                                                                  |
 | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| `/model`    | Choose the model, speed, and reasoning. Pass a model ID to set it directly: `/model provider/model-id`.                                                      |
+| `/reset`    | Start a fresh session.                                                                                                                                       |
+| `/clear`    | Clear the session's model-message history. `/new` is an alias.                                                                                               |
+| `/compact`  | Compact the current session's context.                                                                                                                       |
+| `/cancel`   | Cancel the current turn without discarding settled context.                                                                                                  |
 | `/login`    | Connect a ChatGPT subscription, Vercel account, or provider API key.                                                                                         |
-| `/model`    | Choose the model and its settings. Pass a model ID to set it directly: `/model provider/model-id`.                                                           |
 | `/add`      | Select and install channels, MCP connections, extensions, and observability integrations. Pass an item address to install it directly: `/add channel/slack`. |
 | `/deploy`   | Deploy the agent to Vercel production. Installs the Vercel CLI, signs in, and links the directory if needed.                                                 |
-| `/info`     | Show the resolved application, compiled artifacts, discovery diagnostics, and messaging routes.                                                              |
-| `/loglevel` | Choose which server and agent logs appear in the transcript.                                                                                                 |
 | `/traces`   | Open the local trace viewer. Pass a trace ID prefix to open a specific trace.                                                                                |
-| `/reset`    | Start a fresh session.                                                                                                                                       |
-| `/cancel`   | Cancel the current turn without discarding settled context.                                                                                                  |
-| `/clear`    | Clear the session's model-message history. `/new` is an alias.                                                                                               |
-| `/compact`  | Compact the current session's context.                                                                                                                       |
-| `/exit`     | Quit the UI.                                                                                                                                                 |
+| `/loglevel` | Choose which server and agent logs appear in the transcript.                                                                                                 |
+| `/info`     | Show the resolved application, compiled artifacts, discovery diagnostics, and messaging routes.                                                              |
 | `/help`     | List available commands.                                                                                                                                     |
+| `/exit`     | Quit the UI.                                                                                                                                                 |
 
 `/login`, `/model`, `/add`, `/deploy`, `/info`, and `/traces` are available when `eve dev` runs locally. They are unavailable when the UI connects to a server with `--url`.
 
@@ -59,7 +59,17 @@ Local discovery runs only in development. Deployments need explicitly provisione
 
 ### Models and settings
 
-`/model` opens the model picker and settings. Each completed selection applies immediately and returns to chat; there is no final Done step. A successful login or model change takes effect on the next prompt.
+`/model` walks through model, speed, and reasoning in order:
+
+1. Choose a model. Type to filter the list.
+2. Choose **Standard** or **Fast** speed, when supported.
+3. Choose a reasoning level, when the model supports reasoning settings. **Provider default** leaves the reasoning level to the provider.
+
+The picker highlights your current settings when they are compatible with the selected model and skips settings that cannot be changed. Use `↑` and `↓` to move, then `Enter` to advance or apply the final choice. `Esc` or `←` returns to the previous step; at the model list, either key cancels. `Ctrl+C` cancels from any step.
+
+Changes apply together after the final choice, then the picker returns to chat. Cancelling leaves your model and settings unchanged.
+
+A successful login or model change takes effect on the next prompt.
 
 OpenAI, ChatGPT, and Gateway connections default to `gpt-5.6-luna-fast`; Anthropic defaults to `claude-sonnet-5`. An explicitly authored compatible model stays selected. If a new default is unavailable, eve offers the connection's available models. Dynamic or custom model expressions must be edited in `agent.ts`.
 
@@ -82,6 +92,8 @@ Required authorization or deployment setup still runs for the selected item. Pre
 
 Type a message and press `Enter` to send it. When the agent asks a question or requests tool approval, respond in the prompt shown by the UI. Connection authorization can open a browser; keep local `eve dev` running until the browser returns to it.
 
+The activity line shows **Thinking** while the model reasons or waits to respond, **Generating** while it writes a response or tool input, and **Running** while tools execute. A blinking dot and elapsed time indicate progress, with token counts shown when available. The activity line disappears when the turn finishes or needs your input.
+
 While a turn is running, `Enter` sends your message immediately as steering. Before assistant output begins, the runtime interrupts pending model generation and continues the same turn with your correction. Executing tools finish safely. After output begins, steering applies at the next workflow boundary and preserves streamed text.
 
 Slash commands wait until the turn ends, except `/cancel`, which cancels directly. If the session does not support steering, messages queue for the next turn. Press `Esc` or `Ctrl+C` to cancel a turn with no queued messages. With queued messages, these keys select the oldest message for steering, or for the next turn if steering is unavailable. If a direct cancellation requested with `/cancel` or `Ctrl+C` does not settle, press `Ctrl+C` to stop waiting. The UI then returns to the prompt and asks you to press `Ctrl+C` again to exit. At an idle prompt, press `Ctrl+C` twice to exit.
@@ -102,7 +114,7 @@ By default, the UI shows `stderr` logs. Use `/loglevel <all|stderr|sandbox|none>
 
 Every `eve dev` process writes diagnostic logs to `.eve/logs/`, regardless of the display mode. Read them with [`eve logs`](../reference/cli#eve-logs).
 
-Use `/traces` to inspect traces recorded during local development. See [Instrumentation](../observability/instrumentation#local-traces) for trace capture and retention settings.
+Use `/traces` to inspect traces recorded during local development. See [Local traces](../observability/otel#local-traces) for trace capture and retention settings.
 
 ## Display options
 

package/docs/guides/evaluate.md

@@ -151,6 +151,10 @@ You can also call `evaluate` outside a tool; it does not require an active eve
 session. Each call performs its own evaluation. `auto` uses this function
 and adds the per-turn routing behavior described below.
 
+## Judge eval results
+
+Eval authors can use `t.judge(...)` to turn evaluation answers into scored assertions, including batches of questions sharing one state. It uses this same `evaluate` implementation and default model. See [Judge](../evals/judge) for criteria, rubrics, and thresholds.
+
 ## Evaluate tool approvals
 
 Use `auto({ model? })` when an evaluation model should decide whether a

package/docs/guides/frontend/overview.mdx

@@ -172,7 +172,7 @@ supplied `session` remains externally owned and is never replaced by prewarming
 
 `data.messages` are eve-owned `EveMessage[]`. Common text, reasoning, file, and dynamic-tool parts follow the [AI SDK `UIMessage`](https://ai-sdk.dev/docs/reference/ai-sdk-core/ui-message) rendering convention, but the types are not interchangeable. eve also exposes authorization and HITL metadata, and a file part's URL can be absent. Adapt those parts before passing messages to an API typed as `UIMessage[]`.
 
-When the root agent delegates, its stream emits `subagent.called` with the child's `childSessionId`, then `subagent.completed` after admission with a working task receipt. Later task notifications wake the parent with completion, failure, or cancellation. Detailed child progress lives on the child session's stream instead of being flattened into the root `data.messages`. Use the lower-level [TypeScript client](../client/overview#sessions) to attach to that ID when your UI needs live subagent activity. See [What the parent sees](../../subagents#what-the-parent-sees) for the complete contract.
+When the root agent delegates, its stream emits `subagent.called` with the child's `childSessionId`, then `subagent.completed` when the parent records a successful result, for both blocking and background invocations. A background working receipt appears in `action.result`, without a completion event. The child session can remain available for reuse. Task notifications wake the parent with completion, failure, or cancellation. Detailed child progress lives on the child session's stream instead of being flattened into the root `data.messages`. Use the lower-level [TypeScript client](../client/overview#sessions) to attach to that ID when your UI needs live subagent activity. See [What the parent sees](../../subagents#what-the-parent-sees) for the complete contract.
 
 ## Sending and streaming
 

package/docs/guides/hooks.md

@@ -50,7 +50,7 @@ export default githubChannel({
 
 A GitHub channel event handler cannot fire for a Slack-owned session, so platform-specific side effects do not depend on an early-return guard. On a built-in channel, an authored handler replaces that channel's default handler for the same event key. Check the channel page before overriding events that deliver replies, progress, errors, or human-input prompts.
 
-Use `ctx.channel.kind` inside a global hook only when the operation is otherwise agent-wide and conditional handling is intentional. For typed channel metadata in dynamic resolvers or instrumentation, import the channel definition and narrow with `isChannel`; see [Add runtime context](../observability/instrumentation#add-runtime-context).
+Use `ctx.channel.kind` inside a global hook only when the operation is otherwise agent-wide and conditional handling is intentional. For typed channel metadata in dynamic resolvers or instrumentation, import the channel definition and narrow with `isChannel`; see [OpenTelemetry runtime context](../observability/otel#add-runtime-context).
 
 ## Hook structure and context
 

package/docs/guides/instrumentation/{providers.mdx → instrumentation.mdx}

@@ -1,42 +1,30 @@
 ---
-title: "Instrumentation Providers"
-description: "Configure experimental instrumentation providers, handle lifecycle events, and control the content each provider receives."
-url: "/observability/instrumentation-providers"
+title: "Instrumentation"
+description: "Configure lifecycle instrumentation, handle runtime events, and control the content each destination receives."
+url: "/observability/instrumentation"
 ---
 
-<Callout type="warning" title="Experimental">
-  This API is experimental and may change without a deprecation period.
-</Callout>
+Add lifecycle instrumentation as files under `agent/instrumentation/`. Each
+file handles eve runtime events independently. Configure trace exporters and
+shared span settings with the [built-in OpenTelemetry APIs](./otel).
 
-Instrumentation providers split observability into files under `agent/instrumentation/`. Each provider handles eve lifecycle events without owning the rest of the telemetry pipeline. Configure OpenTelemetry destinations with the [built-in OpenTelemetry APIs](./otel).
+The instrumentation directory replaces `agent/instrumentation.ts`. See
+[Migrate instrumentation](/docs/observability/instrumentation-migration) when
+upgrading from `eve < 0.62.0`.
 
-Enable it explicitly in `agent.ts`:
+## Add instrumentation
 
-```ts title="agent/agent.ts"
-import { defineAgent } from "eve";
-
-export default defineAgent({
-  model: "anthropic/claude-sonnet-5",
-  experimental: {
-    instrumentationProviders: true,
-  },
-});
-```
-
-The provider directory replaces `agent/instrumentation.ts`; the two layouts cannot be used together. See [Instrumentation](../observability/instrumentation) for the current single-file API.
-
-## Add a provider
-
-The filename identifies the provider slot. A lifecycle-event provider file
-must default-export `defineInstrumentation(...)` or `disableInstrumentation()`.
+The filename identifies the instrumentation slot. A lifecycle-event file must
+default-export `defineInstrumentation(...)` or `disableInstrumentation()`.
 See [OpenTelemetry](./otel) for the supported OTel declarations.
 
 ```text
 agent/instrumentation/
-  audit.ts  lifecycle event provider
+  audit.ts  lifecycle instrumentation
 ```
 
-This provider records action timing and identity without receiving tool arguments or results:
+This instrumentation records action timing and identity without receiving tool
+arguments or results:
 
 ```ts title="agent/instrumentation/audit.ts"
 import { defineInstrumentation } from "eve/instrumentation";
@@ -60,12 +48,15 @@ export default defineInstrumentation({
 });
 ```
 
-`ctx.state` is JSON storage scoped to this provider and operation. It survives durable suspension and is released after the terminal event.
+`ctx.state` is JSON storage scoped to this file and operation. It survives
+durable suspension and is released after the terminal event.
 
 ## Control inputs and outputs
 
-Each provider has an independent `tracePolicy`. It decides whether the provider
-receives a trace and whether its events include input or output content.
+Each lifecycle file created with `defineInstrumentation(...)` has an
+independent `tracePolicy`. It decides whether that file receives a trace and
+whether its events include input or output content. OpenTelemetry destinations
+share the process-wide policy declared by `otel(...)`.
 
 ```ts title="agent/instrumentation/audit.ts"
 import { defineInstrumentation } from "eve/instrumentation";
@@ -92,8 +83,8 @@ The function receives `agentName`, `channel`, `audience`, `mode`,
 `"unknown"`; `environment` is `"development"`, `"preview"`, or
 `"production"`.
 
-Without a policy, or when it returns `true`, eve sends the provider metadata
-for every trace. It includes input and output content in these cases:
+Without a policy, or when it returns `true`, eve sends metadata for every
+trace. It includes input and output content in these cases:
 
 | Environment           | Audience               | Content            |
 | --------------------- | ---------------------- | ------------------ |
@@ -101,20 +92,21 @@ for every trace. It includes input and output content in these cases:
 | Preview or production | `public`               | Inputs and outputs |
 | Preview or production | `private` or `unknown` | Metadata only      |
 
-Return `{ emit: false }` to not sample the trace for this provider. Return
+Return `{ emit: false }` to not sample the trace for this file. Return
 `{ emit: true, recordInputs, recordOutputs }` to choose the two content
 directions explicitly. Inputs include model prompts, tool arguments, channel
 input, and user responses. Outputs include model responses, tool results,
-requests for user input, provider metadata, and error details. Omitted content
-fields are unavailable to the handler.
+requests for user input, model provider metadata, and error details. Omitted
+content fields are unavailable to the handler.
 
 The channel assigns the audience once when it creates the session. It controls
 content capture, not access. See [Audience](../channels/overview#audience) for
 how the channel derives `public`, `private`, and `unknown`.
 
-## Redact fields in a custom provider
+## Redact fields
 
-Lifecycle events are immutable snapshots. Copy the fields you need into a destination-specific payload and redact that copy before sending it:
+Lifecycle events are immutable snapshots. Copy the fields you need into a
+destination-specific payload and redact that copy before sending it:
 
 ```ts title="agent/instrumentation/audit.ts"
 import { defineInstrumentation } from "eve/instrumentation";
@@ -145,7 +137,7 @@ function redactApiKey(value: unknown): unknown {
 }
 
 async function sendAuditRecord(record: unknown): Promise<void> {
-  // Send the sanitized record to your provider.
+  // Send the sanitized record to your destination.
   void record;
 }
 ```
@@ -154,15 +146,19 @@ Prefer `recordInputs: false` or `recordOutputs: false` when the destination does
 
 ## Lifecycle events
 
-Providers can handle session, channel delivery, turn, model attempt, model call, action, input request, and tool call events. Start and terminal events share an `idempotencyKey`, which can serve as a destination row ID.
+Instrumentation can handle session, channel delivery, turn, model attempt,
+model call, action, input request, and tool call events. Start and terminal
+events share an `idempotencyKey`, which can serve as a destination row ID.
 
 An ordinary tool emits both `action.*` and `tool.call.*` events. Use `action.*` for eve's durable dispatch lifecycle, including tools, skills, subagents, and remote agents. Use `tool.call.*` only when you need the AI SDK's in-process tool execution boundary.
 
-Handlers for different providers run concurrently and are failure-isolated. Do not depend on provider execution order. Use `flush` to drain buffered records and `shutdown` to release resources.
+Handlers from different files run concurrently and are failure-isolated. Do
+not depend on execution order. Use `flush` to drain buffered records and
+`shutdown` to release resources.
 
 ## What to read next
 
-- [Instrumentation](../observability/instrumentation): configure the current single-file API
-- [OpenTelemetry](./otel): configure OTel destinations and managed exports.
+- [Migrate instrumentation](/docs/observability/instrumentation-migration): replace `agent/instrumentation.ts`.
+- [OpenTelemetry](/docs/observability/otel): configure OTel destinations and managed exports.
 - [Local development](../guides/dev-tui): inspect local traces in the TUI
-- [Hooks](../guides/hooks): react to runtime events outside the instrumentation provider API
+- [Hooks](../guides/hooks): react to runtime events outside the instrumentation lifecycle

package/docs/guides/instrumentation/meta.json

@@ -1,5 +1,5 @@
 {
   "title": "Observability",
   "defaultOpen": false,
-  "pages": ["overview", "providers", "otel"]
+  "pages": ["instrumentation", "otel", "migration"]
 }