eve 0.6.0-beta.15 → 0.6.0-beta.17

package/CHANGELOG.md

@@ -1,5 +1,37 @@
 # eve
 
+## 0.6.0-beta.17
+
+### Minor Changes
+
+- 20f0fc8: The dev TUI's `/channels` is now an action list: pick an unregistered channel to run its add flow, return to the repainted list (added channels show locked), and leave with the Done row or Esc. Web Chat (the Next.js app) is now addable in projects that already have the scaffolded `agent/channels/eve.ts` session channel — the row locks only when the project actually depends on `next`, and the REPL channel shows as its own locked row.
+- 20f0fc8: The dev TUI now discovers its slash commands: typing `/` opens a suggestion list above the prompt with each command's argument hint and description — `↑`/`↓` move the highlight, `Tab` completes, `Enter` runs, `Esc` dismisses — and a new `/help` command prints the full table. Unknown `/text` is still sent to the agent as a normal message.
+- 20f0fc8: New `eve link` and `eve deploy` commands, also available inside the `eve dev` TUI as `/vercel`, `/channels`, and `/deploy` when the server runs locally. `eve link` picks the Vercel team and project and pulls AI Gateway credentials into `.env.local`; `eve deploy` links when needed and remains successful when its production URL cannot be verified; AI Gateway authentication failures in the TUI now suggest `/vercel`, and the dev server picks up pulled credentials without a restart.
+- 20f0fc8: The dev TUI's bare `/model` now opens the same searchable AI Gateway catalog picker that `eve init` uses — full catalog with the featured shortlist, pre-selected on the model the runtime is serving — instead of a short hand-curated list. `/model <provider/model-id>` still applies directly.
+- 20f0fc8: The dev TUI's `/vercel` now asks which model provider you want before any linking: pick AI Gateway and connect via a project or by pasting your own `AI_GATEWAY_API_KEY` (saved to `.env.local`), or pick another provider to get wiring instructions instead.
+
+### Patch Changes
+
+- 20f0fc8: `eve channels add` and the dev TUI `/channels` command now run `pnpm install` after recording channels, so a running `eve dev` can load newly scaffolded channel modules (for example `@vercel/connect` for Slack) immediately instead of failing to resolve them until the next deploy.
+- 20f0fc8: `eve dev` no longer clears the terminal scrollback when the terminal UI starts, and skips the redundant `server listening at` line in TUI mode — the header already shows the URL. Headless and REPL modes still print it.
+- 20f0fc8: Make `eve init` install the newest Eve prerelease through the npm `beta` dist-tag, including when scaffolding Web Chat.
+- 20f0fc8: Replace removed `create-eve` and `eve setup` guidance with the current `eve init` and `eve link` commands.
+- 20f0fc8: Restore the `eve link` and `eve deploy` command registrations after the `eve init` migration.
+- 20f0fc8: Keep superseded Vercel project-name warnings inside the `/vercel` panel when linking ultimately succeeds.
+- 20f0fc8: Scaffolding channels, connections, and new projects from an unstamped dev build (for example under `pnpm dev`'s watch emit) now resolves dependency versions from the live workspace catalog instead of failing with "unstamped version token". Published builds are unchanged: they are stamped at build time, and an unstamped token outside a dev tree still fails loudly.
+
+## 0.6.0-beta.16
+
+### Minor Changes
+
+- 5a6ac17: Add a required `evals/evals.config.ts` (authored with `defineEvalConfig`) that declares run-wide eval defaults: a mandatory scorer `model`, plus optional run-level `reporters`, `maxConcurrency`, and `timeoutMs`. Model-backed scorers now fall back to the config `model`, so `model` is optional on `defineEval` and a shared reporter (e.g. one `Braintrust()`) no longer needs to be repeated in every eval. CLI flags and per-eval values still take precedence over the config defaults.
+- 5a6ac17: `defineEval` is now always a single case, with identity fully derived from the file path — `cases`, `load`, `task`, per-case `id`, and `maxConcurrency` are removed. Declare `input` or `run` (plus `expected`, `checks`, `scores`, `parseOutput`, …) at the top level, organize related evals with directory nesting (`evals/runtime/multi-turn.eval.ts` → `runtime/multi-turn`), and default-export an array of `defineEval(...)` values for dataset fan-out (ids get a zero-padded index suffix, e.g. `weather/0000`). The runner now executes eval files concurrently (default 8, `--max-concurrency`), positional `eve eval` ids match by directory prefix, `--case` is removed, reporters use a run-level lifecycle (`onRunStart`/`onEvalComplete`/`onRunComplete`), check/scorer args expose `evaluation` instead of `case`, and artifacts land under one `.eve/evals/<timestamp>/` directory per run.
+
+### Patch Changes
+
+- a8363e6: Fix `authorization.required` not being emitted when a tool combines `needsApproval` with interactive auth. Approval-resume auth signals are now routed through the authorization park path instead of being replayed to the model as a plain tool result.
+- a8363e6: Authorization-pending tool results no longer expose OAuth URLs, user codes, or hook URLs to the model. Channels still receive full `authorization.required` events.
+
 ## 0.6.0-beta.15
 
 ### Minor Changes

package/README.md

@@ -52,7 +52,7 @@ Every authored directory has a typed helper. Import each from the matching subpa
 | `eveChannel(...)`, `slackChannel(...)`, `vercelOidc(...)`                                                           | `eve/channels/eve`, `/slack`, `/auth` | reused from `channels/<name>.ts`                 |
 | `defineSandbox(...)`                                                                                                | `eve/sandbox`                         | `sandbox.ts` (or `sandbox/sandbox.ts`)           |
 | `defineSchedule(...)`                                                                                               | `eve/schedules`                       | `schedules/<name>.ts` (or `schedules/<name>.md`) |
-| `defineEval(...)`                                                                                                   | `eve/evals`                           | `evals/<name>.eval.ts`                           |
+| `defineEval(...)`, `defineEvalConfig(...)`                                                                          | `eve/evals`                           | `evals/<name>.eval.ts`, `evals/evals.config.ts`  |
 
 Runtime accessors live on the subpath that owns the concern:
 

package/dist/docs/public/advanced/dev-tui.md

@@ -19,10 +19,12 @@ On startup the TUI prints a header for the connected agent — the model, instru
  · Subagents     researcher
  · Server        http://localhost:3000
 
- Type to chat  ·  ↑ history  ·  /new reset session  ·  /exit quit  ·  Ctrl+C interrupt
+ Type to chat  ·  ↑ history  ·  /new reset session  ·  /model /vercel /channels /deploy  ·  /exit quit  ·  Ctrl+C interrupt
 ```
 
-From there the conversation streams straight into your terminal's normal scrollback — your prompts, the agent's replies, reasoning, tool calls, nested subagents, connection-authorization prompts, and any captured `stdout`/`stderr` — so you keep native scrolling, copy/paste, and a transcript that persists after you exit. Each turn is rendered without boxes: a colored gutter glyph marks who's speaking, tool calls collapse to a one-line summary (`✓ get_weather  city="SF" → 73°F`), and a subagent's work is indented beneath its `◆` header. A sticky line at the bottom shows the input prompt or the live status (spinner, token usage). Press `Enter` to send; `Ctrl+C` interrupts a running turn or quits at the prompt. Two slash commands: `/new` starts a fresh session and `/exit` quits.
+From there the conversation streams straight into your terminal's normal scrollback — your prompts, the agent's replies, reasoning, tool calls, nested subagents, connection-authorization prompts, and any captured `stdout`/`stderr` — so you keep native scrolling, copy/paste, and a transcript that persists after you exit. Each turn is rendered without boxes: a colored gutter glyph marks who's speaking, tool calls collapse to a one-line summary (`✓ get_weather  city="SF" → 73°F`), and a subagent's work is indented beneath its `◆` header. A sticky line at the bottom shows the input prompt or the live status (spinner, token usage). Press `Enter` to send; `Ctrl+C` interrupts a running turn or quits at the prompt. Slash commands: `/new` starts a fresh session and `/exit` quits.
+
+When `eve dev` runs the server locally, three more slash commands manage the project without leaving the session. `/vercel` opens with two questions — which model provider to use (picking something other than AI Gateway shows wiring instructions for your own provider and stops there) and how to connect to AI Gateway (paste your own `AI_GATEWAY_API_KEY`, saved straight to `.env.local`, or connect via a project) — and the project path then walks the same Vercel team/project pickers as setup (picking again re-links) and pulls the project's environment so an AI Gateway credential lands in `.env.local` — the dev server reloads env files automatically, no restart needed. `/channels` shows the agent's channel list — already-registered channels render as locked rows — and adds the one you pick, including the Slack Connect provisioning, then installs the dependencies the scaffold added so the dev server can load the new channels right away; after each addition the list repaints with the new channel locked, until Done (or Esc) leaves the flow. `/deploy` ships the agent to Vercel production, linking first when the directory is unlinked. Each command echoes as an invocation line, asks through a bordered panel that takes the input area's place — one question at a time, clearly separate from the chat transcript — and finishes with a one-line `⎿` result; loading states stay on the ephemeral status line instead of piling into the transcript. These commands are not available when connected to a remote server with `--url`, and when a turn fails because AI Gateway authentication is missing or stale, the error points you at `/vercel` directly. The TUI also checks at startup: a missing model-provider setup surfaces as an attention line — `⚠ 1 setup issue: model provider not linked · /vercel` — so the fix is visible before the first message fails, and each command's outcome hangs under it with the `⎿` connector. Local sessions also get `/model`: bare `/model` opens the same searchable model picker setup uses (the AI Gateway catalog, pre-selected on the model the runtime is serving), `/model <provider/model-id>` applies one directly, and the change is written into your agent's authored source — the command reports success only after the reloaded runtime confirms the new id.
 
 The prompt input behaves like a shell line editor: `↑`/`↓` cycle through the messages you've sent this session, `←`/`→`, Home/End, and `Ctrl+A`/`Ctrl+E` move the caret, and `Ctrl+U`/`Ctrl+K`/`Ctrl+W` kill the line, the rest of the line, or the previous word. If a turn fails terminally — the server session dies or the connection drops — the TUI starts a fresh session and notes it inline so you can keep going (server-side context resets with the old session). Errors render compactly, with docs links highlighted, and a code bug escaping your agent's own code shows its stack trace dim beneath the error headline. Captured server `stdout`/`stderr` renders as dim, indented log runs behind a `│` rule — consecutive lines from the same source share one label, and nothing is ever hidden.
 

package/dist/docs/public/evals/cases.mdx

@@ -1,96 +1,95 @@
 ---
-title: "Cases and tasks"
-description: "Author data cases, load fixtures with loaders, and script multi-turn cases with run(ctx)."
+title: "Cases"
+description: "Author prompt evals, script multi-turn evals with run(ctx), and fan one file out over a dataset."
 ---
 
-A case is one graded unit of work: the runner executes it against the target, captures every event, and applies [checks](./checks) and [scores](./scores) to the result. A case is either a data case or a scripted case.
+Each eval file is one graded case: the runner executes it against the target, captures every event, and applies [checks](./checks) and [scores](./scores) to the result. An eval is either a prompt eval (`input`) or a scripted eval (`run`).
 
-## Data cases
+## Prompt evals
 
-Data cases pair an `input` with an `expected`, plus optional `checks`, `scores`, `requires`, `tags`, and `metadata`. `input` can be a string or an object. `expected` is optional, which is handy when you only care about behavior:
+Prompt evals pair an `input` with an optional `expected`. `input` can be a string or an object (objects are `JSON.stringify`d); the runner sends it as a single user turn. `expected` is optional, which is handy when you only care about behavior:
 
-```ts title="evals/weather.eval.ts"
+```ts title="evals/weather/brooklyn-forecast.eval.ts"
 import { defineEval } from "eve/evals";
 import { Checks } from "eve/evals/checks";
 
 export default defineEval({
+  input: "What is the weather in Brooklyn?",
+  expected: "Sunny",
   checks: [Checks.didNotFail()],
   scores: [],
-  cases: [
-    { id: "brooklyn-forecast", input: "What is the weather in Brooklyn?", expected: "Sunny" },
-    {
-      id: "no-tools-for-greetings",
-      input: "Hello!",
-      checks: [Checks.toolNotCalled("get_weather")],
-    },
-  ],
 });
 ```
 
-Eval-level `checks` and `scores` apply to every case; case-level entries append to them.
-
-## Loading cases from fixtures
-
-List cases inline, or load them dynamically with `load`. The loaders (`loadJson`, `loadYaml` from `eve/evals/loaders`) resolve paths relative to the app root:
-
-```ts title="evals/sql.eval.ts"
+```ts title="evals/weather/no-tools-for-greetings.eval.ts"
 import { defineEval } from "eve/evals";
-import { loadYaml } from "eve/evals/loaders";
-import { Text } from "eve/evals/scores";
+import { Checks } from "eve/evals/checks";
 
 export default defineEval({
-  async load() {
-    const doc = await loadYaml("evals/data/cases.yaml");
-    return (doc.evals as readonly { task: string; prompt: string; sql: string }[]).map((row) => ({
-      id: row.task,
-      input: row.prompt,
-      expected: row.sql,
-    }));
-  },
-  scores: [Text.exact()],
+  input: "Hello!",
+  checks: [Checks.didNotFail(), Checks.toolNotCalled("get_weather")],
+  scores: [],
 });
 ```
 
-Pass either `cases` or `load`, never both. The loaders are meant for fixtures, not runtime agent code.
+## Organizing with directories
+
+Identity is the file path, so directories are the grouping mechanism. `evals/weather/brooklyn-forecast.eval.ts` gets the id `weather/brooklyn-forecast`, and `eve eval weather` runs everything under `evals/weather/`. Shared constants and helpers live in sibling non-eval files (any name that doesn't end in `.eval.ts`):
+
+```text
+evals/
+├── weather/
+│   ├── shared.ts                    # helpers — not an eval
+│   ├── brooklyn-forecast.eval.ts
+│   └── no-tools-for-greetings.eval.ts
+└── smoke.eval.ts
+```
+
+## Datasets: exporting an array
 
-## Tasks
+To fan one file out over a dataset, default-export an array of `defineEval(...)` values. Eval modules are ESM, so top-level `await` can load anything. Ids derive from the file name plus a zero-padded index (`sql/0000`, `sql/0001`, …, in array order). The loaders (`loadJson`, `loadYaml` from `eve/evals/loaders`) parse fixture files relative to the app root:
 
-The eval-level `task` controls how the runner turns a data case into agent work:
+```ts title="evals/sql.eval.ts"
+import { defineEval } from "eve/evals";
+import { loadYaml } from "eve/evals/loaders";
+import { Text } from "eve/evals/scores";
 
-- `task.prompt` covers the single-string case — a template that interpolates the case input.
-- `task.messages` sets up a static multi-turn conversation.
-- `task.run` gives you imperative control flow for every case in the eval.
-- `task.parseOutput` compares a transformed result rather than the raw final message.
+const doc = await loadYaml("evals/data/cases.yaml");
+const rows = doc.evals as readonly { task: string; prompt: string; sql: string }[];
+
+export default rows.map((row) =>
+  defineEval({
+    description: row.task,
+    input: row.prompt,
+    expected: row.sql,
+    scores: [Text.exact()],
+  }),
+);
+```
 
-A task can specify at most one of `prompt`, `messages`, and `run`. When none is set, the case `input` is sent as a single user message.
+The loaders are meant for fixtures, not runtime agent code.
 
-## Scripted cases
+## Scripted evals
 
-Scripted cases define their own `run(ctx)` and do not need `input`. Use them for smoke-style behavior: multi-turn branching, HITL approvals, structured output, attachments, and multiple independent sessions.
+Scripted evals define `run(ctx)` instead of `input`. Use them for smoke-style behavior: multi-turn branching, HITL approvals, structured output, attachments, and multiple independent sessions.
 
-```ts title="evals/approvals.eval.ts"
+```ts title="evals/approve-tool.eval.ts"
 import { defineEval } from "eve/evals";
 import { Checks } from "eve/evals/checks";
 
 export default defineEval({
-  checks: [Checks.didNotFail()],
+  async run({ session }) {
+    await session.send("run `pwd`");
+    session.expectInputRequests({ toolName: "bash" });
+    const turn = await session.respondAll("approve");
+    return turn.message;
+  },
+  checks: [Checks.didNotFail(), Checks.toolCalled("bash", { input: { command: /pwd/ } })],
   scores: [],
-  cases: [
-    {
-      id: "approve-tool",
-      async run({ session }) {
-        await session.send("run `pwd`");
-        session.expectInputRequests({ toolName: "bash" });
-        const turn = await session.respondAll("approve");
-        return turn.message;
-      },
-      checks: [Checks.toolCalled("bash", { input: { command: /pwd/ } })],
-    },
-  ],
 });
 ```
 
-The return value of `run` becomes the case output that scorers grade. Throwing marks the case failed with the error message in the result.
+The return value of `run` becomes the output that scorers grade (set `parseOutput` to transform the raw result instead). Throwing marks the eval failed with the error message in the result.
 
 ## The session API
 
@@ -104,12 +103,12 @@ The return value of `run` becomes the case output that scorers grade. Throwing m
 
 Each `send` resolves to an `EveEvalTurn` carrying the turn's `message`, `events`, and status. `turn.expectOk()` throws only when the turn ended failed — a session left open for a next message is the normal end state of a successful turn.
 
-Events from every eval session are captured in the case result and artifacts. `ctx.log(message)` records debug lines into the case artifact; `--verbose` also streams them to stdout as cases run.
+Events from every eval session are captured in the result and artifacts. `ctx.log(message)` records debug lines into the eval artifact; `--verbose` also streams them to stdout as evals run.
 
 For driving sessions created outside the eval — by a channel webhook or a schedule — see [Targets and requirements](./targets).
 
 ## What to read next
 
-- [Checks](./checks): assert on what the case did
+- [Checks](./checks): assert on what the eval did
 - [Scores](./scores): grade how well it did it
 - [TypeScript client](../client/messages): the send/turn protocol eval sessions build on

package/dist/docs/public/evals/checks.mdx

@@ -1,11 +1,9 @@
 ---
 title: "Checks"
-description: "Hard assertions over runs, tool calls, and output — any failed check fails the case and the exit code."
+description: "Hard assertions over runs, tool calls, and output — any failed check fails the eval and the exit code."
 ---
 
-Checks are hard assertions. Any failed check marks the case failed and `eve eval` exits non-zero. Use them for the things that must hold — the run completed, the right tool ran, the output parses. For graded, non-fatal signals, use [scores](./scores) instead.
-
-Checks exist at the eval level (applied to every case) and the case level (appended to the eval's).
+Checks are hard assertions. Any failed check marks the eval failed and `eve eval` exits non-zero. Use them for the things that must hold — the run completed, the right tool ran, the output parses. For graded, non-fatal signals, use [scores](./scores) instead.
 
 ## Built-in checks
 
@@ -39,7 +37,7 @@ Checks.subagentCalled("weather", {
 
 ## Custom checks
 
-A custom check is a plain function receiving `{ case, result, target }` and returning `{ name, passed, message? }`:
+A custom check is a plain function receiving `{ evaluation, result, target }` and returning `{ name, passed, message? }`:
 
 ```ts
 import type { EveEvalCheck } from "eve/evals/checks";
@@ -51,7 +49,7 @@ const repliedFast: EveEvalCheck = ({ result }) => ({
 });
 ```
 
-Write a `message` for the failing path — it is what the console reporter prints under the case line and what lands in JUnit output.
+Write a `message` for the failing path — it is what the console reporter prints under the eval line and what lands in JUnit output.
 
 ## Run status and parking
 
@@ -62,4 +60,4 @@ Write a `message` for the failing path — it is what the console reporter print
 ## What to read next
 
 - [Scores](./scores): graded signals with thresholds
-- [Cases and tasks](./cases): where checks attach
+- [Cases](./cases): where checks attach

package/dist/docs/public/evals/overview.mdx

@@ -9,67 +9,83 @@ Evals exercise the same HTTP surface your users hit. The runner boots (or target
 
 ## `defineEval`
 
-Eve discovers evals under the app-root `evals/` directory, in `.eval.ts` files. The file path is the eval's identity, so you don't author an `id` or `name`.
+Eve discovers evals under the app-root `evals/` directory, in `.eval.ts` files. Each file is exactly one eval — one graded case. The file path is the eval's identity, so you don't author an `id` or `name`; directories group related evals (`evals/weather/brooklyn-forecast.eval.ts` → id `weather/brooklyn-forecast`).
 
 ```text
 my-agent/
 ├── agent/
 ├── evals/
+│   ├── evals.config.ts
 │   ├── smoke.eval.ts
-│   └── weather.eval.ts
+│   └── weather/
+│       ├── brooklyn-forecast.eval.ts
+│       └── no-tools-for-greetings.eval.ts
 └── package.json
 ```
 
-```ts title="evals/weather.eval.ts"
+```ts title="evals/weather/brooklyn-forecast.eval.ts"
 import { defineEval } from "eve/evals";
 import { Checks } from "eve/evals/checks";
 import { Run } from "eve/evals/scores";
 
 export default defineEval({
   description: "Basic message and tool-usage coverage for the weather agent.",
-  cases: [
-    { id: "brooklyn-forecast", input: "What is the weather in Brooklyn?", expected: "Sunny" },
-  ],
+  input: "What is the weather in Brooklyn?",
+  expected: "Sunny",
   checks: [Checks.didNotFail(), Checks.toolCalled("get_weather")],
   scores: [Run.didNotFail()],
 });
 ```
 
-Every eval needs `scores` (an empty array is fine) and either `cases` or `load`. The rest are optional: `description`, `task`, `checks`, `requires`, `model`, `thresholds`, `modelOptions`, `tags`, `metadata`, `maxConcurrency`, `timeoutMs`, `reporters`. The init template adds `evals/**/*.ts` to `tsconfig.json`, so your eval code type-checks alongside the app.
+Every eval needs `scores` (an empty array is fine) and either `input` (a prompt sent as a single turn) or `run` (an imperative script). The rest are optional: `description`, `expected`, `checks`, `requires`, `parseOutput`, `model`, `thresholds`, `modelOptions`, `tags`, `metadata`, `timeoutMs`, `reporters`. The init template adds `evals/**/*.ts` to `tsconfig.json`, so your eval code type-checks alongside the app.
 
-## Two grading tiers
+## `evals.config.ts`
+
+Every `evals/` directory needs exactly one `evals.config.ts` at its root. It declares the defaults every eval shares — most importantly the `model` used by model-backed scorers, so you don't repeat it in each file:
+
+```ts title="evals/evals.config.ts"
+import { defineEvalConfig } from "eve/evals";
+import { Braintrust } from "eve/evals/reporters";
+
+export default defineEvalConfig({
+  model: "openai/gpt-5.4-mini",
+  reporters: [Braintrust({ projectName: "my-agent" })],
+});
+```
 
-Evals grade a case on two distinct tiers:
+`model` is required; `reporters`, `maxConcurrency`, and `timeoutMs` are optional. Config `reporters` observe every eval in the run — set one `Braintrust()` here instead of adding it to each eval. CLI flags (`--max-concurrency`, `--timeout`) and per-eval values take precedence over the config defaults. An eval that needs a different judge model overrides it with its own `model`; otherwise the config `model` applies.
+
+## Two grading tiers
 
-- **[Checks](./checks) are hard assertions.** Any failed check marks the case failed and `eve eval` exits non-zero. Use them for the things that must hold — the run completed, the right tool ran, the output parses.
-- **[Scores](./scores) are soft data.** They land in reports and artifacts, and a below-threshold score marks the case `scored` — visible but not fatal, unless you pass `--strict`.
+Evals are graded on two distinct tiers:
 
-Both exist at the eval level (applied to every case) and the case level (appended to the eval's).
+- **[Checks](./checks) are hard assertions.** Any failed check marks the eval failed and `eve eval` exits non-zero. Use them for the things that must hold — the run completed, the right tool ran, the output parses.
+- **[Scores](./scores) are soft data.** They land in reports and artifacts, and a below-threshold score marks the eval `scored` — visible but not fatal, unless you pass `--strict`.
 
 ## Run it
 
 ```bash
 eve eval                       # run all discovered evals against a local dev server
-eve eval weather               # run selected evals
+eve eval weather               # run one eval, or every eval under evals/weather/
 eve eval --url https://<app>   # target an existing server or deployment
 ```
 
-Exit code `0` means every case passed its checks. See [Running evals](./running) for the full flag list, exit codes, and CI guidance.
+Exit code `0` means every eval passed its checks. See [Running evals](./running) for the full flag list, exit codes, and CI guidance.
 
 ## A good baseline
 
-Most apps do fine with a small smoke eval. Assert behavior with `Checks.didNotFail()` plus one or two content checks, keep case fixtures in `evals/data/`, and only reach for Braintrust once you actually need shared result review or experiment history. In CI, run `eve eval --strict` so threshold misses fail the build too.
+Most apps do fine with a few small smoke evals. Assert behavior with `Checks.didNotFail()` plus one or two content checks, keep dataset fixtures in `evals/data/`, and only reach for Braintrust once you actually need shared result review or experiment history. In CI, run `eve eval --strict` so threshold misses fail the build too.
 
 The rest of this section covers each piece:
 
-- [Cases and tasks](./cases): data cases, loaders, and scripted multi-turn cases
+- [Cases](./cases): prompt evals, scripted multi-turn evals, and dataset fan-out
 - [Checks](./checks): hard assertions over runs, tools, and output
 - [Scores](./scores): deterministic and LLM-judged scorers, with thresholds
-- [Targets and requirements](./targets): local vs remote targets, and gating cases on capabilities
+- [Targets and requirements](./targets): local vs remote targets, and gating evals on capabilities
 - [Reporters](./reporters): Braintrust experiments and JUnit XML
 - [Running evals](./running): the `eve eval` CLI, exit codes, and artifacts
 
 ## What to read next
 
-- [Cases and tasks](./cases): author your first cases
+- [Cases](./cases): author your first evals
 - [Tools](../tools): the surface most evals assert on

package/dist/docs/public/evals/reporters.mdx

@@ -3,27 +3,43 @@ title: "Reporters"
 description: "Ship eval results to Braintrust experiments or JUnit XML — Eve runs and scores everything itself."
 ---
 
-Eve runs and scores everything itself; reporters just ship the results out. The CLI prints a console summary by default — one line per case, failed checks with their messages — and eval-level reporters from `eve/evals/reporters` add destinations on top.
+Eve runs and scores everything itself; reporters just ship the results out. The CLI prints a console summary by default — one line per eval, failed checks with their messages — and reporters from `eve/evals/reporters` add destinations on top.
+
+Reporters attach in two places. Declare them in `evals.config.ts` to observe **every** eval in the run — the usual choice for a shared destination like one Braintrust experiment, so you don't repeat the reporter in each file. Or list them on an individual eval's `reporters` to scope a destination to that eval (or to a group of evals that share one instance).
 
 ## Braintrust
 
-`Braintrust(...)` uploads eval results to Braintrust experiments:
+`Braintrust(...)` uploads eval results to Braintrust experiments. Put one instance in the config so it covers the whole run:
+
+```ts title="evals/evals.config.ts"
+import { defineEvalConfig } from "eve/evals";
+import { Braintrust } from "eve/evals/reporters";
 
-```ts title="evals/weather.eval.ts"
+export default defineEvalConfig({
+  model: "openai/gpt-5.4-mini",
+  reporters: [Braintrust({ projectName: "weather-agent" })],
+});
+```
+
+Need a destination for only some evals? Attach it per eval instead:
+
+```ts title="evals/brooklyn-forecast.eval.ts"
 import { defineEval } from "eve/evals";
 import { Braintrust } from "eve/evals/reporters";
 import { Run } from "eve/evals/scores";
 
 export default defineEval({
-  cases: [{ id: "brooklyn-forecast", input: "What is the weather in Brooklyn?" }],
+  input: "What is the weather in Brooklyn?",
   scores: [Run.didNotFail()],
   reporters: [Braintrust({ projectName: "weather-agent" })],
 });
 ```
 
-The config takes an optional `projectName` and `experimentName`, plus a base experiment (by name or id) to diff against. Checks log as binary scores under a `check:` prefix so experiments diff check regressions the same way they diff score regressions. Eval and case `metadata` ride along to reporters.
+The reporter config takes an optional `projectName` and `experimentName`, plus a base experiment (by name or id) to diff against. Checks log as binary scores under a `check:` prefix so experiments diff check regressions the same way they diff score regressions. Eval `metadata` rides along to reporters.
+
+A reporter instance observes the evals that reference it: share one instance across several evals — the config, a `shared.ts` export, or every entry of a dataset array — and their results land in a single experiment. Listing the same config reporter on an eval too does not double-report it.
 
-Braintrust needs its SDK installed in the app and credentials in the environment. Pass `--skip-report` to run the eval without shipping results — useful locally when iterating.
+Braintrust needs its SDK installed in the app and credentials in the environment. Pass `--skip-report` to run the eval without shipping results (this also suppresses config reporters) — useful locally when iterating.
 
 ## JUnit
 
@@ -33,7 +49,7 @@ Braintrust needs its SDK installed in the app and credentials in the environment
 eve eval --strict --junit .eve/junit.xml
 ```
 
-Failed checks and execution errors land as failure messages on the matching test case, so CI surfaces them inline.
+Each eval becomes one `<testcase>` named by its path-derived id; failed checks and execution errors land as failure messages on the matching test case, so CI surfaces them inline.
 
 ## Custom reporters
 

package/dist/docs/public/evals/running.mdx

@@ -3,39 +3,40 @@ title: "Running evals"
 description: "The eve eval CLI: flags, filters, exit codes, artifacts, and how to wire evals into CI."
 ---
 
-`eve eval` discovers every `.eval.ts` file under `evals/`, boots a local dev server (or targets a remote one), runs the cases, and prints a per-case summary.
+`eve eval` discovers every `.eval.ts` file under `evals/`, boots a local dev server (or targets a remote one), runs the evals concurrently, and prints a per-eval summary.
 
 ```bash
 eve eval                       # run all discovered evals locally
-eve eval weather smoke         # run selected evals
+eve eval weather smoke         # run selected evals (an id, or a directory prefix)
 eve eval --url https://<app>   # target a remote app instead of a local host
 eve eval --mock-models         # local dev target uses deterministic mock models
-eve eval --tag fast            # only cases (or evals) carrying a tag
-eve eval --case brooklyn-forecast  # only specific case ids
+eve eval --tag fast            # only evals carrying a tag
 eve eval --strict              # below-threshold scores also fail the exit code
 eve eval --no-skips            # unmet requirements fail instead of skipping
-eve eval --timeout 60000       # per-case timeout in milliseconds
-eve eval --max-concurrency 4   # cap concurrent cases per eval
+eve eval --timeout 60000       # per-eval timeout in milliseconds
+eve eval --max-concurrency 4   # cap concurrent eval executions (default 8)
 eve eval --junit .eve/junit.xml  # write JUnit XML
-eve eval --list                # print discovered evals and cases without running
-eve eval --verbose             # stream per-case ctx.log lines to stdout
+eve eval --list                # print discovered evals without running
+eve eval --verbose             # stream per-eval ctx.log lines to stdout
 eve eval --json                # machine-readable output
-eve eval --skip-report         # skip eval-defined reporters (e.g. Braintrust)
+eve eval --skip-report         # skip config and eval-defined reporters (e.g. Braintrust)
 ```
 
+Positional ids match exactly or by directory prefix: `eve eval weather` runs `evals/weather.eval.ts`, every eval under `evals/weather/`, and every entry of an array-exported `weather.eval.ts`.
+
 ## Exit codes
 
 | Code | Means                                                                            |
 | ---- | -------------------------------------------------------------------------------- |
-| `0`  | Every case passed its checks (and thresholds, under `--strict`)                  |
-| `1`  | Any case failed — a failed check, an execution error, or a strict threshold miss |
+| `0`  | Every eval passed its checks (and thresholds, under `--strict`)                  |
+| `1`  | Any eval failed — a failed check, an execution error, or a strict threshold miss |
 | `2`  | Configuration error                                                              |
 
 Unmet [requirements](./targets) skip visibly without affecting the exit code unless you pass `--no-skips`.
 
 ## Artifacts
 
-Each run drops per-eval artifacts under `.eve/evals/<timestamp>-<eval-id>/`, including per-case check results, verdicts, captured event streams, and `ctx.log` lines. The console output stays tight on purpose; when a case fails, the artifact has the full story.
+Each run drops artifacts under `.eve/evals/<timestamp>/`: a run `summary.json`, a `results.jsonl` index, and per-eval check results, verdicts, captured event streams, and `ctx.log` lines under `evals/`. The console output stays tight on purpose; when an eval fails, the artifact has the full story.
 
 ## CI
 
@@ -46,8 +47,8 @@ eve eval --strict --mock-models --junit .eve/junit.xml
 ```
 
 - `--strict` turns threshold misses into failures, so score regressions block the merge.
-- `--mock-models` keeps the default leg deterministic and credential-free. Put real-model cases in their own eval files gated on `requires: ["env:..."]`, and add `--no-skips` on legs that must prove those ran.
-- `--junit` gives the CI provider per-case annotations; upload the `.eve/evals/` directory as a failure artifact for the full event streams.
+- `--mock-models` keeps the default leg deterministic and credential-free. Put real-model evals in their own files gated on `requires: ["env:..."]`, and add `--no-skips` on legs that must prove those ran.
+- `--junit` gives the CI provider per-eval annotations; upload the `.eve/evals/` directory as a failure artifact for the full event streams.
 
 Against a deployed app, swap `--mock-models` for `--url`:
 

package/dist/docs/public/evals/scores.mdx

@@ -1,13 +1,13 @@
 ---
 title: "Scores"
-description: "Grade eval cases with deterministic scorers or LLM judges, and gate them with thresholds."
+description: "Grade evals with deterministic scorers or LLM judges, and gate them with thresholds."
 ---
 
-Scores are soft data. They land in reports and artifacts, and a below-threshold score marks the case `scored` — visible but not fatal, unless you pass `--strict`. Use them to grade quality fractionally where a [check](./checks) would assert it absolutely.
+Scores are soft data. They land in reports and artifacts, and a below-threshold score marks the eval `scored` — visible but not fatal, unless you pass `--strict`. Use them to grade quality fractionally where a [check](./checks) would assert it absolutely.
 
 ## Choosing a scorer
 
-Scorers live in namespaces on `eve/evals/scores`. Pick the cheapest one that captures what "correct" means here. The deterministic scorers run instantly for free; an LLM judge runs once per case and burns tokens, so save it for when nothing simpler will do.
+Scorers live in namespaces on `eve/evals/scores`. Pick the cheapest one that captures what "correct" means here. The deterministic scorers run instantly for free; an LLM judge runs once per eval and burns tokens, so save it for when nothing simpler will do.
 
 | Need                                                       | Use                                                                                           |
 | ---------------------------------------------------------- | --------------------------------------------------------------------------------------------- |
@@ -21,7 +21,7 @@ Scorers live in namespaces on `eve/evals/scores`. Pick the cheapest one that cap
 | LLM-judged SQL semantic equivalence                        | `Autoevals.sql()`                                                                             |
 | LLM-judged free-form criteria (no `expected` to match)     | `Autoevals.closedQA({ criteria: "..." })`                                                     |
 
-Each scorer gets the flattened `input`, `output`, and `expected` strings along with the full case and task result — including derived facts: typed tool calls (name, input, output, error state), subagent calls, HITL input requests, and whether the run parked. `Run.usedTool` accepts the same matcher options as `Checks.toolCalled`. Return `null` from a scorer to skip a case.
+Each scorer gets the flattened `input`, `output`, and `expected` strings along with the full eval and task result — including derived facts: typed tool calls (name, input, output, error state), subagent calls, HITL input requests, and whether the run parked. `Run.usedTool` accepts the same matcher options as `Checks.toolCalled`. Return `null` from a scorer to skip it.
 
 ## Thresholds
 
@@ -32,23 +32,24 @@ import { defineEval } from "eve/evals";
 import { Run, Text } from "eve/evals/scores";
 
 export default defineEval({
-  cases: [{ id: "hello", input: "Hello", expected: "Hello" }],
+  input: "Hello",
+  expected: "Hello",
   scores: [Run.didNotFail(), Text.includes()],
   thresholds: { "run.didNotFail": 1, "text.includes": 0.5 },
 });
 ```
 
-A case below a threshold gets the `scored` verdict — reported, but only fatal under `eve eval --strict`.
+An eval below a threshold gets the `scored` verdict — reported, but only fatal under `eve eval --strict`.
 
 ## The scorer model
 
-`model` is only required when a model-backed scorer (one of the `Autoevals` wrappers) is present without its own per-scorer model override — and it's the scorer model, not the agent's. Eve only uses it for model-backed scoring, never to swap out the agent under test. Pass a string id (e.g. `"anthropic/claude-opus-4.8"`) to route through the Vercel AI Gateway, or hand it an AI SDK model instance to use that directly.
+Model-backed scorers (the `Autoevals` wrappers) need a judge model — the scorer model, not the agent's. Eve only uses it for scoring, never to swap out the agent under test. The default lives in [`evals.config.ts`](./overview#evalsconfigts) as the required `model`, so most evals inherit it without setting anything. Pass a string id (e.g. `"anthropic/claude-opus-4.8"`) to route through the Vercel AI Gateway, or an AI SDK model instance to use it directly.
 
-For provider-specific scorer-model settings, use `modelOptions.providerOptions`. Individual Autoevals scorers can also take their own `model` / `modelOptions`, which win over the eval default.
+Override the default on a single eval by setting that eval's own `model`. For provider-specific scorer-model settings, use `modelOptions.providerOptions`. Individual Autoevals scorers can also take their own `model` / `modelOptions`, which win over both the eval and config defaults.
 
 ## Concurrency and timeouts
 
-`maxConcurrency` caps parallelism and `timeoutMs` bounds each case. Leave them off and `eve eval` runs up to 8 cases per eval at once. Lower `maxConcurrency` when cases contend for a shared resource — a rate-limited connection, or a sandbox-heavy fixture.
+`timeoutMs` bounds one eval's execution: the eval's own value wins, then `evals.config.ts`'s default, and `eve eval --timeout <ms>` overrides both for a run. The runner executes up to 8 evals at once — set a default `maxConcurrency` in `evals.config.ts` or pass `--max-concurrency <n>` (which wins) to change that, and lower it when evals contend for a shared resource: a rate-limited connection, or a sandbox-heavy fixture.
 
 ## What to read next