eve 0.6.0-beta.15 → 0.6.0-beta.16

package/CHANGELOG.md

@@ -1,5 +1,17 @@
 # eve
 
+## 0.6.0-beta.16
+
+### Minor Changes
+
+- 5a6ac17: Add a required `evals/evals.config.ts` (authored with `defineEvalConfig`) that declares run-wide eval defaults: a mandatory scorer `model`, plus optional run-level `reporters`, `maxConcurrency`, and `timeoutMs`. Model-backed scorers now fall back to the config `model`, so `model` is optional on `defineEval` and a shared reporter (e.g. one `Braintrust()`) no longer needs to be repeated in every eval. CLI flags and per-eval values still take precedence over the config defaults.
+- 5a6ac17: `defineEval` is now always a single case, with identity fully derived from the file path — `cases`, `load`, `task`, per-case `id`, and `maxConcurrency` are removed. Declare `input` or `run` (plus `expected`, `checks`, `scores`, `parseOutput`, …) at the top level, organize related evals with directory nesting (`evals/runtime/multi-turn.eval.ts` → `runtime/multi-turn`), and default-export an array of `defineEval(...)` values for dataset fan-out (ids get a zero-padded index suffix, e.g. `weather/0000`). The runner now executes eval files concurrently (default 8, `--max-concurrency`), positional `eve eval` ids match by directory prefix, `--case` is removed, reporters use a run-level lifecycle (`onRunStart`/`onEvalComplete`/`onRunComplete`), check/scorer args expose `evaluation` instead of `case`, and artifacts land under one `.eve/evals/<timestamp>/` directory per run.
+
+### Patch Changes
+
+- a8363e6: Fix `authorization.required` not being emitted when a tool combines `needsApproval` with interactive auth. Approval-resume auth signals are now routed through the authorization park path instead of being replayed to the model as a plain tool result.
+- a8363e6: Authorization-pending tool results no longer expose OAuth URLs, user codes, or hook URLs to the model. Channels still receive full `authorization.required` events.
+
 ## 0.6.0-beta.15
 
 ### Minor Changes

package/README.md

@@ -52,7 +52,7 @@ Every authored directory has a typed helper. Import each from the matching subpa
 | `eveChannel(...)`, `slackChannel(...)`, `vercelOidc(...)`                                                           | `eve/channels/eve`, `/slack`, `/auth` | reused from `channels/<name>.ts`                 |
 | `defineSandbox(...)`                                                                                                | `eve/sandbox`                         | `sandbox.ts` (or `sandbox/sandbox.ts`)           |
 | `defineSchedule(...)`                                                                                               | `eve/schedules`                       | `schedules/<name>.ts` (or `schedules/<name>.md`) |
-| `defineEval(...)`                                                                                                   | `eve/evals`                           | `evals/<name>.eval.ts`                           |
+| `defineEval(...)`, `defineEvalConfig(...)`                                                                          | `eve/evals`                           | `evals/<name>.eval.ts`, `evals/evals.config.ts`  |
 
 Runtime accessors live on the subpath that owns the concern:
 

package/dist/docs/public/evals/cases.mdx

@@ -1,96 +1,95 @@
 ---
-title: "Cases and tasks"
-description: "Author data cases, load fixtures with loaders, and script multi-turn cases with run(ctx)."
+title: "Cases"
+description: "Author prompt evals, script multi-turn evals with run(ctx), and fan one file out over a dataset."
 ---
 
-A case is one graded unit of work: the runner executes it against the target, captures every event, and applies [checks](./checks) and [scores](./scores) to the result. A case is either a data case or a scripted case.
+Each eval file is one graded case: the runner executes it against the target, captures every event, and applies [checks](./checks) and [scores](./scores) to the result. An eval is either a prompt eval (`input`) or a scripted eval (`run`).
 
-## Data cases
+## Prompt evals
 
-Data cases pair an `input` with an `expected`, plus optional `checks`, `scores`, `requires`, `tags`, and `metadata`. `input` can be a string or an object. `expected` is optional, which is handy when you only care about behavior:
+Prompt evals pair an `input` with an optional `expected`. `input` can be a string or an object (objects are `JSON.stringify`d); the runner sends it as a single user turn. `expected` is optional, which is handy when you only care about behavior:
 
-```ts title="evals/weather.eval.ts"
+```ts title="evals/weather/brooklyn-forecast.eval.ts"
 import { defineEval } from "eve/evals";
 import { Checks } from "eve/evals/checks";
 
 export default defineEval({
+  input: "What is the weather in Brooklyn?",
+  expected: "Sunny",
   checks: [Checks.didNotFail()],
   scores: [],
-  cases: [
-    { id: "brooklyn-forecast", input: "What is the weather in Brooklyn?", expected: "Sunny" },
-    {
-      id: "no-tools-for-greetings",
-      input: "Hello!",
-      checks: [Checks.toolNotCalled("get_weather")],
-    },
-  ],
 });
 ```
 
-Eval-level `checks` and `scores` apply to every case; case-level entries append to them.
-
-## Loading cases from fixtures
-
-List cases inline, or load them dynamically with `load`. The loaders (`loadJson`, `loadYaml` from `eve/evals/loaders`) resolve paths relative to the app root:
-
-```ts title="evals/sql.eval.ts"
+```ts title="evals/weather/no-tools-for-greetings.eval.ts"
 import { defineEval } from "eve/evals";
-import { loadYaml } from "eve/evals/loaders";
-import { Text } from "eve/evals/scores";
+import { Checks } from "eve/evals/checks";
 
 export default defineEval({
-  async load() {
-    const doc = await loadYaml("evals/data/cases.yaml");
-    return (doc.evals as readonly { task: string; prompt: string; sql: string }[]).map((row) => ({
-      id: row.task,
-      input: row.prompt,
-      expected: row.sql,
-    }));
-  },
-  scores: [Text.exact()],
+  input: "Hello!",
+  checks: [Checks.didNotFail(), Checks.toolNotCalled("get_weather")],
+  scores: [],
 });
 ```
 
-Pass either `cases` or `load`, never both. The loaders are meant for fixtures, not runtime agent code.
+## Organizing with directories
+
+Identity is the file path, so directories are the grouping mechanism. `evals/weather/brooklyn-forecast.eval.ts` gets the id `weather/brooklyn-forecast`, and `eve eval weather` runs everything under `evals/weather/`. Shared constants and helpers live in sibling non-eval files (any name that doesn't end in `.eval.ts`):
+
+```text
+evals/
+├── weather/
+│   ├── shared.ts                    # helpers — not an eval
+│   ├── brooklyn-forecast.eval.ts
+│   └── no-tools-for-greetings.eval.ts
+└── smoke.eval.ts
+```
+
+## Datasets: exporting an array
 
-## Tasks
+To fan one file out over a dataset, default-export an array of `defineEval(...)` values. Eval modules are ESM, so top-level `await` can load anything. Ids derive from the file name plus a zero-padded index (`sql/0000`, `sql/0001`, …, in array order). The loaders (`loadJson`, `loadYaml` from `eve/evals/loaders`) parse fixture files relative to the app root:
 
-The eval-level `task` controls how the runner turns a data case into agent work:
+```ts title="evals/sql.eval.ts"
+import { defineEval } from "eve/evals";
+import { loadYaml } from "eve/evals/loaders";
+import { Text } from "eve/evals/scores";
 
-- `task.prompt` covers the single-string case — a template that interpolates the case input.
-- `task.messages` sets up a static multi-turn conversation.
-- `task.run` gives you imperative control flow for every case in the eval.
-- `task.parseOutput` compares a transformed result rather than the raw final message.
+const doc = await loadYaml("evals/data/cases.yaml");
+const rows = doc.evals as readonly { task: string; prompt: string; sql: string }[];
+
+export default rows.map((row) =>
+  defineEval({
+    description: row.task,
+    input: row.prompt,
+    expected: row.sql,
+    scores: [Text.exact()],
+  }),
+);
+```
 
-A task can specify at most one of `prompt`, `messages`, and `run`. When none is set, the case `input` is sent as a single user message.
+The loaders are meant for fixtures, not runtime agent code.
 
-## Scripted cases
+## Scripted evals
 
-Scripted cases define their own `run(ctx)` and do not need `input`. Use them for smoke-style behavior: multi-turn branching, HITL approvals, structured output, attachments, and multiple independent sessions.
+Scripted evals define `run(ctx)` instead of `input`. Use them for smoke-style behavior: multi-turn branching, HITL approvals, structured output, attachments, and multiple independent sessions.
 
-```ts title="evals/approvals.eval.ts"
+```ts title="evals/approve-tool.eval.ts"
 import { defineEval } from "eve/evals";
 import { Checks } from "eve/evals/checks";
 
 export default defineEval({
-  checks: [Checks.didNotFail()],
+  async run({ session }) {
+    await session.send("run `pwd`");
+    session.expectInputRequests({ toolName: "bash" });
+    const turn = await session.respondAll("approve");
+    return turn.message;
+  },
+  checks: [Checks.didNotFail(), Checks.toolCalled("bash", { input: { command: /pwd/ } })],
   scores: [],
-  cases: [
-    {
-      id: "approve-tool",
-      async run({ session }) {
-        await session.send("run `pwd`");
-        session.expectInputRequests({ toolName: "bash" });
-        const turn = await session.respondAll("approve");
-        return turn.message;
-      },
-      checks: [Checks.toolCalled("bash", { input: { command: /pwd/ } })],
-    },
-  ],
 });
 ```
 
-The return value of `run` becomes the case output that scorers grade. Throwing marks the case failed with the error message in the result.
+The return value of `run` becomes the output that scorers grade (set `parseOutput` to transform the raw result instead). Throwing marks the eval failed with the error message in the result.
 
 ## The session API
 
@@ -104,12 +103,12 @@ The return value of `run` becomes the case output that scorers grade. Throwing m
 
 Each `send` resolves to an `EveEvalTurn` carrying the turn's `message`, `events`, and status. `turn.expectOk()` throws only when the turn ended failed — a session left open for a next message is the normal end state of a successful turn.
 
-Events from every eval session are captured in the case result and artifacts. `ctx.log(message)` records debug lines into the case artifact; `--verbose` also streams them to stdout as cases run.
+Events from every eval session are captured in the result and artifacts. `ctx.log(message)` records debug lines into the eval artifact; `--verbose` also streams them to stdout as evals run.
 
 For driving sessions created outside the eval — by a channel webhook or a schedule — see [Targets and requirements](./targets).
 
 ## What to read next
 
-- [Checks](./checks): assert on what the case did
+- [Checks](./checks): assert on what the eval did
 - [Scores](./scores): grade how well it did it
 - [TypeScript client](../client/messages): the send/turn protocol eval sessions build on

package/dist/docs/public/evals/checks.mdx

@@ -1,11 +1,9 @@
 ---
 title: "Checks"
-description: "Hard assertions over runs, tool calls, and output — any failed check fails the case and the exit code."
+description: "Hard assertions over runs, tool calls, and output — any failed check fails the eval and the exit code."
 ---
 
-Checks are hard assertions. Any failed check marks the case failed and `eve eval` exits non-zero. Use them for the things that must hold — the run completed, the right tool ran, the output parses. For graded, non-fatal signals, use [scores](./scores) instead.
-
-Checks exist at the eval level (applied to every case) and the case level (appended to the eval's).
+Checks are hard assertions. Any failed check marks the eval failed and `eve eval` exits non-zero. Use them for the things that must hold — the run completed, the right tool ran, the output parses. For graded, non-fatal signals, use [scores](./scores) instead.
 
 ## Built-in checks
 
@@ -39,7 +37,7 @@ Checks.subagentCalled("weather", {
 
 ## Custom checks
 
-A custom check is a plain function receiving `{ case, result, target }` and returning `{ name, passed, message? }`:
+A custom check is a plain function receiving `{ evaluation, result, target }` and returning `{ name, passed, message? }`:
 
 ```ts
 import type { EveEvalCheck } from "eve/evals/checks";
@@ -51,7 +49,7 @@ const repliedFast: EveEvalCheck = ({ result }) => ({
 });
 ```
 
-Write a `message` for the failing path — it is what the console reporter prints under the case line and what lands in JUnit output.
+Write a `message` for the failing path — it is what the console reporter prints under the eval line and what lands in JUnit output.
 
 ## Run status and parking
 
@@ -62,4 +60,4 @@ Write a `message` for the failing path — it is what the console reporter print
 ## What to read next
 
 - [Scores](./scores): graded signals with thresholds
-- [Cases and tasks](./cases): where checks attach
+- [Cases](./cases): where checks attach

package/dist/docs/public/evals/overview.mdx

@@ -9,67 +9,83 @@ Evals exercise the same HTTP surface your users hit. The runner boots (or target
 
 ## `defineEval`
 
-Eve discovers evals under the app-root `evals/` directory, in `.eval.ts` files. The file path is the eval's identity, so you don't author an `id` or `name`.
+Eve discovers evals under the app-root `evals/` directory, in `.eval.ts` files. Each file is exactly one eval — one graded case. The file path is the eval's identity, so you don't author an `id` or `name`; directories group related evals (`evals/weather/brooklyn-forecast.eval.ts` → id `weather/brooklyn-forecast`).
 
 ```text
 my-agent/
 ├── agent/
 ├── evals/
+│   ├── evals.config.ts
 │   ├── smoke.eval.ts
-│   └── weather.eval.ts
+│   └── weather/
+│       ├── brooklyn-forecast.eval.ts
+│       └── no-tools-for-greetings.eval.ts
 └── package.json
 ```
 
-```ts title="evals/weather.eval.ts"
+```ts title="evals/weather/brooklyn-forecast.eval.ts"
 import { defineEval } from "eve/evals";
 import { Checks } from "eve/evals/checks";
 import { Run } from "eve/evals/scores";
 
 export default defineEval({
   description: "Basic message and tool-usage coverage for the weather agent.",
-  cases: [
-    { id: "brooklyn-forecast", input: "What is the weather in Brooklyn?", expected: "Sunny" },
-  ],
+  input: "What is the weather in Brooklyn?",
+  expected: "Sunny",
   checks: [Checks.didNotFail(), Checks.toolCalled("get_weather")],
   scores: [Run.didNotFail()],
 });
 ```
 
-Every eval needs `scores` (an empty array is fine) and either `cases` or `load`. The rest are optional: `description`, `task`, `checks`, `requires`, `model`, `thresholds`, `modelOptions`, `tags`, `metadata`, `maxConcurrency`, `timeoutMs`, `reporters`. The init template adds `evals/**/*.ts` to `tsconfig.json`, so your eval code type-checks alongside the app.
+Every eval needs `scores` (an empty array is fine) and either `input` (a prompt sent as a single turn) or `run` (an imperative script). The rest are optional: `description`, `expected`, `checks`, `requires`, `parseOutput`, `model`, `thresholds`, `modelOptions`, `tags`, `metadata`, `timeoutMs`, `reporters`. The init template adds `evals/**/*.ts` to `tsconfig.json`, so your eval code type-checks alongside the app.
 
-## Two grading tiers
+## `evals.config.ts`
+
+Every `evals/` directory needs exactly one `evals.config.ts` at its root. It declares the defaults every eval shares — most importantly the `model` used by model-backed scorers, so you don't repeat it in each file:
+
+```ts title="evals/evals.config.ts"
+import { defineEvalConfig } from "eve/evals";
+import { Braintrust } from "eve/evals/reporters";
+
+export default defineEvalConfig({
+  model: "openai/gpt-5.4-mini",
+  reporters: [Braintrust({ projectName: "my-agent" })],
+});
+```
 
-Evals grade a case on two distinct tiers:
+`model` is required; `reporters`, `maxConcurrency`, and `timeoutMs` are optional. Config `reporters` observe every eval in the run — set one `Braintrust()` here instead of adding it to each eval. CLI flags (`--max-concurrency`, `--timeout`) and per-eval values take precedence over the config defaults. An eval that needs a different judge model overrides it with its own `model`; otherwise the config `model` applies.
+
+## Two grading tiers
 
-- **[Checks](./checks) are hard assertions.** Any failed check marks the case failed and `eve eval` exits non-zero. Use them for the things that must hold — the run completed, the right tool ran, the output parses.
-- **[Scores](./scores) are soft data.** They land in reports and artifacts, and a below-threshold score marks the case `scored` — visible but not fatal, unless you pass `--strict`.
+Evals are graded on two distinct tiers:
 
-Both exist at the eval level (applied to every case) and the case level (appended to the eval's).
+- **[Checks](./checks) are hard assertions.** Any failed check marks the eval failed and `eve eval` exits non-zero. Use them for the things that must hold — the run completed, the right tool ran, the output parses.
+- **[Scores](./scores) are soft data.** They land in reports and artifacts, and a below-threshold score marks the eval `scored` — visible but not fatal, unless you pass `--strict`.
 
 ## Run it
 
 ```bash
 eve eval                       # run all discovered evals against a local dev server
-eve eval weather               # run selected evals
+eve eval weather               # run one eval, or every eval under evals/weather/
 eve eval --url https://<app>   # target an existing server or deployment
 ```
 
-Exit code `0` means every case passed its checks. See [Running evals](./running) for the full flag list, exit codes, and CI guidance.
+Exit code `0` means every eval passed its checks. See [Running evals](./running) for the full flag list, exit codes, and CI guidance.
 
 ## A good baseline
 
-Most apps do fine with a small smoke eval. Assert behavior with `Checks.didNotFail()` plus one or two content checks, keep case fixtures in `evals/data/`, and only reach for Braintrust once you actually need shared result review or experiment history. In CI, run `eve eval --strict` so threshold misses fail the build too.
+Most apps do fine with a few small smoke evals. Assert behavior with `Checks.didNotFail()` plus one or two content checks, keep dataset fixtures in `evals/data/`, and only reach for Braintrust once you actually need shared result review or experiment history. In CI, run `eve eval --strict` so threshold misses fail the build too.
 
 The rest of this section covers each piece:
 
-- [Cases and tasks](./cases): data cases, loaders, and scripted multi-turn cases
+- [Cases](./cases): prompt evals, scripted multi-turn evals, and dataset fan-out
 - [Checks](./checks): hard assertions over runs, tools, and output
 - [Scores](./scores): deterministic and LLM-judged scorers, with thresholds
-- [Targets and requirements](./targets): local vs remote targets, and gating cases on capabilities
+- [Targets and requirements](./targets): local vs remote targets, and gating evals on capabilities
 - [Reporters](./reporters): Braintrust experiments and JUnit XML
 - [Running evals](./running): the `eve eval` CLI, exit codes, and artifacts
 
 ## What to read next
 
-- [Cases and tasks](./cases): author your first cases
+- [Cases](./cases): author your first evals
 - [Tools](../tools): the surface most evals assert on

package/dist/docs/public/evals/reporters.mdx

@@ -3,27 +3,43 @@ title: "Reporters"
 description: "Ship eval results to Braintrust experiments or JUnit XML — Eve runs and scores everything itself."
 ---
 
-Eve runs and scores everything itself; reporters just ship the results out. The CLI prints a console summary by default — one line per case, failed checks with their messages — and eval-level reporters from `eve/evals/reporters` add destinations on top.
+Eve runs and scores everything itself; reporters just ship the results out. The CLI prints a console summary by default — one line per eval, failed checks with their messages — and reporters from `eve/evals/reporters` add destinations on top.
+
+Reporters attach in two places. Declare them in `evals.config.ts` to observe **every** eval in the run — the usual choice for a shared destination like one Braintrust experiment, so you don't repeat the reporter in each file. Or list them on an individual eval's `reporters` to scope a destination to that eval (or to a group of evals that share one instance).
 
 ## Braintrust
 
-`Braintrust(...)` uploads eval results to Braintrust experiments:
+`Braintrust(...)` uploads eval results to Braintrust experiments. Put one instance in the config so it covers the whole run:
+
+```ts title="evals/evals.config.ts"
+import { defineEvalConfig } from "eve/evals";
+import { Braintrust } from "eve/evals/reporters";
 
-```ts title="evals/weather.eval.ts"
+export default defineEvalConfig({
+  model: "openai/gpt-5.4-mini",
+  reporters: [Braintrust({ projectName: "weather-agent" })],
+});
+```
+
+Need a destination for only some evals? Attach it per eval instead:
+
+```ts title="evals/brooklyn-forecast.eval.ts"
 import { defineEval } from "eve/evals";
 import { Braintrust } from "eve/evals/reporters";
 import { Run } from "eve/evals/scores";
 
 export default defineEval({
-  cases: [{ id: "brooklyn-forecast", input: "What is the weather in Brooklyn?" }],
+  input: "What is the weather in Brooklyn?",
   scores: [Run.didNotFail()],
   reporters: [Braintrust({ projectName: "weather-agent" })],
 });
 ```
 
-The config takes an optional `projectName` and `experimentName`, plus a base experiment (by name or id) to diff against. Checks log as binary scores under a `check:` prefix so experiments diff check regressions the same way they diff score regressions. Eval and case `metadata` ride along to reporters.
+The reporter config takes an optional `projectName` and `experimentName`, plus a base experiment (by name or id) to diff against. Checks log as binary scores under a `check:` prefix so experiments diff check regressions the same way they diff score regressions. Eval `metadata` rides along to reporters.
+
+A reporter instance observes the evals that reference it: share one instance across several evals — the config, a `shared.ts` export, or every entry of a dataset array — and their results land in a single experiment. Listing the same config reporter on an eval too does not double-report it.
 
-Braintrust needs its SDK installed in the app and credentials in the environment. Pass `--skip-report` to run the eval without shipping results — useful locally when iterating.
+Braintrust needs its SDK installed in the app and credentials in the environment. Pass `--skip-report` to run the eval without shipping results (this also suppresses config reporters) — useful locally when iterating.
 
 ## JUnit
 
@@ -33,7 +49,7 @@ Braintrust needs its SDK installed in the app and credentials in the environment
 eve eval --strict --junit .eve/junit.xml
 ```
 
-Failed checks and execution errors land as failure messages on the matching test case, so CI surfaces them inline.
+Each eval becomes one `<testcase>` named by its path-derived id; failed checks and execution errors land as failure messages on the matching test case, so CI surfaces them inline.
 
 ## Custom reporters
 

package/dist/docs/public/evals/running.mdx

@@ -3,39 +3,40 @@ title: "Running evals"
 description: "The eve eval CLI: flags, filters, exit codes, artifacts, and how to wire evals into CI."
 ---
 
-`eve eval` discovers every `.eval.ts` file under `evals/`, boots a local dev server (or targets a remote one), runs the cases, and prints a per-case summary.
+`eve eval` discovers every `.eval.ts` file under `evals/`, boots a local dev server (or targets a remote one), runs the evals concurrently, and prints a per-eval summary.
 
 ```bash
 eve eval                       # run all discovered evals locally
-eve eval weather smoke         # run selected evals
+eve eval weather smoke         # run selected evals (an id, or a directory prefix)
 eve eval --url https://<app>   # target a remote app instead of a local host
 eve eval --mock-models         # local dev target uses deterministic mock models
-eve eval --tag fast            # only cases (or evals) carrying a tag
-eve eval --case brooklyn-forecast  # only specific case ids
+eve eval --tag fast            # only evals carrying a tag
 eve eval --strict              # below-threshold scores also fail the exit code
 eve eval --no-skips            # unmet requirements fail instead of skipping
-eve eval --timeout 60000       # per-case timeout in milliseconds
-eve eval --max-concurrency 4   # cap concurrent cases per eval
+eve eval --timeout 60000       # per-eval timeout in milliseconds
+eve eval --max-concurrency 4   # cap concurrent eval executions (default 8)
 eve eval --junit .eve/junit.xml  # write JUnit XML
-eve eval --list                # print discovered evals and cases without running
-eve eval --verbose             # stream per-case ctx.log lines to stdout
+eve eval --list                # print discovered evals without running
+eve eval --verbose             # stream per-eval ctx.log lines to stdout
 eve eval --json                # machine-readable output
-eve eval --skip-report         # skip eval-defined reporters (e.g. Braintrust)
+eve eval --skip-report         # skip config and eval-defined reporters (e.g. Braintrust)
 ```
 
+Positional ids match exactly or by directory prefix: `eve eval weather` runs `evals/weather.eval.ts`, every eval under `evals/weather/`, and every entry of an array-exported `weather.eval.ts`.
+
 ## Exit codes
 
 | Code | Means                                                                            |
 | ---- | -------------------------------------------------------------------------------- |
-| `0`  | Every case passed its checks (and thresholds, under `--strict`)                  |
-| `1`  | Any case failed — a failed check, an execution error, or a strict threshold miss |
+| `0`  | Every eval passed its checks (and thresholds, under `--strict`)                  |
+| `1`  | Any eval failed — a failed check, an execution error, or a strict threshold miss |
 | `2`  | Configuration error                                                              |
 
 Unmet [requirements](./targets) skip visibly without affecting the exit code unless you pass `--no-skips`.
 
 ## Artifacts
 
-Each run drops per-eval artifacts under `.eve/evals/<timestamp>-<eval-id>/`, including per-case check results, verdicts, captured event streams, and `ctx.log` lines. The console output stays tight on purpose; when a case fails, the artifact has the full story.
+Each run drops artifacts under `.eve/evals/<timestamp>/`: a run `summary.json`, a `results.jsonl` index, and per-eval check results, verdicts, captured event streams, and `ctx.log` lines under `evals/`. The console output stays tight on purpose; when an eval fails, the artifact has the full story.
 
 ## CI
 
@@ -46,8 +47,8 @@ eve eval --strict --mock-models --junit .eve/junit.xml
 ```
 
 - `--strict` turns threshold misses into failures, so score regressions block the merge.
-- `--mock-models` keeps the default leg deterministic and credential-free. Put real-model cases in their own eval files gated on `requires: ["env:..."]`, and add `--no-skips` on legs that must prove those ran.
-- `--junit` gives the CI provider per-case annotations; upload the `.eve/evals/` directory as a failure artifact for the full event streams.
+- `--mock-models` keeps the default leg deterministic and credential-free. Put real-model evals in their own files gated on `requires: ["env:..."]`, and add `--no-skips` on legs that must prove those ran.
+- `--junit` gives the CI provider per-eval annotations; upload the `.eve/evals/` directory as a failure artifact for the full event streams.
 
 Against a deployed app, swap `--mock-models` for `--url`:
 

package/dist/docs/public/evals/scores.mdx

@@ -1,13 +1,13 @@
 ---
 title: "Scores"
-description: "Grade eval cases with deterministic scorers or LLM judges, and gate them with thresholds."
+description: "Grade evals with deterministic scorers or LLM judges, and gate them with thresholds."
 ---
 
-Scores are soft data. They land in reports and artifacts, and a below-threshold score marks the case `scored` — visible but not fatal, unless you pass `--strict`. Use them to grade quality fractionally where a [check](./checks) would assert it absolutely.
+Scores are soft data. They land in reports and artifacts, and a below-threshold score marks the eval `scored` — visible but not fatal, unless you pass `--strict`. Use them to grade quality fractionally where a [check](./checks) would assert it absolutely.
 
 ## Choosing a scorer
 
-Scorers live in namespaces on `eve/evals/scores`. Pick the cheapest one that captures what "correct" means here. The deterministic scorers run instantly for free; an LLM judge runs once per case and burns tokens, so save it for when nothing simpler will do.
+Scorers live in namespaces on `eve/evals/scores`. Pick the cheapest one that captures what "correct" means here. The deterministic scorers run instantly for free; an LLM judge runs once per eval and burns tokens, so save it for when nothing simpler will do.
 
 | Need                                                       | Use                                                                                           |
 | ---------------------------------------------------------- | --------------------------------------------------------------------------------------------- |
@@ -21,7 +21,7 @@ Scorers live in namespaces on `eve/evals/scores`. Pick the cheapest one that cap
 | LLM-judged SQL semantic equivalence                        | `Autoevals.sql()`                                                                             |
 | LLM-judged free-form criteria (no `expected` to match)     | `Autoevals.closedQA({ criteria: "..." })`                                                     |
 
-Each scorer gets the flattened `input`, `output`, and `expected` strings along with the full case and task result — including derived facts: typed tool calls (name, input, output, error state), subagent calls, HITL input requests, and whether the run parked. `Run.usedTool` accepts the same matcher options as `Checks.toolCalled`. Return `null` from a scorer to skip a case.
+Each scorer gets the flattened `input`, `output`, and `expected` strings along with the full eval and task result — including derived facts: typed tool calls (name, input, output, error state), subagent calls, HITL input requests, and whether the run parked. `Run.usedTool` accepts the same matcher options as `Checks.toolCalled`. Return `null` from a scorer to skip it.
 
 ## Thresholds
 
@@ -32,23 +32,24 @@ import { defineEval } from "eve/evals";
 import { Run, Text } from "eve/evals/scores";
 
 export default defineEval({
-  cases: [{ id: "hello", input: "Hello", expected: "Hello" }],
+  input: "Hello",
+  expected: "Hello",
   scores: [Run.didNotFail(), Text.includes()],
   thresholds: { "run.didNotFail": 1, "text.includes": 0.5 },
 });
 ```
 
-A case below a threshold gets the `scored` verdict — reported, but only fatal under `eve eval --strict`.
+An eval below a threshold gets the `scored` verdict — reported, but only fatal under `eve eval --strict`.
 
 ## The scorer model
 
-`model` is only required when a model-backed scorer (one of the `Autoevals` wrappers) is present without its own per-scorer model override — and it's the scorer model, not the agent's. Eve only uses it for model-backed scoring, never to swap out the agent under test. Pass a string id (e.g. `"anthropic/claude-opus-4.8"`) to route through the Vercel AI Gateway, or hand it an AI SDK model instance to use that directly.
+Model-backed scorers (the `Autoevals` wrappers) need a judge model — the scorer model, not the agent's. Eve only uses it for scoring, never to swap out the agent under test. The default lives in [`evals.config.ts`](./overview#evalsconfigts) as the required `model`, so most evals inherit it without setting anything. Pass a string id (e.g. `"anthropic/claude-opus-4.8"`) to route through the Vercel AI Gateway, or an AI SDK model instance to use it directly.
 
-For provider-specific scorer-model settings, use `modelOptions.providerOptions`. Individual Autoevals scorers can also take their own `model` / `modelOptions`, which win over the eval default.
+Override the default on a single eval by setting that eval's own `model`. For provider-specific scorer-model settings, use `modelOptions.providerOptions`. Individual Autoevals scorers can also take their own `model` / `modelOptions`, which win over both the eval and config defaults.
 
 ## Concurrency and timeouts
 
-`maxConcurrency` caps parallelism and `timeoutMs` bounds each case. Leave them off and `eve eval` runs up to 8 cases per eval at once. Lower `maxConcurrency` when cases contend for a shared resource — a rate-limited connection, or a sandbox-heavy fixture.
+`timeoutMs` bounds one eval's execution: the eval's own value wins, then `evals.config.ts`'s default, and `eve eval --timeout <ms>` overrides both for a run. The runner executes up to 8 evals at once — set a default `maxConcurrency` in `evals.config.ts` or pass `--max-concurrency <n>` (which wins) to change that, and lower it when evals contend for a shared resource: a rate-limited connection, or a sandbox-heavy fixture.
 
 ## What to read next
 

package/dist/docs/public/evals/targets.mdx

@@ -1,30 +1,25 @@
 ---
 title: "Targets and requirements"
-description: "Point evals at a local dev server or a deployment, and gate cases on target capabilities with requires."
+description: "Point evals at a local dev server or a deployment, and gate evals on target capabilities with requires."
 ---
 
 An eval target is always an HTTP URL. `eve eval` starts a local dev server, while `eve eval --url <url>` runs against an existing server or deployment — the same eval files work for both, which is what makes evals usable as end-to-end tests in CI.
 
-The runner polls `/eve/v1/health`, verifies `/eve/v1/info`, and exposes the live target as `ctx.target` inside scripted cases.
+The runner polls `/eve/v1/health`, verifies `/eve/v1/info`, and exposes the live target as `ctx.target` inside scripted evals.
 
 ## Target helpers
 
-```ts title="evals/schedules.eval.ts"
+```ts title="evals/heartbeat.eval.ts"
 import { defineEval } from "eve/evals";
 
 export default defineEval({
   requires: ["mockModels", "devRoutes"],
   scores: [],
-  cases: [
-    {
-      id: "heartbeat",
-      async run({ target }) {
-        const { sessionIds } = await target.dispatchSchedule("heartbeat");
-        const session = await target.attachSession(sessionIds[0]!);
-        return session.events;
-      },
-    },
-  ],
+  async run({ target }) {
+    const { sessionIds } = await target.dispatchSchedule("heartbeat");
+    const session = await target.attachSession(sessionIds[0]!);
+    return session.events;
+  },
 });
 ```
 
@@ -36,7 +31,7 @@ Sessions attached this way are full `EveEvalSession`s: you can keep driving them
 
 ## Requirements
 
-Use `requires` to declare assumptions the runner verifies before executing a case:
+Use `requires` to declare assumptions the runner verifies before executing an eval:
 
 | Requirement    | Means                                                                 |
 | -------------- | --------------------------------------------------------------------- |
@@ -44,13 +39,13 @@ Use `requires` to declare assumptions the runner verifies before executing a cas
 | `"devRoutes"`  | `/eve/v1/info` reports dev-only routes are mounted                    |
 | `"env:NAME"`   | The eval process has environment variable `NAME` set                  |
 
-Eval-level requirements apply to every case; case-level requirements append. Unmet requirements produce a visible `skipped` verdict and do not affect the exit code. Pass `--no-skips` when a CI leg must prove full coverage.
+Unmet requirements produce a visible `skipped` verdict and do not affect the exit code. Pass `--no-skips` when a CI leg must prove full coverage.
 
 ## Mock models
 
-Deterministic evals — the kind you want in CI — should not depend on a live model. `eve eval --mock-models` starts the local dev server with deterministic authored models, and `requires: ["mockModels"]` makes the dependency explicit so the case skips instead of flaking anywhere else.
+Deterministic evals — the kind you want in CI — should not depend on a live model. `eve eval --mock-models` starts the local dev server with deterministic authored models, and `requires: ["mockModels"]` makes the dependency explicit so the eval skips instead of flaking anywhere else.
 
-`--mock-models` is invalid with `--url` because remote target capabilities are discovered, not set by the runner. For cases that genuinely need a real model — judging nuanced behavior, exercising provider-side tools — gate them on credentials instead (`requires: ["env:AI_GATEWAY_API_KEY"]`) and keep them in their own eval file so a tag filter can select or exclude them.
+`--mock-models` is invalid with `--url` because remote target capabilities are discovered, not set by the runner. For evals that genuinely need a real model — judging nuanced behavior, exercising provider-side tools — gate them on credentials instead (`requires: ["env:AI_GATEWAY_API_KEY"]`) and keep them in their own eval files so a tag filter can select or exclude them.
 
 ## What to read next
 

package/dist/docs/public/reference/cli.md

@@ -96,19 +96,18 @@ Local dev keeps immutable runtime source snapshots under `.eve/dev-runtime/snaps
 eve eval [evalId...] [--url <url>] [options]
 ```
 
-Runs all discovered evals when no eval ids are given. Exits `0` when every case passed its checks, `1` when any case failed (a failed check, an execution error, or a `--strict` threshold miss), `2` on configuration errors.
-
-| Flag                    | Effect                                                |
-| ----------------------- | ----------------------------------------------------- |
-| `--url <url>`           | Remote agent URL (skip local host startup)            |
-| `--tag <tag...>`        | Run only cases (or evals) carrying a tag              |
-| `--case <id...>`        | Run only specific case ids                            |
-| `--strict`              | Below-threshold scores also fail the exit code        |
-| `--list`                | Print discovered evals and cases without running them |
-| `--timeout <ms>`        | Per-case timeout in milliseconds                      |
-| `--max-concurrency <n>` | Max concurrent case executions per eval               |
-| `--json`                | Output results as JSON                                |
-| `--skip-report`         | Skip eval-defined reporters (e.g. Braintrust)         |
+Runs all discovered evals when no eval ids are given; ids match exactly or by directory prefix (`eve eval weather` runs everything under `evals/weather/`). Exits `0` when every eval passed its checks, `1` when any eval failed (a failed check, an execution error, or a `--strict` threshold miss), `2` on configuration errors.
+
+| Flag                    | Effect                                         |
+| ----------------------- | ---------------------------------------------- |
+| `--url <url>`           | Remote agent URL (skip local host startup)     |
+| `--tag <tag...>`        | Run only evals carrying a tag                  |
+| `--strict`              | Below-threshold scores also fail the exit code |
+| `--list`                | Print discovered evals without running them    |
+| `--timeout <ms>`        | Per-eval timeout in milliseconds               |
+| `--max-concurrency <n>` | Max concurrent eval executions (default 8)     |
+| `--json`                | Output results as JSON                         |
+| `--skip-report`         | Skip eval-defined reporters (e.g. Braintrust)  |
 
 See [Evals](../evals/overview) for authoring evals.