eve 0.6.0-beta.14 → 0.6.0-beta.16

package/CHANGELOG.md

@@ -1,5 +1,28 @@
 # eve
 
+## 0.6.0-beta.16
+
+### Minor Changes
+
+- 5a6ac17: Add a required `evals/evals.config.ts` (authored with `defineEvalConfig`) that declares run-wide eval defaults: a mandatory scorer `model`, plus optional run-level `reporters`, `maxConcurrency`, and `timeoutMs`. Model-backed scorers now fall back to the config `model`, so `model` is optional on `defineEval` and a shared reporter (e.g. one `Braintrust()`) no longer needs to be repeated in every eval. CLI flags and per-eval values still take precedence over the config defaults.
+- 5a6ac17: `defineEval` is now always a single case, with identity fully derived from the file path — `cases`, `load`, `task`, per-case `id`, and `maxConcurrency` are removed. Declare `input` or `run` (plus `expected`, `checks`, `scores`, `parseOutput`, …) at the top level, organize related evals with directory nesting (`evals/runtime/multi-turn.eval.ts` → `runtime/multi-turn`), and default-export an array of `defineEval(...)` values for dataset fan-out (ids get a zero-padded index suffix, e.g. `weather/0000`). The runner now executes eval files concurrently (default 8, `--max-concurrency`), positional `eve eval` ids match by directory prefix, `--case` is removed, reporters use a run-level lifecycle (`onRunStart`/`onEvalComplete`/`onRunComplete`), check/scorer args expose `evaluation` instead of `case`, and artifacts land under one `.eve/evals/<timestamp>/` directory per run.
+
+### Patch Changes
+
+- a8363e6: Fix `authorization.required` not being emitted when a tool combines `needsApproval` with interactive auth. Approval-resume auth signals are now routed through the authorization park path instead of being replayed to the model as a plain tool result.
+- a8363e6: Authorization-pending tool results no longer expose OAuth URLs, user codes, or hook URLs to the model. Channels still receive full `authorization.required` events.
+
+## 0.6.0-beta.15
+
+### Minor Changes
+
+- 0a8d93b: Add `eve init <name>` for non-interactive agent creation, with optional Web Chat through `--web`. The command installs dependencies, initializes Git, and starts the development server, but does not provision or deploy a Vercel project.
+- f733fe5: The deterministic mock model adapter (`EVE_MOCK_AUTHORED_MODELS=1`) now synthesizes tool inputs for string properties anchored to quoted spans in the message (e.g. `with label "x"`), honors exact-reply fixture directives (`reply with the exact string … and nothing else`, `include the exact token … verbatim`) found in the current turn's user messages and per-turn client context, discovers static skill advertisements embedded in instruction text (not just standalone announcements), derives skill activation from `load_skill` calls in history so skills are never re-loaded in a loop, and no longer matches `load_skill` by explicit name. This widens what agent smoke evals can assert deterministically without a real model.
+
+### Patch Changes
+
+- 0a8d93b: Scaffolded projects now carry a `packageExtensions` entry in `pnpm-workspace.yaml` that restores eve's missing `oxc-parser` dependency, so fresh agents boot against the published `eve` 0.6.0-beta.13/14 builds that import it without declaring it. The entry lives in `pnpm-workspace.yaml` because pnpm 11 reads settings only from there, not from the `package.json` `pnpm` field. Remove it from the template once the scaffolded `eve` range only resolves to versions that declare `oxc-parser` themselves.
+
 ## 0.6.0-beta.14
 
 ### Minor Changes

package/README.md

@@ -52,7 +52,7 @@ Every authored directory has a typed helper. Import each from the matching subpa
 | `eveChannel(...)`, `slackChannel(...)`, `vercelOidc(...)`                                                           | `eve/channels/eve`, `/slack`, `/auth` | reused from `channels/<name>.ts`                 |
 | `defineSandbox(...)`                                                                                                | `eve/sandbox`                         | `sandbox.ts` (or `sandbox/sandbox.ts`)           |
 | `defineSchedule(...)`                                                                                               | `eve/schedules`                       | `schedules/<name>.ts` (or `schedules/<name>.md`) |
-| `defineEval(...)`                                                                                                   | `eve/evals`                           | `evals/<name>.eval.ts`                           |
+| `defineEval(...)`, `defineEvalConfig(...)`                                                                          | `eve/evals`                           | `evals/<name>.eval.ts`, `evals/evals.config.ts`  |
 
 Runtime accessors live on the subpath that owns the concern:
 
@@ -105,15 +105,16 @@ export default defineAgent({
 ## Quick Start
 
 ```bash
-pnpm create eve@beta
-cd my-agent
-pnpm dev
+pnpm dlx eve@beta init my-agent
 ```
 
-The wizard scaffolds the project, picks a model, and (for the REPL channel) installs dependencies and starts the dev server for you. To scaffold into the current empty directory, run `pnpm create eve@beta .`.
+`eve init` writes a new agent with Eve's default model. Pass `--web` to add the
+Web Chat application. It installs dependencies, initializes Git, and starts the
+development server. It does not create a Vercel project or deploy the agent.
 
 CLI commands:
 
+- `eve init <name>` — create a new agent
 - `eve info` — discovery results and compiled artifacts
 - `eve build` — compile `.eve/` and build the host output
 - `eve start` — serve the built `.output/` app

package/dist/docs/evals-v2-plan.md

@@ -812,13 +812,13 @@ weather-result-reaches-parent` reruns it in isolation.
 
 ### CI
 
-`.github/workflows/smoke.yml` discovery changes from globbing
+`.github/workflows/e2e.yml` discovery changes from globbing
 `e2e/tests/*/*.ts` to globbing fixture apps with `evals/` directories. Each
 matrix leg provisions, then runs the evals against the resulting URL:
 
 ```sh
 node e2e/provision/<group>.ts &        # build + sidecars + env + eve start
-pnpm --filter <fixture-app> exec eve eval --strict --json --url "$TARGET_URL"
+pnpm --filter <fixture-app> exec eve eval --strict --url "$TARGET_URL"
 ```
 
 twice (direct / `EVE_EXPERIMENTAL_CODE_MODE=1` set by the provisioner),
@@ -842,7 +842,7 @@ Phase 5.
 ## Implementation phases
 
 Each phase ships independently, keeps `pnpm test` green, and includes docs
-(`docs/public/advanced/evals.md`) + changesets per repo policy.
+(now the top-level `docs/public/evals/` section) + changesets per repo policy.
 
 ### Phase 1 — Assertions and pass/fail (no breaking interaction changes)
 
@@ -906,6 +906,31 @@ addresses. Nothing in phase 4 deploys anywhere.
     area policy; shrink `e2e/lib`/`e2e/target` into `e2e/provision/`; update
     `e2e/README.md` and AGENTS.md smoke-test guidance to point at `eve eval`.
 
+> **Decisions made during the phase-4 migration (June 2026):**
+>
+> - **Hybrid mock/real split.** Evals default to the deterministic mock
+>   (`requires: ["mockModels"]`). Behaviors the mock cannot express — the
+>   built-in `agent` tool, Workflow fan-out, provider-executed `web_search`,
+>   url-file attachment reads — live in separate `real-model`-tagged eval
+>   files run against a non-mock server only when model credentials are
+>   present (provisioners log a visible skip otherwise). Net: real-model CI
+>   legs dropped from ~156 to ~7 cases.
+> - **Code-mode e2e dropped entirely.** No `EVE_EXPERIMENTAL_CODE_MODE`
+>   matrix dimension and no codemode group port; the `agent-codemode` fixture
+>   was deleted. Purpose-built code-mode fixtures/evals will be designed
+>   later.
+> - **Mock adapter extensions** (in `eve`): anchored quoted-span tool-input
+>   synthesis, exact-reply directives in current-turn user text (proves
+>   clientContext delivery), embedded static-skill advert discovery, and
+>   history-derived skill-activation tracking.
+> - **Coverage consciously dropped:** semantic multi-turn recall and the
+>   channel-rekey marker recall (pure model behaviors), `subagent-output-schema`
+>   moved to the real leg, and the eval-CLI self-smoke (`evals.ts`) — now the
+>   CI mechanism itself.
+> - Sidecar fake providers expose a `GET /_probe/requests` inspection
+>   endpoint (`e2e/lib/http-test-server.ts`) so evals assert captured
+>   traffic across the provisioner process boundary.
+
 ### Phase 5 — Remote leg: deploy fixtures and eval the deployment
 
 After phase 4, CI gains a second e2e variant that runs the **same eval
@@ -918,7 +943,7 @@ skip.
     preview env sets `EVE_MOCK_AUTHORED_MODELS=1` (determinism, zero model
     spend) and the workflow backend credentials.
 16. CI job: deploy the preview, capture the URL, then
-    `eve eval --strict --json --junit ... --url "$DEPLOY_URL"` with
+    `eve eval --strict --junit ... --url "$DEPLOY_URL"` with
     `VERCEL_AUTOMATION_BYPASS_SECRET` in the job env (the eval client's
     existing remote-auth cascade handles protection bypass / OIDC). No
     `--no-skips` on this leg: schedule evals (`devRoutes`) and local-sidecar

package/dist/docs/public/README.md

@@ -30,7 +30,7 @@ Read in this order:
 14. [TypeScript Client](./client/overview.mdx)
 15. [Subagents](./subagents.mdx)
 16. [Schedules](./schedules.mdx)
-17. [Evals](./advanced/evals.md)
+17. [Evals](./evals/overview.mdx)
 18. [Auth And Route Protection](./advanced/auth-and-route-protection.md)
 19. [Vercel Deployment](./advanced/deployment.md)
 20. [CLI, Build, And Debugging](./reference/cli.md)

package/dist/docs/public/advanced/auth-and-route-protection.md

@@ -184,7 +184,7 @@ export default defineChannel({
 
 ## Replace `placeholderAuth` before production
 
-`pnpm create eve@beta` sometimes scaffolds `agent/channels/eve.ts` with a `placeholderAuth()` guardrail:
+`eve init` scaffolds `agent/channels/eve.ts` with a `placeholderAuth()` guardrail:
 
 ```ts
 import { eveChannel } from "eve/channels/eve";

package/dist/docs/public/advanced/instrumentation.md

@@ -162,4 +162,4 @@ When `eve build` fails on discovery errors, the CLI prints the full diagnostics
 - [`agent.ts`](../agent-config)
 - [Hooks](./hooks): observe the runtime event stream
 - [Dev TUI](./dev-tui): drive the agent locally
-- [Evals](./evals): repeatable scored checks
+- [Evals](../evals/overview): repeatable scored checks

package/dist/docs/public/advanced/meta.json

@@ -14,7 +14,6 @@
     "deployment",
     "instrumentation",
     "dev-tui",
-    "evals",
     "execution-model-and-durability",
     "sessions-runs-and-streaming"
   ]

package/dist/docs/public/channels/eve.mdx

@@ -36,7 +36,7 @@ The `auth` option decides who can call these routes. The built-in helpers are me
 
 Neither admits browser users or external clients in production. For a public app, wire the channel to your own auth (Clerk, Auth.js, or your own OIDC/JWT verification).
 
-`pnpm create eve@beta` scaffolds an `agent/channels/eve.ts` with a production placeholder so you replace it before going live. The generated channel allows Vercel OIDC and localhost, and includes `placeholderAuth()`, which returns a setup-focused 401 in production until you swap it for real auth. Delete the file and Eve falls back to `[localDev(), vercelOidc()]`, which still does not admit browser users in production.
+`eve init` scaffolds an `agent/channels/eve.ts` with a production placeholder so you replace it before going live. The generated channel allows Vercel OIDC and localhost, and includes `placeholderAuth()`, which returns a setup-focused 401 in production until you swap it for real auth. Delete the file and Eve falls back to `[localDev(), vercelOidc()]`, which still does not admit browser users in production.
 
 For the full auth model and helper list, see [Auth & route protection](../advanced/auth-and-route-protection).
 

package/dist/docs/public/evals/cases.mdx

@@ -0,0 +1,114 @@
+---
+title: "Cases"
+description: "Author prompt evals, script multi-turn evals with run(ctx), and fan one file out over a dataset."
+---
+
+Each eval file is one graded case: the runner executes it against the target, captures every event, and applies [checks](./checks) and [scores](./scores) to the result. An eval is either a prompt eval (`input`) or a scripted eval (`run`).
+
+## Prompt evals
+
+Prompt evals pair an `input` with an optional `expected`. `input` can be a string or an object (objects are `JSON.stringify`d); the runner sends it as a single user turn. `expected` is optional, which is handy when you only care about behavior:
+
+```ts title="evals/weather/brooklyn-forecast.eval.ts"
+import { defineEval } from "eve/evals";
+import { Checks } from "eve/evals/checks";
+
+export default defineEval({
+  input: "What is the weather in Brooklyn?",
+  expected: "Sunny",
+  checks: [Checks.didNotFail()],
+  scores: [],
+});
+```
+
+```ts title="evals/weather/no-tools-for-greetings.eval.ts"
+import { defineEval } from "eve/evals";
+import { Checks } from "eve/evals/checks";
+
+export default defineEval({
+  input: "Hello!",
+  checks: [Checks.didNotFail(), Checks.toolNotCalled("get_weather")],
+  scores: [],
+});
+```
+
+## Organizing with directories
+
+Identity is the file path, so directories are the grouping mechanism. `evals/weather/brooklyn-forecast.eval.ts` gets the id `weather/brooklyn-forecast`, and `eve eval weather` runs everything under `evals/weather/`. Shared constants and helpers live in sibling non-eval files (any name that doesn't end in `.eval.ts`):
+
+```text
+evals/
+├── weather/
+│   ├── shared.ts                    # helpers — not an eval
+│   ├── brooklyn-forecast.eval.ts
+│   └── no-tools-for-greetings.eval.ts
+└── smoke.eval.ts
+```
+
+## Datasets: exporting an array
+
+To fan one file out over a dataset, default-export an array of `defineEval(...)` values. Eval modules are ESM, so top-level `await` can load anything. Ids derive from the file name plus a zero-padded index (`sql/0000`, `sql/0001`, …, in array order). The loaders (`loadJson`, `loadYaml` from `eve/evals/loaders`) parse fixture files relative to the app root:
+
+```ts title="evals/sql.eval.ts"
+import { defineEval } from "eve/evals";
+import { loadYaml } from "eve/evals/loaders";
+import { Text } from "eve/evals/scores";
+
+const doc = await loadYaml("evals/data/cases.yaml");
+const rows = doc.evals as readonly { task: string; prompt: string; sql: string }[];
+
+export default rows.map((row) =>
+  defineEval({
+    description: row.task,
+    input: row.prompt,
+    expected: row.sql,
+    scores: [Text.exact()],
+  }),
+);
+```
+
+The loaders are meant for fixtures, not runtime agent code.
+
+## Scripted evals
+
+Scripted evals define `run(ctx)` instead of `input`. Use them for smoke-style behavior: multi-turn branching, HITL approvals, structured output, attachments, and multiple independent sessions.
+
+```ts title="evals/approve-tool.eval.ts"
+import { defineEval } from "eve/evals";
+import { Checks } from "eve/evals/checks";
+
+export default defineEval({
+  async run({ session }) {
+    await session.send("run `pwd`");
+    session.expectInputRequests({ toolName: "bash" });
+    const turn = await session.respondAll("approve");
+    return turn.message;
+  },
+  checks: [Checks.didNotFail(), Checks.toolCalled("bash", { input: { command: /pwd/ } })],
+  scores: [],
+});
+```
+
+The return value of `run` becomes the output that scorers grade (set `parseOutput` to transform the raw result instead). Throwing marks the eval failed with the error message in the result.
+
+## The session API
+
+`ctx.session` is the primary `EveEvalSession`; `ctx.newSession()` creates another independent session against the same target:
+
+- `session.send(input)` sends a turn and waits for it to settle. It accepts the same input as `ClientSession.send()` — a string or a structured message.
+- `session.sendFile(text, path, mediaType?)` attaches a local file as a data URL.
+- `session.expectInputRequests(filter?)` asserts the previous turn parked on HITL input and returns the pending requests.
+- `session.respondAll(optionId)` answers every pending input request with the same option and sends the responses as the next turn.
+- `session.events` is the full typed event stream captured so far.
+
+Each `send` resolves to an `EveEvalTurn` carrying the turn's `message`, `events`, and status. `turn.expectOk()` throws only when the turn ended failed — a session left open for a next message is the normal end state of a successful turn.
+
+Events from every eval session are captured in the result and artifacts. `ctx.log(message)` records debug lines into the eval artifact; `--verbose` also streams them to stdout as evals run.
+
+For driving sessions created outside the eval — by a channel webhook or a schedule — see [Targets and requirements](./targets).
+
+## What to read next
+
+- [Checks](./checks): assert on what the eval did
+- [Scores](./scores): grade how well it did it
+- [TypeScript client](../client/messages): the send/turn protocol eval sessions build on

package/dist/docs/public/evals/checks.mdx

@@ -0,0 +1,63 @@
+---
+title: "Checks"
+description: "Hard assertions over runs, tool calls, and output — any failed check fails the eval and the exit code."
+---
+
+Checks are hard assertions. Any failed check marks the eval failed and `eve eval` exits non-zero. Use them for the things that must hold — the run completed, the right tool ran, the output parses. For graded, non-fatal signals, use [scores](./scores) instead.
+
+## Built-in checks
+
+Built-in checks live on `eve/evals/checks`:
+
+| Check                                                         | Asserts                                                                           |
+| ------------------------------------------------------------- | --------------------------------------------------------------------------------- |
+| `Checks.completed()`                                          | The run did not fail and did not park on unanswered HITL input                    |
+| `Checks.waiting()`                                            | The run parked on HITL input (for approval-shaped evals)                          |
+| `Checks.didNotFail()`                                         | No terminal failure and no `turn.failed`/`step.failed` events (parked runs pass)  |
+| `Checks.messageIncludes(token)`                               | Joined assistant text contains `token` (string or RegExp)                         |
+| `Checks.outputEquals(value)` / `Checks.outputMatches(schema)` | Deep equality / Standard Schema (e.g. Zod) validation of the parsed output        |
+| `Checks.toolCalled(name, opts?)`                              | A matching tool call happened (`input`, `output`, `isError`, `times` constraints) |
+| `Checks.toolNotCalled(name)`                                  | No call to `name`                                                                 |
+| `Checks.toolOrder([...names])`                                | Tool names appear in order (other calls may interleave)                           |
+| `Checks.noFailedActions()`                                    | No tool, subagent, or skill action reported a failure                             |
+| `Checks.subagentCalled(name, opts?)`                          | A subagent delegation happened (`remoteUrl`, `output` constraints)                |
+| `Checks.event(predicate, label)`                              | Escape hatch: any predicate over the typed event stream                           |
+
+## Matchers
+
+Matcher options accept a literal (objects partial-deep-match), a RegExp, or a function — return a boolean to act as a predicate, or return a value to compare against (handy for runner-assigned values like environment-provided URLs):
+
+```ts
+Checks.toolCalled("bash", { input: { command: /^pwd/ }, isError: false, times: 1 });
+Checks.subagentCalled("weather", {
+  remoteUrl: () => process.env.WEATHER_AGENT_URL!,
+  output: /72F/,
+});
+```
+
+## Custom checks
+
+A custom check is a plain function receiving `{ evaluation, result, target }` and returning `{ name, passed, message? }`:
+
+```ts
+import type { EveEvalCheck } from "eve/evals/checks";
+
+const repliedFast: EveEvalCheck = ({ result }) => ({
+  name: "replied-fast",
+  passed: result.durationMs < 5_000,
+  message: `took ${result.durationMs}ms`,
+});
+```
+
+Write a `message` for the failing path — it is what the console reporter prints under the eval line and what lands in JUnit output.
+
+## Run status and parking
+
+`result.status` is `"waiting"` whenever the session is left open for a next message — the normal end state of a successful turn. Parking on unanswered HITL input is tracked separately as `result.derived.parked`, which is what `Checks.completed()` and `Checks.waiting()` key off.
+
+`result.derived` also carries typed tool calls (name, input, output, error state), subagent calls, and HITL input requests, so custom checks rarely need to walk the raw event stream. When they do, `Checks.event(predicate, label)` covers it.
+
+## What to read next
+
+- [Scores](./scores): graded signals with thresholds
+- [Cases](./cases): where checks attach

package/dist/docs/public/evals/meta.json

@@ -0,0 +1,4 @@
+{
+  "title": "Evals",
+  "pages": ["overview", "cases", "checks", "scores", "targets", "reporters", "running"]
+}

package/dist/docs/public/evals/overview.mdx

@@ -0,0 +1,91 @@
+---
+title: "Evals"
+description: "Define repeatable scored checks for an Eve agent with defineEval and run them with eve eval."
+---
+
+An eval is a scored check that runs your agent against real sessions and grades the result. Use it to catch regressions when you change a prompt or a tool: compare the output against expected text, JSON, SQL, or behavior, and optionally ship the results to Braintrust.
+
+Evals exercise the same HTTP surface your users hit. The runner boots (or targets) a real agent server, drives sessions through the [TypeScript client](../client/overview) protocol, and grades what comes back — so a passing eval means the agent actually booted, accepted a request, and produced the result you asserted.
+
+## `defineEval`
+
+Eve discovers evals under the app-root `evals/` directory, in `.eval.ts` files. Each file is exactly one eval — one graded case. The file path is the eval's identity, so you don't author an `id` or `name`; directories group related evals (`evals/weather/brooklyn-forecast.eval.ts` → id `weather/brooklyn-forecast`).
+
+```text
+my-agent/
+├── agent/
+├── evals/
+│   ├── evals.config.ts
+│   ├── smoke.eval.ts
+│   └── weather/
+│       ├── brooklyn-forecast.eval.ts
+│       └── no-tools-for-greetings.eval.ts
+└── package.json
+```
+
+```ts title="evals/weather/brooklyn-forecast.eval.ts"
+import { defineEval } from "eve/evals";
+import { Checks } from "eve/evals/checks";
+import { Run } from "eve/evals/scores";
+
+export default defineEval({
+  description: "Basic message and tool-usage coverage for the weather agent.",
+  input: "What is the weather in Brooklyn?",
+  expected: "Sunny",
+  checks: [Checks.didNotFail(), Checks.toolCalled("get_weather")],
+  scores: [Run.didNotFail()],
+});
+```
+
+Every eval needs `scores` (an empty array is fine) and either `input` (a prompt sent as a single turn) or `run` (an imperative script). The rest are optional: `description`, `expected`, `checks`, `requires`, `parseOutput`, `model`, `thresholds`, `modelOptions`, `tags`, `metadata`, `timeoutMs`, `reporters`. The init template adds `evals/**/*.ts` to `tsconfig.json`, so your eval code type-checks alongside the app.
+
+## `evals.config.ts`
+
+Every `evals/` directory needs exactly one `evals.config.ts` at its root. It declares the defaults every eval shares — most importantly the `model` used by model-backed scorers, so you don't repeat it in each file:
+
+```ts title="evals/evals.config.ts"
+import { defineEvalConfig } from "eve/evals";
+import { Braintrust } from "eve/evals/reporters";
+
+export default defineEvalConfig({
+  model: "openai/gpt-5.4-mini",
+  reporters: [Braintrust({ projectName: "my-agent" })],
+});
+```
+
+`model` is required; `reporters`, `maxConcurrency`, and `timeoutMs` are optional. Config `reporters` observe every eval in the run — set one `Braintrust()` here instead of adding it to each eval. CLI flags (`--max-concurrency`, `--timeout`) and per-eval values take precedence over the config defaults. An eval that needs a different judge model overrides it with its own `model`; otherwise the config `model` applies.
+
+## Two grading tiers
+
+Evals are graded on two distinct tiers:
+
+- **[Checks](./checks) are hard assertions.** Any failed check marks the eval failed and `eve eval` exits non-zero. Use them for the things that must hold — the run completed, the right tool ran, the output parses.
+- **[Scores](./scores) are soft data.** They land in reports and artifacts, and a below-threshold score marks the eval `scored` — visible but not fatal, unless you pass `--strict`.
+
+## Run it
+
+```bash
+eve eval                       # run all discovered evals against a local dev server
+eve eval weather               # run one eval, or every eval under evals/weather/
+eve eval --url https://<app>   # target an existing server or deployment
+```
+
+Exit code `0` means every eval passed its checks. See [Running evals](./running) for the full flag list, exit codes, and CI guidance.
+
+## A good baseline
+
+Most apps do fine with a few small smoke evals. Assert behavior with `Checks.didNotFail()` plus one or two content checks, keep dataset fixtures in `evals/data/`, and only reach for Braintrust once you actually need shared result review or experiment history. In CI, run `eve eval --strict` so threshold misses fail the build too.
+
+The rest of this section covers each piece:
+
+- [Cases](./cases): prompt evals, scripted multi-turn evals, and dataset fan-out
+- [Checks](./checks): hard assertions over runs, tools, and output
+- [Scores](./scores): deterministic and LLM-judged scorers, with thresholds
+- [Targets and requirements](./targets): local vs remote targets, and gating evals on capabilities
+- [Reporters](./reporters): Braintrust experiments and JUnit XML
+- [Running evals](./running): the `eve eval` CLI, exit codes, and artifacts
+
+## What to read next
+
+- [Cases](./cases): author your first evals
+- [Tools](../tools): the surface most evals assert on

package/dist/docs/public/evals/reporters.mdx

@@ -0,0 +1,61 @@
+---
+title: "Reporters"
+description: "Ship eval results to Braintrust experiments or JUnit XML — Eve runs and scores everything itself."
+---
+
+Eve runs and scores everything itself; reporters just ship the results out. The CLI prints a console summary by default — one line per eval, failed checks with their messages — and reporters from `eve/evals/reporters` add destinations on top.
+
+Reporters attach in two places. Declare them in `evals.config.ts` to observe **every** eval in the run — the usual choice for a shared destination like one Braintrust experiment, so you don't repeat the reporter in each file. Or list them on an individual eval's `reporters` to scope a destination to that eval (or to a group of evals that share one instance).
+
+## Braintrust
+
+`Braintrust(...)` uploads eval results to Braintrust experiments. Put one instance in the config so it covers the whole run:
+
+```ts title="evals/evals.config.ts"
+import { defineEvalConfig } from "eve/evals";
+import { Braintrust } from "eve/evals/reporters";
+
+export default defineEvalConfig({
+  model: "openai/gpt-5.4-mini",
+  reporters: [Braintrust({ projectName: "weather-agent" })],
+});
+```
+
+Need a destination for only some evals? Attach it per eval instead:
+
+```ts title="evals/brooklyn-forecast.eval.ts"
+import { defineEval } from "eve/evals";
+import { Braintrust } from "eve/evals/reporters";
+import { Run } from "eve/evals/scores";
+
+export default defineEval({
+  input: "What is the weather in Brooklyn?",
+  scores: [Run.didNotFail()],
+  reporters: [Braintrust({ projectName: "weather-agent" })],
+});
+```
+
+The reporter config takes an optional `projectName` and `experimentName`, plus a base experiment (by name or id) to diff against. Checks log as binary scores under a `check:` prefix so experiments diff check regressions the same way they diff score regressions. Eval `metadata` rides along to reporters.
+
+A reporter instance observes the evals that reference it: share one instance across several evals — the config, a `shared.ts` export, or every entry of a dataset array — and their results land in a single experiment. Listing the same config reporter on an eval too does not double-report it.
+
+Braintrust needs its SDK installed in the app and credentials in the environment. Pass `--skip-report` to run the eval without shipping results (this also suppresses config reporters) — useful locally when iterating.
+
+## JUnit
+
+`JUnit({ filePath })` writes JUnit XML for CI annotations. The `--junit <path>` CLI flag does the same thing without touching the eval file, which is usually the better fit — CI owns the output path, not the eval:
+
+```bash
+eve eval --strict --junit .eve/junit.xml
+```
+
+Each eval becomes one `<testcase>` named by its path-derived id; failed checks and execution errors land as failure messages on the matching test case, so CI surfaces them inline.
+
+## Custom reporters
+
+A reporter implements the `EvalReporter` interface from `eve/evals/reporters` and receives the same structured results the built-ins do. Reach for one only when a destination isn't covered — the per-run artifacts under `.eve/evals/` already capture everything for ad-hoc inspection.
+
+## What to read next
+
+- [Running evals](./running): console output, `--json`, and artifacts
+- [Scores](./scores): what the reported numbers mean

package/dist/docs/public/evals/running.mdx

@@ -0,0 +1,63 @@
+---
+title: "Running evals"
+description: "The eve eval CLI: flags, filters, exit codes, artifacts, and how to wire evals into CI."
+---
+
+`eve eval` discovers every `.eval.ts` file under `evals/`, boots a local dev server (or targets a remote one), runs the evals concurrently, and prints a per-eval summary.
+
+```bash
+eve eval                       # run all discovered evals locally
+eve eval weather smoke         # run selected evals (an id, or a directory prefix)
+eve eval --url https://<app>   # target a remote app instead of a local host
+eve eval --mock-models         # local dev target uses deterministic mock models
+eve eval --tag fast            # only evals carrying a tag
+eve eval --strict              # below-threshold scores also fail the exit code
+eve eval --no-skips            # unmet requirements fail instead of skipping
+eve eval --timeout 60000       # per-eval timeout in milliseconds
+eve eval --max-concurrency 4   # cap concurrent eval executions (default 8)
+eve eval --junit .eve/junit.xml  # write JUnit XML
+eve eval --list                # print discovered evals without running
+eve eval --verbose             # stream per-eval ctx.log lines to stdout
+eve eval --json                # machine-readable output
+eve eval --skip-report         # skip config and eval-defined reporters (e.g. Braintrust)
+```
+
+Positional ids match exactly or by directory prefix: `eve eval weather` runs `evals/weather.eval.ts`, every eval under `evals/weather/`, and every entry of an array-exported `weather.eval.ts`.
+
+## Exit codes
+
+| Code | Means                                                                            |
+| ---- | -------------------------------------------------------------------------------- |
+| `0`  | Every eval passed its checks (and thresholds, under `--strict`)                  |
+| `1`  | Any eval failed — a failed check, an execution error, or a strict threshold miss |
+| `2`  | Configuration error                                                              |
+
+Unmet [requirements](./targets) skip visibly without affecting the exit code unless you pass `--no-skips`.
+
+## Artifacts
+
+Each run drops artifacts under `.eve/evals/<timestamp>/`: a run `summary.json`, a `results.jsonl` index, and per-eval check results, verdicts, captured event streams, and `ctx.log` lines under `evals/`. The console output stays tight on purpose; when an eval fails, the artifact has the full story.
+
+## CI
+
+A solid CI invocation is strict, deterministic, and machine-reportable:
+
+```bash
+eve eval --strict --mock-models --junit .eve/junit.xml
+```
+
+- `--strict` turns threshold misses into failures, so score regressions block the merge.
+- `--mock-models` keeps the default leg deterministic and credential-free. Put real-model evals in their own files gated on `requires: ["env:..."]`, and add `--no-skips` on legs that must prove those ran.
+- `--junit` gives the CI provider per-eval annotations; upload the `.eve/evals/` directory as a failure artifact for the full event streams.
+
+Against a deployed app, swap `--mock-models` for `--url`:
+
+```bash
+eve eval --strict --url "$DEPLOY_URL" --junit .eve/junit.xml
+```
+
+## What to read next
+
+- [Targets and requirements](./targets): what `--url`, `--mock-models`, and `--no-skips` interact with
+- [Reporters](./reporters): Braintrust and JUnit output
+- [CLI reference](../reference/cli): the rest of the `eve` CLI

package/dist/docs/public/evals/scores.mdx

@@ -0,0 +1,57 @@
+---
+title: "Scores"
+description: "Grade evals with deterministic scorers or LLM judges, and gate them with thresholds."
+---
+
+Scores are soft data. They land in reports and artifacts, and a below-threshold score marks the eval `scored` — visible but not fatal, unless you pass `--strict`. Use them to grade quality fractionally where a [check](./checks) would assert it absolutely.
+
+## Choosing a scorer
+
+Scorers live in namespaces on `eve/evals/scores`. Pick the cheapest one that captures what "correct" means here. The deterministic scorers run instantly for free; an LLM judge runs once per eval and burns tokens, so save it for when nothing simpler will do.
+
+| Need                                                       | Use                                                                                           |
+| ---------------------------------------------------------- | --------------------------------------------------------------------------------------------- |
+| Grade agent behavior (run succeeded, used the right tools) | `Run.didNotFail()`, `Run.usedTool(name, opts?)`, `Run.usedNoTools()`, `Run.maxToolCalls(max)` |
+| Exact string match                                         | `Text.exact()`, `Text.includes()`                                                             |
+| Fuzzy text match (typos, whitespace)                       | `Text.levenshtein()`                                                                          |
+| Exact JSON match                                           | `Json.deepEqual()`                                                                            |
+| Exact SQL match (after normalization)                      | `Sql.exactNormalized()`                                                                       |
+| LLM-judged factual correctness vs an expected answer       | `Autoevals.factuality()`                                                                      |
+| LLM-judged summary quality                                 | `Autoevals.summary()`                                                                         |
+| LLM-judged SQL semantic equivalence                        | `Autoevals.sql()`                                                                             |
+| LLM-judged free-form criteria (no `expected` to match)     | `Autoevals.closedQA({ criteria: "..." })`                                                     |
+
+Each scorer gets the flattened `input`, `output`, and `expected` strings along with the full eval and task result — including derived facts: typed tool calls (name, input, output, error state), subagent calls, HITL input requests, and whether the run parked. `Run.usedTool` accepts the same matcher options as `Checks.toolCalled`. Return `null` from a scorer to skip it.
+
+## Thresholds
+
+By default a scorer has to hit an exact match to pass. `thresholds` loosens that, mapping each scorer name to the minimum score you'll accept:
+
+```ts
+import { defineEval } from "eve/evals";
+import { Run, Text } from "eve/evals/scores";
+
+export default defineEval({
+  input: "Hello",
+  expected: "Hello",
+  scores: [Run.didNotFail(), Text.includes()],
+  thresholds: { "run.didNotFail": 1, "text.includes": 0.5 },
+});
+```
+
+An eval below a threshold gets the `scored` verdict — reported, but only fatal under `eve eval --strict`.
+
+## The scorer model
+
+Model-backed scorers (the `Autoevals` wrappers) need a judge model — the scorer model, not the agent's. Eve only uses it for scoring, never to swap out the agent under test. The default lives in [`evals.config.ts`](./overview#evalsconfigts) as the required `model`, so most evals inherit it without setting anything. Pass a string id (e.g. `"anthropic/claude-opus-4.8"`) to route through the Vercel AI Gateway, or an AI SDK model instance to use it directly.
+
+Override the default on a single eval by setting that eval's own `model`. For provider-specific scorer-model settings, use `modelOptions.providerOptions`. Individual Autoevals scorers can also take their own `model` / `modelOptions`, which win over both the eval and config defaults.
+
+## Concurrency and timeouts
+
+`timeoutMs` bounds one eval's execution: the eval's own value wins, then `evals.config.ts`'s default, and `eve eval --timeout <ms>` overrides both for a run. The runner executes up to 8 evals at once — set a default `maxConcurrency` in `evals.config.ts` or pass `--max-concurrency <n>` (which wins) to change that, and lower it when evals contend for a shared resource: a rate-limited connection, or a sandbox-heavy fixture.
+
+## What to read next
+
+- [Checks](./checks): hard assertions that fail the build
+- [Reporters](./reporters): ship scores to Braintrust experiments