eve 0.6.0-beta.13 → 0.6.0-beta.15

package/CHANGELOG.md

@@ -1,5 +1,26 @@
 # eve
 
+## 0.6.0-beta.15
+
+### Minor Changes
+
+- 0a8d93b: Add `eve init <name>` for non-interactive agent creation, with optional Web Chat through `--web`. The command installs dependencies, initializes Git, and starts the development server, but does not provision or deploy a Vercel project.
+- f733fe5: The deterministic mock model adapter (`EVE_MOCK_AUTHORED_MODELS=1`) now synthesizes tool inputs for string properties anchored to quoted spans in the message (e.g. `with label "x"`), honors exact-reply fixture directives (`reply with the exact string … and nothing else`, `include the exact token … verbatim`) found in the current turn's user messages and per-turn client context, discovers static skill advertisements embedded in instruction text (not just standalone announcements), derives skill activation from `load_skill` calls in history so skills are never re-loaded in a loop, and no longer matches `load_skill` by explicit name. This widens what agent smoke evals can assert deterministically without a real model.
+
+### Patch Changes
+
+- 0a8d93b: Scaffolded projects now carry a `packageExtensions` entry in `pnpm-workspace.yaml` that restores eve's missing `oxc-parser` dependency, so fresh agents boot against the published `eve` 0.6.0-beta.13/14 builds that import it without declaring it. The entry lives in `pnpm-workspace.yaml` because pnpm 11 reads settings only from there, not from the `package.json` `pnpm` field. Remove it from the template once the scaffolded `eve` range only resolves to versions that declare `oxc-parser` themselves.
+
+## 0.6.0-beta.14
+
+### Minor Changes
+
+- 9d806d0: Add a `/model` command to the `eve dev` TUI. Run it bare to pick from a list (the running model flagged, a catalog-validated shortlist, and a freeform entry), or `/model <provider/model-id>` to set one directly. The pick rewrites `agent.ts`, and the dev server's HMR watcher applies it on the next prompt.
+
+### Patch Changes
+
+- c99668f: Correct the public bash tool helper export from `defineBeveTool` to `defineBashTool`.
+
 ## 0.6.0-beta.13
 
 ### Minor Changes

package/README.md

@@ -45,7 +45,7 @@ Every authored directory has a typed helper. Import each from the matching subpa
 | ------------------------------------------------------------------------------------------------------------------- | ------------------------------------- | ------------------------------------------------ |
 | `defineAgent(...)`                                                                                                  | `eve`                                 | `agent.ts`, `subagents/<id>/agent.ts`            |
 | `defineInstructions(...)`                                                                                           | `eve/instructions`                    | `instructions.ts` (or `instructions.md`)         |
-| `defineTool(...)`, `defineBeveTool(...)`, `defineReadFileTool(...)`, `defineWriteFileTool(...)`, `disableTool(...)` | `eve/tools`                           | `tools/<name>.ts`                                |
+| `defineTool(...)`, `defineBashTool(...)`, `defineReadFileTool(...)`, `defineWriteFileTool(...)`, `disableTool(...)` | `eve/tools`                           | `tools/<name>.ts`                                |
 | `defineSkill(...)`, `getSkill(...)`                                                                                 | `eve/skills`                          | `skills/<name>.ts` (or `skills/<name>.md`)       |
 | `defineHook(...)`                                                                                                   | `eve/hooks`                           | `hooks/<slug>.ts`                                |
 | `defineChannel(...)`, `POST`, `GET`                                                                                 | `eve/channels`                        | `channels/<name>.ts`                             |
@@ -105,15 +105,16 @@ export default defineAgent({
 ## Quick Start
 
 ```bash
-pnpm create eve@beta
-cd my-agent
-pnpm dev
+pnpm dlx eve@beta init my-agent
 ```
 
-The wizard scaffolds the project, picks a model, and (for the REPL channel) installs dependencies and starts the dev server for you. To scaffold into the current empty directory, run `pnpm create eve@beta .`.
+`eve init` writes a new agent with Eve's default model. Pass `--web` to add the
+Web Chat application. It installs dependencies, initializes Git, and starts the
+development server. It does not create a Vercel project or deploy the agent.
 
 CLI commands:
 
+- `eve init <name>` — create a new agent
 - `eve info` — discovery results and compiled artifacts
 - `eve build` — compile `.eve/` and build the host output
 - `eve start` — serve the built `.output/` app

package/dist/docs/evals-v2-plan.md

@@ -812,13 +812,13 @@ weather-result-reaches-parent` reruns it in isolation.
 
 ### CI
 
-`.github/workflows/smoke.yml` discovery changes from globbing
+`.github/workflows/e2e.yml` discovery changes from globbing
 `e2e/tests/*/*.ts` to globbing fixture apps with `evals/` directories. Each
 matrix leg provisions, then runs the evals against the resulting URL:
 
 ```sh
 node e2e/provision/<group>.ts &        # build + sidecars + env + eve start
-pnpm --filter <fixture-app> exec eve eval --strict --json --url "$TARGET_URL"
+pnpm --filter <fixture-app> exec eve eval --strict --url "$TARGET_URL"
 ```
 
 twice (direct / `EVE_EXPERIMENTAL_CODE_MODE=1` set by the provisioner),
@@ -842,7 +842,7 @@ Phase 5.
 ## Implementation phases
 
 Each phase ships independently, keeps `pnpm test` green, and includes docs
-(`docs/public/advanced/evals.md`) + changesets per repo policy.
+(now the top-level `docs/public/evals/` section) + changesets per repo policy.
 
 ### Phase 1 — Assertions and pass/fail (no breaking interaction changes)
 
@@ -906,6 +906,31 @@ addresses. Nothing in phase 4 deploys anywhere.
     area policy; shrink `e2e/lib`/`e2e/target` into `e2e/provision/`; update
     `e2e/README.md` and AGENTS.md smoke-test guidance to point at `eve eval`.
 
+> **Decisions made during the phase-4 migration (June 2026):**
+>
+> - **Hybrid mock/real split.** Evals default to the deterministic mock
+>   (`requires: ["mockModels"]`). Behaviors the mock cannot express — the
+>   built-in `agent` tool, Workflow fan-out, provider-executed `web_search`,
+>   url-file attachment reads — live in separate `real-model`-tagged eval
+>   files run against a non-mock server only when model credentials are
+>   present (provisioners log a visible skip otherwise). Net: real-model CI
+>   legs dropped from ~156 to ~7 cases.
+> - **Code-mode e2e dropped entirely.** No `EVE_EXPERIMENTAL_CODE_MODE`
+>   matrix dimension and no codemode group port; the `agent-codemode` fixture
+>   was deleted. Purpose-built code-mode fixtures/evals will be designed
+>   later.
+> - **Mock adapter extensions** (in `eve`): anchored quoted-span tool-input
+>   synthesis, exact-reply directives in current-turn user text (proves
+>   clientContext delivery), embedded static-skill advert discovery, and
+>   history-derived skill-activation tracking.
+> - **Coverage consciously dropped:** semantic multi-turn recall and the
+>   channel-rekey marker recall (pure model behaviors), `subagent-output-schema`
+>   moved to the real leg, and the eval-CLI self-smoke (`evals.ts`) — now the
+>   CI mechanism itself.
+> - Sidecar fake providers expose a `GET /_probe/requests` inspection
+>   endpoint (`e2e/lib/http-test-server.ts`) so evals assert captured
+>   traffic across the provisioner process boundary.
+
 ### Phase 5 — Remote leg: deploy fixtures and eval the deployment
 
 After phase 4, CI gains a second e2e variant that runs the **same eval
@@ -918,7 +943,7 @@ skip.
     preview env sets `EVE_MOCK_AUTHORED_MODELS=1` (determinism, zero model
     spend) and the workflow backend credentials.
 16. CI job: deploy the preview, capture the URL, then
-    `eve eval --strict --json --junit ... --url "$DEPLOY_URL"` with
+    `eve eval --strict --junit ... --url "$DEPLOY_URL"` with
     `VERCEL_AUTOMATION_BYPASS_SECRET` in the job env (the eval client's
     existing remote-auth cascade handles protection bypass / OIDC). No
     `--no-skips` on this leg: schedule evals (`devRoutes`) and local-sidecar

package/dist/docs/public/README.md

@@ -30,7 +30,7 @@ Read in this order:
 14. [TypeScript Client](./client/overview.mdx)
 15. [Subagents](./subagents.mdx)
 16. [Schedules](./schedules.mdx)
-17. [Evals](./advanced/evals.md)
+17. [Evals](./evals/overview.mdx)
 18. [Auth And Route Protection](./advanced/auth-and-route-protection.md)
 19. [Vercel Deployment](./advanced/deployment.md)
 20. [CLI, Build, And Debugging](./reference/cli.md)

package/dist/docs/public/advanced/auth-and-route-protection.md

@@ -184,7 +184,7 @@ export default defineChannel({
 
 ## Replace `placeholderAuth` before production
 
-`pnpm create eve@beta` sometimes scaffolds `agent/channels/eve.ts` with a `placeholderAuth()` guardrail:
+`eve init` scaffolds `agent/channels/eve.ts` with a `placeholderAuth()` guardrail:
 
 ```ts
 import { eveChannel } from "eve/channels/eve";

package/dist/docs/public/advanced/instrumentation.md

@@ -162,4 +162,4 @@ When `eve build` fails on discovery errors, the CLI prints the full diagnostics
 - [`agent.ts`](../agent-config)
 - [Hooks](./hooks): observe the runtime event stream
 - [Dev TUI](./dev-tui): drive the agent locally
-- [Evals](./evals): repeatable scored checks
+- [Evals](../evals/overview): repeatable scored checks

package/dist/docs/public/advanced/meta.json

@@ -14,7 +14,6 @@
     "deployment",
     "instrumentation",
     "dev-tui",
-    "evals",
     "execution-model-and-durability",
     "sessions-runs-and-streaming"
   ]

package/dist/docs/public/channels/eve.mdx

@@ -36,7 +36,7 @@ The `auth` option decides who can call these routes. The built-in helpers are me
 
 Neither admits browser users or external clients in production. For a public app, wire the channel to your own auth (Clerk, Auth.js, or your own OIDC/JWT verification).
 
-`pnpm create eve@beta` scaffolds an `agent/channels/eve.ts` with a production placeholder so you replace it before going live. The generated channel allows Vercel OIDC and localhost, and includes `placeholderAuth()`, which returns a setup-focused 401 in production until you swap it for real auth. Delete the file and Eve falls back to `[localDev(), vercelOidc()]`, which still does not admit browser users in production.
+`eve init` scaffolds an `agent/channels/eve.ts` with a production placeholder so you replace it before going live. The generated channel allows Vercel OIDC and localhost, and includes `placeholderAuth()`, which returns a setup-focused 401 in production until you swap it for real auth. Delete the file and Eve falls back to `[localDev(), vercelOidc()]`, which still does not admit browser users in production.
 
 For the full auth model and helper list, see [Auth & route protection](../advanced/auth-and-route-protection).
 

package/dist/docs/public/evals/cases.mdx

@@ -0,0 +1,115 @@
+---
+title: "Cases and tasks"
+description: "Author data cases, load fixtures with loaders, and script multi-turn cases with run(ctx)."
+---
+
+A case is one graded unit of work: the runner executes it against the target, captures every event, and applies [checks](./checks) and [scores](./scores) to the result. A case is either a data case or a scripted case.
+
+## Data cases
+
+Data cases pair an `input` with an `expected`, plus optional `checks`, `scores`, `requires`, `tags`, and `metadata`. `input` can be a string or an object. `expected` is optional, which is handy when you only care about behavior:
+
+```ts title="evals/weather.eval.ts"
+import { defineEval } from "eve/evals";
+import { Checks } from "eve/evals/checks";
+
+export default defineEval({
+  checks: [Checks.didNotFail()],
+  scores: [],
+  cases: [
+    { id: "brooklyn-forecast", input: "What is the weather in Brooklyn?", expected: "Sunny" },
+    {
+      id: "no-tools-for-greetings",
+      input: "Hello!",
+      checks: [Checks.toolNotCalled("get_weather")],
+    },
+  ],
+});
+```
+
+Eval-level `checks` and `scores` apply to every case; case-level entries append to them.
+
+## Loading cases from fixtures
+
+List cases inline, or load them dynamically with `load`. The loaders (`loadJson`, `loadYaml` from `eve/evals/loaders`) resolve paths relative to the app root:
+
+```ts title="evals/sql.eval.ts"
+import { defineEval } from "eve/evals";
+import { loadYaml } from "eve/evals/loaders";
+import { Text } from "eve/evals/scores";
+
+export default defineEval({
+  async load() {
+    const doc = await loadYaml("evals/data/cases.yaml");
+    return (doc.evals as readonly { task: string; prompt: string; sql: string }[]).map((row) => ({
+      id: row.task,
+      input: row.prompt,
+      expected: row.sql,
+    }));
+  },
+  scores: [Text.exact()],
+});
+```
+
+Pass either `cases` or `load`, never both. The loaders are meant for fixtures, not runtime agent code.
+
+## Tasks
+
+The eval-level `task` controls how the runner turns a data case into agent work:
+
+- `task.prompt` covers the single-string case — a template that interpolates the case input.
+- `task.messages` sets up a static multi-turn conversation.
+- `task.run` gives you imperative control flow for every case in the eval.
+- `task.parseOutput` compares a transformed result rather than the raw final message.
+
+A task can specify at most one of `prompt`, `messages`, and `run`. When none is set, the case `input` is sent as a single user message.
+
+## Scripted cases
+
+Scripted cases define their own `run(ctx)` and do not need `input`. Use them for smoke-style behavior: multi-turn branching, HITL approvals, structured output, attachments, and multiple independent sessions.
+
+```ts title="evals/approvals.eval.ts"
+import { defineEval } from "eve/evals";
+import { Checks } from "eve/evals/checks";
+
+export default defineEval({
+  checks: [Checks.didNotFail()],
+  scores: [],
+  cases: [
+    {
+      id: "approve-tool",
+      async run({ session }) {
+        await session.send("run `pwd`");
+        session.expectInputRequests({ toolName: "bash" });
+        const turn = await session.respondAll("approve");
+        return turn.message;
+      },
+      checks: [Checks.toolCalled("bash", { input: { command: /pwd/ } })],
+    },
+  ],
+});
+```
+
+The return value of `run` becomes the case output that scorers grade. Throwing marks the case failed with the error message in the result.
+
+## The session API
+
+`ctx.session` is the primary `EveEvalSession`; `ctx.newSession()` creates another independent session against the same target:
+
+- `session.send(input)` sends a turn and waits for it to settle. It accepts the same input as `ClientSession.send()` — a string or a structured message.
+- `session.sendFile(text, path, mediaType?)` attaches a local file as a data URL.
+- `session.expectInputRequests(filter?)` asserts the previous turn parked on HITL input and returns the pending requests.
+- `session.respondAll(optionId)` answers every pending input request with the same option and sends the responses as the next turn.
+- `session.events` is the full typed event stream captured so far.
+
+Each `send` resolves to an `EveEvalTurn` carrying the turn's `message`, `events`, and status. `turn.expectOk()` throws only when the turn ended failed — a session left open for a next message is the normal end state of a successful turn.
+
+Events from every eval session are captured in the case result and artifacts. `ctx.log(message)` records debug lines into the case artifact; `--verbose` also streams them to stdout as cases run.
+
+For driving sessions created outside the eval — by a channel webhook or a schedule — see [Targets and requirements](./targets).
+
+## What to read next
+
+- [Checks](./checks): assert on what the case did
+- [Scores](./scores): grade how well it did it
+- [TypeScript client](../client/messages): the send/turn protocol eval sessions build on

package/dist/docs/public/evals/checks.mdx

@@ -0,0 +1,65 @@
+---
+title: "Checks"
+description: "Hard assertions over runs, tool calls, and output — any failed check fails the case and the exit code."
+---
+
+Checks are hard assertions. Any failed check marks the case failed and `eve eval` exits non-zero. Use them for the things that must hold — the run completed, the right tool ran, the output parses. For graded, non-fatal signals, use [scores](./scores) instead.
+
+Checks exist at the eval level (applied to every case) and the case level (appended to the eval's).
+
+## Built-in checks
+
+Built-in checks live on `eve/evals/checks`:
+
+| Check                                                         | Asserts                                                                           |
+| ------------------------------------------------------------- | --------------------------------------------------------------------------------- |
+| `Checks.completed()`                                          | The run did not fail and did not park on unanswered HITL input                    |
+| `Checks.waiting()`                                            | The run parked on HITL input (for approval-shaped evals)                          |
+| `Checks.didNotFail()`                                         | No terminal failure and no `turn.failed`/`step.failed` events (parked runs pass)  |
+| `Checks.messageIncludes(token)`                               | Joined assistant text contains `token` (string or RegExp)                         |
+| `Checks.outputEquals(value)` / `Checks.outputMatches(schema)` | Deep equality / Standard Schema (e.g. Zod) validation of the parsed output        |
+| `Checks.toolCalled(name, opts?)`                              | A matching tool call happened (`input`, `output`, `isError`, `times` constraints) |
+| `Checks.toolNotCalled(name)`                                  | No call to `name`                                                                 |
+| `Checks.toolOrder([...names])`                                | Tool names appear in order (other calls may interleave)                           |
+| `Checks.noFailedActions()`                                    | No tool, subagent, or skill action reported a failure                             |
+| `Checks.subagentCalled(name, opts?)`                          | A subagent delegation happened (`remoteUrl`, `output` constraints)                |
+| `Checks.event(predicate, label)`                              | Escape hatch: any predicate over the typed event stream                           |
+
+## Matchers
+
+Matcher options accept a literal (objects partial-deep-match), a RegExp, or a function — return a boolean to act as a predicate, or return a value to compare against (handy for runner-assigned values like environment-provided URLs):
+
+```ts
+Checks.toolCalled("bash", { input: { command: /^pwd/ }, isError: false, times: 1 });
+Checks.subagentCalled("weather", {
+  remoteUrl: () => process.env.WEATHER_AGENT_URL!,
+  output: /72F/,
+});
+```
+
+## Custom checks
+
+A custom check is a plain function receiving `{ case, result, target }` and returning `{ name, passed, message? }`:
+
+```ts
+import type { EveEvalCheck } from "eve/evals/checks";
+
+const repliedFast: EveEvalCheck = ({ result }) => ({
+  name: "replied-fast",
+  passed: result.durationMs < 5_000,
+  message: `took ${result.durationMs}ms`,
+});
+```
+
+Write a `message` for the failing path — it is what the console reporter prints under the case line and what lands in JUnit output.
+
+## Run status and parking
+
+`result.status` is `"waiting"` whenever the session is left open for a next message — the normal end state of a successful turn. Parking on unanswered HITL input is tracked separately as `result.derived.parked`, which is what `Checks.completed()` and `Checks.waiting()` key off.
+
+`result.derived` also carries typed tool calls (name, input, output, error state), subagent calls, and HITL input requests, so custom checks rarely need to walk the raw event stream. When they do, `Checks.event(predicate, label)` covers it.
+
+## What to read next
+
+- [Scores](./scores): graded signals with thresholds
+- [Cases and tasks](./cases): where checks attach

package/dist/docs/public/evals/meta.json

@@ -0,0 +1,4 @@
+{
+  "title": "Evals",
+  "pages": ["overview", "cases", "checks", "scores", "targets", "reporters", "running"]
+}

package/dist/docs/public/evals/overview.mdx

@@ -0,0 +1,75 @@
+---
+title: "Evals"
+description: "Define repeatable scored checks for an Eve agent with defineEval and run them with eve eval."
+---
+
+An eval is a scored check that runs your agent against real sessions and grades the result. Use it to catch regressions when you change a prompt or a tool: compare the output against expected text, JSON, SQL, or behavior, and optionally ship the results to Braintrust.
+
+Evals exercise the same HTTP surface your users hit. The runner boots (or targets) a real agent server, drives sessions through the [TypeScript client](../client/overview) protocol, and grades what comes back — so a passing eval means the agent actually booted, accepted a request, and produced the result you asserted.
+
+## `defineEval`
+
+Eve discovers evals under the app-root `evals/` directory, in `.eval.ts` files. The file path is the eval's identity, so you don't author an `id` or `name`.
+
+```text
+my-agent/
+├── agent/
+├── evals/
+│   ├── smoke.eval.ts
+│   └── weather.eval.ts
+└── package.json
+```
+
+```ts title="evals/weather.eval.ts"
+import { defineEval } from "eve/evals";
+import { Checks } from "eve/evals/checks";
+import { Run } from "eve/evals/scores";
+
+export default defineEval({
+  description: "Basic message and tool-usage coverage for the weather agent.",
+  cases: [
+    { id: "brooklyn-forecast", input: "What is the weather in Brooklyn?", expected: "Sunny" },
+  ],
+  checks: [Checks.didNotFail(), Checks.toolCalled("get_weather")],
+  scores: [Run.didNotFail()],
+});
+```
+
+Every eval needs `scores` (an empty array is fine) and either `cases` or `load`. The rest are optional: `description`, `task`, `checks`, `requires`, `model`, `thresholds`, `modelOptions`, `tags`, `metadata`, `maxConcurrency`, `timeoutMs`, `reporters`. The init template adds `evals/**/*.ts` to `tsconfig.json`, so your eval code type-checks alongside the app.
+
+## Two grading tiers
+
+Evals grade a case on two distinct tiers:
+
+- **[Checks](./checks) are hard assertions.** Any failed check marks the case failed and `eve eval` exits non-zero. Use them for the things that must hold — the run completed, the right tool ran, the output parses.
+- **[Scores](./scores) are soft data.** They land in reports and artifacts, and a below-threshold score marks the case `scored` — visible but not fatal, unless you pass `--strict`.
+
+Both exist at the eval level (applied to every case) and the case level (appended to the eval's).
+
+## Run it
+
+```bash
+eve eval                       # run all discovered evals against a local dev server
+eve eval weather               # run selected evals
+eve eval --url https://<app>   # target an existing server or deployment
+```
+
+Exit code `0` means every case passed its checks. See [Running evals](./running) for the full flag list, exit codes, and CI guidance.
+
+## A good baseline
+
+Most apps do fine with a small smoke eval. Assert behavior with `Checks.didNotFail()` plus one or two content checks, keep case fixtures in `evals/data/`, and only reach for Braintrust once you actually need shared result review or experiment history. In CI, run `eve eval --strict` so threshold misses fail the build too.
+
+The rest of this section covers each piece:
+
+- [Cases and tasks](./cases): data cases, loaders, and scripted multi-turn cases
+- [Checks](./checks): hard assertions over runs, tools, and output
+- [Scores](./scores): deterministic and LLM-judged scorers, with thresholds
+- [Targets and requirements](./targets): local vs remote targets, and gating cases on capabilities
+- [Reporters](./reporters): Braintrust experiments and JUnit XML
+- [Running evals](./running): the `eve eval` CLI, exit codes, and artifacts
+
+## What to read next
+
+- [Cases and tasks](./cases): author your first cases
+- [Tools](../tools): the surface most evals assert on

package/dist/docs/public/evals/reporters.mdx

@@ -0,0 +1,45 @@
+---
+title: "Reporters"
+description: "Ship eval results to Braintrust experiments or JUnit XML — Eve runs and scores everything itself."
+---
+
+Eve runs and scores everything itself; reporters just ship the results out. The CLI prints a console summary by default — one line per case, failed checks with their messages — and eval-level reporters from `eve/evals/reporters` add destinations on top.
+
+## Braintrust
+
+`Braintrust(...)` uploads eval results to Braintrust experiments:
+
+```ts title="evals/weather.eval.ts"
+import { defineEval } from "eve/evals";
+import { Braintrust } from "eve/evals/reporters";
+import { Run } from "eve/evals/scores";
+
+export default defineEval({
+  cases: [{ id: "brooklyn-forecast", input: "What is the weather in Brooklyn?" }],
+  scores: [Run.didNotFail()],
+  reporters: [Braintrust({ projectName: "weather-agent" })],
+});
+```
+
+The config takes an optional `projectName` and `experimentName`, plus a base experiment (by name or id) to diff against. Checks log as binary scores under a `check:` prefix so experiments diff check regressions the same way they diff score regressions. Eval and case `metadata` ride along to reporters.
+
+Braintrust needs its SDK installed in the app and credentials in the environment. Pass `--skip-report` to run the eval without shipping results — useful locally when iterating.
+
+## JUnit
+
+`JUnit({ filePath })` writes JUnit XML for CI annotations. The `--junit <path>` CLI flag does the same thing without touching the eval file, which is usually the better fit — CI owns the output path, not the eval:
+
+```bash
+eve eval --strict --junit .eve/junit.xml
+```
+
+Failed checks and execution errors land as failure messages on the matching test case, so CI surfaces them inline.
+
+## Custom reporters
+
+A reporter implements the `EvalReporter` interface from `eve/evals/reporters` and receives the same structured results the built-ins do. Reach for one only when a destination isn't covered — the per-run artifacts under `.eve/evals/` already capture everything for ad-hoc inspection.
+
+## What to read next
+
+- [Running evals](./running): console output, `--json`, and artifacts
+- [Scores](./scores): what the reported numbers mean

package/dist/docs/public/evals/running.mdx

@@ -0,0 +1,62 @@
+---
+title: "Running evals"
+description: "The eve eval CLI: flags, filters, exit codes, artifacts, and how to wire evals into CI."
+---
+
+`eve eval` discovers every `.eval.ts` file under `evals/`, boots a local dev server (or targets a remote one), runs the cases, and prints a per-case summary.
+
+```bash
+eve eval                       # run all discovered evals locally
+eve eval weather smoke         # run selected evals
+eve eval --url https://<app>   # target a remote app instead of a local host
+eve eval --mock-models         # local dev target uses deterministic mock models
+eve eval --tag fast            # only cases (or evals) carrying a tag
+eve eval --case brooklyn-forecast  # only specific case ids
+eve eval --strict              # below-threshold scores also fail the exit code
+eve eval --no-skips            # unmet requirements fail instead of skipping
+eve eval --timeout 60000       # per-case timeout in milliseconds
+eve eval --max-concurrency 4   # cap concurrent cases per eval
+eve eval --junit .eve/junit.xml  # write JUnit XML
+eve eval --list                # print discovered evals and cases without running
+eve eval --verbose             # stream per-case ctx.log lines to stdout
+eve eval --json                # machine-readable output
+eve eval --skip-report         # skip eval-defined reporters (e.g. Braintrust)
+```
+
+## Exit codes
+
+| Code | Means                                                                            |
+| ---- | -------------------------------------------------------------------------------- |
+| `0`  | Every case passed its checks (and thresholds, under `--strict`)                  |
+| `1`  | Any case failed — a failed check, an execution error, or a strict threshold miss |
+| `2`  | Configuration error                                                              |
+
+Unmet [requirements](./targets) skip visibly without affecting the exit code unless you pass `--no-skips`.
+
+## Artifacts
+
+Each run drops per-eval artifacts under `.eve/evals/<timestamp>-<eval-id>/`, including per-case check results, verdicts, captured event streams, and `ctx.log` lines. The console output stays tight on purpose; when a case fails, the artifact has the full story.
+
+## CI
+
+A solid CI invocation is strict, deterministic, and machine-reportable:
+
+```bash
+eve eval --strict --mock-models --junit .eve/junit.xml
+```
+
+- `--strict` turns threshold misses into failures, so score regressions block the merge.
+- `--mock-models` keeps the default leg deterministic and credential-free. Put real-model cases in their own eval files gated on `requires: ["env:..."]`, and add `--no-skips` on legs that must prove those ran.
+- `--junit` gives the CI provider per-case annotations; upload the `.eve/evals/` directory as a failure artifact for the full event streams.
+
+Against a deployed app, swap `--mock-models` for `--url`:
+
+```bash
+eve eval --strict --url "$DEPLOY_URL" --junit .eve/junit.xml
+```
+
+## What to read next
+
+- [Targets and requirements](./targets): what `--url`, `--mock-models`, and `--no-skips` interact with
+- [Reporters](./reporters): Braintrust and JUnit output
+- [CLI reference](../reference/cli): the rest of the `eve` CLI

package/dist/docs/public/evals/scores.mdx

@@ -0,0 +1,56 @@
+---
+title: "Scores"
+description: "Grade eval cases with deterministic scorers or LLM judges, and gate them with thresholds."
+---
+
+Scores are soft data. They land in reports and artifacts, and a below-threshold score marks the case `scored` — visible but not fatal, unless you pass `--strict`. Use them to grade quality fractionally where a [check](./checks) would assert it absolutely.
+
+## Choosing a scorer
+
+Scorers live in namespaces on `eve/evals/scores`. Pick the cheapest one that captures what "correct" means here. The deterministic scorers run instantly for free; an LLM judge runs once per case and burns tokens, so save it for when nothing simpler will do.
+
+| Need                                                       | Use                                                                                           |
+| ---------------------------------------------------------- | --------------------------------------------------------------------------------------------- |
+| Grade agent behavior (run succeeded, used the right tools) | `Run.didNotFail()`, `Run.usedTool(name, opts?)`, `Run.usedNoTools()`, `Run.maxToolCalls(max)` |
+| Exact string match                                         | `Text.exact()`, `Text.includes()`                                                             |
+| Fuzzy text match (typos, whitespace)                       | `Text.levenshtein()`                                                                          |
+| Exact JSON match                                           | `Json.deepEqual()`                                                                            |
+| Exact SQL match (after normalization)                      | `Sql.exactNormalized()`                                                                       |
+| LLM-judged factual correctness vs an expected answer       | `Autoevals.factuality()`                                                                      |
+| LLM-judged summary quality                                 | `Autoevals.summary()`                                                                         |
+| LLM-judged SQL semantic equivalence                        | `Autoevals.sql()`                                                                             |
+| LLM-judged free-form criteria (no `expected` to match)     | `Autoevals.closedQA({ criteria: "..." })`                                                     |
+
+Each scorer gets the flattened `input`, `output`, and `expected` strings along with the full case and task result — including derived facts: typed tool calls (name, input, output, error state), subagent calls, HITL input requests, and whether the run parked. `Run.usedTool` accepts the same matcher options as `Checks.toolCalled`. Return `null` from a scorer to skip a case.
+
+## Thresholds
+
+By default a scorer has to hit an exact match to pass. `thresholds` loosens that, mapping each scorer name to the minimum score you'll accept:
+
+```ts
+import { defineEval } from "eve/evals";
+import { Run, Text } from "eve/evals/scores";
+
+export default defineEval({
+  cases: [{ id: "hello", input: "Hello", expected: "Hello" }],
+  scores: [Run.didNotFail(), Text.includes()],
+  thresholds: { "run.didNotFail": 1, "text.includes": 0.5 },
+});
+```
+
+A case below a threshold gets the `scored` verdict — reported, but only fatal under `eve eval --strict`.
+
+## The scorer model
+
+`model` is only required when a model-backed scorer (one of the `Autoevals` wrappers) is present without its own per-scorer model override — and it's the scorer model, not the agent's. Eve only uses it for model-backed scoring, never to swap out the agent under test. Pass a string id (e.g. `"anthropic/claude-opus-4.8"`) to route through the Vercel AI Gateway, or hand it an AI SDK model instance to use that directly.
+
+For provider-specific scorer-model settings, use `modelOptions.providerOptions`. Individual Autoevals scorers can also take their own `model` / `modelOptions`, which win over the eval default.
+
+## Concurrency and timeouts
+
+`maxConcurrency` caps parallelism and `timeoutMs` bounds each case. Leave them off and `eve eval` runs up to 8 cases per eval at once. Lower `maxConcurrency` when cases contend for a shared resource — a rate-limited connection, or a sandbox-heavy fixture.
+
+## What to read next
+
+- [Checks](./checks): hard assertions that fail the build
+- [Reporters](./reporters): ship scores to Braintrust experiments