niceeval 0.4.5 → 0.5.0

package/docs-site/.mintignore

@@ -1,13 +0,0 @@
-# Mintlify automatically ignores these files and directories:
-# .git, .github, .claude, .agents, .idea, node_modules,
-# README.md, LICENSE.md, CHANGELOG.md, CONTRIBUTING.md
-
-# Draft content
-drafts/
-*.draft.mdx
-
-# Scraped source content (not documentation pages)
-sources/
-
-# Agent-only writing reference, not part of the published docs surface
-docs-ref/

package/docs-site/AGENTS.md

@@ -1,46 +0,0 @@
-# Mintlify 文档站指南
-
-`docs-site/` 是 NiceEval 的公开 Mintlify 文档站，面向正在学习和使用 NiceEval 的用户。这里的文档要回答“用户怎么完成任务”，不是记录内部设计讨论；内部设计、取舍和源码地图放在仓库根目录的 `docs/`。
-
-## 目录结构
-
-- `docs.json`：Mintlify 导航、主题、logo、navbar、redirects。
-- 顶层 `*.mdx`：英文入口页，例如 introduction、quickstart、installation。
-- `concepts/`：英文核心概念页，解释心智模型。
-- `guides/`：英文任务指南，按工作流组织。
-- `reference/`：英文 API / CLI 参考，列完整字段和选项。
-- `zh/`：中文文档。中文定位、概念命名和场景示例是公开叙事的准绳；英文页或 README 与中文冲突时，先按中文和当前代码核对，再同步其它入口。
-
-## 术语表
-
-- **NiceEval**：产品名。中文正文表达产品时用 `NiceEval`；命令、包名、配置文件、代码标识里用 `niceeval`。
-- **Eval**：一个评测用例。中文可以写“Eval”或“评估用例”
-- **Experiment**：可签入的运行配置。中文写“实验”或保留 `Experiment`，用于说明 agent、model、flags、runs、budget、sandbox 等运行维度。
-- **Adapter**：适配器，负责连接被测系统、鉴权、调用接口、把返回翻译成标准事件流。页面标题和导航可用 `Adapter`。
-- **Agent**：NiceEval 看到的被测对象连接。不要把 Agent 写成某个固定协议；具体协议属于 Adapter。
-- **Sandbox**：沙箱后端，回答“在哪里隔离运行”。不要和 Adapter 混成一层。
-- **Turn**：一次 `t.send()` / `t.respond()` 的结果。中文可写“一轮”或保留 `Turn`。
-- **StreamEvent / events**：标准事件流，是断言和报告读取的事实来源。
-- **HITL**：human-in-the-loop，人工介入。第一次出现时写全称或中文解释。
-- **OTel 接入**：Tier 2 的接入方式。只在讲 `send + OTel` 时使用，不要把 OTel 写进 Tier 1。
-- **Flags**：experiment 传入的 feature flags，经 `ctx.flags` 到 Adapter，经 `t.flags` 到 eval。不要写成 CLI flags，除非指命令行参数。
-- **Runner**：运行器。面向用户文档里避免写 “NiceEval core”；需要表达执行主体时写 NiceEval 或 runner。
-
-## 写作规则
-- 英语单词应该以大写开头
-- 只在 @docs-site/zh 下面更新中文版本，英语版本由其它 AI 翻译
-- 新增或重命名页面时同时更新 `docs.json`，必要时加 redirect，避免旧链接断掉。
-- 链接示例必须指向真实存在的 `examples/` 目录；当前完整示例主要在 `examples/zh/`。
-- 工作流写成 guide，字段全集写成 reference，概念边界写成 concepts。不要把一个页面同时写成教程、设计文档和 API 字典。
-- 命令、路径、flag、文件名、包名、代码标识用反引号。
-- 文案使用主动语态和短句。错误信息、限制和前置条件要直接说清楚下一步。
-- 写作指南: docs-site/docs-ref/00-index.md
-
-## 校验
-
-改 `docs-site/` 后，从仓库根目录运行：
-
-```sh
-PATH=/opt/homebrew/opt/node@22/bin:$PATH pnpm run docs:validate
-PATH=/opt/homebrew/opt/node@22/bin:$PATH pnpm run docs:links
-```

package/docs-site/concepts/adapter.mdx

@@ -1,287 +0,0 @@
----
-title: "Adapter: Connect an Agent to NiceEval"
-sidebarTitle: "Adapter"
-description: "The Adapter is the adapter you write. This page explains what the send function receives and returns, how system-under-test configuration is passed in, the three integration tiers, and where the capabilities on t come from."
----
-
-In NiceEval, an `Adapter` is the adaptation layer between the runner and the system under test. The Adapter implements the `send` function: it forwards eval-side actions such as `t.send()` to your application, then translates the application's response into NiceEval's standard `Turn`.
-
-Follow-up reading: [Write Send](/guides/write-send)
-
-## The adapter
-
-If the system under test speaks the standard OpenAI Chat Completions or Responses protocol, use an official adapter directly. For a homegrown frontend-backend protocol (HTTP, gRPC, WebSocket — anything works), write an Adapter: it knows how to authenticate, how to call your application, and how to translate the response into the standard event stream.
-
-One full round trip of `t.send()` crosses this boundary twice:
-
-![Full round trip of t.send: the eval calls t.send, the runner assembles TurnInput and ctx, the adapter calls your application and returns a Turn with the standard event stream.](/images/agent-turn-roundtrip-en.svg)
-
-Outbound: the driving verbs in the eval (`t.send` / `t.sendFile` / `t.respond`) assemble a `TurnInput` and the `ctx` for this conversation line, then call the Adapter's `send` once.
-
-Inbound: the Adapter translates the application's raw response into a `Turn` — where `events` is an array of event objects ordered by when they happened (real values below).
-
-The system under test's URL, auth, and protocol details are passed in through the Adapter factory's configuration. `defineExperiment` receives an already configured agent instance; the Adapter consumes that configuration inside the `send` closure, and forwards per-turn dynamic values (such as `ctx.model`, `ctx.flags`, `ctx.telemetry`) to the application with each request:
-
-```ts
-// agents/web-agent.ts
-import { defineAgent } from "niceeval/adapter";
-import type { Agent } from "niceeval/adapter";
-
-interface WebAgentOptions {
-  baseUrl: string;
-  apiKey?: string;
-}
-
-export function webAgent(options: WebAgentOptions): Agent {
-  const baseUrl = options.baseUrl.replace(/\/$/, "");
-
-  return defineAgent({
-    name: "web-agent",
-    async send(input, ctx) {
-      const response = await fetch(`${baseUrl}/api/turn`, {
-        method: "POST",
-        headers: {
-          "content-type": "application/json",
-          ...(options.apiKey ? { authorization: `Bearer ${options.apiKey}` } : {}),
-          ...ctx.telemetry?.headers,
-        },
-        body: JSON.stringify({
-          message: input.text,
-          files: input.files,
-          model: ctx.model,
-          flags: ctx.flags,
-        }),
-        signal: ctx.signal,
-      });
-
-      if (!response.ok) {
-        return {
-          status: "failed",
-          events: [{ type: "error", message: `HTTP ${response.status}` }],
-        };
-      }
-
-      const body = await response.json();
-      return {
-        status: "completed",
-        events: body.events,
-        data: body.data,
-        usage: body.usage,
-      };
-    },
-  });
-}
-
-// experiments/staging.ts
-import { defineExperiment } from "niceeval";
-import { webAgent } from "../agents/web-agent.ts";
-
-export default defineExperiment({
-  agent: webAgent({
-    baseUrl: "https://staging.example.com",
-    apiKey: process.env.STAGING_AGENT_API_KEY,
-  }),
-  model: "gpt-5.4",
-  flags: { promptVariant: "concise" },
-});
-```
-
-## Integration tiers
-
-Grouped by where the Adapter connects and what extra observability data it receives, integration comes in three tiers: **Tier 1 is send only** (not a single line of application code changes, and the full assertion set is already complete at this tier), **Tier 2 is send + OTel** (the application sends spans to NiceEval, buying the call waterfall in `niceeval view`), **Tier 3 is application changes + experiment flags** (feature A/B). What each tier costs, what it buys, and when to move up: see [Tier](/concepts/tier).
-
-The eval-side driving API — `t.send()`, `t.sendFile()`, `t.newSession()`, and HITL's `t.respond()` / `t.respondAll()` —
-all uniformly call the Adapter's `send`.
-
-How they converge and how to handle them in `send`: see [Write Send](/guides/write-send).
-
-## The send function
-
-Whether the Adapter connects to an HTTP service or a CLI in a sandbox, the interface exposed to the runner is exactly the same:
-
-```ts
-interface Agent {
-  readonly name: string;
-  send(input: TurnInput, ctx: AgentContext): Promise<Turn>;
-  // Optional: setup / teardown (once-per-sandbox lifecycle);
-  // optional observability members are not part of send — see "OTel Integration"
-}
-```
-
-`send` is the only function you must implement. There are only three types in its signature; take them one at a time.
-
-### Input: `TurnInput`
-
-```ts
-interface TurnInput {
-  readonly text: string;                          // Every turn: the text being sent; on answer turns it's the answer text (multiple answers joined by newlines)
-  readonly files?: readonly InputFile[];          // t.sendFile turns only: attached files (base64)
-  readonly responses?: readonly InputResponse[];  // Answer turns only (t.respond / t.respondAll): one structured answer per request
-  readonly outputSchema?: JsonSchema;             // Only when this turn declares output: the structured output requirement, lowered to JSON Schema
-}
-
-interface InputFile {
-  readonly filename?: string;     // Optional, for the adapter / model to reference
-  readonly mimeType: string;      // e.g. image/png
-  readonly dataBase64: string;    // base64 content, JSON-friendly, can go straight into the request body
-}
-
-interface InputResponse {
-  readonly requestId: string;    // Always present: which input.requested request this answers; used to match up when multiple requests are paused
-  readonly optionId?: string;    // Either this or text: the answer hit one of the request's option ids (approve / deny...)
-  readonly text?: string;        // Either this or optionId: a free-text answer (the request has no options, or the answer matches none)
-}
-```
-
-Whether the eval side calls `t.send()`, `t.sendFile()`, or `t.respond()`, it arrives at the Adapter as one ordinary `send` — the entire difference is in the fields:
-
-If your application interface does not accept files, ignore `files`.
-
-| Eval side | `text` | `files` | `responses` | `outputSchema` |
-|---|---|---|---|---|
-| `t.send(text)` | Sent text | — | — | Only when `output` is declared |
-| `t.sendFile(path, text?)` | Caption (may be empty string) | Present | — | Same |
-| `t.respond(...)` / `t.respondAll(...)` | Answer text | — | One per request | Same |
-
-`files`: ignore it if the interface does not accept files;
-`outputSchema`: forward it if the application interface accepts a schema (Chat Completions' `response_format`, Responses' `text.format`); it's fine if it doesn't — validation happens in the runner anyway (see the `Turn.data` rule below).
-
-#### Inputs for the different answers
-
-On HITL answer turns, the human verdict arrives in structured form via `input.responses` — the Adapter never needs to parse `text` to guess which sentence answers which request or whether it counts as approval. Every answer carries a `requestId`; `optionId` and `text` are mutually exclusive: if the answer hits one of the request's option ids it becomes `optionId` (the eval side has already validated it exists, so no typo can slip through silently); otherwise the whole sentence lands in `text` as free text. The four typical shapes:
-
-```ts
-// 1) A single pending request, answer hits an option (approve / deny work the same, only the optionId differs)
-await t.respond("approve");
-// → { text: "approve",
-//     responses: [{ requestId: "req_1", optionId: "approve" }] }
-
-// 2) Multiple requests paused at once — the object form matches them up explicitly
-await t.respond({ request, optionId: "deny" });
-// → { text: "deny",
-//     responses: [{ requestId: request.id, optionId: "deny" }] }
-
-// 3) The answer is none of the options → free text (requests waiting for extra info look like this)
-await t.respond("Change the recipient to ceo@corp.com");
-// → { text: "Change the recipient to ceo@corp.com",
-//     responses: [{ requestId: "req_1", text: "Change the recipient to ceo@corp.com" }] }
-
-// 4) respondAll: one answer per pending request, same optionId
-await t.respondAll("approve");
-// → { text: "approve\napprove",
-//     responses: [{ requestId: "req_1", optionId: "approve" },
-//                 { requestId: "req_2", optionId: "approve" }] }
-```
-
-The Adapter's corresponding obligations: hand the verdict back to the application by `requestId` (don't guess by order); for calls a human rejected, set the `action.result` event's `status` to `"rejected"` rather than `"failed"` — rejection is a human decision, not a tool failure, so `noFailedActions()` doesn't misfire and `calledTool(..., { status: "rejected" })` can assert it precisely.
-
-### Context: `AgentContext`
-
-```ts
-interface AgentContext {
-  // The three that are present every turn
-  readonly signal: AbortSignal;    // The runner's timeout and cancellation: pass it through to every request you make
-  readonly session: AgentSession;  // The state slot for this conversation line
-  log(msg: string): void;          // Writes to this turn's log, visible in niceeval view
-
-  // The three that appear per integration tier (see "Integration tiers")
-  readonly model?: string;         // Tier 1 model comparison: experiment.model passed through; forward it if the application interface accepts model selection
-  readonly reasoningEffort?: string;  // Reasoning effort: experiment.reasoningEffort passed through, same ownership as model
-  readonly telemetry?: Telemetry;  // Tier 2: this turn's W3C trace context (headers, a fresh traceparent per turn)
-  readonly flags: Readonly<Record<string, unknown>>;  // Tier 3 feature A/B: experiment.flags passed through; {} when unset
-
-  readonly sandbox?: Sandbox;      // Only Agents constructed by defineSandboxAgent receive this
-}
-
-interface AgentSession {
-  // Session continuation: server keeps the history (the shape for interfaces that accept a session id)
-  readonly id?: string;                      // The session id recorded for this line; undefined on a new conversation line
-  capture(id: string | undefined): void;     // Record the returned id; only lands if none has been recorded yet
-
-  // Session continuation: client carries the full history (Chat Completions shape)
-  history<TMsg>(): { get(): TMsg[]; commit(messages: TMsg[]): void };
-
-  // HITL: store and retrieve the paused-turn state (take clears on read — consumed once)
-  hold<T>(state: T): void;
-  take<T>(): T | undefined;
-
-  // Escape hatch: a free-form state slot, starts as {}, the framework never writes to it
-  readonly state: Record<string, unknown>;
-}
-```
-
-There are no flags to check on `ctx`. All three tier fields have "pass-through" semantics: `model` and `flags` are declared by the experiment and handed over verbatim by the runner — the Adapter's only job is to forward them to the application with the request, never to interpret them; `telemetry` only appears when OTel integration is configured, and inside `send` you only need to spread its `headers` into the request headers — the receiving endpoint is fixed in `defineConfig` and pointed at by the application at startup, not passed from here; see [OTel Integration](/guides/connect-otel).
-
-`session` is the private state of one conversation line, and NiceEval promises exactly one thing about it: **every `send` on the same conversation line receives the same `ctx.session`, and a new conversation line (the first turn of an eval, or after `t.newSession()`) receives a brand-new one.** The session continuation accessors (`id`/`capture`, `history`) and the HITL paused-turn accessors (`hold`/`take`) all live on it, and "the first turn" is simply the natural shape of a new conversation line — `id` is `undefined`, `history.get()` is an empty array, with no branch to write; `state` is the escape hatch beyond those accessors, and the framework never writes data into it.
-
-### Return: `Turn`
-
-```ts
-interface Turn {
-  readonly events: StreamEvent[];          // ★ The standard event stream — every Adapter's core product
-  readonly data?: JsonValue;               // This turn's structured output; only filled when input.outputSchema is present, validated by the runner against the declaration
-  readonly status: "completed" | "failed" | "waiting";  // waiting = paused for a human (HITL)
-  readonly usage?: Usage;
-}
-```
-
-`events` is just a **plain JS array**: one object per thing that happened this turn, in the order it actually happened. For a turn like "How warm is Beijing today?", where the agent checks the weather before answering, `events` looks like this:
-
-```ts
-[
-  { type: "action.called", callId: "c1", name: "get_weather", input: { city: "Beijing" } },
-  { type: "action.result", callId: "c1", output: { temp: 21 }, status: "completed" },
-  { type: "message", role: "assistant", text: "Beijing is 21°C today." },
-]
-```
-
-There are nine object types in total (`message`, `action.*`, `input.requested`… full list in the [Events Reference](/reference/events)). Assertions read this exact array: `t.calledTool("get_weather")` counts `action.called`, `t.reply` takes the last assistant `message`. Note that each `send` returns only **this turn's** array — stitching turns into a whole conversation line is the runner's job, covered in the next section. In most cases you don't hand-write these objects either: an official converter's return value is already a complete `Turn` with `events`, `usage`, and `status` filled in — how to pick a converter is covered in [Write Send](/guides/write-send).
-
-`data` is not a "put whatever" pocket; it has exactly one rule: **what goes in it is declared by the eval with a schema on the send; only a declaration produces data, and what you get is the declared type.**
-
-```ts
-// Eval side: the output declaration is both the requirement given to the application and the type source for turn.data
-const turn = await t.send("What's the total on this invoice?", {
-  output: z.object({ amount: z.number(), currency: z.string() }),
-});
-turn.data.amount;   // Typed as number — not unknown, no manual narrowing
-```
-
-The declaration, lowered to JSON Schema, reaches the Adapter via `input.outputSchema`; the Adapter puts the structured output from the application's response into `data` (e.g. `JSON.parse(reply.content)`). **Validation is enforced by the runner**: if `data` doesn't match the declaration, the turn fails immediately with the diff reported — there is no path where "a wrong object got stuffed in and the assertion silently passed". Conversely, turns with no `output` declaration have no `data`: applications that only return text never touch this field — don't copy the raw response body into it to fill space.
-
-### After send returns: where the four Turn fields go
-
-The moment `send` returns, the Adapter's work is done — everything after that is the runner's job. Each of the four fields has a definite destination:
-
-| Turn field | What the runner does with it | Where the eval author feels it |
-|---|---|---|
-| `events` | Appended to this conversation line's event stream; `deriveRunFacts` folds it into facts (`toolCalls`, `parked`, `messageCount`…); the last assistant `message` also updates `t.reply` | All scoped assertions: `t.calledTool()`, `t.messageIncludes()`…; `t.events` is directly queryable |
-| `status` | Records the turn's outcome; on `"waiting"`, collects this turn's `input.requested` into the pending list | `t.succeeded()` / `t.parked()`; after `"waiting"`, continue with `t.respond()` |
-| `usage` | Accumulated per turn into the conversation line and the whole run | `maxTokens` / `maxCost` scorers, usage in reports |
-| `data` | Validated against the `output` schema declared on the send — mismatch fails the turn immediately; saved only if it passes | `turn.data` (strongly typed by the declaration), `outputEquals` |
-
-The handle you get from `await t.send()` is a view onto this one Turn: turn-level assertions (things like `draft.parked()`) look only at this turn's `events`, while `t`-level assertions look at the whole conversation line's accumulation. So the Adapter never needs to "cooperate" with any assertion — fill in these four fields correctly, and assertions, scoring, and reporting all happen automatically downstream.
-
-## Where capabilities come from: proof by construction, not a questionnaire
-
-
-| What unlocks on `t` | What the evidence is | What you write |
-|---|---|---|
-| `t.send`, `t.sendFile`, `t.check`, `t.judge`… | Every Agent has them | Nothing |
-| Multiple `t.send()`, `t.reply`, `t.newSession()` | `send` wired up `ctx.session`'s continuation accessors (`history()` or `id` + `capture()`) — without them every turn is a fresh conversation | Nothing — the accessors are right on `ctx` |
-| `t.respond()` / `t.parked()` and other HITL | send has returned `"waiting"` + an `input.requested` event | Nothing — doing it is having it |
-| `t.calledTool()` / `t.toolOrder()` and other positive assertions | `action.*` present in events | Nothing — having events means you can assert |
-| `t.notCalledTool()` / `t.usedNoTools()` and other negative assertions being **trustworthy** | Events come from an official converter with a completeness proof | Nothing — the proof travels with the converter's return value |
-| `t.sandbox`, `t.sandbox.fileChanged()`, etc. | Constructed by `defineSandboxAgent` | Nothing |
-| `EvalResult.trace`, the `niceeval view` waterfall | A `tracing` block exists (spans only feed the waterfall, never assertions) | Nothing — you were writing the block anyway |
-
-The negative-assertion row is the only tiered one: `notCalledTool` asserts "it did not happen", which presupposes "the events are complete" — and that only holds when the source itself carries a completeness contract (SDK-native event stream pass-through, AI SDK's `result.steps`, Responses' `output`). Hand-written mappings carry no proof, so negative assertions warn at runtime that they are untrustworthy instead of silently passing. This also means there is no "declared but can't deliver" drift: trustworthiness follows where the events came from, not the author's self-assessment. The precise per-capability obligations are in the [Capabilities Reference](/reference/capabilities).
-
-
-## Related reading
-
-- [Write Send](/guides/write-send) — The hands-on tutorial: from sending one message to a full integration, seven steps, each unlocking a set of assertions.
-- [Connect Your Agent](/guides/connect-your-agent) — The integration overview: minimal integration, parameter channels, and the incremental map.
-- [Drive](/concepts/drive) — The eval-side view: how to use `t.send()`, `t.newSession()`, and HITL.
-- [Assert](/concepts/assert) — The complete assertion vocabulary driven by the standard event stream.
-- [Architecture Overview](/concepts/overview) — The four layers and their boundaries.

package/docs-site/concepts/assert.mdx

@@ -1,243 +0,0 @@
----
-title: "niceeval assertions: values, scoped facts, tests, and cost"
-sidebarTitle: "Assert"
-description: "niceeval's assertion vocabulary — value assertions, scoped assertions, test-as-scoring, and efficiency checks — plus gate vs soft severity and the outcome rules that fold them into a result."
----
-
-Assertion is the process of taking everything an agent did during an eval — every message, tool call, file change, and token spent — and folding it into a single, actionable outcome. niceeval gives you four assertion mechanisms that complement each other: some check values immediately, some assess the whole run after it completes, some execute tests inside the sandbox, and some measure efficiency. All four produce the same `Assertion` type and feed into the same outcome rules. For the fifth mechanism — asking a language model to judge open-ended quality — see [Judge](/concepts/judge).
-
-## The four assertion mechanisms
-
-<CardGroup cols={2}>
-  <Card title="1. Value assertions" icon="equals">
-    `t.check(value, matcher)` and `t.require(value, matcher)` evaluate a specific value immediately against a matcher from `niceeval/expect`. Use these for facts you can verify inline.
-  </Card>
-  <Card title="2. Scoped assertions" icon="crosshairs">
-    `t.succeeded()`, `t.calledTool()`, `t.messageIncludes()`, and friends are registered during `test(t)` but evaluated **after** the function returns, against the complete turn data. Use these for whole-run facts.
-  </Card>
-  <Card title="3. Test-as-scoring" icon="flask">
-    For sandbox evals, run project tests, build scripts, or focused probe commands from `test(t)`. Use this for coding tasks where file content and build results are the ground truth.
-  </Card>
-  <Card title="4. Efficiency assertions" icon="gauge">
-    `t.maxTokens()` and `t.maxCost()` turn token usage and estimated cost into scoreable dimensions. An agent that answers correctly but burns ten times the expected tokens should not score the same as one that answers efficiently.
-  </Card>
-</CardGroup>
-
-## Gate vs soft severity
-
-Every assertion carries a **severity** that determines how it influences the final outcome. There are exactly two severities:
-
-<Tabs>
-  <Tab title="gate">
-    A gate assertion is a hard requirement. If it fails, the entire eval is immediately classified as `failed` — regardless of how well every other assertion passed. Use gate for facts that must be true: "the agent called the correct tool", "the response parsed as valid JSON", "no shell commands errored."
-
-    Most matchers in `niceeval/expect` (`includes`, `equals`, `matches`, `satisfies`) default to gate. Scoped assertions like `t.succeeded()` and `t.calledTool()` also default to gate.
-  </Tab>
-  <Tab title="soft">
-    A soft assertion is a quality score with a numeric threshold. If the score falls below the threshold, the eval becomes `passed` rather than `failed` — a signal that there is a quality regression, but not a hard breakage. Soft failures only count as failures when you run with `--strict`.
-
-    Use soft for continuous judgments where the answer is "how good" rather than "correct or not": similarity scoring, LLM-as-judge factuality ratings, cost budgets you want to track without blocking CI.
-
-    Matchers that produce a continuous score (`similarity`) and all judge calls default to soft.
-  </Tab>
-</Tabs>
-
-You can override the default severity with a chain method on any matcher or assertion:
-
-```ts
-t.check(t.reply, includes("confirmed"));          // gate (default)
-t.check(t.reply, similarity(expected).gate());    // promote to gate
-t.maxTokens(80_000).atLeast(0.7);                 // demote to soft with a threshold
-```
-
-## Outcome rules
-
-Once all assertions are collected, the runner folds them into a single outcome in this order:
-
-```
-Execution error (timeout / thrown exception / author mistake)  →  failed
-Explicit t.skip(reason) was called                             →  skipped
-Any gate assertion failed                                       →  failed
-All gates passed, but at least one soft is below its threshold →  passed (flagged under --strict)
-Otherwise                                                       →  passed
-```
-
-<CardGroup cols={2}>
-  <Card title="passed" icon="circle-check" color="#22c55e">
-    No errors, all gate assertions passed, all soft assertions met their thresholds.
-  </Card>
-  <Card title="failed" icon="circle-xmark" color="#ef4444">
-    Execution error **or** at least one gate assertion did not pass. Hard failure.
-  </Card>
-  <Card title="passed (scored)" icon="chart-bar" color="#f59e0b">
-    All gates passed, but at least one soft fell below its threshold. Quality regression — silent by default, red under `--strict`.
-  </Card>
-  <Card title="skipped" icon="forward" color="#6b7280">
-    `t.skip("reason")` was called. Excluded from pass-rate calculations entirely.
-  </Card>
-</CardGroup>
-
-When you run an eval more than once (`runs > 1`), the per-eval summary becomes a **pass rate** (the fraction of runs that produced `passed`) and an average latency, rather than a single outcome.
-
-## 1. Value assertions — `niceeval/expect` matchers
-
-`t.check(value, assertion)` evaluates the assertion immediately and records the result. `t.require(value, assertion)` does the same but **throws immediately** if the assertion fails, aborting the rest of the test function. Use `t.require` for preconditions: if a required fact is false, there is no point continuing.
-
-The matchers available from `niceeval/expect`:
-
-```ts
-import {
-  includes,    // substring or regex match          (default: gate)
-  equals,      // deep equality                     (default: gate)
-  matches,     // Standard Schema (Zod etc.) check  (default: gate)
-  similarity,  // normalized Levenshtein 0–1        (default: soft)
-  satisfies,   // custom predicate + label          (default: gate)
-} from "niceeval/expect";
-```
-
-Usage examples:
-
-```ts
-// Check that the agent's reply contains a specific string
-t.check(t.reply, includes("order confirmed"));
-
-// Deep-equal check on structured output
-t.check(turn.data, equals({ status: "refund", amount: 42 }));
-
-// Validate structured output against a Zod schema
-t.check(turn.data, matches(z.object({ intent: z.enum(["refund", "ship"]) })));
-
-// Similarity with an explicit threshold
-t.check(t.reply, similarity("expected answer").atLeast(0.8));
-
-// Custom predicate
-t.check(turn.data, satisfies((d) => d.total > 0, "total is positive"));
-```
-
-Matchers are pure functions — `(value) => number` — so you can write your own and pass them to `t.check` without any special registration.
-
-## 2. Scoped assertions
-
-Scoped assertions are registered during `test(t)` but evaluated **after the function returns**, against the complete accumulated turn data. They read from the standard event stream that `t.send()` produces (see [Drive](/concepts/drive)) and its derived facts — so as long as your adapter produces correct events, these assertions work identically for every agent.
-
-<Warning>
-Scoped assertions only appear on `t` if the agent has declared the corresponding capability. Calling `t.calledTool()` when the agent has not declared `toolObservability: true` is a compile error.
-</Warning>
-
-### Run / session dimension
-
-```ts
-t.succeeded();                  // run completed with no failed actions and no unresolved HITL
-t.parked();                     // cleanly stopped on a HITL input.requested event
-t.messageIncludes("Regards,");  // all message events concatenated contain this string/regex
-```
-
-### Tool / action dimension
-
-```ts
-t.calledTool("bash", { input: { command: /^pwd/ }, count: 1 });
-t.notCalledTool("shell", { input: { command: /npm i/ } });
-t.toolOrder(["read_file", "write_file"]);   // relative order of tool calls
-t.usedNoTools();
-t.maxToolCalls(5);
-t.loadedSkill("memory-v2");                // sugar for calledTool("load_skill", ...)
-t.calledSubagent("researcher", { remoteUrl: /api\.example/ });
-t.noFailedActions();                       // no tool, subagent, or skill has status "failed"
-```
-
-The `input` argument to `calledTool` and `notCalledTool` supports a small matching language: a plain object performs deep partial matching, a `RegExp` matches against the serialized input, and a predicate function receives the raw input value.
-
-### Event stream dimension (low-level escape hatch)
-
-```ts
-t.event("input.requested", { count: 1 });
-t.notEvent("error");
-t.eventOrder(["action.called", "subagent.called"]);
-t.eventsSatisfy("read before write", (events) => /* custom predicate */ true);
-```
-
-All scoped assertions above are syntactic sugar for these low-level event stream queries. When none of the higher-level assertions fit your use case, you can drop down to `eventsSatisfy` and write an arbitrary predicate over the raw `StreamEvent[]`.
-
-### Structured output (on `turn`, not `t`)
-
-```ts
-const turn = await t.send("Return the result as JSON");
-turn.outputEquals({ status: "ok" });                         // deep equality on turn.data
-turn.outputMatches(z.object({ status: z.string() }));        // Standard Schema validation
-```
-
-### Workspace dimension (sandbox agents only)
-
-```ts
-t.sandbox.fileChanged("src/Button.tsx");
-t.sandbox.fileDeleted("src/old.ts");
-t.sandbox.diff.isEmpty();                      // no repository files were modified this turn
-t.sandbox.notInDiff(/sk-[A-Za-z0-9]/);         // diff contains no secrets / inline styles
-t.check(await t.sandbox.runCommand("npm", ["test"]), commandSucceeded());         // npm test exited 0
-t.check(await t.sandbox.runCommand("npm", ["run", "build"]), commandSucceeded()); // npm run build exited 0
-t.sandbox.noFailedShellCommands();
-```
-
-`t.sandbox.diff` is a queryable object. `t.sandbox.diff.get("src/Button.tsx")` returns the file's post-change content; `t.sandbox.diff.isEmpty()` checks whether any files changed; `t.sandbox.diff.matches(re)` and `t.sandbox.notInDiff(re)` run a regex over the full diff text.
-
-Scoped assertions follow one rule everywhere: **the receiver decides the scope, not the assertion name.** `t.*` aggregates every turn of the whole eval run (including any `t.newSession()` sessions); `session.*` (from `t.newSession()`) scopes to that one session; `turn.*` (from `t.send()`'s return value) scopes to that single turn only. Same vocabulary, different receiver — see [Drive](/concepts/drive) for what each receiver is.
-
-## 3. Test-as-scoring (sandbox evals)
-
-For sandbox coding evals, run validation commands inside `test(t)` and record their result as assertions.
-
-```ts
-import { commandSucceeded, includes } from "niceeval/expect";
-
-const testResult = await t.sandbox.runCommand("npm", ["test"]);
-t.check(testResult, commandSucceeded());
-
-const src = await t.sandbox.readSourceFiles({ extensions: ["ts"] });
-t.check(src.text(), includes(/z\.object\s*\(/));
-```
-
-You can also assert behavior through the standard event stream: `t.calledTool(...)`, `t.sandbox.noFailedShellCommands()`, `t.eventsSatisfy(...)`, and diff assertions such as `t.sandbox.fileChanged(...)`.
-
-## 4. Efficiency / cost assertions
-
-Token usage is a first-class scoring dimension. An agent that answers correctly but burns far more tokens than expected should not score identically to one that answers efficiently.
-
-```ts
-t.maxTokens(50_000);            // hard token limit for the entire run (gate by default)
-t.maxCost(0.5);                 // estimated cost cap in USD (requires a price table in config)
-t.maxTokens(80_000).atLeast(0.7);     // soft variant — tracked but only red under --strict
-t.check(t.usage.outputTokens, satisfies((n) => n < 10_000, "not verbose"));
-```
-
-`t.usage` is available anywhere inside `test(t)` and exposes `{ inputTokens, outputTokens, cacheReadTokens?, … }`. For sandbox agents, token counts are extracted from the transcript by the adapter; for remote agents, they are returned in `Turn.usage`.
-
-## Custom scorers
-
-A value assertion is just a function `(value) => number | Promise<number>`. You can write custom matchers using `makeAssertion`:
-
-```ts
-import { makeAssertion } from "niceeval/expect";
-import type { Assertion } from "niceeval/expect";
-
-function jsonValid(): Assertion {
-  return makeAssertion({
-    name: "jsonValid",
-    severity: "gate",
-    score: (value) => {
-      try { JSON.parse(String(value)); return 1; }
-      catch { return 0; }
-    },
-  });
-}
-
-t.check(t.reply, jsonValid());
-```
-
-Custom matchers compose with the same chain methods as built-ins: `.gate()`, `.atLeast(0.7)`.
-
-## Related pages
-
-- [Drive](/concepts/drive) — `t.send()`, `t.newSession()`, and HITL: how you produce the Turn data these assertions read from.
-- [Judge](/concepts/judge) — the fifth scoring mechanism, for open-ended quality that can't be expressed as a fixed rule.
-- [Adapter](/concepts/adapter) — how the standard event stream is produced, which scoped assertions depend on.
-- [Evals](/concepts/evals) — how assertions fold into the eval lifecycle and outcome types.