niceeval 0.4.4 → 0.4.5

package/docs-site/reference/builtin-agents.mdx

@@ -0,0 +1,161 @@
+---
+title: "Built-in agent capabilities reference"
+sidebarTitle: "Built-in Agent Capabilities"
+description: "Which capabilities NiceEval's built-in claude-code, codex, and bub adapters implement, which assertions they map to, and known limits."
+---
+
+The three built-in sandbox agents (`claude-code`, `codex`, `bub`) do not implement exactly the same capabilities — capabilities have no declaration layer and are proven entirely by construction and actual behavior, see the [capabilities reference](/reference/capabilities). This page inventories what each built-in agent does per capability, plus two built-ins for connecting AI SDK apps: the non-invasive HTTP adapter `uiMessageStreamAgent` (with HITL) and the result converter `fromAiSdk`.
+
+## Capability overview
+
+| Capability | Corresponding assertions / API |
+|---|---|
+| Send/receive messages (base contract) | `t.send()` (can be called multiple times), `t.reply`, `turn.outputEquals` / `outputMatches`, status-based `t.succeeded()` |
+| Event stream completeness | The whole scoped assertion set: `calledTool` / `toolOrder` / `usedNoTools` / `maxToolCalls` / `messageIncludes` / `noFailedActions` / `event` / `eventOrder`, etc.; **negative assertions (`notCalledTool` and friends) are only trustworthy with a complete event stream** |
+| Session continuation | Cross-turn memory assertions, `t.newSession()` session isolation |
+| HITL (human in the loop) | `t.parked()`, `t.requireInputRequest()`, `t.respond()` / `t.respondAll()` |
+| `tracing` | `EvalResult.trace`, the call waterfall in `niceeval view` |
+
+## What each built-in agent implements
+
+| Agent | Send/receive | Event stream | Session continuation | HITL | tracing | Notes |
+|---|---|---|---|---|---|---|
+| `claude-code` | ✅ | ✅ | ✅ (`claude --resume <id>`) | ❌ | ❌ | Built-in parser emits `compaction` events automatically; `t.event("compaction")` works |
+| `codex` | ✅ | ✅ | ✅ (`codex exec resume <id>`) | ❌ | ✅ (`http/json` → OTLP) | Built-in parser emits `compaction` events automatically |
+| `bub` | ✅ | ✅ | ✅ (`--session-id` + tape continuation) | ❌ | ✅ (`http/protobuf` → OTLP) | Built-in parser emits `compaction` events automatically |
+
+All three are constructed with `defineSandboxAgent` (`Agent.kind` is always `"sandbox"`), so `t.sandbox.fileChanged()` / diff assertions, file IO, and command execution work on all three agents regardless of this table.
+
+<Warning>
+None of the three built-in sandbox agents currently support HITL: their `send` only returns `"completed"` / `"failed"`, never `"waiting"`, and never emits `input.requested` events. When you need `t.respond()` / `t.requireInputRequest()`: if the system under test is an AI SDK `useChat` backend, use the built-in `uiMessageStreamAgent` below (v7 tool approval maps natively to HITL); for anything else write your own adapter: implement the `waiting` status + `input.requested` event + resume handoff — [`examples/zh/tier1/pi-sdk`](https://github.com/CorrectRoadH/niceeval/tree/main/examples/zh/tier1/pi-sdk) / [`examples/zh/tier1/claude-sdk`](https://github.com/CorrectRoadH/niceeval/tree/main/examples/zh/tier1/claude-sdk) both have ready-made hand-written HITL references.
+</Warning>
+
+## Per-agent details
+
+### claude-code
+
+- Connection: spawns `claude --print --dangerously-skip-permissions` in the sandbox and reads back the latest transcript from `~/.claude/projects/**/*.jsonl`.
+- Session continuation: appends `--resume <id>` when `ctx.session.id` has a value; the session id parsed from the transcript is written back via `ctx.session.capture()`.
+- Auth: `ANTHROPIC_API_KEY` (overridable via the `apiKey` option), optional `ANTHROPIC_BASE_URL`.
+- Supports `mcpServers` and `skills` (GitHub `org/repo`) options.
+- Does not declare `tracing`: no OTLP receiver is started and `EvalResult.trace` is empty.
+
+### codex
+
+- Connection: runs `codex exec --json` in the sandbox (`codex exec resume <id> --json` when continuing), using stdout JSONL as the transcript.
+- Auth: `CODEX_API_KEY` (not `OPENAI_API_KEY`), optional `CODEX_BASE_URL` for OpenAI-compatible proxies.
+- `tracing` is configured through the `[otel.trace_exporter.otlp-http]` section of `~/.codex/config.toml`, protocol `http/json`.
+- Supports `mcpServers` and `skills` options.
+
+### bub
+
+- Connection: runs `bub run` + `--session-id` in the sandbox and reads the tape from `~/.bub/tapes/<hash>.jsonl` as the transcript.
+- Auth: `BUB_API_KEY` + `BUB_API_BASE` (OpenAI-compatible proxy).
+- `tracing` is injected via environment variables (`OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`, etc.), protocol `http/protobuf`.
+- Supports the `pythonPlugins` option.
+- Installed via `uv tool install` (not an npm package); the first install builds a checkpoint cache to speed up later sandboxes.
+
+## `uiMessageStreamAgent`: built-in non-invasive adapter for AI SDK apps (with HITL)
+
+`uiMessageStreamAgent` (exported from `niceeval/adapter`) talks non-invasively to an HTTP endpoint speaking the [UI Message Stream protocol](https://ai-sdk.dev/docs/ai-sdk-ui/stream-protocol) (the standard SSE of AI SDK `useChat` backends) — it only `fetch`es, never imports app code, so it connects wherever the app is deployed:
+
+```ts
+import { uiMessageStreamAgent } from "niceeval/adapter";
+
+export default uiMessageStreamAgent({
+  name: "my-assistant",
+  url: "https://my-app.example.com/api/chat",
+  body: (ctx) => ({ model: ctx.model }),   // When the app supports request-level model selection, model comparison needs zero changes
+});
+```
+
+Capabilities it handles for you:
+
+- **Send/receive + event stream**: SSE frames are reduced by the `ai` package's official framework-agnostic reducer `readUIMessageStream` (the same one inside `useChat`); tool calls/results/message text are built directly from message parts — **the app is not required to wire up OTel**.
+- **Session continuation**: the protocol is server-stateless, "client carries the full history" — the factory stores the whole `UIMessage[]` with `ctx.session.history()` and replays it verbatim each turn; a new conversation line (after `t.newSession()`) gets an empty history.
+- **HITL**: AI SDK v7 tool approval (tools with `needsApproval: true`) maps natively — when a part stops at `approval-requested`, the whole turn is `status: "waiting"` + `input.requested`; `t.respond("approve" / "deny")` is translated into `approval-responded`, rewriting that part in place and resending `messages` verbatim to trigger the server to continue (identical protocol behavior to a real frontend's `addToolApprovalResponse()` + `sendMessage()`; there is no separate approve endpoint). Denied calls land in the event stream as `rejected`, and by default carry a "do not retry" reason (`denyReason` can override) — without one, models often resend the very same call.
+- **usage / waterfall**: UI Message Stream protocol frames carry no usage, so usage assertions like `t.maxTokens` have no data by default on this built-in (an app putting usage into message metadata is the app's own protocol extension). The waterfall is a separate matter: when the app has OTel instrumentation (e.g. the official `@ai-sdk/otel`), send spans to NiceEval per [OTel integration](/guides/connect-otel) and `niceeval view` gets a full waterfall — spans only feed the waterfall, never assertions.
+
+You need `ai` installed in the eval project (optional peer dependency; the protocol reducer comes from it). Full runnable example: [`examples/zh/tier1/ai-sdk-v7`](https://github.com/CorrectRoadH/niceeval/tree/main/examples/zh/tier1/ai-sdk-v7).
+
+## SDK event stream converters: `fromClaudeSdkMessages` / `fromPiAgentEvents` / `fromCodexThreadEvents`
+
+Each agent SDK's streaming protocol is a generic protocol defined by the SDK, not some app's private format — the native-frame-to-standard-event mapping knowledge lives in the official package (exported from `niceeval/adapter`), so when writing your own non-invasive adapter only the **transport glue** remains (which endpoint the app puts the stream on, which endpoint approvals go through). Driving the loop frame by frame is itself an official piece (`driveFrameStream`, see the next section), so no hand-written `for` loop:
+
+```ts
+import { sseJsonFrames, fromClaudeSdkMessages, driveFrameStream } from "niceeval/adapter";
+
+const frames = sseJsonFrames<SDKMessage>(res.body);   // Standard SSE → per-frame JSON
+const stream = fromClaudeSdkMessages();               // SDKMessage → standard events
+return driveFrameStream(frames, stream, ctx);         // Feed stream.add() frame by frame, summarize into a Turn
+```
+
+- **`fromClaudeSdkMessages`** (Claude Agent SDK `SDKMessage`): `assistant` text/tool_use blocks, `user` tool_result blocks, `system`/`permission_denied` (→ `rejected`), `result` usage; `markRejected()` registers denied calls.
+- **`fromPiAgentEvents`** (pi-agent-core `AgentEvent`): text/thinking/usage from `message_end`, tool pairs from `tool_execution_start/end`.
+- **`fromCodexThreadEvents`** (Codex SDK `ThreadEvent`): message-class frames from `agent_message`/`reasoning`, tool items (`command_execution` / `mcp_tool_call` / `file_change` / `web_search` → `action.called` + `action.result`), usage from `turn.completed`, error frames. The waterfall needs the codex CLI's native OTLP (the `[otel]` block in config.toml) sending spans to NiceEval, normalized with the official `mapCodexSpans`.
+
+Runnable references: the claude-sdk / pi-sdk / codex-sdk adapters under [`examples/zh/tier1`](https://github.com/CorrectRoadH/niceeval/tree/main/examples/zh/tier1) are exactly these three converters + `driveFrameStream` + transport glue.
+
+## Generic "assembly" pieces: `driveFrameStream` / `deltaStream` + `ctx.session`
+
+Inside a hand-written send, only three truly independent parts exist: transport (how to send), reduce (raw data → events, handled by the converters above), and orchestration (session continuation + HITL pause/resume). The third part is protocol-agnostic, pure control-flow pattern — the accessors for session continuation (`id`/`capture`, `history`) and the HITL paused-turn scene (`hold`/`take`) hang directly off `ctx.session` (`AgentSession`, see the [Adapter concept](/concepts/adapter#context-agentcontext)); nothing to import separately and nothing to declare. What actually remains reusable across protocols is just two official pieces:
+
+| Piece | What it solves |
+|---|---|
+| `driveFrameStream(cursor, reducer, ctx, onFrame?)` | Feeds the reducer frame by frame, handles transport frames, detects the HITL pause signal — the loop itself collapsed into one function. On pause the `cursor` is not closed: store it with `ctx.session.hold(cursor)` inside `onFrame`, and on the answer turn `ctx.session.take()` retrieves it and keeps reading, without issuing a new request |
+| `deltaStream(spec)` | Generic accumulator for token-by-token / per-argument deltas — for protocols with "no whole-chunk landing frames" like raw OpenAI/Anthropic streaming APIs or your own hand-rolled token-by-token backend; declare "which operation this frame belongs to" (text delta / tool-argument delta / finalize) and it manages assembly timing itself |
+
+```ts
+// First thing in send: check for a held scene (HITL answer turn)
+const pending = ctx.session.take<Pending>();
+if (pending) {
+  // ...match verdicts from input.responses, hand the approval back to the app...
+  return driveFrameStream(pending.cursor, pending.stream, ctx, onFrame);
+}
+
+// Normal turn: session continuation via ctx.session.id / capture; on pause, ctx.session.hold stores the cursor
+return driveFrameStream(cursor, stream, ctx, (frame, derived) => {
+  ctx.session.capture(stream.sessionId);
+  const gated = derived.find((e) => e.type === "action.called" && e.name === GATED_TOOL_NAME);
+  if (gated?.type === "action.called") {
+    ctx.session.hold<Pending>({ cursor, stream, toolUseId: gated.callId });
+    return { pause: { id: gated.callId, action: GATED_TOOL_NAME, options: [{ id: "approve" }, { id: "deny" }] } };
+  }
+});
+```
+
+The three Tier 1 examples (claude-sdk and pi-sdk use `driveFrameStream` + `ctx.session.hold`/`take`; codex-sdk uses only `driveFrameStream`) have been rewritten on these pieces — the adapters are down to transport plus the "does this frame need extra handling" decision in `onFrame`, with no hand-written loops or module-level Maps.
+
+## `fromAiSdk`: AI SDK result → event stream converter
+
+`fromAiSdk` (exported from `niceeval/adapter`) is used inside adapters you write yourself (e.g. server-side direct construction for an HTTP web agent, see `examples/zh/ai-sdk/`). It maps an AI SDK `generateText` / `streamText` result into `{ events, usage, status }` — exact `toolCallId` pairing, order-faithful, `tool-error` mapped to a failed `action.result`, usage aggregation (v4 / v5 / v7 field drift all covered) — laid straight into the `Turn`:
+
+```ts
+import { defineAgent, fromAiSdk } from "niceeval/adapter";
+
+export default defineAgent({
+  name: "my-ai-sdk-agent",
+  async send(input) {
+    const result = await generateText({ model, tools, prompt: input.text });
+    return { ...fromAiSdk(result), data: result.text };
+  },
+});
+```
+
+The `status` comes from the converter: `"waiting"` when there is a tool approval pending human review (with an `input.requested` event attached), otherwise `"completed"`. Session continuation (multi-turn resume), handing HITL verdicts back, and tracing depend on how you write your own `send`.
+
+## How to choose
+
+- The system under test is an AI SDK app (`useChat` backend): the built-in `uiMessageStreamAgent` connects non-invasively with zero mapping (including HITL).
+- The system under test is another agent system (HTTP / gRPC service): connect non-invasively and hand-write the event mapping (official converters cover most of it) — all five examples under [`examples/zh/tier1`](https://github.com/CorrectRoadH/niceeval/tree/main/examples/zh/tier1) take this path; if the app already has OTel instrumentation, wire it up for a bonus waterfall.
+- You need the call waterfall (the trace in `niceeval view`): all three of `claude-code` / `codex` / `bub` emit OTLP themselves (claude-code via the CLI's beta telemetry; its spans carry structure and timing only); adapters you write yourself just declare `tracing` (see the OTel guide).
+- You need human approval / multi-step confirmation (HITL): `uiMessageStreamAgent` supports it natively (AI SDK v7 tool approval); none of the three sandbox agents do; for other systems write your own adapter, assembled from `driveFrameStream` + `ctx.session.hold`/`take` (the pi-sdk / claude-sdk Tier 1 examples have ready-made patterns).
+- You want to run a coding agent that edits code, inspect diffs, and judge tool calls: `claude-code` / `codex` / `bub` (send/receive + event stream + session continuation + workspace + sandbox).
+- Your backend protocol is home-grown with no official SDK converter: do not hand-write send from scratch — pick a reduce shape (a small mapping for whole-chunk landing, `deltaStream` for token-by-token deltas), take `ctx.session.id`/`capture` or `ctx.session.history` for session continuation, add `ctx.session.hold`/`take` if you have HITL, and the assembly is your send.
+
+## Related reading
+
+- [Connect your agent](/guides/connect-your-agent) — how to implement each capability in your own adapter and which assertions it maps to.
+- [Sandbox Agent](/guides/sandbox-agent) — how to run the built-in sandbox agents and write your own.
+- [defineAgent reference](/reference/define-agent) — full parameters of `defineAgent` / `defineSandboxAgent`.
+- [OTel integration](/guides/connect-otel) — send your app's spans to NiceEval in exchange for the call waterfall in `niceeval view`.

package/docs-site/reference/capabilities.mdx

@@ -0,0 +1,76 @@
+---
+title: "Capabilities reference"
+sidebarTitle: "Capabilities"
+description: "How NiceEval knows what your adapter has done: there are no declarative capability bits — everything is proven by construction and actual behavior."
+---
+
+Capabilities answer one question: **what has your adapter actually done** — which decides which assertions are trustworthy and which services the runner prepares for you.
+
+There is no `capabilities` field on `defineAgent` / `defineSandboxAgent` — **nothing needs to be declared**. Every capability is proven by how the agent is constructed or by what it actually returns. The decision table is below.
+
+## Where capabilities come from: one decision table
+
+| Capability | How it is obtained | What you write |
+|---|---|---|
+| `t.sandbox` / `t.sandbox.fileChanged()` and other filesystem assertions | Constructed with `defineSandboxAgent` (`Agent.kind` is always `"sandbox"`) | Nothing |
+| `tracing` (OTLP receiver → `niceeval view` waterfall) | CLI-style agents write the `tracing` field; long-running apps configure a fixed `telemetry` port in the config | Nothing (the configuration is what you would write anyway) |
+| Tool/action assertions (`t.calledTool()` etc., including negative assertions) | Use an official converter (`fromClaudeSdkMessages`, `fromAiSdk`, etc.) — the `action.*` events in the return value are exactly what you get | Nothing; with manual mapping, event completeness is decided by your mapping code |
+| Cross-turn memory, `t.newSession()` isolation | Used `ctx.session` (`id`/`capture`, `history`) to continue context | Nothing — wire it up and continuation works; skip it and every turn is a fresh conversation, with no error either way |
+| HITL (`t.respond()`) | `send` has returned `status: "waiting"` + `input.requested` | Nothing |
+
+A capability you have not implemented = the corresponding events/fields never appear, and the related assertion group is unavailable (not an "error" — the assertions simply have nothing to judge from); basic assertions (text, structured output, judge) are always available.
+
+## Why there is no declaration layer
+
+An earlier design had boolean declarations like `capabilities: { conversation, toolObservability }` to cover "the parts code cannot prove". That layer has been removed, because what it promised should never have been guaranteed by a declaration in the first place:
+
+- **Construction is proof**: using `defineSandboxAgent` means the system under test edits files in a sandbox — there is no need to declare `workspace: true` on top of that.
+- **Fields are proof**: writing `tracing` (how to hand the OTLP endpoint to the CLI) makes tracing real. The field itself is the evidence of use; a boolean sitting next to it would just repeat it.
+- **Source is proof**: `fromAiSdk(result)` consumes `result.steps`, which the AI SDK contract guarantees is the **complete** tool-loop record, so the return value carries its own proof of a complete event stream; built-in CLI parsers work the same way. With manual mapping, whether events are complete depends on whether your mapping code covers everything — that is not something a declaration can guarantee, and declaring it would not make the code any more complete.
+- **Behavior is proof**: HITL has no capability bit — `send` returns `status: "waiting"` + an `input.requested` event, so the behavior is directly observable.
+- **Wire it up and it continues**: session continuation is no longer a "declared to exist" switch. `ctx.session` is always there; use the `id`/`capture` or `history` accessors and context continues, skip them and every turn is a brand-new session — both are normal states, neither errors, and the difference is purely how your adapter is actually written.
+
+```ts
+export default defineAgent({
+  name: "my-ai-sdk-agent",
+  async send(input, ctx) {
+    const result = await generateText({ model, tools, prompt: input.text });
+    return { ...fromAiSdk(result), data: result.text };
+    // ↑ The return value carries complete tool events; no capability declaration needed
+  },
+});
+```
+
+## What happens when a manual mapping misses events
+
+With the declaration layer gone, "false claims" no longer exist either — whatever tool events your manual mapping misses, assertions like `t.calledTool()` see exactly the post-omission picture, and negative assertions (`notCalledTool()`) may "look like they pass" simply because the events were never written. That is not a framework misjudgment; it is a completeness problem in the manual mapping itself: **whether to trust a negative assertion depends on whether you have confirmed your mapping code covers every tool call** — there is no framework-provided middle layer vouching for you.
+
+## Item-by-item quick reference
+
+### Tool/action event completeness
+
+Meaning: whether every tool call in the turn appears in events as `action.called` + `action.result` ([events reference](/reference/events)). There are only two sources: official converters guarantee it per the contract of the protocol under test; manual mapping is decided by your code. OTel spans are not a source — spans only feed the waterfall, never the event stream (see [OTel integration](/guides/connect-otel)).
+
+### Session continuation
+
+Meaning: using `ctx.session.id` / `capture` or `ctx.session.history` to connect requests to the same backend session / accumulated history. With it, cross-turn memory assertions and `t.newSession()` isolation assertions become meaningful. Both implementation styles are shown in [Write send](/guides/write-send).
+
+### `tracing` — emitting OTLP traces
+
+What the runner consumes: at runtime it starts a local OTLP receiver and hands this turn's trace context to you via `ctx.telemetry.headers`; normalized spans are drawn as a waterfall in `niceeval view` — drawing only, never feeding assertions. Source: CLI-style agents write the `tracing` field (`env` / `configure` hands the endpoint to the process under test); long-running apps configure a fixed port with `telemetry: { port }` in `niceeval.config.ts` (see [OTel integration](/guides/connect-otel)).
+
+### `kind: "sandbox"` — sandbox and file changes
+
+What the runner consumes: it prepares a sandbox (Docker / Vercel / E2B) for each attempt and hands it to the adapter via `ctx.sandbox`; this unlocks diff assertions like `t.sandbox.fileChanged()`. Source: constructed with `defineSandboxAgent` — an internal discriminant field you never write.
+
+### Compaction visibility
+
+Meaning: when the agent under test compacts its context, it emits a `compaction` event, making `t.event("compaction")` meaningful. Source: built-in parsers emit it automatically; a hand-written parser has it if you covered that logic.
+
+## Related reading
+
+- [Events reference](/reference/events) — exactly what the event stream promises.
+- [Adapter concept](/concepts/adapter) — the full `ctx`/`Turn` contract and where capabilities come from.
+- [Connect your agent](/guides/connect-your-agent) — how to follow each integration path.
+- [OTel integration](/guides/connect-otel) — the full walkthrough of spans feeding only the waterfall, never assertions.
+- [Built-in agent capabilities](/reference/builtin-agents) — what each built-in adapter has done.

package/docs-site/reference/cli.mdx

@@ -0,0 +1,120 @@
+---
+title: "NiceEval CLI: commands, flags, and exit codes reference"
+sidebarTitle: "CLI"
+description: "NiceEval CLI reference for exp, init, list, clean, and view, plus experiment selection, eval filtering, sandbox, concurrency, budget, and JUnit CI output."
+---
+
+The NiceEval CLI is the entry point for discovery, execution, and result viewing. Actual eval execution follows an experiment-first model: **`exp` selects a commit-ready run configuration**, and only the positional arguments after the experiment filter eval IDs by prefix. Agent, model, and feature flags live in `experiments/`, not in temporary CLI flags.
+
+## Commands
+
+<CardGroup cols={2}>
+  <Card title="npx niceeval exp &lt;group&gt; [id-prefix]" icon="flask">
+    Run a named experiment group or config. Optional trailing positional args filter by eval ID prefix.
+  </Card>
+  <Card title="npx niceeval init" icon="wand-magic-sparkles">
+    Generate `evals/`, `niceeval.config.ts`, and sample files.
+  </Card>
+  <Card title="npx niceeval list" icon="list">
+    Discover and print all evals without running them.
+  </Card>
+  <Card title="npx niceeval clean" icon="trash">
+    Remove historical `.niceeval/` artifacts.
+  </Card>
+  <Card title="npx niceeval view" icon="eye">
+    Open the result viewer for the most recent run.
+  </Card>
+</CardGroup>
+
+## Output language
+
+NiceEval supports localized CLI and runtime copy without configuration in `niceeval.config.ts`. When you need fixed output language, use environment variables:
+
+```bash
+NICEEVAL_LANG=en npx niceeval list
+NICEEVAL_LANG=zh-CN npx niceeval list
+```
+
+Detection order is `NICEEVAL_LANG`, `NICEEVAL_LOCALE`, `LC_ALL`, `LC_MESSAGES`, then `LANG`. Values starting with `zh` use `zh-CN`; everything else uses `en`. If nothing is set, the default is `zh-CN`. This affects terminal/runtime copy only. It does not change machine fields in result JSON or translate judge prompts.
+
+## `npx niceeval exp [group|config] [id-prefix...]`
+
+```bash
+# Run every experiment under experiments/
+npx niceeval exp
+
+# Run one experiment group
+npx niceeval exp compare-models
+
+# Inside that group, only run evals whose ID starts with weather
+npx niceeval exp compare-models weather
+```
+
+<Note>
+  Eval filters only appear after the experiment selector. Bare `npx niceeval weather` does not run. Use `npx niceeval exp local weather` or `npx niceeval exp compare weather`.
+</Note>
+
+## Common flags
+
+<ParamField query="--agent" type="string">
+  Experiment runs do not support this flag. To switch agents, add or duplicate a config file in `experiments/`.
+</ParamField>
+
+> There is no CLI flag for selecting a sandbox backend. Backend choice only lives in code: set `sandbox` on the experiment (`defineExperiment`) or in `niceeval.config.ts` (`defineConfig`) as a project-level fallback, using `dockerSandbox()` / `vercelSandbox()` / `e2bSandbox()` from `niceeval/sandbox`.
+
+<ParamField query="--model" type="string">
+  Experiment runs do not support this flag. To switch models, add or duplicate an experiment file and change `model`.
+</ParamField>
+
+<ParamField query="--max-concurrency" type="number">
+  Set how many evals may run at the same time.
+</ParamField>
+
+<ParamField query="--runs" type="number">
+  Set how many times each eval runs, commonly for pass@N.
+</ParamField>
+
+<ParamField query="--early-exit" type="boolean">
+  Stop the remaining attempts for an eval after one attempt passes.
+</ParamField>
+
+<ParamField query="--timeout-ms" type="number">
+  Timeout for one eval.
+</ParamField>
+
+<ParamField query="--budget" type="number">
+  Budget limit for the whole run.
+</ParamField>
+
+<ParamField query="--strict" type="boolean">
+  Recommended in CI. Make failures show up more explicitly in the exit code.
+</ParamField>
+
+## `list`
+
+```bash
+npx niceeval list
+```
+
+Use this to confirm eval discovery, IDs, and config loading.
+
+## `exp`
+
+```bash
+npx niceeval exp compare-models
+npx niceeval exp compare-models weather-tool
+```
+
+Run a named experiment and compare agents, models, or flags. Positional args after the experiment filter eval IDs by prefix.
+
+## `view`
+
+```bash
+npx niceeval view
+```
+
+Open the local result viewer. By default it shows the latest run under `.niceeval/`.
+
+## Exit codes
+
+The CI rule is simple: when a gate fails or the run itself fails, the command should exit non-zero. When everything passes, it should exit 0. Use `--strict` when you want the stricter CI behavior explicitly.

package/docs-site/reference/define-agent.mdx

@@ -0,0 +1,168 @@
+---
+title: "defineAgent and defineSandboxAgent: adapter reference"
+sidebarTitle: "defineAgent"
+description: "Reference for defineAgent and defineSandboxAgent: AgentContext, AgentSession, the Sandbox interface, and shared sandbox helpers."
+---
+
+Every NiceEval agent is an adapter: code that knows how to drive a specific backend and translate its output into the standard event stream. The runner only calls `agent.send(input, ctx)`.
+
+![Full round trip of one t.send: the eval calls t.send, the runner assembles TurnInput and ctx, the adapter returns a Turn with standard events.](/images/agent-turn-roundtrip-en.svg)
+
+## `defineAgent`
+
+Use this for direct agent integrations:
+
+```ts
+import { defineAgent } from "niceeval/adapter";
+```
+
+<ParamField body="name" type="string" required>
+  Unique agent name. Experiments reference the agent object, and reports group results by this name.
+</ParamField>
+
+<ParamField body="tracing" type="AgentTracing">
+  OTLP export configuration, used by CLI / child-process agents: declares how to hand the receiver endpoint to the process under test (`env` / `configure`). Long-running HTTP apps do not need this field — configure a fixed port with `telemetry: { port }` in `niceeval.config.ts` instead (see [OTel integration](/guides/connect-otel)). Spans only feed the `niceeval view` waterfall, never assertions.
+</ParamField>
+
+<ParamField body="spanMapper" type="(spans: TraceSpan[]) => TraceSpan[]">
+  Thin mapper from native spans to canonical GenAI semconv; it only affects semantic coloring in the waterfall. Omit it to fall back to the generic heuristic. For what maps to what in the waterfall and how to write your own (`tagSpan` / `heuristicTag`), see [OTel integration](/guides/connect-otel).
+</ParamField>
+
+<ParamField body="send" type="(input: TurnInput, ctx: AgentContext) => Promise<Turn>" required>
+  Core function that drives the agent. Must return a standard `Turn`.
+</ParamField>
+
+`Agent.kind` produced by `defineAgent` is always `"remote"` (an internal discriminant field you never declare). There are no capability bits on `t` that need declaring to unlock — filesystem assertions like `t.sandbox` exist only on agents constructed with `defineSandboxAgent` (`kind: "sandbox"`); everything else is judged by which events `send` actually returned and whether `ctx.session` was used. See the [capabilities reference](/reference/capabilities).
+
+## `input` (`TurnInput`)
+
+<ResponseField name="text" type="string">
+  Text passed by the current `t.send(...)`.
+</ResponseField>
+
+<ResponseField name="files" type="readonly InputFile[] | undefined">
+  Files attached to this turn (images, etc.). Adapters that do not support multimodal input can ignore it.
+</ResponseField>
+
+<ResponseField name="responses" type="readonly InputResponse[] | undefined">
+  Present only on answer turns (`t.respond` / `t.respondAll`): per-request structured answers, matched by `requestId`.
+</ResponseField>
+
+## `ctx` (`AgentContext`)
+
+<ResponseField name="signal" type="AbortSignal">
+  For timeouts and cancellation; pass it through to every request you make.
+</ResponseField>
+
+<ResponseField name="model" type="string | undefined">
+  Model comparison for Tier 1: `experiment.model`, passed through; forward it if the app's interface accepts a model selection.
+</ResponseField>
+
+<ResponseField name="flags" type="Readonly<Record<string, unknown>>">
+  Feature A/B for Tier 3: `experiment.flags`, passed through; `{}` when not configured.
+</ResponseField>
+
+<ResponseField name="telemetry" type="Telemetry | undefined">
+  Present only when OTel integration is configured: this turn's W3C trace context (`headers`, a fresh `traceparent` per turn).
+</ResponseField>
+
+<ResponseField name="sandbox" type="Sandbox | undefined">
+  Only agents constructed with `defineSandboxAgent` receive this.
+</ResponseField>
+
+<ResponseField name="session" type="AgentSession">
+  State slot for the current conversation line. Every `send` on the same conversation line gets the same `ctx.session`; a new conversation line (the eval's first turn, or after `t.newSession()`) gets a fresh one:
+
+  - `id?: string` / `capture(id): void` — session continuation when the server keeps history: `id` is the session id recorded on this line (`undefined` on a new line), `capture` records the returned id (it only lands when none has been recorded yet).
+  - `history<TMsg>(): { get(): TMsg[]; commit(messages: TMsg[]): void }` — session continuation when the client carries the full history.
+  - `hold<T>(state: T): void` / `take<T>(): T | undefined` — storage for the HITL paused-turn scene; `take` clears on read (consume once).
+  - `state: Record<string, unknown>` — escape hatch, starts as `{}`, never written by the framework.
+
+  Full contract in the [Adapter concept](/concepts/adapter#context-agentcontext).
+</ResponseField>
+
+<ResponseField name="log" type="(msg: string) => void">
+  Writes a message to the run log, visible in `niceeval view`.
+</ResponseField>
+
+## Direct agent example
+
+```ts
+export default defineAgent({
+  name: "echo",
+  async send(input) {
+    return {
+      status: "completed",
+      events: [{ type: "message", role: "assistant", text: input.text }],
+    };
+  },
+});
+```
+
+## `defineSandboxAgent`
+
+Use this for coding agent CLIs. The produced `Agent.kind` is always `"sandbox"` — filesystem assertions like `t.sandbox` and `t.sandbox.fileChanged()` unlock only on this kind of agent. `ctx.sandbox` is the handle to the current isolated environment:
+
+<ResponseField name="runCommand(cmd, args?, opts?)" type="(cmd: string, args?: string[], opts?: CommandOptions) => Promise<CommandResult>">
+  Run a command.
+</ResponseField>
+
+<ResponseField name="runShell(script, opts?)" type="(script: string, opts?: CommandOptions) => Promise<CommandResult>">
+  Run a shell script.
+</ResponseField>
+
+<ResponseField name="readFile(path)" type="(path: string) => Promise<string>">
+  Read a file inside the sandbox.
+</ResponseField>
+
+<ResponseField name="fileExists(path)" type="(path: string) => Promise<boolean>">
+  Check whether a file exists inside the sandbox.
+</ResponseField>
+
+<ResponseField name="writeFiles(files, targetDir?)" type="(files: Record<string, string>, targetDir?: string) => Promise<void>">
+  Write file contents by path.
+</ResponseField>
+
+<ResponseField name="uploadFiles(files, targetDir?)" type="(files: SandboxFile[], targetDir?: string) => Promise<void>">
+  Upload local files into the sandbox.
+</ResponseField>
+
+<ResponseField name="uploadDirectory(localDir, targetDir?, opts?)" type="(localDir: string, targetDir?: string, opts?: { ignore?: string[] }) => Promise<void>">
+  Upload an entire local directory, typically used to write the eval's starting files.
+</ResponseField>
+
+<ResponseField name="sandboxId" type="string">
+  Id of the current sandbox instance.
+</ResponseField>
+
+<ResponseField name="stop()" type="() => Promise<void>">
+  Stop the sandbox.
+</ResponseField>
+
+```ts
+import { defineSandboxAgent } from "niceeval/adapter";
+
+export default defineSandboxAgent({
+  name: "my-cli-agent",
+  async send(input, ctx) {
+    await ctx.sandbox.runCommand("my-agent", ["run", input.text]);
+    return { status: "completed", events: [] };
+  },
+});
+```
+
+## Registration
+
+```ts
+import { defineExperiment } from "niceeval";
+import echo from "./agents/echo";
+
+export default defineExperiment({
+  agent: echo,
+  runs: 1,
+});
+```
+
+## `ctx` and `t`
+
+`ctx` is the run context seen by the adapter side; `t` is the test context seen by the eval author. They use the same run data but have different responsibilities.

package/docs-site/reference/define-config.mdx

@@ -0,0 +1,41 @@
+---
+title: "defineConfig: project defaults"
+sidebarTitle: "defineConfig"
+description: "defineConfig reference: judge, reporters, concurrency, timeout, and sandbox defaults."
+---
+
+Use `defineConfig` in `niceeval.config.ts` for project-wide defaults. Agent, model, flags, runs, and per-experiment budget belong in `experiments/` files via `defineExperiment`.
+
+```ts
+import { defineConfig } from "niceeval";
+import { JUnit } from "niceeval/reporters";
+
+export default defineConfig({
+  judge: { model: "gpt-5.4-mini" },
+  reporters: [JUnit(".niceeval/junit.xml")],
+  maxConcurrency: 4,
+  timeoutMs: 300_000,
+});
+```
+
+<ParamField body="judge" type="{ model: string; baseUrl?: string; apiKeyEnv?: string }">
+  Default judge configuration for `t.judge.autoevals.*`. Individual evals can override this with `defineEval({ judge })`.
+</ParamField>
+
+<ParamField body="reporters" type="Reporter[]">
+  Additional reporters that observe the whole run, such as `Braintrust(...)` for uploading to an experiment platform. See [Reporters](../guides/reporters). For temporary JUnit output, use the CLI flag `--junit <path>`.
+</ParamField>
+
+<ParamField body="maxConcurrency" type="number">
+  Maximum concurrent attempts. Sandbox backends may choose a lower default when this is omitted.
+</ParamField>
+
+<ParamField body="timeoutMs" type="number">
+  Per-attempt timeout in milliseconds.
+</ParamField>
+
+<ParamField body="sandbox" type="SandboxSpec">
+  Default sandbox backend for sandbox agents, built with `dockerSandbox()` / `vercelSandbox()` / `e2bSandbox()` from `niceeval/sandbox` — bare strings like `"docker"` are no longer accepted here. Required for sandbox agents (directly on the experiment, or via this project-wide fallback: `exp.sandbox ?? config.sandbox`) — niceeval no longer auto-detects a backend from env vars, and there is no CLI flag for choosing one.
+</ParamField>
+
+niceeval keeps environment preparation in ordinary code. Prepare files inside `test(t)` with `t.sandbox.writeFiles`, `uploadFiles`, or `uploadDirectory`; put adapter setup in the Adapter.