niceeval 0.4.5 → 0.5.0

package/docs-site/introduction.mdx

@@ -1,141 +0,0 @@
----
-title: "NiceEval: an eval tool for agents"
-sidebarTitle: "Introduction"
-description: "NiceEval helps teams measure, evaluate, and improve AI in production: compare models, iterate on agents, catch regressions, and improve with real data."
----
-
-[NiceEval](https://niceeval.com/) is an agent eval tool that helps teams measure, evaluate, and improve AI in production. With NiceEval, teams can compare models, iterate on agents, catch regressions, and keep improving their AI applications using real user data. It aims for low-friction DX: most projects can integrate in about 10 minutes, write the first eval, and start checking agent behavior reliably.
-
-NiceEval is local-first at its core: your evals run in your own environment, so you don't need to create an account or connect to a platform before running your first eval. When your team needs to share eval results or track regressions, you can push a Report to platforms like BrainTrust, or export a custom report.
-
-It can evaluate coding-agent plugins, hooks, and Skills for Claude Code / Codex, and it can also evaluate your own AI agent application directly. Whether your agent is built on AI SDK, LangGraph, Pi, or a custom loop, you can connect it through an Adapter into the same eval surface.
-
-## 30 seconds of DX
-
-An eval is just a TypeScript file: send the agent a message, assert on what it did.
-
-```ts
-// evals/eval-tool-call.eval.ts
-import { defineEval } from "niceeval";
-
-export default defineEval({
-  description: "Verify the agent calls the weather tool and answers from its result",
-
-  async test(t) {
-    const turn = await t.send("What's the weather in Beijing today?");
-    t.succeeded();
-
-    await t.group("calls get_weather with the right city", () => {
-      t.calledTool("get_weather", { input: { city: "Beijing" } });
-      t.messageIncludes(/°C|sunny|cloudy|rain/);
-    });
-
-    const second = await t.send("What about Shanghai tomorrow?");
-    second.messageIncludes("Shanghai");
-
-    t.judge.autoevals
-      .closedQA("Does the reply use the tool's weather data instead of making up a temperature?")
-      .atLeast(0.7);
-  },
-});
-```
-
-```sh
-pnpm exec niceeval exp local eval-tool-call  # run only eval-tool-call under the local experiment
-pnpm exec niceeval view                      # view eval results
-```
-
-## Why NiceEval when DeepEval, LangFuse, and BrainTrust already exist
-
-NiceEval is an Agent-Native eval tool. The Dataset / golden pattern of building an Input and an Expected Output doesn't fit real agent evaluation — agents today need to be checked at a finer grain: multi-turn conversations, multi-agent collaboration, tool calls, skill loading. NiceEval is designed for exactly that: assertions land directly on tool calls, message content, structured output, and usage, instead of requiring you to hand-build a golden dataset first.
-
-Tools like Langfuse and Braintrust lean more toward tracing and observability. For the path of "write evals, run evals, inspect results", that is often heavier than necessary. NiceEval is designed around that path from the start, with a simpler developer experience. The two aren't mutually exclusive either: NiceEval coexists with LangFuse and BrainTrust — use them for tracing, or upload eval results to both.
-
-## What you can evaluate
-
-<CardGroup cols={2}>
-  <Card title="Coding agents" icon="code">
-    Put Claude Code, Codex, or bub into a sandbox, give them a task, then verify the result with real tests and file assertions.
-  </Card>
-  <Card title="Your own AI agent application" icon="globe">
-    Connect to your app directly and assert on replies, tool calls, and structured outputs.
-  </Card>
-</CardGroup>
-
-## Two integration modes
-
-<Tabs>
-  <Tab title="Sandbox mode">
-    Fits coding agents such as Codex and Claude Code that need a real file system and command execution.
-
-    ```text
-       evals/*.eval.ts
-            │
-            ▼
-       ┌────────────┐
-       │  niceeval  │
-       └────────────┘
-            │
-            │ Adapter (official)
-            ▼
-       ┌───────────────────────────────┐
-       │        Docker Sandbox         │
-       │    ┌───────────────────────┐  │
-       │    │ Codex / Claude Code / │  │
-       │    │  apps that need an    │  │
-       │    │  isolated workspace   │  │
-       │    └───────────────────────┘  │
-       └───────────────────────────────┘
-    ```
-  </Tab>
-  <Tab title="Direct mode">
-    Fits your own AI agent app and does not require Docker.
-
-    ```text
-       evals/*.eval.ts
-            │
-            ▼
-       ┌────────────┐
-       │  niceeval  │
-       └────────────┘
-            │
-            │ Adapter (official or custom)
-            ▼
-       ┌────────────────────────────┐
-       │   Your own AI agent app    │
-       │ (AI SDK · LangGraph · Pi · │
-       │  custom agent loop, etc.)  │
-       └────────────────────────────┘
-    ```
-  </Tab>
-</Tabs>
-
-## Core concepts at a glance
-
-| Concept | In one line |
-|---|---|
-| Eval | A test case: written in `evals/*.eval.ts`, describing what to check. |
-| Experiment | A checked-in run configuration: which Adapter, which model, which flags. |
-| Adapter | The layer that connects to the system under test: implement one `send`, get back a standard event stream. |
-| Sandbox | Only needed for coding agents that require an isolated workspace; a direct web agent doesn't need one. |
-| Tier | Three levels of Adapter integration effort: Tier 1 wires up `send` only, Tier 2 adds OTel for a call waterfall, Tier 3 makes invasive changes for feature A/B testing. |
-
-See the full glossary in the [architecture overview](/concepts/overview).
-
-## Start from your scenario
-
-<CardGroup cols={3}>
-  <Card title="Evaluate a Claude Code / Codex plugin" icon="puzzle-piece" href="/example/claude-code-codex-plugin">
-    Drop the plugin into a real workspace and compare pass rate, cost, latency, and behavior with and without it.
-  </Card>
-  <Card title="Evaluate a Claude Code / Codex Skill" icon="wand-magic-sparkles" href="/example/claude-code-codex-skill">
-    Compare coding-agent tasks with and without the Skill to measure whether it actually improves results.
-  </Card>
-  <Card title="Evaluate your own AI agent application" icon="robot" href="/example/ai-agent-application">
-    Connect directly to your app's API and assert on tool calls, image understanding, and multi-turn conversations.
-  </Card>
-</CardGroup>
-
-## What to read next
-
-[Quickstart](/quickstart) walks through installation, writing the first eval, and getting a report.

package/docs-site/quickstart.mdx

@@ -1,136 +0,0 @@
----
-title: "Set up evals for your agent project"
-sidebarTitle: "Quickstart"
-description: "Install NiceEval, write three files, and run the first eval against your own application in about 10 minutes."
----
-
-Connecting your own application takes three files: an **adapter** that calls your app, an **experiment** that chooses what to run, and an **eval** that declares what must be true. Below is the shortest complete path. Scenario-specific guides and the coding-agent handoff path come after it.
-
-## Let AI wire it up for you (recommended)
-
-<Steps>
-  <Step title="Install">
-    ```text
-    READ https://niceeval.com/INIT.md and install niceeval for this repo.
-    ```
-  </Step>
-  <Step title="Run evals">
-    ```bash
-    pnpm exec niceeval exp <experiment-name>
-    ```
-  </Step>
-  <Step title="View results">
-    ```bash
-    pnpm exec niceeval view
-    ```
-  </Step>
-</Steps>
-
-## Manual integration
-
-```bash
-pnpm add -D niceeval
-pnpm exec niceeval init
-```
-
-### Adapter
-
-The adapter is the layer between your agent and NiceEval. There are multiple integration tiers, from non-invasive to partially instrumented, which unlock more useful eval features. See [Tier](/concepts/tier).
-
-```ts
-// agents/my-bot.ts
-import { defineAgent } from "niceeval/adapter";
-
-export default defineAgent({
-  name: "my-bot",
-  async send(input, ctx) {
-    const r = await fetch("http://localhost:3000/chat", {
-      method: "POST",
-      headers: { "content-type": "application/json" },
-      body: JSON.stringify({ message: input.text, model: ctx.model }),
-      signal: ctx.signal,
-    });
-    const body = await r.json();
-    return {
-      status: r.ok ? "completed" : "failed",
-      events: [{ type: "message", role: "assistant", text: body.reply }],
-    };
-  },
-});
-```
-
-### Experiment
-
-```ts
-// experiments/my-bot.ts
-import { defineExperiment } from "niceeval";
-import myBot from "../agents/my-bot.ts";
-
-export default defineExperiment({
-  description: "my-bot baseline",
-  model: "gpt-4o",
-  agent: myBot,
-  runs: 1,
-});
-```
-
-### Eval
-
-```ts
-// evals/refund-policy.eval.ts
-import { defineEval } from "niceeval";
-import { includes } from "niceeval/expect";
-
-export default defineEval({
-  description: "Refund policy question",
-  async test(t) {
-    await t.send("What is your refund policy?");
-    t.succeeded();
-    t.check(t.reply, includes("30 days"));
-  },
-});
-```
-
-```bash
-pnpm exec niceeval exp my-bot
-pnpm exec niceeval view
-```
-
-At this point the first eval is running.
-
-This minimal adapter is **single-turn**. The second turn will not remember the first until you teach the app how to continue conversation history. Multi-turn sessions, tool-call events that unlock `t.calledTool()`, HITL, and tracing are all optional increments you add later. Follow [Connect Your Agent](/guides/connect-your-agent) step by step. Existing evals do not need to be rewritten.
-
-## Start from a scenario
-
-If you already know what you want to evaluate, jump straight into the matching path:
-
-<CardGroup cols={3}>
-  <Card title="Eval your Claude Code / Codex plugin" icon="plug" href="/example/claude-code-codex-plugin">
-    For plugins, hooks, MCP servers, and project-level coding-agent extensions.
-  </Card>
-  <Card title="Eval your Claude Code / Codex Skill" icon="wand-magic-sparkles" href="/example/claude-code-codex-skill">
-    For verifying whether a Skill is triggered, follows the intended workflow, and genuinely improves task success.
-  </Card>
-  <Card title="Eval your AI agent application" icon="globe" href="/example/ai-agent-application">
-    For full examples built on AI SDK, LangGraph, Pi, or a custom agent app.
-  </Card>
-</CardGroup>
-
-## Put it in CI
-
-```yaml
-name: evals
-on: [pull_request]
-jobs:
-  evals:
-    runs-on: ubuntu-latest
-    steps:
-      - uses: actions/checkout@v4
-      - uses: actions/setup-node@v4
-      - run: npm ci
-      - run: npm exec niceeval exp my-bot
-```
-
-<Tip>
-  Next, read [Authoring Evals](/guides/authoring) and [Scoring Guide](/guides/scoring-guide), then replace the example with your real behavior.
-</Tip>

package/docs-site/reference/builtin-agents.mdx

@@ -1,161 +0,0 @@
----
-title: "Built-in agent capabilities reference"
-sidebarTitle: "Built-in Agent Capabilities"
-description: "Which capabilities NiceEval's built-in claude-code, codex, and bub adapters implement, which assertions they map to, and known limits."
----
-
-The three built-in sandbox agents (`claude-code`, `codex`, `bub`) do not implement exactly the same capabilities — capabilities have no declaration layer and are proven entirely by construction and actual behavior, see the [capabilities reference](/reference/capabilities). This page inventories what each built-in agent does per capability, plus two built-ins for connecting AI SDK apps: the non-invasive HTTP adapter `uiMessageStreamAgent` (with HITL) and the result converter `fromAiSdk`.
-
-## Capability overview
-
-| Capability | Corresponding assertions / API |
-|---|---|
-| Send/receive messages (base contract) | `t.send()` (can be called multiple times), `t.reply`, `turn.outputEquals` / `outputMatches`, status-based `t.succeeded()` |
-| Event stream completeness | The whole scoped assertion set: `calledTool` / `toolOrder` / `usedNoTools` / `maxToolCalls` / `messageIncludes` / `noFailedActions` / `event` / `eventOrder`, etc.; **negative assertions (`notCalledTool` and friends) are only trustworthy with a complete event stream** |
-| Session continuation | Cross-turn memory assertions, `t.newSession()` session isolation |
-| HITL (human in the loop) | `t.parked()`, `t.requireInputRequest()`, `t.respond()` / `t.respondAll()` |
-| `tracing` | `EvalResult.trace`, the call waterfall in `niceeval view` |
-
-## What each built-in agent implements
-
-| Agent | Send/receive | Event stream | Session continuation | HITL | tracing | Notes |
-|---|---|---|---|---|---|---|
-| `claude-code` | ✅ | ✅ | ✅ (`claude --resume <id>`) | ❌ | ❌ | Built-in parser emits `compaction` events automatically; `t.event("compaction")` works |
-| `codex` | ✅ | ✅ | ✅ (`codex exec resume <id>`) | ❌ | ✅ (`http/json` → OTLP) | Built-in parser emits `compaction` events automatically |
-| `bub` | ✅ | ✅ | ✅ (`--session-id` + tape continuation) | ❌ | ✅ (`http/protobuf` → OTLP) | Built-in parser emits `compaction` events automatically |
-
-All three are constructed with `defineSandboxAgent` (`Agent.kind` is always `"sandbox"`), so `t.sandbox.fileChanged()` / diff assertions, file IO, and command execution work on all three agents regardless of this table.
-
-<Warning>
-None of the three built-in sandbox agents currently support HITL: their `send` only returns `"completed"` / `"failed"`, never `"waiting"`, and never emits `input.requested` events. When you need `t.respond()` / `t.requireInputRequest()`: if the system under test is an AI SDK `useChat` backend, use the built-in `uiMessageStreamAgent` below (v7 tool approval maps natively to HITL); for anything else write your own adapter: implement the `waiting` status + `input.requested` event + resume handoff — [`examples/zh/tier1/pi-sdk`](https://github.com/CorrectRoadH/niceeval/tree/main/examples/zh/tier1/pi-sdk) / [`examples/zh/tier1/claude-sdk`](https://github.com/CorrectRoadH/niceeval/tree/main/examples/zh/tier1/claude-sdk) both have ready-made hand-written HITL references.
-</Warning>
-
-## Per-agent details
-
-### claude-code
-
-- Connection: spawns `claude --print --dangerously-skip-permissions` in the sandbox and reads back the latest transcript from `~/.claude/projects/**/*.jsonl`.
-- Session continuation: appends `--resume <id>` when `ctx.session.id` has a value; the session id parsed from the transcript is written back via `ctx.session.capture()`.
-- Auth: `ANTHROPIC_API_KEY` (overridable via the `apiKey` option), optional `ANTHROPIC_BASE_URL`.
-- Supports `mcpServers` and `skills` (GitHub `org/repo`) options.
-- Does not declare `tracing`: no OTLP receiver is started and `EvalResult.trace` is empty.
-
-### codex
-
-- Connection: runs `codex exec --json` in the sandbox (`codex exec resume <id> --json` when continuing), using stdout JSONL as the transcript.
-- Auth: `CODEX_API_KEY` (not `OPENAI_API_KEY`), optional `CODEX_BASE_URL` for OpenAI-compatible proxies.
-- `tracing` is configured through the `[otel.trace_exporter.otlp-http]` section of `~/.codex/config.toml`, protocol `http/json`.
-- Supports `mcpServers` and `skills` options.
-
-### bub
-
-- Connection: runs `bub run` + `--session-id` in the sandbox and reads the tape from `~/.bub/tapes/<hash>.jsonl` as the transcript.
-- Auth: `BUB_API_KEY` + `BUB_API_BASE` (OpenAI-compatible proxy).
-- `tracing` is injected via environment variables (`OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`, etc.), protocol `http/protobuf`.
-- Supports the `pythonPlugins` option.
-- Installed via `uv tool install` (not an npm package); the first install builds a checkpoint cache to speed up later sandboxes.
-
-## `uiMessageStreamAgent`: built-in non-invasive adapter for AI SDK apps (with HITL)
-
-`uiMessageStreamAgent` (exported from `niceeval/adapter`) talks non-invasively to an HTTP endpoint speaking the [UI Message Stream protocol](https://ai-sdk.dev/docs/ai-sdk-ui/stream-protocol) (the standard SSE of AI SDK `useChat` backends) — it only `fetch`es, never imports app code, so it connects wherever the app is deployed:
-
-```ts
-import { uiMessageStreamAgent } from "niceeval/adapter";
-
-export default uiMessageStreamAgent({
-  name: "my-assistant",
-  url: "https://my-app.example.com/api/chat",
-  body: (ctx) => ({ model: ctx.model }),   // When the app supports request-level model selection, model comparison needs zero changes
-});
-```
-
-Capabilities it handles for you:
-
-- **Send/receive + event stream**: SSE frames are reduced by the `ai` package's official framework-agnostic reducer `readUIMessageStream` (the same one inside `useChat`); tool calls/results/message text are built directly from message parts — **the app is not required to wire up OTel**.
-- **Session continuation**: the protocol is server-stateless, "client carries the full history" — the factory stores the whole `UIMessage[]` with `ctx.session.history()` and replays it verbatim each turn; a new conversation line (after `t.newSession()`) gets an empty history.
-- **HITL**: AI SDK v7 tool approval (tools with `needsApproval: true`) maps natively — when a part stops at `approval-requested`, the whole turn is `status: "waiting"` + `input.requested`; `t.respond("approve" / "deny")` is translated into `approval-responded`, rewriting that part in place and resending `messages` verbatim to trigger the server to continue (identical protocol behavior to a real frontend's `addToolApprovalResponse()` + `sendMessage()`; there is no separate approve endpoint). Denied calls land in the event stream as `rejected`, and by default carry a "do not retry" reason (`denyReason` can override) — without one, models often resend the very same call.
-- **usage / waterfall**: UI Message Stream protocol frames carry no usage, so usage assertions like `t.maxTokens` have no data by default on this built-in (an app putting usage into message metadata is the app's own protocol extension). The waterfall is a separate matter: when the app has OTel instrumentation (e.g. the official `@ai-sdk/otel`), send spans to NiceEval per [OTel integration](/guides/connect-otel) and `niceeval view` gets a full waterfall — spans only feed the waterfall, never assertions.
-
-You need `ai` installed in the eval project (optional peer dependency; the protocol reducer comes from it). Full runnable example: [`examples/zh/tier1/ai-sdk-v7`](https://github.com/CorrectRoadH/niceeval/tree/main/examples/zh/tier1/ai-sdk-v7).
-
-## SDK event stream converters: `fromClaudeSdkMessages` / `fromPiAgentEvents` / `fromCodexThreadEvents`
-
-Each agent SDK's streaming protocol is a generic protocol defined by the SDK, not some app's private format — the native-frame-to-standard-event mapping knowledge lives in the official package (exported from `niceeval/adapter`), so when writing your own non-invasive adapter only the **transport glue** remains (which endpoint the app puts the stream on, which endpoint approvals go through). Driving the loop frame by frame is itself an official piece (`driveFrameStream`, see the next section), so no hand-written `for` loop:
-
-```ts
-import { sseJsonFrames, fromClaudeSdkMessages, driveFrameStream } from "niceeval/adapter";
-
-const frames = sseJsonFrames<SDKMessage>(res.body);   // Standard SSE → per-frame JSON
-const stream = fromClaudeSdkMessages();               // SDKMessage → standard events
-return driveFrameStream(frames, stream, ctx);         // Feed stream.add() frame by frame, summarize into a Turn
-```
-
-- **`fromClaudeSdkMessages`** (Claude Agent SDK `SDKMessage`): `assistant` text/tool_use blocks, `user` tool_result blocks, `system`/`permission_denied` (→ `rejected`), `result` usage; `markRejected()` registers denied calls.
-- **`fromPiAgentEvents`** (pi-agent-core `AgentEvent`): text/thinking/usage from `message_end`, tool pairs from `tool_execution_start/end`.
-- **`fromCodexThreadEvents`** (Codex SDK `ThreadEvent`): message-class frames from `agent_message`/`reasoning`, tool items (`command_execution` / `mcp_tool_call` / `file_change` / `web_search` → `action.called` + `action.result`), usage from `turn.completed`, error frames. The waterfall needs the codex CLI's native OTLP (the `[otel]` block in config.toml) sending spans to NiceEval, normalized with the official `mapCodexSpans`.
-
-Runnable references: the claude-sdk / pi-sdk / codex-sdk adapters under [`examples/zh/tier1`](https://github.com/CorrectRoadH/niceeval/tree/main/examples/zh/tier1) are exactly these three converters + `driveFrameStream` + transport glue.
-
-## Generic "assembly" pieces: `driveFrameStream` / `deltaStream` + `ctx.session`
-
-Inside a hand-written send, only three truly independent parts exist: transport (how to send), reduce (raw data → events, handled by the converters above), and orchestration (session continuation + HITL pause/resume). The third part is protocol-agnostic, pure control-flow pattern — the accessors for session continuation (`id`/`capture`, `history`) and the HITL paused-turn scene (`hold`/`take`) hang directly off `ctx.session` (`AgentSession`, see the [Adapter concept](/concepts/adapter#context-agentcontext)); nothing to import separately and nothing to declare. What actually remains reusable across protocols is just two official pieces:
-
-| Piece | What it solves |
-|---|---|
-| `driveFrameStream(cursor, reducer, ctx, onFrame?)` | Feeds the reducer frame by frame, handles transport frames, detects the HITL pause signal — the loop itself collapsed into one function. On pause the `cursor` is not closed: store it with `ctx.session.hold(cursor)` inside `onFrame`, and on the answer turn `ctx.session.take()` retrieves it and keeps reading, without issuing a new request |
-| `deltaStream(spec)` | Generic accumulator for token-by-token / per-argument deltas — for protocols with "no whole-chunk landing frames" like raw OpenAI/Anthropic streaming APIs or your own hand-rolled token-by-token backend; declare "which operation this frame belongs to" (text delta / tool-argument delta / finalize) and it manages assembly timing itself |
-
-```ts
-// First thing in send: check for a held scene (HITL answer turn)
-const pending = ctx.session.take<Pending>();
-if (pending) {
-  // ...match verdicts from input.responses, hand the approval back to the app...
-  return driveFrameStream(pending.cursor, pending.stream, ctx, onFrame);
-}
-
-// Normal turn: session continuation via ctx.session.id / capture; on pause, ctx.session.hold stores the cursor
-return driveFrameStream(cursor, stream, ctx, (frame, derived) => {
-  ctx.session.capture(stream.sessionId);
-  const gated = derived.find((e) => e.type === "action.called" && e.name === GATED_TOOL_NAME);
-  if (gated?.type === "action.called") {
-    ctx.session.hold<Pending>({ cursor, stream, toolUseId: gated.callId });
-    return { pause: { id: gated.callId, action: GATED_TOOL_NAME, options: [{ id: "approve" }, { id: "deny" }] } };
-  }
-});
-```
-
-The three Tier 1 examples (claude-sdk and pi-sdk use `driveFrameStream` + `ctx.session.hold`/`take`; codex-sdk uses only `driveFrameStream`) have been rewritten on these pieces — the adapters are down to transport plus the "does this frame need extra handling" decision in `onFrame`, with no hand-written loops or module-level Maps.
-
-## `fromAiSdk`: AI SDK result → event stream converter
-
-`fromAiSdk` (exported from `niceeval/adapter`) is used inside adapters you write yourself (e.g. server-side direct construction for an HTTP web agent, see `examples/zh/ai-sdk/`). It maps an AI SDK `generateText` / `streamText` result into `{ events, usage, status }` — exact `toolCallId` pairing, order-faithful, `tool-error` mapped to a failed `action.result`, usage aggregation (v4 / v5 / v7 field drift all covered) — laid straight into the `Turn`:
-
-```ts
-import { defineAgent, fromAiSdk } from "niceeval/adapter";
-
-export default defineAgent({
-  name: "my-ai-sdk-agent",
-  async send(input) {
-    const result = await generateText({ model, tools, prompt: input.text });
-    return { ...fromAiSdk(result), data: result.text };
-  },
-});
-```
-
-The `status` comes from the converter: `"waiting"` when there is a tool approval pending human review (with an `input.requested` event attached), otherwise `"completed"`. Session continuation (multi-turn resume), handing HITL verdicts back, and tracing depend on how you write your own `send`.
-
-## How to choose
-
-- The system under test is an AI SDK app (`useChat` backend): the built-in `uiMessageStreamAgent` connects non-invasively with zero mapping (including HITL).
-- The system under test is another agent system (HTTP / gRPC service): connect non-invasively and hand-write the event mapping (official converters cover most of it) — all five examples under [`examples/zh/tier1`](https://github.com/CorrectRoadH/niceeval/tree/main/examples/zh/tier1) take this path; if the app already has OTel instrumentation, wire it up for a bonus waterfall.
-- You need the call waterfall (the trace in `niceeval view`): all three of `claude-code` / `codex` / `bub` emit OTLP themselves (claude-code via the CLI's beta telemetry; its spans carry structure and timing only); adapters you write yourself just declare `tracing` (see the OTel guide).
-- You need human approval / multi-step confirmation (HITL): `uiMessageStreamAgent` supports it natively (AI SDK v7 tool approval); none of the three sandbox agents do; for other systems write your own adapter, assembled from `driveFrameStream` + `ctx.session.hold`/`take` (the pi-sdk / claude-sdk Tier 1 examples have ready-made patterns).
-- You want to run a coding agent that edits code, inspect diffs, and judge tool calls: `claude-code` / `codex` / `bub` (send/receive + event stream + session continuation + workspace + sandbox).
-- Your backend protocol is home-grown with no official SDK converter: do not hand-write send from scratch — pick a reduce shape (a small mapping for whole-chunk landing, `deltaStream` for token-by-token deltas), take `ctx.session.id`/`capture` or `ctx.session.history` for session continuation, add `ctx.session.hold`/`take` if you have HITL, and the assembly is your send.
-
-## Related reading
-
-- [Connect your agent](/guides/connect-your-agent) — how to implement each capability in your own adapter and which assertions it maps to.
-- [Sandbox Agent](/guides/sandbox-agent) — how to run the built-in sandbox agents and write your own.
-- [defineAgent reference](/reference/define-agent) — full parameters of `defineAgent` / `defineSandboxAgent`.
-- [OTel integration](/guides/connect-otel) — send your app's spans to NiceEval in exchange for the call waterfall in `niceeval view`.

package/docs-site/reference/capabilities.mdx

@@ -1,76 +0,0 @@
----
-title: "Capabilities reference"
-sidebarTitle: "Capabilities"
-description: "How NiceEval knows what your adapter has done: there are no declarative capability bits — everything is proven by construction and actual behavior."
----
-
-Capabilities answer one question: **what has your adapter actually done** — which decides which assertions are trustworthy and which services the runner prepares for you.
-
-There is no `capabilities` field on `defineAgent` / `defineSandboxAgent` — **nothing needs to be declared**. Every capability is proven by how the agent is constructed or by what it actually returns. The decision table is below.
-
-## Where capabilities come from: one decision table
-
-| Capability | How it is obtained | What you write |
-|---|---|---|
-| `t.sandbox` / `t.sandbox.fileChanged()` and other filesystem assertions | Constructed with `defineSandboxAgent` (`Agent.kind` is always `"sandbox"`) | Nothing |
-| `tracing` (OTLP receiver → `niceeval view` waterfall) | CLI-style agents write the `tracing` field; long-running apps configure a fixed `telemetry` port in the config | Nothing (the configuration is what you would write anyway) |
-| Tool/action assertions (`t.calledTool()` etc., including negative assertions) | Use an official converter (`fromClaudeSdkMessages`, `fromAiSdk`, etc.) — the `action.*` events in the return value are exactly what you get | Nothing; with manual mapping, event completeness is decided by your mapping code |
-| Cross-turn memory, `t.newSession()` isolation | Used `ctx.session` (`id`/`capture`, `history`) to continue context | Nothing — wire it up and continuation works; skip it and every turn is a fresh conversation, with no error either way |
-| HITL (`t.respond()`) | `send` has returned `status: "waiting"` + `input.requested` | Nothing |
-
-A capability you have not implemented = the corresponding events/fields never appear, and the related assertion group is unavailable (not an "error" — the assertions simply have nothing to judge from); basic assertions (text, structured output, judge) are always available.
-
-## Why there is no declaration layer
-
-An earlier design had boolean declarations like `capabilities: { conversation, toolObservability }` to cover "the parts code cannot prove". That layer has been removed, because what it promised should never have been guaranteed by a declaration in the first place:
-
-- **Construction is proof**: using `defineSandboxAgent` means the system under test edits files in a sandbox — there is no need to declare `workspace: true` on top of that.
-- **Fields are proof**: writing `tracing` (how to hand the OTLP endpoint to the CLI) makes tracing real. The field itself is the evidence of use; a boolean sitting next to it would just repeat it.
-- **Source is proof**: `fromAiSdk(result)` consumes `result.steps`, which the AI SDK contract guarantees is the **complete** tool-loop record, so the return value carries its own proof of a complete event stream; built-in CLI parsers work the same way. With manual mapping, whether events are complete depends on whether your mapping code covers everything — that is not something a declaration can guarantee, and declaring it would not make the code any more complete.
-- **Behavior is proof**: HITL has no capability bit — `send` returns `status: "waiting"` + an `input.requested` event, so the behavior is directly observable.
-- **Wire it up and it continues**: session continuation is no longer a "declared to exist" switch. `ctx.session` is always there; use the `id`/`capture` or `history` accessors and context continues, skip them and every turn is a brand-new session — both are normal states, neither errors, and the difference is purely how your adapter is actually written.
-
-```ts
-export default defineAgent({
-  name: "my-ai-sdk-agent",
-  async send(input, ctx) {
-    const result = await generateText({ model, tools, prompt: input.text });
-    return { ...fromAiSdk(result), data: result.text };
-    // ↑ The return value carries complete tool events; no capability declaration needed
-  },
-});
-```
-
-## What happens when a manual mapping misses events
-
-With the declaration layer gone, "false claims" no longer exist either — whatever tool events your manual mapping misses, assertions like `t.calledTool()` see exactly the post-omission picture, and negative assertions (`notCalledTool()`) may "look like they pass" simply because the events were never written. That is not a framework misjudgment; it is a completeness problem in the manual mapping itself: **whether to trust a negative assertion depends on whether you have confirmed your mapping code covers every tool call** — there is no framework-provided middle layer vouching for you.
-
-## Item-by-item quick reference
-
-### Tool/action event completeness
-
-Meaning: whether every tool call in the turn appears in events as `action.called` + `action.result` ([events reference](/reference/events)). There are only two sources: official converters guarantee it per the contract of the protocol under test; manual mapping is decided by your code. OTel spans are not a source — spans only feed the waterfall, never the event stream (see [OTel integration](/guides/connect-otel)).
-
-### Session continuation
-
-Meaning: using `ctx.session.id` / `capture` or `ctx.session.history` to connect requests to the same backend session / accumulated history. With it, cross-turn memory assertions and `t.newSession()` isolation assertions become meaningful. Both implementation styles are shown in [Write send](/guides/write-send).
-
-### `tracing` — emitting OTLP traces
-
-What the runner consumes: at runtime it starts a local OTLP receiver and hands this turn's trace context to you via `ctx.telemetry.headers`; normalized spans are drawn as a waterfall in `niceeval view` — drawing only, never feeding assertions. Source: CLI-style agents write the `tracing` field (`env` / `configure` hands the endpoint to the process under test); long-running apps configure a fixed port with `telemetry: { port }` in `niceeval.config.ts` (see [OTel integration](/guides/connect-otel)).
-
-### `kind: "sandbox"` — sandbox and file changes
-
-What the runner consumes: it prepares a sandbox (Docker / Vercel / E2B) for each attempt and hands it to the adapter via `ctx.sandbox`; this unlocks diff assertions like `t.sandbox.fileChanged()`. Source: constructed with `defineSandboxAgent` — an internal discriminant field you never write.
-
-### Compaction visibility
-
-Meaning: when the agent under test compacts its context, it emits a `compaction` event, making `t.event("compaction")` meaningful. Source: built-in parsers emit it automatically; a hand-written parser has it if you covered that logic.
-
-## Related reading
-
-- [Events reference](/reference/events) — exactly what the event stream promises.
-- [Adapter concept](/concepts/adapter) — the full `ctx`/`Turn` contract and where capabilities come from.
-- [Connect your agent](/guides/connect-your-agent) — how to follow each integration path.
-- [OTel integration](/guides/connect-otel) — the full walkthrough of spans feeding only the waterfall, never assertions.
-- [Built-in agent capabilities](/reference/builtin-agents) — what each built-in adapter has done.

package/docs-site/reference/cli.mdx

@@ -1,120 +0,0 @@
----
-title: "NiceEval CLI: commands, flags, and exit codes reference"
-sidebarTitle: "CLI"
-description: "NiceEval CLI reference for exp, init, list, clean, and view, plus experiment selection, eval filtering, sandbox, concurrency, budget, and JUnit CI output."
----
-
-The NiceEval CLI is the entry point for discovery, execution, and result viewing. Actual eval execution follows an experiment-first model: **`exp` selects a commit-ready run configuration**, and only the positional arguments after the experiment filter eval IDs by prefix. Agent, model, and feature flags live in `experiments/`, not in temporary CLI flags.
-
-## Commands
-
-<CardGroup cols={2}>
-  <Card title="npx niceeval exp &lt;group&gt; [id-prefix]" icon="flask">
-    Run a named experiment group or config. Optional trailing positional args filter by eval ID prefix.
-  </Card>
-  <Card title="npx niceeval init" icon="wand-magic-sparkles">
-    Generate `evals/`, `niceeval.config.ts`, and sample files.
-  </Card>
-  <Card title="npx niceeval list" icon="list">
-    Discover and print all evals without running them.
-  </Card>
-  <Card title="npx niceeval clean" icon="trash">
-    Remove historical `.niceeval/` artifacts.
-  </Card>
-  <Card title="npx niceeval view" icon="eye">
-    Open the result viewer for the most recent run.
-  </Card>
-</CardGroup>
-
-## Output language
-
-NiceEval supports localized CLI and runtime copy without configuration in `niceeval.config.ts`. When you need fixed output language, use environment variables:
-
-```bash
-NICEEVAL_LANG=en npx niceeval list
-NICEEVAL_LANG=zh-CN npx niceeval list
-```
-
-Detection order is `NICEEVAL_LANG`, `NICEEVAL_LOCALE`, `LC_ALL`, `LC_MESSAGES`, then `LANG`. Values starting with `zh` use `zh-CN`; everything else uses `en`. If nothing is set, the default is `zh-CN`. This affects terminal/runtime copy only. It does not change machine fields in result JSON or translate judge prompts.
-
-## `npx niceeval exp [group|config] [id-prefix...]`
-
-```bash
-# Run every experiment under experiments/
-npx niceeval exp
-
-# Run one experiment group
-npx niceeval exp compare-models
-
-# Inside that group, only run evals whose ID starts with weather
-npx niceeval exp compare-models weather
-```
-
-<Note>
-  Eval filters only appear after the experiment selector. Bare `npx niceeval weather` does not run. Use `npx niceeval exp local weather` or `npx niceeval exp compare weather`.
-</Note>
-
-## Common flags
-
-<ParamField query="--agent" type="string">
-  Experiment runs do not support this flag. To switch agents, add or duplicate a config file in `experiments/`.
-</ParamField>
-
-> There is no CLI flag for selecting a sandbox backend. Backend choice only lives in code: set `sandbox` on the experiment (`defineExperiment`) or in `niceeval.config.ts` (`defineConfig`) as a project-level fallback, using `dockerSandbox()` / `vercelSandbox()` / `e2bSandbox()` from `niceeval/sandbox`.
-
-<ParamField query="--model" type="string">
-  Experiment runs do not support this flag. To switch models, add or duplicate an experiment file and change `model`.
-</ParamField>
-
-<ParamField query="--max-concurrency" type="number">
-  Set how many evals may run at the same time.
-</ParamField>
-
-<ParamField query="--runs" type="number">
-  Set how many times each eval runs, commonly for pass@N.
-</ParamField>
-
-<ParamField query="--early-exit" type="boolean">
-  Stop the remaining attempts for an eval after one attempt passes.
-</ParamField>
-
-<ParamField query="--timeout-ms" type="number">
-  Timeout for one eval.
-</ParamField>
-
-<ParamField query="--budget" type="number">
-  Budget limit for the whole run.
-</ParamField>
-
-<ParamField query="--strict" type="boolean">
-  Recommended in CI. Make failures show up more explicitly in the exit code.
-</ParamField>
-
-## `list`
-
-```bash
-npx niceeval list
-```
-
-Use this to confirm eval discovery, IDs, and config loading.
-
-## `exp`
-
-```bash
-npx niceeval exp compare-models
-npx niceeval exp compare-models weather-tool
-```
-
-Run a named experiment and compare agents, models, or flags. Positional args after the experiment filter eval IDs by prefix.
-
-## `view`
-
-```bash
-npx niceeval view
-```
-
-Open the local result viewer. By default it shows the latest run under `.niceeval/`.
-
-## Exit codes
-
-The CI rule is simple: when a gate fails or the run itself fails, the command should exit non-zero. When everything passes, it should exit 0. Use `--strict` when you want the stricter CI behavior explicitly.