niceeval 0.4.6 → 0.5.1

package/docs-site/concepts/drive.mdx

@@ -1,118 +0,0 @@
----
-title: "How NiceEval drives agents: send, sessions, and HITL"
-sidebarTitle: "Drive"
-description: "t.send() and the Turn it returns, t.sendFile(), multi-turn conversations, t.newSession(), and HITL through t.respond() / t.respondAll()."
----
-
-Before assertions or judge scoring, you need to make the agent do something. **Drive** is the part of `test(t)` that sends input and gets results back: `t.send()`, `t.sendFile()`, `t.newSession()`, and HITL through `t.respond()` / `t.respondAll()`. Every drive API produces a **Turn**. All assertions and judge checks in NiceEval read from Turn data. See [Assert](/concepts/assert) and [Judge](/concepts/judge) for the scoring side.
-
-## `t.send()` and the `Turn` it returns
-
-`t.send(input)` is the core verb. Under the hood it calls the Adapter's `send(input, ctx)` and normalizes the return value into a `Turn`:
-
-```ts
-const turn = await t.send("What's the weather like in Brooklyn today?");
-
-turn.message;    // last assistant message
-turn.data;       // structured output, only when this send declared output
-turn.status;     // "completed" | "failed" | "waiting"
-turn.events;     // StreamEvent[] for this turn
-turn.usage;      // { inputTokens, outputTokens, ... } when the adapter reports it
-turn.succeeded(); // scored assertion: this turn didn't fail and isn't parked on HITL
-```
-
-`t.reply` is shorthand for the last assistant message in the main session, equivalent to `turn.message` from the most recent `t.send()`. `t.events` is the full accumulated event stream for the main session.
-
-When you need structured output, declare `output` on the send call. The declaration is both a requirement for the app and the type source for `turn.data`:
-
-```ts
-const turn = await t.send("What is the total on this invoice?", {
-  output: z.object({ amount: z.number(), currency: z.string() }),
-});
-t.check(turn.data.amount, equals(3200));
-```
-
-The runner validates `data` against that schema immediately after the adapter returns. If the shape does not match, the turn fails and the diff is reported. If you did not declare `output`, there is no `turn.data`.
-
-<Tip>
-`turn.succeeded()` scopes to this turn only: it checks this turn's `status` and whether it's parked on an unanswered HITL request. Call it right after a `t.send()` that later turns depend on, so a failure surfaces here instead of a pile of confusing follow-on failures.
-</Tip>
-
-## One turn with a file: `t.sendFile()`
-
-`t.sendFile(path, text?)` reads a local file relative to the eval directory, infers the MIME type from the extension, and attaches it to the turn input as a data URL:
-
-```ts
-const turn = await t.sendFile("fixtures/invoice.png", "What is the total on this invoice?");
-t.check(turn.message, includes("$"));
-```
-
-## Multi-turn conversations
-
-Every `await t.send(...)` is a new turn on the same conversation. Keep each return value in a local variable when you want to assert on that turn specifically:
-
-```ts
-const draft = await t.send("Draft a follow-up email.");
-draft.succeeded();
-t.check(draft.message, includes("Best"));
-draft.judge.autoevals.closedQA("Is the tone professional?").atLeast(0.6);
-
-await t.send("Looks good, send it.");
-t.calledTool("send_email");
-```
-
-Whether multi-turn calls actually continue the same conversation depends on whether the adapter's `send` uses the continuation helpers from `ctx.session` (`history()` or `id` + `capture()`). If it does not, each turn becomes a fresh conversation. See [Adapter](/concepts/adapter) and [How to write send](/guides/write-send).
-
-## Independent sessions: `t.newSession()`
-
-`t.newSession()` opens a second conversation line that runs in parallel with the main one and should not share state. The returned handle has the same drive APIs (`send`, `sendFile`, `respond`, `respondAll`) and the same scoped assertions, but only sees its own events:
-
-```ts
-await t.send("My name is Alice.");
-await t.send("What is my name?");
-t.check(t.reply, includes("Alice"));
-
-const fresh = t.newSession();
-await fresh.send("What is my name?");
-t.check(fresh.reply, satisfies((r) => !r.includes("Alice"), "no memory leak"));
-```
-
-<Warning>
-Do not assume `t.newSession()` guarantees isolation by itself. The runner only guarantees a fresh session handle. The adapter still has to open a fresh underlying conversation. If the adapter always resumes the same backend context, `t.newSession()` will quietly share state. Verify isolation with an eval like the one above before trusting it.
-</Warning>
-
-## Human-in-the-loop (HITL)
-
-Some agents stop in the middle of a turn and wait for approval or missing information instead of finishing immediately. In that case the turn ends with `status: "waiting"` and one or more `input.requested` events that describe what it is waiting for. The full model is in [HITL](/concepts/hitl); here the focus is how evals drive it.
-
-```ts
-const draft = await t.send("Draft a follow-up email, but don't send it until I confirm.");
-draft.parked();
-
-const request = t.requireInputRequest({
-  prompt: /send it/,
-  optionIds: ["approve", "reject"],
-});
-
-await t.respond({ request, optionId: "approve" });
-t.calledTool("send_email");
-```
-
-`t.requireInputRequest(filter)` turns a pending HITL request into a concrete value you can inspect and answer. It throws when 0 or more than 1 requests match, so fill as many filter fields as you can (`id`, `prompt`, `display`, `action`, `optionIds`, `input`) to disambiguate. `t.respond(...)` answers it and emits the next turn. String arguments answer requests in order; object arguments such as `{ request, optionId }` target a specific request directly. Under the hood this is still just another `send`: the answer text goes into `input.text`, and structured answers go into `input.responses`.
-
-When the current turn has multiple pending requests of the same kind and they should all get the same answer, use `t.respondAll(optionId)`:
-
-```ts
-await t.send("Ask for approval on each of these file changes.");
-t.requireInputRequest({ display: /approval/ });
-
-await t.respondAll("approve");
-t.succeeded();
-```
-
-## Related reading
-
-- [HITL](/concepts/hitl) - The full concept for paused turns, request/response handshakes, and adapter obligations.
-- [Assert](/concepts/assert) - Assertions that read from `Turn.events` and `Turn.data`.
-- [Judge](/concepts/judge) - What `t.judge`, `session.judge`, and `turn.judge` score by default.
-- [Adapter](/concepts/adapter) - Where session continuation, HITL, and tool assertions come from.

package/docs-site/concepts/evals.mdx

@@ -1,108 +0,0 @@
----
-title: "Evals in NiceEval: lifecycle, outcome, and files"
-sidebarTitle: "Evals"
-description: "An eval is a test case: description, optional agent reference, and test function. Learn how evals are discovered, scheduled, scored, and reported."
----
-
-An eval is a runnable test case. It usually lives in a `*.eval.ts` file and is declared with `defineEval`.
-
-## What an eval contains
-
-```ts
-import { defineEval } from "niceeval";
-import { includes } from "niceeval/expect";
-
-export default defineEval({
-  description: "Brooklyn weather query",
-  async test(t) {
-    await t.send("What's the weather like in Brooklyn today?");
-    t.succeeded();
-    t.calledTool("get_weather", { input: { city: "Brooklyn" }, count: 1 });
-    t.check(t.reply, includes("sunny"));
-  },
-});
-```
-
-| Field | Meaning |
-|---|---|
-| `description` | Human-readable description shown in reports |
-| `agent` | Which Adapter to use, usually supplied by the experiment |
-| `test(t)` | Interaction and assertion logic |
-
-<Note>
-  Do not hand-write `id` or `name`. NiceEval derives the eval ID from the file path.
-</Note>
-
-## Path is identity
-
-`evals/weather/brooklyn.eval.ts` has ID `weather/brooklyn`. Positional arguments after the experiment name filter by ID prefix:
-
-```bash
-npx niceeval exp local weather
-npx niceeval exp local weather/brooklyn
-```
-
-## Lifecycle
-
-<Steps>
-  <Step title="Discovery">Load `*.eval.ts` files and fixture directories.</Step>
-  <Step title="Scheduling">Build the execution plan from concurrency, cache, runs, attempts, and early exit.</Step>
-  <Step title="agent.send">`t.send()` calls the selected adapter and receives a standard `Turn`.</Step>
-  <Step title="Scoring">NiceEval collects value assertions, scoped assertions, judge scores, and test results.</Step>
-  <Step title="Outcome">All assertion results fold into one final outcome.</Step>
-  <Step title="Report">Reporters write console output and `.niceeval/` artifacts.</Step>
-</Steps>
-
-## Outcome types
-
-<CardGroup cols={2}>
-  <Card title="passed" icon="circle-check" color="#22c55e">
-    All gate assertions passed and no hard failure occurred.
-  </Card>
-  <Card title="failed" icon="circle-xmark" color="#ef4444">
-    A gate assertion failed or the run itself failed.
-  </Card>
-  <Card title="passed" icon="chart-bar" color="#f59e0b">
-    No gate failed, but soft scores are preserved as scores.
-  </Card>
-  <Card title="skipped" icon="forward" color="#6b7280">
-    The eval was skipped, usually through `t.skip(reason)`.
-  </Card>
-</CardGroup>
-
-## Gate and soft
-
-`gate` is a hard threshold. Failure makes the eval fail. `soft` contributes a score without necessarily failing the eval. See [Assert](/concepts/assert).
-
-## File convention
-
-Only files ending in `.eval.ts` are discovered. Use directories for grouping:
-
-```text
-evals/
-└─ billing/
-   └─ refund.eval.ts  # id: billing/refund
-```
-
-## Dataset fan-out
-
-A file can also default-export an array of `defineEval(...)` calls to generate multiple cases from the same logic:
-
-```ts
-export default rows.map((row) =>
-  defineEval({
-    description: row.task,
-    async test(t) {
-      await t.send(row.prompt);
-      t.check(t.reply, equals(row.expected));
-    },
-  }),
-);
-```
-
-This generates IDs such as `sql/0000` and `sql/0001`. See [Dataset Fan-out](/guides/dataset-fanout).
-
-## Related reading
-
-- [Experiment](/concepts/experiment) - The other half: who to evaluate, how to run it, and why evals are separate from experiments.
-- [Assert](/concepts/assert) - The full outcome rules for gate and soft assertions.

package/docs-site/concepts/experiment.mdx

@@ -1,37 +0,0 @@
----
-title: "Experiment: who to evaluate and how to run"
-sidebarTitle: "Experiment"
-description: "An experiment is a commit-ready run configuration: which agent to evaluate, which model to use, which flags to enable, and how many times to run the same eval suite."
----
-
-An **experiment** chooses which agent to evaluate, which model to use, which flags to enable, and how many times to run. Use it for A/B tests, model comparisons, and feature comparisons.
-
-## What an experiment contains
-
-```ts
-import { defineExperiment } from "niceeval";
-import { webAgent } from "../agents/web-agent.ts";
-
-export default defineExperiment({
-  description: "Concise prompt variant",
-  agent: webAgent({ baseUrl: "https://staging.example.com" }), // configured agent instance
-  model: "gpt-5.4",                    // forwarded to the adapter through ctx.model
-  flags: { promptVariant: "concise" }, // forwarded through ctx.flags; visible in evals as t.flags
-  runs: 3,                             // how many times to run each eval
-});
-```
-
-- `agent`: who to evaluate. Put a configured instance here. The subject under test URL and credentials belong in the adapter factory, not as ad hoc experiment fields.
-- `model` / `flags`: transparent pass-through values. NiceEval does not interpret them. They are forwarded through `ctx`, and the adapter decides how to use them. This is the path that [Tier](/concepts/tier) uses for model comparisons (Tier 1) and feature A/B tests (Tier 3).
-- `runs`, `budget`, concurrency, `sandbox`, and related settings: how to run, and how much to run. See [Write Experiments](/guides/write-experiment) for the full field list.
-
-## Matrix comparisons
-
-Write one experiment file per variant you want to compare. Two models means two files that differ by one `model` line. Prompt A/B means two files that differ by one flag. The same eval suite runs under each experiment, and `niceeval view` shows pass rate, cost, and latency side by side. See [Experiments](/guides/experiments) for which comparisons fit this pattern and how to read the output.
-
-## Related reading
-
-- [Write Experiments](/guides/write-experiment) - The full `defineExperiment` field set: runs, budget, concurrency, and sandbox.
-- [Experiments](/guides/experiments) - How to organize and read cross-agent, cross-model, and cross-flag comparisons.
-- [Evals](/concepts/evals) - The other half: what an eval is, plus its lifecycle and outcomes.
-- [Tier](/concepts/tier) - Where `model` and `flags` become effective.

package/docs-site/concepts/hitl.mdx

@@ -1,83 +0,0 @@
----
-title: "Human-in-the-loop (HITL): pause, answer, continue"
-sidebarTitle: "HITL"
-description: "What human-in-the-loop means in Claude Code, Codex, and AI SDK apps, and how NiceEval turns it into an eval path through waiting turns, input.requested events, and t.respond()."
----
-
-HITL (human-in-the-loop) means the agent pauses in the middle of execution and hands the decision back to a person. It is not an error. It is an intentional safety gate or information gate: approve sensitive actions, provide missing information, or choose between multiple options.
-
-You probably already use it every day:
-
-- **Claude Code** pauses before risky commands or out-of-scope file edits and asks for approval.
-- **Codex** approval modes such as suggest and auto-edit decide which actions need confirmation first.
-- **AI SDK apps** often surface tool calls in the UI when the tool does not have `execute`, let the human confirm, then return the result with `addToolResult`.
-
-That branch needs regression coverage like any other behavior: does the action happen after approval, stay blocked after rejection, and stop at the right point? But evals run automatically, without a human. NiceEval compresses the human into three programmable actions: notice that the agent parked (`t.parked()`), inspect what it is waiting for (`t.requireInputRequest()`), and answer in place of the human (`t.respond()`).
-
-## How a pause is represented: `waiting` + `input.requested`
-
-The `Turn.status` returned from `send` has three values: `"completed"`, `"failed"`, and `"waiting"`. `"waiting"` means the turn has not finished. The agent is paused for human input. It is not a failure. Once the answer arrives, the same turn continues instead of restarting the request or opening a new conversation.
-
-The paused turn must also say **what** it is waiting for. Each pending question emits one `input.requested` event with a stable request `id`, the `action` where it paused, and optional `options` such as approve / deny. Both parts matter: `waiting` tells evals they should answer, and `input.requested` tells them what to answer.
-
-## One complete handshake
-
-`t.respond` does not use a separate channel. It is just **another normal `send`**, and the human decision arrives as structured input:
-
-![A full HITL handshake: t.send reaches an approval gate, returns waiting with input.requested, t.parked and t.requireInputRequest inspect the pause, then t.respond sends the structured decision and the same turn continues to completion.](/images/hitl-handshake-zh.svg)
-
-Two easy mistakes to avoid:
-
-- **The response turn is not a new conversation.** It continues on the same session line, so `ctx.session` still points to the same conversation.
-- **The adapter should not infer answers from free text.** `input.responses` already carries `{ requestId, optionId }` per request, or `{ requestId, text }` for free text answers.
-
-## On the eval side: three actions
-
-```ts
-const draft = await t.send("Send a weekly status email to my manager.");
-draft.parked();
-
-const req = t.requireInputRequest({ action: "send_email" });
-
-await t.respond({ request: req, optionId: "approve" });
-t.calledTool("send_email", { status: "completed" });
-```
-
-- `t.parked()` asserts that the turn really stopped in `waiting`.
-- `t.requireInputRequest(filter)` selects exactly one pending request, matching fields such as `id`, `prompt`, `action`, or `optionIds`.
-- `t.respond(...)` answers it and emits the next turn. Use `t.respondAll(optionId)` when multiple pending requests of the same kind should get the same answer.
-
-Approval and rejection are two branches of the same gate. Write both evals if you want full coverage: one for "it happened after approval" and one for "it stayed blocked after rejection".
-
-## On the adapter side: two obligations, one continuation
-
-HITL asks the adapter to do three things. The first two happen on the paused turn, the third happens when the answer arrives:
-
-1. Return `status: "waiting"` honestly when the turn pauses.
-2. Emit one `input.requested` event for each pending question, with a stable `id` and as many populated fields as possible.
-3. On the next `send`, hand the answer back to the application by `requestId`, then continue from the paused point instead of replaying the original request.
-
-The paused execution state lives on the session line in `ctx.session`, typically as `hold(...)` on the paused turn and `take()` on the response turn. See [How to write send](/guides/write-send#step-5-hitl) for the full streaming skeleton.
-
-Like the rest of NiceEval capabilities, HITL is not declared with a boolean flag. **If you implement it, it exists.** If `send` returns `"waiting"` and emits `input.requested`, `t.respond` works. If it does not, the eval fails explicitly at `parked()` or `requireInputRequest()`.
-
-## Rejection is not a failure
-
-Human denial and tool failure are different things. Use different statuses in the event stream. A human-blocked tool call should produce `action.result` with `status: "rejected"`, not `"failed"`. That way `t.noFailedActions()` still passes for an intentional rejection, and `t.calledTool("deploy", { status: "rejected" })` can assert the rejection branch precisely.
-
-## Which agents do not need HITL
-
-Not every agent has this branch. Skip the whole feature when it does not exist:
-
-- **Single-shot agents** that answer in one pass and never wait for a person.
-- **Approval gates disabled in the runtime mode**. Claude Code and Codex support HITL as products, but the built-in `claude-code`, `codex`, and `bub` sandbox agents usually run fully automated and therefore do not expose HITL.
-- **Human review outside the execution loop**, such as a ticket system that reviews agent output later. That does not happen inside `send`, so NiceEval should not model it as HITL.
-
-The rule is simple: use HITL only when your agent's execution loop has a branch that stops for a human and then continues from the same point.
-
-## Related reading
-
-- [Drive](/concepts/drive#human-in-the-loop-hitl) - The full eval-side APIs: `t.parked()`, `t.requireInputRequest()`, `t.respond()`, and `t.respondAll()`.
-- [Adapter](/concepts/adapter#inputs-for-the-different-answers) - The structured adapter inputs that carry human replies.
-- [How to write send](/guides/write-send) - The practical adapter-side skeleton with `ctx.session.hold` / `take`.
-- [Connect Your Agent](/guides/connect-your-agent) - The full integration map.

package/docs-site/concepts/judge.mdx

@@ -1,129 +0,0 @@
----
-title: "LLM-as-judge in niceeval: scoring open-ended output"
-sidebarTitle: "Judge"
-description: "How t.judge, session.judge, and turn.judge use a separate evaluator model to score factuality, closed-ended quality, and summarization — plus model resolution and severity."
----
-
-Judge assertions are the fifth scoring mechanism, alongside the four covered in [Assert](/concepts/assert). Use them when correctness cannot be expressed as a rule — for open-ended prose, tone, factual consistency, or summarization quality. The evaluator model is **entirely separate** from the agent under test, which prevents self-evaluation bias: the same model grading its own output would be prone to rating itself generously.
-
-```ts
-t.judge.autoevals.factuality(reference).atLeast(0.8);   // factual consistency with a reference text
-t.judge.autoevals.closedQA("Is the response appropriate for a 10-year-old?");
-t.judge.autoevals.summarizes(sourceDocument);            // faithful summarization check
-t.judge.autoevals.closedQA("Custom scoring rubric description", { on: t.reply });
-```
-
-## The three judge shapes
-
-`t.judge.autoevals` exposes exactly three fixed forms, whatever you're evaluating maps onto one of them:
-
-- **`closedQA(question, opts?)`** — a closed-ended judgment: does the output satisfy this criterion? Best for rubric-style checks ("is the tone polite and specific", "does this avoid inventing policy details").
-- **`factuality(reference, opts?)`** — factual consistency between the output and a reference text you provide.
-- **`summarizes(source, opts?)`** — whether the output is a faithful summary of a source document.
-
-The `{ on }` option specifies which value to evaluate; it defaults based on the receiver (see below). The `{ model }` option overrides the judge model for a single call.
-
-## Where judge lives: `t`, `session`, and `turn`
-
-Like scoped assertions, judge calls follow the rule that **the receiver decides the default material, not the call site**:
-
-```ts
-t.judge.autoevals.closedQA("...")        // defaults to the primary session's full conversation text
-session.judge.autoevals.closedQA("...")  // (from t.newSession()) defaults to that session's conversation text
-turn.judge.autoevals.closedQA("...")     // (from t.send()'s return value) defaults to turn.message only
-```
-
-`t.judge` and `session.judge` are session-level: they're the right choice for judging overall reply quality or cross-turn consistency across a whole conversation. `turn.judge` is turn-level: it only sees the message from that one turn, which is what you want when different turns in a multi-turn eval need different rubrics:
-
-```ts
-const turn1 = await t.send("What's in this image?");                 // turn 1: image
-const turn2 = await t.send("What color is the background?");         // turns 2–3: text follow-ups
-const turn3 = await t.send("What color is the shape in the middle?"); // testing cross-turn memory
-
-t.judge.autoevals
-  .closedQA("Does the assistant stay grounded in the image from turn 1 throughout?")
-  .atLeast(0.7);
-
-turn3.judge.autoevals.closedQA("Does this turn answer the shape's color?").gate();
-```
-
-To judge material that isn't the conversation itself — a sandbox diff, file content, or anything else — pass it explicitly with `{ on }`, regardless of which receiver you're calling from:
-
-```ts
-t.judge.autoevals.closedQA("Is the generated code idiomatic TypeScript?", {
-  on: t.sandbox.diff.get("src/handler.ts"),
-}).atLeast(0.7);
-```
-
-## Judge model resolution
-
-The judge model is resolved from most- to least-specific:
-
-```
-Per-call { model } option
-  ↓
-Per-eval judge.model
-  ↓
-Global config judge.model
-```
-
-```ts
-// niceeval.config.ts — global default
-defineConfig({ judge: { model: "anthropic/claude-haiku-4-5" } });
-
-// A specific eval that needs a more capable judge
-defineEval({
-  judge: { model: "anthropic/claude-opus-4-8" },
-  async test(t) {
-    t.judge.autoevals.factuality(reference);  // uses claude-opus-4-8 for this eval
-  },
-});
-
-// A single call override
-t.judge.autoevals.closedQA("rubric", { on: t.reply, model: "openai/gpt-4o" });
-```
-
-If all three levels are unset, NiceEval falls back to the environment variable `NICEEVAL_JUDGE_MODEL`. If that is also unset, the judge call fails as a configuration error. There is no built-in default judge model.
-
-## Judge endpoint and key: OpenAI-compatible protocol
-
-Judge calls use an **OpenAI-compatible `/chat/completions`** endpoint. OpenAI, DeepSeek, and any compatible gateway can act as the judge backend:
-
-```ts
-defineConfig({
-  judge: {
-    model: "deepseek-chat",
-    baseUrl: "https://api.deepseek.com/v1",
-    apiKeyEnv: "DEEPSEEK_API_KEY",
-  },
-});
-```
-
-`baseUrl` and the API key are both resolved from most-specific to most-generic:
-
-| Item | Resolution order |
-|---|---|
-| Endpoint | `judge.baseUrl` -> `NICEEVAL_JUDGE_BASE` -> `CODEX_BASE_URL` -> `OPENAI_BASE_URL` -> `https://api.openai.com/v1` |
-| Key | env var named by `judge.apiKeyEnv` -> `NICEEVAL_JUDGE_KEY` -> `CODEX_API_KEY` -> `OPENAI_API_KEY` |
-
-<Warning>
-  When no API key resolves, judge assertions are skipped silently: no error, no score, and the eval keeps running. After configuring keys, run one eval with `t.judge` and confirm the score appears in `niceeval view`.
-</Warning>
-
-## Severity: judge defaults to soft
-
-Judge calls are scoring functions like any other assertion, so they follow the same gate/soft mechanics described in [Assert · Gate vs soft severity](/concepts/assert#gate-vs-soft-severity) — but their **default** is different from most value matchers:
-
-```ts
-t.judge.autoevals.closedQA("Is the tone polite?");              // no threshold → soft, pure score, never fails the eval
-t.judge.autoevals.closedQA("Is the tone polite?").atLeast(0.7); // soft with a threshold → fails only under --strict
-t.judge.autoevals.closedQA("Is the tone polite?").gate();       // promoted to gate → fails immediately if below threshold
-```
-
-With no `.atLeast()` and no `.gate()`, a bare judge call is recorded purely as a quality score — it shows up on the eval's score chip but can never fail the run on its own. This is the right default for judge: it's a probabilistic model scoring another probabilistic model's output, and treating every rubric miss as a hard failure would make suites flaky. Reach for `.gate()` only when the judged property is truly load-bearing (for example, "the response must not fabricate account details").
-
-## Related pages
-
-- [Assert](/concepts/assert) — gate vs soft severity in full, plus the outcome rules judge scores feed into.
-- [Drive](/concepts/drive) — `t.send()`, `t.newSession()`, and the `Turn` / session handles that `t.judge`, `session.judge`, and `turn.judge` hang off of.
-- [Evals](/concepts/evals) — how judge scores fold into the eval lifecycle and outcome types.

package/docs-site/concepts/overview.mdx

@@ -1,98 +0,0 @@
----
-title: "NiceEval architecture: evals, agents, and sandboxes"
-sidebarTitle: "Overview"
-description: "Understand how NiceEval, Adapters, and sandbox backends work together to evaluate any AI agent through one API."
----
-
-NiceEval separates **what you evaluate** from **how the subject under test is connected**. NiceEval discovers evals, schedules runs, scores assertions, and writes reports. Adapters call the system under test. Sandbox backends isolate filesystems for coding-agent tasks.
-
-## Four layers
-
-```text
-Eval files / fixtures
-        |
-        v
-NiceEval
-        |
-        v
-Adapter
-        |
-        v
-Subject under test / Sandbox backend
-```
-
-## What NiceEval owns
-
-<CardGroup cols={2}>
-  <Card title="Eval discovery" icon="magnifying-glass">
-    Finds `*.eval.ts` files and fixture directories, then derives stable IDs from paths.
-  </Card>
-  <Card title="Scheduling" icon="timeline">
-    Controls concurrency, retries, attempts, runs, and early exit.
-  </Card>
-  <Card title="Assertions and scoring" icon="chart-bar">
-    Collects `t.check`, scoped assertions, judge scores, and test results.
-  </Card>
-  <Card title="Cache" icon="database">
-    Skips cases that already passed when the fingerprint is unchanged.
-  </Card>
-  <Card title="Reports" icon="file-lines">
-    Writes console, JSON, JUnit, and local viewer data.
-  </Card>
-  <Card title="Artifacts" icon="folder-open">
-    Saves summaries, event streams, transcripts, diffs, traces, and logs.
-  </Card>
-</CardGroup>
-
-## Adapter
-
-<Note>
-  `Agent` is the abstraction NiceEval calls. `Adapter` is the concrete implementation you write. NiceEval does not know your private protocol, CLI arguments, or auth rules.
-</Note>
-
-That boundary keeps the runner and scorers generic. Evaluating your own AI app, Claude Code, Codex, or a custom agent should not change core scoring behavior.
-
-## Sandbox
-
-<Tabs>
-  <Tab title="Docker">
-    Local container backend for coding-agent evals in development and CI.
-  </Tab>
-  <Tab title="Vercel Sandbox">
-    Cloud sandbox backend for stronger isolation or larger resources.
-  </Tab>
-  <Tab title="Third-party backend">
-    Any service can integrate by implementing the `Sandbox` interface.
-  </Tab>
-</Tabs>
-
-## Key terms
-
-<AccordionGroup>
-  <Accordion title="Eval">A test case: description, optional agent reference, and `test(t)`.</Accordion>
-  <Accordion title="Agent">The connection NiceEval calls by name. It returns a standard `Turn`.</Accordion>
-  <Accordion title="Adapter">The concrete implementation that knows how to call your runtime or CLI.</Accordion>
-  <Accordion title="Sandbox">The isolated runtime used by coding agents.</Accordion>
-  <Accordion title="Turn">The immutable result snapshot from one `t.send()` or `t.respond()`.</Accordion>
-  <Accordion title="Artifact">A structured result file written after a run.</Accordion>
-  <Accordion title="Experiment">A checked-in run configuration for comparing agents, models, flags, runs, budgets, and sandboxes.</Accordion>
-</AccordionGroup>
-
-## End-to-end flow
-
-<Steps>
-  <Step title="Discovery">Find eval files and fixtures.</Step>
-  <Step title="Scheduling">Plan attempts from concurrency, cache, runs, and early-exit settings.</Step>
-  <Step title="Agent send">Call the selected adapter and receive a standard `Turn`.</Step>
-  <Step title="Scoring">Run assertions, judge checks, and tests.</Step>
-  <Step title="Outcome">Fold results into `passed`, `failed`, `errored`, or `skipped`.</Step>
-  <Step title="Reporting & artifacts">Write reports and structured files.</Step>
-</Steps>
-
-## Related
-
-- [Evals](/concepts/evals) - what an eval is and how its lifecycle works.
-- [Adapter](/concepts/adapter) - how to write an adapter and use it in an experiment.
-- [Drive](/concepts/drive) - `t.send()`, sessions, and HITL from the eval side.
-- [Assert](/concepts/assert) - assertion vocabulary and outcome rules.
-- [Judge](/concepts/judge) - LLM-as-judge for open-ended quality.

package/docs-site/concepts/tier.mdx

@@ -1,42 +0,0 @@
----
-title: "Integration Tiers: Three Levels of Investment, Three Sets of Capabilities"
-sidebarTitle: "Tier"
-description: "Grouped by where the Adapter connects and what extra observability data it receives, integration comes in three tiers: Tier 1 is send only, Tier 2 adds OTel, Tier 3 modifies the application to expose experiment flags. What each tier buys and when to move up."
----
-
-Connecting to NiceEval is not an all-or-nothing one-time project. Grouped by where the Adapter connects and what extra observability data it receives, integration comes in three tiers — each tier adds one kind of investment on top of the previous one and buys a new set of capabilities, and every eval you have already written keeps working unchanged the whole way up.
-
-## Tier 1: send only
-
-Your application code stays untouched; the integration is a single Adapter file — `send` talks to the same interface your user-facing frontend already uses.
-
-**The full assertion set lives at this tier**: text assertions and judges, structured output validation, multi-turn and session isolation, HITL approval flows, tool assertions from event mapping, and usage assertions from the `usage` returned by `send` — every verdict is based entirely on the `Turn` returned by `send`, and the next two tiers add no further assertions. If the application interface itself exposes model selection, **model comparison** experiments also live at this tier — the experiment's `model` reaches the Adapter via `ctx.model` and is simply forwarded with the request.
-
-Most teams start here, and for many this tier is all they ever need.
-
-## Tier 2: send + OTel
-
-Same `send`, same event mapping — the only change is that your application also sends its OTel spans to NiceEval. If the app is already instrumented (AI SDK telemetry, LangGraph, OpenLLMetry / OpenInference, hand-rolled gen_ai spans), this is zero change; if not, what you add is a piece of generic OTel initialization — that is observability infrastructure, not an eval-specific modification.
-
-What this tier buys is **observability**: the call waterfall in `niceeval view` — every model call and tool execution inside the application, each with its own latency and tokens, laid out as a per-turn timeline. Assertions are unaffected: spans only feed the waterfall; they never enter the event stream or feed assertions. See [OTel Integration](/guides/connect-otel) for the walkthrough.
-
-## Tier 3: application changes + experiment flags
-
-When you want to evaluate "which prompt / tool set / feature toggle works better" — a **feature A/B test** — the point of comparison lives inside the application, out of reach for the first two tiers. This tier changes the application's internal code to expose the variants as externally selectable configuration: the experiment's `flags` pass through `ctx.flags` to the Adapter, the Adapter hands them to the application with the request (an HTTP header, a request body field, or an environment variable all work), and the application switches variants by flag.
-
-Note that what gets modified is the application (exposing variants as configuration), not the integration surface — the Adapter itself stays the same and still calls the application only through its external interface.
-
-## Non-intrusive is the baseline for the first two tiers
-
-The first two tiers are both **non-intrusive**: you start the application your own way (`pnpm start`, deployed wherever) — the eval side never spawns the application process or opens a new port. The Adapter only talks to **the same interface your user-facing frontend already uses** (an HTTP endpoint, an SSE stream), and reports a clear "start the application first" error when it cannot connect. The only difference between the two tiers is observability data: Tier 1 has only the events returned by `send`; Tier 2 adds a copy of spans sent to NiceEval, and the waterfall appears with them.
-
-## How to move up
-
-The three tiers are progressive, not mutually exclusive: start with Tier 1 to get baselines and model comparison running; move up to Tier 2 when you want the call waterfall; move up to Tier 3 when you need to compare variants inside the application. Each upgrade only adds something to the Adapter or the application — how to organize comparisons on the experiment side is covered in [Experiments](/concepts/experiment).
-
-## Related reading
-
-- [Connect Your Agent](/guides/connect-your-agent) — The integration overview: minimal integration and parameter channels.
-- [Adapter](/concepts/adapter) — The contract itself: what `send` receives and returns, and how `ctx.model` / `ctx.telemetry` / `ctx.flags` appear per tier.
-- [OTel Integration](/guides/connect-otel) — The full Tier 2 walkthrough.
-- [Experiments](/concepts/experiment) — How model / flags comparisons are declared on the experiment side.

package/docs-site/docs-ref/00-index.md

@@ -1,23 +0,0 @@
-# 给 Agent 的 docs 写作指南
-
-整理日期:2026-07-04。蒸馏自 [`../dev-docs-dx-sources/`](../dev-docs-dx-sources/00-index.md) 的来源级提取(Diátaxis、Divio、Google / Microsoft / GitHub 风格指南、Write the Docs、The Good Docs Project、MDN、I'd Rather Be Writing、GetDX / Twilio / Phodal 的 DX 资料)。
-
-这套指南回答"怎么写好一页用户文档",按写作流程组织。仓库特定的术语表、目录职责和校验命令以 [`docs-site/AGENTS.md`](../../AGENTS.md) 为准,本指南不重复维护;两者冲突时以 AGENTS.md 为准。
-
-## 使用顺序
-
-写任何 `docs-site/` 页面前,按顺序过一遍:
-
-1. [01-page-types.md](./01-page-types.md):先判定这页是 Tutorial、How-to、Reference、Explanation 还是 Troubleshooting。这是最重要的一步,页面类型错了后面全错。
-3. [03-style-rules.md](./03-style-rules.md):行文、步骤、可扫描性、标题与搜索意图。
-4. [04-code-examples.md](./04-code-examples.md):代码示例的写法和维护义务。
-5. [05-dx-failure-paths.md](./05-dx-failure-paths.md):失败路径覆盖和"文档是 DX 的一部分"视角。
-6. [06-checklists.md](./06-checklists.md):动笔前、按类型、发布前三层检查清单。
-
-## 三条最高原则
-
-如果只记三件事:
-
-1. **一页只服务一种需求。** 学习、干活、查事实、理解,四选一。页面边界模糊是文档难用的首要原因。(Diátaxis)
-2. **"刚好足够"胜过"全面覆盖"。** 更多内容会让所有内容更难找;优先写高频高价值路径,不为完整性堆低频分支。(GitHub Docs)
-3. **错误的文档比没有文档更糟。** 每个声明的命令、字段、路径、能力,写之前先对当前代码或真实运行结果核实;未实现的能力不进正式文档。(Write the Docs / MDN)