niceeval 0.4.4 → 0.4.5

package/docs-site/concepts/assert.mdx

@@ -0,0 +1,243 @@
+---
+title: "niceeval assertions: values, scoped facts, tests, and cost"
+sidebarTitle: "Assert"
+description: "niceeval's assertion vocabulary — value assertions, scoped assertions, test-as-scoring, and efficiency checks — plus gate vs soft severity and the outcome rules that fold them into a result."
+---
+
+Assertion is the process of taking everything an agent did during an eval — every message, tool call, file change, and token spent — and folding it into a single, actionable outcome. niceeval gives you four assertion mechanisms that complement each other: some check values immediately, some assess the whole run after it completes, some execute tests inside the sandbox, and some measure efficiency. All four produce the same `Assertion` type and feed into the same outcome rules. For the fifth mechanism — asking a language model to judge open-ended quality — see [Judge](/concepts/judge).
+
+## The four assertion mechanisms
+
+<CardGroup cols={2}>
+  <Card title="1. Value assertions" icon="equals">
+    `t.check(value, matcher)` and `t.require(value, matcher)` evaluate a specific value immediately against a matcher from `niceeval/expect`. Use these for facts you can verify inline.
+  </Card>
+  <Card title="2. Scoped assertions" icon="crosshairs">
+    `t.succeeded()`, `t.calledTool()`, `t.messageIncludes()`, and friends are registered during `test(t)` but evaluated **after** the function returns, against the complete turn data. Use these for whole-run facts.
+  </Card>
+  <Card title="3. Test-as-scoring" icon="flask">
+    For sandbox evals, run project tests, build scripts, or focused probe commands from `test(t)`. Use this for coding tasks where file content and build results are the ground truth.
+  </Card>
+  <Card title="4. Efficiency assertions" icon="gauge">
+    `t.maxTokens()` and `t.maxCost()` turn token usage and estimated cost into scoreable dimensions. An agent that answers correctly but burns ten times the expected tokens should not score the same as one that answers efficiently.
+  </Card>
+</CardGroup>
+
+## Gate vs soft severity
+
+Every assertion carries a **severity** that determines how it influences the final outcome. There are exactly two severities:
+
+<Tabs>
+  <Tab title="gate">
+    A gate assertion is a hard requirement. If it fails, the entire eval is immediately classified as `failed` — regardless of how well every other assertion passed. Use gate for facts that must be true: "the agent called the correct tool", "the response parsed as valid JSON", "no shell commands errored."
+
+    Most matchers in `niceeval/expect` (`includes`, `equals`, `matches`, `satisfies`) default to gate. Scoped assertions like `t.succeeded()` and `t.calledTool()` also default to gate.
+  </Tab>
+  <Tab title="soft">
+    A soft assertion is a quality score with a numeric threshold. If the score falls below the threshold, the eval becomes `passed` rather than `failed` — a signal that there is a quality regression, but not a hard breakage. Soft failures only count as failures when you run with `--strict`.
+
+    Use soft for continuous judgments where the answer is "how good" rather than "correct or not": similarity scoring, LLM-as-judge factuality ratings, cost budgets you want to track without blocking CI.
+
+    Matchers that produce a continuous score (`similarity`) and all judge calls default to soft.
+  </Tab>
+</Tabs>
+
+You can override the default severity with a chain method on any matcher or assertion:
+
+```ts
+t.check(t.reply, includes("confirmed"));          // gate (default)
+t.check(t.reply, similarity(expected).gate());    // promote to gate
+t.maxTokens(80_000).atLeast(0.7);                 // demote to soft with a threshold
+```
+
+## Outcome rules
+
+Once all assertions are collected, the runner folds them into a single outcome in this order:
+
+```
+Execution error (timeout / thrown exception / author mistake)  →  failed
+Explicit t.skip(reason) was called                             →  skipped
+Any gate assertion failed                                       →  failed
+All gates passed, but at least one soft is below its threshold →  passed (flagged under --strict)
+Otherwise                                                       →  passed
+```
+
+<CardGroup cols={2}>
+  <Card title="passed" icon="circle-check" color="#22c55e">
+    No errors, all gate assertions passed, all soft assertions met their thresholds.
+  </Card>
+  <Card title="failed" icon="circle-xmark" color="#ef4444">
+    Execution error **or** at least one gate assertion did not pass. Hard failure.
+  </Card>
+  <Card title="passed (scored)" icon="chart-bar" color="#f59e0b">
+    All gates passed, but at least one soft fell below its threshold. Quality regression — silent by default, red under `--strict`.
+  </Card>
+  <Card title="skipped" icon="forward" color="#6b7280">
+    `t.skip("reason")` was called. Excluded from pass-rate calculations entirely.
+  </Card>
+</CardGroup>
+
+When you run an eval more than once (`runs > 1`), the per-eval summary becomes a **pass rate** (the fraction of runs that produced `passed`) and an average latency, rather than a single outcome.
+
+## 1. Value assertions — `niceeval/expect` matchers
+
+`t.check(value, assertion)` evaluates the assertion immediately and records the result. `t.require(value, assertion)` does the same but **throws immediately** if the assertion fails, aborting the rest of the test function. Use `t.require` for preconditions: if a required fact is false, there is no point continuing.
+
+The matchers available from `niceeval/expect`:
+
+```ts
+import {
+  includes,    // substring or regex match          (default: gate)
+  equals,      // deep equality                     (default: gate)
+  matches,     // Standard Schema (Zod etc.) check  (default: gate)
+  similarity,  // normalized Levenshtein 0–1        (default: soft)
+  satisfies,   // custom predicate + label          (default: gate)
+} from "niceeval/expect";
+```
+
+Usage examples:
+
+```ts
+// Check that the agent's reply contains a specific string
+t.check(t.reply, includes("order confirmed"));
+
+// Deep-equal check on structured output
+t.check(turn.data, equals({ status: "refund", amount: 42 }));
+
+// Validate structured output against a Zod schema
+t.check(turn.data, matches(z.object({ intent: z.enum(["refund", "ship"]) })));
+
+// Similarity with an explicit threshold
+t.check(t.reply, similarity("expected answer").atLeast(0.8));
+
+// Custom predicate
+t.check(turn.data, satisfies((d) => d.total > 0, "total is positive"));
+```
+
+Matchers are pure functions — `(value) => number` — so you can write your own and pass them to `t.check` without any special registration.
+
+## 2. Scoped assertions
+
+Scoped assertions are registered during `test(t)` but evaluated **after the function returns**, against the complete accumulated turn data. They read from the standard event stream that `t.send()` produces (see [Drive](/concepts/drive)) and its derived facts — so as long as your adapter produces correct events, these assertions work identically for every agent.
+
+<Warning>
+Scoped assertions only appear on `t` if the agent has declared the corresponding capability. Calling `t.calledTool()` when the agent has not declared `toolObservability: true` is a compile error.
+</Warning>
+
+### Run / session dimension
+
+```ts
+t.succeeded();                  // run completed with no failed actions and no unresolved HITL
+t.parked();                     // cleanly stopped on a HITL input.requested event
+t.messageIncludes("Regards,");  // all message events concatenated contain this string/regex
+```
+
+### Tool / action dimension
+
+```ts
+t.calledTool("bash", { input: { command: /^pwd/ }, count: 1 });
+t.notCalledTool("shell", { input: { command: /npm i/ } });
+t.toolOrder(["read_file", "write_file"]);   // relative order of tool calls
+t.usedNoTools();
+t.maxToolCalls(5);
+t.loadedSkill("memory-v2");                // sugar for calledTool("load_skill", ...)
+t.calledSubagent("researcher", { remoteUrl: /api\.example/ });
+t.noFailedActions();                       // no tool, subagent, or skill has status "failed"
+```
+
+The `input` argument to `calledTool` and `notCalledTool` supports a small matching language: a plain object performs deep partial matching, a `RegExp` matches against the serialized input, and a predicate function receives the raw input value.
+
+### Event stream dimension (low-level escape hatch)
+
+```ts
+t.event("input.requested", { count: 1 });
+t.notEvent("error");
+t.eventOrder(["action.called", "subagent.called"]);
+t.eventsSatisfy("read before write", (events) => /* custom predicate */ true);
+```
+
+All scoped assertions above are syntactic sugar for these low-level event stream queries. When none of the higher-level assertions fit your use case, you can drop down to `eventsSatisfy` and write an arbitrary predicate over the raw `StreamEvent[]`.
+
+### Structured output (on `turn`, not `t`)
+
+```ts
+const turn = await t.send("Return the result as JSON");
+turn.outputEquals({ status: "ok" });                         // deep equality on turn.data
+turn.outputMatches(z.object({ status: z.string() }));        // Standard Schema validation
+```
+
+### Workspace dimension (sandbox agents only)
+
+```ts
+t.sandbox.fileChanged("src/Button.tsx");
+t.sandbox.fileDeleted("src/old.ts");
+t.sandbox.diff.isEmpty();                      // no repository files were modified this turn
+t.sandbox.notInDiff(/sk-[A-Za-z0-9]/);         // diff contains no secrets / inline styles
+t.check(await t.sandbox.runCommand("npm", ["test"]), commandSucceeded());         // npm test exited 0
+t.check(await t.sandbox.runCommand("npm", ["run", "build"]), commandSucceeded()); // npm run build exited 0
+t.sandbox.noFailedShellCommands();
+```
+
+`t.sandbox.diff` is a queryable object. `t.sandbox.diff.get("src/Button.tsx")` returns the file's post-change content; `t.sandbox.diff.isEmpty()` checks whether any files changed; `t.sandbox.diff.matches(re)` and `t.sandbox.notInDiff(re)` run a regex over the full diff text.
+
+Scoped assertions follow one rule everywhere: **the receiver decides the scope, not the assertion name.** `t.*` aggregates every turn of the whole eval run (including any `t.newSession()` sessions); `session.*` (from `t.newSession()`) scopes to that one session; `turn.*` (from `t.send()`'s return value) scopes to that single turn only. Same vocabulary, different receiver — see [Drive](/concepts/drive) for what each receiver is.
+
+## 3. Test-as-scoring (sandbox evals)
+
+For sandbox coding evals, run validation commands inside `test(t)` and record their result as assertions.
+
+```ts
+import { commandSucceeded, includes } from "niceeval/expect";
+
+const testResult = await t.sandbox.runCommand("npm", ["test"]);
+t.check(testResult, commandSucceeded());
+
+const src = await t.sandbox.readSourceFiles({ extensions: ["ts"] });
+t.check(src.text(), includes(/z\.object\s*\(/));
+```
+
+You can also assert behavior through the standard event stream: `t.calledTool(...)`, `t.sandbox.noFailedShellCommands()`, `t.eventsSatisfy(...)`, and diff assertions such as `t.sandbox.fileChanged(...)`.
+
+## 4. Efficiency / cost assertions
+
+Token usage is a first-class scoring dimension. An agent that answers correctly but burns far more tokens than expected should not score identically to one that answers efficiently.
+
+```ts
+t.maxTokens(50_000);            // hard token limit for the entire run (gate by default)
+t.maxCost(0.5);                 // estimated cost cap in USD (requires a price table in config)
+t.maxTokens(80_000).atLeast(0.7);     // soft variant — tracked but only red under --strict
+t.check(t.usage.outputTokens, satisfies((n) => n < 10_000, "not verbose"));
+```
+
+`t.usage` is available anywhere inside `test(t)` and exposes `{ inputTokens, outputTokens, cacheReadTokens?, … }`. For sandbox agents, token counts are extracted from the transcript by the adapter; for remote agents, they are returned in `Turn.usage`.
+
+## Custom scorers
+
+A value assertion is just a function `(value) => number | Promise<number>`. You can write custom matchers using `makeAssertion`:
+
+```ts
+import { makeAssertion } from "niceeval/expect";
+import type { Assertion } from "niceeval/expect";
+
+function jsonValid(): Assertion {
+  return makeAssertion({
+    name: "jsonValid",
+    severity: "gate",
+    score: (value) => {
+      try { JSON.parse(String(value)); return 1; }
+      catch { return 0; }
+    },
+  });
+}
+
+t.check(t.reply, jsonValid());
+```
+
+Custom matchers compose with the same chain methods as built-ins: `.gate()`, `.atLeast(0.7)`.
+
+## Related pages
+
+- [Drive](/concepts/drive) — `t.send()`, `t.newSession()`, and HITL: how you produce the Turn data these assertions read from.
+- [Judge](/concepts/judge) — the fifth scoring mechanism, for open-ended quality that can't be expressed as a fixed rule.
+- [Adapter](/concepts/adapter) — how the standard event stream is produced, which scoped assertions depend on.
+- [Evals](/concepts/evals) — how assertions fold into the eval lifecycle and outcome types.

package/docs-site/concepts/drive.mdx

@@ -0,0 +1,118 @@
+---
+title: "How NiceEval drives agents: send, sessions, and HITL"
+sidebarTitle: "Drive"
+description: "t.send() and the Turn it returns, t.sendFile(), multi-turn conversations, t.newSession(), and HITL through t.respond() / t.respondAll()."
+---
+
+Before assertions or judge scoring, you need to make the agent do something. **Drive** is the part of `test(t)` that sends input and gets results back: `t.send()`, `t.sendFile()`, `t.newSession()`, and HITL through `t.respond()` / `t.respondAll()`. Every drive API produces a **Turn**. All assertions and judge checks in NiceEval read from Turn data. See [Assert](/concepts/assert) and [Judge](/concepts/judge) for the scoring side.
+
+## `t.send()` and the `Turn` it returns
+
+`t.send(input)` is the core verb. Under the hood it calls the Adapter's `send(input, ctx)` and normalizes the return value into a `Turn`:
+
+```ts
+const turn = await t.send("What's the weather like in Brooklyn today?");
+
+turn.message;    // last assistant message
+turn.data;       // structured output, only when this send declared output
+turn.status;     // "completed" | "failed" | "waiting"
+turn.events;     // StreamEvent[] for this turn
+turn.usage;      // { inputTokens, outputTokens, ... } when the adapter reports it
+turn.succeeded(); // scored assertion: this turn didn't fail and isn't parked on HITL
+```
+
+`t.reply` is shorthand for the last assistant message in the main session, equivalent to `turn.message` from the most recent `t.send()`. `t.events` is the full accumulated event stream for the main session.
+
+When you need structured output, declare `output` on the send call. The declaration is both a requirement for the app and the type source for `turn.data`:
+
+```ts
+const turn = await t.send("What is the total on this invoice?", {
+  output: z.object({ amount: z.number(), currency: z.string() }),
+});
+t.check(turn.data.amount, equals(3200));
+```
+
+The runner validates `data` against that schema immediately after the adapter returns. If the shape does not match, the turn fails and the diff is reported. If you did not declare `output`, there is no `turn.data`.
+
+<Tip>
+`turn.succeeded()` scopes to this turn only: it checks this turn's `status` and whether it's parked on an unanswered HITL request. Call it right after a `t.send()` that later turns depend on, so a failure surfaces here instead of a pile of confusing follow-on failures.
+</Tip>
+
+## One turn with a file: `t.sendFile()`
+
+`t.sendFile(path, text?)` reads a local file relative to the eval directory, infers the MIME type from the extension, and attaches it to the turn input as a data URL:
+
+```ts
+const turn = await t.sendFile("fixtures/invoice.png", "What is the total on this invoice?");
+t.check(turn.message, includes("$"));
+```
+
+## Multi-turn conversations
+
+Every `await t.send(...)` is a new turn on the same conversation. Keep each return value in a local variable when you want to assert on that turn specifically:
+
+```ts
+const draft = await t.send("Draft a follow-up email.");
+draft.succeeded();
+t.check(draft.message, includes("Best"));
+draft.judge.autoevals.closedQA("Is the tone professional?").atLeast(0.6);
+
+await t.send("Looks good, send it.");
+t.calledTool("send_email");
+```
+
+Whether multi-turn calls actually continue the same conversation depends on whether the adapter's `send` uses the continuation helpers from `ctx.session` (`history()` or `id` + `capture()`). If it does not, each turn becomes a fresh conversation. See [Adapter](/concepts/adapter) and [How to write send](/guides/write-send).
+
+## Independent sessions: `t.newSession()`
+
+`t.newSession()` opens a second conversation line that runs in parallel with the main one and should not share state. The returned handle has the same drive APIs (`send`, `sendFile`, `respond`, `respondAll`) and the same scoped assertions, but only sees its own events:
+
+```ts
+await t.send("My name is Alice.");
+await t.send("What is my name?");
+t.check(t.reply, includes("Alice"));
+
+const fresh = t.newSession();
+await fresh.send("What is my name?");
+t.check(fresh.reply, satisfies((r) => !r.includes("Alice"), "no memory leak"));
+```
+
+<Warning>
+Do not assume `t.newSession()` guarantees isolation by itself. The runner only guarantees a fresh session handle. The adapter still has to open a fresh underlying conversation. If the adapter always resumes the same backend context, `t.newSession()` will quietly share state. Verify isolation with an eval like the one above before trusting it.
+</Warning>
+
+## Human-in-the-loop (HITL)
+
+Some agents stop in the middle of a turn and wait for approval or missing information instead of finishing immediately. In that case the turn ends with `status: "waiting"` and one or more `input.requested` events that describe what it is waiting for. The full model is in [HITL](/concepts/hitl); here the focus is how evals drive it.
+
+```ts
+const draft = await t.send("Draft a follow-up email, but don't send it until I confirm.");
+draft.parked();
+
+const request = t.requireInputRequest({
+  prompt: /send it/,
+  optionIds: ["approve", "reject"],
+});
+
+await t.respond({ request, optionId: "approve" });
+t.calledTool("send_email");
+```
+
+`t.requireInputRequest(filter)` turns a pending HITL request into a concrete value you can inspect and answer. It throws when 0 or more than 1 requests match, so fill as many filter fields as you can (`id`, `prompt`, `display`, `action`, `optionIds`, `input`) to disambiguate. `t.respond(...)` answers it and emits the next turn. String arguments answer requests in order; object arguments such as `{ request, optionId }` target a specific request directly. Under the hood this is still just another `send`: the answer text goes into `input.text`, and structured answers go into `input.responses`.
+
+When the current turn has multiple pending requests of the same kind and they should all get the same answer, use `t.respondAll(optionId)`:
+
+```ts
+await t.send("Ask for approval on each of these file changes.");
+t.requireInputRequest({ display: /approval/ });
+
+await t.respondAll("approve");
+t.succeeded();
+```
+
+## Related reading
+
+- [HITL](/concepts/hitl) - The full concept for paused turns, request/response handshakes, and adapter obligations.
+- [Assert](/concepts/assert) - Assertions that read from `Turn.events` and `Turn.data`.
+- [Judge](/concepts/judge) - What `t.judge`, `session.judge`, and `turn.judge` score by default.
+- [Adapter](/concepts/adapter) - Where session continuation, HITL, and tool assertions come from.

package/docs-site/concepts/evals.mdx

@@ -0,0 +1,108 @@
+---
+title: "Evals in NiceEval: lifecycle, outcome, and files"
+sidebarTitle: "Evals"
+description: "An eval is a test case: description, optional agent reference, and test function. Learn how evals are discovered, scheduled, scored, and reported."
+---
+
+An eval is a runnable test case. It usually lives in a `*.eval.ts` file and is declared with `defineEval`.
+
+## What an eval contains
+
+```ts
+import { defineEval } from "niceeval";
+import { includes } from "niceeval/expect";
+
+export default defineEval({
+  description: "Brooklyn weather query",
+  async test(t) {
+    await t.send("What's the weather like in Brooklyn today?");
+    t.succeeded();
+    t.calledTool("get_weather", { input: { city: "Brooklyn" }, count: 1 });
+    t.check(t.reply, includes("sunny"));
+  },
+});
+```
+
+| Field | Meaning |
+|---|---|
+| `description` | Human-readable description shown in reports |
+| `agent` | Which Adapter to use, usually supplied by the experiment |
+| `test(t)` | Interaction and assertion logic |
+
+<Note>
+  Do not hand-write `id` or `name`. NiceEval derives the eval ID from the file path.
+</Note>
+
+## Path is identity
+
+`evals/weather/brooklyn.eval.ts` has ID `weather/brooklyn`. Positional arguments after the experiment name filter by ID prefix:
+
+```bash
+npx niceeval exp local weather
+npx niceeval exp local weather/brooklyn
+```
+
+## Lifecycle
+
+<Steps>
+  <Step title="Discovery">Load `*.eval.ts` files and fixture directories.</Step>
+  <Step title="Scheduling">Build the execution plan from concurrency, cache, runs, attempts, and early exit.</Step>
+  <Step title="agent.send">`t.send()` calls the selected adapter and receives a standard `Turn`.</Step>
+  <Step title="Scoring">NiceEval collects value assertions, scoped assertions, judge scores, and test results.</Step>
+  <Step title="Outcome">All assertion results fold into one final outcome.</Step>
+  <Step title="Report">Reporters write console output and `.niceeval/` artifacts.</Step>
+</Steps>
+
+## Outcome types
+
+<CardGroup cols={2}>
+  <Card title="passed" icon="circle-check" color="#22c55e">
+    All gate assertions passed and no hard failure occurred.
+  </Card>
+  <Card title="failed" icon="circle-xmark" color="#ef4444">
+    A gate assertion failed or the run itself failed.
+  </Card>
+  <Card title="passed" icon="chart-bar" color="#f59e0b">
+    No gate failed, but soft scores are preserved as scores.
+  </Card>
+  <Card title="skipped" icon="forward" color="#6b7280">
+    The eval was skipped, usually through `t.skip(reason)`.
+  </Card>
+</CardGroup>
+
+## Gate and soft
+
+`gate` is a hard threshold. Failure makes the eval fail. `soft` contributes a score without necessarily failing the eval. See [Assert](/concepts/assert).
+
+## File convention
+
+Only files ending in `.eval.ts` are discovered. Use directories for grouping:
+
+```text
+evals/
+└─ billing/
+   └─ refund.eval.ts  # id: billing/refund
+```
+
+## Dataset fan-out
+
+A file can also default-export an array of `defineEval(...)` calls to generate multiple cases from the same logic:
+
+```ts
+export default rows.map((row) =>
+  defineEval({
+    description: row.task,
+    async test(t) {
+      await t.send(row.prompt);
+      t.check(t.reply, equals(row.expected));
+    },
+  }),
+);
+```
+
+This generates IDs such as `sql/0000` and `sql/0001`. See [Dataset Fan-out](/guides/dataset-fanout).
+
+## Related reading
+
+- [Experiment](/concepts/experiment) - The other half: who to evaluate, how to run it, and why evals are separate from experiments.
+- [Assert](/concepts/assert) - The full outcome rules for gate and soft assertions.

package/docs-site/concepts/experiment.mdx

@@ -0,0 +1,37 @@
+---
+title: "Experiment: who to evaluate and how to run"
+sidebarTitle: "Experiment"
+description: "An experiment is a commit-ready run configuration: which agent to evaluate, which model to use, which flags to enable, and how many times to run the same eval suite."
+---
+
+An **experiment** chooses which agent to evaluate, which model to use, which flags to enable, and how many times to run. Use it for A/B tests, model comparisons, and feature comparisons.
+
+## What an experiment contains
+
+```ts
+import { defineExperiment } from "niceeval";
+import { webAgent } from "../agents/web-agent.ts";
+
+export default defineExperiment({
+  description: "Concise prompt variant",
+  agent: webAgent({ baseUrl: "https://staging.example.com" }), // configured agent instance
+  model: "gpt-5.4",                    // forwarded to the adapter through ctx.model
+  flags: { promptVariant: "concise" }, // forwarded through ctx.flags; visible in evals as t.flags
+  runs: 3,                             // how many times to run each eval
+});
+```
+
+- `agent`: who to evaluate. Put a configured instance here. The subject under test URL and credentials belong in the adapter factory, not as ad hoc experiment fields.
+- `model` / `flags`: transparent pass-through values. NiceEval does not interpret them. They are forwarded through `ctx`, and the adapter decides how to use them. This is the path that [Tier](/concepts/tier) uses for model comparisons (Tier 1) and feature A/B tests (Tier 3).
+- `runs`, `budget`, concurrency, `sandbox`, and related settings: how to run, and how much to run. See [Write Experiments](/guides/write-experiment) for the full field list.
+
+## Matrix comparisons
+
+Write one experiment file per variant you want to compare. Two models means two files that differ by one `model` line. Prompt A/B means two files that differ by one flag. The same eval suite runs under each experiment, and `niceeval view` shows pass rate, cost, and latency side by side. See [Experiments](/guides/experiments) for which comparisons fit this pattern and how to read the output.
+
+## Related reading
+
+- [Write Experiments](/guides/write-experiment) - The full `defineExperiment` field set: runs, budget, concurrency, and sandbox.
+- [Experiments](/guides/experiments) - How to organize and read cross-agent, cross-model, and cross-flag comparisons.
+- [Evals](/concepts/evals) - The other half: what an eval is, plus its lifecycle and outcomes.
+- [Tier](/concepts/tier) - Where `model` and `flags` become effective.

package/docs-site/concepts/hitl.mdx

@@ -0,0 +1,83 @@
+---
+title: "Human-in-the-loop (HITL): pause, answer, continue"
+sidebarTitle: "HITL"
+description: "What human-in-the-loop means in Claude Code, Codex, and AI SDK apps, and how NiceEval turns it into an eval path through waiting turns, input.requested events, and t.respond()."
+---
+
+HITL (human-in-the-loop) means the agent pauses in the middle of execution and hands the decision back to a person. It is not an error. It is an intentional safety gate or information gate: approve sensitive actions, provide missing information, or choose between multiple options.
+
+You probably already use it every day:
+
+- **Claude Code** pauses before risky commands or out-of-scope file edits and asks for approval.
+- **Codex** approval modes such as suggest and auto-edit decide which actions need confirmation first.
+- **AI SDK apps** often surface tool calls in the UI when the tool does not have `execute`, let the human confirm, then return the result with `addToolResult`.
+
+That branch needs regression coverage like any other behavior: does the action happen after approval, stay blocked after rejection, and stop at the right point? But evals run automatically, without a human. NiceEval compresses the human into three programmable actions: notice that the agent parked (`t.parked()`), inspect what it is waiting for (`t.requireInputRequest()`), and answer in place of the human (`t.respond()`).
+
+## How a pause is represented: `waiting` + `input.requested`
+
+The `Turn.status` returned from `send` has three values: `"completed"`, `"failed"`, and `"waiting"`. `"waiting"` means the turn has not finished. The agent is paused for human input. It is not a failure. Once the answer arrives, the same turn continues instead of restarting the request or opening a new conversation.
+
+The paused turn must also say **what** it is waiting for. Each pending question emits one `input.requested` event with a stable request `id`, the `action` where it paused, and optional `options` such as approve / deny. Both parts matter: `waiting` tells evals they should answer, and `input.requested` tells them what to answer.
+
+## One complete handshake
+
+`t.respond` does not use a separate channel. It is just **another normal `send`**, and the human decision arrives as structured input:
+
+![A full HITL handshake: t.send reaches an approval gate, returns waiting with input.requested, t.parked and t.requireInputRequest inspect the pause, then t.respond sends the structured decision and the same turn continues to completion.](/images/hitl-handshake-zh.svg)
+
+Two easy mistakes to avoid:
+
+- **The response turn is not a new conversation.** It continues on the same session line, so `ctx.session` still points to the same conversation.
+- **The adapter should not infer answers from free text.** `input.responses` already carries `{ requestId, optionId }` per request, or `{ requestId, text }` for free text answers.
+
+## On the eval side: three actions
+
+```ts
+const draft = await t.send("Send a weekly status email to my manager.");
+draft.parked();
+
+const req = t.requireInputRequest({ action: "send_email" });
+
+await t.respond({ request: req, optionId: "approve" });
+t.calledTool("send_email", { status: "completed" });
+```
+
+- `t.parked()` asserts that the turn really stopped in `waiting`.
+- `t.requireInputRequest(filter)` selects exactly one pending request, matching fields such as `id`, `prompt`, `action`, or `optionIds`.
+- `t.respond(...)` answers it and emits the next turn. Use `t.respondAll(optionId)` when multiple pending requests of the same kind should get the same answer.
+
+Approval and rejection are two branches of the same gate. Write both evals if you want full coverage: one for "it happened after approval" and one for "it stayed blocked after rejection".
+
+## On the adapter side: two obligations, one continuation
+
+HITL asks the adapter to do three things. The first two happen on the paused turn, the third happens when the answer arrives:
+
+1. Return `status: "waiting"` honestly when the turn pauses.
+2. Emit one `input.requested` event for each pending question, with a stable `id` and as many populated fields as possible.
+3. On the next `send`, hand the answer back to the application by `requestId`, then continue from the paused point instead of replaying the original request.
+
+The paused execution state lives on the session line in `ctx.session`, typically as `hold(...)` on the paused turn and `take()` on the response turn. See [How to write send](/guides/write-send#step-5-hitl) for the full streaming skeleton.
+
+Like the rest of NiceEval capabilities, HITL is not declared with a boolean flag. **If you implement it, it exists.** If `send` returns `"waiting"` and emits `input.requested`, `t.respond` works. If it does not, the eval fails explicitly at `parked()` or `requireInputRequest()`.
+
+## Rejection is not a failure
+
+Human denial and tool failure are different things. Use different statuses in the event stream. A human-blocked tool call should produce `action.result` with `status: "rejected"`, not `"failed"`. That way `t.noFailedActions()` still passes for an intentional rejection, and `t.calledTool("deploy", { status: "rejected" })` can assert the rejection branch precisely.
+
+## Which agents do not need HITL
+
+Not every agent has this branch. Skip the whole feature when it does not exist:
+
+- **Single-shot agents** that answer in one pass and never wait for a person.
+- **Approval gates disabled in the runtime mode**. Claude Code and Codex support HITL as products, but the built-in `claude-code`, `codex`, and `bub` sandbox agents usually run fully automated and therefore do not expose HITL.
+- **Human review outside the execution loop**, such as a ticket system that reviews agent output later. That does not happen inside `send`, so NiceEval should not model it as HITL.
+
+The rule is simple: use HITL only when your agent's execution loop has a branch that stops for a human and then continues from the same point.
+
+## Related reading
+
+- [Drive](/concepts/drive#human-in-the-loop-hitl) - The full eval-side APIs: `t.parked()`, `t.requireInputRequest()`, `t.respond()`, and `t.respondAll()`.
+- [Adapter](/concepts/adapter#inputs-for-the-different-answers) - The structured adapter inputs that carry human replies.
+- [How to write send](/guides/write-send) - The practical adapter-side skeleton with `ctx.session.hold` / `take`.
+- [Connect Your Agent](/guides/connect-your-agent) - The full integration map.