niceeval 0.4.6 → 0.5.0

package/docs-site/reference/define-agent.mdx

@@ -1,168 +0,0 @@
----
-title: "defineAgent and defineSandboxAgent: adapter reference"
-sidebarTitle: "defineAgent"
-description: "Reference for defineAgent and defineSandboxAgent: AgentContext, AgentSession, the Sandbox interface, and shared sandbox helpers."
----
-
-Every NiceEval agent is an adapter: code that knows how to drive a specific backend and translate its output into the standard event stream. The runner only calls `agent.send(input, ctx)`.
-
-![Full round trip of one t.send: the eval calls t.send, the runner assembles TurnInput and ctx, the adapter returns a Turn with standard events.](/images/agent-turn-roundtrip-en.svg)
-
-## `defineAgent`
-
-Use this for direct agent integrations:
-
-```ts
-import { defineAgent } from "niceeval/adapter";
-```
-
-<ParamField body="name" type="string" required>
-  Unique agent name. Experiments reference the agent object, and reports group results by this name.
-</ParamField>
-
-<ParamField body="tracing" type="AgentTracing">
-  OTLP export configuration, used by CLI / child-process agents: declares how to hand the receiver endpoint to the process under test (`env` / `configure`). Long-running HTTP apps do not need this field — configure a fixed port with `telemetry: { port }` in `niceeval.config.ts` instead (see [OTel integration](/guides/connect-otel)). Spans only feed the `niceeval view` waterfall, never assertions.
-</ParamField>
-
-<ParamField body="spanMapper" type="(spans: TraceSpan[]) => TraceSpan[]">
-  Thin mapper from native spans to canonical GenAI semconv; it only affects semantic coloring in the waterfall. Omit it to fall back to the generic heuristic. For what maps to what in the waterfall and how to write your own (`tagSpan` / `heuristicTag`), see [OTel integration](/guides/connect-otel).
-</ParamField>
-
-<ParamField body="send" type="(input: TurnInput, ctx: AgentContext) => Promise<Turn>" required>
-  Core function that drives the agent. Must return a standard `Turn`.
-</ParamField>
-
-`Agent.kind` produced by `defineAgent` is always `"remote"` (an internal discriminant field you never declare). There are no capability bits on `t` that need declaring to unlock — filesystem assertions like `t.sandbox` exist only on agents constructed with `defineSandboxAgent` (`kind: "sandbox"`); everything else is judged by which events `send` actually returned and whether `ctx.session` was used. See the [capabilities reference](/reference/capabilities).
-
-## `input` (`TurnInput`)
-
-<ResponseField name="text" type="string">
-  Text passed by the current `t.send(...)`.
-</ResponseField>
-
-<ResponseField name="files" type="readonly InputFile[] | undefined">
-  Files attached to this turn (images, etc.). Adapters that do not support multimodal input can ignore it.
-</ResponseField>
-
-<ResponseField name="responses" type="readonly InputResponse[] | undefined">
-  Present only on answer turns (`t.respond` / `t.respondAll`): per-request structured answers, matched by `requestId`.
-</ResponseField>
-
-## `ctx` (`AgentContext`)
-
-<ResponseField name="signal" type="AbortSignal">
-  For timeouts and cancellation; pass it through to every request you make.
-</ResponseField>
-
-<ResponseField name="model" type="string | undefined">
-  Model comparison for Tier 1: `experiment.model`, passed through; forward it if the app's interface accepts a model selection.
-</ResponseField>
-
-<ResponseField name="flags" type="Readonly<Record<string, unknown>>">
-  Feature A/B for Tier 3: `experiment.flags`, passed through; `{}` when not configured.
-</ResponseField>
-
-<ResponseField name="telemetry" type="Telemetry | undefined">
-  Present only when OTel integration is configured: this turn's W3C trace context (`headers`, a fresh `traceparent` per turn).
-</ResponseField>
-
-<ResponseField name="sandbox" type="Sandbox | undefined">
-  Only agents constructed with `defineSandboxAgent` receive this.
-</ResponseField>
-
-<ResponseField name="session" type="AgentSession">
-  State slot for the current conversation line. Every `send` on the same conversation line gets the same `ctx.session`; a new conversation line (the eval's first turn, or after `t.newSession()`) gets a fresh one:
-
-  - `id?: string` / `capture(id): void` — session continuation when the server keeps history: `id` is the session id recorded on this line (`undefined` on a new line), `capture` records the returned id (it only lands when none has been recorded yet).
-  - `history<TMsg>(): { get(): TMsg[]; commit(messages: TMsg[]): void }` — session continuation when the client carries the full history.
-  - `hold<T>(state: T): void` / `take<T>(): T | undefined` — storage for the HITL paused-turn scene; `take` clears on read (consume once).
-  - `state: Record<string, unknown>` — escape hatch, starts as `{}`, never written by the framework.
-
-  Full contract in the [Adapter concept](/concepts/adapter#context-agentcontext).
-</ResponseField>
-
-<ResponseField name="log" type="(msg: string) => void">
-  Writes a message to the run log, visible in `niceeval view`.
-</ResponseField>
-
-## Direct agent example
-
-```ts
-export default defineAgent({
-  name: "echo",
-  async send(input) {
-    return {
-      status: "completed",
-      events: [{ type: "message", role: "assistant", text: input.text }],
-    };
-  },
-});
-```
-
-## `defineSandboxAgent`
-
-Use this for coding agent CLIs. The produced `Agent.kind` is always `"sandbox"` — filesystem assertions like `t.sandbox` and `t.sandbox.fileChanged()` unlock only on this kind of agent. `ctx.sandbox` is the handle to the current isolated environment:
-
-<ResponseField name="runCommand(cmd, args?, opts?)" type="(cmd: string, args?: string[], opts?: CommandOptions) => Promise<CommandResult>">
-  Run a command.
-</ResponseField>
-
-<ResponseField name="runShell(script, opts?)" type="(script: string, opts?: CommandOptions) => Promise<CommandResult>">
-  Run a shell script.
-</ResponseField>
-
-<ResponseField name="readFile(path)" type="(path: string) => Promise<string>">
-  Read a file inside the sandbox.
-</ResponseField>
-
-<ResponseField name="fileExists(path)" type="(path: string) => Promise<boolean>">
-  Check whether a file exists inside the sandbox.
-</ResponseField>
-
-<ResponseField name="writeFiles(files, targetDir?)" type="(files: Record<string, string>, targetDir?: string) => Promise<void>">
-  Write file contents by path.
-</ResponseField>
-
-<ResponseField name="uploadFiles(files, targetDir?)" type="(files: SandboxFile[], targetDir?: string) => Promise<void>">
-  Upload local files into the sandbox.
-</ResponseField>
-
-<ResponseField name="uploadDirectory(localDir, targetDir?, opts?)" type="(localDir: string, targetDir?: string, opts?: { ignore?: string[] }) => Promise<void>">
-  Upload an entire local directory, typically used to write the eval's starting files.
-</ResponseField>
-
-<ResponseField name="sandboxId" type="string">
-  Id of the current sandbox instance.
-</ResponseField>
-
-<ResponseField name="stop()" type="() => Promise<void>">
-  Stop the sandbox.
-</ResponseField>
-
-```ts
-import { defineSandboxAgent } from "niceeval/adapter";
-
-export default defineSandboxAgent({
-  name: "my-cli-agent",
-  async send(input, ctx) {
-    await ctx.sandbox.runCommand("my-agent", ["run", input.text]);
-    return { status: "completed", events: [] };
-  },
-});
-```
-
-## Registration
-
-```ts
-import { defineExperiment } from "niceeval";
-import echo from "./agents/echo";
-
-export default defineExperiment({
-  agent: echo,
-  runs: 1,
-});
-```
-
-## `ctx` and `t`
-
-`ctx` is the run context seen by the adapter side; `t` is the test context seen by the eval author. They use the same run data but have different responsibilities.

package/docs-site/reference/define-config.mdx

@@ -1,41 +0,0 @@
----
-title: "defineConfig: project defaults"
-sidebarTitle: "defineConfig"
-description: "defineConfig reference: judge, reporters, concurrency, timeout, and sandbox defaults."
----
-
-Use `defineConfig` in `niceeval.config.ts` for project-wide defaults. Agent, model, flags, runs, and per-experiment budget belong in `experiments/` files via `defineExperiment`.
-
-```ts
-import { defineConfig } from "niceeval";
-import { JUnit } from "niceeval/reporters";
-
-export default defineConfig({
-  judge: { model: "gpt-5.4-mini" },
-  reporters: [JUnit(".niceeval/junit.xml")],
-  maxConcurrency: 4,
-  timeoutMs: 300_000,
-});
-```
-
-<ParamField body="judge" type="{ model: string; baseUrl?: string; apiKeyEnv?: string }">
-  Default judge configuration for `t.judge.autoevals.*`. Individual evals can override this with `defineEval({ judge })`.
-</ParamField>
-
-<ParamField body="reporters" type="Reporter[]">
-  Additional reporters that observe the whole run, such as `Braintrust(...)` for uploading to an experiment platform. See [Reporters](../guides/reporters). For temporary JUnit output, use the CLI flag `--junit <path>`.
-</ParamField>
-
-<ParamField body="maxConcurrency" type="number">
-  Maximum concurrent attempts. Sandbox backends may choose a lower default when this is omitted.
-</ParamField>
-
-<ParamField body="timeoutMs" type="number">
-  Per-attempt timeout in milliseconds.
-</ParamField>
-
-<ParamField body="sandbox" type="SandboxSpec">
-  Default sandbox backend for sandbox agents, built with `dockerSandbox()` / `vercelSandbox()` / `e2bSandbox()` from `niceeval/sandbox` — bare strings like `"docker"` are no longer accepted here. Required for sandbox agents (directly on the experiment, or via this project-wide fallback: `exp.sandbox ?? config.sandbox`) — niceeval no longer auto-detects a backend from env vars, and there is no CLI flag for choosing one.
-</ParamField>
-
-niceeval keeps environment preparation in ordinary code. Prepare files inside `test(t)` with `t.sandbox.writeFiles`, `uploadFiles`, or `uploadDirectory`; put adapter setup in the Adapter.

package/docs-site/reference/define-eval.mdx

@@ -1,160 +0,0 @@
----
-title: "defineEval: declare, configure, and run NiceEval evals"
-sidebarTitle: "defineEval"
-description: "Reference for defineEval options, the test context t, Turn return values, sandbox helpers, and dataset array exports."
----
-
-`defineEval` is the main entry point for authoring evals. Each eval file normally calls it once and default-exports the result.
-
-```ts
-import { defineEval } from "niceeval";
-
-export default defineEval({
-  description: "Brooklyn weather query",
-  async test(t) {
-    await t.send("What's the weather like in Brooklyn today?");
-    t.succeeded();
-  },
-});
-```
-
-<Note>
-  Do not provide `id` or `name`. NiceEval derives the eval ID from the file path.
-</Note>
-
-## `defineEval` options
-
-<ParamField body="description" type="string">
-  Human-readable description shown in console output and reports.
-</ParamField>
-
-<ParamField body="agent" type="string">
-  Agent name for this eval. Usually omitted so the experiment supplies the agent.
-</ParamField>
-
-<ParamField body="tags" type="string[]">
-  Tags for filtering or organizing evals.
-</ParamField>
-
-<ParamField body="judge" type="JudgeConfig">
-  Judge model override for this eval.
-</ParamField>
-
-<ParamField body="reporters" type="Reporter[]">
-  Eval-specific reporters. The instance only observes evals that reference it. If multiple evals share the same instance, their results land in the same destination. See [Reporters](../guides/reporters).
-</ParamField>
-
-<ParamField body="timeoutMs" type="number">
-  Timeout override.
-</ParamField>
-
-<ParamField body="metadata" type="Record<string, unknown>">
-  Extra metadata for reports or external tools.
-</ParamField>
-
-<ParamField body="test" type="(t: TestContext) => Promise<void>" required>
-  Eval body. Drive the agent and declare assertions here.
-</ParamField>
-
-## Test context: `t`
-
-### Always available
-
-<ParamField body="t.send(text)" type="(text: string, opts?) => Promise<Turn>">
-  Sends a message to the agent and returns the current turn.
-</ParamField>
-
-<ParamField body="t.sendFile(path, text?, opts?)" type="function">
-  Sends a file plus optional text.
-</ParamField>
-
-<ParamField body="t.check(value, assertion)" type="(value: unknown, assertion: Assertion) => void">
-  Records a value assertion. Failure does not immediately abort the test.
-</ParamField>
-
-<ParamField body="t.require(value, assertion)" type="(value: unknown, assertion: Assertion) => void">
-  Records a prerequisite assertion. Failure aborts the test immediately.
-</ParamField>
-
-<ParamField body="t.skip(reason)" type="(reason: string) => void">
-  Skips the eval.
-</ParamField>
-
-### Conversation
-
-<ParamField body="t.reply" type="string">
-  Last assistant text message in the current session.
-</ParamField>
-
-<ParamField body="t.newSession()" type="() => TestContext">
-  Starts an independent session line.
-</ParamField>
-
-### Tool observability
-
-<ParamField body="t.calledTool(name, opts?)" type="function">
-  Asserts that a tool was called.
-</ParamField>
-
-<ParamField body="t.notCalledTool(name, opts?)" type="function">
-  Asserts that a tool was not called. This is only trustworthy when the event stream is complete.
-</ParamField>
-
-<ParamField body="t.event(type, opts?)" type="function">
-  Low-level event stream assertion.
-</ParamField>
-
-### Workspace / sandbox
-
-<ParamField body="t.sandbox.fileChanged(path)" type="function">
-  Asserts that a workspace file changed.
-</ParamField>
-
-<ParamField body="t.sandbox.runCommand(command, args?)" type="function">
-  Runs a verification command in the sandbox.
-</ParamField>
-
-## Judge assertions
-
-```ts
-t.judge.autoevals.factuality(expected, { on: t.reply }).atLeast(0.8);
-t.judge.autoevals.closedQA(question, { on: t.reply }).atLeast(0.7);
-t.judge.autoevals.closedQA(rubric, { on: t.reply }).atLeast(0.75);
-```
-
-## `Turn`
-
-<ResponseField name="events" type="StreamEvent[]">
-  Standard event stream for this turn.
-</ResponseField>
-
-<ResponseField name="data" type="unknown | undefined">
-  Structured output, present only when declared by the send call.
-</ResponseField>
-
-<ResponseField name="status" type='"completed" | "failed" | "waiting"'>
-  Current turn status.
-</ResponseField>
-
-<ResponseField name="message" type="string">
-  Assistant text for this turn.
-</ResponseField>
-
-<ResponseField name="toolCalls" type="ToolCall[]">
-  Tool calls for this turn.
-</ResponseField>
-
-## Dataset export
-
-```ts
-export default rows.map((row) =>
-  defineEval({
-    description: row.task,
-    async test(t) {
-      await t.send(row.prompt);
-    },
-  }),
-);
-```
-
-Array exports generate stable IDs such as `file/0000`, `file/0001`, and so on.

package/docs-site/reference/events.mdx

@@ -1,131 +0,0 @@
----
-title: "Standard event stream reference"
-sidebarTitle: "Events"
-description: "The nine StreamEvent variants: what each one means, when adapters emit it, and which assertions consume it. Producing this stream is the adapter's core job."
----
-
-An adapter `send` returns a `Turn`, and its `events: StreamEvent[]` is the **only data source for assertions**: `t.calledTool()`, `t.reply`, `toolOrder`, `noFailedActions`, and more all read from this stream. Once your adapter translates "what the agent did this turn" into this stream, the whole assertion surface becomes available.
-
-## Turn: the return value of `send`
-
-```ts
-interface Turn {
-  events: StreamEvent[];                           // events for this turn, in real order
-  data?: unknown;                                 // structured output -> outputEquals / outputMatches
-  status: "completed" | "failed" | "waiting";     // waiting = paused for human input
-  usage?: { inputTokens: number; outputTokens: number }; // -> maxTokens / maxCost / cost reports
-}
-```
-
-`data` means "the structured product of this turn". Only fill it when the application answer itself is structured, such as extraction, classification, or form filling. `outputEquals` / `outputMatches` read from it. If the app only returns text, leave it unset. Do not copy raw response bodies or message text into `data`, and do not serialize structured output back into `events`. Include `usage` only when you have it. Do not invent numbers.
-
-## Event table
-
-| Event | Meaning | Consumed by |
-|---|---|---|
-| `message` | the agent (or user) said something | `t.reply`, `messageIncludes`, judge inputs |
-| `action.called` | a tool call started | `calledTool`, `toolOrder`, `maxToolCalls`, `notCalledTool` |
-| `action.result` | the result of that call | `calledTool` status matching, `noFailedActions` |
-| `subagent.called` | delegated to a subagent | `calledSubagent` |
-| `subagent.completed` | the subagent returned | `calledSubagent` status, `noFailedActions` |
-| `input.requested` | paused and waiting for human input | `t.parked()`, `t.requireInputRequest()` |
-| `thinking` | reasoning text | `event("thinking")`, viewer display |
-| `compaction` | context got compacted | `event("compaction")` |
-| `error` | the turn hit an error | `event("error")`, viewer display |
-
-Any event can also be checked with `event(type)`, `notEvent(type)`, `eventOrder(types)`, or `eventsSatisfy(label, predicate)`.
-
-## Event details
-
-### `message` - what was said
-
-```ts
-{ type: "message", role: "assistant" | "user", text: string }
-```
-
-Emit one `message` with `role: "assistant"` for each chunk of assistant text. **Tool results are not assistant messages.** Do not wrap tool output as `message`, or `t.reply` will read the wrong thing. User `message` events are usually recorded by NiceEval automatically.
-
-### `action.called` / `action.result` - which tool was used, and what happened
-
-```ts
-{ type: "action.called", callId: string, name: string, input: JsonValue }
-{ type: "action.result", callId: string, output?: JsonValue,
-  status: "completed" | "failed" | "rejected" }
-```
-
-- Pair each `action.called` with an `action.result` that has the same `callId`, so concurrent calls do not get mixed up. Use the explicit ID from your source when it exists.
-- Fill `status` truthfully: actual tool execution failure is `"failed"`. Human denial is `"rejected"`.
-- Keep `name` as the tool's original name.
-
-### `subagent.called` / `subagent.completed` - which subagent was delegated to
-
-```ts
-{ type: "subagent.called", callId: string, name: string, remoteUrl?: string }
-{ type: "subagent.completed", callId: string, output?: JsonValue,
-  status: "completed" | "failed" }
-```
-
-Emit this pair when the subject under test delegates work to a subagent and waits for it to return.
-
-### `input.requested` - paused for human input
-
-```ts
-{ type: "input.requested", request: {
-    id?: string,
-    action?: string,
-    input?: JsonValue,
-    prompt?: string,
-    options?: { id: string, label?: string }[],
-} }
-```
-
-When the agent pauses for a person, emit one event per pending question and return overall turn `status: "waiting"`. `t.requireInputRequest(filter)` matches this `request` structure field by field, so populate as many fields as you can. See the HITL part of [Connect Your Agent](/guides/connect-your-agent).
-
-### `thinking` / `compaction` / `error`
-
-```ts
-{ type: "thinking", text: string }
-{ type: "compaction", reason?: string }
-{ type: "error", message: string }
-```
-
-Emit them when your source gives them to you. Do not fabricate them. `compaction` mostly appears in coding-agent CLIs when context is compressed automatically.
-
-## Three mapping rules
-
-1. **Order is fact.** Keep events in real execution order. `toolOrder` and `eventOrder` depend on that sequence.
-2. **Match `callId`s.** Every `called` event should have the corresponding `result` with the same ID.
-3. **Be honest about completeness.** If you declare `toolObservability: true`, all tool calls must be in the stream. If you omit some of them, negative assertions can silently pass. If your manual mapping cannot be complete, [do not claim that capability](/reference/capabilities).
-
-## A complete mapping example
-
-When the agent response contains step records, the mapping is usually a small loop:
-
-```ts
-import type { StreamEvent } from "niceeval";
-
-function toStreamEvents(body: MyBotResponse): StreamEvent[] {
-  const events: StreamEvent[] = [];
-  for (const step of body.steps) {
-    if (step.type === "tool_call") {
-      events.push({ type: "action.called", callId: step.id, name: step.tool, input: step.args });
-      events.push({
-        type: "action.result",
-        callId: step.id,
-        output: step.result,
-        status: step.error ? "failed" : "completed",
-      });
-    }
-    if (step.type === "text") {
-      events.push({ type: "message", role: "assistant", text: step.text });
-    }
-  }
-  return events;
-}
-```
-
-## Related reading
-
-- [Connect Your Agent](/guides/connect-your-agent) - The from-scratch integration tutorial.
-- [Capabilities](/reference/capabilities) - What it means to claim "the event stream is complete".
-- [Authoring Evals](/guides/authoring) - The assertions that consume this stream.

package/docs-site/reference/expect.mdx

@@ -1,112 +0,0 @@
----
-title: "niceeval/expect matchers and custom assertion reference"
-sidebarTitle: "Assertion Matchers"
-description: "Reference for niceeval/expect: includes, equals, matches, similarity, satisfies. Chain `.gate()` or `.atLeast(0.7)`, or build custom matchers with `makeAssertion`."
----
-
-`niceeval/expect` provides a set of composable matchers for `t.check()` and `t.require()`. A matcher returns an `Assertion` with a default severity, either `gate` or `soft`.
-
-```ts
-import { includes, equals, matches, similarity, satisfies } from "niceeval/expect";
-```
-
-## How matchers are used
-
-```ts
-t.check(t.reply, includes("confirmed"));
-t.check(turn.data, equals({ intent: "refund" }));
-t.require(turn.status, equals("completed"));
-```
-
-| Method | On failure | Best for |
-|---|---|---|
-| `t.check` | record result, keep executing | most assertions |
-| `t.require` | throw immediately, abort the test | preconditions where continuing makes no sense |
-
-## Matchers
-
-### `includes`
-
-```ts
-includes(substring: string | RegExp): Assertion
-```
-
-Assert that a string contains the given substring, or matches a regex.
-
-```ts
-t.check(t.reply, includes("Paris"));
-t.check(t.reply, includes(/order #\d+/i));
-```
-
-### `equals`
-
-```ts
-equals(expected: unknown): Assertion
-```
-
-Assert deep structural equality, useful for JSON, arrays, and objects.
-
-```ts
-t.check(turn.data, equals({ intent: "refund" }));
-t.check(turn.data, equals(["a", "b", "c"]));
-```
-
-### `matches`
-
-```ts
-matches(pattern: RegExp): Assertion
-```
-
-Assert that a string matches a regex.
-
-```ts
- t.check(t.reply, matches(/#[A-Z0-9]{8}/));
-```
-
-### `similarity`
-
-```ts
-similarity(expected: string): Assertion
-```
-
-Use this for semantic or textual similarity checks, usually together with a threshold.
-
-```ts
-t.check(t.reply, similarity("The capital of France is Paris.").atLeast(0.8));
-```
-
-### `satisfies`
-
-```ts
-satisfies(predicate: (value: unknown) => boolean): Assertion
-```
-
-Check a value with a custom predicate.
-
-```ts
-t.check(turn.data, satisfies((value) => Array.isArray(value)));
-```
-
-## Gate and soft
-
-```ts
-t.check(t.reply, includes("required").gate());
-t.check(t.reply, includes("nice to have").atLeast(0.7));
-```
-
-- `gate()`: failure makes the eval fail.
-- `soft()`: contributes to scoring without necessarily failing the eval.
-
-## Custom matchers
-
-```ts
-import { makeAssertion } from "niceeval/expect";
-
-const validEmail = makeAssertion("valid email", (value) => {
-  return typeof value === "string" && value.includes("@") ? 1 : 0;
-});
-
-t.check(t.reply, validEmail());
-```
-
-Custom matchers should return a score between 0 and 1 and use a clear name.

package/docs-site/tracker.js

@@ -1,8 +0,0 @@
-(function () {
-  var script = document.createElement("script");
-  script.src = "https://api.goshipfast.com/tracker.js";
-  script.async = true;
-  script.setAttribute("data-project", "cmr24oe2n006qlj10lr9t38n8");
-  script.setAttribute("data-endpoint", "https://api.goshipfast.com");
-  document.head.appendChild(script);
-})();

package/src/report/react/data.ts

@@ -1,17 +0,0 @@
-// niceeval/report/react 的数据契约(组件 props 类型)。
-//
-// 契约的家在计算层(../types.ts,照 docs/reports.md「计算函数与数据契约」),这里 re-export,
-// 「算」与「画」两侧共用同一份类型;本文件零运行时代码——组件不做任何聚合或数据操作。
-
-export type {
-  AttemptRef,
-  MetricColumn,
-  MetricCell,
-  TableData,
-  MatrixData,
-  ScoreboardData,
-  ScatterData,
-  OverviewData,
-  DeltaData,
-  CaseListData,
-} from "../types.ts";