niceeval 0.4.5 → 0.5.0

package/docs-site/guides/connect-your-agent.mdx

@@ -1,221 +0,0 @@
----
-title: "Connect your agent"
-sidebarTitle: "Connect Agent"
-description: "The full picture of integration: write an adapter, configure an experiment, and get the first eval running; how configuration and parameters are declared in the experiment and consumed by the Adapter. Event streams, multi-turn, HITL, and tracing are increments, each with its own page."
----
-
-Connecting a subject under test to NiceEval means writing one **adapter** for it: `defineAgent` wrapping a `send` function — it receives one turn of input, drives your agent, and returns that turn's result. The entire integration is this one file.
-
-This page answers only two questions: **how this file and its two companions (experiment, eval) get running end to end**, and **how parameters flow from the experiment to the Adapter and on to your app**. Event streams, multi-turn, HITL, and tracing are all optional increments that come later, each with its own page; a map is at the end.
-
-## Pick a path first: what is your subject under test?
-
-<CardGroup cols={3}>
-  <Card title="AI SDK app" icon="bolt" href="/reference/builtin-agents">
-    Apps built with the Vercel AI SDK (`useChat` backend): the built-in `uiMessageStreamAgent` connects to their HTTP interface non-intrusively — zero mapping, HITL included, with a complete before/after example. You do not need this page.
-  </Card>
-  <Card title="Coding agent CLI" icon="terminal" href="/guides/sandbox-agent">
-    Evaluating claude-code / codex / bub and other file-editing coding agents: use the built-in sandbox agent.
-  </Card>
-  <Card title="Other AI Agent" icon="plug">
-    Custom agent loops, LangGraph / OpenAI Agents SDK apps, deployed agents: hand-write `send` (this page + [Write Send](/guides/write-send)). If the app already emits OTel, hook it up while you are at it — `niceeval view` gains a call waterfall ([OTel integration](/guides/connect-otel)).
-  </Card>
-</CardGroup>
-
-## Get the minimal integration running
-
-Assume the project has already run `npx niceeval init` (so `niceeval.config.ts` and the `evals/` directory exist). Three files, one job each: **the adapter connects to the system under test, the experiment decides who to evaluate and how to run, the eval writes the interaction and assertions.**
-
-**1. Write the adapter.** The minimal integration fills only `status` and `events`: whatever your agent said goes into one `message` event.
-
-```ts
-// agents/my-bot.ts
-import { defineAgent } from "niceeval/adapter";
-
-export default defineAgent({
-  name: "my-bot",
-  async send(input, ctx) {
-    const r = await fetch("http://localhost:3000/chat", {
-      method: "POST",
-      headers: { "content-type": "application/json" },
-      body: JSON.stringify({ message: input.text }),
-      signal: ctx.signal,
-    });
-    const body = await r.json();
-    return {
-      status: r.ok ? "completed" : "failed",
-      events: [{ type: "message", role: "assistant", text: body.reply }],
-    };
-  },
-});
-```
-
-Hardcoding the URL is fine for now — how to pass it in from the experiment is covered below in ["How parameters flow in"](#how-parameters-flow-in-the-experiment-declares-the-adapter-consumes). The full `send` contract (every field of `TurnInput` / `AgentContext` / `Turn`) is in [Adapter](/concepts/adapter).
-
-Even when the agent runtime and the evals live in the same codebase, still go through HTTP — do not replace the `fetch` with an in-process function call:
-
-- **A direct call does not evaluate the path your users take.** The HTTP layer, serialization, middleware, and streaming are all bypassed; a passing eval does not mean production behavior is correct.
-- **No process isolation, so results are not reproducible.** Crashes, global state, and memory leaks in the code under test drag the runner down with them; concurrent evals pollute each other through shared state.
-- **Timeouts and cancellation stop working.** `ctx.signal` can only actually kill a runaway turn when it is attached to a request; an in-process function call cannot be stopped.
-- **The adapter gets welded to this codebase.** An HTTP adapter runs against local / staging / production by swapping `baseUrl` (see the two experiment files below); a direct call cannot.
-
-Starting a local dev server costs one command; none of the above is worth trading for it.
-
-**2. Register it in an experiment:**
-
-```ts
-// experiments/my-bot.ts
-import { defineExperiment } from "niceeval";
-import myBot from "../agents/my-bot.ts";
-
-export default defineExperiment({
-  description: "my-bot baseline",
-  agent: myBot,
-  runs: 1,
-});
-```
-
-**3. Write the first eval, then run:**
-
-```ts
-// evals/refund-policy.eval.ts
-import { defineEval } from "niceeval";
-import { includes } from "niceeval/expect";
-
-export default defineEval({
-  description: "Refund policy Q&A",
-  async test(t) {
-    await t.send("What is your refund policy?");
-    t.succeeded();
-    t.check(t.reply, includes("30 days"));
-    t.judge.autoevals.closedQA("Does the answer explain the refund window?").atLeast(0.7);
-  },
-});
-```
-
-```bash
-npx niceeval exp my-bot        # run all evals under this experiment
-npx niceeval exp my-bot refund # only those whose ID starts with refund
-npx niceeval view              # inspect results in the local viewer
-```
-
-**What success looks like**: one verdict line per eval in the terminal, ending with a `N passed, N failed` summary; `npx niceeval view` shows each eval's per-turn inputs, events, and scoring detail.
-
-If it does not run, triage by where the error shows up, in three buckets:
-
-- **`fetch` throws outright** (connection refused, etc.): the app is not up, or the URL in `send` is wrong — first send the same request to that endpoint with `curl` to confirm.
-- **`t.succeeded()` fails and the turn verdict is failed**: the request went out but the app returned non-2xx. Print the response body with `ctx.log()` inside `send`; it shows up in this turn's log in `niceeval view`.
-- **Only content assertions fail**: the integration itself already works — compare the actual value of `t.reply` in `view`, then adjust the assertion or the app.
-
-At this point the integration is complete: text assertions and judge scoring both work. For more assertions (tools, multi-turn, approval flows), see the increment map at the end.
-
-## How parameters flow in: the experiment declares, the Adapter consumes
-
-Configuration has exactly two channels; keep them apart and the integration stays untangled:
-
-1. **Static configuration goes through the Adapter factory.** URL, auth, protocol details — the "what does this environment look like" kind of configuration — are written as factory options in the experiment file: the `agent` field of `defineExperiment` holds an **already-configured instance**.
-2. **Per-turn dynamic values go through `ctx`.** The `model` and `flags` declared by the experiment are handed to `send` verbatim via `ctx` every turn; the Adapter does not interpret their meaning, it only forwards them to the app with the request.
-
-Putting the two channels together means turning step 1's my-bot from "an instance with a hardcoded URL" into "a factory that receives configuration": the default export becomes a function returning `defineAgent(...)`, the hardcoded spots in `send` become reads of factory options; the `model` and `flags` declared by the experiment arrive via `ctx` every turn and are forwarded inside `send` with the request:
-
-```ts
-// agents/my-bot.ts — factory: static configuration goes into the closure
-import { defineAgent } from "niceeval/adapter";
-import type { Agent } from "niceeval/adapter";
-
-export function myBot(options: { baseUrl: string }): Agent {
-  return defineAgent({
-    name: "my-bot",
-    async send(input, ctx) {
-      const r = await fetch(`${options.baseUrl}/chat`, {   // ← factory option: where to connect
-        method: "POST",
-        headers: { "content-type": "application/json" },
-        body: JSON.stringify({
-          message: input.text,
-          model: ctx.model,     // ← experiment.model: forward it if the app's interface accepts a model choice
-          flags: ctx.flags,     // ← experiment.flags: hand to the app as-is; the app switches on flags
-        }),
-        signal: ctx.signal,     // ← runner timeout and cancellation: attach to every request
-      });
-      const body = await r.json();
-      return {
-        status: r.ok ? "completed" : "failed",
-        events: [{ type: "message", role: "assistant", text: body.reply }],
-      };
-    },
-  });
-}
-```
-
-Auth headers, protocol switches, and the like are static configuration too — add them to `options` the same way. The experiment side changes in two small places: a named import of the factory, and the `agent` field goes from "referencing an instance" to "calling the factory":
-
-```ts
-// experiments/my-bot.ts — where parameters are declared
-import { defineExperiment } from "niceeval";
-import { myBot } from "../agents/my-bot.ts";
-
-export default defineExperiment({
-  description: "my-bot baseline",
-  agent: myBot({ baseUrl: "http://localhost:3000" }),  // ← static configuration: passed at instantiation
-  model: "gpt-5.4",                                    // ← dynamic value: reaches send via ctx.model
-  flags: { promptVariant: "concise" },                 // ← dynamic value: reaches send via ctx.flags
-});
-```
-
-The `ctx` fields a turn may use, and how to consume them:
-
-| `ctx` field | Who provides it | How the Adapter consumes it |
-|---|---|---|
-| `signal` | The runner (timeout and cancellation) | Attach it to every outgoing request |
-| `model` | The experiment's `model` | Forward it with the request if the app's interface accepts a model choice; ignore it otherwise |
-| `flags` | The experiment's `flags` | Forward as-is (request body or header both work); the app switches variants on flags |
-| `telemetry` | Present when OTel integration is configured | Touch only `headers`: a fresh W3C `traceparent` per turn, spread into the request headers. The receiver endpoint is the same on every run — pin it in `defineConfig` and point the app at it on startup; it is not passed from send — see [OTel integration](/guides/connect-otel) |
-| `session` | The runner (one per session line) | The accessors for session continuation and HITL held state all live on it: `history()`, `id` / `capture()`, `hold()` / `take()` — see [Write Send](/guides/write-send) |
-| `log(msg)` | The runner | Written into this turn's log, visible in `niceeval view` |
-
-Running the same agent against local and production separately is just two experiment files with two sets of factory options:
-
-```ts
-// experiments/local.ts
-export default defineExperiment({
-  agent: myBot({ baseUrl: "http://localhost:3000" }),
-});
-
-// experiments/prod.ts
-export default defineExperiment({
-  agent: myBot({ baseUrl: "https://api.example.com" }),
-});
-```
-
-```bash
-npx niceeval exp local
-npx niceeval exp prod
-```
-
-Do not put URLs into CLI positional arguments — the positional arguments after the experiment name only filter eval IDs. For the full experiment fields (`runs`, `budget`, concurrency, `sandbox`), see [Write experiments](/guides/write-experiment).
-
-## The increments after: a map
-
-After the minimal integration, every increment you give the adapter unlocks a family of assertions, and the evals you have already written stay untouched:
-
-| What you want to unlock | What to add to the adapter | Where to look |
-|---|---|---|
-| Tool assertions (`calledTool` / `toolOrder` / negative assertions) | Map the app's response into the standard event stream | [Write Send](/guides/write-send), [Events reference](/reference/events) |
-| Multi-turn conversations, `t.newSession()` isolation | Connect `ctx.session`: `history()` or `id` + `capture()` | [Write Send](/guides/write-send) |
-| Approval flows (HITL, human-in-the-loop) | Pause the turn with `waiting` + `input.requested`, resume on the answer turn | [HITL](/concepts/hitl) |
-| The call waterfall in `niceeval view` | The app sends OTel spans to NiceEval (does not affect assertions) | [OTel integration](/guides/connect-otel) |
-| Feature A/B comparison | The app exposes variants as configuration switchable by `flags` | [Tier](/concepts/tier), [Write experiments](/guides/write-experiment) |
-
-Which tier each increment lands in, and what each tier buys, is in [Tier](/concepts/tier).
-
-## Reference implementations
-
-Complete runnable references: the five non-intrusive integration examples under [`examples/zh/tier1`](https://github.com/CorrectRoadH/niceeval/tree/main/examples/zh/tier1) (ai-sdk-v7, claude-sdk, codex-sdk, pi-sdk, langgraph), covering event-stream assertions, multi-turn isolation, HITL approve / reject, and the trace waterfall. When hand-writing `send`, all you really write is the transport and the mapping table — the state slots for session continuation and HITL pause/resume are on `ctx.session`, and frame-by-frame driving is official machinery; see [Built-in agent capabilities](/reference/builtin-agents).
-
-## Related reading
-
-- [Write Send](/guides/write-send) — the complete tutorial for hand-writing an Adapter: seven progressive steps, from sending one message to HITL, OTel, and flags.
-- [Adapter](/concepts/adapter) — the `send` contract: `TurnInput` / `AgentContext` / `Turn`, field by field.
-- [OTel integration](/guides/connect-otel) — send the app's spans to NiceEval too, in exchange for the call waterfall in `niceeval view`.
-- [Tier](/concepts/tier) — what each of the three integration tiers costs and buys.
-- [Write experiments](/guides/write-experiment) — the full `defineExperiment` fields.

package/docs-site/guides/dataset-fanout.mdx

@@ -1,98 +0,0 @@
----
-title: "Dataset fan-out: run evals across many test cases"
-sidebarTitle: "Datasets"
-description: "Export an array from a .eval.ts file to fan out into many evals. Use loadYaml or loadJson to drive evals from external datasets with stable IDs."
----
-
-Dataset fan-out is a good fit when many test cases share the same structure and only the inputs change. Typical examples are SQL generation, intent classification, retrieval QA, and tool selection.
-
-## How fan-out works
-
-A `.eval.ts` file can export an array by default:
-
-```ts
-import { defineEval } from "niceeval";
-import { equals } from "niceeval/expect";
-
-const rows = [
-  { task: "Count users", prompt: "Count all users", sql: "SELECT COUNT(*) FROM users;" },
-  { task: "Recent orders", prompt: "Find recent orders", sql: "SELECT * FROM orders ORDER BY created_at DESC LIMIT 10;" },
-];
-
-export default rows.map((row) =>
-  defineEval({
-    description: row.task,
-    async test(t) {
-      await t.send(row.prompt);
-      t.check(t.reply, equals(row.sql));
-    },
-  }),
-);
-```
-
-## Generated ID format
-
-If the file is `evals/sql.eval.ts`, the generated IDs are:
-
-```text
-sql/0000
-sql/0001
-```
-
-The numeric suffix is zero-padded so IDs stay stable and easy to filter.
-
-## Loading from YAML and JSON
-
-```ts
-import { defineEval } from "niceeval";
-import { loadYaml } from "niceeval/loaders";
-import { equals } from "niceeval/expect";
-
-const doc = await loadYaml("evals/data/sql-cases.yaml");
-const rows = doc.cases as { task: string; prompt: string; sql: string }[];
-
-export default rows.map((row) =>
-  defineEval({
-    description: row.task,
-    async test(t) {
-      await t.send(row.prompt);
-      t.check(t.reply, equals(row.sql));
-    },
-  }),
-);
-```
-
-```yaml
-cases:
-  - task: Count users
-    prompt: Count all rows in the users table
-    sql: SELECT COUNT(*) FROM users;
-```
-
-## Filtering dataset evals
-
-Because each eval in a fan-out set shares the same file-level ID prefix, you can run them all with a single CLI argument:
-
-```bash
-# Run the whole dataset
-npx niceeval exp local sql
-
-# Run only the first case
-npx niceeval exp local sql/0000
-
-```
-
-## Datasets vs separate files
-
-<Tabs>
-  <Tab title="Use a dataset">
-    The cases have exactly the same structure. Only the input and expected output change.
-  </Tab>
-  <Tab title="Use separate files">
-    Each case has a different flow, different assertions, or needs a different agent configuration.
-  </Tab>
-</Tabs>
-
-<Tip>
-  Datasets are good for broad horizontal coverage. Separate eval files are better when the behavior itself is complex.
-</Tip>

package/docs-site/guides/experiments.mdx

@@ -1,71 +0,0 @@
----
-title: "Experiment matrix: compare agents and models with run matrices"
-sidebarTitle: "Experiments"
-description: "Use NiceEval experiments to run the same evals across multiple agents, models, and feature flags, then compare pass rate, cost, and latency."
----
-
-Experiments compare multiple run configurations. Typical questions include: which coding agent has the higher pass rate on the same task set, whether a prompt change lowers cost, and how latency trades off against quality across models.
-
-## Basic shape
-
-**One experiment file = one configuration** (`one agent x one model`). `model` is a single string, not an array. To compare across models or agents, put multiple files in the same **experiment group directory** and keep everything else fixed:
-
-```text
-experiments/
-  compare-models/
-    gpt-5.4.ts
-    deepseek-v4-pro.ts
-```
-
-```ts
-import { defineExperiment } from "niceeval";
-import { webAgent } from "../../adapter/adapter.ts";
-
-export default defineExperiment({
-  description: "gpt-5.4 comparison model",
-  agent: webAgent({ baseUrl: "http://127.0.0.1:5188" }),
-  model: "gpt-5.4",
-  runs: 2,
-  earlyExit: true,
-});
-```
-
-```bash
-npx niceeval exp compare-models
-```
-
-Each configuration stays in its own file, which makes naming, diffing, and review straightforward. The directory structure itself explains which files belong to the same comparison.
-
-See [Write Experiments](/guides/write-experiment) for the full `defineExperiment` field list and how `flags` flow into adapters.
-
-## What this is good for
-
-- Comparing different Adapters
-- Comparing different models. Tier 1 is enough as long as the application exposes model choice and the value is forwarded through `ctx.model`
-- Comparing prompts or feature flags. This usually needs Tier 3, because the application has to expose the variant as an experiment-selectable config and forward it through `flags` -> `ctx.flags`
-- Comparing different sandbox backends
-- Measuring pass@N for the same task
-
-See [Tier](/concepts/tier) for what Tier 1, Tier 2, and Tier 3 mean.
-
-## Reading the results
-
-Experiment output is typically shown per `(agent, model, eval)` cell:
-
-```text
-api-validation   claude-code+zod-skill   pass@3 = 3/3 (100%)   mean 34s
-api-validation   claude-code             pass@3 = 1/3 (33%)    mean 41s
-```
-
-Beyond pass rate, you should also compare mean time, tokens, cost, and failure types.
-
-## Design advice
-
-- Keep the eval set stable so the comparison does not mix in extra variables.
-- Run multiple attempts per cell, especially for non-deterministic coding agents.
-- Make the budget and concurrency explicit.
-- Group failure modes instead of looking only at the total score.
-
-## Relation to ordinary runs
-
-`npx niceeval exp <experiment>` answers "did this eval suite pass under this configuration?" An experiment group answers "which configuration is better?" Both use the same evals, adapters, scoring, and artifacts.

package/docs-site/guides/fixtures.mdx

@@ -1,147 +0,0 @@
----
-title: "Sandbox fixtures: evaluate coding agents with tasks"
-sidebarTitle: "Fixtures"
-description: "Use .eval.ts files to prepare isolated workspaces, send real coding-agent tasks, and verify results with files, commands, diffs, and judges."
----
-
-When you evaluate a coding agent, a chat assertion is not enough. You usually need to give it a real project, let it read and edit files, run commands, and then inspect the result. The current recommended niceeval shape is a normal `.eval.ts` file that explicitly prepares the sandbox workspace, sends the task, and records assertions.
-
-See [`examples/zh/coding-agent-skill`](https://github.com/CorrectRoadH/niceeval/tree/main/examples/zh/coding-agent-skill) for a runnable reference.
-
-## Recommended structure
-
-```text
-evals/
-├─ api-validation.eval.ts
-├─ config-schema.eval.ts
-└─ ponytail-safe-path.eval.ts
-
-workspaces/
-└─ ts-starter/
-   ├─ package.json
-   ├─ tsconfig.json
-   └─ src/
-
-skills/
-└─ zod.md
-
-experiments/
-├─ baseline.ts
-└─ with-skill.ts
-```
-
-`workspaces/` contains the starting project the agent can see. The `.eval.ts` file decides when to upload it, what prompt to send, and what to verify. `experiments/` decides which agent/model runs and whether a Skill or plugin is injected.
-
-## Prepare the workspace
-
-Upload the starting project inside the eval:
-
-```ts
-import { defineEval } from "niceeval";
-
-export default defineEval({
-  description: "Add request-body validation to an Express route",
-  async test(t) {
-    await t.sandbox.uploadDirectory("../workspaces/ts-starter");
-
-    await t.send("Implement POST /users in src/routes/users.ts.").then((turn) => turn.succeeded());
-
-    t.sandbox.fileChanged("src/routes/users.ts");
-  },
-});
-```
-
-Use `t.sandbox.writeFiles()` to add small seed files inside a specific eval, such as a Python probe or a focused TODO file.
-
-## Write the task prompt
-
-The prompt should read like a real work order. Describe the goal and constraints, but do not leak the verification answer.
-
-```ts
-await t
-  .send(
-    `Implement environment validation in src/config/env.ts.
-Requirements:
-- Define EnvSchema with Zod
-- Convert PORT to a number
-- DATABASE_URL must be a valid URL
-- JWT_SECRET must be at least 32 characters
-- Export the env object and Env type`,
-  )
-  .then((turn) => turn.succeeded());
-```
-
-Leave implicit requirements, such as path traversal defense or reuse of existing helpers, for the verification phase. That is how you measure whether a Skill or plugin helped the agent infer missing context.
-
-## Verify files and code
-
-```ts
-import { excludes, includes } from "niceeval/expect";
-
-const src = await t.sandbox.readSourceFiles({ extensions: ["ts"] });
-const config = src.fileMatching(/env/);
-const code = config?.content ?? "";
-
-t.check(code, includes(/z\.object\s*\(/));
-t.check(code, includes(/EnvSchema\.parse\s*\(\s*process\.env/));
-t.check(code, excludes(/process\.env\.\w+\s*\?\?/));
-t.sandbox.fileChanged("src/config/env.ts");
-```
-
-`t.sandbox.fileChanged()` is a scoped assertion. It is evaluated from the sandbox diff after the run.
-
-## Run project tests or probes
-
-Real commands are often more reliable than text matching for coding-agent tasks:
-
-```ts
-await t.sandbox.writeFiles({
-  "_test_traversal.py": [
-    "import sys, os",
-    "sys.path.insert(0, '.')",
-    "from uploads import safe_upload_path",
-    "base = '/var/uploads'",
-    "try:",
-    "    p = safe_upload_path(base, '../../etc/passwd')",
-    "    assert os.path.commonpath([base, os.path.abspath(p)]) == base",
-    "except (ValueError, PermissionError, AssertionError):",
-    "    pass",
-    "print('ok')",
-  ].join("\n"),
-});
-
-const result = await t.sandbox.runCommand("python3", ["_test_traversal.py"]);
-t.check(result.stdout.trim(), includes(/ok/));
-```
-
-You can also run the project's own test, lint, or build scripts:
-
-```ts
-const result = await t.sandbox.runCommand("npm", ["test"]);
-t.check(result.exitCode, includes(0));
-```
-
-## Use experiments for baselines
-
-Do not put agent names, plugin names, or URLs into CLI positionals. Experiments decide which agent to run and how; eval positionals only filter eval IDs.
-
-```bash
-pnpm exec niceeval exp compare
-pnpm exec niceeval exp compare api-validation
-```
-
-A typical A/B setup:
-
-- `experiments/baseline.ts`: plain agent.
-- `experiments/with-skill.ts`: same agent plus a setup hook that writes `CLAUDE.md`.
-- Same eval set, model, runs, and budget in both cells.
-
-## When to use sandbox fixtures
-
-- The agent must modify real files.
-- You need to run tests, builds, or lint.
-- You need to inspect diffs.
-- You are evaluating Claude Code, Codex, or another coding agent.
-- You are comparing whether a Skill, plugin, Hook, or MCP server improves real tasks.
-
-If the system under test is a direct agent application, a direct adapter is lighter.

package/docs-site/guides/reporters.mdx

@@ -1,113 +0,0 @@
----
-title: "Report Results to Braintrust and Other Destinations"
-sidebarTitle: "Reporters"
-description: "Use built-in reporters to send eval results to Braintrust experiments, JUnit XML, JSON files, or custom destinations."
----
-
-[NiceEval](https://niceeval.com/) runs and scores evals itself. Reporters send the results elsewhere. Console output and `.niceeval/` artifacts are built-in reporters and are always enabled. Other reporters are imported from `niceeval/reporters` and mounted as needed.
-
-There are two mount points:
-
-- `reporters` in `niceeval.config.ts`: observes every eval in the run. Shared destinations, such as one Braintrust experiment, usually belong here.
-- `reporters` on a single eval: the instance only observes that eval. If multiple evals reference the same instance, their results are merged into the same destination. Listing an instance on an eval again after it was already configured globally does not report it twice.
-
-## Braintrust
-
-`Braintrust(...)` uploads one NiceEval run as a Braintrust experiment, with one row per attempt. Put it in config to cover the whole run:
-
-```ts niceeval.config.ts
-import { defineConfig } from "niceeval";
-import { Braintrust } from "niceeval/reporters";
-
-export default defineConfig({
-  reporters: [Braintrust({ project: "weather-agent" })],
-});
-```
-
-Mount it on an eval when only some evals should report:
-
-```ts evals/forecast.eval.ts
-import { defineEval } from "niceeval";
-import { Braintrust } from "niceeval/reporters";
-
-export default defineEval({
-  reporters: [Braintrust({ project: "weather-agent" })],
-  async test(t) {
-    await t.send("Is Beijing good for cycling today?");
-    t.succeeded();
-  },
-});
-```
-
-Prerequisites: install the optional `braintrust` package (`npm install braintrust`; other features work without it) and set `BRAINTRUST_API_KEY`. Pass `apiKey` when the key must be provided explicitly in code. After the run, the terminal prints the experiment URL.
-
-### Reporting Shape
-
-- Each attempt becomes one Braintrust row. `runs: 3` creates three rows, distinguished by `attempt` in metadata.
-- Soft assertions are recorded as scores by name. Gate assertions are recorded under a `gate:` prefix, so gate regressions and soft score regressions appear in the same Braintrust experiment diff surface.
-- Metrics include start/end time, token usage, and estimated cost. Missing data is omitted rather than filled with zero.
-- Metadata includes `agent`, `model`, `experiment`, `flags`, `outcome`, and failed assertion details, so Braintrust filters can slice by those dimensions.
-
-### Options
-
-<ParamField body="project" type="string">
-  Braintrust project name. Defaults to `niceeval`.
-</ParamField>
-
-<ParamField body="projectId" type="string">
-  Braintrust project ID. Provide either `project` or `projectId`.
-</ParamField>
-
-<ParamField body="experiment" type="string">
-  Experiment name. If omitted, Braintrust names it automatically.
-</ParamField>
-
-<ParamField body="baseExperiment" type="string">
-  Existing experiment name to use as the diff base.
-</ParamField>
-
-<ParamField body="baseExperimentId" type="string">
-  Existing experiment ID to use as the diff base.
-</ParamField>
-
-<ParamField body="update" type="boolean">
-  When `true`, updates an existing experiment with the same name instead of creating a new one.
-</ParamField>
-
-<ParamField body="metadata" type="Record<string, unknown>">
-  Experiment-level metadata. Merged with NiceEval's automatic fields; this value wins on name conflicts.
-</ParamField>
-
-<ParamField body="apiKey" type="string">
-  Braintrust API key. Defaults to the SDK reading `BRAINTRUST_API_KEY`.
-</ParamField>
-
-## JUnit and JSON
-
-`JUnit(path)` writes JUnit XML so failures appear in CI test report UIs. For temporary output, use the CLI flag `--junit <path>` without changing config. `Json(path)` writes the full `RunSummary` as JSON for downstream scripts or dashboards. See [CI Integration](./ci-integration) for a complete CI setup.
-
-## Custom Reporters
-
-A reporter is an object that implements optional callbacks and receives the same structured results as built-in reporters:
-
-```ts
-import type { Reporter } from "niceeval";
-
-const notify: Reporter = {
-  async onEvalComplete(result) {
-    if (result.outcome === "failed") {
-      await fetch("http://localhost:3000/notify", {
-        method: "POST",
-        body: JSON.stringify({ eval: result.id, agent: result.agent }),
-      });
-    }
-  },
-};
-```
-
-- `onRunStart(evals, agent, shape)`: run start, with the eval list that will actually run and the run shape.
-- `onEvalComplete(result)`: fires as soon as each attempt completes. Callbacks are serialized and do not interleave.
-- `onRunComplete(summary)`: run complete, with the aggregate summary.
-- `onEvent(event)`: lower-level event stream, such as `eval:start` and `run:budgetExceeded`.
-
-Reporter errors only print diagnostics; they do not stop the run. Custom reporters are only needed for destinations not covered by built-ins. `.niceeval/` artifacts already record all results, so post-run analysis can read them directly. See [Viewing Results](./viewing-results).