niceeval 0.4.6 → 0.5.1

package/docs-site/guides/runner.mdx

@@ -1,79 +0,0 @@
----
-title: "How the NiceEval runner schedules and executes evals"
-sidebarTitle: "Runner"
-description: "The NiceEval runner discovers evals, schedules them with bounded concurrency, caches results, retries flaky infrastructure, and enforces budget guards."
----
-
-The runner is the execution engine that turns eval files into an actual run. It handles discovery, filtering, scheduling, caching, attempts, reporters, and artifact output.
-
-## Discovery
-
-The runner reads from `evals/`:
-
-- `*.eval.ts` files
-- dataset files that export eval arrays
-
-```bash
-npx niceeval list
-```
-
-`list` only discovers and prints IDs. It does not execute evals.
-
-## Filtering
-
-In the `exp` command, positional arguments after the experiment name are eval ID prefixes:
-
-```bash
-npx niceeval exp local
-npx niceeval exp local weather
-npx niceeval exp local fixtures/button
-```
-
-Do not put the agent name into positional arguments. Agent, model, and flags belong in the experiment.
-
-## Concurrency
-
-```bash
-npx niceeval exp local --max-concurrency 8
-```
-
-Remote HTTP agents can usually handle higher concurrency. Local Docker sandbox runs usually need lower concurrency to avoid CPU, memory, and disk contention.
-
-## `runs` and `early-exit`
-
-```bash
-npx niceeval exp local fixtures/button --runs 5 --early-exit
-```
-
-`runs` is for pass-rate measurement. `early-exit` stops the remaining attempts for the same eval after one attempt passes.
-
-## Cache
-
-NiceEval can skip already-passing results based on a fingerprint of inputs, configuration, and relevant files. Cache is useful for iteration speed, but turn it off or clear it explicitly when you are debugging non-deterministic behavior.
-
-## Timeout and budget
-
-```bash
-npx niceeval exp local --timeout-ms 300000 --budget 5
-```
-
-Timeout protects a single eval. Budget protects the cost of the entire run.
-
-## Reporter
-
-After each eval finishes, the runner hands the result to reporters:
-
-- the console reporter provides live feedback
-- JSON artifacts support later analysis
-- the JUnit reporter fits CI
-
-## Output directory
-
-Each run writes to `.niceeval/<timestamp>/`, including `summary.json` and per-attempt artifacts such as `events.json`, `sources.json`, `trace.json`, `o11y.json`, and `diff.json`.
-
-## Recommended debugging flow
-
-1. Run `npx niceeval list` to confirm discovery.
-2. Narrow to one eval with `npx niceeval exp <experiment> <id-prefix>`.
-3. After a failure, run `npx niceeval view` to inspect transcript and diff.
-4. Then widen back out to the full suite or experiment.

package/docs-site/guides/sandbox-agent.mdx

@@ -1,138 +0,0 @@
----
-title: "Sandbox agents: evaluate Claude Code, Codex, and bub"
-sidebarTitle: "Sandbox Agents"
-description: "Use NiceEval built-in agents or a custom adapter to run coding-agent CLIs inside Docker or cloud sandboxes."
----
-
-Sandbox agents start a coding-agent CLI inside an isolated environment, give it a workspace and a task, let it edit files and run commands, then collect transcript, diff, and test results.
-
-## Built-in sandbox agents
-
-<CardGroup cols={3}>
-  <Card title="claude-code" icon="code">
-    Runs the Anthropic Claude Code CLI. Requires `ANTHROPIC_API_KEY`.
-  </Card>
-  <Card title="codex" icon="terminal">
-    Runs the OpenAI Codex CLI. Requires `CODEX_API_KEY`.
-  </Card>
-  <Card title="bub" icon="robot">
-    Runs the bub coding agent. Authentication follows the bub CLI's own rules.
-  </Card>
-</CardGroup>
-
-## Run a built-in agent
-
-Set `sandbox` in the experiment, or as a project-level fallback in `niceeval.config.ts`, to choose where NiceEval creates the isolated environment:
-
-```ts
-import { defineExperiment } from "niceeval";
-import { claudeCodeAgent } from "niceeval/adapter";
-import { dockerSandbox } from "niceeval/sandbox";
-
-export default defineExperiment({
-  agent: claudeCodeAgent(),
-  model: "claude-sonnet-4-6",
-  sandbox: dockerSandbox(),
-});
-```
-
-```sh
-export ANTHROPIC_API_KEY=sk-ant-...
-npx niceeval exp local fixtures/button
-
-npx niceeval exp local fixtures/button --runs 10 --early-exit
-```
-
-<Note>
-  There is no CLI flag for choosing the backend. Backend selection lives entirely in code. If neither the experiment nor `niceeval.config.ts` sets `sandbox`, NiceEval fails when it tries to create the sandbox.
-</Note>
-
-The built-in agents exported from `niceeval/adapter` are factory functions. Put auth, proxy, MCP, or GitHub Skill setup in the factory arguments. Keep the model in the experiment's `model` field, and the sandbox backend in `sandbox`:
-
-```ts
-import { defineExperiment } from "niceeval";
-import { claudeCodeAgent } from "niceeval/adapter";
-import { dockerSandbox } from "niceeval/sandbox";
-
-export default defineExperiment({
-  agent: claudeCodeAgent({
-    apiKey: process.env.ANTHROPIC_API_KEY,
-    baseUrl: process.env.ANTHROPIC_BASE_URL,
-    maxTurns: 8,
-    mcpServers: [
-      {
-        name: "browser",
-        command: "npx",
-        args: ["-y", "@anthropic/mcp-browser"],
-      },
-    ],
-    skills: ["Effect-TS/skills"],
-  }),
-  model: "claude-sonnet-4-6",
-  sandbox: dockerSandbox(),
-});
-```
-
-## Agent environment variables
-
-| Agent | Required variables |
-|---|---|
-| `claude-code` | `ANTHROPIC_API_KEY` |
-| `codex` | `CODEX_API_KEY` |
-| `bub` | whatever the bub CLI requires |
-
-## Workflow
-
-```text
-createSandbox
-  -> git init && git commit
-  -> adapter.setup?
-  -> test(t): uploadDirectory(...)
-  -> adapter.send(input, ctx)
-  -> test(t): runCommand(...)
-  -> collectGeneratedFiles()
-  -> sandbox.stop()
-```
-
-The starting files and verification commands both live in `test(t)`. During agent execution, it can only see the files you already wrote into the sandbox.
-
-## Custom sandbox agent
-
-```ts
-import { defineSandboxAgent } from "niceeval/adapter";
-
-export default defineSandboxAgent({
-  name: "my-agent",
-  async send(input, ctx) {
-    await ctx.sandbox.runCommand("my-agent", ["run", input.text, "--json-out", "agent-events.json"]);
-    const transcript = await ctx.sandbox.readFile("agent-events.json");
-    return {
-      status: "completed",
-      events: parseTranscript(transcript),
-    };
-  },
-});
-```
-
-## `ctx.model` and `ctx.flags`
-
-The model and custom flags passed by the experiment appear in the adapter context. The adapter decides how to map them to CLI arguments or HTTP payloads.
-
-## Use a custom agent in an experiment
-
-```ts
-import { defineExperiment } from "niceeval";
-import { dockerSandbox } from "niceeval/sandbox";
-import myAgent from "./agents/my-agent";
-
-export default defineExperiment({
-  agent: myAgent,
-  model: "claude-sonnet-4-6",
-  sandbox: dockerSandbox(),
-  runs: 3,
-});
-```
-
-<Warning>
-  Do not add agent-specific branches inside the runner. Put differing behavior in the adapter.
-</Warning>

package/docs-site/guides/sandbox-backends.mdx

@@ -1,64 +0,0 @@
----
-title: "Sandbox backends: Docker, Vercel, and third-party providers"
-sidebarTitle: "Sandbox Backends"
-description: "NiceEval runs coding agents inside Docker or Vercel sandboxes. Learn how to choose a backend, configure permissions, and improve performance with warm pools."
----
-
-The sandbox backend is the infrastructure that creates and manages isolated runtime environments. NiceEval wraps all backends behind the same `Sandbox` interface, so adapters do not need to know whether the current backend is local Docker, a Vercel micro-VM, or a third-party provider.
-
-## The `Sandbox` interface
-
-Common adapter operations include:
-
-| Method | Use |
-|---|---|
-| `runCommand(cmd, args)` | run a command |
-| `runShell(script)` | run a shell script |
-| `readFile(path)` | read a file |
-| `writeFiles(files)` | write a batch of files |
-| `uploadFiles(files)` | upload a workspace or test files |
-| `workdir` | the backend's real working directory; omitted `cwd` / `targetDir` resolve here |
-| `runCommand(..., { cwd })` | temporarily switch directory for one command; relative paths resolve from `workdir` |
-| `stop()` | destroy the environment |
-
-## Choose a backend
-
-Backend selection only lives in code. There is no CLI flag, and NiceEval does not auto-detect a backend. Set the `sandbox` field in the experiment, or set it in `niceeval.config.ts` as a project-level fallback:
-
-```ts
-import { defineExperiment } from "niceeval";
-import { dockerSandbox } from "niceeval/sandbox";
-
-export default defineExperiment({
-  agent: myCodingAgent,
-  model: "claude-sonnet-4-6",
-  sandbox: dockerSandbox(), // or vercelSandbox() / e2bSandbox()
-});
-```
-
-If neither the experiment nor `niceeval.config.ts` sets `sandbox`, NiceEval fails when it tries to create the environment instead of guessing one.
-
-## Docker
-
-Docker is good for local development and standard CI. It is simple, controllable, and has no cloud dependency. The tradeoff is limited machine resources, plus slower cold starts and dependency installs.
-
-## Vercel Sandbox
-
-Vercel Sandbox is good when you want cloud isolation, more resources, or a more stable environment. It requires the right token or OIDC setup.
-
-## Third-party backends
-
-Any provider that implements the `Sandbox` interface can be plugged in. Adapters still depend only on the interface, not on provider-specific APIs.
-
-## Permissions and root
-
-Different backends impose different constraints on root access, networking, the file system, and process lifecycle. When writing fixtures, keep host assumptions to a minimum. Put dependencies in `package.json` or fixture setup instead of depending on the machine outside the sandbox.
-
-## Performance advice
-
-- Keep fixture dependencies small.
-- Use caching or warm-up mechanisms.
-- Tune `sandboxConcurrency` so local Docker does not run out of resources.
-- Split slow checks into required gates and optional soft checks.
-
-Warm pools and reuse belong to the runner / scheduler layer. See [Runner](/guides/runner).

package/docs-site/guides/scoring-guide.mdx

@@ -1,92 +0,0 @@
----
-title: "Scoring guide: assertions, judge, and cost limits"
-sidebarTitle: "Scoring Guide"
-description: "Evaluate any NiceEval run with five scoring mechanisms: value assertions, scoped assertions, LLM-as-judge, test-as-scoring, and efficiency checks."
----
-
-A good eval should decompose "did this succeed?" into signals you can explain. NiceEval lets you mix precise assertions, semantic judge checks, event-stream checks, and real tests.
-
-## Choose the assertion type
-
-| Goal | Recommended mechanism |
-|---|---|
-| Reply includes a fixed field | `includes` / `matches` |
-| Structured JSON matches exactly | `equals` |
-| A tool call happened | `t.calledTool` / `t.notCalledTool` |
-| Output is semantically correct | `t.judge.*` |
-| Code actually works | `t.check(await t.sandbox.runCommand("npm", ["test"]), commandSucceeded())` |
-| Cost stays under control | `t.maxTokens` / `t.maxCost` |
-
-## Value assertions
-
-```ts
-import { includes, equals, matches } from "niceeval/expect";
-
-t.check(t.reply, includes("refund"));
-t.check(turn.data, equals({ intent: "refund" }));
-t.check(t.reply, matches(/order #\d+/));
-```
-
-Value assertions fit results that are precise, stable, and low ambiguity.
-
-## Scoped assertions
-
-```ts
-t.succeeded();
-t.calledTool("search");
-t.usedNoTools();
-t.sandbox.fileChanged("src/app.ts");
-t.check(await t.sandbox.runCommand("npm", ["test"]), commandSucceeded());
-```
-
-These check facts about the run, usually from the standard event stream or sandbox artifacts.
-
-## LLM-as-judge
-
-```ts
-t.judge.autoevals.factuality("Refunds are available within 30 days.", { on: t.reply }).atLeast(0.8);
-t.judge.autoevals.closedQA("The answer is specific, polite, and does not invent policy.", { on: t.reply }).atLeast(0.75);
-```
-
-Judge checks are good for semantic quality. They are not a replacement for all deterministic assertions. Prefer exact checks when you can make them.
-
-## Gate vs soft
-
-```ts
-t.check(t.reply, includes("refund").gate());
-t.check(t.reply, includes("friendly tone").atLeast(0.7));
-```
-
-- `gate`: failure is a run failure
-- `soft`: keep the score for comparison without turning it into a hard failure
-
-## Use tests to score code tasks
-
-Run project tests or short probe scripts inside `.eval.ts`:
-
-```ts
-import { commandSucceeded, includes } from "niceeval/expect";
-
-const testResult = await t.sandbox.runCommand("npm", ["test"]);
-t.check(testResult, commandSucceeded());
-
-const src = await t.sandbox.readSourceFiles({ extensions: ["ts"] });
-t.check(src.text(), includes(/z\.object\s*\(/));
-```
-
-## Cost and efficiency
-
-```ts
-t.maxTokens(25_000);
-t.maxCost(0.1);
-```
-
-This is especially useful for coding agents and long-chain agents, where repeated retries or excessive tool use can hide quality problems.
-
-## Practical advice
-
-- Start with one or two gates to establish the minimum bar.
-- Add soft scores to compare quality across variants.
-- Use judge for complex semantics, but give it a clear rubric.
-- For coding-agent results, prefer real tests when possible.
-- Make failure messages explicit so reports are actionable.

package/docs-site/guides/viewing-results.mdx

@@ -1,195 +0,0 @@
----
-title: "View NiceEval results and debug agent behavior"
-sidebarTitle: "Viewing Results"
-description: "After every run, NiceEval writes structured artifacts into .niceeval/. Use npx niceeval view to inspect transcripts, diffs, event streams, and pass rates."
----
-
-After every run, [NiceEval](https://niceeval.com/) writes structured results into `.niceeval/<timestamp>/`. The console gives immediate feedback. The viewer is for deeper failure analysis.
-
-## Console output
-
-```text
-Discovered 3 evals
-
-  ✓ classify (12ms)
-  ✓ weather/brooklyn (456ms)
-  ✗ api-validation (38s)
-    - gate: expected src/routes/users.ts to include .safeParse() [FAILED]
-
-Results:  2 passed, 1 failed, 0 errored, 0 skipped
-```
-
-Each line includes the eval ID, outcome, and duration. Failures also show the assertion type and error message.
-
-## `.niceeval/<timestamp>/`
-
-Typical structure:
-
-```text
-.niceeval/
-└─ 2025-01-15T14-23-00/
-   ├─ summary.json
-   └─ artifacts/
-      └─ compare__web-agent__gpt-5.4__weather-tool__0/
-         ├─ events.json
-         ├─ sources.json
-         ├─ trace.json
-         ├─ o11y.json
-         └─ diff.json
-```
-
-## `niceeval view`
-
-```bash
-npx niceeval view
-```
-
-This opens the local viewer and, by default, shows the most recent run. You can browse evals, inspect transcripts, read diffs, inspect the event stream, and review assertions. Nothing is uploaded to an external service.
-
-<Tip>
-  Run `npx niceeval view` right after a failure to open the artifacts from the run you just made.
-</Tip>
-
-## Exporting and Static Hosting
-
-There is one export mode for publishing results: `--out <dir>` exports the whole viewer site with everything the viewer can inspect. If you want a custom report shape, build one with the report primitives.
-
-### Built-in Viewer Site: Static Hosting
-
-```bash
-npx niceeval view --out site
-```
-
-NiceEval writes `<dir>/index.html` and copies the artifacts the viewer reads (`sources.json`, `events.json`, `trace.json`) into `<dir>/artifact/`. Deploy the whole directory to any static host, such as Vercel, GitHub Pages, or `python3 -m http.server`; code views, transcripts, and trace waterfalls behave the same as local `niceeval view`.
-
-`--out` only accepts a directory. There is no single-file export: code views, transcripts, and traces depend on artifact files, and a single HTML file cannot carry the full evidence. To share results, host the whole site and send the link.
-
-Two limits:
-
-- `diff.json` and `o11y.json` are not copied. The viewer does not read them, and diffs can reach hundreds of MB.
-- Opening `index.html` directly with `file://` prevents the browser from fetching artifacts, so the code view will say source is unavailable. Use an HTTP server for local preview.
-
-The simplest flow is to export on the machine that ran the eval and deploy the output directory directly. `.niceeval/` usually does not go into version control. If the build happens on another machine, such as CI or a hosting platform, use `copyRun` first to copy a slim snapshot into the repo, then export from that snapshot directory:
-
-```typescript
-// scripts/snapshot.ts -- run after evals finish, then commit the output.
-import { openResults, latestPerExperiment, copyRun } from "niceeval/results";
-
-const results = await openResults(".niceeval");
-const { snapshots } = latestPerExperiment(results.snapshots);
-await copyRun(snapshots, "site-data/run", {
-  artifacts: ["sources", "events", "trace"],
-});
-```
-
-```bash
-npx niceeval view --out site site-data/run   # Build from the snapshot in CI.
-```
-
-`latestPerExperiment` chooses the latest snapshot per experiment, not the latest run overall. If one run only covered part of the experiment set, the other experiments still keep their previous results.
-
-### Custom Report Page: Your Own View
-
-The built-in viewer has a fixed leaderboard shape. For your own view, such as a scorecard, benchmark board, or quality x cost scatter plot, build it into your app with report primitives: `niceeval/results` reads results, `niceeval/report` computes data, and `niceeval/report/react` renders components.
-
-There is a serializable boundary between compute and render. Compute functions such as `table()`, `scatter()`, and `overview()` read the file system and only run in Node, such as a build script, server, or CI. Components only accept computed JSON data, perform zero IO, and can render anywhere. That gives static publishing two shapes.
-
-**Option 1: compute at build time and export a fully static page.** Build the page on a machine that can read the result data, such as Next.js RSC or any SSG. The output is ordinary static files:
-
-```tsx
-// app/evals/page.tsx
-import { openResults, latestPerExperiment } from "niceeval/results";
-import { table, passRate, costUSD } from "niceeval/report";
-import { MetricTable } from "niceeval/report/react";
-import "niceeval/report/react/styles.css";
-
-export default async function EvalsPage() {
-  const results = await openResults(".niceeval");
-  const { snapshots } = latestPerExperiment(results.snapshots);
-  const data = await table(snapshots, {
-    rows: "agent",
-    columns: [passRate, costUSD],
-    sort: passRate,
-  });
-  return <MetricTable data={data} />;
-}
-```
-
-**Option 2: write JSON in CI and render it from any frontend.** Compute where the results exist, then render wherever your page lives, such as a Vite SPA or an existing internal dashboard:
-
-```typescript
-// scripts/build-report-data.ts -- run in CI after evals finish.
-import { writeFile } from "node:fs/promises";
-import { openResults, latestPerExperiment } from "niceeval/results";
-import { table, passRate, costUSD } from "niceeval/report";
-
-const results = await openResults(".niceeval");
-const { snapshots } = latestPerExperiment(results.snapshots);
-await writeFile("public/report.json", JSON.stringify({
-  board: await table(snapshots, { rows: "agent", columns: [passRate, costUSD] }),
-}));
-```
-
-```tsx
-// Frontend: any React app.
-import { MetricTable } from "niceeval/report/react";
-
-const doc = await fetch("/report.json").then((r) => r.json());
-
-<MetricTable data={doc.board} />;
-```
-
-The components are complete without hydration: sorting is precomputed on the data side, drill-downs are normal links, and expand/collapse uses `<details>`. Static exports do not need client-side JavaScript. Styles ship with the package (`niceeval/report/react/styles.css`); load your own CSS after it to override.
-
-Your custom page owns the conclusion. You do not need to redraw every attempt transcript and trace yourself. Export the built-in viewer to a subpath, such as `npx niceeval view --out <site-dir>/evidence`, then generate deep links with the component `attemptHref`:
-
-```tsx
-<MetricTable
-  data={data}
-  attemptHref={(ref) => `/evidence/#/attempt/${encodeURIComponent(ref.run)}/${ref.result}`}
-/>
-```
-
-## Artifact Reference
-
-### `summary.json`
-
-The full-run summary: run ID, pass / fail counts, duration, cost, and per-eval status.
-
-### `events.json`
-
-The standard event stream. It is the low-level fact source for tool calls, messages, commands, and errors.
-
-### `diff.json`
-
-The file diff produced by a sandbox eval.
-
-### `sources.json`
-
-Eval source locations and assertion locations, which let the viewer map failures back to code.
-
-### `trace.json`
-
-OTLP traces or normalized span data.
-
-### `o11y.json`
-
-Observation summaries such as tool calls, commands, usage, and cost.
-
-## Outcome Meanings
-
-<Tabs>
-  <Tab title="passed">All gates passed.</Tab>
-  <Tab title="failed">At least one gate failed, or a soft assertion missed its threshold under `--strict`.</Tab>
-  <Tab title="errored">The environment, timeout, adapter, or agent runtime failed.</Tab>
-  <Tab title="skipped">The eval was intentionally skipped.</Tab>
-</Tabs>
-
-Soft assertion scores are recorded in `assertions[].score`; outside `--strict`, they do not create another outcome.
-
-## Debugging Advice
-
-- Start with `summary.json` to find the failing assertion.
-- Then inspect the transcript view to understand the agent's decisions.
-- For coding-agent failures, inspect `diff.json` and command output.
-- For tool-call problems, inspect `events.json`.

package/docs-site/guides/write-experiment.mdx

@@ -1,81 +0,0 @@
----
-title: "Write experiments with defineExperiment"
-sidebarTitle: "Write Experiments"
-description: "Use experiments/ to select an agent, pass model and flags, and configure runs, budget, concurrency, and sandbox."
----
-
-An experiment is a checked-in run configuration. It says which adapter, model, flags, run count, budget, and sandbox to use for the same set of evals. CLI positional arguments select eval ID prefixes; they do not temporarily change the agent or run configuration.
-
-## Minimal experiment
-
-```ts
-// experiments/local.ts
-import { defineExperiment } from "niceeval";
-import { webAgent } from "../agents/web-agent.ts";
-
-export default defineExperiment({
-  agent: webAgent({ baseUrl: "http://127.0.0.1:5188" }),
-});
-```
-
-`agent` is an already configured agent instance. URL, auth, and protocol details normally go into the adapter factory.
-
-## Pass model and flags
-
-```ts
-// experiments/concise.ts
-import { defineExperiment } from "niceeval";
-import { webAgent } from "../agents/web-agent.ts";
-
-export default defineExperiment({
-  description: "concise prompt variant",
-  agent: webAgent({
-    baseUrl: "https://staging.example.com",
-    apiKey: process.env.STAGING_AGENT_API_KEY,
-  }),
-  model: "gpt-5.4",
-  flags: {
-    promptVariant: "concise",
-    webResearch: false,
-  },
-  runs: 3,
-  earlyExit: true,
-  budget: 5,
-});
-```
-
-`model` reaches the adapter as `ctx.model`. `flags` reaches the adapter as `ctx.flags` and evals as `t.flags`. NiceEval passes both through without interpreting them.
-
-## Organize a comparison
-
-One experiment file is one configuration cell. To compare multiple models, agents, or flags, put several files under one experiment group:
-
-```text
-experiments/
-  prompt-variants/
-    baseline.ts
-    concise.ts
-    with-retrieval.ts
-```
-
-```bash
-npx niceeval exp prompt-variants
-```
-
-## Common fields
-
-| Field | Purpose |
-|---|---|
-| `agent` | Select and configure the adapter |
-| `model` | Single model name passed through `ctx.model` |
-| `reasoningEffort` | Single reasoning effort level (e.g. `"high"`), passed through `ctx.reasoningEffort` and `t.reasoningEffort`, same ownership as `model` |
-| `flags` | JSON object passed through `ctx.flags` and `t.flags` |
-| `runs` | Maximum runs per eval and configuration |
-| `earlyExit` | Stop repeated runs when one passes |
-| `evals` | Restrict which evals this configuration runs |
-| `timeoutMs` | Attempt timeout |
-| `budget` | Budget limit for this configuration |
-| `maxConcurrency` | Concurrency limit for this configuration |
-| `sandbox` | Sandbox backend for sandbox agents, such as `dockerSandbox()` or `e2bSandbox()` |
-
-See [Experiment Matrix](/guides/experiments) for comparison design.