niceeval 0.4.6 → 0.5.0

package/docs-site/example/claude-code-codex-plugin.mdx

@@ -1,162 +0,0 @@
----
-title: "Eval Your Claude Code / Codex Plugin"
-sidebarTitle: "Eval CC/Codex Plugins"
-description: "Use niceeval to put a coding-agent plugin in a real workspace and compare pass rate, latency, cost, and behavior with and without the plugin."
----
-
-If you're building a plugin for Claude Code, Codex, or another coding agent, the useful eval is not "can the agent mention the plugin?" It is: put the plugin in a real workspace, ask the agent to complete real development tasks, and compare pass rate, latency, cost, and diffs against a no-plugin baseline.
-
-The runnable reference is [`examples/zh/coding-agent-skill`](https://github.com/CorrectRoadH/niceeval/tree/main/examples/zh/coding-agent-skill). Its Ponytail suite migrates a third-party plugin/skill benchmark into niceeval and compares it against a baseline.
-
-## Directory structure
-
-```text
-examples/zh/coding-agent-skill/
-├─ niceeval.config.ts
-├─ skills/
-│  └─ ponytail.md
-├─ workspaces/
-│  └─ ts-starter/
-├─ evals/
-│  ├─ ponytail-safe-path.eval.ts
-│  ├─ ponytail-csv-sum.eval.ts
-│  └─ ponytail-reuse.eval.ts
-└─ experiments/
-   ├─ ponytail.ts
-   └─ ponytail-baseline.ts
-```
-
-- `skills/ponytail.md` is the plugin/skill content injected into the agent workspace.
-- `workspaces/ts-starter/` is the initial project uploaded into each sandbox.
-- `evals/*.eval.ts` uploads the workspace, sends the task, and verifies the result.
-- `experiments/ponytail.ts` injects the plugin; `experiments/ponytail-baseline.ts` does not.
-
-## Install the plugin in an experiment
-
-Plugin on/off belongs in the experiment, not in the eval ID or CLI positionals. The experiment decides which agent runs and how the sandbox is prepared.
-
-```ts
-import { defineExperiment } from "niceeval";
-import { claudeCodeAgent } from "niceeval/adapter";
-import { dockerSandbox } from "niceeval/sandbox";
-import { readFileSync } from "node:fs";
-import { dirname, join } from "node:path";
-import { fileURLToPath } from "node:url";
-
-const __dir = dirname(fileURLToPath(import.meta.url));
-const ponytailSkill = readFileSync(join(__dir, "../skills/ponytail.md"), "utf-8");
-const baseAgent = claudeCodeAgent();
-
-const ponytailAgent = {
-  ...baseAgent,
-  name: "claude-code+ponytail",
-  async setup(sb, ctx) {
-    const cleanup = await baseAgent.setup?.(sb, ctx);
-    await sb.writeFiles({ "CLAUDE.md": ponytailSkill });
-    return cleanup;
-  },
-} satisfies typeof baseAgent;
-
-export default defineExperiment({
-  description: "claude-code + ponytail plugin",
-  agent: ponytailAgent,
-  model: "claude-sonnet-4-6",
-  sandbox: dockerSandbox(),
-  runs: 3,
-  earlyExit: false,
-  budget: 15,
-  evals: (id) => id.startsWith("ponytail-"),
-});
-```
-
-The baseline keeps the same model, runs, budget, and eval filter, but uses plain `claudeCodeAgent()`. That keeps the comparison attributable to the plugin.
-
-## Write plugin tasks
-
-Each task is a normal `.eval.ts` file. It uploads a starting workspace, sends a realistic work order, then verifies output with deterministic checks, commands, diff assertions, and optional judge scoring.
-
-```ts
-import { defineEval } from "niceeval";
-import { includes } from "niceeval/expect";
-
-export default defineEval({
-  description: "Implement an upload path resolver without explicitly mentioning traversal",
-
-  async test(t) {
-    await t.sandbox.uploadDirectory("../workspaces/ts-starter");
-    await t.sandbox.writeFiles({
-      "uploads.py": [
-        "import os",
-        "",
-        "def safe_upload_path(base_dir: str, filename: str) -> str:",
-        "    # TODO: implement",
-        "    pass",
-      ].join("\n"),
-    });
-
-    await t
-      .send(
-        "Complete safe_upload_path(base_dir, filename) in uploads.py. " +
-          "Return the full server path for a user-uploaded file. base_dir is the allowed root.",
-      )
-      .then((turn) => turn.succeeded());
-
-    const src = await t.sandbox.readSourceFiles({ extensions: ["py"] });
-    const code = src.text();
-    t.check(code, includes(/commonpath|realpath|abspath/));
-
-    await t.sandbox.writeFiles({
-      "_test_traversal.py": [
-        "import sys, os",
-        "sys.path.insert(0, '.')",
-        "from uploads import safe_upload_path",
-        "base = '/var/uploads'",
-        "try:",
-        "    p = safe_upload_path(base, '../../etc/passwd')",
-        "    assert os.path.commonpath([base, os.path.abspath(p)]) == base",
-        "except (ValueError, PermissionError, AssertionError):",
-        "    pass",
-        "print('ok')",
-      ].join("\n"),
-    });
-
-    const result = await t.sandbox.runCommand("python3", ["_test_traversal.py"]);
-    t.check(result.stdout.trim(), includes(/ok/));
-    t.sandbox.fileChanged("uploads.py");
-  },
-});
-```
-
-The prompt does not leak the safety answer. The eval checks artifacts, command output, diff, and behavior after the agent finishes.
-
-## Run the comparison
-
-```bash
-cd examples/zh/coding-agent-skill
-pnpm install
-pnpm exec niceeval exp ponytail ponytail-baseline
-pnpm exec niceeval view
-```
-
-The first argument after `exp` selects the experiment group or file. Later positionals only filter eval IDs; they are not agent names, plugin names, URLs, or runtime config.
-
-## What to inspect
-
-- Whether `pass@N` improves.
-- Whether average latency, tokens, and estimated cost remain acceptable.
-- Whether failing transcripts show the agent ignored the plugin process.
-- Whether the diff only touches task-relevant files.
-- Whether behavior assertions capture the plugin's value, such as more reuse, less overengineering, or proactive safety checks.
-
-## Copy to your agent
-
-```text
-READ docs-site/example/claude-code-codex-plugin.mdx and install niceeval for this repo.
-Use examples/zh/coding-agent-skill as the reference shape: put the plugin in skills/, inject it from an experiment setup hook, write one .eval.ts task against a real workspace, then add a no-plugin baseline experiment.
-```
-
-## Next steps
-
-- [Fixtures](/guides/fixtures) — manage sandbox workspaces, tasks, and verification in `.eval.ts`.
-- [Sandbox Agent](/guides/sandbox-agent) — built-in `claude-code`, `codex`, and custom sandbox agents.
-- [Viewing Results](/guides/viewing-results) — inspect transcripts, diffs, event streams, and traces.

package/docs-site/example/claude-code-codex-skill.mdx

@@ -1,147 +0,0 @@
----
-title: "Eval Your Skill on Claude Code / Codex"
-sidebarTitle: "Eval Skills in Agents"
-description: "Compare coding-agent tasks with and without a Skill by using real workspaces, assertions, and judge scoring."
----
-
-A Skill's quality is not only whether the final answer is correct. It is whether the agent follows the intended process in the right situation, avoids known anti-patterns, and reliably improves pass rate, cost, and latency. The recommended niceeval shape is to inject the Skill into a sandboxed coding-agent workspace, then compare against a no-Skill baseline.
-
-The runnable reference is [`examples/zh/coding-agent-skill`](https://github.com/CorrectRoadH/niceeval/tree/main/examples/zh/coding-agent-skill). It contains two suites:
-
-- Zod skill: a local `skills/zod.md` that steers request-body and environment-variable validation.
-- Ponytail plugin: a migrated third-party benchmark for safety, reuse, and simplicity.
-
-## Directory structure
-
-```text
-examples/zh/coding-agent-skill/
-├─ skills/
-│  ├─ zod.md
-│  └─ ponytail.md
-├─ workspaces/
-│  └─ ts-starter/
-├─ evals/
-│  ├─ api-validation.eval.ts
-│  ├─ config-schema.eval.ts
-│  ├─ ponytail-safe-path.eval.ts
-│  ├─ ponytail-csv-sum.eval.ts
-│  └─ ponytail-reuse.eval.ts
-└─ experiments/
-   ├─ with-skill.ts
-   ├─ baseline.ts
-   ├─ ponytail.ts
-   └─ ponytail-baseline.ts
-```
-
-## Define the Skill experiment
-
-Skill injection belongs to the experiment. The example writes `skills/zod.md` into `CLAUDE.md` during sandbox setup, so Claude Code reads it when it starts.
-
-```ts
-import { defineExperiment } from "niceeval";
-import { claudeCodeAgent } from "niceeval/adapter";
-import { dockerSandbox } from "niceeval/sandbox";
-import { readFileSync } from "node:fs";
-import { dirname, join } from "node:path";
-import { fileURLToPath } from "node:url";
-
-const __dir = dirname(fileURLToPath(import.meta.url));
-const zodSkill = readFileSync(join(__dir, "../skills/zod.md"), "utf-8");
-const baseAgent = claudeCodeAgent();
-
-const zodAgent = {
-  ...baseAgent,
-  name: "claude-code+zod-skill",
-  async setup(sb, ctx) {
-    const cleanup = await baseAgent.setup?.(sb, ctx);
-    await sb.writeFiles({ "CLAUDE.md": zodSkill });
-    return cleanup;
-  },
-} satisfies typeof baseAgent;
-
-export default defineExperiment({
-  description: "claude-code + zod skill",
-  agent: zodAgent,
-  model: "claude-sonnet-4-6",
-  sandbox: dockerSandbox(),
-  runs: 3,
-  earlyExit: false,
-  budget: 10,
-  evals: (id) => !id.startsWith("ponytail-"),
-});
-```
-
-The baseline keeps the same eval filter, model, runs, and budget, but uses plain `claudeCodeAgent()`.
-
-## Write the eval
-
-A `.eval.ts` file prepares the workspace, sends the task, and verifies the result. This is a shortened version of the Zod request-body validation task.
-
-```ts
-import { defineEval } from "niceeval";
-import { excludes, includes } from "niceeval/expect";
-
-export default defineEval({
-  description: "Validate POST /users request bodies with Zod and structured errors",
-
-  async test(t) {
-    await t.sandbox.uploadDirectory("../workspaces/ts-starter");
-
-    await t
-      .send(
-        `Implement POST /users in src/routes/users.ts.
-Requirements:
-- Define CreateUserSchema with Zod
-- Validate req.body with .safeParse(), not .parse()
-- On validation failure, return HTTP 422 with { errors: result.error.issues }
-- On success, return HTTP 201`,
-      )
-      .then((turn) => turn.succeeded());
-
-    const src = await t.sandbox.readSourceFiles({ extensions: ["ts"] });
-    const route = src.fileMatching(/users/);
-    const code = route?.content ?? "";
-
-    t.check(code, includes(/z\.object\s*\(/));
-    t.check(code, includes(/\.safeParse\s*\(/));
-    t.check(code, includes(/422|UNPROCESSABLE/));
-    t.check(code, excludes(/JSON\.parse/));
-    t.sandbox.fileChanged("src/routes/users.ts");
-
-    t.judge.autoevals
-      .closedQA("Does the code use the idiomatic Zod validation pattern?", { on: code })
-      .atLeast(0.75);
-  },
-});
-```
-
-Prefer verifying the final artifact and real tests. Only make Skill-load or tool-call events hard gates when those event names are stable for the agent.
-
-## Run
-
-```bash
-cd examples/zh/coding-agent-skill
-pnpm install
-
-# Zod Skill A/B comparison
-pnpm exec niceeval exp compare
-
-# Ponytail plugin A/B comparison
-pnpm exec niceeval exp ponytail ponytail-baseline
-
-pnpm exec niceeval view
-```
-
-## What to evaluate
-
-- Whether the Skill improves pass rate.
-- Whether it reduces anti-patterns such as raw `JSON.parse`, unsafe path joins, or needless rewrites.
-- Whether the agent consistently changes the intended files.
-- Whether cost and latency remain acceptable.
-- Whether failures mean the Skill was not read, was read but ignored, or the eval target was unclear.
-
-## Next steps
-
-- [Fixtures](/guides/fixtures) — organize sandbox workspaces, tasks, and verification.
-- [Experiments](/guides/experiments) — run with-Skill vs without-Skill controlled experiments.
-- [Scoring Guide](/guides/scoring-guide) — score final results and behavioral constraints together.

package/docs-site/example/showcase.mdx

@@ -1,39 +0,0 @@
----
-title: "Showcase"
-sidebarTitle: "Showcase"
-description: "Real projects and live sites that build evals with NiceEval, collected over time."
----
-
-This page collects real projects and live sites that use [NiceEval](https://niceeval.com/) to build evals. Each case is a runnable, reproducible repository, not an excerpt. Open the source repository to inspect the full eval, experiment, and sandbox configuration.
-
-## coding-agent-memory-evals
-
-Tape, Nowledge Mem, mem9, db9.ai, OpenClaw... memory implementations for coding agents are diverging, but the ecosystem needs a reproducible eval surface to answer whether those mechanisms help in practice. [coding-agent-memory-evals](https://github.com/CorrectRoadH/coding-agent-memory-evals) builds that benchmark with NiceEval: the same coding tasks, the same model (`gpt-5.4`), and only the agent memory mechanism changes. It compares [bub](https://github.com/CorrectRoadH/coding-agent-memory-evals), which includes [Tape](https://tape.systems/), against codex without the corresponding persistent memory.
-
-- Online report: [coding-agent-memory-evals.vercel.app](https://coding-agent-memory-evals.vercel.app/)
-- Source repository: [github.com/CorrectRoadH/coding-agent-memory-evals](https://github.com/CorrectRoadH/coding-agent-memory-evals)
-
-The primary score only checks whether the development task itself was completed, such as build, tests, or the upstream verifier. It does not add a separate question like "did the agent remember this fact?" The value of memory shows up in secondary metrics such as duration, tokens, cost, and `pass^k`.
-
-### Cases
-
-Each eval connects the original task and verification script from a public benchmark. These are not rewritten assertions.
-
-| Eval | Source | What It Verifies |
-|---|---|---|
-| [`agent-029-use-cache-directive`](https://github.com/CorrectRoadH/coding-agent-memory-evals/blob/main/evals/memory/agent-029-use-cache-directive.eval.ts) | next-evals | Whether a Server Action uses `use cache` + `revalidateTag` for eventually consistent directory-data refreshes. |
-| [`agent-030-app-router-migration-hard`](https://github.com/CorrectRoadH/coding-agent-memory-evals/blob/main/evals/memory/agent-030-app-router-migration-hard.eval.ts) | next-evals | Whether a complex Pages Router app is fully migrated to App Router, legacy files are deleted, and legacy APIs do not remain. |
-| [`agent-037-updatetag-cache`](https://github.com/CorrectRoadH/coding-agent-memory-evals/blob/main/evals/memory/agent-037-updatetag-cache.eval.ts) | next-evals | Whether read-your-own-writes scenarios use `updateTag` instead of only `revalidateTag`. |
-| [`terminal-cancel-async-tasks`](https://github.com/CorrectRoadH/coding-agent-memory-evals/blob/main/evals/memory/terminal-cancel-async-tasks.eval.ts) | [Terminal-Bench](https://www.tbench.ai/) | Implements a cancellable concurrency-limited async task runner and runs the original `pytest` verifier. |
-| [`repomod-hello-world-api`](https://github.com/CorrectRoadH/coding-agent-memory-evals/blob/main/evals/memory/repomod-hello-world-api.eval.ts) | RepoMod-Bench | Migrates a Flask API to Java Spring Boot, starts the service, and runs upstream hidden HTTP tests. |
-| [`tool-call-observability`](https://github.com/CorrectRoadH/coding-agent-memory-evals/blob/main/evals/memory/tool-call-observability.eval.ts) | Custom smoke | Whether the agent's shell tool calls are correctly recorded as standard events. |
-
-### Comparison Experiments
-
-[`experiments/compare/`](https://github.com/CorrectRoadH/coding-agent-memory-evals/tree/main/experiments/compare) is the only comparable experiment set. It pins the same model (`gpt-5.4`) and the same sandbox (`docker`), and changes only the agent: [`bub-gpt-5.4.ts`](https://github.com/CorrectRoadH/coding-agent-memory-evals/blob/main/experiments/compare/bub-gpt-5.4.ts) vs. [`codex-gpt-5.4.ts`](https://github.com/CorrectRoadH/coding-agent-memory-evals/blob/main/experiments/compare/codex-gpt-5.4.ts). That makes the difference attributable to the memory mechanism rather than the model or runtime environment. `niceeval exp compare` runs the whole set, and the result snapshot is published to the online report.
-
-## Next Steps
-
-- [Connect Your Agent](/guides/connect-your-agent) — integrate a coding-agent adapter from scratch.
-- [Experiments](/guides/experiments) — organize comparable experiment matrices.
-- [Evaluate Skill Impact](/example/claude-code-codex-skill) — use the same "with / without X" comparison pattern for Skills.