niceeval 0.4.4 → 0.4.5

package/docs-site/example/claude-code-codex-skill.mdx

@@ -0,0 +1,147 @@
+---
+title: "Eval Your Skill on Claude Code / Codex"
+sidebarTitle: "Eval Skills in Agents"
+description: "Compare coding-agent tasks with and without a Skill by using real workspaces, assertions, and judge scoring."
+---
+
+A Skill's quality is not only whether the final answer is correct. It is whether the agent follows the intended process in the right situation, avoids known anti-patterns, and reliably improves pass rate, cost, and latency. The recommended niceeval shape is to inject the Skill into a sandboxed coding-agent workspace, then compare against a no-Skill baseline.
+
+The runnable reference is [`examples/zh/coding-agent-skill`](https://github.com/CorrectRoadH/niceeval/tree/main/examples/zh/coding-agent-skill). It contains two suites:
+
+- Zod skill: a local `skills/zod.md` that steers request-body and environment-variable validation.
+- Ponytail plugin: a migrated third-party benchmark for safety, reuse, and simplicity.
+
+## Directory structure
+
+```text
+examples/zh/coding-agent-skill/
+├─ skills/
+│  ├─ zod.md
+│  └─ ponytail.md
+├─ workspaces/
+│  └─ ts-starter/
+├─ evals/
+│  ├─ api-validation.eval.ts
+│  ├─ config-schema.eval.ts
+│  ├─ ponytail-safe-path.eval.ts
+│  ├─ ponytail-csv-sum.eval.ts
+│  └─ ponytail-reuse.eval.ts
+└─ experiments/
+   ├─ with-skill.ts
+   ├─ baseline.ts
+   ├─ ponytail.ts
+   └─ ponytail-baseline.ts
+```
+
+## Define the Skill experiment
+
+Skill injection belongs to the experiment. The example writes `skills/zod.md` into `CLAUDE.md` during sandbox setup, so Claude Code reads it when it starts.
+
+```ts
+import { defineExperiment } from "niceeval";
+import { claudeCodeAgent } from "niceeval/adapter";
+import { dockerSandbox } from "niceeval/sandbox";
+import { readFileSync } from "node:fs";
+import { dirname, join } from "node:path";
+import { fileURLToPath } from "node:url";
+
+const __dir = dirname(fileURLToPath(import.meta.url));
+const zodSkill = readFileSync(join(__dir, "../skills/zod.md"), "utf-8");
+const baseAgent = claudeCodeAgent();
+
+const zodAgent = {
+  ...baseAgent,
+  name: "claude-code+zod-skill",
+  async setup(sb, ctx) {
+    const cleanup = await baseAgent.setup?.(sb, ctx);
+    await sb.writeFiles({ "CLAUDE.md": zodSkill });
+    return cleanup;
+  },
+} satisfies typeof baseAgent;
+
+export default defineExperiment({
+  description: "claude-code + zod skill",
+  agent: zodAgent,
+  model: "claude-sonnet-4-6",
+  sandbox: dockerSandbox(),
+  runs: 3,
+  earlyExit: false,
+  budget: 10,
+  evals: (id) => !id.startsWith("ponytail-"),
+});
+```
+
+The baseline keeps the same eval filter, model, runs, and budget, but uses plain `claudeCodeAgent()`.
+
+## Write the eval
+
+A `.eval.ts` file prepares the workspace, sends the task, and verifies the result. This is a shortened version of the Zod request-body validation task.
+
+```ts
+import { defineEval } from "niceeval";
+import { excludes, includes } from "niceeval/expect";
+
+export default defineEval({
+  description: "Validate POST /users request bodies with Zod and structured errors",
+
+  async test(t) {
+    await t.sandbox.uploadDirectory("../workspaces/ts-starter");
+
+    await t
+      .send(
+        `Implement POST /users in src/routes/users.ts.
+Requirements:
+- Define CreateUserSchema with Zod
+- Validate req.body with .safeParse(), not .parse()
+- On validation failure, return HTTP 422 with { errors: result.error.issues }
+- On success, return HTTP 201`,
+      )
+      .then((turn) => turn.succeeded());
+
+    const src = await t.sandbox.readSourceFiles({ extensions: ["ts"] });
+    const route = src.fileMatching(/users/);
+    const code = route?.content ?? "";
+
+    t.check(code, includes(/z\.object\s*\(/));
+    t.check(code, includes(/\.safeParse\s*\(/));
+    t.check(code, includes(/422|UNPROCESSABLE/));
+    t.check(code, excludes(/JSON\.parse/));
+    t.sandbox.fileChanged("src/routes/users.ts");
+
+    t.judge.autoevals
+      .closedQA("Does the code use the idiomatic Zod validation pattern?", { on: code })
+      .atLeast(0.75);
+  },
+});
+```
+
+Prefer verifying the final artifact and real tests. Only make Skill-load or tool-call events hard gates when those event names are stable for the agent.
+
+## Run
+
+```bash
+cd examples/zh/coding-agent-skill
+pnpm install
+
+# Zod Skill A/B comparison
+pnpm exec niceeval exp compare
+
+# Ponytail plugin A/B comparison
+pnpm exec niceeval exp ponytail ponytail-baseline
+
+pnpm exec niceeval view
+```
+
+## What to evaluate
+
+- Whether the Skill improves pass rate.
+- Whether it reduces anti-patterns such as raw `JSON.parse`, unsafe path joins, or needless rewrites.
+- Whether the agent consistently changes the intended files.
+- Whether cost and latency remain acceptable.
+- Whether failures mean the Skill was not read, was read but ignored, or the eval target was unclear.
+
+## Next steps
+
+- [Fixtures](/guides/fixtures) — organize sandbox workspaces, tasks, and verification.
+- [Experiments](/guides/experiments) — run with-Skill vs without-Skill controlled experiments.
+- [Scoring Guide](/guides/scoring-guide) — score final results and behavioral constraints together.

package/docs-site/example/showcase.mdx

@@ -0,0 +1,39 @@
+---
+title: "Showcase"
+sidebarTitle: "Showcase"
+description: "Real projects and live sites that build evals with NiceEval, collected over time."
+---
+
+This page collects real projects and live sites that use [NiceEval](https://niceeval.com/) to build evals. Each case is a runnable, reproducible repository, not an excerpt. Open the source repository to inspect the full eval, experiment, and sandbox configuration.
+
+## coding-agent-memory-evals
+
+Tape, Nowledge Mem, mem9, db9.ai, OpenClaw... memory implementations for coding agents are diverging, but the ecosystem needs a reproducible eval surface to answer whether those mechanisms help in practice. [coding-agent-memory-evals](https://github.com/CorrectRoadH/coding-agent-memory-evals) builds that benchmark with NiceEval: the same coding tasks, the same model (`gpt-5.4`), and only the agent memory mechanism changes. It compares [bub](https://github.com/CorrectRoadH/coding-agent-memory-evals), which includes [Tape](https://tape.systems/), against codex without the corresponding persistent memory.
+
+- Online report: [coding-agent-memory-evals.vercel.app](https://coding-agent-memory-evals.vercel.app/)
+- Source repository: [github.com/CorrectRoadH/coding-agent-memory-evals](https://github.com/CorrectRoadH/coding-agent-memory-evals)
+
+The primary score only checks whether the development task itself was completed, such as build, tests, or the upstream verifier. It does not add a separate question like "did the agent remember this fact?" The value of memory shows up in secondary metrics such as duration, tokens, cost, and `pass^k`.
+
+### Cases
+
+Each eval connects the original task and verification script from a public benchmark. These are not rewritten assertions.
+
+| Eval | Source | What It Verifies |
+|---|---|---|
+| [`agent-029-use-cache-directive`](https://github.com/CorrectRoadH/coding-agent-memory-evals/blob/main/evals/memory/agent-029-use-cache-directive.eval.ts) | next-evals | Whether a Server Action uses `use cache` + `revalidateTag` for eventually consistent directory-data refreshes. |
+| [`agent-030-app-router-migration-hard`](https://github.com/CorrectRoadH/coding-agent-memory-evals/blob/main/evals/memory/agent-030-app-router-migration-hard.eval.ts) | next-evals | Whether a complex Pages Router app is fully migrated to App Router, legacy files are deleted, and legacy APIs do not remain. |
+| [`agent-037-updatetag-cache`](https://github.com/CorrectRoadH/coding-agent-memory-evals/blob/main/evals/memory/agent-037-updatetag-cache.eval.ts) | next-evals | Whether read-your-own-writes scenarios use `updateTag` instead of only `revalidateTag`. |
+| [`terminal-cancel-async-tasks`](https://github.com/CorrectRoadH/coding-agent-memory-evals/blob/main/evals/memory/terminal-cancel-async-tasks.eval.ts) | [Terminal-Bench](https://www.tbench.ai/) | Implements a cancellable concurrency-limited async task runner and runs the original `pytest` verifier. |
+| [`repomod-hello-world-api`](https://github.com/CorrectRoadH/coding-agent-memory-evals/blob/main/evals/memory/repomod-hello-world-api.eval.ts) | RepoMod-Bench | Migrates a Flask API to Java Spring Boot, starts the service, and runs upstream hidden HTTP tests. |
+| [`tool-call-observability`](https://github.com/CorrectRoadH/coding-agent-memory-evals/blob/main/evals/memory/tool-call-observability.eval.ts) | Custom smoke | Whether the agent's shell tool calls are correctly recorded as standard events. |
+
+### Comparison Experiments
+
+[`experiments/compare/`](https://github.com/CorrectRoadH/coding-agent-memory-evals/tree/main/experiments/compare) is the only comparable experiment set. It pins the same model (`gpt-5.4`) and the same sandbox (`docker`), and changes only the agent: [`bub-gpt-5.4.ts`](https://github.com/CorrectRoadH/coding-agent-memory-evals/blob/main/experiments/compare/bub-gpt-5.4.ts) vs. [`codex-gpt-5.4.ts`](https://github.com/CorrectRoadH/coding-agent-memory-evals/blob/main/experiments/compare/codex-gpt-5.4.ts). That makes the difference attributable to the memory mechanism rather than the model or runtime environment. `niceeval exp compare` runs the whole set, and the result snapshot is published to the online report.
+
+## Next Steps
+
+- [Connect Your Agent](/guides/connect-your-agent) — integrate a coding-agent adapter from scratch.
+- [Experiments](/guides/experiments) — organize comparable experiment matrices.
+- [Evaluate Skill Impact](/example/claude-code-codex-skill) — use the same "with / without X" comparison pattern for Skills.