@cursor/july 0.1.1

package/skills/ab/SKILL.md

@@ -0,0 +1,139 @@
+---
+name: agent-serve-ab
+description: >-
+  Author defineAB live A/B metrics plug-ins under agent/ab. Splits traffic into
+  sticky variants and calls onSample with performance metrics as the agent runs
+  — not a test runner. Use when instrumenting production/dev traffic for variant
+  comparison; use defineEval for regression gates.
+---
+
+# agent-serve A/B metrics (`defineAB`)
+
+`defineAB` is a **live metrics plug-in**. At session creation the engine runs
+`split` and appends durable `ab.assigned` events; the collector folds the
+session event stream and calls `onSample`. There is **no** `agent-serve ab`
+CLI and **no** assertion API.
+
+Human-facing reference: [`docs/ab.md`](../../docs/ab.md).
+
+| | `defineEval` | `defineAB` |
+| --- | --- | --- |
+| Job | Regression gates on frozen fixtures | Collect metrics on live runs |
+| Location | `evals/**/*.eval.ts` | `agent/ab.ts` or `agent/ab/<name>.ts` |
+| How it runs | `agent-serve eval` | Automatically under `serve` / `run` |
+| Driver | `t.send` + gates | `split` → `ab.assigned` + `onSample` |
+
+## Authoring
+
+```ts
+import { defineAB, splitBySessionHash, splitIf } from "@anysphere/agent-serve/ab";
+
+export default defineAB({
+  name: "concise-instructions",
+  variants: {
+    control: { label: "Baseline" },
+    treatment: {
+      label: "Shorter",
+      // Optional: appended to agent instructions for this arm.
+      instructions: "Keep replies to one short paragraph.",
+    },
+  },
+  split: splitBySessionHash({ holdout: 0.1 }),
+  derive: {
+    // Keep sync + cheap — runs on every event, including message deltas.
+    weatherCalls: (event) =>
+      event.type === "action.result" && event.data.toolName === "get_weather"
+        ? 1
+        : null,
+  },
+  onSample(sample) {
+    console.log(sample.variant, sample.metrics.toolCalls, sample.metrics.wallTimeMs);
+  },
+});
+```
+
+In tools, read the arm from session context:
+
+```ts
+async execute(input, ctx) {
+  if (ctx.session.abs?.["concise-instructions"] === "treatment") {
+    // treatment-specific behavior
+  }
+}
+```
+
+
+Discovery: `agent/ab.ts` and `agent/ab/*.ts`. Name defaults to file stem or `config.name`.
+
+Optional project defaults in `agent/ab.config.ts` (retention + persistence):
+
+```ts
+import {
+  defineABConfig,
+  persistABSamplesToDir,
+  persistABSnapshotsToDir,
+} from "@anysphere/agent-serve/ab";
+
+export default defineABConfig({
+  // maxPlaygroundSessions: 200, // optional; default 200; A/Bs tab / GET /v1/abs only
+  persistSamples: persistABSamplesToDir(".agent-serve/ab-samples"),
+  persistSnapshots: persistABSnapshotsToDir(".agent-serve/ab-snapshots"),
+});
+```
+
+Session `events.ndjson` remains the assignment/metrics source of truth.
+`persistSamples` / `persistSnapshots` are author-controlled exports (like
+eval `persistRuns`).
+
+### Stream model
+
+Typical new-session prefix:
+
+1. `session.started`
+2. one `ab.assigned` per experiment (`variant` key or `null` skip)
+3. `message.received` / `turn.started` / …
+
+Enrollment is **at session creation** (before the first turn) so each arm can
+vary instructions and tool behavior from a known assignment. Failed/`null`
+splits persist as `ab.assigned` with `variant: null` (sticky skip). The
+collector is a pure fold over the log. `SessionRecord.abs` is a convenience
+mirror, not a metrics store. After park/restart it replays `events.ndjson` to
+rebuild counters without re-firing `onSample`.
+
+**Evals are separate:** the eval harness creates sessions with
+`purpose: "eval"` (playground Evals / `agent-serve eval`). Those skip
+enrollment entirely — no `ab.assigned`, no `onSample`, omitted from
+`GET /v1/abs`. Ordinary chat / `agent-serve run` / Slack stay `"live"`.
+Do not use `splitIf` to filter evals; the framework already does.
+
+After enrollment, arms are on `session.abs` (experiment → variant | `null`)
+for tools/hooks, and optional per-variant `instructions` are appended to the
+agent prompt for that session (a treatment arm can, say, switch `get_weather`
+to °C for its sessions).
+
+### Split helpers
+
+| Helper | Behavior |
+| --- | --- |
+| `splitBySessionHash({ weights?, holdout?, salt? })` | Sticky hash → arm |
+| `splitByRandom({ weights?, holdout? })` | Random once, then sticky |
+| `splitAlways("control")` | Pin all new sessions |
+| `splitNone()` | Skip everyone (pause in place) |
+| `splitIf(pred, inner)` | Skip unless `pred` |
+| `(ctx) => …` | Custom — variant key or `null` |
+
+### Built-in metrics
+
+`turns`, `turnFailures`, `toolCalls`, `toolErrors`, token fields, `wallTimeMs`,
+plus `custom.*` from `derive`. `onSample` fires on `turn.completed` / `turn.failed`.
+
+Playground **A/Bs** tab and `GET /v1/abs` fold the same durable streams into
+per-session assignments and aggregate arm totals (no separate metrics store).
+
+### `derive` cost
+
+`derive` extractors run on **every** session event (same cost model as hooks).
+Keep them synchronous and cheap — no network, no disk, no heavy parsing.
+
+Before wiring: pick experiment name + arm labels, a split helper (and who
+to leave out), and a real `onSample` sink — do not invent credentials.

package/skills/create-agent/SKILL.md

@@ -0,0 +1,219 @@
+---
+name: agent-serve-create-agent
+description: >-
+  Scaffold a new agent-serve agent through a guided AskQuestion interview —
+  purpose, name, runtime, model, channels, MCP connections, capabilities —
+  then verify it serves and hand off to hillclimbing. Read this skill when
+  creating a new agent.
+paths:
+  - packages/agent-serve/**/*
+---
+
+# Create an agent-serve agent
+
+Stand up one new agent project via a short interview, scaffold it, get
+channels working
+with the user, prove a turn runs end-to-end, and set up the improvement loop.
+Sibling skills: `framework-map` for orientation, `setup-slack` for Slack
+wiring, `github` for webhook-driven agents, `evals` for the eval API,
+`ab` for live A/B metrics, `hillclimb` for iteration once the agent runs.
+
+Read `framework-map/SKILL.md` first if you haven't; treat the package
+`AGENTS.md` and `README.md` as ground truth for
+framework behavior. Run the CLI with Node, never Bun (Bun corrupts harness
+tool-result streams): use the installed `agent-serve` bin, or from a source
+checkout `pnpm exec tsx src/bin/agent-serve.ts <command> …` — written as
+`agent-serve …` below.
+
+## Interview
+
+Walk the user through the decisions with the `AskQuestion` tool: batched
+questions, concrete options, the recommended option listed first and labeled
+`(recommended)` so one click accepts it. Rules:
+
+- **Mine the invoking message first.** Never ask a question it already
+  answers. Target: at most two rounds before files appear.
+- **Stage the rounds.** Round 1 fixes identity; round 2 fixes the surface and
+  is tailored by round 1 (no MCP question for a chat toy, no cloud-repos
+  question for local runtime).
+- Use multi-select (`allow_multiple: true`) for channels, MCP connections, and
+  capabilities; single-select elsewhere. Add an "Other" option wherever a
+  custom answer is plausible (name, model, MCP server) — its freeform text is
+  the answer.
+- **Ask about decisions, not implementation.** Instructions wording, tool
+  names, file layout, env prefix, and eval shape are your job — derive them
+  from the purpose and show the result instead of asking.
+- If `AskQuestion` is unavailable or the user says "just go", state the
+  defaults you are assuming in one short list and proceed.
+
+### Round 1 — identity
+
+| Question | Options |
+| --- | --- |
+| **Purpose** | chat assistant · PR / repo automation · domain assistant over APIs/tools · scheduled reporter · Other (describe) |
+| **Name** | 2–3 slug suggestions derived from the purpose + Other. Slug = directory name: `[A-Za-z0-9][A-Za-z0-9_-]*`, not `v1`/`playground`/`docs`; lowercase-hyphenated |
+| **Location** | `./<slug>` (default) · another directory (user names it) |
+
+### Round 2 — surface
+
+| Question | Options | Guidance |
+| --- | --- | --- |
+| **Runtime** | `local` (default) · `cloud` | Local = Cursor SDK harness on this machine; server tools, skills, sandbox seeds, and tool approvals all work. Cloud = Cursor cloud agents; needs `cloud.repos`, no server tools/skills/sandbox seeds (instructions are prepended to the first prompt). Recommend local unless the purpose needs a repo checkout on a cloud VM. |
+| **Model** | `grok-4.5` with `effort=high`, `fast=true` (default) · Other model id | Effort and fast are params, not id suffixes: `{ id: "grok-4.5", params: [{ id: "effort", value: "high" }, { id: "fast", value: "true" }] }` — the SDK rejects suffix-style ids. |
+| **Channels** (multi) | playground + HTTP API only (always mounted) · Slack · GitHub webhooks · custom webhook · schedule (cron) | Slack is Socket Mode with one app + token pair per agent. GitHub verifies `X-Hub-Signature-256` when a secret is set. Discord/Teams packs do not exist yet — offer custom webhook instead. |
+| **MCP connections** (multi) | none · remote server (URL + header auth) · local stdio server · Other | One file per server under `agent/mcp-connections/`; filename = server name; tokens via env vars, never hardcoded. MCP connections serve the model **and** host code (`ctx.host.mcp`, channel `host.mcp`). |
+| **Capabilities** (multi) | server tools · agent tools (scripts) · skills · subagents · hooks · sandbox seed files · tool approvals (HITL) · evals (recommend) | Server tools, skills, sandbox seeds, approvals are local-runtime only. Agent tools (`execution: "agent"`) work on both runtimes. Always recommend at least one smoke eval. |
+
+Close the round with a one-screen plan — chosen options plus the file tree you
+intend to create — and a final gate: "Scaffold it" / "Adjust something". Only
+write files after approval.
+
+## Fill in the blanks
+
+Everything the interview did not fix is yours to decide. Derive it from the
+purpose, and reach for the proven shape instead of
+inventing:
+
+| Purpose looks like | Proven shape |
+| --- | --- |
+| Slack chat assistant | minimal `slackChannel()` pack with suggested prompts |
+| PR review with host-prepared evidence | channel `callTool` prep into a trimmed workspace `pr/` tree |
+| GitHub-webhook automation | `githubChannel()` hooks, per-PR debounce, replay fixtures |
+| Domain assistant over live APIs / MCP | server + agent tools, a stdio MCP connection, a skill, a schedule |
+| Wrap an existing pipeline/tool | GitHub `{ task }` hook driving a deterministic loop under `agent/lib/` |
+
+Defaults that make first cuts good:
+
+- **Minimal surface.** Instructions + 1–2 real tools + the chosen channels +
+  one smoke eval. Unpicked capabilities are follow-ups, not scope.
+- **Instructions** (`agent/instructions.md`, required): a few lines — identity,
+  when to use which tool, output shape. Not an essay.
+- **Tool per file**, filename = tool name the model sees. Prefer
+  `execution: "server"` with a zod `inputSchema`; use `execution: "agent"`
+  (script) only when the tool must run where the agent runs (or runtime is
+  cloud). Gate side-effecting tools with `needsApproval: true`.
+- **Env prefix** for Slack tokens = upper-snake slug (`my-agent` →
+  `MY_AGENT_SLACK_*`); `agent-serve slack init` derives it for you.
+- **Host prep beats model wandering:** when the purpose has a deterministic
+  setup step (fetch a PR, seed files), do it in the channel handler via
+  `callTool` / `workspaceFiles` rather than instructing the model to do it.
+
+## Scaffold
+
+```bash
+agent-serve init ./<slug>
+```
+
+`init` writes `package.json`, `agent/agent.ts`, `agent/instructions.md`, and a
+demo `agent/tools/echo.ts`; it refuses to overwrite existing files. Then shape
+it to the plan:
+
+1. `tsconfig.json` — add one so the type-checker covers the project.
+2. `agent/agent.ts` — chosen model/runtime via `defineAgent({...})`; add the
+   `cloud: { repos: [...] }` block for cloud runtime.
+3. `agent/instructions.md` — real instructions; delete or replace `echo.ts`
+   with the real tools.
+4. Add the chosen `channels/`, `mcp-connections/`, `skills/`, `subagents/<id>/`
+   (needs `description`), `schedules/`, `hooks/`, `sandbox/workspace/`. For
+   Slack, do not hand-write the channel: `agent-serve slack init --dir
+   ./<slug> --name "<Name>"` generates `agent/channels/slack.ts` (with
+   the env prefix) plus manifests and `env.example`; customize the generated
+   file (e.g. `suggestedPrompts`) afterwards.
+5. `evals/evals.config.ts` — `defineEvalConfig({ maxConcurrency: 20 })`
+   (playground runs are memory-only unless you add `persistRuns`)
+   (`maxConcurrency` is required; hard-capped at 200 due to model provider
+   request limits) plus eval cases under `evals/**/*.eval.ts` — prefer one
+   suite file with `cases: [{ id, test }, …]` for related datapoints, or a
+   single `test` for one smoke. Assert `t.succeeded()` + `t.calledTool(...)`
+   (case id = file path, or `<fileId>/<case.id>`). Full assertion API and
+   fixture strategy: `evals/SKILL.md` (sibling skill).
+
+Stick to deps agent-serve already
+ships (`zod`, `@modelcontextprotocol/sdk`, `tsx`); a new npm dep needs its own
+install story and is a smell for a first cut.
+
+## Verify
+
+No API key needed for the structural half — run these first and fix every
+error diagnostic:
+
+```bash
+agent-serve validate --dir ./<slug>            # zero errors (warnings explain runtime mismatches)
+agent-serve info --dir ./<slug> --json         # surface matches the plan
+agent-serve call <tool> --dir ./<slug> --input '{…}'   # server tools, deterministic, no model
+agent-serve eval --dir ./<slug> --list
+npx tsc --noEmit -p ./<slug>                   # or the project's own type-check task
+```
+
+The typecheck matters: everything above runs through tsx, which strips types
+without checking them. The classic miss is a tool `execute` return that does
+not satisfy `ToolExecuteResult` — returns must be JSON-shaped
+(`ToolResultValue`), so use plain object literals or `type` aliases, not
+`interface` types (their lack of an index signature fails assignability).
+
+`serve` also boots without a key (good for checking channel mounts and the
+playground shell), but model turns need `CURSOR_API_KEY` — ask the user for
+one if the environment lacks it (minted in the Cursor dashboard). If no key
+is forthcoming, finish every key-free check, confirm `run` fails with only
+the clean API-key error, and hand these to the user as their next steps:
+
+```bash
+agent-serve run --dir ./<slug> --message "<fixture prompt>"   # JSON trajectory
+agent-serve serve --dir ./<slug> --dev
+# playground: http://127.0.0.1:3000/<slug>/playground
+agent-serve eval --dir ./<slug>
+```
+
+Serve only the new agent's directory during bring-up — pointing `serve` at a
+parent folder also mounts every sibling agent it contains.
+Runtime state lands in `<project>/.agent-serve/` (gitignored); keep durable
+`--state-root` outside the monorepo so sessions don't inherit ambient
+`AGENTS.md`.
+
+## Channel setup with the user
+
+Each surface beyond the playground has a human gate — drive the user through
+it, verify with doctor commands, and smoke-test before declaring victory.
+
+- **Slack** — follow `packages/agent-serve/skills/setup-slack/SKILL.md` end
+  to end (init → human creates apps → doctor → `@mention` smoke).
+- **GitHub** — follow `packages/agent-serve/skills/github/SKILL.md`
+  (fixtures / replay / forward / `--cursor-events`).
+- **Custom webhook** — `POST /<slug>/v1/channels/<id>/<route>`. Default
+  `localDevStrict()` is loopback-only; add `bearerAuth(...)` before any
+  tunnel or shared host.
+- **Schedules** — never auto-fire under `--dev`; dispatch via
+  `POST /<slug>/v1/dev/schedules/<id>`.
+
+## Hand off to hillclimbing
+
+Scaffolding proves the agent runs; it says nothing about quality. Once the
+smoke turn passes, propose the improvement loop — one last `AskQuestion` if
+choices are open:
+
+1. **Fixtures** — 1–3 fixed inputs worth optimizing against (a real PR URL, a
+   canonical Slack question, a saved webhook payload).
+2. **Success criteria** — what "better" means first (correct tool choice,
+   fewer tool calls, faster wall time, output quality).
+3. **Freeze line** — what must not change (API contract, output shape).
+
+Those are exactly the preconditions in `skills/hillclimb/SKILL.md`; suggest
+the user (or you) follow that skill with the fixtures you agreed on. Seed
+the loop by pointing
+at the smoke run's session (`.agent-serve/<slug>/sessions/<id>/events.ndjson`
+or the trace under `.agent-serve/traces/`) as the baseline measurement — or,
+when no API key was available, name the user's first real turn as the
+baseline instead. For GitHub agents, snapshot replay fixtures now
+(`agent-serve github replay ... --dry-run --out fixtures/github`) so the
+loop starts deterministic.
+
+## Working agreements
+
+- Interview → plan gate → scaffold → verify → channel setup → hillclimb
+  handoff. Do not skip verification because the files "look right".
+- Fill blanks yourself; re-ask only when an answer contradicts the framework
+  (e.g. server tools on cloud runtime — `validate` warns about these).
+- Do not deploy, post to real Slack/GitHub surfaces beyond the smoke tests the
+  user asked for, or commit without being asked.
+- Keep the first cut reviewable: if the plan grows past ~10 files, cut scope
+  and say what you deferred.

package/skills/debug/SKILL.md

@@ -0,0 +1,71 @@
+---
+name: agent-serve-debug
+description: >-
+  Diagnose a misbehaving agent-serve agent, server, or playground — blank
+  playground, sessions missing, HMR not reloading, failing reads/retry
+  loops, webhook 401s, 403/409 responses, approvals or reminders lost,
+  orphaned processes. Use when an agent-serve project runs but behaves
+  wrong locally.
+---
+
+# Debugging agent-serve locally
+
+Read `framework-map/SKILL.md` (sibling skill) first if you don't know the
+project structure or session model. Everything below assumes Node/tsx (`pnpm exec tsx
+src/bin/agent-serve.ts ...` in the everysphere monorepo).
+
+## Triage in order
+
+```bash
+# 1. Is discovery clean? serve refuses to start on error diagnostics.
+agent-serve validate --dir <project>
+
+# 2. What is actually running, and where?
+lsof -iTCP:3000 -sTCP:LISTEN; lsof -iTCP:5273 -sTCP:LISTEN
+ps aux | rg -i 'agent-serve|vite' | rg -v rg
+
+# 3. What does the server think it serves?
+curl -s localhost:3000/v1/agents | jq .        # multi-agent index
+curl -s localhost:3000/<slug>/v1/info | jq .   # per-agent surface
+
+# 4. What did the session actually do? The stream never lies.
+curl -sN 'localhost:3000/<slug>/v1/session/<id>/stream?startIndex=0' | head -50
+# or read <stateRoot>/<slug>/sessions/<id>/events.ndjson directly
+```
+
+## Symptom → cause
+
+| Symptom | Cause / fix |
+| --- | --- |
+| Playground blank or "no agents" even though Vite assets are built | The SPA is static; it needs the agent-serve backend on `:3000`. Start `serve` — building `dist/playground/` alone serves nothing. |
+| Edits under `playground/src` don't show up in the browser | You're on the **static** `dist/playground` bundle, not HMR. Only `serve --dev` starts Vite HMR (`:5273`) and prints `playground (HMR): ...` — open that URL, not `:3000`. In the monorepo, `mise //packages/agent-serve:start` = all examples + multi-agent HMR; pin one slug with `AGENT_SERVE_BASE=/<slug>`. |
+| Webhook / schedule sessions exist on disk but the playground session list is empty | The list shows the calling principal's sessions. `--dev` (loopback) or `--allow-anonymous` (trusted shared host) switches it to `includeAll`. Otherwise deep-link `/<slug>/playground?sessionId=ses_...` or read `sessions/` on disk. |
+| Every built-in read/grep the model makes fails; turns crawl through retry loops | You ran the CLI under **Bun**. Kill it, rerun under Node/tsx. (`NGHTTP2_FRAME_SIZE_ERROR` in SDK logs is the tell.) |
+| `gh webhook forward` / `agent-serve github forward` deliveries all 401 — but hook creation succeeded | `GITHUB_TOKEN`/`GH_TOKEN` in the env. The relay authenticates with the gh CLI login and rejects env tokens. `GITHUB_TOKEN= GH_TOKEN= agent-serve github forward ...` |
+| `Hook already exists` starting a forwarder | GitHub allows one forwarder per repo. Use one `github forward --dir <parent>` (it fans out to every matching channel) instead of N processes; kill stale forwarders. |
+| Agent's answers reference monorepo rules / AGENTS.md it shouldn't know | Session workspace sits inside the monorepo, so the harness loaded ancestor config. `defineAgent({ local: { cwd } })` outside the repo, or `--state-root` under `/tmp`. |
+| Port 3000/5273 in use; stray processes after crashes | `lsof -iTCP:3000 -sTCP:LISTEN`, kill the pids; also check companion processes (e.g. vite). |
+| Pending tool approval vanished after server restart | By design: parked SDK calls don't survive host restart; they resolve as interrupted. Re-run the turn. |
+| Schedule / reminder never fires under `--dev` | Dev never auto-fires. `POST /<slug>/v1/dev/schedules/<id>` or `POST /<slug>/v1/dev/reminders/<id>` (list at `GET /v1/dev/reminders`). |
+| Reminder disarmed after restart with `handler_lost_on_restart` | `run`-handler reminders are in-memory; re-arm them from the code path that created them (enroll hook / policy), or use prompt-based reminders. |
+| `409` on a follow-up | Stale `continuationToken` (each accepted follow-up rotates it), busy session, or a task/schedule session (not followable). |
+| `409 session_busy` on `agent-serve call --session` | Session-bound deterministic calls serialize with model turns; wait or use an ephemeral call (drop `--session`). |
+| `403` on stream/follow-up | Caller principal ≠ session owner. In dev, create and inspect with the same auth; beyond loopback pass `--bearer-token` and send it. |
+| Works on localhost, 401/blocked through a tunnel or LAN | Default `localDevStrict()` auth only admits direct loopback, **rejects proxy-forwarding headers** (`X-Forwarded-For`, ...), and requires a loopback `Host`. Use `--bearer-token <secret>` (or authored `bearerAuth`) — `--allow-anonymous` only for trusted-network demos (and never with Cursor account MCP connections). |
+| Channel route won't compile: body/query schema type error | `GET` requires a Zod `querySchema`, `POST`/`PUT`/`PATCH` a Zod `bodySchema` — plain JSON Schema objects don't type-check. Use `z.object({})` / `z.unknown()` for open surfaces. Empty POST bodies are coerced to `{}` before validation. |
+| Slack channel prints `channel idle ... missing credentials` | Expected: tokens absent. Multi-agent needs `<PREFIX>_SLACK_BOT_TOKEN` + `<PREFIX>_SLACK_APP_TOKEN` per agent. `agent-serve slack doctor --prefix <PREFIX>`. |
+| Turn fails immediately with an API-key error | Model turns need `CURSOR_API_KEY`; everything structural (validate/info/call/serve bring-up) doesn't. |
+| Server tools / skills / sandbox silently absent | Runtime is `cloud` — those are local-only. `validate` prints exactly this warning; read it. |
+| `validate` clean, `run` works, CI typecheck fails | tsx never typechecked it. See invariant 3 in `framework-map/SKILL.md` (JSON-shaped tool returns; `type` not `interface`). |
+
+## Reading a turn like an expert
+
+- `actions.requested` / `action.result` pairs are the tool trajectory;
+  count them by `toolName` before blaming latency, and split **host** work
+  (channel `callTool`, prep) from **model** tools.
+- `turn.failed` with `"turn interrupted"` means a follow-up or `/stop`
+  preempted it — that's the designed behavior, not a crash.
+- Escapes outside the session workspace in read/grep paths mean the
+  harness is fighting your evidence layout, not that the model is broken.
+- `agent-serve trajectory --events <file>` renders any saved NDJSON; the
+  playground "Open trace" does the same visually.

package/skills/evals/SKILL.md

@@ -0,0 +1,217 @@
+---
+name: agent-serve-evals
+description: >-
+  Author and run agent-serve defineEval cases (single- or multi-datapoint).
+  Use when writing, fixing, seeding, or hillclimbing evals; AskQuestion
+  whether to generate samples or upload data. Materialize API-backed
+  fixtures first. Live A/B metrics: defineAB (skills/ab), not defineEval.
+---
+
+# agent-serve evals
+
+Evals are the ratchet that makes iteration trustworthy: a fixed input, a
+model turn, and gates over the recorded trajectory. They live at the
+**project root** `evals/**/*.eval.ts` (never `agent/evals/` — that slot is
+ignored).
+
+For **live A/B metrics** while the agent serves traffic, use `defineAB`
+under `agent/ab` (`packages/agent-serve/skills/ab/SKILL.md`). That plug-in
+enrolls via sticky `split` and calls `onSample`; it is not a test runner
+and does not replace `defineEval` gates.
+
+**Identity**
+
+- File id = path under `evals/` without `.eval.ts`
+  (`evals/weather.eval.ts` → `weather`).
+- Single datapoint (`test`): case id = file id
+  (`evals/weather/nyc.eval.ts` → `weather/nyc`).
+- Multiple datapoints (`cases`): case id = `<fileId>/<case.id>`
+  (`evals/weather.eval.ts` + `{ id: "nyc" }` → `weather/nyc`).
+
+```bash
+agent-serve eval --dir . --list              # all datapoints
+agent-serve eval --dir . --json              # run all
+agent-serve eval --dir . weather/nyc         # one datapoint
+agent-serve eval --dir . weather             # every datapoint in that file
+agent-serve eval --dir . weather forecast    # several files
+agent-serve eval --dir . --tag smoke
+agent-serve eval --dir . --verbose
+```
+
+`eval` (like `run`) boots an ephemeral server on port 0 with a temp state
+root outside the project, so cases don't inherit ambient monorepo rules and
+don't pollute `.agent-serve/`. Point `--url` at a running server instead to
+eval a live agent. Model turns require `CURSOR_API_KEY`.
+
+## Seeding / adding samples (required fork)
+
+Whenever the task is to **create, seed, or expand** eval cases (not merely
+run or tweak an existing gate), start with the `AskQuestion` tool. Do not
+assume a source of fixtures.
+
+### Step 1 — who supplies the samples?
+
+Use `AskQuestion` (single-select). Put the recommended option first and
+label it `(recommended)`. Do **not** put code blocks inside the question
+prompt (they render poorly); keep the prompt short plain text.
+
+| Question | Options |
+| --- | --- |
+| **How should we get eval samples?** | Generate test eval samples for me `(recommended)` · I will add / upload the data manually |
+
+If `AskQuestion` is unavailable, ask the same fork in plain chat and wait.
+
+### Step 2a — user chose manual
+
+Tell the user the **shape** of the data (see “The API” below) and **how to
+upload it**. Do not invent cases until they provide files (or paste content
+to write). After they add data, run `agent-serve eval --dir <project>
+--list` to confirm discovery, then wire any missing gates.
+
+If the uploaded data is **API-backed** (PR URLs, pinned SHAs, gold labels,
+snapshot ids) rather than self-contained prompts/diffs, **materialize
+fixtures first** — see “When eval data needs API calls to render” below.
+
+**How to upload:**
+
+1. Add or edit `.eval.ts` files under the project’s `evals/` directory.
+2. Ensure `evals/evals.config.ts` exists with `maxConcurrency`.
+3. Optional fixtures beside the case or under `fixtures/`.
+4. Tell the agent once the files are in place — or paste case bodies in chat
+   and ask the agent to write the file.
+5. Verify: `agent-serve eval --dir <project> --list` shows the new ids.
+
+### Step 2b — user chose agent-generated samples
+
+1. Use `AskQuestion` again for count (single-select), recommended first:
+
+   | Question | Options |
+   | --- | --- |
+   | **How many test samples should I create?** | `3` `(recommended)` · `5` · `10` · Other (enter a number) |
+
+2. Inspect the agent (tools, instructions, existing `evals/`) and invent
+   that many **distinct** datapoints: frozen prompts + decision/shape gates,
+   not exact prose. Follow “What good cases assert” below.
+3. **Append** — prefer adding entries to an existing file’s `cases` array
+   when the suite fits; otherwise create a new `evals/<suite>.eval.ts` with
+   `cases`. Never overwrite or weaken an existing datapoint. Case ids must
+   not collide with `agent-serve eval --dir <project> --list`.
+4. Create `evals/evals.config.ts` if missing (`maxConcurrency: 20` is fine
+   for now; hard limit 200 for model provider request limits).
+5. Re-list to confirm, then optionally run `--tag smoke` if you tagged any.
+
+## The API
+
+```ts
+import { defineEval, includes, equals, satisfies } from "@anysphere/agent-serve/evals";
+
+// Single datapoint (case id = file path under evals/)
+export default defineEval({
+  description: "Uses get_weather for temperature questions.",
+  tags: ["smoke"],
+  timeoutMs: 120_000,
+  async test(t) {
+    await t.send("What's the temperature in NYC?");
+    t.succeeded();
+    t.calledTool("get_weather");
+    t.notCalledTool("save_weather_note");
+    t.messageIncludes(/°|[FC]/);
+    t.check(t.reply, includes("NYC"));
+  },
+});
+
+// Multiple datapoints in one file (ids = <fileId>/<case.id>)
+export default defineEval({
+  tags: ["smoke", "weather"],
+  cases: [
+    {
+      id: "nyc",
+      description: "NYC temperature.",
+      async test(t) {
+        await t.send("What's the temperature in NYC?");
+        t.succeeded();
+        t.calledTool("get_weather");
+        t.check(t.reply, includes(/°|[FC]/));
+      },
+    },
+    {
+      id: "paris",
+      async test(t) {
+        await t.send("What's the weather in Paris?");
+        t.succeeded();
+        t.calledTool("get_weather");
+      },
+    },
+  ],
+});
+```
+
+Project-wide defaults in `evals/evals.config.ts` (required once):
+
+```ts
+import {
+  defineEvalConfig,
+  persistEvalRunsToDir,
+} from "@anysphere/agent-serve/evals";
+
+export default defineEvalConfig({
+  maxConcurrency: 20, // required; hard-capped at 200 (model provider limits)
+  // Playground batches are memory-only unless you set persistRuns:
+  // maxPlaygroundRuns: 50, // optional; default 20; playground history only
+  // persistRuns: persistEvalRunsToDir(".agent-serve/eval-runs"),
+});
+```
+
+- Either a single `test(t)` **or** a `cases: [{ id, test }, …]` array — not
+  both. Prefer `cases` for multiple related datapoints.
+- `t.send(message, options?)` runs one turn and waits for park/fail.
+  Options: `workspaceDir`, `workspaceFiles` (`{ path: contents }` seeded
+  before the first turn). Multiple `t.send` calls share the session.
+- `t.reply` / `t.events` / `t.sessionId` for assertions and correlation.
+- Builders: `includes(string | RegExp)`, `equals(value)`,
+  `satisfies(predicate, label)`.
+
+## What good cases assert
+
+Gate **decisions and shape, not prose**. Model wording varies run to run;
+tool choice, tool avoidance, and output structure shouldn't.
+
+1. `t.succeeded()` — always, first.
+2. The tool decision: `calledTool` for the intended path, `notCalledTool`
+   for the tempting wrong one (the pair is stronger than either alone).
+3. Output shape: a regex for the contract (`/°|[FC]/`, a JSON marker, a
+   findings-block fence) — not exact sentences.
+4. For structured output, parse `t.reply` and `t.check` fields with
+   `satisfies` rather than substring-matching JSON.
+
+Anti-patterns: asserting exact phrasing; more than ~5 gates in one case
+(split it); cases that depend on live external state that drifts (pin the
+input instead — see fixtures).
+
+## Fixture strategy by agent type
+
+| Agent surface | Fixture source |
+| --- | --- |
+| Chat / domain assistant | A canonical prompt string, chosen once and frozen |
+| Tool-heavy | `agent-serve call <tool> --dir . --input '{...}'` first, then the prompt that should trigger it |
+| GitHub webhook | `github replay … --dry-run --out fixtures/github` (see `packages/agent-serve/skills/github/SKILL.md`) |
+| PR reviewer with host prep | A small fixed PR the team controls; assert on findings shape, not counts |
+| Workspace-dependent | `workspaceFiles` in `t.send` — never machine-local paths |
+
+### When eval data needs API calls to render (rate limits)
+
+If samples are **pointers** (PR URL, SHAs, `gold_labels`) rather than
+self-contained evidence, do **not** wire live `t.send(prUrl)` matrices.
+Materialize once under `fixtures/` at pinned SHAs, point cases at those
+files, gate on decision/shape against gold, and keep a tiny `--tag smoke`
+subset for any full-pipeline runs.
+
+## The ratchet rule (hillclimbing)
+
+Every hillclimb round that **keeps** a change must land an eval that would
+have failed before the change. Never weaken a gate to make a round pass.
+
+```ts
+const toolResults = t.events.filter((e) => e.type === "action.result");
+t.check(toolResults.length, satisfies((n) => (n as number) <= 4, "≤4 tool calls"));
+```