@cursor/july 0.1.94 → 0.1.96

package/skills/debug/SKILL.md

@@ -1,78 +1,44 @@
 ---
 name: agentsdk-debug
 description: >-
-  Diagnose a misbehaving Agent SDK agent, server, or playground — blank
-  playground, sessions missing, HMR not reloading, failing reads/retry
-  loops, webhook 401s, 403/409 responses, approvals or reminders lost,
-  orphaned processes. Use when an Agent SDK project runs but behaves
-  wrong locally.
+  Diagnose a misbehaving local Agent SDK agent: blank playground,
+  missing sessions, failed reads, webhook 401s, 403/409, lost
+  approvals. Use when a project runs but behaves wrong locally.
 ---
 
 # Debugging the Agent SDK locally
 
-Public docs index: `node_modules/@cursor/july/dist/docs/llms.txt`. A running
-host serves the same index at `/docs/llms.txt`.
+Local `agent-sdk serve`. Hosted start or health failures:
+`docs/troubleshooting.md` and `docs/deployment.md`.
 
-This skill is **local** `agent-sdk serve` / playground. A **hosted**
-deployment that is stopped, slow to start, or failing `/v1/health` is
-`.cursor/skills/debugging/debug-agent-serve-deployment/SKILL.md`.
+1. **Validate.** `agent-sdk validate --dir <project>`
+2. **Info.** `curl -s http://127.0.0.1:3000/<slug>/v1/info | jq .`
+3. **Trace.** Session events under the project state directory.
+   Then match the table.
 
-Read `framework-map/SKILL.md` (sibling skill) first if you don't know the
-project structure or session model. Everything below assumes Node/tsx (`pnpm exec tsx
-src/bin/agent-serve.ts ...` in the everysphere monorepo).
-
-## Triage in order
-
-```bash
-# 1. Is discovery clean? serve refuses to start on error diagnostics.
-agent-sdk validate --dir <project>
-
-# 2. What is actually running, and where?
-lsof -iTCP:3000 -sTCP:LISTEN; lsof -iTCP:5273 -sTCP:LISTEN
-ps aux | rg -i 'agent-serve|vite' | rg -v rg
-
-# 3. What does the server think it serves?
-curl -s localhost:3000/v1/agents | jq .        # multi-agent index
-curl -s localhost:3000/<slug>/v1/info | jq .   # per-agent surface
-
-# 4. What did the session actually do? The stream never lies.
-curl -sN 'localhost:3000/<slug>/v1/session/<id>/stream?startIndex=0' | head -50
-# or read <stateRoot>/<slug>/sessions/<id>/events.ndjson directly
-```
-
-## Symptom → cause
-
-| Symptom | Cause / fix |
+| Symptom | Fix |
 | --- | --- |
-| Playground blank or "no agents" even though Vite assets are built | The SPA is static; it needs the Agent SDK backend on `:3000`. Start `serve` — building `dist/playground/` alone serves nothing. |
-| Edits under `playground/src` don't show up in the browser | You're on the **static** `dist/playground` bundle, not HMR. Only `serve --dev` / `dev` starts Vite HMR (`:5273`) and prints that as `playground` — open that URL, not `:3000`. In the monorepo, `mise //packages/agent-serve:start` = all examples + multi-agent HMR; pin one slug with `AGENT_SERVE_BASE=/<slug>`. |
-| Webhook / schedule sessions exist on disk but the playground session list is empty | The list shows the calling principal's sessions. `--dev` (loopback) or `--allow-anonymous` (trusted shared host) switches it to `includeAll`. Otherwise deep-link `/<slug>/playground?sessionId=ses_...` or read `sessions/` on disk. |
-| Every built-in read/grep the model makes fails; turns crawl through retry loops | You ran the CLI under **Bun**. Kill it, rerun under Node/tsx. (`NGHTTP2_FRAME_SIZE_ERROR` in SDK logs is the tell.) |
-| `gh webhook forward` / `agent-sdk github forward` deliveries all 401 — but hook creation succeeded | `GITHUB_TOKEN`/`GH_TOKEN` in the env. The relay authenticates with the gh CLI login and rejects env tokens. `GITHUB_TOKEN= GH_TOKEN= agent-sdk github forward ...` |
-| `Hook already exists` starting a forwarder | GitHub allows one forwarder per repo. Use one `github forward --dir <parent>` (it fans out to every matching channel) instead of N processes; kill stale forwarders. |
-| Agent's answers reference monorepo rules / AGENTS.md it shouldn't know | Session workspace sits inside another git checkout, so the harness loaded ancestor config. Nested projects default `local.cwd` to `~/.cache/agent-serve/<dir>`. Point cwd at a checkout only when the agent should inherit that tree. |
-| Yellow box lists Datadog/Linear tools; model recites `GetDynamicTools` / IDE `cursor` tools and never calls MCP | Attached MCP is behind meta-tools, or cwd still leaked IDE MCP. Set `advertiseTools: true` for named tools on local turns. Confirm `GET /v1/info` `local.cwd` and `connections[].advertiseTools`. |
-| Port 3000/5273 in use; stray processes after crashes | `lsof -iTCP:3000 -sTCP:LISTEN`, kill the pids; also check companion processes (e.g. vite). |
-| Pending tool approval vanished after server restart | By design: parked SDK calls don't survive host restart; they resolve as interrupted. Re-run the turn. |
-| Schedule / reminder never fires under `--dev` | Dev never auto-fires. `POST /<slug>/v1/dev/schedules/<id>` or `POST /<slug>/v1/dev/reminders/<id>` (list at `GET /v1/dev/reminders`). |
-| Reminder disarmed after restart with `handler_lost_on_restart` | `run`-handler reminders are in-memory; re-arm them from the code path that created them (enroll hook / policy), or use prompt-based reminders. |
-| `409` on a follow-up | Stale `continuationToken` (each accepted follow-up rotates it), busy session, or a task/schedule session (not followable). |
-| `409 session_busy` on `agent-sdk call --session` | Session-bound deterministic calls serialize with model turns; wait or use an ephemeral call (drop `--session`). |
-| `403` on stream/follow-up | Caller principal ≠ session owner. In dev, create and inspect with the same auth; beyond loopback pass `--bearer-token` and send it. |
-| Works on localhost, 401/blocked through a tunnel or LAN | Default `localDevStrict()` auth only admits direct loopback, **rejects proxy-forwarding headers** (`X-Forwarded-For`, ...), and requires a loopback `Host`. Use `--bearer-token <secret>` (or authored `bearerAuth`) — `--allow-anonymous` only for trusted-network demos (and never with Cursor account MCP connections). |
-| Channel route won't compile: body/query schema type error | `GET` requires a Zod `querySchema`, `POST`/`PUT`/`PATCH` a Zod `bodySchema` — plain JSON Schema objects don't type-check. Use `z.object({})` / `z.unknown()` for open surfaces. Empty POST bodies are coerced to `{}` before validation. |
-| Slack channel prints `channel idle ... missing credentials` | Expected: tokens absent. Multi-agent needs `<PREFIX>_SLACK_BOT_TOKEN` + `<PREFIX>_SLACK_APP_TOKEN` per agent. `agent-sdk slack doctor --prefix <PREFIX>`. |
-| Turn fails immediately with an API-key error | Model turns need `CURSOR_API_KEY`; everything structural (validate/info/call/serve bring-up) doesn't. |
-| Server tools / skills / sandbox silently absent | Runtime is `cloud` — those are local-only. `validate` prints exactly this warning; read it. |
-| `validate` clean, `run` works, CI typecheck fails | tsx never typechecked it. See invariant 3 in `framework-map/SKILL.md` (JSON-shaped tool returns; `type` not `interface`). |
-
-## Reading a turn like an expert
-
-- `actions.requested` / `action.result` pairs are the tool trajectory;
-  count them by `toolName` before blaming latency, and split **host** work
-  (channel `callTool`, prep) from **model** tools.
-- `turn.failed` with `"turn interrupted"` means a follow-up or `/stop`
-  preempted it — that's the designed behavior, not a crash.
-- Escapes outside the session workspace in read/grep paths mean the
-  harness is fighting your evidence layout, not that the model is broken.
-- `agent-sdk trajectory --events <file>` renders any saved NDJSON.
+| Playground blank / "no agents" | Start `serve`. A built SPA with no backend serves nothing. |
+| Playground UI edits do not show | Open the URL `serve --dev` prints as `playground`, not `:3000`. |
+| Sessions on disk, empty playground list | List is the calling principal. `--dev` or `--allow-anonymous` shows all. Or `?sessionId=ses_...`. |
+| Built-in read/grep fail; retry loops | Bun. Rerun under Node. The tell is `NGHTTP2_FRAME_SIZE_ERROR`. |
+| `github forward` 401s; hook created | Blank `GITHUB_TOKEN`/`GH_TOKEN`. Relay uses `gh` login. |
+| `Hook already exists` | One forwarder per repo. `forward --dir <parent>`. |
+| Answers cite ancestor `AGENTS.md` | Nested checkout. Default `local.cwd` is a cache directory under `~/.cache`. |
+| Model lists IDE `cursor` tools, never MCP | `advertiseTools: true`. Check `GET /v1/info`. |
+| Port 3000 in use | `lsof -iTCP:3000 -sTCP:LISTEN` and kill that pid. |
+| Approval vanished after restart | Parked calls do not survive restart. Re-run. |
+| Schedule / reminder silent under `--dev` | Dev never auto-fires. `POST /<slug>/v1/dev/schedules/<id>`. |
+| `409` on follow-up | Stale `continuationToken`, busy session, or a task session. |
+| `409 session_busy` on `call --session` | Wait, or drop `--session`. |
+| `403` on stream | Wrong principal. Same auth as create; beyond loopback send `--bearer-token`. |
+| Works on loopback, blocked via tunnel | `localDevStrict()` rejects forwarded headers. Use `--bearer-token`. Never `--allow-anonymous` with account MCP. |
+| Slack `channel idle … missing credentials` | Expected. `slack doctor --prefix <PREFIX>`. |
+| Immediate API-key error | Model turns need `CURSOR_API_KEY`. |
+| Approvals or sandbox seeds missing | `runtime: "cloud"`. Those are local only. `validate` warns. |
+| `validate` clean, CI typecheck fails | tsx skipped types. JSON-shaped returns; `type` not `interface`. |
+
+Count `action.result` by `toolName` before blaming latency.
+`turn.failed` + `"turn interrupted"` is a follow-up or stop, not a
+crash. `agent-sdk trajectory --events <file>` renders a saved
+trace.

package/skills/evals/SKILL.md

@@ -1,129 +1,58 @@
 ---
 name: agentsdk-evals
 description: >-
-  Author and run Agent SDK defineEval cases (single- or multi-datapoint).
-  Use when writing, fixing, seeding, or hillclimbing evals; AskQuestion
-  whether to generate samples or upload data. Materialize API-backed
-  fixtures first. Live A/B metrics: defineAB (skills/ab), not defineEval.
+  Author and run defineEval cases. Use when writing, fixing, seeding,
+  or hillclimbing evals. Ask whether to generate samples or take
+  uploaded data. Materialize API-backed fixtures first. Live A/B is
+  defineAB (skills/ab), not defineEval.
 ---
 
 # Agent SDK evals
 
-Public docs index: `node_modules/@cursor/july/dist/docs/llms.txt`. A running
-host serves the same index at `/docs/llms.txt`.
+Fixed input, model turn, gates on the trajectory. Files live at
+project-root `evals/**/*.eval.ts`. `agent/evals/` is ignored.
 
-Evals are the ratchet that makes iteration trustworthy: a fixed input, a
-model turn, and gates over the recorded trajectory. They live at the
-**project root** `evals/**/*.eval.ts` (never `agent/evals/` — that slot is
-ignored).
-
-For **live A/B metrics** while the agent serves traffic, use `defineAB`
-under `agent/ab` (`packages/agent-serve/skills/ab/SKILL.md`). That plug-in
-enrolls via sticky `split` and calls `onSample`; it is not a test runner
-and does not replace `defineEval` gates.
-
-**Identity**
-
-- File id = path under `evals/` without `.eval.ts`
-  (`evals/weather.eval.ts` → `weather`).
-- Single datapoint (`test`): case id = file id
-  (`evals/weather/nyc.eval.ts` → `weather/nyc`).
-- Multiple datapoints (`cases`): case id = `<fileId>/<case.id>`
-  (`evals/weather.eval.ts` + `{ id: "nyc" }` → `weather/nyc`).
+Live traffic variants: `skills/ab/SKILL.md`. That is not a test
+runner.
 
 ```bash
-agent-sdk eval --dir . --list              # all datapoints
-agent-sdk eval --dir . --json              # run all
-agent-sdk eval --dir . weather/nyc         # one datapoint
-agent-sdk eval --dir . weather             # every datapoint in that file
-agent-sdk eval --dir . weather forecast    # several files
+agent-sdk eval --dir . --list
+agent-sdk eval --dir . --json
+agent-sdk eval --dir . weather/nyc
 agent-sdk eval --dir . --tag smoke
-agent-sdk eval --dir . --verbose
 ```
 
-`eval` (like `run`) boots an ephemeral server on port 0 with a temp state
-root outside the project, so cases don't inherit ambient monorepo rules and
-don't pollute `.agent-serve/`. Point `--url` at a running server instead to
-eval a live agent. Model turns require `CURSOR_API_KEY`.
-
-## Seeding / adding samples (required fork)
+| Form | Case id |
+| --- | --- |
+| `evals/weather.eval.ts` + `test` | `weather` |
+| `evals/weather/nyc.eval.ts` + `test` | `weather/nyc` |
+| `evals/weather.eval.ts` + `{ id: "nyc" }` | `weather/nyc` |
 
-Whenever the task is to **create, seed, or expand** eval cases (not merely
-run or tweak an existing gate), start with the `AskQuestion` tool. Do not
-assume a source of fixtures.
+`eval` boots an ephemeral server and a temp state root. `--url`
+points at a running agent. Model turns need `CURSOR_API_KEY`.
 
-### Step 1 — who supplies the samples?
+## Seeding
 
-Use `AskQuestion` (single-select). Put the recommended option first and
-label it `(recommended)`. Do **not** put code blocks inside the question
-prompt (they render poorly); keep the prompt short plain text.
+Creating or expanding cases: `AskQuestion` first.
 
 | Question | Options |
 | --- | --- |
 | **How should we get eval samples?** | Generate test eval samples for me `(recommended)` · I will add / upload the data manually |
 
-If `AskQuestion` is unavailable, ask the same fork in plain chat and wait.
-
-### Step 2a — user chose manual
-
-Tell the user the **shape** of the data (see “The API” below) and **how to
-upload it**. Do not invent cases until they provide files (or paste content
-to write). After they add data, run `agent-sdk eval --dir <project>
---list` to confirm discovery, then wire any missing gates.
-
-If the uploaded data is **API-backed** (PR URLs, pinned SHAs, gold labels,
-snapshot ids) rather than self-contained prompts/diffs, **materialize
-fixtures first** — see “When eval data needs API calls to render” below.
-
-**How to upload:**
-
-1. Add or edit `.eval.ts` files under the project’s `evals/` directory.
-2. Ensure `evals/evals.config.ts` exists with `maxConcurrency`.
-3. Optional fixtures beside the case or under `fixtures/`.
-4. Tell the agent once the files are in place — or paste case bodies in chat
-   and ask the agent to write the file.
-5. Verify: `agent-sdk eval --dir <project> --list` shows the new ids.
-
-### Step 2b — user chose agent-generated samples
-
-1. Use `AskQuestion` again for count (single-select), recommended first:
-
-   | Question | Options |
-   | --- | --- |
-   | **How many test samples should I create?** | `3` `(recommended)` · `5` · `10` · Other (enter a number) |
-
-2. Inspect the agent (tools, instructions, existing `evals/`) and invent
-   that many **distinct** datapoints: frozen prompts + decision/shape gates,
-   not exact prose. Follow “What good cases assert” below.
-3. **Append** — prefer adding entries to an existing file’s `cases` array
-   when the suite fits; otherwise create a new `evals/<suite>.eval.ts` with
-   `cases`. Never overwrite or weaken an existing datapoint. Case ids must
-   not collide with `agent-sdk eval --dir <project> --list`.
-4. Create `evals/evals.config.ts` if missing (`maxConcurrency: 20` is fine
-   for now; hard limit 200 for model provider request limits).
-5. Re-list to confirm, then optionally run `--tag smoke` if you tagged any.
+1. **Manual.** They provide files or paste. Show the shape below.
+   Do not invent cases. Then `eval --list` and wire gates.
+   API-backed pointers (PR URLs, SHAs, gold labels): materialize
+   under `fixtures/` first.
+2. **Generated.** Ask count (`3` recommended). Append to an
+   existing `cases` array when it fits. Never overwrite or weaken
+   a datapoint. Create `evals/evals.config.ts` if missing
+   (`maxConcurrency: 20`; cap 200).
 
-## The API
+## API
 
 ```ts
-import { defineEval, includes, equals, satisfies } from "@cursor/july/evals";
+import { defineEval, includes, satisfies } from "@cursor/july/evals";
 
-// Single datapoint (case id = file path under evals/)
-export default defineEval({
-  description: "Uses get_weather for temperature questions.",
-  tags: ["smoke"],
-  timeoutMs: 120_000,
-  async test(t) {
-    await t.send("What's the temperature in NYC?");
-    t.succeeded();
-    t.calledTool("get_weather");
-    t.notCalledTool("save_weather_note");
-    t.messageIncludes(/°|[FC]/);
-    t.check(t.reply, includes("NYC"));
-  },
-});
-
-// Multiple datapoints in one file (ids = <fileId>/<case.id>)
 export default defineEval({
   tags: ["smoke", "weather"],
   cases: [
@@ -134,100 +63,46 @@ export default defineEval({
         await t.send("What's the temperature in NYC?");
         t.succeeded();
         t.calledTool("get_weather");
+        t.notCalledTool("save_weather_note");
         t.check(t.reply, includes(/°|[FC]/));
       },
     },
-    {
-      id: "paris",
-      async test(t) {
-        await t.send("What's the weather in Paris?");
-        t.succeeded();
-        t.calledTool("get_weather");
-      },
-    },
   ],
 });
 ```
 
-Project-wide defaults in `evals/evals.config.ts` (required once):
-
 ```ts
-import {
-  defineEvalConfig,
-  persistEvalRunsToDir,
-} from "@cursor/july/evals";
+import { defineEvalConfig } from "@cursor/july/evals";
 
 export default defineEvalConfig({
-  maxConcurrency: 20, // required; hard-capped at 200 (model provider limits)
-  // Playground batches are memory-only unless you set persistRuns:
-  // maxPlaygroundRuns: 50, // optional; default 20; playground history only
-  // persistRuns: persistEvalRunsToDir(".agent-serve/eval-runs"),
+  maxConcurrency: 20,
 });
 ```
 
-- Either a single `test(t)` **or** a `cases: [{ id, test }, …]` array — not
-  both. Prefer `cases` for multiple related datapoints.
-- `t.send(message, options?)` runs one turn and waits for park/fail.
-  Options: `workspaceDir`, `workspaceFiles` (`{ path: contents }` seeded
-  before the first turn). Multiple `t.send` calls share the session.
-- `t.reply` / `t.events` / `t.sessionId` for assertions and correlation.
-- `t.metric(name, value)` — structured scores on the case snapshot (playground
-  badges + CLI). Prefer this over encoding scores only in `t.log`.
-- Builders: `includes(string | RegExp)`, `equals(value)`,
-  `satisfies(predicate, label)`.
-
-### Playground customization (optional)
-
-Drop `agent/playground/evals.tsx` exporting `agentName` (must match
-`defineAgent({ name })`) and
-`default: { RunSummary?, CaseMetrics?, CaseExtras? }`. The playground
-loads these via Vite glob (`factory/*`). Without a
-module, cases still show default `t.metric` badges.
-
-## What good cases assert
-
-Gate **decisions and shape, not prose**. Model wording varies run to run;
-tool choice, tool avoidance, and output structure shouldn't.
-
-1. `t.succeeded()` — always, first.
-2. The tool decision: `calledTool` for the intended path, `notCalledTool`
-   for the tempting wrong one (the pair is stronger than either alone).
-3. Output shape: a regex for the contract (`/°|[FC]/`, a JSON marker, a
-   findings-block fence) — not exact sentences.
-4. For structured output, parse `t.reply` and `t.check` fields with
-   `satisfies` rather than substring-matching JSON.
-5. When a shape gate keeps failing on model formatting, tighten the output
-   contract in instructions and re-run — do not move rendering into a host
-   tool to make the gate pass. That trades a prompt fix for permanent code.
-
-Anti-patterns: asserting exact phrasing; more than ~5 gates in one case
-(split it); cases that depend on live external state that drifts (pin the
-input instead — see fixtures).
-
-## Fixture strategy by agent type
-
-| Agent surface | Fixture source |
-| --- | --- |
-| Chat / domain assistant | A canonical prompt string, chosen once and frozen |
-| Tool-heavy | `agent-sdk call <tool> --dir . --input '{...}'` first, then the prompt that should trigger it |
-| GitHub webhook | `github replay … --dry-run --out fixtures/github` (see `packages/agent-serve/skills/github/SKILL.md`) |
-| PR reviewer with host prep | A small fixed PR the team controls; assert on findings shape, not counts |
-| Workspace-dependent | `workspaceFiles` in `t.send` — never machine-local paths |
+Either `test(t)` or `cases`, not both. `t.send` waits for park/fail.
+`workspaceFiles` seeds the first turn. Assert with `t.succeeded()`,
+`calledTool` / `notCalledTool`, `t.check(t.reply, …)`, `t.metric`.
 
-### When eval data needs API calls to render (rate limits)
+## What to gate
 
-If samples are **pointers** (PR URL, SHAs, `gold_labels`) rather than
-self-contained evidence, do **not** wire live `t.send(prUrl)` matrices.
-Materialize once under `fixtures/` at pinned SHAs, point cases at those
-files, gate on decision/shape against gold, and keep a tiny `--tag smoke`
-subset for any full-pipeline runs.
+Decisions and shape, not prose.
 
-## The ratchet rule (hillclimbing)
+1. `t.succeeded()` first
+2. Intended tool + the tempting wrong one
+3. A shape regex or `satisfies` on parsed fields
+4. If formatting keeps failing, tighten instructions. Do not move
+   rendering into a host tool.
 
-Every hillclimb round that **keeps** a change must land an eval that would
-have failed before the change. Never weaken a gate to make a round pass.
+Anti-patterns: exact phrasing; more than ~5 gates (split); live
+drifting inputs (pin them).
 
-```ts
-const toolResults = t.events.filter((e) => e.type === "action.result");
-t.check(toolResults.length, satisfies((n) => (n as number) <= 4, "≤4 tool calls"));
-```
+| Surface | Fixture |
+| --- | --- |
+| Chat | One frozen prompt |
+| Tool-heavy | `agent-sdk call` first, then the prompt |
+| GitHub | `github replay … --dry-run --out fixtures/github` |
+| Host-prep PR review | A team-owned PR; gate findings shape, not counts |
+| Workspace | `workspaceFiles` in `t.send` |
+
+Every kept hillclimb change lands an eval that would have failed
+before it. Never weaken a gate to pass a round.