@cursor/july 0.1.1

package/docs/evals.md

@@ -0,0 +1,383 @@
+---
+title: "Evals"
+description: "Define repeatable checks with defineEval, run them with agentkit eval, and use them as regression checks."
+---
+
+# Evals
+
+An eval is a repeatable check that runs your agent against a fixed input
+and gates the recorded trajectory: the run completed, the right tool
+ran, the reply has the right shape. Evals are how you know a prompt
+tweak helped, a refactor didn't regress the agent, and last month's fix
+is still holding.
+
+Evals exercise the same surface your users hit. The runner starts (or
+targets) a real agent server, drives sessions over the public API, and
+grades what comes back. A passing eval means the agent started,
+accepted a message, and did what you asserted.
+
+> [!NOTE]
+> Import paths here use `@cursor/july/evals`. On projects still
+> using `@cursor/july`, swap the import and run
+> `agent-serve eval`. See
+> [Run the CLI](./README.md#run-the-cli) for the full rename table.
+
+## Define evals with `defineEval`
+
+agentkit discovers evals under the project-root `evals/` directory,
+in `.eval.ts` or `.eval.js` files. That's a sibling of `agent/`, never
+inside it (`agent/evals/` is silently ignored). TypeScript is the normal
+authoring format.
+
+The file path is the eval's identity, so you don't author an id.
+Directories group related evals: `evals/builds/api.eval.ts` becomes id
+`builds/api`. An `index` filename collapses to its directory, so
+`evals/builds/index.eval.ts` becomes `builds`.
+
+An eval is a single `async test(t)`. You drive the agent with `t` and
+assert on the run with the same `t`:
+
+```ts
+// evals/readiness.eval.ts
+import { defineEval, includes } from "@cursor/july/evals";
+
+export default defineEval({
+  description: "Inspects a PR without approving it.",
+  tags: ["smoke"],
+  timeoutMs: 120_000,
+  async test(t) {
+    await t.send(
+      "Is https://github.com/acme/checkout/pull/42 ready to approve?"
+    );
+    t.succeeded();
+    t.calledTool("inspect_pr");
+    t.notCalledTool("approve_pr");
+    t.check(t.reply, includes(/ready|approve/i));
+  },
+});
+```
+
+One file can also hold several datapoints through `cases` (provide
+either `test` or `cases`, not both). Each case id becomes
+`<fileId>/<case.id>`:
+
+```ts
+// evals/prs.eval.ts → prs/checkout, prs/search
+export default defineEval({
+  tags: ["smoke", "prs"],
+  cases: [
+    {
+      id: "checkout",
+      description: "Checkout PR readiness.",
+      async test(t) {
+        await t.send(
+          "Is https://github.com/acme/checkout/pull/42 ready to approve?"
+        );
+        t.succeeded();
+        t.calledTool("inspect_pr");
+      },
+    },
+    {
+      id: "search",
+      async test(t) {
+        await t.send(
+          "Check https://github.com/acme/search/pull/7 before approval."
+        );
+        t.succeeded();
+        t.calledTool("inspect_pr");
+      },
+    },
+  ],
+});
+```
+
+Case ids must be single path segments, unique within the file.
+Each case can set its own `description`, `tags`, and `timeoutMs`. A
+case-level value replaces the file-level value for that datapoint.
+
+## Configure eval runs
+
+Each project with evals needs `evals/evals.config.ts` or
+`evals/evals.config.js`, and it must set `maxConcurrency`. Each case
+issues real model-provider requests, so concurrency is capped hard at
+200. Existing projects use 20. Discovery with `eval --list` works
+without this file, but running a case does not.
+
+```ts
+import {
+  defineEvalConfig,
+  persistEvalRunsToDir,
+} from "@cursor/july/evals";
+
+export default defineEvalConfig({
+  maxConcurrency: 20, // required
+  // timeoutMs: 180_000, // optional project-wide default
+  // maxPlaygroundRuns: 50, // playground /v1/dev/evals history only (default 20)
+  //
+  // Playground batches default to **process memory only** — they disappear
+  // when `serve` exits. Opt into durable storage:
+  // persistRuns: persistEvalRunsToDir(".agent-serve/eval-runs"),
+  // Or implement { load, save, delete } yourself (S3, DB, …).
+});
+```
+
+The timeout order is case or file `timeoutMs`, CLI `--timeout-ms`,
+project config `timeoutMs`, then the 180-second runner default.
+
+Playground batch retention is separate from case concurrency:
+
+| Option | Default | Meaning |
+| --- | --- | --- |
+| `maxPlaygroundRuns` | `20` | Max batches in the playground / `/v1/dev/evals*` history (not CLI `eval`) |
+| `persistRuns` | unset | Optional `{ load, save, delete }` so batches survive process restart (`delete` required for durable prune) |
+
+Without `persistRuns`, navigating away and back still works while the same
+`serve` process is up; a restart clears history.
+
+## Drive and assert with `t`
+
+`t` is both the driver and the assertion surface. You write ordinary
+control flow, sending turns and asserting inline.
+
+Drive the agent with `t.send(message, options?)`. It runs one turn and
+waits for the session to park or fail. Multiple sends in one case share
+the session, which is how you write multi-turn evals. The return value
+contains the turn's `message`, `sessionId`, `events`, `toolCalls`, and
+`ok` state.
+
+Read the full case state with `t.reply` (the last assistant text),
+`t.events` (every captured session event across turns), and
+`t.sessionId`.
+
+Assert with the gates:
+
+| Gate | Checks |
+| --- | --- |
+| `t.succeeded()` | the captured trajectory has at least one turn and did not fail |
+| `t.calledTool(name)` | `name` appears in the captured tool calls |
+| `t.notCalledTool(name)` | `name` does not appear in the captured tool calls |
+| `t.messageIncludes(token)` | the last assistant reply matches a string or `RegExp` |
+| `t.check(value, expectation)` | any value, against a builder |
+
+`calledTool` reads the recorded trajectory. A requested call counts even
+when its result has not arrived. To require a completed result, inspect
+`t.events` for an `action.result` event.
+
+The expectation builders are `includes(string | RegExp)`,
+`equals(value)`, and `satisfies(predicate, label)`. `includes`
+stringifies its input, `equals` compares values deeply, and `satisfies`
+runs your predicate. `t.log(message)` records a debug line for the CLI
+and playground result.
+
+Two `t.send` options make cases self-contained: `workspaceFiles`
+(`{ path: contents }`, seeded into the session workspace before the
+first turn) and `workspaceDir` (an absolute harness cwd). Prefer
+`workspaceFiles` over machine-local paths, so the case carries its own
+fixtures. Both options only apply to the first `t.send`, when the runner
+creates the session.
+
+```ts
+const toolResults = t.events.filter((e) => e.type === "action.result");
+t.check(
+  toolResults.length,
+  satisfies((n) => (n as number) <= 4, "at most 4 tool calls")
+);
+```
+
+A case with no explicit gates falls back to whether at least one turn
+completed successfully. Add `t.succeeded()` and behavior-specific gates
+anyway. They make the contract visible during review.
+
+## Run evals from the CLI
+
+The `eval` command discovers, filters, and runs cases.
+
+Run the CLI under Node 22.13 or newer. Do not use Bun. Its HTTP/2 client
+breaks tool-result streams and causes eval turns to fail.
+
+```bash
+agentkit eval --dir . --list                         # discover only
+agentkit eval --dir .                                # run all
+agentkit eval --dir . builds/checkout                # one datapoint
+agentkit eval --dir . builds search                  # several ids or prefixes
+agentkit eval --dir . --tag smoke --tag pull-request # any matching tag
+agentkit eval --dir . --json --no-stream             # machine-readable results
+agentkit eval --dir . --verbose                      # logs + reply snippets
+```
+
+Id filters use OR semantics. Each filter selects an exact id and its
+descendants. For example, `builds` selects `builds`,
+`builds/checkout`, and every other case below that path. Repeated tags
+also use OR semantics. When you provide both ids and tags, a case must
+match both groups.
+
+`eval` boots an ephemeral server on port 0 with a temp state root
+outside the project, so cases don't inherit ambient monorepo rules and
+don't pollute `.agentkit/`. Point `--url` at a running server to eval
+a live agent instead:
+
+```bash
+agentkit eval --dir . \
+  --url http://127.0.0.1:3000/weather-agent \
+  --bearer-token "$AGENT_TOKEN"
+```
+
+The eval definitions still come from `--dir`; `--url` only changes the
+agent that receives the turns. For a locally mounted multi-agent
+directory, `--slug weather-agent` chooses the target. Use
+`--state-root` to keep ephemeral session state at a chosen path,
+`--timeout-ms` to override the project timeout, and `--no-stream` to
+keep live progress off stderr. A TTY streams turn progress by default.
+`--verbose` still writes `t.log` lines to stderr and adds reply snippets
+to text results.
+
+Model turns need a Cursor credential from `agentkit login` or
+`CURSOR_API_KEY`.
+
+The exit code is `0` when every selected case passes, `1` when any case
+fails, and `2` when no case matches. `--list` exits `0`, including when
+it finds no cases.
+
+For a compact command index, see [CLI: eval](./reference/cli.md#eval).
+
+### JSON results
+
+Use `--json --no-stream` in scripts and CI. The top-level result carries
+the totals and one result per case:
+
+```json
+{
+  "ok": true,
+  "passed": 1,
+  "failed": 0,
+  "results": [
+    {
+      "id": "readiness",
+      "ok": true,
+      "assertions": [{ "name": "succeeded", "passed": true }],
+      "sessionId": "ses_123",
+      "inputs": ["Is checkout pull request 42 ready to approve?"],
+      "toolCalls": [{ "toolName": "inspect_pr", "isError": false }],
+      "logs": [],
+      "durationMs": 12340
+    }
+  ]
+}
+```
+
+Each case result can also include `description`, `finalText`, `tools`,
+`error`, and tool arguments or output. This shape lets CI report the
+failed assertion without parsing terminal text.
+
+## Run evals in the playground
+
+Start the server with `--dev`, open the playground, and choose
+**Evals**. You can run every case or one case, watch progress, and open
+the resulting session trace.
+
+```bash
+agentkit serve --dir . --dev
+```
+
+Playground runs target the live server instead of an ephemeral one.
+Their sessions appear in the session list. One eval batch can run at a
+time. By default those batches are **in-memory only** (capped by
+`maxPlaygroundRuns`); set `persistRuns` in `evals.config.ts` if you need them
+after a serve restart — see [Configure eval runs](#configure-eval-runs).
+
+The UI uses the playground eval routes (available without `--dev`):
+`GET /v1/dev/evals` lists datapoints and config (includes `maxPlaygroundRuns` /
+`durableRuns`),
+`GET /v1/dev/evals/runs` rehydrates recent batches after navigation,
+`POST /v1/dev/evals/runs` starts a batch, and
+`GET /v1/dev/evals/runs/:runId` polls it. See
+[Playground eval routes](./reference/http-api.md#playground-eval-routes).
+The start request returns `202` while cases run in the background.
+Poll until the snapshot status becomes `completed` or `failed`.
+Configuration errors appear on a failed snapshot.
+
+## What good cases assert
+
+Gate decisions and shape, not prose. Model wording varies run to run.
+Tool choice, tool avoidance, and output structure are the stable
+contract.
+
+1. `t.succeeded()`: always, first.
+2. The tool decision: `calledTool` for the intended path,
+   `notCalledTool` for the likely wrong alternative. The pair is
+   stronger than either alone.
+3. Output shape: a regex for the contract (`/ready|blocked/i`, a JSON
+   marker, a findings-block fence), never exact sentences.
+4. For structured output, parse `t.reply` and check fields with
+   `satisfies` instead of substring-matching JSON.
+
+The common failure modes: asserting exact phrasing, packing more than
+about five gates into one case (split it), and cases that depend on live
+external state that drifts (pin the input; see fixtures).
+
+## Pick fixtures by agent type
+
+The right fixture depends on the surface under test.
+
+| Agent surface | Fixture |
+| --- | --- |
+| Chat / domain assistant | A canonical prompt string, chosen once and frozen |
+| Tool-heavy | Run `agentkit call <tool>` first to pin what the tool returns, then freeze the prompt that triggers it |
+| GitHub webhook | `agentkit github replay <pr> --events '*' --dry-run --out fixtures/github` snapshots real payloads for offline replay ([GitHub guide](./guides/github.md)) |
+| PR reviewer with host preparation | Diff, metadata, and gold labels pinned to commit SHAs; keep any live PR matrix small |
+| Workspace-dependent | `workspaceFiles` in `t.send` options, never developer-machine paths |
+
+Tag the fast, reliably passing core `smoke` and run `--tag smoke` in the
+inner loop. Leave slow or flaky-prone cases untagged for explicit runs.
+
+### Materialize API-backed fixtures
+
+An input that only points at external data, such as a pull request URL,
+snapshot id, or pair of commit SHAs, is not self-contained. Fetch it
+once and commit the rendered fixture before you expand the suite.
+
+1. Save the diff, metadata, and labels under `fixtures/` at pinned
+   revisions.
+2. Seed those files with `workspaceFiles`, or read them from the fixture
+   directory.
+3. Assert decisions and output shape against the saved evidence.
+4. Keep a small `smoke` subset for any remaining live pipeline checks.
+
+`maxConcurrency` limits parallel datapoints. It does not limit model or
+API fan-out inside one datapoint. Materialized fixtures prevent a large
+suite from exhausting provider and GitHub rate limits. The
+[evals skill](../skills/evals/SKILL.md) has the full fixture workflow.
+
+## Keep improvements with regression evals
+
+Every [hillclimb](./hillclimbing.md) round that keeps a change must land
+an eval that would have failed before the change. If you can't express
+the improvement as a gate (a `calledTool` shift, a bounded
+`action.result` count, an output-shape regex), the improvement is
+unverified, and it'll regress silently.
+
+The rule cuts the other way too: never weaken an existing gate to make a
+round pass. That's the freeze line moving, and it turns your regression
+suite into a list of checks that no longer protect anything.
+
+## Compare variants on live traffic
+
+Use `defineAB` to compare variant metrics on live sessions. It is not a
+test runner and has no `agentkit ab` command. Keep `defineEval` as the
+regression ratchet. Eval sessions do not enroll or change live metrics.
+See [Live A/B metrics](./ab.md) for assignment, behavior, collection,
+and inspection.
+
+## What's next
+
+Continue with these pages:
+
+- [Live A/B metrics](./ab.md): sticky variants and cumulative metrics
+  on live sessions
+- [Hillclimbing](./hillclimbing.md): the loop evals make trustworthy
+- [Building agents with agents](./building-with-agents.md): have a
+  coding agent write the first suite
+- [GitHub guide](./guides/github.md): deterministic webhook fixtures
+  with `github replay`
+- [Sessions and streaming](./reference/sessions.md): the events
+  `t.events` contains

package/docs/guides/agent-to-agent.md

@@ -0,0 +1,118 @@
+---
+title: "Agent-to-agent"
+description: "Every agent is an MCP server. Peer MCP connections let one agent delegate to another that keeps its own tools, sessions, and context."
+---
+
+# Agent-to-agent
+
+Every mounted agent is also an MCP server. So agents can delegate to each
+other without extra infrastructure. One agent hands a question to
+another, and the peer answers in its own session with its own
+instructions, tools, and context. Any external MCP client can do the
+same. A peer MCP connection makes the wiring one line.
+
+This guide wires a `concierge` agent that delegates weather questions
+to a `weather-agent` peer mounted on the same host.
+
+## The MCP endpoint
+
+Each agent serves the Model Context Protocol over streamable HTTP at
+`/<slug>/v1/mcp` (or `/v1/mcp` in single mode). The surface is stateless
+(session identity travels in tool arguments) and runs the same route
+auth chain as the session API. It exposes three tools:
+
+| Tool | Behavior |
+| --- | --- |
+| `ask` | Send a message. Runs a model turn in this agent's own session and returns `{ status, sessionId, reply }`. Omit `sessionId` for a fresh session; pass it back to follow up. |
+| `check` | Wait for or poll a running session (`waitSeconds: 0` polls without blocking). |
+| `call_tool` | Call one of the agent's server tools directly, no model turn. Registered only when the agent has server tools. |
+
+Waits are bounded at roughly 50 seconds, below typical MCP client
+request timeouts: a long turn returns `status: "running"` and the caller
+keeps waiting with `check`. Sessions created this way live on the `mcp`
+channel, bind to the calling principal, and show up in the playground
+and `GET /v1/sessions` like any other session.
+
+Any MCP client can attach to this endpoint. It isn't only for other
+agentkit agents.
+
+## Wire a peer MCP connection
+
+A peer MCP connection points one agent at another mounted on the same
+serve host. Author an MCP connection whose transport is the peer's slug:
+
+```ts
+// agents/concierge/agent/mcp-connections/weather.ts
+import { defineConnection } from "@cursor/july/connections";
+
+export default defineConnection({
+  agent: "weather-agent",
+  description: "Delegate weather questions to the weather agent.",
+});
+```
+
+The parent model now sees the peer's `ask` and `check` (and `call_tool`)
+tools under the `weather` server name. It reads like subagent
+delegation, except the peer is a full agent that keeps its own
+instructions, tools, MCP connections, and sessions.
+
+```bash
+agentkit serve --dir ./agents --dev
+agentkit chat --url http://127.0.0.1:3000/concierge \
+  --message "What's the weather in Paris right now?"
+# concierge → weather.ask → weather-agent's own session/tools → reply
+```
+
+## Peer or subagent?
+
+The two delegation mechanisms solve different problems.
+
+| | Subagent (`agent/subagents/<id>/`) | Peer (`defineConnection({ agent })`) |
+| --- | --- | --- |
+| Runs as | an SDK custom subagent inside the parent's harness | an independent agent on the same host |
+| Own tools, MCP connections, sessions | no, inherits the parent's surface | yes, everything is its own |
+| Visible to others | only its parent | any MCP client, other agents, its own playground |
+| Reach for it when | splitting one job into specialist roles | composing independently useful agents |
+
+## How peer URLs resolve
+
+Peer URLs resolve when the server starts, so the ephemeral ports `run`
+and `eval` use work too. Local-runtime turns (and host-side
+`ctx.host.mcp` or channel-handler calls) reach the peer over loopback,
+which works under the default auth. Cloud-runtime turns execute on a VM
+that cannot reach this host's loopback address: pass
+`--public-url https://agentkit.example.com` (or
+`serve(dir, { publicUrl })`) so peers resolve to a reachable URL.
+Without one, peers are omitted from cloud turns and the server warns at
+startup. With `--bearer-token`, the token is attached to peer calls
+automatically so they pass the target agent's auth chain.
+
+## Guardrails
+
+Unknown peer slugs and self-references fail `serve` at startup, so you
+find out immediately rather than at delegation time. Peers require the
+multi-agent layout (each agent mounted under its slug, the default).
+
+The framework does not provide cross-host loop protection. If agent A's
+instructions delegate to B and B's delegate back to A, they can recurse.
+Scope each agent's delegation instructions narrowly. The concierge
+delegates *weather questions* to `weather-agent`, not everything.
+
+## Call a peer from host code
+
+Peer MCP connections are ordinary MCP connections, so deterministic host code
+can use them too. A channel handler or server tool can call
+`host.mcp.callTool("weather", "ask", { message: "…" })` without any
+model turn deciding to. See
+[MCP connections](../reference/connections.md#every-mcp-connection-is-available-in-three-places)
+for the three places every MCP connection is available.
+
+## What's next
+
+Continue with these pages:
+
+- [MCP connections](../reference/connections.md): all four MCP connection
+  transports
+- [Subagents](../reference/subagents.md): the in-harness alternative
+- [HTTP API](../reference/http-api.md#mcp-endpoint): the endpoint
+  contract

package/docs/guides/cloud-runtime.md

@@ -0,0 +1,122 @@
+---
+title: "Cloud runtime"
+description: "Run turns on Cursor cloud agents instead of the local harness: when to switch, what changes, and the patterns that hold up."
+---
+
+# Cloud runtime
+
+By default, turns execute on the Cursor SDK's local harness, on the same
+machine as the server. Set `runtime: "cloud"` and turns execute on
+Cursor cloud agents instead. They're ephemeral VMs that carry a repo
+checkout, run `gh`, `git`, and tests for real, and scale past what one
+host's disk and CPU can do. The serve host keeps handling routing, host
+preparation, sessions, and bookkeeping.
+
+A canonical use is a PR driver whose triage runs on cloud VMs. The
+patterns in this guide come from running one against real PR traffic.
+
+## When to switch
+
+A guideline from running PR agents at scale: per-PR worktrees on the serve host
+don't scale to hundreds of engineers opening PRs. When the job needs a
+repo checkout at scale, use cloud. The signals:
+
+- The agent must run repo commands (tests, builds, `git`) against many
+  different refs concurrently.
+- Turns are long and heavy, and you don't want them competing with the
+  server for resources.
+- The work product is a PR or branch the VM can push, not a local file.
+
+Stay local when the agent is conversational, tool-driven against APIs,
+or works over host-prepared evidence. Local turns are cheaper, start
+faster, and support the full authored surface.
+
+## Configure it
+
+Cloud runtime is two fields on the agent config.
+
+```ts
+import { defineAgent } from "@cursor/july";
+
+export default defineAgent({
+  runtime: "cloud",
+  cloud: {
+    repos: [{ url: "https://github.com/org/repo", startingRef: "main" }],
+    // env / envVars / … forwarded to the Cursor SDK
+  },
+});
+```
+
+The host must be signed in (`agentkit login` or `CURSOR_API_KEY`).
+
+> [!IMPORTANT]
+> Cloud agents run against the Cursor backend under the signed-in
+> account, and every wake spends real cloud-agent budget. Decide
+> explicitly what may trigger one.
+
+## What changes on cloud
+
+Cloud turns run on a VM without your authored files, so the runtime
+mapping shifts:
+
+| Folder or file | Local runtime | Cloud runtime |
+| --- | --- | --- |
+| `instructions.*` | `AGENTS.md` in the session workspace | prepended to the first prompt |
+| Server tools (`execution: "server"`) | in-process SDK custom tools | not available |
+| Agent tools (`execution: "agent"`) | scripts in the session workspace | catalog + script bodies on the first prompt |
+| `skills/*` | `.cursor/skills/` in the workspace | only if present in the cloud repo |
+| `mcp-connections/*.ts` | SDK `mcpServers` | SDK `mcpServers` (peers need `--public-url`) |
+| `sandbox/workspace/**` | seeded into the session workspace | ignored |
+| Tool approvals (`needsApproval`) | supported | not supported |
+
+`agentkit validate` warns when `runtime: "cloud"` is combined with
+local-only capabilities. Those warnings match this table.
+
+Two more behaviors are cloud-specific. Sessions persist a separate SDK
+agent id (`bc-…`), emitted on the stream as `agent.bound` with a URL to
+the cloud conversation. Cloud ids are minted during the first send. And
+peer MCP connections resolve to `--public-url` for cloud turns, because a VM
+cannot reach the host's loopback; without one, peers are omitted from
+cloud turns and the server warns at startup.
+
+## Hybrid: local agent, cloud sessions
+
+A local-runtime agent can still open cloud-attached sessions per send.
+Channel handlers may pass a `cloud` block (repos pinned to a PR ref, say)
+in `send` options, and Slack handlers may return `cloud` from a mention
+hook. A PR driver works this way: chat stays local, and the `drive`
+flow attaches the PR to a cloud VM. The agent-level `cloud` config is the
+base that per-session options merge over.
+
+## Patterns that hold up
+
+These come from running a PR driver against real PR traffic:
+
+- One cloud agent per unit of work (per PR, say). Store the `bc-…` id
+  keyed by the work unit (an affinity store written from an
+  `agent.bound` hook) so webhook wakes resume the same conversation
+  instead of booting a fresh VM per event.
+- Stable continuation keys (`pr:owner/repo#N`) so every wake lands on
+  the same session within a channel.
+- Keep the host deterministic: fetch briefs and metadata on the host,
+  send the VM a compact prompt, and let the VM re-read source of truth
+  with its own `gh` and `git` instead of trusting payload snapshots.
+- Limit exposure: add repository allowlists on webhook channels,
+  because every wake spends the account's budget.
+
+## Verify cloud agents
+
+`agentkit run` and `eval` work unchanged. The trajectory records the
+same event vocabulary plus `agent.bound` with the cloud URL, so you can
+open the cloud conversation for any session. Cloud turns take minutes.
+Pass generous `--timeout-ms` values, and keep curl timeouts long when
+driving channels directly.
+
+## What's next
+
+Continue with these pages:
+
+- [Agent config](../reference/agent-config.md): the `runtime` and
+  `cloud` fields precisely
+- [GitHub guide](./github.md): the webhook patterns that pair with
+  cloud triage