@cursor/july 0.1.108 → 0.1.111

package/docs/ab.md

@@ -1,337 +0,0 @@
----
-title: "Live A/B metrics"
-description: "Assign sticky variants with defineAB, collect live performance metrics, and inspect per-arm results."
----
-
-# Live A/B metrics
-
-Use `defineAB` to compare variants on live agent sessions. New sessions
-receive a sticky assignment in each enrolled experiment. The Agent SDK folds
-their durable event streams into tool, token, failure, and wall-time
-metrics. You can send cumulative samples to your metrics backend and
-inspect aggregates in the playground.
-
-`defineAB` compares live variants through sticky assignment,
-instruction overlays, optional tool branches, and cumulative metrics.
-Metric callbacks observe the result without approving, rejecting, or
-failing a turn. Use [evals](./evals.md) for pass/fail regression checks
-on fixed inputs.
-
-## Choose live A/B metrics or evals
-
-Both features read the session event stream, but they answer different
-questions.
-
-| | Live A/B metrics | Evals |
-| --- | --- | --- |
-| Question | How do variants compare on live sessions? | Does the agent still meet a fixed contract? |
-| Location | `agent/ab.ts` or `agent/ab/<name>.ts` | `evals/**/*.eval.ts` |
-| Input | Dev or production traffic | Frozen prompts and fixtures |
-| Output | Cumulative metrics by session and arm | Pass/fail assertions |
-| How it runs | Automatically on new live sessions | `agent-sdk eval` |
-
-There is no `agent-sdk ab` command or assertion API.
-
-## Define an experiment
-
-Author one experiment in `agent/ab.ts`, add more under
-`agent/ab/<name>.ts`, or use both forms. Each file defines one
-experiment. The experiment name comes from `name` when set. Otherwise,
-the Agent SDK uses `ab` for `agent/ab.ts` and the file stem for files under
-`agent/ab/`.
-
-```ts
-// agent/ab/concise-weather.ts
-import {
-  defineAB,
-  splitBySessionHash,
-} from "@cursor/july/ab";
-
-export default defineAB({
-  name: "concise-weather",
-  variants: {
-    control: {
-      label: "Baseline",
-    },
-    treatment: {
-      label: "Short replies",
-      description: "Adds a one-paragraph response limit.",
-      instructions: "Keep weather replies to one short paragraph.",
-    },
-  },
-  split: splitBySessionHash({
-    weights: { control: 1, treatment: 1 },
-    holdout: 0.1,
-  }),
-  derive: {
-    weatherCalls: (event) =>
-      event.type === "action.result" &&
-      event.data.toolName === "get_weather"
-        ? 1
-        : null,
-  },
-  onSample(sample) {
-    console.log(
-      sample.experiment,
-      sample.variant,
-      sample.metrics.toolCalls,
-      sample.metrics.wallTimeMs
-    );
-  },
-});
-```
-
-Every definition needs:
-
-- At least two variants. Variant keys cannot be empty or contain `/` or
-  `\`.
-- A `split` function that returns a variant key or `null`.
-- An `onSample` callback for completed or failed turns.
-
-`label` and `description` appear with the arm in result surfaces.
-`instructions` changes the prompt for sessions in that arm. `derive`
-adds custom counters.
-
-Duplicate experiment names are validation errors. Check discovery
-before you serve:
-
-```bash
-agent-sdk validate --dir .
-agent-sdk info --dir . --json
-```
-
-The `abs` field in `info` lists the discovered experiment names.
-
-## Assign sticky variants
-
-Enrollment happens once, when a live session is created and before its
-first turn:
-
-1. The Agent SDK records `session.started`.
-2. Each experiment runs its `split` function.
-3. The Agent SDK records one durable `ab.assigned` event per experiment.
-4. The selected arms become available on `session.abs`.
-5. Variant instruction overlays reach the first model turn.
-
-A split can return a variant key or `null`. A null assignment is a
-sticky skip for that experiment. It increments the experiment's
-`skipped` total, still appears in the snapshot's `sessions` list with
-`variant: null`, and does not collect arm metrics or call `onSample`.
-
-Use the split helper that matches your rollout:
-
-| Helper | Behavior |
-| --- | --- |
-| `splitBySessionHash({ weights?, holdout?, salt? })` | Hashes the session id into a reproducible arm; the recommended default |
-| `splitByRandom({ weights?, holdout? })` | Draws once when the session starts, then persists the result |
-| `splitAlways("control")` | Pins every new session to one arm |
-| `splitNone()` | Skips every new session without deleting the experiment |
-| `splitIf(predicate, inner)` | Runs `inner` only when the predicate passes |
-| Custom `split(ctx)` | Returns a declared variant key or `null` |
-
-The split context includes the agent name, channel id, session info,
-experiment name, and declared variant keys. For example, enroll only
-Slack sessions:
-
-```ts
-split: splitIf(
-  (ctx) => ctx.channel.id === "slack",
-  splitBySessionHash()
-),
-```
-
-Weights default to equal. Non-positive weights leave an arm out of the
-draw, and at least one arm must have a positive weight. `holdout` is the
-fraction of sessions assigned `null`, from `0` through `1`. Change
-`salt` to reshuffle future hash assignments without renaming the
-experiment.
-
-If a custom split throws or returns an unknown variant, the Agent SDK logs
-the error and records `variant: null`. The failed decision becomes a
-sticky skip instead of breaking the session.
-
-Enrollment only applies to new sessions. Adding an experiment does not
-assign existing conversations. Follow-ups keep the session's original
-arms. Keep experiment names and variant keys stable while you collect
-and compare results.
-
-## Change behavior by variant
-
-Variant instructions are appended to the agent's base instructions.
-Local sessions receive the merged instructions in `AGENTS.md` before
-every turn. Cloud sessions receive them in the first-turn preamble
-only. For cloud follow-ups, branch through `session.abs` when the arm
-must remain visible to deterministic behavior.
-
-Tools can branch on the assignment through `ctx.session.abs`. Hooks can
-read the same field for logging or export:
-
-```ts
-const treatment =
-  ctx.session.abs?.["concise-weather"] === "treatment";
-
-if (treatment) {
-  return conciseWeatherResult;
-}
-
-return baselineWeatherResult;
-```
-
-This makes the assignment available to deterministic code as well as
-the model prompt. Use both patterns together when one experiment must
-steer the prompt and host code at once.
-
-`defineAB` does not select a different model or runtime for each arm.
-Keep those settings in `agent/agent.ts`, or write explicit host logic
-when your experiment needs another behavior lever.
-
-The split and selected arm can affect agent behavior. `derive` and
-`onSample` only observe the resulting event stream. Errors in either
-callback are logged and never fail the turn.
-
-## Collect built-in and custom metrics
-
-Metrics accumulate for each session and experiment. When one session
-joins several experiments, every enrolled experiment folds the same
-turn and tool events into its own counters.
-
-| Metric | How the Agent SDK calculates it |
-| --- | --- |
-| `turns` | Adds one on `turn.completed` or `turn.failed` |
-| `turnFailures` | Adds one on `turn.failed` |
-| `toolCalls` | Adds one for each `action.result` |
-| `toolErrors` | Adds one when `action.result.data.isError` is true |
-| `inputTokens`, `outputTokens` | Adds usage from completed turns |
-| `cacheReadTokens`, `cacheWriteTokens` | Adds cache usage from completed turns |
-| `costUsd` | Sums the estimated turn cost recorded on `turn.completed` (turns whose model has no known rates contribute 0) |
-| `wallTimeMs` | Sums the time from `turn.started` to its completed or failed event |
-| `custom` | Sums finite numeric deltas returned by `derive` |
-
-`onSample` fires after every `turn.completed` and `turn.failed` event
-for an enrolled arm. The sample contains:
-
-| Field | Value |
-| --- | --- |
-| `experiment` | Experiment name |
-| `variant`, `variantLabel?` | Sticky arm and optional display label |
-| `sessionId`, `channelId` | Source session |
-| `metrics` | Cumulative metrics through this turn |
-| `reason` | `turn.completed` or `turn.failed` |
-| `at` | Terminal event timestamp |
-
-The metrics are cumulative, not per-turn deltas. A second sample from
-the same session includes the first turn's counts.
-
-Each `derive` extractor runs on every session event for its enrolled
-experiment, including streamed `message.appended` events. Keep it
-synchronous and cheap. Return a finite number to add a delta, or
-`null` to skip the event. Send samples to your metrics service from
-`onSample`; do not perform network or disk work in `derive`.
-
-Skipped sessions never call `onSample`. Errors from `derive` or
-`onSample` are logged, then metric collection continues.
-
-## Inspect assignments and results
-
-Open the playground's **A/Bs** tab to see aggregate arm totals and
-per-session assignments. The tab reads `GET /v1/abs`.
-
-The response has two views of the same durable data:
-
-| Field | Contents |
-| --- | --- |
-| `experiments` | Declared variants, skipped-session count, arm session counts, and aggregate metrics |
-| `sessions` | Visible sessions with their assignments and cumulative metrics |
-
-`GET /v1/abs` returns sessions visible to the current principal by
-default. In `--dev`, loopback requests include every session. Add
-`--allow-anonymous` to include every session from non-loopback callers
-too.
-
-The session event stream is the source of truth for assignment + fold.
-`GET /v1/abs` recomputes aggregates from those logs. Any
-`agent/storage.ts` exports samples and snapshots durably: an authored
-`abs` table when the backend has a native shape for it, or the table
-derived over the KV core otherwise. See
-[Storage](./storage.md#eval-and-a-b-tables).
-
-## Configure the playground fold window
-
-Assignments and foldable metrics already persist in each session's
-event stream. The optional `agent/ab.config.ts` only caps how many
-sessions the playground and `GET /v1/abs` fold:
-
-```ts
-import { defineABConfig } from "@cursor/july/ab";
-
-export default defineABConfig({
-  // Optional. Defaults to 200. Only affects GET /v1/abs / A/Bs tab.
-  maxPlaygroundSessions: 500,
-});
-```
-
-`maxPlaygroundSessions` keeps the newest sessions in the fold. It does
-not prune session logs or change assignment. For export to S3, a DB, or
-your metrics vendor, send samples from `onSample` or declare a storage
-`abs` table.
-
-## Keep assignments durable
-
-The append-only event stream is the source of truth. Each
-`ab.assigned` event persists a variant key or null skip. Built-in
-metrics come from the turn and tool events that follow it.
-
-After a server restart or a parked session resumes, the live collector
-replays the stream to rebuild cumulative counters. Replay does not call
-`onSample` (or write to the storage `abs` table) for historical turns.
-Only a new completed or failed turn emits another sample.
-
-The snapshot API also replays `derive` across the full stream, so
-custom totals match the current extractor. Changing a derive function
-can change historical snapshot totals. Treat metric definitions as
-versioned experiment code.
-
-## Keep eval traffic separate
-
-Sessions created by `agent-sdk eval` and the playground Evals runner use
-`purpose: "eval"`. They skip A/B enrollment entirely:
-
-- No split function runs.
-- No `ab.assigned` event is recorded.
-- No `onSample` callback fires.
-- The session is omitted from `GET /v1/abs`.
-
-Ordinary chat, `agent-sdk run`, Slack, GitHub, and other channel sessions
-use the live purpose. You do not need `splitIf` to exclude eval traffic.
-
-## Know the boundaries
-
-`defineAB` provides sticky assignment, variant instructions,
-`session.abs` for tools, cumulative metrics, and local inspection. It
-does not provide:
-
-- A test command, assertion API, or pass/fail result
-- Statistical significance calculations
-- An experiment rollout or lifecycle service
-- Per-variant model or runtime configuration
-- A built-in analytics warehouse (bring your own via `onSample` or the
-  storage `abs` table)
-
-Use [evals](./evals.md) to protect known behavior. Use `onSample` or a
-storage `abs` table when you need sample/snapshot exports beyond the
-session event log.
-
-## What's next
-
-Continue with these pages:
-
-- [Evals](./evals.md): pass/fail regression checks on fixed inputs
-- [Hillclimbing](./hillclimbing.md): improve an agent against fixed
-  fixtures
-- [Hooks](./reference/hooks.md): other event-stream consumers
-- [Sessions and streaming](./reference/sessions.md): the
-  `ab.assigned` event and durable log
-- [Playground](./reference/playground.md): the A/Bs tab
-- [HTTP API](./reference/http-api.md): `GET /v1/abs`
-- [Live A/B metrics skill](./skills/ab.md): have a coding agent
-  wire an experiment

package/docs/concepts.md

@@ -1,173 +0,0 @@
----
-title: "How the Agent SDK works"
-description: "Understand projects, sessions, event streams, channels, and runtimes in plain language."
----
-
-# How the Agent SDK works
-
-An agent is a folder of instructions and capabilities. The Agent SDK
-discovers those files, runs conversations, and records what happened.
-
-## What happens when someone sends a message?
-
-Follow one message through the system:
-
-1. A channel receives the message from HTTP, Slack, GitHub, or another
-   webhook.
-2. The channel starts a session or continues an existing one.
-3. The runtime gives the model its instructions, tools, and workspace.
-4. The model replies and can call tools along the way.
-5. The Agent SDK appends every message and tool call to the session's
-   event stream.
-
-The channel is the front door. The runtime does the work. The event
-stream is the record you inspect later.
-
-## How do files become an agent?
-
-Each capability has a home in the project. The path tells
-the Agent SDK what to load. The filename becomes the capability's
-name. For example, `agent/tools/get_weather.ts` creates a tool named
-`get_weather`.
-
-| Path | What it is |
-| --- | --- |
-| `agent/agent.ts` | Model and runtime settings |
-| `agent/instructions.md` | The always-on system prompt |
-| `agent/tools/<name>.ts` | Typed actions the model can call |
-| `agent/skills/*` | Procedures loaded when needed |
-| `agent/mcp-connections/<name>.ts` | Tools from external MCP servers |
-| `agent/host-connections/<name>.ts` | Privileged MCP servers for host tools only |
-| `agent/channels/*.ts` | HTTP, Slack, and GitHub entry points |
-| `agent/ab.ts` or `agent/ab/*.ts` | Sticky variants and live performance metrics |
-| `agent/result.ts` | Optional host `commit` on the final assistant text |
-| `evals/**/*.eval.ts` | Repeatable checks at the project root |
-
-Other folders add subagents, extensions, hooks, schedules, and workspace
-files. You don't register them elsewhere. Run `agent-sdk validate` to
-catch invalid files before serving the project.
-
-See [Project layout](./reference/project-layout.md) for the folder
-structure.
-
-## How does the Agent SDK identify a conversation?
-
-A session is one durable conversation. It has two identifiers:
-
-- **`continuationToken`** tells a channel which conversation to resume.
-  A Slack channel can use its thread ID. A GitHub channel can use the
-  pull request. The built-in HTTP API returns an opaque token and
-  rotates it after each accepted follow-up.
-- **`sessionId`** identifies the stored session. Use it to stream
-  events, inspect the session, resolve approvals, or bind a tool call to
-  the session.
-
-Use the continuation token to keep talking. Use the session ID to
-observe or manage the conversation.
-
-## How do I see what an agent did?
-
-Each session records an append-only event stream. It includes:
-
-- Messages and streamed text
-- Requested tool calls and their results
-- Approval requests and decisions
-- Turn completion and token usage
-
-Sessions and their event streams survive server restarts. The
-playground renders the stream. Evals assert against it. The
-`agent-sdk trajectory` command turns a saved stream into a short
-summary.
-
-When a run surprises you, inspect its event stream first. See
-[Sessions and streaming](./reference/sessions.md) for the event
-vocabulary.
-
-## What does a channel control?
-
-A channel connects the agent to a surface such as HTTP, Slack, GitHub,
-or a custom webhook. It controls:
-
-- Routes and input schemas
-- Authentication
-- Conversation identity
-- How replies return to the user
-
-The built-in HTTP session API is always available. Custom routes accept
-loopback callers by default. Add an auth policy before sharing them over
-a network.
-
-Channels should also prepare deterministic input for the model. For
-example, a GitHub channel can fetch the pull request, collect the diff,
-and seed the workspace before the turn starts. The model can then focus
-on the review instead of gathering files.
-
-See [Channels](./reference/channels.md) for route and authentication
-details.
-
-## Where does a turn run?
-
-Choose a runtime in `agent/agent.ts`:
-
-| | Local (default) | Cloud |
-| --- | --- | --- |
-| Turn runs on | The server host | A Cursor cloud agent |
-| Server tools | Supported | Supported when the server has `--public-url` or `--cloud-tools-url`; the cloud turn reaches them over authenticated HTTP MCP. Without one of those flags, the server warns and cloud turns omit them. |
-| Approvals (`needsApproval`) | Supported | Not supported (local runtime only) |
-| Agent tool scripts | Supported | Supported |
-| Skills | Added to the session workspace | Available automatically |
-| Seeded files | Added to the session workspace | Available in a shared directory the first prompt names |
-| Repository | You provide it | The cloud agent checks it out |
-
-Use the local runtime when the host has the tools and files the agent
-needs. Use the cloud runtime when each turn needs an isolated repository
-checkout. `agent-sdk validate` warns when a cloud agent uses a
-local-only capability.
-
-## What files can a local session access?
-
-Each local session gets its own workspace. The Agent SDK writes the
-instructions as `AGENTS.md`, installs authored skills, copies sandbox
-files, and adds agent tool scripts.
-
-The workspace is a real Cursor project. It can inherit `AGENTS.md` and
-`.cursor` settings from parent directories. Nested git checkouts default
-`local.cwd` to a per-project cache directory under `~/.cache`. Point
-`cwd` at a checkout only when the agent should inherit that tree. `run`
-and `eval` already use a temporary state root.
-
-Session files live under `--state-root`. See
-[Sessions](./reference/sessions.md#where-does-the-agent-sdk-store-session-data)
-for the layout.
-
-## How can one agent call another?
-
-Every mounted agent also serves MCP at `/<slug>/v1/mcp`. Another agent
-or MCP client can use `ask`, `check`, and `call_tool` to delegate work.
-A peer MCP connection such as
-`defineConnection({ agent: "weather-agent" })` adds those tools to the
-calling agent.
-
-See [Agent-to-agent](./guides/agent-to-agent.md) for a complete example.
-
-## Which rules prevent common setup problems?
-
-- Use Node 22.13 or newer. Bun isn't supported.
-- Put evals under the project-root `evals/` directory, not
-  `agent/evals/`.
-- Run a TypeScript check before shipping. `validate` and `run` execute
-  TypeScript but don't type-check it.
-- Return JSON-shaped values from tool `execute` functions.
-- Keep local session workspaces away from parent rules you don't want
-  the agent to inherit.
-- Sign in or set `CURSOR_API_KEY` before starting a model turn.
-  Discovery, validation, direct tool calls, and server startup work
-  without a credential.
-
-## Related
-
-- [Quickstart](./quickstart.md)
-- [Project layout](./reference/project-layout.md)
-- [Sessions and streaming](./reference/sessions.md)
-- [Channels](./reference/channels.md)
-- [Live A/B metrics](./ab.md)