@cursor/july 0.1.107 → 0.1.109

package/docs/ab.md

@@ -1,337 +0,0 @@
----
-title: "Live A/B metrics"
-description: "Assign sticky variants with defineAB, collect live performance metrics, and inspect per-arm results."
----
-
-# Live A/B metrics
-
-Use `defineAB` to compare variants on live agent sessions. New sessions
-receive a sticky assignment in each enrolled experiment. The Agent SDK folds
-their durable event streams into tool, token, failure, and wall-time
-metrics. You can send cumulative samples to your metrics backend and
-inspect aggregates in the playground.
-
-`defineAB` compares live variants through sticky assignment,
-instruction overlays, optional tool branches, and cumulative metrics.
-Metric callbacks observe the result without approving, rejecting, or
-failing a turn. Use [evals](./evals.md) for pass/fail regression checks
-on fixed inputs.
-
-## Choose live A/B metrics or evals
-
-Both features read the session event stream, but they answer different
-questions.
-
-| | Live A/B metrics | Evals |
-| --- | --- | --- |
-| Question | How do variants compare on live sessions? | Does the agent still meet a fixed contract? |
-| Location | `agent/ab.ts` or `agent/ab/<name>.ts` | `evals/**/*.eval.ts` |
-| Input | Dev or production traffic | Frozen prompts and fixtures |
-| Output | Cumulative metrics by session and arm | Pass/fail assertions |
-| How it runs | Automatically on new live sessions | `agent-sdk eval` |
-
-There is no `agent-sdk ab` command or assertion API.
-
-## Define an experiment
-
-Author one experiment in `agent/ab.ts`, add more under
-`agent/ab/<name>.ts`, or use both forms. Each file defines one
-experiment. The experiment name comes from `name` when set. Otherwise,
-the Agent SDK uses `ab` for `agent/ab.ts` and the file stem for files under
-`agent/ab/`.
-
-```ts
-// agent/ab/concise-weather.ts
-import {
-  defineAB,
-  splitBySessionHash,
-} from "@cursor/july/ab";
-
-export default defineAB({
-  name: "concise-weather",
-  variants: {
-    control: {
-      label: "Baseline",
-    },
-    treatment: {
-      label: "Short replies",
-      description: "Adds a one-paragraph response limit.",
-      instructions: "Keep weather replies to one short paragraph.",
-    },
-  },
-  split: splitBySessionHash({
-    weights: { control: 1, treatment: 1 },
-    holdout: 0.1,
-  }),
-  derive: {
-    weatherCalls: (event) =>
-      event.type === "action.result" &&
-      event.data.toolName === "get_weather"
-        ? 1
-        : null,
-  },
-  onSample(sample) {
-    console.log(
-      sample.experiment,
-      sample.variant,
-      sample.metrics.toolCalls,
-      sample.metrics.wallTimeMs
-    );
-  },
-});
-```
-
-Every definition needs:
-
-- At least two variants. Variant keys cannot be empty or contain `/` or
-  `\`.
-- A `split` function that returns a variant key or `null`.
-- An `onSample` callback for completed or failed turns.
-
-`label` and `description` appear with the arm in result surfaces.
-`instructions` changes the prompt for sessions in that arm. `derive`
-adds custom counters.
-
-Duplicate experiment names are validation errors. Check discovery
-before you serve:
-
-```bash
-agent-sdk validate --dir .
-agent-sdk info --dir . --json
-```
-
-The `abs` field in `info` lists the discovered experiment names.
-
-## Assign sticky variants
-
-Enrollment happens once, when a live session is created and before its
-first turn:
-
-1. The Agent SDK records `session.started`.
-2. Each experiment runs its `split` function.
-3. The Agent SDK records one durable `ab.assigned` event per experiment.
-4. The selected arms become available on `session.abs`.
-5. Variant instruction overlays reach the first model turn.
-
-A split can return a variant key or `null`. A null assignment is a
-sticky skip for that experiment. It increments the experiment's
-`skipped` total, still appears in the snapshot's `sessions` list with
-`variant: null`, and does not collect arm metrics or call `onSample`.
-
-Use the split helper that matches your rollout:
-
-| Helper | Behavior |
-| --- | --- |
-| `splitBySessionHash({ weights?, holdout?, salt? })` | Hashes the session id into a reproducible arm; the recommended default |
-| `splitByRandom({ weights?, holdout? })` | Draws once when the session starts, then persists the result |
-| `splitAlways("control")` | Pins every new session to one arm |
-| `splitNone()` | Skips every new session without deleting the experiment |
-| `splitIf(predicate, inner)` | Runs `inner` only when the predicate passes |
-| Custom `split(ctx)` | Returns a declared variant key or `null` |
-
-The split context includes the agent name, channel id, session info,
-experiment name, and declared variant keys. For example, enroll only
-Slack sessions:
-
-```ts
-split: splitIf(
-  (ctx) => ctx.channel.id === "slack",
-  splitBySessionHash()
-),
-```
-
-Weights default to equal. Non-positive weights leave an arm out of the
-draw, and at least one arm must have a positive weight. `holdout` is the
-fraction of sessions assigned `null`, from `0` through `1`. Change
-`salt` to reshuffle future hash assignments without renaming the
-experiment.
-
-If a custom split throws or returns an unknown variant, the Agent SDK logs
-the error and records `variant: null`. The failed decision becomes a
-sticky skip instead of breaking the session.
-
-Enrollment only applies to new sessions. Adding an experiment does not
-assign existing conversations. Follow-ups keep the session's original
-arms. Keep experiment names and variant keys stable while you collect
-and compare results.
-
-## Change behavior by variant
-
-Variant instructions are appended to the agent's base instructions.
-Local sessions receive the merged instructions in `AGENTS.md` before
-every turn. Cloud sessions receive them in the first-turn preamble
-only. For cloud follow-ups, branch through `session.abs` when the arm
-must remain visible to deterministic behavior.
-
-Tools can branch on the assignment through `ctx.session.abs`. Hooks can
-read the same field for logging or export:
-
-```ts
-const treatment =
-  ctx.session.abs?.["concise-weather"] === "treatment";
-
-if (treatment) {
-  return conciseWeatherResult;
-}
-
-return baselineWeatherResult;
-```
-
-This makes the assignment available to deterministic code as well as
-the model prompt. Use both patterns together when one experiment must
-steer the prompt and host code at once.
-
-`defineAB` does not select a different model or runtime for each arm.
-Keep those settings in `agent/agent.ts`, or write explicit host logic
-when your experiment needs another behavior lever.
-
-The split and selected arm can affect agent behavior. `derive` and
-`onSample` only observe the resulting event stream. Errors in either
-callback are logged and never fail the turn.
-
-## Collect built-in and custom metrics
-
-Metrics accumulate for each session and experiment. When one session
-joins several experiments, every enrolled experiment folds the same
-turn and tool events into its own counters.
-
-| Metric | How the Agent SDK calculates it |
-| --- | --- |
-| `turns` | Adds one on `turn.completed` or `turn.failed` |
-| `turnFailures` | Adds one on `turn.failed` |
-| `toolCalls` | Adds one for each `action.result` |
-| `toolErrors` | Adds one when `action.result.data.isError` is true |
-| `inputTokens`, `outputTokens` | Adds usage from completed turns |
-| `cacheReadTokens`, `cacheWriteTokens` | Adds cache usage from completed turns |
-| `costUsd` | Sums the estimated turn cost recorded on `turn.completed` (turns whose model has no known rates contribute 0) |
-| `wallTimeMs` | Sums the time from `turn.started` to its completed or failed event |
-| `custom` | Sums finite numeric deltas returned by `derive` |
-
-`onSample` fires after every `turn.completed` and `turn.failed` event
-for an enrolled arm. The sample contains:
-
-| Field | Value |
-| --- | --- |
-| `experiment` | Experiment name |
-| `variant`, `variantLabel?` | Sticky arm and optional display label |
-| `sessionId`, `channelId` | Source session |
-| `metrics` | Cumulative metrics through this turn |
-| `reason` | `turn.completed` or `turn.failed` |
-| `at` | Terminal event timestamp |
-
-The metrics are cumulative, not per-turn deltas. A second sample from
-the same session includes the first turn's counts.
-
-Each `derive` extractor runs on every session event for its enrolled
-experiment, including streamed `message.appended` events. Keep it
-synchronous and cheap. Return a finite number to add a delta, or
-`null` to skip the event. Send samples to your metrics service from
-`onSample`; do not perform network or disk work in `derive`.
-
-Skipped sessions never call `onSample`. Errors from `derive` or
-`onSample` are logged, then metric collection continues.
-
-## Inspect assignments and results
-
-Open the playground's **A/Bs** tab to see aggregate arm totals and
-per-session assignments. The tab reads `GET /v1/abs`.
-
-The response has two views of the same durable data:
-
-| Field | Contents |
-| --- | --- |
-| `experiments` | Declared variants, skipped-session count, arm session counts, and aggregate metrics |
-| `sessions` | Visible sessions with their assignments and cumulative metrics |
-
-`GET /v1/abs` returns sessions visible to the current principal by
-default. In `--dev`, loopback requests include every session. Add
-`--allow-anonymous` to include every session from non-loopback callers
-too.
-
-The session event stream is the source of truth for assignment + fold.
-`GET /v1/abs` recomputes aggregates from those logs. Any
-`agent/storage.ts` exports samples and snapshots durably: an authored
-`abs` table when the backend has a native shape for it, or the table
-derived over the KV core otherwise. See
-[Storage](./storage.md#eval-and-a-b-tables).
-
-## Configure the playground fold window
-
-Assignments and foldable metrics already persist in each session's
-event stream. The optional `agent/ab.config.ts` only caps how many
-sessions the playground and `GET /v1/abs` fold:
-
-```ts
-import { defineABConfig } from "@cursor/july/ab";
-
-export default defineABConfig({
-  // Optional. Defaults to 200. Only affects GET /v1/abs / A/Bs tab.
-  maxPlaygroundSessions: 500,
-});
-```
-
-`maxPlaygroundSessions` keeps the newest sessions in the fold. It does
-not prune session logs or change assignment. For export to S3, a DB, or
-your metrics vendor, send samples from `onSample` or declare a storage
-`abs` table.
-
-## Keep assignments durable
-
-The append-only event stream is the source of truth. Each
-`ab.assigned` event persists a variant key or null skip. Built-in
-metrics come from the turn and tool events that follow it.
-
-After a server restart or a parked session resumes, the live collector
-replays the stream to rebuild cumulative counters. Replay does not call
-`onSample` (or write to the storage `abs` table) for historical turns.
-Only a new completed or failed turn emits another sample.
-
-The snapshot API also replays `derive` across the full stream, so
-custom totals match the current extractor. Changing a derive function
-can change historical snapshot totals. Treat metric definitions as
-versioned experiment code.
-
-## Keep eval traffic separate
-
-Sessions created by `agent-sdk eval` and the playground Evals runner use
-`purpose: "eval"`. They skip A/B enrollment entirely:
-
-- No split function runs.
-- No `ab.assigned` event is recorded.
-- No `onSample` callback fires.
-- The session is omitted from `GET /v1/abs`.
-
-Ordinary chat, `agent-sdk run`, Slack, GitHub, and other channel sessions
-use the live purpose. You do not need `splitIf` to exclude eval traffic.
-
-## Know the boundaries
-
-`defineAB` provides sticky assignment, variant instructions,
-`session.abs` for tools, cumulative metrics, and local inspection. It
-does not provide:
-
-- A test command, assertion API, or pass/fail result
-- Statistical significance calculations
-- An experiment rollout or lifecycle service
-- Per-variant model or runtime configuration
-- A built-in analytics warehouse (bring your own via `onSample` or the
-  storage `abs` table)
-
-Use [evals](./evals.md) to protect known behavior. Use `onSample` or a
-storage `abs` table when you need sample/snapshot exports beyond the
-session event log.
-
-## What's next
-
-Continue with these pages:
-
-- [Evals](./evals.md): pass/fail regression checks on fixed inputs
-- [Hillclimbing](./hillclimbing.md): improve an agent against fixed
-  fixtures
-- [Hooks](./reference/hooks.md): other event-stream consumers
-- [Sessions and streaming](./reference/sessions.md): the
-  `ab.assigned` event and durable log
-- [Playground](./reference/playground.md): the A/Bs tab
-- [HTTP API](./reference/http-api.md): `GET /v1/abs`
-- [Live A/B metrics skill](./skills/ab.md): have a coding agent
-  wire an experiment

package/skills/ab/SKILL.md

@@ -1,58 +0,0 @@
----
-name: agentsdk-ab
-description: >-
-  Author defineAB live A/B metrics under agent/ab. Sticky variants
-  and onSample on live runs. Not a test runner. Use when comparing
-  variants on traffic; use defineEval for regression gates.
----
-
-# Agent SDK A/B metrics (`defineAB`)
-
-Live metrics plug-in. No `agent-sdk ab` CLI. No assertion API.
-Reference: `docs/ab.md`.
-
-| | `defineEval` | `defineAB` |
-| --- | --- | --- |
-| Job | Gates on frozen fixtures | Metrics on live runs |
-| Location | `evals/**/*.eval.ts` | `agent/ab.ts` or `agent/ab/<name>.ts` |
-| How it runs | `agent-sdk eval` | Under `serve` / `run` |
-
-```ts
-import { defineAB, splitBySessionHash } from "@cursor/july/ab";
-
-export default defineAB({
-  name: "concise-instructions",
-  variants: {
-    control: { label: "Baseline" },
-    treatment: {
-      label: "Shorter",
-      instructions: "Keep replies to one short paragraph.",
-    },
-  },
-  split: splitBySessionHash({ holdout: 0.1 }),
-  derive: {
-    weatherCalls: (event) =>
-      event.type === "action.result" && event.data.toolName === "get_weather"
-        ? 1
-        : null,
-  },
-  onSample(sample) {
-    console.log(sample.variant, sample.metrics.toolCalls, sample.metrics.wallTimeMs);
-  },
-});
-```
-
-```ts
-async execute(input, ctx) {
-  if (ctx.session.abs?.["concise-instructions"] === "treatment") {
-    // treatment-specific behavior
-  }
-}
-```
-
-Enrollment is at session creation. Eval sessions skip it. Do not
-use `splitIf` to filter evals. Split helpers and `onSample`
-fields: `docs/ab.md`.
-
-Pick a name, arm labels, a split, and a real `onSample` sink. Do
-not invent credentials.