@cursor/july 0.1.107 → 0.1.109

package/dist/docs/ab.md

@@ -1,332 +0,0 @@
-# Live A/B metrics
-
-Use `defineAB` to compare variants on live agent sessions. New sessions
-receive a sticky assignment in each enrolled experiment. The Agent SDK folds
-their durable event streams into tool, token, failure, and wall-time
-metrics. You can send cumulative samples to your metrics backend and
-inspect aggregates in the playground.
-
-`defineAB` compares live variants through sticky assignment,
-instruction overlays, optional tool branches, and cumulative metrics.
-Metric callbacks observe the result without approving, rejecting, or
-failing a turn. Use [evals](/docs/evals.md) for pass/fail regression checks
-on fixed inputs.
-
-## Choose live A/B metrics or evals
-
-Both features read the session event stream, but they answer different
-questions.
-
-| | Live A/B metrics | Evals |
-| --- | --- | --- |
-| Question | How do variants compare on live sessions? | Does the agent still meet a fixed contract? |
-| Location | `agent/ab.ts` or `agent/ab/<name>.ts` | `evals/**/*.eval.ts` |
-| Input | Dev or production traffic | Frozen prompts and fixtures |
-| Output | Cumulative metrics by session and arm | Pass/fail assertions |
-| How it runs | Automatically on new live sessions | `agent-sdk eval` |
-
-There is no `agent-sdk ab` command or assertion API.
-
-## Define an experiment
-
-Author one experiment in `agent/ab.ts`, add more under
-`agent/ab/<name>.ts`, or use both forms. Each file defines one
-experiment. The experiment name comes from `name` when set. Otherwise,
-the Agent SDK uses `ab` for `agent/ab.ts` and the file stem for files under
-`agent/ab/`.
-
-```ts
-// agent/ab/concise-weather.ts
-import {
-  defineAB,
-  splitBySessionHash,
-} from "@cursor/july/ab";
-
-export default defineAB({
-  name: "concise-weather",
-  variants: {
-    control: {
-      label: "Baseline",
-    },
-    treatment: {
-      label: "Short replies",
-      description: "Adds a one-paragraph response limit.",
-      instructions: "Keep weather replies to one short paragraph.",
-    },
-  },
-  split: splitBySessionHash({
-    weights: { control: 1, treatment: 1 },
-    holdout: 0.1,
-  }),
-  derive: {
-    weatherCalls: (event) =>
-      event.type === "action.result" &&
-      event.data.toolName === "get_weather"
-        ? 1
-        : null,
-  },
-  onSample(sample) {
-    console.log(
-      sample.experiment,
-      sample.variant,
-      sample.metrics.toolCalls,
-      sample.metrics.wallTimeMs
-    );
-  },
-});
-```
-
-Every definition needs:
-
-- At least two variants. Variant keys cannot be empty or contain `/` or
-  `\`.
-- A `split` function that returns a variant key or `null`.
-- An `onSample` callback for completed or failed turns.
-
-`label` and `description` appear with the arm in result surfaces.
-`instructions` changes the prompt for sessions in that arm. `derive`
-adds custom counters.
-
-Duplicate experiment names are validation errors. Check discovery
-before you serve:
-
-```bash
-agent-sdk validate --dir .
-agent-sdk info --dir . --json
-```
-
-The `abs` field in `info` lists the discovered experiment names.
-
-## Assign sticky variants
-
-Enrollment happens once, when a live session is created and before its
-first turn:
-
-1. The Agent SDK records `session.started`.
-2. Each experiment runs its `split` function.
-3. The Agent SDK records one durable `ab.assigned` event per experiment.
-4. The selected arms become available on `session.abs`.
-5. Variant instruction overlays reach the first model turn.
-
-A split can return a variant key or `null`. A null assignment is a
-sticky skip for that experiment. It increments the experiment's
-`skipped` total, still appears in the snapshot's `sessions` list with
-`variant: null`, and does not collect arm metrics or call `onSample`.
-
-Use the split helper that matches your rollout:
-
-| Helper | Behavior |
-| --- | --- |
-| `splitBySessionHash({ weights?, holdout?, salt? })` | Hashes the session id into a reproducible arm; the recommended default |
-| `splitByRandom({ weights?, holdout? })` | Draws once when the session starts, then persists the result |
-| `splitAlways("control")` | Pins every new session to one arm |
-| `splitNone()` | Skips every new session without deleting the experiment |
-| `splitIf(predicate, inner)` | Runs `inner` only when the predicate passes |
-| Custom `split(ctx)` | Returns a declared variant key or `null` |
-
-The split context includes the agent name, channel id, session info,
-experiment name, and declared variant keys. For example, enroll only
-Slack sessions:
-
-```ts
-split: splitIf(
-  (ctx) => ctx.channel.id === "slack",
-  splitBySessionHash()
-),
-```
-
-Weights default to equal. Non-positive weights leave an arm out of the
-draw, and at least one arm must have a positive weight. `holdout` is the
-fraction of sessions assigned `null`, from `0` through `1`. Change
-`salt` to reshuffle future hash assignments without renaming the
-experiment.
-
-If a custom split throws or returns an unknown variant, the Agent SDK logs
-the error and records `variant: null`. The failed decision becomes a
-sticky skip instead of breaking the session.
-
-Enrollment only applies to new sessions. Adding an experiment does not
-assign existing conversations. Follow-ups keep the session's original
-arms. Keep experiment names and variant keys stable while you collect
-and compare results.
-
-## Change behavior by variant
-
-Variant instructions are appended to the agent's base instructions.
-Local sessions receive the merged instructions in `AGENTS.md` before
-every turn. Cloud sessions receive them in the first-turn preamble
-only. For cloud follow-ups, branch through `session.abs` when the arm
-must remain visible to deterministic behavior.
-
-Tools can branch on the assignment through `ctx.session.abs`. Hooks can
-read the same field for logging or export:
-
-```ts
-const treatment =
-  ctx.session.abs?.["concise-weather"] === "treatment";
-
-if (treatment) {
-  return conciseWeatherResult;
-}
-
-return baselineWeatherResult;
-```
-
-This makes the assignment available to deterministic code as well as
-the model prompt. Use both patterns together when one experiment must
-steer the prompt and host code at once.
-
-`defineAB` does not select a different model or runtime for each arm.
-Keep those settings in `agent/agent.ts`, or write explicit host logic
-when your experiment needs another behavior lever.
-
-The split and selected arm can affect agent behavior. `derive` and
-`onSample` only observe the resulting event stream. Errors in either
-callback are logged and never fail the turn.
-
-## Collect built-in and custom metrics
-
-Metrics accumulate for each session and experiment. When one session
-joins several experiments, every enrolled experiment folds the same
-turn and tool events into its own counters.
-
-| Metric | How the Agent SDK calculates it |
-| --- | --- |
-| `turns` | Adds one on `turn.completed` or `turn.failed` |
-| `turnFailures` | Adds one on `turn.failed` |
-| `toolCalls` | Adds one for each `action.result` |
-| `toolErrors` | Adds one when `action.result.data.isError` is true |
-| `inputTokens`, `outputTokens` | Adds usage from completed turns |
-| `cacheReadTokens`, `cacheWriteTokens` | Adds cache usage from completed turns |
-| `costUsd` | Sums the estimated turn cost recorded on `turn.completed` (turns whose model has no known rates contribute 0) |
-| `wallTimeMs` | Sums the time from `turn.started` to its completed or failed event |
-| `custom` | Sums finite numeric deltas returned by `derive` |
-
-`onSample` fires after every `turn.completed` and `turn.failed` event
-for an enrolled arm. The sample contains:
-
-| Field | Value |
-| --- | --- |
-| `experiment` | Experiment name |
-| `variant`, `variantLabel?` | Sticky arm and optional display label |
-| `sessionId`, `channelId` | Source session |
-| `metrics` | Cumulative metrics through this turn |
-| `reason` | `turn.completed` or `turn.failed` |
-| `at` | Terminal event timestamp |
-
-The metrics are cumulative, not per-turn deltas. A second sample from
-the same session includes the first turn's counts.
-
-Each `derive` extractor runs on every session event for its enrolled
-experiment, including streamed `message.appended` events. Keep it
-synchronous and cheap. Return a finite number to add a delta, or
-`null` to skip the event. Send samples to your metrics service from
-`onSample`; do not perform network or disk work in `derive`.
-
-Skipped sessions never call `onSample`. Errors from `derive` or
-`onSample` are logged, then metric collection continues.
-
-## Inspect assignments and results
-
-Open the playground's **A/Bs** tab to see aggregate arm totals and
-per-session assignments. The tab reads `GET /v1/abs`.
-
-The response has two views of the same durable data:
-
-| Field | Contents |
-| --- | --- |
-| `experiments` | Declared variants, skipped-session count, arm session counts, and aggregate metrics |
-| `sessions` | Visible sessions with their assignments and cumulative metrics |
-
-`GET /v1/abs` returns sessions visible to the current principal by
-default. In `--dev`, loopback requests include every session. Add
-`--allow-anonymous` to include every session from non-loopback callers
-too.
-
-The session event stream is the source of truth for assignment + fold.
-`GET /v1/abs` recomputes aggregates from those logs. Any
-`agent/storage.ts` exports samples and snapshots durably: an authored
-`abs` table when the backend has a native shape for it, or the table
-derived over the KV core otherwise. See
-[Storage](/docs/storage.md#eval-and-a-b-tables).
-
-## Configure the playground fold window
-
-Assignments and foldable metrics already persist in each session's
-event stream. The optional `agent/ab.config.ts` only caps how many
-sessions the playground and `GET /v1/abs` fold:
-
-```ts
-import { defineABConfig } from "@cursor/july/ab";
-
-export default defineABConfig({
-  // Optional. Defaults to 200. Only affects GET /v1/abs / A/Bs tab.
-  maxPlaygroundSessions: 500,
-});
-```
-
-`maxPlaygroundSessions` keeps the newest sessions in the fold. It does
-not prune session logs or change assignment. For export to S3, a DB, or
-your metrics vendor, send samples from `onSample` or declare a storage
-`abs` table.
-
-## Keep assignments durable
-
-The append-only event stream is the source of truth. Each
-`ab.assigned` event persists a variant key or null skip. Built-in
-metrics come from the turn and tool events that follow it.
-
-After a server restart or a parked session resumes, the live collector
-replays the stream to rebuild cumulative counters. Replay does not call
-`onSample` (or write to the storage `abs` table) for historical turns.
-Only a new completed or failed turn emits another sample.
-
-The snapshot API also replays `derive` across the full stream, so
-custom totals match the current extractor. Changing a derive function
-can change historical snapshot totals. Treat metric definitions as
-versioned experiment code.
-
-## Keep eval traffic separate
-
-Sessions created by `agent-sdk eval` and the playground Evals runner use
-`purpose: "eval"`. They skip A/B enrollment entirely:
-
-- No split function runs.
-- No `ab.assigned` event is recorded.
-- No `onSample` callback fires.
-- The session is omitted from `GET /v1/abs`.
-
-Ordinary chat, `agent-sdk run`, Slack, GitHub, and other channel sessions
-use the live purpose. You do not need `splitIf` to exclude eval traffic.
-
-## Know the boundaries
-
-`defineAB` provides sticky assignment, variant instructions,
-`session.abs` for tools, cumulative metrics, and local inspection. It
-does not provide:
-
-- A test command, assertion API, or pass/fail result
-- Statistical significance calculations
-- An experiment rollout or lifecycle service
-- Per-variant model or runtime configuration
-- A built-in analytics warehouse (bring your own via `onSample` or the
-  storage `abs` table)
-
-Use [evals](/docs/evals.md) to protect known behavior. Use `onSample` or a
-storage `abs` table when you need sample/snapshot exports beyond the
-session event log.
-
-## What's next
-
-Continue with these pages:
-
-- [Evals](/docs/evals.md): pass/fail regression checks on fixed inputs
-- [Hillclimbing](/docs/hillclimbing.md): improve an agent against fixed
-  fixtures
-- [Hooks](/docs/reference/hooks.md): other event-stream consumers
-- [Sessions and streaming](/docs/reference/sessions.md): the
-  `ab.assigned` event and durable log
-- [Playground](/docs/reference/playground.md): the A/Bs tab
-- [HTTP API](/docs/reference/http-api.md): `GET /v1/abs`
-- [Live A/B metrics skill](/docs/skills/ab.md): have a coding agent
-  wire an experiment

package/dist/docs/assets/ab.md.mlVgqvSk.js

@@ -1,54 +0,0 @@
-import{_ as s,c as a,o as t,ag as i}from"./chunks/framework.BCISBCiQ.js";const k=JSON.parse('{"title":"Live A/B metrics","description":"Assign sticky variants with defineAB, collect live performance metrics, and inspect per-arm results.","frontmatter":{"title":"Live A/B metrics","description":"Assign sticky variants with defineAB, collect live performance metrics, and inspect per-arm results."},"headers":[],"relativePath":"ab.md","filePath":"ab.md"}'),n={name:"ab.md"};function o(l,e,r,d,h,p){return t(),a("div",null,[...e[0]||(e[0]=[i(`<h1 id="live-a-b-metrics" tabindex="-1">Live A/B metrics <a class="header-anchor" href="#live-a-b-metrics" aria-label="Permalink to &quot;Live A/B metrics&quot;">​</a></h1><p>Use <code>defineAB</code> to compare variants on live agent sessions. New sessions receive a sticky assignment in each enrolled experiment. The Agent SDK folds their durable event streams into tool, token, failure, and wall-time metrics. You can send cumulative samples to your metrics backend and inspect aggregates in the playground.</p><p><code>defineAB</code> compares live variants through sticky assignment, instruction overlays, optional tool branches, and cumulative metrics. Metric callbacks observe the result without approving, rejecting, or failing a turn. Use <a href="./evals.html">evals</a> for pass/fail regression checks on fixed inputs.</p><h2 id="choose-live-a-b-metrics-or-evals" tabindex="-1">Choose live A/B metrics or evals <a class="header-anchor" href="#choose-live-a-b-metrics-or-evals" aria-label="Permalink to &quot;Choose live A/B metrics or evals&quot;">​</a></h2><p>Both features read the session event stream, but they answer different questions.</p><table tabindex="0"><thead><tr><th></th><th>Live A/B metrics</th><th>Evals</th></tr></thead><tbody><tr><td>Question</td><td>How do variants compare on live sessions?</td><td>Does the agent still meet a fixed contract?</td></tr><tr><td>Location</td><td><code>agent/ab.ts</code> or <code>agent/ab/&lt;name&gt;.ts</code></td><td><code>evals/**/*.eval.ts</code></td></tr><tr><td>Input</td><td>Dev or production traffic</td><td>Frozen prompts and fixtures</td></tr><tr><td>Output</td><td>Cumulative metrics by session and arm</td><td>Pass/fail assertions</td></tr><tr><td>How it runs</td><td>Automatically on new live sessions</td><td><code>agent-sdk eval</code></td></tr></tbody></table><p>There is no <code>agent-sdk ab</code> command or assertion API.</p><h2 id="define-an-experiment" tabindex="-1">Define an experiment <a class="header-anchor" href="#define-an-experiment" aria-label="Permalink to &quot;Define an experiment&quot;">​</a></h2><p>Author one experiment in <code>agent/ab.ts</code>, add more under <code>agent/ab/&lt;name&gt;.ts</code>, or use both forms. Each file defines one experiment. The experiment name comes from <code>name</code> when set. Otherwise, the Agent SDK uses <code>ab</code> for <code>agent/ab.ts</code> and the file stem for files under <code>agent/ab/</code>.</p><div class="language-ts vp-adaptive-theme"><button title="Copy Code" class="copy"></button><span class="lang">ts</span><pre class="shiki shiki-themes github-light github-dark vp-code" tabindex="0"><code><span class="line"><span style="--shiki-light:#6A737D;--shiki-dark:#6A737D;">// agent/ab/concise-weather.ts</span></span>
-<span class="line"><span style="--shiki-light:#D73A49;--shiki-dark:#F97583;">import</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;"> {</span></span>
-<span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">  defineAB,</span></span>
-<span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">  splitBySessionHash,</span></span>
-<span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">} </span><span style="--shiki-light:#D73A49;--shiki-dark:#F97583;">from</span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;"> &quot;@cursor/july/ab&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">;</span></span>
-<span class="line"></span>
-<span class="line"><span style="--shiki-light:#D73A49;--shiki-dark:#F97583;">export</span><span style="--shiki-light:#D73A49;--shiki-dark:#F97583;"> default</span><span style="--shiki-light:#6F42C1;--shiki-dark:#B392F0;"> defineAB</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">({</span></span>
-<span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">  name: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;concise-weather&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
-<span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">  variants: {</span></span>
-<span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">    control: {</span></span>
-<span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">      label: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;Baseline&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
-<span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">    },</span></span>
-<span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">    treatment: {</span></span>
-<span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">      label: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;Short replies&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
-<span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">      description: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;Adds a one-paragraph response limit.&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
-<span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">      instructions: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;Keep weather replies to one short paragraph.&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
-<span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">    },</span></span>
-<span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">  },</span></span>
-<span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">  split: </span><span style="--shiki-light:#6F42C1;--shiki-dark:#B392F0;">splitBySessionHash</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">({</span></span>
-<span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">    weights: { control: </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">1</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">, treatment: </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">1</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;"> },</span></span>
-<span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">    holdout: </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">0.1</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
-<span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">  }),</span></span>
-<span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">  derive: {</span></span>
-<span class="line"><span style="--shiki-light:#6F42C1;--shiki-dark:#B392F0;">    weatherCalls</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: (</span><span style="--shiki-light:#E36209;--shiki-dark:#FFAB70;">event</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">) </span><span style="--shiki-light:#D73A49;--shiki-dark:#F97583;">=&gt;</span></span>
-<span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">      event.type </span><span style="--shiki-light:#D73A49;--shiki-dark:#F97583;">===</span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;"> &quot;action.result&quot;</span><span style="--shiki-light:#D73A49;--shiki-dark:#F97583;"> &amp;&amp;</span></span>
-<span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">      event.data.toolName </span><span style="--shiki-light:#D73A49;--shiki-dark:#F97583;">===</span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;"> &quot;get_weather&quot;</span></span>
-<span class="line"><span style="--shiki-light:#D73A49;--shiki-dark:#F97583;">        ?</span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;"> 1</span></span>
-<span class="line"><span style="--shiki-light:#D73A49;--shiki-dark:#F97583;">        :</span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;"> null</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
-<span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">  },</span></span>
-<span class="line"><span style="--shiki-light:#6F42C1;--shiki-dark:#B392F0;">  onSample</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">(</span><span style="--shiki-light:#E36209;--shiki-dark:#FFAB70;">sample</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">) {</span></span>
-<span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">    console.</span><span style="--shiki-light:#6F42C1;--shiki-dark:#B392F0;">log</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">(</span></span>
-<span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">      sample.experiment,</span></span>
-<span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">      sample.variant,</span></span>
-<span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">      sample.metrics.toolCalls,</span></span>
-<span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">      sample.metrics.wallTimeMs</span></span>
-<span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">    );</span></span>
-<span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">  },</span></span>
-<span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">});</span></span></code></pre></div><p>Every definition needs:</p><ul><li>At least two variants. Variant keys cannot be empty or contain <code>/</code> or <code>\\</code>.</li><li>A <code>split</code> function that returns a variant key or <code>null</code>.</li><li>An <code>onSample</code> callback for completed or failed turns.</li></ul><p><code>label</code> and <code>description</code> appear with the arm in result surfaces. <code>instructions</code> changes the prompt for sessions in that arm. <code>derive</code> adds custom counters.</p><p>Duplicate experiment names are validation errors. Check discovery before you serve:</p><div class="language-bash vp-adaptive-theme"><button title="Copy Code" class="copy"></button><span class="lang">bash</span><pre class="shiki shiki-themes github-light github-dark vp-code" tabindex="0"><code><span class="line"><span style="--shiki-light:#6F42C1;--shiki-dark:#B392F0;">agent-sdk</span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;"> validate</span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;"> --dir</span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;"> .</span></span>
-<span class="line"><span style="--shiki-light:#6F42C1;--shiki-dark:#B392F0;">agent-sdk</span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;"> info</span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;"> --dir</span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;"> .</span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;"> --json</span></span></code></pre></div><p>The <code>abs</code> field in <code>info</code> lists the discovered experiment names.</p><h2 id="assign-sticky-variants" tabindex="-1">Assign sticky variants <a class="header-anchor" href="#assign-sticky-variants" aria-label="Permalink to &quot;Assign sticky variants&quot;">​</a></h2><p>Enrollment happens once, when a live session is created and before its first turn:</p><ol><li>The Agent SDK records <code>session.started</code>.</li><li>Each experiment runs its <code>split</code> function.</li><li>The Agent SDK records one durable <code>ab.assigned</code> event per experiment.</li><li>The selected arms become available on <code>session.abs</code>.</li><li>Variant instruction overlays reach the first model turn.</li></ol><p>A split can return a variant key or <code>null</code>. A null assignment is a sticky skip for that experiment. It increments the experiment&#39;s <code>skipped</code> total, still appears in the snapshot&#39;s <code>sessions</code> list with <code>variant: null</code>, and does not collect arm metrics or call <code>onSample</code>.</p><p>Use the split helper that matches your rollout:</p><table tabindex="0"><thead><tr><th>Helper</th><th>Behavior</th></tr></thead><tbody><tr><td><code>splitBySessionHash({ weights?, holdout?, salt? })</code></td><td>Hashes the session id into a reproducible arm; the recommended default</td></tr><tr><td><code>splitByRandom({ weights?, holdout? })</code></td><td>Draws once when the session starts, then persists the result</td></tr><tr><td><code>splitAlways(&quot;control&quot;)</code></td><td>Pins every new session to one arm</td></tr><tr><td><code>splitNone()</code></td><td>Skips every new session without deleting the experiment</td></tr><tr><td><code>splitIf(predicate, inner)</code></td><td>Runs <code>inner</code> only when the predicate passes</td></tr><tr><td>Custom <code>split(ctx)</code></td><td>Returns a declared variant key or <code>null</code></td></tr></tbody></table><p>The split context includes the agent name, channel id, session info, experiment name, and declared variant keys. For example, enroll only Slack sessions:</p><div class="language-ts vp-adaptive-theme"><button title="Copy Code" class="copy"></button><span class="lang">ts</span><pre class="shiki shiki-themes github-light github-dark vp-code" tabindex="0"><code><span class="line"><span style="--shiki-light:#6F42C1;--shiki-dark:#B392F0;">split</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#6F42C1;--shiki-dark:#B392F0;">splitIf</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">(</span></span>
-<span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">  (</span><span style="--shiki-light:#E36209;--shiki-dark:#FFAB70;">ctx</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">) </span><span style="--shiki-light:#D73A49;--shiki-dark:#F97583;">=&gt;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;"> ctx.channel.id </span><span style="--shiki-light:#D73A49;--shiki-dark:#F97583;">===</span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;"> &quot;slack&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
-<span class="line"><span style="--shiki-light:#6F42C1;--shiki-dark:#B392F0;">  splitBySessionHash</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">()</span></span>
-<span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">),</span></span></code></pre></div><p>Weights default to equal. Non-positive weights leave an arm out of the draw, and at least one arm must have a positive weight. <code>holdout</code> is the fraction of sessions assigned <code>null</code>, from <code>0</code> through <code>1</code>. Change <code>salt</code> to reshuffle future hash assignments without renaming the experiment.</p><p>If a custom split throws or returns an unknown variant, the Agent SDK logs the error and records <code>variant: null</code>. The failed decision becomes a sticky skip instead of breaking the session.</p><p>Enrollment only applies to new sessions. Adding an experiment does not assign existing conversations. Follow-ups keep the session&#39;s original arms. Keep experiment names and variant keys stable while you collect and compare results.</p><h2 id="change-behavior-by-variant" tabindex="-1">Change behavior by variant <a class="header-anchor" href="#change-behavior-by-variant" aria-label="Permalink to &quot;Change behavior by variant&quot;">​</a></h2><p>Variant instructions are appended to the agent&#39;s base instructions. Local sessions receive the merged instructions in <code>AGENTS.md</code> before every turn. Cloud sessions receive them in the first-turn preamble only. For cloud follow-ups, branch through <code>session.abs</code> when the arm must remain visible to deterministic behavior.</p><p>Tools can branch on the assignment through <code>ctx.session.abs</code>. Hooks can read the same field for logging or export:</p><div class="language-ts vp-adaptive-theme"><button title="Copy Code" class="copy"></button><span class="lang">ts</span><pre class="shiki shiki-themes github-light github-dark vp-code" tabindex="0"><code><span class="line"><span style="--shiki-light:#D73A49;--shiki-dark:#F97583;">const</span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;"> treatment</span><span style="--shiki-light:#D73A49;--shiki-dark:#F97583;"> =</span></span>
-<span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">  ctx.session.abs?.[</span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;concise-weather&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">] </span><span style="--shiki-light:#D73A49;--shiki-dark:#F97583;">===</span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;"> &quot;treatment&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">;</span></span>
-<span class="line"></span>
-<span class="line"><span style="--shiki-light:#D73A49;--shiki-dark:#F97583;">if</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;"> (treatment) {</span></span>
-<span class="line"><span style="--shiki-light:#D73A49;--shiki-dark:#F97583;">  return</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;"> conciseWeatherResult;</span></span>
-<span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">}</span></span>
-<span class="line"></span>
-<span class="line"><span style="--shiki-light:#D73A49;--shiki-dark:#F97583;">return</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;"> baselineWeatherResult;</span></span></code></pre></div><p>This makes the assignment available to deterministic code as well as the model prompt. Use both patterns together when one experiment must steer the prompt and host code at once.</p><p><code>defineAB</code> does not select a different model or runtime for each arm. Keep those settings in <code>agent/agent.ts</code>, or write explicit host logic when your experiment needs another behavior lever.</p><p>The split and selected arm can affect agent behavior. <code>derive</code> and <code>onSample</code> only observe the resulting event stream. Errors in either callback are logged and never fail the turn.</p><h2 id="collect-built-in-and-custom-metrics" tabindex="-1">Collect built-in and custom metrics <a class="header-anchor" href="#collect-built-in-and-custom-metrics" aria-label="Permalink to &quot;Collect built-in and custom metrics&quot;">​</a></h2><p>Metrics accumulate for each session and experiment. When one session joins several experiments, every enrolled experiment folds the same turn and tool events into its own counters.</p><table tabindex="0"><thead><tr><th>Metric</th><th>How the Agent SDK calculates it</th></tr></thead><tbody><tr><td><code>turns</code></td><td>Adds one on <code>turn.completed</code> or <code>turn.failed</code></td></tr><tr><td><code>turnFailures</code></td><td>Adds one on <code>turn.failed</code></td></tr><tr><td><code>toolCalls</code></td><td>Adds one for each <code>action.result</code></td></tr><tr><td><code>toolErrors</code></td><td>Adds one when <code>action.result.data.isError</code> is true</td></tr><tr><td><code>inputTokens</code>, <code>outputTokens</code></td><td>Adds usage from completed turns</td></tr><tr><td><code>cacheReadTokens</code>, <code>cacheWriteTokens</code></td><td>Adds cache usage from completed turns</td></tr><tr><td><code>costUsd</code></td><td>Sums the estimated turn cost recorded on <code>turn.completed</code> (turns whose model has no known rates contribute 0)</td></tr><tr><td><code>wallTimeMs</code></td><td>Sums the time from <code>turn.started</code> to its completed or failed event</td></tr><tr><td><code>custom</code></td><td>Sums finite numeric deltas returned by <code>derive</code></td></tr></tbody></table><p><code>onSample</code> fires after every <code>turn.completed</code> and <code>turn.failed</code> event for an enrolled arm. The sample contains:</p><table tabindex="0"><thead><tr><th>Field</th><th>Value</th></tr></thead><tbody><tr><td><code>experiment</code></td><td>Experiment name</td></tr><tr><td><code>variant</code>, <code>variantLabel?</code></td><td>Sticky arm and optional display label</td></tr><tr><td><code>sessionId</code>, <code>channelId</code></td><td>Source session</td></tr><tr><td><code>metrics</code></td><td>Cumulative metrics through this turn</td></tr><tr><td><code>reason</code></td><td><code>turn.completed</code> or <code>turn.failed</code></td></tr><tr><td><code>at</code></td><td>Terminal event timestamp</td></tr></tbody></table><p>The metrics are cumulative, not per-turn deltas. A second sample from the same session includes the first turn&#39;s counts.</p><p>Each <code>derive</code> extractor runs on every session event for its enrolled experiment, including streamed <code>message.appended</code> events. Keep it synchronous and cheap. Return a finite number to add a delta, or <code>null</code> to skip the event. Send samples to your metrics service from <code>onSample</code>; do not perform network or disk work in <code>derive</code>.</p><p>Skipped sessions never call <code>onSample</code>. Errors from <code>derive</code> or <code>onSample</code> are logged, then metric collection continues.</p><h2 id="inspect-assignments-and-results" tabindex="-1">Inspect assignments and results <a class="header-anchor" href="#inspect-assignments-and-results" aria-label="Permalink to &quot;Inspect assignments and results&quot;">​</a></h2><p>Open the playground&#39;s <strong>A/Bs</strong> tab to see aggregate arm totals and per-session assignments. The tab reads <code>GET /v1/abs</code>.</p><p>The response has two views of the same durable data:</p><table tabindex="0"><thead><tr><th>Field</th><th>Contents</th></tr></thead><tbody><tr><td><code>experiments</code></td><td>Declared variants, skipped-session count, arm session counts, and aggregate metrics</td></tr><tr><td><code>sessions</code></td><td>Visible sessions with their assignments and cumulative metrics</td></tr></tbody></table><p><code>GET /v1/abs</code> returns sessions visible to the current principal by default. In <code>--dev</code>, loopback requests include every session. Add <code>--allow-anonymous</code> to include every session from non-loopback callers too.</p><p>The session event stream is the source of truth for assignment + fold. <code>GET /v1/abs</code> recomputes aggregates from those logs. Any <code>agent/storage.ts</code> exports samples and snapshots durably: an authored <code>abs</code> table when the backend has a native shape for it, or the table derived over the KV core otherwise. See <a href="./storage.html#eval-and-a-b-tables">Storage</a>.</p><h2 id="configure-the-playground-fold-window" tabindex="-1">Configure the playground fold window <a class="header-anchor" href="#configure-the-playground-fold-window" aria-label="Permalink to &quot;Configure the playground fold window&quot;">​</a></h2><p>Assignments and foldable metrics already persist in each session&#39;s event stream. The optional <code>agent/ab.config.ts</code> only caps how many sessions the playground and <code>GET /v1/abs</code> fold:</p><div class="language-ts vp-adaptive-theme"><button title="Copy Code" class="copy"></button><span class="lang">ts</span><pre class="shiki shiki-themes github-light github-dark vp-code" tabindex="0"><code><span class="line"><span style="--shiki-light:#D73A49;--shiki-dark:#F97583;">import</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;"> { defineABConfig } </span><span style="--shiki-light:#D73A49;--shiki-dark:#F97583;">from</span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;"> &quot;@cursor/july/ab&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">;</span></span>
-<span class="line"></span>
-<span class="line"><span style="--shiki-light:#D73A49;--shiki-dark:#F97583;">export</span><span style="--shiki-light:#D73A49;--shiki-dark:#F97583;"> default</span><span style="--shiki-light:#6F42C1;--shiki-dark:#B392F0;"> defineABConfig</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">({</span></span>
-<span class="line"><span style="--shiki-light:#6A737D;--shiki-dark:#6A737D;">  // Optional. Defaults to 200. Only affects GET /v1/abs / A/Bs tab.</span></span>
-<span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">  maxPlaygroundSessions: </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">500</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
-<span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">});</span></span></code></pre></div><p><code>maxPlaygroundSessions</code> keeps the newest sessions in the fold. It does not prune session logs or change assignment. For export to S3, a DB, or your metrics vendor, send samples from <code>onSample</code> or declare a storage <code>abs</code> table.</p><h2 id="keep-assignments-durable" tabindex="-1">Keep assignments durable <a class="header-anchor" href="#keep-assignments-durable" aria-label="Permalink to &quot;Keep assignments durable&quot;">​</a></h2><p>The append-only event stream is the source of truth. Each <code>ab.assigned</code> event persists a variant key or null skip. Built-in metrics come from the turn and tool events that follow it.</p><p>After a server restart or a parked session resumes, the live collector replays the stream to rebuild cumulative counters. Replay does not call <code>onSample</code> (or write to the storage <code>abs</code> table) for historical turns. Only a new completed or failed turn emits another sample.</p><p>The snapshot API also replays <code>derive</code> across the full stream, so custom totals match the current extractor. Changing a derive function can change historical snapshot totals. Treat metric definitions as versioned experiment code.</p><h2 id="keep-eval-traffic-separate" tabindex="-1">Keep eval traffic separate <a class="header-anchor" href="#keep-eval-traffic-separate" aria-label="Permalink to &quot;Keep eval traffic separate&quot;">​</a></h2><p>Sessions created by <code>agent-sdk eval</code> and the playground Evals runner use <code>purpose: &quot;eval&quot;</code>. They skip A/B enrollment entirely:</p><ul><li>No split function runs.</li><li>No <code>ab.assigned</code> event is recorded.</li><li>No <code>onSample</code> callback fires.</li><li>The session is omitted from <code>GET /v1/abs</code>.</li></ul><p>Ordinary chat, <code>agent-sdk run</code>, Slack, GitHub, and other channel sessions use the live purpose. You do not need <code>splitIf</code> to exclude eval traffic.</p><h2 id="know-the-boundaries" tabindex="-1">Know the boundaries <a class="header-anchor" href="#know-the-boundaries" aria-label="Permalink to &quot;Know the boundaries&quot;">​</a></h2><p><code>defineAB</code> provides sticky assignment, variant instructions, <code>session.abs</code> for tools, cumulative metrics, and local inspection. It does not provide:</p><ul><li>A test command, assertion API, or pass/fail result</li><li>Statistical significance calculations</li><li>An experiment rollout or lifecycle service</li><li>Per-variant model or runtime configuration</li><li>A built-in analytics warehouse (bring your own via <code>onSample</code> or the storage <code>abs</code> table)</li></ul><p>Use <a href="./evals.html">evals</a> to protect known behavior. Use <code>onSample</code> or a storage <code>abs</code> table when you need sample/snapshot exports beyond the session event log.</p><h2 id="what-s-next" tabindex="-1">What&#39;s next <a class="header-anchor" href="#what-s-next" aria-label="Permalink to &quot;What&#39;s next&quot;">​</a></h2><p>Continue with these pages:</p><ul><li><a href="./evals.html">Evals</a>: pass/fail regression checks on fixed inputs</li><li><a href="./hillclimbing.html">Hillclimbing</a>: improve an agent against fixed fixtures</li><li><a href="./reference/hooks.html">Hooks</a>: other event-stream consumers</li><li><a href="./reference/sessions.html">Sessions and streaming</a>: the <code>ab.assigned</code> event and durable log</li><li><a href="./reference/playground.html">Playground</a>: the A/Bs tab</li><li><a href="./reference/http-api.html">HTTP API</a>: <code>GET /v1/abs</code></li><li><a href="./skills/ab.html">Live A/B metrics skill</a>: have a coding agent wire an experiment</li></ul>`,67)])])}const u=s(n,[["render",o]]);export{k as __pageData,u as default};

package/dist/docs/assets/ab.md.mlVgqvSk.lean.js

@@ -1 +0,0 @@
-import{_ as s,c as a,o as t,ag as i}from"./chunks/framework.BCISBCiQ.js";const k=JSON.parse('{"title":"Live A/B metrics","description":"Assign sticky variants with defineAB, collect live performance metrics, and inspect per-arm results.","frontmatter":{"title":"Live A/B metrics","description":"Assign sticky variants with defineAB, collect live performance metrics, and inspect per-arm results."},"headers":[],"relativePath":"ab.md","filePath":"ab.md"}'),n={name:"ab.md"};function o(l,e,r,d,h,p){return t(),a("div",null,[...e[0]||(e[0]=[i("",67)])])}const u=s(n,[["render",o]]);export{k as __pageData,u as default};