@statelyai/agent 2.0.0-alpha.28 → 2.0.0-alpha.29

package/skills/generate-machine/SKILL.md

@@ -1,16 +1,38 @@
 ---
 name: generate-machine
-description: Author a @statelyai/agent workflow as JSON and check it before it runs. Use when asked to generate or author an agent machine, create a workflow config, write an AgentWorkflowConfig, or produce an LLM-authored state machine for an agent — and when a generated config fails Ajv validation or agent lint and needs repair.
+description: Author a @statelyai/agent workflow as JSON and check it before it runs. Use when asked to generate or author an agent machine, create a workflow config, write an AgentWorkflowConfig, or produce an LLM-authored state machine for an agent — and when a generated config fails verifyAgentConfig, Ajv validation, or agent lint and needs repair.
 ---
 
 # Generate an agent machine
 
-An agent machine is data. You author a JSON `AgentWorkflowConfig`, then run it through gates that all work with no API key: Ajv → `fromConfig` → lint → simulate. Do not hand back a config that has not passed all four.
+An agent machine is data. You author a JSON `AgentWorkflowConfig`, then run it through `verifyAgentConfig`, whose gates all work with no API key: validate → vocabulary → lower → lint → explore. Do not hand back a config that has not passed every gate.
 
 ```
-author → validate (validateAgentConfig) → lower (fromConfig) → lint (lintAgentMachine) → simulate → hand back
+split environment/policy → author (description = goal) → verifyAgentConfig → repair (≤ 3) → persist traces → hand back
 ```
 
+To improve an existing policy from its traces, use the `improve-machine` skill instead.
+
+## 0. Split environment and policy
+
+Decide what is fixed before you write any JSON.
+
+- **Environment** (TypeScript, fixed): the rules of the world and the business invariants, as states and guards. People author it and verify it with `canReach` / `explorePaths` from `@statelyai/agent/testing`. You never generate it, and no generated config may change it.
+- **Policy** (JSON, generated): how the agent works toward the goal: which step next, which tool, when to ask a person, cheap judgment vs model call. This is the `AgentWorkflowConfig` you author.
+- The policy plugs into the environment at its decision points and outputs one event; the environment's guards decide whether to accept it.
+- No split needed for a single judgment inside a fixed workflow, or for a workflow where every step is dictated by rules. Author one machine.
+
+If the request asks you to generate a business rule (an approval threshold, a payout limit), stop: that belongs in the environment, in TypeScript, reviewed by a person. Say so.
+
+Before editing an existing machine, read it as text:
+
+```ts
+import { describeMachine } from "@statelyai/agent";
+const text = describeMachine(machine); // "Machine <id>", then "Goal: …", states, transitions, events
+```
+
+Pass `{ snapshot }` to mark the active states with `*`. Concept page: `docs/environment-and-policy.md`.
+
 ## 1. Read the schema
 
 The config shape is fixed by a shipped JSON Schema. Read it first; it is the contract.
@@ -20,23 +42,32 @@ The config shape is fixed by a shipped JSON Schema. Read it first; it is the con
 
 ## 2. Author the config
 
+Start with the root `description`: it is the machine's **goal**, the overall objective plus measurable success criteria. `describeMachine` renders it first as `Goal: …`, and an improver later climbs toward it. Write criteria a scorer can compute from outcomes (terminal states, effects in the world, cost), not "be helpful":
+
+```json
+"description": "Resolve each support ticket in one session. Success: escalations only for tickets that need a human; every reply approved before sending; at most 2 model calls per ticket."
+```
+
 Rules the runtime enforces. Each one maps to a build error or a lint diagnostic:
 
 - Every value is a JSON literal or a whole-string `"{{ }}"` expression reading a dot path on `input`, `context`, or `event`. No JavaScript.
 - Guard expressions are truthy-only: no comparisons, operators, or negation.
-- A model call is either a named `requests` entry invoked with `src: "<requestName>"`, or an inline invoke with `src: "agent.decide"` plus an `allowedEvents` list.
+- A model call is either a named `requests` entry invoked with `src: "<requestName>"`, or an inline invoke with `src: "agent.decide"` plus an `allowedEvents` list. A `requests` entry's templates read only its own `input` (`"{{ input.ticket }}"`); the invoke's `input` maps context into it.
 - A state invoking `agent.decide` MUST handle every allowed event in its `on`.
 - An `agent.decide` invoke has no `onDone` (a decision produces no output). Use `onError` for the retries-exhausted path.
 - A request invoke resolves with `{ result, messages }`, so `onDone.assign` reads its result from `"{{ event.output.result.<field> }}"` (`"{{ event.output.result }}"` for a plain-text request). A non-request `actors` invoke reads `"{{ event.output.<field> }}"` directly.
 - Every path must reach a `"type": "final"` state, and each final state needs an `output` when `schemas.output` is declared.
 - Do not invent guard or action names. Only `"{{ }}"` guards, `assign`, and `emit` exist unless the host tells you which named guards/actions it implements.
 - Model refs are strings (`"openai/gpt-5.4-mini"`); the host resolves them.
+- To make a decision a closed choice, add `payloads` to the `agent.decide` input: one payload list per event type, without `type` (`"payloads": { "REFUND": [{ "amount": 20 }, { "amount": 50 }], "ESCALATE": [] }`). The model then picks one listed event instead of writing a payload. A type with no key is offered as `{ type }`; `[]` withholds it.
+- An idle human step can carry `meta.interaction` for the host to render with `getInteraction`: a `label`, `events` (one button per event type), `choices` (one button per full event, `{ "label": "Refund $20", "event": { "type": "REFUND", "amount": 20 } }`, or a context path string holding such a list), and `textEvent` for free text. A bare `"textEvent": "REJECT"` puts the text in a `text` field; `{ "type": "REJECT", "field": "reason" }` names the field. `meta` is not template-evaluated.
 
 Reference config — decision, text request, idle human step, one final state:
 
 ```json
 {
   "id": "support-ticket",
+  "description": "Resolve each support ticket in one session. Success: escalations only for tickets that need a human; every reply approved before sending; at most 2 model calls per ticket.",
   "schemas": {
     "input": {
       "type": "object",
@@ -73,7 +104,7 @@ Reference config — decision, text request, idle human step, one final state:
     "draftReply": {
       "model": "openai/gpt-5.4-mini",
       "system": "Draft a short, courteous support reply.",
-      "prompt": "{{ context.ticket }}",
+      "prompt": "{{ input.ticket }}",
       "input": {
         "type": "object",
         "properties": { "ticket": { "type": "string" } },
@@ -132,36 +163,27 @@ Reference config — decision, text request, idle human step, one final state:
 }
 ```
 
-## 3. Validate with `validateAgentConfig`
-
-`validateAgentConfig` from `@statelyai/agent/validate` checks a candidate against the shipped workflow schema with Ajv 2020 built in. It needs `ajv` installed (an optional peer of the package). Run it before anything else touches the config.
-
-```ts
-import { validateAgentConfig } from "@statelyai/agent/validate";
-import type { AgentWorkflowConfig } from "@statelyai/agent";
-
-function validateGeneratedConfig(candidate: unknown): AgentWorkflowConfig {
-  const result = validateAgentConfig(candidate);
-  if (result.valid) return candidate as AgentWorkflowConfig;
-  throw new Error(
-    result.errors.map((error) => `${error.path || "(root)"} ${error.message}`).join("\n"),
-  );
-}
-```
+## 3. Verify with `verifyAgentConfig`
 
-Each error carries a JSON Pointer `path`, so a repair prompt can name the exact bad field.
+`verifyAgentConfig(config, options)` from `@statelyai/agent/validate` runs every pre-run gate and stops at the first failure. It never throws on a bad config; each gate is a record `{ name, ok, detail? }`. It needs `ajv` installed (an optional peer of the package).
 
-## 4. Lower with `fromConfig`
+| Gate         | Checks with                            | Fails on                                                                                                                                                   |
+| ------------ | -------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `validate`   | `validateAgentConfig` (shipped schema) | malformed config shape                                                                                                                                     |
+| `vocabulary` | `allow` lists                          | an invoke src, named guard, named action, `agent.decide` event, or model ref not listed                                                                    |
+| `lower`      | `setupAgent.fromConfig`                | unresolved named guard/action, `onDone` on an `agent.decide` invoke                                                                                        |
+| `lint`       | `lintAgentMachine`                     | error findings (warnings too with `lint: { warnings: true }`)                                                                                              |
+| `explore`    | `explorePaths`, once per input         | a path ending outside `explore.terminals` (default `['done']`), a loop past `maxDepth`, a `maxPaths`/`maxNodes` cap, untried payloads (`unexpandedEvents`) |
 
-`fromConfig` needs a `compileSchema` that turns the JSON Schemas _inside_ the config into Standard Schema validators. Two engines, two jobs: the 2020 build above checks the config document; this one compiles the per-field schemas. The library bundles no JSON Schema engine, by design.
+`compileSchema` turns the JSON Schemas _inside_ the config into Standard Schema validators for `fromConfig`. The library bundles no JSON Schema engine:
 
 ```ts
 import Ajv from "ajv";
-import { setupAgent, type SchemaCompiler, type StandardSchemaV1 } from "@statelyai/agent";
+import type { SchemaCompiler, StandardSchemaV1 } from "@statelyai/agent";
 
 const ajv = new Ajv({ strict: false });
 
-export const ajvCompileSchema: SchemaCompiler = (jsonSchema, name): StandardSchemaV1 => {
+export const compileSchema: SchemaCompiler = (jsonSchema, name): StandardSchemaV1 => {
   const validate = ajv.compile(jsonSchema);
   return {
     "~standard": {
@@ -175,87 +197,139 @@ export const ajvCompileSchema: SchemaCompiler = (jsonSchema, name): StandardSche
                 message: `${name}${e.instancePath} ${e.message}`,
               })),
             },
-      // Expose the source JSON Schema so lint's serializability checks can read the shape.
+      // Expose the source JSON Schema so lint and describeMachine can read the shape.
       jsonSchema: { input: () => jsonSchema },
     },
   };
 };
+```
 
-const { machine, schemas } = setupAgent.fromConfig(config, { compileSchema: ajvCompileSchema });
+Run the gates. List the host's real names in `allow`; an omitted key is unrestricted:
+
+```ts
+import { verifyAgentConfig } from "@statelyai/agent/validate";
+
+const result = await verifyAgentConfig(candidate, {
+  compileSchema,
+  allow: {
+    actors: [],
+    requests: ["draftReply"],
+    guards: [],
+    actions: [],
+    events: ["ESCALATE", "REPLY", "APPROVE", "REJECT"],
+    models: ["openai/gpt-5.4-mini"],
+  },
+  explore: {
+    inputs: [{ ticket: "Export downloads a 0-byte CSV on Safari." }],
+    outputs: { draftReply: [{ reply: "" }] },
+    // ESCALATE's `reason` is a free-form string: give exploration one to try.
+    payloads: () => ({ ESCALATE: [{ reason: "needs a human" }] }),
+  },
+});
+// result.ok, result.gates, result.config (after validate), result.machine (after lower)
 ```
 
-Lowering is itself a gate: it throws on an unresolved named guard/action and on an `onDone` attached to an `agent.decide` invoke.
+- Exploration forks past idle waits on every event the waiting state accepts, so a human step still ends `done`. Add `terminals: ["done", "idle"]` only when a path may legitimately stop at a wait with nothing to accept.
+- `guards` / `actions`: named implementations, passed to `fromConfig`. `actors`: bound with `machine.provide({ actors })` after lowering.
+- `explore` keys other than `inputs` and `terminals` pass through to `explorePaths`: `outputs` (output lists per request or actor src: one element is a canned output, two or more fork one branch each), `errors` (fork a failing branch per src), `payloads` (fork one branch per payload at decisions and idle waits), `dedupeBy` (explore each snapshot key once; the key must cover every field a guard reads), `maxDepth` (default 8), `maxPaths` (default 200), `maxNodes` (default 100,000).
+- Stub each request's output from its declared `output` schema (`""` for string, `0` for number, `[]` for array, recurse on `properties`). Add an `errors` entry for every src whose `onError` path matters.
+- Pick `explore.inputs` that reach every branch the goal cares about. Exploration covers only the inputs you give it.
+- A passing `lint` gate carries its warnings in `detail`. Report them.
 
-## 5. Lint
+## 4. Repair loop
+
+A failed gate's `detail` names the offending path, state, or name, one problem per line. Feed it back verbatim with the rejected config, and run the gates again:
 
 ```ts
-import { lintAgentMachine } from "@statelyai/agent/testing";
-const diagnostics = lintAgentMachine(machine);
-lintAgentMachine(machine, { throw: true }); // throws AgentLintError on error-severity findings
+let prompt = originalTask;
+for (let attempt = 0; attempt < 3; attempt++) {
+  const candidate = await author({ system, prompt });
+  const result = await verifyAgentConfig(candidate, verifyOptions);
+  if (result.ok) return result;
+  const failed = result.gates.find((gate) => !gate.ok)!;
+  prompt = `${originalTask}\n\nYour previous config:\n${JSON.stringify(candidate)}\n\nGate '${failed.name}' failed:\n${failed.detail}\nReturn a corrected config.`;
+}
 ```
 
-Every check applies to config-built machines, reachability included — the lowering keeps the config's transition targets, so `unreachable-state` and `missing-final` read the real graph. Do not disable checks.
+Keep the cap at 3. A config that fails three repairs is usually asking for something the data form cannot express: author it in TypeScript with `setupAgent` instead, or report that the vocabulary is missing something.
 
-The one that bites most often is `decide-without-events`: an `allowedEvents` list with no matching `on` transitions produces a decision the machine can never deliver.
+## 5. Manual fallback
 
-## 6. Simulate a dry run
-
-Lint is structural. A dry run proves a path actually settles. No API key needed.
+Without `verifyAgentConfig` (older package, or a gate you need to run alone), run the same gates by hand:
 
 ```ts
-import { simulateAgent } from "@statelyai/agent/testing";
+import { setupAgent, type AgentWorkflowConfig } from "@statelyai/agent";
+import { validateAgentConfig } from "@statelyai/agent/validate";
+import { explorePaths, lintAgentMachine, simulateAgent } from "@statelyai/agent/testing";
+
+const validation = validateAgentConfig(candidate); // { valid, errors: [{ path, message }] }, path is a JSON Pointer
+if (!validation.valid) {
+  // Stop here: feed the errors back as the repair prompt. Never lower an invalid config.
+  throw new Error(validation.errors.map((e) => `${e.path} ${e.message}`).join("\n"));
+}
+const { machine } = setupAgent.fromConfig(candidate as AgentWorkflowConfig, { compileSchema }); // throws on unresolved names
+lintAgentMachine(machine, { throw: true }); // throws AgentLintError on error findings
 const dryRun = await simulateAgent(machine, {
   input: { ticket: "" },
   script: {
     decisions: { "agent.decide": [{ type: "REPLY" }] },
     text: { draftReply: [{ reply: "" }] },
   },
-});
-// dryRun.status: 'done' | 'idle' | 'exhausted'
+}); // dryRun.status: 'done' | 'idle' | 'exhausted' ('exhausted' = loops, reject)
+const report = await explorePaths(machine, {
+  input: { ticket: "" },
+  outputs: { draftReply: [{ reply: "" }] },
+  payloads: () => ({ ESCALATE: [{ reason: "needs a human" }] }),
+}); // report.unexpandedEvents lists types that still need a payloads entry
 ```
 
-Derive the script from the config rather than guessing: take the first entry of each decision's `allowedEvents`, and stub each request's output from its declared `output` schema (`""` for string, `0` for number, `[]` for array, recurse on `properties`).
-
-- `'exhausted'` → the machine loops. Reject it.
-- `'idle'` → it stopped at a human step. Expected when the config has one.
-- To cover every branch instead of one path, use `explorePaths` / `canReach` (its result is `{ reachable, witness }`).
+The lint check that bites most often is `decide-without-events`: an `allowedEvents` list with no matching `on` transitions. Do not disable checks.
 
-## 7. Repair loop
+## 6. Persist traces from day one
 
-Every gate throws with a message naming the offending field, state, or path. Feed that message back verbatim and regenerate:
+An improver needs data. Wire an event log store and a thread id per run before the first real run, so every run is a trace:
 
 ```ts
-for (let attempt = 0; attempt < 3; attempt++) {
-  try {
-    return buildAndVet(await author({ system, prompt }));
-  } catch (error) {
-    prompt = `${originalTask}\n\nYour previous config was rejected:\n${(error as Error).message}\nReturn a corrected config.`;
-  }
-}
+import { runAgent } from "@statelyai/agent";
+import { createInMemoryEventLogStore } from "@statelyai/agent/log";
+
+const store = createInMemoryEventLogStore(); // tests only; implement AgentEventLogStore on your database
+const result = await runAgent(machine, {
+  input,
+  executors,
+  store,
+  threadId: `tickets:${ticketId}`,
+});
 ```
 
-Keep the cap at ~3. A config that fails three schema-shaped repairs is usually asking for something the data form cannot express — author it in TypeScript with `setupAgent` instead.
+- `threadId` is required with `store`. One thread per run.
+- Give thread ids a prefix an improver can list by, such as `tickets:<ticketId>`. `store.listThreads({ prefix, limit, cursor })` pages through them. It is optional on a custom `AgentEventLogStore`, so with a store that lacks it, record each thread id with its scenario or ticket.
+- Failed decision attempts are journaled on the entry they explain; `getDecisionAttempts(entries)` from `@statelyai/agent/log` reads them back.
+- `replay(machine, await store.read(threadId))` from `@statelyai/agent/log` rebuilds the final state with no model calls. Pass the machine that recorded the thread: with an environment and a policy, the environment; `getUsageFromEvents(entries)` sums token usage.
 
-## 8. Hand back
+## 7. Hand back
 
 Deliver the config JSON plus how to run it:
 
 ```ts
-import { runToQuiescence } from "@statelyai/agent";
+import { runAgent } from "@statelyai/agent";
 import { createAiSdkExecutors } from "@statelyai/agent/ai-sdk";
 
-const result = await runToQuiescence(
-  createAgentRuntime(machine, { executors: createAiSdkExecutors({ resolveModel }) }),
-  { input: { ticket: "Export downloads a 0-byte CSV on Safari." } },
-);
+const run = await runAgent(result.machine!, {
+  input: { ticket: "Export downloads a 0-byte CSV on Safari." },
+  executors: createAiSdkExecutors({ resolveModel }),
+  store,
+  threadId,
+});
 ```
 
-Say which gates passed, list any warning-severity lint diagnostics, and report the dry-run status. Model refs are strings, so the host supplies `resolveModel`.
+Say which gates passed (list `result.gates`), list any lint warnings, state the goal and its success criteria, and say where traces are written. Model refs are strings, so the host supplies `resolveModel`.
 
 ## Known limits
 
 - **Named guards and actions are host-resolved.** A config carries no functions. `guard: "isReady"` only works if the host passes an `isReady` implementation to `fromConfig`; an unresolved name is a build-time throw. Either list the host's names in the prompt or forbid named guards/actions entirely.
 - **Expressions are dot-path templates only.** `"{{ context.a.b }}"` — no comparisons, no arithmetic, no method calls, no partial interpolation inside a larger string.
-- **Ajv validity is not semantic validity.** A config can validate and still invoke a request that does not exist or route to a state that solves nothing. That is what lint and simulate are for.
-- **A dry run covers one path.** Use `explorePaths` when the branch structure matters.
-- **Model refs and tool names are unchecked strings.** They resolve at run time, in the host.
+- **Ajv validity is not semantic validity.** A config can validate and still invoke a request that does not exist or route to a state that solves nothing. That is what the `vocabulary`, `lint`, and `explore` gates are for.
+- **Exploration covers the inputs you give it.** A branch no `explore.inputs` entry reaches is unverified.
+- **Model refs are checked only with `allow.models`.** Tool names stay unchecked strings that resolve at run time, in the host.
+- **The gates check structure, not quality.** Whether a config is better than another is measured by replaying scenarios against an objective. See the `improve-machine` skill.

package/skills/improve-machine/SKILL.md

@@ -0,0 +1,190 @@
+---
+name: improve-machine
+description: Improve a @statelyai/agent policy machine (a JSON AgentWorkflowConfig) from its run traces toward its goal, inside a fixed environment machine, with gates and a held-out promotion rule. Use when asked to "improve this agent", "self-improve", "optimize the playbook from traces", "tune the policy", "make the agent cheaper or more accurate", or "why is my agent failing" for a machine whose runs are persisted to an event log store.
+---
+
+# Improve an agent machine
+
+You improve a **policy**: the JSON `AgentWorkflowConfig` that decides how the agent works toward its goal. It runs inside an **environment**: a TypeScript machine whose states and guards encode the rules of the world and the business invariants. One round changes the policy once and measures it:
+
+```
+locate inputs → read traces → diagnose → propose one diff → verifyAgentConfig (≤ 3 repairs) → replay train → replay held-out → promote or discard → record
+```
+
+To author a policy from scratch, use the `generate-machine` skill. Concept page: `docs/environment-and-policy.md`.
+
+## Hard rules
+
+- **Never edit the environment.** Not its states, guards, invariants, event schemas, or the vocabulary it exposes. When the goal is unreachable without an environment change, write a change request for a person (template in [reference.md](reference.md)) and stop.
+- **Never edit a machine while it runs.** A promoted policy applies to new runs only.
+- **The objective comes from ground truth.** Score outcomes from the traces (terminal states, outputs) or from effects recorded in the world. Never ask a model to grade its own runs.
+- **One cause per round, smallest diff.** A diff that fixes two things cannot tell you which one helped.
+- **Diagnose from training scenarios only.** Held-out scenarios are for the promotion decision. Never read their failures to shape a change, and never move a scenario between splits.
+- **Stop after 3 rounds without a promotion** (or the limit the user gives).
+
+## 1. Locate the inputs
+
+Find all six before changing anything. If one is missing, stop and ask.
+
+| Input          | Where to look                                                                                    | If missing                                                                  |
+| -------------- | ------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------- |
+| Goal           | Root `description` of the policy config or the environment machine                               | Draft one with measurable success criteria; ask the user to confirm it      |
+| Objective      | A scorer over outcomes (an eval file, a `score*` function, a benchmark)                          | Write one from the goal's success criteria; **ask the user to confirm it**  |
+| Traces         | The `store` passed to `createAgentRuntime` / `runAgent`; its `listThreads`, or recorded ids      | Wire a store first (see `generate-machine` step 6) and collect runs         |
+| Scenarios      | A scenario list with expected outcomes, split into train and held-out                            | Propose a fixed split (about 70/30, every category in both); ask to confirm |
+| Vocabulary     | The actors, requests, named guards, actions, decision events, and model refs the host implements | Read them off the host code; never invent names                             |
+| Policy history | The policy files and a `CHANGELOG.md` next to them                                               | Create the changelog with the current policy as `v0`                        |
+
+The objective defines "better". Do not start a round until the user has confirmed it. A good objective has:
+
+- a primary metric from the goal (accuracy against expected outcomes, resolution rate),
+- a hard constraint: invariant violations, which must be 0,
+- a cost term (model calls, tokens, guard rejections) used as a tie-breaker or a stated budget.
+
+## 2. Read
+
+Read both machines as text. `describeMachine` puts the goal first:
+
+```ts
+import { describeMachine, setupAgent } from "@statelyai/agent";
+
+const { machine: policy } = setupAgent.fromConfig(policyConfig, { compileSchema });
+const environmentText = describeMachine(environment);
+const policyText = describeMachine(policy);
+```
+
+Then find the training threads and compute one outcome row per episode:
+
+```ts
+import { getDecisionAttempts, getUsageFromEvents, replay } from "@statelyai/agent/log";
+
+// Thread ids name the split and policy version: "train:v<n>:<scenario>:r<k>".
+const prefix = `train:v${currentVersion}:`;
+const threadIds: string[] = [];
+let cursor: string | undefined;
+do {
+  const page = await store.listThreads!({ prefix, limit: 100, cursor });
+  threadIds.push(...page.threadIds);
+  cursor = page.cursor;
+} while (cursor !== undefined);
+
+const rows = [];
+for (const threadId of threadIds) {
+  const scenario = scenarioOf(threadId);
+  const entries = await store.read(threadId);
+  const { snapshot } = replay(environment, entries); // the recording machine; pure fold, no model calls
+  rows.push({
+    threadId,
+    expected: scenario.expected,
+    state: snapshot.value,
+    status: snapshot.status,
+    output: snapshot.output,
+    steps: entries.length,
+    tokens: getUsageFromEvents(entries).totalTokens ?? 0,
+    failedAttempts: getDecisionAttempts(entries).flatMap(({ attempts }) => attempts),
+  });
+}
+```
+
+- `store.listThreads({ prefix, limit, cursor })` pages through the threads that hold at least one entry. `cursor` is present only while more ids remain. It is optional on a custom `AgentEventLogStore`; `createInMemoryEventLogStore` implements it. Without it, use the thread ids recorded with each scenario.
+- Prefix thread ids with the split and version so a `listThreads` prefix returns one version's training runs and never a held-out thread. `scenarioOf` is your mapping from a thread id back to its scenario.
+- `store.read(threadId)` returns a thread's entries in log order.
+- `replay(machine, entries)` needs the machine, at the version, that recorded the thread. Here that is the environment: its runtime records every thread, and the policy runs only as a decider at its boundary, so never replay with the policy machine. A thread recorded by a different environment version throws `AgentMachineVersionMismatchError`. Score each thread with the version that recorded it.
+- `getDecisionAttempts(entries)` returns `{ index, eventType, attempts }` for each entry whose decision needed retries: the failed tries before the event landed, or every try of an exhausted decision (`eventType` is then `xstate.error.actor.*`). Each attempt has `failure` (`unknown-event`, `invalid-payload`, or `rejected-by-guard`), `reason`, and the attempted `event` when there was one. A first-try decision records nothing.
+- `agent.decide` invokes journal their attempts on their own. With `decideNext` at the boundary, the host must pass them with the event, `runToQuiescence(runtime, { event, attempts })` or `runtime.transition(state, event, { attempts })`, or they never reach the log.
+- Compute the objective over the rows. This is the baseline for the round.
+
+## 3. Diagnose
+
+Group the failing training episodes by cause. Check these clusters in order:
+
+1. **Invariant violations.** Any violation is a stop: the environment is wrong, not the policy. Write a change request.
+2. **Wrong terminal state.** The episode ended in `escalated` when the label says `issued`, or the reverse. Look at which decision led there.
+3. **Guard rejections.** The policy keeps proposing events the environment refuses: `rejected-by-guard` entries in the rows' `failedAttempts`. The policy lacks a check the environment already makes.
+4. **Loops and budget exhaustion.** Repeated state values, or a terminal reached because a turn or retry budget ran out.
+5. **Cost hotspots.** Correct episodes that spend model calls where a `choice` over known facts, or a cheap judgment, would decide.
+
+Pick the cluster with the largest effect on the objective. Write the hypothesis down before you edit anything (template in [reference.md](reference.md)): cause, evidence (thread ids), the change, and which metric should move.
+
+## 4. Propose
+
+Make the smallest policy diff that addresses the one cause.
+
+- Typical diffs: add a `choice` branch over facts already in context; replace a model call with a judgment actor from the vocabulary; narrow an `agent.decide` `allowedEvents` list; turn an open decision into a closed choice with `payloads`; tighten a prompt; add an `onError` path; lower a model ref to a cheaper one in the vocabulary.
+- Use only names in the vocabulary. A named guard the host does not implement fails the `lower` gate.
+- Keep the config `id`. Bump `version`.
+- Keep the root `description` (the goal) unchanged. Changing the goal is the user's decision.
+
+## 5. Gate
+
+Run `verifyAgentConfig` from `@statelyai/agent/validate` with the vocabulary and probe inputs. Probe inputs are one input per training scenario category, chosen to reach every branch the diff touches.
+
+```ts
+import { verifyAgentConfig } from "@statelyai/agent/validate";
+
+const verified = await verifyAgentConfig(candidate, {
+  compileSchema,
+  guards: hostGuards,
+  actors: hostActors,
+  allow: { actors, requests, guards, actions, events, models },
+  explore: {
+    inputs: probeInputs,
+    terminals: ["done"],
+    // Both outcomes, so the branches on either side of the classifier are probed.
+    outputs: {
+      classifyTicket: [
+        { isRefundRequest: true, confidence: 0.9 },
+        { isRefundRequest: false, confidence: 0.9 },
+      ],
+    },
+    errors: { classifyTicket: new Error("down") },
+  },
+});
+const failed = verified.gates.find((gate) => !gate.ok);
+```
+
+- Gates run in order and stop at the first failure: `validate`, `vocabulary`, `lower`, `lint`, `explore`.
+- A failed gate's `detail` names the offending path, state, or name. Send it back verbatim with the rejected config and regenerate. Cap repairs at 3. After 3, discard the candidate, record it, and move to the next cause.
+- `explore` keys besides `inputs` and `terminals` pass through to `explorePaths`: `outputs` (output lists per src: one element is a canned output, two or more fork one branch each), `errors`, `payloads` (one branch per payload), `dedupeBy` (explore each snapshot key once; the key must cover every field a guard reads), `maxDepth`, `maxPaths`.
+- The environment's own `canReach` proofs cover every policy, because the environment's guards judge every proposed event. You did not change the environment, so you do not re-prove it.
+
+## 6. Replay
+
+Run the candidate on the training scenarios, then on the held-out scenarios. Write every replay to the store under new thread ids that name the split and version (`train:v3:<scenario>:r1`, `heldout:v3:<scenario>:r1`), so the next round can list them by prefix.
+
+- Run the current policy and the candidate on the same scenarios, the same world state, and the same model refs.
+- Model calls are nondeterministic. Run each scenario k times (k ≥ 3 when calls are not cached) and compare means.
+- Compute the objective, invariant violations, and cost per split, for both versions.
+
+## 7. Promote or discard
+
+State the tolerance before you replay. Promote only when every condition holds:
+
+- invariant violations are 0 on both splits,
+- the objective on the held-out split is better than the current policy's,
+- no single scenario regresses beyond the tolerance (for example, a scenario that passed in all k runs now fails in any),
+- cost stays within the stated budget, if there is one.
+
+Otherwise discard the candidate and keep the current policy. Either way, append an entry to the `CHANGELOG.md` next to the policies: version, hypothesis, diff summary, gate results, before and after per split, and the decision. Record discarded candidates too; they stop the next round from retrying the same idea.
+
+## 8. Stop and report
+
+Stop when any of these holds:
+
+- 3 rounds in a row without a promotion,
+- the goal's success criteria are met on the held-out split,
+- the remaining failures need an environment change (write a change request),
+- the vocabulary lacks a capability the fix needs (ask for it; name the actor, request, or guard and its input and output).
+
+Report to the user: rounds run, each candidate with its hypothesis and decision, the objective before and after on both splits, violations, cost, and any change requests.
+
+## Worked example
+
+`examples/self-improving-refunds` in the `statelyai/agent` repo runs this loop end to end:
+
+- the refund desk is the environment: business rules as guards, and an `illegallyIssued` sentinel proven unreachable,
+- `playbooks/*.json` are policy versions, with the vocabulary listed in `playbooks/README.md`,
+- `improve.ts` is the loop: propose, gate, replay, promote,
+- `benchmark.ts` compares a raw tool loop, the desk with a raw model deciding, and the desk with a playbook deciding.
+
+Read the directory before you start. File contents change between releases.

package/skills/improve-machine/reference.md

@@ -0,0 +1,96 @@
+# Improve-machine templates
+
+Templates for the `improve-machine` skill: a round checklist, the hypothesis, the proposer prompt, the changelog entry, and the environment change request.
+
+## Round checklist
+
+```
+[ ] Goal confirmed (root description, measurable success criteria)
+[ ] Objective confirmed by the user (primary metric, violations = 0, cost term)
+[ ] Train / held-out split fixed; tolerance stated
+[ ] Vocabulary listed (actors, requests, guards, actions, events, models)
+[ ] Baseline: objective, violations, cost per split for the current version
+[ ] Failures clustered (train only); one cause picked
+[ ] Hypothesis written
+[ ] Candidate: smallest diff, id kept, version bumped, description unchanged
+[ ] verifyAgentConfig passed (repairs used: _/3)
+[ ] Replayed train, then held-out, k runs each, new thread ids
+[ ] Promotion rule applied; CHANGELOG entry written
+[ ] Stop condition checked
+```
+
+## Hypothesis
+
+```
+Cause: <one sentence, the mechanism, not the symptom>
+Evidence: <thread ids and what each shows>
+Change: <the one diff, in words>
+Expected effect: <metric> moves from <x> to about <y>; nothing else changes
+Risk: <scenarios that could regress, and why>
+```
+
+## Proposer prompt
+
+Use this when a model proposes the candidate. Fill every slot from real data.
+
+```
+You improve a policy machine. You may change only the policy config below.
+
+{describeMachine(environment)}
+
+Current policy (version {n}):
+{JSON.stringify(policyConfig, null, 2)}
+
+Vocabulary (use no other names):
+- actors: {actors}
+- requests: {requests}
+- named guards: {guards}
+- named actions: {actions}
+- decision events: {events}
+- model refs: {models}
+
+Objective on the training scenarios: {objective summary}
+Diagnosed cause: {hypothesis.cause}
+Evidence: {failing rows: scenario, expected, reached state, rejections, tokens}
+
+Return one complete AgentWorkflowConfig as JSON. Make the smallest change that
+addresses the diagnosed cause. Keep "id" and "description". Set "version" to {n + 1}.
+```
+
+On a failed gate, append:
+
+```
+Your previous config was rejected.
+Gate '{gate.name}' failed:
+{gate.detail}
+Return a corrected config.
+```
+
+## Changelog entry
+
+```md
+## v{n} — {promoted | discarded}
+
+- Hypothesis: {cause} → {change}
+- Diff: {states or transitions touched}
+- Gates: validate ✓ vocabulary ✓ lower ✓ lint ✓ explore ✓ (repairs: {r})
+- Train: objective {a} → {b}, violations {v}, cost {c0} → {c1}
+- Held-out: objective {a} → {b}, violations {v}, cost {c0} → {c1}
+- Regressions: {none | scenario ids}
+- Decision: {why}
+```
+
+## Environment change request
+
+Write this instead of editing the environment.
+
+```md
+# Environment change request: {short title}
+
+- Goal criterion blocked: {the success criterion no policy can meet}
+- Evidence: {thread ids; the state and guard that block it}
+- Policies tried: {versions and why each could not get there}
+- Proposed change: {the state, guard, or event to add or change, in words}
+- Invariants affected: {which, and the canReach check to rerun}
+- Owner: a person reviews and merges this; the improver does not.
+```