@statelyai/agent 2.0.0-alpha.28 → 2.0.0-alpha.30

package/skills/add-evals/SKILL.md

@@ -0,0 +1,129 @@
+---
+name: add-evals
+description: Write evals for a @statelyai/agent or XState agent machine at the right level — rule checks with no model, decision evals at one decision state, outcome evals over whole runs, and per-state stats from traces — each tied to the states it covers. Use when asked to "write evals", "add evals for the approval logic", "test the agent", "measure the agent", "build an eval suite", or "what should I eval" for an agent built as a state machine.
+---
+
+# Add evals to an agent machine
+
+The machine tells you what to eval and at which level. Every eval names the states it covers, so a failing eval points at a part of the machine.
+
+```
+read the machine → map the request to states → pick the level per state → write the evals → baseline → hand back
+```
+
+This is step 3 of 4. Before: `loop-to-machine`, `add-traces`. Next: `improve-machine`.
+
+## Hard rules
+
+- **The cheapest level that can catch the failure.** A rule is checked without a model. Only judgment needs a model in the eval.
+- **Ground truth, never a model's opinion of itself.** Outcome evals read terminal states and effects in the world. A model grades text only where no ground truth exists, and that eval says so in its name.
+- **Labels belong to people.** Draft expected outcomes and expected events; ask the user to confirm them before they gate anything. Keep drafts in their own list, marked unconfirmed, and out of every gate.
+- **Never edit the machine to make an eval pass.** A failing eval is a finding. Report it.
+- **Held-out scenarios stay held out.** Evals may run on them; nobody tunes against them.
+
+## 1. Read the machine
+
+```ts
+import { describeMachine } from "@statelyai/agent";
+console.log(describeMachine(machine)); // goal, states, transitions, events
+```
+
+List:
+
+- the decision states: an `agent.decide` invoke, a `decision` tag, or a state a host answers with `decideNext`;
+- the person's steps: idle states a person answers (`meta.interaction`, the runner's `onIdle`). What the person decides is ground truth, not something to eval; the routing after their answer is a rule;
+- the guards. `describeMachine` shows where a transition function can go and marks it `(conditional)`, but not the guard's name or logic. Read the source for those;
+- the sentinel states and the final states.
+
+## 2. Map the request to states
+
+Turn the user's words into states. "Evals for the approval logic" means every state and guard that touches approval: the guard that routes a refund over the threshold, the approval step and each answer to it, the sentinel check before the payment, and the final states they lead to. Write the mapping down:
+
+| Asked about | States and guards                                                                                  |
+| ----------- | -------------------------------------------------------------------------------------------------- |
+| approval    | `deciding` (REFUND: over threshold → `awaitingApproval`), `awaitingApproval` (APPROVE, REJECT), `paying` (sentinel check), `issued`, `closed` |
+
+If the request names nothing ("write evals"), cover every decision state, every sentinel, and the run as a whole.
+
+## 3. Pick the level
+
+| Level        | Catches                                              | Runs a model          | Tools                                                              |
+| ------------ | ---------------------------------------------------- | --------------------- | ------------------------------------------------------------------ |
+| **Rule**     | A guard, an ordering, an invariant that must hold    | No                    | Exported guard functions, `canReach`, `explorePaths`, `simulateAgent` |
+| **Decision** | A wrong choice at one decision state                 | The decider under test | The decider called on recorded or written decision requests        |
+| **Request**  | A bad text output from one request                   | That request          | `runSeam`, scored on `seamOutput` and the path after it. Not applicable when the machine has no text requests; say so |
+| **Outcome**  | A wrong end result for a whole scenario              | The whole agent       | Scenarios, the runner, an objective over terminal states and effects |
+| **Stats**    | Which state the failures cluster in                  | No (reads traces)     | `foldTraces`                                                       |
+
+A guard or ordering is always a rule eval. Use a decision eval when the guards leave more than one legal option. Use an outcome eval when a person can label the right end result.
+
+## 4. Write them
+
+Name every eval `<state>: <what it checks>`, so the report reads as a map of the machine. In `test.each` titles, `$name` is a placeholder: write amounts as `100 dollars`, not `$100`.
+
+**Rule evals**, from `@statelyai/agent/testing`:
+
+```ts
+test("paying: no refund over 100 dollars issues without approval", async () => {
+  const result = await canReach(machine, "ruleBroken", { input, payloads, outputs, errors });
+  expect(result.complete).toBe(true); // reachable: false is a proof only when the search finished
+  expect(result.reachable, JSON.stringify(result.witness)).toBe(false);
+});
+
+test("deciding: REFUND over the order amount is refused", () => {
+  expect(refundIsLegal({ ...context, order: { ...order, amount: 40 } }, 41)).toBe(false);
+});
+```
+
+Use `explorePaths` to assert the set of reachable final states, and `await simulateAgent(...)` for one scripted path. Cover both sides of every threshold. The two treat a closed choice differently:
+
+- `explorePaths` and `canReach` use `options.payloads` **instead of** the decision's own offered options, so they explore values the machine would never offer. That is what a boundary proof needs.
+- `simulateAgent` keeps the offered options: a scripted event the decision did not offer is refused, and the run ends in the decision's `onError`. Script only offered events, or pick an input that offers the value.
+
+**Decision evals.** A row is a decision request and the event a person says is right. Write rows by hand, or take them from recorded traces: the first attempt of each decision is a `request.start` event with `request.kind === "decision"` and `request.attempts.length === 0`, and its `request` carries `name`, `options`, and `metadata`. Cast the serialized `request` to `AgentExecutorDecisionRequest`, and compare events without caring about key order. If the decider reads anything outside the request (the ticket, the tool history), store that in the row too.
+
+```ts
+// Your helper: rows for one decision state, from traces or a fixture file.
+const rows: Array<{ id: string; request: AgentExecutorDecisionRequest; expected: ChosenEvent }> = loadDecisionRows("deciding");
+
+test.each(rows)("deciding: $id", async ({ request, expected }) => {
+  const { event } = await decide(request); // the policy decider, or the model executor
+  expect(event).toEqual(expected);
+});
+```
+
+Which deciders gate CI: a deterministic reference decider (a scripted or rules-based one) gates. A live model is a measurement, not a unit test: run it outside CI, several times, and report the rate. A seeded random decider is measured over several seeds, never gated.
+
+**Request evals.** For a text request (a drafted question, a summary), `runSeam(machine, { input, scripts, seam: { request: "draftQuestion" }, candidate })` runs the whole machine with only that request live. Score `seamOutput`, and the states the run reached after it.
+
+**Outcome evals.** One function over a batch of scenario runs, reading ground truth. Score the effects as well as the outcome: a run that ends `issued` with the wrong amount is wrong.
+
+```ts
+export function objective(runs: Array<{ scenario: Scenario; outcome: string; refund?: Refund; violations: number; modelCalls: number }>) {
+  return {
+    accuracy: runs.filter((r) => r.outcome === r.scenario.expected && sameRefund(r.refund, r.scenario.expectedRefund)).length / runs.length,
+    violations: runs.reduce((n, r) => n + r.violations, 0), // must be 0
+    modelCalls: runs.reduce((n, r) => n + r.modelCalls, 0), // cost
+  };
+}
+```
+
+Read the terminal state from the event logs `add-traces` recorded (`replay(machine, entries).snapshot`, `replay` from `@statelyai/agent/log`). Effects and violations live in the world, not the log: save them next to the thread id when the run ends, so the objective can be recomputed without running anything. Use `matchesTrajectory` only for an ordering a person requires (approval before payment), never to pin the whole path; build its input from the `machine.transition` trace events' `snapshot.value`.
+
+**Stats.** `foldTraces(traces, { failed })` (from `@statelyai/agent/testing`) returns `{ state, visits, failures, withheld }` per state. A failed run counts against **every** state it visited, so states after a bad decision rank as high as the decision itself. Use the table to see which states failing runs pass through, then look at the decision states among them, their `withheld` counts, and their decision attempts. `failed` sees only the trace: to judge a run against its label, keep a map from trace to scenario.
+
+## 5. Coverage
+
+If every decider scores the same on the confirmed scenarios, the scenarios cannot tell a better decider from a worse one. Draft scenarios on the boundaries the evals found (an amount just under and over a threshold, a partial request), label them as drafts, and ask the user to confirm them and add them to the scenario set. They become part of the train or held-out split only once confirmed.
+
+## 6. Baseline
+
+Run every eval once on the current machine and record the numbers: which pass, the outcome scores per split, the stats table. Failing evals are findings, not bugs in the eval, unless the label was wrong; then ask.
+
+## 7. Hand back
+
+1. The state mapping from step 2.
+2. The evals by level, each named by state.
+3. The baseline numbers, train and held-out separately.
+4. Every failing eval and the state it points at.
+5. Next step: `improve-machine`, with the objective from step 4 as its scorer.

package/skills/add-traces/SKILL.md

@@ -0,0 +1,135 @@
+---
+name: add-traces
+description: Make every run of a @statelyai/agent or XState machine leave a replayable, stamped trace — an event log thread plus trace events — so evals and self-improvement can read what happened and what produced it. Use when asked to "add tracing", "add logging", "persist runs", "stamp traces", "record what the agent did", or "set up observability" for an agent machine, or before writing evals or improving a policy from traces.
+---
+
+# Add traces to an agent machine
+
+Every run should leave two records, both stamped with what produced it:
+
+| Record           | What it is                                                                     | Read by                                              |
+| ---------------- | ------------------------------------------------------------------------------ | ---------------------------------------------------- |
+| Event log thread | The external inputs the machine consumed, in order. The source of truth        | `replay`, `getDecisionAttempts`, `getUsageFromEvents` |
+| Trace events     | Observations: transitions with `from`, decision requests with offered options  | `foldTraces`, OpenTelemetry, debugging               |
+
+```
+version the machines → pick a store → name threads → stamp → collect trace events → prove replay → hand back
+```
+
+This is step 2 of 4. Before: `loop-to-machine`. Next: `add-evals`, then `improve-machine`.
+
+## Hard rules
+
+- **Traces never change behavior.** Recording is an observer. No branch reads a trace.
+- **A trace must name what produced it.** A run with no machine version or policy version cannot be used to compare versions, so it is useless to the improver.
+- **Replay is the test.** A log that does not replay to the run's final state with zero model calls is not a trace yet.
+- **No secrets or personal data in metadata or thread ids.** Use ids, not names or emails. The log itself holds the machine's input and every event verbatim (a ticket's text, a customer's reply), and trace snapshots hold the context: in production, redact or encrypt them per your data rules, and ask the user which applies.
+
+## 1. Version the machines
+
+Declare `version` on every machine whose runs you record: the environment and, if there is one, the policy machine.
+
+```ts
+export const deskMachine = deskSetup.createMachine({ id: "refund-desk", version: "1", ... });
+```
+
+Also name the **decider version**: what answers the decisions. It is the policy machine's `version` when there is one; otherwise a label for the model and prompt that decide (`gpt-6-luna.p1`), or for a stand-in model in tests (`careful`). Runs by different deciders must never share a version label.
+
+- Every log entry carries `machineVersion`. Without a declared `version` it falls back to a structural hash that skips functions, and in XState v6 the guards and targets live in transition functions. Changing a guard would not change the hash.
+- Bump `version` on every behavior change. Adding recording is not one: it only observes. `improve-machine` bumps the policy's on promotion.
+
+## 2. Pick a store
+
+- **Tests and local runs:** `createInMemoryEventLogStore()` from `@statelyai/agent/log`, or write each thread's `result.events` to a JSON file after the run.
+- **Production:** implement `AgentEventLogStore` against your database and check it with `assertEventLogStoreConformance`. Implement the optional `listThreads` so a split can be paged by prefix.
+
+Pass it with a `threadId`. The store is write-ahead: no model call starts until the entries before it are durable.
+
+```ts
+const result = await runAgent(deskMachine, { input, executors, store, threadId, metadata });
+await result.drain(); // late usage entries, before you read the store or exit
+```
+
+**A thread that already has entries is resumed, not overwritten.** Give every new run a new thread id; reuse one only to continue that run.
+
+## 3. Name threads
+
+A thread id is how runs are found later. Use a prefix scheme and keep it in one function:
+
+| Runs                     | Thread id                                    |
+| ------------------------ | -------------------------------------------- |
+| Scenario runs for evals  | `<split>:<deciderVersion>:<scenarioId>:r<k>`, `k` = the repeat index |
+| Production               | `prod:<domain id>` (ticket id, session id)   |
+
+`store.listThreads?.({ prefix: "train:careful:" })` then pages through one split of one decider. `listThreads` is optional on a custom store.
+
+## 4. Stamp each run
+
+`metadata` lands on the log's first (`@agent.init`) entry. Put there everything needed to tell two runs' producers apart:
+
+```ts
+const metadata = {
+  environmentVersion: deskMachine.version ?? "unversioned",
+  deciderVersion, // see step 1
+  models: { decider: "openai/gpt-6-luna" }, // the machine's model refs → what the host resolved them to
+  scenario: scenario.id, // eval runs only
+  split: scenario.split, // eval runs only
+};
+```
+
+Read it back with `entries[0].metadata`. A resumed run keeps the stamp of the run that started the log. `executionId` is reserved. The value type is `JsonValue` from `@statelyai/agent/log`.
+
+## 5. Collect trace events
+
+`onTrace` sees every trace event. Serialize and keep them per thread:
+
+```ts
+import { serializeTraceEvent, type JsonSerializableTraceEvent } from "@statelyai/agent";
+
+const trace: JsonSerializableTraceEvent[] = [];
+const result = await runAgent(deskMachine, { ..., onTrace: (event) => trace.push(serializeTraceEvent(event)) });
+await result.drain();
+
+// For local batches: one file per thread. Encoding keeps colons and path separators out of the name.
+const file = encodeURIComponent(threadId);
+await mkdir("runs", { recursive: true });
+await writeFile(`runs/${file}.events.json`, JSON.stringify(result.events));
+await writeFile(`runs/${file}.trace.json`, JSON.stringify(trace));
+```
+
+Decide with the user whether `runs/` is committed or gitignored.
+
+- In production, send them to OpenTelemetry instead: `createOtelTraceHandler({ tracer })` from `@statelyai/agent/otel` is an `onTrace` handler.
+- **A policy run inside a decider** is a separate run. Pass it the same `onTrace` (and its own `store` thread, `<threadId>:policy:<n>`) if you want per-state stats for the policy too.
+- **A plain XState machine** (built with `createMachine` or `setup`, decided with `decideNext`): drive it with `createAgentRuntime` and `runToQuiescence`, which accept plain machines, and pass each `decideNext` result as `{ event, attempts }` so refused choices reach the log. Push `{ type: "request.start", request: result.request }` onto the trace before sending the event, since a host-driven decision emits no trace itself.
+- **A plain `createActor` run you cannot change:** attach `recordLog(actor)` from `@statelyai/agent/log` before `actor.start()`, and pass `inspect: traceTransitions((event) => trace.push(serializeTraceEvent(event)))` to `createActor`. `recordLog` cannot record decision attempts.
+
+## 6. Prove it
+
+Write these tests, with the stand-in model from `loop-to-machine`:
+
+```ts
+import { getDecisionAttempts, replay } from "@statelyai/agent/log";
+import { foldTraces } from "@statelyai/agent/testing";
+
+test("a recorded run replays to its final state with no model calls", async () => {
+  const result = await runAgent(deskMachine, { input, executors, store, threadId: "test:careful:small-clean:r1", metadata });
+  await result.drain();
+  const entries = await store.read("test:careful:small-clean:r1");
+  const { snapshot } = replay(deskMachine, entries); // a pure fold: nothing runs
+  expect(snapshot.value).toEqual(result.snapshot.value);
+  expect(entries[0]!.metadata).toMatchObject({ environmentVersion: "1" });
+});
+```
+
+- `foldTraces(traces, { failed })` over a batch returns a row for every decision state. `failed` judges one whole trace; start with "the last event is not a `run.end` with status `done`, or the run ended in a failure or sentinel state", and ask the user whether a wrong outcome counts too. `withheld` is above 0 only for a closed choice (`payloads`), where the guards refused an option.
+- `getDecisionAttempts(entries)` lists the refused choices of a run. A stand-in model that picks from the offered options can never propose an illegal one, so write a raw decide executor for this test that proposes a refused event first: `decide: async (request) => ({ event: request.attempts.length ? request.options![0]!.event : { type: "REFUND", amount: 1000 } })`.
+- Running a scenario with and without recording leaves the same world and the same final state.
+
+## 7. Hand back
+
+1. Where runs are stored and how thread ids are built.
+2. The stamp fields.
+3. Where trace events go.
+4. The replay test passing.
+5. Next step: `add-evals`.

package/skills/generate-machine/SKILL.md

@@ -1,16 +1,38 @@
 ---
 name: generate-machine
-description: Author a @statelyai/agent workflow as JSON and check it before it runs. Use when asked to generate or author an agent machine, create a workflow config, write an AgentWorkflowConfig, or produce an LLM-authored state machine for an agent — and when a generated config fails Ajv validation or agent lint and needs repair.
+description: Author a @statelyai/agent workflow as JSON and check it before it runs. Use when asked to generate or author an agent machine, create a workflow config, write an AgentWorkflowConfig, or produce an LLM-authored state machine for an agent — and when a generated config fails verifyAgentConfig, Ajv validation, or agent lint and needs repair.
 ---
 
 # Generate an agent machine
 
-An agent machine is data. You author a JSON `AgentWorkflowConfig`, then run it through gates that all work with no API key: Ajv → `fromConfig` → lint → simulate. Do not hand back a config that has not passed all four.
+An agent machine is data. You author a JSON `AgentWorkflowConfig`, then run it through `verifyAgentConfig`, whose gates all work with no API key: validate → vocabulary → lower → lint → explore. Do not hand back a config that has not passed every gate.
 
 ```
-author → validate (validateAgentConfig) → lower (fromConfig) → lint (lintAgentMachine) → simulate → hand back
+split environment/policy → author (description = goal) → verifyAgentConfig → repair (≤ 3) → persist traces → hand back
 ```
 
+To improve an existing policy from its traces, use the `improve-machine` skill instead.
+
+## 0. Split environment and policy
+
+Decide what is fixed before you write any JSON.
+
+- **Environment** (TypeScript, fixed): the rules of the world and the business invariants, as states and guards. People author it and verify it with `canReach` / `explorePaths` from `@statelyai/agent/testing`. You never generate it, and no generated config may change it.
+- **Policy** (JSON, generated): how the agent works toward the goal: which step next, which tool, when to ask a person, cheap judgment vs model call. This is the `AgentWorkflowConfig` you author.
+- The policy plugs into the environment at its decision points and outputs one event; the environment's guards decide whether to accept it.
+- No split needed for a single judgment inside a fixed workflow, or for a workflow where every step is dictated by rules. Author one machine.
+
+If the request asks you to generate a business rule (an approval threshold, a payout limit), stop: that belongs in the environment, in TypeScript, reviewed by a person. Say so.
+
+Before editing an existing machine, read it as text:
+
+```ts
+import { describeMachine } from "@statelyai/agent";
+const text = describeMachine(machine); // "Machine <id>", then "Goal: …", states, transitions, events
+```
+
+Pass `{ snapshot }` to mark the active states with `*`. Concept page: [Environment and policy](https://github.com/statelyai/agent/blob/main/docs/environment-and-policy.md).
+
 ## 1. Read the schema
 
 The config shape is fixed by a shipped JSON Schema. Read it first; it is the contract.
@@ -20,23 +42,32 @@ The config shape is fixed by a shipped JSON Schema. Read it first; it is the con
 
 ## 2. Author the config
 
+Start with the root `description`: it is the machine's **goal**, the overall objective plus measurable success criteria. `describeMachine` renders it first as `Goal: …`, and an improver later climbs toward it. Write criteria a scorer can compute from outcomes (terminal states, effects in the world, cost), not "be helpful":
+
+```json
+"description": "Resolve each support ticket in one session. Success: escalations only for tickets that need a human; every reply approved before sending; at most 2 model calls per ticket."
+```
+
 Rules the runtime enforces. Each one maps to a build error or a lint diagnostic:
 
 - Every value is a JSON literal or a whole-string `"{{ }}"` expression reading a dot path on `input`, `context`, or `event`. No JavaScript.
 - Guard expressions are truthy-only: no comparisons, operators, or negation.
-- A model call is either a named `requests` entry invoked with `src: "<requestName>"`, or an inline invoke with `src: "agent.decide"` plus an `allowedEvents` list.
+- A model call is either a named `requests` entry invoked with `src: "<requestName>"`, or an inline invoke with `src: "agent.decide"` plus an `allowedEvents` list. A `requests` entry's templates read only its own `input` (`"{{ input.ticket }}"`); the invoke's `input` maps context into it.
 - A state invoking `agent.decide` MUST handle every allowed event in its `on`.
 - An `agent.decide` invoke has no `onDone` (a decision produces no output). Use `onError` for the retries-exhausted path.
 - A request invoke resolves with `{ result, messages }`, so `onDone.assign` reads its result from `"{{ event.output.result.<field> }}"` (`"{{ event.output.result }}"` for a plain-text request). A non-request `actors` invoke reads `"{{ event.output.<field> }}"` directly.
 - Every path must reach a `"type": "final"` state, and each final state needs an `output` when `schemas.output` is declared.
 - Do not invent guard or action names. Only `"{{ }}"` guards, `assign`, and `emit` exist unless the host tells you which named guards/actions it implements.
-- Model refs are strings (`"openai/gpt-5.4-mini"`); the host resolves them.
+- Model refs are strings (`"openai/gpt-6-luna"`); the host resolves them.
+- To make a decision a closed choice, add `payloads` to the `agent.decide` input: one payload list per event type, without `type` (`"payloads": { "REFUND": [{ "amount": 20 }, { "amount": 50 }], "ESCALATE": [] }`). The model then picks one listed event instead of writing a payload. A type with no key is offered as `{ type }`; `[]` withholds it.
+- An idle human step can carry `meta.interaction` for the host to render with `getInteraction`: a `label`, `events` (one button per event type), `choices` (one button per full event, `{ "label": "Refund $20", "event": { "type": "REFUND", "amount": 20 } }`, or a context path string holding such a list), and `textEvent` for free text. A bare `"textEvent": "REJECT"` puts the text in a `text` field; `{ "type": "REJECT", "field": "reason" }` names the field. `meta` is not template-evaluated.
 
 Reference config — decision, text request, idle human step, one final state:
 
 ```json
 {
   "id": "support-ticket",
+  "description": "Resolve each support ticket in one session. Success: escalations only for tickets that need a human; every reply approved before sending; at most 2 model calls per ticket.",
   "schemas": {
     "input": {
       "type": "object",
@@ -71,9 +102,9 @@ Reference config — decision, text request, idle human step, one final state:
   "context": { "ticket": "{{ input.ticket }}" },
   "requests": {
     "draftReply": {
-      "model": "openai/gpt-5.4-mini",
+      "model": "openai/gpt-6-luna",
       "system": "Draft a short, courteous support reply.",
-      "prompt": "{{ context.ticket }}",
+      "prompt": "{{ input.ticket }}",
       "input": {
         "type": "object",
         "properties": { "ticket": { "type": "string" } },
@@ -93,7 +124,7 @@ Reference config — decision, text request, idle human step, one final state:
         "id": "triageDecision",
         "src": "agent.decide",
         "input": {
-          "model": "openai/gpt-5.4-mini",
+          "model": "openai/gpt-6-luna",
           "system": "Decide whether this ticket needs escalation or a drafted reply.",
           "prompt": "{{ context.ticket }}",
           "allowedEvents": ["ESCALATE", "REPLY"],
@@ -132,36 +163,27 @@ Reference config — decision, text request, idle human step, one final state:
 }
 ```
 
-## 3. Validate with `validateAgentConfig`
-
-`validateAgentConfig` from `@statelyai/agent/validate` checks a candidate against the shipped workflow schema with Ajv 2020 built in. It needs `ajv` installed (an optional peer of the package). Run it before anything else touches the config.
-
-```ts
-import { validateAgentConfig } from "@statelyai/agent/validate";
-import type { AgentWorkflowConfig } from "@statelyai/agent";
-
-function validateGeneratedConfig(candidate: unknown): AgentWorkflowConfig {
-  const result = validateAgentConfig(candidate);
-  if (result.valid) return candidate as AgentWorkflowConfig;
-  throw new Error(
-    result.errors.map((error) => `${error.path || "(root)"} ${error.message}`).join("\n"),
-  );
-}
-```
+## 3. Verify with `verifyAgentConfig`
 
-Each error carries a JSON Pointer `path`, so a repair prompt can name the exact bad field.
+`verifyAgentConfig(config, options)` from `@statelyai/agent/validate` runs every pre-run gate and stops at the first failure. It never throws on a bad config; each gate is a record `{ name, ok, detail? }`. It needs `ajv` installed (an optional peer of the package).
 
-## 4. Lower with `fromConfig`
+| Gate         | Checks with                            | Fails on                                                                                                                                                   |
+| ------------ | -------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `validate`   | `validateAgentConfig` (shipped schema) | malformed config shape                                                                                                                                     |
+| `vocabulary` | `allow` lists                          | an invoke src, named guard, named action, `agent.decide` event, or model ref not listed                                                                    |
+| `lower`      | `setupAgent.fromConfig`                | unresolved named guard/action, `onDone` on an `agent.decide` invoke                                                                                        |
+| `lint`       | `lintAgentMachine`                     | error findings (warnings too with `lint: { warnings: true }`)                                                                                              |
+| `explore`    | `explorePaths`, once per input         | a path ending outside `explore.terminals` (default `['done']`), a loop past `maxDepth`, a `maxPaths`/`maxNodes` cap, untried payloads (`unexpandedEvents`) |
 
-`fromConfig` needs a `compileSchema` that turns the JSON Schemas _inside_ the config into Standard Schema validators. Two engines, two jobs: the 2020 build above checks the config document; this one compiles the per-field schemas. The library bundles no JSON Schema engine, by design.
+`compileSchema` turns the JSON Schemas _inside_ the config into Standard Schema validators for `fromConfig`. The library bundles no JSON Schema engine:
 
 ```ts
 import Ajv from "ajv";
-import { setupAgent, type SchemaCompiler, type StandardSchemaV1 } from "@statelyai/agent";
+import type { SchemaCompiler, StandardSchemaV1 } from "@statelyai/agent";
 
 const ajv = new Ajv({ strict: false });
 
-export const ajvCompileSchema: SchemaCompiler = (jsonSchema, name): StandardSchemaV1 => {
+export const compileSchema: SchemaCompiler = (jsonSchema, name): StandardSchemaV1 => {
   const validate = ajv.compile(jsonSchema);
   return {
     "~standard": {
@@ -175,87 +197,139 @@ export const ajvCompileSchema: SchemaCompiler = (jsonSchema, name): StandardSche
                 message: `${name}${e.instancePath} ${e.message}`,
               })),
             },
-      // Expose the source JSON Schema so lint's serializability checks can read the shape.
+      // Expose the source JSON Schema so lint and describeMachine can read the shape.
       jsonSchema: { input: () => jsonSchema },
     },
   };
 };
+```
 
-const { machine, schemas } = setupAgent.fromConfig(config, { compileSchema: ajvCompileSchema });
+Run the gates. List the host's real names in `allow`; an omitted key is unrestricted:
+
+```ts
+import { verifyAgentConfig } from "@statelyai/agent/validate";
+
+const result = await verifyAgentConfig(candidate, {
+  compileSchema,
+  allow: {
+    actors: [],
+    requests: ["draftReply"],
+    guards: [],
+    actions: [],
+    events: ["ESCALATE", "REPLY", "APPROVE", "REJECT"],
+    models: ["openai/gpt-6-luna"],
+  },
+  explore: {
+    inputs: [{ ticket: "Export downloads a 0-byte CSV on Safari." }],
+    outputs: { draftReply: [{ reply: "" }] },
+    // ESCALATE's `reason` is a free-form string: give exploration one to try.
+    payloads: () => ({ ESCALATE: [{ reason: "needs a human" }] }),
+  },
+});
+// result.ok, result.gates, result.config (after validate), result.machine (after lower)
 ```
 
-Lowering is itself a gate: it throws on an unresolved named guard/action and on an `onDone` attached to an `agent.decide` invoke.
+- Exploration forks past idle waits on every event the waiting state accepts, so a human step still ends `done`. Add `terminals: ["done", "idle"]` only when a path may legitimately stop at a wait with nothing to accept.
+- `guards` / `actions`: named implementations, passed to `fromConfig`. `actors`: bound with `machine.provide({ actors })` after lowering.
+- `explore` keys other than `inputs` and `terminals` pass through to `explorePaths`: `outputs` (output lists per request or actor src: one element is a canned output, two or more fork one branch each), `errors` (fork a failing branch per src), `payloads` (fork one branch per payload at decisions and idle waits), `dedupeBy` (explore each snapshot key once; the key must cover every field a guard reads), `maxDepth` (default 8), `maxPaths` (default 200), `maxNodes` (default 100,000).
+- Stub each request's output from its declared `output` schema (`""` for string, `0` for number, `[]` for array, recurse on `properties`). Add an `errors` entry for every src whose `onError` path matters.
+- Pick `explore.inputs` that reach every branch the goal cares about. Exploration covers only the inputs you give it.
+- A passing `lint` gate carries its warnings in `detail`. Report them.
 
-## 5. Lint
+## 4. Repair loop
+
+A failed gate's `detail` names the offending path, state, or name, one problem per line. Feed it back verbatim with the rejected config, and run the gates again:
 
 ```ts
-import { lintAgentMachine } from "@statelyai/agent/testing";
-const diagnostics = lintAgentMachine(machine);
-lintAgentMachine(machine, { throw: true }); // throws AgentLintError on error-severity findings
+let prompt = originalTask;
+for (let attempt = 0; attempt < 3; attempt++) {
+  const candidate = await author({ system, prompt });
+  const result = await verifyAgentConfig(candidate, verifyOptions);
+  if (result.ok) return result;
+  const failed = result.gates.find((gate) => !gate.ok)!;
+  prompt = `${originalTask}\n\nYour previous config:\n${JSON.stringify(candidate)}\n\nGate '${failed.name}' failed:\n${failed.detail}\nReturn a corrected config.`;
+}
 ```
 
-Every check applies to config-built machines, reachability included — the lowering keeps the config's transition targets, so `unreachable-state` and `missing-final` read the real graph. Do not disable checks.
+Keep the cap at 3. A config that fails three repairs is usually asking for something the data form cannot express: author it in TypeScript with `setupAgent` instead, or report that the vocabulary is missing something.
 
-The one that bites most often is `decide-without-events`: an `allowedEvents` list with no matching `on` transitions produces a decision the machine can never deliver.
+## 5. Manual fallback
 
-## 6. Simulate a dry run
-
-Lint is structural. A dry run proves a path actually settles. No API key needed.
+Without `verifyAgentConfig` (older package, or a gate you need to run alone), run the same gates by hand:
 
 ```ts
-import { simulateAgent } from "@statelyai/agent/testing";
+import { setupAgent, type AgentWorkflowConfig } from "@statelyai/agent";
+import { validateAgentConfig } from "@statelyai/agent/validate";
+import { explorePaths, lintAgentMachine, simulateAgent } from "@statelyai/agent/testing";
+
+const validation = validateAgentConfig(candidate); // { valid, errors: [{ path, message }] }, path is a JSON Pointer
+if (!validation.valid) {
+  // Stop here: feed the errors back as the repair prompt. Never lower an invalid config.
+  throw new Error(validation.errors.map((e) => `${e.path} ${e.message}`).join("\n"));
+}
+const { machine } = setupAgent.fromConfig(candidate as AgentWorkflowConfig, { compileSchema }); // throws on unresolved names
+lintAgentMachine(machine, { throw: true }); // throws AgentLintError on error findings
 const dryRun = await simulateAgent(machine, {
   input: { ticket: "" },
   script: {
     decisions: { "agent.decide": [{ type: "REPLY" }] },
     text: { draftReply: [{ reply: "" }] },
   },
-});
-// dryRun.status: 'done' | 'idle' | 'exhausted'
+}); // dryRun.status: 'done' | 'idle' | 'exhausted' ('exhausted' = loops, reject)
+const report = await explorePaths(machine, {
+  input: { ticket: "" },
+  outputs: { draftReply: [{ reply: "" }] },
+  payloads: () => ({ ESCALATE: [{ reason: "needs a human" }] }),
+}); // report.unexpandedEvents lists types that still need a payloads entry
 ```
 
-Derive the script from the config rather than guessing: take the first entry of each decision's `allowedEvents`, and stub each request's output from its declared `output` schema (`""` for string, `0` for number, `[]` for array, recurse on `properties`).
-
-- `'exhausted'` → the machine loops. Reject it.
-- `'idle'` → it stopped at a human step. Expected when the config has one.
-- To cover every branch instead of one path, use `explorePaths` / `canReach` (its result is `{ reachable, witness }`).
+The lint check that bites most often is `decide-without-events`: an `allowedEvents` list with no matching `on` transitions. Do not disable checks.
 
-## 7. Repair loop
+## 6. Persist traces from day one
 
-Every gate throws with a message naming the offending field, state, or path. Feed that message back verbatim and regenerate:
+An improver needs data. Wire an event log store and a thread id per run before the first real run, so every run is a trace:
 
 ```ts
-for (let attempt = 0; attempt < 3; attempt++) {
-  try {
-    return buildAndVet(await author({ system, prompt }));
-  } catch (error) {
-    prompt = `${originalTask}\n\nYour previous config was rejected:\n${(error as Error).message}\nReturn a corrected config.`;
-  }
-}
+import { runAgent } from "@statelyai/agent";
+import { createInMemoryEventLogStore } from "@statelyai/agent/log";
+
+const store = createInMemoryEventLogStore(); // tests only; implement AgentEventLogStore on your database
+const result = await runAgent(machine, {
+  input,
+  executors,
+  store,
+  threadId: `tickets:${ticketId}`,
+});
 ```
 
-Keep the cap at ~3. A config that fails three schema-shaped repairs is usually asking for something the data form cannot express — author it in TypeScript with `setupAgent` instead.
+- `threadId` is required with `store`. One thread per run.
+- Give thread ids a prefix an improver can list by, such as `tickets:<ticketId>`. `store.listThreads({ prefix, limit, cursor })` pages through them. It is optional on a custom `AgentEventLogStore`, so with a store that lacks it, record each thread id with its scenario or ticket.
+- Failed decision attempts are journaled on the entry they explain; `getDecisionAttempts(entries)` from `@statelyai/agent/log` reads them back.
+- `replay(machine, await store.read(threadId))` from `@statelyai/agent/log` rebuilds the final state with no model calls. Pass the machine that recorded the thread: with an environment and a policy, the environment; `getUsageFromEvents(entries)` sums token usage.
 
-## 8. Hand back
+## 7. Hand back
 
 Deliver the config JSON plus how to run it:
 
 ```ts
-import { runToQuiescence } from "@statelyai/agent";
+import { runAgent } from "@statelyai/agent";
 import { createAiSdkExecutors } from "@statelyai/agent/ai-sdk";
 
-const result = await runToQuiescence(
-  createAgentRuntime(machine, { executors: createAiSdkExecutors({ resolveModel }) }),
-  { input: { ticket: "Export downloads a 0-byte CSV on Safari." } },
-);
+const run = await runAgent(result.machine!, {
+  input: { ticket: "Export downloads a 0-byte CSV on Safari." },
+  executors: createAiSdkExecutors({ resolveModel }),
+  store,
+  threadId,
+});
 ```
 
-Say which gates passed, list any warning-severity lint diagnostics, and report the dry-run status. Model refs are strings, so the host supplies `resolveModel`.
+Say which gates passed (list `result.gates`), list any lint warnings, state the goal and its success criteria, and say where traces are written. Model refs are strings, so the host supplies `resolveModel`.
 
 ## Known limits
 
 - **Named guards and actions are host-resolved.** A config carries no functions. `guard: "isReady"` only works if the host passes an `isReady` implementation to `fromConfig`; an unresolved name is a build-time throw. Either list the host's names in the prompt or forbid named guards/actions entirely.
 - **Expressions are dot-path templates only.** `"{{ context.a.b }}"` — no comparisons, no arithmetic, no method calls, no partial interpolation inside a larger string.
-- **Ajv validity is not semantic validity.** A config can validate and still invoke a request that does not exist or route to a state that solves nothing. That is what lint and simulate are for.
-- **A dry run covers one path.** Use `explorePaths` when the branch structure matters.
-- **Model refs and tool names are unchecked strings.** They resolve at run time, in the host.
+- **Ajv validity is not semantic validity.** A config can validate and still invoke a request that does not exist or route to a state that solves nothing. That is what the `vocabulary`, `lint`, and `explore` gates are for.
+- **Exploration covers the inputs you give it.** A branch no `explore.inputs` entry reaches is unverified.
+- **Model refs are checked only with `allow.models`.** Tool names stay unchecked strings that resolve at run time, in the host.
+- **The gates check structure, not quality.** Whether a config is better than another is measured by replaying scenarios against an objective. See the `improve-machine` skill.