@statelyai/agent 2.0.0-alpha.29 → 2.0.0-alpha.30

package/skills/improve-machine/SKILL.md

@@ -1,17 +1,17 @@
 ---
 name: improve-machine
-description: Improve a @statelyai/agent policy machine (a JSON AgentWorkflowConfig) from its run traces toward its goal, inside a fixed environment machine, with gates and a held-out promotion rule. Use when asked to "improve this agent", "self-improve", "optimize the playbook from traces", "tune the policy", "make the agent cheaper or more accurate", or "why is my agent failing" for a machine whose runs are persisted to an event log store.
+description: Improve a @statelyai/agent policy machine (a TypeScript setupAgent machine or a JSON AgentWorkflowConfig) from its run traces toward its goal, inside a fixed environment machine, with gates and a held-out promotion rule, and hand the result to a person as a pull request. Use when asked to "improve this agent", "self-improve", "optimize the playbook from traces", "tune the policy", "make the agent cheaper or more accurate", or "why is my agent failing" for a machine whose runs are persisted to an event log store.
 ---
 
 # Improve an agent machine
 
-You improve a **policy**: the JSON `AgentWorkflowConfig` that decides how the agent works toward its goal. It runs inside an **environment**: a TypeScript machine whose states and guards encode the rules of the world and the business invariants. One round changes the policy once and measures it:
+You improve a **policy**: the machine that decides how the agent works toward its goal, written in TypeScript or as a JSON `AgentWorkflowConfig`. It runs inside an **environment**: a TypeScript machine whose states and guards encode the rules of the world and the business invariants. One round changes the policy once and measures it:
 
 ```
-locate inputs → read traces → diagnose → propose one diff → verifyAgentConfig (≤ 3 repairs) → replay train → replay held-out → promote or discard → record
+locate inputs → read traces → diagnose → propose one diff → gate (≤ 3 repairs) → replay train → replay held-out → recommend or discard → record → pull request
 ```
 
-To author a policy from scratch, use the `generate-machine` skill. Concept page: `docs/environment-and-policy.md`.
+This is step 4 of 4, after `loop-to-machine`, `add-traces`, and `add-evals`. To author a JSON policy from scratch, use the `generate-machine` skill. Concept page: [Environment and policy](https://github.com/statelyai/agent/blob/main/docs/environment-and-policy.md).
 
 ## Hard rules
 
@@ -19,8 +19,32 @@ To author a policy from scratch, use the `generate-machine` skill. Concept page:
 - **Never edit a machine while it runs.** A promoted policy applies to new runs only.
 - **The objective comes from ground truth.** Score outcomes from the traces (terminal states, outputs) or from effects recorded in the world. Never ask a model to grade its own runs.
 - **One cause per round, smallest diff.** A diff that fixes two things cannot tell you which one helped.
-- **Diagnose from training scenarios only.** Held-out scenarios are for the promotion decision. Never read their failures to shape a change, and never move a scenario between splits.
+- **Diagnose from training scenarios only.** Held-out scenarios are for the promotion decision. Never read their failures to shape a change, and never move a scenario between splits. Eval files may list baseline failures for every split; read only the training rows.
 - **Stop after 3 rounds without a promotion** (or the limit the user gives).
+- **Propose, never promote.** A round ends in a recommendation. A recommended candidate goes to a person as a pull request with its scorecard; merging it is the promotion. Never merge, deploy, or switch the running policy yourself.
+
+## Policy format
+
+The rounds are the same for both formats. What differs is what the diff may touch and how it is gated.
+
+| | TypeScript policy | JSON policy |
+| --- | --- | --- |
+| The policy | A `setupAgent` machine in its own file(s), with a declared `version` | An `AgentWorkflowConfig` file |
+| May change | The policy file(s) and their `CHANGELOG.md`; once, the line in the runner that routes the decision to the policy; tests that pin measured numbers (see the pull request) | The config's `states` and `context` |
+| Gates | `tsc`, `lintAgentMachine`, `explorePaths` on the policy, the environment check, the overfit check | `verifyAgentConfig` |
+| Vocabulary | What the policy file imports and the events the environment offers | The `allow` lists |
+
+**No policy yet.** `loop-to-machine` leaves a single model decision inside the environment when the old code had no policy logic. Then:
+
+- Record `v0` in a new `CHANGELOG.md` next to the environment (in a `policy/` directory): "the model decides directly". This entry goes in the working tree.
+- The first recommended candidate introduces the policy as `v1`: a policy file that reproduces the current behavior (it passes the environment's prompt and facts through to the model) plus the one rule the diagnosis found, and the routing line in the runner (`withPolicy(modelExecutors)`). The pattern is in `../loop-to-machine/policy.md`.
+- Its `description` is the environment's goal with measurable success criteria. If the environment's goal has none, draft them from the confirmed objective and list the draft under questions for the user.
+- If the policy needs a fact the environment does not pass in the decision's `metadata.facts`, that is an environment change request.
+
+For a TypeScript policy, before the first round:
+
+- **List the environment files** (the environment machine, its guards, the world and its tools) and record a hash of each. Any change to them fails the round. The runner is not one of them, but its diff may contain only the routing line.
+- **Keep the policy a machine.** A diff may add states, `choice` branches, guards over facts, helpers in the policy file, or change a prompt or a model ref. It may not turn the policy into a free function, call tools the environment owns, or import from the environment anything but its schemas, types, constants, and exported guard functions.
 
 ## 1. Locate the inputs
 
@@ -30,16 +54,16 @@ Find all six before changing anything. If one is missing, stop and ask.
 | -------------- | ------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------- |
 | Goal           | Root `description` of the policy config or the environment machine                               | Draft one with measurable success criteria; ask the user to confirm it      |
 | Objective      | A scorer over outcomes (an eval file, a `score*` function, a benchmark)                          | Write one from the goal's success criteria; **ask the user to confirm it**  |
-| Traces         | The `store` passed to `createAgentRuntime` / `runAgent`; its `listThreads`, or recorded ids      | Wire a store first (see `generate-machine` step 6) and collect runs         |
+| Traces         | The `store` passed to `createAgentRuntime` / `runAgent`; its `listThreads`, or recorded ids      | Wire a store first (the `add-traces` skill) and collect runs                |
 | Scenarios      | A scenario list with expected outcomes, split into train and held-out                            | Propose a fixed split (about 70/30, every category in both); ask to confirm |
 | Vocabulary     | The actors, requests, named guards, actions, decision events, and model refs the host implements | Read them off the host code; never invent names                             |
 | Policy history | The policy files and a `CHANGELOG.md` next to them                                               | Create the changelog with the current policy as `v0`                        |
 
-The objective defines "better". Do not start a round until the user has confirmed it. A good objective has:
+The objective defines "better". Do not start a round until the user has confirmed it, including how two versions compare when it has more than one field. A good objective has:
 
-- a primary metric from the goal (accuracy against expected outcomes, resolution rate),
+- a primary metric from the goal (accuracy against expected outcomes and effects, resolution rate). With several, state the rule: one primary, or none worse and one better,
 - a hard constraint: invariant violations, which must be 0,
-- a cost term (model calls, tokens, guard rejections) used as a tie-breaker or a stated budget.
+- a cost term (model calls, tokens, guard rejections) used as a tie-breaker or a stated budget. Count the calls that reach a model executor, not the environment's decision requests: a policy that answers with a rule makes no model call, and a cost term that counts requests cannot see the saving.
 
 ## 2. Read
 
@@ -48,6 +72,7 @@ Read both machines as text. `describeMachine` puts the goal first:
 ```ts
 import { describeMachine, setupAgent } from "@statelyai/agent";
 
+// A TypeScript policy: import it. A JSON policy: lower it.
 const { machine: policy } = setupAgent.fromConfig(policyConfig, { compileSchema });
 const environmentText = describeMachine(environment);
 const policyText = describeMachine(policy);
@@ -100,9 +125,10 @@ Group the failing training episodes by cause. Check these clusters in order:
 
 1. **Invariant violations.** Any violation is a stop: the environment is wrong, not the policy. Write a change request.
 2. **Wrong terminal state.** The episode ended in `escalated` when the label says `issued`, or the reverse. Look at which decision led there.
-3. **Guard rejections.** The policy keeps proposing events the environment refuses: `rejected-by-guard` entries in the rows' `failedAttempts`. The policy lacks a check the environment already makes.
-4. **Loops and budget exhaustion.** Repeated state values, or a terminal reached because a turn or retry budget ran out.
-5. **Cost hotspots.** Correct episodes that spend model calls where a `choice` over known facts, or a cheap judgment, would decide.
+3. **Right terminal state, wrong effect.** The episode ended `issued` with the wrong amount, or reached the right state by a costlier path. Compare the chosen event with the offered options at each decision.
+4. **Guard rejections.** The policy keeps proposing events the environment refuses: `rejected-by-guard` entries in the rows' `failedAttempts`. The policy lacks a check the environment already makes.
+5. **Loops and budget exhaustion.** Repeated state values, or a terminal reached because a turn or retry budget ran out.
+6. **Cost hotspots.** Correct episodes that spend model calls where a `choice` over known facts, or a cheap judgment, would decide.
 
 Pick the cluster with the largest effect on the objective. Write the hypothesis down before you edit anything (template in [reference.md](reference.md)): cause, evidence (thread ids), the change, and which metric should move.
 
@@ -111,12 +137,29 @@ Pick the cluster with the largest effect on the objective. Write the hypothesis
 Make the smallest policy diff that addresses the one cause.
 
 - Typical diffs: add a `choice` branch over facts already in context; replace a model call with a judgment actor from the vocabulary; narrow an `agent.decide` `allowedEvents` list; turn an open decision into a closed choice with `payloads`; tighten a prompt; add an `onError` path; lower a model ref to a cheaper one in the vocabulary.
-- Use only names in the vocabulary. A named guard the host does not implement fails the `lower` gate.
-- Keep the config `id`. Bump `version`.
+- TypeScript policy: compare facts directly in a guard or `choice` instead of adding a named guard; keep helpers in the policy file.
+- JSON policy: use only names in the vocabulary. A named guard the host does not implement fails the `lower` gate.
+- Keep the machine `id`. Bump `version`.
 - Keep the root `description` (the goal) unchanged. Changing the goal is the user's decision.
 
 ## 5. Gate
 
+### TypeScript policy
+
+Run every check; a failure's output is the repair prompt. Cap repairs at 3.
+
+1. **Environment unchanged:** every environment file hashes as recorded. A change is never repaired; discard the candidate.
+2. **Types:** `tsc --noEmit` passes.
+3. **Lint:** `lintAgentMachine(policy)` from `@statelyai/agent/testing` reports no errors.
+4. **Paths:** for probe inputs built from recorded decision requests (one per training scenario that reaches the decision: its facts and offered options), `canReach(policy, (snapshot) => snapshot.status === "done" && !isOfferedEvent(snapshot.output), { input })` is `{ reachable: false, complete: true }`: every final output is one of the offered events. Pass `outputs` for any actor the policy invokes, and `errors: { "agent.decide": new Error("down") }` so the fallback path is checked too.
+5. **Source rules:** the policy file is still one `setupAgent` machine; its imports from the environment are only schemas, types, constants, and exported guard functions; it names no actor the environment owns. These are source checks: read the diff, or grep its imports and actor names.
+6. **Overfit:** the diff contains no literal from the scenarios (ids, ticket text, customer ids, order ids). A branch that matches one scenario's data is memorization; discard it.
+7. **Existing evals:** the rule and decision evals from `add-evals` still pass. Tests that pin measured numbers of the current version (a baseline failure list, a call count) are expected to change; update them in the same pull request and list each change in its body.
+
+The environment's own `canReach` proofs still cover every policy, because the environment's guards judge every proposed event.
+
+### JSON policy
+
 Run `verifyAgentConfig` from `@statelyai/agent/validate` with the vocabulary and probe inputs. Probe inputs are one input per training scenario category, chosen to reach every branch the diff touches.
 
 ```ts
@@ -150,15 +193,15 @@ const failed = verified.gates.find((gate) => !gate.ok);
 
 ## 6. Replay
 
-Run the candidate on the training scenarios, then on the held-out scenarios. Write every replay to the store under new thread ids that name the split and version (`train:v3:<scenario>:r1`, `heldout:v3:<scenario>:r1`), so the next round can list them by prefix.
+Run the candidate on the training scenarios, then on the held-out scenarios. Write every replay to the store under new thread ids that name the split and version, in the scheme `add-traces` set up (`train:v3:<scenario>:r1`, `held-out:v3:<scenario>:r1`), so the next round can list them by prefix. With no persistent store, write each thread's `result.events` to a file.
 
 - Run the current policy and the candidate on the same scenarios, the same world state, and the same model refs.
 - Model calls are nondeterministic. Run each scenario k times (k ≥ 3 when calls are not cached) and compare means.
 - Compute the objective, invariant violations, and cost per split, for both versions.
 
-## 7. Promote or discard
+## 7. Recommend or discard
 
-State the tolerance before you replay. Promote only when every condition holds:
+State the tolerance before you replay. Recommend only when every condition holds:
 
 - invariant violations are 0 on both splits,
 - the objective on the held-out split is better than the current policy's,
@@ -167,6 +210,14 @@ State the tolerance before you replay. Promote only when every condition holds:
 
 Otherwise discard the candidate and keep the current policy. Either way, append an entry to the `CHANGELOG.md` next to the policies: version, hypothesis, diff summary, gate results, before and after per split, and the decision. Record discarded candidates too; they stop the next round from retrying the same idea.
 
+### The pull request
+
+A recommended candidate becomes one pull request (or, with no remote, one branch and a diff file), opened against the repository's integration branch:
+
+- **The diff:** the policy file(s) and the new `CHANGELOG.md` entry; the runner's routing line, when this candidate introduces the policy; and updated pinned measurements, each listed in the body. Nothing else.
+- **The body:** the hypothesis, the scorecard (objective, violations, and cost before and after, train and held-out separately, k runs each), every scenario that changed outcome, the gate results, and the thread ids a reviewer can replay.
+- **One candidate per pull request.** Leave your working tree on the current policy. Later rounds build on the current policy, not on an unmerged candidate.
+
 ## 8. Stop and report
 
 Stop when any of these holds:
@@ -176,11 +227,11 @@ Stop when any of these holds:
 - the remaining failures need an environment change (write a change request),
 - the vocabulary lacks a capability the fix needs (ask for it; name the actor, request, or guard and its input and output).
 
-Report to the user: rounds run, each candidate with its hypothesis and decision, the objective before and after on both splits, violations, cost, and any change requests.
+Report to the user: rounds run, each candidate with its hypothesis and decision, the pull request link (or, with no remote, the branch and diff file), the objective before and after on both splits, violations, cost, and any change requests.
 
 ## Worked example
 
-`examples/self-improving-refunds` in the `statelyai/agent` repo runs this loop end to end:
+[`examples/self-improving-refunds`](https://github.com/statelyai/agent/blob/main/examples/self-improving-refunds) in the `statelyai/agent` repo (not in the installed package) runs this loop end to end for a JSON policy:
 
 - the refund desk is the environment: business rules as guards, and an `illegallyIssued` sentinel proven unreachable,
 - `playbooks/*.json` are policy versions, with the vocabulary listed in `playbooks/README.md`,

package/skills/improve-machine/reference.md

@@ -12,10 +12,11 @@ Templates for the `improve-machine` skill: a round checklist, the hypothesis, th
 [ ] Baseline: objective, violations, cost per split for the current version
 [ ] Failures clustered (train only); one cause picked
 [ ] Hypothesis written
-[ ] Candidate: smallest diff, id kept, version bumped, description unchanged
-[ ] verifyAgentConfig passed (repairs used: _/3)
+[ ] Candidate: smallest diff, id kept, version bumped, description unchanged (or, for a new policy: v0 recorded, v1 reproduces current behavior plus one rule)
+[ ] Gates passed: TS (env unchanged, tsc, lint, paths, source rules, overfit, evals) or JSON (verifyAgentConfig) (repairs used: _/3)
 [ ] Replayed train, then held-out, k runs each, new thread ids
 [ ] Promotion rule applied; CHANGELOG entry written
+[ ] Recommended candidate opened as one pull request with its scorecard; not merged
 [ ] Stop condition checked
 ```
 
@@ -39,7 +40,7 @@ You improve a policy machine. You may change only the policy config below.
 {describeMachine(environment)}
 
 Current policy (version {n}):
-{JSON.stringify(policyConfig, null, 2)}
+{the policy file's source, or JSON.stringify(policyConfig, null, 2)}
 
 Vocabulary (use no other names):
 - actors: {actors}
@@ -53,8 +54,12 @@ Objective on the training scenarios: {objective summary}
 Diagnosed cause: {hypothesis.cause}
 Evidence: {failing rows: scenario, expected, reached state, rejections, tokens}
 
-Return one complete AgentWorkflowConfig as JSON. Make the smallest change that
-addresses the diagnosed cause. Keep "id" and "description". Set "version" to {n + 1}.
+Return the complete policy: the TypeScript file, or one AgentWorkflowConfig as
+JSON. Make the smallest change that addresses the diagnosed cause. Keep "id" and
+"description". Set "version" to {n + 1}. Never match on a scenario's data.
+For a TypeScript policy, keep the result one `setupAgent` machine. Import from
+the environment only schemas, types, constants, and exported guard functions.
+Do not invoke tools the environment owns.
 ```
 
 On a failed gate, append:
@@ -69,11 +74,11 @@ Return a corrected config.
 ## Changelog entry
 
 ```md
-## v{n} — {promoted | discarded}
+## v{n} — {recommended | discarded}
 
 - Hypothesis: {cause} → {change}
 - Diff: {states or transitions touched}
-- Gates: validate ✓ vocabulary ✓ lower ✓ lint ✓ explore ✓ (repairs: {r})
+- Gates: {TS: environment ✓ tsc ✓ lint ✓ paths ✓ overfit ✓ evals ✓ | JSON: validate ✓ vocabulary ✓ lower ✓ lint ✓ explore ✓} (repairs: {r})
 - Train: objective {a} → {b}, violations {v}, cost {c0} → {c1}
 - Held-out: objective {a} → {b}, violations {v}, cost {c0} → {c1}
 - Regressions: {none | scenario ids}

package/skills/loop-to-machine/SKILL.md

@@ -0,0 +1,183 @@
+---
+name: loop-to-machine
+description: Convert existing agent code (a while-loop around a model, a tool loop, ad-hoc if/else control flow) into an XState v6 state machine with @statelyai/agent, and prove with a parity report that it behaves like the code it replaces. Use when asked to "turn this agent into a state machine", "port my loop to xstate", "make this a statechart", "refactor this agent into a machine", or "split environment and policy" for code that already runs.
+---
+
+# Convert agent code to a machine
+
+You replace the control flow of working agent code with a machine, and show where the machine behaves like the old code and where it differs on purpose. The model calls, tools, and retry code move across unchanged.
+
+```
+inventory → pin the baseline → environment → (policy) → runner → proofs → parity → hand back
+```
+
+This is step 1 of 4. Next: `add-traces`, then `add-evals`, then `improve-machine`. To author a machine with no existing code, use `generate-machine`.
+
+## Hard rules
+
+- **Pin the baseline before you change anything.** The parity test runs against the old code, which stays in the repo, untouched, until a person deletes it.
+- **Never promise "behaves exactly like".** Promise what the parity report shows: same outcome and effects on these scenarios, with every difference labeled.
+- **Never copy a rule break.** Where the old code broke a rule, the machine blocks it. That difference is `intended`, not a regression.
+- **Rules are code, not prompt text.** A rule that lived only in a system prompt becomes a guard. Keep the prompt line only as context for the model.
+- **Business rules belong to people.** If a rule is ambiguous (which threshold, which order of checks), ask. Never pick one silently.
+
+## 1. Inventory the code
+
+Read the loop and fill this table before writing anything. Each row maps to a machine element.
+
+| In the code                                                                  | Becomes                                                                  |
+| ---------------------------------------------------------------------------- | ------------------------------------------------------------------------ |
+| A tool with a side effect someone must gate (refund, send)                   | An actor invoked by one state, entered only through a guarded transition |
+| A read-only tool (search, lookup)                                            | An actor the machine invokes itself, or a tool on a text request         |
+| A rule: an `if`, a prompt line, a check after the fact                       | A guard on the transition it protects                                    |
+| A point where the model picks what to do                                     | A decision state: `agent.decide`, usually a closed choice                |
+| A value the model pulled out of the input (an order id, an amount)           | Code when the format is fixed (a regex); a text request when it is not   |
+| A step every correct run takes in the same order (verify, then look up)      | A state the machine enters itself; the model no longer chooses it        |
+| A person in the loop: `await approve()`, a returned `pending`, or a tool that asks a person (`askCustomer`, `requestApproval`), even a synchronous one | An idle state with `meta.interaction`, answered by `onIdle` in the runner |
+| A cap (`turns < 8`, a retry counter) on a loop the machine keeps             | A `choice` state against an exported constant                            |
+| A cap on a loop the machine removed                                          | Nothing. Record that it is gone and why                                  |
+| Each distinct return                                                         | A final state                                                            |
+| A terminal tool with a side effect (`escalate`, `close`)                     | A state that invokes it, then the final state; its `onError` goes to a `failed` final |
+
+List the rules as a numbered list in your notes. Every one must end as a guard, a state ordering, or a written reason it was dropped.
+
+## 2. Pin the baseline
+
+The parity test exists before the machine does.
+
+1. **Scenarios.** Reuse the code's tests, fixtures, or benchmark cases. Each has an `id` and, where people know it, the `expected` outcome. If there are none, draft 6–10 that cover every terminal and every rule, and ask the user to confirm them.
+2. **Deterministic stand-in models.** Parity with a live model drifts between runs. Use the code's scripted or mock models if it has them; otherwise write one keyed on the input (see [`examples/mock-model.ts`](https://github.com/statelyai/agent/blob/main/examples/mock-model.ts) in the `statelyai/agent` repo). You want two: one that follows the rules, and one that breaks them. A seeded random model needs its seed derived from the scenario id and a fresh instance per run, or every scenario replays the same draws.
+3. **Ground truth.** Read outcomes and effects from the world, not from the agent's own report. `effects` holds **every write the world records**: the refunds, and also the approvals, the verifications, the questions asked, the escalations. A refund issued without its approval looks correct if `effects` holds only refunds. Text a model wrote (an escalation reason) differs on every run: compare its count or presence, not the text. If the world can check rules after the fact, return those violations. If it cannot, write the check as a plain function over the effects.
+4. **The harness check.** Run the old code on both sides with each stand-in model. Each row's baseline must equal its candidate (the runs are deterministic), and with the rule-following model every row must be `same`. With a rule-breaking model, rows are `regression` by design: the candidate breaks rules.
+
+```ts
+import { compareRuns, type ParityResult, type ParityRun } from "@statelyai/agent/testing";
+
+function toParityRun(run: { outcome: string; world: World }): ParityRun {
+  return {
+    outcome: run.outcome,
+    // Every write the world records; model-written text as a count.
+    effects: { refunds: run.world.refunds, approvals: run.world.approvals, verified: [...run.world.verified], escalations: run.world.escalations.length },
+    violations: run.world.violations, // rules broken, from ground truth
+  };
+}
+
+// A model factory, called once per scenario and per side. Construct a stateful or seeded
+// model inside it (seeded from s.id); returning a shared stateless object is fine.
+const runWith = (runner: typeof runLoop, model: (s: Scenario) => Model) => async (s: Scenario) =>
+  toParityRun(await runner({ ...s, model: model(s), world: createWorld(s) }));
+
+const MODELS: Array<[string, (s: Scenario) => Model]> = [
+  ["careful", () => carefulModel],
+  ["sloppy", () => sloppyModel],
+];
+
+test.each(MODELS)("harness: the loop matches itself (%s)", async (_, model) => {
+  const report = await compareRuns({ scenarios: SCENARIOS, baseline: runWith(runLoop, model), candidate: runWith(runLoop, model) });
+  for (const row of report.scenarios) expect(row.candidate, row.id).toEqual(row.baseline);
+});
+```
+
+## 3. Author the environment
+
+The environment is the machine: states, guards that encode the rules, side-effecting actors, an invariant sentinel. Use `setupAgent` (from `@statelyai/agent`) with zod schemas for `input`, `context`, `output`, and every event. Export the event schemas as one object, so a policy can reuse them later.
+
+A gated action:
+
+```ts
+deciding: {
+  tags: ["decision"],
+  invoke: {
+    src: "agent.decide",
+    input: ({ context }) => ({
+      model: "decider", // a key the host's executors resolve
+      name: "chooseResolution", // tests, traces, and a later policy find it by this name
+      prompt: `Ticket: ${context.ticket}\nOrder: ${JSON.stringify(context.order)}`,
+      allowedEvents: ["REFUND", "ESCALATE", "CLOSE"],
+      // A closed choice: the decider picks one of these. The runtime drops the ones the guards refuse.
+      payloads: { REFUND: refundAmounts(context).map((amount) => ({ amount })) },
+      // The facts a decider reads. Arrives as `request.metadata.facts`.
+      metadata: { facts: { order: context.order, requestedAmount: context.requestedAmount } },
+    }),
+    onError: { target: "escalated" }, // retries exhausted, or no option legal
+  },
+  on: {
+    REFUND: ({ context, event }) =>
+      refundIsLegal(context, event.amount)
+        ? { target: event.amount > APPROVAL_THRESHOLD ? "awaitingApproval" : "paying", context: { amount: event.amount } }
+        : undefined, // undefined rejects: the decision retries with feedback
+    ESCALATE: { target: "escalated" },
+    CLOSE: { target: "closed" },
+  },
+},
+```
+
+- **Do the deterministic steps yourself.** Identity, lookup, the ownership check: states the machine runs in order. A model only answers what genuinely needs judgment.
+- **Export the guards** as plain functions (`refundIsLegal`) so tests can call them, and build the machine in a factory (`createDeskMachine({ refundIsLegal })`) so a test can pass a loosened guard.
+- **Add an invariant sentinel** for the rule that must never break: a final state with a description starting `SENTINEL:`, entered from an `always` on the state that invokes the side effect (`paying`). An `always` is taken before the state's invoke starts, so the effect never runs. Write the check as its own function, not a call to the guard, so loosening the guard does not loosen the check.
+- **A person's step** is an idle state whose `meta.interaction` tells a host what to render: `{ label: "Approve a refund of ${amount}?", events: { APPROVE: { label: "Approve" }, REJECT: { label: "Reject" } } }`, or `textEvent: { type: "CUSTOMER_REPLIED", field: "reply" }` for free text.
+- **Actors are placeholders.** Declare each tool as `createAsyncLogic<Output, Input>({ run })` from `xstate` in `setupAgent({ actors })` (XState v6 has no `fromPromise`). Give the type parameters, or `onDone`'s `output` is untyped. Pass the real implementations at run time with `runAgent(machine, { actors })`.
+- **Final outputs carry the machine's result** (`{ outcome, orderId, amount }`), JSON only. The runner rebuilds the old return type from it and from the world (step 5).
+- **Declare `version`** on `createMachine` (`version: "1"`). Behavior lives in transition functions, which the structural hash cannot see; traces stamp this string.
+
+## 4. A policy, only if the code already has one
+
+Keep the decision in the environment as one `agent.decide`, and do not write a policy machine, unless the old code already has decision logic beyond one model call: a rule ladder, heuristics that pick a strategy, a cheap check before the model. A policy that only passes the options through to a model is boilerplate. `improve-machine` introduces a policy when the traces show a rule worth adding.
+
+If the code does have that logic, port it as a policy machine that answers `chooseResolution`. The pattern, with code that compiles, is in [policy.md](policy.md).
+
+## 5. The runner
+
+Write `runMachine` with the old function's signature and result type, so callers and the parity test change one line.
+
+- `runAgent(machine, { input, executors, actors, onIdle })`.
+- `onIdle` answers each idle state with the old code's call (`tools.askCustomer(...)`, `tools.requestApproval(...)`). Write the returned events with `as const` on `type`, or the event union does not narrow.
+- `executors.decide` adapts the code's model to the decision: the closed choice arrives as `request.options` (each `{ event }`), and the facts as `request.metadata?.facts`, typed `unknown`: parse it with the exported facts schema. Every executor slot is optional in the type; check or assert the ones you call.
+- Rebuild the old result from `result.output` and the world. A field that has no meaning anymore (a turn count with no loop) keeps its type and gets a comment saying what it now holds.
+
+## 6. Proofs that need no model
+
+From `@statelyai/agent/testing`:
+
+- `lintAgentMachine(machine)`: no errors.
+- `canReach(machine, "<sentinel>", options)` returns `reachable: false` **and** `complete: true`, for each distinct input shape (an input with an order id and one without lead to different paths; loop over them).
+- `options.payloads` lists values on both sides of every guard boundary (`0, 40, 41, 100, 101, 240, 241, 1000`). Without it, a closed choice is explored only with its own `payloads`, and `complete: true` then proves nothing about the boundaries.
+- `options.outputs` lists every output each actor can return (found and not found, owned and not, refunded and not); `options.errors` makes each actor fail once.
+- A loosened guard makes `canReach` true. Write this test too, or the proof proves nothing.
+- Each normal final state is reachable, so the search is not vacuous.
+
+Type the options with `ExplorePathsOptions` from `@statelyai/agent/testing`. `errors` takes one error per actor src, and `"agent.decide"` is a valid key.
+
+## 7. Parity
+
+Switch the parity test's candidate to `runMachine`, and run it with each stand-in model:
+
+```ts
+test.each(MODELS)("parity with the loop: %s", async (_, model) => {
+  const report = await compareRuns({ scenarios: SCENARIOS, baseline: runWith(runLoop, model), candidate: runWith(runMachine, model) });
+  const unreviewed = report.scenarios.filter((row) => row.label === "regression" && !REVIEWED[row.id]);
+  expect(unreviewed).toEqual([]);
+  // A reviewed difference still has to be exactly the difference a person accepted.
+  for (const row of report.scenarios.filter((row) => REVIEWED[row.id])) REVIEWED[row.id]!(row);
+});
+
+// Effect-only differences a person accepted: who, and the exact difference.
+const REVIEWED: Record<string, (row: ParityResult<Scenario>) => void> = {
+  // "no-order-id-no-reply": (row) => { /* accepted by <name>: asks once, not three times */ expect(row.candidate.outcome).toBe(row.baseline.outcome); ... },
+};
+```
+
+- Every `regression` row: fix the machine, or, when the old behavior was wrong but broke no rule, ask the user. If the outcomes differ, the answer may become the scenario's `expected`. If only effects differ (the loop asked three times, the machine once), list the row in a reviewed-differences list in the test, with the exact difference asserted and the person who accepted it. Never edit a scenario to make a row pass.
+- Every `intended` row: check that its reason is a real rule. A candidate that fixed a rule break with the same outcome is `intended` too.
+- Run once with a live model if a key is available. Report it as one sample, not a guarantee.
+
+## 8. Hand back
+
+Report, in this order:
+
+1. The files: environment, runner, tests (and a policy, if the code had one). The old code is untouched.
+2. The rule list from step 1, each mapped to its guard, its state, or the reason it was dropped.
+3. The parity table per stand-in model: `same / intended / regression`, and one line per `intended` row with its reason.
+4. The `canReach` results and the loosened-guard test.
+5. Anything you had to decide that the user should confirm.
+6. Next step: `add-traces`.

package/skills/loop-to-machine/policy.md

@@ -0,0 +1,158 @@
+# A policy machine
+
+A policy answers one decision of the environment. It is a `setupAgent` machine, so `describeMachine`, traces, and evals work on it, and `improve-machine` can change it one diff at a time.
+
+- **Input:** the facts it reads, the options the environment offered, and the environment's prompt. **Output:** one event, from a final state.
+- **It proposes; the environment disposes.** The environment's guards still judge whatever the policy returns, so the policy needs no copy of the rules to be safe. It may import the environment's schemas, types, constants, and exported guard functions to decide well, and nothing else from it.
+- **Its first version reproduces the current behavior.** When it still asks the model, it passes the environment's `system`, `prompt`, and `metadata.facts` through unchanged, so the model sees what it saw before.
+- **Declare `version`** and bump it on every change. Its `description` is the environment's goal with measurable success criteria; ask the user to confirm it.
+- **Where it lives:** one file next to the environment (`policy.ts`, or `policy/` with its `CHANGELOG.md`), holding the machine, `policyDecider`, and `withPolicy`.
+
+## What the environment exports
+
+The event schemas and the facts schema, so the policy reuses them instead of redeclaring them. The `agent.decide` input passes the facts as `metadata: { facts }`.
+
+```ts
+// In the environment file (exported there):
+export const factsSchema = z.object({
+  order: z.object({ orderId: z.string(), amount: z.number(), refunded: z.boolean() }).nullable(),
+  requestedAmount: z.number().nullable(),
+});
+export const deskEvents = {
+  REFUND: z.object({ amount: z.number() }),
+  ESCALATE: z.object({}),
+  CLOSE: z.object({}),
+};
+```
+
+## The policy
+
+```ts
+import { z } from "zod";
+import { runAgent, setupAgent, type AgentDecisionExecutor, type AgentRequestExecutors } from "@statelyai/agent";
+import { deskEvents, factsSchema } from "./desk.js";
+
+/** One proposal: an environment event with its `type`. */
+const proposalSchema = z.discriminatedUnion("type", [
+  deskEvents.REFUND.extend({ type: z.literal("REFUND") }),
+  deskEvents.ESCALATE.extend({ type: z.literal("ESCALATE") }),
+  deskEvents.CLOSE.extend({ type: z.literal("CLOSE") }),
+]);
+type Proposal = z.infer<typeof proposalSchema>;
+
+const policySetup = setupAgent({
+  input: z.object({
+    facts: factsSchema,
+    options: z.array(proposalSchema),
+    // The environment's own prompt, passed through so the model sees what it saw before.
+    system: z.string().optional(),
+    prompt: z.string().optional(),
+  }),
+  context: z.object({
+    facts: factsSchema,
+    options: z.array(proposalSchema),
+    system: z.string().optional(),
+    prompt: z.string().optional(),
+    chosen: proposalSchema.nullable(),
+  }),
+  output: proposalSchema,
+  // Only the events the policy can propose, not every environment event.
+  events: { REFUND: deskEvents.REFUND, ESCALATE: deskEvents.ESCALATE, CLOSE: deskEvents.CLOSE },
+});
+
+/** The offered options of one type, as `payloads` entries (`type` removed). */
+const payloadsOf = (options: readonly Proposal[], type: Proposal["type"]) =>
+  options.filter((option) => option.type === type).map(({ type: _type, ...payload }) => payload);
+
+const offers = (options: readonly Proposal[], type: Proposal["type"]) =>
+  options.some((option) => option.type === type);
+
+export const policyMachine = policySetup.createMachine({
+  id: "resolution-policy",
+  version: "1",
+  description: "Pick one offered resolution per ticket. Success: the desk's outcome matches the expected outcome.",
+  context: ({ input }) => ({ ...input, chosen: null }),
+  initial: "checking",
+  states: {
+    // A rule ported from the old code: an already-refunded order closes, no model call.
+    checking: {
+      type: "choice",
+      choice: ({ context }) =>
+        context.facts.order?.refunded && offers(context.options, "CLOSE")
+          ? { target: "chosen", context: { chosen: { type: "CLOSE" as const } } }
+          : { target: "asking" },
+    },
+    asking: {
+      invoke: {
+        src: "agent.decide",
+        input: ({ context }) => ({
+          model: "decider",
+          name: "pickResolution",
+          system: context.system,
+          prompt: context.prompt ?? `Facts: ${JSON.stringify(context.facts)}`,
+          allowedEvents: ["REFUND", "ESCALATE", "CLOSE"],
+          // Only what the environment offered; `[]` withholds a type.
+          payloads: {
+            REFUND: payloadsOf(context.options, "REFUND"),
+            ESCALATE: payloadsOf(context.options, "ESCALATE"),
+            CLOSE: payloadsOf(context.options, "CLOSE"),
+          },
+          // Same shape as the environment's request, so the model adapter reads the facts unchanged.
+          metadata: { facts: context.facts },
+        }),
+        onError: { target: "gaveUp" },
+      },
+      on: {
+        REFUND: ({ event }) => ({
+          target: "chosen",
+          context: { chosen: { type: "REFUND" as const, amount: event.amount } },
+        }),
+        ESCALATE: { target: "chosen", context: { chosen: { type: "ESCALATE" as const } } },
+        CLOSE: { target: "chosen", context: { chosen: { type: "CLOSE" as const } } },
+      },
+    },
+    chosen: { type: "final", output: ({ context }) => context.chosen! },
+    gaveUp: { type: "final", output: { type: "ESCALATE" as const } },
+  },
+});
+
+/** Runs the policy as the environment's decider. */
+export function policyDecider(executors: AgentRequestExecutors): AgentDecisionExecutor {
+  return async (request, info) => {
+    const result = await runAgent(policyMachine, {
+      input: {
+        facts: factsSchema.parse(request.metadata?.facts),
+        options: (request.options ?? []).map((option) => proposalSchema.parse(option.event)),
+        system: request.system,
+        prompt: request.prompt,
+      },
+      executors,
+      signal: info?.signal,
+    });
+    if (result.status !== "done") throw new Error(`policy settled '${result.status}'`);
+    return { event: result.output };
+  };
+}
+```
+
+- Every state that invokes `agent.decide` needs an `onError`, or `lintAgentMachine` warns. A policy that cannot choose falls back to the safest event the environment always accepts (here `ESCALATE`).
+- A rule the policy applies without a model is a `choice` state over the facts, checked against what was offered (`offers`), since the environment may have withheld it.
+
+## Running it as the environment's decider
+
+```ts
+/** The model's executors, with the policy answering the environment's `chooseResolution`. */
+export function withPolicy(modelExecutors: AgentRequestExecutors): AgentRequestExecutors {
+  const decideWithPolicy = policyDecider(modelExecutors);
+  return {
+    ...modelExecutors,
+    decide: (request, info) => {
+      if (request.name === "chooseResolution") return decideWithPolicy(request, info);
+      if (!modelExecutors.decide) throw new Error("no decide executor");
+      return modelExecutors.decide(request, info);
+    },
+  };
+}
+```
+
+Pass `withPolicy(modelExecutors)` as the environment's `executors` in the runner. That one line is the only runner change a policy needs. Decisions with any other name still reach the model.