@statelyai/agent 2.0.0-alpha.34 → 2.0.0-alpha.36
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +124 -0
- package/dist/act-B9hLQls6.mjs +19 -0
- package/dist/act-CrX0JwWY.cjs +42 -0
- package/dist/ai-sdk.cjs +63 -7
- package/dist/ai-sdk.d.cts +13 -3
- package/dist/ai-sdk.d.mts +13 -3
- package/dist/ai-sdk.mjs +63 -7
- package/dist/{decision-BEsATcr1.cjs → decision-CMeuNZ1g.cjs} +262 -4
- package/dist/{decision-COkJNjOW.mjs → decision-m5hGmHpF.mjs} +239 -5
- package/dist/decision-object-B5zGSPFp.d.mts +15 -0
- package/dist/decision-object-DBcLWVvv.d.cts +15 -0
- package/dist/{run-agent-s5D32w99.cjs → effects-B23bFZEs.cjs} +948 -102
- package/dist/{run-agent-Ct_GZw_f.mjs → effects-CuOUNYtZ.mjs} +914 -104
- package/dist/{event-log-C7jeH0y8.mjs → event-log-DkVlk3Rh.mjs} +30 -15
- package/dist/{event-log-BWSSLUff.cjs → event-log-Dkc-yIYu.cjs} +41 -14
- package/dist/{event-log-store-BzXehP7R.d.cts → event-log-store-BBS8so8a.d.mts} +2 -2
- package/dist/{event-log-store-W49TCJXi.mjs → event-log-store-Brx9tOHt.mjs} +1 -1
- package/dist/{event-log-store-CleWpsp5.cjs → event-log-store-Cl-YWDnq.cjs} +1 -1
- package/dist/{event-log-store-CWjN6lmM.d.mts → event-log-store-bc4ICGcS.d.cts} +2 -2
- package/dist/index.cjs +660 -286
- package/dist/index.d.cts +168 -9
- package/dist/index.d.mts +168 -9
- package/dist/index.mjs +639 -271
- package/dist/internal-C3Lqu538.d.cts +43 -0
- package/dist/internal-DUcdhk-i.d.mts +43 -0
- package/dist/log.cjs +2 -2
- package/dist/log.d.cts +1 -1
- package/dist/log.d.mts +1 -1
- package/dist/log.mjs +2 -2
- package/dist/machines.cjs +457 -41
- package/dist/machines.d.cts +114 -43
- package/dist/machines.d.mts +114 -43
- package/dist/machines.mjs +457 -42
- package/dist/{model-source-DmhTMK39.cjs → model-source-B4WQ8Idp.cjs} +1 -1
- package/dist/{model-source-CK3F4YuU.mjs → model-source-kjGwqAV8.mjs} +1 -1
- package/dist/openai.cjs +149 -36
- package/dist/openai.d.cts +28 -10
- package/dist/openai.d.mts +28 -10
- package/dist/openai.mjs +149 -36
- package/dist/otel.d.cts +1 -1
- package/dist/otel.d.mts +1 -1
- package/dist/{run-agent-BGdx5913.d.mts → run-agent-BcQ-9NRT.d.cts} +91 -9
- package/dist/{run-agent-B1-Eqviq.d.cts → run-agent-C86drZtN.d.mts} +91 -9
- package/dist/{setup-agent-05uw4dPZ.mjs → setup-agent-BDmNr6PC.mjs} +9 -3
- package/dist/{setup-agent-BtfeFfF6.d.cts → setup-agent-BIKcbVXV.d.mts} +4 -4
- package/dist/{setup-agent-CXEBURdo.cjs → setup-agent-CmEToB8y.cjs} +9 -3
- package/dist/{setup-agent-C6nqzv_X.d.mts → setup-agent-jlo4vtaZ.d.cts} +4 -4
- package/dist/tanstack-ai.cjs +62 -4
- package/dist/tanstack-ai.d.cts +18 -6
- package/dist/tanstack-ai.d.mts +18 -6
- package/dist/tanstack-ai.mjs +62 -4
- package/dist/testing.cjs +536 -67
- package/dist/testing.d.cts +197 -180
- package/dist/testing.d.mts +197 -180
- package/dist/testing.mjs +532 -67
- package/dist/{text-logic-CV_4KGr6.d.cts → text-logic-BqxalgMr.d.mts} +87 -4
- package/dist/{text-logic-Ckh-rJFV.d.mts → text-logic-CSf9bqpP.d.cts} +87 -4
- package/dist/{types-Dp5_Ru4R.d.cts → types-DQUOOaZO.d.cts} +1 -1
- package/dist/{types-Dp5_Ru4R.d.mts → types-DQUOOaZO.d.mts} +1 -1
- package/dist/{usage-BDiJKyv9.d.cts → usage-DoP0Oi84.d.mts} +1 -1
- package/dist/{usage-DOoVUGVq.d.mts → usage-HtnrUsl9.d.cts} +1 -1
- package/dist/utils-7lSa4H-i.d.cts +403 -0
- package/dist/utils-CBbKw2fM.d.mts +403 -0
- package/dist/validate.cjs +44 -12
- package/dist/validate.d.cts +6 -6
- package/dist/validate.d.mts +6 -6
- package/dist/validate.mjs +44 -12
- package/dist/verify-Bink-Drw.cjs +2304 -0
- package/dist/{verify-BQpE8hUz.d.cts → verify-Buh8XSUr.d.mts} +208 -14
- package/dist/{verify-CqeuEmIk.d.mts → verify-CciG8UP-.d.cts} +208 -14
- package/dist/verify-Dpz1u5Pe.mjs +2251 -0
- package/dist/with-model-calls-B_WLmOSl.cjs +248 -0
- package/dist/with-model-calls-BkOwdroS.mjs +213 -0
- package/docs/act.md +102 -0
- package/docs/advanced.md +105 -0
- package/docs/any-stack.md +62 -0
- package/docs/choosing-a-run-mode.md +55 -0
- package/docs/debugging.md +29 -0
- package/docs/decisions.md +473 -0
- package/docs/environment-and-policy.md +188 -0
- package/docs/evals.md +509 -0
- package/docs/event-log.md +311 -0
- package/docs/from-a-loop.md +528 -0
- package/docs/generate-machines.md +110 -0
- package/docs/hosts.md +414 -0
- package/docs/human-in-the-loop.md +177 -0
- package/docs/index.md +48 -0
- package/docs/judgments.md +281 -0
- package/docs/langgraph-comparison.md +198 -0
- package/docs/machines-as-data.md +454 -0
- package/docs/machines-presets.md +198 -0
- package/docs/machines.md +497 -0
- package/docs/messages.md +67 -0
- package/docs/meta.json +52 -0
- package/docs/models-and-providers.md +268 -0
- package/docs/multi-agent.md +68 -0
- package/docs/observability.md +109 -0
- package/docs/patterns.md +31 -0
- package/docs/persistence.md +136 -0
- package/docs/quickstart.md +121 -0
- package/docs/roadmap.md +22 -0
- package/docs/scope.md +75 -0
- package/docs/statechart-policy-examples.md +69 -0
- package/docs/steps.md +54 -0
- package/docs/strangler-fig.md +323 -0
- package/docs/text-requests.md +250 -0
- package/docs/thinking-in-state-machines.md +324 -0
- package/docs/tools.md +249 -0
- package/docs/upgrading.md +37 -0
- package/docs/usage-and-budgets.md +516 -0
- package/docs/validate-and-repair.md +253 -0
- package/docs/verify.md +487 -0
- package/docs/xstate-4-and-5-apps.md +161 -0
- package/llms.txt +53 -0
- package/package.json +10 -3
- package/readme.md +30 -15
- package/skills/add-evals/SKILL.md +3 -1
- package/skills/add-traces/SKILL.md +6 -1
- package/skills/build-machine/SKILL.md +81 -0
- package/skills/generate-machine/SKILL.md +1 -1
- package/skills/graph-engineering/SKILL.md +13 -3
- package/skills/improve-machine/SKILL.md +21 -2
- package/skills/loop-to-machine/SKILL.md +147 -190
- package/skills/loop-to-machine/reference.md +260 -0
- package/dist/utils-BnXdWih3.d.mts +0 -108
- package/dist/utils-ebVpCYoF.d.cts +0 -108
- package/dist/verify-Bpahmr34.mjs +0 -1241
- package/dist/verify-DCLZa9R0.cjs +0 -1288
- package/dist/with-model-calls-BXYdBdcu.cjs +0 -45
- package/dist/with-model-calls-Bv4V4EHC.mjs +0 -34
|
@@ -1,248 +1,205 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: loop-to-machine
|
|
3
|
-
description: Convert existing agent code (a while-loop around a model, a tool loop, ad-hoc if/else control flow) into an XState v6 state machine with @statelyai/agent, and prove with a parity report that it behaves like the code it replaces. Use when asked to "turn this agent into a state machine", "port my loop to xstate", "make this a statechart", "refactor this agent into a machine", or "split environment and policy" for code that already runs.
|
|
3
|
+
description: Convert existing agent code (a while-loop around a model, a tool loop, a tool-call scheduler, ad-hoc if/else control flow) into an XState v6 state machine with @statelyai/agent, and prove with a parity report that it behaves like the code it replaces. Use when asked to "turn this agent into a state machine", "port my loop to xstate", "make this a statechart", "refactor this agent into a machine", or "split environment and policy" for code that already runs.
|
|
4
4
|
---
|
|
5
5
|
|
|
6
6
|
# Convert agent code to a machine
|
|
7
7
|
|
|
8
|
-
You replace the control flow of working agent code with a machine, and show where
|
|
8
|
+
You replace the control flow of working agent code with a machine, and show where it behaves like the old code and where it differs on purpose. Model calls, tools, and retry code move across unchanged.
|
|
9
9
|
|
|
10
10
|
```
|
|
11
11
|
inventory → pin the baseline → environment → (policy) → runner → proofs → parity → hand back
|
|
12
12
|
```
|
|
13
13
|
|
|
14
|
-
|
|
14
|
+
## Default track: wrap first, carve per commit
|
|
15
15
|
|
|
16
|
-
|
|
16
|
+
For an async workflow (not a hand-rolled loop you are asked to replace wholesale), start here, and use the sections below for each carved phase.
|
|
17
17
|
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
- **Never copy a rule break.** Where the old code broke a rule, the machine blocks it. That difference is `intended`, not a regression.
|
|
21
|
-
- **Rules are code, not prompt text.** A rule that lived only in a system prompt becomes a guard. Keep the prompt line only as context for the model.
|
|
22
|
-
- **Business rules belong to people.** If a rule is ambiguous (which threshold, which order of checks), ask. Never pick one silently.
|
|
18
|
+
1. **Step 0: wrap and record.** Move every side effect behind one `effects` object of async functions (the only edit to the old code). Then `wrapWorkflow({ id, effects, run: (input, effects) => old(input, effects) })` from `@statelyai/agent`. Pin the baseline with `checkParity` (`@statelyai/agent/testing`): `{ scenarios, effects: (s) => fakes, before: old, after: (input, fx) => step0.run(input, { effects: fx }) }`; every row `equivalent`. Ship it: inspection, traces, the event log, `signal`, a recorded baseline.
|
|
19
|
+
2. **Carve one phase per commit.** One phase becomes states (§3); the rest stays the old helper, invoked as a `createAsyncLogic` actor built per run from `effects`. Keep the `(input, effects)` runner signature with a third `options` argument passed through to `runAgent` (`threadId`, `store`, `metadata`, `onTrace`, `inspect`), so traces and inspection never need a copy of the runner. Move any prompt or warning text the carved phase sends into a versioned request config as you carve it: that file is what `improve-machine` may later edit. Run `checkParity` against the old code after each carve. Add the §6 proofs for the carved phase only.
|
|
23
20
|
|
|
24
|
-
|
|
21
|
+
A wrapped workflow is a black box: one state, no mid-function resume, no `canReach` about its inside. Do not report proofs before the phase is carved. Worked example: `examples/strangler-fig`. A class whose loop calls its own private model/tool methods: replace those methods on the instance inside `run` so they call `effects` (no edit to the class). Effect-TS generators, Redux thunks, LangGraph graphs, closures in DI factories, and step loops inside the model SDK have their own seams: [docs/strangler-fig.md](../../docs/strangler-fig.md#seams-for-other-shapes). Effects bound to one call's live objects: create the workflow once, pass `run(input, { effects })` per call. Scrub generated ids and timestamps: `compare.normalize` for effect arguments, `normalizeOutput` for the output, `normalizeError` for thrown errors.
|
|
25
22
|
|
|
26
|
-
|
|
23
|
+
**A phase in the middle of a private method:** copy only that phase's lines into the carved actor, list the copied line ranges in a comment, and let `checkParity` call-args catch drift. When a public per-turn hook exists, replace that method on the instance instead and let the machine decide at that boundary ([reference.md](reference.md#a-phase-in-the-middle-of-a-method)).
|
|
27
24
|
|
|
28
|
-
|
|
29
|
-
| ---------------------------------------------------------------------------- | ------------------------------------------------------------------------ |
|
|
30
|
-
| A tool with a side effect someone must gate (refund, send) | An actor invoked by one state, entered only through a guarded transition |
|
|
31
|
-
| A read-only tool (search, lookup) | An actor the machine invokes itself, or a tool on a text request |
|
|
32
|
-
| A rule: an `if`, a prompt line, a check after the fact | A guard on the transition it protects |
|
|
33
|
-
| A model call that produces content (a draft, generated code, a summary) | A text request in `setupAgent({ requests })`, invoked by one state |
|
|
34
|
-
| One SDK tool call that both picks the action and carries its content (`edit_graph({ … })`, `send_email({ … })`) | A text request with those tools, then guards on the returned call: exactly one, a tool that was offered, input that passes the tool's schema. Keep it a text request: `agent.decide` would rename the tools to `send_event_<TYPE>` with the description "Choose the '<TYPE>' move.", so the model would no longer see your tool names and descriptions |
|
|
35
|
-
| A tool the browser runs (no `execute`; the client sends results back) | The request ends with the tool calls; an idle state waits for a `TOOL_RESULTS` event the route sends when the client posts results |
|
|
36
|
-
| A point where the model picks what to do | A decision state: `agent.decide`, usually a closed choice |
|
|
37
|
-
| A check on the model's output, then a retry or a repair call | An actor or guard that checks it, and a `choice` state that routes to the repair request, the next candidate, or a failure final |
|
|
38
|
-
| A value the model pulled out of the input (an order id, an amount) | Code when the format is fixed (a regex); a text request when it is not |
|
|
39
|
-
| A step every correct run takes in the same order (verify, then look up) | A state the machine enters itself; the model no longer chooses it |
|
|
40
|
-
| A person in the loop: `await approve()`, a returned `pending`, or a tool that asks a person (`askCustomer`, `requestApproval`), even a synchronous one | An idle state with `meta.interaction`, answered by `onIdle` in the runner |
|
|
41
|
-
| A cap (`turns < 8`, a retry counter) on a loop the machine keeps | A `choice` state against an exported constant |
|
|
42
|
-
| A cap on a loop the machine removed | Nothing. Record that it is gone and why |
|
|
43
|
-
| Each distinct return | A final state |
|
|
44
|
-
| A terminal tool with a side effect (`escalate`, `close`) | A state that invokes it, then the final state; its `onError` goes to a `failed` final |
|
|
45
|
-
|
|
46
|
-
List the rules as a numbered list in your notes. Every one must end as a guard, a state ordering, or a written reason it was dropped.
|
|
47
|
-
|
|
48
|
-
A loop with no decision point (generate, validate, repair) is still worth converting: the machine makes the retry cap, the validation rules, and every failure path explicit and provable. It has no `agent.decide`, so step 4 and the decision parts of step 5 do not apply.
|
|
25
|
+
This is step 1 of 4. Next: `add-traces`, then `add-evals`, then `improve-machine`. To author a machine with no existing code, use `generate-machine`. Recording, replay, and the full parity test code are in [reference.md](reference.md).
|
|
49
26
|
|
|
50
|
-
##
|
|
27
|
+
## Hard rules
|
|
51
28
|
|
|
52
|
-
The parity test
|
|
29
|
+
- **Pin the baseline before you change anything.** The parity test runs against the old code, which stays in the repo, untouched, until a person deletes it. When the old code cannot run, the baseline is a written behavioral spec (see "No model in the loop").
|
|
30
|
+
- **Never promise "behaves exactly like".** Promise what the parity evidence shows, every difference labeled.
|
|
31
|
+
- **Never copy a rule break.** Where the old code broke a rule, the machine blocks it. That difference is `intended`, not a regression.
|
|
32
|
+
- **Rules are code, not prompt text.** A prompt-only rule becomes a guard; the prompt line stays only as context.
|
|
33
|
+
- **Business rules belong to people.** Ambiguous rule (which threshold, which order)? Ask; never pick silently.
|
|
34
|
+
- **Port call shapes unchanged first.** An SDK tool call stays a text request with the same tools. Changing it (to `agent.decide` or `agent.act`) is a separate step with its own `INTENDED` divergences.
|
|
53
35
|
|
|
54
|
-
|
|
55
|
-
2. **Deterministic stand-in models.** Parity with a live model drifts between runs. Use the code's scripted or mock models if it has them; otherwise write one: a function from the request (its prompt, or its `name`) to a fixed answer. ([`examples/mock-model.ts`](https://github.com/statelyai/agent/blob/next/examples/mock-model.ts) in the `statelyai/agent` repo is one, built on the AI SDK's `MockLanguageModelV3`; it is not in the package.) You want two: one that follows the rules, and one that breaks them. A seeded random model needs its seed derived from the scenario id and a fresh instance per run, or every scenario replays the same draws.
|
|
56
|
-
3. **Ground truth.** Read outcomes and effects from the world, not from the agent's own report. `effects` holds **every write the world records**: the refunds, and also the approvals, the verifications, the questions asked, the escalations. A refund issued without its approval looks correct if `effects` holds only refunds. Text a model wrote (an escalation reason) differs on every run: compare its count or presence, not the text. If the world can check rules after the fact, return those violations. If it cannot, write the check as a plain function over the effects.
|
|
57
|
-
4. **A baseline with no function to call.** When the loop lives inline in a framework handler (a tRPC procedure, a route), call the handler itself in the test, with its model client and storage replaced by the stand-ins (a module mock, or the framework's test caller). Do not extract the loop into a function first: that changes the code you are trying to pin.
|
|
58
|
-
5. **Record golden runs.** When the code calls the model through the AI SDK or the `openai` client, record each scenario's effects: every model call, every tool call with its arguments and result, and the output. Wrap the model and tools with a recorder from `@statelyai/agent/testing`, run the old code, and save the recording as a JSON fixture. Step 7 replays it against the machine with no API key and no side effects.
|
|
36
|
+
## 0. Before you write code
|
|
59
37
|
|
|
60
|
-
|
|
61
|
-
|
|
38
|
+
- Text requests need exactly one non-empty `prompt` or `messages`. `setupAgent` throws `invalid-request` when both are omitted; resolving neither or both at run time fails validation.
|
|
39
|
+
- Actors: `createAsyncLogic<Output, Input>({ run })` from `xstate` (no `fromPromise` in v6); without the generics `onDone`'s `output` is untyped.
|
|
40
|
+
- `onIdle` return events need `as const` on `type`, or the union does not narrow.
|
|
41
|
+
- In `onError`, the error is `event.error`; there is no `error` argument.
|
|
42
|
+
- Context, input, events, outputs: JSON only. No `Date`, `Error`, `Map`, class instances, functions; store ISO strings and `String(error)`.
|
|
43
|
+
- Every executor slot is optional in the type; assert the ones you call.
|
|
44
|
+
- `request.metadata.facts` is `unknown`; parse it with the exported facts schema.
|
|
45
|
+
- A `type: "choice"` state has no `enq` or actions; when routing must emit, use `always` on a normal state.
|
|
46
|
+
- `compareRuns` scenarios reserve `expected: string` for the outcome; name other expectations differently.
|
|
47
|
+
- `explorePaths` `errors[src]`: an array forks one failing branch per element (`[[...]]` for an array-valued failure). `outputs` and `errors` branches are alternatives, not a sequence: drive "fails twice, then succeeds" from a context counter.
|
|
62
48
|
|
|
63
|
-
|
|
64
|
-
const model = recorder.model(openai("gpt-5")); // or recorder.openai(new OpenAI())
|
|
65
|
-
const tools = recorder.tools(appTools); // AI SDK tools; recorder.tool(name, fn) for plain functions
|
|
66
|
-
recorder.output(await runLoop({ ...scenario, model, tools }));
|
|
67
|
-
writeFileSync(`fixtures/${scenario.id}.json`, JSON.stringify(recorder.recording));
|
|
68
|
-
```
|
|
49
|
+
### Inspect one event
|
|
69
50
|
|
|
70
|
-
|
|
71
|
-
6. **The harness check.** Run the old code on both sides with each stand-in model. Each row's baseline must equal its candidate (the runs are deterministic), and with the rule-following model every row must be `same`. With a rule-breaking model, rows are `regression` by design: the candidate breaks rules.
|
|
51
|
+
`transition(machine, snapshot, event)` from `xstate` is pure: it returns the next snapshot and the effects it would run, and runs none of them. Use it to answer "what happens if this event arrives here?" before writing a test or running the loop.
|
|
72
52
|
|
|
73
53
|
```ts
|
|
74
|
-
import {
|
|
75
|
-
|
|
76
|
-
|
|
77
|
-
|
|
78
|
-
|
|
79
|
-
|
|
80
|
-
|
|
81
|
-
|
|
82
|
-
};
|
|
83
|
-
}
|
|
84
|
-
|
|
85
|
-
// A model factory, called once per scenario and per side. Construct a stateful or seeded
|
|
86
|
-
// model inside it (seeded from s.id); returning a shared stateless object is fine.
|
|
87
|
-
const runWith = (runner: typeof runLoop, model: (s: Scenario) => Model) => async (s: Scenario) =>
|
|
88
|
-
toParityRun(await runner({ ...s, model: model(s), world: createWorld(s) }));
|
|
89
|
-
|
|
90
|
-
const MODELS: Array<[string, (s: Scenario) => Model]> = [
|
|
91
|
-
["careful", () => carefulModel],
|
|
92
|
-
["sloppy", () => sloppyModel],
|
|
93
|
-
];
|
|
94
|
-
|
|
95
|
-
test.each(MODELS)("harness: the loop matches itself (%s)", async (_, model) => {
|
|
96
|
-
const report = await compareRuns({ scenarios: SCENARIOS, baseline: runWith(runLoop, model), candidate: runWith(runLoop, model) });
|
|
97
|
-
for (const row of report.scenarios) expect(row.candidate, row.id).toEqual(row.baseline);
|
|
54
|
+
import { initialTransition, transition } from "xstate";
|
|
55
|
+
|
|
56
|
+
const [s0] = initialTransition(machine, { maxTurns: 2 });
|
|
57
|
+
const actorId = Object.keys(s0.children)[0]; // the running invoke
|
|
58
|
+
const [s1, effects] = transition(machine, s0, {
|
|
59
|
+
type: "xstate.done.actor",
|
|
60
|
+
actorId,
|
|
61
|
+
output: { done: false },
|
|
98
62
|
});
|
|
63
|
+
s1.value; // next state
|
|
64
|
+
s1.context; // next context
|
|
65
|
+
effects; // what would run (spawn, start, stop, ...); nothing ran
|
|
99
66
|
```
|
|
100
67
|
|
|
101
|
-
|
|
68
|
+
- Same snapshot object back and no effects: the event is unhandled there.
|
|
69
|
+
- Fail an invoke with `{ type: "xstate.error.actor", actorId, error }`.
|
|
70
|
+
- To start mid-run, pass a live snapshot: `restoreActor(machine, persisted).getSnapshot()` (`xstate/persist`) for a persisted one, or the snapshot `replay` (`@statelyai/agent/log`) folds from a journal. A raw persisted snapshot throws.
|
|
102
71
|
|
|
103
|
-
|
|
72
|
+
## 1. Inventory the code
|
|
104
73
|
|
|
105
|
-
|
|
74
|
+
Read the loop and fill this table before writing anything. Each row maps to a machine element.
|
|
106
75
|
|
|
107
|
-
|
|
108
|
-
|
|
109
|
-
|
|
110
|
-
|
|
111
|
-
|
|
112
|
-
|
|
113
|
-
|
|
114
|
-
|
|
115
|
-
|
|
116
|
-
|
|
117
|
-
|
|
118
|
-
|
|
119
|
-
|
|
120
|
-
|
|
121
|
-
|
|
122
|
-
|
|
123
|
-
|
|
124
|
-
|
|
125
|
-
|
|
126
|
-
|
|
127
|
-
|
|
128
|
-
|
|
129
|
-
|
|
130
|
-
|
|
131
|
-
|
|
132
|
-
|
|
133
|
-
|
|
134
|
-
|
|
76
|
+
| In the code | Becomes |
|
|
77
|
+
| --- | --- |
|
|
78
|
+
| A tool with a side effect someone must gate (refund, send) | An actor invoked by one state, entered only through a guarded transition |
|
|
79
|
+
| A read-only tool (search, lookup) | An actor the machine invokes, or a tool on a text request |
|
|
80
|
+
| A rule: an `if`, a prompt line, a check after the fact | A guard on the transition it protects |
|
|
81
|
+
| A model call that produces content | A text request in `setupAgent({ requests })`, invoked by one state |
|
|
82
|
+
| One SDK tool call that picks the action and carries its content (`send_email({ … })`) | A text request with those tools, then shape guards on the returned call: exactly one, offered, schema-valid. A tool that keeps `execute` runs inside the SDK (old timing); to gate it, drop `execute` and invoke the real work after the verdict. Not `agent.decide` yet: it hides your tool names and descriptions |
|
|
83
|
+
| N tool calls per model turn, each needing approval | First port: the request returns the calls; a state holds them in context and waits for a verdict per call. Later step: `agent.act` ([docs/act.md](../../docs/act.md)), each call an event the state queues |
|
|
84
|
+
| A persisted "always allow" rule | Data loaded into input or context, read by a guard. Saving a rule is a recorded effect |
|
|
85
|
+
| A tool the browser runs (no `execute`) | The request ends with the tool calls; an idle state waits for `TOOL_RESULTS` |
|
|
86
|
+
| A point where the model picks what to do | A decision state: `agent.decide`, usually a closed choice |
|
|
87
|
+
| A check on model output, then retry or repair | A checking actor or guard, and a `choice` state routing to repair, next candidate, or a failure final |
|
|
88
|
+
| Provider retry/backoff around the SDK call | Stays in the executor, unchanged |
|
|
89
|
+
| Context overflow caught → compact → retry | A compaction request state, entered from the call's `onError` on an overflow error, with a retry cap |
|
|
90
|
+
| Recursive re-entry (`this.run()` inside the loop) | A transition back to the first state, under the recursion's cap (or a new one, `intended`) |
|
|
91
|
+
| A value pulled out of the input | Code when the format is fixed; a text request when not |
|
|
92
|
+
| A step every correct run takes in order | A state the machine enters itself |
|
|
93
|
+
| A person in the loop (`await approve()`, a returned `pending`, an ask-a-person tool) | An idle state with `meta.interaction` and `onIdle`, or an invoked wait actor (§5) |
|
|
94
|
+
| `AbortSignal` or a cancel method | `runAgent`'s `signal`, or a `CANCEL` event (§5) |
|
|
95
|
+
| Hooks or bus emissions the UI depends on | `enq.emit(...)` (declared under `emitted`) or `onTransition`, same payloads |
|
|
96
|
+
| Message history in a class field | Context when a guard or prompt reads it; else host-owned, passed as `messages`. Write down which |
|
|
97
|
+
| A streaming iterator | The `streamText` executor keeps it and pipes to the old consumer. In a wrapped workflow, an effect returning it is observed: `effect.done` carries the chunks once read, a mid-stream throw is `effect.error` |
|
|
98
|
+
| A step loop inside the model SDK (`stopWhen`, `maxSteps`) | One executor or effect, unchanged. Carve the control flow around it: handoff, approval, caps |
|
|
99
|
+
| A synchronous host callback, a diagnostic logger | Not an effect: the machine factory closes over it |
|
|
100
|
+
| An effect result that is a live object, a write to host state by reference | A name, id, or change in the result, with a host-side registry. Parity cannot see the live object or the write |
|
|
101
|
+
| Mutable state on the class instance | JSON context fields, or host state passed as input |
|
|
102
|
+
| A cap on a loop the machine keeps / removed | A `choice` state against an exported constant / nothing, with the reason recorded |
|
|
103
|
+
| A class-shaped runtime (`run`/`continue`/`abort`/`restore`/`snapshot`) | A subclass overriding the loop methods to call `runAgent`, inheriting the rest. The conversation may stay host-owned at first (a snapshot then cannot resume alone); moving it into context is a later labeled step |
|
|
104
|
+
| A function that returns a live stream at once (`streamText`-shaped) | Return at once from a `TransformStream` the run writes into: an `emit` effect writes each chunk, the run closes the writer when it succeeds and aborts it with the error when it fails, and the result's promise fields resolve from the run. The machine owns the loop; the caller reads the stream as before |
|
|
105
|
+
| A live receiver the effects need (a class instance, an open stream, a cancellation token) | A host registry keyed by id: context and events carry the id, each effect looks the object up |
|
|
106
|
+
| Each distinct return | A final state |
|
|
107
|
+
| A terminal tool with a side effect (`escalate`) | A state that invokes it, then the final; `onError` to a `failed` final |
|
|
108
|
+
|
|
109
|
+
Number the rules in your notes. Each ends as a guard, a state ordering, or a written reason it was dropped.
|
|
110
|
+
|
|
111
|
+
A loop with no decision point is still worth converting: caps, rules, and failure paths become provable. Skip step 4 and the decision parts of step 5.
|
|
135
112
|
|
|
136
|
-
|
|
137
|
-
- **Export the guards** as plain functions (`refundIsLegal`) so tests can call them, and build the machine in a factory (`createDeskMachine({ refundIsLegal })`) so a test can pass a loosened guard.
|
|
138
|
-
- **Add an invariant sentinel** for the rule that must never break: a final state with a description starting `SENTINEL:`, entered from an `always` on the state that invokes the side effect (`paying`). An `always` is taken before the state's invoke starts, so the effect never runs. Write the check as its own function, not a call to the guard, so loosening the guard does not loosen the check.
|
|
139
|
-
- **A person's step** is an idle state whose `meta.interaction` tells a host what to render: `{ label: "Approve a refund of ${amount}?", events: { APPROVE: { label: "Approve" }, REJECT: { label: "Reject" } } }`, or `textEvent: { type: "CUSTOMER_REPLIED", field: "reply" }` for free text.
|
|
140
|
-
- **Actors are placeholders.** Declare each tool as `createAsyncLogic<Output, Input>({ run })` from `xstate` in `setupAgent({ actors })` (XState v6 has no `fromPromise`). Give the type parameters, or `onDone`'s `output` is untyped. Pass the real implementations at run time with `runAgent(machine, { actors })`.
|
|
141
|
-
- **Final outputs carry the machine's result** (`{ outcome, orderId, amount }`), JSON only. The runner rebuilds the old return type from it and from the world (step 5).
|
|
142
|
-
- **Declare `version`** on `createMachine` (`version: "1"`). Behavior lives in transition functions, which the structural hash cannot see; traces stamp this string.
|
|
113
|
+
## 2. Pin the baseline
|
|
143
114
|
|
|
144
|
-
|
|
115
|
+
The parity test exists before the machine does.
|
|
145
116
|
|
|
146
|
-
|
|
117
|
+
1. **Scenarios.** Reuse the code's tests, fixtures, or benchmarks: an `id` each and, where known, the `expected` outcome. If none exist, draft 6–10 covering every terminal and rule; ask the user to confirm them.
|
|
118
|
+
2. **Deterministic stand-in models.** The code's mocks, or a function from the request (prompt or `name`) to a fixed answer ([`examples/mock-model.ts`](https://github.com/statelyai/agent/blob/next/examples/mock-model.ts), repo only). Two: one follows the rules, one breaks them. A seeded model seeds from the scenario id, fresh per run.
|
|
119
|
+
3. **Ground truth.** Outcomes and effects from the world, not the agent's report. `effects` holds **every write the world records**: refunds, approvals, verifications, questions asked, escalations. Model-written text: compare count or presence. Violations come from the world, or a plain function over the effects.
|
|
120
|
+
4. **A baseline with no function to call.** Loop inline in a framework handler: call the handler with stand-in model and storage. Do not extract the loop first.
|
|
121
|
+
5. **Record golden runs.** `createRecorder` (`@statelyai/agent/testing`) wraps model and tools; run the old code per scenario and stand-in, save JSON fixtures. AI SDK model → `recorder.model`; `openai` client → `recorder.openai`; anything else → `recorder.client` on the SDK object (`methods: ["messages.create"]`), or `recorder.fn` on a function. Code, streams, and `redact`/`serialize`/`revive`: [reference.md](reference.md#any-other-model-client).
|
|
122
|
+
6. **The harness check.** The old code on both sides of `compareRuns`, per stand-in: baseline equals candidate in every row; all `same` with the rule-following model, `regression` by design with the rule-breaker. [reference.md](reference.md#the-parity-harness).
|
|
147
123
|
|
|
148
|
-
|
|
124
|
+
### No model in the loop
|
|
149
125
|
|
|
150
|
-
|
|
126
|
+
For code with no model call (a tool scheduler, an approval queue), and old code that cannot run (a class wired to a bus, no callable entry). A tool runner whose text request picks tools has a model in the loop: keep recorders and replay (§7); use the spec baseline only where the old code cannot run.
|
|
151
127
|
|
|
152
|
-
|
|
128
|
+
- **Skip recorders only when nothing calls a model.** Then replay (§7) does not apply.
|
|
129
|
+
- **Baseline as spec.** When the old code cannot run, `baseline: async (s) => SPEC[s.id]`: outcome, effects, and violations a person confirmed from the old code and its docs, before the machine exists. Skip the harness check; the hand-back says the baseline is a spec, not a run.
|
|
130
|
+
- **Stand-ins become scripted worlds:** the proposed calls, the order approvals arrive in, which tool fails.
|
|
153
131
|
|
|
154
|
-
|
|
155
|
-
- **A streaming route keeps its response, not its function signature.** `createAiSdkExecutors().streamText` forwards text chunks only, so a UI stream with tool calls cannot come out of it. Write the `streamText` executor around the old call, pipe its UI stream to the route's response yourself, and return the final messages as `{ result }`.
|
|
132
|
+
## 3. Author the environment
|
|
156
133
|
|
|
157
|
-
- `
|
|
158
|
-
- `onIdle` answers each idle state with the old code's call (`tools.askCustomer(...)`, `tools.requestApproval(...)`). Write the returned events with `as const` on `type`, or the event union does not narrow.
|
|
159
|
-
- `executors.decide` adapts the code's model to the decision: the closed choice arrives as `request.options` (each `{ event }`), and the facts as `request.metadata?.facts`, typed `unknown`: parse it with the exported facts schema. Every executor slot is optional in the type; check or assert the ones you call.
|
|
160
|
-
- In an `onError` function, the error is `event.error`: `onError: ({ event }) => ({ target: "failed", context: { error: String(event.error) } })`. There is no `error` argument.
|
|
161
|
-
- Make input and events JSON-safe at the boundary: no `Date`, `Map`, class instances, or functions (fields set to `undefined` are dropped). With a store, a value the log cannot hold fails the run; without one, the log stops with a warning.
|
|
162
|
-
- Rebuild the old result from `result.output` and the world. A field that has no meaning anymore (a turn count with no loop) keeps its type and gets a comment saying what it now holds.
|
|
134
|
+
The environment is the machine: states, guards that encode the rules, side-effecting actors, an invariant sentinel. Use `setupAgent` (from `@statelyai/agent`) with schemas for `input`, `context`, `output`, and every event. A zod `z.object` strips keys it does not list at the boundary; list every field the old code reads, or use `z.looseObject`. It takes any Standard Schema library (zod, valibot, arktype); the package ships none, so install one. Pass the schemas as top-level fields, `setupAgent({ context, input, output, events })` (`context` is required), or as one `schemas: createAgentSchemas({ ... })`; a partial object under `schemas` does not typecheck. Export the event schemas as one object for a later policy. A gated action (closed-choice `agent.decide`, guard in the transition) is in [reference.md](reference.md#a-gated-action).
|
|
163
135
|
|
|
164
|
-
|
|
136
|
+
- **Do the deterministic steps yourself.** Identity, lookup, ownership: states in order. A model answers only what needs judgment.
|
|
137
|
+
- **Write simple guards inline** (`context.turn < context.maxTurns`): `describeMachine(machine, { source: true })` prints them. Export a guard as a plain function only when a test must loosen it, build the machine in a factory (`createDeskMachine({ refundIsLegal })`), and pass the same functions as `helpers` so the read-back still shows their bodies: `describeMachine(createDeskMachine(guards), { source: true, helpers: guards })`.
|
|
138
|
+
- **Add an invariant sentinel** for the rule that must never break: a final state described `SENTINEL: …`, entered from an `always` on the side-effect state (`paying`), which runs before the invoke starts. The check is its own function, not the guard, so a loosened guard does not loosen it.
|
|
139
|
+
- **A person's step** is an idle state whose `meta.interaction` tells a host what to render: `{ label, events: { APPROVE: { label: "Approve" }, REJECT: { label: "Reject" } } }`, or `textEvent: { type: "CUSTOMER_REPLIED", field: "reply" }` for free text.
|
|
140
|
+
- **Actors are placeholders** in `setupAgent({ actors })`; pass the real ones with `runAgent(machine, { actors })`.
|
|
141
|
+
- **Final outputs carry the result** (`{ outcome, orderId, amount }`); the runner rebuilds the old return type from it and the world.
|
|
142
|
+
- **Declare `version`** in `createMachine({ id, version: "1", ... })` on every recorded machine; bump it on every behavior change. Traces stamp it (`add-traces` §1).
|
|
165
143
|
|
|
166
|
-
|
|
144
|
+
## 4. A policy, only if the code already has one
|
|
167
145
|
|
|
168
|
-
|
|
169
|
-
- `canReach(machine, "<sentinel>", options)` returns `reachable: false` **and** `complete: true`, for each distinct input shape (an input with an order id and one without lead to different paths; loop over them).
|
|
170
|
-
- `options.payloads` is a function of the snapshot that returns payload lists per event type. List values on both sides of every guard boundary: `0`, the threshold and one over, and each order amount the outputs below can produce and one over. Without it, a closed choice is explored only with its own `payloads`, and `complete: true` then proves nothing about the boundaries.
|
|
171
|
-
- A free-text event from a person's step (`CUSTOMER_REPLIED`) needs an entry too, one payload per branch it can take, or the search cannot send it.
|
|
146
|
+
Keep the decision in the environment as one `agent.decide`, with no policy machine, unless the old code has decision logic beyond one model call: a rule ladder, strategy heuristics, a cheap check before the model. `improve-machine` introduces a policy when traces show a rule worth adding. If the code has that logic, port it as a policy machine that answers the decision: [policy.md](policy.md).
|
|
172
147
|
|
|
173
|
-
|
|
174
|
-
const proofOptions: ExplorePathsOptions = {
|
|
175
|
-
payloads: () => ({
|
|
176
|
-
REFUND: [0, 40, 41, 100, 101, 240, 241, 800, 801].map((amount) => ({ amount })),
|
|
177
|
-
CUSTOMER_REPLIED: [{ reply: "It's ORD-1001" }, { reply: "no idea" }],
|
|
178
|
-
}),
|
|
179
|
-
outputs: { lookupOrder: [ownedOrder, otherCustomersOrder, refundedOrder, { found: false }] },
|
|
180
|
-
errors: { lookupOrder: new Error("down"), "agent.decide": new Error("exhausted") },
|
|
181
|
-
};
|
|
182
|
-
```
|
|
183
|
-
- `options.outputs` lists every output each actor can return (found and not found, owned and not, refunded and not); `options.errors` makes each actor fail once.
|
|
184
|
-
- **A machine with a loop** (a repair cycle, a resubmit) needs `options.dedupeBy`, or the search runs to `maxDepth` and reports `complete: false`. Return every context field a guard or choice reads: `dedupeBy: (s) => JSON.stringify([s.value, s.context.attempts, s.context.lastError])`. `outputs` are canned per actor src, not per call: to explore "fails twice, then succeeds", let the guard read a counter from context instead of the output.
|
|
185
|
-
- `explorePaths(...).reachedStates` lists only states a run settles in; a `choice` state it passes through is not listed.
|
|
186
|
-
- A loosened guard makes `canReach` true. Write this test too, or the proof proves nothing.
|
|
187
|
-
- Each normal final state is reachable, so the search is not vacuous.
|
|
148
|
+
## 5. The runner
|
|
188
149
|
|
|
189
|
-
|
|
150
|
+
Write `runMachine` with the old function's signature and result type.
|
|
190
151
|
|
|
191
|
-
|
|
152
|
+
- **The existing SDK call becomes the executor.** `generateText` (or `streamText`) in `executors`, as the old call: same model, system prompt, tools, options, retry code; return `{ result }`. Pass the conversation as `messages: ({ input }) => input.messages`.
|
|
153
|
+
- **A streaming route keeps its response, not its signature.** `createAiSdkExecutors().streamText` forwards text chunks only, so wrap the old call: pipe its stream to the old consumer, then return `{ result: finalValue, messages: responseMessages }` so later requests can rebuild the assistant and tool history from `output.messages`.
|
|
154
|
+
- `runAgent(machine, { input, executors, actors, onIdle })`. `onIdle` answers each idle state with the old code's call.
|
|
155
|
+
- `executors.decide` reads `request.options` (each `{ event }`) and `request.metadata?.facts`.
|
|
156
|
+
- JSON-safe input and events: with a store, an unloggable value fails the run.
|
|
157
|
+
- Rebuild the old result from `result.output` and the world. A field with no meaning now keeps its type and a comment.
|
|
192
158
|
|
|
193
|
-
|
|
159
|
+
**Approvals off a message bus.** The rule: nothing else runs while the answer is awaited → idle state + `onIdle`. Other calls in flight, or answers in any order → never idle, so invoke a wait actor: `createCallbackLogic` that subscribes to the bus, sends `APPROVED`/`REJECTED` with the call id to its parent, and unsubscribes on exit. For N calls per batch (per-call approve/reject/always, a barrier before execute, cancel-all), use `createApprovalBatchMachine` from `@statelyai/agent/machines` ([docs/machines-presets.md](../../docs/machines-presets.md)). Only when it does not fit, hand-write a child machine per batch.
|
|
194
160
|
|
|
195
|
-
|
|
196
|
-
test.each(MODELS)("parity with the loop: %s", async (_, model) => {
|
|
197
|
-
const report = await compareRuns({ scenarios: SCENARIOS, baseline: runWith(runLoop, model), candidate: runWith(runMachine, model) });
|
|
198
|
-
const unreviewed = report.scenarios.filter((row) => row.label === "regression" && !REVIEWED[row.id]);
|
|
199
|
-
expect(unreviewed).toEqual([]);
|
|
200
|
-
// A reviewed difference still has to be exactly the difference a person accepted.
|
|
201
|
-
for (const row of report.scenarios.filter((row) => REVIEWED[row.id])) REVIEWED[row.id]!(row);
|
|
202
|
-
});
|
|
161
|
+
**Host events mid-run** (cancel, abort, a user message while a request runs):
|
|
203
162
|
|
|
204
|
-
|
|
205
|
-
|
|
206
|
-
|
|
207
|
-
};
|
|
208
|
-
```
|
|
163
|
+
- A legacy body that owns abort (it returns its own `stopped` result) during a partial carve: pass the signal to the legacy body only, not to `runAgent`, so its result stands and reaches the machine as a normal final.
|
|
164
|
+
- Abort: `runAgent`'s `signal` alone is a hard stop: the run settles `{ status: "error", cause: "aborted" }` and the machine sees nothing. Add `abortWith: { event, timeout? }` when the machine should handle it (a `cancelled` final, cleanup invokes); it falls back to `aborted` if the event is unhandled, if the machine is still active once the event has been applied and the run settles, or on timeout: the handled path must reach a final or idle state before you can rely on its cleanup.
|
|
165
|
+
- A cancel the machine handles (cleanup, a `cancelled` final): a `CANCEL` event on the parent state. With `runAgent`, pass `inbox: createAgentInbox()` and call `inbox.send(event)`; an inbox serves one run. A hand-written `createAgentRuntime` loop uses `runtime.send(event)` ([docs/hosts.md](../../docs/hosts.md)).
|
|
209
166
|
|
|
210
|
-
|
|
167
|
+
## 6. Proofs that need no model
|
|
211
168
|
|
|
212
|
-
|
|
213
|
-
import { compareRecordings, createRecorder, type EffectDivergence, type Recording } from "@statelyai/agent/testing";
|
|
214
|
-
|
|
215
|
-
test.each(SCENARIOS)("equivalent to the recorded loop: $id", async (scenario) => {
|
|
216
|
-
const recording: Recording = JSON.parse(readFileSync(`fixtures/${scenario.id}.json`, "utf8"));
|
|
217
|
-
const replay = createRecorder({ replay: recording });
|
|
218
|
-
const model = replay.model(openai("gpt-5")); // never called in replay
|
|
219
|
-
replay.output(await runMachine({ ...scenario, model, tools: replay.tools(appTools) }));
|
|
220
|
-
const report = compareRecordings(recording, replay.recording);
|
|
221
|
-
expect(report.divergences.map(({ reason, message }) => ({ reason, message }))).toEqual(INTENDED[scenario.id] ?? []);
|
|
222
|
-
});
|
|
169
|
+
From `@statelyai/agent/testing` (example options in [reference.md](reference.md#proof-options)):
|
|
223
170
|
|
|
224
|
-
|
|
225
|
-
|
|
226
|
-
|
|
171
|
+
- `lintAgentMachine`: no errors. An invoke whose error must reach the caller unchanged has no `onError` (the runner rethrows `result.error`); pass `disable: ["invoke-without-on-error"]` for that machine. For one invoke that cannot fail (a callback bridge), pass `ignore: [{ code: "invoke-without-on-error", path: "<state>" }]` instead.
|
|
172
|
+
- `canReach(machine, "<sentinel>", options)` returns `reachable: false` **and** `complete: true`, for each distinct input shape (with and without an order id; loop over them).
|
|
173
|
+
- `options.payloads(snapshot)` returns payload lists per event type, on both sides of every guard boundary (`0`, the threshold and one over, each producible amount and one over); without it `complete: true` proves nothing about boundaries. A person's free-text event needs one payload per branch.
|
|
174
|
+
- `outputs`: every output each actor can return; `errors`: each actor fails. An actor that never settles (a callback bridge, a long-lived listener) gets `outputs: { src: [] }`: its invoke stays pending and only events move the machine.
|
|
175
|
+
- **A loop** needs `options.dedupeBy`, or the search hits `maxDepth` with `complete: false`. Key on the state value plus the fields read at the loop head (counters, flags); drop fields that only change within one pass. A search over a few seconds: cut the key first. A counter cap of N needs `maxDepth` above N times the events per pass; with the default the search ends `complete: false`.
|
|
176
|
+
- A state passed through within one step (left at once by `always`, or a `choice` state) counts as reached: `canReach` returns the path through it and `explorePaths` lists it in `reachedStates`, so it works as a sentinel.
|
|
177
|
+
- **A machine that spawns or invokes children**: pass `children: "coupled"`, `childEvents` (only the types the host sends a child directly; the parent's sends need no entry), and `hostEvents` when child status events must not be forged by the host. A witness is then a run the machines can make together. Read `vacuous`. Use `interleave: "pending"` only for the invariant that needs host events raced against in-flight work, and for a long-lived invoke that hands calls back through an inbox: without it, events that arrive while the invoke runs are never explored.
|
|
178
|
+
- A loosened guard makes `canReach` true; test that too.
|
|
179
|
+
- Each normal final is reachable (the search is not vacuous).
|
|
180
|
+
|
|
181
|
+
Type options with `ExplorePathsOptions` and callback snapshots with `ExploreSnapshot` (`@statelyai/agent/testing`), not your app's `AnyMachineSnapshot`.
|
|
182
|
+
|
|
183
|
+
## 7. Parity
|
|
227
184
|
|
|
228
|
-
|
|
229
|
-
- Values that differ on every run (generated ids, branch names, timestamps) fail `tool-input`. Pass the same `normalize` to `createRecorder({ replay, normalize })` and `compareRecordings(a, b, { normalize })` to redact them on both sides; never edit the fixtures.
|
|
230
|
-
- `divergences` lists every difference, lined up so one extra call does not shift the rest; `divergence` is the first.
|
|
231
|
-
- In replay, a model call past the recorded ones or a tool call the old code never made throws `RecordingDivergenceError`, and `compareRecordings` names it.
|
|
232
|
-
- `model-call-shape`: the machine changed a call's tools or output format, for example a model-picked action that became `agent.decide`. The recorded answer does not fit the new call, so equivalence ends there. Port the call unchanged first, prove equivalence, then change it in a separate step whose divergences are `INTENDED`. To keep replaying past a call you changed on purpose, pass the same `modelCalls` function to `createRecorder({ replay, modelCalls })` and `compareRecordings`: `(expected, actual) => expected.call === 2 || sameModelCallShape(expected, actual)`. The recorded answer must still fit the new call.
|
|
233
|
-
- A guard that blocks a rule break diverges on purpose. List it in `INTENDED` with the exact divergence.
|
|
185
|
+
Switch the candidate to `runMachine`; run per stand-in (or against the spec). Code in [reference.md](reference.md#the-parity-harness).
|
|
234
186
|
|
|
235
|
-
- Every `regression` row: fix the machine, or
|
|
236
|
-
-
|
|
237
|
-
-
|
|
187
|
+
- Every `regression` row: fix the machine, or ask the user when the old behavior was wrong but broke no rule. Differing outcomes: the answer may become `expected`. Differing effects only: list the row in a reviewed-differences map, exact difference asserted, with who accepted it. Never edit a scenario to make a row pass.
|
|
188
|
+
- **Run the old suite against the port**, the strongest evidence when the old code has tests: mock the old module (`vi.mock('../old-module', ...)`) with a subclass whose loop methods call `runAgent`, plus a canary counter the machine increments. Assert the exact count (the N runs the suite drives), not `> 0`. A test that spies on internals the port no longer calls is a structural failure: mark it `it.fails` with a comment naming why; never delete it. Snapshots are stored under the importing file's name, so a `toMatchSnapshot` there writes a new snapshot and passes: copy the old `__snapshots__/<old>.snap` to the swap file's name and run with `CI=true`, so a missing snapshot fails.
|
|
189
|
+
- **Swap pitfalls.** A mock factory that imports the port deadlocks when the port imports the mocked module: load the port with a dynamic import in `beforeAll` and call it from inside the factory's functions. A mock that misses the code's own import of the module: patch the class prototype from a setup file that imports the module dynamically. Code in [reference.md](reference.md#old-suite-swap).
|
|
190
|
+
- **Timing.** `runAgent` starts asynchronously, so fire-and-forget work the old code starts at the same time may settle in another order. An old test asserting exact action order across concurrent tasks is structural only after you confirm the actions commute and their order is not part of the contract; an order users or storage see is behavioral.
|
|
191
|
+
- Every `intended` row: its reason is a real rule. A fixed rule break with the same outcome is `intended` too.
|
|
192
|
+
- With recordings, replay each against the machine (`createRecorder({ replay })`, `compareRecordings`): no key, no side effects, every divergence in `INTENDED` with who accepted it, including a guard that blocks a rule break. Semantics in [reference.md](reference.md#replay-equivalence).
|
|
193
|
+
- With no model in the loop, parity is the whole proof: the spec baseline plus §6. Say so.
|
|
238
194
|
|
|
239
195
|
## 8. Hand back
|
|
240
196
|
|
|
241
197
|
Report, in this order:
|
|
242
198
|
|
|
243
|
-
1. The files
|
|
244
|
-
2. The rule list
|
|
245
|
-
3. The parity table per stand-in model: `same / intended / regression`,
|
|
199
|
+
1. The files (environment, runner, tests, any policy); the old code untouched.
|
|
200
|
+
2. The rule list, each mapped to a guard, a state, or why it was dropped.
|
|
201
|
+
3. The parity table per stand-in model: `same / intended / regression`, one line per `intended` row with its reason. Say whether the baseline was a run or a spec.
|
|
246
202
|
4. The `canReach` results and the loosened-guard test.
|
|
247
|
-
5.
|
|
248
|
-
6.
|
|
203
|
+
5. The machine rendered: `describeMachine(machine)` output, and how to watch it live in the Stately inspector (install `@statelyai/sdk` and its optional peer `ws` for a local relay, and pass `inspector.inspect` as `runAgent`'s `inspect`; see [docs/observability.md](../../docs/observability.md)).
|
|
204
|
+
6. Anything you had to decide that the user should confirm, including where message history lives.
|
|
205
|
+
7. Next step: `add-traces`.
|