@statelyai/agent 2.0.0-alpha.39 → 2.0.0-alpha.41

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (45) hide show
  1. package/CHANGELOG.md +35 -0
  2. package/dist/evals.cjs +1 -1
  3. package/dist/evals.d.cts +1 -1
  4. package/dist/evals.d.mts +1 -1
  5. package/dist/evals.mjs +1 -1
  6. package/dist/index.cjs +17 -10
  7. package/dist/index.d.cts +29 -12
  8. package/dist/index.d.mts +29 -12
  9. package/dist/index.mjs +17 -10
  10. package/dist/machine-input-0YTPcnCm.mjs +27 -0
  11. package/dist/machine-input-DU3owLUr.cjs +32 -0
  12. package/dist/otel.d.cts +1 -1
  13. package/dist/otel.d.mts +1 -1
  14. package/dist/policy.cjs +1 -1
  15. package/dist/policy.d.cts +1 -1
  16. package/dist/policy.d.mts +1 -1
  17. package/dist/policy.mjs +1 -1
  18. package/dist/{run-agent-Dq0oSFRm.d.cts → run-agent-BO9RSb7i.d.cts} +16 -7
  19. package/dist/{run-agent-DHlq87G7.d.mts → run-agent-BxkeuAYR.d.mts} +16 -7
  20. package/dist/{run-agent-De3xVFGw.cjs → run-agent-COdR9FpM.cjs} +26 -27
  21. package/dist/{run-agent-BE7DOWD6.mjs → run-agent-D3gW9wfd.mjs} +27 -28
  22. package/dist/testing.cjs +377 -18
  23. package/dist/testing.d.cts +181 -5
  24. package/dist/testing.d.mts +181 -5
  25. package/dist/testing.mjs +377 -20
  26. package/dist/{utils-DAclF-kw.d.cts → utils-BMxUcjay.d.cts} +57 -1
  27. package/dist/{utils-B8k-YPYI.d.mts → utils-BV7pxhD-.d.mts} +57 -1
  28. package/dist/validate.cjs +1 -1
  29. package/dist/validate.d.cts +1 -1
  30. package/dist/validate.d.mts +1 -1
  31. package/dist/validate.mjs +1 -1
  32. package/dist/{verify-H3Pr2gNw.d.mts → verify-C1UU84eq.d.mts} +22 -2
  33. package/dist/{verify-BM0sGmsf.d.cts → verify-DlN6tK8_.d.cts} +22 -2
  34. package/dist/{verify-Dek6gHCS.cjs → verify-IVYsI026.cjs} +84 -17
  35. package/dist/{verify-CWlg3j7B.mjs → verify-lKyu3buS.mjs} +84 -17
  36. package/docs/evals.md +1 -0
  37. package/docs/hosts.md +5 -3
  38. package/docs/index.md +2 -2
  39. package/docs/meta.json +1 -0
  40. package/docs/strangler-fig.md +36 -29
  41. package/docs/verify-in-place.md +326 -0
  42. package/docs/verify.md +70 -4
  43. package/llms.txt +1 -0
  44. package/package.json +1 -1
  45. package/readme.md +1 -0
package/docs/hosts.md CHANGED
@@ -309,7 +309,7 @@ Each operation is described in [The runtime loop](steps.md).
309
309
 
310
310
  ### Host events mid-run
311
311
 
312
- `runtime.send(event)` puts a host event in the run's mailbox, behind whatever already arrived. A `nextEvent()` parked on a pending request resolves with it; the loop applies it with `transition` like any other event.
312
+ `runtime.send(event)` queues a host event behind earlier host sends and ahead of completions the run holds but has not handed out. A `nextEvent()` parked on a pending request resolves with it; the loop applies it with `transition` like any other event.
313
313
 
314
314
  ```ts no-check
315
315
  const runtime = createAgentRuntime(machine, { executors });
@@ -325,7 +325,7 @@ for (let event; (event = await runtime.nextEvent());) {
325
325
  const result = await runtime.finish();
326
326
  ```
327
327
 
328
- - **Ordering.** One FIFO with request completions, child messages, and timers. A send during a pending invoke is handed out by the next `nextEvent()`; the invoke keeps running unless the transition leaves its state, which aborts its `signal`.
328
+ - **Ordering.** Host sends keep their own order and go ahead of request completions, child messages, and timers the run already holds, which keep theirs. A send during a pending invoke is handed out by the next `nextEvent()`; the invoke keeps running unless the transition leaves its state, which aborts its `signal`.
329
329
  - **Journal.** A sent event is a log entry, the same as `runAgent({ event })`. Replaying `result.events` lands on the same state.
330
330
  - **Unhandled.** A sent event the state has no transition for is reported as `result.ignored`.
331
331
  - **Before `start`.** Queued; the first `nextEvent()` hands it out. Returns `true`.
@@ -380,6 +380,8 @@ const result = await runAgent(machine, {
380
380
  - the loop never applies it;
381
381
  - the machine handled it but settles still `active` (an idle wait nothing will answer), immediately and without waiting for `timeout`;
382
382
  - `timeout` ms pass while the machine is still active and working.
383
+ - `event` is a function and it throws; the run's `error` is what it threw.
384
+ - **Per reason.** `event` can be a function of `signal.reason`, so a pause and a stop send different events: `abortWith: { event: (reason) => ({ type: reason === "pause" ? ("PAUSE" as const) : ("STOP" as const) }) }`. As with `onIdle`, an event built in a function needs `as const` for its `type`.
383
385
  - **Journal.** The abort event is a log entry; replay reaches the same final state.
384
386
 
385
387
  ### runAgent inbox
@@ -395,7 +397,7 @@ const result = await run;
395
397
 
396
398
  - **Same as `runtime.send`.** Each `inbox.send` goes to the run's mailbox and returns the same boolean. See [Host events mid-run](#host-events-mid-run).
397
399
  - **Before start.** Queued; delivered after the init event and after `runAgent`'s own `event`. Returns `true`.
398
- - **While idle.** An inbox event resumes the run, ahead of a pending `onIdle` answer; that answer is dropped. Without `onIdle`, `runAgent` returns at the first idle.
400
+ - **While idle.** An inbox event resumes the run, ahead of a pending `onIdle` answer; that answer is dropped. Without `onIdle`, `runAgent` returns at the first idle. With `onIdle: "inbox"`, an idle state waits for the next `inbox.send` (a resume after a pause) and has no answer of its own.
399
401
  - **Ignored.** An inbox event the state does not handle is reported as `result.ignored`. Unlike an ignored `onIdle` answer, it does not end the `onIdle` loop.
400
402
  - **After `runAgent` returns.** Returns `false`.
401
403
  - **One call.** An inbox serves one `runAgent` call; reusing it throws `AgentError` `inbox-in-use`.
package/docs/index.md CHANGED
@@ -11,7 +11,7 @@ Stately Agent is XState plus typed model requests, decisions, and host executors
11
11
  ## Start here
12
12
 
13
13
  - Greenfield, a new agent: [Quickstart](quickstart.md), then [Thinking in state machines](thinking-in-state-machines.md) or a copied [agent pattern](patterns.md).
14
- - Brownfield, an existing agent loop: [Migrating from a loop](from-a-loop.md), or the [`loop-to-machine`](https://github.com/statelyai/skills/blob/main/plugins/stately-agent/skills/loop-to-machine/SKILL.md) skill from [statelyai/skills](https://github.com/statelyai/skills).
14
+ - Brownfield, an existing agent loop: [Verify in place](verify-in-place.md) to check it against a model without changing it, or [Migrating from a loop](from-a-loop.md), or the [`loop-to-machine`](https://github.com/statelyai/skills/blob/main/plugins/stately-agent/skills/loop-to-machine/SKILL.md) skill from [statelyai/skills](https://github.com/statelyai/skills).
15
15
  - Brownfield, an existing async workflow: [Strangler-fig retrofit](strangler-fig.md). Wrap it unchanged with `wrapWorkflow`, then move one phase at a time into states, checked with `checkParity`.
16
16
  - [Choosing a run mode](choosing-a-run-mode.md)
17
17
  - [Advanced: bare XState actors and hand-written executors](advanced.md)
@@ -35,7 +35,7 @@ Stately Agent is XState plus typed model requests, decisions, and host executors
35
35
  | ------------------------------ | ------------------------------------------------------------------------------------------ |
36
36
  | `@statelyai/agent` | Authoring, running, human interaction, and the executor contract |
37
37
  | `@statelyai/agent/evals` | Trace analytics: `foldTraces`, `traceCoverage`, `repeatRuns`, `judgeAgreement` |
38
- | `@statelyai/agent/testing` | `lintAgentMachine`, `simulateAgent`, `canReach`, `checkParity`, trajectories, seams |
38
+ | `@statelyai/agent/testing` | `lintAgentMachine`, `canReach`, `checkConformance`, `checkParity`, trajectories, seams |
39
39
  | `@statelyai/agent/log` | The event log: `replay`, `forkEventLog`, `getDecisionAttempts`, hand-built entries, stores |
40
40
  | `@statelyai/agent/ai-sdk` | `createAiSdkExecutors`: the Vercel AI SDK adapter, for language and decision models |
41
41
  | `@statelyai/agent/openai` | Executors over the raw `openai` package |
package/docs/meta.json CHANGED
@@ -5,6 +5,7 @@
5
5
  "index",
6
6
  "quickstart",
7
7
  "thinking-in-state-machines",
8
+ "verify-in-place",
8
9
  "from-a-loop",
9
10
  "strangler-fig",
10
11
  "scope",
@@ -36,20 +36,21 @@ export async function runTicket(ticket: Ticket, effects: SupportEffects): Promis
36
36
  When the loop is a class method that calls its own private methods for the model and the tools, there is no `(input, effects)` function to pass. Wrap it with no edit to the class (a phase in the middle of a method is extracted at step 0, see [A phase in the middle of a method](#a-phase-in-the-middle-of-a-method)):
37
37
 
38
38
  - `run: (input, effects) => { const executor = createExecutor(input); executor.callModel = (...args) => effects.callModel(...args); return executor.run(); }`, with each effect calling the original method.
39
+ - One wrapped workflow serves every instance: the instance goes in as `live`, and the workflow reads it from its context. `run: (input, effects, { live }: { live: Agent }) => live.run(input)`, then `step0.run(input, { live: agent, effects })`. With `live` typed, every `run` call must pass it.
39
40
  - Route the old test suite through the wrap with a factory or a static `create` stand-in, not a subclass, when the constructor is private.
40
41
  - Install each replacement with `Object.defineProperty` (non-enumerable). A plain assignment adds an own enumerable field, which the repo's state snapshots then serialize.
41
42
  - The seam is on both sides of `checkParity`: `before` runs the class with the seam installed, so a serialization bug in the seam shows up on both sides and parity reports equivalent. Also compare the untouched class against the port by output, with no seam on the original, and add scenarios for edge results: empty, `null`, non-JSON values, `BigInt`.
42
43
 
43
44
  ### Seams for other shapes
44
45
 
45
- | Shape | Seam |
46
- | --------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
47
- | Effect-TS generator | Transliterate the generator to async, statement for statement, and bridge each effect from Effect to Promise, keeping the fiber context, typed failures, and interrupts. The parity baseline is the async transliteration; the old suite covers the original. |
48
- | Redux thunk | The thunk's `extra` argument plus its nested dispatches. Wrap `dispatch` so every nested thunk gets the effects-backed `extra`. |
49
- | LangGraph graph | Wrap `graph.invoke`. Fake the model factory with the test runner's module mock (`vi.mock`); the fake calls the run's effects. |
50
- | A closure inside a DI or Layer factory | Copy the module, identical outside the carved function, and swap it in with a module mock. |
51
- | Async generator the consumer pulls (yields events, `.return()` on break, `.throw()` to interrupt) | Step 0: wrap the host call that drains the generator, with the collected events as the output. Carve: the machine runs the loop head and each pass is an actor. `createGeneratorBridge` carries the pull contract across: the actor calls `bridge.forward(legacyPass())` and the original generator becomes `yield* bridge.stream(runAgent(...))`. Each value waits for the consumer's next pull, `.throw()` reaches the legacy generator, and `.return()` unwinds it and awaits the run before it resolves. Code after the `yield*` does not run on `.return()`; put cleanup in a `finally`. |
52
- | A step loop inside the model SDK (`stopWhen`, `maxSteps`) | No loop to carve. The SDK call stays one effect; carve the control flow around it, such as handoff or approval. |
46
+ | Shape | Seam |
47
+ | ------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
48
+ | Effect-TS generator | Transliterate the generator to async, statement for statement, and bridge each effect from Effect to Promise, keeping the fiber context, typed failures, and interrupts. The parity baseline is the async transliteration; the old suite covers the original. |
49
+ | Redux thunk | The thunk's `extra` argument plus its nested dispatches. Wrap `dispatch` so every nested thunk gets the effects-backed `extra`. |
50
+ | LangGraph graph | Wrap `graph.invoke`. Fake the model factory with the test runner's module mock (`vi.mock`); the fake calls the run's effects. |
51
+ | A closure inside a DI or Layer factory | Copy the module, identical outside the carved function, and swap it in with a module mock. |
52
+ | Async generator the consumer pulls (yields events, `.return()` on break, `.throw()` to interrupt) | Step 0: wrap the host call that drains the generator, with the collected events as the output. Carve: the machine runs the loop head and each pass is an actor. `createGeneratorBridge` carries the pull contract across: the actor calls `bridge.forward(legacyPass())` and the original generator becomes `yield* bridge.stream(runAgent(...))`. Each value waits for the consumer's next pull, `.throw()` reaches the legacy generator, and `.return()` unwinds it and awaits the run before it resolves. Code after the `yield*` does not run on `.return()`; put cleanup in a `finally`. As with `yield*`, a consumer that pulls a value yielded while unwinding resumes the pass: it completes normally (`forward` reports `closed: false`), and the code after the `yield*` runs. |
53
+ | A step loop inside the model SDK (`stopWhen`, `maxSteps`) | No loop to carve. The SDK call stays one effect; carve the control flow around it, such as handoff or approval. |
53
54
 
54
55
  ## Step 0: wrap
55
56
 
@@ -112,9 +113,9 @@ Each effect call is two events, `effect.call` then `effect.done` or `effect.erro
112
113
 
113
114
  `support.run(input, options)` takes these `runAgent` options: `signal`, `inspect`, `onTransition`, `onTrace`, `onEvent`, `metadata`, and `verification`. Two more are its own:
114
115
 
115
- | Option | Purpose |
116
- | ---------- | ------------------------------------------------------------------------------------------- |
117
- | `effects` | Replaces some effects for one run: fakes, stubs, replay stand-ins |
116
+ | Option | Purpose |
117
+ | ---------- | --------------------------------------------------------------------------------------------- |
118
+ | `effects` | Replaces some effects for one run: fakes, stubs, replay stand-ins |
118
119
  | `recorder` | Wraps every effect with `recorder.fn(name, fn)` and records the result with `recorder.output` |
119
120
 
120
121
  Effects bound to one call's live objects (a request context, a span) go in per call: create the workflow once and pass `run(input, { effects })` on every call. The `effects` given to `wrapWorkflow` are then the defaults.
@@ -237,7 +238,9 @@ import { readFileSync } from "node:fs";
237
238
 
238
239
  const lines = readFileSync(require.resolve("vendor-lib/dist/runtime.js"), "utf8").split("\n");
239
240
  const hash = (from: number, to: number) =>
240
- createHash("sha256").update(lines.slice(from - 1, to).join("\n")).digest("hex");
241
+ createHash("sha256")
242
+ .update(lines.slice(from - 1, to).join("\n"))
243
+ .digest("hex");
241
244
 
242
245
  test("copied ranges match the original", () => {
243
246
  expect(hash(412, 470)).toBe("<sha256 recorded when you copied>");
@@ -277,13 +280,13 @@ What it does per scenario:
277
280
  2. Runs `after` on replay stand-ins: each call is answered from the recording, and no effect runs.
278
281
  3. Compares the two recordings with `compareRecordings`: effect names, call order, arguments, results, and the output.
279
282
 
280
- | Field | Value |
281
- | ---------------------------- | ---------------------------------------------------------------------- |
282
- | `report.equivalent` | `true` when every scenario is |
283
- | `row.divergences` | `compareRecordings` divergences: `call-args`, `missing-effect`, … |
284
- | `row.before` / `row.after` | `{ output }` or `{ error }` |
285
- | `row.recording` / `row.replay` | The baseline recording, and what `after` did in replay |
286
- | `report.scenarios` | The rows, one per scenario, in order |
283
+ | Field | Value |
284
+ | ------------------------------ | ----------------------------------------------------------------- |
285
+ | `report.equivalent` | `true` when every scenario is |
286
+ | `row.divergences` | `compareRecordings` divergences: `call-args`, `missing-effect`, … |
287
+ | `row.before` / `row.after` | `{ output }` or `{ error }` |
288
+ | `row.recording` / `row.replay` | The baseline recording, and what `after` did in replay |
289
+ | `report.scenarios` | The rows, one per scenario, in order |
287
290
 
288
291
  - An `after` that throws is a row with `after: { error }`, never a thrown `checkParity`. A call the recording cannot answer throws `RecordingDivergenceError` inside `after`.
289
292
  - When both sides throw, they match on the error's name and message.
@@ -350,20 +353,24 @@ Run the module's own tests against the port by swapping the module.
350
353
 
351
354
  Each carved phase adds what the machine can prove:
352
355
 
353
- | Step | Change | Gained |
354
- | ---- | ------------------------------ | --------------------------------------------------------------- |
355
- | 0 | none | transitions, traces, event log, `signal`, a recorded baseline |
356
- | 1 | classify retry becomes states | `canReach` proves the classifier runs at most `MAX_ATTEMPTS` times |
357
- | n | the last phase becomes states | the machine is the workflow; drop the wrapper |
356
+ | Step | Change | Gained |
357
+ | ---- | ----------------------------- | ------------------------------------------------------------------ |
358
+ | 0 | none | transitions, traces, event log, `signal`, a recorded baseline |
359
+ | 1 | classify retry becomes states | `canReach` proves the classifier runs at most `MAX_ATTEMPTS` times |
360
+ | n | the last phase becomes states | the machine is the workflow; drop the wrapper |
358
361
 
359
362
  The step 1 proof in `examples/strangler-fig/index.test.ts`:
360
363
 
361
364
  ```ts no-check
362
- const tooMany = await canReach(supportMachine, (snapshot) => snapshot.context.attempts > MAX_ATTEMPTS, {
363
- input: ticket,
364
- outputs: { classify: [classification], resolve: [resolved] },
365
- errors: { classify: new Error("model overloaded") },
366
- });
365
+ const tooMany = await canReach(
366
+ supportMachine,
367
+ (snapshot) => snapshot.context.attempts > MAX_ATTEMPTS,
368
+ {
369
+ input: ticket,
370
+ outputs: { classify: [classification], resolve: [resolved] },
371
+ errors: { classify: new Error("model overloaded") },
372
+ },
373
+ );
367
374
  // { reachable: false, complete: true }
368
375
  ```
369
376
 
@@ -0,0 +1,326 @@
1
+ ---
2
+ title: Verify in place
3
+ description: Model an existing agent workflow, prove the model's rules, and check recorded runs of the unchanged code against it. Migrating the code to a machine is optional.
4
+ ---
5
+
6
+ > **Alpha:** `@statelyai/agent` 2.0 is in alpha. APIs can change between releases; pin an exact version. Feedback: [github.com/statelyai/agent](https://github.com/statelyai/agent/issues).
7
+
8
+ There are two ways to use `@statelyai/agent`:
9
+
10
+ - **Author** a new workflow as a machine. The machine is the code, and it enforces its rules at runtime. Start with the [Quickstart](quickstart.md).
11
+ - **Verify in place.** Keep the code you have, write a model of its control flow, prove the model's rules, and check recorded runs of the code against the model. This page covers that track.
12
+
13
+ You can migrate the code to a machine later. The model and its proofs carry over. See [Migrate later](#migrate-later).
14
+
15
+ The worked example is [examples/verify-in-place](https://github.com/statelyai/agent/tree/next/examples/verify-in-place). Every snippet on this page comes from it.
16
+
17
+ ## What you get
18
+
19
+ - A model file in the repo, reviewed like other code.
20
+ - Proofs of the model's rules that run in CI with no model calls.
21
+ - A check that recorded runs of the code conform to the model, and a report of the first step where a run does not.
22
+ - Coverage: the states and transitions of the model that no recorded run reached.
23
+
24
+ The claim is: the model is proved, and these recorded runs conform to it. A run you did not record can still break a rule. After a migration, the machine enforces the rules, and the claim covers every run.
25
+
26
+ ## The existing code
27
+
28
+ The example is a refund agent. Every side effect goes through one `effects` object, and the rules live in `if` statements:
29
+
30
+ ```ts no-check
31
+ export async function handleRefundRequest(
32
+ fx: RefundEffects,
33
+ request: { orderId: string; message: string },
34
+ ): Promise<RefundOutcome> {
35
+ const order = await fx.lookupOrder(request.orderId);
36
+ for (let decisions = 0; decisions < MAX_DECISIONS; decisions++) {
37
+ const decision = await fx.decide({ request: request.message, order, decisions });
38
+ if (decision.type === "reply") return "replied";
39
+ if (decision.type === "investigate") continue;
40
+ if (decision.amount > order.total) return "over-total";
41
+ if (decision.amount > APPROVAL_THRESHOLD) {
42
+ const approved = await fx.requestApproval({
43
+ orderId: order.orderId,
44
+ amount: decision.amount,
45
+ });
46
+ if (!approved) return "declined";
47
+ }
48
+ await fx.issueRefund({ orderId: order.orderId, amount: decision.amount });
49
+ return "refunded";
50
+ }
51
+ return "escalated";
52
+ }
53
+ ```
54
+
55
+ The check does not edit this file. Code whose side effects are spread across modules needs a seam first: route them through one object, or record them at the client with `createRecorder`'s `recorder.client`, `recorder.model` for an AI SDK model, and `recorder.tools` for AI SDK tools. See [Evals](evals.md#any-model-client).
56
+
57
+ ### AI SDK code
58
+
59
+ Code that builds its model inline (`openai("gpt-…")`) and passes module-level tools to `generateText` has nothing to inject. Mock `"ai"` in the test and route its calls through the recorder with `recordAiSdk`:
60
+
61
+ ```ts no-check
62
+ // seam.ts: must not import "ai", or the mock below waits on itself.
63
+ let current: AiSdkSeam | undefined;
64
+ export const currentSeam = () => current;
65
+ export function startRecording(standIn: LanguageModelV3) {
66
+ const recorder = createRecorder();
67
+ current = { recorder, model: recorder.model(standIn) };
68
+ return recorder;
69
+ }
70
+
71
+ // support-agent.test.ts
72
+ vi.mock("ai", async (importOriginal) => {
73
+ const ai = await importOriginal<typeof import("ai")>();
74
+ const { recordAiSdk } = await import("@statelyai/agent/testing");
75
+ const { currentSeam } = await import("./seam");
76
+ return { ...ai, ...recordAiSdk(ai, currentSeam) };
77
+ });
78
+ ```
79
+
80
+ - Each `generateText` and `streamText` call runs on the real AI SDK with the seam's `model` in place of the code's, and its `tools` wrapped with `recorder.tools`. The code's own tool loop runs.
81
+ - A call made while `currentSeam()` returns `undefined` throws, so no test reaches a real provider.
82
+ - A tool that has `execute` runs inside the `generateText` call. A check the code makes after the call returns comes after the tool already ran: model the order the calls happen in.
83
+ - `recorder.model` records each provider attempt, and `effect.call` counts attempts. `recordAiSdk` stamps each attempt of a `recorder.model` wrapper from `createRecorder` with `aiSdkCall`, the `generateText` or `streamText` call it belongs to, including steps that use a model from `prepareStep`: attempts with the same `aiSdkCall` are one call the code made, its tool-loop steps and the SDK's retries.
84
+ - A failed attempt followed by the same request has `retriedBy` set to the attempt that resent it. When that attempt has the same `aiSdkCall`, the SDK retried it (`maxRetries`): skip it in the mapping. When it has a different `aiSdkCall`, the code resent it: keep it, so the model checks the code's retry path and its call limit. An attempt with no `aiSdkCall` is one whose resender is unknown: keep it. That is any attempt not routed through `recordAiSdk`, any routed attempt whose model is not a `recorder.model` wrapper passed as is (a proxy around one, or one with a replaced method), and any attempt recorded through a recorder `createRecorder` did not make.
85
+
86
+ ```ts no-check
87
+ const sdkRetried = (effect: RecordedModelCall, effects: RecordedEffect[]) =>
88
+ effect.aiSdkCall !== undefined &&
89
+ effect.retriedBy !== undefined &&
90
+ effects.some(
91
+ (e) => e.kind === "model" && e.call === effect.retriedBy && e.aiSdkCall === effect.aiSdkCall,
92
+ );
93
+ ```
94
+
95
+ ## Step 1: the model file
96
+
97
+ Write the same control flow as a machine in `model.ts`. Each unit of work is an actor declared by name, with a placeholder implementation, so the model runs none of the code. Each rule is a state or a guard:
98
+
99
+ ```ts no-check
100
+ deciding: {
101
+ invoke: {
102
+ src: "decide",
103
+ input: ({ context }) => ({ decisions: context.decisions }),
104
+ onDone: ({ context, event }) => {
105
+ const decision = event.output;
106
+ const decisions = context.decisions + 1;
107
+ if (decision.type === "reply") return { target: "replied", context: { decisions } };
108
+ if (decision.type === "investigate")
109
+ return { target: "checking", context: { decisions } };
110
+ const amount = decision.amount;
111
+ if (amount > context.order!.total)
112
+ return { target: "overTotal", context: { decisions, amount } };
113
+ return amount > APPROVAL_THRESHOLD
114
+ ? { target: "awaitingApproval", context: { decisions, amount } }
115
+ : { target: "refunding", context: { decisions, amount } };
116
+ },
117
+ },
118
+ },
119
+ awaitingApproval: {
120
+ on: {
121
+ APPROVE: { target: "refunding", context: { approved: true } },
122
+ REJECT: { target: "declined" },
123
+ },
124
+ },
125
+ ```
126
+
127
+ - Declare every actor in `setupAgent({ actors })` and invoke it by `src`. `checkConformance` answers named actors from the recording, and rejects a model whose invoke names its logic inline.
128
+ - A person's input, such as an approval, is an event the model waits for in an idle state.
129
+ - The `stately` skill in [statelyai/skills](https://github.com/statelyai/skills) can draft a model from code. Review it like code: the proofs below hold for the model, not for behavior the model leaves out.
130
+
131
+ ## Step 2: proofs
132
+
133
+ Prove the model's rules with `canReach`, as ordinary tests. Give each actor the outputs it can return:
134
+
135
+ ```ts no-check
136
+ const proofOptions = {
137
+ input: { orderId: "P-1" },
138
+ outputs: {
139
+ lookupOrder: [{ orderId: "P-1", total: 300 }],
140
+ decide: [
141
+ { type: "refund", amount: 50 },
142
+ { type: "refund", amount: 250 },
143
+ { type: "refund", amount: 400 },
144
+ { type: "reply", message: "ok" },
145
+ { type: "investigate" },
146
+ ],
147
+ issueRefund: [undefined],
148
+ },
149
+ maxDepth: 12,
150
+ };
151
+
152
+ const result = await canReach(
153
+ refundModel,
154
+ (snapshot) =>
155
+ snapshot.matches("refunding") &&
156
+ snapshot.context.amount > APPROVAL_THRESHOLD &&
157
+ !snapshot.context.approved,
158
+ proofOptions,
159
+ );
160
+ // { reachable: false, complete: true }
161
+ ```
162
+
163
+ - `reachable: false` is a proof only with `complete: true`, and only over the outputs and payloads you list. The search forks on those values and no others: a refund of 150 is not explored unless `outputs` lists it. List each rule's boundary values (just under and over a threshold, the cap reached).
164
+ - Check that the search reaches the path the rule guards, for example a large refund through approval. Otherwise a model where that path does not exist proves the rule vacuously.
165
+ - See [Testing and verification](verify.md#reachability-checks) for payloads, ordering targets, and bounds.
166
+
167
+ ## Step 3: record runs of the code
168
+
169
+ Wrap each function of the `effects` object with `recorder.fn` and run the code as it is:
170
+
171
+ ```ts no-check
172
+ const recorder = createRecorder();
173
+ const outcome = await handleRefundRequest(
174
+ {
175
+ lookupOrder: recorder.fn("lookupOrder", lookupOrder),
176
+ decide: recorder.fn("decide", decide),
177
+ requestApproval: recorder.fn("requestApproval", requestApproval),
178
+ issueRefund: recorder.fn("issueRefund", issueRefund),
179
+ },
180
+ request,
181
+ );
182
+ recorder.output(outcome); // the return value, checked in step 5
183
+ const recording = recorder.recording;
184
+ ```
185
+
186
+ The example records scripted scenarios with stand-in functions. A recording from a real run works the same way; save it as JSON and check it later.
187
+
188
+ ## Step 4: map recorded calls to observations
189
+
190
+ An observation is one thing the run did, in the model's terms. Map the recording's effects with `flatMap`:
191
+
192
+ ```ts no-check
193
+ function toObservations(recording: Recording): Observation[] {
194
+ return recording.effects.flatMap((effect): Observation[] => {
195
+ if (effect.kind === "output") return [{ kind: "returned", outcome: effect.value }];
196
+ if (effect.kind !== "call") return [];
197
+ switch (effect.name) {
198
+ // A call that threw carries its error, so its actor fails with it.
199
+ case "lookupOrder":
200
+ return [
201
+ {
202
+ kind: "lookup",
203
+ orderId: (effect.args as [string])[0],
204
+ order: effect.result as Order,
205
+ error: effect.error,
206
+ },
207
+ ];
208
+ case "decide":
209
+ return [{ kind: "decision", decision: effect.result as Decision, error: effect.error }];
210
+ case "requestApproval": {
211
+ const [{ amount }] = effect.args as [{ amount: number }];
212
+ return [{ kind: "approval", approved: effect.result as boolean, amount }];
213
+ }
214
+ case "issueRefund":
215
+ return [
216
+ {
217
+ kind: "refund",
218
+ ...(effect.args as [{ orderId: string; amount: number }])[0],
219
+ error: effect.error,
220
+ },
221
+ ];
222
+ default:
223
+ return [];
224
+ }
225
+ });
226
+ }
227
+ ```
228
+
229
+ Keep the mapping to what the call shows. Do not decide in the mapping what the code should have done next: that is the model's job, and a mapping that repeats the code's logic makes the check pass for any run.
230
+
231
+ ## Step 5: check conformance
232
+
233
+ `checkConformance` runs the model with each actor answered from the observations:
234
+
235
+ ```ts no-check
236
+ const report = await checkConformance(refundModel, {
237
+ input: { orderId: scenario.order.orderId },
238
+ observations: toObservations(recording),
239
+ actors: {
240
+ lookupOrder: {
241
+ observes: "lookup",
242
+ matches: ({ input, observation }) => input === observation.orderId,
243
+ output: ({ observation }) => observation.order,
244
+ error: ({ observation }) => observation.error,
245
+ },
246
+ decide: {
247
+ observes: "decision",
248
+ output: ({ observation }) => observation.decision,
249
+ error: ({ observation }) => observation.error,
250
+ },
251
+ issueRefund: {
252
+ observes: "refund",
253
+ matches: ({ input, observation }) =>
254
+ input.orderId === observation.orderId && input.amount === observation.amount,
255
+ error: ({ observation }) => observation.error,
256
+ },
257
+ },
258
+ events: {
259
+ approval: {
260
+ event: (observation) => ({ type: observation.approved ? "APPROVE" : "REJECT" }),
261
+ matches: ({ observation, context }) => observation.amount === context.amount,
262
+ },
263
+ },
264
+ output: {
265
+ observes: "returned",
266
+ matches: ({ output, observation }) => output.outcome === observation.outcome,
267
+ },
268
+ });
269
+ ```
270
+
271
+ - An actor with `observes` consumes the next observation, which must be of that kind. `output` builds the actor's output from it.
272
+ - `matches` compares the actor's input with the observation. Here, the refund the code issued must be for the order and amount the model would refund.
273
+ - `events` sends an observation to the model as an event while the model waits, here the person's approval. Its `matches` compares the observation with the model's context: the person approved the amount the model is waiting to refund.
274
+ - `output` checks the code's return value. The model must finish where the code returned, and `matches` compares the model's output with the recorded one. Pass it for runs where the code threw too: record the throw as the outcome (`recorder.output({ threw: error instanceof Error ? error.message : String(error) })` in a `catch`, since code can throw a value that is not an `Error`) and model each failure the code throws as a final state whose output `matches` compares. Give that final state an output in the same shape as a return, such as `{ outcome: { threw: message } }`, and compare the fields, such as `output.outcome.threw === observation.outcome.threw`: the recorded outcome and the model's output are different objects, so `===` on the objects never matches. A model that errors instead of finishing leaves that outcome unconsumed, which deviates. Without `output`, a run that otherwise conforms reports a recorded throw as `extra-observations`, like a return value. Leaving the throw out of the observations leaves the outcome unchecked.
275
+ - An actor the recording cannot see takes no `observes`. It runs unchecked and is listed in `report.unobserved`.
276
+ - When an actor's output carries state the recording does not show, build it in `output` from the model's own `input` plus the observation.
277
+
278
+ All runs of the unchanged code conform. The full option and deviation reference is in [Testing and verification](verify.md#recorded-runs-against-a-model).
279
+
280
+ ## Step 6: coverage
281
+
282
+ Pass each report's `trace` to `traceCoverage`:
283
+
284
+ ```ts no-check
285
+ const coverage = traceCoverage(refundModel, reports.map((report) => report.trace));
286
+ coverage.unvisitedStates; // []
287
+ coverage.unvisitedHandlers; // transitions no recorded run took
288
+ ```
289
+
290
+ An unvisited state or handler is a part of the model no recorded run exercised. Add a scenario that reaches it, or confirm that it cannot happen in production.
291
+
292
+ ## A regression, caught
293
+
294
+ `regression.ts` is the code after a plausible change: repeat customers skip approval on refunds up to twice the threshold. The happy-path tests still pass. The conformance check reports:
295
+
296
+ ```ts no-check
297
+ {
298
+ reason: "extra-observations",
299
+ expected: "idle, waiting for an event",
300
+ observed: { kind: "refund", amount: 150 },
301
+ state: "awaitingApproval",
302
+ }
303
+ ```
304
+
305
+ The model was waiting for a person in `awaitingApproval`, and the code issued the refund instead. Only the scenario with a repeat customer and a mid-size refund shows it. The check finds a bug only on a run that exercises it. Coverage shows which parts of the model the recorded runs exercised.
306
+
307
+ ## What the check covers
308
+
309
+ - It checks the order and the content of what the recording shows against what the model allows, and the code's return value when you pass `output`.
310
+ - It does not see effects the recording does not capture. An actor without `observes` is unchecked, and the report lists it.
311
+ - It does not prove anything about runs you did not record. The proofs cover every path through the outputs and payloads listed in [Step 2](#step-2-proofs), not every value an actor could return. Conformance ties those paths to the recorded runs.
312
+ - A wrong model can conform and prove its rules. Review the model, and treat a deviation as a question about the code or the model.
313
+
314
+ ## Migrate later
315
+
316
+ When you want the rules enforced at runtime instead of checked after the fact, migrate the code to a machine. The model becomes the machine, its placeholder actors become the code's functions, and the same proofs run against it.
317
+
318
+ - [Migrating from a hand-rolled loop](from-a-loop.md) walks through a migration.
319
+ - The [`loop-to-machine`](https://github.com/statelyai/skills/blob/main/plugins/stately-agent/skills/loop-to-machine/SKILL.md) skill runs it with a coding agent.
320
+ - `checkParity` checks the migrated machine against the original code. See [Strangler-fig retrofit](strangler-fig.md#check-parity).
321
+
322
+ ## Related
323
+
324
+ - [Testing and verification](verify.md): `canReach`, `explorePaths`, and the `checkConformance` reference.
325
+ - [Evals](evals.md): `traceCoverage`, recorders, and parity.
326
+ - [examples/verify-in-place](https://github.com/statelyai/agent/tree/next/examples/verify-in-place): the code on this page, with tests.