@statelyai/agent 2.0.0-alpha.39 → 2.0.0-alpha.41
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +35 -0
- package/dist/evals.cjs +1 -1
- package/dist/evals.d.cts +1 -1
- package/dist/evals.d.mts +1 -1
- package/dist/evals.mjs +1 -1
- package/dist/index.cjs +17 -10
- package/dist/index.d.cts +29 -12
- package/dist/index.d.mts +29 -12
- package/dist/index.mjs +17 -10
- package/dist/machine-input-0YTPcnCm.mjs +27 -0
- package/dist/machine-input-DU3owLUr.cjs +32 -0
- package/dist/otel.d.cts +1 -1
- package/dist/otel.d.mts +1 -1
- package/dist/policy.cjs +1 -1
- package/dist/policy.d.cts +1 -1
- package/dist/policy.d.mts +1 -1
- package/dist/policy.mjs +1 -1
- package/dist/{run-agent-Dq0oSFRm.d.cts → run-agent-BO9RSb7i.d.cts} +16 -7
- package/dist/{run-agent-DHlq87G7.d.mts → run-agent-BxkeuAYR.d.mts} +16 -7
- package/dist/{run-agent-De3xVFGw.cjs → run-agent-COdR9FpM.cjs} +26 -27
- package/dist/{run-agent-BE7DOWD6.mjs → run-agent-D3gW9wfd.mjs} +27 -28
- package/dist/testing.cjs +377 -18
- package/dist/testing.d.cts +181 -5
- package/dist/testing.d.mts +181 -5
- package/dist/testing.mjs +377 -20
- package/dist/{utils-DAclF-kw.d.cts → utils-BMxUcjay.d.cts} +57 -1
- package/dist/{utils-B8k-YPYI.d.mts → utils-BV7pxhD-.d.mts} +57 -1
- package/dist/validate.cjs +1 -1
- package/dist/validate.d.cts +1 -1
- package/dist/validate.d.mts +1 -1
- package/dist/validate.mjs +1 -1
- package/dist/{verify-H3Pr2gNw.d.mts → verify-C1UU84eq.d.mts} +22 -2
- package/dist/{verify-BM0sGmsf.d.cts → verify-DlN6tK8_.d.cts} +22 -2
- package/dist/{verify-Dek6gHCS.cjs → verify-IVYsI026.cjs} +84 -17
- package/dist/{verify-CWlg3j7B.mjs → verify-lKyu3buS.mjs} +84 -17
- package/docs/evals.md +1 -0
- package/docs/hosts.md +5 -3
- package/docs/index.md +2 -2
- package/docs/meta.json +1 -0
- package/docs/strangler-fig.md +36 -29
- package/docs/verify-in-place.md +326 -0
- package/docs/verify.md +70 -4
- package/llms.txt +1 -0
- package/package.json +1 -1
- package/readme.md +1 -0
package/docs/hosts.md
CHANGED
|
@@ -309,7 +309,7 @@ Each operation is described in [The runtime loop](steps.md).
|
|
|
309
309
|
|
|
310
310
|
### Host events mid-run
|
|
311
311
|
|
|
312
|
-
`runtime.send(event)`
|
|
312
|
+
`runtime.send(event)` queues a host event behind earlier host sends and ahead of completions the run holds but has not handed out. A `nextEvent()` parked on a pending request resolves with it; the loop applies it with `transition` like any other event.
|
|
313
313
|
|
|
314
314
|
```ts no-check
|
|
315
315
|
const runtime = createAgentRuntime(machine, { executors });
|
|
@@ -325,7 +325,7 @@ for (let event; (event = await runtime.nextEvent());) {
|
|
|
325
325
|
const result = await runtime.finish();
|
|
326
326
|
```
|
|
327
327
|
|
|
328
|
-
- **Ordering.**
|
|
328
|
+
- **Ordering.** Host sends keep their own order and go ahead of request completions, child messages, and timers the run already holds, which keep theirs. A send during a pending invoke is handed out by the next `nextEvent()`; the invoke keeps running unless the transition leaves its state, which aborts its `signal`.
|
|
329
329
|
- **Journal.** A sent event is a log entry, the same as `runAgent({ event })`. Replaying `result.events` lands on the same state.
|
|
330
330
|
- **Unhandled.** A sent event the state has no transition for is reported as `result.ignored`.
|
|
331
331
|
- **Before `start`.** Queued; the first `nextEvent()` hands it out. Returns `true`.
|
|
@@ -380,6 +380,8 @@ const result = await runAgent(machine, {
|
|
|
380
380
|
- the loop never applies it;
|
|
381
381
|
- the machine handled it but settles still `active` (an idle wait nothing will answer), immediately and without waiting for `timeout`;
|
|
382
382
|
- `timeout` ms pass while the machine is still active and working.
|
|
383
|
+
- `event` is a function and it throws; the run's `error` is what it threw.
|
|
384
|
+
- **Per reason.** `event` can be a function of `signal.reason`, so a pause and a stop send different events: `abortWith: { event: (reason) => ({ type: reason === "pause" ? ("PAUSE" as const) : ("STOP" as const) }) }`. As with `onIdle`, an event built in a function needs `as const` for its `type`.
|
|
383
385
|
- **Journal.** The abort event is a log entry; replay reaches the same final state.
|
|
384
386
|
|
|
385
387
|
### runAgent inbox
|
|
@@ -395,7 +397,7 @@ const result = await run;
|
|
|
395
397
|
|
|
396
398
|
- **Same as `runtime.send`.** Each `inbox.send` goes to the run's mailbox and returns the same boolean. See [Host events mid-run](#host-events-mid-run).
|
|
397
399
|
- **Before start.** Queued; delivered after the init event and after `runAgent`'s own `event`. Returns `true`.
|
|
398
|
-
- **While idle.** An inbox event resumes the run, ahead of a pending `onIdle` answer; that answer is dropped. Without `onIdle`, `runAgent` returns at the first idle.
|
|
400
|
+
- **While idle.** An inbox event resumes the run, ahead of a pending `onIdle` answer; that answer is dropped. Without `onIdle`, `runAgent` returns at the first idle. With `onIdle: "inbox"`, an idle state waits for the next `inbox.send` (a resume after a pause) and has no answer of its own.
|
|
399
401
|
- **Ignored.** An inbox event the state does not handle is reported as `result.ignored`. Unlike an ignored `onIdle` answer, it does not end the `onIdle` loop.
|
|
400
402
|
- **After `runAgent` returns.** Returns `false`.
|
|
401
403
|
- **One call.** An inbox serves one `runAgent` call; reusing it throws `AgentError` `inbox-in-use`.
|
package/docs/index.md
CHANGED
|
@@ -11,7 +11,7 @@ Stately Agent is XState plus typed model requests, decisions, and host executors
|
|
|
11
11
|
## Start here
|
|
12
12
|
|
|
13
13
|
- Greenfield, a new agent: [Quickstart](quickstart.md), then [Thinking in state machines](thinking-in-state-machines.md) or a copied [agent pattern](patterns.md).
|
|
14
|
-
- Brownfield, an existing agent loop: [Migrating from a loop](from-a-loop.md), or the [`loop-to-machine`](https://github.com/statelyai/skills/blob/main/plugins/stately-agent/skills/loop-to-machine/SKILL.md) skill from [statelyai/skills](https://github.com/statelyai/skills).
|
|
14
|
+
- Brownfield, an existing agent loop: [Verify in place](verify-in-place.md) to check it against a model without changing it, or [Migrating from a loop](from-a-loop.md), or the [`loop-to-machine`](https://github.com/statelyai/skills/blob/main/plugins/stately-agent/skills/loop-to-machine/SKILL.md) skill from [statelyai/skills](https://github.com/statelyai/skills).
|
|
15
15
|
- Brownfield, an existing async workflow: [Strangler-fig retrofit](strangler-fig.md). Wrap it unchanged with `wrapWorkflow`, then move one phase at a time into states, checked with `checkParity`.
|
|
16
16
|
- [Choosing a run mode](choosing-a-run-mode.md)
|
|
17
17
|
- [Advanced: bare XState actors and hand-written executors](advanced.md)
|
|
@@ -35,7 +35,7 @@ Stately Agent is XState plus typed model requests, decisions, and host executors
|
|
|
35
35
|
| ------------------------------ | ------------------------------------------------------------------------------------------ |
|
|
36
36
|
| `@statelyai/agent` | Authoring, running, human interaction, and the executor contract |
|
|
37
37
|
| `@statelyai/agent/evals` | Trace analytics: `foldTraces`, `traceCoverage`, `repeatRuns`, `judgeAgreement` |
|
|
38
|
-
| `@statelyai/agent/testing` | `lintAgentMachine`, `
|
|
38
|
+
| `@statelyai/agent/testing` | `lintAgentMachine`, `canReach`, `checkConformance`, `checkParity`, trajectories, seams |
|
|
39
39
|
| `@statelyai/agent/log` | The event log: `replay`, `forkEventLog`, `getDecisionAttempts`, hand-built entries, stores |
|
|
40
40
|
| `@statelyai/agent/ai-sdk` | `createAiSdkExecutors`: the Vercel AI SDK adapter, for language and decision models |
|
|
41
41
|
| `@statelyai/agent/openai` | Executors over the raw `openai` package |
|
package/docs/meta.json
CHANGED
package/docs/strangler-fig.md
CHANGED
|
@@ -36,20 +36,21 @@ export async function runTicket(ticket: Ticket, effects: SupportEffects): Promis
|
|
|
36
36
|
When the loop is a class method that calls its own private methods for the model and the tools, there is no `(input, effects)` function to pass. Wrap it with no edit to the class (a phase in the middle of a method is extracted at step 0, see [A phase in the middle of a method](#a-phase-in-the-middle-of-a-method)):
|
|
37
37
|
|
|
38
38
|
- `run: (input, effects) => { const executor = createExecutor(input); executor.callModel = (...args) => effects.callModel(...args); return executor.run(); }`, with each effect calling the original method.
|
|
39
|
+
- One wrapped workflow serves every instance: the instance goes in as `live`, and the workflow reads it from its context. `run: (input, effects, { live }: { live: Agent }) => live.run(input)`, then `step0.run(input, { live: agent, effects })`. With `live` typed, every `run` call must pass it.
|
|
39
40
|
- Route the old test suite through the wrap with a factory or a static `create` stand-in, not a subclass, when the constructor is private.
|
|
40
41
|
- Install each replacement with `Object.defineProperty` (non-enumerable). A plain assignment adds an own enumerable field, which the repo's state snapshots then serialize.
|
|
41
42
|
- The seam is on both sides of `checkParity`: `before` runs the class with the seam installed, so a serialization bug in the seam shows up on both sides and parity reports equivalent. Also compare the untouched class against the port by output, with no seam on the original, and add scenarios for edge results: empty, `null`, non-JSON values, `BigInt`.
|
|
42
43
|
|
|
43
44
|
### Seams for other shapes
|
|
44
45
|
|
|
45
|
-
| Shape
|
|
46
|
-
|
|
|
47
|
-
| Effect-TS generator
|
|
48
|
-
| Redux thunk
|
|
49
|
-
| LangGraph graph
|
|
50
|
-
| A closure inside a DI or Layer factory
|
|
51
|
-
| Async generator the consumer pulls (yields events, `.return()` on break, `.throw()` to interrupt) | Step 0: wrap the host call that drains the generator, with the collected events as the output. Carve: the machine runs the loop head and each pass is an actor. `createGeneratorBridge` carries the pull contract across: the actor calls `bridge.forward(legacyPass())` and the original generator becomes `yield* bridge.stream(runAgent(...))`. Each value waits for the consumer's next pull, `.throw()` reaches the legacy generator, and `.return()` unwinds it and awaits the run before it resolves. Code after the `yield*` does not run on `.return()`; put cleanup in a `finally`. |
|
|
52
|
-
| A step loop inside the model SDK (`stopWhen`, `maxSteps`)
|
|
46
|
+
| Shape | Seam |
|
|
47
|
+
| ------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
48
|
+
| Effect-TS generator | Transliterate the generator to async, statement for statement, and bridge each effect from Effect to Promise, keeping the fiber context, typed failures, and interrupts. The parity baseline is the async transliteration; the old suite covers the original. |
|
|
49
|
+
| Redux thunk | The thunk's `extra` argument plus its nested dispatches. Wrap `dispatch` so every nested thunk gets the effects-backed `extra`. |
|
|
50
|
+
| LangGraph graph | Wrap `graph.invoke`. Fake the model factory with the test runner's module mock (`vi.mock`); the fake calls the run's effects. |
|
|
51
|
+
| A closure inside a DI or Layer factory | Copy the module, identical outside the carved function, and swap it in with a module mock. |
|
|
52
|
+
| Async generator the consumer pulls (yields events, `.return()` on break, `.throw()` to interrupt) | Step 0: wrap the host call that drains the generator, with the collected events as the output. Carve: the machine runs the loop head and each pass is an actor. `createGeneratorBridge` carries the pull contract across: the actor calls `bridge.forward(legacyPass())` and the original generator becomes `yield* bridge.stream(runAgent(...))`. Each value waits for the consumer's next pull, `.throw()` reaches the legacy generator, and `.return()` unwinds it and awaits the run before it resolves. Code after the `yield*` does not run on `.return()`; put cleanup in a `finally`. As with `yield*`, a consumer that pulls a value yielded while unwinding resumes the pass: it completes normally (`forward` reports `closed: false`), and the code after the `yield*` runs. |
|
|
53
|
+
| A step loop inside the model SDK (`stopWhen`, `maxSteps`) | No loop to carve. The SDK call stays one effect; carve the control flow around it, such as handoff or approval. |
|
|
53
54
|
|
|
54
55
|
## Step 0: wrap
|
|
55
56
|
|
|
@@ -112,9 +113,9 @@ Each effect call is two events, `effect.call` then `effect.done` or `effect.erro
|
|
|
112
113
|
|
|
113
114
|
`support.run(input, options)` takes these `runAgent` options: `signal`, `inspect`, `onTransition`, `onTrace`, `onEvent`, `metadata`, and `verification`. Two more are its own:
|
|
114
115
|
|
|
115
|
-
| Option | Purpose
|
|
116
|
-
| ---------- |
|
|
117
|
-
| `effects` | Replaces some effects for one run: fakes, stubs, replay stand-ins
|
|
116
|
+
| Option | Purpose |
|
|
117
|
+
| ---------- | --------------------------------------------------------------------------------------------- |
|
|
118
|
+
| `effects` | Replaces some effects for one run: fakes, stubs, replay stand-ins |
|
|
118
119
|
| `recorder` | Wraps every effect with `recorder.fn(name, fn)` and records the result with `recorder.output` |
|
|
119
120
|
|
|
120
121
|
Effects bound to one call's live objects (a request context, a span) go in per call: create the workflow once and pass `run(input, { effects })` on every call. The `effects` given to `wrapWorkflow` are then the defaults.
|
|
@@ -237,7 +238,9 @@ import { readFileSync } from "node:fs";
|
|
|
237
238
|
|
|
238
239
|
const lines = readFileSync(require.resolve("vendor-lib/dist/runtime.js"), "utf8").split("\n");
|
|
239
240
|
const hash = (from: number, to: number) =>
|
|
240
|
-
createHash("sha256")
|
|
241
|
+
createHash("sha256")
|
|
242
|
+
.update(lines.slice(from - 1, to).join("\n"))
|
|
243
|
+
.digest("hex");
|
|
241
244
|
|
|
242
245
|
test("copied ranges match the original", () => {
|
|
243
246
|
expect(hash(412, 470)).toBe("<sha256 recorded when you copied>");
|
|
@@ -277,13 +280,13 @@ What it does per scenario:
|
|
|
277
280
|
2. Runs `after` on replay stand-ins: each call is answered from the recording, and no effect runs.
|
|
278
281
|
3. Compares the two recordings with `compareRecordings`: effect names, call order, arguments, results, and the output.
|
|
279
282
|
|
|
280
|
-
| Field
|
|
281
|
-
|
|
|
282
|
-
| `report.equivalent`
|
|
283
|
-
| `row.divergences`
|
|
284
|
-
| `row.before` / `row.after`
|
|
285
|
-
| `row.recording` / `row.replay` | The baseline recording, and what `after` did in replay
|
|
286
|
-
| `report.scenarios`
|
|
283
|
+
| Field | Value |
|
|
284
|
+
| ------------------------------ | ----------------------------------------------------------------- |
|
|
285
|
+
| `report.equivalent` | `true` when every scenario is |
|
|
286
|
+
| `row.divergences` | `compareRecordings` divergences: `call-args`, `missing-effect`, … |
|
|
287
|
+
| `row.before` / `row.after` | `{ output }` or `{ error }` |
|
|
288
|
+
| `row.recording` / `row.replay` | The baseline recording, and what `after` did in replay |
|
|
289
|
+
| `report.scenarios` | The rows, one per scenario, in order |
|
|
287
290
|
|
|
288
291
|
- An `after` that throws is a row with `after: { error }`, never a thrown `checkParity`. A call the recording cannot answer throws `RecordingDivergenceError` inside `after`.
|
|
289
292
|
- When both sides throw, they match on the error's name and message.
|
|
@@ -350,20 +353,24 @@ Run the module's own tests against the port by swapping the module.
|
|
|
350
353
|
|
|
351
354
|
Each carved phase adds what the machine can prove:
|
|
352
355
|
|
|
353
|
-
| Step | Change
|
|
354
|
-
| ---- |
|
|
355
|
-
| 0 | none
|
|
356
|
-
| 1 | classify retry becomes states
|
|
357
|
-
| n | the last phase becomes states
|
|
356
|
+
| Step | Change | Gained |
|
|
357
|
+
| ---- | ----------------------------- | ------------------------------------------------------------------ |
|
|
358
|
+
| 0 | none | transitions, traces, event log, `signal`, a recorded baseline |
|
|
359
|
+
| 1 | classify retry becomes states | `canReach` proves the classifier runs at most `MAX_ATTEMPTS` times |
|
|
360
|
+
| n | the last phase becomes states | the machine is the workflow; drop the wrapper |
|
|
358
361
|
|
|
359
362
|
The step 1 proof in `examples/strangler-fig/index.test.ts`:
|
|
360
363
|
|
|
361
364
|
```ts no-check
|
|
362
|
-
const tooMany = await canReach(
|
|
363
|
-
|
|
364
|
-
|
|
365
|
-
|
|
366
|
-
|
|
365
|
+
const tooMany = await canReach(
|
|
366
|
+
supportMachine,
|
|
367
|
+
(snapshot) => snapshot.context.attempts > MAX_ATTEMPTS,
|
|
368
|
+
{
|
|
369
|
+
input: ticket,
|
|
370
|
+
outputs: { classify: [classification], resolve: [resolved] },
|
|
371
|
+
errors: { classify: new Error("model overloaded") },
|
|
372
|
+
},
|
|
373
|
+
);
|
|
367
374
|
// { reachable: false, complete: true }
|
|
368
375
|
```
|
|
369
376
|
|
|
@@ -0,0 +1,326 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: Verify in place
|
|
3
|
+
description: Model an existing agent workflow, prove the model's rules, and check recorded runs of the unchanged code against it. Migrating the code to a machine is optional.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
> **Alpha:** `@statelyai/agent` 2.0 is in alpha. APIs can change between releases; pin an exact version. Feedback: [github.com/statelyai/agent](https://github.com/statelyai/agent/issues).
|
|
7
|
+
|
|
8
|
+
There are two ways to use `@statelyai/agent`:
|
|
9
|
+
|
|
10
|
+
- **Author** a new workflow as a machine. The machine is the code, and it enforces its rules at runtime. Start with the [Quickstart](quickstart.md).
|
|
11
|
+
- **Verify in place.** Keep the code you have, write a model of its control flow, prove the model's rules, and check recorded runs of the code against the model. This page covers that track.
|
|
12
|
+
|
|
13
|
+
You can migrate the code to a machine later. The model and its proofs carry over. See [Migrate later](#migrate-later).
|
|
14
|
+
|
|
15
|
+
The worked example is [examples/verify-in-place](https://github.com/statelyai/agent/tree/next/examples/verify-in-place). Every snippet on this page comes from it.
|
|
16
|
+
|
|
17
|
+
## What you get
|
|
18
|
+
|
|
19
|
+
- A model file in the repo, reviewed like other code.
|
|
20
|
+
- Proofs of the model's rules that run in CI with no model calls.
|
|
21
|
+
- A check that recorded runs of the code conform to the model, and a report of the first step where a run does not.
|
|
22
|
+
- Coverage: the states and transitions of the model that no recorded run reached.
|
|
23
|
+
|
|
24
|
+
The claim is: the model is proved, and these recorded runs conform to it. A run you did not record can still break a rule. After a migration, the machine enforces the rules, and the claim covers every run.
|
|
25
|
+
|
|
26
|
+
## The existing code
|
|
27
|
+
|
|
28
|
+
The example is a refund agent. Every side effect goes through one `effects` object, and the rules live in `if` statements:
|
|
29
|
+
|
|
30
|
+
```ts no-check
|
|
31
|
+
export async function handleRefundRequest(
|
|
32
|
+
fx: RefundEffects,
|
|
33
|
+
request: { orderId: string; message: string },
|
|
34
|
+
): Promise<RefundOutcome> {
|
|
35
|
+
const order = await fx.lookupOrder(request.orderId);
|
|
36
|
+
for (let decisions = 0; decisions < MAX_DECISIONS; decisions++) {
|
|
37
|
+
const decision = await fx.decide({ request: request.message, order, decisions });
|
|
38
|
+
if (decision.type === "reply") return "replied";
|
|
39
|
+
if (decision.type === "investigate") continue;
|
|
40
|
+
if (decision.amount > order.total) return "over-total";
|
|
41
|
+
if (decision.amount > APPROVAL_THRESHOLD) {
|
|
42
|
+
const approved = await fx.requestApproval({
|
|
43
|
+
orderId: order.orderId,
|
|
44
|
+
amount: decision.amount,
|
|
45
|
+
});
|
|
46
|
+
if (!approved) return "declined";
|
|
47
|
+
}
|
|
48
|
+
await fx.issueRefund({ orderId: order.orderId, amount: decision.amount });
|
|
49
|
+
return "refunded";
|
|
50
|
+
}
|
|
51
|
+
return "escalated";
|
|
52
|
+
}
|
|
53
|
+
```
|
|
54
|
+
|
|
55
|
+
The check does not edit this file. Code whose side effects are spread across modules needs a seam first: route them through one object, or record them at the client with `createRecorder`'s `recorder.client`, `recorder.model` for an AI SDK model, and `recorder.tools` for AI SDK tools. See [Evals](evals.md#any-model-client).
|
|
56
|
+
|
|
57
|
+
### AI SDK code
|
|
58
|
+
|
|
59
|
+
Code that builds its model inline (`openai("gpt-…")`) and passes module-level tools to `generateText` has nothing to inject. Mock `"ai"` in the test and route its calls through the recorder with `recordAiSdk`:
|
|
60
|
+
|
|
61
|
+
```ts no-check
|
|
62
|
+
// seam.ts: must not import "ai", or the mock below waits on itself.
|
|
63
|
+
let current: AiSdkSeam | undefined;
|
|
64
|
+
export const currentSeam = () => current;
|
|
65
|
+
export function startRecording(standIn: LanguageModelV3) {
|
|
66
|
+
const recorder = createRecorder();
|
|
67
|
+
current = { recorder, model: recorder.model(standIn) };
|
|
68
|
+
return recorder;
|
|
69
|
+
}
|
|
70
|
+
|
|
71
|
+
// support-agent.test.ts
|
|
72
|
+
vi.mock("ai", async (importOriginal) => {
|
|
73
|
+
const ai = await importOriginal<typeof import("ai")>();
|
|
74
|
+
const { recordAiSdk } = await import("@statelyai/agent/testing");
|
|
75
|
+
const { currentSeam } = await import("./seam");
|
|
76
|
+
return { ...ai, ...recordAiSdk(ai, currentSeam) };
|
|
77
|
+
});
|
|
78
|
+
```
|
|
79
|
+
|
|
80
|
+
- Each `generateText` and `streamText` call runs on the real AI SDK with the seam's `model` in place of the code's, and its `tools` wrapped with `recorder.tools`. The code's own tool loop runs.
|
|
81
|
+
- A call made while `currentSeam()` returns `undefined` throws, so no test reaches a real provider.
|
|
82
|
+
- A tool that has `execute` runs inside the `generateText` call. A check the code makes after the call returns comes after the tool already ran: model the order the calls happen in.
|
|
83
|
+
- `recorder.model` records each provider attempt, and `effect.call` counts attempts. `recordAiSdk` stamps each attempt of a `recorder.model` wrapper from `createRecorder` with `aiSdkCall`, the `generateText` or `streamText` call it belongs to, including steps that use a model from `prepareStep`: attempts with the same `aiSdkCall` are one call the code made, its tool-loop steps and the SDK's retries.
|
|
84
|
+
- A failed attempt followed by the same request has `retriedBy` set to the attempt that resent it. When that attempt has the same `aiSdkCall`, the SDK retried it (`maxRetries`): skip it in the mapping. When it has a different `aiSdkCall`, the code resent it: keep it, so the model checks the code's retry path and its call limit. An attempt with no `aiSdkCall` is one whose resender is unknown: keep it. That is any attempt not routed through `recordAiSdk`, any routed attempt whose model is not a `recorder.model` wrapper passed as is (a proxy around one, or one with a replaced method), and any attempt recorded through a recorder `createRecorder` did not make.
|
|
85
|
+
|
|
86
|
+
```ts no-check
|
|
87
|
+
const sdkRetried = (effect: RecordedModelCall, effects: RecordedEffect[]) =>
|
|
88
|
+
effect.aiSdkCall !== undefined &&
|
|
89
|
+
effect.retriedBy !== undefined &&
|
|
90
|
+
effects.some(
|
|
91
|
+
(e) => e.kind === "model" && e.call === effect.retriedBy && e.aiSdkCall === effect.aiSdkCall,
|
|
92
|
+
);
|
|
93
|
+
```
|
|
94
|
+
|
|
95
|
+
## Step 1: the model file
|
|
96
|
+
|
|
97
|
+
Write the same control flow as a machine in `model.ts`. Each unit of work is an actor declared by name, with a placeholder implementation, so the model runs none of the code. Each rule is a state or a guard:
|
|
98
|
+
|
|
99
|
+
```ts no-check
|
|
100
|
+
deciding: {
|
|
101
|
+
invoke: {
|
|
102
|
+
src: "decide",
|
|
103
|
+
input: ({ context }) => ({ decisions: context.decisions }),
|
|
104
|
+
onDone: ({ context, event }) => {
|
|
105
|
+
const decision = event.output;
|
|
106
|
+
const decisions = context.decisions + 1;
|
|
107
|
+
if (decision.type === "reply") return { target: "replied", context: { decisions } };
|
|
108
|
+
if (decision.type === "investigate")
|
|
109
|
+
return { target: "checking", context: { decisions } };
|
|
110
|
+
const amount = decision.amount;
|
|
111
|
+
if (amount > context.order!.total)
|
|
112
|
+
return { target: "overTotal", context: { decisions, amount } };
|
|
113
|
+
return amount > APPROVAL_THRESHOLD
|
|
114
|
+
? { target: "awaitingApproval", context: { decisions, amount } }
|
|
115
|
+
: { target: "refunding", context: { decisions, amount } };
|
|
116
|
+
},
|
|
117
|
+
},
|
|
118
|
+
},
|
|
119
|
+
awaitingApproval: {
|
|
120
|
+
on: {
|
|
121
|
+
APPROVE: { target: "refunding", context: { approved: true } },
|
|
122
|
+
REJECT: { target: "declined" },
|
|
123
|
+
},
|
|
124
|
+
},
|
|
125
|
+
```
|
|
126
|
+
|
|
127
|
+
- Declare every actor in `setupAgent({ actors })` and invoke it by `src`. `checkConformance` answers named actors from the recording, and rejects a model whose invoke names its logic inline.
|
|
128
|
+
- A person's input, such as an approval, is an event the model waits for in an idle state.
|
|
129
|
+
- The `stately` skill in [statelyai/skills](https://github.com/statelyai/skills) can draft a model from code. Review it like code: the proofs below hold for the model, not for behavior the model leaves out.
|
|
130
|
+
|
|
131
|
+
## Step 2: proofs
|
|
132
|
+
|
|
133
|
+
Prove the model's rules with `canReach`, as ordinary tests. Give each actor the outputs it can return:
|
|
134
|
+
|
|
135
|
+
```ts no-check
|
|
136
|
+
const proofOptions = {
|
|
137
|
+
input: { orderId: "P-1" },
|
|
138
|
+
outputs: {
|
|
139
|
+
lookupOrder: [{ orderId: "P-1", total: 300 }],
|
|
140
|
+
decide: [
|
|
141
|
+
{ type: "refund", amount: 50 },
|
|
142
|
+
{ type: "refund", amount: 250 },
|
|
143
|
+
{ type: "refund", amount: 400 },
|
|
144
|
+
{ type: "reply", message: "ok" },
|
|
145
|
+
{ type: "investigate" },
|
|
146
|
+
],
|
|
147
|
+
issueRefund: [undefined],
|
|
148
|
+
},
|
|
149
|
+
maxDepth: 12,
|
|
150
|
+
};
|
|
151
|
+
|
|
152
|
+
const result = await canReach(
|
|
153
|
+
refundModel,
|
|
154
|
+
(snapshot) =>
|
|
155
|
+
snapshot.matches("refunding") &&
|
|
156
|
+
snapshot.context.amount > APPROVAL_THRESHOLD &&
|
|
157
|
+
!snapshot.context.approved,
|
|
158
|
+
proofOptions,
|
|
159
|
+
);
|
|
160
|
+
// { reachable: false, complete: true }
|
|
161
|
+
```
|
|
162
|
+
|
|
163
|
+
- `reachable: false` is a proof only with `complete: true`, and only over the outputs and payloads you list. The search forks on those values and no others: a refund of 150 is not explored unless `outputs` lists it. List each rule's boundary values (just under and over a threshold, the cap reached).
|
|
164
|
+
- Check that the search reaches the path the rule guards, for example a large refund through approval. Otherwise a model where that path does not exist proves the rule vacuously.
|
|
165
|
+
- See [Testing and verification](verify.md#reachability-checks) for payloads, ordering targets, and bounds.
|
|
166
|
+
|
|
167
|
+
## Step 3: record runs of the code
|
|
168
|
+
|
|
169
|
+
Wrap each function of the `effects` object with `recorder.fn` and run the code as it is:
|
|
170
|
+
|
|
171
|
+
```ts no-check
|
|
172
|
+
const recorder = createRecorder();
|
|
173
|
+
const outcome = await handleRefundRequest(
|
|
174
|
+
{
|
|
175
|
+
lookupOrder: recorder.fn("lookupOrder", lookupOrder),
|
|
176
|
+
decide: recorder.fn("decide", decide),
|
|
177
|
+
requestApproval: recorder.fn("requestApproval", requestApproval),
|
|
178
|
+
issueRefund: recorder.fn("issueRefund", issueRefund),
|
|
179
|
+
},
|
|
180
|
+
request,
|
|
181
|
+
);
|
|
182
|
+
recorder.output(outcome); // the return value, checked in step 5
|
|
183
|
+
const recording = recorder.recording;
|
|
184
|
+
```
|
|
185
|
+
|
|
186
|
+
The example records scripted scenarios with stand-in functions. A recording from a real run works the same way; save it as JSON and check it later.
|
|
187
|
+
|
|
188
|
+
## Step 4: map recorded calls to observations
|
|
189
|
+
|
|
190
|
+
An observation is one thing the run did, in the model's terms. Map the recording's effects with `flatMap`:
|
|
191
|
+
|
|
192
|
+
```ts no-check
|
|
193
|
+
function toObservations(recording: Recording): Observation[] {
|
|
194
|
+
return recording.effects.flatMap((effect): Observation[] => {
|
|
195
|
+
if (effect.kind === "output") return [{ kind: "returned", outcome: effect.value }];
|
|
196
|
+
if (effect.kind !== "call") return [];
|
|
197
|
+
switch (effect.name) {
|
|
198
|
+
// A call that threw carries its error, so its actor fails with it.
|
|
199
|
+
case "lookupOrder":
|
|
200
|
+
return [
|
|
201
|
+
{
|
|
202
|
+
kind: "lookup",
|
|
203
|
+
orderId: (effect.args as [string])[0],
|
|
204
|
+
order: effect.result as Order,
|
|
205
|
+
error: effect.error,
|
|
206
|
+
},
|
|
207
|
+
];
|
|
208
|
+
case "decide":
|
|
209
|
+
return [{ kind: "decision", decision: effect.result as Decision, error: effect.error }];
|
|
210
|
+
case "requestApproval": {
|
|
211
|
+
const [{ amount }] = effect.args as [{ amount: number }];
|
|
212
|
+
return [{ kind: "approval", approved: effect.result as boolean, amount }];
|
|
213
|
+
}
|
|
214
|
+
case "issueRefund":
|
|
215
|
+
return [
|
|
216
|
+
{
|
|
217
|
+
kind: "refund",
|
|
218
|
+
...(effect.args as [{ orderId: string; amount: number }])[0],
|
|
219
|
+
error: effect.error,
|
|
220
|
+
},
|
|
221
|
+
];
|
|
222
|
+
default:
|
|
223
|
+
return [];
|
|
224
|
+
}
|
|
225
|
+
});
|
|
226
|
+
}
|
|
227
|
+
```
|
|
228
|
+
|
|
229
|
+
Keep the mapping to what the call shows. Do not decide in the mapping what the code should have done next: that is the model's job, and a mapping that repeats the code's logic makes the check pass for any run.
|
|
230
|
+
|
|
231
|
+
## Step 5: check conformance
|
|
232
|
+
|
|
233
|
+
`checkConformance` runs the model with each actor answered from the observations:
|
|
234
|
+
|
|
235
|
+
```ts no-check
|
|
236
|
+
const report = await checkConformance(refundModel, {
|
|
237
|
+
input: { orderId: scenario.order.orderId },
|
|
238
|
+
observations: toObservations(recording),
|
|
239
|
+
actors: {
|
|
240
|
+
lookupOrder: {
|
|
241
|
+
observes: "lookup",
|
|
242
|
+
matches: ({ input, observation }) => input === observation.orderId,
|
|
243
|
+
output: ({ observation }) => observation.order,
|
|
244
|
+
error: ({ observation }) => observation.error,
|
|
245
|
+
},
|
|
246
|
+
decide: {
|
|
247
|
+
observes: "decision",
|
|
248
|
+
output: ({ observation }) => observation.decision,
|
|
249
|
+
error: ({ observation }) => observation.error,
|
|
250
|
+
},
|
|
251
|
+
issueRefund: {
|
|
252
|
+
observes: "refund",
|
|
253
|
+
matches: ({ input, observation }) =>
|
|
254
|
+
input.orderId === observation.orderId && input.amount === observation.amount,
|
|
255
|
+
error: ({ observation }) => observation.error,
|
|
256
|
+
},
|
|
257
|
+
},
|
|
258
|
+
events: {
|
|
259
|
+
approval: {
|
|
260
|
+
event: (observation) => ({ type: observation.approved ? "APPROVE" : "REJECT" }),
|
|
261
|
+
matches: ({ observation, context }) => observation.amount === context.amount,
|
|
262
|
+
},
|
|
263
|
+
},
|
|
264
|
+
output: {
|
|
265
|
+
observes: "returned",
|
|
266
|
+
matches: ({ output, observation }) => output.outcome === observation.outcome,
|
|
267
|
+
},
|
|
268
|
+
});
|
|
269
|
+
```
|
|
270
|
+
|
|
271
|
+
- An actor with `observes` consumes the next observation, which must be of that kind. `output` builds the actor's output from it.
|
|
272
|
+
- `matches` compares the actor's input with the observation. Here, the refund the code issued must be for the order and amount the model would refund.
|
|
273
|
+
- `events` sends an observation to the model as an event while the model waits, here the person's approval. Its `matches` compares the observation with the model's context: the person approved the amount the model is waiting to refund.
|
|
274
|
+
- `output` checks the code's return value. The model must finish where the code returned, and `matches` compares the model's output with the recorded one. Pass it for runs where the code threw too: record the throw as the outcome (`recorder.output({ threw: error instanceof Error ? error.message : String(error) })` in a `catch`, since code can throw a value that is not an `Error`) and model each failure the code throws as a final state whose output `matches` compares. Give that final state an output in the same shape as a return, such as `{ outcome: { threw: message } }`, and compare the fields, such as `output.outcome.threw === observation.outcome.threw`: the recorded outcome and the model's output are different objects, so `===` on the objects never matches. A model that errors instead of finishing leaves that outcome unconsumed, which deviates. Without `output`, a run that otherwise conforms reports a recorded throw as `extra-observations`, like a return value. Leaving the throw out of the observations leaves the outcome unchecked.
|
|
275
|
+
- An actor the recording cannot see takes no `observes`. It runs unchecked and is listed in `report.unobserved`.
|
|
276
|
+
- When an actor's output carries state the recording does not show, build it in `output` from the model's own `input` plus the observation.
|
|
277
|
+
|
|
278
|
+
All runs of the unchanged code conform. The full option and deviation reference is in [Testing and verification](verify.md#recorded-runs-against-a-model).
|
|
279
|
+
|
|
280
|
+
## Step 6: coverage
|
|
281
|
+
|
|
282
|
+
Pass each report's `trace` to `traceCoverage`:
|
|
283
|
+
|
|
284
|
+
```ts no-check
|
|
285
|
+
const coverage = traceCoverage(refundModel, reports.map((report) => report.trace));
|
|
286
|
+
coverage.unvisitedStates; // []
|
|
287
|
+
coverage.unvisitedHandlers; // transitions no recorded run took
|
|
288
|
+
```
|
|
289
|
+
|
|
290
|
+
An unvisited state or handler is a part of the model no recorded run exercised. Add a scenario that reaches it, or confirm that it cannot happen in production.
|
|
291
|
+
|
|
292
|
+
## A regression, caught
|
|
293
|
+
|
|
294
|
+
`regression.ts` is the code after a plausible change: repeat customers skip approval on refunds up to twice the threshold. The happy-path tests still pass. The conformance check reports:
|
|
295
|
+
|
|
296
|
+
```ts no-check
|
|
297
|
+
{
|
|
298
|
+
reason: "extra-observations",
|
|
299
|
+
expected: "idle, waiting for an event",
|
|
300
|
+
observed: { kind: "refund", amount: 150 },
|
|
301
|
+
state: "awaitingApproval",
|
|
302
|
+
}
|
|
303
|
+
```
|
|
304
|
+
|
|
305
|
+
The model was waiting for a person in `awaitingApproval`, and the code issued the refund instead. Only the scenario with a repeat customer and a mid-size refund shows it. The check finds a bug only on a run that exercises it. Coverage shows which parts of the model the recorded runs exercised.
|
|
306
|
+
|
|
307
|
+
## What the check covers
|
|
308
|
+
|
|
309
|
+
- It checks the order and the content of what the recording shows against what the model allows, and the code's return value when you pass `output`.
|
|
310
|
+
- It does not see effects the recording does not capture. An actor without `observes` is unchecked, and the report lists it.
|
|
311
|
+
- It does not prove anything about runs you did not record. The proofs cover every path through the outputs and payloads listed in [Step 2](#step-2-proofs), not every value an actor could return. Conformance ties those paths to the recorded runs.
|
|
312
|
+
- A wrong model can conform and prove its rules. Review the model, and treat a deviation as a question about the code or the model.
|
|
313
|
+
|
|
314
|
+
## Migrate later
|
|
315
|
+
|
|
316
|
+
When you want the rules enforced at runtime instead of checked after the fact, migrate the code to a machine. The model becomes the machine, its placeholder actors become the code's functions, and the same proofs run against it.
|
|
317
|
+
|
|
318
|
+
- [Migrating from a hand-rolled loop](from-a-loop.md) walks through a migration.
|
|
319
|
+
- The [`loop-to-machine`](https://github.com/statelyai/skills/blob/main/plugins/stately-agent/skills/loop-to-machine/SKILL.md) skill runs it with a coding agent.
|
|
320
|
+
- `checkParity` checks the migrated machine against the original code. See [Strangler-fig retrofit](strangler-fig.md#check-parity).
|
|
321
|
+
|
|
322
|
+
## Related
|
|
323
|
+
|
|
324
|
+
- [Testing and verification](verify.md): `canReach`, `explorePaths`, and the `checkConformance` reference.
|
|
325
|
+
- [Evals](evals.md): `traceCoverage`, recorders, and parity.
|
|
326
|
+
- [examples/verify-in-place](https://github.com/statelyai/agent/tree/next/examples/verify-in-place): the code on this page, with tests.
|