@statelyai/agent 2.0.0-alpha.37 → 2.0.0-alpha.39

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (135) hide show
  1. package/CHANGELOG.md +51 -0
  2. package/dist/{act-CrX0JwWY.cjs → act-B0CVbmo4.cjs} +1 -1
  3. package/dist/{act-B9hLQls6.mjs → act-Bsz-hUfD.mjs} +1 -1
  4. package/dist/ai-sdk.cjs +4 -4
  5. package/dist/ai-sdk.d.cts +2 -3
  6. package/dist/ai-sdk.d.mts +2 -3
  7. package/dist/ai-sdk.mjs +4 -4
  8. package/dist/{decision-m5hGmHpF.mjs → decision-B41YBx5U.mjs} +1 -1
  9. package/dist/{decision-CMeuNZ1g.cjs → decision-D8bcIGXQ.cjs} +1 -1
  10. package/dist/{decision-object-DBcLWVvv.d.cts → decision-object-BB8XV5Ec.d.cts} +1 -1
  11. package/dist/{decision-object-B5zGSPFp.d.mts → decision-object-CH_sE694.d.mts} +1 -1
  12. package/dist/effects-B9IvUPVh.mjs +23 -0
  13. package/dist/effects-DrWcUGgc.cjs +34 -0
  14. package/dist/evals.cjs +810 -0
  15. package/dist/evals.d.cts +595 -0
  16. package/dist/evals.d.mts +595 -0
  17. package/dist/evals.mjs +801 -0
  18. package/dist/{event-log-DkVlk3Rh.mjs → event-log-CEq_bVXQ.mjs} +8 -4
  19. package/dist/{event-log-Dkc-yIYu.cjs → event-log-dSOnXHuG.cjs} +7 -15
  20. package/dist/{event-log-store-Brx9tOHt.mjs → event-log-store-DLAfNM34.mjs} +1 -1
  21. package/dist/{event-log-store-BBS8so8a.d.mts → event-log-store-DYun526m.d.mts} +15 -14
  22. package/dist/{event-log-store-Cl-YWDnq.cjs → event-log-store-Dey9ufA4.cjs} +1 -1
  23. package/dist/{event-log-store-bc4ICGcS.d.cts → event-log-store-aFXK6yZX.d.cts} +15 -14
  24. package/dist/index.cjs +136 -98
  25. package/dist/index.d.cts +73 -134
  26. package/dist/index.d.mts +73 -134
  27. package/dist/index.mjs +117 -75
  28. package/dist/internal-BzlpzSux.d.cts +14 -0
  29. package/dist/internal-BzlpzSux.d.mts +14 -0
  30. package/dist/log.cjs +2 -3
  31. package/dist/log.d.cts +2 -2
  32. package/dist/log.d.mts +2 -2
  33. package/dist/log.mjs +3 -3
  34. package/dist/machines.cjs +1 -719
  35. package/dist/machines.d.cts +2 -424
  36. package/dist/machines.d.mts +2 -424
  37. package/dist/machines.mjs +2 -709
  38. package/dist/{model-source-kjGwqAV8.mjs → model-source-BrsLDvHL.mjs} +1 -1
  39. package/dist/{model-source-B4WQ8Idp.cjs → model-source-YpxjmWuX.cjs} +1 -1
  40. package/dist/openai.cjs +3 -3
  41. package/dist/openai.d.cts +2 -3
  42. package/dist/openai.d.mts +2 -3
  43. package/dist/openai.mjs +3 -3
  44. package/dist/otel.d.cts +1 -1
  45. package/dist/otel.d.mts +1 -1
  46. package/dist/policy.cjs +77 -0
  47. package/dist/policy.d.cts +143 -0
  48. package/dist/policy.d.mts +143 -0
  49. package/dist/policy.mjs +75 -0
  50. package/dist/{effects-BT_cSY0j.mjs → run-agent-BE7DOWD6.mjs} +150 -184
  51. package/dist/{run-agent-DlPzBWj-.d.mts → run-agent-DHlq87G7.d.mts} +18 -34
  52. package/dist/{effects-DDseYtJk.cjs → run-agent-De3xVFGw.cjs} +149 -201
  53. package/dist/{run-agent-30mH2moE.d.cts → run-agent-Dq0oSFRm.d.cts} +18 -34
  54. package/dist/{setup-agent-BDmNr6PC.mjs → setup-agent-CLm6QVGy.mjs} +3 -3
  55. package/dist/{setup-agent-BaNyz6Q3.d.cts → setup-agent-Cv8Hk3mX.d.cts} +12 -11
  56. package/dist/{setup-agent-C1CG_nfX.d.mts → setup-agent-DgFAhaPJ.d.mts} +12 -11
  57. package/dist/{setup-agent-CmEToB8y.cjs → setup-agent-Dv5VpuHe.cjs} +3 -3
  58. package/dist/tanstack-ai.cjs +4 -4
  59. package/dist/tanstack-ai.d.cts +2 -3
  60. package/dist/tanstack-ai.d.mts +2 -3
  61. package/dist/tanstack-ai.mjs +4 -4
  62. package/dist/testing.cjs +122 -996
  63. package/dist/testing.d.cts +36 -695
  64. package/dist/testing.d.mts +36 -695
  65. package/dist/testing.mjs +112 -977
  66. package/dist/{text-logic-BqxalgMr.d.mts → text-logic-x17uLBwA.d.cts} +238 -2
  67. package/dist/{text-logic-CSf9bqpP.d.cts → text-logic-x17uLBwA.d.mts} +238 -2
  68. package/dist/trajectory-BFjXaooB.d.cts +96 -0
  69. package/dist/trajectory-BFjXaooB.d.mts +96 -0
  70. package/dist/trajectory-BUXj_1Ue.mjs +152 -0
  71. package/dist/trajectory-CoZSv4wF.cjs +157 -0
  72. package/dist/{usage-Bm5nvpow.d.cts → usage-B4B-TQSV.d.mts} +1 -1
  73. package/dist/{usage-BnikdOca.d.mts → usage-DvGcoK6x.d.cts} +1 -1
  74. package/dist/{utils-7lSa4H-i.d.cts → utils-B8k-YPYI.d.mts} +23 -17
  75. package/dist/{utils-CBbKw2fM.d.mts → utils-DAclF-kw.d.cts} +23 -17
  76. package/dist/validate.cjs +4 -4
  77. package/dist/validate.d.cts +2 -2
  78. package/dist/validate.d.mts +2 -2
  79. package/dist/validate.mjs +4 -4
  80. package/dist/{verify-DJHfEfm6.d.cts → verify-BM0sGmsf.d.cts} +42 -6
  81. package/dist/{verify-ChyX6jq0.mjs → verify-CWlg3j7B.mjs} +153 -45
  82. package/dist/{verify-CwlombbC.cjs → verify-Dek6gHCS.cjs} +153 -45
  83. package/dist/{verify-i9Hiwc5g.d.mts → verify-H3Pr2gNw.d.mts} +42 -6
  84. package/dist/{with-model-calls-B_WLmOSl.cjs → with-model-calls-D50Ca6qW.cjs} +2 -2
  85. package/dist/{with-model-calls-BkOwdroS.mjs → with-model-calls-uY-KKliA.mjs} +2 -2
  86. package/docs/advanced.md +5 -5
  87. package/docs/choosing-a-run-mode.md +7 -7
  88. package/docs/debugging.md +6 -8
  89. package/docs/decisions.md +6 -5
  90. package/docs/environment-and-policy.md +1 -1
  91. package/docs/evals.md +57 -43
  92. package/docs/event-log.md +14 -16
  93. package/docs/from-a-loop.md +3 -3
  94. package/docs/generate-machines.md +4 -4
  95. package/docs/hosts.md +22 -29
  96. package/docs/human-in-the-loop.md +5 -8
  97. package/docs/index.md +4 -4
  98. package/docs/judgments.md +5 -7
  99. package/docs/langgraph-comparison.md +6 -6
  100. package/docs/machines-as-data.md +12 -17
  101. package/docs/machines-presets.md +17 -120
  102. package/docs/machines.md +13 -13
  103. package/docs/models-and-providers.md +19 -25
  104. package/docs/multi-agent.md +1 -1
  105. package/docs/observability.md +6 -8
  106. package/docs/persistence.md +15 -25
  107. package/docs/quickstart.md +4 -4
  108. package/docs/statechart-policy-examples.md +1 -1
  109. package/docs/steps.md +5 -6
  110. package/docs/strangler-fig.md +72 -7
  111. package/docs/text-requests.md +10 -12
  112. package/docs/thinking-in-state-machines.md +8 -8
  113. package/docs/tools.md +10 -12
  114. package/docs/upgrading.md +1 -1
  115. package/docs/usage-and-budgets.md +51 -56
  116. package/docs/validate-and-repair.md +9 -14
  117. package/docs/verify.md +53 -7
  118. package/docs/xstate-4-and-5-apps.md +19 -28
  119. package/llms.txt +1 -1
  120. package/package.json +31 -6
  121. package/readme.md +32 -24
  122. package/dist/internal-C3Lqu538.d.cts +0 -43
  123. package/dist/internal-DUcdhk-i.d.mts +0 -43
  124. package/dist/types-DQUOOaZO.d.cts +0 -238
  125. package/dist/types-DQUOOaZO.d.mts +0 -238
  126. package/skills/add-evals/SKILL.md +0 -152
  127. package/skills/add-traces/SKILL.md +0 -152
  128. package/skills/build-machine/SKILL.md +0 -84
  129. package/skills/generate-machine/SKILL.md +0 -335
  130. package/skills/graph-engineering/SKILL.md +0 -43
  131. package/skills/improve-machine/SKILL.md +0 -261
  132. package/skills/improve-machine/reference.md +0 -101
  133. package/skills/loop-to-machine/SKILL.md +0 -205
  134. package/skills/loop-to-machine/policy.md +0 -154
  135. package/skills/loop-to-machine/reference.md +0 -260
@@ -1,154 +0,0 @@
1
- # A policy machine
2
-
3
- A policy answers one decision of the environment. It is a `setupAgent` machine, so `describeMachine`, traces, and evals work on it, and `improve-machine` can change it one diff at a time.
4
-
5
- - **Input:** the facts it reads, the options the environment offered, and the environment's prompt. **Output:** one event, from a final state.
6
- - **It proposes; the environment disposes.** The environment's guards still judge whatever the policy returns, so the policy needs no copy of the rules to be safe. It may import the environment's schemas, types, constants, and exported guard functions to decide well, and nothing else from it.
7
- - **Its first version reproduces the current behavior.** When it still asks the model, it passes the environment's `system`, `prompt`, and `metadata.facts` through unchanged, so the model sees what it saw before.
8
- - **Declare `version`** and bump it on every change. Its `description` is the environment's goal with measurable success criteria; ask the user to confirm it.
9
- - **Where it lives:** one file next to the environment (`policy.ts`, or `policy/` with its `CHANGELOG.md`), holding the machine, `policyDecider`, and `withPolicy`.
10
-
11
- ## What the environment exports
12
-
13
- The event schemas and the facts schema, so the policy reuses them instead of redeclaring them. The `agent.decide` input passes the facts as `metadata: { facts }`. The facts must carry everything any decider reads: the policy's rules, and the adapter that turns a request into a call to your model (the ticket, the customer, the order's owner). A field the adapter needs and the facts lack changes what the model sees, so the policy no longer reproduces the current behavior.
14
-
15
- ```ts
16
- // In the environment file (exported there):
17
- // Everything a decider reads: the policy's rules, and the model adapter when the policy asks the model.
18
- export const factsSchema = z.object({
19
- ticket: z.string(),
20
- customerId: z.string(),
21
- order: z.object({ orderId: z.string(), customerId: z.string(), amount: z.number(), refunded: z.boolean() }).nullable(),
22
- requestedAmount: z.number().nullable(),
23
- });
24
- export const deskEvents = {
25
- REFUND: z.object({ amount: z.number() }),
26
- ESCALATE: z.object({}),
27
- CLOSE: z.object({}),
28
- };
29
- ```
30
-
31
- ## The policy
32
-
33
- ```ts
34
- import { z } from "zod";
35
- import { createPolicyDecider, setupAgent, type AgentDecisionExecutor, type AgentRequestExecutors } from "@statelyai/agent";
36
- import { deskEvents, factsSchema } from "./desk.js";
37
-
38
- /** One proposal: an environment event with its `type`. */
39
- const proposalSchema = z.discriminatedUnion("type", [
40
- deskEvents.REFUND.extend({ type: z.literal("REFUND") }),
41
- deskEvents.ESCALATE.extend({ type: z.literal("ESCALATE") }),
42
- deskEvents.CLOSE.extend({ type: z.literal("CLOSE") }),
43
- ]);
44
- type Proposal = z.infer<typeof proposalSchema>;
45
-
46
- const policySetup = setupAgent({
47
- input: z.object({
48
- facts: factsSchema,
49
- options: z.array(proposalSchema),
50
- // The environment's own prompt, passed through so the model sees what it saw before.
51
- system: z.string().optional(),
52
- prompt: z.string().optional(),
53
- }),
54
- context: z.object({
55
- facts: factsSchema,
56
- options: z.array(proposalSchema),
57
- system: z.string().optional(),
58
- prompt: z.string().optional(),
59
- chosen: proposalSchema.nullable(),
60
- }),
61
- output: proposalSchema,
62
- // Only the events the policy can propose, not every environment event.
63
- events: { REFUND: deskEvents.REFUND, ESCALATE: deskEvents.ESCALATE, CLOSE: deskEvents.CLOSE },
64
- });
65
-
66
- /** The offered options of one type, as `payloads` entries (`type` removed). */
67
- const payloadsOf = (options: readonly Proposal[], type: Proposal["type"]) =>
68
- options.filter((option) => option.type === type).map(({ type: _type, ...payload }) => payload);
69
-
70
- const offers = (options: readonly Proposal[], type: Proposal["type"]) =>
71
- options.some((option) => option.type === type);
72
-
73
- export const policyMachine = policySetup.createMachine({
74
- id: "resolution-policy",
75
- version: "1",
76
- description: "Pick one offered resolution per ticket. Success: the desk's outcome matches the expected outcome.",
77
- context: ({ input }) => ({ ...input, chosen: null }),
78
- initial: "checking",
79
- states: {
80
- // A rule ported from the old code: an already-refunded order closes, no model call.
81
- checking: {
82
- type: "choice",
83
- choice: ({ context }) =>
84
- context.facts.order?.refunded && offers(context.options, "CLOSE")
85
- ? { target: "chosen", context: { chosen: { type: "CLOSE" as const } } }
86
- : { target: "asking" },
87
- },
88
- asking: {
89
- invoke: {
90
- src: "agent.decide",
91
- input: ({ context }) => ({
92
- model: "decider",
93
- name: "pickResolution",
94
- system: context.system,
95
- prompt: context.prompt ?? `Facts: ${JSON.stringify(context.facts)}`,
96
- allowedEvents: ["REFUND", "ESCALATE", "CLOSE"],
97
- // Only what the environment offered; `[]` withholds a type.
98
- payloads: {
99
- REFUND: payloadsOf(context.options, "REFUND"),
100
- ESCALATE: payloadsOf(context.options, "ESCALATE"),
101
- CLOSE: payloadsOf(context.options, "CLOSE"),
102
- },
103
- // Same shape as the environment's request, so the model adapter reads the facts unchanged.
104
- metadata: { facts: context.facts },
105
- }),
106
- onError: { target: "gaveUp" },
107
- },
108
- on: {
109
- REFUND: ({ event }) => ({
110
- target: "chosen",
111
- context: { chosen: { type: "REFUND" as const, amount: event.amount } },
112
- }),
113
- ESCALATE: { target: "chosen", context: { chosen: { type: "ESCALATE" as const } } },
114
- CLOSE: { target: "chosen", context: { chosen: { type: "CLOSE" as const } } },
115
- },
116
- },
117
- chosen: { type: "final", output: ({ context }) => context.chosen! },
118
- gaveUp: { type: "final", output: { type: "ESCALATE" as const } },
119
- },
120
- });
121
-
122
- /**
123
- * Runs the policy as the environment's decider. createPolicyDecider passes the request's
124
- * `metadata.facts`, offered events, `system`, and `prompt` as the policy's input (its input
125
- * schema validates them), and returns the policy's output event with its token usage.
126
- */
127
- export function policyDecider(executors: AgentRequestExecutors): AgentDecisionExecutor {
128
- return createPolicyDecider(policyMachine, { executors });
129
- }
130
- ```
131
-
132
- - Every state that invokes `agent.decide` needs an `onError`, or `lintAgentMachine` warns. A policy that cannot choose falls back to the safest event the environment always accepts (here `ESCALATE`).
133
- - A rule the policy applies without a model is a `choice` state over the facts, checked against what was offered (`offers`), since the environment may have withheld it.
134
-
135
- ## Running it as the environment's decider
136
-
137
- ```ts
138
- /** The model's executors, with the policy answering the environment's `chooseResolution`. */
139
- export function withPolicy(modelExecutors: AgentRequestExecutors): AgentRequestExecutors {
140
- const decideWithPolicy = policyDecider(modelExecutors);
141
- return {
142
- ...modelExecutors,
143
- decide: (request, info) => {
144
- if (request.name === "chooseResolution") return decideWithPolicy(request, info);
145
- if (!modelExecutors.decide) throw new Error("no decide executor");
146
- return modelExecutors.decide(request, info);
147
- },
148
- };
149
- }
150
- ```
151
-
152
- Pass `withPolicy(modelExecutors)` as the environment's `executors` in the runner. That one line is the only runner change a policy needs. Decisions with any other name still reach the model.
153
-
154
- A runner that already takes a decide executor for the decision (`runMachine({ decide })`) needs no routing: pass `policyDecider(modelExecutors)` there. If your model is not an executor (a function that returns an option index, say), wrap it in a decide executor first, the same adapter the runner uses, and keep that adapter out of the environment's imports.
@@ -1,260 +0,0 @@
1
- # Convert agent code to a machine: reference
2
-
3
- Code and semantics behind [SKILL.md](SKILL.md). The rules are in the skill; this page is how.
4
-
5
- ## A gated action
6
-
7
- An `agent.decide` with a closed choice, and the rule as a guard in the transition:
8
-
9
- ```ts
10
- deciding: {
11
- tags: ["decision"],
12
- invoke: {
13
- src: "agent.decide",
14
- input: ({ context }) => ({
15
- model: "decider", // a key the host's executors resolve
16
- name: "chooseResolution", // tests, traces, and a later policy find it by this name
17
- prompt: `Ticket: ${context.ticket}\nOrder: ${JSON.stringify(context.order)}`,
18
- allowedEvents: ["REFUND", "ESCALATE", "CLOSE"],
19
- // A closed choice: the decider picks one of these. The runtime drops the ones the guards refuse.
20
- payloads: { REFUND: refundAmounts(context).map((amount) => ({ amount })) },
21
- // The facts a decider reads. Arrives as `request.metadata.facts`.
22
- metadata: { facts: { order: context.order, requestedAmount: context.requestedAmount } },
23
- }),
24
- onError: { target: "escalating" }, // retries exhausted, or no option legal
25
- },
26
- on: {
27
- REFUND: ({ context, event }) =>
28
- refundIsLegal(context, event.amount)
29
- ? { target: event.amount > APPROVAL_THRESHOLD ? "awaitingApproval" : "paying", context: { amount: event.amount } }
30
- : undefined, // undefined rejects: the decision retries with feedback
31
- ESCALATE: { target: "escalating" },
32
- CLOSE: { target: "closing" },
33
- },
34
- },
35
- ```
36
-
37
- ## Proof options
38
-
39
- ```ts
40
- const proofOptions: ExplorePathsOptions = {
41
- payloads: () => ({
42
- REFUND: [0, 40, 41, 100, 101, 240, 241, 800, 801].map((amount) => ({ amount })),
43
- CUSTOMER_REPLIED: [{ reply: "It's ORD-1001" }, { reply: "no idea" }],
44
- }),
45
- outputs: { lookupOrder: [ownedOrder, otherCustomersOrder, refundedOrder, { found: false }] },
46
- errors: { lookupOrder: new Error("down"), "agent.decide": new Error("exhausted") },
47
- };
48
- ```
49
-
50
- `errors[src]` takes one failure or an array of them (one failing branch per element; wrap an array-valued failure as `[[...]]`), and `"agent.decide"` is a valid key.
51
-
52
- ## A phase in the middle of a method
53
-
54
- When the phase has no seam at its edges, copy only its lines into the carved actor and mark the copy:
55
-
56
- ```ts
57
- // Copied verbatim from src/runtime.ts:412-470 (prepareToolCall) and :88-97 (normalizeArgs).
58
- // checkParity call-args fail if this copy drifts from the original.
59
- const prepare = createAsyncLogic<Prepared, ToolCall>({
60
- run: async ({ input }) => {
61
- /* copied lines */
62
- },
63
- });
64
- ```
65
-
66
- When a public per-turn hook exists (an `onTurnEnd` or `beforeToolCall` method), replace it on the instance instead. Each call hands that boundary's facts to the machine and awaits its answer, which an actor resolves; the rest of the method stays legacy. Race the legacy `run()` against the machine so exits that never reach the hook (stop, provider errors) still settle it.
67
-
68
- ## Old-suite swap
69
-
70
- The port imports the module the old suite mocks, so the factory must not import the port. Hold it in `vi.hoisted` and load it in `beforeAll`:
71
-
72
- ```ts
73
- import { beforeAll, vi } from "vitest";
74
-
75
- const ported = vi.hoisted(() => ({
76
- module: undefined as undefined | typeof import("../src/port"),
77
- }));
78
- vi.mock("../../src/runner", async (importOriginal) => ({
79
- ...(await importOriginal<typeof import("../../src/runner")>()),
80
- createRunner: (...args: Parameters<typeof import("../src/port").createRunner>) =>
81
- ported.module!.createRunner(...args),
82
- }));
83
- beforeAll(async () => {
84
- ported.module = await import("../src/port");
85
- });
86
- await import("../../src/runner.test"); // the old suite, unchanged
87
- ```
88
-
89
- When the mock misses the code's own import of the module, patch the prototype from a setup file. Both imports are dynamic, so the module loads after the test file's mocks:
90
-
91
- ```ts
92
- // old-suite.setup.ts, listed in the config's setupFiles
93
- import { beforeAll } from "vitest";
94
-
95
- beforeAll(async () => {
96
- const { Executor } = await import("../../src/executor");
97
- const { carvedRun } = await import("../src/port");
98
- Executor.prototype.run = function (this: InstanceType<typeof Executor>, ...args) {
99
- return carvedRun(this, ...args);
100
- };
101
- });
102
- ```
103
-
104
- ## Recording golden runs
105
-
106
- Wrap the model and tools, run the old code, save the recording. One recording per scenario and per stand-in. Record with the live model to capture real answers, or with a stand-in; either way the recording is the script for replay.
107
-
108
- ```ts
109
- import { createRecorder } from "@statelyai/agent/testing";
110
-
111
- const recorder = createRecorder();
112
- const model = recorder.model(openai("gpt-5")); // or recorder.openai(new OpenAI())
113
- const tools = recorder.tools(appTools); // AI SDK tools; recorder.tool(name, fn) for plain functions
114
- recorder.output(await runLoop({ ...scenario, model, tools }));
115
- writeFileSync(`fixtures/${scenario.id}.json`, JSON.stringify(recorder.recording));
116
- ```
117
-
118
- ### Effects-object workflows: `checkParity`
119
-
120
- When every side effect goes through one `effects` object of async functions, `checkParity` (`@statelyai/agent/testing`) does the record, replay, and compare above in one call: it wraps each effect with `recorder.fn`, records `before`, replays `after` with no effect running, and returns one row per scenario (`equivalent`, `divergences`, `before`/`after` as `{ output }` or `{ error }`, `recording`).
121
-
122
- ```ts
123
- const report = await checkParity({
124
- scenarios, // { id, input }[]
125
- effects: (scenario) => createFakeEffects(scenario),
126
- before: (input, effects) => oldWorkflow(input, effects),
127
- after: (input, effects) => step1.run(input, { effects }), // or a runAgent runner
128
- });
129
- expect(report.scenarios.filter((row) => !row.equivalent)).toEqual([]);
130
- ```
131
-
132
- `wrapWorkflow(...).run(input, { recorder })` records a golden run from a normal run.
133
-
134
- ### Any other model client
135
-
136
- `recorder.client(name, sdk, { methods })` wraps the named async methods of an SDK object and keeps its type; `recorder.fn(name, asyncFn)` wraps a client that is a function. Record the hand-rolled loop with the wrapped client, then put the same wrapper inside the machine's executor for replay:
137
-
138
- ```ts
139
- import Anthropic from "@anthropic-ai/sdk";
140
- import { runAgent } from "@statelyai/agent";
141
- import { compareRecordings, createRecorder } from "@statelyai/agent/testing";
142
-
143
- const methods = ["messages.create"];
144
-
145
- // Record: the hand-rolled loop gets the wrapped client.
146
- const recorder = createRecorder();
147
- recorder.output(await runLoop(recorder.client("anthropic", new Anthropic(), { methods }), scenario));
148
-
149
- // Replay: the same wrapper inside the machine's executor; the client is never called.
150
- const replay = createRecorder({ replay: recorder.recording });
151
- const client = replay.client("anthropic", {} as Anthropic, { methods });
152
- const run = await runAgent(machine, {
153
- input: scenario,
154
- executors: {
155
- // askModel: the old loop's call, moved unchanged.
156
- generateText: async (request) => ({ result: await askModel(client, request.messages as Anthropic.MessageParam[]) }),
157
- },
158
- });
159
- if (run.status === "done") replay.output(run.output);
160
- const { divergences } = compareRecordings(recorder.recording, replay.recording);
161
- expect(divergences).toEqual(INTENDED[scenario.id] ?? []); // the replay test fails on any unreviewed difference
162
- ```
163
-
164
- - Each call is a `call` effect named `<name>.<method>` (`anthropic.messages.create`) with its arguments and its result or error.
165
- - Replay answers each call with the next unused recorded call of the same name, in order, whatever its arguments; `compareRecordings` reports differing arguments as `call-args`.
166
- - A result that is an async iterable (a stream) records its chunks; replay returns an async iterable of them.
167
- - `redact(args)` records only what must match (model, offered tool names) and keeps secrets out. `serialize` maps a non-JSON result or chunk; `revive` rebuilds the replayed value, such as a stream object with helper methods.
168
-
169
- ## The parity harness
170
-
171
- Each run becomes a `ParityRun`: the outcome, every write the world records, and the rules broken. The harness check runs the old code on both sides:
172
-
173
- ```ts
174
- import { compareRuns, type ParityResult, type ParityRun } from "@statelyai/agent/testing";
175
-
176
- function toParityRun(run: { outcome: string; world: World }): ParityRun {
177
- return {
178
- outcome: run.outcome,
179
- // Every write the world records; model-written text as a count.
180
- effects: { refunds: run.world.refunds, approvals: run.world.approvals, verified: [...run.world.verified], escalations: run.world.escalations.length },
181
- violations: run.world.violations, // rules broken, from ground truth
182
- };
183
- }
184
-
185
- // A model factory, called once per scenario and per side. Construct a stateful or seeded
186
- // model inside it (seeded from s.id); returning a shared stateless object is fine.
187
- const runWith = (runner: typeof runLoop, model: (s: Scenario) => Model) => async (s: Scenario) =>
188
- toParityRun(await runner({ ...s, model: model(s), world: createWorld(s) }));
189
-
190
- const MODELS: Array<[string, (s: Scenario) => Model]> = [
191
- ["careful", () => carefulModel],
192
- ["sloppy", () => sloppyModel],
193
- ];
194
-
195
- test.each(MODELS)("harness: the loop matches itself (%s)", async (_, model) => {
196
- const report = await compareRuns({ scenarios: SCENARIOS, baseline: runWith(runLoop, model), candidate: runWith(runLoop, model) });
197
- for (const row of report.scenarios) expect(row.candidate, row.id).toEqual(row.baseline);
198
- });
199
- ```
200
-
201
- The parity test switches the candidate to `runMachine`:
202
-
203
- ```ts
204
- test.each(MODELS)("parity with the loop: %s", async (_, model) => {
205
- const report = await compareRuns({ scenarios: SCENARIOS, baseline: runWith(runLoop, model), candidate: runWith(runMachine, model) });
206
- const unreviewed = report.scenarios.filter((row) => row.label === "regression" && !REVIEWED[row.id]);
207
- expect(unreviewed).toEqual([]);
208
- // A reviewed difference still has to be exactly the difference a person accepted.
209
- for (const row of report.scenarios.filter((row) => REVIEWED[row.id])) REVIEWED[row.id]!(row);
210
- });
211
-
212
- // Effect-only differences a person accepted: who, and the exact difference.
213
- const REVIEWED: Record<string, (row: ParityResult<Scenario>) => void> = {
214
- // "no-order-id-no-reply": (row) => { /* accepted by <name>: asks once, not three times */ expect(row.candidate.outcome).toBe(row.baseline.outcome); ... },
215
- };
216
- ```
217
-
218
- With no model in the loop, a spec baseline replaces the old code's run, and the candidate runs in a scripted world (the proposed calls, the order approvals arrive in, which tool fails):
219
-
220
- ```ts
221
- // Confirmed by a person from the old code and its docs, before the machine existed.
222
- const SPEC: Record<string, ParityRun> = {
223
- /* "two-calls-one-rejected": { outcome: "completed", effects: { executed: ["read"], rejected: ["write"] }, violations: [] }, */
224
- };
225
-
226
- const report = await compareRuns({
227
- scenarios: SCENARIOS,
228
- baseline: async (s) => SPEC[s.id]!,
229
- candidate: async (s) => toParityRun(await runMachine({ ...s, world: createScriptedWorld(s) })),
230
- });
231
- ```
232
-
233
- ## Replay equivalence
234
-
235
- Replay answers each model and tool call from the recording, so the machine needs no key and causes no side effects:
236
-
237
- ```ts
238
- import { compareRecordings, createRecorder, type EffectDivergence, type Recording } from "@statelyai/agent/testing";
239
-
240
- test.each(SCENARIOS)("equivalent to the recorded loop: $id", async (scenario) => {
241
- const recording: Recording = JSON.parse(readFileSync(`fixtures/${scenario.id}.json`, "utf8"));
242
- const replay = createRecorder({ replay: recording });
243
- const model = replay.model(openai("gpt-5")); // never called in replay
244
- replay.output(await runMachine({ ...scenario, model, tools: replay.tools(appTools) }));
245
- const report = compareRecordings(recording, replay.recording);
246
- expect(report.divergences.map(({ reason, message }) => ({ reason, message }))).toEqual(INTENDED[scenario.id] ?? []);
247
- });
248
-
249
- // Differences a person accepted: who, and the exact divergences.
250
- const INTENDED: Record<string, Array<Pick<EffectDivergence, "reason" | "message">>> = {};
251
- ```
252
-
253
- - Equivalent means the same model calls (count and shape: offered tool names, response format), the same tool calls with the same arguments, and the same output. Prompts and states may differ. Tool calls that ran at the same time compare in any order; sequential ones keep theirs.
254
- - Values that differ on every run (generated ids, branch names, timestamps) fail `tool-input`. Pass the same `normalize` to `createRecorder({ replay, normalize })` and `compareRecordings(a, b, { normalize })` to redact them on both sides; never edit the fixtures. `normalize(effect, index, effects)` returns `undefined` to leave an effect out of `compareRecordings`, such as a repeat of an idempotent write; replay still answers from it. Drop an effect only when repeating it is harmless: a dropped `send` or `charge` hides a duplicate.
255
- - `divergences` lists every difference, lined up so one extra call does not shift the rest; `divergence` is the first.
256
- - Replay answers each model call with the earliest unused recorded call of the same method and shape, and each tool call with the earliest unused recorded call of the same name and input, not by position. A call the machine added or dropped does not shift the answers after it, as long as no other call has the same method and shape (or tool name and input). Calls that do are answered in recorded order, so an added or dropped one takes or leaves the next one's answer.
257
- - A call no recorded answer fits throws `RecordingDivergenceError`, and `compareRecordings` names it. To run the machine's own path instead (its retry, its added check), pass `fallback: { model: standIn, tools: true }`: a stand-in model answers the call and the wrapped tool runs. Those effects are marked `unrecorded` and still count as extra.
258
- - `model-call-shape`: the machine changed a call's tools or output format, for example a model-picked action that became `agent.decide`. Port the call unchanged first, prove equivalence, then change it in a separate step whose divergences are `INTENDED`. To keep replaying past a call you changed on purpose, pass `translate(recorded, actual)` to `createRecorder`: it returns the answer that fits the new call (`{ response }`, `{ stream }`, or `{ error }`), or `undefined` to leave that recorded call alone, and `compareRecordings` counts the pair as the same call. When the recorded answer already fits, a `modelCalls` function passed to both is enough: `(expected, actual) => expected.call === 2 || sameModelCallShape(expected, actual)`.
259
- - When the machine asks the model a different way altogether (another protocol, one structured draft instead of a tool loop), compare what it did, not how it asked: `compareRecordings(a, b, { kinds: ["tool", "output"] })`, with the machine recorded on a stand-in model.
260
- - A guard that blocks a rule break diverges on purpose. List it in `INTENDED` with the exact divergence.