@statelyai/agent 2.0.0-alpha.38 → 2.0.0-alpha.40

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (137) hide show
  1. package/CHANGELOG.md +59 -0
  2. package/dist/{act-CrX0JwWY.cjs → act-B0CVbmo4.cjs} +1 -1
  3. package/dist/{act-B9hLQls6.mjs → act-Bsz-hUfD.mjs} +1 -1
  4. package/dist/ai-sdk.cjs +4 -4
  5. package/dist/ai-sdk.d.cts +2 -3
  6. package/dist/ai-sdk.d.mts +2 -3
  7. package/dist/ai-sdk.mjs +4 -4
  8. package/dist/{decision-m5hGmHpF.mjs → decision-B41YBx5U.mjs} +1 -1
  9. package/dist/{decision-CMeuNZ1g.cjs → decision-D8bcIGXQ.cjs} +1 -1
  10. package/dist/{decision-object-DBcLWVvv.d.cts → decision-object-BB8XV5Ec.d.cts} +1 -1
  11. package/dist/{decision-object-B5zGSPFp.d.mts → decision-object-CH_sE694.d.mts} +1 -1
  12. package/dist/effects-B9IvUPVh.mjs +23 -0
  13. package/dist/effects-DrWcUGgc.cjs +34 -0
  14. package/dist/evals.cjs +810 -0
  15. package/dist/evals.d.cts +595 -0
  16. package/dist/evals.d.mts +595 -0
  17. package/dist/evals.mjs +801 -0
  18. package/dist/{event-log-DkVlk3Rh.mjs → event-log-CEq_bVXQ.mjs} +8 -4
  19. package/dist/{event-log-Dkc-yIYu.cjs → event-log-dSOnXHuG.cjs} +7 -15
  20. package/dist/{event-log-store-Brx9tOHt.mjs → event-log-store-DLAfNM34.mjs} +1 -1
  21. package/dist/{event-log-store-BBS8so8a.d.mts → event-log-store-DYun526m.d.mts} +15 -14
  22. package/dist/{event-log-store-Cl-YWDnq.cjs → event-log-store-Dey9ufA4.cjs} +1 -1
  23. package/dist/{event-log-store-bc4ICGcS.d.cts → event-log-store-aFXK6yZX.d.cts} +15 -14
  24. package/dist/index.cjs +142 -97
  25. package/dist/index.d.cts +97 -141
  26. package/dist/index.d.mts +97 -141
  27. package/dist/index.mjs +123 -74
  28. package/dist/internal-BzlpzSux.d.cts +14 -0
  29. package/dist/internal-BzlpzSux.d.mts +14 -0
  30. package/dist/log.cjs +2 -3
  31. package/dist/log.d.cts +2 -2
  32. package/dist/log.d.mts +2 -2
  33. package/dist/log.mjs +3 -3
  34. package/dist/machine-input-0YTPcnCm.mjs +27 -0
  35. package/dist/machine-input-DU3owLUr.cjs +32 -0
  36. package/dist/machines.cjs +1 -719
  37. package/dist/machines.d.cts +2 -424
  38. package/dist/machines.d.mts +2 -424
  39. package/dist/machines.mjs +2 -709
  40. package/dist/{model-source-kjGwqAV8.mjs → model-source-BrsLDvHL.mjs} +1 -1
  41. package/dist/{model-source-B4WQ8Idp.cjs → model-source-YpxjmWuX.cjs} +1 -1
  42. package/dist/openai.cjs +3 -3
  43. package/dist/openai.d.cts +2 -3
  44. package/dist/openai.d.mts +2 -3
  45. package/dist/openai.mjs +3 -3
  46. package/dist/otel.d.cts +1 -1
  47. package/dist/otel.d.mts +1 -1
  48. package/dist/policy.cjs +77 -0
  49. package/dist/policy.d.cts +143 -0
  50. package/dist/policy.d.mts +143 -0
  51. package/dist/policy.mjs +75 -0
  52. package/dist/{run-agent-DlPzBWj-.d.mts → run-agent-BO9RSb7i.d.cts} +33 -40
  53. package/dist/{run-agent-30mH2moE.d.cts → run-agent-BxkeuAYR.d.mts} +33 -40
  54. package/dist/{effects-DDseYtJk.cjs → run-agent-COdR9FpM.cjs} +175 -228
  55. package/dist/{effects-BT_cSY0j.mjs → run-agent-D3gW9wfd.mjs} +176 -211
  56. package/dist/{setup-agent-BDmNr6PC.mjs → setup-agent-CLm6QVGy.mjs} +3 -3
  57. package/dist/{setup-agent-BaNyz6Q3.d.cts → setup-agent-Cv8Hk3mX.d.cts} +12 -11
  58. package/dist/{setup-agent-C1CG_nfX.d.mts → setup-agent-DgFAhaPJ.d.mts} +12 -11
  59. package/dist/{setup-agent-CmEToB8y.cjs → setup-agent-Dv5VpuHe.cjs} +3 -3
  60. package/dist/tanstack-ai.cjs +4 -4
  61. package/dist/tanstack-ai.d.cts +2 -3
  62. package/dist/tanstack-ai.d.mts +2 -3
  63. package/dist/tanstack-ai.mjs +4 -4
  64. package/dist/testing.cjs +122 -996
  65. package/dist/testing.d.cts +36 -695
  66. package/dist/testing.d.mts +36 -695
  67. package/dist/testing.mjs +112 -977
  68. package/dist/{text-logic-BqxalgMr.d.mts → text-logic-x17uLBwA.d.cts} +238 -2
  69. package/dist/{text-logic-CSf9bqpP.d.cts → text-logic-x17uLBwA.d.mts} +238 -2
  70. package/dist/trajectory-BFjXaooB.d.cts +96 -0
  71. package/dist/trajectory-BFjXaooB.d.mts +96 -0
  72. package/dist/trajectory-BUXj_1Ue.mjs +152 -0
  73. package/dist/trajectory-CoZSv4wF.cjs +157 -0
  74. package/dist/{usage-Bm5nvpow.d.cts → usage-B4B-TQSV.d.mts} +1 -1
  75. package/dist/{usage-BnikdOca.d.mts → usage-DvGcoK6x.d.cts} +1 -1
  76. package/dist/{utils-7lSa4H-i.d.cts → utils-B8k-YPYI.d.mts} +23 -17
  77. package/dist/{utils-CBbKw2fM.d.mts → utils-DAclF-kw.d.cts} +23 -17
  78. package/dist/validate.cjs +4 -4
  79. package/dist/validate.d.cts +2 -2
  80. package/dist/validate.d.mts +2 -2
  81. package/dist/validate.mjs +4 -4
  82. package/dist/{verify-DJHfEfm6.d.cts → verify-CIu4mqIZ.d.mts} +56 -6
  83. package/dist/{verify-i9Hiwc5g.d.mts → verify-CTHlmBbh.d.cts} +56 -6
  84. package/dist/{verify-ChyX6jq0.mjs → verify-Cacz6XSk.mjs} +170 -46
  85. package/dist/{verify-CwlombbC.cjs → verify-D8UDqU4l.cjs} +170 -46
  86. package/dist/{with-model-calls-B_WLmOSl.cjs → with-model-calls-D50Ca6qW.cjs} +2 -2
  87. package/dist/{with-model-calls-BkOwdroS.mjs → with-model-calls-uY-KKliA.mjs} +2 -2
  88. package/docs/advanced.md +5 -5
  89. package/docs/choosing-a-run-mode.md +7 -7
  90. package/docs/debugging.md +6 -8
  91. package/docs/decisions.md +5 -4
  92. package/docs/environment-and-policy.md +1 -1
  93. package/docs/evals.md +57 -43
  94. package/docs/event-log.md +13 -15
  95. package/docs/from-a-loop.md +3 -3
  96. package/docs/generate-machines.md +4 -4
  97. package/docs/hosts.md +25 -30
  98. package/docs/human-in-the-loop.md +5 -8
  99. package/docs/index.md +4 -4
  100. package/docs/judgments.md +5 -7
  101. package/docs/langgraph-comparison.md +6 -6
  102. package/docs/machines-as-data.md +7 -12
  103. package/docs/machines-presets.md +17 -120
  104. package/docs/machines.md +13 -13
  105. package/docs/models-and-providers.md +19 -25
  106. package/docs/multi-agent.md +1 -1
  107. package/docs/observability.md +6 -8
  108. package/docs/persistence.md +10 -20
  109. package/docs/quickstart.md +3 -3
  110. package/docs/statechart-policy-examples.md +1 -1
  111. package/docs/steps.md +5 -6
  112. package/docs/strangler-fig.md +105 -33
  113. package/docs/text-requests.md +10 -12
  114. package/docs/thinking-in-state-machines.md +8 -8
  115. package/docs/tools.md +10 -12
  116. package/docs/upgrading.md +1 -1
  117. package/docs/usage-and-budgets.md +51 -56
  118. package/docs/validate-and-repair.md +9 -14
  119. package/docs/verify.md +54 -7
  120. package/docs/xstate-4-and-5-apps.md +19 -28
  121. package/llms.txt +1 -1
  122. package/package.json +31 -6
  123. package/readme.md +32 -24
  124. package/dist/internal-C3Lqu538.d.cts +0 -43
  125. package/dist/internal-DUcdhk-i.d.mts +0 -43
  126. package/dist/types-DQUOOaZO.d.cts +0 -238
  127. package/dist/types-DQUOOaZO.d.mts +0 -238
  128. package/skills/add-evals/SKILL.md +0 -152
  129. package/skills/add-traces/SKILL.md +0 -152
  130. package/skills/build-machine/SKILL.md +0 -84
  131. package/skills/generate-machine/SKILL.md +0 -335
  132. package/skills/graph-engineering/SKILL.md +0 -43
  133. package/skills/improve-machine/SKILL.md +0 -261
  134. package/skills/improve-machine/reference.md +0 -101
  135. package/skills/loop-to-machine/SKILL.md +0 -205
  136. package/skills/loop-to-machine/policy.md +0 -154
  137. package/skills/loop-to-machine/reference.md +0 -260
@@ -1,238 +0,0 @@
1
- //#region src/types.d.ts
2
- /**
3
- * The [Standard Schema](https://standardschema.dev) interface. Every schema
4
- * this library accepts (context, events, input/output, tool schemas, …) is a
5
- * `StandardSchemaV1` — Zod, Valibot, ArkType, and hand-written validators all
6
- * implement it, so the library never depends on a specific validation
7
- * library. JSON workflow configs use a caller-provided {@link SchemaCompiler}.
8
- */
9
- interface StandardSchemaV1<Input = unknown, Output = Input> {
10
- readonly "~standard": {
11
- readonly version: 1;
12
- readonly vendor: string;
13
- readonly validate: (value: unknown) => any;
14
- readonly types?: {
15
- readonly input: Input;
16
- readonly output: Output;
17
- };
18
- readonly jsonSchema?: {
19
- readonly input?: (...args: any[]) => unknown;
20
- readonly output?: (...args: any[]) => unknown;
21
- };
22
- };
23
- }
24
- /** The validated output type of a {@link StandardSchemaV1}. */
25
- type InferOutput<T> = T extends StandardSchemaV1<any, infer O> ? O : never;
26
- /**
27
- * The *pre*-validation input type of a {@link StandardSchemaV1}: what a caller
28
- * passes in, before defaults are filled and transforms applied. Standard Schema
29
- * carries both sides (`~standard.types.input` / `.output`), so a schema
30
- * declaring a defaulted field makes that field optional here and required in
31
- * {@link InferOutput} — which is exactly the split between what a run
32
- * accepts as machine `input` and what the `context` factory then sees.
33
- */
34
- type InferInput<T> = T extends StandardSchemaV1<infer I, any> ? I : never;
35
- /**
36
- * Phantom brand carrying a machine's declared input schema on the machine type.
37
- *
38
- * XState resolves `schemas.input` to a single type and uses it for both
39
- * `createActor`'s `input` option and the `context: ({ input })` factory, so the
40
- * caller-facing and factory-facing sides cannot differ there. `setupAgent`'s
41
- * `createMachine` brands the machine's input type with the schema itself, which
42
- * lets `AgentInputFrom` recover the looser input side for a run's `input` while the
43
- * `context` factory keeps the strict validated side.
44
- *
45
- * The key is a `~`-prefixed phantom property (the same convention Standard
46
- * Schema uses for `~standard`) rather than a `unique symbol`: a symbol would
47
- * have to be exported as a runtime value for declaration emit to name it in
48
- * every machine type it touches.
49
- */
50
- type WithAgentInputSchema<TInputSchema> = {
51
- readonly "~agent.inputSchema"?: TInputSchema;
52
- };
53
- /** An event schema's output, widened to `unknown` when it validates an empty object (no payload fields). */
54
- type EventPayload<T> = T extends Record<string, never> ? unknown : T;
55
- /**
56
- * One entry in an event schema map: a Standard Schema for the event's
57
- * payload, or the `{}` shorthand for a payload-less event
58
- * (`events: { CONFIRM: {} }` ≡ `events: { CONFIRM: z.object({}) }`).
59
- */
60
- type AgentEventSchemaInput = StandardSchemaV1 | Record<string, never>;
61
- /** An event schema map as authored: payload schemas and/or `{}` payload-less shorthands, keyed by event type. */
62
- type AgentEventSchemaInputMap = Record<string, AgentEventSchemaInput>;
63
- /** Resolves an authored event schema map's `{}` shorthands to real (empty-payload) schemas — the type-level counterpart of the runtime normalization in `createAgentSchemas`. */
64
- type NormalizedEventSchemas<T extends AgentEventSchemaInputMap> = { [K in keyof T]: T[K] extends StandardSchemaV1 ? T[K] : StandardSchemaV1<{}>; };
65
- /**
66
- * The discriminated event union derived from a machine's event schema map
67
- * (e.g. `{ ASK: z.object({ question: z.string() }) }` → `{ type: 'ASK';
68
- * question: string }`; a `{}` shorthand entry yields its bare `{ type: K }`).
69
- * Used internally by {@link createAgentSchemas} and `setupAgent` to type a
70
- * machine's `TEvent`.
71
- */
72
- type EventUnion<T extends AgentEventSchemaInputMap> = { [K in keyof T & string]: {
73
- type: K;
74
- } & (T[K] extends StandardSchemaV1 ? EventPayload<InferOutput<T[K]>> : unknown); }[keyof T & string];
75
- /** Raw binary or string content for an {@link ImagePart}/{@link FilePart}. */
76
- type DataContent = string | Uint8Array | ArrayBuffer;
77
- /** Provider-specific passthrough options, keyed by provider name (e.g. `{ anthropic: { cacheControl: ... } }`). */
78
- type ProviderOptions = Record<string, Record<string, unknown>>;
79
- /** A plain-text segment of a multi-part {@link AgentMessage} content array. */
80
- interface TextPart {
81
- type: "text";
82
- text: string;
83
- providerOptions?: ProviderOptions;
84
- }
85
- /**
86
- * Binary (`Uint8Array`/`ArrayBuffer`) and `URL` values are not
87
- * JSON-serializable. Machines that persist snapshots/event logs should use
88
- * URL strings or base64-encoded strings in `image` instead.
89
- */
90
- interface ImagePart {
91
- type: "image";
92
- image: DataContent | URL;
93
- mediaType?: string;
94
- providerOptions?: ProviderOptions;
95
- }
96
- /**
97
- * Binary (`Uint8Array`/`ArrayBuffer`) and `URL` values are not
98
- * JSON-serializable. Machines that persist snapshots/event logs should use
99
- * URL strings or base64-encoded strings in `data` instead.
100
- */
101
- interface FilePart {
102
- type: "file";
103
- data: DataContent | URL;
104
- mediaType: string;
105
- filename?: string;
106
- providerOptions?: ProviderOptions;
107
- }
108
- /** A model-issued tool call, as an {@link AssistantMessage} content part. */
109
- interface ToolCallPart {
110
- type: "tool-call";
111
- toolCallId: string;
112
- toolName: string;
113
- input: unknown;
114
- providerOptions?: ProviderOptions;
115
- }
116
- /** The result payload of a {@link ToolResultPart}, discriminated by shape (plain text/JSON, or an error variant of either). */
117
- type ToolResultOutput = {
118
- type: "text";
119
- value: string;
120
- } | {
121
- type: "json";
122
- value: unknown;
123
- } | {
124
- type: "error-text";
125
- value: string;
126
- } | {
127
- type: "error-json";
128
- value: unknown;
129
- } | {
130
- type: "content";
131
- value: Array<TextPart | ImagePart>;
132
- };
133
- /** A tool's result, as a {@link ToolMessage} content part. */
134
- interface ToolResultPart {
135
- type: "tool-result";
136
- toolCallId: string;
137
- toolName: string;
138
- output: ToolResultOutput;
139
- providerOptions?: ProviderOptions;
140
- }
141
- /** A system-role {@link AgentMessage}. Create with {@link systemMessage}. */
142
- type SystemMessage = {
143
- role: "system";
144
- content: string;
145
- providerOptions?: ProviderOptions;
146
- };
147
- /** A user-role {@link AgentMessage}, optionally multimodal. Create with {@link userMessage}. */
148
- type UserMessage = {
149
- role: "user";
150
- content: string | Array<TextPart | ImagePart | FilePart>;
151
- providerOptions?: ProviderOptions;
152
- };
153
- /** An assistant-role {@link AgentMessage}, which may carry tool calls/results inline. Create with {@link assistantMessage}. */
154
- type AssistantMessage = {
155
- role: "assistant";
156
- content: string | Array<TextPart | FilePart | ToolCallPart | ToolResultPart>;
157
- providerOptions?: ProviderOptions;
158
- };
159
- /** A tool-role {@link AgentMessage} carrying one or more tool results. Create with {@link toolMessage}. */
160
- type ToolMessage = {
161
- role: "tool";
162
- content: Array<ToolResultPart>;
163
- providerOptions?: ProviderOptions;
164
- };
165
- /**
166
- * Optional framework-neutral message helpers. Requests also accept native
167
- * framework message values directly; core never converts between formats.
168
- * Store messages as plain context state: a text request's `onDone` output
169
- * carries the executor's response messages as `output.messages`.
170
- */
171
- type AgentMessage = SystemMessage | UserMessage | AssistantMessage | ToolMessage;
172
- /**
173
- * A schema value on an {@link AgentToolDescriptor}: deliberately just `object`,
174
- * because an SDK-native tool's `inputSchema` is the SDK's own union type (a Zod
175
- * schema, the SDK's `Schema`, a lazy thunk, ...) and must assign structurally
176
- * with no cast. The contract is runtime, not static: core narrows with
177
- * `isStandardSchema` and reads {@link StandardSchemaV1} schemas via
178
- * `getJsonSchema`; anything else passes through to the executor untouched.
179
- */
180
- type AgentToolSchema = object;
181
- /**
182
- * A tool exposed to a text request, described for both the model and
183
- * (optionally) host execution. This is a **minimal structural contract**: any
184
- * object matching it — an AI SDK `tool({...})`, an MCP-style descriptor, or a
185
- * hand-written `{ description, inputSchema, execute }` — is a valid entry, and
186
- * extra properties (`providerOptions`, `toModelOutput`, …) pass through
187
- * untouched via the index signature. `execute` is typed permissively so a
188
- * native tool's `(input, options)` executor is structurally assignable; the
189
- * SDK you built the tool with owns its precise input typing.
190
- */
191
- interface AgentToolDescriptor {
192
- /**
193
- * Text shown to the model. Typed loosely enough to accept an SDK tool whose
194
- * description is computed per call (the AI SDK v7 `tool({ description })`
195
- * accepts a function of the call's context); core never reads it.
196
- */
197
- description?: string | ((...args: any[]) => string);
198
- inputSchema?: AgentToolSchema;
199
- outputSchema?: AgentToolSchema;
200
- execute?: (...args: any[]) => unknown;
201
- [key: string]: unknown;
202
- }
203
- /** A bare tool implementation (no description/schema) — shorthand for {@link AgentToolDescriptor.execute}. */
204
- type AgentToolExecute = (input?: unknown) => unknown | Promise<unknown>;
205
- /** A tool entry in {@link AgentTools}: either a full descriptor or a bare execute function. */
206
- type AgentTool = AgentToolDescriptor | AgentToolExecute;
207
- /** The `tools` map passed on an {@link AgentTextRequest}, keyed by tool name. */
208
- type AgentTools = Record<string, AgentTool | undefined>;
209
- /** How a text request's model should select among its `tools`; `{ type: 'tool', name }` forces one specific tool. */
210
- type AgentToolChoice = "auto" | "none" | "required" | {
211
- type: "tool";
212
- name: string;
213
- };
214
- /** The event chosen and raised by a decision. */
215
- type ChosenEvent = {
216
- type: string;
217
- [key: string]: unknown;
218
- };
219
- type EventWildcardsOf<TEvent extends string> = TEvent extends `${infer Head}.${infer Rest}` ? `${Head}.*` | `${Head}.${EventWildcardsOf<Rest>}` : never;
220
- /** One `allowedEvents` entry: an exact declared event type, `'*'` (every event), or a `'prefix.*'` wildcard derived from the declared dotted event types. */
221
- type AllowedEventPattern<TEvent extends string = string> = TEvent | "*" | EventWildcardsOf<TEvent>;
222
- /**
223
- * Candidate event types for a decision (declared on the `agent.decide`
224
- * builtin's `allowedEvents` input). A single
225
- * entry or an array; entries are exact event types or wildcard patterns
226
- * (`'*'` for every event, `'todo.*'` for a dotted namespace). The effective
227
- * candidate set offered to the model is this declaration **intersected with
228
- * the snapshot's currently-legal events** (via `getAcceptedEvents`) —
229
- * omitting `allowedEvents` means "all currently-legal events." A resolver
230
- * function can therefore only ever narrow, never widen, the real surface.
231
- * Wildcards expand against the live snapshot, so they need a snapshot-aware
232
- * host (`createAgentRuntime`).
233
- */
234
- type AllowedEvents<TEvent extends string = string, TInput = unknown> = AllowedEventPattern<TEvent> | readonly AllowedEventPattern<TEvent>[] | ((args: {
235
- input: TInput;
236
- }) => AllowedEventPattern<TEvent> | readonly AllowedEventPattern<TEvent>[]);
237
- //#endregion
238
- export { UserMessage as A, StandardSchemaV1 as C, ToolMessage as D, ToolCallPart as E, ToolResultOutput as O, ProviderOptions as S, TextPart as T, FilePart as _, AgentToolChoice as a, InferOutput as b, AgentToolSchema as c, AllowedEvents as d, AssistantMessage as f, EventUnion as g, EventPayload as h, AgentTool as i, WithAgentInputSchema as j, ToolResultPart as k, AgentTools as l, DataContent as m, AgentEventSchemaInputMap as n, AgentToolDescriptor as o, ChosenEvent as p, AgentMessage as r, AgentToolExecute as s, AgentEventSchemaInput as t, AllowedEventPattern as u, ImagePart as v, SystemMessage as w, NormalizedEventSchemas as x, InferInput as y };
@@ -1,238 +0,0 @@
1
- //#region src/types.d.ts
2
- /**
3
- * The [Standard Schema](https://standardschema.dev) interface. Every schema
4
- * this library accepts (context, events, input/output, tool schemas, …) is a
5
- * `StandardSchemaV1` — Zod, Valibot, ArkType, and hand-written validators all
6
- * implement it, so the library never depends on a specific validation
7
- * library. JSON workflow configs use a caller-provided {@link SchemaCompiler}.
8
- */
9
- interface StandardSchemaV1<Input = unknown, Output = Input> {
10
- readonly "~standard": {
11
- readonly version: 1;
12
- readonly vendor: string;
13
- readonly validate: (value: unknown) => any;
14
- readonly types?: {
15
- readonly input: Input;
16
- readonly output: Output;
17
- };
18
- readonly jsonSchema?: {
19
- readonly input?: (...args: any[]) => unknown;
20
- readonly output?: (...args: any[]) => unknown;
21
- };
22
- };
23
- }
24
- /** The validated output type of a {@link StandardSchemaV1}. */
25
- type InferOutput<T> = T extends StandardSchemaV1<any, infer O> ? O : never;
26
- /**
27
- * The *pre*-validation input type of a {@link StandardSchemaV1}: what a caller
28
- * passes in, before defaults are filled and transforms applied. Standard Schema
29
- * carries both sides (`~standard.types.input` / `.output`), so a schema
30
- * declaring a defaulted field makes that field optional here and required in
31
- * {@link InferOutput} — which is exactly the split between what a run
32
- * accepts as machine `input` and what the `context` factory then sees.
33
- */
34
- type InferInput<T> = T extends StandardSchemaV1<infer I, any> ? I : never;
35
- /**
36
- * Phantom brand carrying a machine's declared input schema on the machine type.
37
- *
38
- * XState resolves `schemas.input` to a single type and uses it for both
39
- * `createActor`'s `input` option and the `context: ({ input })` factory, so the
40
- * caller-facing and factory-facing sides cannot differ there. `setupAgent`'s
41
- * `createMachine` brands the machine's input type with the schema itself, which
42
- * lets `AgentInputFrom` recover the looser input side for a run's `input` while the
43
- * `context` factory keeps the strict validated side.
44
- *
45
- * The key is a `~`-prefixed phantom property (the same convention Standard
46
- * Schema uses for `~standard`) rather than a `unique symbol`: a symbol would
47
- * have to be exported as a runtime value for declaration emit to name it in
48
- * every machine type it touches.
49
- */
50
- type WithAgentInputSchema<TInputSchema> = {
51
- readonly "~agent.inputSchema"?: TInputSchema;
52
- };
53
- /** An event schema's output, widened to `unknown` when it validates an empty object (no payload fields). */
54
- type EventPayload<T> = T extends Record<string, never> ? unknown : T;
55
- /**
56
- * One entry in an event schema map: a Standard Schema for the event's
57
- * payload, or the `{}` shorthand for a payload-less event
58
- * (`events: { CONFIRM: {} }` ≡ `events: { CONFIRM: z.object({}) }`).
59
- */
60
- type AgentEventSchemaInput = StandardSchemaV1 | Record<string, never>;
61
- /** An event schema map as authored: payload schemas and/or `{}` payload-less shorthands, keyed by event type. */
62
- type AgentEventSchemaInputMap = Record<string, AgentEventSchemaInput>;
63
- /** Resolves an authored event schema map's `{}` shorthands to real (empty-payload) schemas — the type-level counterpart of the runtime normalization in `createAgentSchemas`. */
64
- type NormalizedEventSchemas<T extends AgentEventSchemaInputMap> = { [K in keyof T]: T[K] extends StandardSchemaV1 ? T[K] : StandardSchemaV1<{}>; };
65
- /**
66
- * The discriminated event union derived from a machine's event schema map
67
- * (e.g. `{ ASK: z.object({ question: z.string() }) }` → `{ type: 'ASK';
68
- * question: string }`; a `{}` shorthand entry yields its bare `{ type: K }`).
69
- * Used internally by {@link createAgentSchemas} and `setupAgent` to type a
70
- * machine's `TEvent`.
71
- */
72
- type EventUnion<T extends AgentEventSchemaInputMap> = { [K in keyof T & string]: {
73
- type: K;
74
- } & (T[K] extends StandardSchemaV1 ? EventPayload<InferOutput<T[K]>> : unknown); }[keyof T & string];
75
- /** Raw binary or string content for an {@link ImagePart}/{@link FilePart}. */
76
- type DataContent = string | Uint8Array | ArrayBuffer;
77
- /** Provider-specific passthrough options, keyed by provider name (e.g. `{ anthropic: { cacheControl: ... } }`). */
78
- type ProviderOptions = Record<string, Record<string, unknown>>;
79
- /** A plain-text segment of a multi-part {@link AgentMessage} content array. */
80
- interface TextPart {
81
- type: "text";
82
- text: string;
83
- providerOptions?: ProviderOptions;
84
- }
85
- /**
86
- * Binary (`Uint8Array`/`ArrayBuffer`) and `URL` values are not
87
- * JSON-serializable. Machines that persist snapshots/event logs should use
88
- * URL strings or base64-encoded strings in `image` instead.
89
- */
90
- interface ImagePart {
91
- type: "image";
92
- image: DataContent | URL;
93
- mediaType?: string;
94
- providerOptions?: ProviderOptions;
95
- }
96
- /**
97
- * Binary (`Uint8Array`/`ArrayBuffer`) and `URL` values are not
98
- * JSON-serializable. Machines that persist snapshots/event logs should use
99
- * URL strings or base64-encoded strings in `data` instead.
100
- */
101
- interface FilePart {
102
- type: "file";
103
- data: DataContent | URL;
104
- mediaType: string;
105
- filename?: string;
106
- providerOptions?: ProviderOptions;
107
- }
108
- /** A model-issued tool call, as an {@link AssistantMessage} content part. */
109
- interface ToolCallPart {
110
- type: "tool-call";
111
- toolCallId: string;
112
- toolName: string;
113
- input: unknown;
114
- providerOptions?: ProviderOptions;
115
- }
116
- /** The result payload of a {@link ToolResultPart}, discriminated by shape (plain text/JSON, or an error variant of either). */
117
- type ToolResultOutput = {
118
- type: "text";
119
- value: string;
120
- } | {
121
- type: "json";
122
- value: unknown;
123
- } | {
124
- type: "error-text";
125
- value: string;
126
- } | {
127
- type: "error-json";
128
- value: unknown;
129
- } | {
130
- type: "content";
131
- value: Array<TextPart | ImagePart>;
132
- };
133
- /** A tool's result, as a {@link ToolMessage} content part. */
134
- interface ToolResultPart {
135
- type: "tool-result";
136
- toolCallId: string;
137
- toolName: string;
138
- output: ToolResultOutput;
139
- providerOptions?: ProviderOptions;
140
- }
141
- /** A system-role {@link AgentMessage}. Create with {@link systemMessage}. */
142
- type SystemMessage = {
143
- role: "system";
144
- content: string;
145
- providerOptions?: ProviderOptions;
146
- };
147
- /** A user-role {@link AgentMessage}, optionally multimodal. Create with {@link userMessage}. */
148
- type UserMessage = {
149
- role: "user";
150
- content: string | Array<TextPart | ImagePart | FilePart>;
151
- providerOptions?: ProviderOptions;
152
- };
153
- /** An assistant-role {@link AgentMessage}, which may carry tool calls/results inline. Create with {@link assistantMessage}. */
154
- type AssistantMessage = {
155
- role: "assistant";
156
- content: string | Array<TextPart | FilePart | ToolCallPart | ToolResultPart>;
157
- providerOptions?: ProviderOptions;
158
- };
159
- /** A tool-role {@link AgentMessage} carrying one or more tool results. Create with {@link toolMessage}. */
160
- type ToolMessage = {
161
- role: "tool";
162
- content: Array<ToolResultPart>;
163
- providerOptions?: ProviderOptions;
164
- };
165
- /**
166
- * Optional framework-neutral message helpers. Requests also accept native
167
- * framework message values directly; core never converts between formats.
168
- * Store messages as plain context state: a text request's `onDone` output
169
- * carries the executor's response messages as `output.messages`.
170
- */
171
- type AgentMessage = SystemMessage | UserMessage | AssistantMessage | ToolMessage;
172
- /**
173
- * A schema value on an {@link AgentToolDescriptor}: deliberately just `object`,
174
- * because an SDK-native tool's `inputSchema` is the SDK's own union type (a Zod
175
- * schema, the SDK's `Schema`, a lazy thunk, ...) and must assign structurally
176
- * with no cast. The contract is runtime, not static: core narrows with
177
- * `isStandardSchema` and reads {@link StandardSchemaV1} schemas via
178
- * `getJsonSchema`; anything else passes through to the executor untouched.
179
- */
180
- type AgentToolSchema = object;
181
- /**
182
- * A tool exposed to a text request, described for both the model and
183
- * (optionally) host execution. This is a **minimal structural contract**: any
184
- * object matching it — an AI SDK `tool({...})`, an MCP-style descriptor, or a
185
- * hand-written `{ description, inputSchema, execute }` — is a valid entry, and
186
- * extra properties (`providerOptions`, `toModelOutput`, …) pass through
187
- * untouched via the index signature. `execute` is typed permissively so a
188
- * native tool's `(input, options)` executor is structurally assignable; the
189
- * SDK you built the tool with owns its precise input typing.
190
- */
191
- interface AgentToolDescriptor {
192
- /**
193
- * Text shown to the model. Typed loosely enough to accept an SDK tool whose
194
- * description is computed per call (the AI SDK v7 `tool({ description })`
195
- * accepts a function of the call's context); core never reads it.
196
- */
197
- description?: string | ((...args: any[]) => string);
198
- inputSchema?: AgentToolSchema;
199
- outputSchema?: AgentToolSchema;
200
- execute?: (...args: any[]) => unknown;
201
- [key: string]: unknown;
202
- }
203
- /** A bare tool implementation (no description/schema) — shorthand for {@link AgentToolDescriptor.execute}. */
204
- type AgentToolExecute = (input?: unknown) => unknown | Promise<unknown>;
205
- /** A tool entry in {@link AgentTools}: either a full descriptor or a bare execute function. */
206
- type AgentTool = AgentToolDescriptor | AgentToolExecute;
207
- /** The `tools` map passed on an {@link AgentTextRequest}, keyed by tool name. */
208
- type AgentTools = Record<string, AgentTool | undefined>;
209
- /** How a text request's model should select among its `tools`; `{ type: 'tool', name }` forces one specific tool. */
210
- type AgentToolChoice = "auto" | "none" | "required" | {
211
- type: "tool";
212
- name: string;
213
- };
214
- /** The event chosen and raised by a decision. */
215
- type ChosenEvent = {
216
- type: string;
217
- [key: string]: unknown;
218
- };
219
- type EventWildcardsOf<TEvent extends string> = TEvent extends `${infer Head}.${infer Rest}` ? `${Head}.*` | `${Head}.${EventWildcardsOf<Rest>}` : never;
220
- /** One `allowedEvents` entry: an exact declared event type, `'*'` (every event), or a `'prefix.*'` wildcard derived from the declared dotted event types. */
221
- type AllowedEventPattern<TEvent extends string = string> = TEvent | "*" | EventWildcardsOf<TEvent>;
222
- /**
223
- * Candidate event types for a decision (declared on the `agent.decide`
224
- * builtin's `allowedEvents` input). A single
225
- * entry or an array; entries are exact event types or wildcard patterns
226
- * (`'*'` for every event, `'todo.*'` for a dotted namespace). The effective
227
- * candidate set offered to the model is this declaration **intersected with
228
- * the snapshot's currently-legal events** (via `getAcceptedEvents`) —
229
- * omitting `allowedEvents` means "all currently-legal events." A resolver
230
- * function can therefore only ever narrow, never widen, the real surface.
231
- * Wildcards expand against the live snapshot, so they need a snapshot-aware
232
- * host (`createAgentRuntime`).
233
- */
234
- type AllowedEvents<TEvent extends string = string, TInput = unknown> = AllowedEventPattern<TEvent> | readonly AllowedEventPattern<TEvent>[] | ((args: {
235
- input: TInput;
236
- }) => AllowedEventPattern<TEvent> | readonly AllowedEventPattern<TEvent>[]);
237
- //#endregion
238
- export { UserMessage as A, StandardSchemaV1 as C, ToolMessage as D, ToolCallPart as E, ToolResultOutput as O, ProviderOptions as S, TextPart as T, FilePart as _, AgentToolChoice as a, InferOutput as b, AgentToolSchema as c, AllowedEvents as d, AssistantMessage as f, EventUnion as g, EventPayload as h, AgentTool as i, WithAgentInputSchema as j, ToolResultPart as k, AgentTools as l, DataContent as m, AgentEventSchemaInputMap as n, AgentToolDescriptor as o, ChosenEvent as p, AgentMessage as r, AgentToolExecute as s, AgentEventSchemaInput as t, AllowedEventPattern as u, ImagePart as v, SystemMessage as w, NormalizedEventSchemas as x, InferInput as y };
@@ -1,152 +0,0 @@
1
- ---
2
- name: add-evals
3
- description: Write evals for a @statelyai/agent or XState agent machine at the right level — rule checks with no model, decision evals at one decision state, outcome evals over whole runs, and per-state stats from traces — each tied to the states it covers. Use when asked to "write evals", "add evals for the approval logic", "test the agent", "measure the agent", "build an eval suite", or "what should I eval" for an agent built as a state machine.
4
- ---
5
-
6
- # Add evals to an agent machine
7
-
8
- The machine tells you what to eval and at which level. Every eval names the states it covers, so a failing eval points at a part of the machine.
9
-
10
- ```
11
- read the machine → map the request to states → pick the level per state → write the evals → baseline → hand back
12
- ```
13
-
14
- This is step 3 of 4. Before: `loop-to-machine`, `add-traces`. Next: `improve-machine`.
15
-
16
- ## Hard rules
17
-
18
- - **The cheapest level that can catch the failure.** A rule is checked without a model. Only judgment needs a model in the eval.
19
- - **Ground truth, never a model's opinion of itself.** Outcome evals read terminal states and effects in the world. A model grades text only where no ground truth exists, and that eval says so in its name.
20
- - **Labels belong to people.** Draft expected outcomes and expected events; ask the user to confirm them before they gate anything. Keep drafts in their own list, marked unconfirmed, and out of every gate.
21
- - **Never edit the machine to make an eval pass.** A failing eval is a finding. Report it.
22
- - **Held-out scenarios stay held out.** Evals may run on them; nobody tunes against them.
23
-
24
- ## 1. Read the machine
25
-
26
- ```ts
27
- import { describeMachine } from "@statelyai/agent";
28
- console.log(describeMachine(machine)); // goal, states, transitions, events
29
- ```
30
-
31
- List:
32
-
33
- - the decision states: an `agent.decide` invoke, a `decision` tag, or a state a host answers with `decideNext`. A machine whose model choice is the output of a text request (a tool call it then checks) has no decision states: eval that choice at the **request** level with `runSeam`;
34
- - the person's steps: idle states a person answers (`meta.interaction`, the runner's `onIdle`). What the person decides is ground truth, not something to eval; the routing after their answer is a rule;
35
- - the guards. `describeMachine` shows where a transition function can go and marks it `(conditional)`; `describeMachine(machine, { source: true })` also prints each function, so its guard logic is in the same read;
36
- - the sentinel states and the final states.
37
-
38
- **No model decisions.** When the machine has no decision states and its model only picks tools inside one text request (a scheduler, a tool runner), eval two levels: **rule** evals for every guard, cap, and approval rule (an "always allow" rule on both sides of its pattern; the barrier before execute; cancel), and **stats** from traces (`foldTraces` per state, approve/reject counts per tool). Skip decision evals. Add request evals only if a person can label which tool calls were right. The objective is rule violations (0) plus the per-state failure and cost numbers `improve-machine` reads.
39
-
40
- ## 2. Map the request to states
41
-
42
- Turn the user's words into states. "Evals for the approval logic" means every state and guard that touches approval: the guard that routes a refund over the threshold, the approval step and each answer to it, the sentinel check before the payment, and the final states they lead to. Write the mapping down:
43
-
44
- | Asked about | States and guards |
45
- | ----------- | -------------------------------------------------------------------------------------------------- |
46
- | approval | `deciding` (REFUND: over threshold → `awaitingApproval`), `awaitingApproval` (APPROVE, REJECT), `paying` (sentinel check), `issued`, `closed` |
47
-
48
- If the request names nothing ("write evals"), cover every decision state, every sentinel, and the run as a whole.
49
-
50
- ## 3. Pick the level
51
-
52
- | Level | Catches | Runs a model | Tools |
53
- | ------------ | ---------------------------------------------------- | --------------------- | ------------------------------------------------------------------ |
54
- | **Rule** | A guard, an ordering, an invariant that must hold | No | Exported guard functions, `canReach`, `explorePaths`, `simulateAgent` |
55
- | **Decision** | A wrong choice at one decision state | The decider under test | The decider called on recorded or written decision requests |
56
- | **Request** | A bad text output from one request | That request | `runSeam`, scored on `seamOutput` and the path after it. Not applicable when the machine has no text requests; say so |
57
- | **Outcome** | A wrong end result for a whole scenario | The whole agent | Scenarios, the runner, an objective over terminal states and effects |
58
- | **Stats** | Which state the failures cluster in | No (reads traces) | `foldTraces` |
59
-
60
- A guard or ordering is always a rule eval. Use a decision eval when the guards leave more than one legal option. Use an outcome eval when a person can label the right end result.
61
-
62
- Draft labels gate nothing, but rule evals need no labels, so they gate CI from day one. The label-based evals start gating as the user confirms labels (`draft: false`). Say so in the hand-back, so nobody reads a green CI as "the outcome evals pass".
63
-
64
- ## 4. Write them
65
-
66
- Name every eval `<state>: <what it checks>`, so the report reads as a map of the machine. An outcome eval over a whole run takes the final state it expects (`issued: partial-refund refunds 50 dollars`). In `test.each` titles, `$name` is a placeholder: write amounts as `100 dollars`, not `$100`.
67
-
68
- **Rule evals**, from `@statelyai/agent/testing`:
69
-
70
- ```ts
71
- test("paying: no refund over 100 dollars issues without approval", async () => {
72
- const result = await canReach(machine, "<sentinel>", { input, payloads, outputs, errors });
73
- expect(result.complete).toBe(true); // reachable: false is a proof only when the search finished
74
- expect(result.reachable, JSON.stringify(result.witness)).toBe(false);
75
- });
76
-
77
- test("deciding: REFUND over the order amount never reaches paying", async () => {
78
- const result = await canReach(machine, (s) => s.matches("paying") && s.context.amount > s.context.order.amount, { input, payloads, outputs });
79
- expect(result.complete).toBe(true);
80
- expect(result.reachable).toBe(false);
81
- });
82
- ```
83
-
84
- Check guards through the machine, not by importing the guard and calling it: an imported `refundIsLegal` is always the one in the source, so it passes on a machine built with a loosened guard. A unit test of the guard function is fine as well, never instead.
85
-
86
- Use `explorePaths` to assert the set of reachable final states, and `await simulateAgent(...)` for one scripted path. Cover both sides of every threshold. The two treat a closed choice differently:
87
-
88
- - `explorePaths` and `canReach` use `options.payloads` **instead of** the decision's own offered options, so they explore values the machine would never offer. That is what a boundary proof needs.
89
- - `simulateAgent` keeps the offered options: a scripted event the decision did not offer is refused, and the run ends in the decision's `onError`. Script only offered events, or pick an input that offers the value.
90
-
91
- **Decision evals.** A row is a decision request and the event a person says is right. Write rows by hand, or take them from recorded traces with `getDecisionRequests(trace, { name })` from `@statelyai/agent/testing`: each decision's first attempt, with the states it was asked in, its `options`, and its `metadata`. Compare events without caring about key order. If the decider reads anything outside the request (the ticket, the tool history), store that in the row too.
92
-
93
- ```ts
94
- import { getDecisionRequests } from "@statelyai/agent/testing";
95
-
96
- // One row per recorded decision; `expected` is the label a person confirmed for that decision.
97
- // A scenario can decide more than once, so label each decision by its position, not the scenario alone.
98
- const rows = traces.flatMap(({ scenario, trace }) =>
99
- getDecisionRequests(trace, { name: "chooseResolution" }).map(({ request }, i) => ({
100
- id: `${scenario.id}#${i}`,
101
- request,
102
- expected: labels[`${scenario.id}#${i}`],
103
- })),
104
- );
105
-
106
- test.each(rows)("deciding: $id", async ({ request, expected }) => {
107
- const { event } = await decide(request); // the policy decider, or the model executor
108
- expect(event).toEqual(expected);
109
- });
110
- ```
111
-
112
- Which deciders gate CI: a deterministic reference decider (a scripted or rules-based one) gates. A live model is a measurement, not a unit test: run it outside CI, several times, and report the rate. A seeded random decider is measured over several seeds, never gated.
113
-
114
- **Request evals.** For a text request (a drafted question, a summary), `runSeam(machine, { input, scripts, seam: { request: "draftQuestion" }, candidate })` runs the whole machine with only that request live. Score `seamOutput`, and the states the run reached after it.
115
-
116
- **Outcome evals.** One function over a batch of scenario runs, reading ground truth. Score the effects as well as the outcome: a run that ends `issued` with the wrong amount is wrong.
117
-
118
- ```ts
119
- export function objective(runs: Array<{ scenario: Scenario; outcome: string; refund?: Refund; violations: number; modelCalls: number }>) {
120
- return {
121
- accuracy: runs.filter((r) => r.outcome === r.scenario.expected && sameRefund(r.refund, r.scenario.expectedRefund)).length / runs.length,
122
- violations: runs.reduce((n, r) => n + r.violations, 0), // must be 0
123
- modelCalls: runs.reduce((n, r) => n + r.modelCalls, 0), // cost
124
- };
125
- }
126
- ```
127
-
128
- Read the terminal state from the event logs `add-traces` recorded (`replay(machine, entries).snapshot`, `replay` from `@statelyai/agent/log`). Effects and violations live in the world, not the log: save them next to the thread id when the run ends, so the objective can be recomputed without running anything. Use `matchesTrajectory` only for an ordering a person requires (approval before payment), never to pin the whole path; build its input from the `machine.transition` trace events' `snapshot.value`.
129
-
130
- **Reliability.** A live model is measured over repeated runs. `repeatRuns({ scenarios, k, run, passed })` (from `@statelyai/agent/testing`) runs each scenario `k` times and reports `passes`, `passAll` (pass^k), `distinctPaths`, and `divergedAt`, the state where its runs went different ways. Report `report.passHat(k)` next to `report.passHat(1)`. For runs recorded elsewhere, `summarizeRepeats` takes them per scenario.
131
-
132
- **Judged evals.** Where a model grades text because no ground truth exists, measure the judge first: `judgeAgreement(rows, { positive })` over confirmed labels, grouped by the state or request it judges. Report the true positive and true negative rates, and gate on the judge only after the user accepts them. Use `report.correct(p)` to turn the judge's rate on unlabeled runs into an estimate.
133
-
134
- **Stats.** `foldTraces(traces, { failed, blame })` (from `@statelyai/agent/testing`) returns `{ state, visits, failures, withheld, ignored, blamed }` per state. A failed run counts in `failures` against **every** state it visited, so states after a bad decision rank as high as the decision itself. `blame` counts each failed run once, against the state it started failing in: `blame: (trace) => firstDivergence(trace, reference)?.state`, where the reference is a passing run of the same scenario or the trajectory a person expects. Sort by `blamed`, then look at the decision states among the top rows, their `withheld` counts, and their decision attempts. `failed` sees only the trace: to judge a run against its label, keep a map from trace to scenario.
135
-
136
- ## 5. Coverage
137
-
138
- Run `traceCoverage(machine, traces, { reachable })` (from `@statelyai/agent/testing`) over the traces of every scenario, with `reachable` set to an `explorePaths` report. List its `unvisitedStates` and `unvisitedHandlers`: each one is a part of the machine no scenario exercises. Draft a scenario for each, as below. Handlers in `unobservable` (`always` transitions) do not show in traces; say so, and do not draft scenarios for them.
139
-
140
- If every decider scores the same on the confirmed scenarios, the scenarios cannot tell a better decider from a worse one. Draft scenarios on the boundaries the evals found (an amount just under and over a threshold, a partial request), label them as drafts, and ask the user to confirm them and add them to the scenario set. They become part of the train or held-out split only once confirmed.
141
-
142
- ## 6. Baseline
143
-
144
- Run every eval once on the current machine and record the numbers: which pass, the outcome scores per split, the stats table. Failing evals are findings, not bugs in the eval, unless the label was wrong; then ask.
145
-
146
- ## 7. Hand back
147
-
148
- 1. The state mapping from step 2.
149
- 2. The evals by level, each named by state.
150
- 3. The baseline numbers, train and held-out separately, with pass^k for live-model evals and the coverage ratios.
151
- 4. Every failing eval and the state it points at.
152
- 5. Next step: `improve-machine`, with the objective from step 4 as its scorer.