@statelyai/agent 2.0.0-alpha.36 → 2.0.0-alpha.38
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +33 -0
- package/dist/{effects-CuOUNYtZ.mjs → effects-BT_cSY0j.mjs} +95 -11
- package/dist/{effects-B23bFZEs.cjs → effects-DDseYtJk.cjs} +95 -11
- package/dist/index.cjs +6 -6
- package/dist/index.d.cts +3 -3
- package/dist/index.d.mts +3 -3
- package/dist/index.mjs +6 -6
- package/dist/otel.d.cts +1 -1
- package/dist/otel.d.mts +1 -1
- package/dist/{run-agent-BcQ-9NRT.d.cts → run-agent-30mH2moE.d.cts} +2 -2
- package/dist/{run-agent-C86drZtN.d.mts → run-agent-DlPzBWj-.d.mts} +2 -2
- package/dist/{setup-agent-jlo4vtaZ.d.cts → setup-agent-BaNyz6Q3.d.cts} +1 -1
- package/dist/{setup-agent-BIKcbVXV.d.mts → setup-agent-C1CG_nfX.d.mts} +1 -1
- package/dist/testing.cjs +2411 -1799
- package/dist/testing.d.cts +167 -4
- package/dist/testing.d.mts +167 -4
- package/dist/testing.mjs +2412 -1801
- package/dist/{usage-HtnrUsl9.d.cts → usage-Bm5nvpow.d.cts} +5 -0
- package/dist/{usage-DoP0Oi84.d.mts → usage-BnikdOca.d.mts} +5 -0
- package/dist/validate.cjs +1 -1
- package/dist/validate.d.cts +2 -2
- package/dist/validate.d.mts +2 -2
- package/dist/validate.mjs +1 -1
- package/dist/{verify-Dpz1u5Pe.mjs → verify-ChyX6jq0.mjs} +75 -20
- package/dist/{verify-Bink-Drw.cjs → verify-CwlombbC.cjs} +75 -20
- package/dist/{verify-CciG8UP-.d.cts → verify-DJHfEfm6.d.cts} +9 -4
- package/dist/{verify-Buh8XSUr.d.mts → verify-i9Hiwc5g.d.mts} +9 -4
- package/docs/decisions.md +1 -1
- package/docs/event-log.md +3 -2
- package/docs/hosts.md +2 -2
- package/docs/machines-as-data.md +5 -5
- package/docs/persistence.md +76 -5
- package/docs/quickstart.md +1 -1
- package/docs/steps.md +2 -0
- package/docs/usage-and-budgets.md +2 -0
- package/docs/verify.md +7 -2
- package/package.json +1 -1
- package/skills/build-machine/SKILL.md +4 -1
package/docs/persistence.md
CHANGED
|
@@ -4,7 +4,7 @@ The [event log](event-log.md) is the durable artifact. A snapshot is a cache ove
|
|
|
4
4
|
|
|
5
5
|
- The log is an ordered array of the external inputs the machine received. A run is fully reconstructed from it.
|
|
6
6
|
- The persisted snapshot is the machine's serialized state at one point. It lets a resume skip the fold.
|
|
7
|
-
- Live run state
|
|
7
|
+
- Live run state (the actor, in-flight requests, stream chunks) is derived and disposable.
|
|
8
8
|
|
|
9
9
|
## Persist the log
|
|
10
10
|
|
|
@@ -19,7 +19,7 @@ const result = await runToQuiescence(createAgentRuntime(machine, { store, thread
|
|
|
19
19
|
- A rejected write stops the run: `{ status: "error", cause: "journal" }`.
|
|
20
20
|
- `threadId` is required whenever `store` is given.
|
|
21
21
|
|
|
22
|
-
`onEvent` remains the observer seam
|
|
22
|
+
`onEvent` remains the observer seam. It is synchronous and never awaited. Persisting there is at-least-once: a crash between an entry and the host's flush loses it. See [Record a log](event-log.md#record-a-log).
|
|
23
23
|
|
|
24
24
|
`result.events` is a complete, self-contained segment: its first entry is the reserved `@agent.init` entry, so it replays with no side channel.
|
|
25
25
|
|
|
@@ -70,9 +70,80 @@ if (result.ignored) {
|
|
|
70
70
|
- A log that already reached a final state settles immediately with the recorded output.
|
|
71
71
|
- The resumed result's `events` extends the same log, so the whole history stays replayable.
|
|
72
72
|
|
|
73
|
+
## Crash testing
|
|
74
|
+
|
|
75
|
+
`checkCrashRecovery` from `@statelyai/agent/testing` kills a run at every point it could die and checks that the resume matches a run that never died.
|
|
76
|
+
|
|
77
|
+
```ts
|
|
78
|
+
import { createAsyncLogic } from "xstate";
|
|
79
|
+
import { z } from "zod";
|
|
80
|
+
import { setupAgent, createTextLogic } from "@statelyai/agent";
|
|
81
|
+
import { checkCrashRecovery } from "@statelyai/agent/testing";
|
|
82
|
+
|
|
83
|
+
const machine = setupAgent({
|
|
84
|
+
context: z.object({ order: z.string() }),
|
|
85
|
+
events: {},
|
|
86
|
+
actors: {
|
|
87
|
+
lookup: createAsyncLogic({ run: async () => "order-7" }),
|
|
88
|
+
reply: createTextLogic({ model: "m", prompt: "Reply." }),
|
|
89
|
+
},
|
|
90
|
+
}).createMachine({
|
|
91
|
+
context: { order: "" },
|
|
92
|
+
initial: "lookup",
|
|
93
|
+
states: {
|
|
94
|
+
lookup: {
|
|
95
|
+
invoke: {
|
|
96
|
+
src: "lookup",
|
|
97
|
+
onDone: ({ event }) => ({ target: "reply", context: { order: event.output } }),
|
|
98
|
+
},
|
|
99
|
+
},
|
|
100
|
+
reply: { invoke: { src: "reply", onDone: { target: "done" } } },
|
|
101
|
+
done: { type: "final" },
|
|
102
|
+
},
|
|
103
|
+
});
|
|
104
|
+
|
|
105
|
+
const report = await checkCrashRecovery(machine, {
|
|
106
|
+
executors: { generateText: async () => ({ result: "Your order shipped." }) },
|
|
107
|
+
});
|
|
108
|
+
report.failures; // [] when every crash point recovers
|
|
109
|
+
```
|
|
110
|
+
|
|
111
|
+
It runs the machine once uninterrupted, then once per crash point, each resumed from the store's log by a fresh runtime:
|
|
112
|
+
|
|
113
|
+
- **Log points:** before the first append, and after each append. The next write never lands.
|
|
114
|
+
- **Effect points:** while each model call or plain actor is in flight. It ran, but its result was never recorded.
|
|
115
|
+
|
|
116
|
+
Each resumed run is checked against the uninterrupted one. Each failure names its crash point and one of these problems:
|
|
117
|
+
|
|
118
|
+
| `problem` | Meaning |
|
|
119
|
+
| ----------------- | ------------------------------------------------------------------------------------------- |
|
|
120
|
+
| `reran` | An effect ran again although its completion was durable before the crash |
|
|
121
|
+
| `skipped` | An effect of the uninterrupted run never ran |
|
|
122
|
+
| `unexpected` | An effect ran that the uninterrupted run never made, such as a call under a new `callKey` |
|
|
123
|
+
| `request-changed` | A model call re-executed under its `callKey` with a different request |
|
|
124
|
+
| `outcome` | The resume ended with a different status, output, state, or context |
|
|
125
|
+
| `replay` | The final log does not replay to the resumed state |
|
|
126
|
+
| `undelivered` | A `sendDuring` event the crash lost, which the resume never reached the point to send again |
|
|
127
|
+
| `stuck` | The crashed run or its resume did not settle within `timeout` (default 5000 ms) |
|
|
128
|
+
| `threw` | The crashed run, its resume, or the `store` factory threw |
|
|
129
|
+
|
|
130
|
+
- `report.unreached` lists the points no run died at, with why: the crashed run never started that effect, or one write took the log past that length. A point is tried 3 times before it counts, since concurrent effects can settle in another order and merge two writes. Their resumes were not checked, so `report.ok` is `false`. An effect point is the n-th run of its key, so concurrent effects starting in another order still hit it.
|
|
131
|
+
- An effect in flight at the crash may run twice. That is the at-least-once contract (see [Idempotency keys](hosts.md#idempotency-keys)), and it is not reported.
|
|
132
|
+
- Model calls are keyed by `callKey` without the execution id, and plain actors by invoke site. Only actors bound through string-keyed sources are seen.
|
|
133
|
+
- Model calls run live only in the uninterrupted run. After a crash, a call it made gets its recorded answer, after the same delay, so a fake may answer by call order.
|
|
134
|
+
- Plain actors run live every time, so they must give the same result however often they run. A fake that keeps state passes `reset`, which runs before the baseline and each crashed run. It does not run before the resume, so the fake keeps what the crashed run did, as a real dependency would.
|
|
135
|
+
- The outcome check compares the root's final status, output, state, and context, not spawned children's. A child's progress counts only through what it reports to the root. It compares final states, so a machine whose result depends on which concurrent call finishes first can fail it after a crash. Make the result order-independent, or test that part without concurrency.
|
|
136
|
+
- `events` are host events, delivered one per idle settle. A resume skips the ones its log already holds.
|
|
137
|
+
- `sendDuring` sends host events while the run is busy: `{ during: "draft#1", event: { type: "CANCEL" } }` goes out the first time that effect starts, keyed as in `report.baseline.effects`: `<site>#<n>` for a model call, `actor:<site>` for a plain actor. Pass `occurrence` (from 0) to target a later start of the same key, such as a retry's second POST. After a crash, one the log does not hold is sent again, as a retrying host would, as its effect starts again or, when that effect's completion was durable, at the point the uninterrupted log holds it, so it lands in the same order. A `during` key (or `occurrence`) the uninterrupted run never starts throws, so a typo cannot pass silently.
|
|
138
|
+
- `timers: "host"` makes the harness the host scheduler for root `after` timers (see [Host-owned timers](steps.md#host-owned-timers)). The run settles `idle` with them pending. An id-less `{ type: "xstate.timer" }` in `events` is a timer's turn: the timer due soonest fires there. Once `events` run out, the pending timers fire in due order, without waiting for their delays. Each is due its delay after it was scheduled, on a clock that advances as timers fire. The default, in-process timers, fire after their real delay.
|
|
139
|
+
- An uninterrupted run that ends with some of `events` undelivered throws, so an event a final state never sees cannot pass silently.
|
|
140
|
+
- `store` takes a factory for your own store adapter, one fresh store per run. A store that acknowledges a write it then loses shows up as `reran`.
|
|
141
|
+
|
|
142
|
+
Run it in CI next to the machine's other tests. It needs no API key.
|
|
143
|
+
|
|
73
144
|
## Snapshot as cache
|
|
74
145
|
|
|
75
|
-
`persist()` output carries `agentMeta: { machineId, version, logId, logIndex }
|
|
146
|
+
`persist()` output carries `agentMeta: { machineId, version, logId, logIndex }`: the lineage and position of the log it caches.
|
|
76
147
|
|
|
77
148
|
Pass `snapshot` alongside `events` to skip the fold:
|
|
78
149
|
|
|
@@ -84,13 +155,13 @@ const resumed = await runToQuiescence(createAgentRuntime(machine, { executors })
|
|
|
84
155
|
});
|
|
85
156
|
```
|
|
86
157
|
|
|
87
|
-
- Fast path: `agentMeta.logId` names the log, `logIndex` equals the log length, and the snapshot's state hash matches the tail entry's `verification.stateHash`. Length alone is not enough
|
|
158
|
+
- Fast path: `agentMeta.logId` names the log, `logIndex` equals the log length, and the snapshot's state hash matches the tail entry's `verification.stateHash`. Length alone is not enough, because a fork or a sibling thread can sit at the same index.
|
|
88
159
|
- Otherwise the log is replayed and the run resumes from the replayed state.
|
|
89
160
|
- A cache that disagrees with the state the log replays to throws `AgentSnapshotDivergedError` (code `snapshot-diverged`). Drop the snapshot and resume from `events` alone.
|
|
90
161
|
|
|
91
162
|
A snapshot is never authoritative within a version. Snapshot only at quiescent points: a mid-flight snapshot cannot carry in-flight request state, but the log can.
|
|
92
163
|
|
|
93
|
-
Resuming from a snapshot with no log also yields a self-contained log
|
|
164
|
+
Resuming from a snapshot with no log also yields a self-contained log (the new segment's init entry carries that snapshot), so every result is replayable.
|
|
94
165
|
|
|
95
166
|
## Version bridge
|
|
96
167
|
|
package/docs/quickstart.md
CHANGED
|
@@ -85,7 +85,7 @@ Requests have a semantic `name`, resolved `input`, model reference, schemas,
|
|
|
85
85
|
prompt/messages, and tools. The machine owns control flow; the executor owns
|
|
86
86
|
the provider call.
|
|
87
87
|
|
|
88
|
-
An executor is
|
|
88
|
+
An executor is a function, so the machine runs anywhere a function does.
|
|
89
89
|
When several requests each need their own canned answer, the executor routes on
|
|
90
90
|
`request.name`. See [Evals](evals.md#testing-without-a-provider).
|
|
91
91
|
|
package/docs/steps.md
CHANGED
|
@@ -40,6 +40,8 @@ with an external timer pending. When it fires, deliver
|
|
|
40
40
|
`{ type: "xstate.timer", id }` through `transition` or as the `event` passed to
|
|
41
41
|
`runToQuiescence`.
|
|
42
42
|
|
|
43
|
+
A resumed run calls `schedule` again for each timer still pending, with what is left of its delay by the wall clock. Treat `schedule` as an upsert by `id`: a host that already holds that timer keeps its own due time. An `id` names a timer by its state and delay, not by arming, so a state entered again schedules the same `id` once the earlier timer has fired or been cancelled.
|
|
44
|
+
|
|
43
45
|
An invoked child machine's `after` timers always run in process, and the run
|
|
44
46
|
waits for them, whichever `timers` option you pass.
|
|
45
47
|
|
|
@@ -190,6 +190,7 @@ A wrapper is still host-side. The machine cannot read its counts or react to the
|
|
|
190
190
|
- Delivery is opt-in. The event is delivered only when an active state declares a transition that matches it, either `'@agent.usage'` or the wildcard `'*'`. Otherwise no usage event is sent to the machine. Wildcard matching follows XState semantics, so a catch-all `on: { '*': … }` does receive `@agent.usage`. Narrow the handler on `event.type` when the catch-all is meant for other events.
|
|
191
191
|
- Declare the handler at machine level. A root `on: { '@agent.usage': … }` catches every call regardless of which state made it. A state-scoped handler drops calls made in other states.
|
|
192
192
|
- Delivery goes to the run's root machine, including usage from requests inside an invoked child machine. Attribute those calls with `id`, `src`, and `model`.
|
|
193
|
+
- Usage counts real spend across a crash. A call that re-executes on resume reports its usage again, so a token budget sees both attempts when the first attempt's usage was journaled before the crash. A call in flight at the crash that never reported usage is not counted.
|
|
193
194
|
|
|
194
195
|
The payload has this shape.
|
|
195
196
|
|
|
@@ -202,6 +203,7 @@ The payload has this shape.
|
|
|
202
203
|
src?: string, // the request's invoke src
|
|
203
204
|
model?: string, // the model ref it targeted
|
|
204
205
|
name?: string, // a text request's registered name
|
|
206
|
+
callKey?: string, // the call's idempotency key, with an event log
|
|
205
207
|
budget?: { modelCalls, maxModelCalls }, // counted against maxModelCalls so far, this call included
|
|
206
208
|
}
|
|
207
209
|
```
|
package/docs/verify.md
CHANGED
|
@@ -310,9 +310,11 @@ const result = await canReach(waitingMachine, "timedOut");
|
|
|
310
310
|
- The event is `{ type: '@agent.usage', kind: 'text', id, src, model, name?, usage }`, the run's shape without `budget`.
|
|
311
311
|
- Each listed value forks one branch, recorded in the path before `xstate.done.actor`.
|
|
312
312
|
- One value with one canned output is no fork: the output resolves straight through, but the usage event is still recorded in the path.
|
|
313
|
-
- A usage
|
|
313
|
+
- A usage value the active states refuse (`snapshot.can` is false) is dropped, as a run drops it: the request settles with no usage event delivered or recorded.
|
|
314
|
+
- A usage transition that leaves the invoking state, or restarts its invoke, cancels the request: its completion is not recorded, and a restarted invoke settles later on its own.
|
|
314
315
|
- A settling request with no `usage` entry while a usage transition is active lists `@agent.usage` in `vacuous`. Completeness is unchanged.
|
|
315
|
-
-
|
|
316
|
+
- With `children: "coupled"`, a child's text request with an entry forks one branch per value. The usage event goes to the root before the child's completion reaches its parent. The root takes it or drops it by the same rules. A child's request with no entry is not listed in `vacuous`.
|
|
317
|
+
- Requests inside `"outputs"` or `"explore"` children, and decisions, deliver no usage.
|
|
316
318
|
|
|
317
319
|
```ts no-check
|
|
318
320
|
const overBudget = await canReach(researchMachine, "outOfBudget", {
|
|
@@ -360,7 +362,9 @@ const report = await explorePaths(parentMachine, {
|
|
|
360
362
|
- The parent's host events arrive at its idle waits, including while every child waits on the host. While a child or the parent has work pending, they race it only with `interleave: "pending"`.
|
|
361
363
|
- What a child's move sends its own children, and their replies, happen in the same move. A child's send to a sibling is delivered to that sibling, entry sends included. A child's sends reach their targets in the order it made them, each run to quiescence before the next, as a live run does. A send to an actor the search does not model lists the event type in `unexpandedEvents`, so the report is not complete.
|
|
362
364
|
- Child moves count against `maxDepth` like the parent's own branches. On a path they carry `actor` with the child's id (a grandchild's as `child.grandchild`): the host events it received, its choices, its settled work. The events it sends the parent appear as the parent receives them.
|
|
365
|
+
- A target predicate and `dedupeBy` see each coupled child's explored state: `snapshot.children[id].getSnapshot()` returns the state the search stepped the child to, its own coupled children likewise. A parent's send and the child's receipt are one state, so a cross-actor invariant such as "every chunk the parent forwarded, a child received" can be a target.
|
|
363
366
|
- With `dedupeBy`, each child's state joins the key. Without it, every interleaving is explored in full.
|
|
367
|
+
- A coupled child's text requests deliver `@agent.usage` to the root, as a run does. See [Usage events](#usage-events).
|
|
364
368
|
- A child that stops on `needs-output` or `max-depth` ends the parent branch the same way, so the parent report is not complete.
|
|
365
369
|
|
|
366
370
|
Pass `hostEvents` when some types the parent accepts come only from its children, such as a child's status events. The search then offers only the listed types at the parent's waits.
|
|
@@ -479,6 +483,7 @@ For machines authored as data, validate the config with `validateAgentConfig(con
|
|
|
479
483
|
|
|
480
484
|
## Related
|
|
481
485
|
|
|
486
|
+
- [Crash testing](persistence.md#crash-testing): `checkCrashRecovery` kills a run at every log append and in-flight effect, and checks each resume.
|
|
482
487
|
- [Debugging](debugging.md): scripted reproduction and the diagnostic codes in context.
|
|
483
488
|
- [Evals](evals.md): scoring runs on output, trajectory, and budget.
|
|
484
489
|
- [Quickstart](quickstart.md#define-and-run-one-artifact): a plain-function executor for `runToQuiescence`.
|
package/package.json
CHANGED
|
@@ -62,7 +62,10 @@ expect(result).toMatchObject({ reachable: false, complete: true });
|
|
|
62
62
|
|
|
63
63
|
## 5. Crash and resume
|
|
64
64
|
|
|
65
|
-
|
|
65
|
+
Call `checkCrashRecovery(machine, { input, events, executors })` from `@statelyai/agent/testing` for each scenario, and assert `report.failures` is `[]`. It kills the run after every log append and while every effect is in flight, resumes each from the log, and checks the resume against an uninterrupted run. See `docs/persistence.md#crash-testing`.
|
|
66
|
+
|
|
67
|
+
- Executors and fake tools must answer from the request, not call order. A fake that keeps state passes `reset`.
|
|
68
|
+
- A failure names the crash point and the problem (`reran`, `skipped`, `outcome`, `stuck`, …). Fix the machine or the effect, never the check.
|
|
66
69
|
|
|
67
70
|
- A completed model call never runs again. An in-flight one re-runs with the same `info.callKey`.
|
|
68
71
|
- A plain actor (`createAsyncLogic`) in flight at the crash re-runs. It gets no call key, so derive an idempotency key from the domain (the ticket id, the booking key) and dedupe in the effect.
|