@statelyai/agent 2.0.0-alpha.36 → 2.0.0-alpha.38

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (38) hide show
  1. package/CHANGELOG.md +33 -0
  2. package/dist/{effects-CuOUNYtZ.mjs → effects-BT_cSY0j.mjs} +95 -11
  3. package/dist/{effects-B23bFZEs.cjs → effects-DDseYtJk.cjs} +95 -11
  4. package/dist/index.cjs +6 -6
  5. package/dist/index.d.cts +3 -3
  6. package/dist/index.d.mts +3 -3
  7. package/dist/index.mjs +6 -6
  8. package/dist/otel.d.cts +1 -1
  9. package/dist/otel.d.mts +1 -1
  10. package/dist/{run-agent-BcQ-9NRT.d.cts → run-agent-30mH2moE.d.cts} +2 -2
  11. package/dist/{run-agent-C86drZtN.d.mts → run-agent-DlPzBWj-.d.mts} +2 -2
  12. package/dist/{setup-agent-jlo4vtaZ.d.cts → setup-agent-BaNyz6Q3.d.cts} +1 -1
  13. package/dist/{setup-agent-BIKcbVXV.d.mts → setup-agent-C1CG_nfX.d.mts} +1 -1
  14. package/dist/testing.cjs +2411 -1799
  15. package/dist/testing.d.cts +167 -4
  16. package/dist/testing.d.mts +167 -4
  17. package/dist/testing.mjs +2412 -1801
  18. package/dist/{usage-HtnrUsl9.d.cts → usage-Bm5nvpow.d.cts} +5 -0
  19. package/dist/{usage-DoP0Oi84.d.mts → usage-BnikdOca.d.mts} +5 -0
  20. package/dist/validate.cjs +1 -1
  21. package/dist/validate.d.cts +2 -2
  22. package/dist/validate.d.mts +2 -2
  23. package/dist/validate.mjs +1 -1
  24. package/dist/{verify-Dpz1u5Pe.mjs → verify-ChyX6jq0.mjs} +75 -20
  25. package/dist/{verify-Bink-Drw.cjs → verify-CwlombbC.cjs} +75 -20
  26. package/dist/{verify-CciG8UP-.d.cts → verify-DJHfEfm6.d.cts} +9 -4
  27. package/dist/{verify-Buh8XSUr.d.mts → verify-i9Hiwc5g.d.mts} +9 -4
  28. package/docs/decisions.md +1 -1
  29. package/docs/event-log.md +3 -2
  30. package/docs/hosts.md +2 -2
  31. package/docs/machines-as-data.md +5 -5
  32. package/docs/persistence.md +76 -5
  33. package/docs/quickstart.md +1 -1
  34. package/docs/steps.md +2 -0
  35. package/docs/usage-and-budgets.md +2 -0
  36. package/docs/verify.md +7 -2
  37. package/package.json +1 -1
  38. package/skills/build-machine/SKILL.md +4 -1
@@ -4,7 +4,7 @@ The [event log](event-log.md) is the durable artifact. A snapshot is a cache ove
4
4
 
5
5
  - The log is an ordered array of the external inputs the machine received. A run is fully reconstructed from it.
6
6
  - The persisted snapshot is the machine's serialized state at one point. It lets a resume skip the fold.
7
- - Live run state — the actor, in-flight requests, stream chunks — is derived and disposable.
7
+ - Live run state (the actor, in-flight requests, stream chunks) is derived and disposable.
8
8
 
9
9
  ## Persist the log
10
10
 
@@ -19,7 +19,7 @@ const result = await runToQuiescence(createAgentRuntime(machine, { store, thread
19
19
  - A rejected write stops the run: `{ status: "error", cause: "journal" }`.
20
20
  - `threadId` is required whenever `store` is given.
21
21
 
22
- `onEvent` remains the observer seam — synchronous, never awaited. Persisting there is at-least-once: a crash between an entry and the host's flush loses it. See [Record a log](event-log.md#record-a-log).
22
+ `onEvent` remains the observer seam. It is synchronous and never awaited. Persisting there is at-least-once: a crash between an entry and the host's flush loses it. See [Record a log](event-log.md#record-a-log).
23
23
 
24
24
  `result.events` is a complete, self-contained segment: its first entry is the reserved `@agent.init` entry, so it replays with no side channel.
25
25
 
@@ -70,9 +70,80 @@ if (result.ignored) {
70
70
  - A log that already reached a final state settles immediately with the recorded output.
71
71
  - The resumed result's `events` extends the same log, so the whole history stays replayable.
72
72
 
73
+ ## Crash testing
74
+
75
+ `checkCrashRecovery` from `@statelyai/agent/testing` kills a run at every point it could die and checks that the resume matches a run that never died.
76
+
77
+ ```ts
78
+ import { createAsyncLogic } from "xstate";
79
+ import { z } from "zod";
80
+ import { setupAgent, createTextLogic } from "@statelyai/agent";
81
+ import { checkCrashRecovery } from "@statelyai/agent/testing";
82
+
83
+ const machine = setupAgent({
84
+ context: z.object({ order: z.string() }),
85
+ events: {},
86
+ actors: {
87
+ lookup: createAsyncLogic({ run: async () => "order-7" }),
88
+ reply: createTextLogic({ model: "m", prompt: "Reply." }),
89
+ },
90
+ }).createMachine({
91
+ context: { order: "" },
92
+ initial: "lookup",
93
+ states: {
94
+ lookup: {
95
+ invoke: {
96
+ src: "lookup",
97
+ onDone: ({ event }) => ({ target: "reply", context: { order: event.output } }),
98
+ },
99
+ },
100
+ reply: { invoke: { src: "reply", onDone: { target: "done" } } },
101
+ done: { type: "final" },
102
+ },
103
+ });
104
+
105
+ const report = await checkCrashRecovery(machine, {
106
+ executors: { generateText: async () => ({ result: "Your order shipped." }) },
107
+ });
108
+ report.failures; // [] when every crash point recovers
109
+ ```
110
+
111
+ It runs the machine once uninterrupted, then once per crash point, each resumed from the store's log by a fresh runtime:
112
+
113
+ - **Log points:** before the first append, and after each append. The next write never lands.
114
+ - **Effect points:** while each model call or plain actor is in flight. It ran, but its result was never recorded.
115
+
116
+ Each resumed run is checked against the uninterrupted one. Each failure names its crash point and one of these problems:
117
+
118
+ | `problem` | Meaning |
119
+ | ----------------- | ------------------------------------------------------------------------------------------- |
120
+ | `reran` | An effect ran again although its completion was durable before the crash |
121
+ | `skipped` | An effect of the uninterrupted run never ran |
122
+ | `unexpected` | An effect ran that the uninterrupted run never made, such as a call under a new `callKey` |
123
+ | `request-changed` | A model call re-executed under its `callKey` with a different request |
124
+ | `outcome` | The resume ended with a different status, output, state, or context |
125
+ | `replay` | The final log does not replay to the resumed state |
126
+ | `undelivered` | A `sendDuring` event the crash lost, which the resume never reached the point to send again |
127
+ | `stuck` | The crashed run or its resume did not settle within `timeout` (default 5000 ms) |
128
+ | `threw` | The crashed run, its resume, or the `store` factory threw |
129
+
130
+ - `report.unreached` lists the points no run died at, with why: the crashed run never started that effect, or one write took the log past that length. A point is tried 3 times before it counts, since concurrent effects can settle in another order and merge two writes. Their resumes were not checked, so `report.ok` is `false`. An effect point is the n-th run of its key, so concurrent effects starting in another order still hit it.
131
+ - An effect in flight at the crash may run twice. That is the at-least-once contract (see [Idempotency keys](hosts.md#idempotency-keys)), and it is not reported.
132
+ - Model calls are keyed by `callKey` without the execution id, and plain actors by invoke site. Only actors bound through string-keyed sources are seen.
133
+ - Model calls run live only in the uninterrupted run. After a crash, a call it made gets its recorded answer, after the same delay, so a fake may answer by call order.
134
+ - Plain actors run live every time, so they must give the same result however often they run. A fake that keeps state passes `reset`, which runs before the baseline and each crashed run. It does not run before the resume, so the fake keeps what the crashed run did, as a real dependency would.
135
+ - The outcome check compares the root's final status, output, state, and context, not spawned children's. A child's progress counts only through what it reports to the root. It compares final states, so a machine whose result depends on which concurrent call finishes first can fail it after a crash. Make the result order-independent, or test that part without concurrency.
136
+ - `events` are host events, delivered one per idle settle. A resume skips the ones its log already holds.
137
+ - `sendDuring` sends host events while the run is busy: `{ during: "draft#1", event: { type: "CANCEL" } }` goes out the first time that effect starts, keyed as in `report.baseline.effects`: `<site>#<n>` for a model call, `actor:<site>` for a plain actor. Pass `occurrence` (from 0) to target a later start of the same key, such as a retry's second POST. After a crash, one the log does not hold is sent again, as a retrying host would, as its effect starts again or, when that effect's completion was durable, at the point the uninterrupted log holds it, so it lands in the same order. A `during` key (or `occurrence`) the uninterrupted run never starts throws, so a typo cannot pass silently.
138
+ - `timers: "host"` makes the harness the host scheduler for root `after` timers (see [Host-owned timers](steps.md#host-owned-timers)). The run settles `idle` with them pending. An id-less `{ type: "xstate.timer" }` in `events` is a timer's turn: the timer due soonest fires there. Once `events` run out, the pending timers fire in due order, without waiting for their delays. Each is due its delay after it was scheduled, on a clock that advances as timers fire. The default, in-process timers, fire after their real delay.
139
+ - An uninterrupted run that ends with some of `events` undelivered throws, so an event a final state never sees cannot pass silently.
140
+ - `store` takes a factory for your own store adapter, one fresh store per run. A store that acknowledges a write it then loses shows up as `reran`.
141
+
142
+ Run it in CI next to the machine's other tests. It needs no API key.
143
+
73
144
  ## Snapshot as cache
74
145
 
75
- `persist()` output carries `agentMeta: { machineId, version, logId, logIndex }` — the lineage and position of the log it caches.
146
+ `persist()` output carries `agentMeta: { machineId, version, logId, logIndex }`: the lineage and position of the log it caches.
76
147
 
77
148
  Pass `snapshot` alongside `events` to skip the fold:
78
149
 
@@ -84,13 +155,13 @@ const resumed = await runToQuiescence(createAgentRuntime(machine, { executors })
84
155
  });
85
156
  ```
86
157
 
87
- - Fast path: `agentMeta.logId` names the log, `logIndex` equals the log length, and the snapshot's state hash matches the tail entry's `verification.stateHash`. Length alone is not enough — a fork or a sibling thread can sit at the same index.
158
+ - Fast path: `agentMeta.logId` names the log, `logIndex` equals the log length, and the snapshot's state hash matches the tail entry's `verification.stateHash`. Length alone is not enough, because a fork or a sibling thread can sit at the same index.
88
159
  - Otherwise the log is replayed and the run resumes from the replayed state.
89
160
  - A cache that disagrees with the state the log replays to throws `AgentSnapshotDivergedError` (code `snapshot-diverged`). Drop the snapshot and resume from `events` alone.
90
161
 
91
162
  A snapshot is never authoritative within a version. Snapshot only at quiescent points: a mid-flight snapshot cannot carry in-flight request state, but the log can.
92
163
 
93
- Resuming from a snapshot with no log also yields a self-contained log — the new segment's init entry carries that snapshot — so every result is replayable.
164
+ Resuming from a snapshot with no log also yields a self-contained log (the new segment's init entry carries that snapshot), so every result is replayable.
94
165
 
95
166
  ## Version bridge
96
167
 
@@ -85,7 +85,7 @@ Requests have a semantic `name`, resolved `input`, model reference, schemas,
85
85
  prompt/messages, and tools. The machine owns control flow; the executor owns
86
86
  the provider call.
87
87
 
88
- An executor is just a function, so the machine runs anywhere a function does.
88
+ An executor is a function, so the machine runs anywhere a function does.
89
89
  When several requests each need their own canned answer, the executor routes on
90
90
  `request.name`. See [Evals](evals.md#testing-without-a-provider).
91
91
 
package/docs/steps.md CHANGED
@@ -40,6 +40,8 @@ with an external timer pending. When it fires, deliver
40
40
  `{ type: "xstate.timer", id }` through `transition` or as the `event` passed to
41
41
  `runToQuiescence`.
42
42
 
43
+ A resumed run calls `schedule` again for each timer still pending, with what is left of its delay by the wall clock. Treat `schedule` as an upsert by `id`: a host that already holds that timer keeps its own due time. An `id` names a timer by its state and delay, not by arming, so a state entered again schedules the same `id` once the earlier timer has fired or been cancelled.
44
+
43
45
  An invoked child machine's `after` timers always run in process, and the run
44
46
  waits for them, whichever `timers` option you pass.
45
47
 
@@ -190,6 +190,7 @@ A wrapper is still host-side. The machine cannot read its counts or react to the
190
190
  - Delivery is opt-in. The event is delivered only when an active state declares a transition that matches it, either `'@agent.usage'` or the wildcard `'*'`. Otherwise no usage event is sent to the machine. Wildcard matching follows XState semantics, so a catch-all `on: { '*': … }` does receive `@agent.usage`. Narrow the handler on `event.type` when the catch-all is meant for other events.
191
191
  - Declare the handler at machine level. A root `on: { '@agent.usage': … }` catches every call regardless of which state made it. A state-scoped handler drops calls made in other states.
192
192
  - Delivery goes to the run's root machine, including usage from requests inside an invoked child machine. Attribute those calls with `id`, `src`, and `model`.
193
+ - Usage counts real spend across a crash. A call that re-executes on resume reports its usage again, so a token budget sees both attempts when the first attempt's usage was journaled before the crash. A call in flight at the crash that never reported usage is not counted.
193
194
 
194
195
  The payload has this shape.
195
196
 
@@ -202,6 +203,7 @@ The payload has this shape.
202
203
  src?: string, // the request's invoke src
203
204
  model?: string, // the model ref it targeted
204
205
  name?: string, // a text request's registered name
206
+ callKey?: string, // the call's idempotency key, with an event log
205
207
  budget?: { modelCalls, maxModelCalls }, // counted against maxModelCalls so far, this call included
206
208
  }
207
209
  ```
package/docs/verify.md CHANGED
@@ -310,9 +310,11 @@ const result = await canReach(waitingMachine, "timedOut");
310
310
  - The event is `{ type: '@agent.usage', kind: 'text', id, src, model, name?, usage }`, the run's shape without `budget`.
311
311
  - Each listed value forks one branch, recorded in the path before `xstate.done.actor`.
312
312
  - One value with one canned output is no fork: the output resolves straight through, but the usage event is still recorded in the path.
313
- - A usage transition that leaves the invoking state cancels the request: its completion is not recorded.
313
+ - A usage value the active states refuse (`snapshot.can` is false) is dropped, as a run drops it: the request settles with no usage event delivered or recorded.
314
+ - A usage transition that leaves the invoking state, or restarts its invoke, cancels the request: its completion is not recorded, and a restarted invoke settles later on its own.
314
315
  - A settling request with no `usage` entry while a usage transition is active lists `@agent.usage` in `vacuous`. Completeness is unchanged.
315
- - Requests inside child machines and decisions deliver no usage.
316
+ - With `children: "coupled"`, a child's text request with an entry forks one branch per value. The usage event goes to the root before the child's completion reaches its parent. The root takes it or drops it by the same rules. A child's request with no entry is not listed in `vacuous`.
317
+ - Requests inside `"outputs"` or `"explore"` children, and decisions, deliver no usage.
316
318
 
317
319
  ```ts no-check
318
320
  const overBudget = await canReach(researchMachine, "outOfBudget", {
@@ -360,7 +362,9 @@ const report = await explorePaths(parentMachine, {
360
362
  - The parent's host events arrive at its idle waits, including while every child waits on the host. While a child or the parent has work pending, they race it only with `interleave: "pending"`.
361
363
  - What a child's move sends its own children, and their replies, happen in the same move. A child's send to a sibling is delivered to that sibling, entry sends included. A child's sends reach their targets in the order it made them, each run to quiescence before the next, as a live run does. A send to an actor the search does not model lists the event type in `unexpandedEvents`, so the report is not complete.
362
364
  - Child moves count against `maxDepth` like the parent's own branches. On a path they carry `actor` with the child's id (a grandchild's as `child.grandchild`): the host events it received, its choices, its settled work. The events it sends the parent appear as the parent receives them.
365
+ - A target predicate and `dedupeBy` see each coupled child's explored state: `snapshot.children[id].getSnapshot()` returns the state the search stepped the child to, its own coupled children likewise. A parent's send and the child's receipt are one state, so a cross-actor invariant such as "every chunk the parent forwarded, a child received" can be a target.
363
366
  - With `dedupeBy`, each child's state joins the key. Without it, every interleaving is explored in full.
367
+ - A coupled child's text requests deliver `@agent.usage` to the root, as a run does. See [Usage events](#usage-events).
364
368
  - A child that stops on `needs-output` or `max-depth` ends the parent branch the same way, so the parent report is not complete.
365
369
 
366
370
  Pass `hostEvents` when some types the parent accepts come only from its children, such as a child's status events. The search then offers only the listed types at the parent's waits.
@@ -479,6 +483,7 @@ For machines authored as data, validate the config with `validateAgentConfig(con
479
483
 
480
484
  ## Related
481
485
 
486
+ - [Crash testing](persistence.md#crash-testing): `checkCrashRecovery` kills a run at every log append and in-flight effect, and checks each resume.
482
487
  - [Debugging](debugging.md): scripted reproduction and the diagnostic codes in context.
483
488
  - [Evals](evals.md): scoring runs on output, trajectory, and budget.
484
489
  - [Quickstart](quickstart.md#define-and-run-one-artifact): a plain-function executor for `runToQuiescence`.
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@statelyai/agent",
3
- "version": "2.0.0-alpha.36",
3
+ "version": "2.0.0-alpha.38",
4
4
  "description": "Make invalid agent actions impossible. Agent logic as state machines: deterministic, inspectable, resumable, runs anywhere.",
5
5
  "keywords": [
6
6
  "agent",
@@ -62,7 +62,10 @@ expect(result).toMatchObject({ reachable: false, complete: true });
62
62
 
63
63
  ## 5. Crash and resume
64
64
 
65
- Run with a store and a thread id, kill the run mid-flight (throw from an executor, or exit the process), then call `runAgent` again with the same store and `threadId`.
65
+ Call `checkCrashRecovery(machine, { input, events, executors })` from `@statelyai/agent/testing` for each scenario, and assert `report.failures` is `[]`. It kills the run after every log append and while every effect is in flight, resumes each from the log, and checks the resume against an uninterrupted run. See `docs/persistence.md#crash-testing`.
66
+
67
+ - Executors and fake tools must answer from the request, not call order. A fake that keeps state passes `reset`.
68
+ - A failure names the crash point and the problem (`reran`, `skipped`, `outcome`, `stuck`, …). Fix the machine or the effect, never the check.
66
69
 
67
70
  - A completed model call never runs again. An in-flight one re-runs with the same `info.callKey`.
68
71
  - A plain actor (`createAsyncLogic`) in flight at the crash re-runs. It gets no call key, so derive an idempotency key from the domain (the ticket id, the booking key) and dedupe in the effect.