workflow 5.0.0-beta.11 → 5.0.0-beta.13
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/docs/ai/index.mdx +21 -18
- package/docs/api-reference/workflow-ai/durable-agent.mdx +7 -45
- package/docs/changelog/eager-processing.mdx +67 -413
- package/docs/changelog/resilient-start.mdx +31 -283
- package/docs/cookbook/agent-patterns/durable-agent.mdx +10 -146
- package/docs/cookbook/index.mdx +1 -1
- package/docs/errors/corrupted-event-log.mdx +5 -5
- package/docs/errors/index.mdx +3 -0
- package/docs/errors/replay-divergence.mdx +27 -0
- package/docs/foundations/streaming.mdx +13 -22
- package/docs/internal/index.mdx +2 -0
- package/docs/internal/meta.json +6 -1
- package/docs/internal/nitro-native-build.mdx +38 -0
- package/docs/internal/nitro-web-ui.mdx +24 -0
- package/package.json +11 -11
|
@@ -7,321 +7,69 @@ description: Overhaul run start logic to tolerate world storage unavailability,
|
|
|
7
7
|
|
|
8
8
|
## Motivation
|
|
9
9
|
|
|
10
|
-
When `world` storage is unavailable but the queue is up,
|
|
11
|
-
`start()` previously failed entirely because `world.events.create(run_created)`
|
|
12
|
-
is called before `world.queue()`. This change decouples run creation from queue
|
|
13
|
-
dispatch so that runs can still be accepted when storage is degraded.
|
|
10
|
+
When `world` storage is unavailable but the queue is up, `start()` previously failed entirely because `world.events.create(run_created)` is called before `world.queue()`. This change decouples run creation from queue dispatch so that runs can still be accepted when storage is degraded.
|
|
14
11
|
|
|
15
|
-
Additionally, the runtime previously called `world.runs.get(runId)` before
|
|
16
|
-
`run_started`, adding an extra round-trip. By always calling `run_started`
|
|
17
|
-
directly, we save that round-trip and can return pre-loaded events in the
|
|
18
|
-
response to skip the initial `events.list` call, reducing TTFB.
|
|
12
|
+
Additionally, the runtime previously called `world.runs.get(runId)` before `run_started`, adding an extra round-trip. By always calling `run_started` directly, we save that round-trip and can return pre-loaded events in the response to skip the initial `events.list` call, reducing TTFB.
|
|
19
13
|
|
|
20
14
|
## Design
|
|
21
15
|
|
|
22
|
-
### `start()` changes
|
|
16
|
+
### `start()` changes
|
|
23
17
|
|
|
24
|
-
- `world.events.create` (run_created) and `world.queue` are now called **in parallel**
|
|
25
|
-
|
|
26
|
-
- If `events.create` errors with **
|
|
27
|
-
creation failed but the run was accepted — creation will be re-tried async by the
|
|
28
|
-
runtime when it processes the queue message. The returned `Run` instance is marked
|
|
29
|
-
with `resilientStart = true`.
|
|
30
|
-
- If `events.create` errors with **409** (EntityConflictError), the run already exists
|
|
31
|
-
(e.g., the queue handler's resilient start path created it first due to a cold-start
|
|
32
|
-
race). This is treated as success.
|
|
18
|
+
- `world.events.create` (run_created) and `world.queue` are now called **in parallel** via `Promise.allSettled`.
|
|
19
|
+
- If `events.create` errors with **429 or 5xx**, we log a warning saying that run creation failed but the run was accepted — creation will be re-tried async by the runtime when it processes the queue message. The returned `Run` instance is marked with `resilientStart = true`.
|
|
20
|
+
- If `events.create` errors with **409** (EntityConflictError), the run already exists (e.g., the queue handler's resilient start path created it first due to a cold-start race). This is treated as success.
|
|
33
21
|
- If `world.queue` fails, we still throw — the run truly failed and was not enqueued.
|
|
34
|
-
- The queue invocation now receives all the run inputs (`input`, `deploymentId`,
|
|
35
|
-
|
|
36
|
-
create the run later if needed.
|
|
37
|
-
- When the runtime re-enqueues itself, it does **not** pass these inputs — only the
|
|
38
|
-
first queue cycle carries them.
|
|
22
|
+
- The queue invocation now receives all the run inputs (`input`, `deploymentId`, `workflowName`, `specVersion`, `executionContext`) via `runInput` so the runtime can create the run later if needed.
|
|
23
|
+
- When the runtime re-enqueues itself, it does **not** pass these inputs — only the first queue cycle carries them.
|
|
39
24
|
|
|
40
|
-
### `workflowEntrypoint` changes
|
|
25
|
+
### `workflowEntrypoint` changes
|
|
41
26
|
|
|
42
|
-
- When calling `world.events.create` with `run_started`, we now also always pass the
|
|
43
|
-
run input that was sent through the queue, if available. The response will still be on off:
|
|
44
|
-
- **200 with event (now running)**: As usual, but the server could have used the run input to create the run if it didn't exist yet. The response will be opaque to the runtime.
|
|
45
|
-
- **200 without event (already running)**: As usual
|
|
46
|
-
- **409 or 410 (already finished)**: As usual
|
|
27
|
+
- When calling `world.events.create` with `run_started`, we now also always pass the run input that was sent through the queue, if available. The world is responsible for creating the run if it doesn't already exist.
|
|
47
28
|
|
|
48
|
-
### `Run.returnValue` polling
|
|
29
|
+
### `Run.returnValue` polling
|
|
49
30
|
|
|
50
|
-
- When `resilientStart` is true on the Run instance (run_created failed), the
|
|
51
|
-
|
|
52
|
-
(1s + 3s + 6s = 10s total) to give the queue time to deliver and the runtime
|
|
53
|
-
to create the run via `run_started`.
|
|
54
|
-
- When `resilientStart` is false (normal path), 404 fails immediately — no delay
|
|
55
|
-
for the common case of a wrong run ID.
|
|
31
|
+
- When `resilientStart` is true on the Run instance (run_created failed), the `pollReturnValue` loop retries on `WorkflowRunNotFoundError` up to 3 times (1s + 3s + 6s = 10s total) to give the queue time to deliver and the runtime to create the run via `run_started`.
|
|
32
|
+
- When `resilientStart` is false (normal path), 404 fails immediately — no delay for the common case of a wrong run ID.
|
|
56
33
|
|
|
57
|
-
### World
|
|
34
|
+
### World contract changes
|
|
58
35
|
|
|
59
|
-
- Posting `run_started` to a **non-existent** run is now allowed when the run input is
|
|
60
|
-
|
|
61
|
-
1. Creates a `run_created` event first (so the event log is consistent).
|
|
62
|
-
2. Strips the input from the `run_started` event data (it lives on `run_created`).
|
|
63
|
-
3. Then creates the `run_started` event normally.
|
|
64
|
-
4. Emits a log and a Datadog metric (`workflow_server.resilient_start.run_created_via_run_started`)
|
|
65
|
-
to track when this fallback path is hit.
|
|
66
|
-
- When `run_started` encounters an **already-running** run, all worlds return `{ run }`
|
|
67
|
-
with `event: undefined` instead of throwing. No duplicate event is created.
|
|
36
|
+
- Posting `run_started` to a **non-existent** run is now allowed when the run input is sent along with the payload. The world creates a `run_created` event first (so the event log is consistent), then creates the `run_started` event normally.
|
|
37
|
+
- When `run_started` encounters an **already-running** run, all worlds return `{ run }` with `event: undefined` instead of throwing. No duplicate event is created.
|
|
68
38
|
|
|
69
39
|
### Queue transport changes
|
|
70
40
|
|
|
71
|
-
`Uint8Array` values (the serialized workflow input in `runInput`) don't survive plain
|
|
72
|
-
JSON serialization. Each world uses a transport that preserves binary data:
|
|
41
|
+
`Uint8Array` values (the serialized workflow input in `runInput`) don't survive plain JSON serialization. Each world uses a transport that preserves binary data:
|
|
73
42
|
|
|
74
|
-
- **world-vercel**: CBOR transport — CBOR-encodes the entire queue payload into a
|
|
75
|
-
|
|
76
|
-
- **world-
|
|
77
|
-
from `fs.ts` that encode Uint8Array as `{ __type: 'Uint8Array', data: '<base64>' }`.
|
|
78
|
-
- **world-postgres**: Inline typed JSON transport — same tagged-envelope approach as
|
|
79
|
-
world-local, inlined since world-postgres doesn't import from world-local.
|
|
43
|
+
- **world-vercel**: CBOR transport — CBOR-encodes the entire queue payload into a `Buffer` and uses `BufferTransport` from `@vercel/queue`. Uint8Array survives natively.
|
|
44
|
+
- **world-local**: `TypedJsonTransport` — encodes Uint8Array as `{ __type: 'Uint8Array', data: '<base64>' }`.
|
|
45
|
+
- **world-postgres**: Inline typed JSON transport — same tagged-envelope approach as world-local.
|
|
80
46
|
|
|
81
47
|
## Decisions
|
|
82
48
|
|
|
83
|
-
1. **Parallel not sequential**: We chose `Promise.allSettled` over sequential calls to
|
|
84
|
-
minimize latency in the happy path.
|
|
49
|
+
1. **Parallel not sequential**: We chose `Promise.allSettled` over sequential calls to minimize latency in the happy path.
|
|
85
50
|
|
|
86
|
-
2. **Already-running returns run without event**: When `run_started` encounters an
|
|
87
|
-
already-running run, all worlds return `{ run }` with `event: undefined` (no
|
|
88
|
-
`events` array) instead of throwing. The runtime detects this by checking for
|
|
89
|
-
`result.event === undefined`. This avoids an extra `world.runs.get` round-trip.
|
|
51
|
+
2. **Already-running returns run without event**: When `run_started` encounters an already-running run, all worlds return `{ run }` with `event: undefined` (no `events` array) instead of throwing. The runtime detects this by checking for `result.event === undefined`. This avoids an extra `world.runs.get` round-trip.
|
|
90
52
|
|
|
91
|
-
3. **Events in 200 response**: We only return events on the 200 path (first caller).
|
|
92
|
-
On the already-running path, we fall back to the normal `events.list` call. This is
|
|
93
|
-
correct because only on 200 can we be certain we know the full event history.
|
|
53
|
+
3. **Events in 200 response**: We only return events on the 200 path (first caller). On the already-running path, we fall back to the normal `events.list` call. This is correct because only on 200 can we be certain we know the full event history.
|
|
94
54
|
|
|
95
|
-
4. **Conditional 404 retry on Run.returnValue**: Only when `resilientStart = true`
|
|
96
|
-
(run_created failed). Normal runs fail fast on 404.
|
|
55
|
+
4. **Conditional 404 retry on Run.returnValue**: Only when `resilientStart = true` (run_created failed). Normal runs fail fast on 404.
|
|
97
56
|
|
|
98
57
|
## Known concerns
|
|
99
58
|
|
|
100
|
-
### Cold-start race on Vercel
|
|
59
|
+
### Cold-start race on Vercel
|
|
101
60
|
|
|
102
|
-
On Vercel, the parallel dispatch can cause the queue message to be processed before
|
|
103
|
-
`run_created` completes, if `run_created` hits a cold-start lambda. Confirmed via
|
|
104
|
-
Datadog: the `run_started` request hit a warm lambda (23ms) while `run_created` hit
|
|
105
|
-
a cold lambda (727ms), even though `run_created` arrived at the edge 116ms earlier.
|
|
106
|
-
When this happens:
|
|
61
|
+
On Vercel, the parallel dispatch can cause the queue message to be processed before `run_created` completes, if `run_created` hits a cold-start lambda. When this happens:
|
|
107
62
|
|
|
108
63
|
1. The runtime's resilient start path creates the run from `run_started`.
|
|
109
64
|
2. The original `run_created` arrives and gets 409 (EntityConflictError).
|
|
110
65
|
3. `start()` treats the 409 as success (the run exists).
|
|
111
66
|
|
|
112
|
-
|
|
113
|
-
in this case (409 is not a retryable error), so `returnValue` fails fast on 404.
|
|
67
|
+
The `resilientStart` flag is NOT set on the Run instance in this case (409 is not a retryable error), so `returnValue` fails fast on 404.
|
|
114
68
|
|
|
115
|
-
###
|
|
69
|
+
### Atomicity of run entity creation
|
|
116
70
|
|
|
117
|
-
|
|
118
|
-
`events.create(run_created)` finishes writing to the shared filesystem. The
|
|
119
|
-
resilient start path should handle this, but Local Prod tests showed occasional
|
|
120
|
-
runs stuck at `pending` (no `run_started` event), and Windows CI showed
|
|
121
|
-
"Unconsumed event in event log" errors from duplicate `run_created` events.
|
|
71
|
+
The normal `run_created` path and the resilient start path can race on creating the run entity. In `world-local`, both paths use `writeExclusive` (O_CREAT|O_EXCL) — atomic at the OS level, so exactly one writer wins and the other gets EEXIST. The normal path throws `EntityConflictError` on conflict (handled by `start()` as 409); the resilient start path re-reads the run from disk on conflict.
|
|
122
72
|
|
|
123
|
-
|
|
124
|
-
resilient start path. Both used `writeJSON` which checks existence with
|
|
125
|
-
`fs.access()` (non-atomic), so both could pass the check and write separate
|
|
126
|
-
`run_created` events with different event IDs. Fixed by switching both paths to
|
|
127
|
-
`writeExclusive` (O_CREAT|O_EXCL) — see retrospective items 12 and 16.
|
|
73
|
+
In `world-postgres`, the resilient start path uses `onConflictDoNothing` plus a re-read on conflict for the same effect, with the same outcome on either side of the race.
|
|
128
74
|
|
|
129
|
-
|
|
130
|
-
|
|
131
|
-
- [x] ~~Investigate Local Prod test flakiness~~ — resolved via `writeExclusive`
|
|
132
|
-
for run entity creation (retrospective items 12, 16).
|
|
133
|
-
- [ ] Monitor the Datadog metric in production to understand how often the fallback is hit.
|
|
134
|
-
- [x] ~~Events optimization for re-enqueue cycles~~ — decided against. The
|
|
135
|
-
already-running path returns early without writing an event, so preloading
|
|
136
|
-
events there would require an extra filesystem/DB query on every re-enqueue.
|
|
137
|
-
More importantly, on Vercel with at-least-once delivery, multiple lambdas can
|
|
138
|
-
process the same run concurrently — the event snapshot could be stale or
|
|
139
|
-
incomplete. The runtime's fallback to `events.list` is the correct behavior
|
|
140
|
-
for re-enqueue cycles.
|
|
141
|
-
- [x] ~~CborTransport pass-through~~ — refactored. `encode()`/`decode()` now
|
|
142
|
-
live inside `CborTransport.serialize()`/`deserialize()`, matching the pattern
|
|
143
|
-
used by TypedJsonTransport (world-local) and the inline transport
|
|
144
|
-
(world-postgres). Call sites pass plain objects instead of pre-encoded buffers.
|
|
145
|
-
|
|
146
|
-
## Development retrospective
|
|
147
|
-
|
|
148
|
-
Chronological log of mistakes, misunderstandings, and reverted approaches during
|
|
149
|
-
development. Included for future reference when working on similar cross-cutting
|
|
150
|
-
runtime changes.
|
|
151
|
-
|
|
152
|
-
### 1. Uint8Array corruption through JSON queue transport
|
|
153
|
-
|
|
154
|
-
The initial implementation passed `runInput.input` (a `Uint8Array`) directly through
|
|
155
|
-
the queue payload. `Uint8Array` doesn't survive `JSON.stringify` — it becomes
|
|
156
|
-
`{"0":72,"1":101,...}`. This corrupted the workflow input when the resilient start
|
|
157
|
-
path tried to recreate the run from the queue-delivered data.
|
|
158
|
-
|
|
159
|
-
Caught by the `spawnWorkflowFromStepWorkflow` e2e test and the `world-testing`
|
|
160
|
-
embedded tests, which failed with "Invalid input" from devalue's `unflatten()`.
|
|
161
|
-
|
|
162
|
-
Three approaches were tried before landing on the final solution:
|
|
163
|
-
|
|
164
|
-
1. **Base64 encoding** (`btoa`/`atob`) — worked but fragile. The decode side used
|
|
165
|
-
`typeof runInput.input === 'string'` as a discriminant, which was flagged as
|
|
166
|
-
dangerous since non-binary inputs could also be strings.
|
|
167
|
-
2. **`Array.from()`/`new Uint8Array()`** — replaced base64 with a plain number array.
|
|
168
|
-
Two problems: (a) 3x JSON size regression vs base64, and (b) `Array.isArray()`
|
|
169
|
-
false-positives on v1Compat runs where `dehydrateWorkflowArguments` returns
|
|
170
|
-
devalue's flat Array format.
|
|
171
|
-
3. **CBOR + BufferTransport** (final) — world-vercel CBOR-encodes the queue payload;
|
|
172
|
-
world-local and world-postgres use a `TypedJsonTransport` with a tagged envelope.
|
|
173
|
-
|
|
174
|
-
### 2. Forgot to commit world-postgres transport fix (twice)
|
|
175
|
-
|
|
176
|
-
After fixing world-local and world-vercel queue transports, the same `JsonTransport`
|
|
177
|
-
corruption bug existed in world-postgres. The fix was written during a session but
|
|
178
|
-
never committed — lost when the working directory was reset via stash/checkout. This
|
|
179
|
-
happened twice. The fix only landed on the third attempt when it was committed and
|
|
180
|
-
pushed immediately. All 14 Postgres e2e jobs failed each time.
|
|
181
|
-
|
|
182
|
-
### 3. Incorrect diagnosis of Vercel Prod 409 errors
|
|
183
|
-
|
|
184
|
-
Multiple Vercel Prod e2e tests failed with `EntityConflictError: Workflow run with
|
|
185
|
-
ID wrun_... already exists` on `run_created`. The initial assumption was that VQS
|
|
186
|
-
couldn't deliver the queue message fast enough to beat the `run_created` call.
|
|
187
|
-
|
|
188
|
-
Datadog logs showed otherwise: the `run_created` request arrived at Vercel's edge
|
|
189
|
-
116ms before `run_started`, but `run_created` hit a cold-start lambda (727ms) while
|
|
190
|
-
`run_started` hit a warm one (23ms). Cold starts can invert expected execution order.
|
|
191
|
-
|
|
192
|
-
### 4. Removed EntityConflictError catch, then had to restore it
|
|
193
|
-
|
|
194
|
-
The `workflowEntrypoint` error handler originally caught both `EntityConflictError`
|
|
195
|
-
and `RunExpiredError`. When adding the "already-running returns run without event"
|
|
196
|
-
behavior, `EntityConflictError` was removed from the catch since the new worlds
|
|
197
|
-
wouldn't throw it. Reviewer flagged this: old worlds or world-vercel hitting an
|
|
198
|
-
older workflow-server could still throw it. The catch was restored.
|
|
199
|
-
|
|
200
|
-
### 5. Duplicate `startedAt` check
|
|
201
|
-
|
|
202
|
-
After refactoring the `run_started` flow, a `workflowRun.startedAt` null check
|
|
203
|
-
existed both inside the `try` block and after the `catch` block. The second was
|
|
204
|
-
unreachable. Removed after review.
|
|
205
|
-
|
|
206
|
-
### 6. WORKFLOW_SERVER_URL_OVERRIDE left set
|
|
207
|
-
|
|
208
|
-
During development, `WORKFLOW_SERVER_URL_OVERRIDE` was set to a test URL pointing
|
|
209
|
-
at the workflow-server preview deployment and accidentally committed. The Vercel
|
|
210
|
-
bot flagged this. Reset to empty string.
|
|
211
|
-
|
|
212
|
-
### 7. e2e test assertion was wrong
|
|
213
|
-
|
|
214
|
-
The resilient start e2e test stubbed `world.events.create` and asserted
|
|
215
|
-
`createCallCount >= 2`. But the stub only intercepts calls from the test runner
|
|
216
|
-
process — the server uses its own world. `createCallCount` was always 1. Changed
|
|
217
|
-
to `expect(createCallCount).toBe(1)`.
|
|
218
|
-
|
|
219
|
-
### 8. Misattributed Local Prod timeouts as "pre-existing"
|
|
220
|
-
|
|
221
|
-
Local Prod tests showed 60-second timeouts across various tests. Initially dismissed
|
|
222
|
-
as CI flakes. Checking main's CI showed all Local Prod tests pass on main — the
|
|
223
|
-
timeouts are caused by our changes. Should have compared against main immediately.
|
|
224
|
-
|
|
225
|
-
### 9. Attempted to revert parallel dispatch
|
|
226
|
-
|
|
227
|
-
After identifying Local Prod timeouts, `start()` was partially reverted back to
|
|
228
|
-
sequential dispatch. The user pointed out that parallel dispatch is the core value
|
|
229
|
-
proposition of the PR. The revert was undone.
|
|
230
|
-
|
|
231
|
-
### 10. WorkflowRunNotFoundError retry was unconditional
|
|
232
|
-
|
|
233
|
-
The initial `pollReturnValue` retry on `WorkflowRunNotFoundError` applied to all
|
|
234
|
-
`Run` instances. A user calling `getRun()` with a wrong ID would wait 10 seconds
|
|
235
|
-
before getting a 404. Fixed by adding a `resilientStart` flag: only retries when
|
|
236
|
-
`run_created` actually failed.
|
|
237
|
-
|
|
238
|
-
### 11. Changeset `minor` vs `patch`
|
|
239
|
-
|
|
240
|
-
The changeset was created with `"@workflow/core": minor`. Reviewer flagged this as
|
|
241
|
-
violating repo rules ("all changes should be patch"). Changed after discussion.
|
|
242
|
-
|
|
243
|
-
### 12. world-local TOCTOU race causing duplicate `run_created` events (Windows CI)
|
|
244
|
-
|
|
245
|
-
The resilient start path AND the normal `run_created` path in `world-local/events-storage.ts`
|
|
246
|
-
both used `writeJSON` to create the run entity. `writeJSON` checks file existence with
|
|
247
|
-
`fs.access()` then writes via temp+rename — a classic TOCTOU race. On the local world,
|
|
248
|
-
the queue delivers via an async IIFE in the same event loop, so `events.create(run_created)`
|
|
249
|
-
and `events.create(run_started)` (with resilient start) run concurrently:
|
|
250
|
-
|
|
251
|
-
1. Both paths call `fs.access(runPath)` → ENOENT (file doesn't exist yet)
|
|
252
|
-
2. Both proceed to write → the last `fs.rename` wins
|
|
253
|
-
3. Both succeed → both write their own `run_created` event with different event IDs
|
|
254
|
-
4. During replay, the consumer sees two `run_created` events → "Unconsumed event" error
|
|
255
|
-
|
|
256
|
-
This caused consistent failures in `world-testing` embedded tests on Windows CI (`hooks`,
|
|
257
|
-
`supports null bytes in step results`, `retriable and fatal errors` — all timing out at
|
|
258
|
-
60s with "Unconsumed event in event log" errors). Linux CI was not affected because the
|
|
259
|
-
timing was different enough that the race window was rarely hit.
|
|
260
|
-
|
|
261
|
-
Fixed by switching BOTH paths to `writeExclusive` (O_CREAT|O_EXCL), which is atomic at
|
|
262
|
-
the OS level — exactly one writer wins, the other gets EEXIST. The normal `run_created`
|
|
263
|
-
path throws `EntityConflictError` on conflict (handled by `start()` as 409). The resilient
|
|
264
|
-
start path re-reads the run from disk on conflict. Either way, only one `run_created`
|
|
265
|
-
event is written.
|
|
266
|
-
|
|
267
|
-
### 13. Non-atomic run + run_created event in world-postgres resilient path
|
|
268
|
-
|
|
269
|
-
The resilient start path in `world-postgres/storage.ts` did two separate writes (run
|
|
270
|
-
insert, then event insert) without a transaction. If the process crashed between them,
|
|
271
|
-
the run would exist without a `run_created` event — an inconsistent event log.
|
|
272
|
-
|
|
273
|
-
A `drizzle.transaction()` wrapper was attempted but dropped due to TypeScript inference
|
|
274
|
-
issues with drizzle's transaction callback and the insert builder's overloads. The current
|
|
275
|
-
fix keeps the two writes sequential but adds the same conflict-aware re-read pattern as
|
|
276
|
-
world-local: when `onConflictDoNothing` produces no result (run already existed), the run
|
|
277
|
-
is re-read so downstream logic sees the real state. The narrow crash window between the
|
|
278
|
-
two writes is acceptable — if the run insert succeeds but the event insert crashes, the
|
|
279
|
-
run exists and `run_started` will still proceed normally (the event log will be missing a
|
|
280
|
-
`run_created` entry, but the run itself is functional).
|
|
281
|
-
|
|
282
|
-
### 14. Missing `WorkflowRunStatus` span attribute after parallel refactor
|
|
283
|
-
|
|
284
|
-
The `start()` span previously set `Attribute.WorkflowRunStatus(result.run.status)`, but
|
|
285
|
-
this was dropped in the parallel refactor because `result.run` is only available when
|
|
286
|
-
`runCreatedResult` fulfilled. The attribute is now conditionally set when the result is
|
|
287
|
-
available. In the resilient start case (run_created failed), the attribute is omitted
|
|
288
|
-
rather than erroring.
|
|
289
|
-
|
|
290
|
-
### 15. `run_started` eventData leak in world-postgres result
|
|
291
|
-
|
|
292
|
-
The `...data` spread in the result construction leaked `eventData` from `run_started`
|
|
293
|
-
into the returned event object. Storage was already correct (`storedEventData` is
|
|
294
|
-
`undefined` for `run_started`), but the returned result carried the input data. While
|
|
295
|
-
harmless (the runtime doesn't use `result.event.eventData`), it was restored to match
|
|
296
|
-
the pre-refactor behavior where eventData was explicitly stripped from the result.
|
|
297
|
-
|
|
298
|
-
### 16. Normal `run_created` path also needed `writeExclusive` (Windows CI)
|
|
299
|
-
|
|
300
|
-
The initial TOCTOU fix (item 12) only changed the resilient start path to use
|
|
301
|
-
`writeExclusive`. The normal `run_created` entity write still used `writeJSON` which
|
|
302
|
-
checks existence with `fs.access()` then writes via temp+rename — not atomic. On
|
|
303
|
-
Windows CI, the local queue's async IIFE delivered fast enough for both paths to pass
|
|
304
|
-
their existence checks simultaneously, producing two `run_created` events with different
|
|
305
|
-
event IDs. The events consumer saw the duplicate as "Unconsumed event in event log,"
|
|
306
|
-
causing `hooks`, `supports null bytes in step results`, and `retriable and fatal errors`
|
|
307
|
-
tests to time out at 60s. Fixed by also switching the normal `run_created` entity write to
|
|
308
|
-
`writeExclusive`, making both paths use the same atomic gate.
|
|
309
|
-
|
|
310
|
-
### 17. CborTransport was a pass-through wrapper
|
|
311
|
-
|
|
312
|
-
`world-vercel/queue.ts` had `CborTransport` implementing `Transport<Buffer>` with a
|
|
313
|
-
no-op `serialize` (identity function) and a `deserialize` that reassembled chunks into
|
|
314
|
-
a Buffer without decoding. The actual CBOR `encode()`/`decode()` calls happened at the
|
|
315
|
-
call sites — `queue()` pre-encoded before calling `client.send()`, and the handler
|
|
316
|
-
post-decoded after receiving from `client.handleCallback()`. This violated the transport
|
|
317
|
-
abstraction (every other transport does its encoding inside serialize/deserialize) and
|
|
318
|
-
meant the call site had to remember to pre-encode. Refactored to move `encode()`/`decode()`
|
|
319
|
-
into the transport methods and changed the type from `Transport<Buffer>` to
|
|
320
|
-
`Transport<unknown>`.
|
|
321
|
-
|
|
322
|
-
## Follow-up work (additional)
|
|
323
|
-
|
|
324
|
-
- [x] ~~**CborTransport is a pass-through**~~ — Resolved. Moved `encode()`/`decode()`
|
|
325
|
-
into `CborTransport.serialize()`/`CborTransport.deserialize()`. The transport is now
|
|
326
|
-
self-contained: call sites pass plain objects, and the handler receives decoded objects.
|
|
327
|
-
See retrospective item 17.
|
|
75
|
+
The narrow crash window in `world-postgres` between the run insert and the event insert is acceptable — if the run insert succeeds but the event insert crashes, the run exists and `run_started` will still proceed normally (the event log will be missing a `run_created` entry, but the run itself is functional).
|
|
@@ -1,154 +1,18 @@
|
|
|
1
1
|
---
|
|
2
|
-
title:
|
|
3
|
-
description:
|
|
2
|
+
title: DurableAgent is now WorkflowAgent
|
|
3
|
+
description: Use AI SDK v7's WorkflowAgent for durable, resumable AI agents.
|
|
4
4
|
type: guide
|
|
5
|
-
summary:
|
|
5
|
+
summary: Build durable, resumable AI agents with AI SDK v7's WorkflowAgent.
|
|
6
6
|
---
|
|
7
7
|
|
|
8
|
-
|
|
8
|
+
## WorkflowAgent from AI SDK v7
|
|
9
9
|
|
|
10
|
-
|
|
10
|
+
Use AI SDK v7's [`WorkflowAgent`](https://ai-sdk.dev/v7/docs/agents/workflow-agent#workflowagent) for new durable agent work. It replaces `DurableAgent` in v5 and keeps the current agent pattern in the AI SDK package.
|
|
11
11
|
|
|
12
|
-
-
|
|
13
|
-
-
|
|
14
|
-
-
|
|
15
|
-
- Agents that need per-step observability in the workflow event log
|
|
12
|
+
- Import `WorkflowAgent` from `@ai-sdk/workflow` and run it inside a `"use workflow"` function.
|
|
13
|
+
- Stream `ModelCallStreamPart` chunks with `getWritable()`, then convert the run stream to UI message chunks with `createModelCallToUIChunkTransform()` in your route.
|
|
14
|
+
- Mark tool `execute` functions with `"use step"` when they should run as durable workflow steps with retry and observability behavior.
|
|
16
15
|
|
|
17
|
-
<Callout type="
|
|
18
|
-
|
|
16
|
+
<Callout type="warn">
|
|
17
|
+
`DurableAgent` is deprecated in v5 and remains documented for existing code only. Existing `DurableAgent` users can refer to the [`DurableAgent` API reference](/docs/api-reference/workflow-ai/durable-agent) while migrating.
|
|
19
18
|
</Callout>
|
|
20
|
-
|
|
21
|
-
## Pattern
|
|
22
|
-
|
|
23
|
-
Replace `Agent` with `DurableAgent`, wrap the function in `"use workflow"`, mark each tool with `"use step"`, and stream output through `getWritable()`.
|
|
24
|
-
|
|
25
|
-
### Workflow
|
|
26
|
-
|
|
27
|
-
```typescript
|
|
28
|
-
import { DurableAgent } from "@workflow/ai/agent";
|
|
29
|
-
import { getWritable } from "workflow";
|
|
30
|
-
import { z } from "zod";
|
|
31
|
-
import type { ModelMessage, UIMessageChunk } from "ai";
|
|
32
|
-
|
|
33
|
-
async function searchFlights({ from, to, date }: {
|
|
34
|
-
from: string;
|
|
35
|
-
to: string;
|
|
36
|
-
date: string;
|
|
37
|
-
}) {
|
|
38
|
-
"use step"; // [!code highlight]
|
|
39
|
-
const res = await fetch(
|
|
40
|
-
`https://api.example.com/flights?from=${from}&to=${to}&date=${date}`
|
|
41
|
-
);
|
|
42
|
-
if (!res.ok) throw new Error(`Search failed: ${res.status}`);
|
|
43
|
-
return res.json();
|
|
44
|
-
}
|
|
45
|
-
|
|
46
|
-
async function bookFlight({ flightId, passenger }: {
|
|
47
|
-
flightId: string;
|
|
48
|
-
passenger: string;
|
|
49
|
-
}) {
|
|
50
|
-
"use step"; // [!code highlight]
|
|
51
|
-
const res = await fetch("https://api.example.com/bookings", {
|
|
52
|
-
method: "POST",
|
|
53
|
-
headers: { "Content-Type": "application/json" },
|
|
54
|
-
body: JSON.stringify({ flightId, passenger }),
|
|
55
|
-
});
|
|
56
|
-
if (!res.ok) throw new Error(`Booking failed: ${res.status}`);
|
|
57
|
-
return res.json();
|
|
58
|
-
}
|
|
59
|
-
|
|
60
|
-
async function checkWeather({ city }: { city: string }) {
|
|
61
|
-
"use step"; // [!code highlight]
|
|
62
|
-
const res = await fetch(`https://api.weather.com/forecast?city=${city}`);
|
|
63
|
-
return res.json();
|
|
64
|
-
}
|
|
65
|
-
|
|
66
|
-
export async function flightAgent(messages: ModelMessage[]) {
|
|
67
|
-
"use workflow";
|
|
68
|
-
|
|
69
|
-
const agent = new DurableAgent({ // [!code highlight]
|
|
70
|
-
model: "anthropic/claude-haiku-4.5",
|
|
71
|
-
instructions: "You are a helpful flight booking assistant.",
|
|
72
|
-
tools: {
|
|
73
|
-
searchFlights: {
|
|
74
|
-
description: "Search for available flights between two airports",
|
|
75
|
-
inputSchema: z.object({
|
|
76
|
-
from: z.string().describe("Departure airport code"),
|
|
77
|
-
to: z.string().describe("Arrival airport code"),
|
|
78
|
-
date: z.string().describe("Travel date (YYYY-MM-DD)"),
|
|
79
|
-
}),
|
|
80
|
-
execute: searchFlights,
|
|
81
|
-
},
|
|
82
|
-
bookFlight: {
|
|
83
|
-
description: "Book a specific flight for a passenger",
|
|
84
|
-
inputSchema: z.object({
|
|
85
|
-
flightId: z.string().describe("Flight ID from search results"),
|
|
86
|
-
passenger: z.string().describe("Passenger full name"),
|
|
87
|
-
}),
|
|
88
|
-
execute: bookFlight,
|
|
89
|
-
},
|
|
90
|
-
checkWeather: {
|
|
91
|
-
description: "Check the weather forecast for a city",
|
|
92
|
-
inputSchema: z.object({
|
|
93
|
-
city: z.string().describe("City name"),
|
|
94
|
-
}),
|
|
95
|
-
execute: checkWeather,
|
|
96
|
-
},
|
|
97
|
-
},
|
|
98
|
-
});
|
|
99
|
-
|
|
100
|
-
const result = await agent.stream({ // [!code highlight]
|
|
101
|
-
messages,
|
|
102
|
-
writable: getWritable<UIMessageChunk>(), // [!code highlight]
|
|
103
|
-
maxSteps: 10,
|
|
104
|
-
});
|
|
105
|
-
|
|
106
|
-
return { messages: result.messages };
|
|
107
|
-
}
|
|
108
|
-
```
|
|
109
|
-
|
|
110
|
-
### API route
|
|
111
|
-
|
|
112
|
-
```typescript
|
|
113
|
-
import type { UIMessage } from "ai";
|
|
114
|
-
import { convertToModelMessages, createUIMessageStreamResponse } from "ai";
|
|
115
|
-
import { start } from "workflow/api";
|
|
116
|
-
import { flightAgent } from "@/app/workflows/flight-agent";
|
|
117
|
-
|
|
118
|
-
export async function POST(req: Request) {
|
|
119
|
-
const { messages }: { messages: UIMessage[] } = await req.json();
|
|
120
|
-
const modelMessages = await convertToModelMessages(messages); // [!code highlight]
|
|
121
|
-
|
|
122
|
-
const run = await start(flightAgent, [modelMessages]); // [!code highlight]
|
|
123
|
-
|
|
124
|
-
return createUIMessageStreamResponse({ // [!code highlight]
|
|
125
|
-
stream: run.readable,
|
|
126
|
-
headers: {
|
|
127
|
-
"x-workflow-run-id": run.runId,
|
|
128
|
-
},
|
|
129
|
-
});
|
|
130
|
-
}
|
|
131
|
-
```
|
|
132
|
-
|
|
133
|
-
## How it works
|
|
134
|
-
|
|
135
|
-
1. **DurableAgent wraps Agent** — same API as AI SDK's `Agent`, but backed by a workflow. If the process crashes, the agent resumes from the last completed step on replay.
|
|
136
|
-
2. **Tools as steps** — each tool's `execute` function uses `"use step"`, giving it automatic retries, full Node.js access, and an entry in the workflow event log.
|
|
137
|
-
3. **Streaming** — `getWritable<UIMessageChunk>()` streams the agent's output (text chunks, tool calls, tool results) to the client in real time via `createUIMessageStreamResponse`.
|
|
138
|
-
4. **maxSteps** — limits the total number of LLM calls the agent can make, preventing runaway tool loops.
|
|
139
|
-
|
|
140
|
-
## Adapting to your use case
|
|
141
|
-
|
|
142
|
-
- **Change the model** — replace `"anthropic/claude-haiku-4.5"` with any AI Gateway model string (e.g. `"openai/gpt-4o"`, `"anthropic/claude-sonnet-4-5"`).
|
|
143
|
-
- **Add tools** — define a new `"use step"` function with a Zod schema. Each tool automatically gets retries and persistence.
|
|
144
|
-
- **Workflow-level tools** — if a tool needs workflow primitives like `sleep()` or `createHook()`, omit `"use step"` so it runs in the workflow context instead.
|
|
145
|
-
- **Multi-turn** — pass `result.messages` plus new user messages to subsequent `agent.stream()` calls for multi-turn conversations.
|
|
146
|
-
- **Client integration** — use `useChat()` from `@ai-sdk/react` with `WorkflowChatTransport` from `@workflow/ai` for a full chat UI with reconnection support.
|
|
147
|
-
|
|
148
|
-
## Key APIs
|
|
149
|
-
|
|
150
|
-
- [`"use workflow"`](/docs/api-reference/workflow/use-workflow) — declares the orchestrator function
|
|
151
|
-
- [`"use step"`](/docs/api-reference/workflow/use-step) — declares step functions with retries and full Node.js access
|
|
152
|
-
- [`DurableAgent`](/docs/api-reference/workflow-ai/durable-agent) — durable wrapper around AI SDK's Agent
|
|
153
|
-
- [`getWritable()`](/docs/api-reference/workflow/get-writable) — streams agent output to the client
|
|
154
|
-
- [`start()`](/docs/api-reference/workflow-api/start) — starts a workflow run from an API route
|
package/docs/cookbook/index.mdx
CHANGED
|
@@ -8,7 +8,7 @@ A curated collection of workflow patterns with clean, copy-paste code examples f
|
|
|
8
8
|
|
|
9
9
|
## Agent Patterns
|
|
10
10
|
|
|
11
|
-
- [**
|
|
11
|
+
- [**WorkflowAgent**](/cookbook/agent-patterns/durable-agent) — Build durable, resumable AI agents with AI SDK's WorkflowAgent
|
|
12
12
|
- [**Human-in-the-Loop**](/cookbook/agent-patterns/human-in-the-loop) — Pause an agent for human approval, then resume based on the decision
|
|
13
13
|
- [**Agent Cancellation**](/cookbook/agent-patterns/agent-cancellation) — Stop a running agent immediately via `run.cancel()` or gracefully via a hook + `Promise.race`
|
|
14
14
|
|
|
@@ -9,21 +9,21 @@ related:
|
|
|
9
9
|
- /docs/foundations/errors-and-retries
|
|
10
10
|
---
|
|
11
11
|
|
|
12
|
-
This error occurs when the Workflow runtime
|
|
12
|
+
This error occurs when the Workflow runtime repeatedly cannot replay events in the event log. This usually means the event log is in an invalid state, such as duplicate or orphaned events, or that a runtime determinism bug persists across retry attempts.
|
|
13
13
|
|
|
14
|
-
This is a **workflow-level fatal error**. It cannot be caught or handled inside your workflow code.
|
|
14
|
+
This is a **workflow-level fatal error**. It cannot be caught or handled inside your workflow code. The runtime first retries transient replay divergence automatically; it marks the run as failed with this error only after replay still cannot recover.
|
|
15
15
|
|
|
16
16
|
## Error Message
|
|
17
17
|
|
|
18
18
|
```
|
|
19
|
-
|
|
19
|
+
Workflow replay diverged <divergenceCount> times after <maxRecoveryReplays> recovery replays; latest divergent event was <eventId>. Last divergence: <details>
|
|
20
20
|
```
|
|
21
21
|
|
|
22
22
|
## Why This Happens
|
|
23
23
|
|
|
24
24
|
Workflows persist their progress as an ordered event log. During replay, the runtime processes each event in sequence — every event must be consumed by a matching callback (e.g., a step or sleep waiting for its result). When an event has no matching consumer, the runtime cannot advance past it, which would block all subsequent events and hang the workflow indefinitely.
|
|
25
25
|
|
|
26
|
-
Instead of silently hanging, the runtime
|
|
26
|
+
Instead of silently hanging, the runtime retries a divergent replay before failing the workflow and surfacing this terminal error.
|
|
27
27
|
|
|
28
28
|
Common scenarios that produce this error:
|
|
29
29
|
|
|
@@ -45,7 +45,7 @@ npm install workflow@latest
|
|
|
45
45
|
|
|
46
46
|
### 2. Retry the failed run
|
|
47
47
|
|
|
48
|
-
|
|
48
|
+
If this error is displayed, automatic replay recovery has already been exhausted and the run has been marked as `failed`. You can re-run it using the **Re-run** button in the Workflow Dashboard.
|
|
49
49
|
|
|
50
50
|
### 3. Report the issue
|
|
51
51
|
|
package/docs/errors/index.mdx
CHANGED
|
@@ -37,6 +37,9 @@ Fix common mistakes when creating and executing workflows in the **Workflow SDK*
|
|
|
37
37
|
<Card href="/docs/errors/corrupted-event-log" title="corrupted-event-log">
|
|
38
38
|
Learn how to handle corrupted or invalid event logs.
|
|
39
39
|
</Card>
|
|
40
|
+
<Card href="/docs/errors/replay-divergence" title="replay-divergence">
|
|
41
|
+
Learn how workflow replay divergence is recovered automatically.
|
|
42
|
+
</Card>
|
|
40
43
|
<Card href="/docs/errors/step-not-registered" title="step-not-registered">
|
|
41
44
|
Resolve step not registered errors caused by deployment mismatches.
|
|
42
45
|
</Card>
|
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: replay-divergence
|
|
3
|
+
description: A workflow replay temporarily followed a path that did not match its recorded events.
|
|
4
|
+
type: troubleshooting
|
|
5
|
+
summary: Understand automatic recovery when a workflow replay diverges from its event history.
|
|
6
|
+
prerequisites:
|
|
7
|
+
- /docs/foundations/workflows-and-steps
|
|
8
|
+
related:
|
|
9
|
+
- /docs/errors/corrupted-event-log
|
|
10
|
+
- /docs/foundations/errors-and-retries
|
|
11
|
+
---
|
|
12
|
+
|
|
13
|
+
A replay divergence occurs when one invocation of a workflow cannot consume the durable event history using the promises, hooks, sleeps, or steps it created during replay.
|
|
14
|
+
|
|
15
|
+
This is an SDK/runtime signal, not an error thrown by your workflow code. It is not catchable inside a workflow function.
|
|
16
|
+
|
|
17
|
+
## Automatic Recovery
|
|
18
|
+
|
|
19
|
+
A single divergent replay does not prove that persisted history is corrupted. For example, asynchronous delivery ordering may cause one invocation to follow the wrong side of a race while another replay can follow the recorded history correctly.
|
|
20
|
+
|
|
21
|
+
The runtime automatically queues another replay when an invocation reports `REPLAY_DIVERGENCE`. No terminal `run_failed` event is written during these recovery attempts.
|
|
22
|
+
|
|
23
|
+
If recovery replays continue to diverge after the retry budget is exhausted, the runtime marks the run as failed with `CORRUPTED_EVENT_LOG` and records the latest divergent event for diagnosis.
|
|
24
|
+
|
|
25
|
+
## What To Do
|
|
26
|
+
|
|
27
|
+
Most replay divergence signals recover without action. If a run ultimately fails with `CORRUPTED_EVENT_LOG`, update to the latest `workflow` package and report the run ID and error details if the failure persists.
|