workflow 5.0.0-beta.41 → 5.0.0-beta.43
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/docs/ai/resumable-streams.mdx +4 -0
- package/docs/api-reference/workflow-api/resume-webhook.mdx +1 -1
- package/docs/api-reference/workflow-errors/precondition-failed-error.mdx +2 -2
- package/docs/api-reference/workflow-runtime/world/analytics.mdx +4 -0
- package/docs/api-reference/workflow-runtime/world/storage.mdx +12 -1
- package/docs/changelog/batched-event-writes.mdx +79 -0
- package/docs/changelog/meta.json +2 -1
- package/docs/configuration/runtime-tuning.mdx +35 -11
- package/docs/configuration/worlds.mdx +8 -0
- package/docs/errors/corrupted-event-log.mdx +4 -5
- package/docs/errors/replay-divergence.mdx +1 -1
- package/docs/foundations/hooks.mdx +2 -2
- package/docs/foundations/streaming.mdx +24 -0
- package/docs/how-it-works/event-sourcing.mdx +39 -1
- package/docs/testing/index.mdx +2 -2
- package/package.json +11 -11
|
@@ -22,6 +22,10 @@ Where a standard chat implementation would require the user to resend their mess
|
|
|
22
22
|
|
|
23
23
|
Resumable streams come out of the box with Workflow SDK, however, the client needs to recognize that a stream exists, and needs to know which stream to reconnect to, and needs to know where to start from. For this, Workflow SDK provides the [`WorkflowChatTransport`](/docs/api-reference/workflow-ai/workflow-chat-transport) helper, a drop-in transport for the AI SDK that handles client-side resumption logic for you.
|
|
24
24
|
|
|
25
|
+
<Callout type="info">
|
|
26
|
+
When deploying a streaming route to Vercel, enable request cancellation so a browser disconnect terminates that route's abandoned stream reader instead of letting the function run until `FUNCTION_INVOCATION_TIMEOUT`. See [Avoiding Function Timeouts After Client Disconnects](/docs/foundations/streaming#avoiding-function-timeouts-after-client-disconnects).
|
|
27
|
+
</Callout>
|
|
28
|
+
|
|
25
29
|
## Implementing stream resumption
|
|
26
30
|
|
|
27
31
|
Let's add stream resumption to our Flight Booking Agent that we build in the [Building Durable AI Agents](/docs/ai) guide.
|
|
@@ -61,7 +61,7 @@ Throws [`HookNotFoundError`](/docs/api-reference/workflow-errors/hook-not-found-
|
|
|
61
61
|
## Usage Note
|
|
62
62
|
|
|
63
63
|
<Callout type="warn">
|
|
64
|
-
In most cases, you should not need to call `resumeWebhook()` directly. When you use `createWebhook()`, the framework automatically generates a
|
|
64
|
+
In most cases, you should not need to call `resumeWebhook()` directly. When you use `createWebhook()`, the framework automatically generates a webhook token and provides a public URL at `/.well-known/workflow/v1/webhook/:token`. External systems can send HTTP requests directly to that URL.
|
|
65
65
|
|
|
66
66
|
For server-side hook resumption with deterministic tokens, use [`resumeHook()`](/docs/api-reference/workflow-api/resume-hook) with [`createHook()`](/docs/api-reference/workflow/create-hook) instead.
|
|
67
67
|
</Callout>
|
|
@@ -8,9 +8,9 @@ related:
|
|
|
8
8
|
- /docs/api-reference/workflow-errors/entity-conflict-error
|
|
9
9
|
---
|
|
10
10
|
|
|
11
|
-
`PreconditionFailedError` is thrown by world implementations when an event creation is rejected because the client's event-log snapshot is stale
|
|
11
|
+
`PreconditionFailedError` is thrown by world implementations when an event creation is rejected because the client's event-log snapshot is stale: the log already held more events than the position the creation named. It corresponds to HTTP 412 Precondition Failed semantics.
|
|
12
12
|
|
|
13
|
-
|
|
13
|
+
No world in this repository throws it. A stale replay does not need to be refused: its log is a prefix rather than a prefix with a hole in it, replay is deterministic on a prefix, and the write it makes next comes back carrying the events it was pushed past (see [Stale reads](/docs/configuration/runtime-tuning#stale-reads-and-why-nothing-has-to-be-rejected)). The error and the runtime's handling of it remain for a world that would rather refuse than report — one that allocates positions somewhere other than the commit, and so cannot report a gap reliably. Event creations that carry no position are never rejected with it.
|
|
14
14
|
|
|
15
15
|
A world rejects only on evidence and accepts the creation whenever it cannot decide, so this error always means the snapshot really was stale — but not receiving it does not prove the snapshot was current.
|
|
16
16
|
|
|
@@ -22,6 +22,10 @@ keywords:
|
|
|
22
22
|
|
|
23
23
|
`world.analytics` is an optional, read-only namespace for observability surfaces — dashboards, CLIs, and admin tools that list large numbers of runs without touching payload data.
|
|
24
24
|
|
|
25
|
+
For observability and inspection listings, prefer this namespace over
|
|
26
|
+
[`world.runs.list()`](/docs/api-reference/workflow-runtime/world/storage#runslist).
|
|
27
|
+
The storage API remains available for operational and payload-bearing reads.
|
|
28
|
+
|
|
25
29
|
It differs from [Storage](/docs/api-reference/workflow-runtime/world/storage) in two ways:
|
|
26
30
|
|
|
27
31
|
- **Metadata only.** Results never include run input/output, step data, or hook tokens. There is no `resolveData` option.
|
|
@@ -133,7 +133,10 @@ const result = await world.events.listByCorrelationId({ // [!code highlight]
|
|
|
133
133
|
|
|
134
134
|
## world.runs
|
|
135
135
|
|
|
136
|
-
Materialized from run events. Use it
|
|
136
|
+
Materialized from run events. Use it for canonical operational reads, including
|
|
137
|
+
reads that require workflow input or output data. For observability dashboards,
|
|
138
|
+
inspection tools, and historical listings, use
|
|
139
|
+
[`world.analytics.runs.list()`](/docs/api-reference/workflow-runtime/world/analytics#runslist).
|
|
137
140
|
|
|
138
141
|
### runs.get()
|
|
139
142
|
|
|
@@ -163,6 +166,14 @@ const result = await world.runs.list({ // [!code highlight]
|
|
|
163
166
|
|
|
164
167
|
**Returns:** `{ data: WorkflowRun[], cursor?: string }`
|
|
165
168
|
|
|
169
|
+
<Callout type="warn">
|
|
170
|
+
Observability and inspection usage of `world.runs.list()` is deprecated. Use
|
|
171
|
+
[`world.analytics.runs.list()`](/docs/api-reference/workflow-runtime/world/analytics#runslist)
|
|
172
|
+
for metadata-only, plan-aware queries backed by the observability pipeline.
|
|
173
|
+
`world.runs.list()` remains supported for operational and payload-bearing
|
|
174
|
+
reads.
|
|
175
|
+
</Callout>
|
|
176
|
+
|
|
166
177
|
### Cancelling Runs
|
|
167
178
|
|
|
168
179
|
To cancel a run, create a `run_cancelled` event via `world.events.create()` (see [world.events](#worldevents) above), or use the CLI or Web UI helpers.
|
|
@@ -0,0 +1,79 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: Batched event writes
|
|
3
|
+
description: An optional World API (events.createBatch) that appends an ordered set of events in one durable write with per-event outcomes, and a suspension fan-out fold that uses it.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Batched event writes (`events.createBatch`)
|
|
7
|
+
|
|
8
|
+
## Motivation
|
|
9
|
+
|
|
10
|
+
A workflow suspension that schedules several steps and waits previously wrote one event per entity — one `world.events.create` call per `step_created` and `wait_created`. Against a remote World each write is its own network round trip and its own crash boundary. Batching folds a suspension's schedule into **one durable write** with per-event outcomes, cutting request count and making the whole fan-out land atomically per attempt.
|
|
11
|
+
|
|
12
|
+
## The World spec addition
|
|
13
|
+
|
|
14
|
+
`Storage['events']` gains one **optional** method:
|
|
15
|
+
|
|
16
|
+
```ts
|
|
17
|
+
import type {
|
|
18
|
+
BatchEventRequest,
|
|
19
|
+
CreateEventBatchParams,
|
|
20
|
+
EventBatchResult,
|
|
21
|
+
} from '@workflow/world';
|
|
22
|
+
|
|
23
|
+
interface BatchCapableEvents {
|
|
24
|
+
createBatch?(
|
|
25
|
+
runId: string,
|
|
26
|
+
events: BatchEventRequest[],
|
|
27
|
+
params?: CreateEventBatchParams
|
|
28
|
+
): Promise<EventBatchResult>;
|
|
29
|
+
}
|
|
30
|
+
```
|
|
31
|
+
|
|
32
|
+
The supporting types, excerpted (canonical definitions live in `@workflow/world`):
|
|
33
|
+
|
|
34
|
+
{/* @skip-typecheck illustrative excerpts of the canonical @workflow/world types */}
|
|
35
|
+
```ts
|
|
36
|
+
interface BatchEventRequest {
|
|
37
|
+
/** The event — the same discriminated union the single `create` takes. */
|
|
38
|
+
event: CreateEventRequest;
|
|
39
|
+
/** Client event time; under slot identity, the source of the durable createdAt. */
|
|
40
|
+
occurredAt?: Date;
|
|
41
|
+
/** Per-event compute attribution, same as the single create's CreateEventParams. */
|
|
42
|
+
computeInstanceId?: string;
|
|
43
|
+
}
|
|
44
|
+
|
|
45
|
+
type BatchEventItemResult =
|
|
46
|
+
| { status: 200; event: Event; run?: WorkflowRun; step?: Step; wait?: Wait }
|
|
47
|
+
| { status: number; error: string; message: string };
|
|
48
|
+
|
|
49
|
+
interface EventBatchResult {
|
|
50
|
+
/** One entry per submitted event, in request order. */
|
|
51
|
+
results: BatchEventItemResult[];
|
|
52
|
+
}
|
|
53
|
+
```
|
|
54
|
+
|
|
55
|
+
The contract:
|
|
56
|
+
|
|
57
|
+
- **Ordered**: events land in the run's log in request order at consecutive slots. A concurrent writer may push the whole batch to slots above the caller's view; no skipped-event report accompanies the batch result, so a position-tracking caller compares committed slots against its expectation and reloads to observe what interleaved (its local view stays a strict prefix of the log — never a hole).
|
|
58
|
+
- **Per-event outcomes**: the batch is processed as a whole, and each event reports what its own single `create` would have returned — `200` plus the materialized entity, or the single-path status/code (`409`/`conflict` for an event an earlier delivery already applied). Callers reuse their single-path conflict handling per event.
|
|
59
|
+
- **Idempotent on retry — for entity-conditioned shapes**: creates, terminal transitions, and the born-running pair are each guarded by their own entity condition, so retrying a batch of them that (partially) committed converges to per-event `409`s with nothing written twice. A standalone bare `step_started` or a `step_retrying` re-patches its step instead of converging, so `world-vercel` only auto-retries batches whose every event is retry-convergent (everything the runtime folds today is), and rejects `hook_received` in a batch outright.
|
|
60
|
+
- **Method presence is the capability declaration.** A World that doesn't implement it keeps the single-event path; a World that implements it must make each attempt atomic (a lost race leaves nothing behind). `world-vercel` implements it against `POST /v4/runs/:runId/events/batch` (slot-identity runs only, i.e. specVersion ≥ 6). `world-local` and `world-postgres` deliberately do not — batching buys nothing for a local write.
|
|
61
|
+
- **Not batchable** (Worlds reject the request): `run_created`, `run_started`, `run_cancelled`, `hook_created`, `hook_disposed`, `attr_set`, and multiple events targeting one entity — except `step_created` followed by `step_started` for the same step, which creates the step born-running.
|
|
62
|
+
|
|
63
|
+
## The runtime integration (suspension fan-out fold)
|
|
64
|
+
|
|
65
|
+
**On by default.** The suspension handler folds a **clean fan-out** — the suspension's eager `step_created` and `wait_created` writes — into `createBatch` calls of at most 32 events (mirroring the server's transaction budgets). Chunks of a larger fan-out commit **concurrently**: slot assignment is the World's, so parallel chunks race for slot ranges exactly like the pre-fold path's parallel single writes did, and per-entity conditions — not commit order — carry correctness. The fold only engages when the World implements `createBatch`, the run is on slot identity, and the suspension carries no attribute writes, no hook writes, and no resilient step dispatch; everything else keeps the single-event path byte-for-byte.
|
|
66
|
+
|
|
67
|
+
**Per-chunk continuation.** Each chunk's follow-on work starts the moment **that chunk** commits, not when the whole fold does: a chunk's step-execution queue messages publish right off its own commit (publish-after-create holds per step), and only the chunk carrying the inline pairs gates the replay's continuation — trailing chunks' commits and publishes are joined before the invocation can acknowledge its message, so the durability contract ("every create durable before ack") is unchanged.
|
|
68
|
+
|
|
69
|
+
**Pre-claimed inline pairs.** When the fold engages and has company for them (at least two inline steps, or one plus other batchable events), the steps the runtime is about to execute inline join the batch as adjacent `[step_created, step_started]` pairs — the created row carrying the input, the started row a bare ownership-stamped claim the World folds into a born-running create. The inline bodies start straight off the pair chunk's commit (in parallel with the queue publishes and any trailing chunks) with no per-step claim POST at all, and a pair that loses its atomic create-claim to a concurrent delivery skips its body exactly as a lost lazy claim does. A lone inline step with nothing else to batch keeps the optimistic lazy-start path, whose claim overlaps the body.
|
|
70
|
+
|
|
71
|
+
Per-event `409`s are tolerated the same way the single path tolerates `EntityConflictError` (a concurrent delivery already created the entity); any other per-event failure fails the suspension write the way a single-path rejection would. A batch carrying a `step_started` (that is, any batch with inline pairs) is **not** retried in-process on a transport blip: a pair's `409` cannot be told apart from the caller's own earlier attempt having committed it, so recovery goes through queue redelivery instead, where the step's ownership stamp routes it back to the same invocation.
|
|
72
|
+
|
|
73
|
+
`createBatch` is optional, so today only the Vercel World folds at all: every other World keeps the single-event path and never sends a pair.
|
|
74
|
+
|
|
75
|
+
**Escape hatch:** set `WORKFLOW_BATCH_TRANSITIONS=0` (or `false`) to disable batching and restore the exact prior one-write-per-event path — see [`WORKFLOW_BATCH_TRANSITIONS`](/docs/configuration/worlds#workflow_batch_transitions).
|
|
76
|
+
|
|
77
|
+
## Follow-up
|
|
78
|
+
|
|
79
|
+
The deferred sequential transition — holding `step_completed(N)` across the replay turn and committing `[step_completed(N), step_created(N+1), step_started(N+1)]` as one batch at the next lazy start — builds on this contract and ships separately.
|
package/docs/changelog/meta.json
CHANGED
|
@@ -71,17 +71,23 @@ For example, a workflow can run a 10-minute inline step even with `WORKFLOW_REPL
|
|
|
71
71
|
- Only applies to Worlds with atomic, immutable deployments (the Vercel World). A run whose pinned deployment cannot be reached at all fails immediately regardless of this value.
|
|
72
72
|
- Transient or unknown queue publishing failures use normal queue redelivery and do not consume this budget.
|
|
73
73
|
|
|
74
|
-
### `
|
|
74
|
+
### `WORKFLOW_RESILIENT_STEP_DISPATCH`
|
|
75
75
|
|
|
76
|
-
- Default:
|
|
77
|
-
-
|
|
78
|
-
-
|
|
79
|
-
-
|
|
80
|
-
-
|
|
81
|
-
-
|
|
82
|
-
|
|
83
|
-
|
|
84
|
-
|
|
76
|
+
- Default: disabled
|
|
77
|
+
- When a suspension hands newly created steps to the queue, the runtime publishes each step's execution message in parallel with its `step_created` event write instead of sequencing them, cutting a round trip per dispatched step. The message also carries the serialized step input (`stepInput`), so a transient `step_created` write failure (429 / 5xx / transport) still executes the step — the queue consumer idempotently re-ensures the event before running it, converging with the producer's write on the step's correlation ID. This mirrors resilient start (`runInput`) and the lazy hook resume (`hookInput`).
|
|
78
|
+
- It is off by default because the publish races the create's verdict, and a create can come back refused: as a duplicate this replay should stop pursuing, or as a [stale write](#stale-reads-and-why-nothing-has-to-be-rejected) on a World that refuses rather than reports. Either way the message carrying the payload is already out, so the consumer can materialize a step whose create was refused, and nothing orders the verdict before the consumer's redelivery re-ensure. The sequential path is the only one that gives the message a happens-after edge over it.
|
|
79
|
+
- Even when enabled, the runtime falls back to the sequential create-then-publish dispatch when the step input is too large to inline on the queue message, or when the run's queue transport cannot carry binary payloads (pre-CBOR spec versions).
|
|
80
|
+
- Producer-side recoveries are reported on the suspension span as `workflow.step.resilient_dispatch_recovered`; a consumer that materialized the event reports `workflow.step.resilient_dispatch_materialized`.
|
|
81
|
+
- Set `1` to enable it.
|
|
82
|
+
|
|
83
|
+
### Stale reads, and why nothing has to be rejected
|
|
84
|
+
|
|
85
|
+
- Not a variable: this is how a replay working from an out-of-date event log stays correct, and why no World needs a precondition guard to make it so.
|
|
86
|
+
- Three properties do it together. A reader's log is always a **prefix** of the run's log, never a prefix with a hole in it — positions are allocated by the World at commit, so nothing lands behind a position a reader has already passed. Replay is **deterministic on a prefix**: the same prefix always yields the same decisions, so a shorter log does not mean a different run, only a run that has not caught up. And every write **reports what it missed**: a creation names the position it replayed from (`eventCount`), and the World returns the events occupying the positions it was pushed past. The replay merges those and continues, correcting itself on the write rather than on a read.
|
|
87
|
+
- So a stale replay costs a merge, not a rejection. None of the shipped Worlds refuses a write for being stale.
|
|
88
|
+
- A World *may* refuse instead, with 412 ([`PreconditionFailedError`](/docs/api-reference/workflow-errors/precondition-failed-error)) — appropriate when it allocates positions somewhere other than the commit and cannot report a gap reliably. The runtime handles that: it restarts the replay in the same invocation from a corrected event log, and falls back to a re-invocation with a fresh replay once the restart budget is spent. The rejected write is never retried as-is, because a replay working from a corrected log derives different events.
|
|
89
|
+
- A World that does refuse should only ever do so on evidence, and accept the write in every other case. A rejection then always means the position really was stale, while the absence of one proves nothing about currency.
|
|
90
|
+
- Two runtime behaviors follow from the properties above rather than from any fence. The per-step event-log delta optimization (consuming the delta returned by a step's terminal write instead of issuing an extra `events.list` per step) stays active while the run has an open hook: a `hook_received` missed by the delta window is observed one iteration later, and the next write brings it back. And while a hook is open, inline steps take the await-then-run path even when optimistic inline start is enabled — several invocations race for one step's claim there, and awaiting it means the body runs only for the writer that won.
|
|
85
91
|
|
|
86
92
|
### `WORKFLOW_SLOT_GAP_CHECK`
|
|
87
93
|
|
|
@@ -94,7 +100,7 @@ For example, a workflow can run a 10-minute inline step even with `WORKFLOW_REPL
|
|
|
94
100
|
### `WORKFLOW_PRECONDITION_MAX_INPROCESS_RESTARTS`
|
|
95
101
|
|
|
96
102
|
- Default: `3`
|
|
97
|
-
- How many times a single invocation restarts its replay in-process after
|
|
103
|
+
- How many times a single invocation restarts its replay in-process after an event creation is [rejected as stale](#stale-reads-and-why-nothing-has-to-be-rejected) before it falls back to a re-invocation. No shipped World rejects one, so this budget is reserved for a World that chooses to.
|
|
98
104
|
- A restart reloads the event log and rebuilds the workflow from scratch, so it costs a replay but no queue round trip. A World may attach the missing events to its rejection, in which case the first restart needs no event-log request at all.
|
|
99
105
|
|
|
100
106
|
### `WORKFLOW_PRECONDITION_MAX_REINVOCATIONS`
|
|
@@ -208,6 +214,24 @@ For example, a workflow can run a 10-minute inline step even with `WORKFLOW_REPL
|
|
|
208
214
|
- Default: enabled
|
|
209
215
|
- On the Vercel World, lets concurrent event-log requests share one HTTP/2 connection instead of one connection per in-flight request.
|
|
210
216
|
- Set `0` to take the event-log requests off HTTP/2 entirely, back to one request per HTTP/1.1 connection. Use this as the kill switch if HTTP/2 turns out to be at fault for event delivery problems.
|
|
217
|
+
- Has no effect when `WORKFLOW_NODE_HTTP` is enabled, which takes the whole HTTP/2 path away.
|
|
218
|
+
|
|
219
|
+
### `WORKFLOW_NODE_HTTP`
|
|
220
|
+
|
|
221
|
+
- Default: disabled
|
|
222
|
+
- Makes the Vercel and Local Worlds issue their HTTP requests through Node's built-in `node:http` and `node:https` modules, instead of the HTTP client library those Worlds normally use.
|
|
223
|
+
- Set `1` to switch to Node's modules. Read the trade-offs below first: they cost throughput on every deployment, which is why this is opt-in.
|
|
224
|
+
- Use it when that library is not an option: a bundler that mangles it, a runtime that does not ship a working copy of it, or a transport-level fault you want to rule out. It is not reached by way of `fetch()` either, so a runtime whose `fetch()` is built on the same library is still covered.
|
|
225
|
+
|
|
226
|
+
Node's own modules do less than the client they replace, so enabling this drops the per-call-site tuning the Worlds configure:
|
|
227
|
+
|
|
228
|
+
- Event-log requests lose HTTP/2, so concurrent reads and writes no longer share one connection, and the enlarged HTTP/2 receive windows no longer apply. This is the largest difference, and it slows down replays that read a big event log. It does not apply to event writes on [`WORKFLOW_EVENTS_TRANSPORT=ws`](/docs/configuration/worlds#workflow_events_transport), which take neither transport.
|
|
229
|
+
- Requests lose their transport-level retry. Failures still surface to the layers above, which retry event writes and redeliver queue messages, so nothing is silently dropped, but a failure that a same-connection retry would have hidden now costs a full redelivery.
|
|
230
|
+
- Stream close loses its retry of retriable server errors. A transient failure at close can leave a stream marked closing until the run expires, where it would previously have resolved on the retry.
|
|
231
|
+
|
|
232
|
+
Connection pooling, keep-alive, and the request, header, and body deadlines are preserved: pooling and keep-alive are configured on Node's agents, and the deadlines are passed per request — from the Local World's two queue timeouts, and on the Vercel World from the same defaults its HTTP client applies today. Queue sends are the exception in the other direction: that client takes no transport override, so its requests keep using the library either way.
|
|
233
|
+
|
|
234
|
+
A `dispatcher` passed to `createVercelWorld()` still wins over this variable. The variable chooses which transport the World builds when you have not supplied one.
|
|
211
235
|
|
|
212
236
|
## Queue namespace
|
|
213
237
|
|
|
@@ -274,6 +274,14 @@ Platform-provided values such as `VERCEL_DEPLOYMENT_ID`, `VERCEL_PROJECT_ID`, an
|
|
|
274
274
|
- Default: `1000`
|
|
275
275
|
- Maximum stream chunks written in one Vercel World request. Larger batches are split.
|
|
276
276
|
|
|
277
|
+
### `WORKFLOW_BATCH_TRANSITIONS`
|
|
278
|
+
|
|
279
|
+
- Surface: environment variable
|
|
280
|
+
- Default: on
|
|
281
|
+
- Set to `0` (or `false`) to **disable** batched event writes — the escape hatch that restores the exact prior one-write-per-event path.
|
|
282
|
+
|
|
283
|
+
When enabled (the default), a suspension's eager `step_created` and `wait_created` writes fold into batched `events.createBatch` calls (one durable write with per-event outcomes) on Worlds that implement the optional batch API. The fold only engages when the World implements `events.createBatch` (the Vercel World does; Local and Postgres do not), the run's spec version supports slot identity (≥ 6), and the suspension carries no attribute writes, hook writes, or resilient step dispatch — everything else keeps the single-event path unchanged, so disabling is only needed as an operational escape hatch. Batches are capped at 32 events; larger fan-outs commit in successive batches. See the [batched event writes changelog](/docs/changelog/batched-event-writes) for the World API contract.
|
|
284
|
+
|
|
277
285
|
### `WORKFLOW_EVENTS_TRANSPORT`
|
|
278
286
|
|
|
279
287
|
- Factory option: none
|
|
@@ -2,14 +2,14 @@
|
|
|
2
2
|
title: corrupted-event-log
|
|
3
3
|
description: The workflow's event log contains an event that no consumer can process, indicating corruption or invalid state.
|
|
4
4
|
type: troubleshooting
|
|
5
|
-
summary: Resolve corrupted event log errors caused by
|
|
5
|
+
summary: Resolve corrupted event log errors caused by orphaned or unattributable events.
|
|
6
6
|
prerequisites:
|
|
7
7
|
- /docs/foundations/workflows-and-steps
|
|
8
8
|
related:
|
|
9
9
|
- /docs/foundations/errors-and-retries
|
|
10
10
|
---
|
|
11
11
|
|
|
12
|
-
This error occurs when the Workflow runtime repeatedly cannot replay events in the event log. This usually means the event log is in an invalid state, such as
|
|
12
|
+
This error occurs when the Workflow runtime repeatedly cannot replay events in the event log. This usually means the event log is in an invalid state, such as an orphaned event or one no consumer can attribute to anything the workflow did, or that a runtime determinism bug persists across retry attempts.
|
|
13
13
|
|
|
14
14
|
This is a **workflow-level fatal error**. It cannot be caught or handled inside your workflow code. The runtime first retries transient replay divergence automatically; it marks the run as failed with this error only after replay still cannot recover.
|
|
15
15
|
|
|
@@ -29,10 +29,9 @@ Before failing, the runtime retries a divergent replay and surfaces this termina
|
|
|
29
29
|
|
|
30
30
|
Common scenarios that produce this error:
|
|
31
31
|
|
|
32
|
-
1. **
|
|
32
|
+
1. **An unclaimed event that repeats nothing** — A duplicate of a kind the log already records for that entity is read past rather than failing the run, so a second `step_completed` or `wait_completed` is not this error (see [Duplicate Events](/docs/how-it-works/event-sourcing#duplicate-events)). What fails is an unclaimed event with no earlier counterpart to defer to: a `step_started` behind a `step_completed` on a log that never recorded a `step_started`, for instance. No consumer remains for the step, and there is no earlier event of that kind the replay could be reading instead.
|
|
33
33
|
2. **Orphaned events** — A `step_completed` or `wait_completed` event whose `correlationId` doesn't match any step or sleep in the workflow code, so the replay reaches its end still holding it.
|
|
34
|
-
3. **
|
|
35
|
-
4. **A hole in the log** — Events are numbered by their position in the run's log, and those positions are dense, so a position below the log's highest that holds no event means the log the replay loaded is incomplete. The runtime cannot tell a position no write ever occupied from one whose event it failed to read, so it refuses to replay rather than produce a result that may be silently wrong. See [`WORKFLOW_SLOT_GAP_CHECK`](/docs/configuration/runtime-tuning#workflow_slot_gap_check).
|
|
34
|
+
3. **A hole in the log** — Events are numbered by their position in the run's log, and those positions are dense, so a position below the log's highest that holds no event means the log the replay loaded is incomplete. The runtime cannot tell a position no write ever occupied from one whose event it failed to read, so it refuses to replay rather than produce a result that may be silently wrong. See [`WORKFLOW_SLOT_GAP_CHECK`](/docs/configuration/runtime-tuning#workflow_slot_gap_check).
|
|
36
35
|
|
|
37
36
|
## What To Do
|
|
38
37
|
|
|
@@ -20,7 +20,7 @@ A single divergent replay does not prove that persisted history is corrupted. Fo
|
|
|
20
20
|
|
|
21
21
|
The runtime automatically queues another replay when an invocation reports `REPLAY_DIVERGENCE`. No terminal `run_failed` event is written during these recovery attempts.
|
|
22
22
|
|
|
23
|
-
If recovery replays continue to diverge after the
|
|
23
|
+
If recovery replays continue to diverge after the recovery budget is exhausted, the runtime marks the run as failed with `CORRUPTED_EVENT_LOG` and records the latest divergent event for diagnosis.
|
|
24
24
|
|
|
25
25
|
## What To Do
|
|
26
26
|
|
|
@@ -116,7 +116,7 @@ Calling `createHook()` on its own does not register the hook — registration is
|
|
|
116
116
|
|
|
117
117
|
### Custom Tokens for Deterministic Hooks
|
|
118
118
|
|
|
119
|
-
By default, hooks generate
|
|
119
|
+
By default, hooks generate their own token. However, you often want to use a **custom token** that external systems can reconstruct. This is especially useful for long-running workflows where the same workflow instance should handle multiple events.
|
|
120
120
|
|
|
121
121
|
For example, imagine a Slack bot where each channel should have its own workflow instance:
|
|
122
122
|
|
|
@@ -480,7 +480,7 @@ This pattern is especially valuable in larger applications where the workflow an
|
|
|
480
480
|
|
|
481
481
|
### Token Design
|
|
482
482
|
|
|
483
|
-
Custom tokens are available for `createHook()` with server-side `resumeHook()` only. Webhooks (`createWebhook()`) always
|
|
483
|
+
Custom tokens are available for `createHook()` with server-side `resumeHook()` only. Webhooks (`createWebhook()`) always generate their own unique tokens. A generated token is not trivial to guess, but it is not a strong security contract either, so anyone who obtains the URL can invoke an unintended webhook resumption. To prevent unauthenticated run resumptions entirely, prefer a **hook** over the **webhook** convenience and implement your own authentication on the route that calls `resumeHook()`.
|
|
484
484
|
|
|
485
485
|
When using custom tokens with `createHook()`:
|
|
486
486
|
|
|
@@ -58,6 +58,30 @@ export async function POST() {
|
|
|
58
58
|
|
|
59
59
|
When a client makes a request to this endpoint, they'll receive each message as it's written, without waiting for the workflow to complete.
|
|
60
60
|
|
|
61
|
+
### Avoiding Function Timeouts After Client Disconnects
|
|
62
|
+
|
|
63
|
+
On Vercel, `run.readable` and `run.getReadable()` reconnect to Workflow's stream storage while the workflow is still running. By default, a client disconnect does not terminate the Vercel Function serving the stream. If a user closes the page or stops the request, the function can therefore keep reconnecting until it reaches its maximum duration and fails with `FUNCTION_INVOCATION_TIMEOUT`.
|
|
64
|
+
|
|
65
|
+
For streaming routes using the Node.js runtime, opt in to [request cancellation](https://vercel.com/docs/functions/functions-api-reference#enable-cancellation) in `vercel.json`:
|
|
66
|
+
|
|
67
|
+
```json filename="vercel.json"
|
|
68
|
+
{
|
|
69
|
+
"functions": {
|
|
70
|
+
"app/api/stream/route.ts": {
|
|
71
|
+
"supportsCancellation": true
|
|
72
|
+
}
|
|
73
|
+
}
|
|
74
|
+
}
|
|
75
|
+
```
|
|
76
|
+
|
|
77
|
+
Replace the function path with the path or glob for your streaming route. When the downstream client disconnects, Vercel terminates the matching function invocation instead of leaving its stream reader running. The workflow run and its durable stream continue independently, so the client can reconnect through another route invocation later.
|
|
78
|
+
|
|
79
|
+
<Callout type="warn">
|
|
80
|
+
Cancellation applies to every function matching the configured path or glob, even if the route does not listen to `request.signal`. Any other work in that invocation which is not wrapped in [`waitUntil`](https://vercel.com/docs/functions/functions-api-reference/vercel-functions-package#waituntil) or [`after`](https://nextjs.org/docs/app/api-reference/functions/after) can be lost. Only enable it for routes that are safe to terminate when their client disconnects.
|
|
81
|
+
</Callout>
|
|
82
|
+
|
|
83
|
+
This setting prevents abandoned stream readers from consuming the rest of a function invocation. It does not extend the function's maximum duration: an actively connected streaming response can still reach the configured limit, at which point the client should reconnect to the durable stream.
|
|
84
|
+
|
|
61
85
|
### Resuming Streams from a Specific Point
|
|
62
86
|
|
|
63
87
|
Use `run.getReadable({ startIndex })` to resume a stream from a specific position. This is useful for reconnecting after timeouts or network interruptions:
|
|
@@ -94,7 +94,7 @@ flowchart TD
|
|
|
94
94
|
- `cancelled`: Reserved for future use (not currently emitted)
|
|
95
95
|
|
|
96
96
|
<Callout type="info">
|
|
97
|
-
The `step_retrying` event is optional. Steps can retry without it - the retry mechanism works regardless of whether this event is emitted. You may see back-to-back `step_started` events in logs when a step retries after a timeout or when the error is not explicitly captured. See [Errors and Retries](/docs/foundations/errors-and-retries) for more on how retries work.
|
|
97
|
+
The `step_retrying` event is optional. Steps can retry without it - the retry mechanism works regardless of whether this event is emitted. You may see back-to-back `step_started` events in logs when a step retries after a timeout or when the error is not explicitly captured, and also when concurrent replays each commit one (see [Duplicate Events](#duplicate-events)). See [Errors and Retries](/docs/foundations/errors-and-retries) for more on how retries work.
|
|
98
98
|
</Callout>
|
|
99
99
|
|
|
100
100
|
When present, the `step_retrying` event moves a step back to `pending` state and records the error that caused the retry. This provides two benefits:
|
|
@@ -225,6 +225,44 @@ Terminal states represent the end of an entity's lifecycle. Once an entity reach
|
|
|
225
225
|
|
|
226
226
|
Attempting to create an event that would transition an entity out of a terminal state will result in an error. This prevents inconsistent state and ensures the integrity of the event log.
|
|
227
227
|
|
|
228
|
+
That guard sits on the write path. A duplicate the write path does permit — a second `step_created` for a step that is not yet terminal, for example — is handled during replay instead, described next.
|
|
229
|
+
|
|
230
|
+
## Duplicate Events
|
|
231
|
+
|
|
232
|
+
Concurrent invocations replaying the same run share one event log. An invocation working from a stale prefix — one that predates another invocation's write — can commit its own `step_created`, `step_started`, or `wait_created` for an entity the log already records one of. These writes pass the terminal-state guard above, so the write path commits them even when a backend validates transitions atomically with the insert.
|
|
233
|
+
|
|
234
|
+
Those duplicates are committed but inert. The outcome was decided by the first event of its kind at a lower position in the log, and every replay reads that same event at that same position, so a later copy cannot change what the workflow observes.
|
|
235
|
+
|
|
236
|
+
To keep an inert copy from failing an otherwise healthy run, the runtime groups event types into **classes** and tracks, per entity, which classes the current replay has already consumed. When an event is offered to every registered consumer and none wants it, and its class is already recorded for that entity, the replay steps over it instead of reporting a [replay divergence](/docs/errors/replay-divergence) — which, once the recovery budget is exhausted, ends the run with [`CORRUPTED_EVENT_LOG`](/docs/errors/corrupted-event-log).
|
|
237
|
+
|
|
238
|
+
| Class | Event types |
|
|
239
|
+
|-------|-------------|
|
|
240
|
+
| `run_started` | `run_started` |
|
|
241
|
+
| `step_created` | `step_created` |
|
|
242
|
+
| `step_started` | `step_started` |
|
|
243
|
+
| `step_retrying` | `step_retrying` |
|
|
244
|
+
| `step_terminal` | `step_completed`, `step_failed` |
|
|
245
|
+
| `wait_created` | `wait_created` |
|
|
246
|
+
| `wait_completed` | `wait_completed` |
|
|
247
|
+
| `hook_created` | `hook_created` |
|
|
248
|
+
| `hook_disposed` | `hook_disposed` |
|
|
249
|
+
|
|
250
|
+
Types that share a class are the mutually exclusive outcomes of one decision: a step either completes or fails, and the first outcome recorded is the one that counts. Classes are independent of one another, so passing over one does not suppress another. A step whose result is already in the log has still recorded exactly one `step_created`, which is what makes a second one ignorable on its own terms.
|
|
251
|
+
|
|
252
|
+
The two hook classes cover the same shape of duplicate, and replay reaches them less often because the write path resolves most hook duplicates before they reach the log: a run re-creating a hook it already owns converges on the existing `hook_created` rather than appending a second one, and a second `hook_disposed` for the same hook is refused as an idempotent no-op. A log that holds either anyway is read past like any other repeat.
|
|
253
|
+
|
|
254
|
+
The remaining event types belong to no class and are never skipped:
|
|
255
|
+
|
|
256
|
+
- `hook_received`: a hook legitimately receives many payloads under one ID, so a second `hook_received` is not a repeat of anything.
|
|
257
|
+
- `hook_conflict`: records a failed acquisition of a hook's token, which the same run can hit repeatedly over its lifetime as other runs take and release that token. The hook's own consumer stays registered and claims every copy it is offered, so a repeat is consumed rather than reaching the class check.
|
|
258
|
+
- `attr_set`: written on every [`setAttributes()`](/docs/api-reference/workflow/set-attributes) call, so a second write of the same key is a new fact rather than a repeat.
|
|
259
|
+
- `run_created` precedes every replay and is always consumed.
|
|
260
|
+
- `run_completed`, `run_failed`, and `run_cancelled` never reach the check. The runtime exits before replaying the workflow body once the log holds one of them, so no consumer ever takes one and no class is ever recorded for them.
|
|
261
|
+
|
|
262
|
+
Both kinds of skip are logged at `debug`, so neither reaches the console unless you run with `DEBUG=workflow:runtime:*`. A duplicate is a permanent feature of the log: every later replay re-reads it and lands on the same check, so anything printed unconditionally would print once per replay for the life of the run, and there is nothing to act on either way. A repeat that decides a class differently — a `step_failed` behind a `step_completed`, or the reverse — gets its own message, because unlike a re-commit of the same outcome there is no reading in which both writers were right.
|
|
263
|
+
|
|
264
|
+
The observability UI greys out the events it can identify this way, with the reason on hover. Its set is narrower than the runtime's: it reads the log without consumer state, and a consumer for an entity that is still open legitimately claims a repeat — each retry of a step writes another `step_started`. So it marks a repeat only once no consumer can remain for it: past a terminal event for the same entity, or a second `run_started`, of which the log records one per run. On a partial view of the log — one page of a paginated list, or search results — it marks nothing, since which copy came first is a property of the whole log.
|
|
265
|
+
|
|
228
266
|
## Event Correlation
|
|
229
267
|
|
|
230
268
|
Events use a `correlationId` to link related events together. For step, hook, and wait events, the correlation ID identifies the specific entity instance:
|
package/docs/testing/index.mdx
CHANGED
|
@@ -256,7 +256,7 @@ import { createWebhook } from "workflow";
|
|
|
256
256
|
export async function ingestWorkflow(endpointId: string) {
|
|
257
257
|
"use workflow";
|
|
258
258
|
|
|
259
|
-
// Webhook tokens are always
|
|
259
|
+
// Webhook tokens are always generated for you
|
|
260
260
|
using webhook = createWebhook(); // [!code highlight]
|
|
261
261
|
|
|
262
262
|
const request = await webhook; // [!code highlight]
|
|
@@ -282,7 +282,7 @@ describe("ingestWorkflow", () => {
|
|
|
282
282
|
it("should process webhook data", async () => {
|
|
283
283
|
const run = await start(ingestWorkflow, ["ep-1"]);
|
|
284
284
|
|
|
285
|
-
// Discover the
|
|
285
|
+
// Discover the generated webhook token
|
|
286
286
|
const hook = await waitForHook(run); // [!code highlight]
|
|
287
287
|
|
|
288
288
|
// Resume the webhook with a Request object
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "workflow",
|
|
3
|
-
"version": "5.0.0-beta.
|
|
3
|
+
"version": "5.0.0-beta.43",
|
|
4
4
|
"description": "Workflow SDK - Build durable, resilient, and observable workflows",
|
|
5
5
|
"main": "dist/typescript-plugin.cjs",
|
|
6
6
|
"type": "module",
|
|
@@ -59,18 +59,18 @@
|
|
|
59
59
|
},
|
|
60
60
|
"dependencies": {
|
|
61
61
|
"ms": "2.1.3",
|
|
62
|
-
"@workflow/astro": "5.0.0-beta.
|
|
63
|
-
"@workflow/cli": "5.0.0-beta.
|
|
64
|
-
"@workflow/core": "5.0.0-beta.
|
|
65
|
-
"@workflow/errors": "5.0.0-beta.
|
|
62
|
+
"@workflow/astro": "5.0.0-beta.43",
|
|
63
|
+
"@workflow/cli": "5.0.0-beta.43",
|
|
64
|
+
"@workflow/core": "5.0.0-beta.43",
|
|
65
|
+
"@workflow/errors": "5.0.0-beta.17",
|
|
66
66
|
"@workflow/typescript-plugin": "5.0.0-beta.5",
|
|
67
67
|
"@workflow/utils": "5.0.0-beta.8",
|
|
68
|
-
"@workflow/next": "5.0.0-beta.
|
|
69
|
-
"@workflow/nest": "5.0.0-beta.
|
|
70
|
-
"@workflow/nitro": "5.0.0-beta.
|
|
71
|
-
"@workflow/nuxt": "5.0.0-beta.
|
|
72
|
-
"@workflow/sveltekit": "5.0.0-beta.
|
|
73
|
-
"@workflow/rollup": "5.0.0-beta.
|
|
68
|
+
"@workflow/next": "5.0.0-beta.43",
|
|
69
|
+
"@workflow/nest": "5.0.0-beta.43",
|
|
70
|
+
"@workflow/nitro": "5.0.0-beta.43",
|
|
71
|
+
"@workflow/nuxt": "5.0.0-beta.43",
|
|
72
|
+
"@workflow/sveltekit": "5.0.0-beta.43",
|
|
73
|
+
"@workflow/rollup": "5.0.0-beta.43"
|
|
74
74
|
},
|
|
75
75
|
"devDependencies": {
|
|
76
76
|
"@types/ms": "2.1.0",
|