workflow 5.0.0-beta.42 → 5.0.0-beta.44
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +6 -4
- package/dist/internal/builtins.d.ts +4 -4
- package/dist/internal/builtins.js +6 -6
- package/docs/ai/chat-session-modeling.mdx +23 -24
- package/docs/ai/defining-tools.mdx +5 -6
- package/docs/ai/human-in-the-loop.mdx +11 -11
- package/docs/ai/index.mdx +20 -20
- package/docs/ai/message-queueing.mdx +6 -6
- package/docs/ai/meta.json +1 -0
- package/docs/ai/resumable-streams.mdx +32 -28
- package/docs/ai/sleep-and-delays.mdx +9 -9
- package/docs/ai/streaming-updates-from-tools.mdx +4 -4
- package/docs/api-reference/vitest/index.mdx +8 -8
- package/docs/api-reference/workflow/create-hook.mdx +15 -15
- package/docs/api-reference/workflow/create-webhook.mdx +15 -15
- package/docs/api-reference/workflow/define-hook.mdx +10 -10
- package/docs/api-reference/workflow/fatal-error.mdx +2 -2
- package/docs/api-reference/workflow/fetch.mdx +7 -7
- package/docs/api-reference/workflow/get-step-metadata.mdx +2 -2
- package/docs/api-reference/workflow/get-workflow-metadata.mdx +3 -3
- package/docs/api-reference/workflow/get-writable.mdx +7 -7
- package/docs/api-reference/workflow/index.mdx +1 -1
- package/docs/api-reference/workflow/retryable-error.mdx +1 -1
- package/docs/api-reference/workflow/set-attributes.mdx +2 -2
- package/docs/api-reference/workflow/sleep.mdx +3 -3
- package/docs/api-reference/workflow-ai/durable-agent.mdx +42 -42
- package/docs/api-reference/workflow-ai/index.mdx +3 -3
- package/docs/api-reference/workflow-ai/workflow-chat-transport.mdx +28 -28
- package/docs/api-reference/workflow-api/get-hook-by-token.mdx +11 -11
- package/docs/api-reference/workflow-api/get-run.mdx +10 -10
- package/docs/api-reference/workflow-api/index.mdx +2 -4
- package/docs/api-reference/workflow-api/resume-hook.mdx +14 -14
- package/docs/api-reference/workflow-api/resume-webhook.mdx +4 -4
- package/docs/api-reference/workflow-api/start.mdx +16 -15
- package/docs/api-reference/workflow-astro/workflow.mdx +3 -3
- package/docs/api-reference/workflow-errors/entity-conflict-error.mdx +4 -4
- package/docs/api-reference/workflow-errors/hook-conflict-error.mdx +4 -4
- package/docs/api-reference/workflow-errors/hook-not-found-error.mdx +8 -8
- package/docs/api-reference/workflow-errors/index.mdx +6 -6
- package/docs/api-reference/workflow-errors/precondition-failed-error.mdx +9 -9
- package/docs/api-reference/workflow-errors/run-expired-error.mdx +2 -2
- package/docs/api-reference/workflow-errors/run-not-supported-error.mdx +4 -4
- package/docs/api-reference/workflow-errors/step-not-registered-error.mdx +5 -5
- package/docs/api-reference/workflow-errors/throttle-error.mdx +2 -2
- package/docs/api-reference/workflow-errors/too-early-error.mdx +2 -2
- package/docs/api-reference/workflow-errors/workflow-error.mdx +4 -4
- package/docs/api-reference/workflow-errors/workflow-not-registered-error.mdx +5 -6
- package/docs/api-reference/workflow-errors/workflow-run-cancelled-error.mdx +6 -6
- package/docs/api-reference/workflow-errors/workflow-run-failed-error.mdx +5 -5
- package/docs/api-reference/workflow-errors/workflow-run-not-completed-error.mdx +4 -4
- package/docs/api-reference/workflow-errors/workflow-run-not-found-error.mdx +4 -4
- package/docs/api-reference/workflow-errors/workflow-runtime-error.mdx +2 -2
- package/docs/api-reference/workflow-errors/workflow-world-error.mdx +8 -8
- package/docs/api-reference/workflow-globals.mdx +12 -11
- package/docs/api-reference/workflow-nest/configure-workflow-controller.mdx +2 -2
- package/docs/api-reference/workflow-nest/nest-local-builder.mdx +4 -4
- package/docs/api-reference/workflow-nest/workflow-controller.mdx +1 -1
- package/docs/api-reference/workflow-nest/workflow-module.mdx +4 -4
- package/docs/api-reference/workflow-next/with-workflow.mdx +16 -16
- package/docs/api-reference/workflow-nitro/index.mdx +6 -6
- package/docs/api-reference/workflow-nuxt/index.mdx +4 -4
- package/docs/api-reference/workflow-observability/hydrate-data.mdx +5 -5
- package/docs/api-reference/workflow-observability/hydrate-resource-io.mdx +4 -4
- package/docs/api-reference/workflow-observability/index.mdx +6 -8
- package/docs/api-reference/workflow-observability/observability-revivers.mdx +2 -2
- package/docs/api-reference/workflow-observability/parse-class-name.mdx +3 -3
- package/docs/api-reference/workflow-observability/parse-step-name.mdx +4 -4
- package/docs/api-reference/workflow-observability/parse-workflow-name.mdx +4 -4
- package/docs/api-reference/workflow-runtime/create-world.mdx +8 -8
- package/docs/api-reference/workflow-runtime/get-world-handlers.mdx +6 -6
- package/docs/api-reference/workflow-runtime/get-world.mdx +4 -4
- package/docs/api-reference/workflow-runtime/health-check.mdx +1 -1
- package/docs/api-reference/workflow-runtime/index.mdx +2 -4
- package/docs/api-reference/workflow-runtime/set-world.mdx +15 -13
- package/docs/api-reference/workflow-runtime/workflow-entrypoint.mdx +6 -5
- package/docs/api-reference/workflow-runtime/world/analytics.mdx +9 -9
- package/docs/api-reference/workflow-runtime/world/index.mdx +5 -3
- package/docs/api-reference/workflow-runtime/world/queue.mdx +11 -11
- package/docs/api-reference/workflow-runtime/world/storage.mdx +62 -28
- package/docs/api-reference/workflow-runtime/world/streams.mdx +7 -7
- package/docs/api-reference/workflow-serde/index.mdx +1 -1
- package/docs/api-reference/workflow-serde/workflow-deserialize.mdx +2 -2
- package/docs/api-reference/workflow-serde/workflow-serialize.mdx +7 -7
- package/docs/api-reference/workflow-sveltekit/workflow-plugin.mdx +3 -3
- package/docs/api-reference/workflow-vite/workflow.mdx +5 -5
- package/docs/changelog/attributes-mvp.mdx +39 -39
- package/docs/changelog/batched-event-writes.mdx +79 -0
- package/docs/changelog/eager-processing.mdx +63 -63
- package/docs/changelog/index.mdx +3 -3
- package/docs/changelog/lazy-event-creation.md +27 -27
- package/docs/changelog/meta.json +2 -1
- package/docs/changelog/resilient-resume.mdx +5 -5
- package/docs/changelog/resilient-start.mdx +14 -14
- package/docs/changelog/step-message-ownership.mdx +47 -47
- package/docs/changelog/turbo-mode.md +20 -20
- package/docs/comparisons/index.mdx +13 -13
- package/docs/comparisons/workflow-sdk-vs-aws-agentcore.mdx +15 -15
- package/docs/comparisons/workflow-sdk-vs-aws-step-functions.mdx +12 -12
- package/docs/comparisons/workflow-sdk-vs-cloudflare-workflows.mdx +11 -11
- package/docs/comparisons/workflow-sdk-vs-inngest.mdx +19 -19
- package/docs/comparisons/workflow-sdk-vs-temporal.mdx +23 -23
- package/docs/comparisons/workflow-sdk-vs-trigger-dev.mdx +18 -17
- package/docs/configuration/build-and-diagnostics.mdx +5 -5
- package/docs/configuration/cli-and-web-ui.mdx +4 -4
- package/docs/configuration/runtime-tuning.mdx +102 -21
- package/docs/configuration/worlds.mdx +34 -12
- package/docs/cookbook/advanced/child-workflows.mdx +25 -25
- package/docs/cookbook/advanced/publishing-libraries.mdx +40 -40
- package/docs/cookbook/advanced/serializable-steps.mdx +21 -21
- package/docs/cookbook/advanced/upgrading-workflows.mdx +31 -31
- package/docs/cookbook/agent-patterns/agent-cancellation.mdx +20 -20
- package/docs/cookbook/agent-patterns/human-in-the-loop.mdx +22 -22
- package/docs/cookbook/common-patterns/batching.mdx +14 -14
- package/docs/cookbook/common-patterns/idempotency.mdx +9 -9
- package/docs/cookbook/common-patterns/rate-limiting.mdx +3 -3
- package/docs/cookbook/common-patterns/saga.mdx +19 -19
- package/docs/cookbook/common-patterns/scheduling.mdx +23 -23
- package/docs/cookbook/common-patterns/sequential-and-parallel.mdx +26 -26
- package/docs/cookbook/common-patterns/timeouts.mdx +23 -23
- package/docs/cookbook/common-patterns/webhooks.mdx +6 -6
- package/docs/cookbook/common-patterns/workflow-composition.mdx +19 -19
- package/docs/cookbook/index.mdx +22 -22
- package/docs/cookbook/integrations/ai-sdk.mdx +43 -41
- package/docs/cookbook/integrations/chat-sdk.mdx +34 -34
- package/docs/cookbook/integrations/sandbox.mdx +46 -46
- package/docs/deploying.mdx +15 -15
- package/docs/errors/abort-signal-timeout-in-workflow.mdx +12 -12
- package/docs/errors/corrupted-event-log.mdx +13 -14
- package/docs/errors/deployment-mismatch.mdx +14 -14
- package/docs/errors/fetch-in-workflow.mdx +8 -8
- package/docs/errors/hook-conflict.mdx +11 -11
- package/docs/errors/index.mdx +1 -1
- package/docs/errors/node-js-module-in-workflow.mdx +5 -5
- package/docs/errors/replay-divergence.mdx +3 -3
- package/docs/errors/runtime-decryption-failed.mdx +12 -12
- package/docs/errors/serialization-failed.mdx +40 -12
- package/docs/errors/start-invalid-workflow-function.mdx +5 -5
- package/docs/errors/step-executed-multiple-times.mdx +2 -2
- package/docs/errors/step-not-registered.mdx +5 -5
- package/docs/errors/timeout-in-workflow.mdx +8 -8
- package/docs/errors/webhook-invalid-respond-with-value.mdx +18 -18
- package/docs/errors/webhook-response-not-sent.mdx +16 -16
- package/docs/errors/workflow-not-registered.mdx +5 -5
- package/docs/foundations/cancellation.mdx +31 -31
- package/docs/foundations/errors-and-retries.mdx +42 -11
- package/docs/foundations/hooks.mdx +37 -37
- package/docs/foundations/idempotency.mdx +9 -9
- package/docs/foundations/serialization.mdx +21 -22
- package/docs/foundations/starting-workflows.mdx +36 -37
- package/docs/foundations/streaming.mdx +68 -39
- package/docs/foundations/versioning.mdx +3 -3
- package/docs/foundations/workflows-and-steps.mdx +9 -9
- package/docs/getting-started/astro.mdx +16 -16
- package/docs/getting-started/express.mdx +8 -8
- package/docs/getting-started/fastify.mdx +8 -8
- package/docs/getting-started/hono.mdx +8 -8
- package/docs/getting-started/nestjs.mdx +18 -17
- package/docs/getting-started/next.mdx +11 -11
- package/docs/getting-started/nitro.mdx +16 -16
- package/docs/getting-started/nuxt.mdx +8 -8
- package/docs/getting-started/python.mdx +4 -4
- package/docs/getting-started/react-router/v7.mdx +1 -1
- package/docs/getting-started/react-router/v8.mdx +1 -1
- package/docs/getting-started/sveltekit.mdx +14 -14
- package/docs/getting-started/tanstack-start.mdx +12 -12
- package/docs/getting-started/vite.mdx +8 -8
- package/docs/how-it-works/cancellation.mdx +62 -62
- package/docs/how-it-works/code-transform.mdx +66 -54
- package/docs/how-it-works/encryption.mdx +25 -21
- package/docs/how-it-works/event-sourcing.mdx +84 -28
- package/docs/how-it-works/framework-integrations.mdx +12 -12
- package/docs/how-it-works/understanding-directives.mdx +21 -21
- package/docs/internal/index.mdx +6 -6
- package/docs/internal/nitro-native-build.mdx +2 -2
- package/docs/internal/nitro-web-ui.mdx +4 -4
- package/docs/internal/serializable-abort-controller.mdx +7 -7
- package/docs/observability/attributes.mdx +3 -3
- package/docs/observability/index.mdx +14 -10
- package/docs/observability/tracing.mdx +10 -10
- package/docs/testing/index.mdx +35 -35
- package/docs/testing/server-based.mdx +10 -10
- package/docs/whats-new.mdx +185 -0
- package/package.json +12 -12
|
@@ -1,20 +1,20 @@
|
|
|
1
1
|
---
|
|
2
|
-
title: Eager
|
|
3
|
-
description: Combine workflow event replay and step bundles to do work inline where possible, only deferring to queue for parallelism
|
|
2
|
+
title: Eager processing of steps and incremental event replay
|
|
3
|
+
description: Combine workflow event replay and step bundles to do work inline where possible, only deferring to the queue for parallelism.
|
|
4
4
|
type: overview
|
|
5
5
|
---
|
|
6
6
|
|
|
7
|
-
# Eager
|
|
7
|
+
# Eager processing of steps and incremental event replay
|
|
8
8
|
|
|
9
9
|
**Date**: March 2026
|
|
10
10
|
|
|
11
11
|
This is a major internal architecture change to how Workflow DevKit executes workflows and steps. It reduces function invocations and queue overhead by executing steps _inline_ within the same function invocation as the workflow replay, rather than dispatching every step to a separate function via the queue.
|
|
12
12
|
|
|
13
|
-
## Previous
|
|
13
|
+
## Previous architecture
|
|
14
14
|
|
|
15
15
|
The previous architecture used two separate routes, each backed by its own queue trigger:
|
|
16
16
|
|
|
17
|
-
```
|
|
17
|
+
```text
|
|
18
18
|
Queue: __wkf_workflow_* --> /.well-known/workflow/v1/flow (workflow replay in VM)
|
|
19
19
|
|
|
|
20
20
|
suspension (step needed)
|
|
@@ -30,15 +30,15 @@ Queue: __wkf_step_* --> /.well-known/workflow/v1/step (step execution in
|
|
|
30
30
|
(cycle repeats for each step)
|
|
31
31
|
```
|
|
32
32
|
|
|
33
|
-
Each step required **2 queue messages** (step
|
|
33
|
+
Each step required **2 queue messages** (step invocation and workflow continuation) and **2 function invocations**, plus cold-start overhead for each. A serial workflow with 10 steps needed approximately 21 function invocations.
|
|
34
34
|
|
|
35
|
-
## New
|
|
35
|
+
## New architecture
|
|
36
36
|
|
|
37
37
|
The two routes are merged into a single handler at `/.well-known/workflow/v1/flow` using `workflowEntrypoint()`. The step route is no longer generated.
|
|
38
38
|
|
|
39
39
|
The handler runs an inline execution loop:
|
|
40
40
|
|
|
41
|
-
```
|
|
41
|
+
```text
|
|
42
42
|
receive queue message
|
|
43
43
|
|
|
|
44
44
|
+-- if message has stepId+stepName: execute that step, queue workflow continuation, exit
|
|
@@ -68,9 +68,9 @@ suspension with pending operations
|
|
|
68
68
|
|
|
69
69
|
A serial workflow with 10 steps now completes in **1 function invocation**.
|
|
70
70
|
|
|
71
|
-
## Background
|
|
71
|
+
## Background steps (parallel execution)
|
|
72
72
|
|
|
73
|
-
When a workflow suspends with multiple pending steps (
|
|
73
|
+
When a workflow suspends with multiple pending steps (for example, from `Promise.all`), the handler creates `step_created` events for all of them, queues N-1 back to `__wkf_workflow_*` with `stepId` and `stepName` in the message payload, executes 1 step inline, and loops back to replay.
|
|
74
74
|
|
|
75
75
|
Each background step message is handled by a separate function invocation of the same handler. When a message arrives with `stepId` and `stepName`, the handler executes that specific step, then checks if all parallel steps from the batch are done by comparing `step_created` events against terminal events (`step_completed`/`step_failed`):
|
|
76
76
|
|
|
@@ -78,56 +78,56 @@ Each background step message is handled by a separate function invocation of the
|
|
|
78
78
|
- **Steps still pending**: The handler returns without queuing a continuation. The last handler to complete its step will see all steps done and replay inline.
|
|
79
79
|
- **Pending ops (stream writes)**: The handler queues a continuation and returns, so `waitUntil` can flush the pending stream data.
|
|
80
80
|
|
|
81
|
-
### Convergence
|
|
81
|
+
### Convergence after parallel steps
|
|
82
82
|
|
|
83
83
|
When multiple background steps complete near-simultaneously, multiple handlers may observe "all steps done" and attempt to advance the workflow concurrently. The event-sourced architecture plus the invariants below ensure safe convergence:
|
|
84
84
|
|
|
85
|
-
- **`step_created` idempotency
|
|
86
|
-
- **`step_completed` / `step_failed` idempotency
|
|
87
|
-
- **Queue idempotency keys
|
|
88
|
-
- **Deterministic replay
|
|
85
|
+
- **`step_created` idempotency**: Duplicate creates return 409; exactly one handler owns each step.
|
|
86
|
+
- **`step_completed` / `step_failed` idempotency**: Only the first invocation to record a terminal result wins.
|
|
87
|
+
- **Queue idempotency keys**: Background step messages use `correlationId` as the idempotency key.
|
|
88
|
+
- **Deterministic replay**: All invocations produce the same result given the same event log.
|
|
89
89
|
|
|
90
|
-
### Single
|
|
90
|
+
### Single inline executor per step
|
|
91
91
|
|
|
92
92
|
Inline step execution combined with background-step dispatch introduces a new coordination requirement: when multiple handlers reach the same `Promise.all` batch concurrently, we need to guarantee that each step body runs at most once via the inline path. Without that guarantee, the event log accumulates duplicate `step_started` events (including some written *after* `step_completed`, which orphans them on replay) and step bodies run redundantly.
|
|
93
93
|
|
|
94
|
-
The design enforces
|
|
94
|
+
The design enforces one invariant: **exactly one handler owns each step, and only the owner may execute it inline**. Ownership is established by the atomicity of `step_created`:
|
|
95
95
|
|
|
96
|
-
1. **Atomic `step_created
|
|
97
|
-
2. **Suspension handler reports ownership
|
|
98
|
-
3. **Inline execution is gated on ownership
|
|
99
|
-
4. **Queueing is unconditional
|
|
96
|
+
1. **Atomic `step_created`**: `events.create('step_created', correlationId=X)` is serialized per correlation ID in every world. Exactly one concurrent caller succeeds; the rest receive `EntityConflictError`.
|
|
97
|
+
2. **Suspension handler reports ownership**: Only `step_created` writes that succeeded (not those that caught 409) count toward ownership.
|
|
98
|
+
3. **Inline execution is gated on ownership**: A handler that didn't win any `step_created` race performs no inline execution.
|
|
99
|
+
4. **Queueing is unconditional**: For every pending step except the one being executed inline, the handler enqueues a background step message with `idempotencyKey: correlationId`. This makes crash recovery work: if a prior handler wrote `step_created` but crashed before enqueueing, a later handler will enqueue the orphaned step. Concurrent handlers' redundant enqueues deduplicate on the idempotency key.
|
|
100
100
|
|
|
101
101
|
Together these give: every `step_created` event has exactly one inline executor **and** at least one queued dispatch. Step bodies are never executed concurrently, and `step_started` events never land in the log after `step_completed` for the same step.
|
|
102
102
|
|
|
103
|
-
**Retry semantics are preserved**: a step that is currently running (status=`running`) still accepts a second `step_started` write with an incremented attempt counter
|
|
103
|
+
**Retry semantics are preserved**: a step that is currently running (status=`running`) still accepts a second `step_started` write with an incremented attempt counter: this is how queue redelivery after a SIGKILL mid-execution legitimately re-runs the step.
|
|
104
104
|
|
|
105
|
-
## Incremental
|
|
105
|
+
## Incremental event loading
|
|
106
106
|
|
|
107
107
|
The handler caches the event log in memory across loop iterations. Instead of re-fetching the entire event log on each replay:
|
|
108
108
|
|
|
109
|
-
1. **First iteration**:
|
|
110
|
-
2. **Subsequent iterations**:
|
|
109
|
+
1. **First iteration**: Load all events and return the final pagination cursor.
|
|
110
|
+
2. **Subsequent iterations**: Fetch only events created after the saved cursor and append them to the cached array.
|
|
111
111
|
|
|
112
112
|
For a 10-step serial workflow completing in one invocation, the 10th replay loads ~2 new events instead of re-fetching all ~30.
|
|
113
113
|
|
|
114
114
|
Incremental loading depends on the World returning a cursor even on the final page of results. If a World implementation does not return a cursor after the initial load, the handler logs an error and falls back to a full reload.
|
|
115
115
|
|
|
116
|
-
## Timeout
|
|
116
|
+
## Timeout handling
|
|
117
117
|
|
|
118
|
-
The inline execution loop checks wall-clock time before each replay iteration. If the elapsed time exceeds a configurable threshold (
|
|
118
|
+
The inline execution loop checks wall-clock time before each replay iteration. If the elapsed time exceeds a configurable threshold (110s by default for a 120-second function limit), the handler reschedules itself through the queue and returns. Configure the threshold with `WORKFLOW_V2_TIMEOUT_MS`.
|
|
119
119
|
|
|
120
|
-
If a single step takes longer than the timeout threshold, the step runs to completion (or SIGKILL)
|
|
120
|
+
If a single step takes longer than the timeout threshold, the step runs to completion (or SIGKILL). There is no interruption mechanism for in-progress step execution. This is the same behavior as the previous architecture.
|
|
121
121
|
|
|
122
|
-
## Queue
|
|
122
|
+
## Queue message changes
|
|
123
123
|
|
|
124
|
-
The `WorkflowInvokePayload` schema has two new optional fields: `stepId` and `stepName`. When `stepId` is present, the handler executes that specific step before (or instead of) replaying the workflow. Background steps are queued with both set, so the handler knows which step function to call without loading the event log. Previously, `stepName` was resolved by loading all events and searching for the `step_created` event matching the `stepId
|
|
124
|
+
The `WorkflowInvokePayload` schema has two new optional fields: `stepId` and `stepName`. When `stepId` is present, the handler executes that specific step before (or instead of) replaying the workflow. Background steps are queued with both set, so the handler knows which step function to call without loading the event log. Previously, `stepName` was resolved by loading all events and searching for the `step_created` event matching the `stepId`, an O(N) operation on the full event history for every background step arrival.
|
|
125
125
|
|
|
126
126
|
The queue trigger configuration uses `WORKFLOW_QUEUE_TRIGGER` on the `__wkf_workflow_*` topic. The `__wkf_step_*` topic and its separate trigger are no longer generated.
|
|
127
127
|
|
|
128
|
-
## Generated
|
|
128
|
+
## Generated file layout
|
|
129
129
|
|
|
130
|
-
```
|
|
130
|
+
```text
|
|
131
131
|
.well-known/workflow/v1/
|
|
132
132
|
flow/
|
|
133
133
|
route.js # Handler (workflowEntrypoint)
|
|
@@ -141,11 +141,11 @@ The queue trigger configuration uses `WORKFLOW_QUEUE_TRIGGER` on the `__wkf_work
|
|
|
141
141
|
|
|
142
142
|
The `step/` directory is no longer generated.
|
|
143
143
|
|
|
144
|
-
## Design
|
|
144
|
+
## Design notes and tradeoffs
|
|
145
145
|
|
|
146
|
-
### Parent→
|
|
146
|
+
### Parent→child polling holds worker slots
|
|
147
147
|
|
|
148
|
-
`Run#returnValue` is implemented as a polling step: the workflow awaits the child run's terminal status inside a step body. In worker-based
|
|
148
|
+
`Run#returnValue` is implemented as a polling step: the workflow awaits the child run's terminal status inside a step body. In worker-based Worlds (notably `world-postgres`), each such poll occupies a queue worker slot until the child run finishes. Parent workflows that fan out to many child runs, such as recursive workflows like `fibonacciWorkflow`, can therefore consume a large fraction of available workers while holding positions in `Promise.all([...children.map(c => c.returnValue)])`.
|
|
149
149
|
|
|
150
150
|
If `queueConcurrency` is smaller than the peak number of concurrent parent polls plus the workers needed for any in-flight children, the system deadlocks: every slot is held by a parent waiting on a child, but no child can acquire a slot to start.
|
|
151
151
|
|
|
@@ -155,11 +155,11 @@ To prevent deadlock when polling is executed inline by the step executor, `Run#p
|
|
|
155
155
|
|
|
156
156
|
**Follow-up**: Replace the worker-pool sizing requirement with a polling design that does not occupy a worker slot. Options under consideration: moving child-completion polling out of the step body into the suspension layer, or emitting a `run_completed` notification on the parent's stream/queue so the parent only resumes when the child actually finishes.
|
|
157
157
|
|
|
158
|
-
### Mixed
|
|
158
|
+
### Mixed suspensions
|
|
159
159
|
|
|
160
160
|
A suspension may contain steps, hooks, and waits simultaneously. The handler creates events for all, then dispatches everything we are not running inline as a single parallel batch of queue messages:
|
|
161
161
|
|
|
162
|
-
```
|
|
162
|
+
```text
|
|
163
163
|
ownedPendingSteps = pendingSteps.filter(owned by this handler)
|
|
164
164
|
inlineStep = ownedPendingSteps[0] // optional
|
|
165
165
|
|
|
@@ -187,71 +187,71 @@ The retry/throttle and hook-conflict paths still return `{ timeoutSeconds }` sin
|
|
|
187
187
|
|
|
188
188
|
The unified dispatch requires `world-local` to honor `delaySeconds` on the queue (added in the same PR series). Without it, the wait continuation would fire instantly in dev and trigger a spurious replay before the wait elapsed (recoverable via redelivery, but inefficient and observable as duplicate `step_started` events under contention).
|
|
189
189
|
|
|
190
|
-
### VM
|
|
190
|
+
### VM sandboxing
|
|
191
191
|
|
|
192
192
|
Workflow code still runs in a Node.js VM for determinism and sandboxing. Step code runs in the Node.js host context. The only change is that both happen within the same function invocation.
|
|
193
193
|
|
|
194
|
-
### Bundle
|
|
194
|
+
### Bundle size and cold start
|
|
195
195
|
|
|
196
196
|
The combined bundle is larger (contains both step code and workflow VM code). Cold start time increases slightly. The reduction in total function invocations more than compensates.
|
|
197
197
|
|
|
198
|
-
### Step
|
|
198
|
+
### Step retries
|
|
199
199
|
|
|
200
200
|
When an inline step fails with retries remaining:
|
|
201
201
|
|
|
202
|
-
- `RetryableError` with explicit `retryAfter` delay:
|
|
203
|
-
- Transient errors with immediate retry:
|
|
204
|
-
- `FatalError`:
|
|
202
|
+
- `RetryableError` with explicit `retryAfter` delay: Requeue to self with `stepId` and a delay.
|
|
203
|
+
- Transient errors with immediate retry: Requeue to self with `stepId` and a 1s delay.
|
|
204
|
+
- `FatalError`: Fail immediately.
|
|
205
205
|
|
|
206
|
-
### Encryption
|
|
206
|
+
### Encryption key resolution
|
|
207
207
|
|
|
208
208
|
Encryption keys are resolved once before the inline execution loop starts (after the run status is confirmed as `running`) and reused across all iterations. Background step executions resolve the key independently. The key does not change within a run.
|
|
209
209
|
|
|
210
|
-
### Module
|
|
210
|
+
### Module scope duplication in re-bundled output
|
|
211
211
|
|
|
212
|
-
Builders that
|
|
212
|
+
Builders that rebundle the combined output into a single file (standalone Workflow CLI, Vercel Build Output API, and NestJS) produce a layout where esbuild creates isolated module scopes for each source module, even within the same output file. Without intervention, `registerStepFunction` and `getStepFunction` operate on different `Map` instances: steps are registered into one `Map` but looked up from another.
|
|
213
213
|
|
|
214
214
|
The step function registry and the step context storage are `globalThis` singletons (via `Symbol.for`) to ensure all module scopes share the same instances. The same pattern is used for the World singleton and the class serialization registry.
|
|
215
215
|
|
|
216
|
-
### Inline
|
|
216
|
+
### Inline step execution with pending stream operations
|
|
217
217
|
|
|
218
218
|
When a step's arguments or return value include serialized streams (e.g., `WritableStream` from `getWritable()`, or AI SDK streaming steps), the serialization layer creates background `flushablePipe` operations that pipe data to S3. These ops are tracked in an `ops` array and need to complete before the stream data is readable by external consumers.
|
|
219
219
|
|
|
220
|
-
In V1, each step ran in a separate function invocation. After the step completed, `waitUntil(ops)` kept the function alive to flush the ops. In V2, the inline execution loop continues immediately after the step body returns
|
|
220
|
+
In V1, each step ran in a separate function invocation. After the step completed, `waitUntil(ops)` kept the function alive to flush the ops. In V2, the inline execution loop continues immediately after the step body returns, so we need to know whether to keep looping or break out and let `waitUntil` flush.
|
|
221
221
|
|
|
222
222
|
`executeStep()` attempts a 500ms `Promise.race` between the ops settling and a timeout. If ops settle in time (data confirmed on server), it returns `hasPendingOps: false` and the V2 handler continues the inline loop. If ops don't settle in 500ms (e.g., `WritableStream` kept open across steps), it returns `hasPendingOps: true` and the V2 handler breaks the loop and queues a continuation so `waitUntil` can flush them.
|
|
223
223
|
|
|
224
224
|
**Follow-up**: Shrink the 500ms inline-ops budget once we have confidence that the flush-waiter path settles deterministically across all worlds. A signaled "ops drained" event from the world layer would let `executeStep()` proceed without the timeout in the common case.
|
|
225
225
|
|
|
226
|
-
### Buffered
|
|
226
|
+
### Buffered stream flush with waiter promises
|
|
227
227
|
|
|
228
|
-
`WorkflowServerWritableStream` buffers writes and flushes via a 10ms `setTimeout` for batching. Naively, `write()` could return immediately after buffering, but that would cause the `flushablePipe`'s `pendingOps` counter to reach 0 before data actually reached the server
|
|
228
|
+
`WorkflowServerWritableStream` buffers writes and flushes via a 10ms `setTimeout` for batching. Naively, `write()` could return immediately after buffering, but that would cause the `flushablePipe`'s `pendingOps` counter to reach 0 before data actually reached the server: the V2 inline loop would see ops as settled prematurely and produce data-loss races on every step with `WritableStream` serialization.
|
|
229
229
|
|
|
230
230
|
`write()` returns a promise that resolves only after the scheduled flush completes. Multiple writes within the 10ms window still share a single batched HTTP request (the batching optimization is preserved). Each write registers a `{resolve, reject}` pair in a `flushWaiters` array. When the `setTimeout` fires and `flush()` completes the HTTP round-trip, all waiters are resolved (or rejected on error). This makes `pendingOps` accurately reflect server-side data state while keeping network-efficient batching.
|
|
231
231
|
|
|
232
|
-
### Lock-
|
|
232
|
+
### Lock-release polling interval
|
|
233
233
|
|
|
234
|
-
`flushablePipe`'s `pollWritableLock` / `pollReadableLock` use `setInterval` to detect when a user releases their stream lock without closing the stream
|
|
234
|
+
`flushablePipe`'s `pollWritableLock` / `pollReadableLock` use `setInterval` to detect when a user releases their stream lock without closing the stream, since the Web Streams API has no event for that state. The V2 step executor's `opsSettled` race waits for this poll to resolve after each writable-bearing step body returns, so the polling interval sits on the critical path of every streaming step.
|
|
235
235
|
|
|
236
|
-
The interval was lowered from 100ms to 10ms. Per-step wait drops from ~50ms average to ~5ms (scaling linearly with the number of writable-bearing steps in a workflow). Per-tick work is
|
|
236
|
+
The interval was lowered from 100ms to 10ms. Per-step wait drops from ~50ms average to ~5ms (scaling linearly with the number of writable-bearing steps in a workflow). Per-tick work is `writable.locked` plus a `getWriter()`/`releaseLock()` probe (microsecond-scale), so 10× more ticks is not measurable in practice.
|
|
237
237
|
|
|
238
|
-
**Follow-up**: Replace polling entirely with an event-driven release signal
|
|
238
|
+
**Follow-up**: Replace polling entirely with an event-driven release signal (wrap the writable returned from the `WritableStream` reviver with a writer that fires on `releaseLock()`), bringing the wait to ~0ms. This would also remove a source of timing drift between worlds with synchronous storage (`world-local`) and worlds with HTTP-deferred storage (`world-vercel`).
|
|
239
239
|
|
|
240
|
-
### Concurrent `step_started` and
|
|
240
|
+
### Concurrent `step_started` and attempt counter
|
|
241
241
|
|
|
242
|
-
When the V2 handler dispatches N parallel steps as background messages, each background step completion queues a workflow continuation. Up to N continuations may replay concurrently, and each may attempt to start the same not-yet-completed step (since `step_started` succeeds for already-running steps). Each call atomically increments the `attempt` counter
|
|
242
|
+
When the V2 handler dispatches N parallel steps as background messages, each background step completion queues a workflow continuation. Up to N continuations may replay concurrently, and each may attempt to start the same not-yet-completed step (since `step_started` succeeds for already-running steps). Each call atomically increments the `attempt` counter, so with N=5 parallel steps, the counter can reach 5 on the first genuine execution.
|
|
243
243
|
|
|
244
|
-
The max retries check in `executeStep()` only enforces when `step.error` exists
|
|
244
|
+
The max retries check in `executeStep()` only enforces when `step.error` exists, distinguishing actual retries (failed → retry with error) from concurrent first-attempt races (multiple handlers start the same step simultaneously without any prior failure). Concurrent starts are harmless since `step_completed` idempotency ensures only the first completion wins.
|
|
245
245
|
|
|
246
|
-
### Unconsumed
|
|
246
|
+
### Unconsumed event check two-phase drain
|
|
247
247
|
|
|
248
248
|
The `EventsConsumer`'s unconsumed event check uses a two-phase promise queue drain: yield once after the first drain (via `setTimeout(0)`) so cross-VM promise chains can append follow-up async work, then re-drain before checking. This improves timing for scenarios like `step_completed` → for-await loop resume → next hook hydration.
|
|
249
249
|
|
|
250
|
-
The check additionally arms a `DEFERRED_CHECK_DELAY_MS = 100` `setTimeout` after the second drain, since Node.js does not guarantee that `setTimeout(0)` fires after all cross-context microtasks settle. Any `subscribe()` call arriving during that 100ms window cancels the check via version invalidation + `clearTimeout`, so the delay only adds latency to genuine corruption
|
|
250
|
+
The check additionally arms a `DEFERRED_CHECK_DELAY_MS = 100` `setTimeout` after the second drain, since Node.js does not guarantee that `setTimeout(0)` fires after all cross-context microtasks settle. Any `subscribe()` call arriving during that 100ms window cancels the check via version invalidation + `clearTimeout`, so the delay only adds latency to genuine corruption, never to the happy path.
|
|
251
251
|
|
|
252
|
-
**Follow-up**: 100ms is a heuristic chosen empirically. A deterministic settlement signal
|
|
252
|
+
**Follow-up**: 100ms is a heuristic chosen empirically. A deterministic settlement signal (for example, a "VM idle" callback exposed by the workflow VM bridge that fires only after all pending cross-context promise chains have resolved) would let the consumer fire the unconsumed-event check immediately on quiescence instead of waiting for a wall-clock timeout.
|
|
253
253
|
|
|
254
|
-
### Lazy World
|
|
254
|
+
### Lazy World loading
|
|
255
255
|
|
|
256
256
|
Static imports of `world-local` and `world-vercel` from the runtime caused two distinct build-time issues: Next.js production builds pulled both worlds (including their Node-only deps like `debug`'s `tty` requires) into the route module, and Turbopack's NFT (Node File Trace) errored on `process.cwd()` and dynamic `import()` patterns it couldn't statically analyze.
|
|
257
257
|
|
|
@@ -259,8 +259,8 @@ A `getWorldLazy()` accessor (backed by a `globalThis` `Symbol.for` cache) replac
|
|
|
259
259
|
|
|
260
260
|
Because tree-shaking can otherwise drop `world.ts`'s module-load registration entirely, a server-only side-effect module (`@workflow/core/runtime/world-init`) imports `./world.js` purely for its module-load side effect. It's wired via package conditions:
|
|
261
261
|
|
|
262
|
-
- `default`
|
|
263
|
-
- `workflow`
|
|
262
|
+
- `default` maps to the real module and loads `world.ts`.
|
|
263
|
+
- `workflow` maps to an empty stub used by VM and step bundles.
|
|
264
264
|
|
|
265
265
|
This guarantees the world is loaded for routes that consume `start()` without going through the queue-driven flow handler first, while keeping `world.ts` and its server-only deps out of the workflow sandbox bundle.
|
|
266
266
|
|
package/docs/changelog/index.mdx
CHANGED
|
@@ -12,6 +12,6 @@ Stay up to date with the latest changes to Workflow SDK.
|
|
|
12
12
|
|
|
13
13
|
## 2026
|
|
14
14
|
|
|
15
|
-
- [Resilient hook resume](/docs/changelog/resilient-resume)
|
|
16
|
-
- [Eager processing of steps and incremental event replay](/docs/changelog/eager-processing)
|
|
17
|
-
- Serializable AbortController and AbortSignal
|
|
15
|
+
- [Resilient hook resume](/docs/changelog/resilient-resume) (July 2026)
|
|
16
|
+
- [Eager processing of steps and incremental event replay](/docs/changelog/eager-processing) (March 2026)
|
|
17
|
+
- Serializable AbortController and AbortSignal (March 12, 2026)
|
|
@@ -9,13 +9,13 @@ description: Defer step_created for the inline step and fold it into a single st
|
|
|
9
9
|
|
|
10
10
|
The owned-inline runtime path used to write two separate world events for a step it already owns and is about to run inline:
|
|
11
11
|
|
|
12
|
-
1. `step_created
|
|
13
|
-
2. `step_started
|
|
14
|
-
3. `step_completed` / `step_failed
|
|
12
|
+
1. `step_created`, written by the suspension handler (`suspension-handler.ts`)
|
|
13
|
+
2. `step_started`, written by `executeStep` (`step-executor.ts`)
|
|
14
|
+
3. `step_completed` / `step_failed`, written by `executeStep`
|
|
15
15
|
|
|
16
|
-
On the Vercel world each `world.events.create` is a network round-trip, so for a
|
|
16
|
+
On the Vercel world each `world.events.create` is a network round-trip, so for a sequential `"use step"` workflow this is pure latency between steps. Steps (1) and (2) are two round-trips for a step we already own and are about to execute in the same invocation.
|
|
17
17
|
|
|
18
|
-
This change defers `step_created` for that one inline step: `executeStep` sends a single `step_started` carrying the step input, and the world creates the step on the fly
|
|
18
|
+
This change defers `step_created` for that one inline step: `executeStep` sends a single `step_started` carrying the step input, and the world creates the step on the fly. This materializes the step entity **and** a synthetic `step_created` event so replay still observes it. **Two writes per inline step instead of three.** It mirrors the existing [resilient `run_started` → `run_created`](./resilient-start) pattern.
|
|
19
19
|
|
|
20
20
|
Steps that are *queued* (not run inline) keep their eager `step_created` and are unchanged. Only the single inline step per suspension is made lazy.
|
|
21
21
|
|
|
@@ -23,8 +23,8 @@ Steps that are *queued* (not run inline) keep their eager `step_created` and are
|
|
|
23
23
|
|
|
24
24
|
### Suspension handler
|
|
25
25
|
|
|
26
|
-
- `handleSuspension` selects exactly one step to defer
|
|
27
|
-
- For that step it **skips** the `step_created` write and instead returns it as `lazyInlineStep = { correlationId, stepName, dehydratedInput }`. It is **not** added to `createdStepCorrelationIds`
|
|
26
|
+
- `handleSuspension` selects exactly one step to defer: the first uncreated step (`stepItems.find(item => stepsNeedingCreation.has(...))`), which matches the inline candidate the caller would have picked.
|
|
27
|
+
- For that step it **skips** the `step_created` write and instead returns it as `lazyInlineStep = { correlationId, stepName, dehydratedInput }`. It is **not** added to `createdStepCorrelationIds` because ownership is no longer decided here.
|
|
28
28
|
- A `lazyInlineStep` is designated only when there is no `hook.getConflict()` awaiter (`hasAwaitedHookCreation === false`). With an awaiter present nothing runs inline, so nothing is deferred.
|
|
29
29
|
|
|
30
30
|
### `executeStep`
|
|
@@ -34,7 +34,7 @@ Steps that are *queued* (not run inline) keep their eager `step_created` and are
|
|
|
34
34
|
|
|
35
35
|
### World contract
|
|
36
36
|
|
|
37
|
-
- `step_started` accepts an optional `input`. When provided for a non-existent step, the world creates the step entity plus a **synthetic `step_created`** event (so the event log reads `created → started → completed`), then records `step_started
|
|
37
|
+
- `step_started` accepts an optional `input`. When provided for a non-existent step, the world atomically creates the step entity plus a **synthetic `step_created`** event (so the event log reads `created → started → completed`), then records `step_started`.
|
|
38
38
|
- A `stepCreated` signal is added to the event result so callers can tell whether the lazy `step_started` created the step or attached to an existing one.
|
|
39
39
|
- Worlds updated: `world-local`, `world-postgres`, `world-vercel`.
|
|
40
40
|
|
|
@@ -42,12 +42,12 @@ Steps that are *queued* (not run inline) keep their eager `step_created` and are
|
|
|
42
42
|
|
|
43
43
|
### Exactly-one-owner is preserved (the race moved)
|
|
44
44
|
|
|
45
|
-
The guarantee that exactly one handler runs a step's body inline is intact
|
|
45
|
+
The guarantee that exactly one handler runs a step's body inline is intact, but it resolves at a different event:
|
|
46
46
|
|
|
47
47
|
- **Before:** ownership was won at the atomic `step_created` claim in the suspension handler; the loser caught `EntityConflictError` and queued instead.
|
|
48
48
|
- **After:** two concurrent handlers may both *select* the same `lazyInlineStep` (selection is optimistic, before any race). The race is now the world's atomic create-claim inside the lazy `step_started` (lock file in world-local, `onConflictDoNothing` + unique index in world-postgres, `attribute_not_exists` on the server). The loser gets `EntityConflictError`, which `executeStep` maps to `{ type: 'skipped' }`, so it never runs the body.
|
|
49
49
|
|
|
50
|
-
This is safe **because a lazy `step_started` is only ever sent for a brand-new step
|
|
50
|
+
This is safe **because a lazy `step_started` is only ever sent for a brand-new step**. The suspension handler defers only steps with no prior `step_created` (`!hasCreatedEvent`).
|
|
51
51
|
|
|
52
52
|
### Crash recovery is unchanged
|
|
53
53
|
|
|
@@ -55,9 +55,9 @@ On crash recovery the step already has a `step_created` event in the log (`hasCr
|
|
|
55
55
|
|
|
56
56
|
### Materialize before failing unregistered steps
|
|
57
57
|
|
|
58
|
-
The inline path previously assumed an invariant: *by the time `executeStep` runs, the step entity already exists* (the suspension handler created it). The unregistered-step branch relied on this
|
|
58
|
+
The inline path previously assumed an invariant: *by the time `executeStep` runs, the step entity already exists* (the suspension handler created it). The unregistered-step branch relied on this and writes `step_failed` directly, with no preceding `step_started`.
|
|
59
59
|
|
|
60
|
-
With the `step_created` deferred, the entity no longer exists when `executeStep` bails for an unregistered step, so the `step_failed` write hits the world's "step must exist" ordering guard and is rejected
|
|
60
|
+
With the `step_created` deferred, the entity no longer exists when `executeStep` bails for an unregistered step, so the `step_failed` write hits the world's "step must exist" ordering guard and is rejected, wedging the run until it times out. On the lazy path, send the lazy `step_started` first (creating the entity + synthetic `step_created`), then write `step_failed`. The lazy `step_started`'s atomic create-claim still preserves exactly-one-owner. A concurrent winner makes our create reject with `EntityConflictError` → `skipped`, so the failure is never written twice.
|
|
61
61
|
|
|
62
62
|
This is the general rule the deferral introduces: **any inline path that writes a terminal step event must first ensure the deferred step has been materialized.**
|
|
63
63
|
|
|
@@ -65,25 +65,25 @@ This is the general rule the deferral introduces: **any inline path that writes
|
|
|
65
65
|
|
|
66
66
|
The client step consumer (`step.ts`) sets `hasCreatedEvent` only when it observes a `step_created` event, and checks step-name divergence against `stepName`. The lazy path stays replay-correct only because the world writes a **synthetic `step_created`**: replay still sees `created → started → completed`. The input lives on the synthetic `step_created`; the `step_started` row drops the input but keeps `stepName` for the divergence check.
|
|
67
67
|
|
|
68
|
-
This intersects with the inline-delta optimization. The delta returned on the `step_completed` write is consumed by the *next* replay in place of an `events.list`, diffed against `preInlineWriteCursor` (snapshotted before replay). Because the synthetic `step_created`, `step_started`, and `step_completed` are all written *after* that cursor, the world's "events since cursor" delta carries the full triple
|
|
68
|
+
This intersects with the inline-delta optimization. The delta returned on the `step_completed` write is consumed by the *next* replay in place of an `events.list`, diffed against `preInlineWriteCursor` (snapshotted before replay). Because the synthetic `step_created`, `step_started`, and `step_completed` are all written *after* that cursor, the world's "events since cursor" delta carries the full triple, so the next replay does not diverge. The delta gate (one step, no hooks/waits, the lone pending step is the inline one) is unchanged.
|
|
69
69
|
|
|
70
70
|
### Pre-emption by attributes / hook conflicts
|
|
71
71
|
|
|
72
72
|
When `attr_set` events force an immediate in-process replay, or a hook conflict forces a re-invocation, the handler skips the dispatch loop for that pass. The deferred step is therefore **neither created nor queued** on that pass; it is recreated and run on the following replay (where it is still a lazy candidate).
|
|
73
73
|
|
|
74
|
-
This is a small behavioral improvement: previously the eager `step_created` left an orphan "created but never started" event when a step lost an attribute/hook race (e.g. `Promise.race([setAttributes(), step()])` where the attribute write wins and completes the run). With deferral, a step that loses the race is never created
|
|
74
|
+
This is a small behavioral improvement: previously the eager `step_created` left an orphan "created but never started" event when a step lost an attribute/hook race (e.g. `Promise.race([setAttributes(), step()])` where the attribute write wins and completes the run). With deferral, a step that loses the race is never created, reducing event-log garbage.
|
|
75
75
|
|
|
76
76
|
### `hook.getConflict()` awaiter
|
|
77
77
|
|
|
78
|
-
When a `hook.getConflict()` awaiter is present, no `lazyInlineStep` is designated, nothing runs inline, every step gets its eager `step_created` and is queued, and the handler re-invokes immediately so replay resolves the awaiter. This is identical to the pre-change behavior
|
|
78
|
+
When a `hook.getConflict()` awaiter is present, no `lazyInlineStep` is designated, nothing runs inline, every step gets its eager `step_created` and is queued, and the handler re-invokes immediately so replay resolves the awaiter. This is identical to the pre-change behavior. The deferral never serializes the awaiter's parallel continuation behind an inline step.
|
|
79
79
|
|
|
80
80
|
## Rollout and compatibility
|
|
81
81
|
|
|
82
|
-
Server-first. The matching world-vercel backend change must deploy before this ships; the Vercel world targets a single backend whose spec version is always at least the SDK's, so the new SDK only ever talks to an already-upgraded backend. An old SDK against a new backend is safe because the lazy path is strictly additive
|
|
82
|
+
Server-first. The matching world-vercel backend change must deploy before this ships; the Vercel world targets a single backend whose spec version is always at least the SDK's, so the new SDK only ever talks to an already-upgraded backend. An old SDK against a new backend is safe because the lazy path is strictly additive. It triggers only when `step_started` carries both `stepName` and `input`, which old SDKs never send. For `world-local` / `world-postgres` the world ships in the same package as the runtime, so there is no version skew. Detection is by `input` presence on the event, mirroring resilient `run_started`, so no capability negotiation is needed.
|
|
83
83
|
|
|
84
84
|
# Parallel inline steps + optimistic start
|
|
85
85
|
|
|
86
|
-
A follow-up builds two more latency wins on top of lazy inline start. Both are client-side only
|
|
86
|
+
A follow-up builds two more latency wins on top of lazy inline start. Both are client-side only; they reuse the world's lazy create-on-`step_started` support and need no further world/backend changes.
|
|
87
87
|
|
|
88
88
|
## Inline up to N steps in parallel
|
|
89
89
|
|
|
@@ -92,7 +92,7 @@ Previously the owned-inline path ran **exactly one** step inline per suspension
|
|
|
92
92
|
The suspension handler now defers `step_created` for up to **`WORKFLOW_MAX_INLINE_STEPS` (default 3)** steps and returns them as `lazyInlineSteps`. The runtime runs that batch inline **in parallel** (`Promise.all`), each via its own lazy `step_started`, and queues only the steps beyond the cap.
|
|
93
93
|
|
|
94
94
|
- **Selection:** the first N uncreated steps, matching the previous single-step inline candidate. Steps beyond N keep their eager `step_created` and are queued exactly as before.
|
|
95
|
-
- **Result aggregation:** `retry` steps (whose `step_started` succeeded, so the step exists) are re-queued per-step as background steps with their own delay. `throttled` steps are different: a throttle rejects the lazy `step_started` on the create-claim, so the step was *never created* and has no recoverable input
|
|
95
|
+
- **Result aggregation:** `retry` steps (whose `step_started` succeeded, so the step exists) are re-queued per-step as background steps with their own delay. `throttled` steps are different: a throttle rejects the lazy `step_started` on the create-claim, so the step was *never created* and has no recoverable input. Re-queuing it as an input-less background step would make the world reject the bare `step_started` with "Step not found" and redeliver until it fails. Any throttle instead **defers redelivery of the orchestrator** (by the longest throttle backoff in the batch), which re-runs the throttled step inline *with its input* on replay. The runtime only loops back to replay in-process once every inline step has reached a terminal state.
|
|
96
96
|
- **Inline-delta fast path:** still used only for the single-step sequential case (`lazyInlineSteps.length === 1`). With more than one inline step each writes its own events, so a per-write delta would be partial; multi-step batches fall back to a normal incremental `events.list`.
|
|
97
97
|
- **Config:** `WORKFLOW_MAX_INLINE_STEPS` is clamped to 1..16. Setting it to `1` reproduces the previous single-inline-step behavior exactly (a useful kill-switch). Inline bodies run in parallel within one function invocation, so the cap also bounds per-handler memory/CPU fan-out.
|
|
98
98
|
|
|
@@ -100,28 +100,28 @@ The suspension handler now defers `step_created` for up to **`WORKFLOW_MAX_INLIN
|
|
|
100
100
|
|
|
101
101
|
Normally `executeStep` awaits `step_started` (the lazy create-claim round-trip) before running the body. Because the inline path already holds the step input locally, it doesn't actually need that round-trip to begin.
|
|
102
102
|
|
|
103
|
-
When `WORKFLOW_OPTIMISTIC_INLINE_START` is enabled (set it to `1`/`true
|
|
103
|
+
When `WORKFLOW_OPTIMISTIC_INLINE_START` is enabled (set it to `1`/`true`; it is **off by default**), an inline step fires `step_started` **without awaiting it** and starts running the body immediately against locally-synthesized state. A lazy step is always brand-new, so attempt is 1, there is no prior error, and `startedAt` is now. All values are known without the server. The in-flight `step_started` is reconciled immediately before the terminal write:
|
|
104
104
|
|
|
105
105
|
- **Lost the create-claim (409 / `EntityConflictError`)** → discard the body result and return `skipped`; the winning handler owns the terminal write.
|
|
106
106
|
- **Run gone / throttled / too-early** → discard the body result and surface `gone` / `throttled` / `retry`.
|
|
107
|
-
- **Transient (non-translatable) failure** → propagate it, so the queue redelivers
|
|
107
|
+
- **Transient (non-translatable) failure** → propagate it, so the queue redelivers exactly as the await path does today.
|
|
108
108
|
- **Success** → write `step_completed` / `step_failed` / `step_retrying` as usual. Awaiting `step_started` before the terminal write keeps the event log ordered (`created → started → completed`).
|
|
109
109
|
|
|
110
110
|
### Safety and the idempotency tradeoff
|
|
111
111
|
|
|
112
|
-
- **Exactly-one terminal write is preserved.** Optimistic start changes only *when the body runs*, never who writes the terminal event
|
|
112
|
+
- **Exactly-one terminal write is preserved.** Optimistic start changes only *when the body runs*, never who writes the terminal event. The terminal event is still gated by the lazy `step_started` create-claim, which is awaited before the terminal write. Losers return `skipped`.
|
|
113
113
|
- **Bounded to attempt 1.** Only brand-new (`!hasCreatedEvent`) steps are lazy; a retried step already has a `step_created`, so it takes the normal await-then-run path with the real attempt counter. Synthesizing `attempt = 1` locally is therefore always correct.
|
|
114
|
-
- **
|
|
114
|
+
- **Why wider double-execution is off by default.** Running the body before confirming ownership means two handlers racing for the same step's create-claim can *both* run the side effects before either wins (previously the loser 409'd on `step_created` and skipped before running anything). This is unsafe for non-idempotent steps. In particular, two concurrent runs of a step that writes to the **workflow stream** (e.g. an AI agent streaming tokens) can interleave and **corrupt the stream data**. The optimization therefore ships **disabled**; enable it (`WORKFLOW_OPTIMISTIC_INLINE_START=1`) only for deployments whose inline step bodies are idempotent and stream-safe.
|
|
115
115
|
|
|
116
116
|
## Queue messages: inline steps don't pay a round-trip
|
|
117
117
|
|
|
118
|
-
Inline steps that **complete** never enqueue a per-step flow-route message. When every step in an inline batch reaches a terminal state with no pending background ops, the runtime
|
|
118
|
+
Inline steps that **complete** never enqueue a per-step flow-route message. When every step in an inline batch reaches a terminal state with no pending background ops, the runtime continues its in-process loop and replays. A sequential chain (or a clean parallel fan-out) of inline steps therefore runs entirely within one invocation with **zero** queue messages. A workflow whose only work is three parallel inline steps issues no `queue()` calls.
|
|
119
119
|
|
|
120
120
|
The only flow-route messages produced around an inline batch are:
|
|
121
121
|
|
|
122
|
-
- **Pre-batch dispatch
|
|
123
|
-
- **`retry` results
|
|
124
|
-
- **`throttled` results
|
|
125
|
-
- **Pending background ops
|
|
122
|
+
- **Pre-batch dispatch:** The steps *beyond* the inline cap (and any pending wait/sleep continuation). Inline steps are explicitly excluded from this dispatch.
|
|
123
|
+
- **`retry` results:** One delayed message per retried step. A retry *is* the step becoming its own background invocation, so this is expected.
|
|
124
|
+
- **`throttled` results:** A single deferral of the orchestrator message (see above).
|
|
125
|
+
- **Pending background ops:** If any inline step left unflushed stream writes (e.g. output streams to blob storage), the loop breaks and enqueues **one** continuation (aggregated across the batch, not per-step) so `waitUntil` can flush before the next replay reads them.
|
|
126
126
|
|
|
127
127
|
In other words: completed inline steps cost no queue round-trips; only steps that genuinely run as their own background invocations create new flow-route messages.
|
package/docs/changelog/meta.json
CHANGED
|
@@ -1,22 +1,22 @@
|
|
|
1
1
|
---
|
|
2
2
|
title: Resilient hook resume
|
|
3
|
-
description: resumeHook() now tolerates transient event storage failures
|
|
3
|
+
description: resumeHook() now tolerates transient event storage failures when the queue is healthy instead of failing the resume.
|
|
4
4
|
---
|
|
5
5
|
|
|
6
6
|
# Resilient `resumeHook()`
|
|
7
7
|
|
|
8
8
|
## Motivation
|
|
9
9
|
|
|
10
|
-
`resumeHook()` used to write the `hook_received` event and dispatch the workflow queue message strictly one after the other, so every resume paid two sequential round trips and a transient event-storage failure failed the whole resume even when the queue was healthy. This change runs both writes **concurrently**
|
|
10
|
+
`resumeHook()` used to write the `hook_received` event and dispatch the workflow queue message strictly one after the other, so every resume paid two sequential round trips and a transient event-storage failure failed the whole resume even when the queue was healthy. This change runs both writes **concurrently** (cutting a round trip off resume latency) and, on the same path, brings `resumeHook()` to parity with [resilient `start()`](/docs/changelog/resilient-start): a transient event-write failure no longer fails the resume when the payload can still be delivered through the queue.
|
|
11
11
|
|
|
12
12
|
## Design
|
|
13
13
|
|
|
14
|
-
- On the fast path, `resumeHook()` writes the `hook_received` event and dispatches the workflow queue message **concurrently** (`Promise.allSettled`). The queue message carries a `hookInput` payload
|
|
14
|
+
- On the fast path, `resumeHook()` writes the `hook_received` event and dispatches the workflow queue message **concurrently** (`Promise.allSettled`). The queue message carries a `hookInput` payload: the dehydrated hook payload plus a client-minted `resumeId` idempotency key, the hook token, and a payload digest.
|
|
15
15
|
- A `(runId, resumeId)` dedup constraint keeps the two writers converging on **exactly one** `hook_received` event: whichever lands first wins, and the other is resolved server-side as success rather than a duplicate. The queue consumer idempotently re-ensures the event from `hookInput` before replay, so the resume is guaranteed even if the direct write never commits.
|
|
16
16
|
- Replay also deduplicates: `hook_received` events sharing a `resumeId` belong to the same resume attempt, and only the first in the event log is delivered to workflow code. Even if a redelivery materializes the event twice, the payload reaches the workflow exactly once.
|
|
17
|
-
- **Queue dispatch failure is fatal
|
|
17
|
+
- **Queue dispatch failure is fatal**: The run was not retriggered, so no consumer will re-ensure the event, and `resumeHook()` throws. A transient event-write failure (429/5xx, a transport error, or an expected `(runId, resumeId)` conflict with the consumer's own re-ensure) is swallowed because the queue delivery still guarantees the resume; a terminal run surfaces as `HookNotFoundError`, and any other event-write error is rethrown.
|
|
18
18
|
- `resumeHook()` returns `ResumedHook` (exported from `workflow/api`), which extends `Hook` with an optional `resilientResume` flag. The flag is `true` only when the direct write failed transiently and the resume was recovered through the queue; on the happy path and the sequential fallback it is absent.
|
|
19
19
|
|
|
20
20
|
## Compatibility
|
|
21
21
|
|
|
22
|
-
The parallel fast path is gated per resume: it activates only when both the target run's queue consumer and the live backend independently attest dedup support (re-checked on every resume, so rollout and rollback both degrade safely). Otherwise
|
|
22
|
+
The parallel fast path is gated per resume: it activates only when both the target run's queue consumer and the live backend independently attest dedup support (re-checked on every resume, so rollout and rollback both degrade safely). Otherwise (for oversized payloads, legacy runs, or with `WORKFLOW_DISABLE_LAZY_HOOK_RESUME=1`), `resumeHook()` falls back to the original sequential write-then-dispatch path. Because runs keep executing on the deployment they were created on, a resume targeting a run from an older deployment uses the sequential path.
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
---
|
|
2
2
|
title: Resilient run start
|
|
3
|
-
description:
|
|
3
|
+
description: Run start logic tolerates World storage unavailability when the queue is healthy and reduces run start latency.
|
|
4
4
|
---
|
|
5
5
|
|
|
6
6
|
# Resilient `start()`
|
|
@@ -9,18 +9,18 @@ description: Overhaul run start logic to tolerate world storage unavailability,
|
|
|
9
9
|
|
|
10
10
|
When `world` storage is unavailable but the queue is up, `start()` previously failed entirely because `world.events.create(run_created)` is called before `world.queue()`. This change decouples run creation from queue dispatch so that runs can still be accepted when storage is degraded.
|
|
11
11
|
|
|
12
|
-
|
|
12
|
+
The runtime also previously called `world.runs.get(runId)` before `run_started`, adding an extra round trip. Calling `run_started` directly removes that round trip and can return preloaded events in the response to skip the initial `events.list` call, reducing time to first byte (TTFB).
|
|
13
13
|
|
|
14
14
|
## Design
|
|
15
15
|
|
|
16
16
|
### `start()` changes
|
|
17
17
|
|
|
18
18
|
- `world.events.create` (run_created) and `world.queue` are now called **in parallel** via `Promise.allSettled`.
|
|
19
|
-
- If `events.create`
|
|
19
|
+
- If `events.create` returns a **429 or 5xx** error, the runtime logs a warning that run creation failed but the run was accepted. The runtime retries creation asynchronously when it processes the queue message. The returned `Run` instance is marked with `resilientStart = true`.
|
|
20
20
|
- If `events.create` errors with **409** (EntityConflictError), the run already exists (e.g., the queue handler's resilient start path created it first due to a cold-start race). This is treated as success.
|
|
21
|
-
- If `world.queue` fails, we still throw
|
|
21
|
+
- If `world.queue` fails, we still throw: the run truly failed and was not enqueued.
|
|
22
22
|
- The queue invocation now receives all the run inputs (`input`, `deploymentId`, `workflowName`, `specVersion`, `executionContext`) via `runInput` so the runtime can create the run later if needed.
|
|
23
|
-
- When the runtime re-enqueues itself, it does **not** pass these inputs
|
|
23
|
+
- When the runtime re-enqueues itself, it does **not** pass these inputs: only the first queue cycle carries them.
|
|
24
24
|
|
|
25
25
|
### `workflowEntrypoint` changes
|
|
26
26
|
|
|
@@ -29,7 +29,7 @@ Additionally, the runtime previously called `world.runs.get(runId)` before `run_
|
|
|
29
29
|
### `Run.returnValue` polling
|
|
30
30
|
|
|
31
31
|
- When `resilientStart` is true on the Run instance (run_created failed), the `pollReturnValue` loop retries on `WorkflowRunNotFoundError` up to 3 times (1s + 3s + 6s = 10s total) to give the queue time to deliver and the runtime to create the run via `run_started`.
|
|
32
|
-
- When `resilientStart` is false (normal path), 404 fails immediately
|
|
32
|
+
- When `resilientStart` is false (normal path), 404 fails immediately: no delay for the common case of a wrong run ID.
|
|
33
33
|
|
|
34
34
|
### World contract changes
|
|
35
35
|
|
|
@@ -40,15 +40,15 @@ Additionally, the runtime previously called `world.runs.get(runId)` before `run_
|
|
|
40
40
|
|
|
41
41
|
`Uint8Array` values (the serialized workflow input in `runInput`) don't survive plain JSON serialization. Each world uses a transport that preserves binary data:
|
|
42
42
|
|
|
43
|
-
- **world-vercel**: CBOR transport
|
|
44
|
-
- **world-local**: `TypedJsonTransport
|
|
45
|
-
- **world-postgres**:
|
|
43
|
+
- **world-vercel**: Uses Concise Binary Object Representation (CBOR) transport, which CBOR-encodes the entire queue payload into a `Buffer` and uses `BufferTransport` from `@vercel/queue`. `Uint8Array` survives natively.
|
|
44
|
+
- **world-local**: Uses `TypedJsonTransport`, which encodes `Uint8Array` as `{ __type: 'Uint8Array', data: '<base64>' }`.
|
|
45
|
+
- **world-postgres**: Uses inline typed JSON transport, the same tagged-envelope approach as world-local.
|
|
46
46
|
|
|
47
47
|
## Decisions
|
|
48
48
|
|
|
49
|
-
1. **Parallel not sequential**: We chose `Promise.allSettled` over sequential calls to minimize latency in the
|
|
49
|
+
1. **Parallel, not sequential**: We chose `Promise.allSettled` over sequential calls to minimize latency in the successful path.
|
|
50
50
|
|
|
51
|
-
2. **Already-running returns run without event**: When `run_started` encounters an already-running run, all worlds return `{ run }` with `event: undefined` (no `events` array) instead of throwing. The runtime detects this by checking for `result.event === undefined`. This avoids an extra `world.runs.get` round
|
|
51
|
+
2. **Already-running returns run without event**: When `run_started` encounters an already-running run, all worlds return `{ run }` with `event: undefined` (no `events` array) instead of throwing. The runtime detects this by checking for `result.event === undefined`. This avoids an extra `world.runs.get` round trip.
|
|
52
52
|
|
|
53
53
|
3. **Events in 200 response**: We only return events on the 200 path (first caller). On the already-running path, we fall back to the normal `events.list` call. This is correct because only on 200 can we be certain we know the full event history.
|
|
54
54
|
|
|
@@ -64,12 +64,12 @@ On Vercel, the parallel dispatch can cause the queue message to be processed bef
|
|
|
64
64
|
2. The original `run_created` arrives and gets 409 (EntityConflictError).
|
|
65
65
|
3. `start()` treats the 409 as success (the run exists).
|
|
66
66
|
|
|
67
|
-
The `resilientStart` flag is
|
|
67
|
+
The `resilientStart` flag is not set on the `Run` instance in this case (409 is not a retryable error), so `returnValue` fails immediately on 404.
|
|
68
68
|
|
|
69
69
|
### Atomicity of run entity creation
|
|
70
70
|
|
|
71
|
-
The normal `run_created` path and the resilient start path can race on creating the run entity. In `world-local`, both paths use `writeExclusive` (O_CREAT|O_EXCL)
|
|
71
|
+
The normal `run_created` path and the resilient start path can race on creating the run entity. In `world-local`, both paths use `writeExclusive` (O_CREAT|O_EXCL), atomic at the OS level, so exactly one writer wins and the other gets EEXIST. The normal path throws `EntityConflictError` on conflict (handled by `start()` as 409); the resilient start path re-reads the run from disk on conflict.
|
|
72
72
|
|
|
73
73
|
In `world-postgres`, the resilient start path uses `onConflictDoNothing` plus a re-read on conflict for the same effect, with the same outcome on either side of the race.
|
|
74
74
|
|
|
75
|
-
The narrow crash window in `world-postgres` between the run insert and the event insert is acceptable
|
|
75
|
+
The narrow crash window in `world-postgres` between the run insert and the event insert is acceptable: if the run insert succeeds but the event insert crashes, the run exists and `run_started` will still proceed normally (the event log will be missing a `run_created` entry, but the run itself is functional).
|