workflow 5.0.0-beta.43 → 5.0.0-beta.44
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +6 -4
- package/dist/internal/builtins.d.ts +4 -4
- package/dist/internal/builtins.js +6 -6
- package/docs/ai/chat-session-modeling.mdx +23 -24
- package/docs/ai/defining-tools.mdx +5 -6
- package/docs/ai/human-in-the-loop.mdx +11 -11
- package/docs/ai/index.mdx +20 -20
- package/docs/ai/message-queueing.mdx +6 -6
- package/docs/ai/meta.json +1 -0
- package/docs/ai/resumable-streams.mdx +28 -28
- package/docs/ai/sleep-and-delays.mdx +9 -9
- package/docs/ai/streaming-updates-from-tools.mdx +4 -4
- package/docs/api-reference/vitest/index.mdx +8 -8
- package/docs/api-reference/workflow/create-hook.mdx +15 -15
- package/docs/api-reference/workflow/create-webhook.mdx +15 -15
- package/docs/api-reference/workflow/define-hook.mdx +10 -10
- package/docs/api-reference/workflow/fatal-error.mdx +2 -2
- package/docs/api-reference/workflow/fetch.mdx +7 -7
- package/docs/api-reference/workflow/get-step-metadata.mdx +2 -2
- package/docs/api-reference/workflow/get-workflow-metadata.mdx +3 -3
- package/docs/api-reference/workflow/get-writable.mdx +7 -7
- package/docs/api-reference/workflow/index.mdx +1 -1
- package/docs/api-reference/workflow/retryable-error.mdx +1 -1
- package/docs/api-reference/workflow/set-attributes.mdx +2 -2
- package/docs/api-reference/workflow/sleep.mdx +3 -3
- package/docs/api-reference/workflow-ai/durable-agent.mdx +42 -42
- package/docs/api-reference/workflow-ai/index.mdx +3 -3
- package/docs/api-reference/workflow-ai/workflow-chat-transport.mdx +28 -28
- package/docs/api-reference/workflow-api/get-hook-by-token.mdx +11 -11
- package/docs/api-reference/workflow-api/get-run.mdx +10 -10
- package/docs/api-reference/workflow-api/index.mdx +2 -4
- package/docs/api-reference/workflow-api/resume-hook.mdx +14 -14
- package/docs/api-reference/workflow-api/resume-webhook.mdx +3 -3
- package/docs/api-reference/workflow-api/start.mdx +16 -15
- package/docs/api-reference/workflow-astro/workflow.mdx +3 -3
- package/docs/api-reference/workflow-errors/entity-conflict-error.mdx +4 -4
- package/docs/api-reference/workflow-errors/hook-conflict-error.mdx +4 -4
- package/docs/api-reference/workflow-errors/hook-not-found-error.mdx +8 -8
- package/docs/api-reference/workflow-errors/index.mdx +6 -6
- package/docs/api-reference/workflow-errors/precondition-failed-error.mdx +9 -9
- package/docs/api-reference/workflow-errors/run-expired-error.mdx +2 -2
- package/docs/api-reference/workflow-errors/run-not-supported-error.mdx +4 -4
- package/docs/api-reference/workflow-errors/step-not-registered-error.mdx +5 -5
- package/docs/api-reference/workflow-errors/throttle-error.mdx +2 -2
- package/docs/api-reference/workflow-errors/too-early-error.mdx +2 -2
- package/docs/api-reference/workflow-errors/workflow-error.mdx +4 -4
- package/docs/api-reference/workflow-errors/workflow-not-registered-error.mdx +5 -6
- package/docs/api-reference/workflow-errors/workflow-run-cancelled-error.mdx +6 -6
- package/docs/api-reference/workflow-errors/workflow-run-failed-error.mdx +5 -5
- package/docs/api-reference/workflow-errors/workflow-run-not-completed-error.mdx +4 -4
- package/docs/api-reference/workflow-errors/workflow-run-not-found-error.mdx +4 -4
- package/docs/api-reference/workflow-errors/workflow-runtime-error.mdx +2 -2
- package/docs/api-reference/workflow-errors/workflow-world-error.mdx +8 -8
- package/docs/api-reference/workflow-globals.mdx +12 -11
- package/docs/api-reference/workflow-nest/configure-workflow-controller.mdx +2 -2
- package/docs/api-reference/workflow-nest/nest-local-builder.mdx +4 -4
- package/docs/api-reference/workflow-nest/workflow-controller.mdx +1 -1
- package/docs/api-reference/workflow-nest/workflow-module.mdx +4 -4
- package/docs/api-reference/workflow-next/with-workflow.mdx +16 -16
- package/docs/api-reference/workflow-nitro/index.mdx +6 -6
- package/docs/api-reference/workflow-nuxt/index.mdx +4 -4
- package/docs/api-reference/workflow-observability/hydrate-data.mdx +5 -5
- package/docs/api-reference/workflow-observability/hydrate-resource-io.mdx +4 -4
- package/docs/api-reference/workflow-observability/index.mdx +6 -8
- package/docs/api-reference/workflow-observability/observability-revivers.mdx +2 -2
- package/docs/api-reference/workflow-observability/parse-class-name.mdx +3 -3
- package/docs/api-reference/workflow-observability/parse-step-name.mdx +4 -4
- package/docs/api-reference/workflow-observability/parse-workflow-name.mdx +4 -4
- package/docs/api-reference/workflow-runtime/create-world.mdx +8 -8
- package/docs/api-reference/workflow-runtime/get-world-handlers.mdx +6 -6
- package/docs/api-reference/workflow-runtime/get-world.mdx +4 -4
- package/docs/api-reference/workflow-runtime/health-check.mdx +1 -1
- package/docs/api-reference/workflow-runtime/index.mdx +2 -4
- package/docs/api-reference/workflow-runtime/set-world.mdx +15 -13
- package/docs/api-reference/workflow-runtime/workflow-entrypoint.mdx +6 -5
- package/docs/api-reference/workflow-runtime/world/analytics.mdx +9 -9
- package/docs/api-reference/workflow-runtime/world/index.mdx +5 -3
- package/docs/api-reference/workflow-runtime/world/queue.mdx +11 -11
- package/docs/api-reference/workflow-runtime/world/storage.mdx +62 -28
- package/docs/api-reference/workflow-runtime/world/streams.mdx +7 -7
- package/docs/api-reference/workflow-serde/index.mdx +1 -1
- package/docs/api-reference/workflow-serde/workflow-deserialize.mdx +2 -2
- package/docs/api-reference/workflow-serde/workflow-serialize.mdx +7 -7
- package/docs/api-reference/workflow-sveltekit/workflow-plugin.mdx +3 -3
- package/docs/api-reference/workflow-vite/workflow.mdx +5 -5
- package/docs/changelog/attributes-mvp.mdx +39 -39
- package/docs/changelog/batched-event-writes.mdx +12 -12
- package/docs/changelog/eager-processing.mdx +63 -63
- package/docs/changelog/index.mdx +3 -3
- package/docs/changelog/lazy-event-creation.md +27 -27
- package/docs/changelog/resilient-resume.mdx +5 -5
- package/docs/changelog/resilient-start.mdx +14 -14
- package/docs/changelog/step-message-ownership.mdx +47 -47
- package/docs/changelog/turbo-mode.md +20 -20
- package/docs/comparisons/index.mdx +13 -13
- package/docs/comparisons/workflow-sdk-vs-aws-agentcore.mdx +15 -15
- package/docs/comparisons/workflow-sdk-vs-aws-step-functions.mdx +12 -12
- package/docs/comparisons/workflow-sdk-vs-cloudflare-workflows.mdx +11 -11
- package/docs/comparisons/workflow-sdk-vs-inngest.mdx +19 -19
- package/docs/comparisons/workflow-sdk-vs-temporal.mdx +23 -23
- package/docs/comparisons/workflow-sdk-vs-trigger-dev.mdx +18 -17
- package/docs/configuration/build-and-diagnostics.mdx +5 -5
- package/docs/configuration/cli-and-web-ui.mdx +4 -4
- package/docs/configuration/runtime-tuning.mdx +86 -23
- package/docs/configuration/worlds.mdx +28 -14
- package/docs/cookbook/advanced/child-workflows.mdx +25 -25
- package/docs/cookbook/advanced/publishing-libraries.mdx +40 -40
- package/docs/cookbook/advanced/serializable-steps.mdx +21 -21
- package/docs/cookbook/advanced/upgrading-workflows.mdx +31 -31
- package/docs/cookbook/agent-patterns/agent-cancellation.mdx +20 -20
- package/docs/cookbook/agent-patterns/human-in-the-loop.mdx +22 -22
- package/docs/cookbook/common-patterns/batching.mdx +14 -14
- package/docs/cookbook/common-patterns/idempotency.mdx +9 -9
- package/docs/cookbook/common-patterns/rate-limiting.mdx +3 -3
- package/docs/cookbook/common-patterns/saga.mdx +19 -19
- package/docs/cookbook/common-patterns/scheduling.mdx +23 -23
- package/docs/cookbook/common-patterns/sequential-and-parallel.mdx +26 -26
- package/docs/cookbook/common-patterns/timeouts.mdx +23 -23
- package/docs/cookbook/common-patterns/webhooks.mdx +6 -6
- package/docs/cookbook/common-patterns/workflow-composition.mdx +19 -19
- package/docs/cookbook/index.mdx +22 -22
- package/docs/cookbook/integrations/ai-sdk.mdx +43 -41
- package/docs/cookbook/integrations/chat-sdk.mdx +34 -34
- package/docs/cookbook/integrations/sandbox.mdx +46 -46
- package/docs/deploying.mdx +15 -15
- package/docs/errors/abort-signal-timeout-in-workflow.mdx +12 -12
- package/docs/errors/corrupted-event-log.mdx +11 -11
- package/docs/errors/deployment-mismatch.mdx +14 -14
- package/docs/errors/fetch-in-workflow.mdx +8 -8
- package/docs/errors/hook-conflict.mdx +11 -11
- package/docs/errors/index.mdx +1 -1
- package/docs/errors/node-js-module-in-workflow.mdx +5 -5
- package/docs/errors/replay-divergence.mdx +2 -2
- package/docs/errors/runtime-decryption-failed.mdx +12 -12
- package/docs/errors/serialization-failed.mdx +40 -12
- package/docs/errors/start-invalid-workflow-function.mdx +5 -5
- package/docs/errors/step-executed-multiple-times.mdx +2 -2
- package/docs/errors/step-not-registered.mdx +5 -5
- package/docs/errors/timeout-in-workflow.mdx +8 -8
- package/docs/errors/webhook-invalid-respond-with-value.mdx +18 -18
- package/docs/errors/webhook-response-not-sent.mdx +16 -16
- package/docs/errors/workflow-not-registered.mdx +5 -5
- package/docs/foundations/cancellation.mdx +31 -31
- package/docs/foundations/errors-and-retries.mdx +42 -11
- package/docs/foundations/hooks.mdx +35 -35
- package/docs/foundations/idempotency.mdx +9 -9
- package/docs/foundations/serialization.mdx +21 -22
- package/docs/foundations/starting-workflows.mdx +36 -37
- package/docs/foundations/streaming.mdx +46 -41
- package/docs/foundations/versioning.mdx +3 -3
- package/docs/foundations/workflows-and-steps.mdx +9 -9
- package/docs/getting-started/astro.mdx +16 -16
- package/docs/getting-started/express.mdx +8 -8
- package/docs/getting-started/fastify.mdx +8 -8
- package/docs/getting-started/hono.mdx +8 -8
- package/docs/getting-started/nestjs.mdx +18 -17
- package/docs/getting-started/next.mdx +11 -11
- package/docs/getting-started/nitro.mdx +16 -16
- package/docs/getting-started/nuxt.mdx +8 -8
- package/docs/getting-started/python.mdx +4 -4
- package/docs/getting-started/react-router/v7.mdx +1 -1
- package/docs/getting-started/react-router/v8.mdx +1 -1
- package/docs/getting-started/sveltekit.mdx +14 -14
- package/docs/getting-started/tanstack-start.mdx +12 -12
- package/docs/getting-started/vite.mdx +8 -8
- package/docs/how-it-works/cancellation.mdx +62 -62
- package/docs/how-it-works/code-transform.mdx +66 -54
- package/docs/how-it-works/encryption.mdx +25 -21
- package/docs/how-it-works/event-sourcing.mdx +53 -35
- package/docs/how-it-works/framework-integrations.mdx +12 -12
- package/docs/how-it-works/understanding-directives.mdx +21 -21
- package/docs/internal/index.mdx +6 -6
- package/docs/internal/nitro-native-build.mdx +2 -2
- package/docs/internal/nitro-web-ui.mdx +4 -4
- package/docs/internal/serializable-abort-controller.mdx +7 -7
- package/docs/observability/attributes.mdx +3 -3
- package/docs/observability/index.mdx +14 -10
- package/docs/observability/tracing.mdx +10 -10
- package/docs/testing/index.mdx +33 -33
- package/docs/testing/server-based.mdx +10 -10
- package/docs/whats-new.mdx +185 -0
- package/package.json +12 -12
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
---
|
|
2
2
|
title: Inline step message ownership
|
|
3
|
-
description: Why inline step_started now records its owning queue message ID, why ownership is bounded by a lease, and why the alternatives
|
|
3
|
+
description: Why inline step_started now records its owning queue message ID, why ownership is bounded by a lease, and why the alternatives, from queue serialization to heartbeats, were rejected. Fixes duplicate inline step execution (issue #2780).
|
|
4
4
|
---
|
|
5
5
|
|
|
6
6
|
# Inline step message ownership
|
|
@@ -11,13 +11,13 @@ description: Why inline step_started now records its owning queue message ID, wh
|
|
|
11
11
|
> made, and why the alternatives were rejected. Kill switch:
|
|
12
12
|
> `WORKFLOW_INLINE_OWNERSHIP=0`; lease tuning:
|
|
13
13
|
> `WORKFLOW_INLINE_OWNERSHIP_LEASE_SECONDS` (see
|
|
14
|
-
> [Runtime tuning](/
|
|
14
|
+
> [Runtime tuning](/docs/configuration/runtime-tuning)).
|
|
15
15
|
|
|
16
16
|
## What this change does
|
|
17
17
|
|
|
18
18
|
The lazy `step_started` that creates an inline step now also records the queue message ID
|
|
19
19
|
of the invocation running its body (`eventData.ownerMessageId`). While that ownership is
|
|
20
|
-
active
|
|
20
|
+
active (until the step's first `step_retrying` or terminal event), a replay triggered by
|
|
21
21
|
anything other than the owning message does **not** requeue the step. It instead ensures a
|
|
22
22
|
*delayed backstop wake* exists, timed to the remainder of an ownership lease. Only a
|
|
23
23
|
handler processing the owning message (the original invocation, or the queue's redelivery
|
|
@@ -25,12 +25,12 @@ of it after a crash) may re-execute the step before that lease expires.
|
|
|
25
25
|
|
|
26
26
|
## The bug being fixed
|
|
27
27
|
|
|
28
|
-
An inline step has no queue message
|
|
28
|
+
An inline step has no queue message: that is the point of inline execution (it saves the
|
|
29
29
|
dispatch round-trip). But the runtime's crash-recovery rule queued every created,
|
|
30
30
|
non-terminal step **unconditionally** on each replay, relying on the queue's
|
|
31
31
|
`idempotencyKey = correlationId` to dedupe repeats. For inline steps there is no prior
|
|
32
|
-
message to dedupe against, so any wake that replayed the run mid-step
|
|
33
|
-
an elapsed wait continuation, a cancellation
|
|
32
|
+
message to dedupe against, so any wake that replayed the run mid-step (`hook_received`,
|
|
33
|
+
an elapsed wait continuation, a cancellation) enqueued a *first* message for that
|
|
34
34
|
correlation ID. Its consumer sent a bare `step_started` on the `running` step (which is
|
|
35
35
|
allowed: retries legitimately re-start non-terminal steps) and ran the body a second time,
|
|
36
36
|
concurrently with the original. One `step_completed` won the terminal write; every side
|
|
@@ -39,7 +39,7 @@ effect had already happened twice.
|
|
|
39
39
|
Note what is *not* broken: the unconditional requeue is correct for eager steps and is
|
|
40
40
|
itself the crash-recovery mechanism for a handler that wrote `step_created` and died
|
|
41
41
|
before enqueueing. Any fix must suppress the requeue *only* while a live invocation is
|
|
42
|
-
demonstrably running the body
|
|
42
|
+
demonstrably running the body, which turns the bug into a liveness problem.
|
|
43
43
|
|
|
44
44
|
## The core problem is liveness, and why we did not build a liveness mechanism
|
|
45
45
|
|
|
@@ -51,38 +51,38 @@ signal: a crashed function emits nothing, and the event log looks identical eith
|
|
|
51
51
|
The classic answers are heartbeats or a lock store: the executing invocation periodically
|
|
52
52
|
renews a claim, and recovery waits for the claim to go stale. We rejected building one:
|
|
53
53
|
|
|
54
|
-
- **It adds World API surface.** Every backend
|
|
55
|
-
Postgres/Turso/Redis worlds
|
|
54
|
+
- **It adds World API surface.** Every backend (Vercel, local filesystem, community
|
|
55
|
+
Postgres/Turso/Redis worlds) would have to implement a renewable-claim store with
|
|
56
56
|
expiry semantics. The event log is the World contract's one source of truth; a second,
|
|
57
57
|
mutable liveness store beside it is a large contract change for a race fix.
|
|
58
58
|
- **It adds steady-state write load.** Heartbeats cost a write per interval per in-flight
|
|
59
59
|
step, paid by every healthy run to detect the rare crashed one.
|
|
60
60
|
- **It does not remove the hard part.** A heartbeat still needs an expiry to survive a
|
|
61
|
-
crashed heartbeater
|
|
61
|
+
crashed heartbeater: that expiry *is* a lease. Any liveness design degenerates to
|
|
62
62
|
"claim + bounded staleness"; the machinery around it is overhead.
|
|
63
63
|
|
|
64
64
|
Instead, the design reuses a liveness signal the system already has: **the queue's
|
|
65
65
|
delivery state**. An un-acked queue message is precisely "work that a live invocation may
|
|
66
66
|
be processing, which will be redelivered if the processor died." Stamping the owning
|
|
67
67
|
message ID on the step makes the queue's own at-least-once machinery serve as the claim,
|
|
68
|
-
the crash detector, and the recovery driver
|
|
68
|
+
the crash detector, and the recovery driver, with zero new World surface and zero
|
|
69
69
|
steady-state writes beyond one field on an event we already write.
|
|
70
70
|
|
|
71
71
|
### Why the identity is the queue `messageId`
|
|
72
72
|
|
|
73
73
|
`createQueueHandler` already delivers `meta.messageId`, and one enqueued message keeps its
|
|
74
74
|
ID across redelivery attempts. That stability is exactly the property recovery needs: "a
|
|
75
|
-
delivery whose ID matches the stamp" means "the queue redelivered the work that crashed"
|
|
75
|
+
delivery whose ID matches the stamp" means "the queue redelivered the work that crashed":
|
|
76
76
|
permission to re-execute. The requirement is now documented in the
|
|
77
|
-
[Queue contract](/
|
|
77
|
+
[Queue contract](/docs/api-reference/workflow-runtime/world/queue); a World whose queue
|
|
78
78
|
mints fresh IDs per delivery degrades gracefully (the owner check never matches, so
|
|
79
|
-
crashed steps recover via the delayed backstop instead of immediately)
|
|
79
|
+
crashed steps recover via the delayed backstop instead of immediately): it never wedges
|
|
80
80
|
and never duplicates.
|
|
81
81
|
|
|
82
82
|
### Why ownership lives in the event log
|
|
83
83
|
|
|
84
84
|
Ownership state is derived per-replay from the step's events, not held in memory or in a
|
|
85
|
-
side store. Every replayer
|
|
85
|
+
side store. Every replayer (the owner's redelivery, a hook wake, the backstop) computes
|
|
86
86
|
the same answer from the same log, which is the workflow runtime's existing consistency
|
|
87
87
|
model. The rules are chosen so the log alone is sufficient:
|
|
88
88
|
|
|
@@ -90,12 +90,12 @@ model. The rules are chosen so the log alone is sufficient:
|
|
|
90
90
|
sets the owner; an unstamped bare start (a retry attempt driven by a queued step
|
|
91
91
|
message, or an older runtime) clears it. This is why owner recovery must *re-stamp* its
|
|
92
92
|
bare start: an unstamped recovery start would read as "unowned" to a later wake, which
|
|
93
|
-
would immediately requeue the step the owner is re-running
|
|
93
|
+
would immediately requeue the step the owner is re-running, reintroducing the bug on
|
|
94
94
|
the recovery path.
|
|
95
95
|
- **`step_retrying` lapses ownership permanently** for the correlation ID. From the first
|
|
96
96
|
retry on, the step is queue-owned: the retry handoff enqueues a real step message, and
|
|
97
97
|
the ordinary `idempotencyKey = correlationId` dedupe works again. Extending ownership
|
|
98
|
-
across retries was rejected deliberately
|
|
98
|
+
across retries was rejected deliberately: retry backoffs are delay-dominated (the
|
|
99
99
|
owning invocation would hold compute open doing nothing), each attempt would pay a
|
|
100
100
|
replay to re-derive state, and transferring ownership between attempts adds a state
|
|
101
101
|
machine where the queue-owned path already recovers correctly.
|
|
@@ -106,8 +106,8 @@ model. The rules are chosen so the log alone is sufficient:
|
|
|
106
106
|
## Why ownership requires a lease
|
|
107
107
|
|
|
108
108
|
Ownership cannot be unconditional. Note first what the lease is *not* for: an owner that
|
|
109
|
-
crash-loops through the SDK's delivery budget does not wedge the run even without one
|
|
110
|
-
|
|
109
|
+
crash-loops through the SDK's delivery budget does not wedge the run even without one.
|
|
110
|
+
The flow handler fails the run when it receives an over-budget delivery
|
|
111
111
|
(`metadata.attempt > maxQueueDeliveries`). The lease exists because that check, and owner
|
|
112
112
|
recovery itself, both depend on assumptions the World contract does not actually promise:
|
|
113
113
|
|
|
@@ -117,13 +117,13 @@ recovery itself, both depend on assumptions the World contract does not actually
|
|
|
117
117
|
community SQS world with a small `maxReceiveCount`), retention can expire it, an
|
|
118
118
|
operator can purge it. The run is then still `running`, the step stamped and
|
|
119
119
|
non-terminal, and no message exists. With unbounded ownership every future wake defers
|
|
120
|
-
to a ghost
|
|
120
|
+
to a ghost, a permanent wedge; with the lease, the already-armed backstop (or the
|
|
121
121
|
first wake after expiry) recovers the step.
|
|
122
|
-
- **Worlds with unstable message IDs
|
|
122
|
+
- **Worlds with unstable message IDs: there the lease is the *entire* recovery
|
|
123
123
|
mechanism.** If a queue mints a fresh ID per delivery, the owner check never matches,
|
|
124
124
|
including on the crashed owner's own redelivery. Unbounded ownership would defer
|
|
125
125
|
forever to a stamp no delivery can ever match, while each deferring replay acks its own
|
|
126
|
-
message
|
|
126
|
+
message: the run drains to zero messages while still `running`. The lease is what
|
|
127
127
|
makes the graceful-degradation claim in the Queue contract true.
|
|
128
128
|
- **Insurance on the ack invariant.** Correctness leans on "no path acks the owning
|
|
129
129
|
message while an owned step is non-terminal" (see the decision-table invariants). That
|
|
@@ -131,7 +131,7 @@ recovery itself, both depend on assumptions the World contract does not actually
|
|
|
131
131
|
queue implementation bug could violate it. The lease caps the cost of any such bug at
|
|
132
132
|
one bounded stall instead of a permanent wedge.
|
|
133
133
|
- **Failure granularity for poison steps.** Even on the well-behaved exhaustion path,
|
|
134
|
-
lease expiry lets a poison step execute and fail on the background path
|
|
134
|
+
lease expiry lets a poison step execute and fail on the background path, a
|
|
135
135
|
*step-level*, `catch`-able failure the workflow can handle. Unbounded ownership funnels
|
|
136
136
|
the same poison into run-level "exceeded max deliveries", which kills the whole run
|
|
137
137
|
uncatchably, and only after the full backed-off delivery budget.
|
|
@@ -149,12 +149,12 @@ live owner is protected from duplicates."
|
|
|
149
149
|
|
|
150
150
|
The upper clamp exists for clock skew: `lastStartedAt` is server-stamped while `now` is
|
|
151
151
|
the local clock, so a client running behind the server would otherwise compute a remainder
|
|
152
|
-
*longer* than the lease
|
|
152
|
+
*longer* than the lease, and above 900s, a `delaySeconds` that SQS-backed queues reject
|
|
153
153
|
outright.
|
|
154
154
|
|
|
155
155
|
### Why a fixed constant, and why 860 seconds
|
|
156
156
|
|
|
157
|
-
The correct lease is "longer than any invocation can possibly live"
|
|
157
|
+
The correct lease is "longer than any invocation can possibly live". Beyond that point
|
|
158
158
|
the owner is provably dead on platforms that kill invocations. Ideally we would derive it
|
|
159
159
|
from the workflow route's resolved `maxDuration`. **No such signal exists**: builders emit
|
|
160
160
|
`maxDuration: 'max'`, which the platform resolves per-plan at deploy time; there is no
|
|
@@ -164,18 +164,18 @@ lease was rejected because there is nothing to derive it from.
|
|
|
164
164
|
860s is justified by a platform rule rather than a measurement: durations above 800s
|
|
165
165
|
require explicit per-function numeric configuration, so `'max'` resolves to ≤ 800s for
|
|
166
166
|
any builder-emitted workflow route, and 860 dominates it with headroom. The constant is
|
|
167
|
-
|
|
167
|
+
configurable through an environment variable (`WORKFLOW_INLINE_OWNERSHIP_LEASE_SECONDS`, clamped from 1 to 900, where 900 is the
|
|
168
168
|
queue's maximum per-message delay, so a single delayed backstop message always suffices
|
|
169
169
|
and no delay chaining is needed). The code comment carries the 30-minute-`maxDuration`
|
|
170
170
|
beta caveat so the constant is revisited when the platform ceiling moves.
|
|
171
171
|
|
|
172
172
|
On worlds with **no** invocation kill bound (world-local's single process, self-hosted
|
|
173
|
-
deployments), no constant is a death proof
|
|
173
|
+
deployments), no constant is a death proof, which is why the in-process single-flight
|
|
174
174
|
below is a required layer, not an optimization.
|
|
175
175
|
|
|
176
|
-
## Why the non-owner action is a delayed backstop wake
|
|
176
|
+
## Why the non-owner action is a delayed backstop wake: not a skip, and not a step message
|
|
177
177
|
|
|
178
|
-
The naive non-owner behavior is to
|
|
178
|
+
The naive non-owner behavior is to *skip* the requeue. Rejected: a pure skip makes
|
|
179
179
|
lease expiry useless, because nothing is scheduled to *observe* the expiry. If the owner
|
|
180
180
|
dies and no external wake happens to arrive later, the run wedges. The escape hatch has to
|
|
181
181
|
be folded into the suppression itself.
|
|
@@ -184,7 +184,7 @@ So the non-owner enqueues a **plain run continuation** (no `stepId`) with `delay
|
|
|
184
184
|
leaseRemaining`. When it fires, it replays the run and re-enters the same dispatch
|
|
185
185
|
decision table, which handles every state the step can be in by then: terminal → nothing
|
|
186
186
|
pending; queue-owned after `step_retrying` → normal keyed dispatch; owner dead with lease
|
|
187
|
-
expired → immediate dispatch
|
|
187
|
+
expired → immediate dispatch, preserving *step-level* failure semantics for poison steps
|
|
188
188
|
(the step fails and the workflow's `catch` sees it, rather than the run dying on a
|
|
189
189
|
delivery-budget backstop); lease refreshed by owner recovery → re-arm for the new
|
|
190
190
|
remainder.
|
|
@@ -194,11 +194,11 @@ now encoded in `backstopIdempotencyKey`:
|
|
|
194
194
|
|
|
195
195
|
1. **The backstop must not be the step's own message.** The first implementation enqueued
|
|
196
196
|
the step message itself (keyed `correlationId`) with the lease delay. But the owner's
|
|
197
|
-
retry handoff enqueues the step under that *same* key with a ~1s backoff
|
|
197
|
+
retry handoff enqueues the step under that *same* key with a ~1s backoff, and the
|
|
198
198
|
pending backstop absorbed it, turning a 1-second retry into a full-lease stall. Caught
|
|
199
199
|
by the abort-mid-flight e2e wedging on every world-local lane.
|
|
200
200
|
2. **The backstop key must change when ownership is re-stamped.** Queues dedupe an
|
|
201
|
-
idempotency key for the original message's lifetime
|
|
201
|
+
idempotency key for the original message's lifetime, *including while a delivery is
|
|
202
202
|
in flight*. With a fixed `${correlationId}:backstop` key, a backstop firing during a
|
|
203
203
|
lease that owner recovery had refreshed could never publish its own replacement (the
|
|
204
204
|
re-arm deduped against the in-flight backstop itself and was dropped); if the
|
|
@@ -229,7 +229,7 @@ Two invariants keep the table sound:
|
|
|
229
229
|
full lease. All inline bodies are awaited before any ack path, a dev assertion
|
|
230
230
|
(`error`-level log) guards the ordering against refactors, and turbo's `reinvoke()`
|
|
231
231
|
paths are safe by construction: turbo requires delivery attempt 1, while owned-recovery
|
|
232
|
-
steps can only exist on redeliveries (attempt ≥ 2)
|
|
232
|
+
steps can only exist on redeliveries (attempt ≥ 2), since a previous delivery of the same
|
|
233
233
|
message must have stamped them. That mutual exclusion is documented at turbo's
|
|
234
234
|
engagement gate.
|
|
235
235
|
- **Owned recovery must not be silently orphaned by early returns.** The background-step
|
|
@@ -242,7 +242,7 @@ Two invariants keep the table sound:
|
|
|
242
242
|
|
|
243
243
|
The lease bounds *cross-instance* duplication only on platforms that kill invocations. On
|
|
244
244
|
world-local (one process, no kill bound) a delayed backstop can fire while the owning
|
|
245
|
-
execution is still mid-body *in the same process
|
|
245
|
+
execution is still mid-body *in the same process*, and on Fluid Compute, an owner
|
|
246
246
|
redelivery and a backstop can land on the same instance. A module-level map keyed
|
|
247
247
|
`runId:correlationId` absorbs both: the loser awaits the winner's settlement and then acks
|
|
248
248
|
**without executing**.
|
|
@@ -250,25 +250,25 @@ redelivery and a backstop can land on the same instance. A module-level map keye
|
|
|
250
250
|
The loser must not ack-and-skip early (before the winner settles): a crash after an early
|
|
251
251
|
ack would consume the loser's message while the winner's outcome is unknown, potentially
|
|
252
252
|
orphaning the step with no message left to drive it. Awaiting settlement first keeps the
|
|
253
|
-
at-least-once envelope intact
|
|
253
|
+
at-least-once envelope intact: if the loser's own invocation hits its deadline while
|
|
254
254
|
waiting, its message redelivers and re-checks, degrading gracefully to polling.
|
|
255
255
|
|
|
256
256
|
Cross-instance duplicates on multi-instance *self-hosted* worlds during steps longer than
|
|
257
|
-
the lease remain the documented residual (mitigate by raising the lease
|
|
257
|
+
the lease remain the documented residual (mitigate by raising the lease environment variable). This equals
|
|
258
258
|
the trade-off every lease-based system makes; eliminating it entirely requires the
|
|
259
259
|
heartbeat machinery rejected above.
|
|
260
260
|
|
|
261
261
|
## Alternatives considered and rejected
|
|
262
262
|
|
|
263
|
-
- **Flow-route queue concurrency = 1 (serialize all run messages)
|
|
263
|
+
- **Flow-route queue concurrency = 1 (serialize all run messages)**: Removes the
|
|
264
264
|
parallelism the wake mechanism exists for: `Promise.race(step, sleep)` works because
|
|
265
265
|
the wait continuation fires in a *separate* invocation while the inline step blocks
|
|
266
|
-
its handler
|
|
266
|
+
its handler. With one slot, the sleep could never win. It is also a queue-backend
|
|
267
267
|
feature the World contract does not guarantee, and it serializes unrelated work
|
|
268
268
|
(hooks, cancellations) behind long step bodies. Noted as a long-term option only if
|
|
269
269
|
ownership proves unmaintainable.
|
|
270
270
|
- **Inline-eligibility latch (never inline while hooks/waits are open).** The cheapest
|
|
271
|
-
hotfix
|
|
271
|
+
hotfix (the condition is already computed for turbo's latch), but it permanently
|
|
272
272
|
forfeits inline execution for exactly the workflows that use hooks, taxing every run to
|
|
273
273
|
prevent a race that needs an in-flight step to matter. And it is incomplete:
|
|
274
274
|
cancellation can wake *any* run mid-step, hooks or not.
|
|
@@ -278,7 +278,7 @@ heartbeat machinery rejected above.
|
|
|
278
278
|
`step_retrying`.
|
|
279
279
|
- **Heartbeat / lock-store liveness.** Rejected above: new World surface for every
|
|
280
280
|
backend, steady-state write amplification, and it still needs a lease to survive a
|
|
281
|
-
crashed heartbeater
|
|
281
|
+
crashed heartbeater. All cost, same bound.
|
|
282
282
|
- **Deriving the lease from the route's `maxDuration`.** Nothing to derive from: builders
|
|
283
283
|
emit `'max'`, resolved per-plan by the platform at deploy; no runtime or build-time API
|
|
284
284
|
exposes the resolved value.
|
|
@@ -300,7 +300,7 @@ heartbeat machinery rejected above.
|
|
|
300
300
|
replays see unowned steps → exactly today's behavior. Safe, but a pointless window to
|
|
301
301
|
ship into.
|
|
302
302
|
- **Version skew is a non-issue**: runs are pinned to their deployment, and old events
|
|
303
|
-
|
|
303
|
+
lack the field → unowned → current behavior. Nothing to migrate.
|
|
304
304
|
- **Kill switch**: `WORKFLOW_INLINE_OWNERSHIP=0` reverts dispatch to the unconditional
|
|
305
305
|
immediate requeue. Stamping continues (it is inert data), so the switch is purely a
|
|
306
306
|
dispatch-behavior toggle.
|
|
@@ -312,11 +312,11 @@ heartbeat machinery rejected above.
|
|
|
312
312
|
|
|
313
313
|
- **Redelivery-while-alive** (visibility lapse or heartbeat partition inside the queue):
|
|
314
314
|
the owner check passes on a redelivery racing the live owner → duplicate. This equals
|
|
315
|
-
the queue's at-least-once envelope
|
|
315
|
+
the queue's at-least-once envelope (the floor for any client-side design) and is
|
|
316
316
|
strictly rarer than the every-wake duplication being fixed. The in-process single-flight
|
|
317
317
|
absorbs the same-instance case.
|
|
318
318
|
- **Multi-instance self-hosted worlds with steps longer than the lease** (see
|
|
319
|
-
single-flight section): raise the lease
|
|
319
|
+
single-flight section): raise the lease environment variable.
|
|
320
320
|
- **A backstop per ownership epoch**: an owner crash-looping through its redelivery
|
|
321
321
|
budget arms up to one delayed wake per re-stamp. Bounded by the queue's delivery
|
|
322
322
|
budget; each fires as a cheap replay no-op if the step completed.
|
|
@@ -329,7 +329,7 @@ heartbeat machinery rejected above.
|
|
|
329
329
|
immediate requeue).
|
|
330
330
|
- Always-printed `warn` logs when owned recovery runs (a prior delivery died mid-body)
|
|
331
331
|
and when the single-flight absorbs a would-be duplicate (a burst of these means leases
|
|
332
|
-
are expiring under live executions
|
|
332
|
+
are expiring under live executions: raise the lease environment variable). Invariant violations log at
|
|
333
333
|
`error`. Backstop arming logs at `debug` (`DEBUG=workflow:runtime:*`), since it can
|
|
334
334
|
legitimately fire on every wake replay during a long inline step.
|
|
335
335
|
|
|
@@ -340,14 +340,14 @@ heartbeat machinery rejected above.
|
|
|
340
340
|
with `WORKFLOW_INLINE_OWNERSHIP=0` the marker fires twice.
|
|
341
341
|
- **Unit**: the ownership state machine (stamp → wake sees owner → retrying clears → bare
|
|
342
342
|
start clears → re-stamp restores), lease math including the clock-skew clamp, and the
|
|
343
|
-
backstop key's epoch behavior
|
|
343
|
+
backstop key's epoch behavior, including a regression test walking the full
|
|
344
344
|
owner-recovery re-arm sequence against a dedupe model matching world-local's in-flight
|
|
345
345
|
key retention; single-flight winner/loser semantics.
|
|
346
346
|
- **Wire**: schema round-trip tests on both sides, plus backend integration tests
|
|
347
347
|
asserting the field survives materialization and the lazy-input strip, and never leaks
|
|
348
348
|
into synthetic `step_created`.
|
|
349
|
-
- **E2E**:
|
|
350
|
-
shape #1
|
|
349
|
+
- **End-to-end (E2E)**: The full suite including the abort/cancellation lanes that caught backstop
|
|
350
|
+
shape #1: world-vercel prod lanes are mandatory for sign-off, since world-local's
|
|
351
351
|
synchronous single-process behavior masks distributed races.
|
|
352
352
|
|
|
353
353
|
## Open questions
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
---
|
|
2
2
|
title: Turbo mode (fast first invocation)
|
|
3
|
-
description: Fast-path the
|
|
3
|
+
description: Fast-path the first delivery of the first invocation by backgrounding run_started, skipping the initial event-log load, and forcing optimistic inline start. A no-op for everything else.
|
|
4
4
|
---
|
|
5
5
|
|
|
6
6
|
# Turbo mode
|
|
@@ -10,7 +10,7 @@ description: Fast-path the very first delivery of the first invocation — backg
|
|
|
10
10
|
The first invocation of a workflow run is where time-to-first-step matters most, yet it pays the most fixed network latency before any user code runs. Three round-trips sit on that critical path today:
|
|
11
11
|
|
|
12
12
|
1. **`run_started` is awaited.** The handler writes `run_started` and waits for it to return the run entity before doing anything else.
|
|
13
|
-
2. **The event log is loaded.** A full `events.list` runs before the first replay
|
|
13
|
+
2. **The event log is loaded.** A full `events.list` runs before the first replay, even though on the first delivery nothing has written any events yet.
|
|
14
14
|
3. **Optimistic inline start is off by default.** The [optimistic inline start](./lazy-event-creation#optimistic-inline-start-opt-in-off-by-default) optimization (running a step body before its `step_started` is confirmed) is off by default because under contention two handlers can both run a body and corrupt non-idempotent side effects.
|
|
15
15
|
|
|
16
16
|
Turbo mode removes all three costs **for the first delivery of the first invocation only**, where each is provably safe to remove, then gets out of the way. For every subsequent invocation it is a complete no-op.
|
|
@@ -19,69 +19,69 @@ Turbo mode removes all three costs **for the first delivery of the first invocat
|
|
|
19
19
|
|
|
20
20
|
When the handler detects the first delivery of the first invocation, it:
|
|
21
21
|
|
|
22
|
-
1. **Backgrounds `run_started`.** The event is written without awaiting; the run entity is synthesized locally from the queued run input (status `running`, `startedAt` now) so replay can begin immediately. The `run_started` round-trip overlaps replay instead of blocking it. This reuses the [resilient start](./resilient-start) contract
|
|
22
|
+
1. **Backgrounds `run_started`.** The event is written without awaiting; the run entity is synthesized locally from the queued run input (status `running`, `startedAt` now) so replay can begin immediately. The `run_started` round-trip overlaps replay instead of blocking it. This reuses the [resilient start](./resilient-start) contract: `run_started` carrying the run input creates the run on the fly (synthetic `run_created`) if it doesn't exist yet. Because turbo uses this `run_started` purely as a write barrier and never reads its response, it also asks the World to **skip the `run_started` event-log preload** (the list+resolve the World normally returns so a run can skip its first `events.list`). That preload would be wasted work here. Since the first `step_started` is chained on the `run_started` barrier, trimming the `run_started` request directly shortens the wait before the first durable `step_started` (and therefore time-to-second-step). A World that ignores the hint stays correct; the runtime falls back to `events.list` if it ever needs the log.
|
|
23
23
|
2. **Skips the initial event-log load.** Nothing has been written, so the first replay runs against an empty log. The second loop iteration does a normal incremental load once the first step's events exist.
|
|
24
24
|
3. **Forces optimistic inline start** for that invocation, independent of `WORKFLOW_OPTIMISTIC_INLINE_START`. The step body runs immediately against locally-synthesized state; only the `step_started` network write waits for the backgrounded `run_started`.
|
|
25
25
|
|
|
26
|
-
The
|
|
26
|
+
The first step body starts after the in-process replay, with `run_started` and `step_started` happening in the background around it and no `events.list` before it.
|
|
27
27
|
|
|
28
28
|
## Why this is safe (and where it stops)
|
|
29
29
|
|
|
30
30
|
### Detection
|
|
31
31
|
|
|
32
|
-
The first-invocation message is the only one that carries the queued **run input**, and the queue delivery **attempt is 1** (a redelivery is attempt ≥ 2). Together with "not a background-step invocation" and "not a divergence recovery", that uniquely identifies the first delivery of the first invocation
|
|
32
|
+
The first-invocation message is the only one that carries the queued **run input**, and the queue delivery **attempt is 1** (a redelivery is attempt ≥ 2). Together with "not a background-step invocation" and "not a divergence recovery", that uniquely identifies the first delivery of the first invocation without a new message field or world/backend change.
|
|
33
33
|
|
|
34
34
|
### The single-handler guarantee
|
|
35
35
|
|
|
36
|
-
Forcing optimistic start is unsafe *in general* because two handlers racing the same step's create-claim can both run the body before one wins. On the first delivery of the first invocation there is **no concurrent peer handler
|
|
36
|
+
Forcing optimistic start is unsafe *in general* because two handlers racing the same step's create-claim can both run the body before one wins. On the first delivery of the first invocation there is **no concurrent peer handler**. The run was created moments ago by `start()`, and only this one message is in flight. The body therefore runs exactly once, and forcing optimistic start is safe here even though the global flag is off.
|
|
37
37
|
|
|
38
38
|
### Turbo exits on the first hook or wait
|
|
39
39
|
|
|
40
|
-
That single-handler guarantee ends the moment the run creates a **hook** or **wait** (or writes attributes)
|
|
40
|
+
That single-handler guarantee ends the moment the run creates a **hook** or **wait** (or writes attributes) because those introduce later resume/parallel invocations that *can* race. Turbo stops forcing optimistic start as soon as a suspension creates any of them. The inline steps of that suspension fall back to the normal await-then-run path, and the rest of the run behaves exactly as it does today. A pure-step suspension (the common hot path) stays on the fast path.
|
|
41
41
|
|
|
42
42
|
### Write ordering is preserved
|
|
43
43
|
|
|
44
44
|
Because `run_started` is backgrounded, every event write is gated on a run-ready barrier so nothing is written before the run exists:
|
|
45
45
|
|
|
46
|
-
- The optimistic `step_started` is **chained** on the barrier
|
|
46
|
+
- The optimistic `step_started` is **chained** on the barrier. The body still runs immediately; only the network write waits.
|
|
47
47
|
- The suspension handler **awaits** the barrier before any eager write (`hook_created`, `wait_created`, overflow `step_created`). The pure inline hot path defers all its steps and writes nothing here, so it never blocks on the barrier.
|
|
48
48
|
- Terminal run writes (`run_completed` / `run_failed`) await the barrier too, so a workflow that finishes with no steps still orders its completion after `run_started`.
|
|
49
49
|
|
|
50
|
-
The event log therefore still reads `run_created → run_started → step_created → step_started → step_completed`. If the backgrounded `run_started` genuinely fails (e.g. the run was
|
|
50
|
+
The event log therefore still reads `run_created → run_started → step_created → step_started → step_completed`. If the backgrounded `run_started` genuinely fails (e.g. the run was canceled in the meantime), the chained writes surface the real error (`gone` / run-not-found) and the message redelivers as a normal, non-turbo attempt.
|
|
51
51
|
|
|
52
|
-
The barrier orders **event** writes. The forced-optimistic first step **body** runs immediately, so any side effects it performs *before* the terminal write
|
|
52
|
+
The barrier orders **event** writes. The forced-optimistic first step **body** runs immediately, so any side effects it performs *before* the terminal write (stream writes via `getWritable()` and the per-step ops flush) are **not** gated on the barrier and can reach the world before the backgrounded `run_started` lands (and are orphaned if it ultimately fails). This is the same exposure as optimistic inline start and is covered by the stream-safety caveat below; deployments whose first step writes to the workflow stream and require strict `run_created → run_started` ordering of stream data should set `WORKFLOW_TURBO=0`.
|
|
53
53
|
|
|
54
54
|
### A run cancelled before its first delivery still runs the first step body
|
|
55
55
|
|
|
56
|
-
The non-turbo path awaits `run_started` up front and, if the run was
|
|
56
|
+
The non-turbo path awaits `run_started` up front and, if the run was canceled or expired between `start()` and this delivery, returns before any workflow/step code runs. Turbo synthesizes `status: 'running'` and runs the first step body optimistically, so such a cancellation is only observed when the backgrounded `run_started` (and the barrier-chained `step_started`) rejects, *after* the body's side effects have executed (they are then discarded via reconciliation). For non-idempotent first steps this is the same "body runs before ownership is confirmed" tradeoff as optimistic inline start; `WORKFLOW_TURBO=0` restores the up-front skip.
|
|
57
57
|
|
|
58
58
|
### `workflowStartedAt` reflects the first delivery's clock
|
|
59
59
|
|
|
60
|
-
Replay matching
|
|
60
|
+
Replay matching (step/wait/hook correlation IDs, the VM seed, and the in-VM `Date.now()`) is derived from a replay-stable timestamp recovered from the run ID, so it does **not** depend on `startedAt` and is identical on every delivery. The one value that still tracks `startedAt` is the user-facing `getWorkflowMetadata().workflowStartedAt`: under turbo the first delivery synthesizes it from the local clock, while a later (non-turbo) delivery loads the server-canonical `startedAt`, so the two can differ by the start→first-delivery latency. Treat `workflowStartedAt` as an approximate, human-facing timestamp. **Do not** branch workflow control flow on it (e.g. `Date.now() - +workflowStartedAt > threshold`), since that can take different paths across deliveries and diverge on replay. For timing logic that must survive replay, use the in-VM `Date.now()` / `new Date()`, which is replay-stable.
|
|
61
61
|
|
|
62
62
|
### Attributes seeded at `start()` survive the skipped event load
|
|
63
63
|
|
|
64
|
-
`start({ attributes })` does **not** disable turbo, and it needs no synthetic event in the empty log. Seed attributes are folded into the `run_created` event's data (not separate `attr_set` events) and ride along in the queued run input, so the locally-synthesized run snapshot carries them
|
|
64
|
+
`start({ attributes })` does **not** disable turbo, and it needs no synthetic event in the empty log. Seed attributes are folded into the `run_created` event's data (not separate `attr_set` events) and ride along in the queued run input, so the locally-synthesized run snapshot carries them. Turbo loses nothing by skipping the initial `events.list`.
|
|
65
65
|
|
|
66
66
|
This is safe specifically because **attributes are write-only inside a workflow**: there is no in-workflow read API today, and `run_created` is consumed structurally during replay without inspecting its attributes. So an empty initial event log replays identically whether or not the run was seeded with attributes.
|
|
67
67
|
|
|
68
|
-
That safety is a standing invariant for any future change: if an in-workflow attribute *read* API is ever added, it MUST read from the run snapshot (which turbo populates from the run input) and **not** by replaying `run_created` / `attr_set` events. Reading from the event log would surface seed attributes as empty on the first turbo delivery only
|
|
68
|
+
That safety is a standing invariant for any future change: if an in-workflow attribute *read* API is ever added, it MUST read from the run snapshot (which turbo populates from the run input) and **not** by replaying `run_created` / `attr_set` events. Reading from the event log would surface seed attributes as empty on the first turbo delivery only, causing a turbo-exclusive divergence from the non-turbo path. `start()` cannot seed hooks or waits, so there is no start-seeded suspension state for the skipped load to miss.
|
|
69
69
|
|
|
70
70
|
## Configuration
|
|
71
71
|
|
|
72
|
-
Turbo mode is **on by default**. Set `WORKFLOW_TURBO=0` (or `false`) to disable it
|
|
72
|
+
Turbo mode is **on by default**. Set `WORKFLOW_TURBO=0` (or `false`) to disable it. Every invocation then takes the existing awaited path. This is a useful kill-switch for deployments whose first-step bodies are not idempotent and stream-safe (the same caveat as optimistic inline start), or for isolating behavior while debugging.
|
|
73
73
|
|
|
74
|
-
Turbo forces optimistic inline start on the first invocation regardless of `WORKFLOW_OPTIMISTIC_INLINE_START` (its single-handler guarantee removes the double-execution race that flag guards against). It does, however, **honor an explicit `WORKFLOW_OPTIMISTIC_INLINE_START=0`**: because forced optimistic start still runs the body before `step_started`/`run_started` is confirmed, an operator who has explicitly disabled optimistic start keeps the await-then-run path even under turbo (the rest of turbo
|
|
74
|
+
Turbo forces optimistic inline start on the first invocation regardless of `WORKFLOW_OPTIMISTIC_INLINE_START` (its single-handler guarantee removes the double-execution race that flag guards against). It does, however, **honor an explicit `WORKFLOW_OPTIMISTIC_INLINE_START=0`**: because forced optimistic start still runs the body before `step_started`/`run_started` is confirmed, an operator who has explicitly disabled optimistic start keeps the await-then-run path even under turbo (the rest of turbo, including backgrounded `run_started` and the skipped initial load, still applies). With the flag unset (the default), turbo forces it on.
|
|
75
75
|
|
|
76
|
-
Turbo mode is purely client-side and builds on the lazy/optimistic inline start support already shipped
|
|
76
|
+
Turbo mode is purely client-side and builds on the lazy/optimistic inline start support already shipped, so it requires no world or backend changes.
|
|
77
77
|
|
|
78
78
|
## Considered: running ahead of durable writes (not implemented)
|
|
79
79
|
|
|
80
|
-
Turbo overlaps the *start* round-trips with a step's body, but it still **awaits each `step_completed` before advancing** to the next step. We explored going further
|
|
80
|
+
Turbo overlaps the *start* round-trips with a step's body, but it still **awaits each `step_completed` before advancing** to the next step. We explored going further with "run-ahead": within a single invocation, execute the workflow forward across a sequential chain *without* awaiting each step's event writes, draining `step_started`/`step_completed` through a background FIFO queue and only blocking on a full drain before acking. A run of three sub-millisecond steps would then fire all the bodies back-to-back while the six event posts caught up in the background, turning per-step latency into `max(Σ body, Σ post)` instead of `Σ(body + post)`.
|
|
81
81
|
|
|
82
82
|
We decided **not** to ship it, for two reasons:
|
|
83
83
|
|
|
84
|
-
1. **Re-execution blast radius on failure.** Awaiting each completion means a crash re-runs essentially one in-flight step. Running ahead leaves many completions undrained at once, so a crash or `maxDuration` SIGTERM re-runs *all* of them on redelivery
|
|
85
|
-
2. **Divergent branches from non-durable results.** Advancing past a step before its result is durable lets the workflow commit to a forward path that a crash-and-redeliver can re-decide differently. A `Promise.race([B, C])` resolved by local timing can pick `B`, run `D(B)`, then crash before `step_completed_B` is durable
|
|
84
|
+
1. **Re-execution blast radius on failure.** Awaiting each completion means a crash re-runs essentially one in-flight step. Running ahead leaves many completions undrained at once, so a crash or `maxDuration` SIGTERM re-runs *all* of them on redelivery. This creates a much larger at-least-once blast radius, precisely on the latency-sensitive runs most likely to pack many steps into one invocation.
|
|
85
|
+
2. **Divergent branches from non-durable results.** Advancing past a step before its result is durable lets the workflow commit to a forward path that a crash-and-redeliver can re-decide differently. A `Promise.race([B, C])` resolved by local timing can pick `B`, run `D(B)`, then crash before `step_completed_B` is durable. The redelivery may re-resolve to `C`, so `D` executed against a winner the durable history never records. The same shape appears for a branch on a non-deterministic step output (`B(v1)` runs, crash, redelivery commits `B(v2)`). Idempotency doesn't cover these because `D(B)`/`D(C)` and `B(v1)`/`B(v2)` are *different* operations, not retries of one. A "run ahead only while at most one result is undurable" gate would contain the race case (a race needs ≥2 concurrent undurable steps) but not the non-deterministic-output case, and that residual hazard plus the re-execution blast radius outweighed the gain.
|
|
86
86
|
|
|
87
87
|
So turbo deliberately stops at forced-optimistic *start* and awaits each `step_completed` before moving on: re-execution after a crash stays deterministic (each step re-runs against the same durable inputs) and bounded (roughly one step, not the whole chain). The idea is recorded here in case a future change (e.g. a determinism signal on steps, or deterministic race resolution) makes run-ahead safe enough to revisit.
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
---
|
|
2
2
|
title: Comparisons
|
|
3
|
-
description: How the Workflow SDK compares to other durable execution, workflow, and AI-agent frameworks
|
|
3
|
+
description: 'How the Workflow SDK compares to other durable execution, workflow, and AI-agent frameworks: Temporal, Cloudflare Workflows, AWS Step Functions, AWS Bedrock AgentCore, Inngest, and trigger.dev.'
|
|
4
4
|
type: overview
|
|
5
5
|
summary: Side-by-side comparisons of the Workflow SDK against Temporal, Cloudflare Workflows, AWS Step Functions, AWS Bedrock AgentCore, Inngest, and trigger.dev.
|
|
6
6
|
related:
|
|
@@ -13,26 +13,26 @@ The Workflow SDK overlaps with several categories: durable execution engines, ba
|
|
|
13
13
|
|
|
14
14
|
## What makes the Workflow SDK different
|
|
15
15
|
|
|
16
|
-
- **It's an open-source SDK, not a hosted product.** Your workflows are plain TypeScript in your existing app. Run them on the managed [Vercel World](/worlds/vercel), or self-host on the [Postgres World](/worlds/postgres)
|
|
16
|
+
- **It's an open-source SDK, not a hosted product.** Your workflows are plain TypeScript in your existing app. Run them on the managed [Vercel World](/worlds/vercel), or self-host on the [Postgres World](/worlds/postgres). The [World abstraction](/worlds/building-a-world) lets you own and swap the storage, queue, and streaming layers independently.
|
|
17
17
|
- **Versioning is safe by default.** Runs are pinned to the immutable deployment that started them, so shipping new code never disturbs in-flight runs. Upgrading a run is explicit and opt-in. See [Versioning](/docs/comparisons/workflow-sdk-vs-temporal#versioning).
|
|
18
|
-
- **
|
|
18
|
+
- **Real-time durable streaming is built in.** Stream partial output, such as large language model (LLM) tokens and progress, to clients with [streams](/docs/foundations/streaming) that survive reconnects, cold starts, and replays. This durability supports chat and agent UIs.
|
|
19
19
|
- **First-class AI agents.** `WorkflowAgent` ships inside the [AI SDK](/docs/ai), turning an agent loop into a durable workflow with automatic step retries and human-in-the-loop pauses.
|
|
20
20
|
|
|
21
21
|
## Snapshot
|
|
22
22
|
|
|
23
23
|
<Callout type="info">
|
|
24
|
-
These comparisons are compiled from each product's public documentation and are **not** based on head-to-head benchmarks
|
|
24
|
+
These comparisons are compiled from each product's public documentation and are **not** based on head-to-head benchmarks: durable engines differ enough that a single number rarely compares cleanly. Treat them as directional and verify current pricing and limits against each vendor's docs.
|
|
25
25
|
</Callout>
|
|
26
26
|
|
|
27
27
|
| Tool | Category | Durability model | Open source / self-host | Language(s) |
|
|
28
28
|
| --- | --- | --- | --- | --- |
|
|
29
|
-
| **Workflow SDK** | Durable functions SDK | Event-log + deterministic replay | ✅ Apache-2.0
|
|
30
|
-
| [Temporal](/docs/comparisons/workflow-sdk-vs-temporal) | Durable execution platform | Event-sourced replay | ✅ MIT server
|
|
29
|
+
| **Workflow SDK** | Durable functions SDK | Event-log + deterministic replay | ✅ Apache-2.0: self-host (Postgres) or Vercel | TypeScript (Python beta) |
|
|
30
|
+
| [Temporal](/docs/comparisons/workflow-sdk-vs-temporal) | Durable execution platform | Event-sourced replay | ✅ MIT server: self-host or Temporal Cloud | Go, Java, TypeScript, Python, .NET, PHP, Ruby |
|
|
31
31
|
| [Cloudflare Workflows](/docs/comparisons/workflow-sdk-vs-cloudflare-workflows) | Durable execution engine | Step-result memoization + replay | ❌ Cloudflare-only | TypeScript (Python beta) |
|
|
32
32
|
| [AWS Step Functions](/docs/comparisons/workflow-sdk-vs-aws-step-functions) | Managed state-machine orchestrator | Declarative ASL state machine | ❌ AWS-only | ASL JSON (tasks: any language) |
|
|
33
33
|
| [AWS Bedrock AgentCore](/docs/comparisons/workflow-sdk-vs-aws-agentcore) | AI-agent hosting platform | Not durable execution (ephemeral sessions) | ❌ AWS-only | Python, Node.js |
|
|
34
|
-
| [Inngest](/docs/comparisons/workflow-sdk-vs-inngest) | Durable functions / event platform | Step-result memoization | ◑ SSPL
|
|
35
|
-
| [trigger.dev](/docs/comparisons/workflow-sdk-vs-trigger-dev) | Durable task platform | Process checkpoint/restore (CRIU) | ✅ Apache-2.0
|
|
34
|
+
| [Inngest](/docs/comparisons/workflow-sdk-vs-inngest) | Durable functions / event platform | Step-result memoization | ◑ SSPL: self-host (community/best-effort) or SaaS | TypeScript (Python/Go pre-1.0) |
|
|
35
|
+
| [trigger.dev](/docs/comparisons/workflow-sdk-vs-trigger-dev) | Durable task platform | Process checkpoint/restore (CRIU) | ✅ Apache-2.0: self-host or Cloud | TypeScript only |
|
|
36
36
|
|
|
37
37
|
## Deep dives
|
|
38
38
|
|
|
@@ -41,16 +41,16 @@ These comparisons are compiled from each product's public documentation and are
|
|
|
41
41
|
The mature, language-agnostic durable-execution platform. You run the workers; Workflow SDK runs in your app.
|
|
42
42
|
</Card>
|
|
43
43
|
<Card href="/docs/comparisons/workflow-sdk-vs-cloudflare-workflows" title="vs Cloudflare Workflows">
|
|
44
|
-
A durable engine on Workers
|
|
44
|
+
A durable engine on Workers and Durable Objects. Both replay; they handle versioning and encryption differently.
|
|
45
45
|
</Card>
|
|
46
46
|
<Card href="/docs/comparisons/workflow-sdk-vs-aws-step-functions" title="vs AWS Step Functions">
|
|
47
|
-
Declarative ASL JSON state machines
|
|
47
|
+
Declarative ASL JSON state machines compared with plain TypeScript control flow.
|
|
48
48
|
</Card>
|
|
49
49
|
<Card href="/docs/comparisons/workflow-sdk-vs-aws-agentcore" title="vs AWS Bedrock AgentCore">
|
|
50
|
-
An AI-agent hosting platform
|
|
50
|
+
An AI-agent hosting platform, not a durable-execution engine. Different axis.
|
|
51
51
|
</Card>
|
|
52
52
|
<Card href="/docs/comparisons/workflow-sdk-vs-inngest" title="vs Inngest">
|
|
53
|
-
Event-driven durable functions that run on your own
|
|
53
|
+
Event-driven durable functions that run on your own infrastructure over HTTP.
|
|
54
54
|
</Card>
|
|
55
55
|
<Card href="/docs/comparisons/workflow-sdk-vs-trigger-dev" title="vs trigger.dev">
|
|
56
56
|
A TypeScript task platform that achieves durability by snapshotting the process (CRIU).
|
|
@@ -58,7 +58,7 @@ These comparisons are compiled from each product's public documentation and are
|
|
|
58
58
|
</Cards>
|
|
59
59
|
|
|
60
60
|
<Callout type="info">
|
|
61
|
-
|
|
61
|
+
Each deep dive includes a concept-mapping section for moving an existing system. The Workflow SDK migration skill can translate code for you:
|
|
62
62
|
|
|
63
63
|
```bash
|
|
64
64
|
npx skills add https://github.com/vercel/workflow --skill migrating-to-workflow-sdk
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
---
|
|
2
2
|
title: Workflow SDK vs AWS Bedrock AgentCore
|
|
3
|
-
description: How the Workflow SDK compares to AWS Bedrock AgentCore
|
|
3
|
+
description: How the Workflow SDK compares to AWS Bedrock AgentCore, a durable-execution framework versus an AI-agent hosting platform. They solve different problems.
|
|
4
4
|
type: conceptual
|
|
5
5
|
summary: AgentCore hosts and operates AI agents in isolated microVMs but is not a durable-execution engine. The Workflow SDK provides durable orchestration and resumable streaming for agents.
|
|
6
6
|
prerequisites:
|
|
@@ -11,10 +11,10 @@ related:
|
|
|
11
11
|
- /docs/foundations/streaming
|
|
12
12
|
---
|
|
13
13
|
|
|
14
|
-
[Amazon Bedrock AgentCore](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/what-is-bedrock-agentcore.html) is AWS's platform for **hosting and operating AI agents
|
|
14
|
+
[Amazon Bedrock AgentCore](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/what-is-bedrock-agentcore.html) is AWS's platform for **hosting and operating AI agents**, with a secure microVM runtime and building blocks for Memory, tool Gateways, and Identity. AgentCore is not a durable-execution engine, so it addresses a different part of the AI agent stack.
|
|
15
15
|
|
|
16
16
|
<Callout type="info">
|
|
17
|
-
**These solve different problems.** AgentCore answers "where do I securely run and operate an agent on AWS?" The Workflow SDK answers "how do I make a multi-step, tool-calling agent loop durable, resumable, and streamable?" AWS itself pairs AgentCore with a durable layer (Step Functions, or Temporal) for resumability
|
|
17
|
+
**These solve different problems.** AgentCore answers "where do I securely run and operate an agent on AWS?" The Workflow SDK answers "how do I make a multi-step, tool-calling agent loop durable, resumable, and streamable?" AWS itself pairs AgentCore with a durable layer (Step Functions, or Temporal) for resumability. The Workflow SDK provides that durable layer natively, in your own app.
|
|
18
18
|
</Callout>
|
|
19
19
|
|
|
20
20
|
## At a glance
|
|
@@ -22,22 +22,22 @@ related:
|
|
|
22
22
|
| | Workflow SDK | AWS Bedrock AgentCore |
|
|
23
23
|
| --- | --- | --- |
|
|
24
24
|
| **Category** | Open-source durable-functions SDK | AI-agent hosting & operations platform |
|
|
25
|
-
| **Durable execution?** | ✅ Event-log replay; the agent loop resumes from its last checkpoint after a crash | ❌ Not built-in
|
|
25
|
+
| **Durable execution?** | ✅ Event-log replay; the agent loop resumes from its last checkpoint after a crash | ❌ Not built-in: sessions are ephemeral microVMs; durability is opt-in via Memory or a framework checkpointer |
|
|
26
26
|
| **What it gives you** | Durable orchestration, steps, hooks, streaming, observability | Runtime (microVM hosting), Harness (managed agent loop), Memory, Gateway (tools/MCP), Identity, Browser, Code Interpreter, Observability, Policy, Evaluations |
|
|
27
|
-
| **Languages** | TypeScript /
|
|
27
|
+
| **Languages** | TypeScript / JavaScript (Python beta) | Python-first for authoring; TypeScript or Python project scaffolding via the `@aws/agentcore` command-line interface (CLI); framework-agnostic (LangGraph, CrewAI, Strands, and others) |
|
|
28
28
|
| **Isolation** | VM-sandboxed workflow code + full-Node steps | Dedicated Firecracker microVM per session (strong hardware isolation) |
|
|
29
29
|
| **Max duration** | No limit ([Vercel World limits](https://vercel.com/docs/workflows/pricing)) | 8-hour hard cap per session |
|
|
30
|
-
| **AI streaming** | Native **durable, resumable** streaming (survives reconnect, cold start, replay) | Live SSE / WebSocket only
|
|
31
|
-
| **Versioning** | Runs pinned to immutable deployment | Immutable runtime versions + endpoints; in-flight sessions stay on their launch version (deployment/rollback only
|
|
30
|
+
| **AI streaming** | Native **durable, resumable** streaming (survives reconnect, cold start, replay) | Live server-sent events (SSE) / WebSocket only: no buffering or replay of missed tokens |
|
|
31
|
+
| **Versioning** | Runs pinned to immutable deployment | Immutable runtime versions + endpoints; in-flight sessions stay on their launch version (deployment/rollback only, no replay) |
|
|
32
32
|
| **Portability** | Apache-2.0; runs anywhere Node runs; World abstraction | AWS-only; agent *code* is portable, the operating platform is not |
|
|
33
|
-
| **Compliance** | Inherits your platform | HIPAA-eligible; SOC
|
|
33
|
+
| **Compliance** | Inherits your platform | HIPAA-eligible; SOC, PCI, and ISO not third-party certified; not FedRAMP-authorized |
|
|
34
34
|
| **Pricing** | SDK free; pay your platform | Per-module consumption (vCPU-hr / GB-hr, etc.) + model inference billed via Bedrock; no minimums |
|
|
35
35
|
|
|
36
|
-
**What the limits mean in practice
|
|
36
|
+
**What the limits mean in practice**: AgentCore's 8-hour session cap means an agent that waits on a human, a long-running job, or a slow external system can't span that wait in one session. Workflow SDK runs have [no duration cap](https://vercel.com/docs/workflows/pricing): they suspend durably at `sleep()` and hooks for hours or weeks.
|
|
37
37
|
|
|
38
38
|
## The core distinction: durability
|
|
39
39
|
|
|
40
|
-
AgentCore Runtime gives each session an isolated microVM with up to 8 hours of runtime, but the compute is **ephemeral
|
|
40
|
+
AgentCore Runtime gives each session an isolated microVM with up to 8 hours of runtime, but the compute is **ephemeral**: on a crash or stop, the next invocation gets a fresh microVM with no automatic replay of the agent loop. AWS's own guidance is to use AgentCore Memory or a framework checkpointer for state durability, and to layer a workflow engine (Step Functions or Temporal) on top when you need durable orchestration.
|
|
41
41
|
|
|
42
42
|
The Workflow SDK *is* that durable layer. With `WorkflowAgent` (in the [AI SDK](/docs/ai)), the agent loop becomes a durable workflow: each model call and tool execution is a checkpointed step, the run resumes mid-loop after a failure, and partial output streams to the client through [resumable streams](/docs/ai/resumable-streams) that survive disconnects and cold starts.
|
|
43
43
|
|
|
@@ -45,11 +45,11 @@ The Workflow SDK *is* that durable layer. With `WorkflowAgent` (in the [AI SDK](
|
|
|
45
45
|
|
|
46
46
|
AgentCore is purpose-built for operating agents on AWS, and brings things the Workflow SDK doesn't try to be:
|
|
47
47
|
|
|
48
|
-
- **MicroVM isolation per session
|
|
49
|
-
- **Managed agent infrastructure
|
|
50
|
-
- **Enterprise/AWS compliance breadth** and VPC/PrivateLink networking.
|
|
48
|
+
- **MicroVM isolation per session**: the strongest hardware isolation among the tools in this section.
|
|
49
|
+
- **Managed agent infrastructure**: Memory, a tool Gateway (turn APIs/Lambda/MCP servers into tools), and Identity (credential vaulting, OAuth) as first-class managed services.
|
|
50
|
+
- **Enterprise/AWS compliance breadth** and Virtual Private Cloud (VPC)/PrivateLink networking.
|
|
51
51
|
|
|
52
|
-
|
|
52
|
+
AgentCore fits applications that run agents inside AWS with managed memory, tools, and identity. The Workflow SDK fits **durable, resumable, streamable** agent execution in your own TypeScript app without an AWS dependency. You can also combine them by hosting on AgentCore and orchestrating durably with the Workflow SDK.
|
|
53
53
|
|
|
54
54
|
---
|
|
55
|
-
*Compiled from public documentation. AgentCore cold-start figures are community-sourced (no published
|
|
55
|
+
*Compiled from public documentation. AgentCore cold-start figures are community-sourced (no published service-level agreement). Verify against [the AgentCore docs](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/what-is-bedrock-agentcore.html). Not based on head-to-head benchmarks.*
|