workflow 5.0.0-beta.4 → 5.0.0-beta.40

package/docs/changelog/step-message-ownership.mdx

@@ -0,0 +1,360 @@
+---
+title: Inline step message ownership
+description: Why inline step_started now records its owning queue message ID, why ownership is bounded by a lease, and why the alternatives — from queue serialization to heartbeats — were rejected. Fixes duplicate inline step execution (issue #2780).
+---
+
+# Inline step message ownership
+
+> Fixes [#2780](https://github.com/vercel/workflow/issues/2780) (duplicate
+> inline step execution when a hook or wait wakes a run mid-step). This page
+> is the design defense for the change: what it does, why each choice was
+> made, and why the alternatives were rejected. Kill switch:
+> `WORKFLOW_INLINE_OWNERSHIP=0`; lease tuning:
+> `WORKFLOW_INLINE_OWNERSHIP_LEASE_SECONDS` (see
+> [Runtime tuning](/v5/docs/configuration/runtime-tuning)).
+
+## What this change does
+
+The lazy `step_started` that creates an inline step now also records the queue message ID
+of the invocation running its body (`eventData.ownerMessageId`). While that ownership is
+active — until the step's first `step_retrying` or terminal event — a replay triggered by
+anything other than the owning message does **not** requeue the step. It instead ensures a
+*delayed backstop wake* exists, timed to the remainder of an ownership lease. Only a
+handler processing the owning message (the original invocation, or the queue's redelivery
+of it after a crash) may re-execute the step before that lease expires.
+
+## The bug being fixed
+
+An inline step has no queue message — that is the point of inline execution (it saves the
+dispatch round-trip). But the runtime's crash-recovery rule queued every created,
+non-terminal step **unconditionally** on each replay, relying on the queue's
+`idempotencyKey = correlationId` to dedupe repeats. For inline steps there is no prior
+message to dedupe against, so any wake that replayed the run mid-step — `hook_received`,
+an elapsed wait continuation, a cancellation — enqueued a *first* message for that
+correlation ID. Its consumer sent a bare `step_started` on the `running` step (which is
+allowed: retries legitimately re-start non-terminal steps) and ran the body a second time,
+concurrently with the original. One `step_completed` won the terminal write; every side
+effect had already happened twice.
+
+Note what is *not* broken: the unconditional requeue is correct for eager steps and is
+itself the crash-recovery mechanism for a handler that wrote `step_created` and died
+before enqueueing. Any fix must suppress the requeue *only* while a live invocation is
+demonstrably running the body — which turns the bug into a liveness problem.
+
+## The core problem is liveness, and why we did not build a liveness mechanism
+
+The missing primitive is the ability to distinguish "this attempt is in flight in a live
+invocation" from "this attempt died with its process." Stateless compute offers no death
+signal: a crashed function emits nothing, and the event log looks identical either way
+(`step_started`, no terminal event).
+
+The classic answers are heartbeats or a lock store: the executing invocation periodically
+renews a claim, and recovery waits for the claim to go stale. We rejected building one:
+
+- **It adds World API surface.** Every backend — Vercel, local filesystem, community
+  Postgres/Turso/Redis worlds — would have to implement a renewable-claim store with
+  expiry semantics. The event log is the World contract's one source of truth; a second,
+  mutable liveness store beside it is a large contract change for a race fix.
+- **It adds steady-state write load.** Heartbeats cost a write per interval per in-flight
+  step, paid by every healthy run to detect the rare crashed one.
+- **It does not remove the hard part.** A heartbeat still needs an expiry to survive a
+  crashed heartbeater — that expiry *is* a lease. Any liveness design degenerates to
+  "claim + bounded staleness"; the machinery around it is overhead.
+
+Instead, the design reuses a liveness signal the system already has: **the queue's
+delivery state**. An un-acked queue message is precisely "work that a live invocation may
+be processing, which will be redelivered if the processor died." Stamping the owning
+message ID on the step makes the queue's own at-least-once machinery serve as the claim,
+the crash detector, and the recovery driver — with zero new World surface and zero
+steady-state writes beyond one field on an event we already write.
+
+### Why the identity is the queue `messageId`
+
+`createQueueHandler` already delivers `meta.messageId`, and one enqueued message keeps its
+ID across redelivery attempts. That stability is exactly the property recovery needs: "a
+delivery whose ID matches the stamp" means "the queue redelivered the work that crashed" —
+permission to re-execute. The requirement is now documented in the
+[Queue contract](/v5/docs/api-reference/workflow-runtime/world/queue); a World whose queue
+mints fresh IDs per delivery degrades gracefully (the owner check never matches, so
+crashed steps recover via the delayed backstop instead of immediately) — it never wedges
+and never duplicates.
+
+### Why ownership lives in the event log
+
+Ownership state is derived per-replay from the step's events, not held in memory or in a
+side store. Every replayer — the owner's redelivery, a hook wake, the backstop — computes
+the same answer from the same log, which is the workflow runtime's existing consistency
+model. The rules are chosen so the log alone is sufficient:
+
+- **Latest `step_started` wins.** A stamped start (inline execution, or owner recovery)
+  sets the owner; an unstamped bare start (a retry attempt driven by a queued step
+  message, or an older runtime) clears it. This is why owner recovery must *re-stamp* its
+  bare start: an unstamped recovery start would read as "unowned" to a later wake, which
+  would immediately requeue the step the owner is re-running — reintroducing the bug on
+  the recovery path.
+- **`step_retrying` lapses ownership permanently** for the correlation ID. From the first
+  retry on, the step is queue-owned: the retry handoff enqueues a real step message, and
+  the ordinary `idempotencyKey = correlationId` dedupe works again. Extending ownership
+  across retries was rejected deliberately — retry backoffs are delay-dominated (the
+  owning invocation would hold compute open doing nothing), each attempt would pay a
+  replay to re-derive state, and transferring ownership between attempts adds a state
+  machine where the queue-owned path already recovers correctly.
+- **Eager steps are untouched.** Their execution is owned by their own step message and
+  its idempotency dedupe; stamping the orchestrator's ID on them would claim work the
+  orchestrator is not performing.
+
+## Why ownership requires a lease
+
+Ownership cannot be unconditional. Note first what the lease is *not* for: an owner that
+crash-loops through the SDK's delivery budget does not wedge the run even without one —
+the flow handler fails the run when it receives an over-budget delivery
+(`metadata.attempt > maxQueueDeliveries`). The lease exists because that check, and owner
+recovery itself, both depend on assumptions the World contract does not actually promise:
+
+- **Owner-message loss the SDK never observes.** The exhaustion check only fires if the
+  queue *delivers* the over-budget attempt. At-least-once doesn't guarantee that: a
+  queue-side redrive policy can dead-letter the message below the SDK's budget (a
+  community SQS world with a small `maxReceiveCount`), retention can expire it, an
+  operator can purge it. The run is then still `running`, the step stamped and
+  non-terminal, and no message exists. With unbounded ownership every future wake defers
+  to a ghost — a permanent wedge; with the lease, the already-armed backstop (or the
+  first wake after expiry) recovers the step.
+- **Worlds with unstable message IDs — there the lease is the *entire* recovery
+  mechanism.** If a queue mints a fresh ID per delivery, the owner check never matches,
+  including on the crashed owner's own redelivery. Unbounded ownership would defer
+  forever to a stamp no delivery can ever match, while each deferring replay acks its own
+  message — the run drains to zero messages while still `running`. The lease is what
+  makes the graceful-degradation claim in the Queue contract true.
+- **Insurance on the ack invariant.** Correctness leans on "no path acks the owning
+  message while an owned step is non-terminal" (see the decision-table invariants). That
+  is an audited property of today's code, not of the contract; a future refactor or a
+  queue implementation bug could violate it. The lease caps the cost of any such bug at
+  one bounded stall instead of a permanent wedge.
+- **Failure granularity for poison steps.** Even on the well-behaved exhaustion path,
+  lease expiry lets a poison step execute and fail on the background path — a
+  *step-level*, `catch`-able failure the workflow can handle. Unbounded ownership funnels
+  the same poison into run-level "exceeded max deliveries", which kills the whole run
+  uncatchably, and only after the full backed-off delivery budget.
+
+In short: unbounded ownership is a bet that message-ID stability, queue delivery
+guarantees, and the ack invariant all hold, forever, on every World. The lease caps the
+cost of losing any of those bets at a single bounded stall.
+
+So ownership is honored only for a bounded window: `leaseRemaining = min(lease, max(0,
+lastStartedAt + lease − now))`, anchored at the latest `step_started`'s server-assigned
+timestamp. Within the window, non-owner replays defer (and arm the backstop); after it,
+dispatch falls back to the pre-existing immediate enqueue. The lease is the upper bound on
+"how long a dead owner can delay recovery," and equally the lower bound on "how long a
+live owner is protected from duplicates."
+
+The upper clamp exists for clock skew: `lastStartedAt` is server-stamped while `now` is
+the local clock, so a client running behind the server would otherwise compute a remainder
+*longer* than the lease — and above 900s, a `delaySeconds` that SQS-backed queues reject
+outright.
+
+### Why a fixed constant, and why 860 seconds
+
+The correct lease is "longer than any invocation can possibly live" — beyond that point
+the owner is provably dead on platforms that kill invocations. Ideally we would derive it
+from the workflow route's resolved `maxDuration`. **No such signal exists**: builders emit
+`maxDuration: 'max'`, which the platform resolves per-plan at deploy time; there is no
+environment variable, request-context deadline, or build-time value to read. Deriving the
+lease was rejected because there is nothing to derive it from.
+
+860s is justified by a platform rule rather than a measurement: durations above 800s
+require explicit per-function numeric configuration, so `'max'` resolves to ≤ 800s for
+any builder-emitted workflow route, and 860 dominates it with headroom. The constant is
+env-tunable (`WORKFLOW_INLINE_OWNERSHIP_LEASE_SECONDS`, clamped to 1..900 — 900 being the
+queue's maximum per-message delay, so a single delayed backstop message always suffices
+and no delay chaining is needed). The code comment carries the 30-minute-`maxDuration`
+beta caveat so the constant is revisited when the platform ceiling moves.
+
+On worlds with **no** invocation kill bound (world-local's single process, self-hosted
+deployments), no constant is a death proof — which is why the in-process single-flight
+below is a required layer, not an optimization.
+
+## Why the non-owner action is a delayed backstop wake — not a skip, and not a step message
+
+The naive non-owner behavior is to simply *skip* the requeue. Rejected: a pure skip makes
+lease expiry useless, because nothing is scheduled to *observe* the expiry. If the owner
+dies and no external wake happens to arrive later, the run wedges. The escape hatch has to
+be folded into the suppression itself.
+
+So the non-owner enqueues a **plain run continuation** (no `stepId`) with `delaySeconds =
+leaseRemaining`. When it fires, it replays the run and re-enters the same dispatch
+decision table, which handles every state the step can be in by then: terminal → nothing
+pending; queue-owned after `step_retrying` → normal keyed dispatch; owner dead with lease
+expired → immediate dispatch — preserving *step-level* failure semantics for poison steps
+(the step fails and the workflow's `catch` sees it, rather than the run dying on a
+delivery-budget backstop); lease refreshed by owner recovery → re-arm for the new
+remainder.
+
+Two shapes of this backstop were tried and rejected by hard evidence, and both lessons are
+now encoded in `backstopIdempotencyKey`:
+
+1. **The backstop must not be the step's own message.** The first implementation enqueued
+   the step message itself (keyed `correlationId`) with the lease delay. But the owner's
+   retry handoff enqueues the step under that *same* key with a ~1s backoff — and the
+   pending backstop absorbed it, turning a 1-second retry into a full-lease stall. Caught
+   by the abort-mid-flight e2e wedging on every world-local lane.
+2. **The backstop key must change when ownership is re-stamped.** Queues dedupe an
+   idempotency key for the original message's lifetime — *including while a delivery is
+   in flight*. With a fixed `${correlationId}:backstop` key, a backstop firing during a
+   lease that owner recovery had refreshed could never publish its own replacement (the
+   re-arm deduped against the in-flight backstop itself and was dropped); if the
+   recovered owner then died with its redelivery budget exhausted, no escape hatch
+   remained. Caught in review. The key is therefore scoped to the **ownership epoch**:
+   `${correlationId}:backstop:${lastStartedAt}`. Wakes within one epoch still dedupe to a
+   single pending backstop; each owner-recovery re-stamp opens a new epoch with a fresh
+   key; pending backstops stay bounded by the owning message's redelivery budget.
+
+## The dispatch decision table
+
+For each pending step in the replay's suspension set that is not designated for lazy
+inline execution:
+
+| Step state | Action |
+|---|---|
+| Uncreated (`!hasCreatedEvent`) | Unchanged: lazy-inline candidate or eager create+queue |
+| Created, ownership active, `owner === myMessageId` | **Execute in this invocation** (owned recovery, alongside the lazy-inline batch; input hydrates from the step entity; the bare `step_started` re-stamps) |
+| Created, ownership active, `owner !== myMessageId` | **Ensure backstop wake**: plain run continuation, `delaySeconds = leaseRemaining`, epoch-scoped idempotency key |
+| Created, ownership lapsed (`step_retrying` seen), never owned (eager / old events), lease expired, or kill-switched | Unchanged: immediate enqueue, `idempotencyKey = correlationId` |
+| Terminal | Unchanged: not in the pending set |
+
+Two invariants keep the table sound:
+
+- **No path acks the owning message while an owned step is non-terminal.** Ack means
+  handler return or `reinvoke()` (which acks and continues under a *new* message ID); an
+  acked owner can never be redelivered, so recovery would fall to the backstop for the
+  full lease. All inline bodies are awaited before any ack path, a dev assertion
+  (`error`-level log) guards the ordering against refactors, and turbo's `reinvoke()`
+  paths are safe by construction: turbo requires delivery attempt 1, while owned-recovery
+  steps can only exist on redeliveries (attempt ≥ 2) — a previous delivery of the same
+  message must have stamped them. That mutual exclusion is documented at turbo's
+  engagement gate.
+- **Owned recovery must not be silently orphaned by early returns.** The background-step
+  fast path used to return without replaying when other steps were still pending; it now
+  falls through to the main loop when one of those pending steps is owned by the arriving
+  message, so a redelivered owner actually performs its recovery instead of leaving it to
+  the backstop.
+
+## Why in-process single-flight is a required layer
+
+The lease bounds *cross-instance* duplication only on platforms that kill invocations. On
+world-local (one process, no kill bound) a delayed backstop can fire while the owning
+execution is still mid-body *in the same process* — and on Fluid compute, an owner
+redelivery and a backstop can land on the same instance. A module-level map keyed
+`runId:correlationId` absorbs both: the loser awaits the winner's settlement and then acks
+**without executing**.
+
+The loser must not ack-and-skip early (before the winner settles): a crash after an early
+ack would consume the loser's message while the winner's outcome is unknown, potentially
+orphaning the step with no message left to drive it. Awaiting settlement first keeps the
+at-least-once envelope intact — if the loser's own invocation hits its deadline while
+waiting, its message redelivers and re-checks, degrading gracefully to polling.
+
+Cross-instance duplicates on multi-instance *self-hosted* worlds during steps longer than
+the lease remain the documented residual (mitigate by raising the lease env). This equals
+the trade-off every lease-based system makes; eliminating it entirely requires the
+heartbeat machinery rejected above.
+
+## Alternatives considered and rejected
+
+- **Flow-route queue concurrency = 1 (serialize all run messages).** Kills the very
+  parallelism the wake mechanism exists for: `Promise.race(step, sleep)` works because
+  the wait continuation fires in a *separate* invocation while the inline step blocks
+  its handler — with one slot, the sleep could never win. It is also a queue-backend
+  feature the World contract does not guarantee, and it serializes unrelated work
+  (hooks, cancellations) behind long step bodies. Noted as a long-term option only if
+  ownership proves unmaintainable.
+- **Inline-eligibility latch (never inline while hooks/waits are open).** The cheapest
+  hotfix — the condition is already computed for turbo's latch — but it permanently
+  forfeits inline execution for exactly the workflows that use hooks, taxing every run to
+  prevent a race that needs an in-flight step to matter. And it is incomplete:
+  cancellation can wake *any* run mid-step, hooks or not.
+- **Inline retries / transferring ownership across attempts.** Rejected above: backoffs
+  are delay-dominated, attempts would pay replay costs to stay inline, and the
+  queue-owned retry path already recovers correctly. Ownership deliberately ends at
+  `step_retrying`.
+- **Heartbeat / lock-store liveness.** Rejected above: new World surface for every
+  backend, steady-state write amplification, and it still needs a lease to survive a
+  crashed heartbeater — all cost, same bound.
+- **Deriving the lease from the route's `maxDuration`.** Nothing to derive from: builders
+  emit `'max'`, resolved per-plan by the platform at deploy; no runtime or build-time API
+  exposes the resolved value.
+- **Fixing it in the backend.** The server sees the same event log and has the same
+  liveness blind spot; it would need its own claim mechanism, and world-local plus every
+  community World would remain broken. The dispatch semantics live in the runtime, so the
+  fix does too.
+
+## Wire format, compatibility, and rollout
+
+- `ownerMessageId` is an optional field on the `step_started` eventData schema
+  (`@workflow/world`). In `@workflow/world-vercel` it rides the v4 frame meta; the
+  compile-time wire guard (`assertEventDataWireContractExhaustive`) fails the build if a
+  schema field is not explicitly routed, which is what forces the split/merge sites to be
+  handled. The backend persists it verbatim on the event row (the lazy-input strip on
+  `step_started` leaves it untouched, and the synthetic `step_created` never carries it)
+  and re-emits it on event lists.
+- **Deploy order: backend before SDK.** An older backend silently drops the meta field →
+  replays see unowned steps → exactly today's behavior. Safe, but a pointless window to
+  ship into.
+- **Version skew is a non-issue**: runs are pinned to their deployment, and old events
+  simply lack the field → unowned → current behavior. Nothing to migrate.
+- **Kill switch**: `WORKFLOW_INLINE_OWNERSHIP=0` reverts dispatch to the unconditional
+  immediate requeue. Stamping continues (it is inert data), so the switch is purely a
+  dispatch-behavior toggle.
+- **Degraded modes are all "today's behavior", never worse**: unstable message IDs (owner
+  check never matches → backstop-lease recovery), missing/unusable event timestamps
+  (lease remaining 0 → immediate enqueue), old backend (field dropped → unowned).
+
+## Accepted residual risks
+
+- **Redelivery-while-alive** (visibility lapse or heartbeat partition inside the queue):
+  the owner check passes on a redelivery racing the live owner → duplicate. This equals
+  the queue's at-least-once envelope — the floor for any client-side design — and is
+  strictly rarer than the every-wake duplication being fixed. The in-process single-flight
+  absorbs the same-instance case.
+- **Multi-instance self-hosted worlds with steps longer than the lease** (see
+  single-flight section): raise the lease env.
+- **A backstop per ownership epoch**: an owner crash-looping through its redelivery
+  budget arms up to one delayed wake per re-stamp. Bounded by the queue's delivery
+  budget; each fires as a cheap replay no-op if the step completed.
+
+## Observability
+
+- Span attributes on `workflow.execute`:
+  `workflow.inline_ownership.owned_recovery_steps` (crash recovery re-executed owned
+  steps) and `workflow.inline_ownership.backstop_wakes_armed` (a replay suppressed an
+  immediate requeue).
+- Always-printed `warn` logs when owned recovery runs (a prior delivery died mid-body)
+  and when the single-flight absorbs a would-be duplicate (a burst of these means leases
+  are expiring under live executions — raise the lease env). Invariant violations log at
+  `error`. Backstop arming logs at `debug` (`DEBUG=workflow:runtime:*`), since it can
+  legitimately fire on every wake replay during a long inline step.
+
+## How the design is protected by tests
+
+- The **#2780 repro** (`workbench/vitest/test/inline-step-ownership.test.ts`): hook
+  resume mid-inline-step → side-effect marker fires exactly once. Verified bidirectional:
+  with `WORKFLOW_INLINE_OWNERSHIP=0` the marker fires twice.
+- **Unit**: the ownership state machine (stamp → wake sees owner → retrying clears → bare
+  start clears → re-stamp restores), lease math including the clock-skew clamp, and the
+  backstop key's epoch behavior — including a regression test walking the full
+  owner-recovery re-arm sequence against a dedupe model matching world-local's in-flight
+  key retention; single-flight winner/loser semantics.
+- **Wire**: schema round-trip tests on both sides, plus backend integration tests
+  asserting the field survives materialization and the lazy-input strip, and never leaks
+  into synthetic `step_created`.
+- **E2E**: the full suite including the abort/cancellation lanes that caught backstop
+  shape #1 — world-vercel prod lanes are mandatory for sign-off, since world-local's
+  synchronous single-process behavior masks distributed races.
+
+## Open questions
+
+1. Confirm platform behavior: `'max'` resolves ≤ 800s even for accounts in the 30-minute
+   beta, and users cannot raise a generated workflow route's `maxDuration` via
+   `vercel.json` `functions` config without the builder seeing it (if they can, the
+   builder should detect the override and fail or bake it into the lease).
+2. Does world-postgres's handler meta deliver a stable ID across retries? If not:
+   document degraded (backstop-lease) mode for community worlds, don't block on it.

package/docs/changelog/turbo-mode.md

@@ -0,0 +1,87 @@
+---
+title: Turbo mode (fast first invocation)
+description: Fast-path the very first delivery of the first invocation — background run_started, skip the initial event-log load, and force optimistic inline start — so a run blazes through its first steps. A no-op for everything else.
+---
+
+# Turbo mode
+
+## Motivation
+
+The first invocation of a workflow run is where time-to-first-step matters most, yet it pays the most fixed network latency before any user code runs. Three round-trips sit on that critical path today:
+
+1. **`run_started` is awaited.** The handler writes `run_started` and waits for it to return the run entity before doing anything else.
+2. **The event log is loaded.** A full `events.list` runs before the first replay — even though on the very first delivery nothing has written any events yet.
+3. **Optimistic inline start is off by default.** The [optimistic inline start](./lazy-event-creation#optimistic-inline-start-opt-in-off-by-default) optimization (running a step body before its `step_started` is confirmed) is off by default because under contention two handlers can both run a body and corrupt non-idempotent side effects.
+
+Turbo mode removes all three costs **for the first delivery of the first invocation only**, where each is provably safe to remove, then gets out of the way. For every subsequent invocation it is a complete no-op.
+
+## What turbo mode does
+
+When the handler detects the first delivery of the first invocation, it:
+
+1. **Backgrounds `run_started`.** The event is written without awaiting; the run entity is synthesized locally from the queued run input (status `running`, `startedAt` now) so replay can begin immediately. The `run_started` round-trip overlaps replay instead of blocking it. This reuses the [resilient start](./resilient-start) contract — `run_started` carrying the run input creates the run on the fly (synthetic `run_created`) if it doesn't exist yet. Because turbo uses this `run_started` purely as a write barrier and never reads its response, it also asks the World to **skip the `run_started` event-log preload** (the list+resolve the World normally returns so a run can skip its first `events.list`). That preload would be wasted work here — and, since the first `step_started` is chained on the `run_started` barrier, trimming the `run_started` request directly shortens the wait before the first durable `step_started` (and therefore time-to-second-step). A World that ignores the hint stays correct; the runtime simply falls back to `events.list` if it ever needs the log.
+2. **Skips the initial event-log load.** Nothing has been written, so the first replay runs against an empty log. The second loop iteration does a normal incremental load once the first step's events exist.
+3. **Forces optimistic inline start** for that invocation, independent of `WORKFLOW_OPTIMISTIC_INLINE_START`. The step body runs immediately against locally-synthesized state; only the `step_started` network write waits for the backgrounded `run_started`.
+
+The net effect: the first step body starts after just the in-process replay, with `run_started` and `step_started` happening in the background around it, and no `events.list` before it.
+
+## Why this is safe (and where it stops)
+
+### Detection
+
+The first-invocation message is the only one that carries the queued **run input**, and the queue delivery **attempt is 1** (a redelivery is attempt ≥ 2). Together with "not a background-step invocation" and "not a divergence recovery", that uniquely identifies the first delivery of the first invocation — with no new message field and no world/backend change.
+
+### The single-handler guarantee
+
+Forcing optimistic start is unsafe *in general* because two handlers racing the same step's create-claim can both run the body before one wins. On the first delivery of the first invocation there is **no concurrent peer handler** — the run was created moments ago by `start()` and only this one message is in flight. So the body runs exactly once, and forcing optimistic start is safe here even though the global flag is off.
+
+### Turbo exits on the first hook or wait
+
+That single-handler guarantee ends the moment the run creates a **hook** or **wait** (or writes attributes): those introduce later resume/parallel invocations that *can* race. So turbo stops forcing optimistic start as soon as a suspension creates any of them — the inline steps of that suspension fall back to the normal await-then-run path, and the rest of the run behaves exactly as it does today. A pure-step suspension (the common hot path) stays on the fast path.
+
+### Write ordering is preserved
+
+Because `run_started` is backgrounded, every event write is gated on a run-ready barrier so nothing is written before the run exists:
+
+- The optimistic `step_started` is **chained** on the barrier — the body still runs immediately, only the network write waits.
+- The suspension handler **awaits** the barrier before any eager write (`hook_created`, `wait_created`, overflow `step_created`). The pure inline hot path defers all its steps and writes nothing here, so it never blocks on the barrier.
+- Terminal run writes (`run_completed` / `run_failed`) await the barrier too, so a workflow that finishes with no steps still orders its completion after `run_started`.
+
+The event log therefore still reads `run_created → run_started → step_created → step_started → step_completed`. If the backgrounded `run_started` genuinely fails (e.g. the run was cancelled in the meantime), the chained writes surface the real error (`gone` / run-not-found) and the message redelivers as a normal, non-turbo attempt.
+
+The barrier orders **event** writes. The forced-optimistic first step **body** runs immediately, so any side effects it performs *before* the terminal write — stream writes via `getWritable()` and the per-step ops flush — are **not** gated on the barrier and can reach the world before the backgrounded `run_started` lands (and are orphaned if it ultimately fails). This is the same exposure as optimistic inline start and is covered by the stream-safety caveat below; deployments whose first step writes to the workflow stream and require strict `run_created → run_started` ordering of stream data should set `WORKFLOW_TURBO=0`.
+
+### A run cancelled before its first delivery still runs the first step body
+
+The non-turbo path awaits `run_started` up front and, if the run was cancelled or expired between `start()` and this delivery, returns before any workflow/step code runs. Turbo synthesizes `status: 'running'` and runs the first step body optimistically, so such a cancellation is only observed when the backgrounded `run_started` (and the barrier-chained `step_started`) rejects — *after* the body's side effects have executed (they are then discarded via reconciliation). For non-idempotent first steps this is the same "body runs before ownership is confirmed" tradeoff as optimistic inline start; `WORKFLOW_TURBO=0` restores the up-front skip.
+
+### `workflowStartedAt` reflects the first delivery's clock
+
+Replay matching — step/wait/hook correlation IDs, the VM seed, and the in-VM `Date.now()` — is derived from a replay-stable timestamp recovered from the run ID, so it does **not** depend on `startedAt` and is identical on every delivery. The one value that still tracks `startedAt` is the user-facing `getWorkflowMetadata().workflowStartedAt`: under turbo the first delivery synthesizes it from the local clock, while a later (non-turbo) delivery loads the server-canonical `startedAt`, so the two can differ by the start→first-delivery latency. Treat `workflowStartedAt` as an approximate, human-facing timestamp — do **not** branch workflow control flow on it (e.g. `Date.now() - +workflowStartedAt > threshold`), since that can take different paths across deliveries and diverge on replay. For timing logic that must survive replay, use the in-VM `Date.now()` / `new Date()`, which is replay-stable.
+
+### Attributes seeded at `start()` survive the skipped event load
+
+`start({ attributes })` does **not** disable turbo, and it needs no synthetic event in the empty log. Seed attributes are folded into the `run_created` event's data (not separate `attr_set` events) and ride along in the queued run input, so the locally-synthesized run snapshot carries them — turbo skipping the initial `events.list` loses nothing.
+
+This is safe specifically because **attributes are write-only inside a workflow**: there is no in-workflow read API today, and `run_created` is consumed structurally during replay without inspecting its attributes. So an empty initial event log replays identically whether or not the run was seeded with attributes.
+
+That safety is a standing invariant for any future change: if an in-workflow attribute *read* API is ever added, it MUST read from the run snapshot (which turbo populates from the run input) and **not** by replaying `run_created` / `attr_set` events. Reading from the event log would surface seed attributes as empty on the first turbo delivery only — a turbo-exclusive divergence from the non-turbo path. `start()` cannot seed hooks or waits, so there is no start-seeded suspension state for the skipped load to miss.
+
+## Configuration
+
+Turbo mode is **on by default**. Set `WORKFLOW_TURBO=0` (or `false`) to disable it — every invocation then takes the existing awaited path. This is a useful kill-switch for deployments whose first-step bodies are not idempotent and stream-safe (the same caveat as optimistic inline start), or for isolating behavior while debugging.
+
+Turbo forces optimistic inline start on the first invocation regardless of `WORKFLOW_OPTIMISTIC_INLINE_START` (its single-handler guarantee removes the double-execution race that flag guards against). It does, however, **honor an explicit `WORKFLOW_OPTIMISTIC_INLINE_START=0`**: because forced optimistic start still runs the body before `step_started`/`run_started` is confirmed, an operator who has explicitly disabled optimistic start keeps the await-then-run path even under turbo (the rest of turbo — backgrounded `run_started`, skipped initial load — still applies). With the flag unset (the default), turbo forces it on.
+
+Turbo mode is purely client-side and builds on the lazy/optimistic inline start support already shipped — it requires no world or backend changes.
+
+## Considered: running ahead of durable writes (not implemented)
+
+Turbo overlaps the *start* round-trips with a step's body, but it still **awaits each `step_completed` before advancing** to the next step. We explored going further — "run-ahead": within a single invocation, execute the workflow forward across a sequential chain *without* awaiting each step's event writes, draining `step_started`/`step_completed` through a background FIFO queue and only blocking on a full drain before acking. A run of three sub-millisecond steps would then fire all the bodies back-to-back while the six event posts caught up in the background, turning per-step latency into `max(Σ body, Σ post)` instead of `Σ(body + post)`.
+
+We decided **not** to ship it, for two reasons:
+
+1. **Re-execution blast radius on failure.** Awaiting each completion means a crash re-runs essentially one in-flight step. Running ahead leaves many completions undrained at once, so a crash or `maxDuration` SIGTERM re-runs *all* of them on redelivery — a much larger at-least-once blast radius, precisely on the latency-sensitive runs most likely to pack many steps into one invocation.
+2. **Divergent branches from non-durable results.** Advancing past a step before its result is durable lets the workflow commit to a forward path that a crash-and-redeliver can re-decide differently. A `Promise.race([B, C])` resolved by local timing can pick `B`, run `D(B)`, then crash before `step_completed_B` is durable — and the redelivery may re-resolve to `C`, so `D` executed against a winner the durable history never records. The same shape appears for a branch on a non-deterministic step output (`B(v1)` runs, crash, redelivery commits `B(v2)`). Idempotency doesn't cover these — `D(B)`/`D(C)` and `B(v1)`/`B(v2)` are *different* operations, not retries of one. A "run ahead only while at most one result is undurable" gate would contain the race case (a race needs ≥2 concurrent undurable steps) but not the non-deterministic-output case, and that residual hazard plus the re-execution blast radius outweighed the gain.
+
+So turbo deliberately stops at forced-optimistic *start* and awaits each `step_completed` before moving on: re-execution after a crash stays deterministic (each step re-runs against the same durable inputs) and bounded (roughly one step, not the whole chain). The idea is recorded here in case a future change (e.g. a determinism signal on steps, or deterministic race resolution) makes run-ahead safe enough to revisit.

package/docs/comparisons/index.mdx

@@ -0,0 +1,66 @@
+---
+title: Comparisons
+description: How the Workflow SDK compares to other durable execution, workflow, and AI-agent frameworks — Temporal, Cloudflare Workflows, AWS Step Functions, AWS Bedrock AgentCore, Inngest, and trigger.dev.
+type: overview
+summary: Side-by-side comparisons of the Workflow SDK against Temporal, Cloudflare Workflows, AWS Step Functions, AWS Bedrock AgentCore, Inngest, and trigger.dev.
+related:
+  - /docs/foundations/workflows-and-steps
+  - /docs/how-it-works/event-sourcing
+  - /worlds/vercel
+---
+
+The Workflow SDK overlaps with several categories: durable execution engines, background-job platforms, managed orchestrators, and AI-agent runtimes. This section compares it against the most common alternatives, one deep dive per tool.
+
+## What makes the Workflow SDK different
+
+- **It's an open-source SDK, not a hosted product.** Your workflows are plain TypeScript in your existing app. Run them on the managed [Vercel World](/worlds/vercel), or self-host on the [Postgres World](/worlds/postgres) — the [World abstraction](/worlds/building-a-world) lets you own and swap the storage, queue, and streaming layers independently.
+- **Versioning is safe by default.** Runs are pinned to the immutable deployment that started them, so shipping new code never disturbs in-flight runs. Upgrading a run is explicit and opt-in. See [Versioning](/docs/comparisons/workflow-sdk-vs-temporal#versioning).
+- **Realtime durable streaming is built in.** Stream partial output (LLM tokens, progress) to clients with [streams](/docs/foundations/streaming) that survive reconnects, cold starts, and replays — essential for chat and agent UIs.
+- **First-class AI agents.** `WorkflowAgent` ships inside the [AI SDK](/docs/ai), turning an agent loop into a durable workflow with automatic step retries and human-in-the-loop pauses.
+
+## Snapshot
+
+<Callout type="info">
+These comparisons are compiled from each product's public documentation and are **not** based on head-to-head benchmarks — durable engines differ enough that a single number rarely compares cleanly. Treat them as directional and verify current pricing and limits against each vendor's docs.
+</Callout>
+
+| Tool | Category | Durability model | Open source / self-host | Language(s) |
+| --- | --- | --- | --- | --- |
+| **Workflow SDK** | Durable functions SDK | Event-log + deterministic replay | ✅ Apache-2.0 — self-host (Postgres) or Vercel | TypeScript (Python beta) |
+| [Temporal](/docs/comparisons/workflow-sdk-vs-temporal) | Durable execution platform | Event-sourced replay | ✅ MIT server — self-host or Temporal Cloud | Go, Java, TS, Python, .NET, PHP, Ruby |
+| [Cloudflare Workflows](/docs/comparisons/workflow-sdk-vs-cloudflare-workflows) | Durable execution engine | Step-result memoization + replay | ❌ Cloudflare-only | TypeScript (Python beta) |
+| [AWS Step Functions](/docs/comparisons/workflow-sdk-vs-aws-step-functions) | Managed state-machine orchestrator | Declarative ASL state machine | ❌ AWS-only | ASL JSON (tasks: any language) |
+| [AWS Bedrock AgentCore](/docs/comparisons/workflow-sdk-vs-aws-agentcore) | AI-agent hosting platform | Not durable execution (ephemeral sessions) | ❌ AWS-only | Python, Node.js |
+| [Inngest](/docs/comparisons/workflow-sdk-vs-inngest) | Durable functions / event platform | Step-result memoization | ◑ SSPL — self-host (community/best-effort) or SaaS | TypeScript (Python/Go pre-1.0) |
+| [trigger.dev](/docs/comparisons/workflow-sdk-vs-trigger-dev) | Durable task platform | Process checkpoint/restore (CRIU) | ✅ Apache-2.0 — self-host or Cloud | TypeScript only |
+
+## Deep dives
+
+<Cards>
+    <Card href="/docs/comparisons/workflow-sdk-vs-temporal" title="vs Temporal">
+        The mature, language-agnostic durable-execution platform. You run the workers; Workflow SDK runs in your app.
+    </Card>
+    <Card href="/docs/comparisons/workflow-sdk-vs-cloudflare-workflows" title="vs Cloudflare Workflows">
+        A durable engine on Workers + Durable Objects. Both replay; they handle versioning and encryption very differently.
+    </Card>
+    <Card href="/docs/comparisons/workflow-sdk-vs-aws-step-functions" title="vs AWS Step Functions">
+        Declarative ASL JSON state machines vs. plain TypeScript control flow.
+    </Card>
+    <Card href="/docs/comparisons/workflow-sdk-vs-aws-agentcore" title="vs AWS Bedrock AgentCore">
+        An AI-agent hosting platform — not a durable-execution engine. Different axis.
+    </Card>
+    <Card href="/docs/comparisons/workflow-sdk-vs-inngest" title="vs Inngest">
+        Event-driven durable functions that run on your own infra over HTTP.
+    </Card>
+    <Card href="/docs/comparisons/workflow-sdk-vs-trigger-dev" title="vs trigger.dev">
+        A TypeScript task platform that achieves durability by snapshotting the process (CRIU).
+    </Card>
+</Cards>
+
+<Callout type="info">
+Moving an existing system over? Each deep dive includes a concept-mapping section, and the Workflow SDK migration skill can translate code for you:
+
+```bash
+npx skills add https://github.com/vercel/workflow --skill migrating-to-workflow-sdk
+```
+</Callout>

package/docs/comparisons/meta.json

@@ -0,0 +1,11 @@
+{
+  "title": "Comparisons",
+  "pages": [
+    "workflow-sdk-vs-temporal",
+    "workflow-sdk-vs-cloudflare-workflows",
+    "workflow-sdk-vs-aws-step-functions",
+    "workflow-sdk-vs-aws-agentcore",
+    "workflow-sdk-vs-inngest",
+    "workflow-sdk-vs-trigger-dev"
+  ]
+}

package/docs/comparisons/workflow-sdk-vs-aws-agentcore.mdx

@@ -0,0 +1,55 @@
+---
+title: Workflow SDK vs AWS Bedrock AgentCore
+description: How the Workflow SDK compares to AWS Bedrock AgentCore — a durable-execution framework versus an AI-agent hosting platform. They solve different problems.
+type: conceptual
+summary: AgentCore hosts and operates AI agents in isolated microVMs but is not a durable-execution engine. The Workflow SDK provides durable orchestration and resumable streaming for agents.
+prerequisites:
+  - /docs/foundations/workflows-and-steps
+related:
+  - /docs/ai
+  - /docs/ai/resumable-streams
+  - /docs/foundations/streaming
+---
+
+[Amazon Bedrock AgentCore](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/what-is-bedrock-agentcore.html) is AWS's platform for **hosting and operating AI agents** — secure microVM runtime, plus building blocks for Memory, tool Gateways, and Identity. It is *not* a durable-execution engine, which makes this less a head-to-head and more a "different axis" comparison.
+
+<Callout type="info">
+**These solve different problems.** AgentCore answers "where do I securely run and operate an agent on AWS?" The Workflow SDK answers "how do I make a multi-step, tool-calling agent loop durable, resumable, and streamable?" AWS itself pairs AgentCore with a durable layer (Step Functions, or Temporal) for resumability — the Workflow SDK provides that durable layer natively, in your own app.
+</Callout>
+
+## At a glance
+
+| | Workflow SDK | AWS Bedrock AgentCore |
+| --- | --- | --- |
+| **Category** | Open-source durable-functions SDK | AI-agent hosting & operations platform |
+| **Durable execution?** | ✅ Event-log replay; the agent loop resumes from its last checkpoint after a crash | ❌ Not built-in — sessions are ephemeral microVMs; durability is opt-in via Memory or a framework checkpointer |
+| **What it gives you** | Durable orchestration, steps, hooks, streaming, observability | Runtime (microVM hosting), Harness (managed agent loop), Memory, Gateway (tools/MCP), Identity, Browser, Code Interpreter, Observability, Policy, Evaluations |
+| **Languages** | TypeScript / JS (Python beta) | Python-first for authoring; TypeScript or Python project scaffolding via the `@aws/agentcore` CLI; framework-agnostic (LangGraph, CrewAI, Strands, etc.) |
+| **Isolation** | VM-sandboxed workflow code + full-Node steps | Dedicated Firecracker microVM per session (strong hardware isolation) |
+| **Max duration** | No limit ([Vercel World limits](https://vercel.com/docs/workflows/pricing)) | 8-hour hard cap per session |
+| **AI streaming** | Native **durable, resumable** streaming (survives reconnect, cold start, replay) | Live SSE / WebSocket only — no buffering or replay of missed tokens |
+| **Versioning** | Runs pinned to immutable deployment | Immutable runtime versions + endpoints; in-flight sessions stay on their launch version (deployment/rollback only — no replay) |
+| **Portability** | Apache-2.0; runs anywhere Node runs; World abstraction | AWS-only; agent *code* is portable, the operating platform is not |
+| **Compliance** | Inherits your platform | HIPAA-eligible; SOC/PCI/ISO not third-party certified; not FedRAMP-authorized |
+| **Pricing** | SDK free; pay your platform | Per-module consumption (vCPU-hr / GB-hr, etc.) + model inference billed via Bedrock; no minimums |
+
+**What the limits mean in practice:** AgentCore's 8-hour session cap means an agent that waits on a human, a long-running job, or a slow external system can't span that wait in one session. Workflow SDK runs have [no duration cap](https://vercel.com/docs/workflows/pricing) — they suspend durably at `sleep()` and hooks for hours or weeks.
+
+## The core distinction: durability
+
+AgentCore Runtime gives each session an isolated microVM with up to 8 hours of runtime, but the compute is **ephemeral** — on a crash or stop, the next invocation gets a fresh microVM with no automatic replay of the agent loop. AWS's own guidance is to use AgentCore Memory or a framework checkpointer for state durability, and to layer a workflow engine (Step Functions or Temporal) on top when you need durable orchestration.
+
+The Workflow SDK *is* that durable layer. With `WorkflowAgent` (in the [AI SDK](/docs/ai)), the agent loop becomes a durable workflow: each model call and tool execution is a checkpointed step, the run resumes mid-loop after a failure, and partial output streams to the client through [resumable streams](/docs/ai/resumable-streams) that survive disconnects and cold starts.
+
+## Where AgentCore leads
+
+AgentCore is purpose-built for operating agents on AWS, and brings things the Workflow SDK doesn't try to be:
+
+- **MicroVM isolation per session** — the strongest hardware isolation among the tools in this section.
+- **Managed agent infrastructure** — Memory, a tool Gateway (turn APIs/Lambda/MCP servers into tools), and Identity (credential vaulting, OAuth) as first-class managed services.
+- **Enterprise/AWS compliance breadth** and VPC/PrivateLink networking.
+
+If your priority is running agents inside AWS with managed memory, tools, and identity, AgentCore is a strong fit. If your priority is **durable, resumable, streamable** agent execution that lives in your own TypeScript app and isn't tied to AWS, the Workflow SDK fits better — and the two can be combined (host on AgentCore, orchestrate durably with the Workflow SDK).
+
+---
+*Compiled from public documentation. AgentCore cold-start figures are community-sourced (no published SLA). Verify against [the AgentCore docs](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/what-is-bedrock-agentcore.html). Not based on head-to-head benchmarks.*