workflow 5.0.0-beta.5 → 5.0.0-beta.50
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +68 -23
- package/dist/api-workflow.d.ts +1 -1
- package/dist/api-workflow.d.ts.map +1 -1
- package/dist/api-workflow.js +1 -1
- package/dist/api.d.ts +3 -3
- package/dist/api.d.ts.map +1 -1
- package/dist/api.js +5 -7
- package/dist/index.d.ts +1 -0
- package/dist/index.d.ts.map +1 -1
- package/dist/index.js +6 -1
- package/dist/internal/builtins.d.ts +20 -3
- package/dist/internal/builtins.d.ts.map +1 -1
- package/dist/internal/builtins.js +68 -4
- package/dist/internal/errors.d.ts +1 -1
- package/dist/internal/errors.d.ts.map +1 -1
- package/dist/internal/errors.js +2 -2
- package/dist/nest-builder.d.ts +2 -0
- package/dist/nest-builder.d.ts.map +1 -0
- package/dist/nest-builder.js +2 -0
- package/dist/nest-vercel-builder.d.ts +2 -0
- package/dist/nest-vercel-builder.d.ts.map +1 -0
- package/dist/nest-vercel-builder.js +2 -0
- package/dist/observability.d.ts +1 -1
- package/dist/observability.js +2 -2
- package/dist/runtime.d.ts +2 -1
- package/dist/runtime.d.ts.map +1 -1
- package/dist/runtime.js +4 -1
- package/docs/ai/chat-session-modeling.mdx +29 -26
- package/docs/ai/defining-tools.mdx +6 -7
- package/docs/ai/human-in-the-loop.mdx +11 -11
- package/docs/ai/index.mdx +50 -45
- package/docs/ai/message-queueing.mdx +16 -16
- package/docs/ai/meta.json +1 -0
- package/docs/ai/resumable-streams.mdx +40 -28
- package/docs/ai/sleep-and-delays.mdx +10 -10
- package/docs/ai/streaming-updates-from-tools.mdx +6 -6
- package/docs/api-reference/index.mdx +24 -0
- package/docs/api-reference/meta.json +8 -0
- package/docs/api-reference/vitest/index.mdx +9 -15
- package/docs/api-reference/workflow/create-hook.mdx +89 -10
- package/docs/api-reference/workflow/create-webhook.mdx +16 -15
- package/docs/api-reference/workflow/define-hook.mdx +35 -33
- package/docs/api-reference/workflow/fatal-error.mdx +30 -8
- package/docs/api-reference/workflow/fetch.mdx +14 -10
- package/docs/api-reference/workflow/get-step-metadata.mdx +2 -2
- package/docs/api-reference/workflow/get-workflow-metadata.mdx +3 -3
- package/docs/api-reference/workflow/get-writable.mdx +7 -7
- package/docs/api-reference/workflow/index.mdx +4 -1
- package/docs/api-reference/workflow/retryable-error.mdx +1 -1
- package/docs/api-reference/workflow/set-attributes.mdx +61 -0
- package/docs/api-reference/workflow/sleep.mdx +4 -4
- package/docs/api-reference/workflow-ai/durable-agent.mdx +48 -86
- package/docs/api-reference/workflow-ai/index.mdx +3 -3
- package/docs/api-reference/workflow-ai/workflow-chat-transport.mdx +67 -24
- package/docs/api-reference/workflow-api/get-hook-by-token.mdx +26 -12
- package/docs/api-reference/workflow-api/get-run.mdx +43 -8
- package/docs/api-reference/workflow-api/index.mdx +6 -10
- package/docs/api-reference/workflow-api/resume-hook.mdx +73 -12
- package/docs/api-reference/workflow-api/resume-webhook.mdx +11 -9
- package/docs/api-reference/workflow-api/start.mdx +60 -13
- package/docs/api-reference/workflow-astro/index.mdx +18 -0
- package/docs/api-reference/workflow-astro/meta.json +4 -0
- package/docs/api-reference/workflow-astro/workflow.mdx +45 -0
- package/docs/api-reference/workflow-errors/entity-conflict-error.mdx +4 -4
- package/docs/api-reference/workflow-errors/hook-conflict-error.mdx +60 -0
- package/docs/api-reference/workflow-errors/hook-not-found-error.mdx +8 -8
- package/docs/api-reference/workflow-errors/index.mdx +88 -0
- package/docs/api-reference/workflow-errors/meta.json +6 -0
- package/docs/api-reference/workflow-errors/precondition-failed-error.mdx +68 -0
- package/docs/api-reference/workflow-errors/run-expired-error.mdx +2 -2
- package/docs/api-reference/workflow-errors/run-not-supported-error.mdx +58 -0
- package/docs/api-reference/workflow-errors/step-not-registered-error.mdx +5 -5
- package/docs/api-reference/workflow-errors/throttle-error.mdx +2 -2
- package/docs/api-reference/workflow-errors/too-early-error.mdx +2 -2
- package/docs/api-reference/workflow-errors/workflow-error.mdx +52 -0
- package/docs/api-reference/workflow-errors/workflow-not-registered-error.mdx +5 -6
- package/docs/api-reference/workflow-errors/workflow-run-cancelled-error.mdx +6 -6
- package/docs/api-reference/workflow-errors/workflow-run-failed-error.mdx +5 -5
- package/docs/api-reference/workflow-errors/workflow-run-not-completed-error.mdx +58 -0
- package/docs/api-reference/workflow-errors/workflow-run-not-found-error.mdx +4 -4
- package/docs/api-reference/workflow-errors/workflow-runtime-error.mdx +58 -0
- package/docs/api-reference/workflow-errors/workflow-world-error.mdx +8 -8
- package/docs/api-reference/workflow-globals.mdx +14 -10
- package/docs/api-reference/workflow-nest/configure-workflow-controller.mdx +33 -0
- package/docs/api-reference/workflow-nest/index.mdx +31 -0
- package/docs/api-reference/workflow-nest/meta.json +9 -0
- package/docs/api-reference/workflow-nest/nest-local-builder.mdx +64 -0
- package/docs/api-reference/workflow-nest/workflow-controller.mdx +40 -0
- package/docs/api-reference/workflow-nest/workflow-module.mdx +74 -0
- package/docs/api-reference/workflow-next/with-workflow.mdx +39 -17
- package/docs/api-reference/workflow-nitro/index.mdx +60 -0
- package/docs/api-reference/workflow-nuxt/index.mdx +48 -0
- package/docs/api-reference/workflow-observability/hydrate-data.mdx +35 -0
- package/docs/api-reference/workflow-observability/hydrate-resource-io.mdx +62 -0
- package/docs/api-reference/workflow-observability/index.mdx +62 -0
- package/docs/api-reference/workflow-observability/meta.json +11 -0
- package/docs/api-reference/workflow-observability/observability-revivers.mdx +50 -0
- package/docs/api-reference/workflow-observability/parse-class-name.mdx +41 -0
- package/docs/api-reference/workflow-observability/parse-step-name.mdx +40 -0
- package/docs/api-reference/workflow-observability/parse-workflow-name.mdx +55 -0
- package/docs/api-reference/workflow-runtime/create-world.mdx +39 -0
- package/docs/api-reference/workflow-runtime/get-world-handlers.mdx +44 -0
- package/docs/api-reference/{workflow-api → workflow-runtime}/get-world.mdx +11 -14
- package/docs/api-reference/workflow-runtime/health-check.mdx +50 -0
- package/docs/api-reference/workflow-runtime/index.mdx +41 -0
- package/docs/api-reference/workflow-runtime/meta.json +12 -0
- package/docs/api-reference/workflow-runtime/set-world.mdx +51 -0
- package/docs/api-reference/workflow-runtime/workflow-entrypoint.mdx +43 -0
- package/docs/api-reference/workflow-runtime/world/analytics.mdx +315 -0
- package/docs/api-reference/workflow-runtime/world/index.mdx +60 -0
- package/docs/api-reference/workflow-runtime/world/meta.json +4 -0
- package/docs/api-reference/workflow-runtime/world/queue.mdx +88 -0
- package/docs/api-reference/{workflow-api → workflow-runtime}/world/storage.mdx +98 -34
- package/docs/api-reference/{workflow-api → workflow-runtime}/world/streams.mdx +8 -8
- package/docs/api-reference/workflow-serde/index.mdx +1 -2
- package/docs/api-reference/workflow-serde/workflow-deserialize.mdx +3 -4
- package/docs/api-reference/workflow-serde/workflow-serialize.mdx +8 -8
- package/docs/api-reference/workflow-sveltekit/index.mdx +18 -0
- package/docs/api-reference/workflow-sveltekit/meta.json +4 -0
- package/docs/api-reference/workflow-sveltekit/workflow-plugin.mdx +42 -0
- package/docs/api-reference/workflow-vite/index.mdx +18 -0
- package/docs/api-reference/workflow-vite/meta.json +4 -0
- package/docs/api-reference/workflow-vite/workflow.mdx +48 -0
- package/docs/changelog/attributes-mvp.mdx +380 -0
- package/docs/changelog/batched-event-writes.mdx +79 -0
- package/docs/changelog/eager-processing.mdx +110 -436
- package/docs/changelog/index.mdx +4 -2
- package/docs/changelog/lazy-event-creation.md +127 -0
- package/docs/changelog/lazy-hook-resume.mdx +78 -0
- package/docs/changelog/meta.json +11 -1
- package/docs/changelog/resilient-resume.mdx +32 -0
- package/docs/changelog/resilient-start.mdx +33 -285
- package/docs/changelog/step-message-ownership.mdx +360 -0
- package/docs/changelog/turbo-mode.md +87 -0
- package/docs/comparisons/index.mdx +66 -0
- package/docs/comparisons/meta.json +11 -0
- package/docs/comparisons/workflow-sdk-vs-aws-agentcore.mdx +55 -0
- package/docs/comparisons/workflow-sdk-vs-aws-step-functions.mdx +111 -0
- package/docs/comparisons/workflow-sdk-vs-cloudflare-workflows.mdx +71 -0
- package/docs/comparisons/workflow-sdk-vs-inngest.mdx +102 -0
- package/docs/comparisons/workflow-sdk-vs-temporal.mdx +123 -0
- package/docs/comparisons/workflow-sdk-vs-trigger-dev.mdx +104 -0
- package/docs/configuration/build-and-diagnostics.mdx +70 -0
- package/docs/configuration/cli-and-web-ui.mdx +241 -0
- package/docs/configuration/framework-options.mdx +165 -0
- package/docs/configuration/index.mdx +32 -0
- package/docs/configuration/meta.json +12 -0
- package/docs/configuration/runtime-tuning.mdx +376 -0
- package/docs/configuration/worlds.mdx +313 -0
- package/docs/cookbook/advanced/child-workflows.mdx +211 -264
- package/docs/cookbook/advanced/meta.json +6 -1
- package/docs/cookbook/advanced/publishing-libraries.mdx +65 -56
- package/docs/cookbook/advanced/serializable-steps.mdx +28 -20
- package/docs/cookbook/advanced/upgrading-workflows.mdx +199 -0
- package/docs/cookbook/agent-patterns/agent-cancellation.mdx +27 -19
- package/docs/cookbook/agent-patterns/durable-agent.mdx +14 -142
- package/docs/cookbook/agent-patterns/human-in-the-loop.mdx +30 -22
- package/docs/cookbook/common-patterns/batching.mdx +18 -14
- package/docs/cookbook/common-patterns/idempotency.mdx +41 -53
- package/docs/cookbook/common-patterns/rate-limiting.mdx +8 -4
- package/docs/cookbook/common-patterns/saga.mdx +23 -19
- package/docs/cookbook/common-patterns/scheduling.mdx +34 -22
- package/docs/cookbook/common-patterns/sequential-and-parallel.mdx +29 -25
- package/docs/cookbook/common-patterns/timeouts.mdx +26 -21
- package/docs/cookbook/common-patterns/webhooks.mdx +10 -6
- package/docs/cookbook/common-patterns/workflow-composition.mdx +30 -27
- package/docs/cookbook/index.mdx +22 -21
- package/docs/cookbook/integrations/ai-sdk.mdx +85 -47
- package/docs/cookbook/integrations/chat-sdk.mdx +50 -33
- package/docs/cookbook/integrations/sandbox.mdx +62 -45
- package/docs/deploying.mdx +95 -0
- package/docs/errors/abort-signal-timeout-in-workflow.mdx +16 -12
- package/docs/errors/corrupted-event-log.mdx +29 -18
- package/docs/errors/deployment-mismatch.mdx +71 -0
- package/docs/errors/fetch-in-workflow.mdx +11 -7
- package/docs/errors/hook-conflict.mdx +69 -13
- package/docs/errors/index.mdx +2 -36
- package/docs/errors/node-js-module-in-workflow.mdx +9 -5
- package/docs/errors/replay-divergence.mdx +27 -0
- package/docs/errors/run-expired.mdx +85 -0
- package/docs/errors/runtime-decryption-failed.mdx +77 -0
- package/docs/errors/serialization-failed.mdx +44 -12
- package/docs/errors/start-invalid-workflow-function.mdx +9 -5
- package/docs/errors/step-executed-multiple-times.mdx +23 -0
- package/docs/errors/step-not-registered.mdx +6 -6
- package/docs/errors/timeout-in-workflow.mdx +12 -8
- package/docs/errors/webhook-invalid-respond-with-value.mdx +18 -18
- package/docs/errors/webhook-response-not-sent.mdx +20 -16
- package/docs/errors/workflow-not-registered.mdx +5 -5
- package/docs/foundations/cancellation.mdx +31 -32
- package/docs/foundations/errors-and-retries.mdx +42 -11
- package/docs/foundations/hooks.mdx +64 -35
- package/docs/foundations/idempotency.mdx +244 -12
- package/docs/foundations/index.mdx +1 -23
- package/docs/foundations/meta.json +2 -1
- package/docs/foundations/serialization.mdx +21 -22
- package/docs/foundations/starting-workflows.mdx +106 -30
- package/docs/foundations/streaming.mdx +107 -59
- package/docs/foundations/versioning.mdx +263 -0
- package/docs/foundations/workflows-and-steps.mdx +9 -9
- package/docs/getting-started/astro.mdx +22 -18
- package/docs/getting-started/express.mdx +15 -11
- package/docs/getting-started/fastify.mdx +15 -11
- package/docs/getting-started/hono.mdx +15 -11
- package/docs/getting-started/index.mdx +10 -3
- package/docs/getting-started/meta.json +3 -1
- package/docs/getting-started/nestjs.mdx +87 -20
- package/docs/getting-started/next.mdx +22 -16
- package/docs/getting-started/nitro.mdx +22 -18
- package/docs/getting-started/nuxt.mdx +15 -11
- package/docs/getting-started/python.mdx +135 -40
- package/docs/getting-started/react-router/index.mdx +33 -0
- package/docs/getting-started/react-router/meta.json +5 -0
- package/docs/getting-started/react-router/v7.mdx +237 -0
- package/docs/getting-started/react-router/v8.mdx +232 -0
- package/docs/getting-started/sveltekit.mdx +20 -16
- package/docs/getting-started/tanstack-start.mdx +17 -13
- package/docs/getting-started/vite.mdx +15 -11
- package/docs/how-it-works/cancellation.mdx +63 -63
- package/docs/how-it-works/code-transform.mdx +83 -67
- package/docs/how-it-works/encryption.mdx +30 -26
- package/docs/how-it-works/event-sourcing.mdx +98 -34
- package/docs/how-it-works/framework-integrations.mdx +96 -337
- package/docs/how-it-works/understanding-directives.mdx +22 -22
- package/docs/internal/index.mdx +6 -4
- package/docs/internal/meta.json +6 -1
- package/docs/internal/nitro-native-build.mdx +38 -0
- package/docs/internal/nitro-web-ui.mdx +24 -0
- package/docs/internal/serializable-abort-controller.mdx +7 -7
- package/docs/meta.json +3 -2
- package/docs/observability/attributes.mdx +134 -0
- package/docs/observability/index.mdx +32 -10
- package/docs/observability/meta.json +1 -1
- package/docs/observability/retention.mdx +93 -0
- package/docs/observability/tracing.mdx +124 -0
- package/docs/testing/index.mdx +36 -36
- package/docs/testing/server-based.mdx +10 -10
- package/docs/whats-new.mdx +186 -0
- package/package.json +17 -14
- package/docs/api-reference/workflow-api/world/index.mdx +0 -58
- package/docs/api-reference/workflow-api/world/meta.json +0 -4
- package/docs/api-reference/workflow-api/world/observability.mdx +0 -164
- package/docs/api-reference/workflow-api/world/queue.mdx +0 -86
- package/docs/deploying/building-a-world.mdx +0 -251
- package/docs/deploying/index.mdx +0 -95
- package/docs/deploying/meta.json +0 -4
- package/docs/deploying/world/local-world.mdx +0 -84
- package/docs/deploying/world/meta.json +0 -4
- package/docs/deploying/world/postgres-world.mdx +0 -224
- package/docs/deploying/world/vercel-world.mdx +0 -179
- package/docs/migration-guides/index.mdx +0 -34
- package/docs/migration-guides/meta.json +0 -9
- package/docs/migration-guides/migrating-from-aws-step-functions.mdx +0 -363
- package/docs/migration-guides/migrating-from-inngest.mdx +0 -314
- package/docs/migration-guides/migrating-from-temporal.mdx +0 -318
- package/docs/migration-guides/migrating-from-trigger-dev.mdx +0 -337
|
@@ -1,21 +1,20 @@
|
|
|
1
1
|
---
|
|
2
|
-
title: Eager
|
|
3
|
-
description: Combine workflow event replay and step bundles to do work inline where possible, only deferring to queue for parallelism
|
|
2
|
+
title: Eager processing of steps and incremental event replay
|
|
3
|
+
description: Combine workflow event replay and step bundles to do work inline where possible, only deferring to the queue for parallelism.
|
|
4
4
|
type: overview
|
|
5
5
|
---
|
|
6
6
|
|
|
7
|
-
# Eager
|
|
7
|
+
# Eager processing of steps and incremental event replay
|
|
8
8
|
|
|
9
9
|
**Date**: March 2026
|
|
10
10
|
|
|
11
|
-
This is a major internal architecture change to how Workflow DevKit executes workflows and steps
|
|
11
|
+
This is a major internal architecture change to how Workflow DevKit executes workflows and steps. It reduces function invocations and queue overhead by executing steps _inline_ within the same function invocation as the workflow replay, rather than dispatching every step to a separate function via the queue.
|
|
12
12
|
|
|
13
|
-
## Previous
|
|
13
|
+
## Previous architecture
|
|
14
14
|
|
|
15
|
-
The previous architecture used two separate routes,
|
|
16
|
-
each backed by its own queue trigger:
|
|
15
|
+
The previous architecture used two separate routes, each backed by its own queue trigger:
|
|
17
16
|
|
|
18
|
-
```
|
|
17
|
+
```text
|
|
19
18
|
Queue: __wkf_workflow_* --> /.well-known/workflow/v1/flow (workflow replay in VM)
|
|
20
19
|
|
|
|
21
20
|
suspension (step needed)
|
|
@@ -31,15 +30,15 @@ Queue: __wkf_step_* --> /.well-known/workflow/v1/step (step execution in
|
|
|
31
30
|
(cycle repeats for each step)
|
|
32
31
|
```
|
|
33
32
|
|
|
34
|
-
Each step required **2 queue messages** (step
|
|
33
|
+
Each step required **2 queue messages** (step invocation and workflow continuation) and **2 function invocations**, plus cold-start overhead for each. A serial workflow with 10 steps needed approximately 21 function invocations.
|
|
35
34
|
|
|
36
|
-
## New
|
|
35
|
+
## New architecture
|
|
37
36
|
|
|
38
37
|
The two routes are merged into a single handler at `/.well-known/workflow/v1/flow` using `workflowEntrypoint()`. The step route is no longer generated.
|
|
39
38
|
|
|
40
39
|
The handler runs an inline execution loop:
|
|
41
40
|
|
|
42
|
-
```
|
|
41
|
+
```text
|
|
43
42
|
receive queue message
|
|
44
43
|
|
|
|
45
44
|
+-- if message has stepId+stepName: execute that step, queue workflow continuation, exit
|
|
@@ -69,120 +68,66 @@ suspension with pending operations
|
|
|
69
68
|
|
|
70
69
|
A serial workflow with 10 steps now completes in **1 function invocation**.
|
|
71
70
|
|
|
72
|
-
##
|
|
73
|
-
|
|
74
|
-
After the workflow suspends with pending steps, the handler executes one step inline:
|
|
75
|
-
|
|
76
|
-
1. Create `step_started` event
|
|
77
|
-
2. Hydrate step input from the event log
|
|
78
|
-
3. Look up the step function via `getStepFunction(stepName)`
|
|
79
|
-
4. Execute the step function
|
|
80
|
-
5. Create `step_completed` or `step_failed` event
|
|
81
|
-
6. Loop back to workflow replay
|
|
82
|
-
|
|
83
|
-
This logic lives in `executeStep()` in `packages/core/src/runtime/step-executor.ts`.
|
|
71
|
+
## Background steps (parallel execution)
|
|
84
72
|
|
|
85
|
-
|
|
73
|
+
When a workflow suspends with multiple pending steps (for example, from `Promise.all`), the handler creates `step_created` events for all of them, queues N-1 back to `__wkf_workflow_*` with `stepId` and `stepName` in the message payload, executes 1 step inline, and loops back to replay.
|
|
86
74
|
|
|
87
|
-
|
|
75
|
+
Each background step message is handled by a separate function invocation of the same handler. When a message arrives with `stepId` and `stepName`, the handler executes that specific step, then checks if all parallel steps from the batch are done by comparing `step_created` events against terminal events (`step_completed`/`step_failed`):
|
|
88
76
|
|
|
89
|
-
|
|
90
|
-
2. Queues N-1 steps back to `__wkf_workflow_*` with `stepId` and `stepName` in the message payload
|
|
91
|
-
3. Executes 1 step inline
|
|
92
|
-
4. Loops back to replay
|
|
93
|
-
|
|
94
|
-
Each background step message is handled by a separate function invocation of the same handler. When a message arrives with `stepId` and `stepName`, the handler executes that specific step, then checks if all parallel steps from the batch are done by loading the event log and comparing `step_created` events against terminal events (`step_completed`/`step_failed`):
|
|
95
|
-
|
|
96
|
-
- **All steps done**: The handler replays the workflow inline, continuing the execution loop without a queue roundtrip. Events are loaded with a cursor so subsequent loop iterations can use incremental loading.
|
|
77
|
+
- **All steps done**: The handler replays the workflow inline, continuing the execution loop without a queue roundtrip.
|
|
97
78
|
- **Steps still pending**: The handler returns without queuing a continuation. The last handler to complete its step will see all steps done and replay inline.
|
|
98
|
-
- **Pending ops (stream writes)**: The handler queues a continuation and returns, so `waitUntil` can flush the pending stream data
|
|
79
|
+
- **Pending ops (stream writes)**: The handler queues a continuation and returns, so `waitUntil` can flush the pending stream data.
|
|
99
80
|
|
|
100
|
-
### Convergence
|
|
81
|
+
### Convergence after parallel steps
|
|
101
82
|
|
|
102
|
-
When multiple background steps complete near-simultaneously, multiple handlers may observe "all steps done" and attempt to advance the workflow concurrently. The event-sourced architecture plus the invariants
|
|
83
|
+
When multiple background steps complete near-simultaneously, multiple handlers may observe "all steps done" and attempt to advance the workflow concurrently. The event-sourced architecture plus the invariants below ensure safe convergence:
|
|
103
84
|
|
|
104
|
-
- **`step_created` idempotency
|
|
105
|
-
- **`step_completed` / `step_failed` idempotency
|
|
106
|
-
- **Queue idempotency keys
|
|
107
|
-
- **Deterministic replay
|
|
85
|
+
- **`step_created` idempotency**: Duplicate creates return 409; exactly one handler owns each step.
|
|
86
|
+
- **`step_completed` / `step_failed` idempotency**: Only the first invocation to record a terminal result wins.
|
|
87
|
+
- **Queue idempotency keys**: Background step messages use `correlationId` as the idempotency key.
|
|
88
|
+
- **Deterministic replay**: All invocations produce the same result given the same event log.
|
|
108
89
|
|
|
109
|
-
### Single
|
|
90
|
+
### Single inline executor per step
|
|
110
91
|
|
|
111
|
-
Inline step execution combined with background-step dispatch introduces a new coordination requirement
|
|
92
|
+
Inline step execution combined with background-step dispatch introduces a new coordination requirement: when multiple handlers reach the same `Promise.all` batch concurrently, we need to guarantee that each step body runs at most once via the inline path. Without that guarantee, the event log accumulates duplicate `step_started` events (including some written *after* `step_completed`, which orphans them on replay) and step bodies run redundantly.
|
|
112
93
|
|
|
113
|
-
The design enforces
|
|
94
|
+
The design enforces one invariant: **exactly one handler owns each step, and only the owner may execute it inline**. Ownership is established by the atomicity of `step_created`:
|
|
114
95
|
|
|
115
|
-
1. **Atomic `step_created
|
|
116
|
-
2. **Suspension handler reports ownership
|
|
117
|
-
3. **Inline execution is gated on ownership
|
|
118
|
-
4. **Queueing is unconditional
|
|
96
|
+
1. **Atomic `step_created`**: `events.create('step_created', correlationId=X)` is serialized per correlation ID in every world. Exactly one concurrent caller succeeds; the rest receive `EntityConflictError`.
|
|
97
|
+
2. **Suspension handler reports ownership**: Only `step_created` writes that succeeded (not those that caught 409) count toward ownership.
|
|
98
|
+
3. **Inline execution is gated on ownership**: A handler that didn't win any `step_created` race performs no inline execution.
|
|
99
|
+
4. **Queueing is unconditional**: For every pending step except the one being executed inline, the handler enqueues a background step message with `idempotencyKey: correlationId`. This makes crash recovery work: if a prior handler wrote `step_created` but crashed before enqueueing, a later handler will enqueue the orphaned step. Concurrent handlers' redundant enqueues deduplicate on the idempotency key.
|
|
119
100
|
|
|
120
|
-
Together these give: every `step_created` event has exactly one inline executor
|
|
101
|
+
Together these give: every `step_created` event has exactly one inline executor **and** at least one queued dispatch. Step bodies are never executed concurrently, and `step_started` events never land in the log after `step_completed` for the same step.
|
|
121
102
|
|
|
122
|
-
**Retry semantics are preserved**:
|
|
103
|
+
**Retry semantics are preserved**: a step that is currently running (status=`running`) still accepts a second `step_started` write with an incremented attempt counter: this is how queue redelivery after a SIGKILL mid-execution legitimately re-runs the step.
|
|
123
104
|
|
|
124
|
-
## Incremental
|
|
105
|
+
## Incremental event loading
|
|
125
106
|
|
|
126
107
|
The handler caches the event log in memory across loop iterations. Instead of re-fetching the entire event log on each replay:
|
|
127
108
|
|
|
128
|
-
1. **First iteration**:
|
|
129
|
-
2. **Subsequent iterations**:
|
|
109
|
+
1. **First iteration**: Load all events and return the final pagination cursor.
|
|
110
|
+
2. **Subsequent iterations**: Fetch only events created after the saved cursor and append them to the cached array.
|
|
130
111
|
|
|
131
112
|
For a 10-step serial workflow completing in one invocation, the 10th replay loads ~2 new events instead of re-fetching all ~30.
|
|
132
113
|
|
|
133
|
-
|
|
134
|
-
|
|
135
|
-
The incremental loading depends on the server returning a cursor even on the final page of results (`hasMore: false`). Previously, `workflow-server` returned `cursor: null` when there were no more pages. This was fixed in the `peter/fix-end-cursor` branch to always return an `eid:<eventId>` cursor when there are events, aligning with `world-local` and `world-postgres` behavior.
|
|
136
|
-
|
|
137
|
-
If a World implementation does not return a cursor after the initial load, the handler logs an error and falls back to a full reload.
|
|
138
|
-
|
|
139
|
-
## Timeout Handling
|
|
114
|
+
Incremental loading depends on the World returning a cursor even on the final page of results. If a World implementation does not return a cursor after the initial load, the handler logs an error and falls back to a full reload.
|
|
140
115
|
|
|
141
|
-
|
|
116
|
+
## Timeout handling
|
|
142
117
|
|
|
143
|
-
The
|
|
118
|
+
The inline execution loop checks wall-clock time before each replay iteration. If the elapsed time exceeds a configurable threshold (110s by default for a 120-second function limit), the handler reschedules itself through the queue and returns. Configure the threshold with `WORKFLOW_V2_TIMEOUT_MS`.
|
|
144
119
|
|
|
145
|
-
If a single step takes longer than the timeout threshold, the step runs to completion (or SIGKILL)
|
|
120
|
+
If a single step takes longer than the timeout threshold, the step runs to completion (or SIGKILL). There is no interruption mechanism for in-progress step execution. This is the same behavior as the previous architecture.
|
|
146
121
|
|
|
147
|
-
## Queue
|
|
122
|
+
## Queue message changes
|
|
148
123
|
|
|
149
|
-
The `WorkflowInvokePayload` schema has two new optional fields:
|
|
150
|
-
|
|
151
|
-
{/*@skip-typecheck - snippet, not runnable code*/}
|
|
152
|
-
|
|
153
|
-
```typescript
|
|
154
|
-
stepId: z.string().optional()
|
|
155
|
-
stepName: z.string().optional()
|
|
156
|
-
```
|
|
157
|
-
|
|
158
|
-
When `stepId` is present, the handler executes that specific step before (or instead of) replaying the workflow. Background steps are queued with both `stepId` and `stepName` set, so the handler knows which step function to call without loading the event log. Previously, `stepName` was resolved by loading all events and searching for the `step_created` event matching the `stepId` — an O(N) operation on the full event history for every background step arrival.
|
|
124
|
+
The `WorkflowInvokePayload` schema has two new optional fields: `stepId` and `stepName`. When `stepId` is present, the handler executes that specific step before (or instead of) replaying the workflow. Background steps are queued with both set, so the handler knows which step function to call without loading the event log. Previously, `stepName` was resolved by loading all events and searching for the `step_created` event matching the `stepId`, an O(N) operation on the full event history for every background step arrival.
|
|
159
125
|
|
|
160
126
|
The queue trigger configuration uses `WORKFLOW_QUEUE_TRIGGER` on the `__wkf_workflow_*` topic. The `__wkf_step_*` topic and its separate trigger are no longer generated.
|
|
161
127
|
|
|
162
|
-
##
|
|
163
|
-
|
|
164
|
-
### Base Builder
|
|
165
|
-
|
|
166
|
-
New method `createCombinedBundle()` in `packages/builders/src/base-builder.ts`:
|
|
167
|
-
|
|
168
|
-
1. Builds the step registrations bundle (same esbuild + SWC step mode as before)
|
|
169
|
-
2. Builds the workflow VM code string (same esbuild + SWC workflow mode as before)
|
|
170
|
-
3. Generates a combined route file that imports the step registrations and uses `workflowEntrypoint(workflowCode)`
|
|
171
|
-
|
|
172
|
-
No changes to the SWC plugin were needed. The two-pass build approach (separate step and workflow SWC modes) still applies.
|
|
173
|
-
|
|
174
|
-
### Framework Builders
|
|
175
|
-
|
|
176
|
-
All framework builders were updated to use `createCombinedBundle()`:
|
|
128
|
+
## Generated file layout
|
|
177
129
|
|
|
178
|
-
|
|
179
|
-
- **NestJS, Nitro, Standalone**: replaces separate `createStepsBundle()` + `createWorkflowsBundle()` calls
|
|
180
|
-
- **SvelteKit, Astro**: same, plus post-processing regex updated to match `workflowEntrypoint`
|
|
181
|
-
- **Vercel Build Output API** (used by Nitro/Astro production): single `flow.func/` with `WORKFLOW_QUEUE_TRIGGER`
|
|
182
|
-
|
|
183
|
-
### Generated File Layout
|
|
184
|
-
|
|
185
|
-
```
|
|
130
|
+
```text
|
|
186
131
|
.well-known/workflow/v1/
|
|
187
132
|
flow/
|
|
188
133
|
route.js # Handler (workflowEntrypoint)
|
|
@@ -196,400 +141,129 @@ All framework builders were updated to use `createCombinedBundle()`:
|
|
|
196
141
|
|
|
197
142
|
The `step/` directory is no longer generated.
|
|
198
143
|
|
|
199
|
-
##
|
|
200
|
-
|
|
201
|
-
`handleSuspension()` in `packages/core/src/runtime/suspension-handler.ts` creates events for all pending operations (hooks, step events, wait events) but does **not** queue step messages. It returns the pending step items so the handler can decide which to execute inline vs. queue to background.
|
|
202
|
-
|
|
203
|
-
## Concerns and Edge Cases
|
|
204
|
-
|
|
205
|
-
### Parent→Child Polling Holds Worker Slots
|
|
206
|
-
|
|
207
|
-
`Run#returnValue` is implemented as a polling step: the workflow awaits the child run's terminal status inside a step body. In worker-based worlds (notably `world-postgres`), each such poll occupies a queue worker slot until the child run finishes. Parent workflows that fan out to many child runs — recursive workflows like `fibonacciWorkflow` are the obvious case — can therefore consume a large fraction of available workers just holding positions in `Promise.all([...children.map(c => c.returnValue)])`.
|
|
208
|
-
|
|
209
|
-
If `queueConcurrency` is smaller than the peak number of concurrent parent polls plus the workers needed for any in-flight children, the system deadlocks: every slot is held by a parent waiting on a child, but no child can acquire a slot to start. An earlier iteration of this work mitigated the deadlock runtime-side by detecting step context (via `contextStorage.getStore()`) and throwing `TooEarlyError` to re-enqueue the polling step, freeing the worker. That runtime guard has been removed — the responsibility now lies with worker-pool sizing.
|
|
210
|
-
|
|
211
|
-
For `world-postgres`, the default `queueConcurrency` is set to **50**, which is comfortably above the ~24 concurrent polls `fibonacciWorkflow(6)` produces at peak. Workflows that fan out more aggressively must raise this ceiling. `packages/core/src/runtime/run.ts`, the `queueConcurrency` option on `createWorld()` for `world-postgres`, and the `fibonacciWorkflow` fixture in `workbench/example/workflows/99_e2e.ts` all carry pointers to this caveat.
|
|
212
|
-
|
|
213
|
-
**Follow-up**: Replace the worker-pool sizing requirement with a polling design that does not occupy a worker slot. Options under consideration: (a) restore a runtime-side `TooEarlyError` re-enqueue path but make it visible to the user (rather than the silent guard the earlier iteration shipped), (b) move child-completion polling out of the step body into the suspension layer so a parent waiting on a child does not consume queue capacity at all, or (c) emit a `run_completed` notification on the parent's stream/queue so the parent only resumes when the child actually finishes. The current `queueConcurrency=50` default is a workaround, not a long-term answer — workflows with deep recursion or large fan-out can still exhaust workers regardless of how high we set the ceiling.
|
|
214
|
-
|
|
215
|
-
### VM Sandboxing
|
|
216
|
-
|
|
217
|
-
Workflow code still runs in a Node.js VM for determinism and sandboxing. Step code runs in the Node.js host context. The only change is that both happen within the same function invocation.
|
|
218
|
-
|
|
219
|
-
### Bundle Size and Cold Start
|
|
220
|
-
|
|
221
|
-
The combined bundle is larger (contains both step code and workflow VM code). Cold start time increases slightly. The reduction in total function invocations more than compensates.
|
|
222
|
-
|
|
223
|
-
### Step Retries
|
|
224
|
-
|
|
225
|
-
When an inline step fails with retries remaining:
|
|
226
|
-
|
|
227
|
-
- `RetryableError` with explicit `retryAfter` delay: re-queue to self with `stepId` and delay
|
|
228
|
-
- Transient errors with immediate retry: re-queue to self with `stepId` (delay = 1s)
|
|
229
|
-
- `FatalError`: fail immediately
|
|
144
|
+
## Design notes and tradeoffs
|
|
230
145
|
|
|
231
|
-
###
|
|
146
|
+
### Parent→child polling holds worker slots
|
|
232
147
|
|
|
233
|
-
|
|
148
|
+
`Run#returnValue` is implemented as a polling step: the workflow awaits the child run's terminal status inside a step body. In worker-based Worlds (notably `world-postgres`), each such poll occupies a queue worker slot until the child run finishes. Parent workflows that fan out to many child runs, such as recursive workflows like `fibonacciWorkflow`, can therefore consume a large fraction of available workers while holding positions in `Promise.all([...children.map(c => c.returnValue)])`.
|
|
234
149
|
|
|
235
|
-
-
|
|
236
|
-
- **Steps + at least one wait**: every step is queued (no inline execution). The handler returns with the wait timeout. Whichever lands first — a step's continuation or the wait timer — drives the next replay.
|
|
237
|
-
- **Hooks / waits only**: handler returns with the wait timeout (or no timeout, for hook-only suspensions). The next continuation is driven by external resume or the wait timer.
|
|
150
|
+
If `queueConcurrency` is smaller than the peak number of concurrent parent polls plus the workers needed for any in-flight children, the system deadlocks: every slot is held by a parent waiting on a child, but no child can acquire a slot to start.
|
|
238
151
|
|
|
239
|
-
|
|
152
|
+
For `world-postgres`, the default `queueConcurrency` is set to **50**. Workflows that fan out more aggressively must raise this ceiling.
|
|
240
153
|
|
|
241
|
-
|
|
154
|
+
To prevent deadlock when polling is executed inline by the step executor, `Run#pollReturnValue()` detects when it's running inside a step executor and throws `TooEarlyError` instead of polling in a blocking loop. The step executor handles `TooEarlyError` by re-queueing the step with a 1-second delay, freeing the worker. Unlike `RetryableError`, `TooEarlyError` does NOT count against `maxRetries`, so polling steps can retry indefinitely until the child completes.
|
|
242
155
|
|
|
243
|
-
|
|
156
|
+
**Follow-up**: Replace the worker-pool sizing requirement with a polling design that does not occupy a worker slot. Options under consideration: moving child-completion polling out of the step body into the suspension layer, or emitting a `run_completed` notification on the parent's stream/queue so the parent only resumes when the child actually finishes.
|
|
244
157
|
|
|
245
|
-
|
|
246
|
-
|
|
247
|
-
### Encryption Key Resolution
|
|
248
|
-
|
|
249
|
-
Encryption keys are resolved once before the inline execution loop starts (after the run status is confirmed as `running`) and reused across all iterations. Background step executions resolve the key independently. The key does not change within a run.
|
|
250
|
-
|
|
251
|
-
## Framework Support
|
|
252
|
-
|
|
253
|
-
All framework integrations have been updated: Next.js (eager and deferred/lazyDiscovery), NestJS, SvelteKit, Astro, Nitro/Nuxt/Hono/Express/Vite, and CLI standalone. The Vercel Build Output API builder (used by Nitro and Astro for production deploys) also uses the combined bundle with `WORKFLOW_QUEUE_TRIGGER`.
|
|
254
|
-
|
|
255
|
-
## Non-Next.js Integration Challenges
|
|
256
|
-
|
|
257
|
-
### Module Scope Duplication in Re-Bundled Output
|
|
258
|
-
|
|
259
|
-
Builders that use `bundleFinalOutput: true` (standalone CLI, Vercel Build Output API, NestJS) produce a single file where esbuild re-bundles the step registrations and the workflow runtime together. esbuild creates isolated module scopes for each source module, even within the same output file. This meant `registerStepFunction` and `getStepFunction` operated on different `Map` instances — steps were registered into one Map but looked up from another.
|
|
260
|
-
|
|
261
|
-
**Fix**: The step function registry (`registeredSteps` Map in `@workflow/core/private`) and the step context storage (`contextStorage` AsyncLocalStorage in `@workflow/core/step/context-storage`) were changed from module-scoped variables to `globalThis` singletons using `Symbol.for`. This ensures all esbuild module scopes share the same instances. The pattern was already used in the codebase for the World singleton and the class serialization registry.
|
|
262
|
-
|
|
263
|
-
### Workflow Package CJS Export Condition
|
|
264
|
-
|
|
265
|
-
The `workflow` package's root export has `"require": "./dist/typescript-plugin.cjs"` for TypeScript editor plugin loading. When esbuild bundles with CJS format, it resolves `import { defineHook } from 'workflow'` via the `require` condition, getting the TS plugin instead of the API.
|
|
266
|
-
|
|
267
|
-
**Fix**: Added a `"node"` condition (`"node": "./dist/index.js"`) before the `"require"` condition in the workflow package's exports. esbuild with `conditions: ['node']` matches `"node"` first and uses the correct API entry. TypeScript's plugin loader doesn't use `conditions: ['node']`, so it still falls through to `"require"` for the TS plugin.
|
|
268
|
-
|
|
269
|
-
### Local World Concurrent Replay Interference
|
|
270
|
-
|
|
271
|
-
The local development world (`world-local`) processes queue messages with high concurrency (default: 1000). With the V2 combined handler, parallel steps generate multiple workflow continuation messages. When these are processed concurrently, each triggers a replay that sees in-flight events from other concurrent replays. This causes "unconsumed event" errors because the event consumer encounters events that don't match any subscriber in the current replay state.
|
|
272
|
-
|
|
273
|
-
In production (Vercel), this doesn't happen — each function invocation is isolated with its own event loading.
|
|
274
|
-
|
|
275
|
-
**Fix**: The `EventsConsumer`'s `onUnconsumedEvent` callback (see "Concurrent Replay Interference with Multi-Batch Workflows" below) handles the concurrent event visibility issue. The V2 inline replay optimization (where the last background step to complete replays inline instead of queuing) further reduces concurrent replays. Redundant step executions from concurrent handlers are harmless due to `step_completed` idempotency — only the first completion wins.
|
|
276
|
-
|
|
277
|
-
### ESM `bundleFinalOutput` and Dynamic Require Errors
|
|
278
|
-
|
|
279
|
-
When `bundleFinalOutput: true` is used with ESM format, esbuild bundles CJS dependencies (like `debug`) into the output. CJS `require()` calls are wrapped in esbuild's `__require` polyfill, which throws "Dynamic require of X is not supported" in ESM contexts where `require` is undefined. This affected all ESM-based framework builders (Nitro, NestJS, SvelteKit, Astro) that were switched to `bundleFinalOutput: true` during the V2 migration.
|
|
280
|
-
|
|
281
|
-
**Fix**: ESM builders use `bundleFinalOutput: false` with `externalizeNonSteps: true`, matching the pre-V2 behavior. The framework's own bundler (Vite, Rollup, Turbopack) handles dependency resolution. The standalone CLI and Vercel Build Output API builders use `bundleFinalOutput: true` with ESM output plus a `createRequire(import.meta.url)` banner (see "V2 Combined Bundle Switched from CJS to ESM" below) so CJS dependencies can still call `require()` for Node.js builtins.
|
|
282
|
-
|
|
283
|
-
### Rollup Tree-Shaking of Step Registrations
|
|
284
|
-
|
|
285
|
-
When `bundleFinalOutput: false` is used with Nitro's rollup pipeline, the step registrations bundle (`steps.mjs`) only contains side-effect code (`registerStepFunction` calls) with no exports. Rollup tree-shakes the entire module because it has no used exports, removing all step registrations from the production bundle. This causes "Step not found" errors at runtime.
|
|
286
|
-
|
|
287
|
-
**Fix**: The steps bundle now exports a sentinel value (`export const __steps_registered = true`), and the combined route file imports it (`import { __steps_registered } from './steps.mjs'`). This gives rollup a used binding to track, preventing it from dropping the module and its side effects.
|
|
288
|
-
|
|
289
|
-
### Concurrent Replay Interference with Multi-Batch Workflows (historic)
|
|
290
|
-
|
|
291
|
-
An earlier iteration of the V2 work hit "Unconsumed event in event log" errors when multiple concurrent handlers raced into the same batch boundary. The diagnosis at the time was that concurrent handlers could see events the current replay hadn't reached yet, and the mitigation was a skip path in `onUnconsumedEvent` that tolerated step/hook/wait lifecycle events whose correlationId had a matching `step_created` / `hook_created` / earlier `wait_completed` in the log.
|
|
292
|
-
|
|
293
|
-
Later work on the "Single Inline Executor Per Step" invariant (described above) identified the actual root cause: duplicate `step_started` events were being written *after* `step_completed` on the same step, because the local world's `step_started` was not atomic w.r.t. terminal state and the main loop was re-picking already-queued steps for inline execution. Fixing those at the source (per-step mutex in `world-local` + ownership-gated inline dispatch + unconditional queueing with idempotency keys) eliminated the unconsumed-step-event path entirely, and fixing the `wait_completed` cursor bug (the main loop manually pushed `wait_completed` events without advancing `eventsCursor`, so the next incremental fetch re-returned them as local-array duplicates) eliminated the wait case.
|
|
294
|
-
|
|
295
|
-
**Current state**: the `onUnconsumedEvent` skip logic has been removed. Any unconsumed event now fatals the run with `CORRUPTED_EVENT_LOG`, matching the original contract from PR #1055. Incremental event loading in `runtime.ts` dedupes by `eventId` to tolerate any residual manual pushes.
|
|
296
|
-
|
|
297
|
-
### Stale V1 Artifacts in Build Caches
|
|
298
|
-
|
|
299
|
-
SvelteKit and Astro's build caches (including Vercel's) may preserve the old V1 `step/` route directory from previous builds. When the V2 builder runs, it no longer generates step routes, but the stale files remain and cause build failures (e.g., importing the removed `stepEntrypoint`). Additionally, SvelteKit's `beforeExit` hook that patches `.vc-config.json` files for Vercel deployments was still trying to configure the non-existent `step.func/` directory.
|
|
300
|
-
|
|
301
|
-
**Fix**: SvelteKit and Astro builders now clean up stale V1 step route directories during build. SvelteKit's Vercel deployment hook was updated to only configure the combined `flow.func/` directory.
|
|
302
|
-
|
|
303
|
-
### Next.js Canary Turbopack and Temp Files
|
|
304
|
-
|
|
305
|
-
The deferred (lazyDiscovery) Next.js builder writes build artifacts with a `.temp` extension to avoid HMR churn, then copies them to their final names. The V2 migration created `__step_registrations.route.js.temp` in the `app/` directory. Canary Turbopack rejects this file as an "Unknown module type" because the `.temp` extension has no associated loader.
|
|
306
|
-
|
|
307
|
-
**Fix**: The step registrations file is written directly to its final name (`__step_registrations.js`) since it doesn't need the temp-file HMR mechanism. Only the route file uses temp naming.
|
|
308
|
-
|
|
309
|
-
### Concurrent `step_started` Inflating Attempt Counter
|
|
310
|
-
|
|
311
|
-
When the V2 handler dispatches N parallel steps as background messages, each background step completion queues a workflow continuation. Up to N continuations may replay concurrently, and each may attempt to start the same not-yet-completed step (since `step_started` succeeds for already-running steps). Each call atomically increments the `attempt` counter. With N=5 parallel steps, the attempt counter can reach 5 on the first genuine execution — exceeding the default `maxRetries + 1 = 4` threshold and prematurely failing the step with "exceeded max retries".
|
|
312
|
-
|
|
313
|
-
This is the same known limitation described in "Convergence After Parallel Steps" above, but with a concrete failure mode: `promiseRaceStressTestWorkflow` (which uses 5 parallel steps with `Promise.race`) consistently failed on Postgres tests.
|
|
314
|
-
|
|
315
|
-
**Fix**: The max retries check in `executeStep()` now only enforces when `step.error` exists — distinguishing actual retries (failed → retry with error) from concurrent first-attempt races (multiple handlers start the same step simultaneously without any prior failure). Concurrent starts are harmless since `step_completed` idempotency ensures only the first completion wins.
|
|
316
|
-
|
|
317
|
-
### Inline Step Execution with Pending Stream Operations
|
|
318
|
-
|
|
319
|
-
When a step's arguments or return value include serialized streams (e.g., `WritableStream` from `getWritable()`, or AI SDK streaming steps), the serialization layer creates background `flushablePipe` operations that pipe data to S3. These ops are tracked in an `ops` array and need to complete before the stream data is readable by external consumers.
|
|
158
|
+
### Mixed suspensions
|
|
320
159
|
|
|
321
|
-
|
|
160
|
+
A suspension may contain steps, hooks, and waits simultaneously. The handler creates events for all, then dispatches everything we are not running inline as a single parallel batch of queue messages:
|
|
322
161
|
|
|
323
|
-
|
|
162
|
+
```text
|
|
163
|
+
ownedPendingSteps = pendingSteps.filter(owned by this handler)
|
|
164
|
+
inlineStep = ownedPendingSteps[0] // optional
|
|
324
165
|
|
|
325
|
-
|
|
166
|
+
dispatches = [
|
|
167
|
+
...for each non-inline pendingStep: queue stepId message (idempotency=correlationId),
|
|
168
|
+
...if soonest pending wait: queue delayed continuation
|
|
169
|
+
(delaySeconds=min(remaining, maxDelay),
|
|
170
|
+
idempotency=waitCorrelationId[:hop|:secondBucket]),
|
|
171
|
+
]
|
|
172
|
+
await Promise.all(dispatches)
|
|
326
173
|
|
|
327
|
-
|
|
328
|
-
|
|
329
|
-
2. **Root cause of the buffered write issue**: `WorkflowServerWritableStream.write()` buffers chunks and schedules a flush via `setTimeout(flush, 10ms)`. The `flushablePipe` calls `await writer.write(chunk)` which returns immediately (data buffered). `pendingOps--` fires before the 10ms timer. The `pollWritableLock` sees `pendingOps === 0` and resolves `state.promise`. The ops appear settled, but data is still in the buffer.
|
|
330
|
-
|
|
331
|
-
3. **Why this only affects Vercel Prod**: On local (world-local), stream writes go to the filesystem — effectively instant. On Vercel (world-vercel), writes go through HTTP to workflow-server → S3, adding 50-100ms latency. The buffered write returns instantly but the HTTP round-trip is deferred. When the V2 loop continues and the function eventually returns, `waitUntil` may not have enough time to flush.
|
|
332
|
-
|
|
333
|
-
**Follow-up**: The flush-waiter design described under "Buffered Stream Flush with Waiter Promises" below is the landed fix and resolves the buffered-write race. The remaining work is to shrink the 500ms inline-ops budget once we have confidence that the flush-waiter path settles deterministically across all worlds (today the budget is a defensive ceiling, not a tuned latency target), and to surface a stronger contract for "ops settled" — currently a 500ms timeout means "probably settled, give up and queue a continuation", which is correct but coarse. A signaled "ops drained" event from the world layer would let `executeStep()` proceed without the timeout in the common case, removing latency for streaming workflows whose ops settle in well under 500ms.
|
|
334
|
-
|
|
335
|
-
### CJS `module.exports` Collision in BOA Bundles (RESOLVED)
|
|
336
|
-
|
|
337
|
-
The Vercel Build Output API (BOA) builder creates a single CJS bundle via `createCombinedBundle` with `bundleFinalOutput: true`. The combined route file imports the steps bundle:
|
|
338
|
-
|
|
339
|
-
```js
|
|
340
|
-
import { __steps_registered } from './__step_registrations.js';
|
|
341
|
-
import { workflowEntrypoint } from 'workflow/runtime';
|
|
342
|
-
export const POST = workflowEntrypoint(workflowCode);
|
|
174
|
+
if (!inlineStep) return
|
|
175
|
+
await executeStep(inlineStep)
|
|
343
176
|
```
|
|
344
177
|
|
|
345
|
-
|
|
346
|
-
|
|
347
|
-
**Symptoms**: The Vercel deployment builds and starts successfully, but the `POST` handler is missing from the function's exports. Queue messages are delivered to the function but nothing processes them. All e2e tests hang indefinitely.
|
|
348
|
-
|
|
349
|
-
**Debugging steps that led to the root cause**:
|
|
350
|
-
|
|
351
|
-
1. Tested the CJS bundle locally with `node -e "require('./index.js')"` — confirmed 92 steps registered, but `module.exports` only contained `{ __steps_registered }`, not `{ POST }`.
|
|
352
|
-
2. Found two `module.exports` assignments in the bundle: line ~45K (from the combined route, exporting `POST`) and line ~95K (from the inlined steps bundle, exporting `__steps_registered`). The second overwrites the first.
|
|
353
|
-
3. Compared with the standalone builder's bundle which had the same steps code wrapped in `__commonJS()` — esbuild's wrapper prevents the inner `module.exports` from leaking.
|
|
354
|
-
|
|
355
|
-
**Fix**: When `bundleFinalOutput` is true, build the steps bundle in **ESM format** regardless of the final output format. The final esbuild pass converts everything to CJS correctly. ESM steps don't have `module.exports`, so there's no collision. The combined route's `export const POST` becomes the sole `module.exports` entry.
|
|
356
|
-
|
|
357
|
-
### Step Error Source Maps on BOA Deployments
|
|
358
|
-
|
|
359
|
-
The V2 combined CJS bundle (`bundleFinalOutput: true`) loses original source file names during re-bundling. Error stack traces show `/var/task/index.js` instead of `99_e2e.ts`. The `hasStepSourceMaps()` utility was updated to return `false` for BOA-builder frameworks (Express, Fastify, Hono, Nitro, Nuxt, Vite, Astro, Example) on Vercel preview, aligning test expectations with the actual bundle behavior.
|
|
360
|
-
|
|
361
|
-
### CLI Health Check Port Mismatch
|
|
362
|
-
|
|
363
|
-
The CLI `health` command defaults to `http://localhost:3000` when `WORKFLOW_LOCAL_BASE_URL` is not set. Different frameworks use different ports (Astro: 4321, SvelteKit: 5173). The e2e test passed `WORKFLOW_LOCAL_BASE_URL` via the spawn env, but the CLI's `getEnvVars()` function had a fixed list of env vars that didn't include `WORKFLOW_LOCAL_BASE_URL`. The env var was set but never read.
|
|
364
|
-
|
|
365
|
-
**Fix**: Added `WORKFLOW_LOCAL_BASE_URL` to the CLI's `getEnvVars()` return object.
|
|
366
|
-
|
|
367
|
-
### Buffered Stream Flush with Waiter Promises
|
|
368
|
-
|
|
369
|
-
`WorkflowServerWritableStream` buffers writes and flushes via a 10ms `setTimeout` for batching. Previously, `write()` returned immediately after buffering, causing the `flushablePipe`'s `pendingOps` counter to reach 0 before data actually reached the server. The V2 inline loop saw ops as settled prematurely and broke on every step with `WritableStream` serialization.
|
|
370
|
-
|
|
371
|
-
**Fix**: `write()` now returns a promise that resolves only after the scheduled flush completes. Multiple writes within the 10ms window still share a single batched HTTP request (the batching optimization is preserved). Each write registers a `{resolve, reject}` pair in a `flushWaiters` array. When the `setTimeout` fires and `flush()` completes the HTTP round-trip, all waiters are resolved (or rejected on error). This makes `pendingOps` accurately reflect server-side data state while keeping network-efficient batching.
|
|
372
|
-
|
|
373
|
-
The 500ms inline ops await in the step executor can now distinguish between:
|
|
374
|
-
|
|
375
|
-
- **Steps where ops settle** (data on server, ~200ms after lock release + flush) → continue loop inline
|
|
376
|
-
- **Steps where ops don't settle** (WritableStream kept open across steps) → break loop
|
|
377
|
-
|
|
378
|
-
### Lock-Release Polling Interval Lowered to 10ms
|
|
379
|
-
|
|
380
|
-
`flushablePipe`'s `pollWritableLock` / `pollReadableLock` use `setInterval` to detect when a user releases their stream lock without closing the stream — the Web Streams API has no event for that state. The V2 step executor's `opsSettled` race waits for this poll to resolve after each writable-bearing step body returns, so the polling interval sits on the critical path of every streaming step.
|
|
381
|
-
|
|
382
|
-
The interval was originally 100ms. Measuring a synthetic workflow with 5 sequential streaming steps (each step receives a shared `WritableStream` argument, writes a few chunks, releases the writer with the stream still open — the same pattern `doStreamStep` / `writeToolOutputToUI` / `writeFinishChunk` use in `DurableAgent.chat`) produced a per-step wait distribution clustered between 22–100ms with a mean of ~58ms. That matches the analytical prediction for a periodic poll with uniformly random offset relative to step return: ~half the interval. Across the 5 steps, polling alone added ~290ms of latency to the workflow even though no step actually had pending I/O — the writes were already flushed, the writer lock was already released, and we were just waiting for the next tick to notice.
|
|
383
|
-
|
|
384
|
-
**Fix**: dropped the polling interval from 100ms to 10ms in `packages/core/src/flushable-stream.ts`. Per-step wait drops from ~50ms average to ~5ms (a 10× improvement, expected to scale linearly with the number of writable-bearing steps in a workflow). For `DurableAgent.chat` with one tool call (4 writable-bearing steps), this removes ~180ms from the streaming chat response's critical path. Per-tick work is just `writable.locked` plus a `getWriter()`/`releaseLock()` probe — both microsecond-scale, so 10× more ticks during a stream's lifetime is not measurable in practice.
|
|
385
|
-
|
|
386
|
-
**Follow-up**: Replace the polling entirely with an event-driven release signal — wrap the writable returned from the `WritableStream` reviver with a writer that fires on `releaseLock()` — bringing the wait to ~0ms. The 10ms polling interval is the cheap path that captures most of the available win without the structural change, but every writable-bearing step still pays a ~5ms tax that the event-driven design would eliminate. The structural change is also worth pursuing because it removes a source of timing drift between `world-local` (filesystem-instant) and `world-vercel` (HTTP-deferred) — both would see truly synchronous lock-release detection rather than periodic-poll detection.
|
|
387
|
-
|
|
388
|
-
### Event Consumer Skip Logic Was Too Broad For Wait Replays
|
|
389
|
-
|
|
390
|
-
The V2 handler needs some tolerance for out-of-order replay, especially around step events created by concurrent continuations. An early follow-up broadened that fallback to all wait lifecycle events too, so `onUnconsumedEvent` would skip `wait_created` and the first `wait_completed` whenever they matched a known wait. In the BOA-backed previews that broke `hookDisposeTestWorkflow`: once the first run disposed its hook and went into `sleep('5s')`, a replay could skip the live wait event before `sleep()` registered its subscriber, leaving the run stuck forever at `wait_created`.
|
|
391
|
-
|
|
392
|
-
**Fix**: Keep the step/hook replay tolerance, but narrow the wait fallback to the one case we actually need: duplicate `wait_completed` events that appear *after* an earlier completion for the same wait. The `hookDispose` e2e was also updated to poll for hook registration/disposal instead of relying on fixed 3-5 second sleeps, which made the Vercel preview timing less brittle.
|
|
393
|
-
|
|
394
|
-
### TooEarlyError Retry Delay in Step Executor
|
|
395
|
-
|
|
396
|
-
The `executeStep()` function handles `TooEarlyError` (thrown when a step's `retryAfter` timestamp hasn't been reached yet) by returning a `retry` result with a timeout. The original implementation used a stale access pattern `(err as any).meta?.retryAfter` copied from an older error shape. The `TooEarlyError` class (from `@workflow/errors`) has `retryAfter` as a direct property (number of seconds), not nested under `.meta`. The stale pattern always evaluated to `undefined`, falling back to a 1-second delay regardless of the server's actual retry-after value.
|
|
397
|
-
|
|
398
|
-
**Fix**: Changed to `err.retryAfter ?? 1`, matching the correct pattern used in `step-handler.ts`.
|
|
399
|
-
|
|
400
|
-
### Health Check Endpoint JSON Response
|
|
401
|
-
|
|
402
|
-
The `withHealthCheck()` wrapper in `helpers.ts` was updated (on main) to return a JSON response with `{ healthy, endpoint, specVersion, workflowCoreVersion }` instead of a plain text string. The V2 branch's e2e test still expected `Content-Type: text/plain` and a text body after merging main, causing the "health check endpoint (HTTP)" test to fail across all frameworks and environments.
|
|
403
|
-
|
|
404
|
-
**Fix**: Updated the e2e test to expect `Content-Type: application/json` and validate the JSON body structure, including a `specVersion >= SPEC_VERSION_CURRENT` range assertion.
|
|
405
|
-
|
|
406
|
-
### V2 Combined Bundle Switched from CJS to ESM
|
|
407
|
-
|
|
408
|
-
The V2 combined bundle was initially emitted as CJS by the standalone CLI and Vercel Build Output API builders, while `main` had already moved those outputs to ESM in [#1562](https://github.com/vercel/workflow/pull/1562). Staying on CJS meant `import.meta.url` was polyfilled (often producing the wrong path in re-bundled contexts), and the `world-testing` server had to import from `flow.js` via `createRequire` to force CJS semantics on what was really a CJS bundle.
|
|
409
|
-
|
|
410
|
-
**Fix**: Align V2 with `main`'s ESM defaults:
|
|
411
|
-
|
|
412
|
-
1. The BOA builder emits `__step_registrations.mjs` and `index.mjs`, writes `"type": "module"` in `package.json`, and sets `handler: "index.mjs"` in `.vc-config.json`.
|
|
413
|
-
2. The standalone builder no longer overrides `format`; it inherits the base builder's `'esm'` default.
|
|
414
|
-
3. The standalone config outputs `step.mjs` / `flow.mjs` instead of `.js`.
|
|
415
|
-
4. The `world-testing` server uses a native `import { POST } from '../.well-known/workflow/v1/flow.mjs'` instead of `createRequire`.
|
|
416
|
-
5. `createCombinedBundle`'s final esbuild pass (for `bundleFinalOutput: true`) now prepends the same `createRequire(import.meta.url)` banner used by the workflow/webhook bundles so CJS dependencies that call `require()` for Node.js builtins (for example the `events` module referenced by bundled libraries) still resolve at runtime.
|
|
417
|
-
6. To avoid a duplicate `__createRequire` declaration, the inner steps bundle that gets inlined by the final pass skips the banner — only the outer bundle emits it. This is threaded through via a new `skipEsmRequireBanner` option on `createStepsBundle`.
|
|
418
|
-
|
|
419
|
-
### World specVersion in Health Check Responses
|
|
178
|
+
The wait timer is queued as its own continuation rather than encoded in the handler's return value (`{ timeoutSeconds }`). This is what makes `Promise.race(step, sleep)` behave correctly: even when the inline step blocks the handler for the full step duration, the wait continuation fires in a separate function invocation. If the sleep wins, that parallel invocation observes `wait_completed` via the "complete elapsed waits" pass and finishes the run; if the step wins, the wait continuation fires later and no-ops on the terminal run via the existing terminal-event check.
|
|
420
179
|
|
|
421
|
-
|
|
180
|
+
Step queueing remains unconditional (covers crash recovery: if a prior handler wrote `step_created` but crashed before queueing, a later handler will queue it; idempotency keys dedupe redundant queues across concurrent handlers).
|
|
422
181
|
|
|
423
|
-
|
|
182
|
+
Wait continuations are likewise deduplicated, keyed on the wait's correlation ID: while a wait is pending, every replay pass over the run re-observes it and would otherwise enqueue another delayed continuation. A key is attached in all cases, since some worlds serialize key-less workflow messages per run, which would park the continuation behind the handler's own inline step execution.
|
|
424
183
|
|
|
425
|
-
|
|
184
|
+
Two situations deliver a continuation while its wait is still pending, and each varies the key so the re-enqueue isn't dropped by a world's dedupe window (which outlives the first delivery): waits longer than the maximum queue delay (23h, bounded by VQS's 24h message retention) are clamped and chained across hops, with the hop index suffixed to the key so each hop dedupes within its window but the chain always advances; and near-elapsed waits (≤2s remaining) use a second-bucketed key suffix so a continuation delivered marginally early (clock skew) can enqueue a fresh short-delay retry. See `runtime/wait-continuation.ts` for the full selection logic.
|
|
426
185
|
|
|
427
|
-
|
|
186
|
+
The retry/throttle and hook-conflict paths still return `{ timeoutSeconds }` since their semantics are "redeliver THIS message after a delay" rather than "schedule a fresh wait timer." Those can be unified in a follow-up.
|
|
428
187
|
|
|
429
|
-
|
|
188
|
+
The unified dispatch requires `world-local` to honor `delaySeconds` on the queue (added in the same PR series). Without it, the wait continuation would fire instantly in dev and trigger a spurious replay before the wait elapsed (recoverable via redelivery, but inefficient and observable as duplicate `step_started` events under contention).
|
|
430
189
|
|
|
431
|
-
|
|
190
|
+
### VM sandboxing
|
|
432
191
|
|
|
433
|
-
|
|
434
|
-
|
|
435
|
-
### Deferred Next.js Builds Re-Ran Eager Discovery
|
|
436
|
-
|
|
437
|
-
The later merge also pulled `BaseBuilder.createCombinedBundle()` into the deferred Next.js path without a way to pass the already-discovered workflow/step/serde entry sets. As a result, `packages/next/src/builder-deferred.ts` quietly fell back to `discoverEntries()` during production builds, re-emitting `Discovering workflow directives ...` and failing the local build tests that assert deferred mode avoids eager input-graph scans.
|
|
438
|
-
|
|
439
|
-
**Fix**: Threaded explicit `discoveredEntries` through `createCombinedBundle()` and passed the deferred builder's tracked workflow/step/serde file sets into that call. Deferred Next.js builds now reuse the socket/cache-driven discovery state instead of re-running the base eager discovery pass.
|
|
440
|
-
|
|
441
|
-
### Deferred Package Steps Fell Back to Compiled `dist/` Files
|
|
442
|
-
|
|
443
|
-
Once deferred discovery stopped re-running the base eager scan, some package-provided steps were only being rediscovered from built artifacts such as `packages/ai/dist/agent/durable-agent.js`. Those compiled files no longer carried every nested `'use step'` directive, so local production Next.js builds could miss registrations like `@workflow/ai/agent`'s `closeStream` helper and fail at runtime with "step is not registered in the current deployment".
|
|
444
|
-
|
|
445
|
-
**Fix**: The deferred Next.js builder now rewrites discovered workspace package paths from `dist/` back to their matching `src/` files when those sources exist. That keeps deferred bundling pointed at the directive-bearing source modules instead of their compiled output.
|
|
446
|
-
|
|
447
|
-
### Workspace Source Step IDs Lost Export Subpaths
|
|
448
|
-
|
|
449
|
-
Switching deferred builds over to workspace `src/` files fixed the missing nested directives, but it exposed a second mismatch in the SWC manifest path logic. `packages/builders/src/module-specifier.ts` only matched package exports against the on-disk file being transformed, so `packages/ai/src/agent/durable-agent.ts` was assigned `@workflow/ai@...` while the runtime still referenced the exported subpath id `@workflow/ai/agent@...`. Local Next.js agent runs then failed with "Step `step//@workflow/ai/agent@...//closeStream` is not registered" even though the source file was finally back in the bundle.
|
|
450
|
-
|
|
451
|
-
**Fix**: `resolveModuleSpecifier()` now treats workspace source files as the source-backed form of their exported `dist/` targets when deriving step ids. That preserves package export subpaths like `@workflow/ai/agent` for id generation while still bundling the directive-bearing `src/` modules.
|
|
452
|
-
|
|
453
|
-
### Tarball-Staged Next.js Builds Still Lost Package Step Sources
|
|
454
|
-
|
|
455
|
-
The local production and Postgres Next.js jobs stage the workbenches by packing workspace packages into tarballs and installing those tarballs into a temporary `node_modules` tree. Deferred discovery was already willing to rewrite workspace `packages/*/dist/*` files back to `src/*`, but the tarballed `@workflow/ai` package did not publish its `src/` tree and the base builder still treated `node_modules/@workflow/*/src/*` as ordinary package imports. That meant the staged CI path fell back to `dist/` again and dropped nested steps like `@workflow/ai/agent`'s `closeStream`, even after the workspace build path had been fixed.
|
|
456
|
-
|
|
457
|
-
**Fix**: Publish `packages/ai/src` in the tarball, treat source-backed `node_modules/@workflow/*/src/*` files like external workspace source files when generating bundle imports, and extend deferred transitive step discovery to follow bare `workflow` / `@workflow/*` package imports during non-watch builds.
|
|
458
|
-
|
|
459
|
-
### Vercel Step Source Map Expectations Were Too Optimistic
|
|
460
|
-
|
|
461
|
-
Merging `main` also pulled in a newer `hasStepSourceMaps()` expectation for Vercel preview deployments. On this branch, the non-Next workbench previews still emit step stacks without source filenames like `99_e2e.ts` or `helpers.ts`, so the Vercel step-error assertions regressed across the BOA-backed workbench matrix even though runtime behavior was otherwise unchanged.
|
|
462
|
-
|
|
463
|
-
**Fix**: Revert the Vercel step source map expectation to the conservative branch behavior so preview e2e only asserts source filenames where this branch actually preserves them. Concretely, `hasStepSourceMaps()` returns `false` for *every* framework on Vercel deployments. Re-applying the blanket Vercel carve-out is what keeps the 11-framework `e2e-vercel-prod` matrix green while V2 source-map coverage catches up. The same pipeline regression also affects `nextjs-webpack` in local dev: pre-V2, webpack dev mode imported step sources directly so error stacks named `99_e2e.ts` / `helpers.ts`; under V2 the step bundle is inlined into the combined flow route and webpack's re-bundling collapses those filenames out of the dev-mode source maps. The helper now returns `false` for `nextjs-webpack` regardless of `DEV_TEST_CONFIG`.
|
|
464
|
-
|
|
465
|
-
**Follow-up**: Wire up consumable inline source maps in the V2 step bundle across the framework integrations — the BOA-backed ones (Astro, Express, Fastify, Hono, Nitro, Nuxt, Vite, plus the standalone `example`), `nextjs-turbopack`, and `nextjs-webpack` (both dev and prod). The plan is to let each builder's `createCombinedBundle()` call carry an esbuild source-map pipeline that survives the framework's downstream re-bundling step, and then re-introduce the per-framework matrix in `hasStepSourceMaps()` so error stacks correctly point at `99_e2e.ts` / `helpers.ts` everywhere. Tracking this as a deferred follow-up rather than blocking the V2 cutover, since the runtime behavior is unaffected — only the surfaced filenames in step error stacks differ.
|
|
466
|
-
|
|
467
|
-
### Community Worlds Still Used The Pre-`world.streams` API
|
|
468
|
-
|
|
469
|
-
The Redis community-world benchmark still loads an external world package that has not adopted the newer `world.streams.*` interface yet. Once the eager-processing changes exercised stream writes through the modern namespace consistently, that adapter started failing with `Cannot read properties of undefined (reading 'writeMulti')` before the benchmark could even start.
|
|
470
|
-
|
|
471
|
-
**Decision**: Community world adapters must implement the `world.streams.*` interface. The runtime legacy stream normalization (`normalizeLegacyWorld`) was removed. Community world e2e tests are skipped until the adapters are updated.
|
|
472
|
-
|
|
473
|
-
### Build Output API Flow Handler Drift
|
|
474
|
-
|
|
475
|
-
The Vercel Build Output API builder still emitted the combined flow function as `index.js`, but the surrounding metadata kept pointing at `index.mjs`. That mismatch meant BOA-based preview deployments published neither `/.well-known/workflow/v1/flow` nor the public manifest, so the Vercel production e2e suite collapsed into manifest `404` errors immediately after deployment.
|
|
476
|
-
|
|
477
|
-
**Fix**: Updated `packages/builders/src/vercel-build-output-api.ts` to point both `.vc-config.json` and manifest extraction at `flow.func/index.js`, which matches the CommonJS file the builder actually writes.
|
|
478
|
-
|
|
479
|
-
### Async World Loading Broke Custom Target Worlds
|
|
480
|
-
|
|
481
|
-
The first lazy-world-loading fix switched package resolution over to `createRequire(import.meta.url)` globally. That solved the Next.js bundling problem for built-in worlds, but it also made custom targets like `@workflow/world-postgres` resolve relative to `@workflow/core` instead of the consuming app. Local Postgres tests then failed at startup with `Cannot find module '@workflow/world-postgres'`.
|
|
482
|
-
|
|
483
|
-
**Fix**: The runtime now creates the package resolver lazily from `process.cwd()/package.json` when possible, falling back to `import.meta.url` only when the app root cannot be resolved. That keeps custom world modules app-relative without reintroducing the eager module-load failure in Next.js builds.
|
|
484
|
-
|
|
485
|
-
### Core Logger Still Pulled `debug` Into Webpack Flow Routes
|
|
486
|
-
|
|
487
|
-
Even after lazy world loading stopped eagerly importing `@workflow/world-vercel`, the generated Next.js webpack flow route still evaluated `packages/core/src/logger.ts` at module load. That file had a top-level `import debug from 'debug'`, which in turn pulled `debug/src/node` and its `tty` dynamic require into `/.well-known/workflow/v1/flow`. Webpack then failed during page-data collection with `Dynamic require of "tty" is not supported`.
|
|
488
|
-
|
|
489
|
-
**Fix**: Replace the static `debug` dependency in the core logger with lightweight `process.env.DEBUG` matching plus `console.debug`. That keeps verbose opt-in logging for local debugging without forcing webpack to bundle `debug` and its Node-only terminal helpers into the flow route.
|
|
490
|
-
|
|
491
|
-
### Deferred Next.js Builder Helper Drift After Merge
|
|
492
|
-
|
|
493
|
-
Merging `main` into the eager-processing branch pulled in a set of helper methods for copied-step import rewriting in `packages/next/src/builder-deferred.ts`, but the corresponding call sites were not present on this branch yet. That left `getRelativeImportSpecifier`, `getStepCopyFileName`, and `rewriteRelativeImportsForCopiedStep` orphaned, and `@workflow/next` failed to build with `TS6133` unused-private-member errors immediately after the merge.
|
|
494
|
-
|
|
495
|
-
**Fix**: Removed the orphaned helper methods during merge resolution and kept the existing deferred-builder behavior unchanged. The copied-step import-rewrite work should land as a complete change set rather than a partial backport from `main`.
|
|
496
|
-
|
|
497
|
-
### Next.js React Step Fixture and `eval('require(...)')`
|
|
498
|
-
|
|
499
|
-
The `nextjs-webpack` e2e suite still failed after the merge in `workflows/8_react_render.tsx`, where the step intentionally did `eval('require("react-dom/server")')` to avoid Next.js linting rules around importing `react-dom/server` directly. That pattern was brittle under webpack rebundling: even though the intermediate step bundle had a `createRequire(import.meta.url)` banner, the rebundled route still failed at runtime with `TypeError: require is not a function`.
|
|
500
|
-
|
|
501
|
-
**Fix**: Updated the React-rendering step fixture in both Next.js workbenches to use `await import('react-dom/server')` instead. The test still exercises server-side React rendering inside a step, but no longer depends on bundler-specific `eval('require(...)')` behavior.
|
|
192
|
+
Workflow code still runs in a Node.js VM for determinism and sandboxing. Step code runs in the Node.js host context. The only change is that both happen within the same function invocation.
|
|
502
193
|
|
|
503
|
-
|
|
194
|
+
### Bundle size and cold start
|
|
504
195
|
|
|
505
|
-
The
|
|
196
|
+
The combined bundle is larger (contains both step code and workflow VM code). Cold start time increases slightly. The reduction in total function invocations more than compensates.
|
|
506
197
|
|
|
507
|
-
|
|
508
|
-
|-----------------|---------------------|-----|
|
|
509
|
-
| Sequential steps (3 adds) | **1** | All steps execute inline |
|
|
510
|
-
| Sequential steps + WritableStream | **1** | Ops settle via flush waiter promises (500ms race) |
|
|
511
|
-
| Sleep (1s) + step | **2** | Sleep requires queue round-trip |
|
|
512
|
-
| Promise.all (2 steps) | **2-3** | Background step + inline replay after all steps done |
|
|
198
|
+
### Step retries
|
|
513
199
|
|
|
514
|
-
|
|
200
|
+
When an inline step fails with retries remaining:
|
|
515
201
|
|
|
516
|
-
|
|
202
|
+
- `RetryableError` with explicit `retryAfter` delay: Requeue to self with `stepId` and a delay.
|
|
203
|
+
- Transient errors with immediate retry: Requeue to self with `stepId` and a 1s delay.
|
|
204
|
+
- `FatalError`: Fail immediately.
|
|
517
205
|
|
|
518
|
-
|
|
206
|
+
### Encryption key resolution
|
|
519
207
|
|
|
520
|
-
|
|
208
|
+
Encryption keys are resolved once before the inline execution loop starts (after the run status is confirmed as `running`) and reused across all iterations. Background step executions resolve the key independently. The key does not change within a run.
|
|
521
209
|
|
|
522
|
-
###
|
|
210
|
+
### Module scope duplication in re-bundled output
|
|
523
211
|
|
|
524
|
-
|
|
212
|
+
Builders that rebundle the combined output into a single file (standalone Workflow CLI, Vercel Build Output API, and NestJS) produce a layout where esbuild creates isolated module scopes for each source module, even within the same output file. Without intervention, `registerStepFunction` and `getStepFunction` operate on different `Map` instances: steps are registered into one `Map` but looked up from another.
|
|
525
213
|
|
|
526
|
-
|
|
214
|
+
The step function registry and the step context storage are `globalThis` singletons (via `Symbol.for`) to ensure all module scopes share the same instances. The same pattern is used for the World singleton and the class serialization registry.
|
|
527
215
|
|
|
528
|
-
|
|
216
|
+
### Inline step execution with pending stream operations
|
|
529
217
|
|
|
530
|
-
|
|
531
|
-
- Removed `stepEntrypoint` re-export from `runtime.ts` (V2 doesn't use separate step routes)
|
|
532
|
-
- Lazy-loaded `getPort` via `createRequire` with opaque specifier to prevent `@workflow/utils/get-port` filesystem operations from being traced
|
|
533
|
-
- `getRuntimeRequire()` uses `process.cwd()` as primary resolution base (for custom world packages like `@workflow/world-postgres` that are app-level deps, not `@workflow/core` deps), with `import.meta.url` fallback
|
|
218
|
+
When a step's arguments or return value include serialized streams (e.g., `WritableStream` from `getWritable()`, or AI SDK streaming steps), the serialization layer creates background `flushablePipe` operations that pipe data to S3. These ops are tracked in an `ops` array and need to complete before the stream data is readable by external consumers.
|
|
534
219
|
|
|
535
|
-
|
|
220
|
+
In V1, each step ran in a separate function invocation. After the step completed, `waitUntil(ops)` kept the function alive to flush the ops. In V2, the inline execution loop continues immediately after the step body returns, so we need to know whether to keep looping or break out and let `waitUntil` flush.
|
|
536
221
|
|
|
537
|
-
|
|
222
|
+
`executeStep()` attempts a 500ms `Promise.race` between the ops settling and a timeout. If ops settle in time (data confirmed on server), it returns `hasPendingOps: false` and the V2 handler continues the inline loop. If ops don't settle in 500ms (e.g., `WritableStream` kept open across steps), it returns `hasPendingOps: true` and the V2 handler breaks the loop and queues a continuation so `waitUntil` can flush them.
|
|
538
223
|
|
|
539
|
-
|
|
224
|
+
**Follow-up**: Shrink the 500ms inline-ops budget once we have confidence that the flush-waiter path settles deterministically across all worlds. A signaled "ops drained" event from the world layer would let `executeStep()` proceed without the timeout in the common case.
|
|
540
225
|
|
|
541
|
-
|
|
542
|
-
2. The dynamic-import fallback inside `get-world-lazy.ts` builds the specifier `./world.js` at runtime to evade bundler tracing — but webpack inlines `get-world-lazy.js` into the route bundle, so the relative specifier resolves against `/var/task/<app>/.next/server/app/<route>/route.js` where no sibling `world.js` exists. Node throws `MODULE_NOT_FOUND`.
|
|
226
|
+
### Buffered stream flush with waiter promises
|
|
543
227
|
|
|
544
|
-
|
|
228
|
+
`WorkflowServerWritableStream` buffers writes and flushes via a 10ms `setTimeout` for batching. Naively, `write()` could return immediately after buffering, but that would cause the `flushablePipe`'s `pendingOps` counter to reach 0 before data actually reached the server: the V2 inline loop would see ops as settled prematurely and produce data-loss races on every step with `WritableStream` serialization.
|
|
545
229
|
|
|
546
|
-
|
|
230
|
+
`write()` returns a promise that resolves only after the scheduled flush completes. Multiple writes within the 10ms window still share a single batched HTTP request (the batching optimization is preserved). Each write registers a `{resolve, reject}` pair in a `flushWaiters` array. When the `setTimeout` fires and `flush()` completes the HTTP round-trip, all waiters are resolved (or rejected on error). This makes `pendingOps` accurately reflect server-side data state while keeping network-efficient batching.
|
|
547
231
|
|
|
548
|
-
|
|
549
|
-
- `workflow` → `./dist/workflow/world-init-stub.js` (empty, used by VM/step bundles)
|
|
232
|
+
### Lock-release polling interval
|
|
550
233
|
|
|
551
|
-
`
|
|
234
|
+
`flushablePipe`'s `pollWritableLock` / `pollReadableLock` use `setInterval` to detect when a user releases their stream lock without closing the stream, since the Web Streams API has no event for that state. The V2 step executor's `opsSettled` race waits for this poll to resolve after each writable-bearing step body returns, so the polling interval sits on the critical path of every streaming step.
|
|
552
235
|
|
|
553
|
-
|
|
236
|
+
The interval was lowered from 100ms to 10ms. Per-step wait drops from ~50ms average to ~5ms (scaling linearly with the number of writable-bearing steps in a workflow). Per-tick work is `writable.locked` plus a `getWriter()`/`releaseLock()` probe (microsecond-scale), so 10× more ticks is not measurable in practice.
|
|
554
237
|
|
|
555
|
-
|
|
238
|
+
**Follow-up**: Replace polling entirely with an event-driven release signal (wrap the writable returned from the `WritableStream` reviver with a writer that fires on `releaseLock()`), bringing the wait to ~0ms. This would also remove a source of timing drift between worlds with synchronous storage (`world-local`) and worlds with HTTP-deferred storage (`world-vercel`).
|
|
556
239
|
|
|
557
|
-
###
|
|
240
|
+
### Concurrent `step_started` and attempt counter
|
|
558
241
|
|
|
559
|
-
When
|
|
242
|
+
When the V2 handler dispatches N parallel steps as background messages, each background step completion queues a workflow continuation. Up to N continuations may replay concurrently, and each may attempt to start the same not-yet-completed step (since `step_started` succeeds for already-running steps). Each call atomically increments the `attempt` counter, so with N=5 parallel steps, the counter can reach 5 on the first genuine execution.
|
|
560
243
|
|
|
561
|
-
|
|
244
|
+
The max retries check in `executeStep()` only enforces when `step.error` exists, distinguishing actual retries (failed → retry with error) from concurrent first-attempt races (multiple handlers start the same step simultaneously without any prior failure). Concurrent starts are harmless since `step_completed` idempotency ensures only the first completion wins.
|
|
562
245
|
|
|
563
|
-
|
|
246
|
+
### Unconsumed event check two-phase drain
|
|
564
247
|
|
|
565
|
-
|
|
248
|
+
The `EventsConsumer`'s unconsumed event check uses a two-phase promise queue drain: yield once after the first drain (via `setTimeout(0)`) so cross-VM promise chains can append follow-up async work, then re-drain before checking. This improves timing for scenarios like `step_completed` → for-await loop resume → next hook hydration.
|
|
566
249
|
|
|
567
|
-
|
|
250
|
+
The check additionally arms a `DEFERRED_CHECK_DELAY_MS = 100` `setTimeout` after the second drain, since Node.js does not guarantee that `setTimeout(0)` fires after all cross-context microtasks settle. Any `subscribe()` call arriving during that 100ms window cancels the check via version invalidation + `clearTimeout`, so the delay only adds latency to genuine corruption, never to the happy path.
|
|
568
251
|
|
|
569
|
-
|
|
252
|
+
**Follow-up**: 100ms is a heuristic chosen empirically. A deterministic settlement signal (for example, a "VM idle" callback exposed by the workflow VM bridge that fires only after all pending cross-context promise chains have resolved) would let the consumer fire the unconsumed-event check immediately on quiescence instead of waiting for a wall-clock timeout.
|
|
570
253
|
|
|
571
|
-
|
|
254
|
+
### Lazy World loading
|
|
572
255
|
|
|
573
|
-
|
|
256
|
+
Static imports of `world-local` and `world-vercel` from the runtime caused two distinct build-time issues: Next.js production builds pulled both worlds (including their Node-only deps like `debug`'s `tty` requires) into the route module, and Turbopack's NFT (Node File Trace) errored on `process.cwd()` and dynamic `import()` patterns it couldn't statically analyze.
|
|
574
257
|
|
|
575
|
-
|
|
258
|
+
A `getWorldLazy()` accessor (backed by a `globalThis` `Symbol.for` cache) replaces the static import in step-side modules. This breaks the static import chain from step code to `world.ts`, preventing both worlds from being bundled into the step registrations.
|
|
576
259
|
|
|
577
|
-
|
|
260
|
+
Because tree-shaking can otherwise drop `world.ts`'s module-load registration entirely, a server-only side-effect module (`@workflow/core/runtime/world-init`) imports `./world.js` purely for its module-load side effect. It's wired via package conditions:
|
|
578
261
|
|
|
579
|
-
|
|
262
|
+
- `default` maps to the real module and loads `world.ts`.
|
|
263
|
+
- `workflow` maps to an empty stub used by VM and step bundles.
|
|
580
264
|
|
|
581
|
-
|
|
582
|
-
|-----------|-----------|--------|
|
|
583
|
-
| Unit Tests | core | 581/581 |
|
|
584
|
-
| Embedded Tests | world-testing | 9/9 (including inline execution) |
|
|
585
|
-
| Local Dev | 14 frameworks | All pass |
|
|
586
|
-
| Local Prod | 14 configurations | All pass |
|
|
587
|
-
| Postgres | 14 frameworks | All pass |
|
|
588
|
-
| Vercel Prod | 11 frameworks | All pass |
|
|
589
|
-
| Vercel Deployments | 15 projects | All succeed |
|
|
590
|
-
| Community Worlds | Turso, MongoDB, Redis | All pass |
|
|
591
|
-
| Windows | e2e | Pass |
|
|
265
|
+
This guarantees the world is loaded for routes that consume `start()` without going through the queue-driven flow handler first, while keeping `world.ts` and its server-only deps out of the workflow sandbox bundle.
|
|
592
266
|
|
|
593
|
-
|
|
267
|
+
### Community Worlds and the `world.streams` API
|
|
594
268
|
|
|
595
|
-
|
|
269
|
+
Community world adapters must implement the `world.streams.*` interface. The runtime legacy stream normalization was removed as part of this work.
|