workflow 5.0.0-beta.9 → 5.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +68 -23
- package/dist/api-workflow.d.ts +3 -1
- package/dist/api-workflow.d.ts.map +1 -1
- package/dist/api-workflow.js +2 -1
- package/dist/api.d.ts +5 -4
- package/dist/api.d.ts.map +1 -1
- package/dist/api.js +6 -7
- package/dist/index.d.ts +1 -0
- package/dist/index.d.ts.map +1 -1
- package/dist/index.js +6 -1
- package/dist/internal/builtins.d.ts +4 -4
- package/dist/internal/builtins.js +6 -6
- package/dist/internal/errors.d.ts +1 -1
- package/dist/internal/errors.d.ts.map +1 -1
- package/dist/internal/errors.js +2 -2
- package/dist/nest-builder.d.ts +2 -0
- package/dist/nest-builder.d.ts.map +1 -0
- package/dist/nest-builder.js +2 -0
- package/dist/nest-vercel-builder.d.ts +2 -0
- package/dist/nest-vercel-builder.d.ts.map +1 -0
- package/dist/nest-vercel-builder.js +2 -0
- package/dist/runtime.d.ts +2 -1
- package/dist/runtime.d.ts.map +1 -1
- package/dist/runtime.js +4 -1
- package/docs/advanced/dynamic-workflows.mdx +224 -0
- package/docs/ai/chat-session-modeling.mdx +176 -422
- package/docs/ai/defining-tools.mdx +6 -7
- package/docs/ai/human-in-the-loop.mdx +11 -11
- package/docs/ai/index.mdx +67 -72
- package/docs/ai/message-queueing.mdx +71 -110
- package/docs/ai/meta.json +1 -0
- package/docs/ai/resumable-streams.mdx +40 -28
- package/docs/ai/sleep-and-delays.mdx +10 -10
- package/docs/ai/streaming-updates-from-tools.mdx +6 -6
- package/docs/api-reference/index.mdx +25 -1
- package/docs/api-reference/meta.json +8 -0
- package/docs/api-reference/vitest/index.mdx +68 -15
- package/docs/api-reference/workflow/create-hook.mdx +166 -10
- package/docs/api-reference/workflow/create-webhook.mdx +16 -15
- package/docs/api-reference/workflow/define-hook.mdx +37 -33
- package/docs/api-reference/workflow/fatal-error.mdx +30 -8
- package/docs/api-reference/workflow/fetch.mdx +14 -10
- package/docs/api-reference/workflow/get-step-metadata.mdx +2 -2
- package/docs/api-reference/workflow/get-workflow-metadata.mdx +3 -3
- package/docs/api-reference/workflow/get-writable.mdx +7 -7
- package/docs/api-reference/workflow/index.mdx +3 -3
- package/docs/api-reference/workflow/retryable-error.mdx +1 -1
- package/docs/api-reference/workflow/set-attributes.mdx +63 -0
- package/docs/api-reference/workflow/sleep.mdx +4 -4
- package/docs/api-reference/workflow-ai/durable-agent.mdx +63 -101
- package/docs/api-reference/workflow-ai/index.mdx +5 -5
- package/docs/api-reference/workflow-ai/workflow-chat-transport.mdx +67 -24
- package/docs/api-reference/workflow-api/get-hook-by-token.mdx +28 -12
- package/docs/api-reference/workflow-api/get-run.mdx +43 -8
- package/docs/api-reference/workflow-api/index.mdx +8 -9
- package/docs/api-reference/workflow-api/register-lifecycle-hooks.mdx +87 -0
- package/docs/api-reference/workflow-api/resume-hook.mdx +73 -12
- package/docs/api-reference/workflow-api/resume-webhook.mdx +11 -9
- package/docs/api-reference/workflow-api/start.mdx +107 -12
- package/docs/api-reference/workflow-astro/index.mdx +18 -0
- package/docs/api-reference/workflow-astro/meta.json +4 -0
- package/docs/api-reference/workflow-astro/workflow.mdx +45 -0
- package/docs/api-reference/workflow-errors/entity-conflict-error.mdx +4 -4
- package/docs/api-reference/workflow-errors/hook-conflict-error.mdx +60 -0
- package/docs/api-reference/workflow-errors/hook-force-claimed-error.mdx +70 -0
- package/docs/api-reference/workflow-errors/hook-not-found-error.mdx +8 -8
- package/docs/api-reference/workflow-errors/index.mdx +91 -0
- package/docs/api-reference/workflow-errors/meta.json +7 -0
- package/docs/api-reference/workflow-errors/precondition-failed-error.mdx +68 -0
- package/docs/api-reference/workflow-errors/run-expired-error.mdx +2 -2
- package/docs/api-reference/workflow-errors/run-not-supported-error.mdx +58 -0
- package/docs/api-reference/workflow-errors/step-not-registered-error.mdx +5 -5
- package/docs/api-reference/workflow-errors/throttle-error.mdx +2 -2
- package/docs/api-reference/workflow-errors/too-early-error.mdx +2 -2
- package/docs/api-reference/workflow-errors/workflow-error.mdx +52 -0
- package/docs/api-reference/workflow-errors/workflow-not-registered-error.mdx +5 -6
- package/docs/api-reference/workflow-errors/workflow-run-cancelled-error.mdx +13 -6
- package/docs/api-reference/workflow-errors/workflow-run-failed-error.mdx +12 -5
- package/docs/api-reference/workflow-errors/workflow-run-not-completed-error.mdx +58 -0
- package/docs/api-reference/workflow-errors/workflow-run-not-found-error.mdx +4 -4
- package/docs/api-reference/workflow-errors/workflow-runtime-error.mdx +58 -0
- package/docs/api-reference/workflow-errors/workflow-world-error.mdx +8 -8
- package/docs/api-reference/workflow-globals.mdx +15 -11
- package/docs/api-reference/workflow-nest/configure-workflow-controller.mdx +37 -0
- package/docs/api-reference/workflow-nest/index.mdx +31 -0
- package/docs/api-reference/workflow-nest/meta.json +9 -0
- package/docs/api-reference/workflow-nest/nest-local-builder.mdx +64 -0
- package/docs/api-reference/workflow-nest/workflow-controller.mdx +46 -0
- package/docs/api-reference/workflow-nest/workflow-module.mdx +134 -0
- package/docs/api-reference/workflow-next/with-workflow.mdx +39 -17
- package/docs/api-reference/workflow-nitro/index.mdx +60 -0
- package/docs/api-reference/workflow-nuxt/index.mdx +48 -0
- package/docs/api-reference/workflow-observability/hydrate-data.mdx +35 -0
- package/docs/api-reference/workflow-observability/hydrate-resource-io.mdx +62 -0
- package/docs/api-reference/workflow-observability/index.mdx +62 -0
- package/docs/api-reference/workflow-observability/meta.json +11 -0
- package/docs/api-reference/workflow-observability/observability-revivers.mdx +50 -0
- package/docs/api-reference/workflow-observability/parse-class-name.mdx +41 -0
- package/docs/api-reference/workflow-observability/parse-step-name.mdx +40 -0
- package/docs/api-reference/workflow-observability/parse-workflow-name.mdx +55 -0
- package/docs/api-reference/workflow-runtime/create-world.mdx +39 -0
- package/docs/api-reference/workflow-runtime/get-world-handlers.mdx +44 -0
- package/docs/api-reference/{workflow-api → workflow-runtime}/get-world.mdx +11 -14
- package/docs/api-reference/workflow-runtime/health-check.mdx +51 -0
- package/docs/api-reference/workflow-runtime/index.mdx +41 -0
- package/docs/api-reference/workflow-runtime/meta.json +12 -0
- package/docs/api-reference/workflow-runtime/set-world.mdx +51 -0
- package/docs/api-reference/workflow-runtime/workflow-entrypoint.mdx +43 -0
- package/docs/api-reference/workflow-runtime/world/analytics.mdx +315 -0
- package/docs/api-reference/workflow-runtime/world/index.mdx +60 -0
- package/docs/api-reference/workflow-runtime/world/meta.json +4 -0
- package/docs/api-reference/workflow-runtime/world/queue.mdx +88 -0
- package/docs/api-reference/{workflow-api → workflow-runtime}/world/storage.mdx +104 -35
- package/docs/api-reference/{workflow-api → workflow-runtime}/world/streams.mdx +8 -8
- package/docs/api-reference/workflow-serde/index.mdx +1 -2
- package/docs/api-reference/workflow-serde/workflow-deserialize.mdx +3 -4
- package/docs/api-reference/workflow-serde/workflow-serialize.mdx +8 -8
- package/docs/api-reference/workflow-sveltekit/index.mdx +18 -0
- package/docs/api-reference/workflow-sveltekit/meta.json +4 -0
- package/docs/api-reference/workflow-sveltekit/workflow-plugin.mdx +42 -0
- package/docs/api-reference/workflow-vite/index.mdx +18 -0
- package/docs/api-reference/workflow-vite/meta.json +4 -0
- package/docs/api-reference/workflow-vite/workflow.mdx +48 -0
- package/docs/changelog/attributes-mvp.mdx +53 -41
- package/docs/changelog/batched-event-writes.mdx +79 -0
- package/docs/changelog/eager-processing.mdx +110 -436
- package/docs/changelog/index.mdx +4 -2
- package/docs/changelog/lazy-event-creation.md +127 -0
- package/docs/changelog/lazy-hook-resume.mdx +78 -0
- package/docs/changelog/meta.json +11 -1
- package/docs/changelog/resilient-resume.mdx +32 -0
- package/docs/changelog/resilient-start.mdx +33 -285
- package/docs/changelog/step-message-ownership.mdx +360 -0
- package/docs/changelog/turbo-mode.md +87 -0
- package/docs/comparisons/index.mdx +66 -0
- package/docs/comparisons/meta.json +11 -0
- package/docs/comparisons/workflow-sdk-vs-aws-agentcore.mdx +55 -0
- package/docs/comparisons/workflow-sdk-vs-aws-step-functions.mdx +111 -0
- package/docs/comparisons/workflow-sdk-vs-cloudflare-workflows.mdx +71 -0
- package/docs/comparisons/workflow-sdk-vs-inngest.mdx +102 -0
- package/docs/comparisons/workflow-sdk-vs-temporal.mdx +123 -0
- package/docs/comparisons/workflow-sdk-vs-trigger-dev.mdx +104 -0
- package/docs/configuration/build-and-diagnostics.mdx +79 -0
- package/docs/configuration/cli-and-web-ui.mdx +241 -0
- package/docs/configuration/framework-options.mdx +165 -0
- package/docs/configuration/index.mdx +32 -0
- package/docs/configuration/meta.json +12 -0
- package/docs/configuration/runtime-tuning.mdx +399 -0
- package/docs/configuration/worlds.mdx +315 -0
- package/docs/cookbook/advanced/child-workflows.mdx +33 -25
- package/docs/cookbook/advanced/publishing-libraries.mdx +65 -56
- package/docs/cookbook/advanced/serializable-steps.mdx +48 -68
- package/docs/cookbook/advanced/upgrading-workflows.mdx +35 -31
- package/docs/cookbook/agent-patterns/agent-cancellation.mdx +78 -60
- package/docs/cookbook/agent-patterns/durable-agent.mdx +23 -135
- package/docs/cookbook/agent-patterns/human-in-the-loop.mdx +180 -195
- package/docs/cookbook/common-patterns/batching.mdx +20 -14
- package/docs/cookbook/common-patterns/idempotency.mdx +41 -53
- package/docs/cookbook/common-patterns/rate-limiting.mdx +8 -4
- package/docs/cookbook/common-patterns/saga.mdx +23 -19
- package/docs/cookbook/common-patterns/scheduling.mdx +30 -22
- package/docs/cookbook/common-patterns/sequential-and-parallel.mdx +29 -25
- package/docs/cookbook/common-patterns/timeouts.mdx +26 -21
- package/docs/cookbook/common-patterns/webhooks.mdx +10 -6
- package/docs/cookbook/common-patterns/workflow-composition.mdx +27 -17
- package/docs/cookbook/index.mdx +22 -22
- package/docs/cookbook/integrations/ai-sdk.mdx +63 -48
- package/docs/cookbook/integrations/chat-sdk.mdx +46 -33
- package/docs/cookbook/integrations/sandbox.mdx +58 -45
- package/docs/deploying.mdx +106 -0
- package/docs/errors/abort-signal-timeout-in-workflow.mdx +16 -12
- package/docs/errors/corrupted-event-log.mdx +39 -18
- package/docs/errors/deployment-mismatch.mdx +71 -0
- package/docs/errors/fetch-in-workflow.mdx +15 -14
- package/docs/errors/hook-conflict.mdx +38 -11
- package/docs/errors/hook-force-claimed.mdx +96 -0
- package/docs/errors/index.mdx +24 -37
- package/docs/errors/node-js-module-in-workflow.mdx +9 -5
- package/docs/errors/replay-divergence.mdx +27 -0
- package/docs/errors/run-expired.mdx +85 -0
- package/docs/errors/runtime-decryption-failed.mdx +77 -0
- package/docs/errors/serialization-failed.mdx +44 -12
- package/docs/errors/start-invalid-workflow-function.mdx +9 -5
- package/docs/errors/step-executed-multiple-times.mdx +23 -0
- package/docs/errors/step-not-registered.mdx +6 -6
- package/docs/errors/timeout-in-workflow.mdx +12 -8
- package/docs/errors/webhook-invalid-respond-with-value.mdx +18 -18
- package/docs/errors/webhook-response-not-sent.mdx +20 -16
- package/docs/errors/workflow-not-registered.mdx +5 -5
- package/docs/foundations/cancellation.mdx +31 -32
- package/docs/foundations/errors-and-retries.mdx +54 -11
- package/docs/foundations/hooks.mdx +98 -35
- package/docs/foundations/idempotency.mdx +267 -12
- package/docs/foundations/index.mdx +1 -26
- package/docs/foundations/serialization.mdx +22 -22
- package/docs/foundations/starting-workflows.mdx +104 -30
- package/docs/foundations/streaming.mdx +108 -60
- package/docs/foundations/versioning.mdx +4 -4
- package/docs/foundations/workflows-and-steps.mdx +9 -9
- package/docs/getting-started/astro.mdx +22 -18
- package/docs/getting-started/express.mdx +15 -11
- package/docs/getting-started/fastify.mdx +15 -11
- package/docs/getting-started/hono.mdx +15 -11
- package/docs/getting-started/index.mdx +10 -3
- package/docs/getting-started/meta.json +3 -1
- package/docs/getting-started/nestjs.mdx +264 -21
- package/docs/getting-started/next.mdx +18 -14
- package/docs/getting-started/nitro.mdx +22 -18
- package/docs/getting-started/nuxt.mdx +15 -11
- package/docs/getting-started/python.mdx +190 -41
- package/docs/getting-started/react-router/index.mdx +33 -0
- package/docs/getting-started/react-router/meta.json +5 -0
- package/docs/getting-started/react-router/v7.mdx +237 -0
- package/docs/getting-started/react-router/v8.mdx +232 -0
- package/docs/getting-started/sveltekit.mdx +20 -16
- package/docs/getting-started/tanstack-start.mdx +17 -13
- package/docs/getting-started/vite.mdx +15 -11
- package/docs/how-it-works/cancellation.mdx +63 -63
- package/docs/how-it-works/code-transform.mdx +82 -66
- package/docs/how-it-works/encryption.mdx +30 -26
- package/docs/how-it-works/event-sourcing.mdx +132 -35
- package/docs/how-it-works/framework-integrations.mdx +96 -337
- package/docs/how-it-works/understanding-directives.mdx +22 -22
- package/docs/internal/index.mdx +6 -4
- package/docs/internal/meta.json +6 -1
- package/docs/internal/nitro-native-build.mdx +38 -0
- package/docs/internal/nitro-web-ui.mdx +24 -0
- package/docs/internal/serializable-abort-controller.mdx +7 -7
- package/docs/meta.json +4 -2
- package/docs/observability/attributes.mdx +91 -21
- package/docs/observability/index.mdx +29 -15
- package/docs/observability/lifecycle-hooks.mdx +95 -0
- package/docs/observability/meta.json +1 -1
- package/docs/observability/retention.mdx +95 -0
- package/docs/observability/tracing.mdx +124 -0
- package/docs/testing/index.mdx +120 -38
- package/docs/testing/server-based.mdx +10 -10
- package/docs/whats-new.mdx +190 -0
- package/docs/worlds/building-a-world.mdx +538 -0
- package/docs/worlds/local.mdx +129 -0
- package/docs/worlds/meta.json +10 -0
- package/docs/worlds/postgres.mdx +424 -0
- package/docs/worlds/upgrading-to-v5.mdx +162 -0
- package/docs/worlds/vercel.mdx +345 -0
- package/package.json +17 -14
- package/docs/api-reference/workflow/experimental-set-attributes.mdx +0 -63
- package/docs/api-reference/workflow-api/world/index.mdx +0 -58
- package/docs/api-reference/workflow-api/world/meta.json +0 -4
- package/docs/api-reference/workflow-api/world/observability.mdx +0 -164
- package/docs/api-reference/workflow-api/world/queue.mdx +0 -86
- package/docs/deploying/building-a-world.mdx +0 -251
- package/docs/deploying/index.mdx +0 -95
- package/docs/deploying/meta.json +0 -4
- package/docs/deploying/world/local-world.mdx +0 -84
- package/docs/deploying/world/meta.json +0 -4
- package/docs/deploying/world/postgres-world.mdx +0 -224
- package/docs/deploying/world/vercel-world.mdx +0 -181
- package/docs/migration-guides/index.mdx +0 -34
- package/docs/migration-guides/meta.json +0 -9
- package/docs/migration-guides/migrating-from-aws-step-functions.mdx +0 -358
- package/docs/migration-guides/migrating-from-inngest.mdx +0 -304
- package/docs/migration-guides/migrating-from-temporal.mdx +0 -313
- package/docs/migration-guides/migrating-from-trigger-dev.mdx +0 -328
|
@@ -1,39 +1,60 @@
|
|
|
1
1
|
---
|
|
2
2
|
title: corrupted-event-log
|
|
3
|
-
description: The workflow's event log contains an event that
|
|
3
|
+
description: The workflow's event log contains an event that cannot be processed or a stored payload that cannot be read.
|
|
4
4
|
type: troubleshooting
|
|
5
|
-
summary: Resolve corrupted event log errors caused by
|
|
5
|
+
summary: Resolve corrupted event log errors caused by invalid events or unreadable stored payloads.
|
|
6
6
|
prerequisites:
|
|
7
7
|
- /docs/foundations/workflows-and-steps
|
|
8
8
|
related:
|
|
9
9
|
- /docs/foundations/errors-and-retries
|
|
10
10
|
---
|
|
11
11
|
|
|
12
|
-
This error occurs when the Workflow runtime
|
|
12
|
+
This error occurs when the Workflow runtime cannot safely replay the event log. The log may be in an invalid state, such as an orphaned event or one no consumer can attribute to anything the workflow did, or it may reference a stored payload that the World can no longer read.
|
|
13
13
|
|
|
14
|
-
This is a **workflow-level fatal error**. It cannot be caught or handled inside your workflow code.
|
|
14
|
+
This is a **workflow-level fatal error**. It cannot be caught or handled inside your workflow code. The runtime retries transient replay divergence automatically, but an unreadable stored payload is terminal immediately because replaying cannot restore it.
|
|
15
15
|
|
|
16
|
-
## Error
|
|
16
|
+
## Error message
|
|
17
17
|
|
|
18
|
+
For replay divergence:
|
|
19
|
+
|
|
20
|
+
```text
|
|
21
|
+
Workflow replay diverged <divergenceCount> times after <maxRecoveryReplays> recovery replays; latest divergent event was <eventId>; divergent event ids: <eventId>, <eventId>, ... Last divergence: <details>
|
|
18
22
|
```
|
|
19
|
-
|
|
23
|
+
|
|
24
|
+
The `divergent event ids` list has one entry per divergence in the recovery chain, oldest first. Every recovery replay diverging at the same event points at a fixed disagreement between the log and the code; ids that wander point at a race with another writer.
|
|
25
|
+
|
|
26
|
+
`<details>` is the last divergence's own message. When the replay could not place an event, it names the invocation that was pending under that event's position at the time, and where the replay's walk over the log stood:
|
|
27
|
+
|
|
28
|
+
```text
|
|
29
|
+
Replay could not consume event: eventType=wait_created, correlationId=wait_<id>, eventId=<eventId>. pending at this id: step <stepName> (step_<id>). consumer: index=<n>, length=<n>, parked=<n>, lastConsumed=<eventId>
|
|
20
30
|
```
|
|
21
31
|
|
|
22
|
-
|
|
32
|
+
Steps, sleeps and hooks draw their ids from one sequence, so `pending at this id` reports the entity the replay put at that position, whatever its kind. In the example, the log recorded a sleep where this replay reached a step named `<stepName>`.
|
|
33
|
+
|
|
34
|
+
For an unreadable stored payload:
|
|
35
|
+
|
|
36
|
+
```text
|
|
37
|
+
the event log references a payload that no longer exists in storage: <details>
|
|
38
|
+
```
|
|
39
|
+
|
|
40
|
+
## Why this happens
|
|
41
|
+
|
|
42
|
+
Workflows persist their progress as an ordered event log. During replay, the runtime processes each event in sequence. Every event must be consumed by a matching callback, such as a step or sleep waiting for its result. An event no callback ever claims is one the runtime would have to drop to finish the run, so it fails the run instead of returning a result that silently ignored it.
|
|
23
43
|
|
|
24
|
-
|
|
44
|
+
A delivery written from outside the replay, such as a hook firing or a step completing on another invocation, can land ahead of the events the replay is writing itself. That is ordinary concurrency rather than corruption, so the runtime holds such an event and offers it to each consumer the replay registers afterwards. The failure comes only when the workflow function returns while an event is still held, at which point no consumer can ever appear. A replay that suspends still holding one reports it on the span (`workflow.events.parked.count`, `.event_id`, `.event_type`) and leaves the decision to the replay that follows.
|
|
25
45
|
|
|
26
|
-
|
|
46
|
+
Before failing on divergence, the runtime retries the replay and surfaces this terminal error only if replay still cannot recover. It does not retry a payload that the World reports as permanently missing.
|
|
27
47
|
|
|
28
48
|
Common scenarios that produce this error:
|
|
29
49
|
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
|
|
50
|
+
- **An unclaimed event that repeats nothing**: A duplicate of a kind the log already records for that entity is read past rather than failing the run, so a second `step_completed` or `wait_completed` is not this error (see [Duplicate Events](/docs/how-it-works/event-sourcing#duplicate-events)). What fails is an unclaimed event with no earlier counterpart to defer to: a `step_started` behind a `step_completed` on a log that never recorded a `step_started`, for instance. No consumer remains for the step, and there is no earlier event of that kind the replay could be reading instead.
|
|
51
|
+
- **Orphaned events**: A `step_completed` or `wait_completed` event whose `correlationId` doesn't match any step or sleep in the workflow code, so the replay reaches its end still holding it.
|
|
52
|
+
- **A hole in the log**: Events are numbered by their position in the run's log, and those positions are dense, so a position below the log's highest that holds no event means the log the replay loaded is incomplete. The runtime cannot tell a position no write ever occupied from one whose event it failed to read, so it refuses to replay rather than produce a result that may be silently wrong. See [`WORKFLOW_SLOT_GAP_CHECK`](/docs/configuration/runtime-tuning#workflow_slot_gap_check).
|
|
53
|
+
- **An unreadable stored payload**: An event row still references a payload object, but the World reports that the object no longer exists in its storage. The same log would fail on every replay, so the run fails immediately instead of retrying forever.
|
|
33
54
|
|
|
34
|
-
## What
|
|
55
|
+
## What to do
|
|
35
56
|
|
|
36
|
-
This error indicates a bug in the Workflow SDK or Workflow server
|
|
57
|
+
This error indicates a bug in the Workflow SDK or Workflow server, not in your workflow code. Your workflow code does not need to change. Follow these steps to resolve the issue:
|
|
37
58
|
|
|
38
59
|
### 1. Upgrade to the latest `workflow` package
|
|
39
60
|
|
|
@@ -45,18 +66,18 @@ npm install workflow@latest
|
|
|
45
66
|
|
|
46
67
|
### 2. Retry the failed run
|
|
47
68
|
|
|
48
|
-
|
|
69
|
+
If this error reports replay divergence, automatic replay recovery has already been exhausted. If it reports an unreadable payload, recovery cannot recreate that payload. In either case, the run has been marked as `failed`. You can re-run the workflow using the **Re-run** button in the Workflow Dashboard; a re-run starts a new run with a new event log.
|
|
49
70
|
|
|
50
71
|
### 3. Report the issue
|
|
51
72
|
|
|
52
|
-
If the error persists after upgrading,
|
|
73
|
+
If the error persists after upgrading, [open an issue on GitHub](https://github.com/vercel/workflow/issues/new) so we can investigate and fix the underlying bug. Include the following details to help us diagnose the problem:
|
|
53
74
|
|
|
54
75
|
- The version of the `workflow` package you are using
|
|
55
76
|
- The run ID(s) of the affected workflow run(s)
|
|
56
|
-
- The error message
|
|
77
|
+
- The complete error message, including any `eventType`, `correlationId`, `eventId`, or payload details
|
|
57
78
|
- Any details about the event log or the workflow that triggered the error
|
|
58
79
|
|
|
59
|
-
## This
|
|
80
|
+
## This error cannot be caught
|
|
60
81
|
|
|
61
82
|
Unlike other workflow errors, a corrupted event log error is **not catchable** inside your workflow function. Because the event log itself is invalid, the runtime cannot safely continue executing any user code. The entire run fails immediately and is marked as `failed`.
|
|
62
83
|
|
|
@@ -0,0 +1,71 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: deployment-mismatch
|
|
3
|
+
description: A workflow run was delivered to a deployment other than the one it is pinned to.
|
|
4
|
+
type: troubleshooting
|
|
5
|
+
summary: Understand how Workflow recovers from a misrouted delivery, and why a run eventually fails with DEPLOYMENT_MISMATCH.
|
|
6
|
+
prerequisites:
|
|
7
|
+
- /docs/foundations/workflows-and-steps
|
|
8
|
+
related:
|
|
9
|
+
- /docs/foundations/versioning
|
|
10
|
+
- /docs/errors/runtime-decryption-failed
|
|
11
|
+
- /docs/foundations/errors-and-retries
|
|
12
|
+
---
|
|
13
|
+
|
|
14
|
+
Every run is pinned to a single deployment when it starts. When a queued workflow or step callback is delivered to a **different** deployment, Workflow does not execute it there. Instead it re-routes the message to the deployment the run is pinned to, and only if the run keeps arriving elsewhere does it fail with the `DEPLOYMENT_MISMATCH` classification.
|
|
15
|
+
|
|
16
|
+
This is an SDK/runtime signal, not an error thrown by your workflow code, and it is not catchable inside a workflow function.
|
|
17
|
+
|
|
18
|
+
## Error message
|
|
19
|
+
|
|
20
|
+
```text
|
|
21
|
+
Workflow run "wrun_..." is pinned to deployment "dpl_A", but was received by deployment "dpl_B". The runtime re-routed the message to "dpl_A" 3 times and it kept arriving elsewhere, so the run was stopped to protect against code-skew errors. Verify that the run's deployment is still available and that queue callbacks are routed to it.
|
|
22
|
+
```
|
|
23
|
+
|
|
24
|
+
When the queue definitively reports that the run's deployment cannot be reached (it was deleted, or aged out of its retention window), no re-route is possible and the message omits the re-routing clause. Transient or unknown publishing failures leave the current delivery unacknowledged so the queue can redeliver it; they do not fail the run or consume this recovery budget.
|
|
25
|
+
|
|
26
|
+
## Why a run is pinned
|
|
27
|
+
|
|
28
|
+
A run's deployment is chosen once, at [`start()`](/docs/api-reference/workflow-api/start):
|
|
29
|
+
|
|
30
|
+
- By default it is the deployment that called `start()`. See [Versioning](/docs/foundations/versioning) for why runs are pinned this way.
|
|
31
|
+
- With `start(workflow, args, { deploymentId })` it is the id you pass, so a run can deliberately target a deployment other than the one that created it.
|
|
32
|
+
- With `deploymentId: "latest"` it is the most recent deployment for the current environment, resolved at start time.
|
|
33
|
+
|
|
34
|
+
Whichever it is, that `deploymentId` is recorded on the run, and every subsequent workflow replay and step execution must happen on that deployment. Continuing on a different one is unsafe:
|
|
35
|
+
|
|
36
|
+
1. **Code skew.** The workflow and step bundles on the receiving deployment may not match the code that produced the run's recorded history, so replay could diverge or produce incorrect results.
|
|
37
|
+
2. **Encryption.** Step inputs and other event-log payloads are encrypted with a per-run key derived from the pinned deployment's key material. A different deployment derives the wrong key and cannot decrypt them, previously the source of a confusing [runtime-decryption-failed](/docs/errors/runtime-decryption-failed) that exhausted retries with no clear cause.
|
|
38
|
+
|
|
39
|
+
So the runtime checks the pinned deployment before it executes anything, and `DEPLOYMENT_MISMATCH` names the result, instead of the mismatch surfacing later as an unrelated decryption failure.
|
|
40
|
+
|
|
41
|
+
## Automatic recovery
|
|
42
|
+
|
|
43
|
+
A deployment that receives a run it does not own first tries to fix the delivery rather than fail the run:
|
|
44
|
+
|
|
45
|
+
1. It re-enqueues the message **explicitly addressed** to the run's own deployment. This is strictly better-addressed than the send that misrouted, which inherited the producing deployment's ambient id.
|
|
46
|
+
2. Delivery is delayed with a short exponential backoff (1s, 2s, 4s).
|
|
47
|
+
3. If the run keeps arriving at the wrong deployment, the run is failed with `DEPLOYMENT_MISMATCH` after `WORKFLOW_DEPLOYMENT_MISMATCH_MAX_RETRIES` attempts (default `3`). Set it to `0` to fail on the first misrouted delivery instead.
|
|
48
|
+
|
|
49
|
+
Nothing is executed on the wrong deployment during recovery: no workflow code, no step body, no `step_started`, and no hook resume. Whatever the delivery was carrying travels with it, so a pending step keeps its identity and a hook resume keeps its payload: they run on the deployment that can actually decrypt them.
|
|
50
|
+
|
|
51
|
+
Recovery attempts do not create events on the run, so a run that self-heals looks completely normal. They are reported on the invocation's trace span (`workflow.deployment.pinned_id`, `workflow.deployment_mismatch.retry_count`, `workflow.deployment_mismatch.recovered`) and as a runtime warning in your function logs.
|
|
52
|
+
|
|
53
|
+
## What to do
|
|
54
|
+
|
|
55
|
+
- **Re-run from the current deployment.** Trigger the workflow again from your latest deployment (or use the **Re-run** button in the Workflow Dashboard). The new run is pinned to the current deployment.
|
|
56
|
+
- **Keep a run's deployment available** for the lifetime of that run. A run whose deployment has been deleted or has aged out cannot be resumed and must be re-run: recovery cannot help, so these fail on the first misrouted delivery. This applies to runs started with an explicit `deploymentId` too: pinning a run to an older deployment keeps it dependent on that deployment for its whole lifetime.
|
|
57
|
+
- **Report it** if the pinned deployment was still available. Include both deployment IDs and the run ID from the error message, plus the trace span attributes above. A run that failed this way despite a reachable target is a routing fault worth investigating rather than something to work around.
|
|
58
|
+
|
|
59
|
+
## This error cannot be caught
|
|
60
|
+
|
|
61
|
+
Like other runtime signals, `DEPLOYMENT_MISMATCH` is **not catchable** inside your workflow function: the run is failed before any workflow or step code executes on the receiving deployment. Check the run status from outside instead:
|
|
62
|
+
|
|
63
|
+
```typescript lineNumbers
|
|
64
|
+
import { getRun } from "workflow/api";
|
|
65
|
+
|
|
66
|
+
const run = getRun("wrun_abc123");
|
|
67
|
+
const status = await run.status;
|
|
68
|
+
if (status === "failed") {
|
|
69
|
+
console.error("Run failed");
|
|
70
|
+
}
|
|
71
|
+
```
|
|
@@ -9,21 +9,25 @@ related:
|
|
|
9
9
|
- /docs/api-reference/workflow/fetch
|
|
10
10
|
---
|
|
11
11
|
|
|
12
|
+
<CopyPrompt
|
|
13
|
+
text="Fix `fetch` usage inside workflow functions. Search workflow files for direct global `fetch(...)` calls and libraries such as AI SDK calls that use fetch. For basic HTTP calls inside a `"use workflow"` function, import `{ fetch }` from `workflow` and replace the global call. For SDK/client calls that need normal Node.js or provider behavior, move the call into a helper function with `"use step"` and call that step from the workflow. Keep all step inputs and outputs serializable. Verify the workflow starts and replays without the fetch-in-workflow error."
|
|
14
|
+
/>
|
|
15
|
+
|
|
12
16
|
This error occurs when you try to use `fetch()` directly in a workflow function, or when a library (like the AI SDK) tries to call `fetch()` under the hood.
|
|
13
17
|
|
|
14
|
-
## Error
|
|
18
|
+
## Error message
|
|
15
19
|
|
|
16
|
-
```
|
|
20
|
+
```text
|
|
17
21
|
Global "fetch" is unavailable in workflow functions. Use the "fetch" step function from "workflow" to make HTTP requests.
|
|
18
22
|
```
|
|
19
23
|
|
|
20
|
-
## Why
|
|
24
|
+
## Why this happens
|
|
21
25
|
|
|
22
26
|
Workflow functions run in a sandboxed environment without direct access to `fetch()`.
|
|
23
27
|
|
|
24
28
|
Many libraries make HTTP requests under the hood. For example, the AI SDK's `generateText()` function calls `fetch()` to make HTTP requests to AI providers. When these libraries run inside a workflow function, they fail because the global `fetch` is not available.
|
|
25
29
|
|
|
26
|
-
## Quick
|
|
30
|
+
## Quick fix
|
|
27
31
|
|
|
28
32
|
Import the `fetch` step function from the `workflow` package and assign it to `globalThis.fetch` inside your workflow function. This version of `fetch` is a step function that wraps the standard `fetch` API, automatically handling serialization and providing retry capabilities. This will also make `fetch()` available to all functions and libraries in the current workflow function.
|
|
29
33
|
|
|
@@ -31,14 +35,13 @@ Import the `fetch` step function from the `workflow` package and assign it to `g
|
|
|
31
35
|
|
|
32
36
|
```typescript lineNumbers title="workflows/ai.ts"
|
|
33
37
|
import { generateText } from "ai";
|
|
34
|
-
import { openai } from "@ai-sdk/openai";
|
|
35
38
|
|
|
36
39
|
export async function chatWorkflow(prompt: string) {
|
|
37
40
|
"use workflow";
|
|
38
41
|
|
|
39
42
|
// Error - generateText() calls fetch() under the hood
|
|
40
43
|
const result = await generateText({ // [!code highlight]
|
|
41
|
-
model:
|
|
44
|
+
model: "spacexai/grok-4.6", // [!code highlight]
|
|
42
45
|
prompt, // [!code highlight]
|
|
43
46
|
}); // [!code highlight]
|
|
44
47
|
|
|
@@ -50,7 +53,6 @@ export async function chatWorkflow(prompt: string) {
|
|
|
50
53
|
|
|
51
54
|
```typescript lineNumbers title="workflows/ai.ts"
|
|
52
55
|
import { generateText } from "ai";
|
|
53
|
-
import { openai } from "@ai-sdk/openai";
|
|
54
56
|
import { fetch } from "workflow"; // [!code highlight]
|
|
55
57
|
|
|
56
58
|
export async function chatWorkflow(prompt: string) {
|
|
@@ -60,7 +62,7 @@ export async function chatWorkflow(prompt: string) {
|
|
|
60
62
|
|
|
61
63
|
// Now generateText() can make HTTP requests via the fetch step
|
|
62
64
|
const result = await generateText({
|
|
63
|
-
model:
|
|
65
|
+
model: "spacexai/grok-4.6",
|
|
64
66
|
prompt,
|
|
65
67
|
});
|
|
66
68
|
|
|
@@ -68,15 +70,14 @@ export async function chatWorkflow(prompt: string) {
|
|
|
68
70
|
}
|
|
69
71
|
```
|
|
70
72
|
|
|
71
|
-
## Common
|
|
73
|
+
## Common scenarios
|
|
72
74
|
|
|
73
|
-
### AI SDK
|
|
75
|
+
### AI SDK integration
|
|
74
76
|
|
|
75
77
|
This is the most common scenario - using AI SDK functions that make HTTP requests:
|
|
76
78
|
|
|
77
79
|
```typescript lineNumbers
|
|
78
80
|
import { generateText, streamText } from "ai";
|
|
79
|
-
import { openai } from "@ai-sdk/openai";
|
|
80
81
|
import { fetch } from "workflow"; // [!code highlight]
|
|
81
82
|
|
|
82
83
|
export async function aiWorkflow(userMessage: string) {
|
|
@@ -84,9 +85,9 @@ export async function aiWorkflow(userMessage: string) {
|
|
|
84
85
|
|
|
85
86
|
globalThis.fetch = fetch; // [!code highlight]
|
|
86
87
|
|
|
87
|
-
// generateText makes HTTP
|
|
88
|
+
// generateText makes an HTTP request under the hood
|
|
88
89
|
const response = await generateText({
|
|
89
|
-
model:
|
|
90
|
+
model: "spacexai/grok-4.6",
|
|
90
91
|
prompt: userMessage,
|
|
91
92
|
});
|
|
92
93
|
|
|
@@ -94,7 +95,7 @@ export async function aiWorkflow(userMessage: string) {
|
|
|
94
95
|
}
|
|
95
96
|
```
|
|
96
97
|
|
|
97
|
-
### Direct API
|
|
98
|
+
### Direct API calls
|
|
98
99
|
|
|
99
100
|
You can also use the fetch step function directly for your own HTTP requests:
|
|
100
101
|
|
|
@@ -10,15 +10,19 @@ related:
|
|
|
10
10
|
- /docs/api-reference/workflow/define-hook
|
|
11
11
|
---
|
|
12
12
|
|
|
13
|
+
<CopyPrompt
|
|
14
|
+
text="Fix hook token conflicts. Find every `createHook({ token })` or typed hook creation site. If multiple waits can exist at the same time, include a unique stable discriminator in the token such as `${workflowRunId}:approval:${itemId}` or `${orderId}:${attempt}` instead of reusing one global token. If duplicate work should join an existing run, catch `HookConflictError` from `@workflow/errors`, read the conflicting run ID from the error/result if available, and use `getRun(runId)` plus `resumeHook()` from `workflow/api` to deliver the payload to the active run. Keep token generation deterministic across retries so replay does not create new hook identities. Verify two concurrent runs and a duplicate request no longer throw hook-conflict unexpectedly."
|
|
15
|
+
/>
|
|
16
|
+
|
|
13
17
|
This error occurs when you try to create a hook with a token that is already in use by another active workflow run. Hook tokens must be unique across all running workflows in your project.
|
|
14
18
|
|
|
15
|
-
## Error
|
|
19
|
+
## Error message
|
|
16
20
|
|
|
17
|
-
```
|
|
21
|
+
```text
|
|
18
22
|
Hook token "<token>" is already in use by another workflow
|
|
19
23
|
```
|
|
20
24
|
|
|
21
|
-
## Why
|
|
25
|
+
## Why this happens
|
|
22
26
|
|
|
23
27
|
Hooks use tokens to identify incoming webhook payloads. When you create a hook with `createHook({ token: "my-token" })`, the Workflow runtime reserves that token for your workflow run. If another workflow run is already using that token, a conflict occurs.
|
|
24
28
|
|
|
@@ -27,9 +31,9 @@ This typically happens when:
|
|
|
27
31
|
1. **Two workflows start simultaneously** with the same hardcoded token
|
|
28
32
|
2. **A previous workflow run is still waiting** for a hook when a new run tries to use the same token
|
|
29
33
|
|
|
30
|
-
## Common
|
|
34
|
+
## Common causes
|
|
31
35
|
|
|
32
|
-
### Hardcoded
|
|
36
|
+
### Hardcoded token values
|
|
33
37
|
|
|
34
38
|
{/* @skip-typecheck: incomplete code sample */}
|
|
35
39
|
```typescript lineNumbers
|
|
@@ -57,7 +61,7 @@ export async function processPayment(orderId: string) {
|
|
|
57
61
|
}
|
|
58
62
|
```
|
|
59
63
|
|
|
60
|
-
### Omitting the
|
|
64
|
+
### Omitting the token (auto-generated)
|
|
61
65
|
|
|
62
66
|
The safest approach is to let the Workflow runtime generate a unique token automatically:
|
|
63
67
|
|
|
@@ -73,7 +77,7 @@ export async function processPayment() {
|
|
|
73
77
|
}
|
|
74
78
|
```
|
|
75
79
|
|
|
76
|
-
## Handling
|
|
80
|
+
## Handling hook conflicts
|
|
77
81
|
|
|
78
82
|
When a hook conflict occurs, awaiting the hook will throw a `HookConflictError`. The error exposes the token that conflicted and, for current worlds, the run ID that currently owns it. `conflictingRunId` remains optional for compatibility with older persisted events and world implementations, so guard it before delegating:
|
|
79
83
|
|
|
@@ -110,7 +114,7 @@ export async function processPayment(orderId: string) {
|
|
|
110
114
|
|
|
111
115
|
This pattern is useful when you want to detect duplicate processing inside the workflow. Runtime APIs such as `resumeHook()` and `getRun()` must be called outside workflow functions, for example from an API route or in a step.
|
|
112
116
|
|
|
113
|
-
### Delegate to the
|
|
117
|
+
### Delegate to the active Run
|
|
114
118
|
|
|
115
119
|
In idempotency flows, a conflict means another active run already owns the hook token. You can return the duplicate-processing payload from the workflow, resume the active hook to deliver the payload to the existing run, then use `getRun(result.runId)` to wait for, stream, or cancel the active run:
|
|
116
120
|
|
|
@@ -152,22 +156,45 @@ export async function POST(request: Request) {
|
|
|
152
156
|
|
|
153
157
|
If the caller needs live output instead of the final result, return `activeRun.getReadable()` from the same branch. If the duplicate request should replace the active work, call `await activeRun.cancel()` after inspecting the run.
|
|
154
158
|
|
|
155
|
-
|
|
159
|
+
### Take the token over
|
|
160
|
+
|
|
161
|
+
If the newest run should always own the token, create the hook with [`experimental_force: true`](/docs/api-reference/workflow/create-hook#take-over-a-token-another-run-holds). The new run takes the token instead of getting `HookConflictError`, and the previous owner's `await hook` rejects with [`HookForceClaimedError`](/docs/errors/hook-force-claimed).
|
|
162
|
+
|
|
163
|
+
This enables zero-downtime transfers for hooks so one run can hand off a hook to another without dropping messages.
|
|
164
|
+
|
|
165
|
+
```typescript lineNumbers
|
|
166
|
+
import { createHook } from "workflow";
|
|
167
|
+
|
|
168
|
+
export async function processPayment(orderId: string) {
|
|
169
|
+
"use workflow";
|
|
170
|
+
|
|
171
|
+
const hook = createHook({
|
|
172
|
+
token: `payment-${orderId}`,
|
|
173
|
+
experimental_force: true, // [!code highlight]
|
|
174
|
+
});
|
|
175
|
+
const payment = await hook;
|
|
176
|
+
}
|
|
177
|
+
```
|
|
178
|
+
|
|
179
|
+
A run started at a Workflow spec version below 8, including runs started by older SDK releases, can't be taken from, so the forced hook still gets `HookConflictError` in that case.
|
|
180
|
+
|
|
181
|
+
## When hook tokens are released
|
|
156
182
|
|
|
157
183
|
Hook tokens are automatically released when:
|
|
158
184
|
|
|
159
185
|
- The workflow run **completes** (successfully or with an error)
|
|
160
|
-
- The workflow run is **
|
|
186
|
+
- The workflow run is **canceled**
|
|
161
187
|
- The hook is explicitly **disposed**
|
|
162
188
|
|
|
163
189
|
After a workflow completes, its hook tokens become available for reuse by other workflows.
|
|
164
190
|
|
|
165
|
-
## Best
|
|
191
|
+
## Best practices
|
|
166
192
|
|
|
167
193
|
1. **Use auto-generated tokens** when possible - they are guaranteed to be unique
|
|
168
194
|
2. **Include unique identifiers** if you need custom tokens (order ID, user ID, etc.)
|
|
169
195
|
3. **Avoid reusing the same token** across multiple concurrent workflow runs
|
|
170
196
|
4. **Consider using webhooks** (`createWebhook`) if you need a fixed, predictable URL that can receive multiple payloads
|
|
197
|
+
5. **Use `experimental_force`** when a newer run should replace the run holding the token
|
|
171
198
|
|
|
172
199
|
## Related
|
|
173
200
|
|
|
@@ -0,0 +1,96 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: hook-force-claimed
|
|
3
|
+
description: Another workflow run took this hook's token over with experimental_force.
|
|
4
|
+
type: troubleshooting
|
|
5
|
+
summary: Handle HookForceClaimedError by letting the replaced run wrap up while the new owner receives the token's payloads.
|
|
6
|
+
prerequisites:
|
|
7
|
+
- /docs/foundations/hooks
|
|
8
|
+
related:
|
|
9
|
+
- /docs/api-reference/workflow/create-hook
|
|
10
|
+
- /docs/api-reference/workflow-errors/hook-force-claimed-error
|
|
11
|
+
---
|
|
12
|
+
|
|
13
|
+
<CopyPrompt
|
|
14
|
+
text="Handle hook-force-claimed. Find every `createHook({ token })` whose token can be taken over by another run created with `experimental_force: true`. Wrap the `await hook` / `for await...of` in a try/catch, check `HookForceClaimedError.is(error)` from `workflow/errors`, and make the replaced run finish cleanly: persist or return anything it still owes (using `error.claimedByRunId` if the new owner should be told), then exit instead of retrying the await. If the run itself should be the one taking over, add `experimental_force: true` to its `createHook()` call and make sure the token is explicit. Verify that starting a second run with the same token wakes the first, that its await rejects with HookForceClaimedError, and that resumeHook() payloads sent after the takeover reach the second run."
|
|
15
|
+
/>
|
|
16
|
+
|
|
17
|
+
This error is thrown to a workflow run that was waiting on a hook when another run created a hook with the same token and [`experimental_force: true`](/docs/api-reference/workflow/create-hook#take-over-a-token-another-run-holds). The token moved to that run; this run's hook is disposed and will not receive anything further.
|
|
18
|
+
|
|
19
|
+
## Error message
|
|
20
|
+
|
|
21
|
+
```text
|
|
22
|
+
Hook token "<token>" was force-claimed by another workflow (run "<runId>")
|
|
23
|
+
```
|
|
24
|
+
|
|
25
|
+
## Why this happens
|
|
26
|
+
|
|
27
|
+
A hook token is owned by one active run at a time. Normally a second run asking for a held token gets [`HookConflictError`](/docs/errors/hook-conflict). With `experimental_force`, the second run takes the token instead:
|
|
28
|
+
|
|
29
|
+
1. The holder's hook is disposed, and a `hook_disposed` event naming the new owner is written to the holder's event log.
|
|
30
|
+
2. The holder is woken. If it was awaiting the hook, that promise rejects with `HookForceClaimedError`. Payloads that arrived before the takeover are still delivered first; a `for await...of` loop drains them and then throws.
|
|
31
|
+
3. Every `resumeHook()` for the token from then on reaches the new owner, including a delivery that was already in flight. Senders are never told the token moved. The one exception is a `resumeWebhook()` request caught inside the handoff window: its body can be sent only once, so it fails with a retryable error instead of being redirected, and the sender's retry reaches the new owner.
|
|
32
|
+
|
|
33
|
+
This is expected behavior, not a failure of the replaced run. It usually means a newer run for the same subject (a channel, a conversation, a device) has started and is meant to take over.
|
|
34
|
+
|
|
35
|
+
## Handling the takeover
|
|
36
|
+
|
|
37
|
+
Catch the error where the hook is awaited and let the run finish. The error tells you which run replaced this one:
|
|
38
|
+
|
|
39
|
+
```typescript lineNumbers
|
|
40
|
+
import { createHook } from "workflow";
|
|
41
|
+
import { HookForceClaimedError } from "workflow/errors";
|
|
42
|
+
|
|
43
|
+
declare function processMessage(message: { text: string }): Promise<void>; // @setup
|
|
44
|
+
declare function flushDraft(channelId: string): Promise<void>; // @setup
|
|
45
|
+
|
|
46
|
+
export async function channelWorkflow(channelId: string) {
|
|
47
|
+
"use workflow";
|
|
48
|
+
|
|
49
|
+
const hook = createHook<{ text: string }>({
|
|
50
|
+
token: `channel:${channelId}`,
|
|
51
|
+
experimental_force: true,
|
|
52
|
+
});
|
|
53
|
+
|
|
54
|
+
try {
|
|
55
|
+
for await (const message of hook) {
|
|
56
|
+
await processMessage(message);
|
|
57
|
+
}
|
|
58
|
+
} catch (error) {
|
|
59
|
+
if (HookForceClaimedError.is(error)) { // [!code highlight]
|
|
60
|
+
// A newer run owns the channel now. Hand off and stop.
|
|
61
|
+
await flushDraft(channelId);
|
|
62
|
+
return { replacedBy: error.claimedByRunId };
|
|
63
|
+
}
|
|
64
|
+
throw error;
|
|
65
|
+
}
|
|
66
|
+
}
|
|
67
|
+
```
|
|
68
|
+
|
|
69
|
+
Anything the replaced run still needs to publish should happen in this branch. Do not create another hook with the same token here unless this run really should take the token back: runs forcing the same token converge on whichever registered last, and every other one receives this error again.
|
|
70
|
+
|
|
71
|
+
## Several runs forcing the same token
|
|
72
|
+
|
|
73
|
+
Any number of runs can force the same token at once without leaving the token or the runs in a bad state. The takeovers chain: each run that loses the token receives `HookForceClaimedError` naming the run that took it, exactly one run ends up owning the token, and a run that crashes halfway through its own takeover is completed by the next request for the token. Which of several simultaneous claimers wins is not defined, so start them in order if the order matters. If a run should fall back rather than keep fighting for the token, catch `HookForceClaimedError` and exit instead of forcing again.
|
|
74
|
+
|
|
75
|
+
## Runs that cannot be taken from
|
|
76
|
+
|
|
77
|
+
A run that was started at a Workflow spec version below 8 (an older SDK release, a Python SDK run, or a deployment with `WORKFLOW_SEALED_LOG=0`) would never learn that its hook was disposed. The World declines to take its token: the forced hook rejects with the ordinary [`HookConflictError`](/docs/errors/hook-conflict), whose `conflictingRunId` names that run, and the run keeps receiving its payloads. Handle it like any other conflict, for example by [delegating to the active run](/docs/errors/hook-conflict#delegate-to-the-active-run). A finished run holding a retained token is taken over at any version.
|
|
78
|
+
|
|
79
|
+
## Deciding who takes over
|
|
80
|
+
|
|
81
|
+
- **The newest run should win.** Create the hook with `experimental_force: true` in the workflow that starts on each new deployment, restart, or session. Older runs get `HookForceClaimedError` and exit.
|
|
82
|
+
- **The first run should win.** Do not use `experimental_force`. Later runs get `HookConflictError` and can [delegate to the active run](/docs/errors/hook-conflict#delegate-to-the-active-run).
|
|
83
|
+
- **Both runs should keep working.** Give them different tokens.
|
|
84
|
+
|
|
85
|
+
## When it is thrown
|
|
86
|
+
|
|
87
|
+
- To `await hook` and to `for await...of` once buffered payloads are drained.
|
|
88
|
+
- On every later `await` of the same hook. `hook.getConflict()` resolves with `null`: the hook was registered, it just no longer holds the token.
|
|
89
|
+
- Never to a run that has already finished. A token retained after the run ended with `experimental_minRetention` is taken over silently.
|
|
90
|
+
|
|
91
|
+
## Related
|
|
92
|
+
|
|
93
|
+
- [Hooks](/docs/foundations/hooks) - Taking over a token from another run
|
|
94
|
+
- [createHook](/docs/api-reference/workflow/create-hook#take-over-a-token-another-run-holds) - The `experimental_force` option
|
|
95
|
+
- [HookForceClaimedError](/docs/api-reference/workflow-errors/hook-force-claimed-error) - Error reference
|
|
96
|
+
- [hook-conflict](/docs/errors/hook-conflict) - The default behavior without `experimental_force`
|
package/docs/errors/index.mdx
CHANGED
|
@@ -9,43 +9,30 @@ related:
|
|
|
9
9
|
|
|
10
10
|
Fix common mistakes when creating and executing workflows in the **Workflow SDK**.
|
|
11
11
|
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
|
|
21
|
-
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
</Card>
|
|
37
|
-
<Card href="/docs/errors/corrupted-event-log" title="corrupted-event-log">
|
|
38
|
-
Learn how to handle corrupted or invalid event logs.
|
|
39
|
-
</Card>
|
|
40
|
-
<Card href="/docs/errors/step-not-registered" title="step-not-registered">
|
|
41
|
-
Resolve step not registered errors caused by deployment mismatches.
|
|
42
|
-
</Card>
|
|
43
|
-
<Card href="/docs/errors/workflow-not-registered" title="workflow-not-registered">
|
|
44
|
-
Resolve workflow not registered errors caused by deployment mismatches.
|
|
45
|
-
</Card>
|
|
46
|
-
</Cards>
|
|
47
|
-
|
|
48
|
-
## Learn More
|
|
12
|
+
## Error codes
|
|
13
|
+
|
|
14
|
+
When a workflow run fails, its `errorCode` identifies the failure category. You can read it from [`WorkflowRunFailedError`](/docs/api-reference/workflow-errors/workflow-run-failed-error), the Workflow CLI's `error.code` field, or the `workflow.error.code` OpenTelemetry span attribute.
|
|
15
|
+
|
|
16
|
+
| Code | Description |
|
|
17
|
+
| --- | --- |
|
|
18
|
+
| `USER_ERROR` | An error thrown by workflow or step code, including an unhandled step failure or `FatalError`. |
|
|
19
|
+
| `RUNTIME_ERROR` | The Workflow runtime encountered an internal error, such as missing runtime data or an invariant failure. Persistent occurrences should be [reported](https://github.com/vercel/workflow/issues). |
|
|
20
|
+
| [`CORRUPTED_EVENT_LOG`](/docs/errors/corrupted-event-log) | The run's event log cannot be replayed because it contains orphaned or mismatched events, a gap, or an unreadable stored payload. |
|
|
21
|
+
| [`REPLAY_DIVERGENCE`](/docs/errors/replay-divergence) | One replay could not consume the event log deterministically. The runtime automatically retries before treating repeated divergence as a corrupted event log. |
|
|
22
|
+
| `MAX_DELIVERIES_EXCEEDED` | The run exceeded the maximum number of queue deliveries, usually because a persistent failure kept causing redelivery. |
|
|
23
|
+
| `MAX_EVENTS_EXCEEDED` | The run reached the World's per-run event limit. Split unbounded work into child workflows before reaching the limit. |
|
|
24
|
+
| `REPLAY_TIMEOUT` | Workflow replay exceeded the configured duration limit. This measures workflow execution and event-log replay between step boundaries, not time spent inside step functions. |
|
|
25
|
+
| `STREAM_ERROR` | Workflow stream infrastructure failed while reading or writing data. This is an SDK or backend failure rather than an error in workflow code. |
|
|
26
|
+
| `WORLD_CONTRACT_ERROR` | A World returned data that violated the SDK contract and could not be retried safely. This usually indicates a World implementation bug. |
|
|
27
|
+
| [`DEPLOYMENT_MISMATCH`](/docs/errors/deployment-mismatch) | A run was delivered to a deployment other than the one it is pinned to, and automatic re-routing did not recover it. |
|
|
28
|
+
|
|
29
|
+
For guidance on catching failures, retry behavior, and inspecting `WorkflowRunFailedError`, see [Errors and retries](/docs/foundations/errors-and-retries).
|
|
30
|
+
|
|
31
|
+
## Troubleshooting guides
|
|
32
|
+
|
|
33
|
+
<AutoCards />
|
|
34
|
+
|
|
35
|
+
## Learn more
|
|
49
36
|
|
|
50
37
|
* [API Reference](/docs/api-reference) - Complete API documentation
|
|
51
38
|
* [Foundations](/docs/foundations) - Architecture and core concepts
|
|
@@ -9,21 +9,25 @@ related:
|
|
|
9
9
|
- /docs/how-it-works/understanding-directives
|
|
10
10
|
---
|
|
11
11
|
|
|
12
|
+
<CopyPrompt
|
|
13
|
+
text="Fix Node.js module usage inside workflow functions. Search workflow files for imports or direct usage of Node-only APIs such as `fs`, `path`, `crypto`, `process`, `http`, or SDK clients. Remove those imports from files/functions that execute under `"use workflow"`. Create helper functions with `"use step"` for filesystem, crypto, environment, network, database, or SDK work, and call those helpers from the workflow. Keep the workflow function limited to deterministic orchestration, serializable values, `sleep`, hooks, and step calls. Verify the workflow starts without node-js-module-in-workflow errors."
|
|
14
|
+
/>
|
|
15
|
+
|
|
12
16
|
This error occurs when you try to import or use Node.js core modules (like `fs`, `http`, `crypto`, `path`, etc.) directly inside a workflow function.
|
|
13
17
|
|
|
14
|
-
## Error
|
|
18
|
+
## Error message
|
|
15
19
|
|
|
16
|
-
```
|
|
20
|
+
```text
|
|
17
21
|
Cannot use Node.js module "fs" in workflow functions. Move this module to a step function.
|
|
18
22
|
```
|
|
19
23
|
|
|
20
|
-
## Why
|
|
24
|
+
## Why this happens
|
|
21
25
|
|
|
22
26
|
Workflow functions run in a sandboxed environment without full Node.js runtime access. This restriction is important for maintaining **determinism** - the ability to replay workflows exactly and resume from where they left off after suspensions or failures.
|
|
23
27
|
|
|
24
28
|
Node.js modules have side effects and non-deterministic behavior that could break workflow replay guarantees.
|
|
25
29
|
|
|
26
|
-
## Quick
|
|
30
|
+
## Quick fix
|
|
27
31
|
|
|
28
32
|
Move any code using Node.js modules to a step function. Step functions have full Node.js runtime access.
|
|
29
33
|
|
|
@@ -64,7 +68,7 @@ async function read(filePath: string) {
|
|
|
64
68
|
}
|
|
65
69
|
```
|
|
66
70
|
|
|
67
|
-
## Common Node.js
|
|
71
|
+
## Common Node.js modules
|
|
68
72
|
|
|
69
73
|
These common Node.js core modules cannot be used in workflow functions:
|
|
70
74
|
|
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: replay-divergence
|
|
3
|
+
description: A workflow replay temporarily followed a path that did not match its recorded events.
|
|
4
|
+
type: troubleshooting
|
|
5
|
+
summary: Understand automatic recovery when a workflow replay diverges from its event history.
|
|
6
|
+
prerequisites:
|
|
7
|
+
- /docs/foundations/workflows-and-steps
|
|
8
|
+
related:
|
|
9
|
+
- /docs/errors/corrupted-event-log
|
|
10
|
+
- /docs/foundations/errors-and-retries
|
|
11
|
+
---
|
|
12
|
+
|
|
13
|
+
A replay divergence occurs when one invocation of a workflow cannot consume the durable event history using the promises, hooks, sleeps, or steps it created during replay.
|
|
14
|
+
|
|
15
|
+
This is an SDK/runtime signal, not an error thrown by your workflow code. It is not catchable inside a workflow function.
|
|
16
|
+
|
|
17
|
+
## Automatic recovery
|
|
18
|
+
|
|
19
|
+
A single divergent replay does not prove that persisted history is corrupted. For example, asynchronous delivery ordering may cause one invocation to follow the wrong side of a race while another replay can follow the recorded history correctly.
|
|
20
|
+
|
|
21
|
+
The runtime automatically queues another replay when an invocation reports `REPLAY_DIVERGENCE`. No terminal `run_failed` event is written during these recovery attempts.
|
|
22
|
+
|
|
23
|
+
If recovery replays continue to diverge after the recovery budget is exhausted, the runtime marks the run as failed with `CORRUPTED_EVENT_LOG` and records the latest divergent event for diagnosis.
|
|
24
|
+
|
|
25
|
+
## What to do
|
|
26
|
+
|
|
27
|
+
Most replay divergence signals recover without action. If a run ultimately fails with `CORRUPTED_EVENT_LOG`, update to the latest `workflow` package and report the run ID and error details if the failure persists.
|