@bastani/atomic 0.9.43 → 0.9.44-alpha.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (137) hide show
  1. package/CHANGELOG.md +42 -0
  2. package/dist/builtin/intercom/CHANGELOG.md +22 -0
  3. package/dist/builtin/intercom/broker/broker.ts +31 -7
  4. package/dist/builtin/intercom/broker/client-message-validation.ts +6 -0
  5. package/dist/builtin/intercom/broker/client.ts +69 -0
  6. package/dist/builtin/intercom/broker/send-signature.ts +4 -1
  7. package/dist/builtin/intercom/index.bundle.mjs +309 -109
  8. package/dist/builtin/intercom/live-route-refusal.ts +20 -0
  9. package/dist/builtin/intercom/package.json +1 -1
  10. package/dist/builtin/intercom/skills/intercom/SKILL.md +11 -6
  11. package/dist/builtin/intercom/types.ts +28 -1
  12. package/dist/builtin/mcp/CHANGELOG.md +7 -0
  13. package/dist/builtin/mcp/package.json +1 -1
  14. package/dist/builtin/subagents/CHANGELOG.md +9 -0
  15. package/dist/builtin/subagents/README.md +8 -8
  16. package/dist/builtin/subagents/agents/worker.md +1 -1
  17. package/dist/builtin/subagents/package.json +1 -1
  18. package/dist/builtin/subagents/skills/agent-browser/SKILL.md +1 -1
  19. package/dist/builtin/subagents/skills/cua-driver/EMBEDDING.md +21 -2
  20. package/dist/builtin/subagents/skills/cua-driver/LINUX.md +2 -2
  21. package/dist/builtin/subagents/skills/cua-driver/MACOS.md +342 -10
  22. package/dist/builtin/subagents/skills/cua-driver/README.md +4 -3
  23. package/dist/builtin/subagents/skills/cua-driver/RUNTIME.md +3 -1
  24. package/dist/builtin/subagents/skills/cua-driver/SKILL.md +7 -4
  25. package/dist/builtin/subagents/skills/cua-driver/WINDOWS.md +6 -5
  26. package/dist/builtin/subagents/skills/cua-driver/WORKFLOW.md +15 -0
  27. package/dist/builtin/subagents/skills/herdr/SKILL.md +22 -10
  28. package/dist/builtin/subagents/skills/subagent/SKILL.md +8 -5
  29. package/dist/builtin/subagents/src/extension/index.bundle.mjs +6 -6
  30. package/dist/builtin/web-access/package.json +1 -1
  31. package/dist/builtin/workflows/CHANGELOG.md +20 -0
  32. package/dist/builtin/workflows/README.md +1 -1
  33. package/dist/builtin/workflows/builtin/{chunk-frqxd4d1.js → chunk-k8d7ehjx.js} +22 -0
  34. package/dist/builtin/workflows/builtin/{chunk-3s95f7ry.js → chunk-qxpymnaf.js} +1 -1
  35. package/dist/builtin/workflows/builtin/{chunk-edhf30ef.js → chunk-zjecnjxx.js} +1 -1
  36. package/dist/builtin/workflows/builtin/goal.js +2 -2
  37. package/dist/builtin/workflows/builtin/index.js +3 -3
  38. package/dist/builtin/workflows/builtin/ralph.js +2 -2
  39. package/dist/builtin/workflows/package.json +1 -1
  40. package/dist/builtin/workflows/src/extension/index.bundle.mjs +1844 -603
  41. package/dist/builtin/workflows/src/extension/local-worker-entry.mjs +1880 -605
  42. package/dist/builtin/workflows/src/index.js +1896 -655
  43. package/dist/builtin/workflows/src/local-worker-entry.mjs +1880 -605
  44. package/dist/cli/args.js +1 -1
  45. package/dist/cli/args.js.map +1 -1
  46. package/dist/core/agent-session-custom-message-commit.d.ts.map +1 -1
  47. package/dist/core/agent-session-custom-message-commit.js +59 -0
  48. package/dist/core/agent-session-custom-message-commit.js.map +1 -1
  49. package/dist/core/agent-session-methods.d.ts +1 -0
  50. package/dist/core/agent-session-methods.d.ts.map +1 -1
  51. package/dist/core/agent-session-methods.js.map +1 -1
  52. package/dist/core/agent-session-prompt.d.ts +4 -0
  53. package/dist/core/agent-session-prompt.d.ts.map +1 -1
  54. package/dist/core/agent-session-prompt.js +10 -2
  55. package/dist/core/agent-session-prompt.js.map +1 -1
  56. package/dist/core/agent-session.d.ts +2 -0
  57. package/dist/core/agent-session.d.ts.map +1 -1
  58. package/dist/core/agent-session.js +9 -0
  59. package/dist/core/agent-session.js.map +1 -1
  60. package/dist/core/builtin-packages.d.ts.map +1 -1
  61. package/dist/core/builtin-packages.js +13 -13
  62. package/dist/core/builtin-packages.js.map +1 -1
  63. package/dist/core/exec.d.ts.map +1 -1
  64. package/dist/core/exec.js +11 -2
  65. package/dist/core/exec.js.map +1 -1
  66. package/dist/core/package-manager-resolver.d.ts.map +1 -1
  67. package/dist/core/package-manager-resolver.js +25 -14
  68. package/dist/core/package-manager-resolver.js.map +1 -1
  69. package/dist/core/package-manager-types.d.ts +8 -0
  70. package/dist/core/package-manager-types.d.ts.map +1 -1
  71. package/dist/core/package-manager-types.js.map +1 -1
  72. package/dist/core/package-manager.d.ts +1 -1
  73. package/dist/core/package-manager.d.ts.map +1 -1
  74. package/dist/core/package-manager.js.map +1 -1
  75. package/dist/core/resource-loader-extension-warnings.d.ts +4 -1
  76. package/dist/core/resource-loader-extension-warnings.d.ts.map +1 -1
  77. package/dist/core/resource-loader-extension-warnings.js +15 -0
  78. package/dist/core/resource-loader-extension-warnings.js.map +1 -1
  79. package/dist/core/resource-loader-package-resources.d.ts.map +1 -1
  80. package/dist/core/resource-loader-package-resources.js +1 -0
  81. package/dist/core/resource-loader-package-resources.js.map +1 -1
  82. package/dist/core/resource-loader-reload.d.ts.map +1 -1
  83. package/dist/core/resource-loader-reload.js +5 -3
  84. package/dist/core/resource-loader-reload.js.map +1 -1
  85. package/dist/extensions/mcp/client/oauth/flow.d.ts.map +1 -1
  86. package/dist/extensions/mcp/client/oauth/flow.js +4 -1
  87. package/dist/extensions/mcp/client/oauth/flow.js.map +1 -1
  88. package/dist/extensions/mcp/client/oauth/types.d.ts +1 -0
  89. package/dist/extensions/mcp/client/oauth/types.d.ts.map +1 -1
  90. package/dist/extensions/mcp/client/oauth/types.js +8 -4
  91. package/dist/extensions/mcp/client/oauth/types.js.map +1 -1
  92. package/dist/extensions/mcp/client/transports/streamable-http.d.ts.map +1 -1
  93. package/dist/extensions/mcp/client/transports/streamable-http.js +2 -1
  94. package/dist/extensions/mcp/client/transports/streamable-http.js.map +1 -1
  95. package/dist/index.d.ts +1 -1
  96. package/dist/index.d.ts.map +1 -1
  97. package/dist/index.js.map +1 -1
  98. package/dist/main.d.ts.map +1 -1
  99. package/dist/main.js +20 -10
  100. package/dist/main.js.map +1 -1
  101. package/dist/modes/interactive/components/model-selector.d.ts.map +1 -1
  102. package/dist/modes/interactive/components/model-selector.js +4 -2
  103. package/dist/modes/interactive/components/model-selector.js.map +1 -1
  104. package/dist/modes/interactive/components/scoped-models-selector.d.ts.map +1 -1
  105. package/dist/modes/interactive/components/scoped-models-selector.js +4 -2
  106. package/dist/modes/interactive/components/scoped-models-selector.js.map +1 -1
  107. package/dist/modes/interactive/components/tool-execution.d.ts +6 -0
  108. package/dist/modes/interactive/components/tool-execution.d.ts.map +1 -1
  109. package/dist/modes/interactive/components/tool-execution.js +11 -1
  110. package/dist/modes/interactive/components/tool-execution.js.map +1 -1
  111. package/dist/modes/interactive/model-search.d.ts +10 -0
  112. package/dist/modes/interactive/model-search.d.ts.map +1 -1
  113. package/dist/modes/interactive/model-search.js +21 -0
  114. package/dist/modes/interactive/model-search.js.map +1 -1
  115. package/dist/package-manager-cli.d.ts.map +1 -1
  116. package/dist/package-manager-cli.js +5 -0
  117. package/dist/package-manager-cli.js.map +1 -1
  118. package/dist/utils/tools-manager.d.ts.map +1 -1
  119. package/dist/utils/tools-manager.js +30 -35
  120. package/dist/utils/tools-manager.js.map +1 -1
  121. package/docs/computer-use.md +5 -5
  122. package/docs/custom-provider/streaming.md +9 -14
  123. package/docs/environment-variables.md +1 -0
  124. package/docs/intercom/operations.md +1 -1
  125. package/docs/intercom.md +16 -2
  126. package/docs/packages.md +2 -0
  127. package/docs/settings.md +2 -2
  128. package/docs/skills.md +2 -2
  129. package/docs/subagents.md +47 -10
  130. package/docs/workflows/api-reference.md +1 -1
  131. package/docs/workflows/authoring.md +1 -1
  132. package/docs/workflows/operations.md +8 -6
  133. package/docs/workflows/reliable-design.md +23 -5
  134. package/examples/extensions/custom-provider-anthropic/index.ts +31 -31
  135. package/examples/extensions/todo.ts +7 -0
  136. package/npm-shrinkwrap.json +32 -32
  137. package/package.json +3 -3
@@ -1858,7 +1858,13 @@ function createStoreContext(state = createStoreState()) {
1858
1858
  }
1859
1859
  return cachedGraphSnapshot;
1860
1860
  }
1861
+ let batchDepth = 0;
1862
+ let batchedNotification = false;
1861
1863
  function notify() {
1864
+ if (batchDepth > 0) {
1865
+ batchedNotification = true;
1866
+ return;
1867
+ }
1862
1868
  observation.capture();
1863
1869
  for (const fn of state.invalidationListeners) {
1864
1870
  try {
@@ -1878,6 +1884,18 @@ function createStoreContext(state = createStoreState()) {
1878
1884
  state.version++;
1879
1885
  notify();
1880
1886
  }
1887
+ function batch(update) {
1888
+ batchDepth++;
1889
+ try {
1890
+ return update();
1891
+ } finally {
1892
+ batchDepth--;
1893
+ if (batchDepth === 0 && batchedNotification) {
1894
+ batchedNotification = false;
1895
+ notify();
1896
+ }
1897
+ }
1898
+ }
1881
1899
  function findRun(runId) {
1882
1900
  return state.runs.find((r) => r.id === runId);
1883
1901
  }
@@ -1931,6 +1949,7 @@ function createStoreContext(state = createStoreState()) {
1931
1949
  snapshot,
1932
1950
  graphSnapshot,
1933
1951
  notify,
1952
+ batch,
1934
1953
  bumpAndNotify,
1935
1954
  findRun,
1936
1955
  findStage,
@@ -3867,6 +3886,9 @@ function createRunStoreMethods(context) {
3867
3886
  subscribeInvalidation(fn) {
3868
3887
  state.invalidationListeners.add(fn);
3869
3888
  return () => state.invalidationListeners.delete(fn);
3889
+ },
3890
+ batch(update) {
3891
+ return context.batch(update);
3870
3892
  }
3871
3893
  };
3872
3894
  }
@@ -5840,22 +5862,37 @@ function childScopedEvidenceExists(checkpoints, boundaryReplayKey, childRunId) {
5840
5862
  }
5841
5863
  function retainReachableRunGroups(grouped, rootRunId) {
5842
5864
  const reachable = new Set([rootRunId]);
5843
- const pending = [rootRunId];
5865
+ const settled = new Set;
5866
+ const pending = [{ runId: rootRunId, live: true }];
5844
5867
  while (pending.length > 0) {
5845
- const drafts = grouped.get(pending.pop());
5868
+ const { runId, live } = pending.pop();
5869
+ const drafts = grouped.get(runId);
5846
5870
  if (drafts === undefined)
5847
5871
  continue;
5848
5872
  for (const draft of drafts) {
5849
- const childRunId = childRunIdFromDraft(draft);
5850
- if (childRunId === undefined || reachable.has(childRunId) || !grouped.has(childRunId))
5873
+ const childRunId = workflowChildFromDraft(draft)?.runId ?? draft.topology?.boundary?.child.runId;
5874
+ if (childRunId === undefined || !grouped.has(childRunId))
5851
5875
  continue;
5852
- reachable.add(childRunId);
5853
- pending.push(childRunId);
5876
+ if (live && childRunIdFromDraft(draft) !== undefined) {
5877
+ if (reachable.has(childRunId))
5878
+ continue;
5879
+ reachable.add(childRunId);
5880
+ pending.push({ runId: childRunId, live: true });
5881
+ } else if (!settled.has(childRunId)) {
5882
+ settled.add(childRunId);
5883
+ pending.push({ runId: childRunId, live: false });
5884
+ }
5854
5885
  }
5855
5886
  }
5856
- for (const runId of grouped.keys())
5857
- if (!reachable.has(runId))
5858
- grouped.delete(runId);
5887
+ let complete = true;
5888
+ for (const runId of grouped.keys()) {
5889
+ if (reachable.has(runId))
5890
+ continue;
5891
+ if (!settled.has(runId))
5892
+ complete = false;
5893
+ grouped.delete(runId);
5894
+ }
5895
+ return complete;
5859
5896
  }
5860
5897
  function childRunIdFromDraft(draft) {
5861
5898
  const boundary = draft.topology?.boundary;
@@ -6771,6 +6808,13 @@ function removeProcessExitCleanup(cleanup) {
6771
6808
  if (processExitCleanups.delete(cleanup) && processExitCleanups.size === 0)
6772
6809
  process.off("exit", runProcessExitCleanups);
6773
6810
  }
6811
+ function snapshotNeedsHistory(tail) {
6812
+ const event = tail.snapshot.event;
6813
+ if (event.type !== RUN_EVENT_TYPES.messageQueued)
6814
+ return false;
6815
+ const payload = event.payload;
6816
+ return hasPendingStageDeliveryDelta(payload.pendingStageMessages);
6817
+ }
6774
6818
  function* invoke(action) {
6775
6819
  try {
6776
6820
  const result = action();
@@ -6886,8 +6930,11 @@ class JournalDurableBackend {
6886
6930
  this.projection.applyWorkflowSnapshot(event.payload.handle);
6887
6931
  this.prompts.set(entry.runId, new PromptReservationState(event.payload.handle.pendingPrompts, event.payload.prompts));
6888
6932
  } else {
6889
- const handle = event.payload;
6890
6933
  const previous = this.projection.getWorkflow(entry.runId);
6934
+ const handle = event.type === RUN_EVENT_TYPES.messageQueued && hasPendingStageDeliveryDelta(event.payload.pendingStageMessages) ? {
6935
+ ...event.payload,
6936
+ pendingStageMessages: decodePendingStageDeliveries(previous?.pendingStageMessages ?? [], event.payload.pendingStageMessages ?? [])
6937
+ } : event.payload;
6891
6938
  this.projection.applyWorkflowSnapshot(handle);
6892
6939
  if (event.type === RUN_EVENT_TYPES.runStarted || previous?.pendingPrompts !== handle.pendingPrompts)
6893
6940
  this.prompts.delete(entry.runId);
@@ -7183,11 +7230,23 @@ class JournalDurableBackend {
7183
7230
  for (const message of messages)
7184
7231
  if (message.runId !== logicalRunId)
7185
7232
  throw new JournalMessageScopeError(logicalRunId, message.runId);
7186
- yield* this.snapshot({
7233
+ const next = {
7187
7234
  ...handle,
7188
7235
  pendingStageMessages: replacePendingStageMessagesForRun(handle.pendingStageMessages ?? [], logicalRunId, messages),
7189
7236
  updatedAt: Math.max(Date.now(), handle.updatedAt + 1)
7190
- }, RUN_EVENT_TYPES.messageQueued);
7237
+ };
7238
+ const encoded = encodePendingStageDeliveries(handle.pendingStageMessages ?? [], next.pendingStageMessages ?? []);
7239
+ const base = this.sequences.get(runId);
7240
+ if (encoded !== undefined && base !== undefined) {
7241
+ try {
7242
+ yield* this.append(runId, { type: RUN_EVENT_TYPES.messageQueued, payload: { ...next, pendingStageMessages: encoded } }, { expectSeq: base });
7243
+ return true;
7244
+ } catch (error) {
7245
+ if (!(error instanceof JournalCasConflictError))
7246
+ throw error;
7247
+ }
7248
+ }
7249
+ yield* this.snapshot(next, RUN_EVENT_TYPES.messageQueued);
7191
7250
  return true;
7192
7251
  }
7193
7252
  async archivePendingStageMessage(runId, message) {
@@ -7281,7 +7340,7 @@ class JournalDurableBackend {
7281
7340
  if (summarized !== undefined && summarized.seq === version.seq && summarized.at === version.at)
7282
7341
  return;
7283
7342
  const tail = yield* invoke(() => journal.snapshotTail(runId));
7284
- if (tail !== undefined)
7343
+ if (tail !== undefined && !snapshotNeedsHistory(tail))
7285
7344
  return this.summarize(runId, tail);
7286
7345
  if (summarized === undefined && version.seq === 0)
7287
7346
  return;
@@ -7507,6 +7566,69 @@ class JournalDurableBackend {
7507
7566
  }
7508
7567
  }
7509
7568
  }
7569
+ function identity(message) {
7570
+ return JSON.stringify([message.runId, message.id]);
7571
+ }
7572
+ function uniqueByIdentity(messages) {
7573
+ const result = new Map;
7574
+ const repeated = new Set;
7575
+ for (const message of messages) {
7576
+ const key = identity(message);
7577
+ if (result.has(key))
7578
+ repeated.add(key);
7579
+ result.set(key, message);
7580
+ }
7581
+ for (const key of repeated)
7582
+ result.delete(key);
7583
+ return result;
7584
+ }
7585
+ function isDeltaEncoded(message) {
7586
+ return message.deliveryDelta !== undefined;
7587
+ }
7588
+ function encodePendingStageDeliveries(previous, next) {
7589
+ const bases = uniqueByIdentity(previous);
7590
+ const nextIdentities = uniqueByIdentity(next);
7591
+ let encodedAny = false;
7592
+ const encoded = next.map((message) => {
7593
+ const base = bases.get(identity(message))?.deliveries;
7594
+ const deliveries = message.deliveries;
7595
+ if (base === undefined || deliveries === undefined || !nextIdentities.has(identity(message)))
7596
+ return message;
7597
+ const drop = [];
7598
+ let kept = 0;
7599
+ base.forEach((delivery, index) => {
7600
+ if (kept < deliveries.length && JSON.stringify(delivery) === JSON.stringify(deliveries[kept]))
7601
+ kept++;
7602
+ else
7603
+ drop.push(index);
7604
+ });
7605
+ const add = deliveries.slice(kept);
7606
+ if (drop.length + add.length >= deliveries.length)
7607
+ return message;
7608
+ encodedAny = true;
7609
+ const { deliveries: _full, ...rest } = message;
7610
+ return { ...rest, deliveryDelta: drop.length === 0 ? { add } : { drop, add } };
7611
+ });
7612
+ return encodedAny ? encoded : undefined;
7613
+ }
7614
+ function decodePendingStageDeliveries(previous, messages) {
7615
+ if (!messages.some(isDeltaEncoded))
7616
+ return messages;
7617
+ const bases = uniqueByIdentity(previous);
7618
+ return messages.map((message) => {
7619
+ if (!isDeltaEncoded(message))
7620
+ return message;
7621
+ const base = bases.get(identity(message))?.deliveries;
7622
+ if (base === undefined)
7623
+ throw new Error(`atomic-workflows: journal delivery change for ${identity(message)} has no base entry`);
7624
+ const { deliveryDelta, ...rest } = message;
7625
+ const dropped = new Set(deliveryDelta.drop ?? []);
7626
+ return { ...rest, deliveries: [...base.filter((_, index) => !dropped.has(index)), ...deliveryDelta.add] };
7627
+ });
7628
+ }
7629
+ function hasPendingStageDeliveryDelta(messages) {
7630
+ return messages?.some(isDeltaEncoded) === true;
7631
+ }
7510
7632
  var PROCESS_HOLDER_ID, processExitCleanups, JournalOwnershipError, JournalBackendClosedError, JournalOperationAbortedError, JournalExecutionRetiredError, JournalMessageScopeError;
7511
7633
  var init_journal_backend = __esm(() => {
7512
7634
  init_workflow_instance_ownership_error();
@@ -15531,7 +15653,7 @@ class ConcurrencyLimiter {
15531
15653
  }
15532
15654
  }
15533
15655
  function createRunLimiter(defaultConcurrency) {
15534
- return new ConcurrencyLimiter(defaultConcurrency ?? 3);
15656
+ return new ConcurrencyLimiter(defaultConcurrency ?? 1000);
15535
15657
  }
15536
15658
 
15537
15659
  // dist/builtin/workflows/src/shared/possible-stages.ts
@@ -17023,7 +17145,7 @@ function kebabCase(name) {
17023
17145
  function errorMessage2(error) {
17024
17146
  return error instanceof Error ? error.message : String(error);
17025
17147
  }
17026
- var DEFAULT_MAX_DEPTH = 4, BUILTIN_BARREL_SPECIFIERS, BUILTIN_SUBPATH_PREFIXES, TRACKED_NODE_METHODS;
17148
+ var DEFAULT_MAX_DEPTH = 3, BUILTIN_BARREL_SPECIFIERS, BUILTIN_SUBPATH_PREFIXES, TRACKED_NODE_METHODS;
17027
17149
  var init_possible_stages = __esm(() => {
17028
17150
  BUILTIN_BARREL_SPECIFIERS = new Set(["@bastani/atomic/workflows/builtin"]);
17029
17151
  BUILTIN_SUBPATH_PREFIXES = ["@bastani/atomic/workflows/builtin/"];
@@ -60578,6 +60700,44 @@ class GraphFrontierTracker {
60578
60700
  }
60579
60701
  }
60580
60702
 
60703
+ // dist/builtin/workflows/src/shared/external-run-routes.ts
60704
+ function claimExternalRunRoute(store, runId, owner) {
60705
+ let owners = registry.owners.get(store);
60706
+ if (owners === undefined) {
60707
+ owners = new Map;
60708
+ registry.owners.set(store, owners);
60709
+ }
60710
+ owners.set(runId, owner);
60711
+ return () => {
60712
+ const current = registry.owners.get(store);
60713
+ if (current?.get(runId) !== owner)
60714
+ return;
60715
+ current.delete(runId);
60716
+ if (current.size === 0)
60717
+ registry.owners.delete(store);
60718
+ };
60719
+ }
60720
+ function externalRunRouteOwner(store, runId) {
60721
+ return registry.owners.get(store)?.get(runId);
60722
+ }
60723
+ function registerPendingStageMessageRouter(store, router) {
60724
+ registry.routers.set(store, router);
60725
+ return () => {
60726
+ if (registry.routers.get(store) === router)
60727
+ registry.routers.delete(store);
60728
+ };
60729
+ }
60730
+ function pendingStageMessageRouter(store) {
60731
+ return registry.routers.get(store);
60732
+ }
60733
+ var registryKey, processState, registry;
60734
+ var init_external_run_routes = __esm(() => {
60735
+ registryKey = Symbol.for("atomic-workflows/external-run-routes@1");
60736
+ processState = globalThis;
60737
+ registry = processState[registryKey] ?? { owners: new WeakMap, routers: new WeakMap };
60738
+ processState[registryKey] = registry;
60739
+ });
60740
+
60581
60741
  // dist/builtin/workflows/src/shared/pending-stage-route-capability.ts
60582
60742
  import { randomUUID as randomUUID11 } from "node:crypto";
60583
60743
  function capabilitiesFor(store) {
@@ -60602,12 +60762,12 @@ function rotateWorkflowPendingStageRouteCapability(store, runId) {
60602
60762
  capabilitiesFor(store).set(runId, capability);
60603
60763
  return capability;
60604
60764
  }
60605
- var routeCapabilitiesKey, processState, routeCapabilities;
60765
+ var routeCapabilitiesKey, processState2, routeCapabilities;
60606
60766
  var init_pending_stage_route_capability = __esm(() => {
60607
60767
  routeCapabilitiesKey = Symbol.for("atomic-workflows/pending-stage-route-capabilities@1");
60608
- processState = globalThis;
60609
- routeCapabilities = processState[routeCapabilitiesKey] ?? new WeakMap;
60610
- processState[routeCapabilitiesKey] = routeCapabilities;
60768
+ processState2 = globalThis;
60769
+ routeCapabilities = processState2[routeCapabilitiesKey] ?? new WeakMap;
60770
+ processState2[routeCapabilitiesKey] = routeCapabilities;
60611
60771
  });
60612
60772
 
60613
60773
  // dist/builtin/workflows/src/shared/pending-stage-route-readiness.ts
@@ -60639,12 +60799,12 @@ function hasWorkflowPendingStageRouteOwner(store) {
60639
60799
  function workflowPendingStageRouteReady(store, runId) {
60640
60800
  return owners.get(store)?.ready(runId);
60641
60801
  }
60642
- var routeOwnersKey, processState2, owners;
60802
+ var routeOwnersKey, processState3, owners;
60643
60803
  var init_pending_stage_route_readiness = __esm(() => {
60644
60804
  routeOwnersKey = Symbol.for("atomic-workflows/pending-stage-route-owners@1");
60645
- processState2 = globalThis;
60646
- owners = processState2[routeOwnersKey] ?? new WeakMap;
60647
- processState2[routeOwnersKey] = owners;
60805
+ processState3 = globalThis;
60806
+ owners = processState3[routeOwnersKey] ?? new WeakMap;
60807
+ processState3[routeOwnersKey] = owners;
60648
60808
  });
60649
60809
 
60650
60810
  // dist/builtin/workflows/src/shared/sticky-stage-delivery-status.ts
@@ -60769,6 +60929,9 @@ function registerPendingStageIntercomBridge(pi, activeStore, ownsRun = () => tru
60769
60929
  notificationId,
60770
60930
  reason
60771
60931
  };
60932
+ const external = externalRunRouteOwner(activeStore, entry.runId);
60933
+ if (external !== undefined)
60934
+ return external.notifyUndeliverable(payload);
60772
60935
  pi.events?.emit?.(PENDING_STAGE_UNDELIVERABLE_EVENT, payload);
60773
60936
  return payload.handled && await payload.completion === true;
60774
60937
  };
@@ -60780,19 +60943,29 @@ function registerPendingStageIntercomBridge(pi, activeStore, ownsRun = () => tru
60780
60943
  return;
60781
60944
  const ownedRunIds = new Set;
60782
60945
  const runs = activeStore.runs();
60946
+ const runById = new Map(runs.map((run) => [run.id, run]));
60947
+ const childrenByBoundary = new Map;
60948
+ for (const run of runs) {
60949
+ if (run.parentRunId === undefined || run.parentStageId === undefined)
60950
+ continue;
60951
+ const key = JSON.stringify([run.parentRunId, run.parentStageId]);
60952
+ const children = childrenByBoundary.get(key) ?? [];
60953
+ children.push(run);
60954
+ childrenByBoundary.set(key, children);
60955
+ }
60783
60956
  const resolveBoundarySegments = createWorkflowBoundarySegmentsResolver(runs);
60784
60957
  for (const run of runs) {
60785
- if (!ownsRun(run.id))
60958
+ if (externalRunRouteOwner(activeStore, run.id) !== undefined || !ownsRun(run.id))
60786
60959
  continue;
60787
- const rootRunId = durableRootRunIdForRun(runs, run.id);
60960
+ const rootRunId = reciprocalWorkflowRootRunId(runById, run.id);
60788
60961
  if (rootRunId === undefined)
60789
60962
  continue;
60790
60963
  ownedRunIds.add(run.id);
60791
- const parentRun = runs.find((candidate) => candidate.id === run.parentRunId);
60964
+ const parentRun = run.parentRunId === undefined ? undefined : runById.get(run.parentRunId);
60792
60965
  const boundary = parentRun?.stages.find((stage) => stage.id === run.parentStageId);
60793
60966
  const parent = parentRun === undefined || boundary === undefined ? undefined : {
60794
60967
  runId: parentRun.id,
60795
- stageKeys: [boundary.id, boundary.name].filter((key) => resolveChildRun(runs, parentRun, key)?.id === run.id)
60968
+ stageKeys: [boundary.id, boundary.name].filter((key) => resolveIndexedChildRun(childrenByBoundary, parentRun, key)?.id === run.id)
60796
60969
  };
60797
60970
  const announcement = {
60798
60971
  runId: run.id,
@@ -60933,16 +61106,22 @@ function registerPendingStageIntercomBridge(pi, activeStore, ownsRun = () => tru
60933
61106
  payload.handled = true;
60934
61107
  payload.completion = queueAndPersist(activeStore, resolvedEvent, workflowInvocationIntercomGroup(rootRunId), stage.pendingStageDeliveryAvailable === true);
60935
61108
  };
60936
- const subscription = pi.events?.on?.(PENDING_STAGE_MESSAGE_EVENT, (payload) => outsideWorkflowExecutions(() => routePendingStageMessage(payload)));
60937
- const stickySubscription = pi.events?.on?.(STICKY_LIVE_DELIVERED_EVENT, (payload) => {
61109
+ const routeStickyLiveDelivered = (payload) => {
60938
61110
  if (disposed || !isStickyLiveDeliveredEvent(payload) || payload.handled)
60939
61111
  return;
60940
61112
  payload.handled = true;
60941
61113
  payload.completion = outsideWorkflowExecutions(() => recordConfirmedStickyDeliveries(activeStore, payload));
61114
+ };
61115
+ const subscription = pi.events?.on?.(PENDING_STAGE_MESSAGE_EVENT, (payload) => outsideWorkflowExecutions(() => routePendingStageMessage(payload)));
61116
+ const stickySubscription = pi.events?.on?.(STICKY_LIVE_DELIVERED_EVENT, routeStickyLiveDelivered);
61117
+ const disposeRouter = registerPendingStageMessageRouter(activeStore, {
61118
+ route: (payload) => outsideWorkflowExecutions(() => routePendingStageMessage(payload)),
61119
+ sticky: routeStickyLiveDelivered
60942
61120
  });
60943
61121
  const dispose = () => {
60944
61122
  disposed = true;
60945
61123
  disposeReadiness();
61124
+ disposeRouter();
60946
61125
  routeCompletions.clear();
60947
61126
  unsubscribeStore();
60948
61127
  if (typeof subscription === "function")
@@ -61068,6 +61247,12 @@ function isMaterializedNonAgentTarget(runs, target) {
61068
61247
  return false;
61069
61248
  return nonAgentNodes(run).some((node) => node.id === stageKey || node.name === stageKey);
61070
61249
  }
61250
+ function resolveIndexedChildRun(childrenByBoundary, parent, segment) {
61251
+ const boundariesById = parent.stages.filter((stage) => stage.id === segment);
61252
+ const boundaries = boundariesById.length > 0 ? boundariesById : parent.stages.filter((stage) => stage.name === segment);
61253
+ const children = boundaries.flatMap((boundary) => childrenByBoundary.get(JSON.stringify([parent.id, boundary.id])) ?? []);
61254
+ return children.length === 1 ? children[0] : undefined;
61255
+ }
61071
61256
  function resolveChildRun(runs, parent, segment) {
61072
61257
  const boundariesById = parent.stages.filter((stage) => stage.id === segment);
61073
61258
  const boundaries = boundariesById.length > 0 ? boundariesById : parent.stages.filter((stage) => stage.name === segment);
@@ -61407,6 +61592,7 @@ var init_pending_stage_intercom = __esm(() => {
61407
61592
  init_execution_context_boundary();
61408
61593
  init_run_owner_backend();
61409
61594
  init_store_backend();
61595
+ init_external_run_routes();
61410
61596
  init_intercom_group();
61411
61597
  init_pending_stage_route_capability();
61412
61598
  init_pending_stage_route_readiness();
@@ -61546,7 +61732,8 @@ function buildSendSignature(to, options) {
61546
61732
  attachments: canonicalizeAttachmentsForSendSignature(options.attachments) ?? [],
61547
61733
  replyTo: options.replyTo ?? null,
61548
61734
  expectsReply: options.expectsReply ?? false,
61549
- replyError: options.replyError ?? null
61735
+ replyError: options.replyError ?? null,
61736
+ ...options.delivery === undefined || options.delivery === "steer" ? {} : { delivery: options.delivery }
61550
61737
  });
61551
61738
  }
61552
61739
 
@@ -61732,6 +61919,8 @@ function isMessage(value2) {
61732
61919
  return false;
61733
61920
  if (message.expectsReply !== undefined && typeof message.expectsReply !== "boolean")
61734
61921
  return false;
61922
+ if (message.delivery !== undefined && message.delivery !== "steer" && message.delivery !== "followUp" && message.delivery !== "interrupt")
61923
+ return false;
61735
61924
  if (message.replyError !== undefined && typeof message.replyError !== "string")
61736
61925
  return false;
61737
61926
  if (message.source !== undefined && !isMessageSource(message.source))
@@ -61865,7 +62054,7 @@ function isWorkflowFutureStageRosterEntries(value2) {
61865
62054
  function isWorkflowStageRosterEntries(value2) {
61866
62055
  return Array.isArray(value2) && value2.every((entry) => typeof entry === "object" && entry !== null && entry.kind === "workflow-stage" && typeof entry.runId === "string" && typeof entry.stageId === "string" && typeof entry.stageName === "string" && typeof entry.target === "string" && (entry.lifecycle === "pending" || entry.lifecycle === "running") && typeof entry.group === "string" && (entry.sessionId === undefined || typeof entry.sessionId === "string"));
61867
62056
  }
61868
- var BROKER_SOCKET, BROKER_CONNECT_TIMEOUT_MS = 1e4, PENDING_STAGE_ROUTE_ACK_TIMEOUT_MS = 1e4, GROUP_REQUEST_TIMEOUT_MS = 5000, PRESENCE_ACK_TIMEOUT_MS = 5000, RESTORING_RECIPIENT_REQUEST_TIMEOUT_MS = 5000, HELD_SEND_GRACE_MS = 5000, IntercomClient;
62057
+ var BROKER_SOCKET, BROKER_CONNECT_TIMEOUT_MS = 1e4, PENDING_STAGE_ROUTE_ACK_TIMEOUT_MS = 1e4, GROUP_REQUEST_TIMEOUT_MS = 5000, PRESENCE_ACK_TIMEOUT_MS = 5000, RESTORING_RECIPIENT_REQUEST_TIMEOUT_MS = 5000, HELD_SEND_GRACE_MS = 5000, IntercomPendingStageRouteRefusedError, PENDING_STAGE_ROUTE_REFUSAL_CODES, IntercomClient;
61869
62058
  var init_client = __esm(() => {
61870
62059
  init_paths();
61871
62060
  init_pending_send_registry();
@@ -61874,6 +62063,21 @@ var init_client = __esm(() => {
61874
62063
  init_live_route_refusal();
61875
62064
  init_retry_policy();
61876
62065
  BROKER_SOCKET = getBrokerSocketPath();
62066
+ IntercomPendingStageRouteRefusedError = class IntercomPendingStageRouteRefusedError extends Error {
62067
+ connectionKept;
62068
+ code;
62069
+ constructor(reason, refusal = { connectionKept: false }) {
62070
+ super(reason);
62071
+ this.name = "IntercomPendingStageRouteRefusedError";
62072
+ this.connectionKept = refusal.connectionKept;
62073
+ this.code = refusal.code;
62074
+ }
62075
+ };
62076
+ PENDING_STAGE_ROUTE_REFUSAL_CODES = new Set([
62077
+ "not_authorized",
62078
+ "invalid_stage_roster",
62079
+ "invalid_possible_stage_roster"
62080
+ ]);
61877
62081
  IntercomClient = class IntercomClient extends EventEmitter {
61878
62082
  returnAddress;
61879
62083
  preferredSessionId;
@@ -61881,6 +62085,8 @@ var init_client = __esm(() => {
61881
62085
  connectTimeoutMs;
61882
62086
  brokerAcknowledgesReceipt = false;
61883
62087
  brokerAcknowledgesPendingStageRoutes = false;
62088
+ multiplexesPendingStageRoutes;
62089
+ brokerRefusesPendingStageRoutesInPlace = false;
61884
62090
  brokerAnswersStatus = false;
61885
62091
  socket = null;
61886
62092
  _sessionId = null;
@@ -61946,6 +62152,7 @@ var init_client = __esm(() => {
61946
62152
  this.preferredSessionId = options.preferredSessionId;
61947
62153
  this.acknowledgesReceipt = options.acknowledgesReceipt === true;
61948
62154
  this.connectTimeoutMs = options.connectTimeoutMs ?? BROKER_CONNECT_TIMEOUT_MS;
62155
+ this.multiplexesPendingStageRoutes = options.multiplexesPendingStageRoutes === true;
61949
62156
  }
61950
62157
  failPending(error) {
61951
62158
  this.pendingSends.rejectAll(error);
@@ -62058,6 +62265,7 @@ var init_client = __esm(() => {
62058
62265
  this._supervisorSessionId = null;
62059
62266
  this.brokerAcknowledgesReceipt = false;
62060
62267
  this.brokerAcknowledgesPendingStageRoutes = false;
62268
+ this.brokerRefusesPendingStageRoutesInPlace = false;
62061
62269
  this.brokerAnswersStatus = false;
62062
62270
  this._messageSource = undefined;
62063
62271
  this.disconnectError = null;
@@ -62136,7 +62344,7 @@ var init_client = __esm(() => {
62136
62344
  }
62137
62345
  switch (brokerMessage.type) {
62138
62346
  case "registered": {
62139
- if (typeof brokerMessage.sessionId !== "string" || brokerMessage.supervisorSessionId !== undefined && typeof brokerMessage.supervisorSessionId !== "string" || brokerMessage.acknowledgesReceipt !== undefined && brokerMessage.acknowledgesReceipt !== true || brokerMessage.acknowledgesPendingStageRoutes !== undefined && brokerMessage.acknowledgesPendingStageRoutes !== true || brokerMessage.brokerStatus !== undefined && brokerMessage.brokerStatus !== true) {
62347
+ if (typeof brokerMessage.sessionId !== "string" || brokerMessage.supervisorSessionId !== undefined && typeof brokerMessage.supervisorSessionId !== "string" || brokerMessage.acknowledgesReceipt !== undefined && brokerMessage.acknowledgesReceipt !== true || brokerMessage.acknowledgesPendingStageRoutes !== undefined && brokerMessage.acknowledgesPendingStageRoutes !== true || brokerMessage.refusesPendingStageRoutesInPlace !== undefined && brokerMessage.refusesPendingStageRoutesInPlace !== true || brokerMessage.brokerStatus !== undefined && brokerMessage.brokerStatus !== true) {
62140
62348
  throw new Error("Invalid registered message");
62141
62349
  }
62142
62350
  if (this._sessionId !== null) {
@@ -62146,6 +62354,7 @@ var init_client = __esm(() => {
62146
62354
  this._supervisorSessionId = typeof brokerMessage.supervisorSessionId === "string" ? brokerMessage.supervisorSessionId : null;
62147
62355
  this.brokerAcknowledgesReceipt = this.acknowledgesReceipt && brokerMessage.acknowledgesReceipt === true;
62148
62356
  this.brokerAcknowledgesPendingStageRoutes = brokerMessage.acknowledgesPendingStageRoutes === true;
62357
+ this.brokerRefusesPendingStageRoutesInPlace = brokerMessage.refusesPendingStageRoutesInPlace === true;
62149
62358
  this.brokerAnswersStatus = brokerMessage.brokerStatus === true;
62150
62359
  this.emit("_registered", { type: "registered", sessionId: brokerMessage.sessionId });
62151
62360
  break;
@@ -62158,6 +62367,12 @@ var init_client = __esm(() => {
62158
62367
  this.emit("_registration_failed", refusal);
62159
62368
  break;
62160
62369
  }
62370
+ const refusedRouteId = typeof brokerMessage.requestId === "string" ? brokerMessage.requestId : "";
62371
+ const refusedRoute = this.pendingStageRouteRegistrations.get(refusedRouteId);
62372
+ if (refusedRoute !== undefined) {
62373
+ this.pendingStageRouteRegistrations.delete(refusedRouteId);
62374
+ refusedRoute.reject(new IntercomPendingStageRouteRefusedError(brokerMessage.reason));
62375
+ }
62161
62376
  this.disconnectError ??= refusal;
62162
62377
  this.failPending(this.disconnectError);
62163
62378
  this.socket?.destroy();
@@ -62283,6 +62498,21 @@ var init_client = __esm(() => {
62283
62498
  this.emit("pending_stage_route_revoked", { runId });
62284
62499
  break;
62285
62500
  }
62501
+ case "pending_stage_route_refused": {
62502
+ const { requestId, runId, code, reason } = brokerMessage;
62503
+ if (typeof requestId !== "string" || typeof runId !== "string" || typeof reason !== "string" || typeof code !== "string") {
62504
+ throw new Error("Invalid pending-stage route refusal");
62505
+ }
62506
+ const pending = this.pendingStageRouteRegistrations.get(requestId);
62507
+ if (!pending)
62508
+ return;
62509
+ this.pendingStageRouteRegistrations.delete(requestId);
62510
+ pending.reject(new IntercomPendingStageRouteRefusedError(reason, {
62511
+ connectionKept: true,
62512
+ ...PENDING_STAGE_ROUTE_REFUSAL_CODES.has(code) ? { code } : {}
62513
+ }));
62514
+ break;
62515
+ }
62286
62516
  case "pending_stage_route_registered": {
62287
62517
  const { requestId } = brokerMessage;
62288
62518
  if (typeof requestId !== "string")
@@ -62786,7 +63016,8 @@ var init_client = __esm(() => {
62786
63016
  stages,
62787
63017
  possibleStages,
62788
63018
  ...parent === undefined ? {} : { parent },
62789
- ...dormant === true ? { dormant: true } : {}
63019
+ ...dormant === true ? { dormant: true } : {},
63020
+ ...this.multiplexesPendingStageRoutes && this.brokerRefusesPendingStageRoutesInPlace ? { refusalKeepsConnection: true } : {}
62790
63021
  });
62791
63022
  } catch (error) {
62792
63023
  clearTimeout(timeout);
@@ -62899,6 +63130,7 @@ var init_client = __esm(() => {
62899
63130
  timestamp: Date.now(),
62900
63131
  replyTo: options.replyTo,
62901
63132
  expectsReply: options.expectsReply,
63133
+ ...options.delivery === undefined || options.delivery === "steer" ? {} : { delivery: options.delivery },
62902
63134
  replyError: options.replyError,
62903
63135
  source: this._messageSource,
62904
63136
  content: { text: options.text, attachments: options.attachments }
@@ -63520,120 +63752,165 @@ var init_reconnect_backoff = __esm(() => {
63520
63752
  BACKOFF_MS = [1000, 2000, 5000, 1e4, 30000];
63521
63753
  });
63522
63754
 
63523
- // dist/builtin/intercom/route-connect-gate.ts
63524
- function createConcurrencyGate(limit) {
63525
- let active = 0;
63526
- const waiting = [];
63527
- const release = () => {
63528
- const next = waiting.shift();
63529
- if (next === undefined)
63530
- active -= 1;
63531
- else
63532
- next();
63533
- };
63534
- return {
63535
- async run(operation) {
63536
- if (active < limit)
63537
- active += 1;
63538
- else
63539
- await new Promise((resolve) => waiting.push(resolve));
63540
- try {
63541
- return await operation();
63542
- } finally {
63543
- release();
63544
- }
63545
- }
63546
- };
63547
- }
63548
- var ROUTE_CLIENT_CONNECT_CONCURRENCY = 4;
63549
-
63550
63755
  // dist/builtin/intercom/workflow-route-owner.ts
63551
63756
  import { randomUUID as randomUUID14 } from "node:crypto";
63552
63757
  import { createEventBus, getBuiltinPackagePaths } from "@bastani/atomic";
63758
+ function closedError() {
63759
+ return new Error("Workflow Intercom route owner closed");
63760
+ }
63553
63761
  function createWorkflowRouteOwner(cwd) {
63554
63762
  const events = createEventBus();
63555
63763
  const config = loadConfig();
63556
63764
  const id = randomUUID14();
63557
63765
  const routes = new Map;
63558
- const gate = createConcurrencyGate(ROUTE_CLIENT_CONNECT_CONCURRENCY);
63766
+ const connections = new Map;
63767
+ const isCurrent = (state) => routes.get(state.route.runId) === state;
63768
+ const connectionKey = (state) => state.isolated ? state.route.runId : state.route.group;
63769
+ function connectionOf(state) {
63770
+ const key = connectionKey(state);
63771
+ let connection = connections.get(key);
63772
+ if (connection === undefined) {
63773
+ connection = { key, group: state.route.group, acknowledged: false, routes: new Map };
63774
+ connections.set(key, connection);
63775
+ }
63776
+ connection.routes.set(state.route.runId, state);
63777
+ return connection;
63778
+ }
63779
+ function detach(state) {
63780
+ const key = connectionKey(state);
63781
+ const connection = connections.get(key);
63782
+ connection?.routes.delete(state.route.runId);
63783
+ if (!state.isolated || connection === undefined)
63784
+ return;
63785
+ connections.delete(key);
63786
+ connection.client?.disconnect().catch(() => {});
63787
+ }
63788
+ function settle(state, registered) {
63789
+ state.registered = registered;
63790
+ if (!registered && isCurrent(state))
63791
+ return;
63792
+ for (const waiter of state.waiters)
63793
+ waiter(registered);
63794
+ state.waiters.clear();
63795
+ }
63559
63796
  function scheduleRetry(state) {
63560
- if (state.closed || state.retry !== undefined)
63797
+ if (!isCurrent(state) || state.retry !== undefined)
63561
63798
  return;
63562
63799
  state.retry = setTimeout(() => {
63563
63800
  state.retry = undefined;
63564
- publish(state, state.route).catch(() => scheduleRetry(state));
63801
+ publish(state, state.route).catch(() => {});
63565
63802
  }, jitteredReconnectDelayMs(state.attempt++));
63566
63803
  state.retry.unref();
63567
63804
  }
63568
- async function connect(state) {
63569
- if (state.client?.isConnected())
63570
- return state.client;
63571
- const [extensionDir] = getBuiltinPackagePaths({
63572
- workflows: false,
63573
- subagents: false,
63574
- mcp: false,
63575
- "web-access": false
63576
- });
63577
- if (extensionDir === undefined)
63578
- throw new Error("Intercom builtin package is unavailable");
63579
- await spawnBrokerIfNeeded(config.brokerCommand, config.brokerArgs, undefined, extensionDir);
63580
- if (state.closed)
63581
- throw new Error("Workflow Intercom route owner closed");
63582
- const client = new IntercomClient(`${id}:pending-stage-route:${state.route.runId}`);
63583
- state.client = client;
63584
- client.on("error", () => {});
63585
- client.on("disconnected", () => {
63586
- if (state.client !== client)
63587
- return;
63588
- state.client = undefined;
63589
- scheduleRetry(state);
63590
- });
63591
- client.on("pending_stage_message", (request) => {
63592
- if (state.closed || state.client !== client)
63593
- return;
63594
- const event = { ...request, handled: false };
63595
- events.emit("atomic:workflow-pending-stage-message", event);
63596
- const completion = event.handled && event.completion !== undefined ? event.completion : Promise.resolve({ outcome: "refused", reason: "Session not found" });
63597
- completion.catch((error) => ({ outcome: "refused", reason: error.message })).then((result) => client.respondPendingStageMessage(request.requestId, result)).catch(() => {});
63598
- });
63599
- client.on("sticky_live_delivered", (notice) => {
63600
- if (state.closed || state.client !== client)
63601
- return;
63602
- events.emit("atomic:workflow-sticky-live-delivered", { ...notice, handled: false });
63603
- });
63604
- await client.connect({
63605
- cwd,
63606
- model: "workflow",
63607
- pid: process.pid,
63608
- startedAt: Date.now(),
63609
- lastActivity: Date.now(),
63610
- group: state.route.group,
63611
- groups: [state.route.group],
63612
- recipientPurpose: "control"
63805
+ function drop(connection, client) {
63806
+ if (connection.client === client) {
63807
+ connection.client = undefined;
63808
+ connection.acknowledged = false;
63809
+ for (const state of connection.routes.values()) {
63810
+ settle(state, false);
63811
+ scheduleRetry(state);
63812
+ }
63813
+ }
63814
+ client.disconnect().catch(() => {});
63815
+ }
63816
+ function connect(connection) {
63817
+ if (connection.client?.isConnected())
63818
+ return Promise.resolve(connection.client);
63819
+ connection.connecting ??= (async () => {
63820
+ const [extensionDir] = getBuiltinPackagePaths({
63821
+ workflows: false,
63822
+ subagents: false,
63823
+ mcp: false,
63824
+ "web-access": false
63825
+ });
63826
+ if (extensionDir === undefined)
63827
+ throw new Error("Intercom builtin package is unavailable");
63828
+ await spawnBrokerIfNeeded(config.brokerCommand, config.brokerArgs, undefined, extensionDir);
63829
+ if (connections.get(connection.key) !== connection)
63830
+ throw closedError();
63831
+ const client = new IntercomClient(`${id}:pending-stage-route:${connection.key}`, {
63832
+ multiplexesPendingStageRoutes: true
63833
+ });
63834
+ connection.client = client;
63835
+ connection.acknowledged = false;
63836
+ client.on("error", () => {});
63837
+ client.on("disconnected", () => drop(connection, client));
63838
+ client.on("pending_stage_message", (request) => {
63839
+ if (connection.client !== client)
63840
+ return;
63841
+ const event = { ...request, handled: false };
63842
+ events.emit("atomic:workflow-pending-stage-message", event);
63843
+ const completion = event.handled && event.completion !== undefined ? event.completion : Promise.resolve({ outcome: "refused", reason: "Session not found" });
63844
+ completion.catch((error) => ({ outcome: "refused", reason: error.message })).then((result) => client.respondPendingStageMessage(request.requestId, result)).catch(() => {});
63845
+ });
63846
+ client.on("sticky_live_delivered", (notice) => {
63847
+ if (connection.client !== client)
63848
+ return;
63849
+ events.emit("atomic:workflow-sticky-live-delivered", { ...notice, handled: false });
63850
+ });
63851
+ try {
63852
+ await client.connect({
63853
+ cwd,
63854
+ model: "workflow",
63855
+ pid: process.pid,
63856
+ startedAt: Date.now(),
63857
+ lastActivity: Date.now(),
63858
+ group: connection.group,
63859
+ groups: [connection.group],
63860
+ recipientPurpose: "control"
63861
+ });
63862
+ if (connections.get(connection.key) !== connection)
63863
+ throw closedError();
63864
+ } catch (error) {
63865
+ drop(connection, client);
63866
+ throw error;
63867
+ }
63868
+ for (const state of connection.routes.values()) {
63869
+ if (state.registered || state.retry === undefined)
63870
+ continue;
63871
+ clearTimeout(state.retry);
63872
+ state.retry = undefined;
63873
+ publish(state, state.route).catch(() => {});
63874
+ }
63875
+ return client;
63876
+ })().finally(() => {
63877
+ connection.connecting = undefined;
63613
63878
  });
63614
- return client;
63879
+ return connection.connecting;
63615
63880
  }
63616
63881
  function publish(state, route) {
63617
- const completion = state.tail.catch(() => {}).then(() => gate.run(async () => {
63618
- if (state.closed)
63619
- throw new Error("Workflow Intercom route owner closed");
63620
- const establishedClient = state.client?.isConnected() ? state.client : undefined;
63882
+ const completion = state.tail.catch(() => {}).then(async () => {
63883
+ if (!isCurrent(state))
63884
+ throw closedError();
63885
+ const connection = connectionOf(state);
63886
+ const established = connection.acknowledged && connection.client?.isConnected() ? connection.client : undefined;
63887
+ let client;
63621
63888
  try {
63622
- const client = await connect(state);
63889
+ client = await connect(connection);
63623
63890
  await client.registerPendingStageRouteAcknowledged(route.runId, route.group, route.capability, route.stages, route.possibleStages, route.parent);
63891
+ if (connection.client !== client || !isCurrent(state))
63892
+ return;
63893
+ connection.acknowledged = true;
63624
63894
  state.attempt = 0;
63625
63895
  clearTimeout(state.retry);
63626
63896
  state.retry = undefined;
63897
+ settle(state, true);
63627
63898
  } catch (error) {
63628
- const client = state.client;
63629
- if (client !== establishedClient || !client?.isConnected()) {
63630
- state.client = undefined;
63631
- await client?.disconnect();
63899
+ if (error instanceof IntercomPendingStageRouteRefusedError && error.connectionKept) {
63900
+ scheduleRetry(state);
63901
+ throw error;
63632
63902
  }
63903
+ if (error instanceof IntercomPendingStageRouteRefusedError && !state.isolated && isCurrent(state)) {
63904
+ detach(state);
63905
+ state.isolated = true;
63906
+ connectionOf(state);
63907
+ }
63908
+ if (client !== undefined && (client !== established || !client.isConnected()))
63909
+ drop(connection, client);
63633
63910
  scheduleRetry(state);
63634
63911
  throw error;
63635
63912
  }
63636
- }));
63913
+ });
63637
63914
  state.tail = completion;
63638
63915
  completion.catch(() => {});
63639
63916
  return completion;
@@ -63642,10 +63919,15 @@ function createWorkflowRouteOwner(cwd) {
63642
63919
  const route = payload;
63643
63920
  let state = routes.get(route.runId);
63644
63921
  if (state === undefined) {
63645
- state = { route, tail: Promise.resolve(), attempt: 0, closed: false };
63922
+ state = { route, tail: Promise.resolve(), attempt: 0, registered: false, isolated: false, waiters: new Set };
63646
63923
  routes.set(route.runId, state);
63924
+ } else if (state.route.group !== route.group) {
63925
+ detach(state);
63926
+ state.isolated = false;
63927
+ settle(state, false);
63647
63928
  }
63648
63929
  state.route = route;
63930
+ connectionOf(state);
63649
63931
  route.completion = publish(state, route);
63650
63932
  });
63651
63933
  events.on("atomic:workflow-pending-stage-undeliverable", (payload) => {
@@ -63656,7 +63938,7 @@ function createWorkflowRouteOwner(cwd) {
63656
63938
  event.handled = true;
63657
63939
  event.completion = (async () => {
63658
63940
  await publish(state, state.route);
63659
- const client = state.client;
63941
+ const client = connections.get(connectionKey(state))?.client;
63660
63942
  if (client === undefined || !client.isConnected())
63661
63943
  return false;
63662
63944
  const text = `Pending workflow stage could not receive intercom message: ${event.reason}`;
@@ -63666,18 +63948,33 @@ function createWorkflowRouteOwner(cwd) {
63666
63948
  });
63667
63949
  return {
63668
63950
  events,
63951
+ registered(runId) {
63952
+ const state = routes.get(runId);
63953
+ if (state === undefined)
63954
+ return Promise.resolve(false);
63955
+ if (state.registered)
63956
+ return Promise.resolve(true);
63957
+ return new Promise((resolve) => state.waiters.add(resolve));
63958
+ },
63669
63959
  async closeGroup(group) {
63670
63960
  const closing = [...routes.values()].filter((state) => state.route.group === group);
63961
+ const closingConnections = [...connections.values()].filter((connection) => connection.group === group);
63962
+ for (const connection of closingConnections) {
63963
+ connections.delete(connection.key);
63964
+ connection.routes.clear();
63965
+ }
63671
63966
  for (const state of closing) {
63672
- state.closed = true;
63673
- clearTimeout(state.retry);
63674
63967
  routes.delete(state.route.runId);
63968
+ clearTimeout(state.retry);
63969
+ state.retry = undefined;
63970
+ settle(state, false);
63675
63971
  }
63676
- await Promise.all(closing.map(async (state) => {
63677
- await state.client?.disconnect();
63678
- await state.tail.catch(() => {});
63679
- await state.client?.disconnect();
63972
+ await Promise.all(closingConnections.map(async (connection) => {
63973
+ await connection.client?.disconnect();
63974
+ await connection.connecting?.catch(() => {});
63680
63975
  }));
63976
+ await Promise.all(closing.map((state) => state.tail.catch(() => {})));
63977
+ await Promise.all(closingConnections.map((connection) => connection.client?.disconnect()));
63681
63978
  }
63682
63979
  };
63683
63980
  }
@@ -63860,7 +64157,7 @@ var init_workflow_infrastructure_error = __esm(() => {
63860
64157
  [/Local worker (?:process )?cleanup (?:was not confirmed|unconfirmed)/, "worker_cleanup_unconfirmed"],
63861
64158
  [/Local worker retry limit/, "worker_retry_limit"],
63862
64159
  [/Local worker (?:startup timed out|exited before readiness)/, "worker_startup_failed"],
63863
- [/Worker exited before delivery acknowledgement/, "worker_lost"],
64160
+ [/Worker exited before delivery acknowledgement|Local worker pool coordinator exited/, "worker_lost"],
63864
64161
  [/Stale work lease/, "stale_work_lease"],
63865
64162
  [/Intercom runtime no longer active/, "intercom_unavailable"],
63866
64163
  [/Storage unavailable/, "storage_unavailable"]
@@ -64159,7 +64456,7 @@ var init_definitions = __esm(() => {
64159
64456
  // dist/builtin/workflows/src/durable/journal/pool-mutations.ts
64160
64457
  import { StorageProvider as StorageProvider2 } from "@bastani/atomic/storage";
64161
64458
  async function validatePoolMutation(tx, journal, root, work, mutation) {
64162
- if (!["checkpoint", "session-create", "session-append", "session-replace"].includes(mutation.kind))
64459
+ if (!["checkpoint", "session-create", "session-replace"].includes(mutation.kind))
64163
64460
  throw new Error("Invalid pool mutation");
64164
64461
  const checkpoint = mutation.checkpoint;
64165
64462
  const prefix = `${work.scope}:`;
@@ -64227,10 +64524,18 @@ async function sessionStore(tx, prefix) {
64227
64524
  return new StorageProvider2().openSessionStore({ kind: "database", database, prefix }, { agentDir: "", domain: "sessions" });
64228
64525
  }
64229
64526
  async function applyPoolSession(tx, prefix, workId, mutation) {
64527
+ const owners = `${prefix}session_owners`;
64528
+ if (mutation.kind === "session-append") {
64529
+ if (typeof mutation.sessionId !== "string" || mutation.sessionId === "")
64530
+ throw new Error("Pool sessions require database identity");
64531
+ const owner = await tx.get(`SELECT work FROM ${owners} WHERE session = ?`, mutation.sessionId);
64532
+ if (owner?.work !== workId)
64533
+ throw new Error("Pool session scope mismatch");
64534
+ return withSessionStore(tx, prefix, (store) => store.append(mutation.sessionId, mutation.entries, { expectSeq: mutation.expectSeq }));
64535
+ }
64230
64536
  const { sessionId, replayKey } = mutation.checkpoint;
64231
64537
  if (!sessionId || mutation.checkpoint.sessionFile !== undefined)
64232
64538
  throw new Error("Pool sessions require database identity");
64233
- const owners = `${prefix}session_owners`;
64234
64539
  if (mutation.kind === "session-create") {
64235
64540
  if (mutation.header.id !== sessionId)
64236
64541
  throw new Error("Pool session identity mismatch");
@@ -64240,10 +64545,12 @@ async function applyPoolSession(tx, prefix, workId, mutation) {
64240
64545
  if (owner?.work !== workId || owner.replay_key !== replayKey)
64241
64546
  throw new Error("Pool session scope mismatch");
64242
64547
  }
64548
+ return withSessionStore(tx, prefix, (store) => mutation.kind === "session-create" ? store.create(mutation.header, mutation.entries) : store.replace(sessionId, mutation.header, mutation.entries, { expectSeq: mutation.expectSeq }));
64549
+ }
64550
+ async function withSessionStore(tx, prefix, write) {
64243
64551
  const store = await sessionStore(tx, prefix);
64244
64552
  try {
64245
- const result = mutation.kind === "session-create" ? await store.create(mutation.header, mutation.entries) : mutation.kind === "session-append" ? await store.append(sessionId, mutation.entries, { expectSeq: mutation.expectSeq }) : await store.replace(sessionId, mutation.header, mutation.entries, { expectSeq: mutation.expectSeq });
64246
- return result.lastSeq;
64553
+ return (await write(store)).lastSeq;
64247
64554
  } finally {
64248
64555
  await store.close();
64249
64556
  }
@@ -64291,8 +64598,11 @@ var init_pool_stage_usage = __esm(() => {
64291
64598
  });
64292
64599
 
64293
64600
  // dist/builtin/workflows/src/durable/journal/sql-pool-queue.ts
64294
- import { randomUUID as randomUUID16 } from "node:crypto";
64601
+ import { createHash as createHash4, randomUUID as randomUUID16 } from "node:crypto";
64295
64602
  import { assertStorageSchemaVersion as assertStorageSchemaVersion2 } from "@bastani/atomic/storage";
64603
+ function requestIdentity(request) {
64604
+ return createHash4("sha256").update(request).digest("hex");
64605
+ }
64296
64606
  function fromRow(row) {
64297
64607
  return {
64298
64608
  ...JSON.parse(row.descriptor),
@@ -64348,10 +64658,11 @@ class SqlPoolQueue {
64348
64658
  const versions = await tx.all(`SELECT version FROM ${pool.meta}`);
64349
64659
  if (versions.length > 1)
64350
64660
  throw new Error("Corrupt pool schema metadata");
64351
- if (versions.length === 0)
64352
- await tx.run(`INSERT INTO ${pool.meta} VALUES (1)`);
64353
- else
64354
- assertStorageSchemaVersion2("journal", versions[0].version, 1);
64661
+ const version = versions[0]?.version;
64662
+ if (version === undefined)
64663
+ await tx.run(`INSERT INTO ${pool.meta} VALUES (?)`, POOL_SCHEMA_VERSION);
64664
+ else if (version !== 1)
64665
+ assertStorageSchemaVersion2("journal", version, POOL_SCHEMA_VERSION);
64355
64666
  await tx.exec(`
64356
64667
  CREATE TABLE IF NOT EXISTS ${pool.roots}(root TEXT PRIMARY KEY REFERENCES ${journal.runs}(run_id) ON DELETE CASCADE, timeout INTEGER NOT NULL);
64357
64668
  CREATE TABLE IF NOT EXISTS ${pool.work}(
@@ -64382,9 +64693,23 @@ class SqlPoolQueue {
64382
64693
  state TEXT NOT NULL DEFAULT 'running', revision INTEGER NOT NULL DEFAULT 0
64383
64694
  )`);
64384
64695
  await tx.exec(poolUsageSchema(journal.runs));
64696
+ if (version === 1)
64697
+ await pool.migrateFromVersion1(tx);
64385
64698
  });
64386
64699
  return pool;
64387
64700
  }
64701
+ async migrateFromVersion1(tx) {
64702
+ await tx.run(`DELETE FROM ${this.operations} WHERE substr(request, 1, 1) = '['`);
64703
+ for (let after = 0;; ) {
64704
+ const rows = await tx.all(`SELECT rowid, request FROM ${this.operations} WHERE rowid > ? ORDER BY rowid LIMIT 256`, after);
64705
+ if (rows.length === 0)
64706
+ break;
64707
+ for (const row of rows)
64708
+ await tx.run(`UPDATE ${this.operations} SET request = ? WHERE rowid = ?`, requestIdentity(String(row.request)), row.rowid);
64709
+ after = Number(rows.at(-1).rowid);
64710
+ }
64711
+ await tx.run(`UPDATE ${this.meta} SET version = ?`, POOL_SCHEMA_VERSION);
64712
+ }
64388
64713
  async authority(tx, lease, timeout) {
64389
64714
  const row = await tx.get(`SELECT l.*, r.status, p.timeout, ${STORE_NOW2} AS now FROM ${this.journal.leases} l JOIN ${this.journal.runs} r ON r.run_id = l.run_id LEFT JOIN ${this.roots} p ON p.root = l.run_id WHERE l.run_id = ?`, lease.runId);
64390
64715
  if (row?.timeout === null && timeout === undefined)
@@ -64525,31 +64850,18 @@ class SqlPoolQueue {
64525
64850
  throw new Error(STALE_WORK_LEASE);
64526
64851
  return row;
64527
64852
  }
64528
- async renew(lease, token, operationId, durationMs) {
64853
+ async renew(lease, token, durationMs) {
64529
64854
  positive(durationMs);
64530
64855
  lease = snapshotLease(lease);
64531
64856
  token = { ...token };
64532
- validateText("key", operationId);
64533
- const request = JSON.stringify([token.epoch, token.generation, token.worker, durationMs]);
64534
64857
  return this.transaction(async (tx) => {
64535
64858
  const { now } = await this.authority(tx, lease);
64536
64859
  const row = await this.current(tx, lease, token);
64537
64860
  if (row.epoch !== lease.epoch || row.expires_at < now)
64538
64861
  throw new Error(STALE_WORK_LEASE);
64539
- const previous = await tx.get(`SELECT * FROM ${this.operations} WHERE work = ? AND generation = ? AND operation = ?`, token.id, token.generation, operationId);
64540
- if (previous) {
64541
- if (previous.request !== request)
64542
- throw new Error("Pool operation identity conflict");
64543
- return JSON.parse(previous.receipt);
64544
- }
64545
- const receipt = {
64546
- ...fromRow(row),
64547
- expiresAt: Math.max(Number(row.expires_at), expiry(now, durationMs)),
64548
- renewedAt: now
64549
- };
64550
- await tx.run(`UPDATE ${this.work} SET expires_at = ?, renewed_at = ? WHERE id = ?`, receipt.expiresAt, now, row.id);
64551
- await tx.run(`INSERT INTO ${this.operations} VALUES (?, ?, ?, ?, ?)`, row.id, token.generation, operationId, request, JSON.stringify(receipt));
64552
- return receipt;
64862
+ const expiresAt = Math.max(Number(row.expires_at), expiry(now, durationMs));
64863
+ await tx.run(`UPDATE ${this.work} SET expires_at = ?, renewed_at = ? WHERE id = ?`, expiresAt, now, row.id);
64864
+ return { ...fromRow(row), expiresAt, renewedAt: now };
64553
64865
  });
64554
64866
  }
64555
64867
  async commit(lease, token, operationId, mutation) {
@@ -64558,6 +64870,7 @@ class SqlPoolQueue {
64558
64870
  validateText("key", operationId);
64559
64871
  assertWorkflowSerializableObject(mutation, "Pool mutation");
64560
64872
  const request = serializeJson({ epoch: token.epoch, worker: token.worker, mutation });
64873
+ const identity = requestIdentity(request);
64561
64874
  mutation = JSON.parse(request).mutation;
64562
64875
  return this.transaction(async (tx) => {
64563
64876
  const { now, status } = await this.authority(tx, lease);
@@ -64566,7 +64879,7 @@ class SqlPoolQueue {
64566
64879
  throw new Error(STALE_WORK_LEASE);
64567
64880
  const previous = await tx.get(`SELECT * FROM ${this.operations} WHERE work = ? AND generation = ? AND operation = ?`, token.id, token.generation, operationId);
64568
64881
  if (previous) {
64569
- if (previous.request !== request)
64882
+ if (previous.request !== identity)
64570
64883
  throw new Error("Pool operation identity conflict");
64571
64884
  return JSON.parse(previous.receipt);
64572
64885
  }
@@ -64574,34 +64887,41 @@ class SqlPoolQueue {
64574
64887
  throw new Error(STALE_WORK_LEASE);
64575
64888
  if (!await this.admitsAttemptWrites(tx, lease, status))
64576
64889
  throw new Error("Pool root is not running");
64577
- const checkpoint = mutation.checkpoint;
64578
- await validatePoolMutation(tx, this.journal, lease.runId, fromRow(row), mutation);
64579
- if (checkpoint.completedAt > Math.max(now, Date.now()) + MAX_WORKER_CLOCK_SKEW_MS)
64580
- throw new Error("Pool checkpoint timestamp is in the future");
64581
- assertWorkflowSerializableObject(checkpoint, "Pool checkpoint");
64582
- const key = JSON.stringify([RUN_EVENT_TYPES.checkpointRecorded, checkpoint.kind, checkpoint.checkpointId]);
64583
- const payload = serializeJson(checkpoint);
64584
- const existing = await tx.get(`SELECT seq, at, payload FROM ${this.journal.events} WHERE run_id = ? AND key = ?`, lease.runId, key);
64585
- if (existing && existing.payload !== payload)
64586
- throw new Error("Pool checkpoint identity conflict");
64587
- if (existing && mutation.kind !== "checkpoint")
64588
- throw new Error("Pool session checkpoint identity conflict");
64589
- const lastSeq = mutation.kind === "checkpoint" ? undefined : await applyPoolSession(tx, `${this.journal.runs}_pool_`, row.id, mutation);
64590
- const latest = await tx.get(`SELECT COALESCE(MAX(seq), 0) AS seq FROM ${this.journal.events} WHERE run_id = ?`, lease.runId);
64591
- const receipt = existing ? { seq: Number(existing.seq), at: Number(existing.at) } : {
64592
- seq: journalCounter(lease.runId, "seq", Number(latest.seq) + 1, "overflow"),
64593
- at: now,
64594
- ...lastSeq === undefined ? {} : { lastSeq }
64595
- };
64596
- if (!existing)
64597
- await tx.run(`INSERT INTO ${this.journal.events} VALUES (?, ?, ?, ?, ?, ?, ?)`, lease.runId, receipt.seq, lease.epoch, key, RUN_EVENT_TYPES.checkpointRecorded, payload, now);
64598
- const billed = poolCheckpointUsage(checkpoint);
64599
- if (billed && !existing)
64600
- await upsertPoolUsage(tx, this.journal.runs, row.id, token.generation, billed.key, billed.usage);
64601
- await tx.run(`INSERT INTO ${this.operations} VALUES (?, ?, ?, ?, ?)`, token.id, token.generation, operationId, request, JSON.stringify(receipt));
64890
+ const receipt = mutation.kind === "session-append" ? await this.appendSession(tx, row, mutation, now) : await this.recordCheckpoint(tx, lease, token, row, mutation, now);
64891
+ await tx.run(`INSERT INTO ${this.operations} VALUES (?, ?, ?, ?, ?)`, token.id, token.generation, operationId, identity, JSON.stringify(receipt));
64602
64892
  return receipt;
64603
64893
  });
64604
64894
  }
64895
+ async appendSession(tx, row, mutation, now) {
64896
+ return { at: now, lastSeq: await applyPoolSession(tx, `${this.journal.runs}_pool_`, row.id, mutation) };
64897
+ }
64898
+ async recordCheckpoint(tx, lease, token, row, mutation, now) {
64899
+ const checkpoint = mutation.checkpoint;
64900
+ await validatePoolMutation(tx, this.journal, lease.runId, fromRow(row), mutation);
64901
+ if (checkpoint.completedAt > Math.max(now, Date.now()) + MAX_WORKER_CLOCK_SKEW_MS)
64902
+ throw new Error("Pool checkpoint timestamp is in the future");
64903
+ assertWorkflowSerializableObject(checkpoint, "Pool checkpoint");
64904
+ const key = JSON.stringify([RUN_EVENT_TYPES.checkpointRecorded, checkpoint.kind, checkpoint.checkpointId]);
64905
+ const payload = serializeJson(checkpoint);
64906
+ const existing = await tx.get(`SELECT seq, at, payload FROM ${this.journal.events} WHERE run_id = ? AND key = ?`, lease.runId, key);
64907
+ if (existing && existing.payload !== payload)
64908
+ throw new Error("Pool checkpoint identity conflict");
64909
+ if (existing && mutation.kind !== "checkpoint")
64910
+ throw new Error("Pool session checkpoint identity conflict");
64911
+ const lastSeq = mutation.kind === "checkpoint" ? undefined : await applyPoolSession(tx, `${this.journal.runs}_pool_`, row.id, mutation);
64912
+ const latest = await tx.get(`SELECT COALESCE(MAX(seq), 0) AS seq FROM ${this.journal.events} WHERE run_id = ?`, lease.runId);
64913
+ const receipt = existing ? { seq: Number(existing.seq), at: Number(existing.at) } : {
64914
+ seq: journalCounter(lease.runId, "seq", Number(latest.seq) + 1, "overflow"),
64915
+ at: now,
64916
+ ...lastSeq === undefined ? {} : { lastSeq }
64917
+ };
64918
+ if (!existing)
64919
+ await tx.run(`INSERT INTO ${this.journal.events} VALUES (?, ?, ?, ?, ?, ?, ?)`, lease.runId, receipt.seq, lease.epoch, key, RUN_EVENT_TYPES.checkpointRecorded, payload, now);
64920
+ const billed = poolCheckpointUsage(checkpoint);
64921
+ if (billed && !existing)
64922
+ await upsertPoolUsage(tx, this.journal.runs, row.id, token.generation, billed.key, billed.usage);
64923
+ return receipt;
64924
+ }
64605
64925
  async complete(lease, token, result) {
64606
64926
  lease = snapshotLease(lease);
64607
64927
  token = { ...token };
@@ -64688,7 +65008,7 @@ function openSqlPoolQueue(journal) {
64688
65008
  return Promise.reject(new Error("Worker pool requires a SQL run journal"));
64689
65009
  return SqlPoolQueue.open(storage);
64690
65010
  }
64691
- var STORE_NOW2 = "CAST(unixepoch('subsec') * 1000 AS INTEGER)", MAX_WORKER_CLOCK_SKEW_MS = 1000, STALE_WORK_LEASE = "Stale work lease";
65011
+ var STORE_NOW2 = "CAST(unixepoch('subsec') * 1000 AS INTEGER)", MAX_WORKER_CLOCK_SKEW_MS = 1000, POOL_SCHEMA_VERSION = 2, STALE_WORK_LEASE = "Stale work lease";
64692
65012
  var init_sql_pool_queue = __esm(() => {
64693
65013
  init_serializable();
64694
65014
  init_boundary_topology();
@@ -64700,6 +65020,15 @@ var init_sql_pool_queue = __esm(() => {
64700
65020
  init_values();
64701
65021
  });
64702
65022
 
65023
+ // dist/builtin/workflows/src/workers/coordinator-protocol.ts
65024
+ var POOL_COORDINATOR_ROLE = "pool-coordinator", POOL_SUMMARY_INTERVAL_MS = 250;
65025
+
65026
+ // dist/builtin/workflows/src/workers/host-options.ts
65027
+ var hostWorkerBindings;
65028
+ var init_host_options = __esm(() => {
65029
+ hostWorkerBindings = Symbol("host-worker-bindings");
65030
+ });
65031
+
64703
65032
  // dist/builtin/workflows/src/durable/journal/sql-pool-usage.ts
64704
65033
  function validate(report) {
64705
65034
  validateText("key", report.key);
@@ -64770,10 +65099,844 @@ var init_sql_pool_usage = __esm(() => {
64770
65099
  init_values();
64771
65100
  });
64772
65101
 
64773
- // dist/builtin/workflows/src/workers/host-options.ts
64774
- var hostWorkerBindings;
64775
- var init_host_options = __esm(() => {
64776
- hostWorkerBindings = Symbol("host-worker-bindings");
65102
+ // dist/builtin/workflows/src/workers/pool-coordinator.ts
65103
+ import { fork } from "node:child_process";
65104
+ import { randomUUID as randomUUID17 } from "node:crypto";
65105
+ import { createWriteStream } from "node:fs";
65106
+ import { mkdir as mkdir6, rm as rm3, rmdir } from "node:fs/promises";
65107
+ import { dirname as dirname14, join as join29, resolve as resolve13 } from "node:path";
65108
+ import { setTimeout as delay2 } from "node:timers/promises";
65109
+ function workerLogDirectory(journalPath, root) {
65110
+ return join29(dirname14(resolve13(journalPath)), "worker-logs", encodeURIComponent(root).replaceAll(".", "%2E"));
65111
+ }
65112
+ function token(work) {
65113
+ return {
65114
+ id: work.id,
65115
+ generation: work.generation,
65116
+ worker: work.worker,
65117
+ epoch: work.epoch,
65118
+ renewedAt: work.renewedAt,
65119
+ expiresAt: work.expiresAt
65120
+ };
65121
+ }
65122
+ function outside(runId) {
65123
+ return new Error(`Pool stage publication is outside the work subtree: ${runId}`);
65124
+ }
65125
+ function isLiveStage(stage) {
65126
+ return (stage.sessionId !== undefined || stage.sessionFile !== undefined) && stage.status !== "completed" && stage.status !== "failed" && stage.status !== "skipped";
65127
+ }
65128
+ function projectRootRun(run) {
65129
+ return {
65130
+ id: run.id,
65131
+ name: run.name,
65132
+ status: run.status,
65133
+ startedAt: run.startedAt,
65134
+ stages: run.stages.map((stage) => ({
65135
+ id: stage.id,
65136
+ name: stage.name,
65137
+ status: stage.status,
65138
+ parentIds: [],
65139
+ toolEvents: [],
65140
+ ...stage.replayKey === undefined ? {} : { replayKey: stage.replayKey },
65141
+ ...stage.intercomGroup === undefined ? {} : { intercomGroup: stage.intercomGroup },
65142
+ ...stage.workflowChildRun === undefined ? {} : { workflowChildRun: stage.workflowChildRun },
65143
+ ...stage.workflowChild === undefined ? {} : { workflowChild: { ...stage.workflowChild, outputs: {} } }
65144
+ }))
65145
+ };
65146
+ }
65147
+
65148
+ class PooledRoutes {
65149
+ root;
65150
+ owner;
65151
+ mirror = createStore();
65152
+ owners = new Map;
65153
+ attempts = new Map;
65154
+ activated = false;
65155
+ bridge;
65156
+ constructor(root, owner) {
65157
+ this.root = root;
65158
+ this.owner = owner;
65159
+ }
65160
+ static async open(root, cwd, rootRun, forward) {
65161
+ await Promise.resolve().then(() => init_workflow_route_owner());
65162
+ const owner = createWorkflowRouteOwner(cwd);
65163
+ const routes = new PooledRoutes(root, owner);
65164
+ if (rootRun !== undefined)
65165
+ routes.updateRoot(rootRun);
65166
+ routes.bridge = registerPendingStageIntercomBridge({
65167
+ events: {
65168
+ emit: (event, payload) => owner.events.emit(event, payload),
65169
+ on: (event, listener) => event === PENDING_STAGE_MESSAGE_EVENT2 || event === STICKY_LIVE_DELIVERED_EVENT2 ? undefined : owner.events.on(event, listener)
65170
+ }
65171
+ }, routes.mirror, (runId) => routes.activated && routes.owners.has(runId));
65172
+ for (const [event, kind] of [
65173
+ [PENDING_STAGE_MESSAGE_EVENT2, "pending-stage-message"],
65174
+ [STICKY_LIVE_DELIVERED_EVENT2, "sticky-live-delivered"]
65175
+ ])
65176
+ owner.events.on(event, (payload) => {
65177
+ const claimed = payload;
65178
+ if (claimed.handled !== false)
65179
+ return;
65180
+ claimed.handled = true;
65181
+ const { handled: _handled, completion: _completion, ...forwarded } = claimed;
65182
+ claimed.completion = forward({ kind, payload: forwarded });
65183
+ });
65184
+ return routes;
65185
+ }
65186
+ updateRoot(rootRun) {
65187
+ this.mirror.recordRunStart({ ...rootRun, inputs: {}, toolNodes: [], rootRunId: this.root });
65188
+ }
65189
+ accept(entries) {
65190
+ this.mirror.batch(() => {
65191
+ for (const { work, runs } of entries) {
65192
+ let attempt = this.attempts.get(work.id);
65193
+ if (attempt !== undefined && work.generation < attempt.generation)
65194
+ continue;
65195
+ if (attempt === undefined || attempt.generation !== work.generation) {
65196
+ if (attempt !== undefined)
65197
+ this.release(attempt);
65198
+ attempt = { generation: work.generation, runs: attempt?.runs ?? new Set, fenced: false };
65199
+ this.attempts.set(work.id, attempt);
65200
+ }
65201
+ if (attempt.fenced)
65202
+ continue;
65203
+ for (const run of runs) {
65204
+ attempt.runs.add(run.id);
65205
+ this.owners.set(run.id, work.id);
65206
+ const { pendingStageMessages: _messages, ...published } = run;
65207
+ this.mirror.recordRunStart({ ...published, rootRunId: this.root });
65208
+ if (run.stages.some((stage) => stage.pendingStageDeliveryAvailable === true))
65209
+ this.activated = true;
65210
+ }
65211
+ }
65212
+ });
65213
+ }
65214
+ fence(work) {
65215
+ const attempt = this.attempts.get(work.id);
65216
+ if (attempt === undefined || attempt.generation > work.generation)
65217
+ return;
65218
+ if (attempt.generation < work.generation) {
65219
+ this.release(attempt);
65220
+ this.attempts.set(work.id, { generation: work.generation, runs: attempt.runs, fenced: true });
65221
+ return;
65222
+ }
65223
+ this.release(attempt);
65224
+ attempt.fenced = true;
65225
+ }
65226
+ async route(work, runId) {
65227
+ this.requireOwned(work, runId);
65228
+ const readiness = workflowPendingStageRouteReady(this.mirror, runId);
65229
+ await readiness?.completion.catch(() => {});
65230
+ readiness?.assertCurrent();
65231
+ const waiting = new AbortController;
65232
+ await Promise.race([
65233
+ this.owner.registered(runId),
65234
+ delay2(ROUTE_REGISTRATION_WAIT_MS, undefined, { signal: waiting.signal, ref: false }).catch(() => {})
65235
+ ]);
65236
+ waiting.abort();
65237
+ this.requireOwned(work, runId);
65238
+ return workflowPendingStageRouteCapability(this.mirror, runId);
65239
+ }
65240
+ async undeliverable(payload) {
65241
+ const event = { ...payload, handled: false };
65242
+ this.owner.events.emit(PENDING_STAGE_UNDELIVERABLE_EVENT2, event);
65243
+ return event.handled && await event.completion === true;
65244
+ }
65245
+ async close() {
65246
+ const bridge = this.bridge;
65247
+ if (bridge === undefined)
65248
+ return;
65249
+ this.bridge = undefined;
65250
+ try {
65251
+ await bridge.flush();
65252
+ } finally {
65253
+ bridge.refresh();
65254
+ bridge();
65255
+ await this.owner.closeGroup(workflowInvocationIntercomGroup(this.root));
65256
+ }
65257
+ }
65258
+ requireOwned(work, runId) {
65259
+ const attempt = this.attempts.get(work.id);
65260
+ if (attempt === undefined || attempt.generation !== work.generation || attempt.fenced)
65261
+ throw new Error(STALE_WORK_LEASE);
65262
+ if (!attempt.runs.has(runId))
65263
+ throw outside(runId);
65264
+ }
65265
+ release(attempt) {
65266
+ this.mirror.batch(() => {
65267
+ for (const id of attempt.runs) {
65268
+ rotateWorkflowPendingStageRouteCapability(this.mirror, id);
65269
+ const run = this.mirror.runs().find((candidate) => candidate.id === id);
65270
+ if (run?.stages.some(isLiveStage) !== true)
65271
+ continue;
65272
+ this.mirror.recordRunStart({
65273
+ ...run,
65274
+ stages: run.stages.map((stage) => {
65275
+ if (!isLiveStage(stage))
65276
+ return stage;
65277
+ const { sessionId: _session, sessionFile: _file, ...rest } = stage;
65278
+ return { ...rest, attachable: false, attached: false };
65279
+ })
65280
+ });
65281
+ }
65282
+ });
65283
+ }
65284
+ }
65285
+
65286
+ class PoolCoordinator {
65287
+ port;
65288
+ children = new Set;
65289
+ incarnations = new Map;
65290
+ readyWorkers = new Set;
65291
+ logs = new Map;
65292
+ cleanups = new Set;
65293
+ controlAcks = new Map;
65294
+ lastHeard = new Map;
65295
+ pendingActivity = new Map;
65296
+ batches = new Map;
65297
+ publicationFailures = new Map;
65298
+ requests = new Map;
65299
+ deliveryRequests = new Map;
65300
+ inflight = new Set;
65301
+ lastTick = performance.now();
65302
+ stalledMs = 0;
65303
+ ticker;
65304
+ recoveryTimer;
65305
+ flushTimer;
65306
+ lastFlushAt = 0;
65307
+ batchSequence = 0;
65308
+ requestSequence = 0;
65309
+ usageChanged = false;
65310
+ budgetBlocked = false;
65311
+ desiredControl;
65312
+ root;
65313
+ lease;
65314
+ options;
65315
+ journal;
65316
+ queue;
65317
+ usageLedger;
65318
+ usageTotal;
65319
+ admission;
65320
+ routes;
65321
+ publishing = false;
65322
+ sources = [];
65323
+ logDirectory;
65324
+ starting;
65325
+ recovering;
65326
+ nextRecoveryAt = 0;
65327
+ cleanupFailure;
65328
+ closed = false;
65329
+ closing;
65330
+ replacements = 0;
65331
+ constructor(port) {
65332
+ this.port = port;
65333
+ port.onMessage((message) => this.receive(message));
65334
+ this.ticker = setInterval(() => this.observeResponsiveness(), TICK_INTERVAL_MS).unref();
65335
+ }
65336
+ async shutdown() {
65337
+ await this.closing?.catch(() => {});
65338
+ while (this.inflight.size > 0)
65339
+ await Promise.allSettled([...this.inflight]);
65340
+ await this.journal?.close().catch(() => {});
65341
+ }
65342
+ receive(message) {
65343
+ if (message.kind === "request-reply") {
65344
+ const pending = this.requests.get(message.id);
65345
+ this.requests.delete(message.id);
65346
+ if (message.error !== undefined)
65347
+ pending?.reject(new Error(message.error));
65348
+ else
65349
+ pending?.resolve(message.value);
65350
+ return;
65351
+ }
65352
+ if ("id" in message) {
65353
+ const command = message;
65354
+ const handled = this.execute(command).then((value2) => this.port.send({ kind: "reply", id: command.id, value: value2 }), (error) => this.port.send({ kind: "reply", id: command.id, error: error.message }));
65355
+ this.inflight.add(handled);
65356
+ handled.finally(() => this.inflight.delete(handled));
65357
+ return;
65358
+ }
65359
+ this.notice(message);
65360
+ }
65361
+ notice(notice) {
65362
+ if (notice.kind === "budget")
65363
+ this.budgetBlocked = notice.blocked;
65364
+ else if (notice.kind === "root")
65365
+ this.routes?.updateRoot(notice.rootRun);
65366
+ else {
65367
+ const batch = this.batches.get(notice.batch);
65368
+ this.batches.delete(notice.batch);
65369
+ const rejected = new Map(notice.rejected.map((entry) => [entry.workId, entry]));
65370
+ for (const entry of rejected.values())
65371
+ this.publicationFailures.set(entry.workId, { generation: entry.generation, error: entry.error });
65372
+ this.routes?.accept((batch?.entries ?? []).filter((entry) => rejected.get(entry.work.id)?.generation !== entry.work.generation));
65373
+ batch?.settle();
65374
+ }
65375
+ }
65376
+ async execute(command) {
65377
+ switch (command.kind) {
65378
+ case "prepare":
65379
+ return this.prepare(command.root, command.lease, command.options, command.control);
65380
+ case "control":
65381
+ return this.control(command.state);
65382
+ case "start":
65383
+ this.starting ??= this.start(command.sources, command.cwd, command.publish, command.rootRun);
65384
+ return this.starting;
65385
+ case "enqueue":
65386
+ return this.enqueue(command);
65387
+ case "transport":
65388
+ return this.transport(command.request, command);
65389
+ case "undeliverable":
65390
+ return await this.routes?.undeliverable(command.payload) ?? false;
65391
+ case "close":
65392
+ this.closing ??= this.close();
65393
+ return this.closing;
65394
+ }
65395
+ }
65396
+ async prepare(root, lease, options, control) {
65397
+ if (this.closed)
65398
+ throw new Error("Local worker pool is closed");
65399
+ if (this.root !== undefined && this.root !== root)
65400
+ throw new Error("Local worker pool belongs to another root");
65401
+ this.root = root;
65402
+ this.options = options;
65403
+ this.desiredControl ??= control;
65404
+ this.journal ??= await resolveRunJournal({ journal: { path: options.journalPath } });
65405
+ this.usageLedger ??= await openSqlPoolUsage(this.journal);
65406
+ this.usageTotal = await this.usageLedger.total(root);
65407
+ this.lease = lease;
65408
+ const queue = await openSqlPoolQueue(this.journal);
65409
+ await queue.initialize(lease, COORDINATOR_LEASE_MS);
65410
+ await queue.releasePreviousIncarnations(lease);
65411
+ for (;; ) {
65412
+ const intended = this.desiredControl ?? "running";
65413
+ await queue.setControl(lease, intended);
65414
+ if (intended === (this.desiredControl ?? "running"))
65415
+ break;
65416
+ }
65417
+ this.queue = queue;
65418
+ return this.usageTotal;
65419
+ }
65420
+ async control(state) {
65421
+ this.desiredControl = state;
65422
+ if (!this.queue || !this.lease)
65423
+ return;
65424
+ const control = await this.queue.setControl(this.lease, state);
65425
+ const deadline = Date.now() + CONTROL_ACKNOWLEDGEMENT_TIMEOUT_MS;
65426
+ while ([...this.readyWorkers].some((worker) => (this.controlAcks.get(worker) ?? -1) < control.revision)) {
65427
+ if (this.desiredControl !== state)
65428
+ return;
65429
+ if (Date.now() >= deadline) {
65430
+ if (state === "quit")
65431
+ break;
65432
+ throw new Error("Worker control acknowledgement unresolved");
65433
+ }
65434
+ await delay2(25);
65435
+ }
65436
+ this.flush();
65437
+ }
65438
+ async start(sources, cwd, publish, rootRun) {
65439
+ if (this.closed)
65440
+ throw new Error("Local worker pool is closed");
65441
+ const root = this.root;
65442
+ if (!this.queue || !this.lease || !this.options || root === undefined || this.lease.runId !== root)
65443
+ throw new Error("Local worker pool requires root preparation");
65444
+ this.sources = sources;
65445
+ this.publishing = publish;
65446
+ if (publish) {
65447
+ this.routes = await PooledRoutes.open(root, cwd, rootRun, (request) => this.request(request));
65448
+ if (this.closed) {
65449
+ await this.routes.close();
65450
+ throw new Error("Local worker pool is closed");
65451
+ }
65452
+ }
65453
+ if (this.options.admission)
65454
+ this.admission = await SqlResourceAdmission.open(this.journal, {
65455
+ ...this.options.admission,
65456
+ root,
65457
+ holder: "coordinator"
65458
+ });
65459
+ this.logDirectory = workerLogDirectory(this.options.journalPath, root);
65460
+ await mkdir6(this.logDirectory, { recursive: true }).catch(() => {
65461
+ return;
65462
+ });
65463
+ this.recoveryTimer = setInterval(() => {
65464
+ this.recoverExpired().catch((error) => this.fail(error));
65465
+ }, RECOVERY_INTERVAL_MS).unref();
65466
+ const workers = Array.from({ length: this.options.workers }, () => this.spawn());
65467
+ try {
65468
+ await Promise.any(workers);
65469
+ } catch (error) {
65470
+ throw error.errors[0];
65471
+ }
65472
+ }
65473
+ async enqueue(command) {
65474
+ if (!this.queue || !this.lease || this.root === undefined)
65475
+ throw new Error("Local worker pool requires root preparation");
65476
+ if (command.rootRun !== undefined)
65477
+ this.routes?.updateRoot(command.rootRun);
65478
+ const work = await this.queue.enqueue(this.lease, command.descriptor, command.boundary);
65479
+ const result = await this.queue.result(this.root, work.id);
65480
+ return { work, ...result === undefined ? {} : { result } };
65481
+ }
65482
+ async transport(request, payload) {
65483
+ const target = this.deliveryRequests.get(request);
65484
+ if (target === undefined || !this.incarnations.has(target.slot.worker))
65485
+ throw new Error("Worker exited before delivery acknowledgement");
65486
+ const { slot, workerRequest } = target;
65487
+ await new Promise((resolve, reject) => {
65488
+ const id = randomUUID17();
65489
+ slot.deliveries.set(id, { resolve, reject });
65490
+ slot.child.send({ id: workerRequest, delivery: { id, from: payload.from, message: payload.message } }, (error) => {
65491
+ if (error) {
65492
+ slot.deliveries.delete(id);
65493
+ reject(error);
65494
+ }
65495
+ });
65496
+ });
65497
+ }
65498
+ request(body) {
65499
+ const id = ++this.requestSequence;
65500
+ return new Promise((resolve, reject) => {
65501
+ this.requests.set(id, { resolve, reject });
65502
+ this.port.send({ kind: "request", id, request: body });
65503
+ });
65504
+ }
65505
+ fail(error) {
65506
+ this.port.send({ kind: "failure", error: error.message });
65507
+ }
65508
+ publish(work, runs) {
65509
+ const failure = this.publicationFailures.get(work.id);
65510
+ if (failure?.generation === work.generation)
65511
+ throw new Error(failure.error);
65512
+ if (!this.publishing)
65513
+ return;
65514
+ this.pendingActivity.set(work.id, { work: token(work), runs });
65515
+ this.scheduleFlush();
65516
+ }
65517
+ scheduleFlush() {
65518
+ if (this.flushTimer !== undefined || this.closed)
65519
+ return;
65520
+ const wait = Math.max(0, this.lastFlushAt + POOL_SUMMARY_INTERVAL_MS - performance.now());
65521
+ this.flushTimer = setTimeout(() => {
65522
+ this.flushTimer = undefined;
65523
+ if (this.batches.size > 0) {
65524
+ this.lastFlushAt = performance.now();
65525
+ this.scheduleFlush();
65526
+ return;
65527
+ }
65528
+ this.flush();
65529
+ }, wait);
65530
+ }
65531
+ flush() {
65532
+ clearTimeout(this.flushTimer);
65533
+ this.flushTimer = undefined;
65534
+ if (this.pendingActivity.size === 0 && !this.usageChanged) {
65535
+ const last = [...this.batches.values()].at(-1);
65536
+ return last === undefined ? Promise.resolve() : new Promise((resolve) => this.chainSettle(last, resolve));
65537
+ }
65538
+ this.lastFlushAt = performance.now();
65539
+ const batch = ++this.batchSequence;
65540
+ const entries = [...this.pendingActivity.values()];
65541
+ this.pendingActivity.clear();
65542
+ const usage = this.usageChanged ? this.usageTotal : undefined;
65543
+ this.usageChanged = false;
65544
+ const settled = new Promise((resolve) => this.batches.set(batch, { entries, settle: resolve }));
65545
+ this.port.send({ kind: "activity", batch, entries, ...usage === undefined ? {} : { usage } });
65546
+ return settled;
65547
+ }
65548
+ chainSettle(batch, resolve) {
65549
+ const settle = batch.settle;
65550
+ batch.settle = () => {
65551
+ settle();
65552
+ resolve();
65553
+ };
65554
+ }
65555
+ noteUsage(usage) {
65556
+ this.usageTotal = usage;
65557
+ this.usageChanged = true;
65558
+ this.scheduleFlush();
65559
+ }
65560
+ async sendResult(workId) {
65561
+ const result = await this.queue.result(this.root, workId);
65562
+ if (result === undefined)
65563
+ return false;
65564
+ this.flush();
65565
+ this.port.send({
65566
+ kind: "result",
65567
+ workId,
65568
+ result,
65569
+ ...this.usageTotal === undefined ? {} : { usage: this.usageTotal }
65570
+ });
65571
+ return true;
65572
+ }
65573
+ fence(work) {
65574
+ this.pendingActivity.delete(work.id);
65575
+ this.routes?.fence(work);
65576
+ this.port.send({ kind: "fence", work: { id: work.id, generation: work.generation } });
65577
+ }
65578
+ spawn() {
65579
+ if (this.closed)
65580
+ return Promise.reject(new Error("Local worker pool is closed"));
65581
+ const root = this.root;
65582
+ const options = this.options;
65583
+ const child = fork(options.workerEntry ?? process.argv[1], [
65584
+ JSON.stringify({
65585
+ journalPath: options.journalPath,
65586
+ root,
65587
+ definitions: this.sources,
65588
+ admission: options.admission
65589
+ })
65590
+ ], { stdio: ["ignore", "ignore", "pipe", "ipc"], execArgv: [] });
65591
+ const disconnected = new Promise((resolve) => child.once("disconnect", () => resolve()));
65592
+ this.children.add(child);
65593
+ if (child.pid !== undefined)
65594
+ this.port.send({ kind: "spawned", pid: child.pid });
65595
+ const worker = randomUUID17();
65596
+ this.incarnations.set(worker, child);
65597
+ const slot = { worker, child, deliveries: new Map };
65598
+ let ready = false;
65599
+ let serial = Promise.resolve();
65600
+ const logPath = join29(this.logDirectory, `${worker}.log`);
65601
+ this.logs.set(child, logPath);
65602
+ let logFailure;
65603
+ const log = createWriteStream(logPath, { flags: "a" });
65604
+ log.on("error", (error) => {
65605
+ logFailure = error.message;
65606
+ });
65607
+ let stderr = "";
65608
+ child.stderr?.on("data", (chunk) => {
65609
+ stderr = (stderr + chunk.toString()).slice(-STDERR_TAIL_CHARACTERS);
65610
+ });
65611
+ child.stderr?.on("error", (error) => {
65612
+ logFailure ??= error.message;
65613
+ log.end();
65614
+ });
65615
+ child.stderr?.pipe(log);
65616
+ const diagnosed = (summary) => new Error(`${summary} (worker stderr log: ${logFailure === undefined ? logPath : `unavailable, ${logFailure}`})${stderr ? `: ${stderr}` : ""}`);
65617
+ const startupTimeoutMs = options.startupTimeoutMs;
65618
+ let timedOut = false;
65619
+ let terminated = false;
65620
+ const terminate = () => {
65621
+ if (terminated)
65622
+ return;
65623
+ terminated = true;
65624
+ child.kill("SIGKILL");
65625
+ };
65626
+ const replied = (error) => {
65627
+ if (error)
65628
+ terminate();
65629
+ };
65630
+ return new Promise((resolve, reject) => {
65631
+ const startup = setTimeout(() => {
65632
+ timedOut = true;
65633
+ terminate();
65634
+ }, startupTimeoutMs);
65635
+ child.on("error", (error) => {
65636
+ clearTimeout(startup);
65637
+ reject(error);
65638
+ terminate();
65639
+ });
65640
+ const closed = new Promise((resolve) => child.once("close", () => resolve()));
65641
+ child.once("exit", () => {
65642
+ Promise.race([closed, delay2(STDERR_DRAIN_GRACE_MS)]).then(() => reject(diagnosed(timedOut ? `Local worker startup timed out after ${startupTimeoutMs} ms` : "Local worker exited before readiness")));
65643
+ this.children.delete(child);
65644
+ this.logs.delete(child);
65645
+ if (child.pid !== undefined)
65646
+ this.port.send({ kind: "exited", pid: child.pid });
65647
+ clearTimeout(startup);
65648
+ this.incarnations.delete(worker);
65649
+ this.readyWorkers.delete(worker);
65650
+ this.controlAcks.delete(worker);
65651
+ this.lastHeard.delete(worker);
65652
+ if (slot.work)
65653
+ this.fence(slot.work);
65654
+ for (const delivery of slot.deliveries.values())
65655
+ delivery.reject(new Error("Worker exited before delivery acknowledgement"));
65656
+ slot.deliveries.clear();
65657
+ const cleanup = disconnected.then(() => serial).then(async () => {
65658
+ try {
65659
+ if (await this.admission?.releaseStoppedSessions(worker))
65660
+ throw new Error("Local worker process cleanup unconfirmed; permits quarantined");
65661
+ } catch (error) {
65662
+ this.cleanupFailure = error instanceof Error ? error : new Error(String(error));
65663
+ throw this.cleanupFailure;
65664
+ }
65665
+ }).then(async () => {
65666
+ if (this.closed || !ready || this.desiredControl === "quit")
65667
+ return;
65668
+ let work = slot.work;
65669
+ if (work && await this.sendResult(work.id)) {
65670
+ work = undefined;
65671
+ slot.work = undefined;
65672
+ this.replacements = 0;
65673
+ }
65674
+ if (this.closed)
65675
+ return;
65676
+ if (work)
65677
+ await this.queue.releaseExited(this.lease, work);
65678
+ const exhausted = work ? work.generation >= 3 : ++this.replacements > 3;
65679
+ if (exhausted)
65680
+ this.fail(diagnosed("Local worker retry limit"));
65681
+ else
65682
+ await this.spawn();
65683
+ }).catch((error) => {
65684
+ this.fail(error);
65685
+ });
65686
+ this.cleanups.add(cleanup);
65687
+ cleanup.finally(() => this.cleanups.delete(cleanup));
65688
+ });
65689
+ child.on("message", (message) => {
65690
+ this.observeResponsiveness();
65691
+ this.lastHeard.set(worker, { at: performance.now(), stalled: this.stalledMs });
65692
+ if (message.kind === "delivery-ack") {
65693
+ const pending = slot.deliveries.get(message.deliveryId);
65694
+ slot.deliveries.delete(message.deliveryId);
65695
+ if (message.error)
65696
+ pending?.reject(new Error(message.error));
65697
+ else
65698
+ pending?.resolve();
65699
+ return;
65700
+ }
65701
+ let routeReady;
65702
+ const handled = serial.then(async () => {
65703
+ if (message.kind === "ready") {
65704
+ if (terminated)
65705
+ return;
65706
+ ready = true;
65707
+ this.readyWorkers.add(worker);
65708
+ clearTimeout(startup);
65709
+ resolve();
65710
+ return;
65711
+ }
65712
+ await this.starting;
65713
+ const queue = this.queue;
65714
+ const lease = this.lease;
65715
+ if (message.kind === "control")
65716
+ return queue.control(lease);
65717
+ if (message.kind === "controlled") {
65718
+ const control = await queue.control(lease);
65719
+ if (message.revision > control.revision)
65720
+ throw new Error("Unknown control revision");
65721
+ this.controlAcks.set(worker, Math.max(this.controlAcks.get(worker) ?? -1, message.revision));
65722
+ return;
65723
+ }
65724
+ if (message.kind === "claim") {
65725
+ if (slot.work)
65726
+ throw new Error("Worker already holds an attempt");
65727
+ if (this.budgetBlocked)
65728
+ return;
65729
+ await this.recoverExpired();
65730
+ slot.work = await queue.claim(lease, worker, this.sources.map((source) => ({
65731
+ definition: `${source.path}#${source.exportName}`,
65732
+ digest: source.digest
65733
+ })), options.leaseMs);
65734
+ return slot.work;
65735
+ }
65736
+ let work = slot.work;
65737
+ if (!work)
65738
+ throw new Error("Worker has no active attempt");
65739
+ if (message.kind === "renew" || work.expiresAt - Date.now() < options.leaseMs / 2) {
65740
+ work = await queue.extend(lease, work, options.leaseMs);
65741
+ slot.work = work;
65742
+ }
65743
+ if (message.kind === "activity")
65744
+ return this.publish(work, message.runs);
65745
+ if (message.kind === "route") {
65746
+ if (this.routes === undefined)
65747
+ return;
65748
+ await this.flush();
65749
+ this.assertPublished(work);
65750
+ routeReady = this.routes.route(work, message.runId);
65751
+ return;
65752
+ }
65753
+ if (message.kind === "delivery-ready") {
65754
+ if (!this.publishing)
65755
+ return;
65756
+ await this.flush();
65757
+ this.assertPublished(work);
65758
+ return this.request({
65759
+ kind: "ready",
65760
+ work: token(work),
65761
+ runId: message.runId,
65762
+ stageId: message.stageId,
65763
+ stageName: message.stageName,
65764
+ ...message.sessionId === undefined ? {} : { sessionId: message.sessionId }
65765
+ });
65766
+ }
65767
+ if (message.kind === "delivery") {
65768
+ if (!message.recipient)
65769
+ throw new Error("Worker delivery requires a session identity");
65770
+ if (!this.publishing)
65771
+ return;
65772
+ await this.flush();
65773
+ this.assertPublished(work);
65774
+ const id = ++this.requestSequence;
65775
+ this.deliveryRequests.set(id, { slot, workerRequest: message.id });
65776
+ try {
65777
+ return await new Promise((resolve, reject) => {
65778
+ this.requests.set(id, { resolve, reject });
65779
+ this.port.send({
65780
+ kind: "request",
65781
+ id,
65782
+ request: {
65783
+ kind: "deliver",
65784
+ work: token(work),
65785
+ runId: message.runId,
65786
+ stageId: message.stageId,
65787
+ stageName: message.stageName,
65788
+ recipient: message.recipient
65789
+ }
65790
+ });
65791
+ });
65792
+ } finally {
65793
+ this.deliveryRequests.delete(id);
65794
+ }
65795
+ }
65796
+ if (message.kind === "renew")
65797
+ return;
65798
+ if (message.kind === "commit") {
65799
+ const receipt = await queue.commit(lease, work, message.operation, message.mutation);
65800
+ this.noteUsage(await this.usageLedger.total(root));
65801
+ return receipt;
65802
+ }
65803
+ if (message.kind === "usage") {
65804
+ this.noteUsage(await this.usageLedger.report(lease, work, message.reports));
65805
+ return;
65806
+ }
65807
+ if (message.kind === "budget") {
65808
+ await queue.assertCurrent(lease, work);
65809
+ if (this.budgetBlocked)
65810
+ throw new Error("Root workflow budget exhausted");
65811
+ return queue.control(lease);
65812
+ }
65813
+ if (message.kind !== "complete")
65814
+ throw new Error("Invalid worker operation");
65815
+ await queue.complete(lease, work, message.result);
65816
+ this.replacements = 0;
65817
+ slot.work = undefined;
65818
+ await this.sendResult(work.id);
65819
+ });
65820
+ const response = handled.then((value2) => routeReady ?? value2);
65821
+ if (message.kind !== "delivery" && message.kind !== "delivery-ready")
65822
+ serial = handled.then(() => {}, () => {});
65823
+ response.then((value2) => child.send({ id: message.id, value: value2 }, replied), (error) => child.send({ id: message.id, error: error.message }, replied));
65824
+ });
65825
+ });
65826
+ }
65827
+ assertPublished(work) {
65828
+ const failure = this.publicationFailures.get(work.id);
65829
+ if (failure?.generation === work.generation)
65830
+ throw new Error(failure.error);
65831
+ }
65832
+ recoverExpired() {
65833
+ if (this.recovering)
65834
+ return this.recovering;
65835
+ const root = this.root;
65836
+ if (this.closed || root === undefined || !this.queue || performance.now() < this.nextRecoveryAt)
65837
+ return Promise.resolve();
65838
+ const queue = this.queue;
65839
+ const lease = this.lease;
65840
+ const leaseMs = this.options.leaseMs;
65841
+ this.recovering = (async () => {
65842
+ await new Promise((resolve) => setImmediate(resolve));
65843
+ await this.admission?.reconcile();
65844
+ const expiredBefore = Date.now();
65845
+ for (const work of await queue.inspect(root)) {
65846
+ if (work.state !== "leased" || work.expiresAt >= expiredBefore)
65847
+ continue;
65848
+ const child = work.worker === null ? undefined : this.incarnations.get(work.worker);
65849
+ if (child && work.worker !== null && this.isLive(work.worker, child)) {
65850
+ await queue.extend(lease, work, leaseMs).catch((error) => {
65851
+ if (error.message !== STALE_WORK_LEASE)
65852
+ throw error;
65853
+ });
65854
+ continue;
65855
+ }
65856
+ if (child && !await this.stop(child).then(() => true, () => false))
65857
+ continue;
65858
+ await queue.expire(lease, work);
65859
+ }
65860
+ })().finally(() => {
65861
+ this.recovering = undefined;
65862
+ this.nextRecoveryAt = performance.now() + RECOVERY_INTERVAL_MS;
65863
+ });
65864
+ return this.recovering;
65865
+ }
65866
+ observeResponsiveness() {
65867
+ const now = performance.now();
65868
+ const gap = now - this.lastTick;
65869
+ if (gap > STALL_THRESHOLD_MS)
65870
+ this.stalledMs += gap - TICK_INTERVAL_MS;
65871
+ this.lastTick = now;
65872
+ }
65873
+ isLive(worker, child) {
65874
+ this.observeResponsiveness();
65875
+ const heard = this.lastHeard.get(worker);
65876
+ return child.exitCode === null && child.signalCode === null && child.connected && heard !== undefined && performance.now() - heard.at - (this.stalledMs - heard.stalled) < this.options.leaseMs;
65877
+ }
65878
+ stop(child) {
65879
+ if (child.pid === undefined || child.exitCode !== null || child.signalCode !== null)
65880
+ return Promise.resolve();
65881
+ return new Promise((resolve, reject) => {
65882
+ const timeout = setTimeout(() => reject(new Error("Local worker cleanup was not confirmed")), 5000);
65883
+ child.once("exit", () => {
65884
+ clearTimeout(timeout);
65885
+ resolve();
65886
+ });
65887
+ child.kill("SIGKILL");
65888
+ });
65889
+ }
65890
+ async close() {
65891
+ this.closed = true;
65892
+ clearInterval(this.ticker);
65893
+ clearInterval(this.recoveryTimer);
65894
+ let failure;
65895
+ let stopping = [];
65896
+ try {
65897
+ await this.recovering?.catch(() => {});
65898
+ stopping = [...this.children].map((child) => ({ child, log: this.logs.get(child) }));
65899
+ await Promise.all(stopping.map(({ child }) => this.stop(child)));
65900
+ await Promise.all([...this.cleanups]);
65901
+ if (this.cleanupFailure)
65902
+ throw this.cleanupFailure;
65903
+ } catch (error) {
65904
+ failure = error;
65905
+ }
65906
+ if (failure === undefined) {
65907
+ const stopped = stopping.flatMap(({ child, log }) => log !== undefined && (child.exitCode !== null || child.signalCode !== null) ? [log] : []);
65908
+ await Promise.all(stopped.map((log) => rm3(log, { force: true }).catch(() => {})));
65909
+ if (this.logDirectory !== undefined)
65910
+ await rmdir(this.logDirectory).catch(() => {});
65911
+ }
65912
+ this.flush();
65913
+ try {
65914
+ await this.routes?.close();
65915
+ } catch (error) {
65916
+ failure ??= error;
65917
+ }
65918
+ for (const batch of this.batches.values())
65919
+ batch.settle();
65920
+ this.batches.clear();
65921
+ for (const pending of this.requests.values())
65922
+ pending.reject(new Error("Local worker pool is closed"));
65923
+ this.requests.clear();
65924
+ if (failure !== undefined)
65925
+ throw failure;
65926
+ }
65927
+ }
65928
+ var CONTROL_ACKNOWLEDGEMENT_TIMEOUT_MS = 5000, RECOVERY_INTERVAL_MS = 500, TICK_INTERVAL_MS = 100, STALL_THRESHOLD_MS = 500, COORDINATOR_LEASE_MS, STDERR_TAIL_CHARACTERS = 8192, STDERR_DRAIN_GRACE_MS = 1000, ROUTE_REGISTRATION_WAIT_MS = 120000, PENDING_STAGE_MESSAGE_EVENT2 = "atomic:workflow-pending-stage-message", STICKY_LIVE_DELIVERED_EVENT2 = "atomic:workflow-sticky-live-delivered", PENDING_STAGE_UNDELIVERABLE_EVENT2 = "atomic:workflow-pending-stage-undeliverable";
65929
+ var init_pool_coordinator = __esm(() => {
65930
+ init_resolution();
65931
+ init_sql_pool_queue();
65932
+ init_sql_pool_usage();
65933
+ init_pending_stage_intercom();
65934
+ init_intercom_group();
65935
+ init_pending_stage_route_capability();
65936
+ init_pending_stage_route_readiness();
65937
+ init_store();
65938
+ init_resource_admission();
65939
+ COORDINATOR_LEASE_MS = 3 * STAGE_SESSION_HEARTBEAT_INTERVAL_MS;
64777
65940
  });
64778
65941
 
64779
65942
  // dist/builtin/workflows/src/workers/protocol.ts
@@ -65023,21 +66186,26 @@ class PoolStageHandoff {
65023
66186
  store;
65024
66187
  queue;
65025
66188
  lease;
66189
+ claimRun;
65026
66190
  attempts = new Map;
65027
66191
  owners = new Map;
65028
66192
  publications = new Map;
65029
66193
  drains = new Map;
65030
- constructor(backend, store, queue, lease) {
66194
+ constructor(backend, store, queue, lease, claimRun) {
65031
66195
  this.backend = backend;
65032
66196
  this.store = store;
65033
66197
  this.queue = queue;
65034
66198
  this.lease = lease;
66199
+ this.claimRun = claimRun;
65035
66200
  }
65036
66201
  publish(work, runs) {
65037
66202
  const previous = this.publications.get(work.id) ?? Promise.resolve();
65038
66203
  const publication = previous.catch(() => {
65039
66204
  return;
65040
- }).then(() => this.apply(work, runs));
66205
+ }).then(async () => {
66206
+ const commit = await this.prepare(work, runs);
66207
+ this.store.batch(commit);
66208
+ });
65041
66209
  const settled = publication.catch(() => {
65042
66210
  return;
65043
66211
  });
@@ -65048,6 +66216,21 @@ class PoolStageHandoff {
65048
66216
  });
65049
66217
  return publication;
65050
66218
  }
66219
+ async publishBatch(publications) {
66220
+ const prepared = [];
66221
+ for (const { work, runs } of publications)
66222
+ prepared.push(await this.prepare(work, runs).catch((error) => error));
66223
+ return this.store.batch(() => prepared.map((commit) => {
66224
+ if (commit instanceof Error)
66225
+ return commit;
66226
+ try {
66227
+ commit();
66228
+ return;
66229
+ } catch (error) {
66230
+ return error instanceof Error ? error : new Error(String(error));
66231
+ }
66232
+ }));
66233
+ }
65051
66234
  async route(work, runId) {
65052
66235
  this.requireOwned(await this.begin(work), runId);
65053
66236
  const readiness = workflowPendingStageRouteReady(this.store, runId);
@@ -65098,7 +66281,7 @@ class PoolStageHandoff {
65098
66281
  const result = await this.queue.result(root, work.id);
65099
66282
  if (result === undefined)
65100
66283
  return;
65101
- await this.backend.hydrateWorkflow(root);
66284
+ await this.backend.catchUpWorkflow(root);
65102
66285
  const terminals = new Map;
65103
66286
  for (const checkpoint of this.backend.listCheckpoints(root)) {
65104
66287
  const boundary = checkpoint.kind === "stage" ? checkpoint.topology?.boundary : undefined;
@@ -65108,6 +66291,9 @@ class PoolStageHandoff {
65108
66291
  if (child?.runId === boundary.child.runId)
65109
66292
  terminals.set(child.runId, { status: child.status, outputs: child.outputs });
65110
66293
  }
66294
+ this.store.batch(() => this.settle(work, result, terminals));
66295
+ }
66296
+ settle(work, result, terminals) {
65111
66297
  for (const run of this.store.runs()) {
65112
66298
  if (run.endedAt !== undefined || this.owners.get(run.id) !== work.id)
65113
66299
  continue;
@@ -65131,22 +66317,31 @@ class PoolStageHandoff {
65131
66317
  if (current !== undefined && current.generation > work.generation)
65132
66318
  return;
65133
66319
  const attempt = current?.generation === work.generation ? current : this.start(work, current);
65134
- this.release(attempt);
66320
+ this.store.batch(() => this.release(attempt));
65135
66321
  attempt.fenced = true;
65136
66322
  }
65137
- async apply(work, runs) {
66323
+ async prepare(work, runs) {
65138
66324
  const subtree = this.subtree(work, runs);
65139
66325
  const root = this.lease.runId;
65140
- const known = new Map(this.store.runs().map((run) => [run.id, run]));
65141
- for (const run of subtree) {
65142
- const existing = known.get(run.id);
65143
- const owner = this.owners.get(run.id);
65144
- if (owner !== undefined && owner !== work.id || owner === undefined && existing !== undefined && (existing.parentRunId !== run.parentRunId || existing.parentStageId !== run.parentStageId))
65145
- throw outside(run.id);
65146
- }
65147
- if (subtree.some((run) => known.get(run.id) === undefined))
65148
- await this.backend.hydrateWorkflow(root);
66326
+ const owned = (known) => {
66327
+ for (const run of subtree) {
66328
+ const existing = known.get(run.id);
66329
+ const owner = this.owners.get(run.id);
66330
+ if (owner !== undefined && owner !== work.id || owner === undefined && existing !== undefined && (existing.parentRunId !== run.parentRunId || existing.parentStageId !== run.parentStageId))
66331
+ throw outside2(run.id);
66332
+ }
66333
+ };
66334
+ const before = new Map(this.store.runs().map((run) => [run.id, run]));
66335
+ owned(before);
66336
+ if (subtree.some((run) => before.get(run.id) === undefined))
66337
+ await this.backend.catchUpWorkflow(root);
65149
66338
  await this.queue.assertCurrent(this.lease, work);
66339
+ return () => this.commit(work, subtree, owned);
66340
+ }
66341
+ commit(work, subtree, owned) {
66342
+ const root = this.lease.runId;
66343
+ const known = new Map(this.store.runs().map((run) => [run.id, run]));
66344
+ owned(known);
65150
66345
  const attempt = this.enter(work);
65151
66346
  const execution = {
65152
66347
  worker: work.worker ?? "",
@@ -65158,11 +66353,13 @@ class PoolStageHandoff {
65158
66353
  ...run,
65159
66354
  rootRunId: root,
65160
66355
  pendingStageMessages: [
65161
- ...this.store.runs().find((current) => current.id === run.id)?.pendingStageMessages ?? pendingStageMessagesForDurableRun(this.backend, run.id, root)
66356
+ ...known.get(run.id)?.pendingStageMessages ?? pendingStageMessagesForDurableRun(this.backend, run.id, root)
65162
66357
  ],
65163
66358
  stages: run.stages.map((stage) => ({ ...stage, workerExecution: execution }))
65164
66359
  }));
65165
66360
  const changed = published.some((run) => activity(known.get(run.id)) !== activity(run));
66361
+ for (const run of published)
66362
+ this.claimRun?.(run.id);
65166
66363
  for (const run of published) {
65167
66364
  attempt.runs.add(run.id);
65168
66365
  this.owners.set(run.id, work.id);
@@ -65187,7 +66384,7 @@ class PoolStageHandoff {
65187
66384
  const byId = new Map;
65188
66385
  for (const run of runs) {
65189
66386
  if (run.id === root || byId.has(run.id))
65190
- throw outside(run.id);
66387
+ throw outside2(run.id);
65191
66388
  byId.set(run.id, run);
65192
66389
  }
65193
66390
  if (runs.length === 0)
@@ -65200,25 +66397,25 @@ class PoolStageHandoff {
65200
66397
  const top = byId.get(work.childRunId);
65201
66398
  if (top !== undefined) {
65202
66399
  if (top.parentRunId !== root || top.parentStageId !== work.boundaryId)
65203
- throw outside(top.id);
66400
+ throw outside2(top.id);
65204
66401
  const pending = [top];
65205
66402
  for (let run = pending.shift();run !== undefined; run = pending.shift()) {
65206
66403
  if (run.rootRunId !== undefined && run.rootRunId !== root)
65207
- throw outside(run.id);
66404
+ throw outside2(run.id);
65208
66405
  ordered.push(run);
65209
66406
  for (const child of runs) {
65210
66407
  if (child.parentRunId !== run.id)
65211
66408
  continue;
65212
66409
  const parentStage = run.stages.find((stage) => stage.id === child.parentStageId);
65213
66410
  if (authoritativeWorkflowChildRunId(parentStage) !== child.id)
65214
- throw outside(child.id);
66411
+ throw outside2(child.id);
65215
66412
  pending.push(child);
65216
66413
  }
65217
66414
  }
65218
66415
  }
65219
66416
  const stray = runs.find((run) => !ordered.includes(run));
65220
66417
  if (stray !== undefined)
65221
- throw outside(stray.id);
66418
+ throw outside2(stray.id);
65222
66419
  return ordered;
65223
66420
  }
65224
66421
  async begin(work) {
@@ -65273,7 +66470,7 @@ class PoolStageHandoff {
65273
66470
  }
65274
66471
  requireOwned(attempt, runId) {
65275
66472
  if (!attempt.runs.has(runId))
65276
- throw outside(runId);
66473
+ throw outside2(runId);
65277
66474
  }
65278
66475
  delivery(attempt, work, runId, stageId, stageName) {
65279
66476
  const key = JSON.stringify([runId, stageId]);
@@ -65291,7 +66488,7 @@ class PoolStageHandoff {
65291
66488
  return delivery;
65292
66489
  }
65293
66490
  }
65294
- function outside(runId) {
66491
+ function outside2(runId) {
65295
66492
  return new Error(`Pool stage publication is outside the work subtree: ${runId}`);
65296
66493
  }
65297
66494
  function isLive(stage) {
@@ -65316,33 +66513,122 @@ var init_stage_handoff = __esm(() => {
65316
66513
  });
65317
66514
 
65318
66515
  // dist/builtin/workflows/src/workers/local-pool.ts
65319
- import { fork } from "node:child_process";
65320
- import { randomUUID as randomUUID17 } from "node:crypto";
65321
- import { createWriteStream } from "node:fs";
65322
- import { mkdir as mkdir6, rm as rm3, rmdir } from "node:fs/promises";
65323
- import { dirname as dirname14, join as join29, resolve as resolve13 } from "node:path";
65324
- import { setTimeout as delay2 } from "node:timers/promises";
66516
+ import { fork as fork2 } from "node:child_process";
66517
+ import { resolve as resolve14 } from "node:path";
66518
+ import { setTimeout as delay3 } from "node:timers/promises";
65325
66519
  import { fileURLToPath as fileURLToPath6 } from "node:url";
65326
66520
  import { isBunBinary as isBunBinary2 } from "@bastani/atomic";
65327
- function workerLogDirectory(journalPath, root) {
65328
- return join29(dirname14(resolve13(journalPath)), "worker-logs", encodeURIComponent(root).replaceAll(".", "%2E"));
66521
+ function defaultWorkerEntry() {
66522
+ return isBunBinary2 ? INTERNAL_WORKFLOW_WORKER_ARG : fileURLToPath6(new URL("./local-worker-entry.mjs", import.meta.url));
66523
+ }
66524
+ function launchCoordinatorProcess() {
66525
+ const child = fork2(defaultWorkerEntry(), [JSON.stringify({ role: POOL_COORDINATOR_ROLE })], {
66526
+ stdio: ["ignore", "ignore", "pipe", "ipc"],
66527
+ execArgv: []
66528
+ });
66529
+ let stderr = "";
66530
+ child.stderr?.on("data", (chunk) => {
66531
+ stderr = (stderr + chunk.toString()).slice(-COORDINATOR_STDERR_TAIL_CHARACTERS);
66532
+ });
66533
+ const listeners = [];
66534
+ let lostReason;
66535
+ const lose = (reason) => {
66536
+ if (lostReason !== undefined)
66537
+ return;
66538
+ lostReason = reason;
66539
+ for (const listener of listeners)
66540
+ listener(reason);
66541
+ };
66542
+ const exited = new Promise((resolve) => child.once("exit", () => resolve()));
66543
+ child.on("error", lose);
66544
+ child.once("exit", (code, signal) => lose(new Error(`Local worker pool coordinator exited (${signal ?? `code ${code}`})${stderr ? `: ${stderr.trim()}` : ""}`)));
66545
+ return {
66546
+ pid: child.pid,
66547
+ port: {
66548
+ send(message) {
66549
+ if (lostReason !== undefined || !child.connected)
66550
+ return;
66551
+ child.send(message, (error) => {
66552
+ if (error)
66553
+ lose(error);
66554
+ });
66555
+ },
66556
+ onMessage(listener) {
66557
+ child.on("message", (message) => listener(message));
66558
+ },
66559
+ onClose(listener) {
66560
+ if (lostReason !== undefined)
66561
+ listener(lostReason);
66562
+ else
66563
+ listeners.push(listener);
66564
+ },
66565
+ async close() {
66566
+ if (child.connected)
66567
+ child.disconnect();
66568
+ if (child.exitCode !== null || child.signalCode !== null)
66569
+ return;
66570
+ const timedOut = await Promise.race([exited.then(() => false), delay3(COORDINATOR_EXIT_GRACE_MS, true)]);
66571
+ if (timedOut) {
66572
+ child.kill("SIGKILL");
66573
+ await exited;
66574
+ }
66575
+ }
66576
+ }
66577
+ };
66578
+ }
66579
+ function launchCoordinatorInProcess() {
66580
+ const mainListeners = [];
66581
+ const coordinatorListeners = [];
66582
+ const copy = (message) => JSON.parse(JSON.stringify(message));
66583
+ const coordinator = new PoolCoordinator({
66584
+ send(message) {
66585
+ const copied = copy(message);
66586
+ setImmediate(() => {
66587
+ for (const listener of mainListeners)
66588
+ listener(copied);
66589
+ });
66590
+ },
66591
+ onMessage(listener) {
66592
+ coordinatorListeners.push(listener);
66593
+ },
66594
+ onClose() {},
66595
+ async close() {}
66596
+ });
66597
+ return {
66598
+ pid: process.pid,
66599
+ port: {
66600
+ send(message) {
66601
+ const copied = copy(message);
66602
+ setImmediate(() => {
66603
+ for (const listener of coordinatorListeners)
66604
+ listener(copied);
66605
+ });
66606
+ },
66607
+ onMessage(listener) {
66608
+ mainListeners.push(listener);
66609
+ },
66610
+ onClose() {},
66611
+ async close() {
66612
+ coordinator.shutdown();
66613
+ }
66614
+ }
66615
+ };
65329
66616
  }
65330
66617
 
65331
66618
  class LocalWorkerPool {
65332
66619
  backend;
65333
66620
  options;
65334
- children = new Set;
65335
- incarnations = new Map;
65336
- readyWorkers = new Set;
65337
- logs = new Map;
65338
- logDirectory;
65339
- cleanups = new Set;
65340
- recovering;
65341
- nextRecoveryAt = 0;
65342
- controlAcks = new Map;
65343
- lastHeard = new Map;
65344
- lastTick = performance.now();
65345
- rootStalledMs = 0;
66621
+ pids = new Set;
66622
+ coordinator;
66623
+ coordinatorLost;
66624
+ commands = new Map;
66625
+ commandSequence = 0;
66626
+ inbound = Promise.resolve();
66627
+ works = new Map;
66628
+ attempts = new Map;
66629
+ results = new Map;
66630
+ changed = Promise.withResolvers();
66631
+ waiting = 0;
65346
66632
  ticker;
65347
66633
  leaseMs;
65348
66634
  desiredControl;
@@ -65350,18 +66636,17 @@ class LocalWorkerPool {
65350
66636
  store;
65351
66637
  handoff;
65352
66638
  releaseRoutes;
66639
+ routeClaims = new Map;
65353
66640
  sources = [];
65354
66641
  queue;
65355
66642
  lease;
65356
- admission;
65357
- usageLedger;
65358
66643
  usageTotal;
65359
66644
  rootBudget;
66645
+ budgetBlocked = false;
66646
+ rootSignature;
65360
66647
  starting;
65361
- cleanupFailure;
65362
66648
  failure;
65363
66649
  closed = false;
65364
- replacements = 0;
65365
66650
  releaseBinding;
65366
66651
  constructor(backend, options) {
65367
66652
  this.backend = backend;
@@ -65376,10 +66661,12 @@ class LocalWorkerPool {
65376
66661
  this.releaseBinding = bindChildWorkerDispatcher(backend, this, options.rootId);
65377
66662
  if (options.admission && options.rootId === undefined)
65378
66663
  workflowResourceAdmission.set(backend, options.admission);
65379
- this.ticker = setInterval(() => this.observeResponsiveness(), ROOT_TICK_INTERVAL_MS).unref();
65380
66664
  }
65381
66665
  get workerPids() {
65382
- return [...this.children].flatMap((child) => child.pid === undefined ? [] : [child.pid]);
66666
+ return [...this.pids];
66667
+ }
66668
+ get coordinatorPid() {
66669
+ return this.coordinator?.pid;
65383
66670
  }
65384
66671
  usage() {
65385
66672
  return this.usageTotal;
@@ -65387,47 +66674,39 @@ class LocalWorkerPool {
65387
66674
  async prepare(root, store) {
65388
66675
  if (this.closed)
65389
66676
  throw new Error("Local worker pool is closed");
65390
- if (!(this.backend.journal instanceof SqlRunJournal) || resolve13(this.options.journalPath) !== resolve13(this.backend.journal.path) || sqlPoolStorage(this.backend.journal)?.tables.runs !== "wf_runs")
66677
+ if (!(this.backend.journal instanceof SqlRunJournal) || resolve14(this.options.journalPath) !== resolve14(this.backend.journal.path) || sqlPoolStorage(this.backend.journal)?.tables.runs !== "wf_runs")
65391
66678
  throw new Error("Local workers require the same default-prefix SQLite journal");
65392
66679
  if (this.root !== undefined && this.root !== root)
65393
66680
  throw new Error("Local worker pool belongs to another root");
65394
66681
  this.root = root;
65395
66682
  if (store)
65396
66683
  this.store = store;
65397
- this.usageLedger ??= await openSqlPoolUsage(this.backend.journal);
65398
- this.usageTotal = await this.usageLedger.total(root);
65399
66684
  const state = await this.backend.journal.leaseState(root);
65400
66685
  if (!state || state.holderId !== this.backend.holderId)
65401
66686
  throw new Error("Local worker pool requires the root owner");
65402
66687
  this.lease = { runId: root, holderId: state.holderId, epoch: state.epoch };
65403
- const queue = await openSqlPoolQueue(this.backend.journal);
65404
- await queue.initialize(this.lease, COORDINATOR_LEASE_MS);
65405
- await queue.releasePreviousIncarnations(this.lease);
65406
- for (;; ) {
65407
- const intended = this.desiredControl ?? "running";
65408
- await queue.setControl(this.lease, intended);
65409
- if (intended === (this.desiredControl ?? "running"))
65410
- break;
65411
- }
65412
- this.queue = queue;
66688
+ const workerEntry = this.options.workerEntry ?? (isBunBinary2 || this.options.coordinator === "in-process" ? defaultWorkerEntry() : undefined);
66689
+ this.usageTotal = await this.command({
66690
+ kind: "prepare",
66691
+ root,
66692
+ lease: this.lease,
66693
+ options: {
66694
+ journalPath: this.options.journalPath,
66695
+ workers: this.options.workers,
66696
+ leaseMs: this.leaseMs,
66697
+ startupTimeoutMs: this.options.startupTimeoutMs ?? DEFAULT_STARTUP_TIMEOUT_MS,
66698
+ ...this.options.admission === undefined ? {} : { admission: this.options.admission },
66699
+ ...workerEntry === undefined ? {} : { workerEntry }
66700
+ },
66701
+ ...this.desiredControl === undefined ? {} : { control: this.desiredControl }
66702
+ });
66703
+ this.queue ??= await openSqlPoolQueue(this.backend.journal);
65413
66704
  }
65414
66705
  async control(state) {
65415
66706
  this.desiredControl = state;
65416
66707
  try {
65417
- if (!this.queue || !this.lease)
65418
- return;
65419
- const control = await this.queue.setControl(this.lease, state);
65420
- const deadline = Date.now() + CONTROL_ACKNOWLEDGEMENT_TIMEOUT_MS;
65421
- while ([...this.readyWorkers].some((worker) => (this.controlAcks.get(worker) ?? -1) < control.revision)) {
65422
- if (this.desiredControl !== state)
65423
- return;
65424
- if (Date.now() >= deadline) {
65425
- if (state === "quit")
65426
- break;
65427
- throw new Error("Worker control acknowledgement unresolved");
65428
- }
65429
- await delay2(25);
65430
- }
66708
+ if (this.coordinator !== undefined)
66709
+ await this.command({ kind: "control", state });
65431
66710
  } finally {
65432
66711
  if (state === "quit")
65433
66712
  await this.close();
@@ -65473,290 +66752,61 @@ class LocalWorkerPool {
65473
66752
  throw new Error(`Worker definition is not importable: ${binding.exportName}`);
65474
66753
  return source;
65475
66754
  }));
65476
- if (this.store) {
65477
- this.releaseRoutes = await acquireStandaloneIntercom(this.store, root, String(this.backend.getWorkflow(root)?.invocationCwd ?? process.cwd()));
66755
+ const cwd = String(this.backend.getWorkflow(root)?.invocationCwd ?? process.cwd());
66756
+ const store = this.store;
66757
+ if (store) {
66758
+ this.releaseRoutes = await acquireStandaloneIntercom(store, root, cwd);
65478
66759
  if (this.closed) {
65479
66760
  await this.releaseRoutes();
65480
66761
  this.releaseRoutes = undefined;
65481
66762
  throw new Error("Local worker pool is closed");
65482
66763
  }
65483
- this.handoff = new PoolStageHandoff(this.backend, this.store, this.queue, this.lease);
65484
- }
65485
- if (this.options.admission)
65486
- this.admission = await SqlResourceAdmission.open(this.backend.journal, {
65487
- ...this.options.admission,
65488
- root,
65489
- holder: "coordinator"
66764
+ const routeOwner = {
66765
+ notifyUndeliverable: async (payload) => !this.closed && await this.command({ kind: "undeliverable", payload }) === true
66766
+ };
66767
+ this.handoff = new PoolStageHandoff(this.backend, store, this.queueView(), this.lease, (runId) => {
66768
+ this.routeClaims.set(runId, claimExternalRunRoute(store, runId, routeOwner));
65490
66769
  });
65491
- this.logDirectory = workerLogDirectory(this.options.journalPath, root);
65492
- await mkdir6(this.logDirectory, { recursive: true }).catch(() => {
65493
- return;
65494
- });
65495
- const workers = Array.from({ length: this.options.workers }, () => this.spawn(root));
65496
- try {
65497
- await Promise.any(workers);
65498
- } catch (error) {
65499
- throw error.errors[0];
65500
66770
  }
65501
- }
65502
- spawn(root) {
65503
- if (this.closed)
65504
- return Promise.reject(new Error("Local worker pool is closed"));
65505
- const child = fork(isBunBinary2 ? INTERNAL_WORKFLOW_WORKER_ARG : fileURLToPath6(new URL("./local-worker-entry.mjs", import.meta.url)), [
65506
- JSON.stringify({
65507
- journalPath: this.options.journalPath,
65508
- root,
65509
- definitions: this.sources,
65510
- admission: this.options.admission
65511
- })
65512
- ], { stdio: ["ignore", "ignore", "pipe", "ipc"], execArgv: [] });
65513
- const disconnected = new Promise((resolve) => child.once("disconnect", () => resolve()));
65514
- this.children.add(child);
65515
- const worker = randomUUID17();
65516
- this.incarnations.set(worker, child);
65517
- let ready = false;
65518
- let serial = Promise.resolve();
65519
- let work;
65520
- const deliveries = new Map;
65521
- const logPath = join29(this.logDirectory, `${worker}.log`);
65522
- this.logs.set(child, logPath);
65523
- let logFailure;
65524
- const log = createWriteStream(logPath, { flags: "a" });
65525
- log.on("error", (error) => {
65526
- logFailure = error.message;
65527
- });
65528
- let stderr = "";
65529
- child.stderr?.on("data", (chunk) => {
65530
- stderr = (stderr + chunk.toString()).slice(-STDERR_TAIL_CHARACTERS);
65531
- });
65532
- child.stderr?.on("error", (error) => {
65533
- logFailure ??= error.message;
65534
- log.end();
65535
- });
65536
- child.stderr?.pipe(log);
65537
- const diagnosed = (summary) => new Error(`${summary} (worker stderr log: ${logFailure === undefined ? logPath : `unavailable, ${logFailure}`})${stderr ? `: ${stderr}` : ""}`);
65538
- const startupTimeoutMs = this.options.startupTimeoutMs ?? DEFAULT_STARTUP_TIMEOUT_MS;
65539
- let timedOut = false;
65540
- let terminated = false;
65541
- const terminate = () => {
65542
- if (terminated)
65543
- return;
65544
- terminated = true;
65545
- child.kill("SIGKILL");
65546
- };
65547
- const replied = (error) => {
65548
- if (error)
65549
- terminate();
65550
- };
65551
- return new Promise((resolve, reject) => {
65552
- const startup = setTimeout(() => {
65553
- timedOut = true;
65554
- terminate();
65555
- }, startupTimeoutMs);
65556
- child.on("error", (error) => {
65557
- clearTimeout(startup);
65558
- reject(error);
65559
- terminate();
65560
- });
65561
- const closed = new Promise((resolve) => child.once("close", () => resolve()));
65562
- child.once("exit", () => {
65563
- Promise.race([closed, delay2(STDERR_DRAIN_GRACE_MS)]).then(() => reject(diagnosed(timedOut ? `Local worker startup timed out after ${startupTimeoutMs} ms` : "Local worker exited before readiness")));
65564
- this.children.delete(child);
65565
- this.logs.delete(child);
65566
- clearTimeout(startup);
65567
- this.incarnations.delete(worker);
65568
- this.readyWorkers.delete(worker);
65569
- this.controlAcks.delete(worker);
65570
- this.lastHeard.delete(worker);
65571
- if (work)
65572
- this.handoff?.fence(work);
65573
- for (const delivery of deliveries.values())
65574
- delivery.reject(new Error("Worker exited before delivery acknowledgement"));
65575
- deliveries.clear();
65576
- const cleanup = disconnected.then(() => serial).then(async () => {
65577
- try {
65578
- if (await this.admission?.releaseStoppedSessions(worker))
65579
- throw new Error("Local worker process cleanup unconfirmed; permits quarantined");
65580
- } catch (error) {
65581
- this.cleanupFailure = error instanceof Error ? error : new Error(String(error));
65582
- throw this.cleanupFailure;
65583
- }
65584
- }).then(async () => {
65585
- if (this.closed || !ready || this.desiredControl === "quit")
65586
- return;
65587
- if (work && await this.queue.result(root, work.id)) {
65588
- work = undefined;
65589
- this.replacements = 0;
65590
- }
65591
- if (this.closed)
65592
- return;
65593
- if (work)
65594
- await this.queue.releaseExited(this.lease, work);
65595
- const exhausted = work ? work.generation >= 3 : ++this.replacements > 3;
65596
- if (exhausted)
65597
- this.failure = diagnosed("Local worker retry limit");
65598
- else
65599
- await this.spawn(root);
65600
- }).catch((error) => {
65601
- this.failure = error;
65602
- });
65603
- this.cleanups.add(cleanup);
65604
- cleanup.finally(() => this.cleanups.delete(cleanup));
65605
- });
65606
- child.on("message", (message) => {
65607
- this.observeResponsiveness();
65608
- this.lastHeard.set(worker, { at: performance.now(), rootStalled: this.rootStalledMs });
65609
- if (message.kind === "delivery-ack") {
65610
- const pending = deliveries.get(message.deliveryId);
65611
- deliveries.delete(message.deliveryId);
65612
- if (message.error)
65613
- pending?.reject(new Error(message.error));
65614
- else
65615
- pending?.resolve();
65616
- return;
65617
- }
65618
- const response = serial.then(async () => {
65619
- if (message.kind === "ready") {
65620
- if (terminated)
65621
- return;
65622
- ready = true;
65623
- this.readyWorkers.add(worker);
65624
- clearTimeout(startup);
65625
- resolve();
65626
- return;
65627
- }
65628
- await this.starting;
65629
- const queue = this.queue;
65630
- const lease = this.lease;
65631
- if (message.kind === "control")
65632
- return queue.control(lease);
65633
- if (message.kind === "controlled") {
65634
- const control = await queue.control(lease);
65635
- if (message.revision > control.revision)
65636
- throw new Error("Unknown control revision");
65637
- this.controlAcks.set(worker, Math.max(this.controlAcks.get(worker) ?? -1, message.revision));
65638
- return;
65639
- }
65640
- if (message.kind === "claim") {
65641
- if (work)
65642
- throw new Error("Worker already holds an attempt");
65643
- const decision = this.rootBudget?.checkpoint();
65644
- if (decision?.kind === "exhausted" || decision?.kind === "wrap_up")
65645
- return;
65646
- await this.recoverExpired(root);
65647
- work = await queue.claim(lease, worker, this.sources.map((source) => ({
65648
- definition: `${source.path}#${source.exportName}`,
65649
- digest: source.digest
65650
- })), this.leaseMs);
65651
- return work;
65652
- }
65653
- if (!work)
65654
- throw new Error("Worker has no active attempt");
65655
- if (message.kind === "renew" || work.expiresAt - Date.now() < this.leaseMs / 2)
65656
- work = await queue.extend(lease, work, this.leaseMs);
65657
- if (message.kind === "activity")
65658
- return this.handoff?.publish(work, message.runs);
65659
- if (message.kind === "route")
65660
- return this.handoff?.route(work, message.runId);
65661
- if (message.kind === "delivery-ready")
65662
- return this.handoff?.ready(work, message.runId, message.stageId, message.stageName, message.sessionId);
65663
- if (message.kind === "delivery") {
65664
- if (!message.recipient)
65665
- throw new Error("Worker delivery requires a session identity");
65666
- return this.handoff?.deliver(work, message.runId, message.stageId, message.stageName, message.recipient, (from, payload) => new Promise((resolve, reject) => {
65667
- const id = randomUUID17();
65668
- deliveries.set(id, { resolve, reject });
65669
- child.send({ id: message.id, delivery: { id, from, message: payload } }, (error) => {
65670
- if (error) {
65671
- deliveries.delete(id);
65672
- reject(error);
65673
- }
65674
- });
65675
- }));
65676
- }
65677
- if (message.kind === "renew")
65678
- return;
65679
- if (message.kind === "commit") {
65680
- const receipt = await queue.commit(lease, work, message.operation, message.mutation);
65681
- this.usageTotal = await this.usageLedger.total(root);
65682
- return receipt;
65683
- }
65684
- if (message.kind === "usage") {
65685
- this.usageTotal = await this.usageLedger.report(lease, work, message.reports);
65686
- return;
65687
- }
65688
- if (message.kind === "budget") {
65689
- await queue.assertCurrent(lease, work);
65690
- const decision = this.rootBudget?.checkpoint();
65691
- if (decision?.kind === "exhausted" || decision?.kind === "wrap_up")
65692
- throw new Error("Root workflow budget exhausted");
65693
- return queue.control(lease);
65694
- }
65695
- if (message.kind !== "complete")
65696
- throw new Error("Invalid worker operation");
65697
- await queue.complete(lease, work, message.result);
65698
- this.replacements = 0;
65699
- work = undefined;
65700
- });
65701
- if (message.kind !== "delivery" && message.kind !== "delivery-ready")
65702
- serial = response.then(() => {}, () => {});
65703
- response.then((value2) => child.send({ id: message.id, value: value2 }, replied), (error) => child.send({ id: message.id, error: error.message }, replied));
65704
- });
66771
+ await this.command({
66772
+ kind: "start",
66773
+ sources: this.sources,
66774
+ cwd,
66775
+ publish: store !== undefined,
66776
+ ...this.rootRun()
65705
66777
  });
65706
66778
  }
65707
- recoverExpired(root) {
65708
- if (this.recovering)
65709
- return this.recovering;
65710
- if (this.closed || performance.now() < this.nextRecoveryAt)
65711
- return Promise.resolve();
65712
- this.recovering = (async () => {
65713
- await new Promise((resolve) => setImmediate(resolve));
65714
- await this.admission?.reconcile();
65715
- const expiredBefore = Date.now();
65716
- for (const work of await this.queue.inspect(root)) {
65717
- if (work.state !== "leased" || work.expiresAt >= expiredBefore)
65718
- continue;
65719
- const child = work.worker === null ? undefined : this.incarnations.get(work.worker);
65720
- if (child && work.worker !== null && this.isLive(work.worker, child)) {
65721
- await this.queue.extend(this.lease, work, this.leaseMs).catch((error) => {
65722
- if (error.message !== STALE_WORK_LEASE)
65723
- throw error;
65724
- });
65725
- continue;
65726
- }
65727
- if (child && !await this.stop(child).then(() => true, () => false))
65728
- continue;
65729
- await this.queue.expire(this.lease, work);
65730
- }
65731
- })().finally(() => {
65732
- this.recovering = undefined;
65733
- this.nextRecoveryAt = performance.now() + RECOVERY_INTERVAL_MS;
65734
- });
65735
- return this.recovering;
66779
+ queueView() {
66780
+ return {
66781
+ assertCurrent: async (_lease, work) => {
66782
+ const current = this.attempts.get(work.id);
66783
+ if (current === undefined || current.fenced || current.generation !== work.generation || current.worker !== work.worker || current.epoch !== work.epoch)
66784
+ throw new Error(STALE_WORK_LEASE);
66785
+ },
66786
+ withCurrentAppend: (lease, work, assertSession, action) => this.queue.withCurrentAppend(lease, work, assertSession, action),
66787
+ result: async (root, id) => this.results.get(id) ?? this.queue.result(root, id)
66788
+ };
65736
66789
  }
65737
- observeResponsiveness() {
65738
- const now = performance.now();
65739
- const gap = now - this.lastTick;
65740
- if (gap > ROOT_STALL_THRESHOLD_MS)
65741
- this.rootStalledMs += gap - ROOT_TICK_INTERVAL_MS;
65742
- this.lastTick = now;
66790
+ rootProjection() {
66791
+ const run = this.store?.runs().find((candidate) => candidate.id === this.root);
66792
+ return run === undefined ? undefined : projectRootRun(run);
65743
66793
  }
65744
- isLive(worker, child) {
65745
- this.observeResponsiveness();
65746
- const heard = this.lastHeard.get(worker);
65747
- return child.exitCode === null && child.signalCode === null && child.connected && heard !== undefined && performance.now() - heard.at - (this.rootStalledMs - heard.rootStalled) < this.leaseMs;
66794
+ rootRun() {
66795
+ const rootRun = this.rootProjection();
66796
+ if (rootRun === undefined)
66797
+ return {};
66798
+ this.rootSignature = JSON.stringify(rootRun);
66799
+ return { rootRun };
65748
66800
  }
65749
- stop(child) {
65750
- if (child.pid === undefined || child.exitCode !== null || child.signalCode !== null)
65751
- return Promise.resolve();
65752
- return new Promise((resolve, reject) => {
65753
- const timeout = setTimeout(() => reject(new Error("Local worker cleanup was not confirmed")), 5000);
65754
- child.once("exit", () => {
65755
- clearTimeout(timeout);
65756
- resolve();
65757
- });
65758
- child.kill("SIGKILL");
65759
- });
66801
+ refreshRootRun() {
66802
+ const rootRun = this.rootProjection();
66803
+ if (rootRun === undefined)
66804
+ return;
66805
+ const signature = JSON.stringify(rootRun);
66806
+ if (signature === this.rootSignature)
66807
+ return;
66808
+ this.rootSignature = signature;
66809
+ this.notice({ kind: "root", rootRun });
65760
66810
  }
65761
66811
  async run(definition, inputs, options) {
65762
66812
  try {
@@ -65789,7 +66839,7 @@ class LocalWorkerPool {
65789
66839
  const selectedSource = this.sources[index];
65790
66840
  if (await definitionDigest(selectedSource.path) !== selectedSource.digest)
65791
66841
  throw new Error("Worker definition compatibility error");
65792
- await this.backend.hydrateWorkflow(root);
66842
+ await this.backend.catchUpWorkflow(root);
65793
66843
  const boundary = this.backend.listCheckpoints(root).find((entry) => entry.kind === "stage" && entry.topology?.boundary?.event === "start" && entry.topology.stageId === options.parentRun.stageId);
65794
66844
  if (boundary?.kind !== "stage")
65795
66845
  throw new Error("Missing durable child boundary");
@@ -65800,39 +66850,232 @@ class LocalWorkerPool {
65800
66850
  config: options.config,
65801
66851
  executionMode: options.executionMode
65802
66852
  }));
65803
- const work = await this.queue.enqueue(this.lease, {
65804
- boundaryId: options.parentRun.stageId,
65805
- childRunId: options.runId,
65806
- scope: options.durableScope.scopePrefix,
65807
- definition: `${selectedSource.path}#${selectedSource.exportName}`,
65808
- digest: selectedSource.digest,
65809
- inputs,
65810
- execution
65811
- }, boundary);
65812
- for (;; ) {
65813
- options.signal?.throwIfAborted();
65814
- if (this.closed) {
66853
+ this.evaluateBudget();
66854
+ const enqueued = await this.command({
66855
+ kind: "enqueue",
66856
+ descriptor: {
66857
+ boundaryId: options.parentRun.stageId,
66858
+ childRunId: options.runId,
66859
+ scope: options.durableScope.scopePrefix,
66860
+ definition: `${selectedSource.path}#${selectedSource.exportName}`,
66861
+ digest: selectedSource.digest,
66862
+ inputs,
66863
+ execution
66864
+ },
66865
+ boundary,
66866
+ ...this.rootRun()
66867
+ });
66868
+ const work = enqueued.work;
66869
+ this.works.set(work.id, work);
66870
+ if (enqueued.result !== undefined && !this.results.has(work.id))
66871
+ this.results.set(work.id, enqueued.result);
66872
+ this.waiting++;
66873
+ this.ticker ??= setInterval(() => {
66874
+ this.evaluateBudget();
66875
+ this.signalChange();
66876
+ }, POOL_SUMMARY_INTERVAL_MS).unref();
66877
+ try {
66878
+ for (;; ) {
66879
+ options.signal?.throwIfAborted();
66880
+ if (this.closed) {
66881
+ if (this.failure)
66882
+ throw this.failure;
66883
+ if (this.desiredControl === "quit")
66884
+ throw new WorkflowGracefulQuitError(root, "worker pool teardown");
66885
+ throw new Error("Local worker pool is closed");
66886
+ }
66887
+ const result = this.results.get(work.id);
66888
+ if (result) {
66889
+ await this.inbound;
66890
+ await this.backend.catchUpWorkflow(root);
66891
+ await this.handoff?.complete(work);
66892
+ return result;
66893
+ }
65815
66894
  if (this.failure)
65816
66895
  throw this.failure;
65817
- if (this.desiredControl === "quit")
65818
- throw new WorkflowGracefulQuitError(root, "worker pool teardown");
65819
- throw new Error("Local worker pool is closed");
66896
+ if (this.budgetBlocked)
66897
+ await this.rootBudget.stopAtBoundaryAsync(definition.name);
66898
+ await this.nextChange(options.signal);
65820
66899
  }
65821
- const result = await this.queue.result(root, work.id);
65822
- if (result) {
65823
- await this.backend.hydrateWorkflow(root);
65824
- await this.handoff?.complete(work);
65825
- return result;
66900
+ } finally {
66901
+ if (--this.waiting === 0) {
66902
+ clearInterval(this.ticker);
66903
+ this.ticker = undefined;
65826
66904
  }
65827
- if (this.failure)
65828
- throw this.failure;
65829
- const decision = this.rootBudget?.checkpoint();
65830
- if (decision?.kind === "exhausted" || decision?.kind === "wrap_up")
65831
- await this.rootBudget.stopAtBoundaryAsync(definition.name);
65832
- await this.recoverExpired(root);
65833
- await delay2(25, undefined, { signal: options.signal });
65834
66905
  }
65835
66906
  }
66907
+ evaluateBudget() {
66908
+ const decision = this.rootBudget?.checkpoint();
66909
+ const blocked = decision?.kind === "exhausted" || decision?.kind === "wrap_up";
66910
+ if (blocked === this.budgetBlocked)
66911
+ return;
66912
+ this.budgetBlocked = blocked;
66913
+ this.notice({ kind: "budget", blocked });
66914
+ }
66915
+ signalChange() {
66916
+ const changed = this.changed;
66917
+ this.changed = Promise.withResolvers();
66918
+ changed.resolve();
66919
+ }
66920
+ nextChange(signal) {
66921
+ const changed = this.changed.promise;
66922
+ if (signal === undefined)
66923
+ return changed;
66924
+ return new Promise((resolve, reject) => {
66925
+ const abort = () => reject(signal.reason);
66926
+ signal.addEventListener("abort", abort, { once: true });
66927
+ changed.then(() => {
66928
+ signal.removeEventListener("abort", abort);
66929
+ resolve();
66930
+ });
66931
+ });
66932
+ }
66933
+ connect() {
66934
+ if (this.coordinator !== undefined)
66935
+ return this.coordinator;
66936
+ const coordinator = this.options.coordinator === "in-process" ? launchCoordinatorInProcess() : launchCoordinatorProcess();
66937
+ this.coordinator = coordinator;
66938
+ coordinator.port.onMessage((event) => {
66939
+ this.inbound = this.inbound.then(() => this.receive(event)).catch(() => {});
66940
+ });
66941
+ coordinator.port.onClose((reason) => {
66942
+ this.coordinatorLost = reason;
66943
+ this.inbound = this.inbound.then(() => this.lost(reason));
66944
+ });
66945
+ return coordinator;
66946
+ }
66947
+ lost(reason) {
66948
+ this.pids.clear();
66949
+ if (!this.closed)
66950
+ this.failure ??= reason;
66951
+ for (const pending of this.commands.values())
66952
+ pending.reject(reason);
66953
+ this.commands.clear();
66954
+ this.signalChange();
66955
+ }
66956
+ command(body) {
66957
+ const coordinator = this.connect();
66958
+ if (this.coordinatorLost !== undefined)
66959
+ return Promise.reject(this.coordinatorLost);
66960
+ const id = ++this.commandSequence;
66961
+ return new Promise((resolve, reject) => {
66962
+ this.commands.set(id, { resolve, reject });
66963
+ coordinator.port.send({ ...body, id });
66964
+ });
66965
+ }
66966
+ notice(notice) {
66967
+ this.coordinator?.port.send(notice);
66968
+ }
66969
+ async receive(event) {
66970
+ switch (event.kind) {
66971
+ case "reply": {
66972
+ const pending = this.commands.get(event.id);
66973
+ this.commands.delete(event.id);
66974
+ if (event.error !== undefined)
66975
+ pending?.reject(new Error(event.error));
66976
+ else
66977
+ pending?.resolve(event.value);
66978
+ return;
66979
+ }
66980
+ case "activity":
66981
+ return this.applyActivity(event.batch, event.entries, event.usage);
66982
+ case "spawned":
66983
+ this.pids.add(event.pid);
66984
+ return;
66985
+ case "exited":
66986
+ this.pids.delete(event.pid);
66987
+ return;
66988
+ case "fence": {
66989
+ const current = this.attempts.get(event.work.id);
66990
+ if (current?.generation === event.work.generation)
66991
+ current.fenced = true;
66992
+ this.handoff?.fence(event.work);
66993
+ return;
66994
+ }
66995
+ case "result":
66996
+ if (event.usage !== undefined)
66997
+ this.usageTotal = event.usage;
66998
+ this.results.set(event.workId, event.result);
66999
+ this.signalChange();
67000
+ return;
67001
+ case "failure":
67002
+ this.failure ??= new Error(event.error);
67003
+ this.signalChange();
67004
+ return;
67005
+ case "request":
67006
+ this.serve(event.id, event.request);
67007
+ return;
67008
+ }
67009
+ }
67010
+ async applyActivity(batch, entries, usage) {
67011
+ if (usage !== undefined)
67012
+ this.usageTotal = usage;
67013
+ const rejected = [];
67014
+ const handoff = this.handoff;
67015
+ if (handoff !== undefined) {
67016
+ const publications = entries.flatMap((entry) => {
67017
+ const work = this.track(entry.work);
67018
+ if (work !== undefined)
67019
+ return [{ work, runs: entry.runs }];
67020
+ rejected.push({ workId: entry.work.id, generation: entry.work.generation, error: STALE_WORK_LEASE });
67021
+ return [];
67022
+ });
67023
+ const outcomes = await handoff.publishBatch(publications);
67024
+ outcomes.forEach((error, index) => {
67025
+ const { work } = publications[index];
67026
+ if (error !== undefined)
67027
+ rejected.push({ workId: work.id, generation: work.generation, error: error.message });
67028
+ });
67029
+ }
67030
+ this.notice({ kind: "published", batch, rejected });
67031
+ this.refreshRootRun();
67032
+ }
67033
+ track(attempt) {
67034
+ const work = this.works.get(attempt.id);
67035
+ if (work === undefined)
67036
+ return;
67037
+ const current = this.attempts.get(attempt.id);
67038
+ if (current === undefined || attempt.generation > current.generation)
67039
+ this.attempts.set(attempt.id, { ...attempt, fenced: false });
67040
+ return { ...work, ...attempt, state: "leased" };
67041
+ }
67042
+ async serve(id, request) {
67043
+ try {
67044
+ const value2 = await this.answer(id, request);
67045
+ this.coordinator?.port.send({ kind: "request-reply", id, ...value2 === undefined ? {} : { value: value2 } });
67046
+ } catch (error) {
67047
+ this.coordinator?.port.send({
67048
+ kind: "request-reply",
67049
+ id,
67050
+ error: error instanceof Error ? error.message : String(error)
67051
+ });
67052
+ }
67053
+ }
67054
+ async answer(id, request) {
67055
+ if (request.kind === "pending-stage-message" || request.kind === "sticky-live-delivered") {
67056
+ const router = this.store === undefined ? undefined : pendingStageMessageRouter(this.store);
67057
+ const event = {
67058
+ ...request.payload,
67059
+ handled: false
67060
+ };
67061
+ if (request.kind === "pending-stage-message")
67062
+ router?.route(event);
67063
+ else
67064
+ router?.sticky(event);
67065
+ if (event.handled && event.completion !== undefined)
67066
+ return await event.completion;
67067
+ return request.kind === "pending-stage-message" ? { outcome: "refused", reason: "Session not found" } : undefined;
67068
+ }
67069
+ const handoff = this.handoff;
67070
+ const work = this.track(request.work);
67071
+ if (handoff === undefined || work === undefined)
67072
+ throw new Error(STALE_WORK_LEASE);
67073
+ if (request.kind === "ready")
67074
+ return handoff.ready(work, request.runId, request.stageId, request.stageName, request.sessionId);
67075
+ return handoff.deliver(work, request.runId, request.stageId, request.stageName, request.recipient, async (from, message) => {
67076
+ await this.command({ kind: "transport", request: id, from, message });
67077
+ });
67078
+ }
65836
67079
  closing;
65837
67080
  close() {
65838
67081
  this.closing ??= this.dispose();
@@ -65841,27 +67084,28 @@ class LocalWorkerPool {
65841
67084
  async dispose() {
65842
67085
  this.closed = true;
65843
67086
  clearInterval(this.ticker);
67087
+ this.ticker = undefined;
65844
67088
  this.releaseBinding();
65845
67089
  if (this.options.rootId === undefined)
65846
67090
  workflowResourceAdmission.delete(this.backend);
67091
+ this.signalChange();
65847
67092
  let failure;
65848
- let stopping = [];
65849
- try {
65850
- await this.recovering?.catch(() => {});
65851
- stopping = [...this.children].map((child) => ({ child, log: this.logs.get(child) }));
65852
- await Promise.all(stopping.map(({ child }) => this.stop(child)));
65853
- await Promise.all([...this.cleanups]);
65854
- if (this.cleanupFailure)
65855
- throw this.cleanupFailure;
65856
- } catch (error) {
65857
- failure = error;
65858
- }
65859
- if (failure === undefined) {
65860
- const stopped = stopping.flatMap(({ child, log }) => log !== undefined && (child.exitCode !== null || child.signalCode !== null) ? [log] : []);
65861
- await Promise.all(stopped.map((log) => rm3(log, { force: true }).catch(() => {})));
65862
- if (this.logDirectory !== undefined)
65863
- await rmdir(this.logDirectory).catch(() => {});
67093
+ const coordinator = this.coordinator;
67094
+ if (coordinator !== undefined) {
67095
+ if (this.coordinatorLost === undefined) {
67096
+ try {
67097
+ await this.command({ kind: "close" });
67098
+ } catch (error) {
67099
+ failure = error;
67100
+ }
67101
+ }
67102
+ await this.inbound;
67103
+ await coordinator.port.close();
67104
+ this.pids.clear();
65864
67105
  }
67106
+ for (const release of this.routeClaims.values())
67107
+ release();
67108
+ this.routeClaims.clear();
65865
67109
  try {
65866
67110
  await this.releaseRoutes?.();
65867
67111
  } catch (error) {
@@ -65871,27 +67115,26 @@ class LocalWorkerPool {
65871
67115
  throw failure;
65872
67116
  }
65873
67117
  }
65874
- var INTERNAL_WORKFLOW_WORKER_ARG = "--internal-workflow-worker", DEFAULT_WORK_LEASE_MS = 30000, MINIMUM_WORK_LEASE_MS, CONTROL_ACKNOWLEDGEMENT_TIMEOUT_MS = 5000, RECOVERY_INTERVAL_MS = 500, ROOT_TICK_INTERVAL_MS = 100, ROOT_STALL_THRESHOLD_MS = 500, COORDINATOR_LEASE_MS, DEFAULT_STARTUP_TIMEOUT_MS = 30000, MAX_TIMER_DELAY_MS = 2147483647, STDERR_TAIL_CHARACTERS = 8192, STDERR_DRAIN_GRACE_MS = 1000;
67118
+ var INTERNAL_WORKFLOW_WORKER_ARG = "--internal-workflow-worker", DEFAULT_WORK_LEASE_MS = 30000, MINIMUM_WORK_LEASE_MS, DEFAULT_STARTUP_TIMEOUT_MS = 30000, MAX_TIMER_DELAY_MS = 2147483647, COORDINATOR_STDERR_TAIL_CHARACTERS = 8192, COORDINATOR_EXIT_GRACE_MS = 5000;
65875
67119
  var init_local_pool = __esm(() => {
65876
67120
  init_internal();
65877
67121
  init_sql_pool_queue();
65878
- init_sql_pool_usage();
65879
67122
  init_sql_run_journal();
65880
67123
  init_workflow_tool_abort();
65881
67124
  init_sdk_intercom();
67125
+ init_external_run_routes();
65882
67126
  init_workflow_infrastructure_error();
65883
67127
  init_admission_binding();
65884
67128
  init_definitions();
65885
67129
  init_dispatch();
65886
67130
  init_host_options();
65887
- init_resource_admission();
67131
+ init_pool_coordinator();
65888
67132
  init_stage_handoff();
65889
67133
  MINIMUM_WORK_LEASE_MS = 2 * WORKER_HEARTBEAT_INTERVAL_MS;
65890
- COORDINATOR_LEASE_MS = 3 * STAGE_SESSION_HEARTBEAT_INTERVAL_MS;
65891
67134
  });
65892
67135
 
65893
67136
  // dist/builtin/workflows/src/workers/public-pool.ts
65894
- import { resolve as resolve14 } from "node:path";
67137
+ import { resolve as resolve15 } from "node:path";
65895
67138
  import { isDeepStrictEqual as isDeepStrictEqual2 } from "node:util";
65896
67139
  function withWorkerSettings(options, settings) {
65897
67140
  const configured = settings?.workflows;
@@ -65940,11 +67183,11 @@ async function activatePublicPool(backend, options) {
65940
67183
  ...options.localWorkers,
65941
67184
  startupTimeoutMs: undefined,
65942
67185
  leaseMs: undefined,
65943
- definitions: options.localWorkers.definitions.map((source) => ({ ...source, path: resolve14(cwd, source.path) }))
67186
+ definitions: options.localWorkers.definitions.map((source) => ({ ...source, path: resolve15(cwd, source.path) }))
65944
67187
  };
65945
67188
  const admission = options.admission && {
65946
67189
  ...options.admission,
65947
- ...options.admission.userDatabase ? { userDatabase: resolve14(cwd, options.admission.userDatabase) } : {}
67190
+ ...options.admission.userDatabase ? { userDatabase: resolve15(cwd, options.admission.userDatabase) } : {}
65948
67191
  };
65949
67192
  const sources = localWorkers && await Promise.all(localWorkers.definitions.map(async (source) => ({ ...source, digest: await definitionDigest(source.path) })));
65950
67193
  const resources = JSON.parse(JSON.stringify({
@@ -72552,9 +73795,9 @@ var init_stage_runner_controller = __esm(() => {
72552
73795
  });
72553
73796
 
72554
73797
  // dist/builtin/workflows/src/runs/foreground/stage-runner-output.ts
72555
- import { createHash as createHash4 } from "node:crypto";
73798
+ import { createHash as createHash5 } from "node:crypto";
72556
73799
  import { mkdir as mkdir7, writeFile as writeFile10 } from "node:fs/promises";
72557
- import { basename as basename5, dirname as dirname15, isAbsolute as isAbsolute11, join as join30, resolve as resolve15 } from "node:path";
73800
+ import { basename as basename5, dirname as dirname15, isAbsolute as isAbsolute11, join as join30, resolve as resolve16 } from "node:path";
72558
73801
  function normalizeMaxOutput2(maxOutput) {
72559
73802
  return {
72560
73803
  bytes: maxOutput?.bytes ?? DEFAULT_MAX_OUTPUT_BYTES,
@@ -72616,7 +73859,7 @@ function formatByteSize(bytes) {
72616
73859
  return `${value2.toFixed(1)} ${units[unitIndex]}`;
72617
73860
  }
72618
73861
  function transcriptPath(runId, outputPath) {
72619
- const digest = createHash4("sha256").update(outputPath).digest("hex").slice(0, 16);
73862
+ const digest = createHash5("sha256").update(outputPath).digest("hex").slice(0, 16);
72620
73863
  return join30(workflowArtifactRunPath(runId), "transcripts", `${digest}-${basename5(outputPath)}.transcript.md`);
72621
73864
  }
72622
73865
  function appendContent(lines, content) {
@@ -72662,7 +73905,7 @@ function renderTranscript(messages, fallbackOutput) {
72662
73905
  `);
72663
73906
  }
72664
73907
  function savedOutputReference(outputPath, fullOutput, transcriptPathValue, transcriptText, transcriptAvailable, warning) {
72665
- const absolutePath = resolve15(outputPath);
73908
+ const absolutePath = resolve16(outputPath);
72666
73909
  const bytes = Buffer.byteLength(fullOutput, "utf8");
72667
73910
  const lines = countLines(fullOutput);
72668
73911
  const transcriptBytes = Buffer.byteLength(transcriptText, "utf8");
@@ -72677,8 +73920,8 @@ function resolveOutputPath(output, runtimeCwd, requestedCwd) {
72677
73920
  return;
72678
73921
  if (isAbsolute11(output))
72679
73922
  return output;
72680
- const baseCwd = requestedCwd === undefined ? runtimeCwd : isAbsolute11(requestedCwd) ? requestedCwd : resolve15(runtimeCwd, requestedCwd);
72681
- return resolve15(baseCwd, output);
73923
+ const baseCwd = requestedCwd === undefined ? runtimeCwd : isAbsolute11(requestedCwd) ? requestedCwd : resolve16(runtimeCwd, requestedCwd);
73924
+ return resolve16(baseCwd, output);
72682
73925
  }
72683
73926
  function degenerateWarning(fullOutput) {
72684
73927
  return fullOutput.trim().length === 0 ? "WARNING: the stage artifact is empty; search the companion transcript for this stage's work." : undefined;
@@ -73687,7 +74930,7 @@ async function run(def, inputs, opts = {}) {
73687
74930
  if (!isWorkflowDefinition(def))
73688
74931
  throw new Error(workflowDefinitionRequirementMessage("run(definition, inputs)", def));
73689
74932
  const depth = opts.depth ?? 0;
73690
- const maxDepth = opts.config?.maxDepth ?? 4;
74933
+ const maxDepth = opts.config?.maxDepth ?? 3;
73691
74934
  if (depth >= maxDepth) {
73692
74935
  return {
73693
74936
  runId: opts.runId ?? crypto.randomUUID(),
@@ -75141,9 +76384,9 @@ var init_sdk_owner_release = __esm(() => {
75141
76384
 
75142
76385
  // dist/builtin/workflows/src/durable/sdk-recovery-scope.ts
75143
76386
  import { realpathSync as realpathSync5 } from "node:fs";
75144
- import { resolve as resolve16 } from "node:path";
76387
+ import { resolve as resolve17 } from "node:path";
75145
76388
  function canonicalDirectory(path) {
75146
- const absolute = resolve16(path);
76389
+ const absolute = resolve17(path);
75147
76390
  try {
75148
76391
  const canonical = realpathSync5.native(absolute);
75149
76392
  return process.platform === "win32" ? canonical.toLowerCase() : canonical;
@@ -75880,9 +77123,7 @@ function runSnapshotsFromCheckpoints(checkpoints, rootRunId, rootRunName, fallba
75880
77123
  }
75881
77124
  for (const group of grouped.values())
75882
77125
  group.sort(compareDraftSourceOrder);
75883
- const groupedCount = grouped.size;
75884
- retainReachableRunGroups(grouped, rootRunId);
75885
- if (policy.failOnUnreachable && grouped.size !== groupedCount)
77126
+ if (!retainReachableRunGroups(grouped, rootRunId) && policy.failOnUnreachable)
75886
77127
  return [];
75887
77128
  if (!validateRunGroups(grouped, rootRunId, toolCheckpoints, checkpoints, policy.retainActive))
75888
77129
  return [];
@@ -87496,7 +88737,7 @@ async function withDormantRunWrite(store, runId, write) {
87496
88737
 
87497
88738
  // dist/builtin/workflows/src/extension/dormant-run-intercom.ts
87498
88739
  var PENDING_STAGE_ROUTE_EVENT2 = "atomic:workflow-pending-stage-route";
87499
- var PENDING_STAGE_MESSAGE_EVENT2 = "atomic:workflow-pending-stage-message";
88740
+ var PENDING_STAGE_MESSAGE_EVENT3 = "atomic:workflow-pending-stage-message";
87500
88741
  var PENDING_STAGE_ROUTE_REVOKED_EVENT = "atomic:workflow-pending-stage-route-revoked";
87501
88742
  var DORMANT_RUN_SUBAGENT_HOLD_EVENT = "atomic:workflow-dormant-run-subagent-hold";
87502
88743
  var DORMANT_RUN_RECHECK_MS = 15000;
@@ -87672,7 +88913,7 @@ function registerDormantRunIntercomRoutes(pi, activeStore, deps) {
87672
88913
  ...event.senderRegistrationName === undefined ? {} : { senderRegistrationName: event.senderRegistrationName },
87673
88914
  ...event.senderReturnAddress === undefined ? {} : { senderReturnAddress: event.senderReturnAddress }
87674
88915
  };
87675
- pi.events?.emit?.(PENDING_STAGE_MESSAGE_EVENT2, forwarded);
88916
+ pi.events?.emit?.(PENDING_STAGE_MESSAGE_EVENT3, forwarded);
87676
88917
  if (forwarded.handled && forwarded.completion !== undefined)
87677
88918
  return await forwarded.completion;
87678
88919
  return { outcome: "refused", reason: "Session not found" };
@@ -87692,7 +88933,7 @@ function registerDormantRunIntercomRoutes(pi, activeStore, deps) {
87692
88933
  }
87693
88934
  return isLive(run.runId) ? await redispatch(event) : { outcome: "refused", reason: `Workflow run ${run.runId} is no longer resumable from this session` };
87694
88935
  };
87695
- const messages = pi.events?.on?.(PENDING_STAGE_MESSAGE_EVENT2, (payload) => {
88936
+ const messages = pi.events?.on?.(PENDING_STAGE_MESSAGE_EVENT3, (payload) => {
87696
88937
  if (disposed || !isDormantMessageEvent(payload) || payload.handled || payload.live === true)
87697
88938
  return;
87698
88939
  const target = parseWorkflowStageTarget(payload.target);
@@ -89610,8 +90851,8 @@ function mergeConfigs(base, override) {
89610
90851
  };
89611
90852
  }
89612
90853
  var WORKFLOW_CONFIG_DEFAULTS = {
89613
- maxDepth: 4,
89614
- defaultConcurrency: 3,
90854
+ maxDepth: 3,
90855
+ defaultConcurrency: 1000,
89615
90856
  persistRuns: true,
89616
90857
  statusFile: false,
89617
90858
  resumeInFlight: "never",
@@ -91378,7 +92619,7 @@ init_registry();
91378
92619
  // dist/builtin/workflows/src/extension/discovery-loaders.ts
91379
92620
  init_workflow_module_loader();
91380
92621
  import { readdir as readdir2, stat as stat2 } from "node:fs/promises";
91381
- import { extname, isAbsolute as isAbsolute13, join as join32, resolve as resolve17 } from "node:path";
92622
+ import { extname, isAbsolute as isAbsolute13, join as join32, resolve as resolve18 } from "node:path";
91382
92623
  async function scanWorkflowDir(dir) {
91383
92624
  try {
91384
92625
  const entries = await readdir2(dir, { withFileTypes: true });
@@ -91422,7 +92663,7 @@ async function loadFromPaths(pathsOrMap, kind, baseCwd, diagnostics, getWorkflow
91422
92663
  const all = [];
91423
92664
  const entries = Array.isArray(pathsOrMap) ? pathsOrMap.map((p) => ({ rawPath: p })) : Object.entries(pathsOrMap).map(([name, p]) => ({ rawPath: p, configuredName: name }));
91424
92665
  for (const { rawPath, configuredName } of entries) {
91425
- const absPath = isAbsolute13(rawPath) ? rawPath : resolve17(baseCwd, rawPath);
92666
+ const absPath = isAbsolute13(rawPath) ? rawPath : resolve18(baseCwd, rawPath);
91426
92667
  let pathStats;
91427
92668
  try {
91428
92669
  pathStats = await stat2(absPath);
@@ -98180,7 +99421,7 @@ var DEFAULT_PROMPT_GUIDANCE = [
98180
99421
  - Bounded loops must create distinct tracked work for every iteration; never reopen an ancestor below its downstream work. Stacked implementation slices are unrolled, not looped: each slice gets a fresh child boundary and distinct tracked nodes, and no slice may add an edge back to the current frontier, itself, or an ancestor. If a retained session receives follow-up without new dependency work, keep it as non-topological activity metadata rather than adding a back-edge. Before launch, sketch nodes and dependencies for every branch and loop and reject any proposed parent edge to the node itself or an ancestor.
98181
99422
  - Treat dynamic graph validation during execution, replay, and journal reconstruction as the authoritative cycle boundary. If runtime topology code changes, require incremental edge checks and reconstruction validation; prompt guidance and TypeScript types cannot replace those runtime checks.
98182
99423
  - Every workflow invocation receives the stable control group \`workflow:<rootRunId>\`. Stages inherit it unless an intentional explicit group is authored. Named groups and \`group: true\` become invocation-owned subgroups under \`workflow:<rootRunId>/...\`, so repeated names cannot collide across runs. The invocation context may list and exactly send/ask live stages in its owned subgroups and queue send to known pending stages; this authority is directional and does not let subgroup siblings see or reach each other. \`intercom list\` also shows possible future targets, and the invocation context may queue sticky sends to their path patterns. Pending and pattern ask remain unsupported. A \`workflow:<rootRunId>/**\` send broadcasts to all live stages and remains sticky for future descendants until root termination. \`group: "default"\` is the explicit non-owned shared-group escape and does not receive pending invocation routing. Do not mint root group names manually.
98183
- - Stages in the same invocation group are peers, not only steerable targets. A live stage can \`intercom list\` its siblings and \`send\`/\`ask\` them directly to debate a finding with evidence, hand off paths and reproductions, claim shared files or expensive steps, or learn what a sibling already verified; it can queue \`send\` to a known pending sibling and \`ask\` a completed sibling that retains a valid conversation, which reopens it for a post-mortem turn. Delegated subagents inside a stage inherit the same group. Named subgroups isolate their members from sibling subgroups. Stages do not coordinate spontaneously: when parallel stages should exchange evidence, say so in each prompt, name the peers or the pattern to discover them, bound the exchange (for example one evidence round), and require each stage to return its own structured result; keep scope and acceptance decisions with the supervisor or a \`ctx.ui\` gate.
99424
+ - Stages in the same invocation group are peers, not only steerable targets. A live stage can \`intercom list\` its siblings and \`send\`/\`ask\` them directly to debate a finding with evidence, hand off paths and reproductions, claim shared files or expensive steps, or learn what a sibling already verified; it can queue \`send\` to a known pending sibling and \`ask\` a completed sibling that retains a valid conversation, which reopens it for a post-mortem turn. Delegated subagents inside a stage inherit the same group. Named subgroups isolate their members from sibling subgroups. Default to peer-to-peer (swarm) exchange; the supervisor is not a message relay. There are no limits on communication between agents; it is encouraged whenever you need to. Stages proactively discover peers and broadcast, debate, claim, hand off, and ask each other directly, and each stage still returns its own structured result. Scale the swarm with the task: for very complex work or when the user asks for heavy parallelism, run hundreds or thousands of agents that self-organize into teams (named subgroups, per-stage subagent sets, child workflows) and coordinate peer-to-peer like an organization.
98184
99425
  - Prefer \`ctx.tool(name, args, fn)\` for workflow-owned TypeScript operations with side effects, such as filesystem writes, network mutations, and external API actions. It creates a tracked, non-attachable durable graph node before invoking \`fn\`; it may be used before, between, after, or without model stages. A completed \`ctx.tool\` call is durably checkpointed with its serializable result, so resume replays that result without rerunning \`fn\`. For checks that may fail during a bounded repair loop, opt into \`failureMode: "return"\`, branch on the typed outcome, and pass only the needed error fields to the repair stage or artifact yourself; Atomic does not inject failure evidence into later prompts. Retries do not bound a hung callback, so every \`ctx.tool\` callback that starts a child process or performs network I/O MUST set an explicit positive finite \`timeoutMs\` and forward the supplied \`signal\` to cancellable work. A timeout applies per attempt, aborts that attempt's signal, and gives retries a fresh deadline and signal. Cancellation and storage faults still throw. Keep pure computation and side-effect-free transformations as ordinary TypeScript. Do not wrap agent-stage internals or every function call indiscriminately; this rule applies to side effects orchestrated directly by the workflow definition.
98185
99426
  - Pass large stage context through files/artifacts and \`reads\` with an explicit \`Read the file at <path>...\` prompt rather than large \`previous\` payloads or injected session history.
98186
99427
  - Wrap critical stage-prompt content in \`<keepContext>\` / \`</keepContext>\`. Tagged content survives compression verbatim regardless of the compression ratio, and every line of the span including the tag lines is protected. A long-running stage will be compacted, and compaction ranks lines individually: an objective is verbose and restated while the constraint that narrows it is usually one line, so the constraint is the cheaper deletion and what survives reads as broader permission than intended. Tag role constraints that bound a stage to part of the work, acceptance criteria and immutable contracts, explicit prohibitions, and identifiers a stage must not lose such as a target branch, worktree path, or run ID. Do not tag bulk context: protected lines count against the keep target rather than raising it, so a large protected span makes the surrounding transcript compress harder — tag the constraint, not the material it applies to.