@bastani/atomic 0.9.43 → 0.9.44-alpha.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (117) hide show
  1. package/CHANGELOG.md +26 -0
  2. package/dist/builtin/intercom/CHANGELOG.md +12 -0
  3. package/dist/builtin/intercom/broker/broker.ts +31 -7
  4. package/dist/builtin/intercom/broker/client.ts +65 -0
  5. package/dist/builtin/intercom/index.bundle.mjs +227 -95
  6. package/dist/builtin/intercom/live-route-refusal.ts +20 -0
  7. package/dist/builtin/intercom/package.json +1 -1
  8. package/dist/builtin/intercom/skills/intercom/SKILL.md +10 -6
  9. package/dist/builtin/intercom/types.ts +20 -1
  10. package/dist/builtin/mcp/CHANGELOG.md +7 -0
  11. package/dist/builtin/mcp/package.json +1 -1
  12. package/dist/builtin/subagents/CHANGELOG.md +9 -0
  13. package/dist/builtin/subagents/README.md +8 -8
  14. package/dist/builtin/subagents/agents/worker.md +1 -1
  15. package/dist/builtin/subagents/package.json +1 -1
  16. package/dist/builtin/subagents/skills/agent-browser/SKILL.md +1 -1
  17. package/dist/builtin/subagents/skills/cua-driver/EMBEDDING.md +21 -2
  18. package/dist/builtin/subagents/skills/cua-driver/LINUX.md +2 -2
  19. package/dist/builtin/subagents/skills/cua-driver/MACOS.md +342 -10
  20. package/dist/builtin/subagents/skills/cua-driver/README.md +4 -3
  21. package/dist/builtin/subagents/skills/cua-driver/RUNTIME.md +3 -1
  22. package/dist/builtin/subagents/skills/cua-driver/SKILL.md +7 -4
  23. package/dist/builtin/subagents/skills/cua-driver/WINDOWS.md +6 -5
  24. package/dist/builtin/subagents/skills/cua-driver/WORKFLOW.md +15 -0
  25. package/dist/builtin/subagents/skills/herdr/SKILL.md +22 -10
  26. package/dist/builtin/subagents/skills/subagent/SKILL.md +8 -5
  27. package/dist/builtin/subagents/src/extension/index.bundle.mjs +6 -6
  28. package/dist/builtin/web-access/package.json +1 -1
  29. package/dist/builtin/workflows/CHANGELOG.md +14 -0
  30. package/dist/builtin/workflows/README.md +1 -1
  31. package/dist/builtin/workflows/builtin/{chunk-frqxd4d1.js → chunk-k8d7ehjx.js} +22 -0
  32. package/dist/builtin/workflows/builtin/{chunk-3s95f7ry.js → chunk-qxpymnaf.js} +1 -1
  33. package/dist/builtin/workflows/builtin/{chunk-edhf30ef.js → chunk-zjecnjxx.js} +1 -1
  34. package/dist/builtin/workflows/builtin/goal.js +2 -2
  35. package/dist/builtin/workflows/builtin/index.js +3 -3
  36. package/dist/builtin/workflows/builtin/ralph.js +2 -2
  37. package/dist/builtin/workflows/package.json +1 -1
  38. package/dist/builtin/workflows/src/extension/index.bundle.mjs +1750 -598
  39. package/dist/builtin/workflows/src/extension/local-worker-entry.mjs +1786 -600
  40. package/dist/builtin/workflows/src/index.js +1802 -650
  41. package/dist/builtin/workflows/src/local-worker-entry.mjs +1786 -600
  42. package/dist/cli/args.js +1 -1
  43. package/dist/cli/args.js.map +1 -1
  44. package/dist/core/exec.d.ts.map +1 -1
  45. package/dist/core/exec.js +11 -2
  46. package/dist/core/exec.js.map +1 -1
  47. package/dist/core/package-manager-resolver.d.ts.map +1 -1
  48. package/dist/core/package-manager-resolver.js +25 -14
  49. package/dist/core/package-manager-resolver.js.map +1 -1
  50. package/dist/core/package-manager-types.d.ts +8 -0
  51. package/dist/core/package-manager-types.d.ts.map +1 -1
  52. package/dist/core/package-manager-types.js.map +1 -1
  53. package/dist/core/package-manager.d.ts +1 -1
  54. package/dist/core/package-manager.d.ts.map +1 -1
  55. package/dist/core/package-manager.js.map +1 -1
  56. package/dist/core/resource-loader-extension-warnings.d.ts +4 -1
  57. package/dist/core/resource-loader-extension-warnings.d.ts.map +1 -1
  58. package/dist/core/resource-loader-extension-warnings.js +15 -0
  59. package/dist/core/resource-loader-extension-warnings.js.map +1 -1
  60. package/dist/core/resource-loader-package-resources.d.ts.map +1 -1
  61. package/dist/core/resource-loader-package-resources.js +1 -0
  62. package/dist/core/resource-loader-package-resources.js.map +1 -1
  63. package/dist/core/resource-loader-reload.d.ts.map +1 -1
  64. package/dist/core/resource-loader-reload.js +5 -3
  65. package/dist/core/resource-loader-reload.js.map +1 -1
  66. package/dist/extensions/mcp/client/oauth/flow.d.ts.map +1 -1
  67. package/dist/extensions/mcp/client/oauth/flow.js +4 -1
  68. package/dist/extensions/mcp/client/oauth/flow.js.map +1 -1
  69. package/dist/extensions/mcp/client/oauth/types.d.ts +1 -0
  70. package/dist/extensions/mcp/client/oauth/types.d.ts.map +1 -1
  71. package/dist/extensions/mcp/client/oauth/types.js +8 -4
  72. package/dist/extensions/mcp/client/oauth/types.js.map +1 -1
  73. package/dist/extensions/mcp/client/transports/streamable-http.d.ts.map +1 -1
  74. package/dist/extensions/mcp/client/transports/streamable-http.js +2 -1
  75. package/dist/extensions/mcp/client/transports/streamable-http.js.map +1 -1
  76. package/dist/index.d.ts +1 -1
  77. package/dist/index.d.ts.map +1 -1
  78. package/dist/index.js.map +1 -1
  79. package/dist/main.d.ts.map +1 -1
  80. package/dist/main.js +20 -10
  81. package/dist/main.js.map +1 -1
  82. package/dist/modes/interactive/components/model-selector.d.ts.map +1 -1
  83. package/dist/modes/interactive/components/model-selector.js +4 -2
  84. package/dist/modes/interactive/components/model-selector.js.map +1 -1
  85. package/dist/modes/interactive/components/scoped-models-selector.d.ts.map +1 -1
  86. package/dist/modes/interactive/components/scoped-models-selector.js +4 -2
  87. package/dist/modes/interactive/components/scoped-models-selector.js.map +1 -1
  88. package/dist/modes/interactive/components/tool-execution.d.ts +6 -0
  89. package/dist/modes/interactive/components/tool-execution.d.ts.map +1 -1
  90. package/dist/modes/interactive/components/tool-execution.js +11 -1
  91. package/dist/modes/interactive/components/tool-execution.js.map +1 -1
  92. package/dist/modes/interactive/model-search.d.ts +10 -0
  93. package/dist/modes/interactive/model-search.d.ts.map +1 -1
  94. package/dist/modes/interactive/model-search.js +21 -0
  95. package/dist/modes/interactive/model-search.js.map +1 -1
  96. package/dist/package-manager-cli.d.ts.map +1 -1
  97. package/dist/package-manager-cli.js +5 -0
  98. package/dist/package-manager-cli.js.map +1 -1
  99. package/dist/utils/tools-manager.d.ts.map +1 -1
  100. package/dist/utils/tools-manager.js +30 -35
  101. package/dist/utils/tools-manager.js.map +1 -1
  102. package/docs/computer-use.md +5 -5
  103. package/docs/custom-provider/streaming.md +9 -14
  104. package/docs/environment-variables.md +1 -0
  105. package/docs/intercom.md +5 -1
  106. package/docs/packages.md +2 -0
  107. package/docs/settings.md +2 -2
  108. package/docs/skills.md +2 -2
  109. package/docs/subagents.md +38 -9
  110. package/docs/workflows/api-reference.md +1 -1
  111. package/docs/workflows/authoring.md +1 -1
  112. package/docs/workflows/operations.md +8 -6
  113. package/docs/workflows/reliable-design.md +23 -5
  114. package/examples/extensions/custom-provider-anthropic/index.ts +31 -31
  115. package/examples/extensions/todo.ts +7 -0
  116. package/npm-shrinkwrap.json +32 -32
  117. package/package.json +3 -3
@@ -1858,7 +1858,13 @@ function createStoreContext(state = createStoreState()) {
1858
1858
  }
1859
1859
  return cachedGraphSnapshot;
1860
1860
  }
1861
+ let batchDepth = 0;
1862
+ let batchedNotification = false;
1861
1863
  function notify() {
1864
+ if (batchDepth > 0) {
1865
+ batchedNotification = true;
1866
+ return;
1867
+ }
1862
1868
  observation.capture();
1863
1869
  for (const fn of state.invalidationListeners) {
1864
1870
  try {
@@ -1878,6 +1884,18 @@ function createStoreContext(state = createStoreState()) {
1878
1884
  state.version++;
1879
1885
  notify();
1880
1886
  }
1887
+ function batch(update) {
1888
+ batchDepth++;
1889
+ try {
1890
+ return update();
1891
+ } finally {
1892
+ batchDepth--;
1893
+ if (batchDepth === 0 && batchedNotification) {
1894
+ batchedNotification = false;
1895
+ notify();
1896
+ }
1897
+ }
1898
+ }
1881
1899
  function findRun(runId) {
1882
1900
  return state.runs.find((r) => r.id === runId);
1883
1901
  }
@@ -1931,6 +1949,7 @@ function createStoreContext(state = createStoreState()) {
1931
1949
  snapshot,
1932
1950
  graphSnapshot,
1933
1951
  notify,
1952
+ batch,
1934
1953
  bumpAndNotify,
1935
1954
  findRun,
1936
1955
  findStage,
@@ -3867,6 +3886,9 @@ function createRunStoreMethods(context) {
3867
3886
  subscribeInvalidation(fn) {
3868
3887
  state.invalidationListeners.add(fn);
3869
3888
  return () => state.invalidationListeners.delete(fn);
3889
+ },
3890
+ batch(update) {
3891
+ return context.batch(update);
3870
3892
  }
3871
3893
  };
3872
3894
  }
@@ -5840,22 +5862,37 @@ function childScopedEvidenceExists(checkpoints, boundaryReplayKey, childRunId) {
5840
5862
  }
5841
5863
  function retainReachableRunGroups(grouped, rootRunId) {
5842
5864
  const reachable = new Set([rootRunId]);
5843
- const pending = [rootRunId];
5865
+ const settled = new Set;
5866
+ const pending = [{ runId: rootRunId, live: true }];
5844
5867
  while (pending.length > 0) {
5845
- const drafts = grouped.get(pending.pop());
5868
+ const { runId, live } = pending.pop();
5869
+ const drafts = grouped.get(runId);
5846
5870
  if (drafts === undefined)
5847
5871
  continue;
5848
5872
  for (const draft of drafts) {
5849
- const childRunId = childRunIdFromDraft(draft);
5850
- if (childRunId === undefined || reachable.has(childRunId) || !grouped.has(childRunId))
5873
+ const childRunId = workflowChildFromDraft(draft)?.runId ?? draft.topology?.boundary?.child.runId;
5874
+ if (childRunId === undefined || !grouped.has(childRunId))
5851
5875
  continue;
5852
- reachable.add(childRunId);
5853
- pending.push(childRunId);
5876
+ if (live && childRunIdFromDraft(draft) !== undefined) {
5877
+ if (reachable.has(childRunId))
5878
+ continue;
5879
+ reachable.add(childRunId);
5880
+ pending.push({ runId: childRunId, live: true });
5881
+ } else if (!settled.has(childRunId)) {
5882
+ settled.add(childRunId);
5883
+ pending.push({ runId: childRunId, live: false });
5884
+ }
5854
5885
  }
5855
5886
  }
5856
- for (const runId of grouped.keys())
5857
- if (!reachable.has(runId))
5858
- grouped.delete(runId);
5887
+ let complete = true;
5888
+ for (const runId of grouped.keys()) {
5889
+ if (reachable.has(runId))
5890
+ continue;
5891
+ if (!settled.has(runId))
5892
+ complete = false;
5893
+ grouped.delete(runId);
5894
+ }
5895
+ return complete;
5859
5896
  }
5860
5897
  function childRunIdFromDraft(draft) {
5861
5898
  const boundary = draft.topology?.boundary;
@@ -15531,7 +15568,7 @@ class ConcurrencyLimiter {
15531
15568
  }
15532
15569
  }
15533
15570
  function createRunLimiter(defaultConcurrency) {
15534
- return new ConcurrencyLimiter(defaultConcurrency ?? 3);
15571
+ return new ConcurrencyLimiter(defaultConcurrency ?? 1000);
15535
15572
  }
15536
15573
 
15537
15574
  // dist/builtin/workflows/src/shared/possible-stages.ts
@@ -17023,7 +17060,7 @@ function kebabCase(name) {
17023
17060
  function errorMessage2(error) {
17024
17061
  return error instanceof Error ? error.message : String(error);
17025
17062
  }
17026
- var DEFAULT_MAX_DEPTH = 4, BUILTIN_BARREL_SPECIFIERS, BUILTIN_SUBPATH_PREFIXES, TRACKED_NODE_METHODS;
17063
+ var DEFAULT_MAX_DEPTH = 3, BUILTIN_BARREL_SPECIFIERS, BUILTIN_SUBPATH_PREFIXES, TRACKED_NODE_METHODS;
17027
17064
  var init_possible_stages = __esm(() => {
17028
17065
  BUILTIN_BARREL_SPECIFIERS = new Set(["@bastani/atomic/workflows/builtin"]);
17029
17066
  BUILTIN_SUBPATH_PREFIXES = ["@bastani/atomic/workflows/builtin/"];
@@ -60578,6 +60615,44 @@ class GraphFrontierTracker {
60578
60615
  }
60579
60616
  }
60580
60617
 
60618
+ // dist/builtin/workflows/src/shared/external-run-routes.ts
60619
+ function claimExternalRunRoute(store, runId, owner) {
60620
+ let owners = registry.owners.get(store);
60621
+ if (owners === undefined) {
60622
+ owners = new Map;
60623
+ registry.owners.set(store, owners);
60624
+ }
60625
+ owners.set(runId, owner);
60626
+ return () => {
60627
+ const current = registry.owners.get(store);
60628
+ if (current?.get(runId) !== owner)
60629
+ return;
60630
+ current.delete(runId);
60631
+ if (current.size === 0)
60632
+ registry.owners.delete(store);
60633
+ };
60634
+ }
60635
+ function externalRunRouteOwner(store, runId) {
60636
+ return registry.owners.get(store)?.get(runId);
60637
+ }
60638
+ function registerPendingStageMessageRouter(store, router) {
60639
+ registry.routers.set(store, router);
60640
+ return () => {
60641
+ if (registry.routers.get(store) === router)
60642
+ registry.routers.delete(store);
60643
+ };
60644
+ }
60645
+ function pendingStageMessageRouter(store) {
60646
+ return registry.routers.get(store);
60647
+ }
60648
+ var registryKey, processState, registry;
60649
+ var init_external_run_routes = __esm(() => {
60650
+ registryKey = Symbol.for("atomic-workflows/external-run-routes@1");
60651
+ processState = globalThis;
60652
+ registry = processState[registryKey] ?? { owners: new WeakMap, routers: new WeakMap };
60653
+ processState[registryKey] = registry;
60654
+ });
60655
+
60581
60656
  // dist/builtin/workflows/src/shared/pending-stage-route-capability.ts
60582
60657
  import { randomUUID as randomUUID11 } from "node:crypto";
60583
60658
  function capabilitiesFor(store) {
@@ -60602,12 +60677,12 @@ function rotateWorkflowPendingStageRouteCapability(store, runId) {
60602
60677
  capabilitiesFor(store).set(runId, capability);
60603
60678
  return capability;
60604
60679
  }
60605
- var routeCapabilitiesKey, processState, routeCapabilities;
60680
+ var routeCapabilitiesKey, processState2, routeCapabilities;
60606
60681
  var init_pending_stage_route_capability = __esm(() => {
60607
60682
  routeCapabilitiesKey = Symbol.for("atomic-workflows/pending-stage-route-capabilities@1");
60608
- processState = globalThis;
60609
- routeCapabilities = processState[routeCapabilitiesKey] ?? new WeakMap;
60610
- processState[routeCapabilitiesKey] = routeCapabilities;
60683
+ processState2 = globalThis;
60684
+ routeCapabilities = processState2[routeCapabilitiesKey] ?? new WeakMap;
60685
+ processState2[routeCapabilitiesKey] = routeCapabilities;
60611
60686
  });
60612
60687
 
60613
60688
  // dist/builtin/workflows/src/shared/pending-stage-route-readiness.ts
@@ -60639,12 +60714,12 @@ function hasWorkflowPendingStageRouteOwner(store) {
60639
60714
  function workflowPendingStageRouteReady(store, runId) {
60640
60715
  return owners.get(store)?.ready(runId);
60641
60716
  }
60642
- var routeOwnersKey, processState2, owners;
60717
+ var routeOwnersKey, processState3, owners;
60643
60718
  var init_pending_stage_route_readiness = __esm(() => {
60644
60719
  routeOwnersKey = Symbol.for("atomic-workflows/pending-stage-route-owners@1");
60645
- processState2 = globalThis;
60646
- owners = processState2[routeOwnersKey] ?? new WeakMap;
60647
- processState2[routeOwnersKey] = owners;
60720
+ processState3 = globalThis;
60721
+ owners = processState3[routeOwnersKey] ?? new WeakMap;
60722
+ processState3[routeOwnersKey] = owners;
60648
60723
  });
60649
60724
 
60650
60725
  // dist/builtin/workflows/src/shared/sticky-stage-delivery-status.ts
@@ -60769,6 +60844,9 @@ function registerPendingStageIntercomBridge(pi, activeStore, ownsRun = () => tru
60769
60844
  notificationId,
60770
60845
  reason
60771
60846
  };
60847
+ const external = externalRunRouteOwner(activeStore, entry.runId);
60848
+ if (external !== undefined)
60849
+ return external.notifyUndeliverable(payload);
60772
60850
  pi.events?.emit?.(PENDING_STAGE_UNDELIVERABLE_EVENT, payload);
60773
60851
  return payload.handled && await payload.completion === true;
60774
60852
  };
@@ -60780,19 +60858,29 @@ function registerPendingStageIntercomBridge(pi, activeStore, ownsRun = () => tru
60780
60858
  return;
60781
60859
  const ownedRunIds = new Set;
60782
60860
  const runs = activeStore.runs();
60861
+ const runById = new Map(runs.map((run) => [run.id, run]));
60862
+ const childrenByBoundary = new Map;
60863
+ for (const run of runs) {
60864
+ if (run.parentRunId === undefined || run.parentStageId === undefined)
60865
+ continue;
60866
+ const key = JSON.stringify([run.parentRunId, run.parentStageId]);
60867
+ const children = childrenByBoundary.get(key) ?? [];
60868
+ children.push(run);
60869
+ childrenByBoundary.set(key, children);
60870
+ }
60783
60871
  const resolveBoundarySegments = createWorkflowBoundarySegmentsResolver(runs);
60784
60872
  for (const run of runs) {
60785
- if (!ownsRun(run.id))
60873
+ if (externalRunRouteOwner(activeStore, run.id) !== undefined || !ownsRun(run.id))
60786
60874
  continue;
60787
- const rootRunId = durableRootRunIdForRun(runs, run.id);
60875
+ const rootRunId = reciprocalWorkflowRootRunId(runById, run.id);
60788
60876
  if (rootRunId === undefined)
60789
60877
  continue;
60790
60878
  ownedRunIds.add(run.id);
60791
- const parentRun = runs.find((candidate) => candidate.id === run.parentRunId);
60879
+ const parentRun = run.parentRunId === undefined ? undefined : runById.get(run.parentRunId);
60792
60880
  const boundary = parentRun?.stages.find((stage) => stage.id === run.parentStageId);
60793
60881
  const parent = parentRun === undefined || boundary === undefined ? undefined : {
60794
60882
  runId: parentRun.id,
60795
- stageKeys: [boundary.id, boundary.name].filter((key) => resolveChildRun(runs, parentRun, key)?.id === run.id)
60883
+ stageKeys: [boundary.id, boundary.name].filter((key) => resolveIndexedChildRun(childrenByBoundary, parentRun, key)?.id === run.id)
60796
60884
  };
60797
60885
  const announcement = {
60798
60886
  runId: run.id,
@@ -60933,16 +61021,22 @@ function registerPendingStageIntercomBridge(pi, activeStore, ownsRun = () => tru
60933
61021
  payload.handled = true;
60934
61022
  payload.completion = queueAndPersist(activeStore, resolvedEvent, workflowInvocationIntercomGroup(rootRunId), stage.pendingStageDeliveryAvailable === true);
60935
61023
  };
60936
- const subscription = pi.events?.on?.(PENDING_STAGE_MESSAGE_EVENT, (payload) => outsideWorkflowExecutions(() => routePendingStageMessage(payload)));
60937
- const stickySubscription = pi.events?.on?.(STICKY_LIVE_DELIVERED_EVENT, (payload) => {
61024
+ const routeStickyLiveDelivered = (payload) => {
60938
61025
  if (disposed || !isStickyLiveDeliveredEvent(payload) || payload.handled)
60939
61026
  return;
60940
61027
  payload.handled = true;
60941
61028
  payload.completion = outsideWorkflowExecutions(() => recordConfirmedStickyDeliveries(activeStore, payload));
61029
+ };
61030
+ const subscription = pi.events?.on?.(PENDING_STAGE_MESSAGE_EVENT, (payload) => outsideWorkflowExecutions(() => routePendingStageMessage(payload)));
61031
+ const stickySubscription = pi.events?.on?.(STICKY_LIVE_DELIVERED_EVENT, routeStickyLiveDelivered);
61032
+ const disposeRouter = registerPendingStageMessageRouter(activeStore, {
61033
+ route: (payload) => outsideWorkflowExecutions(() => routePendingStageMessage(payload)),
61034
+ sticky: routeStickyLiveDelivered
60942
61035
  });
60943
61036
  const dispose = () => {
60944
61037
  disposed = true;
60945
61038
  disposeReadiness();
61039
+ disposeRouter();
60946
61040
  routeCompletions.clear();
60947
61041
  unsubscribeStore();
60948
61042
  if (typeof subscription === "function")
@@ -61068,6 +61162,12 @@ function isMaterializedNonAgentTarget(runs, target) {
61068
61162
  return false;
61069
61163
  return nonAgentNodes(run).some((node) => node.id === stageKey || node.name === stageKey);
61070
61164
  }
61165
+ function resolveIndexedChildRun(childrenByBoundary, parent, segment) {
61166
+ const boundariesById = parent.stages.filter((stage) => stage.id === segment);
61167
+ const boundaries = boundariesById.length > 0 ? boundariesById : parent.stages.filter((stage) => stage.name === segment);
61168
+ const children = boundaries.flatMap((boundary) => childrenByBoundary.get(JSON.stringify([parent.id, boundary.id])) ?? []);
61169
+ return children.length === 1 ? children[0] : undefined;
61170
+ }
61071
61171
  function resolveChildRun(runs, parent, segment) {
61072
61172
  const boundariesById = parent.stages.filter((stage) => stage.id === segment);
61073
61173
  const boundaries = boundariesById.length > 0 ? boundariesById : parent.stages.filter((stage) => stage.name === segment);
@@ -61407,6 +61507,7 @@ var init_pending_stage_intercom = __esm(() => {
61407
61507
  init_execution_context_boundary();
61408
61508
  init_run_owner_backend();
61409
61509
  init_store_backend();
61510
+ init_external_run_routes();
61410
61511
  init_intercom_group();
61411
61512
  init_pending_stage_route_capability();
61412
61513
  init_pending_stage_route_readiness();
@@ -61865,7 +61966,7 @@ function isWorkflowFutureStageRosterEntries(value2) {
61865
61966
  function isWorkflowStageRosterEntries(value2) {
61866
61967
  return Array.isArray(value2) && value2.every((entry) => typeof entry === "object" && entry !== null && entry.kind === "workflow-stage" && typeof entry.runId === "string" && typeof entry.stageId === "string" && typeof entry.stageName === "string" && typeof entry.target === "string" && (entry.lifecycle === "pending" || entry.lifecycle === "running") && typeof entry.group === "string" && (entry.sessionId === undefined || typeof entry.sessionId === "string"));
61867
61968
  }
61868
- var BROKER_SOCKET, BROKER_CONNECT_TIMEOUT_MS = 1e4, PENDING_STAGE_ROUTE_ACK_TIMEOUT_MS = 1e4, GROUP_REQUEST_TIMEOUT_MS = 5000, PRESENCE_ACK_TIMEOUT_MS = 5000, RESTORING_RECIPIENT_REQUEST_TIMEOUT_MS = 5000, HELD_SEND_GRACE_MS = 5000, IntercomClient;
61969
+ var BROKER_SOCKET, BROKER_CONNECT_TIMEOUT_MS = 1e4, PENDING_STAGE_ROUTE_ACK_TIMEOUT_MS = 1e4, GROUP_REQUEST_TIMEOUT_MS = 5000, PRESENCE_ACK_TIMEOUT_MS = 5000, RESTORING_RECIPIENT_REQUEST_TIMEOUT_MS = 5000, HELD_SEND_GRACE_MS = 5000, IntercomPendingStageRouteRefusedError, PENDING_STAGE_ROUTE_REFUSAL_CODES, IntercomClient;
61869
61970
  var init_client = __esm(() => {
61870
61971
  init_paths();
61871
61972
  init_pending_send_registry();
@@ -61874,6 +61975,21 @@ var init_client = __esm(() => {
61874
61975
  init_live_route_refusal();
61875
61976
  init_retry_policy();
61876
61977
  BROKER_SOCKET = getBrokerSocketPath();
61978
+ IntercomPendingStageRouteRefusedError = class IntercomPendingStageRouteRefusedError extends Error {
61979
+ connectionKept;
61980
+ code;
61981
+ constructor(reason, refusal = { connectionKept: false }) {
61982
+ super(reason);
61983
+ this.name = "IntercomPendingStageRouteRefusedError";
61984
+ this.connectionKept = refusal.connectionKept;
61985
+ this.code = refusal.code;
61986
+ }
61987
+ };
61988
+ PENDING_STAGE_ROUTE_REFUSAL_CODES = new Set([
61989
+ "not_authorized",
61990
+ "invalid_stage_roster",
61991
+ "invalid_possible_stage_roster"
61992
+ ]);
61877
61993
  IntercomClient = class IntercomClient extends EventEmitter {
61878
61994
  returnAddress;
61879
61995
  preferredSessionId;
@@ -61881,6 +61997,8 @@ var init_client = __esm(() => {
61881
61997
  connectTimeoutMs;
61882
61998
  brokerAcknowledgesReceipt = false;
61883
61999
  brokerAcknowledgesPendingStageRoutes = false;
62000
+ multiplexesPendingStageRoutes;
62001
+ brokerRefusesPendingStageRoutesInPlace = false;
61884
62002
  brokerAnswersStatus = false;
61885
62003
  socket = null;
61886
62004
  _sessionId = null;
@@ -61946,6 +62064,7 @@ var init_client = __esm(() => {
61946
62064
  this.preferredSessionId = options.preferredSessionId;
61947
62065
  this.acknowledgesReceipt = options.acknowledgesReceipt === true;
61948
62066
  this.connectTimeoutMs = options.connectTimeoutMs ?? BROKER_CONNECT_TIMEOUT_MS;
62067
+ this.multiplexesPendingStageRoutes = options.multiplexesPendingStageRoutes === true;
61949
62068
  }
61950
62069
  failPending(error) {
61951
62070
  this.pendingSends.rejectAll(error);
@@ -62058,6 +62177,7 @@ var init_client = __esm(() => {
62058
62177
  this._supervisorSessionId = null;
62059
62178
  this.brokerAcknowledgesReceipt = false;
62060
62179
  this.brokerAcknowledgesPendingStageRoutes = false;
62180
+ this.brokerRefusesPendingStageRoutesInPlace = false;
62061
62181
  this.brokerAnswersStatus = false;
62062
62182
  this._messageSource = undefined;
62063
62183
  this.disconnectError = null;
@@ -62136,7 +62256,7 @@ var init_client = __esm(() => {
62136
62256
  }
62137
62257
  switch (brokerMessage.type) {
62138
62258
  case "registered": {
62139
- if (typeof brokerMessage.sessionId !== "string" || brokerMessage.supervisorSessionId !== undefined && typeof brokerMessage.supervisorSessionId !== "string" || brokerMessage.acknowledgesReceipt !== undefined && brokerMessage.acknowledgesReceipt !== true || brokerMessage.acknowledgesPendingStageRoutes !== undefined && brokerMessage.acknowledgesPendingStageRoutes !== true || brokerMessage.brokerStatus !== undefined && brokerMessage.brokerStatus !== true) {
62259
+ if (typeof brokerMessage.sessionId !== "string" || brokerMessage.supervisorSessionId !== undefined && typeof brokerMessage.supervisorSessionId !== "string" || brokerMessage.acknowledgesReceipt !== undefined && brokerMessage.acknowledgesReceipt !== true || brokerMessage.acknowledgesPendingStageRoutes !== undefined && brokerMessage.acknowledgesPendingStageRoutes !== true || brokerMessage.refusesPendingStageRoutesInPlace !== undefined && brokerMessage.refusesPendingStageRoutesInPlace !== true || brokerMessage.brokerStatus !== undefined && brokerMessage.brokerStatus !== true) {
62140
62260
  throw new Error("Invalid registered message");
62141
62261
  }
62142
62262
  if (this._sessionId !== null) {
@@ -62146,6 +62266,7 @@ var init_client = __esm(() => {
62146
62266
  this._supervisorSessionId = typeof brokerMessage.supervisorSessionId === "string" ? brokerMessage.supervisorSessionId : null;
62147
62267
  this.brokerAcknowledgesReceipt = this.acknowledgesReceipt && brokerMessage.acknowledgesReceipt === true;
62148
62268
  this.brokerAcknowledgesPendingStageRoutes = brokerMessage.acknowledgesPendingStageRoutes === true;
62269
+ this.brokerRefusesPendingStageRoutesInPlace = brokerMessage.refusesPendingStageRoutesInPlace === true;
62149
62270
  this.brokerAnswersStatus = brokerMessage.brokerStatus === true;
62150
62271
  this.emit("_registered", { type: "registered", sessionId: brokerMessage.sessionId });
62151
62272
  break;
@@ -62158,6 +62279,12 @@ var init_client = __esm(() => {
62158
62279
  this.emit("_registration_failed", refusal);
62159
62280
  break;
62160
62281
  }
62282
+ const refusedRouteId = typeof brokerMessage.requestId === "string" ? brokerMessage.requestId : "";
62283
+ const refusedRoute = this.pendingStageRouteRegistrations.get(refusedRouteId);
62284
+ if (refusedRoute !== undefined) {
62285
+ this.pendingStageRouteRegistrations.delete(refusedRouteId);
62286
+ refusedRoute.reject(new IntercomPendingStageRouteRefusedError(brokerMessage.reason));
62287
+ }
62161
62288
  this.disconnectError ??= refusal;
62162
62289
  this.failPending(this.disconnectError);
62163
62290
  this.socket?.destroy();
@@ -62283,6 +62410,21 @@ var init_client = __esm(() => {
62283
62410
  this.emit("pending_stage_route_revoked", { runId });
62284
62411
  break;
62285
62412
  }
62413
+ case "pending_stage_route_refused": {
62414
+ const { requestId, runId, code, reason } = brokerMessage;
62415
+ if (typeof requestId !== "string" || typeof runId !== "string" || typeof reason !== "string" || typeof code !== "string") {
62416
+ throw new Error("Invalid pending-stage route refusal");
62417
+ }
62418
+ const pending = this.pendingStageRouteRegistrations.get(requestId);
62419
+ if (!pending)
62420
+ return;
62421
+ this.pendingStageRouteRegistrations.delete(requestId);
62422
+ pending.reject(new IntercomPendingStageRouteRefusedError(reason, {
62423
+ connectionKept: true,
62424
+ ...PENDING_STAGE_ROUTE_REFUSAL_CODES.has(code) ? { code } : {}
62425
+ }));
62426
+ break;
62427
+ }
62286
62428
  case "pending_stage_route_registered": {
62287
62429
  const { requestId } = brokerMessage;
62288
62430
  if (typeof requestId !== "string")
@@ -62786,7 +62928,8 @@ var init_client = __esm(() => {
62786
62928
  stages,
62787
62929
  possibleStages,
62788
62930
  ...parent === undefined ? {} : { parent },
62789
- ...dormant === true ? { dormant: true } : {}
62931
+ ...dormant === true ? { dormant: true } : {},
62932
+ ...this.multiplexesPendingStageRoutes && this.brokerRefusesPendingStageRoutesInPlace ? { refusalKeepsConnection: true } : {}
62790
62933
  });
62791
62934
  } catch (error) {
62792
62935
  clearTimeout(timeout);
@@ -63520,120 +63663,165 @@ var init_reconnect_backoff = __esm(() => {
63520
63663
  BACKOFF_MS = [1000, 2000, 5000, 1e4, 30000];
63521
63664
  });
63522
63665
 
63523
- // dist/builtin/intercom/route-connect-gate.ts
63524
- function createConcurrencyGate(limit) {
63525
- let active = 0;
63526
- const waiting = [];
63527
- const release = () => {
63528
- const next = waiting.shift();
63529
- if (next === undefined)
63530
- active -= 1;
63531
- else
63532
- next();
63533
- };
63534
- return {
63535
- async run(operation) {
63536
- if (active < limit)
63537
- active += 1;
63538
- else
63539
- await new Promise((resolve) => waiting.push(resolve));
63540
- try {
63541
- return await operation();
63542
- } finally {
63543
- release();
63544
- }
63545
- }
63546
- };
63547
- }
63548
- var ROUTE_CLIENT_CONNECT_CONCURRENCY = 4;
63549
-
63550
63666
  // dist/builtin/intercom/workflow-route-owner.ts
63551
63667
  import { randomUUID as randomUUID14 } from "node:crypto";
63552
63668
  import { createEventBus, getBuiltinPackagePaths } from "@bastani/atomic";
63669
+ function closedError() {
63670
+ return new Error("Workflow Intercom route owner closed");
63671
+ }
63553
63672
  function createWorkflowRouteOwner(cwd) {
63554
63673
  const events = createEventBus();
63555
63674
  const config = loadConfig();
63556
63675
  const id = randomUUID14();
63557
63676
  const routes = new Map;
63558
- const gate = createConcurrencyGate(ROUTE_CLIENT_CONNECT_CONCURRENCY);
63677
+ const connections = new Map;
63678
+ const isCurrent = (state) => routes.get(state.route.runId) === state;
63679
+ const connectionKey = (state) => state.isolated ? state.route.runId : state.route.group;
63680
+ function connectionOf(state) {
63681
+ const key = connectionKey(state);
63682
+ let connection = connections.get(key);
63683
+ if (connection === undefined) {
63684
+ connection = { key, group: state.route.group, acknowledged: false, routes: new Map };
63685
+ connections.set(key, connection);
63686
+ }
63687
+ connection.routes.set(state.route.runId, state);
63688
+ return connection;
63689
+ }
63690
+ function detach(state) {
63691
+ const key = connectionKey(state);
63692
+ const connection = connections.get(key);
63693
+ connection?.routes.delete(state.route.runId);
63694
+ if (!state.isolated || connection === undefined)
63695
+ return;
63696
+ connections.delete(key);
63697
+ connection.client?.disconnect().catch(() => {});
63698
+ }
63699
+ function settle(state, registered) {
63700
+ state.registered = registered;
63701
+ if (!registered && isCurrent(state))
63702
+ return;
63703
+ for (const waiter of state.waiters)
63704
+ waiter(registered);
63705
+ state.waiters.clear();
63706
+ }
63559
63707
  function scheduleRetry(state) {
63560
- if (state.closed || state.retry !== undefined)
63708
+ if (!isCurrent(state) || state.retry !== undefined)
63561
63709
  return;
63562
63710
  state.retry = setTimeout(() => {
63563
63711
  state.retry = undefined;
63564
- publish(state, state.route).catch(() => scheduleRetry(state));
63712
+ publish(state, state.route).catch(() => {});
63565
63713
  }, jitteredReconnectDelayMs(state.attempt++));
63566
63714
  state.retry.unref();
63567
63715
  }
63568
- async function connect(state) {
63569
- if (state.client?.isConnected())
63570
- return state.client;
63571
- const [extensionDir] = getBuiltinPackagePaths({
63572
- workflows: false,
63573
- subagents: false,
63574
- mcp: false,
63575
- "web-access": false
63576
- });
63577
- if (extensionDir === undefined)
63578
- throw new Error("Intercom builtin package is unavailable");
63579
- await spawnBrokerIfNeeded(config.brokerCommand, config.brokerArgs, undefined, extensionDir);
63580
- if (state.closed)
63581
- throw new Error("Workflow Intercom route owner closed");
63582
- const client = new IntercomClient(`${id}:pending-stage-route:${state.route.runId}`);
63583
- state.client = client;
63584
- client.on("error", () => {});
63585
- client.on("disconnected", () => {
63586
- if (state.client !== client)
63587
- return;
63588
- state.client = undefined;
63589
- scheduleRetry(state);
63590
- });
63591
- client.on("pending_stage_message", (request) => {
63592
- if (state.closed || state.client !== client)
63593
- return;
63594
- const event = { ...request, handled: false };
63595
- events.emit("atomic:workflow-pending-stage-message", event);
63596
- const completion = event.handled && event.completion !== undefined ? event.completion : Promise.resolve({ outcome: "refused", reason: "Session not found" });
63597
- completion.catch((error) => ({ outcome: "refused", reason: error.message })).then((result) => client.respondPendingStageMessage(request.requestId, result)).catch(() => {});
63598
- });
63599
- client.on("sticky_live_delivered", (notice) => {
63600
- if (state.closed || state.client !== client)
63601
- return;
63602
- events.emit("atomic:workflow-sticky-live-delivered", { ...notice, handled: false });
63603
- });
63604
- await client.connect({
63605
- cwd,
63606
- model: "workflow",
63607
- pid: process.pid,
63608
- startedAt: Date.now(),
63609
- lastActivity: Date.now(),
63610
- group: state.route.group,
63611
- groups: [state.route.group],
63612
- recipientPurpose: "control"
63716
+ function drop(connection, client) {
63717
+ if (connection.client === client) {
63718
+ connection.client = undefined;
63719
+ connection.acknowledged = false;
63720
+ for (const state of connection.routes.values()) {
63721
+ settle(state, false);
63722
+ scheduleRetry(state);
63723
+ }
63724
+ }
63725
+ client.disconnect().catch(() => {});
63726
+ }
63727
+ function connect(connection) {
63728
+ if (connection.client?.isConnected())
63729
+ return Promise.resolve(connection.client);
63730
+ connection.connecting ??= (async () => {
63731
+ const [extensionDir] = getBuiltinPackagePaths({
63732
+ workflows: false,
63733
+ subagents: false,
63734
+ mcp: false,
63735
+ "web-access": false
63736
+ });
63737
+ if (extensionDir === undefined)
63738
+ throw new Error("Intercom builtin package is unavailable");
63739
+ await spawnBrokerIfNeeded(config.brokerCommand, config.brokerArgs, undefined, extensionDir);
63740
+ if (connections.get(connection.key) !== connection)
63741
+ throw closedError();
63742
+ const client = new IntercomClient(`${id}:pending-stage-route:${connection.key}`, {
63743
+ multiplexesPendingStageRoutes: true
63744
+ });
63745
+ connection.client = client;
63746
+ connection.acknowledged = false;
63747
+ client.on("error", () => {});
63748
+ client.on("disconnected", () => drop(connection, client));
63749
+ client.on("pending_stage_message", (request) => {
63750
+ if (connection.client !== client)
63751
+ return;
63752
+ const event = { ...request, handled: false };
63753
+ events.emit("atomic:workflow-pending-stage-message", event);
63754
+ const completion = event.handled && event.completion !== undefined ? event.completion : Promise.resolve({ outcome: "refused", reason: "Session not found" });
63755
+ completion.catch((error) => ({ outcome: "refused", reason: error.message })).then((result) => client.respondPendingStageMessage(request.requestId, result)).catch(() => {});
63756
+ });
63757
+ client.on("sticky_live_delivered", (notice) => {
63758
+ if (connection.client !== client)
63759
+ return;
63760
+ events.emit("atomic:workflow-sticky-live-delivered", { ...notice, handled: false });
63761
+ });
63762
+ try {
63763
+ await client.connect({
63764
+ cwd,
63765
+ model: "workflow",
63766
+ pid: process.pid,
63767
+ startedAt: Date.now(),
63768
+ lastActivity: Date.now(),
63769
+ group: connection.group,
63770
+ groups: [connection.group],
63771
+ recipientPurpose: "control"
63772
+ });
63773
+ if (connections.get(connection.key) !== connection)
63774
+ throw closedError();
63775
+ } catch (error) {
63776
+ drop(connection, client);
63777
+ throw error;
63778
+ }
63779
+ for (const state of connection.routes.values()) {
63780
+ if (state.registered || state.retry === undefined)
63781
+ continue;
63782
+ clearTimeout(state.retry);
63783
+ state.retry = undefined;
63784
+ publish(state, state.route).catch(() => {});
63785
+ }
63786
+ return client;
63787
+ })().finally(() => {
63788
+ connection.connecting = undefined;
63613
63789
  });
63614
- return client;
63790
+ return connection.connecting;
63615
63791
  }
63616
63792
  function publish(state, route) {
63617
- const completion = state.tail.catch(() => {}).then(() => gate.run(async () => {
63618
- if (state.closed)
63619
- throw new Error("Workflow Intercom route owner closed");
63620
- const establishedClient = state.client?.isConnected() ? state.client : undefined;
63793
+ const completion = state.tail.catch(() => {}).then(async () => {
63794
+ if (!isCurrent(state))
63795
+ throw closedError();
63796
+ const connection = connectionOf(state);
63797
+ const established = connection.acknowledged && connection.client?.isConnected() ? connection.client : undefined;
63798
+ let client;
63621
63799
  try {
63622
- const client = await connect(state);
63800
+ client = await connect(connection);
63623
63801
  await client.registerPendingStageRouteAcknowledged(route.runId, route.group, route.capability, route.stages, route.possibleStages, route.parent);
63802
+ if (connection.client !== client || !isCurrent(state))
63803
+ return;
63804
+ connection.acknowledged = true;
63624
63805
  state.attempt = 0;
63625
63806
  clearTimeout(state.retry);
63626
63807
  state.retry = undefined;
63808
+ settle(state, true);
63627
63809
  } catch (error) {
63628
- const client = state.client;
63629
- if (client !== establishedClient || !client?.isConnected()) {
63630
- state.client = undefined;
63631
- await client?.disconnect();
63810
+ if (error instanceof IntercomPendingStageRouteRefusedError && error.connectionKept) {
63811
+ scheduleRetry(state);
63812
+ throw error;
63813
+ }
63814
+ if (error instanceof IntercomPendingStageRouteRefusedError && !state.isolated && isCurrent(state)) {
63815
+ detach(state);
63816
+ state.isolated = true;
63817
+ connectionOf(state);
63632
63818
  }
63819
+ if (client !== undefined && (client !== established || !client.isConnected()))
63820
+ drop(connection, client);
63633
63821
  scheduleRetry(state);
63634
63822
  throw error;
63635
63823
  }
63636
- }));
63824
+ });
63637
63825
  state.tail = completion;
63638
63826
  completion.catch(() => {});
63639
63827
  return completion;
@@ -63642,10 +63830,15 @@ function createWorkflowRouteOwner(cwd) {
63642
63830
  const route = payload;
63643
63831
  let state = routes.get(route.runId);
63644
63832
  if (state === undefined) {
63645
- state = { route, tail: Promise.resolve(), attempt: 0, closed: false };
63833
+ state = { route, tail: Promise.resolve(), attempt: 0, registered: false, isolated: false, waiters: new Set };
63646
63834
  routes.set(route.runId, state);
63835
+ } else if (state.route.group !== route.group) {
63836
+ detach(state);
63837
+ state.isolated = false;
63838
+ settle(state, false);
63647
63839
  }
63648
63840
  state.route = route;
63841
+ connectionOf(state);
63649
63842
  route.completion = publish(state, route);
63650
63843
  });
63651
63844
  events.on("atomic:workflow-pending-stage-undeliverable", (payload) => {
@@ -63656,7 +63849,7 @@ function createWorkflowRouteOwner(cwd) {
63656
63849
  event.handled = true;
63657
63850
  event.completion = (async () => {
63658
63851
  await publish(state, state.route);
63659
- const client = state.client;
63852
+ const client = connections.get(connectionKey(state))?.client;
63660
63853
  if (client === undefined || !client.isConnected())
63661
63854
  return false;
63662
63855
  const text = `Pending workflow stage could not receive intercom message: ${event.reason}`;
@@ -63666,18 +63859,33 @@ function createWorkflowRouteOwner(cwd) {
63666
63859
  });
63667
63860
  return {
63668
63861
  events,
63862
+ registered(runId) {
63863
+ const state = routes.get(runId);
63864
+ if (state === undefined)
63865
+ return Promise.resolve(false);
63866
+ if (state.registered)
63867
+ return Promise.resolve(true);
63868
+ return new Promise((resolve) => state.waiters.add(resolve));
63869
+ },
63669
63870
  async closeGroup(group) {
63670
63871
  const closing = [...routes.values()].filter((state) => state.route.group === group);
63872
+ const closingConnections = [...connections.values()].filter((connection) => connection.group === group);
63873
+ for (const connection of closingConnections) {
63874
+ connections.delete(connection.key);
63875
+ connection.routes.clear();
63876
+ }
63671
63877
  for (const state of closing) {
63672
- state.closed = true;
63673
- clearTimeout(state.retry);
63674
63878
  routes.delete(state.route.runId);
63879
+ clearTimeout(state.retry);
63880
+ state.retry = undefined;
63881
+ settle(state, false);
63675
63882
  }
63676
- await Promise.all(closing.map(async (state) => {
63677
- await state.client?.disconnect();
63678
- await state.tail.catch(() => {});
63679
- await state.client?.disconnect();
63883
+ await Promise.all(closingConnections.map(async (connection) => {
63884
+ await connection.client?.disconnect();
63885
+ await connection.connecting?.catch(() => {});
63680
63886
  }));
63887
+ await Promise.all(closing.map((state) => state.tail.catch(() => {})));
63888
+ await Promise.all(closingConnections.map((connection) => connection.client?.disconnect()));
63681
63889
  }
63682
63890
  };
63683
63891
  }
@@ -63860,7 +64068,7 @@ var init_workflow_infrastructure_error = __esm(() => {
63860
64068
  [/Local worker (?:process )?cleanup (?:was not confirmed|unconfirmed)/, "worker_cleanup_unconfirmed"],
63861
64069
  [/Local worker retry limit/, "worker_retry_limit"],
63862
64070
  [/Local worker (?:startup timed out|exited before readiness)/, "worker_startup_failed"],
63863
- [/Worker exited before delivery acknowledgement/, "worker_lost"],
64071
+ [/Worker exited before delivery acknowledgement|Local worker pool coordinator exited/, "worker_lost"],
63864
64072
  [/Stale work lease/, "stale_work_lease"],
63865
64073
  [/Intercom runtime no longer active/, "intercom_unavailable"],
63866
64074
  [/Storage unavailable/, "storage_unavailable"]
@@ -64159,7 +64367,7 @@ var init_definitions = __esm(() => {
64159
64367
  // dist/builtin/workflows/src/durable/journal/pool-mutations.ts
64160
64368
  import { StorageProvider as StorageProvider2 } from "@bastani/atomic/storage";
64161
64369
  async function validatePoolMutation(tx, journal, root, work, mutation) {
64162
- if (!["checkpoint", "session-create", "session-append", "session-replace"].includes(mutation.kind))
64370
+ if (!["checkpoint", "session-create", "session-replace"].includes(mutation.kind))
64163
64371
  throw new Error("Invalid pool mutation");
64164
64372
  const checkpoint = mutation.checkpoint;
64165
64373
  const prefix = `${work.scope}:`;
@@ -64227,10 +64435,18 @@ async function sessionStore(tx, prefix) {
64227
64435
  return new StorageProvider2().openSessionStore({ kind: "database", database, prefix }, { agentDir: "", domain: "sessions" });
64228
64436
  }
64229
64437
  async function applyPoolSession(tx, prefix, workId, mutation) {
64438
+ const owners = `${prefix}session_owners`;
64439
+ if (mutation.kind === "session-append") {
64440
+ if (typeof mutation.sessionId !== "string" || mutation.sessionId === "")
64441
+ throw new Error("Pool sessions require database identity");
64442
+ const owner = await tx.get(`SELECT work FROM ${owners} WHERE session = ?`, mutation.sessionId);
64443
+ if (owner?.work !== workId)
64444
+ throw new Error("Pool session scope mismatch");
64445
+ return withSessionStore(tx, prefix, (store) => store.append(mutation.sessionId, mutation.entries, { expectSeq: mutation.expectSeq }));
64446
+ }
64230
64447
  const { sessionId, replayKey } = mutation.checkpoint;
64231
64448
  if (!sessionId || mutation.checkpoint.sessionFile !== undefined)
64232
64449
  throw new Error("Pool sessions require database identity");
64233
- const owners = `${prefix}session_owners`;
64234
64450
  if (mutation.kind === "session-create") {
64235
64451
  if (mutation.header.id !== sessionId)
64236
64452
  throw new Error("Pool session identity mismatch");
@@ -64240,10 +64456,12 @@ async function applyPoolSession(tx, prefix, workId, mutation) {
64240
64456
  if (owner?.work !== workId || owner.replay_key !== replayKey)
64241
64457
  throw new Error("Pool session scope mismatch");
64242
64458
  }
64459
+ return withSessionStore(tx, prefix, (store) => mutation.kind === "session-create" ? store.create(mutation.header, mutation.entries) : store.replace(sessionId, mutation.header, mutation.entries, { expectSeq: mutation.expectSeq }));
64460
+ }
64461
+ async function withSessionStore(tx, prefix, write) {
64243
64462
  const store = await sessionStore(tx, prefix);
64244
64463
  try {
64245
- const result = mutation.kind === "session-create" ? await store.create(mutation.header, mutation.entries) : mutation.kind === "session-append" ? await store.append(sessionId, mutation.entries, { expectSeq: mutation.expectSeq }) : await store.replace(sessionId, mutation.header, mutation.entries, { expectSeq: mutation.expectSeq });
64246
- return result.lastSeq;
64464
+ return (await write(store)).lastSeq;
64247
64465
  } finally {
64248
64466
  await store.close();
64249
64467
  }
@@ -64291,8 +64509,11 @@ var init_pool_stage_usage = __esm(() => {
64291
64509
  });
64292
64510
 
64293
64511
  // dist/builtin/workflows/src/durable/journal/sql-pool-queue.ts
64294
- import { randomUUID as randomUUID16 } from "node:crypto";
64512
+ import { createHash as createHash4, randomUUID as randomUUID16 } from "node:crypto";
64295
64513
  import { assertStorageSchemaVersion as assertStorageSchemaVersion2 } from "@bastani/atomic/storage";
64514
+ function requestIdentity(request) {
64515
+ return createHash4("sha256").update(request).digest("hex");
64516
+ }
64296
64517
  function fromRow(row) {
64297
64518
  return {
64298
64519
  ...JSON.parse(row.descriptor),
@@ -64348,10 +64569,11 @@ class SqlPoolQueue {
64348
64569
  const versions = await tx.all(`SELECT version FROM ${pool.meta}`);
64349
64570
  if (versions.length > 1)
64350
64571
  throw new Error("Corrupt pool schema metadata");
64351
- if (versions.length === 0)
64352
- await tx.run(`INSERT INTO ${pool.meta} VALUES (1)`);
64353
- else
64354
- assertStorageSchemaVersion2("journal", versions[0].version, 1);
64572
+ const version = versions[0]?.version;
64573
+ if (version === undefined)
64574
+ await tx.run(`INSERT INTO ${pool.meta} VALUES (?)`, POOL_SCHEMA_VERSION);
64575
+ else if (version !== 1)
64576
+ assertStorageSchemaVersion2("journal", version, POOL_SCHEMA_VERSION);
64355
64577
  await tx.exec(`
64356
64578
  CREATE TABLE IF NOT EXISTS ${pool.roots}(root TEXT PRIMARY KEY REFERENCES ${journal.runs}(run_id) ON DELETE CASCADE, timeout INTEGER NOT NULL);
64357
64579
  CREATE TABLE IF NOT EXISTS ${pool.work}(
@@ -64382,9 +64604,23 @@ class SqlPoolQueue {
64382
64604
  state TEXT NOT NULL DEFAULT 'running', revision INTEGER NOT NULL DEFAULT 0
64383
64605
  )`);
64384
64606
  await tx.exec(poolUsageSchema(journal.runs));
64607
+ if (version === 1)
64608
+ await pool.migrateFromVersion1(tx);
64385
64609
  });
64386
64610
  return pool;
64387
64611
  }
64612
+ async migrateFromVersion1(tx) {
64613
+ await tx.run(`DELETE FROM ${this.operations} WHERE substr(request, 1, 1) = '['`);
64614
+ for (let after = 0;; ) {
64615
+ const rows = await tx.all(`SELECT rowid, request FROM ${this.operations} WHERE rowid > ? ORDER BY rowid LIMIT 256`, after);
64616
+ if (rows.length === 0)
64617
+ break;
64618
+ for (const row of rows)
64619
+ await tx.run(`UPDATE ${this.operations} SET request = ? WHERE rowid = ?`, requestIdentity(String(row.request)), row.rowid);
64620
+ after = Number(rows.at(-1).rowid);
64621
+ }
64622
+ await tx.run(`UPDATE ${this.meta} SET version = ?`, POOL_SCHEMA_VERSION);
64623
+ }
64388
64624
  async authority(tx, lease, timeout) {
64389
64625
  const row = await tx.get(`SELECT l.*, r.status, p.timeout, ${STORE_NOW2} AS now FROM ${this.journal.leases} l JOIN ${this.journal.runs} r ON r.run_id = l.run_id LEFT JOIN ${this.roots} p ON p.root = l.run_id WHERE l.run_id = ?`, lease.runId);
64390
64626
  if (row?.timeout === null && timeout === undefined)
@@ -64525,31 +64761,18 @@ class SqlPoolQueue {
64525
64761
  throw new Error(STALE_WORK_LEASE);
64526
64762
  return row;
64527
64763
  }
64528
- async renew(lease, token, operationId, durationMs) {
64764
+ async renew(lease, token, durationMs) {
64529
64765
  positive(durationMs);
64530
64766
  lease = snapshotLease(lease);
64531
64767
  token = { ...token };
64532
- validateText("key", operationId);
64533
- const request = JSON.stringify([token.epoch, token.generation, token.worker, durationMs]);
64534
64768
  return this.transaction(async (tx) => {
64535
64769
  const { now } = await this.authority(tx, lease);
64536
64770
  const row = await this.current(tx, lease, token);
64537
64771
  if (row.epoch !== lease.epoch || row.expires_at < now)
64538
64772
  throw new Error(STALE_WORK_LEASE);
64539
- const previous = await tx.get(`SELECT * FROM ${this.operations} WHERE work = ? AND generation = ? AND operation = ?`, token.id, token.generation, operationId);
64540
- if (previous) {
64541
- if (previous.request !== request)
64542
- throw new Error("Pool operation identity conflict");
64543
- return JSON.parse(previous.receipt);
64544
- }
64545
- const receipt = {
64546
- ...fromRow(row),
64547
- expiresAt: Math.max(Number(row.expires_at), expiry(now, durationMs)),
64548
- renewedAt: now
64549
- };
64550
- await tx.run(`UPDATE ${this.work} SET expires_at = ?, renewed_at = ? WHERE id = ?`, receipt.expiresAt, now, row.id);
64551
- await tx.run(`INSERT INTO ${this.operations} VALUES (?, ?, ?, ?, ?)`, row.id, token.generation, operationId, request, JSON.stringify(receipt));
64552
- return receipt;
64773
+ const expiresAt = Math.max(Number(row.expires_at), expiry(now, durationMs));
64774
+ await tx.run(`UPDATE ${this.work} SET expires_at = ?, renewed_at = ? WHERE id = ?`, expiresAt, now, row.id);
64775
+ return { ...fromRow(row), expiresAt, renewedAt: now };
64553
64776
  });
64554
64777
  }
64555
64778
  async commit(lease, token, operationId, mutation) {
@@ -64558,6 +64781,7 @@ class SqlPoolQueue {
64558
64781
  validateText("key", operationId);
64559
64782
  assertWorkflowSerializableObject(mutation, "Pool mutation");
64560
64783
  const request = serializeJson({ epoch: token.epoch, worker: token.worker, mutation });
64784
+ const identity = requestIdentity(request);
64561
64785
  mutation = JSON.parse(request).mutation;
64562
64786
  return this.transaction(async (tx) => {
64563
64787
  const { now, status } = await this.authority(tx, lease);
@@ -64566,7 +64790,7 @@ class SqlPoolQueue {
64566
64790
  throw new Error(STALE_WORK_LEASE);
64567
64791
  const previous = await tx.get(`SELECT * FROM ${this.operations} WHERE work = ? AND generation = ? AND operation = ?`, token.id, token.generation, operationId);
64568
64792
  if (previous) {
64569
- if (previous.request !== request)
64793
+ if (previous.request !== identity)
64570
64794
  throw new Error("Pool operation identity conflict");
64571
64795
  return JSON.parse(previous.receipt);
64572
64796
  }
@@ -64574,34 +64798,41 @@ class SqlPoolQueue {
64574
64798
  throw new Error(STALE_WORK_LEASE);
64575
64799
  if (!await this.admitsAttemptWrites(tx, lease, status))
64576
64800
  throw new Error("Pool root is not running");
64577
- const checkpoint = mutation.checkpoint;
64578
- await validatePoolMutation(tx, this.journal, lease.runId, fromRow(row), mutation);
64579
- if (checkpoint.completedAt > Math.max(now, Date.now()) + MAX_WORKER_CLOCK_SKEW_MS)
64580
- throw new Error("Pool checkpoint timestamp is in the future");
64581
- assertWorkflowSerializableObject(checkpoint, "Pool checkpoint");
64582
- const key = JSON.stringify([RUN_EVENT_TYPES.checkpointRecorded, checkpoint.kind, checkpoint.checkpointId]);
64583
- const payload = serializeJson(checkpoint);
64584
- const existing = await tx.get(`SELECT seq, at, payload FROM ${this.journal.events} WHERE run_id = ? AND key = ?`, lease.runId, key);
64585
- if (existing && existing.payload !== payload)
64586
- throw new Error("Pool checkpoint identity conflict");
64587
- if (existing && mutation.kind !== "checkpoint")
64588
- throw new Error("Pool session checkpoint identity conflict");
64589
- const lastSeq = mutation.kind === "checkpoint" ? undefined : await applyPoolSession(tx, `${this.journal.runs}_pool_`, row.id, mutation);
64590
- const latest = await tx.get(`SELECT COALESCE(MAX(seq), 0) AS seq FROM ${this.journal.events} WHERE run_id = ?`, lease.runId);
64591
- const receipt = existing ? { seq: Number(existing.seq), at: Number(existing.at) } : {
64592
- seq: journalCounter(lease.runId, "seq", Number(latest.seq) + 1, "overflow"),
64593
- at: now,
64594
- ...lastSeq === undefined ? {} : { lastSeq }
64595
- };
64596
- if (!existing)
64597
- await tx.run(`INSERT INTO ${this.journal.events} VALUES (?, ?, ?, ?, ?, ?, ?)`, lease.runId, receipt.seq, lease.epoch, key, RUN_EVENT_TYPES.checkpointRecorded, payload, now);
64598
- const billed = poolCheckpointUsage(checkpoint);
64599
- if (billed && !existing)
64600
- await upsertPoolUsage(tx, this.journal.runs, row.id, token.generation, billed.key, billed.usage);
64601
- await tx.run(`INSERT INTO ${this.operations} VALUES (?, ?, ?, ?, ?)`, token.id, token.generation, operationId, request, JSON.stringify(receipt));
64801
+ const receipt = mutation.kind === "session-append" ? await this.appendSession(tx, row, mutation, now) : await this.recordCheckpoint(tx, lease, token, row, mutation, now);
64802
+ await tx.run(`INSERT INTO ${this.operations} VALUES (?, ?, ?, ?, ?)`, token.id, token.generation, operationId, identity, JSON.stringify(receipt));
64602
64803
  return receipt;
64603
64804
  });
64604
64805
  }
64806
+ async appendSession(tx, row, mutation, now) {
64807
+ return { at: now, lastSeq: await applyPoolSession(tx, `${this.journal.runs}_pool_`, row.id, mutation) };
64808
+ }
64809
+ async recordCheckpoint(tx, lease, token, row, mutation, now) {
64810
+ const checkpoint = mutation.checkpoint;
64811
+ await validatePoolMutation(tx, this.journal, lease.runId, fromRow(row), mutation);
64812
+ if (checkpoint.completedAt > Math.max(now, Date.now()) + MAX_WORKER_CLOCK_SKEW_MS)
64813
+ throw new Error("Pool checkpoint timestamp is in the future");
64814
+ assertWorkflowSerializableObject(checkpoint, "Pool checkpoint");
64815
+ const key = JSON.stringify([RUN_EVENT_TYPES.checkpointRecorded, checkpoint.kind, checkpoint.checkpointId]);
64816
+ const payload = serializeJson(checkpoint);
64817
+ const existing = await tx.get(`SELECT seq, at, payload FROM ${this.journal.events} WHERE run_id = ? AND key = ?`, lease.runId, key);
64818
+ if (existing && existing.payload !== payload)
64819
+ throw new Error("Pool checkpoint identity conflict");
64820
+ if (existing && mutation.kind !== "checkpoint")
64821
+ throw new Error("Pool session checkpoint identity conflict");
64822
+ const lastSeq = mutation.kind === "checkpoint" ? undefined : await applyPoolSession(tx, `${this.journal.runs}_pool_`, row.id, mutation);
64823
+ const latest = await tx.get(`SELECT COALESCE(MAX(seq), 0) AS seq FROM ${this.journal.events} WHERE run_id = ?`, lease.runId);
64824
+ const receipt = existing ? { seq: Number(existing.seq), at: Number(existing.at) } : {
64825
+ seq: journalCounter(lease.runId, "seq", Number(latest.seq) + 1, "overflow"),
64826
+ at: now,
64827
+ ...lastSeq === undefined ? {} : { lastSeq }
64828
+ };
64829
+ if (!existing)
64830
+ await tx.run(`INSERT INTO ${this.journal.events} VALUES (?, ?, ?, ?, ?, ?, ?)`, lease.runId, receipt.seq, lease.epoch, key, RUN_EVENT_TYPES.checkpointRecorded, payload, now);
64831
+ const billed = poolCheckpointUsage(checkpoint);
64832
+ if (billed && !existing)
64833
+ await upsertPoolUsage(tx, this.journal.runs, row.id, token.generation, billed.key, billed.usage);
64834
+ return receipt;
64835
+ }
64605
64836
  async complete(lease, token, result) {
64606
64837
  lease = snapshotLease(lease);
64607
64838
  token = { ...token };
@@ -64688,7 +64919,7 @@ function openSqlPoolQueue(journal) {
64688
64919
  return Promise.reject(new Error("Worker pool requires a SQL run journal"));
64689
64920
  return SqlPoolQueue.open(storage);
64690
64921
  }
64691
- var STORE_NOW2 = "CAST(unixepoch('subsec') * 1000 AS INTEGER)", MAX_WORKER_CLOCK_SKEW_MS = 1000, STALE_WORK_LEASE = "Stale work lease";
64922
+ var STORE_NOW2 = "CAST(unixepoch('subsec') * 1000 AS INTEGER)", MAX_WORKER_CLOCK_SKEW_MS = 1000, POOL_SCHEMA_VERSION = 2, STALE_WORK_LEASE = "Stale work lease";
64692
64923
  var init_sql_pool_queue = __esm(() => {
64693
64924
  init_serializable();
64694
64925
  init_boundary_topology();
@@ -64700,6 +64931,15 @@ var init_sql_pool_queue = __esm(() => {
64700
64931
  init_values();
64701
64932
  });
64702
64933
 
64934
+ // dist/builtin/workflows/src/workers/coordinator-protocol.ts
64935
+ var POOL_COORDINATOR_ROLE = "pool-coordinator", POOL_SUMMARY_INTERVAL_MS = 250;
64936
+
64937
+ // dist/builtin/workflows/src/workers/host-options.ts
64938
+ var hostWorkerBindings;
64939
+ var init_host_options = __esm(() => {
64940
+ hostWorkerBindings = Symbol("host-worker-bindings");
64941
+ });
64942
+
64703
64943
  // dist/builtin/workflows/src/durable/journal/sql-pool-usage.ts
64704
64944
  function validate(report) {
64705
64945
  validateText("key", report.key);
@@ -64770,10 +65010,844 @@ var init_sql_pool_usage = __esm(() => {
64770
65010
  init_values();
64771
65011
  });
64772
65012
 
64773
- // dist/builtin/workflows/src/workers/host-options.ts
64774
- var hostWorkerBindings;
64775
- var init_host_options = __esm(() => {
64776
- hostWorkerBindings = Symbol("host-worker-bindings");
65013
+ // dist/builtin/workflows/src/workers/pool-coordinator.ts
65014
+ import { fork } from "node:child_process";
65015
+ import { randomUUID as randomUUID17 } from "node:crypto";
65016
+ import { createWriteStream } from "node:fs";
65017
+ import { mkdir as mkdir6, rm as rm3, rmdir } from "node:fs/promises";
65018
+ import { dirname as dirname14, join as join29, resolve as resolve13 } from "node:path";
65019
+ import { setTimeout as delay2 } from "node:timers/promises";
65020
+ function workerLogDirectory(journalPath, root) {
65021
+ return join29(dirname14(resolve13(journalPath)), "worker-logs", encodeURIComponent(root).replaceAll(".", "%2E"));
65022
+ }
65023
+ function token(work) {
65024
+ return {
65025
+ id: work.id,
65026
+ generation: work.generation,
65027
+ worker: work.worker,
65028
+ epoch: work.epoch,
65029
+ renewedAt: work.renewedAt,
65030
+ expiresAt: work.expiresAt
65031
+ };
65032
+ }
65033
+ function outside(runId) {
65034
+ return new Error(`Pool stage publication is outside the work subtree: ${runId}`);
65035
+ }
65036
+ function isLiveStage(stage) {
65037
+ return (stage.sessionId !== undefined || stage.sessionFile !== undefined) && stage.status !== "completed" && stage.status !== "failed" && stage.status !== "skipped";
65038
+ }
65039
+ function projectRootRun(run) {
65040
+ return {
65041
+ id: run.id,
65042
+ name: run.name,
65043
+ status: run.status,
65044
+ startedAt: run.startedAt,
65045
+ stages: run.stages.map((stage) => ({
65046
+ id: stage.id,
65047
+ name: stage.name,
65048
+ status: stage.status,
65049
+ parentIds: [],
65050
+ toolEvents: [],
65051
+ ...stage.replayKey === undefined ? {} : { replayKey: stage.replayKey },
65052
+ ...stage.intercomGroup === undefined ? {} : { intercomGroup: stage.intercomGroup },
65053
+ ...stage.workflowChildRun === undefined ? {} : { workflowChildRun: stage.workflowChildRun },
65054
+ ...stage.workflowChild === undefined ? {} : { workflowChild: { ...stage.workflowChild, outputs: {} } }
65055
+ }))
65056
+ };
65057
+ }
65058
+
65059
+ class PooledRoutes {
65060
+ root;
65061
+ owner;
65062
+ mirror = createStore();
65063
+ owners = new Map;
65064
+ attempts = new Map;
65065
+ activated = false;
65066
+ bridge;
65067
+ constructor(root, owner) {
65068
+ this.root = root;
65069
+ this.owner = owner;
65070
+ }
65071
+ static async open(root, cwd, rootRun, forward) {
65072
+ await Promise.resolve().then(() => init_workflow_route_owner());
65073
+ const owner = createWorkflowRouteOwner(cwd);
65074
+ const routes = new PooledRoutes(root, owner);
65075
+ if (rootRun !== undefined)
65076
+ routes.updateRoot(rootRun);
65077
+ routes.bridge = registerPendingStageIntercomBridge({
65078
+ events: {
65079
+ emit: (event, payload) => owner.events.emit(event, payload),
65080
+ on: (event, listener) => event === PENDING_STAGE_MESSAGE_EVENT2 || event === STICKY_LIVE_DELIVERED_EVENT2 ? undefined : owner.events.on(event, listener)
65081
+ }
65082
+ }, routes.mirror, (runId) => routes.activated && routes.owners.has(runId));
65083
+ for (const [event, kind] of [
65084
+ [PENDING_STAGE_MESSAGE_EVENT2, "pending-stage-message"],
65085
+ [STICKY_LIVE_DELIVERED_EVENT2, "sticky-live-delivered"]
65086
+ ])
65087
+ owner.events.on(event, (payload) => {
65088
+ const claimed = payload;
65089
+ if (claimed.handled !== false)
65090
+ return;
65091
+ claimed.handled = true;
65092
+ const { handled: _handled, completion: _completion, ...forwarded } = claimed;
65093
+ claimed.completion = forward({ kind, payload: forwarded });
65094
+ });
65095
+ return routes;
65096
+ }
65097
+ updateRoot(rootRun) {
65098
+ this.mirror.recordRunStart({ ...rootRun, inputs: {}, toolNodes: [], rootRunId: this.root });
65099
+ }
65100
+ accept(entries) {
65101
+ this.mirror.batch(() => {
65102
+ for (const { work, runs } of entries) {
65103
+ let attempt = this.attempts.get(work.id);
65104
+ if (attempt !== undefined && work.generation < attempt.generation)
65105
+ continue;
65106
+ if (attempt === undefined || attempt.generation !== work.generation) {
65107
+ if (attempt !== undefined)
65108
+ this.release(attempt);
65109
+ attempt = { generation: work.generation, runs: attempt?.runs ?? new Set, fenced: false };
65110
+ this.attempts.set(work.id, attempt);
65111
+ }
65112
+ if (attempt.fenced)
65113
+ continue;
65114
+ for (const run of runs) {
65115
+ attempt.runs.add(run.id);
65116
+ this.owners.set(run.id, work.id);
65117
+ const { pendingStageMessages: _messages, ...published } = run;
65118
+ this.mirror.recordRunStart({ ...published, rootRunId: this.root });
65119
+ if (run.stages.some((stage) => stage.pendingStageDeliveryAvailable === true))
65120
+ this.activated = true;
65121
+ }
65122
+ }
65123
+ });
65124
+ }
65125
+ fence(work) {
65126
+ const attempt = this.attempts.get(work.id);
65127
+ if (attempt === undefined || attempt.generation > work.generation)
65128
+ return;
65129
+ if (attempt.generation < work.generation) {
65130
+ this.release(attempt);
65131
+ this.attempts.set(work.id, { generation: work.generation, runs: attempt.runs, fenced: true });
65132
+ return;
65133
+ }
65134
+ this.release(attempt);
65135
+ attempt.fenced = true;
65136
+ }
65137
+ async route(work, runId) {
65138
+ this.requireOwned(work, runId);
65139
+ const readiness = workflowPendingStageRouteReady(this.mirror, runId);
65140
+ await readiness?.completion.catch(() => {});
65141
+ readiness?.assertCurrent();
65142
+ const waiting = new AbortController;
65143
+ await Promise.race([
65144
+ this.owner.registered(runId),
65145
+ delay2(ROUTE_REGISTRATION_WAIT_MS, undefined, { signal: waiting.signal, ref: false }).catch(() => {})
65146
+ ]);
65147
+ waiting.abort();
65148
+ this.requireOwned(work, runId);
65149
+ return workflowPendingStageRouteCapability(this.mirror, runId);
65150
+ }
65151
+ async undeliverable(payload) {
65152
+ const event = { ...payload, handled: false };
65153
+ this.owner.events.emit(PENDING_STAGE_UNDELIVERABLE_EVENT2, event);
65154
+ return event.handled && await event.completion === true;
65155
+ }
65156
+ async close() {
65157
+ const bridge = this.bridge;
65158
+ if (bridge === undefined)
65159
+ return;
65160
+ this.bridge = undefined;
65161
+ try {
65162
+ await bridge.flush();
65163
+ } finally {
65164
+ bridge.refresh();
65165
+ bridge();
65166
+ await this.owner.closeGroup(workflowInvocationIntercomGroup(this.root));
65167
+ }
65168
+ }
65169
+ requireOwned(work, runId) {
65170
+ const attempt = this.attempts.get(work.id);
65171
+ if (attempt === undefined || attempt.generation !== work.generation || attempt.fenced)
65172
+ throw new Error(STALE_WORK_LEASE);
65173
+ if (!attempt.runs.has(runId))
65174
+ throw outside(runId);
65175
+ }
65176
+ release(attempt) {
65177
+ this.mirror.batch(() => {
65178
+ for (const id of attempt.runs) {
65179
+ rotateWorkflowPendingStageRouteCapability(this.mirror, id);
65180
+ const run = this.mirror.runs().find((candidate) => candidate.id === id);
65181
+ if (run?.stages.some(isLiveStage) !== true)
65182
+ continue;
65183
+ this.mirror.recordRunStart({
65184
+ ...run,
65185
+ stages: run.stages.map((stage) => {
65186
+ if (!isLiveStage(stage))
65187
+ return stage;
65188
+ const { sessionId: _session, sessionFile: _file, ...rest } = stage;
65189
+ return { ...rest, attachable: false, attached: false };
65190
+ })
65191
+ });
65192
+ }
65193
+ });
65194
+ }
65195
+ }
65196
+
65197
+ class PoolCoordinator {
65198
+ port;
65199
+ children = new Set;
65200
+ incarnations = new Map;
65201
+ readyWorkers = new Set;
65202
+ logs = new Map;
65203
+ cleanups = new Set;
65204
+ controlAcks = new Map;
65205
+ lastHeard = new Map;
65206
+ pendingActivity = new Map;
65207
+ batches = new Map;
65208
+ publicationFailures = new Map;
65209
+ requests = new Map;
65210
+ deliveryRequests = new Map;
65211
+ inflight = new Set;
65212
+ lastTick = performance.now();
65213
+ stalledMs = 0;
65214
+ ticker;
65215
+ recoveryTimer;
65216
+ flushTimer;
65217
+ lastFlushAt = 0;
65218
+ batchSequence = 0;
65219
+ requestSequence = 0;
65220
+ usageChanged = false;
65221
+ budgetBlocked = false;
65222
+ desiredControl;
65223
+ root;
65224
+ lease;
65225
+ options;
65226
+ journal;
65227
+ queue;
65228
+ usageLedger;
65229
+ usageTotal;
65230
+ admission;
65231
+ routes;
65232
+ publishing = false;
65233
+ sources = [];
65234
+ logDirectory;
65235
+ starting;
65236
+ recovering;
65237
+ nextRecoveryAt = 0;
65238
+ cleanupFailure;
65239
+ closed = false;
65240
+ closing;
65241
+ replacements = 0;
65242
+ constructor(port) {
65243
+ this.port = port;
65244
+ port.onMessage((message) => this.receive(message));
65245
+ this.ticker = setInterval(() => this.observeResponsiveness(), TICK_INTERVAL_MS).unref();
65246
+ }
65247
+ async shutdown() {
65248
+ await this.closing?.catch(() => {});
65249
+ while (this.inflight.size > 0)
65250
+ await Promise.allSettled([...this.inflight]);
65251
+ await this.journal?.close().catch(() => {});
65252
+ }
65253
+ receive(message) {
65254
+ if (message.kind === "request-reply") {
65255
+ const pending = this.requests.get(message.id);
65256
+ this.requests.delete(message.id);
65257
+ if (message.error !== undefined)
65258
+ pending?.reject(new Error(message.error));
65259
+ else
65260
+ pending?.resolve(message.value);
65261
+ return;
65262
+ }
65263
+ if ("id" in message) {
65264
+ const command = message;
65265
+ const handled = this.execute(command).then((value2) => this.port.send({ kind: "reply", id: command.id, value: value2 }), (error) => this.port.send({ kind: "reply", id: command.id, error: error.message }));
65266
+ this.inflight.add(handled);
65267
+ handled.finally(() => this.inflight.delete(handled));
65268
+ return;
65269
+ }
65270
+ this.notice(message);
65271
+ }
65272
+ notice(notice) {
65273
+ if (notice.kind === "budget")
65274
+ this.budgetBlocked = notice.blocked;
65275
+ else if (notice.kind === "root")
65276
+ this.routes?.updateRoot(notice.rootRun);
65277
+ else {
65278
+ const batch = this.batches.get(notice.batch);
65279
+ this.batches.delete(notice.batch);
65280
+ const rejected = new Map(notice.rejected.map((entry) => [entry.workId, entry]));
65281
+ for (const entry of rejected.values())
65282
+ this.publicationFailures.set(entry.workId, { generation: entry.generation, error: entry.error });
65283
+ this.routes?.accept((batch?.entries ?? []).filter((entry) => rejected.get(entry.work.id)?.generation !== entry.work.generation));
65284
+ batch?.settle();
65285
+ }
65286
+ }
65287
+ async execute(command) {
65288
+ switch (command.kind) {
65289
+ case "prepare":
65290
+ return this.prepare(command.root, command.lease, command.options, command.control);
65291
+ case "control":
65292
+ return this.control(command.state);
65293
+ case "start":
65294
+ this.starting ??= this.start(command.sources, command.cwd, command.publish, command.rootRun);
65295
+ return this.starting;
65296
+ case "enqueue":
65297
+ return this.enqueue(command);
65298
+ case "transport":
65299
+ return this.transport(command.request, command);
65300
+ case "undeliverable":
65301
+ return await this.routes?.undeliverable(command.payload) ?? false;
65302
+ case "close":
65303
+ this.closing ??= this.close();
65304
+ return this.closing;
65305
+ }
65306
+ }
65307
+ async prepare(root, lease, options, control) {
65308
+ if (this.closed)
65309
+ throw new Error("Local worker pool is closed");
65310
+ if (this.root !== undefined && this.root !== root)
65311
+ throw new Error("Local worker pool belongs to another root");
65312
+ this.root = root;
65313
+ this.options = options;
65314
+ this.desiredControl ??= control;
65315
+ this.journal ??= await resolveRunJournal({ journal: { path: options.journalPath } });
65316
+ this.usageLedger ??= await openSqlPoolUsage(this.journal);
65317
+ this.usageTotal = await this.usageLedger.total(root);
65318
+ this.lease = lease;
65319
+ const queue = await openSqlPoolQueue(this.journal);
65320
+ await queue.initialize(lease, COORDINATOR_LEASE_MS);
65321
+ await queue.releasePreviousIncarnations(lease);
65322
+ for (;; ) {
65323
+ const intended = this.desiredControl ?? "running";
65324
+ await queue.setControl(lease, intended);
65325
+ if (intended === (this.desiredControl ?? "running"))
65326
+ break;
65327
+ }
65328
+ this.queue = queue;
65329
+ return this.usageTotal;
65330
+ }
65331
+ async control(state) {
65332
+ this.desiredControl = state;
65333
+ if (!this.queue || !this.lease)
65334
+ return;
65335
+ const control = await this.queue.setControl(this.lease, state);
65336
+ const deadline = Date.now() + CONTROL_ACKNOWLEDGEMENT_TIMEOUT_MS;
65337
+ while ([...this.readyWorkers].some((worker) => (this.controlAcks.get(worker) ?? -1) < control.revision)) {
65338
+ if (this.desiredControl !== state)
65339
+ return;
65340
+ if (Date.now() >= deadline) {
65341
+ if (state === "quit")
65342
+ break;
65343
+ throw new Error("Worker control acknowledgement unresolved");
65344
+ }
65345
+ await delay2(25);
65346
+ }
65347
+ this.flush();
65348
+ }
65349
+ async start(sources, cwd, publish, rootRun) {
65350
+ if (this.closed)
65351
+ throw new Error("Local worker pool is closed");
65352
+ const root = this.root;
65353
+ if (!this.queue || !this.lease || !this.options || root === undefined || this.lease.runId !== root)
65354
+ throw new Error("Local worker pool requires root preparation");
65355
+ this.sources = sources;
65356
+ this.publishing = publish;
65357
+ if (publish) {
65358
+ this.routes = await PooledRoutes.open(root, cwd, rootRun, (request) => this.request(request));
65359
+ if (this.closed) {
65360
+ await this.routes.close();
65361
+ throw new Error("Local worker pool is closed");
65362
+ }
65363
+ }
65364
+ if (this.options.admission)
65365
+ this.admission = await SqlResourceAdmission.open(this.journal, {
65366
+ ...this.options.admission,
65367
+ root,
65368
+ holder: "coordinator"
65369
+ });
65370
+ this.logDirectory = workerLogDirectory(this.options.journalPath, root);
65371
+ await mkdir6(this.logDirectory, { recursive: true }).catch(() => {
65372
+ return;
65373
+ });
65374
+ this.recoveryTimer = setInterval(() => {
65375
+ this.recoverExpired().catch((error) => this.fail(error));
65376
+ }, RECOVERY_INTERVAL_MS).unref();
65377
+ const workers = Array.from({ length: this.options.workers }, () => this.spawn());
65378
+ try {
65379
+ await Promise.any(workers);
65380
+ } catch (error) {
65381
+ throw error.errors[0];
65382
+ }
65383
+ }
65384
+ async enqueue(command) {
65385
+ if (!this.queue || !this.lease || this.root === undefined)
65386
+ throw new Error("Local worker pool requires root preparation");
65387
+ if (command.rootRun !== undefined)
65388
+ this.routes?.updateRoot(command.rootRun);
65389
+ const work = await this.queue.enqueue(this.lease, command.descriptor, command.boundary);
65390
+ const result = await this.queue.result(this.root, work.id);
65391
+ return { work, ...result === undefined ? {} : { result } };
65392
+ }
65393
+ async transport(request, payload) {
65394
+ const target = this.deliveryRequests.get(request);
65395
+ if (target === undefined || !this.incarnations.has(target.slot.worker))
65396
+ throw new Error("Worker exited before delivery acknowledgement");
65397
+ const { slot, workerRequest } = target;
65398
+ await new Promise((resolve, reject) => {
65399
+ const id = randomUUID17();
65400
+ slot.deliveries.set(id, { resolve, reject });
65401
+ slot.child.send({ id: workerRequest, delivery: { id, from: payload.from, message: payload.message } }, (error) => {
65402
+ if (error) {
65403
+ slot.deliveries.delete(id);
65404
+ reject(error);
65405
+ }
65406
+ });
65407
+ });
65408
+ }
65409
+ request(body) {
65410
+ const id = ++this.requestSequence;
65411
+ return new Promise((resolve, reject) => {
65412
+ this.requests.set(id, { resolve, reject });
65413
+ this.port.send({ kind: "request", id, request: body });
65414
+ });
65415
+ }
65416
+ fail(error) {
65417
+ this.port.send({ kind: "failure", error: error.message });
65418
+ }
65419
+ publish(work, runs) {
65420
+ const failure = this.publicationFailures.get(work.id);
65421
+ if (failure?.generation === work.generation)
65422
+ throw new Error(failure.error);
65423
+ if (!this.publishing)
65424
+ return;
65425
+ this.pendingActivity.set(work.id, { work: token(work), runs });
65426
+ this.scheduleFlush();
65427
+ }
65428
+ scheduleFlush() {
65429
+ if (this.flushTimer !== undefined || this.closed)
65430
+ return;
65431
+ const wait = Math.max(0, this.lastFlushAt + POOL_SUMMARY_INTERVAL_MS - performance.now());
65432
+ this.flushTimer = setTimeout(() => {
65433
+ this.flushTimer = undefined;
65434
+ if (this.batches.size > 0) {
65435
+ this.lastFlushAt = performance.now();
65436
+ this.scheduleFlush();
65437
+ return;
65438
+ }
65439
+ this.flush();
65440
+ }, wait);
65441
+ }
65442
+ flush() {
65443
+ clearTimeout(this.flushTimer);
65444
+ this.flushTimer = undefined;
65445
+ if (this.pendingActivity.size === 0 && !this.usageChanged) {
65446
+ const last = [...this.batches.values()].at(-1);
65447
+ return last === undefined ? Promise.resolve() : new Promise((resolve) => this.chainSettle(last, resolve));
65448
+ }
65449
+ this.lastFlushAt = performance.now();
65450
+ const batch = ++this.batchSequence;
65451
+ const entries = [...this.pendingActivity.values()];
65452
+ this.pendingActivity.clear();
65453
+ const usage = this.usageChanged ? this.usageTotal : undefined;
65454
+ this.usageChanged = false;
65455
+ const settled = new Promise((resolve) => this.batches.set(batch, { entries, settle: resolve }));
65456
+ this.port.send({ kind: "activity", batch, entries, ...usage === undefined ? {} : { usage } });
65457
+ return settled;
65458
+ }
65459
+ chainSettle(batch, resolve) {
65460
+ const settle = batch.settle;
65461
+ batch.settle = () => {
65462
+ settle();
65463
+ resolve();
65464
+ };
65465
+ }
65466
+ noteUsage(usage) {
65467
+ this.usageTotal = usage;
65468
+ this.usageChanged = true;
65469
+ this.scheduleFlush();
65470
+ }
65471
+ async sendResult(workId) {
65472
+ const result = await this.queue.result(this.root, workId);
65473
+ if (result === undefined)
65474
+ return false;
65475
+ this.flush();
65476
+ this.port.send({
65477
+ kind: "result",
65478
+ workId,
65479
+ result,
65480
+ ...this.usageTotal === undefined ? {} : { usage: this.usageTotal }
65481
+ });
65482
+ return true;
65483
+ }
65484
+ fence(work) {
65485
+ this.pendingActivity.delete(work.id);
65486
+ this.routes?.fence(work);
65487
+ this.port.send({ kind: "fence", work: { id: work.id, generation: work.generation } });
65488
+ }
65489
+ spawn() {
65490
+ if (this.closed)
65491
+ return Promise.reject(new Error("Local worker pool is closed"));
65492
+ const root = this.root;
65493
+ const options = this.options;
65494
+ const child = fork(options.workerEntry ?? process.argv[1], [
65495
+ JSON.stringify({
65496
+ journalPath: options.journalPath,
65497
+ root,
65498
+ definitions: this.sources,
65499
+ admission: options.admission
65500
+ })
65501
+ ], { stdio: ["ignore", "ignore", "pipe", "ipc"], execArgv: [] });
65502
+ const disconnected = new Promise((resolve) => child.once("disconnect", () => resolve()));
65503
+ this.children.add(child);
65504
+ if (child.pid !== undefined)
65505
+ this.port.send({ kind: "spawned", pid: child.pid });
65506
+ const worker = randomUUID17();
65507
+ this.incarnations.set(worker, child);
65508
+ const slot = { worker, child, deliveries: new Map };
65509
+ let ready = false;
65510
+ let serial = Promise.resolve();
65511
+ const logPath = join29(this.logDirectory, `${worker}.log`);
65512
+ this.logs.set(child, logPath);
65513
+ let logFailure;
65514
+ const log = createWriteStream(logPath, { flags: "a" });
65515
+ log.on("error", (error) => {
65516
+ logFailure = error.message;
65517
+ });
65518
+ let stderr = "";
65519
+ child.stderr?.on("data", (chunk) => {
65520
+ stderr = (stderr + chunk.toString()).slice(-STDERR_TAIL_CHARACTERS);
65521
+ });
65522
+ child.stderr?.on("error", (error) => {
65523
+ logFailure ??= error.message;
65524
+ log.end();
65525
+ });
65526
+ child.stderr?.pipe(log);
65527
+ const diagnosed = (summary) => new Error(`${summary} (worker stderr log: ${logFailure === undefined ? logPath : `unavailable, ${logFailure}`})${stderr ? `: ${stderr}` : ""}`);
65528
+ const startupTimeoutMs = options.startupTimeoutMs;
65529
+ let timedOut = false;
65530
+ let terminated = false;
65531
+ const terminate = () => {
65532
+ if (terminated)
65533
+ return;
65534
+ terminated = true;
65535
+ child.kill("SIGKILL");
65536
+ };
65537
+ const replied = (error) => {
65538
+ if (error)
65539
+ terminate();
65540
+ };
65541
+ return new Promise((resolve, reject) => {
65542
+ const startup = setTimeout(() => {
65543
+ timedOut = true;
65544
+ terminate();
65545
+ }, startupTimeoutMs);
65546
+ child.on("error", (error) => {
65547
+ clearTimeout(startup);
65548
+ reject(error);
65549
+ terminate();
65550
+ });
65551
+ const closed = new Promise((resolve) => child.once("close", () => resolve()));
65552
+ child.once("exit", () => {
65553
+ Promise.race([closed, delay2(STDERR_DRAIN_GRACE_MS)]).then(() => reject(diagnosed(timedOut ? `Local worker startup timed out after ${startupTimeoutMs} ms` : "Local worker exited before readiness")));
65554
+ this.children.delete(child);
65555
+ this.logs.delete(child);
65556
+ if (child.pid !== undefined)
65557
+ this.port.send({ kind: "exited", pid: child.pid });
65558
+ clearTimeout(startup);
65559
+ this.incarnations.delete(worker);
65560
+ this.readyWorkers.delete(worker);
65561
+ this.controlAcks.delete(worker);
65562
+ this.lastHeard.delete(worker);
65563
+ if (slot.work)
65564
+ this.fence(slot.work);
65565
+ for (const delivery of slot.deliveries.values())
65566
+ delivery.reject(new Error("Worker exited before delivery acknowledgement"));
65567
+ slot.deliveries.clear();
65568
+ const cleanup = disconnected.then(() => serial).then(async () => {
65569
+ try {
65570
+ if (await this.admission?.releaseStoppedSessions(worker))
65571
+ throw new Error("Local worker process cleanup unconfirmed; permits quarantined");
65572
+ } catch (error) {
65573
+ this.cleanupFailure = error instanceof Error ? error : new Error(String(error));
65574
+ throw this.cleanupFailure;
65575
+ }
65576
+ }).then(async () => {
65577
+ if (this.closed || !ready || this.desiredControl === "quit")
65578
+ return;
65579
+ let work = slot.work;
65580
+ if (work && await this.sendResult(work.id)) {
65581
+ work = undefined;
65582
+ slot.work = undefined;
65583
+ this.replacements = 0;
65584
+ }
65585
+ if (this.closed)
65586
+ return;
65587
+ if (work)
65588
+ await this.queue.releaseExited(this.lease, work);
65589
+ const exhausted = work ? work.generation >= 3 : ++this.replacements > 3;
65590
+ if (exhausted)
65591
+ this.fail(diagnosed("Local worker retry limit"));
65592
+ else
65593
+ await this.spawn();
65594
+ }).catch((error) => {
65595
+ this.fail(error);
65596
+ });
65597
+ this.cleanups.add(cleanup);
65598
+ cleanup.finally(() => this.cleanups.delete(cleanup));
65599
+ });
65600
+ child.on("message", (message) => {
65601
+ this.observeResponsiveness();
65602
+ this.lastHeard.set(worker, { at: performance.now(), stalled: this.stalledMs });
65603
+ if (message.kind === "delivery-ack") {
65604
+ const pending = slot.deliveries.get(message.deliveryId);
65605
+ slot.deliveries.delete(message.deliveryId);
65606
+ if (message.error)
65607
+ pending?.reject(new Error(message.error));
65608
+ else
65609
+ pending?.resolve();
65610
+ return;
65611
+ }
65612
+ let routeReady;
65613
+ const handled = serial.then(async () => {
65614
+ if (message.kind === "ready") {
65615
+ if (terminated)
65616
+ return;
65617
+ ready = true;
65618
+ this.readyWorkers.add(worker);
65619
+ clearTimeout(startup);
65620
+ resolve();
65621
+ return;
65622
+ }
65623
+ await this.starting;
65624
+ const queue = this.queue;
65625
+ const lease = this.lease;
65626
+ if (message.kind === "control")
65627
+ return queue.control(lease);
65628
+ if (message.kind === "controlled") {
65629
+ const control = await queue.control(lease);
65630
+ if (message.revision > control.revision)
65631
+ throw new Error("Unknown control revision");
65632
+ this.controlAcks.set(worker, Math.max(this.controlAcks.get(worker) ?? -1, message.revision));
65633
+ return;
65634
+ }
65635
+ if (message.kind === "claim") {
65636
+ if (slot.work)
65637
+ throw new Error("Worker already holds an attempt");
65638
+ if (this.budgetBlocked)
65639
+ return;
65640
+ await this.recoverExpired();
65641
+ slot.work = await queue.claim(lease, worker, this.sources.map((source) => ({
65642
+ definition: `${source.path}#${source.exportName}`,
65643
+ digest: source.digest
65644
+ })), options.leaseMs);
65645
+ return slot.work;
65646
+ }
65647
+ let work = slot.work;
65648
+ if (!work)
65649
+ throw new Error("Worker has no active attempt");
65650
+ if (message.kind === "renew" || work.expiresAt - Date.now() < options.leaseMs / 2) {
65651
+ work = await queue.extend(lease, work, options.leaseMs);
65652
+ slot.work = work;
65653
+ }
65654
+ if (message.kind === "activity")
65655
+ return this.publish(work, message.runs);
65656
+ if (message.kind === "route") {
65657
+ if (this.routes === undefined)
65658
+ return;
65659
+ await this.flush();
65660
+ this.assertPublished(work);
65661
+ routeReady = this.routes.route(work, message.runId);
65662
+ return;
65663
+ }
65664
+ if (message.kind === "delivery-ready") {
65665
+ if (!this.publishing)
65666
+ return;
65667
+ await this.flush();
65668
+ this.assertPublished(work);
65669
+ return this.request({
65670
+ kind: "ready",
65671
+ work: token(work),
65672
+ runId: message.runId,
65673
+ stageId: message.stageId,
65674
+ stageName: message.stageName,
65675
+ ...message.sessionId === undefined ? {} : { sessionId: message.sessionId }
65676
+ });
65677
+ }
65678
+ if (message.kind === "delivery") {
65679
+ if (!message.recipient)
65680
+ throw new Error("Worker delivery requires a session identity");
65681
+ if (!this.publishing)
65682
+ return;
65683
+ await this.flush();
65684
+ this.assertPublished(work);
65685
+ const id = ++this.requestSequence;
65686
+ this.deliveryRequests.set(id, { slot, workerRequest: message.id });
65687
+ try {
65688
+ return await new Promise((resolve, reject) => {
65689
+ this.requests.set(id, { resolve, reject });
65690
+ this.port.send({
65691
+ kind: "request",
65692
+ id,
65693
+ request: {
65694
+ kind: "deliver",
65695
+ work: token(work),
65696
+ runId: message.runId,
65697
+ stageId: message.stageId,
65698
+ stageName: message.stageName,
65699
+ recipient: message.recipient
65700
+ }
65701
+ });
65702
+ });
65703
+ } finally {
65704
+ this.deliveryRequests.delete(id);
65705
+ }
65706
+ }
65707
+ if (message.kind === "renew")
65708
+ return;
65709
+ if (message.kind === "commit") {
65710
+ const receipt = await queue.commit(lease, work, message.operation, message.mutation);
65711
+ this.noteUsage(await this.usageLedger.total(root));
65712
+ return receipt;
65713
+ }
65714
+ if (message.kind === "usage") {
65715
+ this.noteUsage(await this.usageLedger.report(lease, work, message.reports));
65716
+ return;
65717
+ }
65718
+ if (message.kind === "budget") {
65719
+ await queue.assertCurrent(lease, work);
65720
+ if (this.budgetBlocked)
65721
+ throw new Error("Root workflow budget exhausted");
65722
+ return queue.control(lease);
65723
+ }
65724
+ if (message.kind !== "complete")
65725
+ throw new Error("Invalid worker operation");
65726
+ await queue.complete(lease, work, message.result);
65727
+ this.replacements = 0;
65728
+ slot.work = undefined;
65729
+ await this.sendResult(work.id);
65730
+ });
65731
+ const response = handled.then((value2) => routeReady ?? value2);
65732
+ if (message.kind !== "delivery" && message.kind !== "delivery-ready")
65733
+ serial = handled.then(() => {}, () => {});
65734
+ response.then((value2) => child.send({ id: message.id, value: value2 }, replied), (error) => child.send({ id: message.id, error: error.message }, replied));
65735
+ });
65736
+ });
65737
+ }
65738
+ assertPublished(work) {
65739
+ const failure = this.publicationFailures.get(work.id);
65740
+ if (failure?.generation === work.generation)
65741
+ throw new Error(failure.error);
65742
+ }
65743
+ recoverExpired() {
65744
+ if (this.recovering)
65745
+ return this.recovering;
65746
+ const root = this.root;
65747
+ if (this.closed || root === undefined || !this.queue || performance.now() < this.nextRecoveryAt)
65748
+ return Promise.resolve();
65749
+ const queue = this.queue;
65750
+ const lease = this.lease;
65751
+ const leaseMs = this.options.leaseMs;
65752
+ this.recovering = (async () => {
65753
+ await new Promise((resolve) => setImmediate(resolve));
65754
+ await this.admission?.reconcile();
65755
+ const expiredBefore = Date.now();
65756
+ for (const work of await queue.inspect(root)) {
65757
+ if (work.state !== "leased" || work.expiresAt >= expiredBefore)
65758
+ continue;
65759
+ const child = work.worker === null ? undefined : this.incarnations.get(work.worker);
65760
+ if (child && work.worker !== null && this.isLive(work.worker, child)) {
65761
+ await queue.extend(lease, work, leaseMs).catch((error) => {
65762
+ if (error.message !== STALE_WORK_LEASE)
65763
+ throw error;
65764
+ });
65765
+ continue;
65766
+ }
65767
+ if (child && !await this.stop(child).then(() => true, () => false))
65768
+ continue;
65769
+ await queue.expire(lease, work);
65770
+ }
65771
+ })().finally(() => {
65772
+ this.recovering = undefined;
65773
+ this.nextRecoveryAt = performance.now() + RECOVERY_INTERVAL_MS;
65774
+ });
65775
+ return this.recovering;
65776
+ }
65777
+ observeResponsiveness() {
65778
+ const now = performance.now();
65779
+ const gap = now - this.lastTick;
65780
+ if (gap > STALL_THRESHOLD_MS)
65781
+ this.stalledMs += gap - TICK_INTERVAL_MS;
65782
+ this.lastTick = now;
65783
+ }
65784
+ isLive(worker, child) {
65785
+ this.observeResponsiveness();
65786
+ const heard = this.lastHeard.get(worker);
65787
+ return child.exitCode === null && child.signalCode === null && child.connected && heard !== undefined && performance.now() - heard.at - (this.stalledMs - heard.stalled) < this.options.leaseMs;
65788
+ }
65789
+ stop(child) {
65790
+ if (child.pid === undefined || child.exitCode !== null || child.signalCode !== null)
65791
+ return Promise.resolve();
65792
+ return new Promise((resolve, reject) => {
65793
+ const timeout = setTimeout(() => reject(new Error("Local worker cleanup was not confirmed")), 5000);
65794
+ child.once("exit", () => {
65795
+ clearTimeout(timeout);
65796
+ resolve();
65797
+ });
65798
+ child.kill("SIGKILL");
65799
+ });
65800
+ }
65801
+ async close() {
65802
+ this.closed = true;
65803
+ clearInterval(this.ticker);
65804
+ clearInterval(this.recoveryTimer);
65805
+ let failure;
65806
+ let stopping = [];
65807
+ try {
65808
+ await this.recovering?.catch(() => {});
65809
+ stopping = [...this.children].map((child) => ({ child, log: this.logs.get(child) }));
65810
+ await Promise.all(stopping.map(({ child }) => this.stop(child)));
65811
+ await Promise.all([...this.cleanups]);
65812
+ if (this.cleanupFailure)
65813
+ throw this.cleanupFailure;
65814
+ } catch (error) {
65815
+ failure = error;
65816
+ }
65817
+ if (failure === undefined) {
65818
+ const stopped = stopping.flatMap(({ child, log }) => log !== undefined && (child.exitCode !== null || child.signalCode !== null) ? [log] : []);
65819
+ await Promise.all(stopped.map((log) => rm3(log, { force: true }).catch(() => {})));
65820
+ if (this.logDirectory !== undefined)
65821
+ await rmdir(this.logDirectory).catch(() => {});
65822
+ }
65823
+ this.flush();
65824
+ try {
65825
+ await this.routes?.close();
65826
+ } catch (error) {
65827
+ failure ??= error;
65828
+ }
65829
+ for (const batch of this.batches.values())
65830
+ batch.settle();
65831
+ this.batches.clear();
65832
+ for (const pending of this.requests.values())
65833
+ pending.reject(new Error("Local worker pool is closed"));
65834
+ this.requests.clear();
65835
+ if (failure !== undefined)
65836
+ throw failure;
65837
+ }
65838
+ }
65839
+ var CONTROL_ACKNOWLEDGEMENT_TIMEOUT_MS = 5000, RECOVERY_INTERVAL_MS = 500, TICK_INTERVAL_MS = 100, STALL_THRESHOLD_MS = 500, COORDINATOR_LEASE_MS, STDERR_TAIL_CHARACTERS = 8192, STDERR_DRAIN_GRACE_MS = 1000, ROUTE_REGISTRATION_WAIT_MS = 120000, PENDING_STAGE_MESSAGE_EVENT2 = "atomic:workflow-pending-stage-message", STICKY_LIVE_DELIVERED_EVENT2 = "atomic:workflow-sticky-live-delivered", PENDING_STAGE_UNDELIVERABLE_EVENT2 = "atomic:workflow-pending-stage-undeliverable";
65840
+ var init_pool_coordinator = __esm(() => {
65841
+ init_resolution();
65842
+ init_sql_pool_queue();
65843
+ init_sql_pool_usage();
65844
+ init_pending_stage_intercom();
65845
+ init_intercom_group();
65846
+ init_pending_stage_route_capability();
65847
+ init_pending_stage_route_readiness();
65848
+ init_store();
65849
+ init_resource_admission();
65850
+ COORDINATOR_LEASE_MS = 3 * STAGE_SESSION_HEARTBEAT_INTERVAL_MS;
64777
65851
  });
64778
65852
 
64779
65853
  // dist/builtin/workflows/src/workers/protocol.ts
@@ -65023,21 +66097,26 @@ class PoolStageHandoff {
65023
66097
  store;
65024
66098
  queue;
65025
66099
  lease;
66100
+ claimRun;
65026
66101
  attempts = new Map;
65027
66102
  owners = new Map;
65028
66103
  publications = new Map;
65029
66104
  drains = new Map;
65030
- constructor(backend, store, queue, lease) {
66105
+ constructor(backend, store, queue, lease, claimRun) {
65031
66106
  this.backend = backend;
65032
66107
  this.store = store;
65033
66108
  this.queue = queue;
65034
66109
  this.lease = lease;
66110
+ this.claimRun = claimRun;
65035
66111
  }
65036
66112
  publish(work, runs) {
65037
66113
  const previous = this.publications.get(work.id) ?? Promise.resolve();
65038
66114
  const publication = previous.catch(() => {
65039
66115
  return;
65040
- }).then(() => this.apply(work, runs));
66116
+ }).then(async () => {
66117
+ const commit = await this.prepare(work, runs);
66118
+ this.store.batch(commit);
66119
+ });
65041
66120
  const settled = publication.catch(() => {
65042
66121
  return;
65043
66122
  });
@@ -65048,6 +66127,21 @@ class PoolStageHandoff {
65048
66127
  });
65049
66128
  return publication;
65050
66129
  }
66130
+ async publishBatch(publications) {
66131
+ const prepared = [];
66132
+ for (const { work, runs } of publications)
66133
+ prepared.push(await this.prepare(work, runs).catch((error) => error));
66134
+ return this.store.batch(() => prepared.map((commit) => {
66135
+ if (commit instanceof Error)
66136
+ return commit;
66137
+ try {
66138
+ commit();
66139
+ return;
66140
+ } catch (error) {
66141
+ return error instanceof Error ? error : new Error(String(error));
66142
+ }
66143
+ }));
66144
+ }
65051
66145
  async route(work, runId) {
65052
66146
  this.requireOwned(await this.begin(work), runId);
65053
66147
  const readiness = workflowPendingStageRouteReady(this.store, runId);
@@ -65098,7 +66192,7 @@ class PoolStageHandoff {
65098
66192
  const result = await this.queue.result(root, work.id);
65099
66193
  if (result === undefined)
65100
66194
  return;
65101
- await this.backend.hydrateWorkflow(root);
66195
+ await this.backend.catchUpWorkflow(root);
65102
66196
  const terminals = new Map;
65103
66197
  for (const checkpoint of this.backend.listCheckpoints(root)) {
65104
66198
  const boundary = checkpoint.kind === "stage" ? checkpoint.topology?.boundary : undefined;
@@ -65108,6 +66202,9 @@ class PoolStageHandoff {
65108
66202
  if (child?.runId === boundary.child.runId)
65109
66203
  terminals.set(child.runId, { status: child.status, outputs: child.outputs });
65110
66204
  }
66205
+ this.store.batch(() => this.settle(work, result, terminals));
66206
+ }
66207
+ settle(work, result, terminals) {
65111
66208
  for (const run of this.store.runs()) {
65112
66209
  if (run.endedAt !== undefined || this.owners.get(run.id) !== work.id)
65113
66210
  continue;
@@ -65131,22 +66228,31 @@ class PoolStageHandoff {
65131
66228
  if (current !== undefined && current.generation > work.generation)
65132
66229
  return;
65133
66230
  const attempt = current?.generation === work.generation ? current : this.start(work, current);
65134
- this.release(attempt);
66231
+ this.store.batch(() => this.release(attempt));
65135
66232
  attempt.fenced = true;
65136
66233
  }
65137
- async apply(work, runs) {
66234
+ async prepare(work, runs) {
65138
66235
  const subtree = this.subtree(work, runs);
65139
66236
  const root = this.lease.runId;
65140
- const known = new Map(this.store.runs().map((run) => [run.id, run]));
65141
- for (const run of subtree) {
65142
- const existing = known.get(run.id);
65143
- const owner = this.owners.get(run.id);
65144
- if (owner !== undefined && owner !== work.id || owner === undefined && existing !== undefined && (existing.parentRunId !== run.parentRunId || existing.parentStageId !== run.parentStageId))
65145
- throw outside(run.id);
65146
- }
65147
- if (subtree.some((run) => known.get(run.id) === undefined))
65148
- await this.backend.hydrateWorkflow(root);
66237
+ const owned = (known) => {
66238
+ for (const run of subtree) {
66239
+ const existing = known.get(run.id);
66240
+ const owner = this.owners.get(run.id);
66241
+ if (owner !== undefined && owner !== work.id || owner === undefined && existing !== undefined && (existing.parentRunId !== run.parentRunId || existing.parentStageId !== run.parentStageId))
66242
+ throw outside2(run.id);
66243
+ }
66244
+ };
66245
+ const before = new Map(this.store.runs().map((run) => [run.id, run]));
66246
+ owned(before);
66247
+ if (subtree.some((run) => before.get(run.id) === undefined))
66248
+ await this.backend.catchUpWorkflow(root);
65149
66249
  await this.queue.assertCurrent(this.lease, work);
66250
+ return () => this.commit(work, subtree, owned);
66251
+ }
66252
+ commit(work, subtree, owned) {
66253
+ const root = this.lease.runId;
66254
+ const known = new Map(this.store.runs().map((run) => [run.id, run]));
66255
+ owned(known);
65150
66256
  const attempt = this.enter(work);
65151
66257
  const execution = {
65152
66258
  worker: work.worker ?? "",
@@ -65158,11 +66264,13 @@ class PoolStageHandoff {
65158
66264
  ...run,
65159
66265
  rootRunId: root,
65160
66266
  pendingStageMessages: [
65161
- ...this.store.runs().find((current) => current.id === run.id)?.pendingStageMessages ?? pendingStageMessagesForDurableRun(this.backend, run.id, root)
66267
+ ...known.get(run.id)?.pendingStageMessages ?? pendingStageMessagesForDurableRun(this.backend, run.id, root)
65162
66268
  ],
65163
66269
  stages: run.stages.map((stage) => ({ ...stage, workerExecution: execution }))
65164
66270
  }));
65165
66271
  const changed = published.some((run) => activity(known.get(run.id)) !== activity(run));
66272
+ for (const run of published)
66273
+ this.claimRun?.(run.id);
65166
66274
  for (const run of published) {
65167
66275
  attempt.runs.add(run.id);
65168
66276
  this.owners.set(run.id, work.id);
@@ -65187,7 +66295,7 @@ class PoolStageHandoff {
65187
66295
  const byId = new Map;
65188
66296
  for (const run of runs) {
65189
66297
  if (run.id === root || byId.has(run.id))
65190
- throw outside(run.id);
66298
+ throw outside2(run.id);
65191
66299
  byId.set(run.id, run);
65192
66300
  }
65193
66301
  if (runs.length === 0)
@@ -65200,25 +66308,25 @@ class PoolStageHandoff {
65200
66308
  const top = byId.get(work.childRunId);
65201
66309
  if (top !== undefined) {
65202
66310
  if (top.parentRunId !== root || top.parentStageId !== work.boundaryId)
65203
- throw outside(top.id);
66311
+ throw outside2(top.id);
65204
66312
  const pending = [top];
65205
66313
  for (let run = pending.shift();run !== undefined; run = pending.shift()) {
65206
66314
  if (run.rootRunId !== undefined && run.rootRunId !== root)
65207
- throw outside(run.id);
66315
+ throw outside2(run.id);
65208
66316
  ordered.push(run);
65209
66317
  for (const child of runs) {
65210
66318
  if (child.parentRunId !== run.id)
65211
66319
  continue;
65212
66320
  const parentStage = run.stages.find((stage) => stage.id === child.parentStageId);
65213
66321
  if (authoritativeWorkflowChildRunId(parentStage) !== child.id)
65214
- throw outside(child.id);
66322
+ throw outside2(child.id);
65215
66323
  pending.push(child);
65216
66324
  }
65217
66325
  }
65218
66326
  }
65219
66327
  const stray = runs.find((run) => !ordered.includes(run));
65220
66328
  if (stray !== undefined)
65221
- throw outside(stray.id);
66329
+ throw outside2(stray.id);
65222
66330
  return ordered;
65223
66331
  }
65224
66332
  async begin(work) {
@@ -65273,7 +66381,7 @@ class PoolStageHandoff {
65273
66381
  }
65274
66382
  requireOwned(attempt, runId) {
65275
66383
  if (!attempt.runs.has(runId))
65276
- throw outside(runId);
66384
+ throw outside2(runId);
65277
66385
  }
65278
66386
  delivery(attempt, work, runId, stageId, stageName) {
65279
66387
  const key = JSON.stringify([runId, stageId]);
@@ -65291,7 +66399,7 @@ class PoolStageHandoff {
65291
66399
  return delivery;
65292
66400
  }
65293
66401
  }
65294
- function outside(runId) {
66402
+ function outside2(runId) {
65295
66403
  return new Error(`Pool stage publication is outside the work subtree: ${runId}`);
65296
66404
  }
65297
66405
  function isLive(stage) {
@@ -65316,33 +66424,122 @@ var init_stage_handoff = __esm(() => {
65316
66424
  });
65317
66425
 
65318
66426
  // dist/builtin/workflows/src/workers/local-pool.ts
65319
- import { fork } from "node:child_process";
65320
- import { randomUUID as randomUUID17 } from "node:crypto";
65321
- import { createWriteStream } from "node:fs";
65322
- import { mkdir as mkdir6, rm as rm3, rmdir } from "node:fs/promises";
65323
- import { dirname as dirname14, join as join29, resolve as resolve13 } from "node:path";
65324
- import { setTimeout as delay2 } from "node:timers/promises";
66427
+ import { fork as fork2 } from "node:child_process";
66428
+ import { resolve as resolve14 } from "node:path";
66429
+ import { setTimeout as delay3 } from "node:timers/promises";
65325
66430
  import { fileURLToPath as fileURLToPath6 } from "node:url";
65326
66431
  import { isBunBinary as isBunBinary2 } from "@bastani/atomic";
65327
- function workerLogDirectory(journalPath, root) {
65328
- return join29(dirname14(resolve13(journalPath)), "worker-logs", encodeURIComponent(root).replaceAll(".", "%2E"));
66432
+ function defaultWorkerEntry() {
66433
+ return isBunBinary2 ? INTERNAL_WORKFLOW_WORKER_ARG : fileURLToPath6(new URL("./local-worker-entry.mjs", import.meta.url));
66434
+ }
66435
+ function launchCoordinatorProcess() {
66436
+ const child = fork2(defaultWorkerEntry(), [JSON.stringify({ role: POOL_COORDINATOR_ROLE })], {
66437
+ stdio: ["ignore", "ignore", "pipe", "ipc"],
66438
+ execArgv: []
66439
+ });
66440
+ let stderr = "";
66441
+ child.stderr?.on("data", (chunk) => {
66442
+ stderr = (stderr + chunk.toString()).slice(-COORDINATOR_STDERR_TAIL_CHARACTERS);
66443
+ });
66444
+ const listeners = [];
66445
+ let lostReason;
66446
+ const lose = (reason) => {
66447
+ if (lostReason !== undefined)
66448
+ return;
66449
+ lostReason = reason;
66450
+ for (const listener of listeners)
66451
+ listener(reason);
66452
+ };
66453
+ const exited = new Promise((resolve) => child.once("exit", () => resolve()));
66454
+ child.on("error", lose);
66455
+ child.once("exit", (code, signal) => lose(new Error(`Local worker pool coordinator exited (${signal ?? `code ${code}`})${stderr ? `: ${stderr.trim()}` : ""}`)));
66456
+ return {
66457
+ pid: child.pid,
66458
+ port: {
66459
+ send(message) {
66460
+ if (lostReason !== undefined || !child.connected)
66461
+ return;
66462
+ child.send(message, (error) => {
66463
+ if (error)
66464
+ lose(error);
66465
+ });
66466
+ },
66467
+ onMessage(listener) {
66468
+ child.on("message", (message) => listener(message));
66469
+ },
66470
+ onClose(listener) {
66471
+ if (lostReason !== undefined)
66472
+ listener(lostReason);
66473
+ else
66474
+ listeners.push(listener);
66475
+ },
66476
+ async close() {
66477
+ if (child.connected)
66478
+ child.disconnect();
66479
+ if (child.exitCode !== null || child.signalCode !== null)
66480
+ return;
66481
+ const timedOut = await Promise.race([exited.then(() => false), delay3(COORDINATOR_EXIT_GRACE_MS, true)]);
66482
+ if (timedOut) {
66483
+ child.kill("SIGKILL");
66484
+ await exited;
66485
+ }
66486
+ }
66487
+ }
66488
+ };
66489
+ }
66490
+ function launchCoordinatorInProcess() {
66491
+ const mainListeners = [];
66492
+ const coordinatorListeners = [];
66493
+ const copy = (message) => JSON.parse(JSON.stringify(message));
66494
+ const coordinator = new PoolCoordinator({
66495
+ send(message) {
66496
+ const copied = copy(message);
66497
+ setImmediate(() => {
66498
+ for (const listener of mainListeners)
66499
+ listener(copied);
66500
+ });
66501
+ },
66502
+ onMessage(listener) {
66503
+ coordinatorListeners.push(listener);
66504
+ },
66505
+ onClose() {},
66506
+ async close() {}
66507
+ });
66508
+ return {
66509
+ pid: process.pid,
66510
+ port: {
66511
+ send(message) {
66512
+ const copied = copy(message);
66513
+ setImmediate(() => {
66514
+ for (const listener of coordinatorListeners)
66515
+ listener(copied);
66516
+ });
66517
+ },
66518
+ onMessage(listener) {
66519
+ mainListeners.push(listener);
66520
+ },
66521
+ onClose() {},
66522
+ async close() {
66523
+ coordinator.shutdown();
66524
+ }
66525
+ }
66526
+ };
65329
66527
  }
65330
66528
 
65331
66529
  class LocalWorkerPool {
65332
66530
  backend;
65333
66531
  options;
65334
- children = new Set;
65335
- incarnations = new Map;
65336
- readyWorkers = new Set;
65337
- logs = new Map;
65338
- logDirectory;
65339
- cleanups = new Set;
65340
- recovering;
65341
- nextRecoveryAt = 0;
65342
- controlAcks = new Map;
65343
- lastHeard = new Map;
65344
- lastTick = performance.now();
65345
- rootStalledMs = 0;
66532
+ pids = new Set;
66533
+ coordinator;
66534
+ coordinatorLost;
66535
+ commands = new Map;
66536
+ commandSequence = 0;
66537
+ inbound = Promise.resolve();
66538
+ works = new Map;
66539
+ attempts = new Map;
66540
+ results = new Map;
66541
+ changed = Promise.withResolvers();
66542
+ waiting = 0;
65346
66543
  ticker;
65347
66544
  leaseMs;
65348
66545
  desiredControl;
@@ -65350,18 +66547,17 @@ class LocalWorkerPool {
65350
66547
  store;
65351
66548
  handoff;
65352
66549
  releaseRoutes;
66550
+ routeClaims = new Map;
65353
66551
  sources = [];
65354
66552
  queue;
65355
66553
  lease;
65356
- admission;
65357
- usageLedger;
65358
66554
  usageTotal;
65359
66555
  rootBudget;
66556
+ budgetBlocked = false;
66557
+ rootSignature;
65360
66558
  starting;
65361
- cleanupFailure;
65362
66559
  failure;
65363
66560
  closed = false;
65364
- replacements = 0;
65365
66561
  releaseBinding;
65366
66562
  constructor(backend, options) {
65367
66563
  this.backend = backend;
@@ -65376,10 +66572,12 @@ class LocalWorkerPool {
65376
66572
  this.releaseBinding = bindChildWorkerDispatcher(backend, this, options.rootId);
65377
66573
  if (options.admission && options.rootId === undefined)
65378
66574
  workflowResourceAdmission.set(backend, options.admission);
65379
- this.ticker = setInterval(() => this.observeResponsiveness(), ROOT_TICK_INTERVAL_MS).unref();
65380
66575
  }
65381
66576
  get workerPids() {
65382
- return [...this.children].flatMap((child) => child.pid === undefined ? [] : [child.pid]);
66577
+ return [...this.pids];
66578
+ }
66579
+ get coordinatorPid() {
66580
+ return this.coordinator?.pid;
65383
66581
  }
65384
66582
  usage() {
65385
66583
  return this.usageTotal;
@@ -65387,47 +66585,39 @@ class LocalWorkerPool {
65387
66585
  async prepare(root, store) {
65388
66586
  if (this.closed)
65389
66587
  throw new Error("Local worker pool is closed");
65390
- if (!(this.backend.journal instanceof SqlRunJournal) || resolve13(this.options.journalPath) !== resolve13(this.backend.journal.path) || sqlPoolStorage(this.backend.journal)?.tables.runs !== "wf_runs")
66588
+ if (!(this.backend.journal instanceof SqlRunJournal) || resolve14(this.options.journalPath) !== resolve14(this.backend.journal.path) || sqlPoolStorage(this.backend.journal)?.tables.runs !== "wf_runs")
65391
66589
  throw new Error("Local workers require the same default-prefix SQLite journal");
65392
66590
  if (this.root !== undefined && this.root !== root)
65393
66591
  throw new Error("Local worker pool belongs to another root");
65394
66592
  this.root = root;
65395
66593
  if (store)
65396
66594
  this.store = store;
65397
- this.usageLedger ??= await openSqlPoolUsage(this.backend.journal);
65398
- this.usageTotal = await this.usageLedger.total(root);
65399
66595
  const state = await this.backend.journal.leaseState(root);
65400
66596
  if (!state || state.holderId !== this.backend.holderId)
65401
66597
  throw new Error("Local worker pool requires the root owner");
65402
66598
  this.lease = { runId: root, holderId: state.holderId, epoch: state.epoch };
65403
- const queue = await openSqlPoolQueue(this.backend.journal);
65404
- await queue.initialize(this.lease, COORDINATOR_LEASE_MS);
65405
- await queue.releasePreviousIncarnations(this.lease);
65406
- for (;; ) {
65407
- const intended = this.desiredControl ?? "running";
65408
- await queue.setControl(this.lease, intended);
65409
- if (intended === (this.desiredControl ?? "running"))
65410
- break;
65411
- }
65412
- this.queue = queue;
66599
+ const workerEntry = this.options.workerEntry ?? (isBunBinary2 || this.options.coordinator === "in-process" ? defaultWorkerEntry() : undefined);
66600
+ this.usageTotal = await this.command({
66601
+ kind: "prepare",
66602
+ root,
66603
+ lease: this.lease,
66604
+ options: {
66605
+ journalPath: this.options.journalPath,
66606
+ workers: this.options.workers,
66607
+ leaseMs: this.leaseMs,
66608
+ startupTimeoutMs: this.options.startupTimeoutMs ?? DEFAULT_STARTUP_TIMEOUT_MS,
66609
+ ...this.options.admission === undefined ? {} : { admission: this.options.admission },
66610
+ ...workerEntry === undefined ? {} : { workerEntry }
66611
+ },
66612
+ ...this.desiredControl === undefined ? {} : { control: this.desiredControl }
66613
+ });
66614
+ this.queue ??= await openSqlPoolQueue(this.backend.journal);
65413
66615
  }
65414
66616
  async control(state) {
65415
66617
  this.desiredControl = state;
65416
66618
  try {
65417
- if (!this.queue || !this.lease)
65418
- return;
65419
- const control = await this.queue.setControl(this.lease, state);
65420
- const deadline = Date.now() + CONTROL_ACKNOWLEDGEMENT_TIMEOUT_MS;
65421
- while ([...this.readyWorkers].some((worker) => (this.controlAcks.get(worker) ?? -1) < control.revision)) {
65422
- if (this.desiredControl !== state)
65423
- return;
65424
- if (Date.now() >= deadline) {
65425
- if (state === "quit")
65426
- break;
65427
- throw new Error("Worker control acknowledgement unresolved");
65428
- }
65429
- await delay2(25);
65430
- }
66619
+ if (this.coordinator !== undefined)
66620
+ await this.command({ kind: "control", state });
65431
66621
  } finally {
65432
66622
  if (state === "quit")
65433
66623
  await this.close();
@@ -65473,290 +66663,61 @@ class LocalWorkerPool {
65473
66663
  throw new Error(`Worker definition is not importable: ${binding.exportName}`);
65474
66664
  return source;
65475
66665
  }));
65476
- if (this.store) {
65477
- this.releaseRoutes = await acquireStandaloneIntercom(this.store, root, String(this.backend.getWorkflow(root)?.invocationCwd ?? process.cwd()));
66666
+ const cwd = String(this.backend.getWorkflow(root)?.invocationCwd ?? process.cwd());
66667
+ const store = this.store;
66668
+ if (store) {
66669
+ this.releaseRoutes = await acquireStandaloneIntercom(store, root, cwd);
65478
66670
  if (this.closed) {
65479
66671
  await this.releaseRoutes();
65480
66672
  this.releaseRoutes = undefined;
65481
66673
  throw new Error("Local worker pool is closed");
65482
66674
  }
65483
- this.handoff = new PoolStageHandoff(this.backend, this.store, this.queue, this.lease);
65484
- }
65485
- if (this.options.admission)
65486
- this.admission = await SqlResourceAdmission.open(this.backend.journal, {
65487
- ...this.options.admission,
65488
- root,
65489
- holder: "coordinator"
66675
+ const routeOwner = {
66676
+ notifyUndeliverable: async (payload) => !this.closed && await this.command({ kind: "undeliverable", payload }) === true
66677
+ };
66678
+ this.handoff = new PoolStageHandoff(this.backend, store, this.queueView(), this.lease, (runId) => {
66679
+ this.routeClaims.set(runId, claimExternalRunRoute(store, runId, routeOwner));
65490
66680
  });
65491
- this.logDirectory = workerLogDirectory(this.options.journalPath, root);
65492
- await mkdir6(this.logDirectory, { recursive: true }).catch(() => {
65493
- return;
65494
- });
65495
- const workers = Array.from({ length: this.options.workers }, () => this.spawn(root));
65496
- try {
65497
- await Promise.any(workers);
65498
- } catch (error) {
65499
- throw error.errors[0];
65500
66681
  }
65501
- }
65502
- spawn(root) {
65503
- if (this.closed)
65504
- return Promise.reject(new Error("Local worker pool is closed"));
65505
- const child = fork(isBunBinary2 ? INTERNAL_WORKFLOW_WORKER_ARG : fileURLToPath6(new URL("./local-worker-entry.mjs", import.meta.url)), [
65506
- JSON.stringify({
65507
- journalPath: this.options.journalPath,
65508
- root,
65509
- definitions: this.sources,
65510
- admission: this.options.admission
65511
- })
65512
- ], { stdio: ["ignore", "ignore", "pipe", "ipc"], execArgv: [] });
65513
- const disconnected = new Promise((resolve) => child.once("disconnect", () => resolve()));
65514
- this.children.add(child);
65515
- const worker = randomUUID17();
65516
- this.incarnations.set(worker, child);
65517
- let ready = false;
65518
- let serial = Promise.resolve();
65519
- let work;
65520
- const deliveries = new Map;
65521
- const logPath = join29(this.logDirectory, `${worker}.log`);
65522
- this.logs.set(child, logPath);
65523
- let logFailure;
65524
- const log = createWriteStream(logPath, { flags: "a" });
65525
- log.on("error", (error) => {
65526
- logFailure = error.message;
65527
- });
65528
- let stderr = "";
65529
- child.stderr?.on("data", (chunk) => {
65530
- stderr = (stderr + chunk.toString()).slice(-STDERR_TAIL_CHARACTERS);
65531
- });
65532
- child.stderr?.on("error", (error) => {
65533
- logFailure ??= error.message;
65534
- log.end();
65535
- });
65536
- child.stderr?.pipe(log);
65537
- const diagnosed = (summary) => new Error(`${summary} (worker stderr log: ${logFailure === undefined ? logPath : `unavailable, ${logFailure}`})${stderr ? `: ${stderr}` : ""}`);
65538
- const startupTimeoutMs = this.options.startupTimeoutMs ?? DEFAULT_STARTUP_TIMEOUT_MS;
65539
- let timedOut = false;
65540
- let terminated = false;
65541
- const terminate = () => {
65542
- if (terminated)
65543
- return;
65544
- terminated = true;
65545
- child.kill("SIGKILL");
65546
- };
65547
- const replied = (error) => {
65548
- if (error)
65549
- terminate();
65550
- };
65551
- return new Promise((resolve, reject) => {
65552
- const startup = setTimeout(() => {
65553
- timedOut = true;
65554
- terminate();
65555
- }, startupTimeoutMs);
65556
- child.on("error", (error) => {
65557
- clearTimeout(startup);
65558
- reject(error);
65559
- terminate();
65560
- });
65561
- const closed = new Promise((resolve) => child.once("close", () => resolve()));
65562
- child.once("exit", () => {
65563
- Promise.race([closed, delay2(STDERR_DRAIN_GRACE_MS)]).then(() => reject(diagnosed(timedOut ? `Local worker startup timed out after ${startupTimeoutMs} ms` : "Local worker exited before readiness")));
65564
- this.children.delete(child);
65565
- this.logs.delete(child);
65566
- clearTimeout(startup);
65567
- this.incarnations.delete(worker);
65568
- this.readyWorkers.delete(worker);
65569
- this.controlAcks.delete(worker);
65570
- this.lastHeard.delete(worker);
65571
- if (work)
65572
- this.handoff?.fence(work);
65573
- for (const delivery of deliveries.values())
65574
- delivery.reject(new Error("Worker exited before delivery acknowledgement"));
65575
- deliveries.clear();
65576
- const cleanup = disconnected.then(() => serial).then(async () => {
65577
- try {
65578
- if (await this.admission?.releaseStoppedSessions(worker))
65579
- throw new Error("Local worker process cleanup unconfirmed; permits quarantined");
65580
- } catch (error) {
65581
- this.cleanupFailure = error instanceof Error ? error : new Error(String(error));
65582
- throw this.cleanupFailure;
65583
- }
65584
- }).then(async () => {
65585
- if (this.closed || !ready || this.desiredControl === "quit")
65586
- return;
65587
- if (work && await this.queue.result(root, work.id)) {
65588
- work = undefined;
65589
- this.replacements = 0;
65590
- }
65591
- if (this.closed)
65592
- return;
65593
- if (work)
65594
- await this.queue.releaseExited(this.lease, work);
65595
- const exhausted = work ? work.generation >= 3 : ++this.replacements > 3;
65596
- if (exhausted)
65597
- this.failure = diagnosed("Local worker retry limit");
65598
- else
65599
- await this.spawn(root);
65600
- }).catch((error) => {
65601
- this.failure = error;
65602
- });
65603
- this.cleanups.add(cleanup);
65604
- cleanup.finally(() => this.cleanups.delete(cleanup));
65605
- });
65606
- child.on("message", (message) => {
65607
- this.observeResponsiveness();
65608
- this.lastHeard.set(worker, { at: performance.now(), rootStalled: this.rootStalledMs });
65609
- if (message.kind === "delivery-ack") {
65610
- const pending = deliveries.get(message.deliveryId);
65611
- deliveries.delete(message.deliveryId);
65612
- if (message.error)
65613
- pending?.reject(new Error(message.error));
65614
- else
65615
- pending?.resolve();
65616
- return;
65617
- }
65618
- const response = serial.then(async () => {
65619
- if (message.kind === "ready") {
65620
- if (terminated)
65621
- return;
65622
- ready = true;
65623
- this.readyWorkers.add(worker);
65624
- clearTimeout(startup);
65625
- resolve();
65626
- return;
65627
- }
65628
- await this.starting;
65629
- const queue = this.queue;
65630
- const lease = this.lease;
65631
- if (message.kind === "control")
65632
- return queue.control(lease);
65633
- if (message.kind === "controlled") {
65634
- const control = await queue.control(lease);
65635
- if (message.revision > control.revision)
65636
- throw new Error("Unknown control revision");
65637
- this.controlAcks.set(worker, Math.max(this.controlAcks.get(worker) ?? -1, message.revision));
65638
- return;
65639
- }
65640
- if (message.kind === "claim") {
65641
- if (work)
65642
- throw new Error("Worker already holds an attempt");
65643
- const decision = this.rootBudget?.checkpoint();
65644
- if (decision?.kind === "exhausted" || decision?.kind === "wrap_up")
65645
- return;
65646
- await this.recoverExpired(root);
65647
- work = await queue.claim(lease, worker, this.sources.map((source) => ({
65648
- definition: `${source.path}#${source.exportName}`,
65649
- digest: source.digest
65650
- })), this.leaseMs);
65651
- return work;
65652
- }
65653
- if (!work)
65654
- throw new Error("Worker has no active attempt");
65655
- if (message.kind === "renew" || work.expiresAt - Date.now() < this.leaseMs / 2)
65656
- work = await queue.extend(lease, work, this.leaseMs);
65657
- if (message.kind === "activity")
65658
- return this.handoff?.publish(work, message.runs);
65659
- if (message.kind === "route")
65660
- return this.handoff?.route(work, message.runId);
65661
- if (message.kind === "delivery-ready")
65662
- return this.handoff?.ready(work, message.runId, message.stageId, message.stageName, message.sessionId);
65663
- if (message.kind === "delivery") {
65664
- if (!message.recipient)
65665
- throw new Error("Worker delivery requires a session identity");
65666
- return this.handoff?.deliver(work, message.runId, message.stageId, message.stageName, message.recipient, (from, payload) => new Promise((resolve, reject) => {
65667
- const id = randomUUID17();
65668
- deliveries.set(id, { resolve, reject });
65669
- child.send({ id: message.id, delivery: { id, from, message: payload } }, (error) => {
65670
- if (error) {
65671
- deliveries.delete(id);
65672
- reject(error);
65673
- }
65674
- });
65675
- }));
65676
- }
65677
- if (message.kind === "renew")
65678
- return;
65679
- if (message.kind === "commit") {
65680
- const receipt = await queue.commit(lease, work, message.operation, message.mutation);
65681
- this.usageTotal = await this.usageLedger.total(root);
65682
- return receipt;
65683
- }
65684
- if (message.kind === "usage") {
65685
- this.usageTotal = await this.usageLedger.report(lease, work, message.reports);
65686
- return;
65687
- }
65688
- if (message.kind === "budget") {
65689
- await queue.assertCurrent(lease, work);
65690
- const decision = this.rootBudget?.checkpoint();
65691
- if (decision?.kind === "exhausted" || decision?.kind === "wrap_up")
65692
- throw new Error("Root workflow budget exhausted");
65693
- return queue.control(lease);
65694
- }
65695
- if (message.kind !== "complete")
65696
- throw new Error("Invalid worker operation");
65697
- await queue.complete(lease, work, message.result);
65698
- this.replacements = 0;
65699
- work = undefined;
65700
- });
65701
- if (message.kind !== "delivery" && message.kind !== "delivery-ready")
65702
- serial = response.then(() => {}, () => {});
65703
- response.then((value2) => child.send({ id: message.id, value: value2 }, replied), (error) => child.send({ id: message.id, error: error.message }, replied));
65704
- });
66682
+ await this.command({
66683
+ kind: "start",
66684
+ sources: this.sources,
66685
+ cwd,
66686
+ publish: store !== undefined,
66687
+ ...this.rootRun()
65705
66688
  });
65706
66689
  }
65707
- recoverExpired(root) {
65708
- if (this.recovering)
65709
- return this.recovering;
65710
- if (this.closed || performance.now() < this.nextRecoveryAt)
65711
- return Promise.resolve();
65712
- this.recovering = (async () => {
65713
- await new Promise((resolve) => setImmediate(resolve));
65714
- await this.admission?.reconcile();
65715
- const expiredBefore = Date.now();
65716
- for (const work of await this.queue.inspect(root)) {
65717
- if (work.state !== "leased" || work.expiresAt >= expiredBefore)
65718
- continue;
65719
- const child = work.worker === null ? undefined : this.incarnations.get(work.worker);
65720
- if (child && work.worker !== null && this.isLive(work.worker, child)) {
65721
- await this.queue.extend(this.lease, work, this.leaseMs).catch((error) => {
65722
- if (error.message !== STALE_WORK_LEASE)
65723
- throw error;
65724
- });
65725
- continue;
65726
- }
65727
- if (child && !await this.stop(child).then(() => true, () => false))
65728
- continue;
65729
- await this.queue.expire(this.lease, work);
65730
- }
65731
- })().finally(() => {
65732
- this.recovering = undefined;
65733
- this.nextRecoveryAt = performance.now() + RECOVERY_INTERVAL_MS;
65734
- });
65735
- return this.recovering;
66690
+ queueView() {
66691
+ return {
66692
+ assertCurrent: async (_lease, work) => {
66693
+ const current = this.attempts.get(work.id);
66694
+ if (current === undefined || current.fenced || current.generation !== work.generation || current.worker !== work.worker || current.epoch !== work.epoch)
66695
+ throw new Error(STALE_WORK_LEASE);
66696
+ },
66697
+ withCurrentAppend: (lease, work, assertSession, action) => this.queue.withCurrentAppend(lease, work, assertSession, action),
66698
+ result: async (root, id) => this.results.get(id) ?? this.queue.result(root, id)
66699
+ };
65736
66700
  }
65737
- observeResponsiveness() {
65738
- const now = performance.now();
65739
- const gap = now - this.lastTick;
65740
- if (gap > ROOT_STALL_THRESHOLD_MS)
65741
- this.rootStalledMs += gap - ROOT_TICK_INTERVAL_MS;
65742
- this.lastTick = now;
66701
+ rootProjection() {
66702
+ const run = this.store?.runs().find((candidate) => candidate.id === this.root);
66703
+ return run === undefined ? undefined : projectRootRun(run);
65743
66704
  }
65744
- isLive(worker, child) {
65745
- this.observeResponsiveness();
65746
- const heard = this.lastHeard.get(worker);
65747
- return child.exitCode === null && child.signalCode === null && child.connected && heard !== undefined && performance.now() - heard.at - (this.rootStalledMs - heard.rootStalled) < this.leaseMs;
66705
+ rootRun() {
66706
+ const rootRun = this.rootProjection();
66707
+ if (rootRun === undefined)
66708
+ return {};
66709
+ this.rootSignature = JSON.stringify(rootRun);
66710
+ return { rootRun };
65748
66711
  }
65749
- stop(child) {
65750
- if (child.pid === undefined || child.exitCode !== null || child.signalCode !== null)
65751
- return Promise.resolve();
65752
- return new Promise((resolve, reject) => {
65753
- const timeout = setTimeout(() => reject(new Error("Local worker cleanup was not confirmed")), 5000);
65754
- child.once("exit", () => {
65755
- clearTimeout(timeout);
65756
- resolve();
65757
- });
65758
- child.kill("SIGKILL");
65759
- });
66712
+ refreshRootRun() {
66713
+ const rootRun = this.rootProjection();
66714
+ if (rootRun === undefined)
66715
+ return;
66716
+ const signature = JSON.stringify(rootRun);
66717
+ if (signature === this.rootSignature)
66718
+ return;
66719
+ this.rootSignature = signature;
66720
+ this.notice({ kind: "root", rootRun });
65760
66721
  }
65761
66722
  async run(definition, inputs, options) {
65762
66723
  try {
@@ -65789,7 +66750,7 @@ class LocalWorkerPool {
65789
66750
  const selectedSource = this.sources[index];
65790
66751
  if (await definitionDigest(selectedSource.path) !== selectedSource.digest)
65791
66752
  throw new Error("Worker definition compatibility error");
65792
- await this.backend.hydrateWorkflow(root);
66753
+ await this.backend.catchUpWorkflow(root);
65793
66754
  const boundary = this.backend.listCheckpoints(root).find((entry) => entry.kind === "stage" && entry.topology?.boundary?.event === "start" && entry.topology.stageId === options.parentRun.stageId);
65794
66755
  if (boundary?.kind !== "stage")
65795
66756
  throw new Error("Missing durable child boundary");
@@ -65800,39 +66761,232 @@ class LocalWorkerPool {
65800
66761
  config: options.config,
65801
66762
  executionMode: options.executionMode
65802
66763
  }));
65803
- const work = await this.queue.enqueue(this.lease, {
65804
- boundaryId: options.parentRun.stageId,
65805
- childRunId: options.runId,
65806
- scope: options.durableScope.scopePrefix,
65807
- definition: `${selectedSource.path}#${selectedSource.exportName}`,
65808
- digest: selectedSource.digest,
65809
- inputs,
65810
- execution
65811
- }, boundary);
65812
- for (;; ) {
65813
- options.signal?.throwIfAborted();
65814
- if (this.closed) {
66764
+ this.evaluateBudget();
66765
+ const enqueued = await this.command({
66766
+ kind: "enqueue",
66767
+ descriptor: {
66768
+ boundaryId: options.parentRun.stageId,
66769
+ childRunId: options.runId,
66770
+ scope: options.durableScope.scopePrefix,
66771
+ definition: `${selectedSource.path}#${selectedSource.exportName}`,
66772
+ digest: selectedSource.digest,
66773
+ inputs,
66774
+ execution
66775
+ },
66776
+ boundary,
66777
+ ...this.rootRun()
66778
+ });
66779
+ const work = enqueued.work;
66780
+ this.works.set(work.id, work);
66781
+ if (enqueued.result !== undefined && !this.results.has(work.id))
66782
+ this.results.set(work.id, enqueued.result);
66783
+ this.waiting++;
66784
+ this.ticker ??= setInterval(() => {
66785
+ this.evaluateBudget();
66786
+ this.signalChange();
66787
+ }, POOL_SUMMARY_INTERVAL_MS).unref();
66788
+ try {
66789
+ for (;; ) {
66790
+ options.signal?.throwIfAborted();
66791
+ if (this.closed) {
66792
+ if (this.failure)
66793
+ throw this.failure;
66794
+ if (this.desiredControl === "quit")
66795
+ throw new WorkflowGracefulQuitError(root, "worker pool teardown");
66796
+ throw new Error("Local worker pool is closed");
66797
+ }
66798
+ const result = this.results.get(work.id);
66799
+ if (result) {
66800
+ await this.inbound;
66801
+ await this.backend.catchUpWorkflow(root);
66802
+ await this.handoff?.complete(work);
66803
+ return result;
66804
+ }
65815
66805
  if (this.failure)
65816
66806
  throw this.failure;
65817
- if (this.desiredControl === "quit")
65818
- throw new WorkflowGracefulQuitError(root, "worker pool teardown");
65819
- throw new Error("Local worker pool is closed");
66807
+ if (this.budgetBlocked)
66808
+ await this.rootBudget.stopAtBoundaryAsync(definition.name);
66809
+ await this.nextChange(options.signal);
65820
66810
  }
65821
- const result = await this.queue.result(root, work.id);
65822
- if (result) {
65823
- await this.backend.hydrateWorkflow(root);
65824
- await this.handoff?.complete(work);
65825
- return result;
66811
+ } finally {
66812
+ if (--this.waiting === 0) {
66813
+ clearInterval(this.ticker);
66814
+ this.ticker = undefined;
65826
66815
  }
65827
- if (this.failure)
65828
- throw this.failure;
65829
- const decision = this.rootBudget?.checkpoint();
65830
- if (decision?.kind === "exhausted" || decision?.kind === "wrap_up")
65831
- await this.rootBudget.stopAtBoundaryAsync(definition.name);
65832
- await this.recoverExpired(root);
65833
- await delay2(25, undefined, { signal: options.signal });
65834
66816
  }
65835
66817
  }
66818
+ evaluateBudget() {
66819
+ const decision = this.rootBudget?.checkpoint();
66820
+ const blocked = decision?.kind === "exhausted" || decision?.kind === "wrap_up";
66821
+ if (blocked === this.budgetBlocked)
66822
+ return;
66823
+ this.budgetBlocked = blocked;
66824
+ this.notice({ kind: "budget", blocked });
66825
+ }
66826
+ signalChange() {
66827
+ const changed = this.changed;
66828
+ this.changed = Promise.withResolvers();
66829
+ changed.resolve();
66830
+ }
66831
+ nextChange(signal) {
66832
+ const changed = this.changed.promise;
66833
+ if (signal === undefined)
66834
+ return changed;
66835
+ return new Promise((resolve, reject) => {
66836
+ const abort = () => reject(signal.reason);
66837
+ signal.addEventListener("abort", abort, { once: true });
66838
+ changed.then(() => {
66839
+ signal.removeEventListener("abort", abort);
66840
+ resolve();
66841
+ });
66842
+ });
66843
+ }
66844
+ connect() {
66845
+ if (this.coordinator !== undefined)
66846
+ return this.coordinator;
66847
+ const coordinator = this.options.coordinator === "in-process" ? launchCoordinatorInProcess() : launchCoordinatorProcess();
66848
+ this.coordinator = coordinator;
66849
+ coordinator.port.onMessage((event) => {
66850
+ this.inbound = this.inbound.then(() => this.receive(event)).catch(() => {});
66851
+ });
66852
+ coordinator.port.onClose((reason) => {
66853
+ this.coordinatorLost = reason;
66854
+ this.inbound = this.inbound.then(() => this.lost(reason));
66855
+ });
66856
+ return coordinator;
66857
+ }
66858
+ lost(reason) {
66859
+ this.pids.clear();
66860
+ if (!this.closed)
66861
+ this.failure ??= reason;
66862
+ for (const pending of this.commands.values())
66863
+ pending.reject(reason);
66864
+ this.commands.clear();
66865
+ this.signalChange();
66866
+ }
66867
+ command(body) {
66868
+ const coordinator = this.connect();
66869
+ if (this.coordinatorLost !== undefined)
66870
+ return Promise.reject(this.coordinatorLost);
66871
+ const id = ++this.commandSequence;
66872
+ return new Promise((resolve, reject) => {
66873
+ this.commands.set(id, { resolve, reject });
66874
+ coordinator.port.send({ ...body, id });
66875
+ });
66876
+ }
66877
+ notice(notice) {
66878
+ this.coordinator?.port.send(notice);
66879
+ }
66880
+ async receive(event) {
66881
+ switch (event.kind) {
66882
+ case "reply": {
66883
+ const pending = this.commands.get(event.id);
66884
+ this.commands.delete(event.id);
66885
+ if (event.error !== undefined)
66886
+ pending?.reject(new Error(event.error));
66887
+ else
66888
+ pending?.resolve(event.value);
66889
+ return;
66890
+ }
66891
+ case "activity":
66892
+ return this.applyActivity(event.batch, event.entries, event.usage);
66893
+ case "spawned":
66894
+ this.pids.add(event.pid);
66895
+ return;
66896
+ case "exited":
66897
+ this.pids.delete(event.pid);
66898
+ return;
66899
+ case "fence": {
66900
+ const current = this.attempts.get(event.work.id);
66901
+ if (current?.generation === event.work.generation)
66902
+ current.fenced = true;
66903
+ this.handoff?.fence(event.work);
66904
+ return;
66905
+ }
66906
+ case "result":
66907
+ if (event.usage !== undefined)
66908
+ this.usageTotal = event.usage;
66909
+ this.results.set(event.workId, event.result);
66910
+ this.signalChange();
66911
+ return;
66912
+ case "failure":
66913
+ this.failure ??= new Error(event.error);
66914
+ this.signalChange();
66915
+ return;
66916
+ case "request":
66917
+ this.serve(event.id, event.request);
66918
+ return;
66919
+ }
66920
+ }
66921
+ async applyActivity(batch, entries, usage) {
66922
+ if (usage !== undefined)
66923
+ this.usageTotal = usage;
66924
+ const rejected = [];
66925
+ const handoff = this.handoff;
66926
+ if (handoff !== undefined) {
66927
+ const publications = entries.flatMap((entry) => {
66928
+ const work = this.track(entry.work);
66929
+ if (work !== undefined)
66930
+ return [{ work, runs: entry.runs }];
66931
+ rejected.push({ workId: entry.work.id, generation: entry.work.generation, error: STALE_WORK_LEASE });
66932
+ return [];
66933
+ });
66934
+ const outcomes = await handoff.publishBatch(publications);
66935
+ outcomes.forEach((error, index) => {
66936
+ const { work } = publications[index];
66937
+ if (error !== undefined)
66938
+ rejected.push({ workId: work.id, generation: work.generation, error: error.message });
66939
+ });
66940
+ }
66941
+ this.notice({ kind: "published", batch, rejected });
66942
+ this.refreshRootRun();
66943
+ }
66944
+ track(attempt) {
66945
+ const work = this.works.get(attempt.id);
66946
+ if (work === undefined)
66947
+ return;
66948
+ const current = this.attempts.get(attempt.id);
66949
+ if (current === undefined || attempt.generation > current.generation)
66950
+ this.attempts.set(attempt.id, { ...attempt, fenced: false });
66951
+ return { ...work, ...attempt, state: "leased" };
66952
+ }
66953
+ async serve(id, request) {
66954
+ try {
66955
+ const value2 = await this.answer(id, request);
66956
+ this.coordinator?.port.send({ kind: "request-reply", id, ...value2 === undefined ? {} : { value: value2 } });
66957
+ } catch (error) {
66958
+ this.coordinator?.port.send({
66959
+ kind: "request-reply",
66960
+ id,
66961
+ error: error instanceof Error ? error.message : String(error)
66962
+ });
66963
+ }
66964
+ }
66965
+ async answer(id, request) {
66966
+ if (request.kind === "pending-stage-message" || request.kind === "sticky-live-delivered") {
66967
+ const router = this.store === undefined ? undefined : pendingStageMessageRouter(this.store);
66968
+ const event = {
66969
+ ...request.payload,
66970
+ handled: false
66971
+ };
66972
+ if (request.kind === "pending-stage-message")
66973
+ router?.route(event);
66974
+ else
66975
+ router?.sticky(event);
66976
+ if (event.handled && event.completion !== undefined)
66977
+ return await event.completion;
66978
+ return request.kind === "pending-stage-message" ? { outcome: "refused", reason: "Session not found" } : undefined;
66979
+ }
66980
+ const handoff = this.handoff;
66981
+ const work = this.track(request.work);
66982
+ if (handoff === undefined || work === undefined)
66983
+ throw new Error(STALE_WORK_LEASE);
66984
+ if (request.kind === "ready")
66985
+ return handoff.ready(work, request.runId, request.stageId, request.stageName, request.sessionId);
66986
+ return handoff.deliver(work, request.runId, request.stageId, request.stageName, request.recipient, async (from, message) => {
66987
+ await this.command({ kind: "transport", request: id, from, message });
66988
+ });
66989
+ }
65836
66990
  closing;
65837
66991
  close() {
65838
66992
  this.closing ??= this.dispose();
@@ -65841,27 +66995,28 @@ class LocalWorkerPool {
65841
66995
  async dispose() {
65842
66996
  this.closed = true;
65843
66997
  clearInterval(this.ticker);
66998
+ this.ticker = undefined;
65844
66999
  this.releaseBinding();
65845
67000
  if (this.options.rootId === undefined)
65846
67001
  workflowResourceAdmission.delete(this.backend);
67002
+ this.signalChange();
65847
67003
  let failure;
65848
- let stopping = [];
65849
- try {
65850
- await this.recovering?.catch(() => {});
65851
- stopping = [...this.children].map((child) => ({ child, log: this.logs.get(child) }));
65852
- await Promise.all(stopping.map(({ child }) => this.stop(child)));
65853
- await Promise.all([...this.cleanups]);
65854
- if (this.cleanupFailure)
65855
- throw this.cleanupFailure;
65856
- } catch (error) {
65857
- failure = error;
65858
- }
65859
- if (failure === undefined) {
65860
- const stopped = stopping.flatMap(({ child, log }) => log !== undefined && (child.exitCode !== null || child.signalCode !== null) ? [log] : []);
65861
- await Promise.all(stopped.map((log) => rm3(log, { force: true }).catch(() => {})));
65862
- if (this.logDirectory !== undefined)
65863
- await rmdir(this.logDirectory).catch(() => {});
67004
+ const coordinator = this.coordinator;
67005
+ if (coordinator !== undefined) {
67006
+ if (this.coordinatorLost === undefined) {
67007
+ try {
67008
+ await this.command({ kind: "close" });
67009
+ } catch (error) {
67010
+ failure = error;
67011
+ }
67012
+ }
67013
+ await this.inbound;
67014
+ await coordinator.port.close();
67015
+ this.pids.clear();
65864
67016
  }
67017
+ for (const release of this.routeClaims.values())
67018
+ release();
67019
+ this.routeClaims.clear();
65865
67020
  try {
65866
67021
  await this.releaseRoutes?.();
65867
67022
  } catch (error) {
@@ -65871,27 +67026,26 @@ class LocalWorkerPool {
65871
67026
  throw failure;
65872
67027
  }
65873
67028
  }
65874
- var INTERNAL_WORKFLOW_WORKER_ARG = "--internal-workflow-worker", DEFAULT_WORK_LEASE_MS = 30000, MINIMUM_WORK_LEASE_MS, CONTROL_ACKNOWLEDGEMENT_TIMEOUT_MS = 5000, RECOVERY_INTERVAL_MS = 500, ROOT_TICK_INTERVAL_MS = 100, ROOT_STALL_THRESHOLD_MS = 500, COORDINATOR_LEASE_MS, DEFAULT_STARTUP_TIMEOUT_MS = 30000, MAX_TIMER_DELAY_MS = 2147483647, STDERR_TAIL_CHARACTERS = 8192, STDERR_DRAIN_GRACE_MS = 1000;
67029
+ var INTERNAL_WORKFLOW_WORKER_ARG = "--internal-workflow-worker", DEFAULT_WORK_LEASE_MS = 30000, MINIMUM_WORK_LEASE_MS, DEFAULT_STARTUP_TIMEOUT_MS = 30000, MAX_TIMER_DELAY_MS = 2147483647, COORDINATOR_STDERR_TAIL_CHARACTERS = 8192, COORDINATOR_EXIT_GRACE_MS = 5000;
65875
67030
  var init_local_pool = __esm(() => {
65876
67031
  init_internal();
65877
67032
  init_sql_pool_queue();
65878
- init_sql_pool_usage();
65879
67033
  init_sql_run_journal();
65880
67034
  init_workflow_tool_abort();
65881
67035
  init_sdk_intercom();
67036
+ init_external_run_routes();
65882
67037
  init_workflow_infrastructure_error();
65883
67038
  init_admission_binding();
65884
67039
  init_definitions();
65885
67040
  init_dispatch();
65886
67041
  init_host_options();
65887
- init_resource_admission();
67042
+ init_pool_coordinator();
65888
67043
  init_stage_handoff();
65889
67044
  MINIMUM_WORK_LEASE_MS = 2 * WORKER_HEARTBEAT_INTERVAL_MS;
65890
- COORDINATOR_LEASE_MS = 3 * STAGE_SESSION_HEARTBEAT_INTERVAL_MS;
65891
67045
  });
65892
67046
 
65893
67047
  // dist/builtin/workflows/src/workers/public-pool.ts
65894
- import { resolve as resolve14 } from "node:path";
67048
+ import { resolve as resolve15 } from "node:path";
65895
67049
  import { isDeepStrictEqual as isDeepStrictEqual2 } from "node:util";
65896
67050
  function withWorkerSettings(options, settings) {
65897
67051
  const configured = settings?.workflows;
@@ -65940,11 +67094,11 @@ async function activatePublicPool(backend, options) {
65940
67094
  ...options.localWorkers,
65941
67095
  startupTimeoutMs: undefined,
65942
67096
  leaseMs: undefined,
65943
- definitions: options.localWorkers.definitions.map((source) => ({ ...source, path: resolve14(cwd, source.path) }))
67097
+ definitions: options.localWorkers.definitions.map((source) => ({ ...source, path: resolve15(cwd, source.path) }))
65944
67098
  };
65945
67099
  const admission = options.admission && {
65946
67100
  ...options.admission,
65947
- ...options.admission.userDatabase ? { userDatabase: resolve14(cwd, options.admission.userDatabase) } : {}
67101
+ ...options.admission.userDatabase ? { userDatabase: resolve15(cwd, options.admission.userDatabase) } : {}
65948
67102
  };
65949
67103
  const sources = localWorkers && await Promise.all(localWorkers.definitions.map(async (source) => ({ ...source, digest: await definitionDigest(source.path) })));
65950
67104
  const resources = JSON.parse(JSON.stringify({
@@ -72552,9 +73706,9 @@ var init_stage_runner_controller = __esm(() => {
72552
73706
  });
72553
73707
 
72554
73708
  // dist/builtin/workflows/src/runs/foreground/stage-runner-output.ts
72555
- import { createHash as createHash4 } from "node:crypto";
73709
+ import { createHash as createHash5 } from "node:crypto";
72556
73710
  import { mkdir as mkdir7, writeFile as writeFile10 } from "node:fs/promises";
72557
- import { basename as basename5, dirname as dirname15, isAbsolute as isAbsolute11, join as join30, resolve as resolve15 } from "node:path";
73711
+ import { basename as basename5, dirname as dirname15, isAbsolute as isAbsolute11, join as join30, resolve as resolve16 } from "node:path";
72558
73712
  function normalizeMaxOutput2(maxOutput) {
72559
73713
  return {
72560
73714
  bytes: maxOutput?.bytes ?? DEFAULT_MAX_OUTPUT_BYTES,
@@ -72616,7 +73770,7 @@ function formatByteSize(bytes) {
72616
73770
  return `${value2.toFixed(1)} ${units[unitIndex]}`;
72617
73771
  }
72618
73772
  function transcriptPath(runId, outputPath) {
72619
- const digest = createHash4("sha256").update(outputPath).digest("hex").slice(0, 16);
73773
+ const digest = createHash5("sha256").update(outputPath).digest("hex").slice(0, 16);
72620
73774
  return join30(workflowArtifactRunPath(runId), "transcripts", `${digest}-${basename5(outputPath)}.transcript.md`);
72621
73775
  }
72622
73776
  function appendContent(lines, content) {
@@ -72662,7 +73816,7 @@ function renderTranscript(messages, fallbackOutput) {
72662
73816
  `);
72663
73817
  }
72664
73818
  function savedOutputReference(outputPath, fullOutput, transcriptPathValue, transcriptText, transcriptAvailable, warning) {
72665
- const absolutePath = resolve15(outputPath);
73819
+ const absolutePath = resolve16(outputPath);
72666
73820
  const bytes = Buffer.byteLength(fullOutput, "utf8");
72667
73821
  const lines = countLines(fullOutput);
72668
73822
  const transcriptBytes = Buffer.byteLength(transcriptText, "utf8");
@@ -72677,8 +73831,8 @@ function resolveOutputPath(output, runtimeCwd, requestedCwd) {
72677
73831
  return;
72678
73832
  if (isAbsolute11(output))
72679
73833
  return output;
72680
- const baseCwd = requestedCwd === undefined ? runtimeCwd : isAbsolute11(requestedCwd) ? requestedCwd : resolve15(runtimeCwd, requestedCwd);
72681
- return resolve15(baseCwd, output);
73834
+ const baseCwd = requestedCwd === undefined ? runtimeCwd : isAbsolute11(requestedCwd) ? requestedCwd : resolve16(runtimeCwd, requestedCwd);
73835
+ return resolve16(baseCwd, output);
72682
73836
  }
72683
73837
  function degenerateWarning(fullOutput) {
72684
73838
  return fullOutput.trim().length === 0 ? "WARNING: the stage artifact is empty; search the companion transcript for this stage's work." : undefined;
@@ -73687,7 +74841,7 @@ async function run(def, inputs, opts = {}) {
73687
74841
  if (!isWorkflowDefinition(def))
73688
74842
  throw new Error(workflowDefinitionRequirementMessage("run(definition, inputs)", def));
73689
74843
  const depth = opts.depth ?? 0;
73690
- const maxDepth = opts.config?.maxDepth ?? 4;
74844
+ const maxDepth = opts.config?.maxDepth ?? 3;
73691
74845
  if (depth >= maxDepth) {
73692
74846
  return {
73693
74847
  runId: opts.runId ?? crypto.randomUUID(),
@@ -75141,9 +76295,9 @@ var init_sdk_owner_release = __esm(() => {
75141
76295
 
75142
76296
  // dist/builtin/workflows/src/durable/sdk-recovery-scope.ts
75143
76297
  import { realpathSync as realpathSync5 } from "node:fs";
75144
- import { resolve as resolve16 } from "node:path";
76298
+ import { resolve as resolve17 } from "node:path";
75145
76299
  function canonicalDirectory(path) {
75146
- const absolute = resolve16(path);
76300
+ const absolute = resolve17(path);
75147
76301
  try {
75148
76302
  const canonical = realpathSync5.native(absolute);
75149
76303
  return process.platform === "win32" ? canonical.toLowerCase() : canonical;
@@ -75880,9 +77034,7 @@ function runSnapshotsFromCheckpoints(checkpoints, rootRunId, rootRunName, fallba
75880
77034
  }
75881
77035
  for (const group of grouped.values())
75882
77036
  group.sort(compareDraftSourceOrder);
75883
- const groupedCount = grouped.size;
75884
- retainReachableRunGroups(grouped, rootRunId);
75885
- if (policy.failOnUnreachable && grouped.size !== groupedCount)
77037
+ if (!retainReachableRunGroups(grouped, rootRunId) && policy.failOnUnreachable)
75886
77038
  return [];
75887
77039
  if (!validateRunGroups(grouped, rootRunId, toolCheckpoints, checkpoints, policy.retainActive))
75888
77040
  return [];
@@ -87496,7 +88648,7 @@ async function withDormantRunWrite(store, runId, write) {
87496
88648
 
87497
88649
  // dist/builtin/workflows/src/extension/dormant-run-intercom.ts
87498
88650
  var PENDING_STAGE_ROUTE_EVENT2 = "atomic:workflow-pending-stage-route";
87499
- var PENDING_STAGE_MESSAGE_EVENT2 = "atomic:workflow-pending-stage-message";
88651
+ var PENDING_STAGE_MESSAGE_EVENT3 = "atomic:workflow-pending-stage-message";
87500
88652
  var PENDING_STAGE_ROUTE_REVOKED_EVENT = "atomic:workflow-pending-stage-route-revoked";
87501
88653
  var DORMANT_RUN_SUBAGENT_HOLD_EVENT = "atomic:workflow-dormant-run-subagent-hold";
87502
88654
  var DORMANT_RUN_RECHECK_MS = 15000;
@@ -87672,7 +88824,7 @@ function registerDormantRunIntercomRoutes(pi, activeStore, deps) {
87672
88824
  ...event.senderRegistrationName === undefined ? {} : { senderRegistrationName: event.senderRegistrationName },
87673
88825
  ...event.senderReturnAddress === undefined ? {} : { senderReturnAddress: event.senderReturnAddress }
87674
88826
  };
87675
- pi.events?.emit?.(PENDING_STAGE_MESSAGE_EVENT2, forwarded);
88827
+ pi.events?.emit?.(PENDING_STAGE_MESSAGE_EVENT3, forwarded);
87676
88828
  if (forwarded.handled && forwarded.completion !== undefined)
87677
88829
  return await forwarded.completion;
87678
88830
  return { outcome: "refused", reason: "Session not found" };
@@ -87692,7 +88844,7 @@ function registerDormantRunIntercomRoutes(pi, activeStore, deps) {
87692
88844
  }
87693
88845
  return isLive(run.runId) ? await redispatch(event) : { outcome: "refused", reason: `Workflow run ${run.runId} is no longer resumable from this session` };
87694
88846
  };
87695
- const messages = pi.events?.on?.(PENDING_STAGE_MESSAGE_EVENT2, (payload) => {
88847
+ const messages = pi.events?.on?.(PENDING_STAGE_MESSAGE_EVENT3, (payload) => {
87696
88848
  if (disposed || !isDormantMessageEvent(payload) || payload.handled || payload.live === true)
87697
88849
  return;
87698
88850
  const target = parseWorkflowStageTarget(payload.target);
@@ -89610,8 +90762,8 @@ function mergeConfigs(base, override) {
89610
90762
  };
89611
90763
  }
89612
90764
  var WORKFLOW_CONFIG_DEFAULTS = {
89613
- maxDepth: 4,
89614
- defaultConcurrency: 3,
90765
+ maxDepth: 3,
90766
+ defaultConcurrency: 1000,
89615
90767
  persistRuns: true,
89616
90768
  statusFile: false,
89617
90769
  resumeInFlight: "never",
@@ -91378,7 +92530,7 @@ init_registry();
91378
92530
  // dist/builtin/workflows/src/extension/discovery-loaders.ts
91379
92531
  init_workflow_module_loader();
91380
92532
  import { readdir as readdir2, stat as stat2 } from "node:fs/promises";
91381
- import { extname, isAbsolute as isAbsolute13, join as join32, resolve as resolve17 } from "node:path";
92533
+ import { extname, isAbsolute as isAbsolute13, join as join32, resolve as resolve18 } from "node:path";
91382
92534
  async function scanWorkflowDir(dir) {
91383
92535
  try {
91384
92536
  const entries = await readdir2(dir, { withFileTypes: true });
@@ -91422,7 +92574,7 @@ async function loadFromPaths(pathsOrMap, kind, baseCwd, diagnostics, getWorkflow
91422
92574
  const all = [];
91423
92575
  const entries = Array.isArray(pathsOrMap) ? pathsOrMap.map((p) => ({ rawPath: p })) : Object.entries(pathsOrMap).map(([name, p]) => ({ rawPath: p, configuredName: name }));
91424
92576
  for (const { rawPath, configuredName } of entries) {
91425
- const absPath = isAbsolute13(rawPath) ? rawPath : resolve17(baseCwd, rawPath);
92577
+ const absPath = isAbsolute13(rawPath) ? rawPath : resolve18(baseCwd, rawPath);
91426
92578
  let pathStats;
91427
92579
  try {
91428
92580
  pathStats = await stat2(absPath);
@@ -98180,7 +99332,7 @@ var DEFAULT_PROMPT_GUIDANCE = [
98180
99332
  - Bounded loops must create distinct tracked work for every iteration; never reopen an ancestor below its downstream work. Stacked implementation slices are unrolled, not looped: each slice gets a fresh child boundary and distinct tracked nodes, and no slice may add an edge back to the current frontier, itself, or an ancestor. If a retained session receives follow-up without new dependency work, keep it as non-topological activity metadata rather than adding a back-edge. Before launch, sketch nodes and dependencies for every branch and loop and reject any proposed parent edge to the node itself or an ancestor.
98181
99333
  - Treat dynamic graph validation during execution, replay, and journal reconstruction as the authoritative cycle boundary. If runtime topology code changes, require incremental edge checks and reconstruction validation; prompt guidance and TypeScript types cannot replace those runtime checks.
98182
99334
  - Every workflow invocation receives the stable control group \`workflow:<rootRunId>\`. Stages inherit it unless an intentional explicit group is authored. Named groups and \`group: true\` become invocation-owned subgroups under \`workflow:<rootRunId>/...\`, so repeated names cannot collide across runs. The invocation context may list and exactly send/ask live stages in its owned subgroups and queue send to known pending stages; this authority is directional and does not let subgroup siblings see or reach each other. \`intercom list\` also shows possible future targets, and the invocation context may queue sticky sends to their path patterns. Pending and pattern ask remain unsupported. A \`workflow:<rootRunId>/**\` send broadcasts to all live stages and remains sticky for future descendants until root termination. \`group: "default"\` is the explicit non-owned shared-group escape and does not receive pending invocation routing. Do not mint root group names manually.
98183
- - Stages in the same invocation group are peers, not only steerable targets. A live stage can \`intercom list\` its siblings and \`send\`/\`ask\` them directly to debate a finding with evidence, hand off paths and reproductions, claim shared files or expensive steps, or learn what a sibling already verified; it can queue \`send\` to a known pending sibling and \`ask\` a completed sibling that retains a valid conversation, which reopens it for a post-mortem turn. Delegated subagents inside a stage inherit the same group. Named subgroups isolate their members from sibling subgroups. Stages do not coordinate spontaneously: when parallel stages should exchange evidence, say so in each prompt, name the peers or the pattern to discover them, bound the exchange (for example one evidence round), and require each stage to return its own structured result; keep scope and acceptance decisions with the supervisor or a \`ctx.ui\` gate.
99335
+ - Stages in the same invocation group are peers, not only steerable targets. A live stage can \`intercom list\` its siblings and \`send\`/\`ask\` them directly to debate a finding with evidence, hand off paths and reproductions, claim shared files or expensive steps, or learn what a sibling already verified; it can queue \`send\` to a known pending sibling and \`ask\` a completed sibling that retains a valid conversation, which reopens it for a post-mortem turn. Delegated subagents inside a stage inherit the same group. Named subgroups isolate their members from sibling subgroups. Default to peer-to-peer (swarm) exchange; the supervisor is not a message relay. There are no limits on communication between agents; it is encouraged whenever you need to. Stages proactively discover peers and broadcast, debate, claim, hand off, and ask each other directly, and each stage still returns its own structured result. Scale the swarm with the task: for very complex work or when the user asks for heavy parallelism, run hundreds or thousands of agents that self-organize into teams (named subgroups, per-stage subagent sets, child workflows) and coordinate peer-to-peer like an organization.
98184
99336
  - Prefer \`ctx.tool(name, args, fn)\` for workflow-owned TypeScript operations with side effects, such as filesystem writes, network mutations, and external API actions. It creates a tracked, non-attachable durable graph node before invoking \`fn\`; it may be used before, between, after, or without model stages. A completed \`ctx.tool\` call is durably checkpointed with its serializable result, so resume replays that result without rerunning \`fn\`. For checks that may fail during a bounded repair loop, opt into \`failureMode: "return"\`, branch on the typed outcome, and pass only the needed error fields to the repair stage or artifact yourself; Atomic does not inject failure evidence into later prompts. Retries do not bound a hung callback, so every \`ctx.tool\` callback that starts a child process or performs network I/O MUST set an explicit positive finite \`timeoutMs\` and forward the supplied \`signal\` to cancellable work. A timeout applies per attempt, aborts that attempt's signal, and gives retries a fresh deadline and signal. Cancellation and storage faults still throw. Keep pure computation and side-effect-free transformations as ordinary TypeScript. Do not wrap agent-stage internals or every function call indiscriminately; this rule applies to side effects orchestrated directly by the workflow definition.
98185
99337
  - Pass large stage context through files/artifacts and \`reads\` with an explicit \`Read the file at <path>...\` prompt rather than large \`previous\` payloads or injected session history.
98186
99338
  - Wrap critical stage-prompt content in \`<keepContext>\` / \`</keepContext>\`. Tagged content survives compression verbatim regardless of the compression ratio, and every line of the span including the tag lines is protected. A long-running stage will be compacted, and compaction ranks lines individually: an objective is verbose and restated while the constraint that narrows it is usually one line, so the constraint is the cheaper deletion and what survives reads as broader permission than intended. Tag role constraints that bound a stage to part of the work, acceptance criteria and immutable contracts, explicit prohibitions, and identifiers a stage must not lose such as a target branch, worktree path, or run ID. Do not tag bulk context: protected lines count against the keep target rather than raising it, so a large protected span makes the surrounding transcript compress harder — tag the constraint, not the material it applies to.