@bastani/atomic 0.9.43 → 0.9.44-alpha.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +26 -0
- package/dist/builtin/intercom/CHANGELOG.md +12 -0
- package/dist/builtin/intercom/broker/broker.ts +31 -7
- package/dist/builtin/intercom/broker/client.ts +65 -0
- package/dist/builtin/intercom/index.bundle.mjs +227 -95
- package/dist/builtin/intercom/live-route-refusal.ts +20 -0
- package/dist/builtin/intercom/package.json +1 -1
- package/dist/builtin/intercom/skills/intercom/SKILL.md +10 -6
- package/dist/builtin/intercom/types.ts +20 -1
- package/dist/builtin/mcp/CHANGELOG.md +7 -0
- package/dist/builtin/mcp/package.json +1 -1
- package/dist/builtin/subagents/CHANGELOG.md +9 -0
- package/dist/builtin/subagents/README.md +8 -8
- package/dist/builtin/subagents/agents/worker.md +1 -1
- package/dist/builtin/subagents/package.json +1 -1
- package/dist/builtin/subagents/skills/agent-browser/SKILL.md +1 -1
- package/dist/builtin/subagents/skills/cua-driver/EMBEDDING.md +21 -2
- package/dist/builtin/subagents/skills/cua-driver/LINUX.md +2 -2
- package/dist/builtin/subagents/skills/cua-driver/MACOS.md +342 -10
- package/dist/builtin/subagents/skills/cua-driver/README.md +4 -3
- package/dist/builtin/subagents/skills/cua-driver/RUNTIME.md +3 -1
- package/dist/builtin/subagents/skills/cua-driver/SKILL.md +7 -4
- package/dist/builtin/subagents/skills/cua-driver/WINDOWS.md +6 -5
- package/dist/builtin/subagents/skills/cua-driver/WORKFLOW.md +15 -0
- package/dist/builtin/subagents/skills/herdr/SKILL.md +22 -10
- package/dist/builtin/subagents/skills/subagent/SKILL.md +8 -5
- package/dist/builtin/subagents/src/extension/index.bundle.mjs +6 -6
- package/dist/builtin/web-access/package.json +1 -1
- package/dist/builtin/workflows/CHANGELOG.md +14 -0
- package/dist/builtin/workflows/README.md +1 -1
- package/dist/builtin/workflows/builtin/{chunk-frqxd4d1.js → chunk-k8d7ehjx.js} +22 -0
- package/dist/builtin/workflows/builtin/{chunk-3s95f7ry.js → chunk-qxpymnaf.js} +1 -1
- package/dist/builtin/workflows/builtin/{chunk-edhf30ef.js → chunk-zjecnjxx.js} +1 -1
- package/dist/builtin/workflows/builtin/goal.js +2 -2
- package/dist/builtin/workflows/builtin/index.js +3 -3
- package/dist/builtin/workflows/builtin/ralph.js +2 -2
- package/dist/builtin/workflows/package.json +1 -1
- package/dist/builtin/workflows/src/extension/index.bundle.mjs +1750 -598
- package/dist/builtin/workflows/src/extension/local-worker-entry.mjs +1786 -600
- package/dist/builtin/workflows/src/index.js +1802 -650
- package/dist/builtin/workflows/src/local-worker-entry.mjs +1786 -600
- package/dist/cli/args.js +1 -1
- package/dist/cli/args.js.map +1 -1
- package/dist/core/exec.d.ts.map +1 -1
- package/dist/core/exec.js +11 -2
- package/dist/core/exec.js.map +1 -1
- package/dist/core/package-manager-resolver.d.ts.map +1 -1
- package/dist/core/package-manager-resolver.js +25 -14
- package/dist/core/package-manager-resolver.js.map +1 -1
- package/dist/core/package-manager-types.d.ts +8 -0
- package/dist/core/package-manager-types.d.ts.map +1 -1
- package/dist/core/package-manager-types.js.map +1 -1
- package/dist/core/package-manager.d.ts +1 -1
- package/dist/core/package-manager.d.ts.map +1 -1
- package/dist/core/package-manager.js.map +1 -1
- package/dist/core/resource-loader-extension-warnings.d.ts +4 -1
- package/dist/core/resource-loader-extension-warnings.d.ts.map +1 -1
- package/dist/core/resource-loader-extension-warnings.js +15 -0
- package/dist/core/resource-loader-extension-warnings.js.map +1 -1
- package/dist/core/resource-loader-package-resources.d.ts.map +1 -1
- package/dist/core/resource-loader-package-resources.js +1 -0
- package/dist/core/resource-loader-package-resources.js.map +1 -1
- package/dist/core/resource-loader-reload.d.ts.map +1 -1
- package/dist/core/resource-loader-reload.js +5 -3
- package/dist/core/resource-loader-reload.js.map +1 -1
- package/dist/extensions/mcp/client/oauth/flow.d.ts.map +1 -1
- package/dist/extensions/mcp/client/oauth/flow.js +4 -1
- package/dist/extensions/mcp/client/oauth/flow.js.map +1 -1
- package/dist/extensions/mcp/client/oauth/types.d.ts +1 -0
- package/dist/extensions/mcp/client/oauth/types.d.ts.map +1 -1
- package/dist/extensions/mcp/client/oauth/types.js +8 -4
- package/dist/extensions/mcp/client/oauth/types.js.map +1 -1
- package/dist/extensions/mcp/client/transports/streamable-http.d.ts.map +1 -1
- package/dist/extensions/mcp/client/transports/streamable-http.js +2 -1
- package/dist/extensions/mcp/client/transports/streamable-http.js.map +1 -1
- package/dist/index.d.ts +1 -1
- package/dist/index.d.ts.map +1 -1
- package/dist/index.js.map +1 -1
- package/dist/main.d.ts.map +1 -1
- package/dist/main.js +20 -10
- package/dist/main.js.map +1 -1
- package/dist/modes/interactive/components/model-selector.d.ts.map +1 -1
- package/dist/modes/interactive/components/model-selector.js +4 -2
- package/dist/modes/interactive/components/model-selector.js.map +1 -1
- package/dist/modes/interactive/components/scoped-models-selector.d.ts.map +1 -1
- package/dist/modes/interactive/components/scoped-models-selector.js +4 -2
- package/dist/modes/interactive/components/scoped-models-selector.js.map +1 -1
- package/dist/modes/interactive/components/tool-execution.d.ts +6 -0
- package/dist/modes/interactive/components/tool-execution.d.ts.map +1 -1
- package/dist/modes/interactive/components/tool-execution.js +11 -1
- package/dist/modes/interactive/components/tool-execution.js.map +1 -1
- package/dist/modes/interactive/model-search.d.ts +10 -0
- package/dist/modes/interactive/model-search.d.ts.map +1 -1
- package/dist/modes/interactive/model-search.js +21 -0
- package/dist/modes/interactive/model-search.js.map +1 -1
- package/dist/package-manager-cli.d.ts.map +1 -1
- package/dist/package-manager-cli.js +5 -0
- package/dist/package-manager-cli.js.map +1 -1
- package/dist/utils/tools-manager.d.ts.map +1 -1
- package/dist/utils/tools-manager.js +30 -35
- package/dist/utils/tools-manager.js.map +1 -1
- package/docs/computer-use.md +5 -5
- package/docs/custom-provider/streaming.md +9 -14
- package/docs/environment-variables.md +1 -0
- package/docs/intercom.md +5 -1
- package/docs/packages.md +2 -0
- package/docs/settings.md +2 -2
- package/docs/skills.md +2 -2
- package/docs/subagents.md +38 -9
- package/docs/workflows/api-reference.md +1 -1
- package/docs/workflows/authoring.md +1 -1
- package/docs/workflows/operations.md +8 -6
- package/docs/workflows/reliable-design.md +23 -5
- package/examples/extensions/custom-provider-anthropic/index.ts +31 -31
- package/examples/extensions/todo.ts +7 -0
- package/npm-shrinkwrap.json +32 -32
- package/package.json +3 -3
|
@@ -1858,7 +1858,13 @@ function createStoreContext(state = createStoreState()) {
|
|
|
1858
1858
|
}
|
|
1859
1859
|
return cachedGraphSnapshot;
|
|
1860
1860
|
}
|
|
1861
|
+
let batchDepth = 0;
|
|
1862
|
+
let batchedNotification = false;
|
|
1861
1863
|
function notify() {
|
|
1864
|
+
if (batchDepth > 0) {
|
|
1865
|
+
batchedNotification = true;
|
|
1866
|
+
return;
|
|
1867
|
+
}
|
|
1862
1868
|
observation.capture();
|
|
1863
1869
|
for (const fn of state.invalidationListeners) {
|
|
1864
1870
|
try {
|
|
@@ -1878,6 +1884,18 @@ function createStoreContext(state = createStoreState()) {
|
|
|
1878
1884
|
state.version++;
|
|
1879
1885
|
notify();
|
|
1880
1886
|
}
|
|
1887
|
+
function batch(update) {
|
|
1888
|
+
batchDepth++;
|
|
1889
|
+
try {
|
|
1890
|
+
return update();
|
|
1891
|
+
} finally {
|
|
1892
|
+
batchDepth--;
|
|
1893
|
+
if (batchDepth === 0 && batchedNotification) {
|
|
1894
|
+
batchedNotification = false;
|
|
1895
|
+
notify();
|
|
1896
|
+
}
|
|
1897
|
+
}
|
|
1898
|
+
}
|
|
1881
1899
|
function findRun(runId) {
|
|
1882
1900
|
return state.runs.find((r) => r.id === runId);
|
|
1883
1901
|
}
|
|
@@ -1931,6 +1949,7 @@ function createStoreContext(state = createStoreState()) {
|
|
|
1931
1949
|
snapshot,
|
|
1932
1950
|
graphSnapshot,
|
|
1933
1951
|
notify,
|
|
1952
|
+
batch,
|
|
1934
1953
|
bumpAndNotify,
|
|
1935
1954
|
findRun,
|
|
1936
1955
|
findStage,
|
|
@@ -3867,6 +3886,9 @@ function createRunStoreMethods(context) {
|
|
|
3867
3886
|
subscribeInvalidation(fn) {
|
|
3868
3887
|
state.invalidationListeners.add(fn);
|
|
3869
3888
|
return () => state.invalidationListeners.delete(fn);
|
|
3889
|
+
},
|
|
3890
|
+
batch(update) {
|
|
3891
|
+
return context.batch(update);
|
|
3870
3892
|
}
|
|
3871
3893
|
};
|
|
3872
3894
|
}
|
|
@@ -5840,22 +5862,37 @@ function childScopedEvidenceExists(checkpoints, boundaryReplayKey, childRunId) {
|
|
|
5840
5862
|
}
|
|
5841
5863
|
function retainReachableRunGroups(grouped, rootRunId) {
|
|
5842
5864
|
const reachable = new Set([rootRunId]);
|
|
5843
|
-
const
|
|
5865
|
+
const settled = new Set;
|
|
5866
|
+
const pending = [{ runId: rootRunId, live: true }];
|
|
5844
5867
|
while (pending.length > 0) {
|
|
5845
|
-
const
|
|
5868
|
+
const { runId, live } = pending.pop();
|
|
5869
|
+
const drafts = grouped.get(runId);
|
|
5846
5870
|
if (drafts === undefined)
|
|
5847
5871
|
continue;
|
|
5848
5872
|
for (const draft of drafts) {
|
|
5849
|
-
const childRunId =
|
|
5850
|
-
if (childRunId === undefined ||
|
|
5873
|
+
const childRunId = workflowChildFromDraft(draft)?.runId ?? draft.topology?.boundary?.child.runId;
|
|
5874
|
+
if (childRunId === undefined || !grouped.has(childRunId))
|
|
5851
5875
|
continue;
|
|
5852
|
-
|
|
5853
|
-
|
|
5876
|
+
if (live && childRunIdFromDraft(draft) !== undefined) {
|
|
5877
|
+
if (reachable.has(childRunId))
|
|
5878
|
+
continue;
|
|
5879
|
+
reachable.add(childRunId);
|
|
5880
|
+
pending.push({ runId: childRunId, live: true });
|
|
5881
|
+
} else if (!settled.has(childRunId)) {
|
|
5882
|
+
settled.add(childRunId);
|
|
5883
|
+
pending.push({ runId: childRunId, live: false });
|
|
5884
|
+
}
|
|
5854
5885
|
}
|
|
5855
5886
|
}
|
|
5856
|
-
|
|
5857
|
-
|
|
5858
|
-
|
|
5887
|
+
let complete = true;
|
|
5888
|
+
for (const runId of grouped.keys()) {
|
|
5889
|
+
if (reachable.has(runId))
|
|
5890
|
+
continue;
|
|
5891
|
+
if (!settled.has(runId))
|
|
5892
|
+
complete = false;
|
|
5893
|
+
grouped.delete(runId);
|
|
5894
|
+
}
|
|
5895
|
+
return complete;
|
|
5859
5896
|
}
|
|
5860
5897
|
function childRunIdFromDraft(draft) {
|
|
5861
5898
|
const boundary = draft.topology?.boundary;
|
|
@@ -15531,7 +15568,7 @@ class ConcurrencyLimiter {
|
|
|
15531
15568
|
}
|
|
15532
15569
|
}
|
|
15533
15570
|
function createRunLimiter(defaultConcurrency) {
|
|
15534
|
-
return new ConcurrencyLimiter(defaultConcurrency ??
|
|
15571
|
+
return new ConcurrencyLimiter(defaultConcurrency ?? 1000);
|
|
15535
15572
|
}
|
|
15536
15573
|
|
|
15537
15574
|
// dist/builtin/workflows/src/shared/possible-stages.ts
|
|
@@ -17023,7 +17060,7 @@ function kebabCase(name) {
|
|
|
17023
17060
|
function errorMessage2(error) {
|
|
17024
17061
|
return error instanceof Error ? error.message : String(error);
|
|
17025
17062
|
}
|
|
17026
|
-
var DEFAULT_MAX_DEPTH =
|
|
17063
|
+
var DEFAULT_MAX_DEPTH = 3, BUILTIN_BARREL_SPECIFIERS, BUILTIN_SUBPATH_PREFIXES, TRACKED_NODE_METHODS;
|
|
17027
17064
|
var init_possible_stages = __esm(() => {
|
|
17028
17065
|
BUILTIN_BARREL_SPECIFIERS = new Set(["@bastani/atomic/workflows/builtin"]);
|
|
17029
17066
|
BUILTIN_SUBPATH_PREFIXES = ["@bastani/atomic/workflows/builtin/"];
|
|
@@ -60578,6 +60615,44 @@ class GraphFrontierTracker {
|
|
|
60578
60615
|
}
|
|
60579
60616
|
}
|
|
60580
60617
|
|
|
60618
|
+
// dist/builtin/workflows/src/shared/external-run-routes.ts
|
|
60619
|
+
function claimExternalRunRoute(store, runId, owner) {
|
|
60620
|
+
let owners = registry.owners.get(store);
|
|
60621
|
+
if (owners === undefined) {
|
|
60622
|
+
owners = new Map;
|
|
60623
|
+
registry.owners.set(store, owners);
|
|
60624
|
+
}
|
|
60625
|
+
owners.set(runId, owner);
|
|
60626
|
+
return () => {
|
|
60627
|
+
const current = registry.owners.get(store);
|
|
60628
|
+
if (current?.get(runId) !== owner)
|
|
60629
|
+
return;
|
|
60630
|
+
current.delete(runId);
|
|
60631
|
+
if (current.size === 0)
|
|
60632
|
+
registry.owners.delete(store);
|
|
60633
|
+
};
|
|
60634
|
+
}
|
|
60635
|
+
function externalRunRouteOwner(store, runId) {
|
|
60636
|
+
return registry.owners.get(store)?.get(runId);
|
|
60637
|
+
}
|
|
60638
|
+
function registerPendingStageMessageRouter(store, router) {
|
|
60639
|
+
registry.routers.set(store, router);
|
|
60640
|
+
return () => {
|
|
60641
|
+
if (registry.routers.get(store) === router)
|
|
60642
|
+
registry.routers.delete(store);
|
|
60643
|
+
};
|
|
60644
|
+
}
|
|
60645
|
+
function pendingStageMessageRouter(store) {
|
|
60646
|
+
return registry.routers.get(store);
|
|
60647
|
+
}
|
|
60648
|
+
var registryKey, processState, registry;
|
|
60649
|
+
var init_external_run_routes = __esm(() => {
|
|
60650
|
+
registryKey = Symbol.for("atomic-workflows/external-run-routes@1");
|
|
60651
|
+
processState = globalThis;
|
|
60652
|
+
registry = processState[registryKey] ?? { owners: new WeakMap, routers: new WeakMap };
|
|
60653
|
+
processState[registryKey] = registry;
|
|
60654
|
+
});
|
|
60655
|
+
|
|
60581
60656
|
// dist/builtin/workflows/src/shared/pending-stage-route-capability.ts
|
|
60582
60657
|
import { randomUUID as randomUUID11 } from "node:crypto";
|
|
60583
60658
|
function capabilitiesFor(store) {
|
|
@@ -60602,12 +60677,12 @@ function rotateWorkflowPendingStageRouteCapability(store, runId) {
|
|
|
60602
60677
|
capabilitiesFor(store).set(runId, capability);
|
|
60603
60678
|
return capability;
|
|
60604
60679
|
}
|
|
60605
|
-
var routeCapabilitiesKey,
|
|
60680
|
+
var routeCapabilitiesKey, processState2, routeCapabilities;
|
|
60606
60681
|
var init_pending_stage_route_capability = __esm(() => {
|
|
60607
60682
|
routeCapabilitiesKey = Symbol.for("atomic-workflows/pending-stage-route-capabilities@1");
|
|
60608
|
-
|
|
60609
|
-
routeCapabilities =
|
|
60610
|
-
|
|
60683
|
+
processState2 = globalThis;
|
|
60684
|
+
routeCapabilities = processState2[routeCapabilitiesKey] ?? new WeakMap;
|
|
60685
|
+
processState2[routeCapabilitiesKey] = routeCapabilities;
|
|
60611
60686
|
});
|
|
60612
60687
|
|
|
60613
60688
|
// dist/builtin/workflows/src/shared/pending-stage-route-readiness.ts
|
|
@@ -60639,12 +60714,12 @@ function hasWorkflowPendingStageRouteOwner(store) {
|
|
|
60639
60714
|
function workflowPendingStageRouteReady(store, runId) {
|
|
60640
60715
|
return owners.get(store)?.ready(runId);
|
|
60641
60716
|
}
|
|
60642
|
-
var routeOwnersKey,
|
|
60717
|
+
var routeOwnersKey, processState3, owners;
|
|
60643
60718
|
var init_pending_stage_route_readiness = __esm(() => {
|
|
60644
60719
|
routeOwnersKey = Symbol.for("atomic-workflows/pending-stage-route-owners@1");
|
|
60645
|
-
|
|
60646
|
-
owners =
|
|
60647
|
-
|
|
60720
|
+
processState3 = globalThis;
|
|
60721
|
+
owners = processState3[routeOwnersKey] ?? new WeakMap;
|
|
60722
|
+
processState3[routeOwnersKey] = owners;
|
|
60648
60723
|
});
|
|
60649
60724
|
|
|
60650
60725
|
// dist/builtin/workflows/src/shared/sticky-stage-delivery-status.ts
|
|
@@ -60769,6 +60844,9 @@ function registerPendingStageIntercomBridge(pi, activeStore, ownsRun = () => tru
|
|
|
60769
60844
|
notificationId,
|
|
60770
60845
|
reason
|
|
60771
60846
|
};
|
|
60847
|
+
const external = externalRunRouteOwner(activeStore, entry.runId);
|
|
60848
|
+
if (external !== undefined)
|
|
60849
|
+
return external.notifyUndeliverable(payload);
|
|
60772
60850
|
pi.events?.emit?.(PENDING_STAGE_UNDELIVERABLE_EVENT, payload);
|
|
60773
60851
|
return payload.handled && await payload.completion === true;
|
|
60774
60852
|
};
|
|
@@ -60780,19 +60858,29 @@ function registerPendingStageIntercomBridge(pi, activeStore, ownsRun = () => tru
|
|
|
60780
60858
|
return;
|
|
60781
60859
|
const ownedRunIds = new Set;
|
|
60782
60860
|
const runs = activeStore.runs();
|
|
60861
|
+
const runById = new Map(runs.map((run) => [run.id, run]));
|
|
60862
|
+
const childrenByBoundary = new Map;
|
|
60863
|
+
for (const run of runs) {
|
|
60864
|
+
if (run.parentRunId === undefined || run.parentStageId === undefined)
|
|
60865
|
+
continue;
|
|
60866
|
+
const key = JSON.stringify([run.parentRunId, run.parentStageId]);
|
|
60867
|
+
const children = childrenByBoundary.get(key) ?? [];
|
|
60868
|
+
children.push(run);
|
|
60869
|
+
childrenByBoundary.set(key, children);
|
|
60870
|
+
}
|
|
60783
60871
|
const resolveBoundarySegments = createWorkflowBoundarySegmentsResolver(runs);
|
|
60784
60872
|
for (const run of runs) {
|
|
60785
|
-
if (!ownsRun(run.id))
|
|
60873
|
+
if (externalRunRouteOwner(activeStore, run.id) !== undefined || !ownsRun(run.id))
|
|
60786
60874
|
continue;
|
|
60787
|
-
const rootRunId =
|
|
60875
|
+
const rootRunId = reciprocalWorkflowRootRunId(runById, run.id);
|
|
60788
60876
|
if (rootRunId === undefined)
|
|
60789
60877
|
continue;
|
|
60790
60878
|
ownedRunIds.add(run.id);
|
|
60791
|
-
const parentRun =
|
|
60879
|
+
const parentRun = run.parentRunId === undefined ? undefined : runById.get(run.parentRunId);
|
|
60792
60880
|
const boundary = parentRun?.stages.find((stage) => stage.id === run.parentStageId);
|
|
60793
60881
|
const parent = parentRun === undefined || boundary === undefined ? undefined : {
|
|
60794
60882
|
runId: parentRun.id,
|
|
60795
|
-
stageKeys: [boundary.id, boundary.name].filter((key) =>
|
|
60883
|
+
stageKeys: [boundary.id, boundary.name].filter((key) => resolveIndexedChildRun(childrenByBoundary, parentRun, key)?.id === run.id)
|
|
60796
60884
|
};
|
|
60797
60885
|
const announcement = {
|
|
60798
60886
|
runId: run.id,
|
|
@@ -60933,16 +61021,22 @@ function registerPendingStageIntercomBridge(pi, activeStore, ownsRun = () => tru
|
|
|
60933
61021
|
payload.handled = true;
|
|
60934
61022
|
payload.completion = queueAndPersist(activeStore, resolvedEvent, workflowInvocationIntercomGroup(rootRunId), stage.pendingStageDeliveryAvailable === true);
|
|
60935
61023
|
};
|
|
60936
|
-
const
|
|
60937
|
-
const stickySubscription = pi.events?.on?.(STICKY_LIVE_DELIVERED_EVENT, (payload) => {
|
|
61024
|
+
const routeStickyLiveDelivered = (payload) => {
|
|
60938
61025
|
if (disposed || !isStickyLiveDeliveredEvent(payload) || payload.handled)
|
|
60939
61026
|
return;
|
|
60940
61027
|
payload.handled = true;
|
|
60941
61028
|
payload.completion = outsideWorkflowExecutions(() => recordConfirmedStickyDeliveries(activeStore, payload));
|
|
61029
|
+
};
|
|
61030
|
+
const subscription = pi.events?.on?.(PENDING_STAGE_MESSAGE_EVENT, (payload) => outsideWorkflowExecutions(() => routePendingStageMessage(payload)));
|
|
61031
|
+
const stickySubscription = pi.events?.on?.(STICKY_LIVE_DELIVERED_EVENT, routeStickyLiveDelivered);
|
|
61032
|
+
const disposeRouter = registerPendingStageMessageRouter(activeStore, {
|
|
61033
|
+
route: (payload) => outsideWorkflowExecutions(() => routePendingStageMessage(payload)),
|
|
61034
|
+
sticky: routeStickyLiveDelivered
|
|
60942
61035
|
});
|
|
60943
61036
|
const dispose = () => {
|
|
60944
61037
|
disposed = true;
|
|
60945
61038
|
disposeReadiness();
|
|
61039
|
+
disposeRouter();
|
|
60946
61040
|
routeCompletions.clear();
|
|
60947
61041
|
unsubscribeStore();
|
|
60948
61042
|
if (typeof subscription === "function")
|
|
@@ -61068,6 +61162,12 @@ function isMaterializedNonAgentTarget(runs, target) {
|
|
|
61068
61162
|
return false;
|
|
61069
61163
|
return nonAgentNodes(run).some((node) => node.id === stageKey || node.name === stageKey);
|
|
61070
61164
|
}
|
|
61165
|
+
function resolveIndexedChildRun(childrenByBoundary, parent, segment) {
|
|
61166
|
+
const boundariesById = parent.stages.filter((stage) => stage.id === segment);
|
|
61167
|
+
const boundaries = boundariesById.length > 0 ? boundariesById : parent.stages.filter((stage) => stage.name === segment);
|
|
61168
|
+
const children = boundaries.flatMap((boundary) => childrenByBoundary.get(JSON.stringify([parent.id, boundary.id])) ?? []);
|
|
61169
|
+
return children.length === 1 ? children[0] : undefined;
|
|
61170
|
+
}
|
|
61071
61171
|
function resolveChildRun(runs, parent, segment) {
|
|
61072
61172
|
const boundariesById = parent.stages.filter((stage) => stage.id === segment);
|
|
61073
61173
|
const boundaries = boundariesById.length > 0 ? boundariesById : parent.stages.filter((stage) => stage.name === segment);
|
|
@@ -61407,6 +61507,7 @@ var init_pending_stage_intercom = __esm(() => {
|
|
|
61407
61507
|
init_execution_context_boundary();
|
|
61408
61508
|
init_run_owner_backend();
|
|
61409
61509
|
init_store_backend();
|
|
61510
|
+
init_external_run_routes();
|
|
61410
61511
|
init_intercom_group();
|
|
61411
61512
|
init_pending_stage_route_capability();
|
|
61412
61513
|
init_pending_stage_route_readiness();
|
|
@@ -61865,7 +61966,7 @@ function isWorkflowFutureStageRosterEntries(value2) {
|
|
|
61865
61966
|
function isWorkflowStageRosterEntries(value2) {
|
|
61866
61967
|
return Array.isArray(value2) && value2.every((entry) => typeof entry === "object" && entry !== null && entry.kind === "workflow-stage" && typeof entry.runId === "string" && typeof entry.stageId === "string" && typeof entry.stageName === "string" && typeof entry.target === "string" && (entry.lifecycle === "pending" || entry.lifecycle === "running") && typeof entry.group === "string" && (entry.sessionId === undefined || typeof entry.sessionId === "string"));
|
|
61867
61968
|
}
|
|
61868
|
-
var BROKER_SOCKET, BROKER_CONNECT_TIMEOUT_MS = 1e4, PENDING_STAGE_ROUTE_ACK_TIMEOUT_MS = 1e4, GROUP_REQUEST_TIMEOUT_MS = 5000, PRESENCE_ACK_TIMEOUT_MS = 5000, RESTORING_RECIPIENT_REQUEST_TIMEOUT_MS = 5000, HELD_SEND_GRACE_MS = 5000, IntercomClient;
|
|
61969
|
+
var BROKER_SOCKET, BROKER_CONNECT_TIMEOUT_MS = 1e4, PENDING_STAGE_ROUTE_ACK_TIMEOUT_MS = 1e4, GROUP_REQUEST_TIMEOUT_MS = 5000, PRESENCE_ACK_TIMEOUT_MS = 5000, RESTORING_RECIPIENT_REQUEST_TIMEOUT_MS = 5000, HELD_SEND_GRACE_MS = 5000, IntercomPendingStageRouteRefusedError, PENDING_STAGE_ROUTE_REFUSAL_CODES, IntercomClient;
|
|
61869
61970
|
var init_client = __esm(() => {
|
|
61870
61971
|
init_paths();
|
|
61871
61972
|
init_pending_send_registry();
|
|
@@ -61874,6 +61975,21 @@ var init_client = __esm(() => {
|
|
|
61874
61975
|
init_live_route_refusal();
|
|
61875
61976
|
init_retry_policy();
|
|
61876
61977
|
BROKER_SOCKET = getBrokerSocketPath();
|
|
61978
|
+
IntercomPendingStageRouteRefusedError = class IntercomPendingStageRouteRefusedError extends Error {
|
|
61979
|
+
connectionKept;
|
|
61980
|
+
code;
|
|
61981
|
+
constructor(reason, refusal = { connectionKept: false }) {
|
|
61982
|
+
super(reason);
|
|
61983
|
+
this.name = "IntercomPendingStageRouteRefusedError";
|
|
61984
|
+
this.connectionKept = refusal.connectionKept;
|
|
61985
|
+
this.code = refusal.code;
|
|
61986
|
+
}
|
|
61987
|
+
};
|
|
61988
|
+
PENDING_STAGE_ROUTE_REFUSAL_CODES = new Set([
|
|
61989
|
+
"not_authorized",
|
|
61990
|
+
"invalid_stage_roster",
|
|
61991
|
+
"invalid_possible_stage_roster"
|
|
61992
|
+
]);
|
|
61877
61993
|
IntercomClient = class IntercomClient extends EventEmitter {
|
|
61878
61994
|
returnAddress;
|
|
61879
61995
|
preferredSessionId;
|
|
@@ -61881,6 +61997,8 @@ var init_client = __esm(() => {
|
|
|
61881
61997
|
connectTimeoutMs;
|
|
61882
61998
|
brokerAcknowledgesReceipt = false;
|
|
61883
61999
|
brokerAcknowledgesPendingStageRoutes = false;
|
|
62000
|
+
multiplexesPendingStageRoutes;
|
|
62001
|
+
brokerRefusesPendingStageRoutesInPlace = false;
|
|
61884
62002
|
brokerAnswersStatus = false;
|
|
61885
62003
|
socket = null;
|
|
61886
62004
|
_sessionId = null;
|
|
@@ -61946,6 +62064,7 @@ var init_client = __esm(() => {
|
|
|
61946
62064
|
this.preferredSessionId = options.preferredSessionId;
|
|
61947
62065
|
this.acknowledgesReceipt = options.acknowledgesReceipt === true;
|
|
61948
62066
|
this.connectTimeoutMs = options.connectTimeoutMs ?? BROKER_CONNECT_TIMEOUT_MS;
|
|
62067
|
+
this.multiplexesPendingStageRoutes = options.multiplexesPendingStageRoutes === true;
|
|
61949
62068
|
}
|
|
61950
62069
|
failPending(error) {
|
|
61951
62070
|
this.pendingSends.rejectAll(error);
|
|
@@ -62058,6 +62177,7 @@ var init_client = __esm(() => {
|
|
|
62058
62177
|
this._supervisorSessionId = null;
|
|
62059
62178
|
this.brokerAcknowledgesReceipt = false;
|
|
62060
62179
|
this.brokerAcknowledgesPendingStageRoutes = false;
|
|
62180
|
+
this.brokerRefusesPendingStageRoutesInPlace = false;
|
|
62061
62181
|
this.brokerAnswersStatus = false;
|
|
62062
62182
|
this._messageSource = undefined;
|
|
62063
62183
|
this.disconnectError = null;
|
|
@@ -62136,7 +62256,7 @@ var init_client = __esm(() => {
|
|
|
62136
62256
|
}
|
|
62137
62257
|
switch (brokerMessage.type) {
|
|
62138
62258
|
case "registered": {
|
|
62139
|
-
if (typeof brokerMessage.sessionId !== "string" || brokerMessage.supervisorSessionId !== undefined && typeof brokerMessage.supervisorSessionId !== "string" || brokerMessage.acknowledgesReceipt !== undefined && brokerMessage.acknowledgesReceipt !== true || brokerMessage.acknowledgesPendingStageRoutes !== undefined && brokerMessage.acknowledgesPendingStageRoutes !== true || brokerMessage.brokerStatus !== undefined && brokerMessage.brokerStatus !== true) {
|
|
62259
|
+
if (typeof brokerMessage.sessionId !== "string" || brokerMessage.supervisorSessionId !== undefined && typeof brokerMessage.supervisorSessionId !== "string" || brokerMessage.acknowledgesReceipt !== undefined && brokerMessage.acknowledgesReceipt !== true || brokerMessage.acknowledgesPendingStageRoutes !== undefined && brokerMessage.acknowledgesPendingStageRoutes !== true || brokerMessage.refusesPendingStageRoutesInPlace !== undefined && brokerMessage.refusesPendingStageRoutesInPlace !== true || brokerMessage.brokerStatus !== undefined && brokerMessage.brokerStatus !== true) {
|
|
62140
62260
|
throw new Error("Invalid registered message");
|
|
62141
62261
|
}
|
|
62142
62262
|
if (this._sessionId !== null) {
|
|
@@ -62146,6 +62266,7 @@ var init_client = __esm(() => {
|
|
|
62146
62266
|
this._supervisorSessionId = typeof brokerMessage.supervisorSessionId === "string" ? brokerMessage.supervisorSessionId : null;
|
|
62147
62267
|
this.brokerAcknowledgesReceipt = this.acknowledgesReceipt && brokerMessage.acknowledgesReceipt === true;
|
|
62148
62268
|
this.brokerAcknowledgesPendingStageRoutes = brokerMessage.acknowledgesPendingStageRoutes === true;
|
|
62269
|
+
this.brokerRefusesPendingStageRoutesInPlace = brokerMessage.refusesPendingStageRoutesInPlace === true;
|
|
62149
62270
|
this.brokerAnswersStatus = brokerMessage.brokerStatus === true;
|
|
62150
62271
|
this.emit("_registered", { type: "registered", sessionId: brokerMessage.sessionId });
|
|
62151
62272
|
break;
|
|
@@ -62158,6 +62279,12 @@ var init_client = __esm(() => {
|
|
|
62158
62279
|
this.emit("_registration_failed", refusal);
|
|
62159
62280
|
break;
|
|
62160
62281
|
}
|
|
62282
|
+
const refusedRouteId = typeof brokerMessage.requestId === "string" ? brokerMessage.requestId : "";
|
|
62283
|
+
const refusedRoute = this.pendingStageRouteRegistrations.get(refusedRouteId);
|
|
62284
|
+
if (refusedRoute !== undefined) {
|
|
62285
|
+
this.pendingStageRouteRegistrations.delete(refusedRouteId);
|
|
62286
|
+
refusedRoute.reject(new IntercomPendingStageRouteRefusedError(brokerMessage.reason));
|
|
62287
|
+
}
|
|
62161
62288
|
this.disconnectError ??= refusal;
|
|
62162
62289
|
this.failPending(this.disconnectError);
|
|
62163
62290
|
this.socket?.destroy();
|
|
@@ -62283,6 +62410,21 @@ var init_client = __esm(() => {
|
|
|
62283
62410
|
this.emit("pending_stage_route_revoked", { runId });
|
|
62284
62411
|
break;
|
|
62285
62412
|
}
|
|
62413
|
+
case "pending_stage_route_refused": {
|
|
62414
|
+
const { requestId, runId, code, reason } = brokerMessage;
|
|
62415
|
+
if (typeof requestId !== "string" || typeof runId !== "string" || typeof reason !== "string" || typeof code !== "string") {
|
|
62416
|
+
throw new Error("Invalid pending-stage route refusal");
|
|
62417
|
+
}
|
|
62418
|
+
const pending = this.pendingStageRouteRegistrations.get(requestId);
|
|
62419
|
+
if (!pending)
|
|
62420
|
+
return;
|
|
62421
|
+
this.pendingStageRouteRegistrations.delete(requestId);
|
|
62422
|
+
pending.reject(new IntercomPendingStageRouteRefusedError(reason, {
|
|
62423
|
+
connectionKept: true,
|
|
62424
|
+
...PENDING_STAGE_ROUTE_REFUSAL_CODES.has(code) ? { code } : {}
|
|
62425
|
+
}));
|
|
62426
|
+
break;
|
|
62427
|
+
}
|
|
62286
62428
|
case "pending_stage_route_registered": {
|
|
62287
62429
|
const { requestId } = brokerMessage;
|
|
62288
62430
|
if (typeof requestId !== "string")
|
|
@@ -62786,7 +62928,8 @@ var init_client = __esm(() => {
|
|
|
62786
62928
|
stages,
|
|
62787
62929
|
possibleStages,
|
|
62788
62930
|
...parent === undefined ? {} : { parent },
|
|
62789
|
-
...dormant === true ? { dormant: true } : {}
|
|
62931
|
+
...dormant === true ? { dormant: true } : {},
|
|
62932
|
+
...this.multiplexesPendingStageRoutes && this.brokerRefusesPendingStageRoutesInPlace ? { refusalKeepsConnection: true } : {}
|
|
62790
62933
|
});
|
|
62791
62934
|
} catch (error) {
|
|
62792
62935
|
clearTimeout(timeout);
|
|
@@ -63520,120 +63663,165 @@ var init_reconnect_backoff = __esm(() => {
|
|
|
63520
63663
|
BACKOFF_MS = [1000, 2000, 5000, 1e4, 30000];
|
|
63521
63664
|
});
|
|
63522
63665
|
|
|
63523
|
-
// dist/builtin/intercom/route-connect-gate.ts
|
|
63524
|
-
function createConcurrencyGate(limit) {
|
|
63525
|
-
let active = 0;
|
|
63526
|
-
const waiting = [];
|
|
63527
|
-
const release = () => {
|
|
63528
|
-
const next = waiting.shift();
|
|
63529
|
-
if (next === undefined)
|
|
63530
|
-
active -= 1;
|
|
63531
|
-
else
|
|
63532
|
-
next();
|
|
63533
|
-
};
|
|
63534
|
-
return {
|
|
63535
|
-
async run(operation) {
|
|
63536
|
-
if (active < limit)
|
|
63537
|
-
active += 1;
|
|
63538
|
-
else
|
|
63539
|
-
await new Promise((resolve) => waiting.push(resolve));
|
|
63540
|
-
try {
|
|
63541
|
-
return await operation();
|
|
63542
|
-
} finally {
|
|
63543
|
-
release();
|
|
63544
|
-
}
|
|
63545
|
-
}
|
|
63546
|
-
};
|
|
63547
|
-
}
|
|
63548
|
-
var ROUTE_CLIENT_CONNECT_CONCURRENCY = 4;
|
|
63549
|
-
|
|
63550
63666
|
// dist/builtin/intercom/workflow-route-owner.ts
|
|
63551
63667
|
import { randomUUID as randomUUID14 } from "node:crypto";
|
|
63552
63668
|
import { createEventBus, getBuiltinPackagePaths } from "@bastani/atomic";
|
|
63669
|
+
function closedError() {
|
|
63670
|
+
return new Error("Workflow Intercom route owner closed");
|
|
63671
|
+
}
|
|
63553
63672
|
function createWorkflowRouteOwner(cwd) {
|
|
63554
63673
|
const events = createEventBus();
|
|
63555
63674
|
const config = loadConfig();
|
|
63556
63675
|
const id = randomUUID14();
|
|
63557
63676
|
const routes = new Map;
|
|
63558
|
-
const
|
|
63677
|
+
const connections = new Map;
|
|
63678
|
+
const isCurrent = (state) => routes.get(state.route.runId) === state;
|
|
63679
|
+
const connectionKey = (state) => state.isolated ? state.route.runId : state.route.group;
|
|
63680
|
+
function connectionOf(state) {
|
|
63681
|
+
const key = connectionKey(state);
|
|
63682
|
+
let connection = connections.get(key);
|
|
63683
|
+
if (connection === undefined) {
|
|
63684
|
+
connection = { key, group: state.route.group, acknowledged: false, routes: new Map };
|
|
63685
|
+
connections.set(key, connection);
|
|
63686
|
+
}
|
|
63687
|
+
connection.routes.set(state.route.runId, state);
|
|
63688
|
+
return connection;
|
|
63689
|
+
}
|
|
63690
|
+
function detach(state) {
|
|
63691
|
+
const key = connectionKey(state);
|
|
63692
|
+
const connection = connections.get(key);
|
|
63693
|
+
connection?.routes.delete(state.route.runId);
|
|
63694
|
+
if (!state.isolated || connection === undefined)
|
|
63695
|
+
return;
|
|
63696
|
+
connections.delete(key);
|
|
63697
|
+
connection.client?.disconnect().catch(() => {});
|
|
63698
|
+
}
|
|
63699
|
+
function settle(state, registered) {
|
|
63700
|
+
state.registered = registered;
|
|
63701
|
+
if (!registered && isCurrent(state))
|
|
63702
|
+
return;
|
|
63703
|
+
for (const waiter of state.waiters)
|
|
63704
|
+
waiter(registered);
|
|
63705
|
+
state.waiters.clear();
|
|
63706
|
+
}
|
|
63559
63707
|
function scheduleRetry(state) {
|
|
63560
|
-
if (state
|
|
63708
|
+
if (!isCurrent(state) || state.retry !== undefined)
|
|
63561
63709
|
return;
|
|
63562
63710
|
state.retry = setTimeout(() => {
|
|
63563
63711
|
state.retry = undefined;
|
|
63564
|
-
publish(state, state.route).catch(() =>
|
|
63712
|
+
publish(state, state.route).catch(() => {});
|
|
63565
63713
|
}, jitteredReconnectDelayMs(state.attempt++));
|
|
63566
63714
|
state.retry.unref();
|
|
63567
63715
|
}
|
|
63568
|
-
|
|
63569
|
-
if (
|
|
63570
|
-
|
|
63571
|
-
|
|
63572
|
-
|
|
63573
|
-
|
|
63574
|
-
|
|
63575
|
-
|
|
63576
|
-
}
|
|
63577
|
-
|
|
63578
|
-
|
|
63579
|
-
|
|
63580
|
-
if (
|
|
63581
|
-
|
|
63582
|
-
|
|
63583
|
-
|
|
63584
|
-
|
|
63585
|
-
|
|
63586
|
-
|
|
63587
|
-
|
|
63588
|
-
|
|
63589
|
-
|
|
63590
|
-
|
|
63591
|
-
|
|
63592
|
-
if (
|
|
63593
|
-
|
|
63594
|
-
const
|
|
63595
|
-
|
|
63596
|
-
|
|
63597
|
-
|
|
63598
|
-
|
|
63599
|
-
|
|
63600
|
-
|
|
63601
|
-
|
|
63602
|
-
|
|
63603
|
-
|
|
63604
|
-
|
|
63605
|
-
|
|
63606
|
-
|
|
63607
|
-
|
|
63608
|
-
|
|
63609
|
-
|
|
63610
|
-
|
|
63611
|
-
|
|
63612
|
-
|
|
63716
|
+
function drop(connection, client) {
|
|
63717
|
+
if (connection.client === client) {
|
|
63718
|
+
connection.client = undefined;
|
|
63719
|
+
connection.acknowledged = false;
|
|
63720
|
+
for (const state of connection.routes.values()) {
|
|
63721
|
+
settle(state, false);
|
|
63722
|
+
scheduleRetry(state);
|
|
63723
|
+
}
|
|
63724
|
+
}
|
|
63725
|
+
client.disconnect().catch(() => {});
|
|
63726
|
+
}
|
|
63727
|
+
function connect(connection) {
|
|
63728
|
+
if (connection.client?.isConnected())
|
|
63729
|
+
return Promise.resolve(connection.client);
|
|
63730
|
+
connection.connecting ??= (async () => {
|
|
63731
|
+
const [extensionDir] = getBuiltinPackagePaths({
|
|
63732
|
+
workflows: false,
|
|
63733
|
+
subagents: false,
|
|
63734
|
+
mcp: false,
|
|
63735
|
+
"web-access": false
|
|
63736
|
+
});
|
|
63737
|
+
if (extensionDir === undefined)
|
|
63738
|
+
throw new Error("Intercom builtin package is unavailable");
|
|
63739
|
+
await spawnBrokerIfNeeded(config.brokerCommand, config.brokerArgs, undefined, extensionDir);
|
|
63740
|
+
if (connections.get(connection.key) !== connection)
|
|
63741
|
+
throw closedError();
|
|
63742
|
+
const client = new IntercomClient(`${id}:pending-stage-route:${connection.key}`, {
|
|
63743
|
+
multiplexesPendingStageRoutes: true
|
|
63744
|
+
});
|
|
63745
|
+
connection.client = client;
|
|
63746
|
+
connection.acknowledged = false;
|
|
63747
|
+
client.on("error", () => {});
|
|
63748
|
+
client.on("disconnected", () => drop(connection, client));
|
|
63749
|
+
client.on("pending_stage_message", (request) => {
|
|
63750
|
+
if (connection.client !== client)
|
|
63751
|
+
return;
|
|
63752
|
+
const event = { ...request, handled: false };
|
|
63753
|
+
events.emit("atomic:workflow-pending-stage-message", event);
|
|
63754
|
+
const completion = event.handled && event.completion !== undefined ? event.completion : Promise.resolve({ outcome: "refused", reason: "Session not found" });
|
|
63755
|
+
completion.catch((error) => ({ outcome: "refused", reason: error.message })).then((result) => client.respondPendingStageMessage(request.requestId, result)).catch(() => {});
|
|
63756
|
+
});
|
|
63757
|
+
client.on("sticky_live_delivered", (notice) => {
|
|
63758
|
+
if (connection.client !== client)
|
|
63759
|
+
return;
|
|
63760
|
+
events.emit("atomic:workflow-sticky-live-delivered", { ...notice, handled: false });
|
|
63761
|
+
});
|
|
63762
|
+
try {
|
|
63763
|
+
await client.connect({
|
|
63764
|
+
cwd,
|
|
63765
|
+
model: "workflow",
|
|
63766
|
+
pid: process.pid,
|
|
63767
|
+
startedAt: Date.now(),
|
|
63768
|
+
lastActivity: Date.now(),
|
|
63769
|
+
group: connection.group,
|
|
63770
|
+
groups: [connection.group],
|
|
63771
|
+
recipientPurpose: "control"
|
|
63772
|
+
});
|
|
63773
|
+
if (connections.get(connection.key) !== connection)
|
|
63774
|
+
throw closedError();
|
|
63775
|
+
} catch (error) {
|
|
63776
|
+
drop(connection, client);
|
|
63777
|
+
throw error;
|
|
63778
|
+
}
|
|
63779
|
+
for (const state of connection.routes.values()) {
|
|
63780
|
+
if (state.registered || state.retry === undefined)
|
|
63781
|
+
continue;
|
|
63782
|
+
clearTimeout(state.retry);
|
|
63783
|
+
state.retry = undefined;
|
|
63784
|
+
publish(state, state.route).catch(() => {});
|
|
63785
|
+
}
|
|
63786
|
+
return client;
|
|
63787
|
+
})().finally(() => {
|
|
63788
|
+
connection.connecting = undefined;
|
|
63613
63789
|
});
|
|
63614
|
-
return
|
|
63790
|
+
return connection.connecting;
|
|
63615
63791
|
}
|
|
63616
63792
|
function publish(state, route) {
|
|
63617
|
-
const completion = state.tail.catch(() => {}).then(
|
|
63618
|
-
if (state
|
|
63619
|
-
throw
|
|
63620
|
-
const
|
|
63793
|
+
const completion = state.tail.catch(() => {}).then(async () => {
|
|
63794
|
+
if (!isCurrent(state))
|
|
63795
|
+
throw closedError();
|
|
63796
|
+
const connection = connectionOf(state);
|
|
63797
|
+
const established = connection.acknowledged && connection.client?.isConnected() ? connection.client : undefined;
|
|
63798
|
+
let client;
|
|
63621
63799
|
try {
|
|
63622
|
-
|
|
63800
|
+
client = await connect(connection);
|
|
63623
63801
|
await client.registerPendingStageRouteAcknowledged(route.runId, route.group, route.capability, route.stages, route.possibleStages, route.parent);
|
|
63802
|
+
if (connection.client !== client || !isCurrent(state))
|
|
63803
|
+
return;
|
|
63804
|
+
connection.acknowledged = true;
|
|
63624
63805
|
state.attempt = 0;
|
|
63625
63806
|
clearTimeout(state.retry);
|
|
63626
63807
|
state.retry = undefined;
|
|
63808
|
+
settle(state, true);
|
|
63627
63809
|
} catch (error) {
|
|
63628
|
-
|
|
63629
|
-
|
|
63630
|
-
|
|
63631
|
-
|
|
63810
|
+
if (error instanceof IntercomPendingStageRouteRefusedError && error.connectionKept) {
|
|
63811
|
+
scheduleRetry(state);
|
|
63812
|
+
throw error;
|
|
63813
|
+
}
|
|
63814
|
+
if (error instanceof IntercomPendingStageRouteRefusedError && !state.isolated && isCurrent(state)) {
|
|
63815
|
+
detach(state);
|
|
63816
|
+
state.isolated = true;
|
|
63817
|
+
connectionOf(state);
|
|
63632
63818
|
}
|
|
63819
|
+
if (client !== undefined && (client !== established || !client.isConnected()))
|
|
63820
|
+
drop(connection, client);
|
|
63633
63821
|
scheduleRetry(state);
|
|
63634
63822
|
throw error;
|
|
63635
63823
|
}
|
|
63636
|
-
})
|
|
63824
|
+
});
|
|
63637
63825
|
state.tail = completion;
|
|
63638
63826
|
completion.catch(() => {});
|
|
63639
63827
|
return completion;
|
|
@@ -63642,10 +63830,15 @@ function createWorkflowRouteOwner(cwd) {
|
|
|
63642
63830
|
const route = payload;
|
|
63643
63831
|
let state = routes.get(route.runId);
|
|
63644
63832
|
if (state === undefined) {
|
|
63645
|
-
state = { route, tail: Promise.resolve(), attempt: 0,
|
|
63833
|
+
state = { route, tail: Promise.resolve(), attempt: 0, registered: false, isolated: false, waiters: new Set };
|
|
63646
63834
|
routes.set(route.runId, state);
|
|
63835
|
+
} else if (state.route.group !== route.group) {
|
|
63836
|
+
detach(state);
|
|
63837
|
+
state.isolated = false;
|
|
63838
|
+
settle(state, false);
|
|
63647
63839
|
}
|
|
63648
63840
|
state.route = route;
|
|
63841
|
+
connectionOf(state);
|
|
63649
63842
|
route.completion = publish(state, route);
|
|
63650
63843
|
});
|
|
63651
63844
|
events.on("atomic:workflow-pending-stage-undeliverable", (payload) => {
|
|
@@ -63656,7 +63849,7 @@ function createWorkflowRouteOwner(cwd) {
|
|
|
63656
63849
|
event.handled = true;
|
|
63657
63850
|
event.completion = (async () => {
|
|
63658
63851
|
await publish(state, state.route);
|
|
63659
|
-
const client = state
|
|
63852
|
+
const client = connections.get(connectionKey(state))?.client;
|
|
63660
63853
|
if (client === undefined || !client.isConnected())
|
|
63661
63854
|
return false;
|
|
63662
63855
|
const text = `Pending workflow stage could not receive intercom message: ${event.reason}`;
|
|
@@ -63666,18 +63859,33 @@ function createWorkflowRouteOwner(cwd) {
|
|
|
63666
63859
|
});
|
|
63667
63860
|
return {
|
|
63668
63861
|
events,
|
|
63862
|
+
registered(runId) {
|
|
63863
|
+
const state = routes.get(runId);
|
|
63864
|
+
if (state === undefined)
|
|
63865
|
+
return Promise.resolve(false);
|
|
63866
|
+
if (state.registered)
|
|
63867
|
+
return Promise.resolve(true);
|
|
63868
|
+
return new Promise((resolve) => state.waiters.add(resolve));
|
|
63869
|
+
},
|
|
63669
63870
|
async closeGroup(group) {
|
|
63670
63871
|
const closing = [...routes.values()].filter((state) => state.route.group === group);
|
|
63872
|
+
const closingConnections = [...connections.values()].filter((connection) => connection.group === group);
|
|
63873
|
+
for (const connection of closingConnections) {
|
|
63874
|
+
connections.delete(connection.key);
|
|
63875
|
+
connection.routes.clear();
|
|
63876
|
+
}
|
|
63671
63877
|
for (const state of closing) {
|
|
63672
|
-
state.closed = true;
|
|
63673
|
-
clearTimeout(state.retry);
|
|
63674
63878
|
routes.delete(state.route.runId);
|
|
63879
|
+
clearTimeout(state.retry);
|
|
63880
|
+
state.retry = undefined;
|
|
63881
|
+
settle(state, false);
|
|
63675
63882
|
}
|
|
63676
|
-
await Promise.all(
|
|
63677
|
-
await
|
|
63678
|
-
await
|
|
63679
|
-
await state.client?.disconnect();
|
|
63883
|
+
await Promise.all(closingConnections.map(async (connection) => {
|
|
63884
|
+
await connection.client?.disconnect();
|
|
63885
|
+
await connection.connecting?.catch(() => {});
|
|
63680
63886
|
}));
|
|
63887
|
+
await Promise.all(closing.map((state) => state.tail.catch(() => {})));
|
|
63888
|
+
await Promise.all(closingConnections.map((connection) => connection.client?.disconnect()));
|
|
63681
63889
|
}
|
|
63682
63890
|
};
|
|
63683
63891
|
}
|
|
@@ -63860,7 +64068,7 @@ var init_workflow_infrastructure_error = __esm(() => {
|
|
|
63860
64068
|
[/Local worker (?:process )?cleanup (?:was not confirmed|unconfirmed)/, "worker_cleanup_unconfirmed"],
|
|
63861
64069
|
[/Local worker retry limit/, "worker_retry_limit"],
|
|
63862
64070
|
[/Local worker (?:startup timed out|exited before readiness)/, "worker_startup_failed"],
|
|
63863
|
-
[/Worker exited before delivery acknowledgement/, "worker_lost"],
|
|
64071
|
+
[/Worker exited before delivery acknowledgement|Local worker pool coordinator exited/, "worker_lost"],
|
|
63864
64072
|
[/Stale work lease/, "stale_work_lease"],
|
|
63865
64073
|
[/Intercom runtime no longer active/, "intercom_unavailable"],
|
|
63866
64074
|
[/Storage unavailable/, "storage_unavailable"]
|
|
@@ -64159,7 +64367,7 @@ var init_definitions = __esm(() => {
|
|
|
64159
64367
|
// dist/builtin/workflows/src/durable/journal/pool-mutations.ts
|
|
64160
64368
|
import { StorageProvider as StorageProvider2 } from "@bastani/atomic/storage";
|
|
64161
64369
|
async function validatePoolMutation(tx, journal, root, work, mutation) {
|
|
64162
|
-
if (!["checkpoint", "session-create", "session-
|
|
64370
|
+
if (!["checkpoint", "session-create", "session-replace"].includes(mutation.kind))
|
|
64163
64371
|
throw new Error("Invalid pool mutation");
|
|
64164
64372
|
const checkpoint = mutation.checkpoint;
|
|
64165
64373
|
const prefix = `${work.scope}:`;
|
|
@@ -64227,10 +64435,18 @@ async function sessionStore(tx, prefix) {
|
|
|
64227
64435
|
return new StorageProvider2().openSessionStore({ kind: "database", database, prefix }, { agentDir: "", domain: "sessions" });
|
|
64228
64436
|
}
|
|
64229
64437
|
async function applyPoolSession(tx, prefix, workId, mutation) {
|
|
64438
|
+
const owners = `${prefix}session_owners`;
|
|
64439
|
+
if (mutation.kind === "session-append") {
|
|
64440
|
+
if (typeof mutation.sessionId !== "string" || mutation.sessionId === "")
|
|
64441
|
+
throw new Error("Pool sessions require database identity");
|
|
64442
|
+
const owner = await tx.get(`SELECT work FROM ${owners} WHERE session = ?`, mutation.sessionId);
|
|
64443
|
+
if (owner?.work !== workId)
|
|
64444
|
+
throw new Error("Pool session scope mismatch");
|
|
64445
|
+
return withSessionStore(tx, prefix, (store) => store.append(mutation.sessionId, mutation.entries, { expectSeq: mutation.expectSeq }));
|
|
64446
|
+
}
|
|
64230
64447
|
const { sessionId, replayKey } = mutation.checkpoint;
|
|
64231
64448
|
if (!sessionId || mutation.checkpoint.sessionFile !== undefined)
|
|
64232
64449
|
throw new Error("Pool sessions require database identity");
|
|
64233
|
-
const owners = `${prefix}session_owners`;
|
|
64234
64450
|
if (mutation.kind === "session-create") {
|
|
64235
64451
|
if (mutation.header.id !== sessionId)
|
|
64236
64452
|
throw new Error("Pool session identity mismatch");
|
|
@@ -64240,10 +64456,12 @@ async function applyPoolSession(tx, prefix, workId, mutation) {
|
|
|
64240
64456
|
if (owner?.work !== workId || owner.replay_key !== replayKey)
|
|
64241
64457
|
throw new Error("Pool session scope mismatch");
|
|
64242
64458
|
}
|
|
64459
|
+
return withSessionStore(tx, prefix, (store) => mutation.kind === "session-create" ? store.create(mutation.header, mutation.entries) : store.replace(sessionId, mutation.header, mutation.entries, { expectSeq: mutation.expectSeq }));
|
|
64460
|
+
}
|
|
64461
|
+
async function withSessionStore(tx, prefix, write) {
|
|
64243
64462
|
const store = await sessionStore(tx, prefix);
|
|
64244
64463
|
try {
|
|
64245
|
-
|
|
64246
|
-
return result.lastSeq;
|
|
64464
|
+
return (await write(store)).lastSeq;
|
|
64247
64465
|
} finally {
|
|
64248
64466
|
await store.close();
|
|
64249
64467
|
}
|
|
@@ -64291,8 +64509,11 @@ var init_pool_stage_usage = __esm(() => {
|
|
|
64291
64509
|
});
|
|
64292
64510
|
|
|
64293
64511
|
// dist/builtin/workflows/src/durable/journal/sql-pool-queue.ts
|
|
64294
|
-
import { randomUUID as randomUUID16 } from "node:crypto";
|
|
64512
|
+
import { createHash as createHash4, randomUUID as randomUUID16 } from "node:crypto";
|
|
64295
64513
|
import { assertStorageSchemaVersion as assertStorageSchemaVersion2 } from "@bastani/atomic/storage";
|
|
64514
|
+
function requestIdentity(request) {
|
|
64515
|
+
return createHash4("sha256").update(request).digest("hex");
|
|
64516
|
+
}
|
|
64296
64517
|
function fromRow(row) {
|
|
64297
64518
|
return {
|
|
64298
64519
|
...JSON.parse(row.descriptor),
|
|
@@ -64348,10 +64569,11 @@ class SqlPoolQueue {
|
|
|
64348
64569
|
const versions = await tx.all(`SELECT version FROM ${pool.meta}`);
|
|
64349
64570
|
if (versions.length > 1)
|
|
64350
64571
|
throw new Error("Corrupt pool schema metadata");
|
|
64351
|
-
|
|
64352
|
-
|
|
64353
|
-
|
|
64354
|
-
|
|
64572
|
+
const version = versions[0]?.version;
|
|
64573
|
+
if (version === undefined)
|
|
64574
|
+
await tx.run(`INSERT INTO ${pool.meta} VALUES (?)`, POOL_SCHEMA_VERSION);
|
|
64575
|
+
else if (version !== 1)
|
|
64576
|
+
assertStorageSchemaVersion2("journal", version, POOL_SCHEMA_VERSION);
|
|
64355
64577
|
await tx.exec(`
|
|
64356
64578
|
CREATE TABLE IF NOT EXISTS ${pool.roots}(root TEXT PRIMARY KEY REFERENCES ${journal.runs}(run_id) ON DELETE CASCADE, timeout INTEGER NOT NULL);
|
|
64357
64579
|
CREATE TABLE IF NOT EXISTS ${pool.work}(
|
|
@@ -64382,9 +64604,23 @@ class SqlPoolQueue {
|
|
|
64382
64604
|
state TEXT NOT NULL DEFAULT 'running', revision INTEGER NOT NULL DEFAULT 0
|
|
64383
64605
|
)`);
|
|
64384
64606
|
await tx.exec(poolUsageSchema(journal.runs));
|
|
64607
|
+
if (version === 1)
|
|
64608
|
+
await pool.migrateFromVersion1(tx);
|
|
64385
64609
|
});
|
|
64386
64610
|
return pool;
|
|
64387
64611
|
}
|
|
64612
|
+
async migrateFromVersion1(tx) {
|
|
64613
|
+
await tx.run(`DELETE FROM ${this.operations} WHERE substr(request, 1, 1) = '['`);
|
|
64614
|
+
for (let after = 0;; ) {
|
|
64615
|
+
const rows = await tx.all(`SELECT rowid, request FROM ${this.operations} WHERE rowid > ? ORDER BY rowid LIMIT 256`, after);
|
|
64616
|
+
if (rows.length === 0)
|
|
64617
|
+
break;
|
|
64618
|
+
for (const row of rows)
|
|
64619
|
+
await tx.run(`UPDATE ${this.operations} SET request = ? WHERE rowid = ?`, requestIdentity(String(row.request)), row.rowid);
|
|
64620
|
+
after = Number(rows.at(-1).rowid);
|
|
64621
|
+
}
|
|
64622
|
+
await tx.run(`UPDATE ${this.meta} SET version = ?`, POOL_SCHEMA_VERSION);
|
|
64623
|
+
}
|
|
64388
64624
|
async authority(tx, lease, timeout) {
|
|
64389
64625
|
const row = await tx.get(`SELECT l.*, r.status, p.timeout, ${STORE_NOW2} AS now FROM ${this.journal.leases} l JOIN ${this.journal.runs} r ON r.run_id = l.run_id LEFT JOIN ${this.roots} p ON p.root = l.run_id WHERE l.run_id = ?`, lease.runId);
|
|
64390
64626
|
if (row?.timeout === null && timeout === undefined)
|
|
@@ -64525,31 +64761,18 @@ class SqlPoolQueue {
|
|
|
64525
64761
|
throw new Error(STALE_WORK_LEASE);
|
|
64526
64762
|
return row;
|
|
64527
64763
|
}
|
|
64528
|
-
async renew(lease, token,
|
|
64764
|
+
async renew(lease, token, durationMs) {
|
|
64529
64765
|
positive(durationMs);
|
|
64530
64766
|
lease = snapshotLease(lease);
|
|
64531
64767
|
token = { ...token };
|
|
64532
|
-
validateText("key", operationId);
|
|
64533
|
-
const request = JSON.stringify([token.epoch, token.generation, token.worker, durationMs]);
|
|
64534
64768
|
return this.transaction(async (tx) => {
|
|
64535
64769
|
const { now } = await this.authority(tx, lease);
|
|
64536
64770
|
const row = await this.current(tx, lease, token);
|
|
64537
64771
|
if (row.epoch !== lease.epoch || row.expires_at < now)
|
|
64538
64772
|
throw new Error(STALE_WORK_LEASE);
|
|
64539
|
-
const
|
|
64540
|
-
|
|
64541
|
-
|
|
64542
|
-
throw new Error("Pool operation identity conflict");
|
|
64543
|
-
return JSON.parse(previous.receipt);
|
|
64544
|
-
}
|
|
64545
|
-
const receipt = {
|
|
64546
|
-
...fromRow(row),
|
|
64547
|
-
expiresAt: Math.max(Number(row.expires_at), expiry(now, durationMs)),
|
|
64548
|
-
renewedAt: now
|
|
64549
|
-
};
|
|
64550
|
-
await tx.run(`UPDATE ${this.work} SET expires_at = ?, renewed_at = ? WHERE id = ?`, receipt.expiresAt, now, row.id);
|
|
64551
|
-
await tx.run(`INSERT INTO ${this.operations} VALUES (?, ?, ?, ?, ?)`, row.id, token.generation, operationId, request, JSON.stringify(receipt));
|
|
64552
|
-
return receipt;
|
|
64773
|
+
const expiresAt = Math.max(Number(row.expires_at), expiry(now, durationMs));
|
|
64774
|
+
await tx.run(`UPDATE ${this.work} SET expires_at = ?, renewed_at = ? WHERE id = ?`, expiresAt, now, row.id);
|
|
64775
|
+
return { ...fromRow(row), expiresAt, renewedAt: now };
|
|
64553
64776
|
});
|
|
64554
64777
|
}
|
|
64555
64778
|
async commit(lease, token, operationId, mutation) {
|
|
@@ -64558,6 +64781,7 @@ class SqlPoolQueue {
|
|
|
64558
64781
|
validateText("key", operationId);
|
|
64559
64782
|
assertWorkflowSerializableObject(mutation, "Pool mutation");
|
|
64560
64783
|
const request = serializeJson({ epoch: token.epoch, worker: token.worker, mutation });
|
|
64784
|
+
const identity = requestIdentity(request);
|
|
64561
64785
|
mutation = JSON.parse(request).mutation;
|
|
64562
64786
|
return this.transaction(async (tx) => {
|
|
64563
64787
|
const { now, status } = await this.authority(tx, lease);
|
|
@@ -64566,7 +64790,7 @@ class SqlPoolQueue {
|
|
|
64566
64790
|
throw new Error(STALE_WORK_LEASE);
|
|
64567
64791
|
const previous = await tx.get(`SELECT * FROM ${this.operations} WHERE work = ? AND generation = ? AND operation = ?`, token.id, token.generation, operationId);
|
|
64568
64792
|
if (previous) {
|
|
64569
|
-
if (previous.request !==
|
|
64793
|
+
if (previous.request !== identity)
|
|
64570
64794
|
throw new Error("Pool operation identity conflict");
|
|
64571
64795
|
return JSON.parse(previous.receipt);
|
|
64572
64796
|
}
|
|
@@ -64574,34 +64798,41 @@ class SqlPoolQueue {
|
|
|
64574
64798
|
throw new Error(STALE_WORK_LEASE);
|
|
64575
64799
|
if (!await this.admitsAttemptWrites(tx, lease, status))
|
|
64576
64800
|
throw new Error("Pool root is not running");
|
|
64577
|
-
const
|
|
64578
|
-
await
|
|
64579
|
-
if (checkpoint.completedAt > Math.max(now, Date.now()) + MAX_WORKER_CLOCK_SKEW_MS)
|
|
64580
|
-
throw new Error("Pool checkpoint timestamp is in the future");
|
|
64581
|
-
assertWorkflowSerializableObject(checkpoint, "Pool checkpoint");
|
|
64582
|
-
const key = JSON.stringify([RUN_EVENT_TYPES.checkpointRecorded, checkpoint.kind, checkpoint.checkpointId]);
|
|
64583
|
-
const payload = serializeJson(checkpoint);
|
|
64584
|
-
const existing = await tx.get(`SELECT seq, at, payload FROM ${this.journal.events} WHERE run_id = ? AND key = ?`, lease.runId, key);
|
|
64585
|
-
if (existing && existing.payload !== payload)
|
|
64586
|
-
throw new Error("Pool checkpoint identity conflict");
|
|
64587
|
-
if (existing && mutation.kind !== "checkpoint")
|
|
64588
|
-
throw new Error("Pool session checkpoint identity conflict");
|
|
64589
|
-
const lastSeq = mutation.kind === "checkpoint" ? undefined : await applyPoolSession(tx, `${this.journal.runs}_pool_`, row.id, mutation);
|
|
64590
|
-
const latest = await tx.get(`SELECT COALESCE(MAX(seq), 0) AS seq FROM ${this.journal.events} WHERE run_id = ?`, lease.runId);
|
|
64591
|
-
const receipt = existing ? { seq: Number(existing.seq), at: Number(existing.at) } : {
|
|
64592
|
-
seq: journalCounter(lease.runId, "seq", Number(latest.seq) + 1, "overflow"),
|
|
64593
|
-
at: now,
|
|
64594
|
-
...lastSeq === undefined ? {} : { lastSeq }
|
|
64595
|
-
};
|
|
64596
|
-
if (!existing)
|
|
64597
|
-
await tx.run(`INSERT INTO ${this.journal.events} VALUES (?, ?, ?, ?, ?, ?, ?)`, lease.runId, receipt.seq, lease.epoch, key, RUN_EVENT_TYPES.checkpointRecorded, payload, now);
|
|
64598
|
-
const billed = poolCheckpointUsage(checkpoint);
|
|
64599
|
-
if (billed && !existing)
|
|
64600
|
-
await upsertPoolUsage(tx, this.journal.runs, row.id, token.generation, billed.key, billed.usage);
|
|
64601
|
-
await tx.run(`INSERT INTO ${this.operations} VALUES (?, ?, ?, ?, ?)`, token.id, token.generation, operationId, request, JSON.stringify(receipt));
|
|
64801
|
+
const receipt = mutation.kind === "session-append" ? await this.appendSession(tx, row, mutation, now) : await this.recordCheckpoint(tx, lease, token, row, mutation, now);
|
|
64802
|
+
await tx.run(`INSERT INTO ${this.operations} VALUES (?, ?, ?, ?, ?)`, token.id, token.generation, operationId, identity, JSON.stringify(receipt));
|
|
64602
64803
|
return receipt;
|
|
64603
64804
|
});
|
|
64604
64805
|
}
|
|
64806
|
+
async appendSession(tx, row, mutation, now) {
|
|
64807
|
+
return { at: now, lastSeq: await applyPoolSession(tx, `${this.journal.runs}_pool_`, row.id, mutation) };
|
|
64808
|
+
}
|
|
64809
|
+
async recordCheckpoint(tx, lease, token, row, mutation, now) {
|
|
64810
|
+
const checkpoint = mutation.checkpoint;
|
|
64811
|
+
await validatePoolMutation(tx, this.journal, lease.runId, fromRow(row), mutation);
|
|
64812
|
+
if (checkpoint.completedAt > Math.max(now, Date.now()) + MAX_WORKER_CLOCK_SKEW_MS)
|
|
64813
|
+
throw new Error("Pool checkpoint timestamp is in the future");
|
|
64814
|
+
assertWorkflowSerializableObject(checkpoint, "Pool checkpoint");
|
|
64815
|
+
const key = JSON.stringify([RUN_EVENT_TYPES.checkpointRecorded, checkpoint.kind, checkpoint.checkpointId]);
|
|
64816
|
+
const payload = serializeJson(checkpoint);
|
|
64817
|
+
const existing = await tx.get(`SELECT seq, at, payload FROM ${this.journal.events} WHERE run_id = ? AND key = ?`, lease.runId, key);
|
|
64818
|
+
if (existing && existing.payload !== payload)
|
|
64819
|
+
throw new Error("Pool checkpoint identity conflict");
|
|
64820
|
+
if (existing && mutation.kind !== "checkpoint")
|
|
64821
|
+
throw new Error("Pool session checkpoint identity conflict");
|
|
64822
|
+
const lastSeq = mutation.kind === "checkpoint" ? undefined : await applyPoolSession(tx, `${this.journal.runs}_pool_`, row.id, mutation);
|
|
64823
|
+
const latest = await tx.get(`SELECT COALESCE(MAX(seq), 0) AS seq FROM ${this.journal.events} WHERE run_id = ?`, lease.runId);
|
|
64824
|
+
const receipt = existing ? { seq: Number(existing.seq), at: Number(existing.at) } : {
|
|
64825
|
+
seq: journalCounter(lease.runId, "seq", Number(latest.seq) + 1, "overflow"),
|
|
64826
|
+
at: now,
|
|
64827
|
+
...lastSeq === undefined ? {} : { lastSeq }
|
|
64828
|
+
};
|
|
64829
|
+
if (!existing)
|
|
64830
|
+
await tx.run(`INSERT INTO ${this.journal.events} VALUES (?, ?, ?, ?, ?, ?, ?)`, lease.runId, receipt.seq, lease.epoch, key, RUN_EVENT_TYPES.checkpointRecorded, payload, now);
|
|
64831
|
+
const billed = poolCheckpointUsage(checkpoint);
|
|
64832
|
+
if (billed && !existing)
|
|
64833
|
+
await upsertPoolUsage(tx, this.journal.runs, row.id, token.generation, billed.key, billed.usage);
|
|
64834
|
+
return receipt;
|
|
64835
|
+
}
|
|
64605
64836
|
async complete(lease, token, result) {
|
|
64606
64837
|
lease = snapshotLease(lease);
|
|
64607
64838
|
token = { ...token };
|
|
@@ -64688,7 +64919,7 @@ function openSqlPoolQueue(journal) {
|
|
|
64688
64919
|
return Promise.reject(new Error("Worker pool requires a SQL run journal"));
|
|
64689
64920
|
return SqlPoolQueue.open(storage);
|
|
64690
64921
|
}
|
|
64691
|
-
var STORE_NOW2 = "CAST(unixepoch('subsec') * 1000 AS INTEGER)", MAX_WORKER_CLOCK_SKEW_MS = 1000, STALE_WORK_LEASE = "Stale work lease";
|
|
64922
|
+
var STORE_NOW2 = "CAST(unixepoch('subsec') * 1000 AS INTEGER)", MAX_WORKER_CLOCK_SKEW_MS = 1000, POOL_SCHEMA_VERSION = 2, STALE_WORK_LEASE = "Stale work lease";
|
|
64692
64923
|
var init_sql_pool_queue = __esm(() => {
|
|
64693
64924
|
init_serializable();
|
|
64694
64925
|
init_boundary_topology();
|
|
@@ -64700,6 +64931,15 @@ var init_sql_pool_queue = __esm(() => {
|
|
|
64700
64931
|
init_values();
|
|
64701
64932
|
});
|
|
64702
64933
|
|
|
64934
|
+
// dist/builtin/workflows/src/workers/coordinator-protocol.ts
|
|
64935
|
+
var POOL_COORDINATOR_ROLE = "pool-coordinator", POOL_SUMMARY_INTERVAL_MS = 250;
|
|
64936
|
+
|
|
64937
|
+
// dist/builtin/workflows/src/workers/host-options.ts
|
|
64938
|
+
var hostWorkerBindings;
|
|
64939
|
+
var init_host_options = __esm(() => {
|
|
64940
|
+
hostWorkerBindings = Symbol("host-worker-bindings");
|
|
64941
|
+
});
|
|
64942
|
+
|
|
64703
64943
|
// dist/builtin/workflows/src/durable/journal/sql-pool-usage.ts
|
|
64704
64944
|
function validate(report) {
|
|
64705
64945
|
validateText("key", report.key);
|
|
@@ -64770,10 +65010,844 @@ var init_sql_pool_usage = __esm(() => {
|
|
|
64770
65010
|
init_values();
|
|
64771
65011
|
});
|
|
64772
65012
|
|
|
64773
|
-
// dist/builtin/workflows/src/workers/
|
|
64774
|
-
|
|
64775
|
-
|
|
64776
|
-
|
|
65013
|
+
// dist/builtin/workflows/src/workers/pool-coordinator.ts
|
|
65014
|
+
import { fork } from "node:child_process";
|
|
65015
|
+
import { randomUUID as randomUUID17 } from "node:crypto";
|
|
65016
|
+
import { createWriteStream } from "node:fs";
|
|
65017
|
+
import { mkdir as mkdir6, rm as rm3, rmdir } from "node:fs/promises";
|
|
65018
|
+
import { dirname as dirname14, join as join29, resolve as resolve13 } from "node:path";
|
|
65019
|
+
import { setTimeout as delay2 } from "node:timers/promises";
|
|
65020
|
+
function workerLogDirectory(journalPath, root) {
|
|
65021
|
+
return join29(dirname14(resolve13(journalPath)), "worker-logs", encodeURIComponent(root).replaceAll(".", "%2E"));
|
|
65022
|
+
}
|
|
65023
|
+
function token(work) {
|
|
65024
|
+
return {
|
|
65025
|
+
id: work.id,
|
|
65026
|
+
generation: work.generation,
|
|
65027
|
+
worker: work.worker,
|
|
65028
|
+
epoch: work.epoch,
|
|
65029
|
+
renewedAt: work.renewedAt,
|
|
65030
|
+
expiresAt: work.expiresAt
|
|
65031
|
+
};
|
|
65032
|
+
}
|
|
65033
|
+
function outside(runId) {
|
|
65034
|
+
return new Error(`Pool stage publication is outside the work subtree: ${runId}`);
|
|
65035
|
+
}
|
|
65036
|
+
function isLiveStage(stage) {
|
|
65037
|
+
return (stage.sessionId !== undefined || stage.sessionFile !== undefined) && stage.status !== "completed" && stage.status !== "failed" && stage.status !== "skipped";
|
|
65038
|
+
}
|
|
65039
|
+
function projectRootRun(run) {
|
|
65040
|
+
return {
|
|
65041
|
+
id: run.id,
|
|
65042
|
+
name: run.name,
|
|
65043
|
+
status: run.status,
|
|
65044
|
+
startedAt: run.startedAt,
|
|
65045
|
+
stages: run.stages.map((stage) => ({
|
|
65046
|
+
id: stage.id,
|
|
65047
|
+
name: stage.name,
|
|
65048
|
+
status: stage.status,
|
|
65049
|
+
parentIds: [],
|
|
65050
|
+
toolEvents: [],
|
|
65051
|
+
...stage.replayKey === undefined ? {} : { replayKey: stage.replayKey },
|
|
65052
|
+
...stage.intercomGroup === undefined ? {} : { intercomGroup: stage.intercomGroup },
|
|
65053
|
+
...stage.workflowChildRun === undefined ? {} : { workflowChildRun: stage.workflowChildRun },
|
|
65054
|
+
...stage.workflowChild === undefined ? {} : { workflowChild: { ...stage.workflowChild, outputs: {} } }
|
|
65055
|
+
}))
|
|
65056
|
+
};
|
|
65057
|
+
}
|
|
65058
|
+
|
|
65059
|
+
class PooledRoutes {
|
|
65060
|
+
root;
|
|
65061
|
+
owner;
|
|
65062
|
+
mirror = createStore();
|
|
65063
|
+
owners = new Map;
|
|
65064
|
+
attempts = new Map;
|
|
65065
|
+
activated = false;
|
|
65066
|
+
bridge;
|
|
65067
|
+
constructor(root, owner) {
|
|
65068
|
+
this.root = root;
|
|
65069
|
+
this.owner = owner;
|
|
65070
|
+
}
|
|
65071
|
+
static async open(root, cwd, rootRun, forward) {
|
|
65072
|
+
await Promise.resolve().then(() => init_workflow_route_owner());
|
|
65073
|
+
const owner = createWorkflowRouteOwner(cwd);
|
|
65074
|
+
const routes = new PooledRoutes(root, owner);
|
|
65075
|
+
if (rootRun !== undefined)
|
|
65076
|
+
routes.updateRoot(rootRun);
|
|
65077
|
+
routes.bridge = registerPendingStageIntercomBridge({
|
|
65078
|
+
events: {
|
|
65079
|
+
emit: (event, payload) => owner.events.emit(event, payload),
|
|
65080
|
+
on: (event, listener) => event === PENDING_STAGE_MESSAGE_EVENT2 || event === STICKY_LIVE_DELIVERED_EVENT2 ? undefined : owner.events.on(event, listener)
|
|
65081
|
+
}
|
|
65082
|
+
}, routes.mirror, (runId) => routes.activated && routes.owners.has(runId));
|
|
65083
|
+
for (const [event, kind] of [
|
|
65084
|
+
[PENDING_STAGE_MESSAGE_EVENT2, "pending-stage-message"],
|
|
65085
|
+
[STICKY_LIVE_DELIVERED_EVENT2, "sticky-live-delivered"]
|
|
65086
|
+
])
|
|
65087
|
+
owner.events.on(event, (payload) => {
|
|
65088
|
+
const claimed = payload;
|
|
65089
|
+
if (claimed.handled !== false)
|
|
65090
|
+
return;
|
|
65091
|
+
claimed.handled = true;
|
|
65092
|
+
const { handled: _handled, completion: _completion, ...forwarded } = claimed;
|
|
65093
|
+
claimed.completion = forward({ kind, payload: forwarded });
|
|
65094
|
+
});
|
|
65095
|
+
return routes;
|
|
65096
|
+
}
|
|
65097
|
+
updateRoot(rootRun) {
|
|
65098
|
+
this.mirror.recordRunStart({ ...rootRun, inputs: {}, toolNodes: [], rootRunId: this.root });
|
|
65099
|
+
}
|
|
65100
|
+
accept(entries) {
|
|
65101
|
+
this.mirror.batch(() => {
|
|
65102
|
+
for (const { work, runs } of entries) {
|
|
65103
|
+
let attempt = this.attempts.get(work.id);
|
|
65104
|
+
if (attempt !== undefined && work.generation < attempt.generation)
|
|
65105
|
+
continue;
|
|
65106
|
+
if (attempt === undefined || attempt.generation !== work.generation) {
|
|
65107
|
+
if (attempt !== undefined)
|
|
65108
|
+
this.release(attempt);
|
|
65109
|
+
attempt = { generation: work.generation, runs: attempt?.runs ?? new Set, fenced: false };
|
|
65110
|
+
this.attempts.set(work.id, attempt);
|
|
65111
|
+
}
|
|
65112
|
+
if (attempt.fenced)
|
|
65113
|
+
continue;
|
|
65114
|
+
for (const run of runs) {
|
|
65115
|
+
attempt.runs.add(run.id);
|
|
65116
|
+
this.owners.set(run.id, work.id);
|
|
65117
|
+
const { pendingStageMessages: _messages, ...published } = run;
|
|
65118
|
+
this.mirror.recordRunStart({ ...published, rootRunId: this.root });
|
|
65119
|
+
if (run.stages.some((stage) => stage.pendingStageDeliveryAvailable === true))
|
|
65120
|
+
this.activated = true;
|
|
65121
|
+
}
|
|
65122
|
+
}
|
|
65123
|
+
});
|
|
65124
|
+
}
|
|
65125
|
+
fence(work) {
|
|
65126
|
+
const attempt = this.attempts.get(work.id);
|
|
65127
|
+
if (attempt === undefined || attempt.generation > work.generation)
|
|
65128
|
+
return;
|
|
65129
|
+
if (attempt.generation < work.generation) {
|
|
65130
|
+
this.release(attempt);
|
|
65131
|
+
this.attempts.set(work.id, { generation: work.generation, runs: attempt.runs, fenced: true });
|
|
65132
|
+
return;
|
|
65133
|
+
}
|
|
65134
|
+
this.release(attempt);
|
|
65135
|
+
attempt.fenced = true;
|
|
65136
|
+
}
|
|
65137
|
+
async route(work, runId) {
|
|
65138
|
+
this.requireOwned(work, runId);
|
|
65139
|
+
const readiness = workflowPendingStageRouteReady(this.mirror, runId);
|
|
65140
|
+
await readiness?.completion.catch(() => {});
|
|
65141
|
+
readiness?.assertCurrent();
|
|
65142
|
+
const waiting = new AbortController;
|
|
65143
|
+
await Promise.race([
|
|
65144
|
+
this.owner.registered(runId),
|
|
65145
|
+
delay2(ROUTE_REGISTRATION_WAIT_MS, undefined, { signal: waiting.signal, ref: false }).catch(() => {})
|
|
65146
|
+
]);
|
|
65147
|
+
waiting.abort();
|
|
65148
|
+
this.requireOwned(work, runId);
|
|
65149
|
+
return workflowPendingStageRouteCapability(this.mirror, runId);
|
|
65150
|
+
}
|
|
65151
|
+
async undeliverable(payload) {
|
|
65152
|
+
const event = { ...payload, handled: false };
|
|
65153
|
+
this.owner.events.emit(PENDING_STAGE_UNDELIVERABLE_EVENT2, event);
|
|
65154
|
+
return event.handled && await event.completion === true;
|
|
65155
|
+
}
|
|
65156
|
+
async close() {
|
|
65157
|
+
const bridge = this.bridge;
|
|
65158
|
+
if (bridge === undefined)
|
|
65159
|
+
return;
|
|
65160
|
+
this.bridge = undefined;
|
|
65161
|
+
try {
|
|
65162
|
+
await bridge.flush();
|
|
65163
|
+
} finally {
|
|
65164
|
+
bridge.refresh();
|
|
65165
|
+
bridge();
|
|
65166
|
+
await this.owner.closeGroup(workflowInvocationIntercomGroup(this.root));
|
|
65167
|
+
}
|
|
65168
|
+
}
|
|
65169
|
+
requireOwned(work, runId) {
|
|
65170
|
+
const attempt = this.attempts.get(work.id);
|
|
65171
|
+
if (attempt === undefined || attempt.generation !== work.generation || attempt.fenced)
|
|
65172
|
+
throw new Error(STALE_WORK_LEASE);
|
|
65173
|
+
if (!attempt.runs.has(runId))
|
|
65174
|
+
throw outside(runId);
|
|
65175
|
+
}
|
|
65176
|
+
release(attempt) {
|
|
65177
|
+
this.mirror.batch(() => {
|
|
65178
|
+
for (const id of attempt.runs) {
|
|
65179
|
+
rotateWorkflowPendingStageRouteCapability(this.mirror, id);
|
|
65180
|
+
const run = this.mirror.runs().find((candidate) => candidate.id === id);
|
|
65181
|
+
if (run?.stages.some(isLiveStage) !== true)
|
|
65182
|
+
continue;
|
|
65183
|
+
this.mirror.recordRunStart({
|
|
65184
|
+
...run,
|
|
65185
|
+
stages: run.stages.map((stage) => {
|
|
65186
|
+
if (!isLiveStage(stage))
|
|
65187
|
+
return stage;
|
|
65188
|
+
const { sessionId: _session, sessionFile: _file, ...rest } = stage;
|
|
65189
|
+
return { ...rest, attachable: false, attached: false };
|
|
65190
|
+
})
|
|
65191
|
+
});
|
|
65192
|
+
}
|
|
65193
|
+
});
|
|
65194
|
+
}
|
|
65195
|
+
}
|
|
65196
|
+
|
|
65197
|
+
class PoolCoordinator {
|
|
65198
|
+
port;
|
|
65199
|
+
children = new Set;
|
|
65200
|
+
incarnations = new Map;
|
|
65201
|
+
readyWorkers = new Set;
|
|
65202
|
+
logs = new Map;
|
|
65203
|
+
cleanups = new Set;
|
|
65204
|
+
controlAcks = new Map;
|
|
65205
|
+
lastHeard = new Map;
|
|
65206
|
+
pendingActivity = new Map;
|
|
65207
|
+
batches = new Map;
|
|
65208
|
+
publicationFailures = new Map;
|
|
65209
|
+
requests = new Map;
|
|
65210
|
+
deliveryRequests = new Map;
|
|
65211
|
+
inflight = new Set;
|
|
65212
|
+
lastTick = performance.now();
|
|
65213
|
+
stalledMs = 0;
|
|
65214
|
+
ticker;
|
|
65215
|
+
recoveryTimer;
|
|
65216
|
+
flushTimer;
|
|
65217
|
+
lastFlushAt = 0;
|
|
65218
|
+
batchSequence = 0;
|
|
65219
|
+
requestSequence = 0;
|
|
65220
|
+
usageChanged = false;
|
|
65221
|
+
budgetBlocked = false;
|
|
65222
|
+
desiredControl;
|
|
65223
|
+
root;
|
|
65224
|
+
lease;
|
|
65225
|
+
options;
|
|
65226
|
+
journal;
|
|
65227
|
+
queue;
|
|
65228
|
+
usageLedger;
|
|
65229
|
+
usageTotal;
|
|
65230
|
+
admission;
|
|
65231
|
+
routes;
|
|
65232
|
+
publishing = false;
|
|
65233
|
+
sources = [];
|
|
65234
|
+
logDirectory;
|
|
65235
|
+
starting;
|
|
65236
|
+
recovering;
|
|
65237
|
+
nextRecoveryAt = 0;
|
|
65238
|
+
cleanupFailure;
|
|
65239
|
+
closed = false;
|
|
65240
|
+
closing;
|
|
65241
|
+
replacements = 0;
|
|
65242
|
+
constructor(port) {
|
|
65243
|
+
this.port = port;
|
|
65244
|
+
port.onMessage((message) => this.receive(message));
|
|
65245
|
+
this.ticker = setInterval(() => this.observeResponsiveness(), TICK_INTERVAL_MS).unref();
|
|
65246
|
+
}
|
|
65247
|
+
async shutdown() {
|
|
65248
|
+
await this.closing?.catch(() => {});
|
|
65249
|
+
while (this.inflight.size > 0)
|
|
65250
|
+
await Promise.allSettled([...this.inflight]);
|
|
65251
|
+
await this.journal?.close().catch(() => {});
|
|
65252
|
+
}
|
|
65253
|
+
receive(message) {
|
|
65254
|
+
if (message.kind === "request-reply") {
|
|
65255
|
+
const pending = this.requests.get(message.id);
|
|
65256
|
+
this.requests.delete(message.id);
|
|
65257
|
+
if (message.error !== undefined)
|
|
65258
|
+
pending?.reject(new Error(message.error));
|
|
65259
|
+
else
|
|
65260
|
+
pending?.resolve(message.value);
|
|
65261
|
+
return;
|
|
65262
|
+
}
|
|
65263
|
+
if ("id" in message) {
|
|
65264
|
+
const command = message;
|
|
65265
|
+
const handled = this.execute(command).then((value2) => this.port.send({ kind: "reply", id: command.id, value: value2 }), (error) => this.port.send({ kind: "reply", id: command.id, error: error.message }));
|
|
65266
|
+
this.inflight.add(handled);
|
|
65267
|
+
handled.finally(() => this.inflight.delete(handled));
|
|
65268
|
+
return;
|
|
65269
|
+
}
|
|
65270
|
+
this.notice(message);
|
|
65271
|
+
}
|
|
65272
|
+
notice(notice) {
|
|
65273
|
+
if (notice.kind === "budget")
|
|
65274
|
+
this.budgetBlocked = notice.blocked;
|
|
65275
|
+
else if (notice.kind === "root")
|
|
65276
|
+
this.routes?.updateRoot(notice.rootRun);
|
|
65277
|
+
else {
|
|
65278
|
+
const batch = this.batches.get(notice.batch);
|
|
65279
|
+
this.batches.delete(notice.batch);
|
|
65280
|
+
const rejected = new Map(notice.rejected.map((entry) => [entry.workId, entry]));
|
|
65281
|
+
for (const entry of rejected.values())
|
|
65282
|
+
this.publicationFailures.set(entry.workId, { generation: entry.generation, error: entry.error });
|
|
65283
|
+
this.routes?.accept((batch?.entries ?? []).filter((entry) => rejected.get(entry.work.id)?.generation !== entry.work.generation));
|
|
65284
|
+
batch?.settle();
|
|
65285
|
+
}
|
|
65286
|
+
}
|
|
65287
|
+
async execute(command) {
|
|
65288
|
+
switch (command.kind) {
|
|
65289
|
+
case "prepare":
|
|
65290
|
+
return this.prepare(command.root, command.lease, command.options, command.control);
|
|
65291
|
+
case "control":
|
|
65292
|
+
return this.control(command.state);
|
|
65293
|
+
case "start":
|
|
65294
|
+
this.starting ??= this.start(command.sources, command.cwd, command.publish, command.rootRun);
|
|
65295
|
+
return this.starting;
|
|
65296
|
+
case "enqueue":
|
|
65297
|
+
return this.enqueue(command);
|
|
65298
|
+
case "transport":
|
|
65299
|
+
return this.transport(command.request, command);
|
|
65300
|
+
case "undeliverable":
|
|
65301
|
+
return await this.routes?.undeliverable(command.payload) ?? false;
|
|
65302
|
+
case "close":
|
|
65303
|
+
this.closing ??= this.close();
|
|
65304
|
+
return this.closing;
|
|
65305
|
+
}
|
|
65306
|
+
}
|
|
65307
|
+
async prepare(root, lease, options, control) {
|
|
65308
|
+
if (this.closed)
|
|
65309
|
+
throw new Error("Local worker pool is closed");
|
|
65310
|
+
if (this.root !== undefined && this.root !== root)
|
|
65311
|
+
throw new Error("Local worker pool belongs to another root");
|
|
65312
|
+
this.root = root;
|
|
65313
|
+
this.options = options;
|
|
65314
|
+
this.desiredControl ??= control;
|
|
65315
|
+
this.journal ??= await resolveRunJournal({ journal: { path: options.journalPath } });
|
|
65316
|
+
this.usageLedger ??= await openSqlPoolUsage(this.journal);
|
|
65317
|
+
this.usageTotal = await this.usageLedger.total(root);
|
|
65318
|
+
this.lease = lease;
|
|
65319
|
+
const queue = await openSqlPoolQueue(this.journal);
|
|
65320
|
+
await queue.initialize(lease, COORDINATOR_LEASE_MS);
|
|
65321
|
+
await queue.releasePreviousIncarnations(lease);
|
|
65322
|
+
for (;; ) {
|
|
65323
|
+
const intended = this.desiredControl ?? "running";
|
|
65324
|
+
await queue.setControl(lease, intended);
|
|
65325
|
+
if (intended === (this.desiredControl ?? "running"))
|
|
65326
|
+
break;
|
|
65327
|
+
}
|
|
65328
|
+
this.queue = queue;
|
|
65329
|
+
return this.usageTotal;
|
|
65330
|
+
}
|
|
65331
|
+
async control(state) {
|
|
65332
|
+
this.desiredControl = state;
|
|
65333
|
+
if (!this.queue || !this.lease)
|
|
65334
|
+
return;
|
|
65335
|
+
const control = await this.queue.setControl(this.lease, state);
|
|
65336
|
+
const deadline = Date.now() + CONTROL_ACKNOWLEDGEMENT_TIMEOUT_MS;
|
|
65337
|
+
while ([...this.readyWorkers].some((worker) => (this.controlAcks.get(worker) ?? -1) < control.revision)) {
|
|
65338
|
+
if (this.desiredControl !== state)
|
|
65339
|
+
return;
|
|
65340
|
+
if (Date.now() >= deadline) {
|
|
65341
|
+
if (state === "quit")
|
|
65342
|
+
break;
|
|
65343
|
+
throw new Error("Worker control acknowledgement unresolved");
|
|
65344
|
+
}
|
|
65345
|
+
await delay2(25);
|
|
65346
|
+
}
|
|
65347
|
+
this.flush();
|
|
65348
|
+
}
|
|
65349
|
+
async start(sources, cwd, publish, rootRun) {
|
|
65350
|
+
if (this.closed)
|
|
65351
|
+
throw new Error("Local worker pool is closed");
|
|
65352
|
+
const root = this.root;
|
|
65353
|
+
if (!this.queue || !this.lease || !this.options || root === undefined || this.lease.runId !== root)
|
|
65354
|
+
throw new Error("Local worker pool requires root preparation");
|
|
65355
|
+
this.sources = sources;
|
|
65356
|
+
this.publishing = publish;
|
|
65357
|
+
if (publish) {
|
|
65358
|
+
this.routes = await PooledRoutes.open(root, cwd, rootRun, (request) => this.request(request));
|
|
65359
|
+
if (this.closed) {
|
|
65360
|
+
await this.routes.close();
|
|
65361
|
+
throw new Error("Local worker pool is closed");
|
|
65362
|
+
}
|
|
65363
|
+
}
|
|
65364
|
+
if (this.options.admission)
|
|
65365
|
+
this.admission = await SqlResourceAdmission.open(this.journal, {
|
|
65366
|
+
...this.options.admission,
|
|
65367
|
+
root,
|
|
65368
|
+
holder: "coordinator"
|
|
65369
|
+
});
|
|
65370
|
+
this.logDirectory = workerLogDirectory(this.options.journalPath, root);
|
|
65371
|
+
await mkdir6(this.logDirectory, { recursive: true }).catch(() => {
|
|
65372
|
+
return;
|
|
65373
|
+
});
|
|
65374
|
+
this.recoveryTimer = setInterval(() => {
|
|
65375
|
+
this.recoverExpired().catch((error) => this.fail(error));
|
|
65376
|
+
}, RECOVERY_INTERVAL_MS).unref();
|
|
65377
|
+
const workers = Array.from({ length: this.options.workers }, () => this.spawn());
|
|
65378
|
+
try {
|
|
65379
|
+
await Promise.any(workers);
|
|
65380
|
+
} catch (error) {
|
|
65381
|
+
throw error.errors[0];
|
|
65382
|
+
}
|
|
65383
|
+
}
|
|
65384
|
+
async enqueue(command) {
|
|
65385
|
+
if (!this.queue || !this.lease || this.root === undefined)
|
|
65386
|
+
throw new Error("Local worker pool requires root preparation");
|
|
65387
|
+
if (command.rootRun !== undefined)
|
|
65388
|
+
this.routes?.updateRoot(command.rootRun);
|
|
65389
|
+
const work = await this.queue.enqueue(this.lease, command.descriptor, command.boundary);
|
|
65390
|
+
const result = await this.queue.result(this.root, work.id);
|
|
65391
|
+
return { work, ...result === undefined ? {} : { result } };
|
|
65392
|
+
}
|
|
65393
|
+
async transport(request, payload) {
|
|
65394
|
+
const target = this.deliveryRequests.get(request);
|
|
65395
|
+
if (target === undefined || !this.incarnations.has(target.slot.worker))
|
|
65396
|
+
throw new Error("Worker exited before delivery acknowledgement");
|
|
65397
|
+
const { slot, workerRequest } = target;
|
|
65398
|
+
await new Promise((resolve, reject) => {
|
|
65399
|
+
const id = randomUUID17();
|
|
65400
|
+
slot.deliveries.set(id, { resolve, reject });
|
|
65401
|
+
slot.child.send({ id: workerRequest, delivery: { id, from: payload.from, message: payload.message } }, (error) => {
|
|
65402
|
+
if (error) {
|
|
65403
|
+
slot.deliveries.delete(id);
|
|
65404
|
+
reject(error);
|
|
65405
|
+
}
|
|
65406
|
+
});
|
|
65407
|
+
});
|
|
65408
|
+
}
|
|
65409
|
+
request(body) {
|
|
65410
|
+
const id = ++this.requestSequence;
|
|
65411
|
+
return new Promise((resolve, reject) => {
|
|
65412
|
+
this.requests.set(id, { resolve, reject });
|
|
65413
|
+
this.port.send({ kind: "request", id, request: body });
|
|
65414
|
+
});
|
|
65415
|
+
}
|
|
65416
|
+
fail(error) {
|
|
65417
|
+
this.port.send({ kind: "failure", error: error.message });
|
|
65418
|
+
}
|
|
65419
|
+
publish(work, runs) {
|
|
65420
|
+
const failure = this.publicationFailures.get(work.id);
|
|
65421
|
+
if (failure?.generation === work.generation)
|
|
65422
|
+
throw new Error(failure.error);
|
|
65423
|
+
if (!this.publishing)
|
|
65424
|
+
return;
|
|
65425
|
+
this.pendingActivity.set(work.id, { work: token(work), runs });
|
|
65426
|
+
this.scheduleFlush();
|
|
65427
|
+
}
|
|
65428
|
+
scheduleFlush() {
|
|
65429
|
+
if (this.flushTimer !== undefined || this.closed)
|
|
65430
|
+
return;
|
|
65431
|
+
const wait = Math.max(0, this.lastFlushAt + POOL_SUMMARY_INTERVAL_MS - performance.now());
|
|
65432
|
+
this.flushTimer = setTimeout(() => {
|
|
65433
|
+
this.flushTimer = undefined;
|
|
65434
|
+
if (this.batches.size > 0) {
|
|
65435
|
+
this.lastFlushAt = performance.now();
|
|
65436
|
+
this.scheduleFlush();
|
|
65437
|
+
return;
|
|
65438
|
+
}
|
|
65439
|
+
this.flush();
|
|
65440
|
+
}, wait);
|
|
65441
|
+
}
|
|
65442
|
+
flush() {
|
|
65443
|
+
clearTimeout(this.flushTimer);
|
|
65444
|
+
this.flushTimer = undefined;
|
|
65445
|
+
if (this.pendingActivity.size === 0 && !this.usageChanged) {
|
|
65446
|
+
const last = [...this.batches.values()].at(-1);
|
|
65447
|
+
return last === undefined ? Promise.resolve() : new Promise((resolve) => this.chainSettle(last, resolve));
|
|
65448
|
+
}
|
|
65449
|
+
this.lastFlushAt = performance.now();
|
|
65450
|
+
const batch = ++this.batchSequence;
|
|
65451
|
+
const entries = [...this.pendingActivity.values()];
|
|
65452
|
+
this.pendingActivity.clear();
|
|
65453
|
+
const usage = this.usageChanged ? this.usageTotal : undefined;
|
|
65454
|
+
this.usageChanged = false;
|
|
65455
|
+
const settled = new Promise((resolve) => this.batches.set(batch, { entries, settle: resolve }));
|
|
65456
|
+
this.port.send({ kind: "activity", batch, entries, ...usage === undefined ? {} : { usage } });
|
|
65457
|
+
return settled;
|
|
65458
|
+
}
|
|
65459
|
+
chainSettle(batch, resolve) {
|
|
65460
|
+
const settle = batch.settle;
|
|
65461
|
+
batch.settle = () => {
|
|
65462
|
+
settle();
|
|
65463
|
+
resolve();
|
|
65464
|
+
};
|
|
65465
|
+
}
|
|
65466
|
+
noteUsage(usage) {
|
|
65467
|
+
this.usageTotal = usage;
|
|
65468
|
+
this.usageChanged = true;
|
|
65469
|
+
this.scheduleFlush();
|
|
65470
|
+
}
|
|
65471
|
+
async sendResult(workId) {
|
|
65472
|
+
const result = await this.queue.result(this.root, workId);
|
|
65473
|
+
if (result === undefined)
|
|
65474
|
+
return false;
|
|
65475
|
+
this.flush();
|
|
65476
|
+
this.port.send({
|
|
65477
|
+
kind: "result",
|
|
65478
|
+
workId,
|
|
65479
|
+
result,
|
|
65480
|
+
...this.usageTotal === undefined ? {} : { usage: this.usageTotal }
|
|
65481
|
+
});
|
|
65482
|
+
return true;
|
|
65483
|
+
}
|
|
65484
|
+
fence(work) {
|
|
65485
|
+
this.pendingActivity.delete(work.id);
|
|
65486
|
+
this.routes?.fence(work);
|
|
65487
|
+
this.port.send({ kind: "fence", work: { id: work.id, generation: work.generation } });
|
|
65488
|
+
}
|
|
65489
|
+
spawn() {
|
|
65490
|
+
if (this.closed)
|
|
65491
|
+
return Promise.reject(new Error("Local worker pool is closed"));
|
|
65492
|
+
const root = this.root;
|
|
65493
|
+
const options = this.options;
|
|
65494
|
+
const child = fork(options.workerEntry ?? process.argv[1], [
|
|
65495
|
+
JSON.stringify({
|
|
65496
|
+
journalPath: options.journalPath,
|
|
65497
|
+
root,
|
|
65498
|
+
definitions: this.sources,
|
|
65499
|
+
admission: options.admission
|
|
65500
|
+
})
|
|
65501
|
+
], { stdio: ["ignore", "ignore", "pipe", "ipc"], execArgv: [] });
|
|
65502
|
+
const disconnected = new Promise((resolve) => child.once("disconnect", () => resolve()));
|
|
65503
|
+
this.children.add(child);
|
|
65504
|
+
if (child.pid !== undefined)
|
|
65505
|
+
this.port.send({ kind: "spawned", pid: child.pid });
|
|
65506
|
+
const worker = randomUUID17();
|
|
65507
|
+
this.incarnations.set(worker, child);
|
|
65508
|
+
const slot = { worker, child, deliveries: new Map };
|
|
65509
|
+
let ready = false;
|
|
65510
|
+
let serial = Promise.resolve();
|
|
65511
|
+
const logPath = join29(this.logDirectory, `${worker}.log`);
|
|
65512
|
+
this.logs.set(child, logPath);
|
|
65513
|
+
let logFailure;
|
|
65514
|
+
const log = createWriteStream(logPath, { flags: "a" });
|
|
65515
|
+
log.on("error", (error) => {
|
|
65516
|
+
logFailure = error.message;
|
|
65517
|
+
});
|
|
65518
|
+
let stderr = "";
|
|
65519
|
+
child.stderr?.on("data", (chunk) => {
|
|
65520
|
+
stderr = (stderr + chunk.toString()).slice(-STDERR_TAIL_CHARACTERS);
|
|
65521
|
+
});
|
|
65522
|
+
child.stderr?.on("error", (error) => {
|
|
65523
|
+
logFailure ??= error.message;
|
|
65524
|
+
log.end();
|
|
65525
|
+
});
|
|
65526
|
+
child.stderr?.pipe(log);
|
|
65527
|
+
const diagnosed = (summary) => new Error(`${summary} (worker stderr log: ${logFailure === undefined ? logPath : `unavailable, ${logFailure}`})${stderr ? `: ${stderr}` : ""}`);
|
|
65528
|
+
const startupTimeoutMs = options.startupTimeoutMs;
|
|
65529
|
+
let timedOut = false;
|
|
65530
|
+
let terminated = false;
|
|
65531
|
+
const terminate = () => {
|
|
65532
|
+
if (terminated)
|
|
65533
|
+
return;
|
|
65534
|
+
terminated = true;
|
|
65535
|
+
child.kill("SIGKILL");
|
|
65536
|
+
};
|
|
65537
|
+
const replied = (error) => {
|
|
65538
|
+
if (error)
|
|
65539
|
+
terminate();
|
|
65540
|
+
};
|
|
65541
|
+
return new Promise((resolve, reject) => {
|
|
65542
|
+
const startup = setTimeout(() => {
|
|
65543
|
+
timedOut = true;
|
|
65544
|
+
terminate();
|
|
65545
|
+
}, startupTimeoutMs);
|
|
65546
|
+
child.on("error", (error) => {
|
|
65547
|
+
clearTimeout(startup);
|
|
65548
|
+
reject(error);
|
|
65549
|
+
terminate();
|
|
65550
|
+
});
|
|
65551
|
+
const closed = new Promise((resolve) => child.once("close", () => resolve()));
|
|
65552
|
+
child.once("exit", () => {
|
|
65553
|
+
Promise.race([closed, delay2(STDERR_DRAIN_GRACE_MS)]).then(() => reject(diagnosed(timedOut ? `Local worker startup timed out after ${startupTimeoutMs} ms` : "Local worker exited before readiness")));
|
|
65554
|
+
this.children.delete(child);
|
|
65555
|
+
this.logs.delete(child);
|
|
65556
|
+
if (child.pid !== undefined)
|
|
65557
|
+
this.port.send({ kind: "exited", pid: child.pid });
|
|
65558
|
+
clearTimeout(startup);
|
|
65559
|
+
this.incarnations.delete(worker);
|
|
65560
|
+
this.readyWorkers.delete(worker);
|
|
65561
|
+
this.controlAcks.delete(worker);
|
|
65562
|
+
this.lastHeard.delete(worker);
|
|
65563
|
+
if (slot.work)
|
|
65564
|
+
this.fence(slot.work);
|
|
65565
|
+
for (const delivery of slot.deliveries.values())
|
|
65566
|
+
delivery.reject(new Error("Worker exited before delivery acknowledgement"));
|
|
65567
|
+
slot.deliveries.clear();
|
|
65568
|
+
const cleanup = disconnected.then(() => serial).then(async () => {
|
|
65569
|
+
try {
|
|
65570
|
+
if (await this.admission?.releaseStoppedSessions(worker))
|
|
65571
|
+
throw new Error("Local worker process cleanup unconfirmed; permits quarantined");
|
|
65572
|
+
} catch (error) {
|
|
65573
|
+
this.cleanupFailure = error instanceof Error ? error : new Error(String(error));
|
|
65574
|
+
throw this.cleanupFailure;
|
|
65575
|
+
}
|
|
65576
|
+
}).then(async () => {
|
|
65577
|
+
if (this.closed || !ready || this.desiredControl === "quit")
|
|
65578
|
+
return;
|
|
65579
|
+
let work = slot.work;
|
|
65580
|
+
if (work && await this.sendResult(work.id)) {
|
|
65581
|
+
work = undefined;
|
|
65582
|
+
slot.work = undefined;
|
|
65583
|
+
this.replacements = 0;
|
|
65584
|
+
}
|
|
65585
|
+
if (this.closed)
|
|
65586
|
+
return;
|
|
65587
|
+
if (work)
|
|
65588
|
+
await this.queue.releaseExited(this.lease, work);
|
|
65589
|
+
const exhausted = work ? work.generation >= 3 : ++this.replacements > 3;
|
|
65590
|
+
if (exhausted)
|
|
65591
|
+
this.fail(diagnosed("Local worker retry limit"));
|
|
65592
|
+
else
|
|
65593
|
+
await this.spawn();
|
|
65594
|
+
}).catch((error) => {
|
|
65595
|
+
this.fail(error);
|
|
65596
|
+
});
|
|
65597
|
+
this.cleanups.add(cleanup);
|
|
65598
|
+
cleanup.finally(() => this.cleanups.delete(cleanup));
|
|
65599
|
+
});
|
|
65600
|
+
child.on("message", (message) => {
|
|
65601
|
+
this.observeResponsiveness();
|
|
65602
|
+
this.lastHeard.set(worker, { at: performance.now(), stalled: this.stalledMs });
|
|
65603
|
+
if (message.kind === "delivery-ack") {
|
|
65604
|
+
const pending = slot.deliveries.get(message.deliveryId);
|
|
65605
|
+
slot.deliveries.delete(message.deliveryId);
|
|
65606
|
+
if (message.error)
|
|
65607
|
+
pending?.reject(new Error(message.error));
|
|
65608
|
+
else
|
|
65609
|
+
pending?.resolve();
|
|
65610
|
+
return;
|
|
65611
|
+
}
|
|
65612
|
+
let routeReady;
|
|
65613
|
+
const handled = serial.then(async () => {
|
|
65614
|
+
if (message.kind === "ready") {
|
|
65615
|
+
if (terminated)
|
|
65616
|
+
return;
|
|
65617
|
+
ready = true;
|
|
65618
|
+
this.readyWorkers.add(worker);
|
|
65619
|
+
clearTimeout(startup);
|
|
65620
|
+
resolve();
|
|
65621
|
+
return;
|
|
65622
|
+
}
|
|
65623
|
+
await this.starting;
|
|
65624
|
+
const queue = this.queue;
|
|
65625
|
+
const lease = this.lease;
|
|
65626
|
+
if (message.kind === "control")
|
|
65627
|
+
return queue.control(lease);
|
|
65628
|
+
if (message.kind === "controlled") {
|
|
65629
|
+
const control = await queue.control(lease);
|
|
65630
|
+
if (message.revision > control.revision)
|
|
65631
|
+
throw new Error("Unknown control revision");
|
|
65632
|
+
this.controlAcks.set(worker, Math.max(this.controlAcks.get(worker) ?? -1, message.revision));
|
|
65633
|
+
return;
|
|
65634
|
+
}
|
|
65635
|
+
if (message.kind === "claim") {
|
|
65636
|
+
if (slot.work)
|
|
65637
|
+
throw new Error("Worker already holds an attempt");
|
|
65638
|
+
if (this.budgetBlocked)
|
|
65639
|
+
return;
|
|
65640
|
+
await this.recoverExpired();
|
|
65641
|
+
slot.work = await queue.claim(lease, worker, this.sources.map((source) => ({
|
|
65642
|
+
definition: `${source.path}#${source.exportName}`,
|
|
65643
|
+
digest: source.digest
|
|
65644
|
+
})), options.leaseMs);
|
|
65645
|
+
return slot.work;
|
|
65646
|
+
}
|
|
65647
|
+
let work = slot.work;
|
|
65648
|
+
if (!work)
|
|
65649
|
+
throw new Error("Worker has no active attempt");
|
|
65650
|
+
if (message.kind === "renew" || work.expiresAt - Date.now() < options.leaseMs / 2) {
|
|
65651
|
+
work = await queue.extend(lease, work, options.leaseMs);
|
|
65652
|
+
slot.work = work;
|
|
65653
|
+
}
|
|
65654
|
+
if (message.kind === "activity")
|
|
65655
|
+
return this.publish(work, message.runs);
|
|
65656
|
+
if (message.kind === "route") {
|
|
65657
|
+
if (this.routes === undefined)
|
|
65658
|
+
return;
|
|
65659
|
+
await this.flush();
|
|
65660
|
+
this.assertPublished(work);
|
|
65661
|
+
routeReady = this.routes.route(work, message.runId);
|
|
65662
|
+
return;
|
|
65663
|
+
}
|
|
65664
|
+
if (message.kind === "delivery-ready") {
|
|
65665
|
+
if (!this.publishing)
|
|
65666
|
+
return;
|
|
65667
|
+
await this.flush();
|
|
65668
|
+
this.assertPublished(work);
|
|
65669
|
+
return this.request({
|
|
65670
|
+
kind: "ready",
|
|
65671
|
+
work: token(work),
|
|
65672
|
+
runId: message.runId,
|
|
65673
|
+
stageId: message.stageId,
|
|
65674
|
+
stageName: message.stageName,
|
|
65675
|
+
...message.sessionId === undefined ? {} : { sessionId: message.sessionId }
|
|
65676
|
+
});
|
|
65677
|
+
}
|
|
65678
|
+
if (message.kind === "delivery") {
|
|
65679
|
+
if (!message.recipient)
|
|
65680
|
+
throw new Error("Worker delivery requires a session identity");
|
|
65681
|
+
if (!this.publishing)
|
|
65682
|
+
return;
|
|
65683
|
+
await this.flush();
|
|
65684
|
+
this.assertPublished(work);
|
|
65685
|
+
const id = ++this.requestSequence;
|
|
65686
|
+
this.deliveryRequests.set(id, { slot, workerRequest: message.id });
|
|
65687
|
+
try {
|
|
65688
|
+
return await new Promise((resolve, reject) => {
|
|
65689
|
+
this.requests.set(id, { resolve, reject });
|
|
65690
|
+
this.port.send({
|
|
65691
|
+
kind: "request",
|
|
65692
|
+
id,
|
|
65693
|
+
request: {
|
|
65694
|
+
kind: "deliver",
|
|
65695
|
+
work: token(work),
|
|
65696
|
+
runId: message.runId,
|
|
65697
|
+
stageId: message.stageId,
|
|
65698
|
+
stageName: message.stageName,
|
|
65699
|
+
recipient: message.recipient
|
|
65700
|
+
}
|
|
65701
|
+
});
|
|
65702
|
+
});
|
|
65703
|
+
} finally {
|
|
65704
|
+
this.deliveryRequests.delete(id);
|
|
65705
|
+
}
|
|
65706
|
+
}
|
|
65707
|
+
if (message.kind === "renew")
|
|
65708
|
+
return;
|
|
65709
|
+
if (message.kind === "commit") {
|
|
65710
|
+
const receipt = await queue.commit(lease, work, message.operation, message.mutation);
|
|
65711
|
+
this.noteUsage(await this.usageLedger.total(root));
|
|
65712
|
+
return receipt;
|
|
65713
|
+
}
|
|
65714
|
+
if (message.kind === "usage") {
|
|
65715
|
+
this.noteUsage(await this.usageLedger.report(lease, work, message.reports));
|
|
65716
|
+
return;
|
|
65717
|
+
}
|
|
65718
|
+
if (message.kind === "budget") {
|
|
65719
|
+
await queue.assertCurrent(lease, work);
|
|
65720
|
+
if (this.budgetBlocked)
|
|
65721
|
+
throw new Error("Root workflow budget exhausted");
|
|
65722
|
+
return queue.control(lease);
|
|
65723
|
+
}
|
|
65724
|
+
if (message.kind !== "complete")
|
|
65725
|
+
throw new Error("Invalid worker operation");
|
|
65726
|
+
await queue.complete(lease, work, message.result);
|
|
65727
|
+
this.replacements = 0;
|
|
65728
|
+
slot.work = undefined;
|
|
65729
|
+
await this.sendResult(work.id);
|
|
65730
|
+
});
|
|
65731
|
+
const response = handled.then((value2) => routeReady ?? value2);
|
|
65732
|
+
if (message.kind !== "delivery" && message.kind !== "delivery-ready")
|
|
65733
|
+
serial = handled.then(() => {}, () => {});
|
|
65734
|
+
response.then((value2) => child.send({ id: message.id, value: value2 }, replied), (error) => child.send({ id: message.id, error: error.message }, replied));
|
|
65735
|
+
});
|
|
65736
|
+
});
|
|
65737
|
+
}
|
|
65738
|
+
assertPublished(work) {
|
|
65739
|
+
const failure = this.publicationFailures.get(work.id);
|
|
65740
|
+
if (failure?.generation === work.generation)
|
|
65741
|
+
throw new Error(failure.error);
|
|
65742
|
+
}
|
|
65743
|
+
recoverExpired() {
|
|
65744
|
+
if (this.recovering)
|
|
65745
|
+
return this.recovering;
|
|
65746
|
+
const root = this.root;
|
|
65747
|
+
if (this.closed || root === undefined || !this.queue || performance.now() < this.nextRecoveryAt)
|
|
65748
|
+
return Promise.resolve();
|
|
65749
|
+
const queue = this.queue;
|
|
65750
|
+
const lease = this.lease;
|
|
65751
|
+
const leaseMs = this.options.leaseMs;
|
|
65752
|
+
this.recovering = (async () => {
|
|
65753
|
+
await new Promise((resolve) => setImmediate(resolve));
|
|
65754
|
+
await this.admission?.reconcile();
|
|
65755
|
+
const expiredBefore = Date.now();
|
|
65756
|
+
for (const work of await queue.inspect(root)) {
|
|
65757
|
+
if (work.state !== "leased" || work.expiresAt >= expiredBefore)
|
|
65758
|
+
continue;
|
|
65759
|
+
const child = work.worker === null ? undefined : this.incarnations.get(work.worker);
|
|
65760
|
+
if (child && work.worker !== null && this.isLive(work.worker, child)) {
|
|
65761
|
+
await queue.extend(lease, work, leaseMs).catch((error) => {
|
|
65762
|
+
if (error.message !== STALE_WORK_LEASE)
|
|
65763
|
+
throw error;
|
|
65764
|
+
});
|
|
65765
|
+
continue;
|
|
65766
|
+
}
|
|
65767
|
+
if (child && !await this.stop(child).then(() => true, () => false))
|
|
65768
|
+
continue;
|
|
65769
|
+
await queue.expire(lease, work);
|
|
65770
|
+
}
|
|
65771
|
+
})().finally(() => {
|
|
65772
|
+
this.recovering = undefined;
|
|
65773
|
+
this.nextRecoveryAt = performance.now() + RECOVERY_INTERVAL_MS;
|
|
65774
|
+
});
|
|
65775
|
+
return this.recovering;
|
|
65776
|
+
}
|
|
65777
|
+
observeResponsiveness() {
|
|
65778
|
+
const now = performance.now();
|
|
65779
|
+
const gap = now - this.lastTick;
|
|
65780
|
+
if (gap > STALL_THRESHOLD_MS)
|
|
65781
|
+
this.stalledMs += gap - TICK_INTERVAL_MS;
|
|
65782
|
+
this.lastTick = now;
|
|
65783
|
+
}
|
|
65784
|
+
isLive(worker, child) {
|
|
65785
|
+
this.observeResponsiveness();
|
|
65786
|
+
const heard = this.lastHeard.get(worker);
|
|
65787
|
+
return child.exitCode === null && child.signalCode === null && child.connected && heard !== undefined && performance.now() - heard.at - (this.stalledMs - heard.stalled) < this.options.leaseMs;
|
|
65788
|
+
}
|
|
65789
|
+
stop(child) {
|
|
65790
|
+
if (child.pid === undefined || child.exitCode !== null || child.signalCode !== null)
|
|
65791
|
+
return Promise.resolve();
|
|
65792
|
+
return new Promise((resolve, reject) => {
|
|
65793
|
+
const timeout = setTimeout(() => reject(new Error("Local worker cleanup was not confirmed")), 5000);
|
|
65794
|
+
child.once("exit", () => {
|
|
65795
|
+
clearTimeout(timeout);
|
|
65796
|
+
resolve();
|
|
65797
|
+
});
|
|
65798
|
+
child.kill("SIGKILL");
|
|
65799
|
+
});
|
|
65800
|
+
}
|
|
65801
|
+
async close() {
|
|
65802
|
+
this.closed = true;
|
|
65803
|
+
clearInterval(this.ticker);
|
|
65804
|
+
clearInterval(this.recoveryTimer);
|
|
65805
|
+
let failure;
|
|
65806
|
+
let stopping = [];
|
|
65807
|
+
try {
|
|
65808
|
+
await this.recovering?.catch(() => {});
|
|
65809
|
+
stopping = [...this.children].map((child) => ({ child, log: this.logs.get(child) }));
|
|
65810
|
+
await Promise.all(stopping.map(({ child }) => this.stop(child)));
|
|
65811
|
+
await Promise.all([...this.cleanups]);
|
|
65812
|
+
if (this.cleanupFailure)
|
|
65813
|
+
throw this.cleanupFailure;
|
|
65814
|
+
} catch (error) {
|
|
65815
|
+
failure = error;
|
|
65816
|
+
}
|
|
65817
|
+
if (failure === undefined) {
|
|
65818
|
+
const stopped = stopping.flatMap(({ child, log }) => log !== undefined && (child.exitCode !== null || child.signalCode !== null) ? [log] : []);
|
|
65819
|
+
await Promise.all(stopped.map((log) => rm3(log, { force: true }).catch(() => {})));
|
|
65820
|
+
if (this.logDirectory !== undefined)
|
|
65821
|
+
await rmdir(this.logDirectory).catch(() => {});
|
|
65822
|
+
}
|
|
65823
|
+
this.flush();
|
|
65824
|
+
try {
|
|
65825
|
+
await this.routes?.close();
|
|
65826
|
+
} catch (error) {
|
|
65827
|
+
failure ??= error;
|
|
65828
|
+
}
|
|
65829
|
+
for (const batch of this.batches.values())
|
|
65830
|
+
batch.settle();
|
|
65831
|
+
this.batches.clear();
|
|
65832
|
+
for (const pending of this.requests.values())
|
|
65833
|
+
pending.reject(new Error("Local worker pool is closed"));
|
|
65834
|
+
this.requests.clear();
|
|
65835
|
+
if (failure !== undefined)
|
|
65836
|
+
throw failure;
|
|
65837
|
+
}
|
|
65838
|
+
}
|
|
65839
|
+
var CONTROL_ACKNOWLEDGEMENT_TIMEOUT_MS = 5000, RECOVERY_INTERVAL_MS = 500, TICK_INTERVAL_MS = 100, STALL_THRESHOLD_MS = 500, COORDINATOR_LEASE_MS, STDERR_TAIL_CHARACTERS = 8192, STDERR_DRAIN_GRACE_MS = 1000, ROUTE_REGISTRATION_WAIT_MS = 120000, PENDING_STAGE_MESSAGE_EVENT2 = "atomic:workflow-pending-stage-message", STICKY_LIVE_DELIVERED_EVENT2 = "atomic:workflow-sticky-live-delivered", PENDING_STAGE_UNDELIVERABLE_EVENT2 = "atomic:workflow-pending-stage-undeliverable";
|
|
65840
|
+
var init_pool_coordinator = __esm(() => {
|
|
65841
|
+
init_resolution();
|
|
65842
|
+
init_sql_pool_queue();
|
|
65843
|
+
init_sql_pool_usage();
|
|
65844
|
+
init_pending_stage_intercom();
|
|
65845
|
+
init_intercom_group();
|
|
65846
|
+
init_pending_stage_route_capability();
|
|
65847
|
+
init_pending_stage_route_readiness();
|
|
65848
|
+
init_store();
|
|
65849
|
+
init_resource_admission();
|
|
65850
|
+
COORDINATOR_LEASE_MS = 3 * STAGE_SESSION_HEARTBEAT_INTERVAL_MS;
|
|
64777
65851
|
});
|
|
64778
65852
|
|
|
64779
65853
|
// dist/builtin/workflows/src/workers/protocol.ts
|
|
@@ -65023,21 +66097,26 @@ class PoolStageHandoff {
|
|
|
65023
66097
|
store;
|
|
65024
66098
|
queue;
|
|
65025
66099
|
lease;
|
|
66100
|
+
claimRun;
|
|
65026
66101
|
attempts = new Map;
|
|
65027
66102
|
owners = new Map;
|
|
65028
66103
|
publications = new Map;
|
|
65029
66104
|
drains = new Map;
|
|
65030
|
-
constructor(backend, store, queue, lease) {
|
|
66105
|
+
constructor(backend, store, queue, lease, claimRun) {
|
|
65031
66106
|
this.backend = backend;
|
|
65032
66107
|
this.store = store;
|
|
65033
66108
|
this.queue = queue;
|
|
65034
66109
|
this.lease = lease;
|
|
66110
|
+
this.claimRun = claimRun;
|
|
65035
66111
|
}
|
|
65036
66112
|
publish(work, runs) {
|
|
65037
66113
|
const previous = this.publications.get(work.id) ?? Promise.resolve();
|
|
65038
66114
|
const publication = previous.catch(() => {
|
|
65039
66115
|
return;
|
|
65040
|
-
}).then(() =>
|
|
66116
|
+
}).then(async () => {
|
|
66117
|
+
const commit = await this.prepare(work, runs);
|
|
66118
|
+
this.store.batch(commit);
|
|
66119
|
+
});
|
|
65041
66120
|
const settled = publication.catch(() => {
|
|
65042
66121
|
return;
|
|
65043
66122
|
});
|
|
@@ -65048,6 +66127,21 @@ class PoolStageHandoff {
|
|
|
65048
66127
|
});
|
|
65049
66128
|
return publication;
|
|
65050
66129
|
}
|
|
66130
|
+
async publishBatch(publications) {
|
|
66131
|
+
const prepared = [];
|
|
66132
|
+
for (const { work, runs } of publications)
|
|
66133
|
+
prepared.push(await this.prepare(work, runs).catch((error) => error));
|
|
66134
|
+
return this.store.batch(() => prepared.map((commit) => {
|
|
66135
|
+
if (commit instanceof Error)
|
|
66136
|
+
return commit;
|
|
66137
|
+
try {
|
|
66138
|
+
commit();
|
|
66139
|
+
return;
|
|
66140
|
+
} catch (error) {
|
|
66141
|
+
return error instanceof Error ? error : new Error(String(error));
|
|
66142
|
+
}
|
|
66143
|
+
}));
|
|
66144
|
+
}
|
|
65051
66145
|
async route(work, runId) {
|
|
65052
66146
|
this.requireOwned(await this.begin(work), runId);
|
|
65053
66147
|
const readiness = workflowPendingStageRouteReady(this.store, runId);
|
|
@@ -65098,7 +66192,7 @@ class PoolStageHandoff {
|
|
|
65098
66192
|
const result = await this.queue.result(root, work.id);
|
|
65099
66193
|
if (result === undefined)
|
|
65100
66194
|
return;
|
|
65101
|
-
await this.backend.
|
|
66195
|
+
await this.backend.catchUpWorkflow(root);
|
|
65102
66196
|
const terminals = new Map;
|
|
65103
66197
|
for (const checkpoint of this.backend.listCheckpoints(root)) {
|
|
65104
66198
|
const boundary = checkpoint.kind === "stage" ? checkpoint.topology?.boundary : undefined;
|
|
@@ -65108,6 +66202,9 @@ class PoolStageHandoff {
|
|
|
65108
66202
|
if (child?.runId === boundary.child.runId)
|
|
65109
66203
|
terminals.set(child.runId, { status: child.status, outputs: child.outputs });
|
|
65110
66204
|
}
|
|
66205
|
+
this.store.batch(() => this.settle(work, result, terminals));
|
|
66206
|
+
}
|
|
66207
|
+
settle(work, result, terminals) {
|
|
65111
66208
|
for (const run of this.store.runs()) {
|
|
65112
66209
|
if (run.endedAt !== undefined || this.owners.get(run.id) !== work.id)
|
|
65113
66210
|
continue;
|
|
@@ -65131,22 +66228,31 @@ class PoolStageHandoff {
|
|
|
65131
66228
|
if (current !== undefined && current.generation > work.generation)
|
|
65132
66229
|
return;
|
|
65133
66230
|
const attempt = current?.generation === work.generation ? current : this.start(work, current);
|
|
65134
|
-
this.release(attempt);
|
|
66231
|
+
this.store.batch(() => this.release(attempt));
|
|
65135
66232
|
attempt.fenced = true;
|
|
65136
66233
|
}
|
|
65137
|
-
async
|
|
66234
|
+
async prepare(work, runs) {
|
|
65138
66235
|
const subtree = this.subtree(work, runs);
|
|
65139
66236
|
const root = this.lease.runId;
|
|
65140
|
-
const
|
|
65141
|
-
|
|
65142
|
-
|
|
65143
|
-
|
|
65144
|
-
|
|
65145
|
-
|
|
65146
|
-
|
|
65147
|
-
|
|
65148
|
-
|
|
66237
|
+
const owned = (known) => {
|
|
66238
|
+
for (const run of subtree) {
|
|
66239
|
+
const existing = known.get(run.id);
|
|
66240
|
+
const owner = this.owners.get(run.id);
|
|
66241
|
+
if (owner !== undefined && owner !== work.id || owner === undefined && existing !== undefined && (existing.parentRunId !== run.parentRunId || existing.parentStageId !== run.parentStageId))
|
|
66242
|
+
throw outside2(run.id);
|
|
66243
|
+
}
|
|
66244
|
+
};
|
|
66245
|
+
const before = new Map(this.store.runs().map((run) => [run.id, run]));
|
|
66246
|
+
owned(before);
|
|
66247
|
+
if (subtree.some((run) => before.get(run.id) === undefined))
|
|
66248
|
+
await this.backend.catchUpWorkflow(root);
|
|
65149
66249
|
await this.queue.assertCurrent(this.lease, work);
|
|
66250
|
+
return () => this.commit(work, subtree, owned);
|
|
66251
|
+
}
|
|
66252
|
+
commit(work, subtree, owned) {
|
|
66253
|
+
const root = this.lease.runId;
|
|
66254
|
+
const known = new Map(this.store.runs().map((run) => [run.id, run]));
|
|
66255
|
+
owned(known);
|
|
65150
66256
|
const attempt = this.enter(work);
|
|
65151
66257
|
const execution = {
|
|
65152
66258
|
worker: work.worker ?? "",
|
|
@@ -65158,11 +66264,13 @@ class PoolStageHandoff {
|
|
|
65158
66264
|
...run,
|
|
65159
66265
|
rootRunId: root,
|
|
65160
66266
|
pendingStageMessages: [
|
|
65161
|
-
...
|
|
66267
|
+
...known.get(run.id)?.pendingStageMessages ?? pendingStageMessagesForDurableRun(this.backend, run.id, root)
|
|
65162
66268
|
],
|
|
65163
66269
|
stages: run.stages.map((stage) => ({ ...stage, workerExecution: execution }))
|
|
65164
66270
|
}));
|
|
65165
66271
|
const changed = published.some((run) => activity(known.get(run.id)) !== activity(run));
|
|
66272
|
+
for (const run of published)
|
|
66273
|
+
this.claimRun?.(run.id);
|
|
65166
66274
|
for (const run of published) {
|
|
65167
66275
|
attempt.runs.add(run.id);
|
|
65168
66276
|
this.owners.set(run.id, work.id);
|
|
@@ -65187,7 +66295,7 @@ class PoolStageHandoff {
|
|
|
65187
66295
|
const byId = new Map;
|
|
65188
66296
|
for (const run of runs) {
|
|
65189
66297
|
if (run.id === root || byId.has(run.id))
|
|
65190
|
-
throw
|
|
66298
|
+
throw outside2(run.id);
|
|
65191
66299
|
byId.set(run.id, run);
|
|
65192
66300
|
}
|
|
65193
66301
|
if (runs.length === 0)
|
|
@@ -65200,25 +66308,25 @@ class PoolStageHandoff {
|
|
|
65200
66308
|
const top = byId.get(work.childRunId);
|
|
65201
66309
|
if (top !== undefined) {
|
|
65202
66310
|
if (top.parentRunId !== root || top.parentStageId !== work.boundaryId)
|
|
65203
|
-
throw
|
|
66311
|
+
throw outside2(top.id);
|
|
65204
66312
|
const pending = [top];
|
|
65205
66313
|
for (let run = pending.shift();run !== undefined; run = pending.shift()) {
|
|
65206
66314
|
if (run.rootRunId !== undefined && run.rootRunId !== root)
|
|
65207
|
-
throw
|
|
66315
|
+
throw outside2(run.id);
|
|
65208
66316
|
ordered.push(run);
|
|
65209
66317
|
for (const child of runs) {
|
|
65210
66318
|
if (child.parentRunId !== run.id)
|
|
65211
66319
|
continue;
|
|
65212
66320
|
const parentStage = run.stages.find((stage) => stage.id === child.parentStageId);
|
|
65213
66321
|
if (authoritativeWorkflowChildRunId(parentStage) !== child.id)
|
|
65214
|
-
throw
|
|
66322
|
+
throw outside2(child.id);
|
|
65215
66323
|
pending.push(child);
|
|
65216
66324
|
}
|
|
65217
66325
|
}
|
|
65218
66326
|
}
|
|
65219
66327
|
const stray = runs.find((run) => !ordered.includes(run));
|
|
65220
66328
|
if (stray !== undefined)
|
|
65221
|
-
throw
|
|
66329
|
+
throw outside2(stray.id);
|
|
65222
66330
|
return ordered;
|
|
65223
66331
|
}
|
|
65224
66332
|
async begin(work) {
|
|
@@ -65273,7 +66381,7 @@ class PoolStageHandoff {
|
|
|
65273
66381
|
}
|
|
65274
66382
|
requireOwned(attempt, runId) {
|
|
65275
66383
|
if (!attempt.runs.has(runId))
|
|
65276
|
-
throw
|
|
66384
|
+
throw outside2(runId);
|
|
65277
66385
|
}
|
|
65278
66386
|
delivery(attempt, work, runId, stageId, stageName) {
|
|
65279
66387
|
const key = JSON.stringify([runId, stageId]);
|
|
@@ -65291,7 +66399,7 @@ class PoolStageHandoff {
|
|
|
65291
66399
|
return delivery;
|
|
65292
66400
|
}
|
|
65293
66401
|
}
|
|
65294
|
-
function
|
|
66402
|
+
function outside2(runId) {
|
|
65295
66403
|
return new Error(`Pool stage publication is outside the work subtree: ${runId}`);
|
|
65296
66404
|
}
|
|
65297
66405
|
function isLive(stage) {
|
|
@@ -65316,33 +66424,122 @@ var init_stage_handoff = __esm(() => {
|
|
|
65316
66424
|
});
|
|
65317
66425
|
|
|
65318
66426
|
// dist/builtin/workflows/src/workers/local-pool.ts
|
|
65319
|
-
import { fork } from "node:child_process";
|
|
65320
|
-
import {
|
|
65321
|
-
import {
|
|
65322
|
-
import { mkdir as mkdir6, rm as rm3, rmdir } from "node:fs/promises";
|
|
65323
|
-
import { dirname as dirname14, join as join29, resolve as resolve13 } from "node:path";
|
|
65324
|
-
import { setTimeout as delay2 } from "node:timers/promises";
|
|
66427
|
+
import { fork as fork2 } from "node:child_process";
|
|
66428
|
+
import { resolve as resolve14 } from "node:path";
|
|
66429
|
+
import { setTimeout as delay3 } from "node:timers/promises";
|
|
65325
66430
|
import { fileURLToPath as fileURLToPath6 } from "node:url";
|
|
65326
66431
|
import { isBunBinary as isBunBinary2 } from "@bastani/atomic";
|
|
65327
|
-
function
|
|
65328
|
-
return
|
|
66432
|
+
function defaultWorkerEntry() {
|
|
66433
|
+
return isBunBinary2 ? INTERNAL_WORKFLOW_WORKER_ARG : fileURLToPath6(new URL("./local-worker-entry.mjs", import.meta.url));
|
|
66434
|
+
}
|
|
66435
|
+
function launchCoordinatorProcess() {
|
|
66436
|
+
const child = fork2(defaultWorkerEntry(), [JSON.stringify({ role: POOL_COORDINATOR_ROLE })], {
|
|
66437
|
+
stdio: ["ignore", "ignore", "pipe", "ipc"],
|
|
66438
|
+
execArgv: []
|
|
66439
|
+
});
|
|
66440
|
+
let stderr = "";
|
|
66441
|
+
child.stderr?.on("data", (chunk) => {
|
|
66442
|
+
stderr = (stderr + chunk.toString()).slice(-COORDINATOR_STDERR_TAIL_CHARACTERS);
|
|
66443
|
+
});
|
|
66444
|
+
const listeners = [];
|
|
66445
|
+
let lostReason;
|
|
66446
|
+
const lose = (reason) => {
|
|
66447
|
+
if (lostReason !== undefined)
|
|
66448
|
+
return;
|
|
66449
|
+
lostReason = reason;
|
|
66450
|
+
for (const listener of listeners)
|
|
66451
|
+
listener(reason);
|
|
66452
|
+
};
|
|
66453
|
+
const exited = new Promise((resolve) => child.once("exit", () => resolve()));
|
|
66454
|
+
child.on("error", lose);
|
|
66455
|
+
child.once("exit", (code, signal) => lose(new Error(`Local worker pool coordinator exited (${signal ?? `code ${code}`})${stderr ? `: ${stderr.trim()}` : ""}`)));
|
|
66456
|
+
return {
|
|
66457
|
+
pid: child.pid,
|
|
66458
|
+
port: {
|
|
66459
|
+
send(message) {
|
|
66460
|
+
if (lostReason !== undefined || !child.connected)
|
|
66461
|
+
return;
|
|
66462
|
+
child.send(message, (error) => {
|
|
66463
|
+
if (error)
|
|
66464
|
+
lose(error);
|
|
66465
|
+
});
|
|
66466
|
+
},
|
|
66467
|
+
onMessage(listener) {
|
|
66468
|
+
child.on("message", (message) => listener(message));
|
|
66469
|
+
},
|
|
66470
|
+
onClose(listener) {
|
|
66471
|
+
if (lostReason !== undefined)
|
|
66472
|
+
listener(lostReason);
|
|
66473
|
+
else
|
|
66474
|
+
listeners.push(listener);
|
|
66475
|
+
},
|
|
66476
|
+
async close() {
|
|
66477
|
+
if (child.connected)
|
|
66478
|
+
child.disconnect();
|
|
66479
|
+
if (child.exitCode !== null || child.signalCode !== null)
|
|
66480
|
+
return;
|
|
66481
|
+
const timedOut = await Promise.race([exited.then(() => false), delay3(COORDINATOR_EXIT_GRACE_MS, true)]);
|
|
66482
|
+
if (timedOut) {
|
|
66483
|
+
child.kill("SIGKILL");
|
|
66484
|
+
await exited;
|
|
66485
|
+
}
|
|
66486
|
+
}
|
|
66487
|
+
}
|
|
66488
|
+
};
|
|
66489
|
+
}
|
|
66490
|
+
function launchCoordinatorInProcess() {
|
|
66491
|
+
const mainListeners = [];
|
|
66492
|
+
const coordinatorListeners = [];
|
|
66493
|
+
const copy = (message) => JSON.parse(JSON.stringify(message));
|
|
66494
|
+
const coordinator = new PoolCoordinator({
|
|
66495
|
+
send(message) {
|
|
66496
|
+
const copied = copy(message);
|
|
66497
|
+
setImmediate(() => {
|
|
66498
|
+
for (const listener of mainListeners)
|
|
66499
|
+
listener(copied);
|
|
66500
|
+
});
|
|
66501
|
+
},
|
|
66502
|
+
onMessage(listener) {
|
|
66503
|
+
coordinatorListeners.push(listener);
|
|
66504
|
+
},
|
|
66505
|
+
onClose() {},
|
|
66506
|
+
async close() {}
|
|
66507
|
+
});
|
|
66508
|
+
return {
|
|
66509
|
+
pid: process.pid,
|
|
66510
|
+
port: {
|
|
66511
|
+
send(message) {
|
|
66512
|
+
const copied = copy(message);
|
|
66513
|
+
setImmediate(() => {
|
|
66514
|
+
for (const listener of coordinatorListeners)
|
|
66515
|
+
listener(copied);
|
|
66516
|
+
});
|
|
66517
|
+
},
|
|
66518
|
+
onMessage(listener) {
|
|
66519
|
+
mainListeners.push(listener);
|
|
66520
|
+
},
|
|
66521
|
+
onClose() {},
|
|
66522
|
+
async close() {
|
|
66523
|
+
coordinator.shutdown();
|
|
66524
|
+
}
|
|
66525
|
+
}
|
|
66526
|
+
};
|
|
65329
66527
|
}
|
|
65330
66528
|
|
|
65331
66529
|
class LocalWorkerPool {
|
|
65332
66530
|
backend;
|
|
65333
66531
|
options;
|
|
65334
|
-
|
|
65335
|
-
|
|
65336
|
-
|
|
65337
|
-
|
|
65338
|
-
|
|
65339
|
-
|
|
65340
|
-
|
|
65341
|
-
|
|
65342
|
-
|
|
65343
|
-
|
|
65344
|
-
|
|
65345
|
-
rootStalledMs = 0;
|
|
66532
|
+
pids = new Set;
|
|
66533
|
+
coordinator;
|
|
66534
|
+
coordinatorLost;
|
|
66535
|
+
commands = new Map;
|
|
66536
|
+
commandSequence = 0;
|
|
66537
|
+
inbound = Promise.resolve();
|
|
66538
|
+
works = new Map;
|
|
66539
|
+
attempts = new Map;
|
|
66540
|
+
results = new Map;
|
|
66541
|
+
changed = Promise.withResolvers();
|
|
66542
|
+
waiting = 0;
|
|
65346
66543
|
ticker;
|
|
65347
66544
|
leaseMs;
|
|
65348
66545
|
desiredControl;
|
|
@@ -65350,18 +66547,17 @@ class LocalWorkerPool {
|
|
|
65350
66547
|
store;
|
|
65351
66548
|
handoff;
|
|
65352
66549
|
releaseRoutes;
|
|
66550
|
+
routeClaims = new Map;
|
|
65353
66551
|
sources = [];
|
|
65354
66552
|
queue;
|
|
65355
66553
|
lease;
|
|
65356
|
-
admission;
|
|
65357
|
-
usageLedger;
|
|
65358
66554
|
usageTotal;
|
|
65359
66555
|
rootBudget;
|
|
66556
|
+
budgetBlocked = false;
|
|
66557
|
+
rootSignature;
|
|
65360
66558
|
starting;
|
|
65361
|
-
cleanupFailure;
|
|
65362
66559
|
failure;
|
|
65363
66560
|
closed = false;
|
|
65364
|
-
replacements = 0;
|
|
65365
66561
|
releaseBinding;
|
|
65366
66562
|
constructor(backend, options) {
|
|
65367
66563
|
this.backend = backend;
|
|
@@ -65376,10 +66572,12 @@ class LocalWorkerPool {
|
|
|
65376
66572
|
this.releaseBinding = bindChildWorkerDispatcher(backend, this, options.rootId);
|
|
65377
66573
|
if (options.admission && options.rootId === undefined)
|
|
65378
66574
|
workflowResourceAdmission.set(backend, options.admission);
|
|
65379
|
-
this.ticker = setInterval(() => this.observeResponsiveness(), ROOT_TICK_INTERVAL_MS).unref();
|
|
65380
66575
|
}
|
|
65381
66576
|
get workerPids() {
|
|
65382
|
-
return [...this.
|
|
66577
|
+
return [...this.pids];
|
|
66578
|
+
}
|
|
66579
|
+
get coordinatorPid() {
|
|
66580
|
+
return this.coordinator?.pid;
|
|
65383
66581
|
}
|
|
65384
66582
|
usage() {
|
|
65385
66583
|
return this.usageTotal;
|
|
@@ -65387,47 +66585,39 @@ class LocalWorkerPool {
|
|
|
65387
66585
|
async prepare(root, store) {
|
|
65388
66586
|
if (this.closed)
|
|
65389
66587
|
throw new Error("Local worker pool is closed");
|
|
65390
|
-
if (!(this.backend.journal instanceof SqlRunJournal) ||
|
|
66588
|
+
if (!(this.backend.journal instanceof SqlRunJournal) || resolve14(this.options.journalPath) !== resolve14(this.backend.journal.path) || sqlPoolStorage(this.backend.journal)?.tables.runs !== "wf_runs")
|
|
65391
66589
|
throw new Error("Local workers require the same default-prefix SQLite journal");
|
|
65392
66590
|
if (this.root !== undefined && this.root !== root)
|
|
65393
66591
|
throw new Error("Local worker pool belongs to another root");
|
|
65394
66592
|
this.root = root;
|
|
65395
66593
|
if (store)
|
|
65396
66594
|
this.store = store;
|
|
65397
|
-
this.usageLedger ??= await openSqlPoolUsage(this.backend.journal);
|
|
65398
|
-
this.usageTotal = await this.usageLedger.total(root);
|
|
65399
66595
|
const state = await this.backend.journal.leaseState(root);
|
|
65400
66596
|
if (!state || state.holderId !== this.backend.holderId)
|
|
65401
66597
|
throw new Error("Local worker pool requires the root owner");
|
|
65402
66598
|
this.lease = { runId: root, holderId: state.holderId, epoch: state.epoch };
|
|
65403
|
-
const
|
|
65404
|
-
await
|
|
65405
|
-
|
|
65406
|
-
|
|
65407
|
-
|
|
65408
|
-
|
|
65409
|
-
|
|
65410
|
-
|
|
65411
|
-
|
|
65412
|
-
|
|
66599
|
+
const workerEntry = this.options.workerEntry ?? (isBunBinary2 || this.options.coordinator === "in-process" ? defaultWorkerEntry() : undefined);
|
|
66600
|
+
this.usageTotal = await this.command({
|
|
66601
|
+
kind: "prepare",
|
|
66602
|
+
root,
|
|
66603
|
+
lease: this.lease,
|
|
66604
|
+
options: {
|
|
66605
|
+
journalPath: this.options.journalPath,
|
|
66606
|
+
workers: this.options.workers,
|
|
66607
|
+
leaseMs: this.leaseMs,
|
|
66608
|
+
startupTimeoutMs: this.options.startupTimeoutMs ?? DEFAULT_STARTUP_TIMEOUT_MS,
|
|
66609
|
+
...this.options.admission === undefined ? {} : { admission: this.options.admission },
|
|
66610
|
+
...workerEntry === undefined ? {} : { workerEntry }
|
|
66611
|
+
},
|
|
66612
|
+
...this.desiredControl === undefined ? {} : { control: this.desiredControl }
|
|
66613
|
+
});
|
|
66614
|
+
this.queue ??= await openSqlPoolQueue(this.backend.journal);
|
|
65413
66615
|
}
|
|
65414
66616
|
async control(state) {
|
|
65415
66617
|
this.desiredControl = state;
|
|
65416
66618
|
try {
|
|
65417
|
-
if (
|
|
65418
|
-
|
|
65419
|
-
const control = await this.queue.setControl(this.lease, state);
|
|
65420
|
-
const deadline = Date.now() + CONTROL_ACKNOWLEDGEMENT_TIMEOUT_MS;
|
|
65421
|
-
while ([...this.readyWorkers].some((worker) => (this.controlAcks.get(worker) ?? -1) < control.revision)) {
|
|
65422
|
-
if (this.desiredControl !== state)
|
|
65423
|
-
return;
|
|
65424
|
-
if (Date.now() >= deadline) {
|
|
65425
|
-
if (state === "quit")
|
|
65426
|
-
break;
|
|
65427
|
-
throw new Error("Worker control acknowledgement unresolved");
|
|
65428
|
-
}
|
|
65429
|
-
await delay2(25);
|
|
65430
|
-
}
|
|
66619
|
+
if (this.coordinator !== undefined)
|
|
66620
|
+
await this.command({ kind: "control", state });
|
|
65431
66621
|
} finally {
|
|
65432
66622
|
if (state === "quit")
|
|
65433
66623
|
await this.close();
|
|
@@ -65473,290 +66663,61 @@ class LocalWorkerPool {
|
|
|
65473
66663
|
throw new Error(`Worker definition is not importable: ${binding.exportName}`);
|
|
65474
66664
|
return source;
|
|
65475
66665
|
}));
|
|
65476
|
-
|
|
65477
|
-
|
|
66666
|
+
const cwd = String(this.backend.getWorkflow(root)?.invocationCwd ?? process.cwd());
|
|
66667
|
+
const store = this.store;
|
|
66668
|
+
if (store) {
|
|
66669
|
+
this.releaseRoutes = await acquireStandaloneIntercom(store, root, cwd);
|
|
65478
66670
|
if (this.closed) {
|
|
65479
66671
|
await this.releaseRoutes();
|
|
65480
66672
|
this.releaseRoutes = undefined;
|
|
65481
66673
|
throw new Error("Local worker pool is closed");
|
|
65482
66674
|
}
|
|
65483
|
-
|
|
65484
|
-
|
|
65485
|
-
|
|
65486
|
-
this.
|
|
65487
|
-
|
|
65488
|
-
root,
|
|
65489
|
-
holder: "coordinator"
|
|
66675
|
+
const routeOwner = {
|
|
66676
|
+
notifyUndeliverable: async (payload) => !this.closed && await this.command({ kind: "undeliverable", payload }) === true
|
|
66677
|
+
};
|
|
66678
|
+
this.handoff = new PoolStageHandoff(this.backend, store, this.queueView(), this.lease, (runId) => {
|
|
66679
|
+
this.routeClaims.set(runId, claimExternalRunRoute(store, runId, routeOwner));
|
|
65490
66680
|
});
|
|
65491
|
-
this.logDirectory = workerLogDirectory(this.options.journalPath, root);
|
|
65492
|
-
await mkdir6(this.logDirectory, { recursive: true }).catch(() => {
|
|
65493
|
-
return;
|
|
65494
|
-
});
|
|
65495
|
-
const workers = Array.from({ length: this.options.workers }, () => this.spawn(root));
|
|
65496
|
-
try {
|
|
65497
|
-
await Promise.any(workers);
|
|
65498
|
-
} catch (error) {
|
|
65499
|
-
throw error.errors[0];
|
|
65500
66681
|
}
|
|
65501
|
-
|
|
65502
|
-
|
|
65503
|
-
|
|
65504
|
-
|
|
65505
|
-
|
|
65506
|
-
|
|
65507
|
-
journalPath: this.options.journalPath,
|
|
65508
|
-
root,
|
|
65509
|
-
definitions: this.sources,
|
|
65510
|
-
admission: this.options.admission
|
|
65511
|
-
})
|
|
65512
|
-
], { stdio: ["ignore", "ignore", "pipe", "ipc"], execArgv: [] });
|
|
65513
|
-
const disconnected = new Promise((resolve) => child.once("disconnect", () => resolve()));
|
|
65514
|
-
this.children.add(child);
|
|
65515
|
-
const worker = randomUUID17();
|
|
65516
|
-
this.incarnations.set(worker, child);
|
|
65517
|
-
let ready = false;
|
|
65518
|
-
let serial = Promise.resolve();
|
|
65519
|
-
let work;
|
|
65520
|
-
const deliveries = new Map;
|
|
65521
|
-
const logPath = join29(this.logDirectory, `${worker}.log`);
|
|
65522
|
-
this.logs.set(child, logPath);
|
|
65523
|
-
let logFailure;
|
|
65524
|
-
const log = createWriteStream(logPath, { flags: "a" });
|
|
65525
|
-
log.on("error", (error) => {
|
|
65526
|
-
logFailure = error.message;
|
|
65527
|
-
});
|
|
65528
|
-
let stderr = "";
|
|
65529
|
-
child.stderr?.on("data", (chunk) => {
|
|
65530
|
-
stderr = (stderr + chunk.toString()).slice(-STDERR_TAIL_CHARACTERS);
|
|
65531
|
-
});
|
|
65532
|
-
child.stderr?.on("error", (error) => {
|
|
65533
|
-
logFailure ??= error.message;
|
|
65534
|
-
log.end();
|
|
65535
|
-
});
|
|
65536
|
-
child.stderr?.pipe(log);
|
|
65537
|
-
const diagnosed = (summary) => new Error(`${summary} (worker stderr log: ${logFailure === undefined ? logPath : `unavailable, ${logFailure}`})${stderr ? `: ${stderr}` : ""}`);
|
|
65538
|
-
const startupTimeoutMs = this.options.startupTimeoutMs ?? DEFAULT_STARTUP_TIMEOUT_MS;
|
|
65539
|
-
let timedOut = false;
|
|
65540
|
-
let terminated = false;
|
|
65541
|
-
const terminate = () => {
|
|
65542
|
-
if (terminated)
|
|
65543
|
-
return;
|
|
65544
|
-
terminated = true;
|
|
65545
|
-
child.kill("SIGKILL");
|
|
65546
|
-
};
|
|
65547
|
-
const replied = (error) => {
|
|
65548
|
-
if (error)
|
|
65549
|
-
terminate();
|
|
65550
|
-
};
|
|
65551
|
-
return new Promise((resolve, reject) => {
|
|
65552
|
-
const startup = setTimeout(() => {
|
|
65553
|
-
timedOut = true;
|
|
65554
|
-
terminate();
|
|
65555
|
-
}, startupTimeoutMs);
|
|
65556
|
-
child.on("error", (error) => {
|
|
65557
|
-
clearTimeout(startup);
|
|
65558
|
-
reject(error);
|
|
65559
|
-
terminate();
|
|
65560
|
-
});
|
|
65561
|
-
const closed = new Promise((resolve) => child.once("close", () => resolve()));
|
|
65562
|
-
child.once("exit", () => {
|
|
65563
|
-
Promise.race([closed, delay2(STDERR_DRAIN_GRACE_MS)]).then(() => reject(diagnosed(timedOut ? `Local worker startup timed out after ${startupTimeoutMs} ms` : "Local worker exited before readiness")));
|
|
65564
|
-
this.children.delete(child);
|
|
65565
|
-
this.logs.delete(child);
|
|
65566
|
-
clearTimeout(startup);
|
|
65567
|
-
this.incarnations.delete(worker);
|
|
65568
|
-
this.readyWorkers.delete(worker);
|
|
65569
|
-
this.controlAcks.delete(worker);
|
|
65570
|
-
this.lastHeard.delete(worker);
|
|
65571
|
-
if (work)
|
|
65572
|
-
this.handoff?.fence(work);
|
|
65573
|
-
for (const delivery of deliveries.values())
|
|
65574
|
-
delivery.reject(new Error("Worker exited before delivery acknowledgement"));
|
|
65575
|
-
deliveries.clear();
|
|
65576
|
-
const cleanup = disconnected.then(() => serial).then(async () => {
|
|
65577
|
-
try {
|
|
65578
|
-
if (await this.admission?.releaseStoppedSessions(worker))
|
|
65579
|
-
throw new Error("Local worker process cleanup unconfirmed; permits quarantined");
|
|
65580
|
-
} catch (error) {
|
|
65581
|
-
this.cleanupFailure = error instanceof Error ? error : new Error(String(error));
|
|
65582
|
-
throw this.cleanupFailure;
|
|
65583
|
-
}
|
|
65584
|
-
}).then(async () => {
|
|
65585
|
-
if (this.closed || !ready || this.desiredControl === "quit")
|
|
65586
|
-
return;
|
|
65587
|
-
if (work && await this.queue.result(root, work.id)) {
|
|
65588
|
-
work = undefined;
|
|
65589
|
-
this.replacements = 0;
|
|
65590
|
-
}
|
|
65591
|
-
if (this.closed)
|
|
65592
|
-
return;
|
|
65593
|
-
if (work)
|
|
65594
|
-
await this.queue.releaseExited(this.lease, work);
|
|
65595
|
-
const exhausted = work ? work.generation >= 3 : ++this.replacements > 3;
|
|
65596
|
-
if (exhausted)
|
|
65597
|
-
this.failure = diagnosed("Local worker retry limit");
|
|
65598
|
-
else
|
|
65599
|
-
await this.spawn(root);
|
|
65600
|
-
}).catch((error) => {
|
|
65601
|
-
this.failure = error;
|
|
65602
|
-
});
|
|
65603
|
-
this.cleanups.add(cleanup);
|
|
65604
|
-
cleanup.finally(() => this.cleanups.delete(cleanup));
|
|
65605
|
-
});
|
|
65606
|
-
child.on("message", (message) => {
|
|
65607
|
-
this.observeResponsiveness();
|
|
65608
|
-
this.lastHeard.set(worker, { at: performance.now(), rootStalled: this.rootStalledMs });
|
|
65609
|
-
if (message.kind === "delivery-ack") {
|
|
65610
|
-
const pending = deliveries.get(message.deliveryId);
|
|
65611
|
-
deliveries.delete(message.deliveryId);
|
|
65612
|
-
if (message.error)
|
|
65613
|
-
pending?.reject(new Error(message.error));
|
|
65614
|
-
else
|
|
65615
|
-
pending?.resolve();
|
|
65616
|
-
return;
|
|
65617
|
-
}
|
|
65618
|
-
const response = serial.then(async () => {
|
|
65619
|
-
if (message.kind === "ready") {
|
|
65620
|
-
if (terminated)
|
|
65621
|
-
return;
|
|
65622
|
-
ready = true;
|
|
65623
|
-
this.readyWorkers.add(worker);
|
|
65624
|
-
clearTimeout(startup);
|
|
65625
|
-
resolve();
|
|
65626
|
-
return;
|
|
65627
|
-
}
|
|
65628
|
-
await this.starting;
|
|
65629
|
-
const queue = this.queue;
|
|
65630
|
-
const lease = this.lease;
|
|
65631
|
-
if (message.kind === "control")
|
|
65632
|
-
return queue.control(lease);
|
|
65633
|
-
if (message.kind === "controlled") {
|
|
65634
|
-
const control = await queue.control(lease);
|
|
65635
|
-
if (message.revision > control.revision)
|
|
65636
|
-
throw new Error("Unknown control revision");
|
|
65637
|
-
this.controlAcks.set(worker, Math.max(this.controlAcks.get(worker) ?? -1, message.revision));
|
|
65638
|
-
return;
|
|
65639
|
-
}
|
|
65640
|
-
if (message.kind === "claim") {
|
|
65641
|
-
if (work)
|
|
65642
|
-
throw new Error("Worker already holds an attempt");
|
|
65643
|
-
const decision = this.rootBudget?.checkpoint();
|
|
65644
|
-
if (decision?.kind === "exhausted" || decision?.kind === "wrap_up")
|
|
65645
|
-
return;
|
|
65646
|
-
await this.recoverExpired(root);
|
|
65647
|
-
work = await queue.claim(lease, worker, this.sources.map((source) => ({
|
|
65648
|
-
definition: `${source.path}#${source.exportName}`,
|
|
65649
|
-
digest: source.digest
|
|
65650
|
-
})), this.leaseMs);
|
|
65651
|
-
return work;
|
|
65652
|
-
}
|
|
65653
|
-
if (!work)
|
|
65654
|
-
throw new Error("Worker has no active attempt");
|
|
65655
|
-
if (message.kind === "renew" || work.expiresAt - Date.now() < this.leaseMs / 2)
|
|
65656
|
-
work = await queue.extend(lease, work, this.leaseMs);
|
|
65657
|
-
if (message.kind === "activity")
|
|
65658
|
-
return this.handoff?.publish(work, message.runs);
|
|
65659
|
-
if (message.kind === "route")
|
|
65660
|
-
return this.handoff?.route(work, message.runId);
|
|
65661
|
-
if (message.kind === "delivery-ready")
|
|
65662
|
-
return this.handoff?.ready(work, message.runId, message.stageId, message.stageName, message.sessionId);
|
|
65663
|
-
if (message.kind === "delivery") {
|
|
65664
|
-
if (!message.recipient)
|
|
65665
|
-
throw new Error("Worker delivery requires a session identity");
|
|
65666
|
-
return this.handoff?.deliver(work, message.runId, message.stageId, message.stageName, message.recipient, (from, payload) => new Promise((resolve, reject) => {
|
|
65667
|
-
const id = randomUUID17();
|
|
65668
|
-
deliveries.set(id, { resolve, reject });
|
|
65669
|
-
child.send({ id: message.id, delivery: { id, from, message: payload } }, (error) => {
|
|
65670
|
-
if (error) {
|
|
65671
|
-
deliveries.delete(id);
|
|
65672
|
-
reject(error);
|
|
65673
|
-
}
|
|
65674
|
-
});
|
|
65675
|
-
}));
|
|
65676
|
-
}
|
|
65677
|
-
if (message.kind === "renew")
|
|
65678
|
-
return;
|
|
65679
|
-
if (message.kind === "commit") {
|
|
65680
|
-
const receipt = await queue.commit(lease, work, message.operation, message.mutation);
|
|
65681
|
-
this.usageTotal = await this.usageLedger.total(root);
|
|
65682
|
-
return receipt;
|
|
65683
|
-
}
|
|
65684
|
-
if (message.kind === "usage") {
|
|
65685
|
-
this.usageTotal = await this.usageLedger.report(lease, work, message.reports);
|
|
65686
|
-
return;
|
|
65687
|
-
}
|
|
65688
|
-
if (message.kind === "budget") {
|
|
65689
|
-
await queue.assertCurrent(lease, work);
|
|
65690
|
-
const decision = this.rootBudget?.checkpoint();
|
|
65691
|
-
if (decision?.kind === "exhausted" || decision?.kind === "wrap_up")
|
|
65692
|
-
throw new Error("Root workflow budget exhausted");
|
|
65693
|
-
return queue.control(lease);
|
|
65694
|
-
}
|
|
65695
|
-
if (message.kind !== "complete")
|
|
65696
|
-
throw new Error("Invalid worker operation");
|
|
65697
|
-
await queue.complete(lease, work, message.result);
|
|
65698
|
-
this.replacements = 0;
|
|
65699
|
-
work = undefined;
|
|
65700
|
-
});
|
|
65701
|
-
if (message.kind !== "delivery" && message.kind !== "delivery-ready")
|
|
65702
|
-
serial = response.then(() => {}, () => {});
|
|
65703
|
-
response.then((value2) => child.send({ id: message.id, value: value2 }, replied), (error) => child.send({ id: message.id, error: error.message }, replied));
|
|
65704
|
-
});
|
|
66682
|
+
await this.command({
|
|
66683
|
+
kind: "start",
|
|
66684
|
+
sources: this.sources,
|
|
66685
|
+
cwd,
|
|
66686
|
+
publish: store !== undefined,
|
|
66687
|
+
...this.rootRun()
|
|
65705
66688
|
});
|
|
65706
66689
|
}
|
|
65707
|
-
|
|
65708
|
-
|
|
65709
|
-
|
|
65710
|
-
|
|
65711
|
-
|
|
65712
|
-
|
|
65713
|
-
|
|
65714
|
-
|
|
65715
|
-
|
|
65716
|
-
|
|
65717
|
-
if (work.state !== "leased" || work.expiresAt >= expiredBefore)
|
|
65718
|
-
continue;
|
|
65719
|
-
const child = work.worker === null ? undefined : this.incarnations.get(work.worker);
|
|
65720
|
-
if (child && work.worker !== null && this.isLive(work.worker, child)) {
|
|
65721
|
-
await this.queue.extend(this.lease, work, this.leaseMs).catch((error) => {
|
|
65722
|
-
if (error.message !== STALE_WORK_LEASE)
|
|
65723
|
-
throw error;
|
|
65724
|
-
});
|
|
65725
|
-
continue;
|
|
65726
|
-
}
|
|
65727
|
-
if (child && !await this.stop(child).then(() => true, () => false))
|
|
65728
|
-
continue;
|
|
65729
|
-
await this.queue.expire(this.lease, work);
|
|
65730
|
-
}
|
|
65731
|
-
})().finally(() => {
|
|
65732
|
-
this.recovering = undefined;
|
|
65733
|
-
this.nextRecoveryAt = performance.now() + RECOVERY_INTERVAL_MS;
|
|
65734
|
-
});
|
|
65735
|
-
return this.recovering;
|
|
66690
|
+
queueView() {
|
|
66691
|
+
return {
|
|
66692
|
+
assertCurrent: async (_lease, work) => {
|
|
66693
|
+
const current = this.attempts.get(work.id);
|
|
66694
|
+
if (current === undefined || current.fenced || current.generation !== work.generation || current.worker !== work.worker || current.epoch !== work.epoch)
|
|
66695
|
+
throw new Error(STALE_WORK_LEASE);
|
|
66696
|
+
},
|
|
66697
|
+
withCurrentAppend: (lease, work, assertSession, action) => this.queue.withCurrentAppend(lease, work, assertSession, action),
|
|
66698
|
+
result: async (root, id) => this.results.get(id) ?? this.queue.result(root, id)
|
|
66699
|
+
};
|
|
65736
66700
|
}
|
|
65737
|
-
|
|
65738
|
-
const
|
|
65739
|
-
|
|
65740
|
-
if (gap > ROOT_STALL_THRESHOLD_MS)
|
|
65741
|
-
this.rootStalledMs += gap - ROOT_TICK_INTERVAL_MS;
|
|
65742
|
-
this.lastTick = now;
|
|
66701
|
+
rootProjection() {
|
|
66702
|
+
const run = this.store?.runs().find((candidate) => candidate.id === this.root);
|
|
66703
|
+
return run === undefined ? undefined : projectRootRun(run);
|
|
65743
66704
|
}
|
|
65744
|
-
|
|
65745
|
-
this.
|
|
65746
|
-
|
|
65747
|
-
|
|
66705
|
+
rootRun() {
|
|
66706
|
+
const rootRun = this.rootProjection();
|
|
66707
|
+
if (rootRun === undefined)
|
|
66708
|
+
return {};
|
|
66709
|
+
this.rootSignature = JSON.stringify(rootRun);
|
|
66710
|
+
return { rootRun };
|
|
65748
66711
|
}
|
|
65749
|
-
|
|
65750
|
-
|
|
65751
|
-
|
|
65752
|
-
|
|
65753
|
-
|
|
65754
|
-
|
|
65755
|
-
|
|
65756
|
-
|
|
65757
|
-
|
|
65758
|
-
child.kill("SIGKILL");
|
|
65759
|
-
});
|
|
66712
|
+
refreshRootRun() {
|
|
66713
|
+
const rootRun = this.rootProjection();
|
|
66714
|
+
if (rootRun === undefined)
|
|
66715
|
+
return;
|
|
66716
|
+
const signature = JSON.stringify(rootRun);
|
|
66717
|
+
if (signature === this.rootSignature)
|
|
66718
|
+
return;
|
|
66719
|
+
this.rootSignature = signature;
|
|
66720
|
+
this.notice({ kind: "root", rootRun });
|
|
65760
66721
|
}
|
|
65761
66722
|
async run(definition, inputs, options) {
|
|
65762
66723
|
try {
|
|
@@ -65789,7 +66750,7 @@ class LocalWorkerPool {
|
|
|
65789
66750
|
const selectedSource = this.sources[index];
|
|
65790
66751
|
if (await definitionDigest(selectedSource.path) !== selectedSource.digest)
|
|
65791
66752
|
throw new Error("Worker definition compatibility error");
|
|
65792
|
-
await this.backend.
|
|
66753
|
+
await this.backend.catchUpWorkflow(root);
|
|
65793
66754
|
const boundary = this.backend.listCheckpoints(root).find((entry) => entry.kind === "stage" && entry.topology?.boundary?.event === "start" && entry.topology.stageId === options.parentRun.stageId);
|
|
65794
66755
|
if (boundary?.kind !== "stage")
|
|
65795
66756
|
throw new Error("Missing durable child boundary");
|
|
@@ -65800,39 +66761,232 @@ class LocalWorkerPool {
|
|
|
65800
66761
|
config: options.config,
|
|
65801
66762
|
executionMode: options.executionMode
|
|
65802
66763
|
}));
|
|
65803
|
-
|
|
65804
|
-
|
|
65805
|
-
|
|
65806
|
-
|
|
65807
|
-
|
|
65808
|
-
|
|
65809
|
-
|
|
65810
|
-
|
|
65811
|
-
|
|
65812
|
-
|
|
65813
|
-
|
|
65814
|
-
|
|
66764
|
+
this.evaluateBudget();
|
|
66765
|
+
const enqueued = await this.command({
|
|
66766
|
+
kind: "enqueue",
|
|
66767
|
+
descriptor: {
|
|
66768
|
+
boundaryId: options.parentRun.stageId,
|
|
66769
|
+
childRunId: options.runId,
|
|
66770
|
+
scope: options.durableScope.scopePrefix,
|
|
66771
|
+
definition: `${selectedSource.path}#${selectedSource.exportName}`,
|
|
66772
|
+
digest: selectedSource.digest,
|
|
66773
|
+
inputs,
|
|
66774
|
+
execution
|
|
66775
|
+
},
|
|
66776
|
+
boundary,
|
|
66777
|
+
...this.rootRun()
|
|
66778
|
+
});
|
|
66779
|
+
const work = enqueued.work;
|
|
66780
|
+
this.works.set(work.id, work);
|
|
66781
|
+
if (enqueued.result !== undefined && !this.results.has(work.id))
|
|
66782
|
+
this.results.set(work.id, enqueued.result);
|
|
66783
|
+
this.waiting++;
|
|
66784
|
+
this.ticker ??= setInterval(() => {
|
|
66785
|
+
this.evaluateBudget();
|
|
66786
|
+
this.signalChange();
|
|
66787
|
+
}, POOL_SUMMARY_INTERVAL_MS).unref();
|
|
66788
|
+
try {
|
|
66789
|
+
for (;; ) {
|
|
66790
|
+
options.signal?.throwIfAborted();
|
|
66791
|
+
if (this.closed) {
|
|
66792
|
+
if (this.failure)
|
|
66793
|
+
throw this.failure;
|
|
66794
|
+
if (this.desiredControl === "quit")
|
|
66795
|
+
throw new WorkflowGracefulQuitError(root, "worker pool teardown");
|
|
66796
|
+
throw new Error("Local worker pool is closed");
|
|
66797
|
+
}
|
|
66798
|
+
const result = this.results.get(work.id);
|
|
66799
|
+
if (result) {
|
|
66800
|
+
await this.inbound;
|
|
66801
|
+
await this.backend.catchUpWorkflow(root);
|
|
66802
|
+
await this.handoff?.complete(work);
|
|
66803
|
+
return result;
|
|
66804
|
+
}
|
|
65815
66805
|
if (this.failure)
|
|
65816
66806
|
throw this.failure;
|
|
65817
|
-
if (this.
|
|
65818
|
-
|
|
65819
|
-
|
|
66807
|
+
if (this.budgetBlocked)
|
|
66808
|
+
await this.rootBudget.stopAtBoundaryAsync(definition.name);
|
|
66809
|
+
await this.nextChange(options.signal);
|
|
65820
66810
|
}
|
|
65821
|
-
|
|
65822
|
-
if (
|
|
65823
|
-
|
|
65824
|
-
|
|
65825
|
-
return result;
|
|
66811
|
+
} finally {
|
|
66812
|
+
if (--this.waiting === 0) {
|
|
66813
|
+
clearInterval(this.ticker);
|
|
66814
|
+
this.ticker = undefined;
|
|
65826
66815
|
}
|
|
65827
|
-
if (this.failure)
|
|
65828
|
-
throw this.failure;
|
|
65829
|
-
const decision = this.rootBudget?.checkpoint();
|
|
65830
|
-
if (decision?.kind === "exhausted" || decision?.kind === "wrap_up")
|
|
65831
|
-
await this.rootBudget.stopAtBoundaryAsync(definition.name);
|
|
65832
|
-
await this.recoverExpired(root);
|
|
65833
|
-
await delay2(25, undefined, { signal: options.signal });
|
|
65834
66816
|
}
|
|
65835
66817
|
}
|
|
66818
|
+
evaluateBudget() {
|
|
66819
|
+
const decision = this.rootBudget?.checkpoint();
|
|
66820
|
+
const blocked = decision?.kind === "exhausted" || decision?.kind === "wrap_up";
|
|
66821
|
+
if (blocked === this.budgetBlocked)
|
|
66822
|
+
return;
|
|
66823
|
+
this.budgetBlocked = blocked;
|
|
66824
|
+
this.notice({ kind: "budget", blocked });
|
|
66825
|
+
}
|
|
66826
|
+
signalChange() {
|
|
66827
|
+
const changed = this.changed;
|
|
66828
|
+
this.changed = Promise.withResolvers();
|
|
66829
|
+
changed.resolve();
|
|
66830
|
+
}
|
|
66831
|
+
nextChange(signal) {
|
|
66832
|
+
const changed = this.changed.promise;
|
|
66833
|
+
if (signal === undefined)
|
|
66834
|
+
return changed;
|
|
66835
|
+
return new Promise((resolve, reject) => {
|
|
66836
|
+
const abort = () => reject(signal.reason);
|
|
66837
|
+
signal.addEventListener("abort", abort, { once: true });
|
|
66838
|
+
changed.then(() => {
|
|
66839
|
+
signal.removeEventListener("abort", abort);
|
|
66840
|
+
resolve();
|
|
66841
|
+
});
|
|
66842
|
+
});
|
|
66843
|
+
}
|
|
66844
|
+
connect() {
|
|
66845
|
+
if (this.coordinator !== undefined)
|
|
66846
|
+
return this.coordinator;
|
|
66847
|
+
const coordinator = this.options.coordinator === "in-process" ? launchCoordinatorInProcess() : launchCoordinatorProcess();
|
|
66848
|
+
this.coordinator = coordinator;
|
|
66849
|
+
coordinator.port.onMessage((event) => {
|
|
66850
|
+
this.inbound = this.inbound.then(() => this.receive(event)).catch(() => {});
|
|
66851
|
+
});
|
|
66852
|
+
coordinator.port.onClose((reason) => {
|
|
66853
|
+
this.coordinatorLost = reason;
|
|
66854
|
+
this.inbound = this.inbound.then(() => this.lost(reason));
|
|
66855
|
+
});
|
|
66856
|
+
return coordinator;
|
|
66857
|
+
}
|
|
66858
|
+
lost(reason) {
|
|
66859
|
+
this.pids.clear();
|
|
66860
|
+
if (!this.closed)
|
|
66861
|
+
this.failure ??= reason;
|
|
66862
|
+
for (const pending of this.commands.values())
|
|
66863
|
+
pending.reject(reason);
|
|
66864
|
+
this.commands.clear();
|
|
66865
|
+
this.signalChange();
|
|
66866
|
+
}
|
|
66867
|
+
command(body) {
|
|
66868
|
+
const coordinator = this.connect();
|
|
66869
|
+
if (this.coordinatorLost !== undefined)
|
|
66870
|
+
return Promise.reject(this.coordinatorLost);
|
|
66871
|
+
const id = ++this.commandSequence;
|
|
66872
|
+
return new Promise((resolve, reject) => {
|
|
66873
|
+
this.commands.set(id, { resolve, reject });
|
|
66874
|
+
coordinator.port.send({ ...body, id });
|
|
66875
|
+
});
|
|
66876
|
+
}
|
|
66877
|
+
notice(notice) {
|
|
66878
|
+
this.coordinator?.port.send(notice);
|
|
66879
|
+
}
|
|
66880
|
+
async receive(event) {
|
|
66881
|
+
switch (event.kind) {
|
|
66882
|
+
case "reply": {
|
|
66883
|
+
const pending = this.commands.get(event.id);
|
|
66884
|
+
this.commands.delete(event.id);
|
|
66885
|
+
if (event.error !== undefined)
|
|
66886
|
+
pending?.reject(new Error(event.error));
|
|
66887
|
+
else
|
|
66888
|
+
pending?.resolve(event.value);
|
|
66889
|
+
return;
|
|
66890
|
+
}
|
|
66891
|
+
case "activity":
|
|
66892
|
+
return this.applyActivity(event.batch, event.entries, event.usage);
|
|
66893
|
+
case "spawned":
|
|
66894
|
+
this.pids.add(event.pid);
|
|
66895
|
+
return;
|
|
66896
|
+
case "exited":
|
|
66897
|
+
this.pids.delete(event.pid);
|
|
66898
|
+
return;
|
|
66899
|
+
case "fence": {
|
|
66900
|
+
const current = this.attempts.get(event.work.id);
|
|
66901
|
+
if (current?.generation === event.work.generation)
|
|
66902
|
+
current.fenced = true;
|
|
66903
|
+
this.handoff?.fence(event.work);
|
|
66904
|
+
return;
|
|
66905
|
+
}
|
|
66906
|
+
case "result":
|
|
66907
|
+
if (event.usage !== undefined)
|
|
66908
|
+
this.usageTotal = event.usage;
|
|
66909
|
+
this.results.set(event.workId, event.result);
|
|
66910
|
+
this.signalChange();
|
|
66911
|
+
return;
|
|
66912
|
+
case "failure":
|
|
66913
|
+
this.failure ??= new Error(event.error);
|
|
66914
|
+
this.signalChange();
|
|
66915
|
+
return;
|
|
66916
|
+
case "request":
|
|
66917
|
+
this.serve(event.id, event.request);
|
|
66918
|
+
return;
|
|
66919
|
+
}
|
|
66920
|
+
}
|
|
66921
|
+
async applyActivity(batch, entries, usage) {
|
|
66922
|
+
if (usage !== undefined)
|
|
66923
|
+
this.usageTotal = usage;
|
|
66924
|
+
const rejected = [];
|
|
66925
|
+
const handoff = this.handoff;
|
|
66926
|
+
if (handoff !== undefined) {
|
|
66927
|
+
const publications = entries.flatMap((entry) => {
|
|
66928
|
+
const work = this.track(entry.work);
|
|
66929
|
+
if (work !== undefined)
|
|
66930
|
+
return [{ work, runs: entry.runs }];
|
|
66931
|
+
rejected.push({ workId: entry.work.id, generation: entry.work.generation, error: STALE_WORK_LEASE });
|
|
66932
|
+
return [];
|
|
66933
|
+
});
|
|
66934
|
+
const outcomes = await handoff.publishBatch(publications);
|
|
66935
|
+
outcomes.forEach((error, index) => {
|
|
66936
|
+
const { work } = publications[index];
|
|
66937
|
+
if (error !== undefined)
|
|
66938
|
+
rejected.push({ workId: work.id, generation: work.generation, error: error.message });
|
|
66939
|
+
});
|
|
66940
|
+
}
|
|
66941
|
+
this.notice({ kind: "published", batch, rejected });
|
|
66942
|
+
this.refreshRootRun();
|
|
66943
|
+
}
|
|
66944
|
+
track(attempt) {
|
|
66945
|
+
const work = this.works.get(attempt.id);
|
|
66946
|
+
if (work === undefined)
|
|
66947
|
+
return;
|
|
66948
|
+
const current = this.attempts.get(attempt.id);
|
|
66949
|
+
if (current === undefined || attempt.generation > current.generation)
|
|
66950
|
+
this.attempts.set(attempt.id, { ...attempt, fenced: false });
|
|
66951
|
+
return { ...work, ...attempt, state: "leased" };
|
|
66952
|
+
}
|
|
66953
|
+
async serve(id, request) {
|
|
66954
|
+
try {
|
|
66955
|
+
const value2 = await this.answer(id, request);
|
|
66956
|
+
this.coordinator?.port.send({ kind: "request-reply", id, ...value2 === undefined ? {} : { value: value2 } });
|
|
66957
|
+
} catch (error) {
|
|
66958
|
+
this.coordinator?.port.send({
|
|
66959
|
+
kind: "request-reply",
|
|
66960
|
+
id,
|
|
66961
|
+
error: error instanceof Error ? error.message : String(error)
|
|
66962
|
+
});
|
|
66963
|
+
}
|
|
66964
|
+
}
|
|
66965
|
+
async answer(id, request) {
|
|
66966
|
+
if (request.kind === "pending-stage-message" || request.kind === "sticky-live-delivered") {
|
|
66967
|
+
const router = this.store === undefined ? undefined : pendingStageMessageRouter(this.store);
|
|
66968
|
+
const event = {
|
|
66969
|
+
...request.payload,
|
|
66970
|
+
handled: false
|
|
66971
|
+
};
|
|
66972
|
+
if (request.kind === "pending-stage-message")
|
|
66973
|
+
router?.route(event);
|
|
66974
|
+
else
|
|
66975
|
+
router?.sticky(event);
|
|
66976
|
+
if (event.handled && event.completion !== undefined)
|
|
66977
|
+
return await event.completion;
|
|
66978
|
+
return request.kind === "pending-stage-message" ? { outcome: "refused", reason: "Session not found" } : undefined;
|
|
66979
|
+
}
|
|
66980
|
+
const handoff = this.handoff;
|
|
66981
|
+
const work = this.track(request.work);
|
|
66982
|
+
if (handoff === undefined || work === undefined)
|
|
66983
|
+
throw new Error(STALE_WORK_LEASE);
|
|
66984
|
+
if (request.kind === "ready")
|
|
66985
|
+
return handoff.ready(work, request.runId, request.stageId, request.stageName, request.sessionId);
|
|
66986
|
+
return handoff.deliver(work, request.runId, request.stageId, request.stageName, request.recipient, async (from, message) => {
|
|
66987
|
+
await this.command({ kind: "transport", request: id, from, message });
|
|
66988
|
+
});
|
|
66989
|
+
}
|
|
65836
66990
|
closing;
|
|
65837
66991
|
close() {
|
|
65838
66992
|
this.closing ??= this.dispose();
|
|
@@ -65841,27 +66995,28 @@ class LocalWorkerPool {
|
|
|
65841
66995
|
async dispose() {
|
|
65842
66996
|
this.closed = true;
|
|
65843
66997
|
clearInterval(this.ticker);
|
|
66998
|
+
this.ticker = undefined;
|
|
65844
66999
|
this.releaseBinding();
|
|
65845
67000
|
if (this.options.rootId === undefined)
|
|
65846
67001
|
workflowResourceAdmission.delete(this.backend);
|
|
67002
|
+
this.signalChange();
|
|
65847
67003
|
let failure;
|
|
65848
|
-
|
|
65849
|
-
|
|
65850
|
-
|
|
65851
|
-
|
|
65852
|
-
|
|
65853
|
-
|
|
65854
|
-
|
|
65855
|
-
|
|
65856
|
-
|
|
65857
|
-
|
|
65858
|
-
|
|
65859
|
-
|
|
65860
|
-
const stopped = stopping.flatMap(({ child, log }) => log !== undefined && (child.exitCode !== null || child.signalCode !== null) ? [log] : []);
|
|
65861
|
-
await Promise.all(stopped.map((log) => rm3(log, { force: true }).catch(() => {})));
|
|
65862
|
-
if (this.logDirectory !== undefined)
|
|
65863
|
-
await rmdir(this.logDirectory).catch(() => {});
|
|
67004
|
+
const coordinator = this.coordinator;
|
|
67005
|
+
if (coordinator !== undefined) {
|
|
67006
|
+
if (this.coordinatorLost === undefined) {
|
|
67007
|
+
try {
|
|
67008
|
+
await this.command({ kind: "close" });
|
|
67009
|
+
} catch (error) {
|
|
67010
|
+
failure = error;
|
|
67011
|
+
}
|
|
67012
|
+
}
|
|
67013
|
+
await this.inbound;
|
|
67014
|
+
await coordinator.port.close();
|
|
67015
|
+
this.pids.clear();
|
|
65864
67016
|
}
|
|
67017
|
+
for (const release of this.routeClaims.values())
|
|
67018
|
+
release();
|
|
67019
|
+
this.routeClaims.clear();
|
|
65865
67020
|
try {
|
|
65866
67021
|
await this.releaseRoutes?.();
|
|
65867
67022
|
} catch (error) {
|
|
@@ -65871,27 +67026,26 @@ class LocalWorkerPool {
|
|
|
65871
67026
|
throw failure;
|
|
65872
67027
|
}
|
|
65873
67028
|
}
|
|
65874
|
-
var INTERNAL_WORKFLOW_WORKER_ARG = "--internal-workflow-worker", DEFAULT_WORK_LEASE_MS = 30000, MINIMUM_WORK_LEASE_MS,
|
|
67029
|
+
var INTERNAL_WORKFLOW_WORKER_ARG = "--internal-workflow-worker", DEFAULT_WORK_LEASE_MS = 30000, MINIMUM_WORK_LEASE_MS, DEFAULT_STARTUP_TIMEOUT_MS = 30000, MAX_TIMER_DELAY_MS = 2147483647, COORDINATOR_STDERR_TAIL_CHARACTERS = 8192, COORDINATOR_EXIT_GRACE_MS = 5000;
|
|
65875
67030
|
var init_local_pool = __esm(() => {
|
|
65876
67031
|
init_internal();
|
|
65877
67032
|
init_sql_pool_queue();
|
|
65878
|
-
init_sql_pool_usage();
|
|
65879
67033
|
init_sql_run_journal();
|
|
65880
67034
|
init_workflow_tool_abort();
|
|
65881
67035
|
init_sdk_intercom();
|
|
67036
|
+
init_external_run_routes();
|
|
65882
67037
|
init_workflow_infrastructure_error();
|
|
65883
67038
|
init_admission_binding();
|
|
65884
67039
|
init_definitions();
|
|
65885
67040
|
init_dispatch();
|
|
65886
67041
|
init_host_options();
|
|
65887
|
-
|
|
67042
|
+
init_pool_coordinator();
|
|
65888
67043
|
init_stage_handoff();
|
|
65889
67044
|
MINIMUM_WORK_LEASE_MS = 2 * WORKER_HEARTBEAT_INTERVAL_MS;
|
|
65890
|
-
COORDINATOR_LEASE_MS = 3 * STAGE_SESSION_HEARTBEAT_INTERVAL_MS;
|
|
65891
67045
|
});
|
|
65892
67046
|
|
|
65893
67047
|
// dist/builtin/workflows/src/workers/public-pool.ts
|
|
65894
|
-
import { resolve as
|
|
67048
|
+
import { resolve as resolve15 } from "node:path";
|
|
65895
67049
|
import { isDeepStrictEqual as isDeepStrictEqual2 } from "node:util";
|
|
65896
67050
|
function withWorkerSettings(options, settings) {
|
|
65897
67051
|
const configured = settings?.workflows;
|
|
@@ -65940,11 +67094,11 @@ async function activatePublicPool(backend, options) {
|
|
|
65940
67094
|
...options.localWorkers,
|
|
65941
67095
|
startupTimeoutMs: undefined,
|
|
65942
67096
|
leaseMs: undefined,
|
|
65943
|
-
definitions: options.localWorkers.definitions.map((source) => ({ ...source, path:
|
|
67097
|
+
definitions: options.localWorkers.definitions.map((source) => ({ ...source, path: resolve15(cwd, source.path) }))
|
|
65944
67098
|
};
|
|
65945
67099
|
const admission = options.admission && {
|
|
65946
67100
|
...options.admission,
|
|
65947
|
-
...options.admission.userDatabase ? { userDatabase:
|
|
67101
|
+
...options.admission.userDatabase ? { userDatabase: resolve15(cwd, options.admission.userDatabase) } : {}
|
|
65948
67102
|
};
|
|
65949
67103
|
const sources = localWorkers && await Promise.all(localWorkers.definitions.map(async (source) => ({ ...source, digest: await definitionDigest(source.path) })));
|
|
65950
67104
|
const resources = JSON.parse(JSON.stringify({
|
|
@@ -72552,9 +73706,9 @@ var init_stage_runner_controller = __esm(() => {
|
|
|
72552
73706
|
});
|
|
72553
73707
|
|
|
72554
73708
|
// dist/builtin/workflows/src/runs/foreground/stage-runner-output.ts
|
|
72555
|
-
import { createHash as
|
|
73709
|
+
import { createHash as createHash5 } from "node:crypto";
|
|
72556
73710
|
import { mkdir as mkdir7, writeFile as writeFile10 } from "node:fs/promises";
|
|
72557
|
-
import { basename as basename5, dirname as dirname15, isAbsolute as isAbsolute11, join as join30, resolve as
|
|
73711
|
+
import { basename as basename5, dirname as dirname15, isAbsolute as isAbsolute11, join as join30, resolve as resolve16 } from "node:path";
|
|
72558
73712
|
function normalizeMaxOutput2(maxOutput) {
|
|
72559
73713
|
return {
|
|
72560
73714
|
bytes: maxOutput?.bytes ?? DEFAULT_MAX_OUTPUT_BYTES,
|
|
@@ -72616,7 +73770,7 @@ function formatByteSize(bytes) {
|
|
|
72616
73770
|
return `${value2.toFixed(1)} ${units[unitIndex]}`;
|
|
72617
73771
|
}
|
|
72618
73772
|
function transcriptPath(runId, outputPath) {
|
|
72619
|
-
const digest =
|
|
73773
|
+
const digest = createHash5("sha256").update(outputPath).digest("hex").slice(0, 16);
|
|
72620
73774
|
return join30(workflowArtifactRunPath(runId), "transcripts", `${digest}-${basename5(outputPath)}.transcript.md`);
|
|
72621
73775
|
}
|
|
72622
73776
|
function appendContent(lines, content) {
|
|
@@ -72662,7 +73816,7 @@ function renderTranscript(messages, fallbackOutput) {
|
|
|
72662
73816
|
`);
|
|
72663
73817
|
}
|
|
72664
73818
|
function savedOutputReference(outputPath, fullOutput, transcriptPathValue, transcriptText, transcriptAvailable, warning) {
|
|
72665
|
-
const absolutePath =
|
|
73819
|
+
const absolutePath = resolve16(outputPath);
|
|
72666
73820
|
const bytes = Buffer.byteLength(fullOutput, "utf8");
|
|
72667
73821
|
const lines = countLines(fullOutput);
|
|
72668
73822
|
const transcriptBytes = Buffer.byteLength(transcriptText, "utf8");
|
|
@@ -72677,8 +73831,8 @@ function resolveOutputPath(output, runtimeCwd, requestedCwd) {
|
|
|
72677
73831
|
return;
|
|
72678
73832
|
if (isAbsolute11(output))
|
|
72679
73833
|
return output;
|
|
72680
|
-
const baseCwd = requestedCwd === undefined ? runtimeCwd : isAbsolute11(requestedCwd) ? requestedCwd :
|
|
72681
|
-
return
|
|
73834
|
+
const baseCwd = requestedCwd === undefined ? runtimeCwd : isAbsolute11(requestedCwd) ? requestedCwd : resolve16(runtimeCwd, requestedCwd);
|
|
73835
|
+
return resolve16(baseCwd, output);
|
|
72682
73836
|
}
|
|
72683
73837
|
function degenerateWarning(fullOutput) {
|
|
72684
73838
|
return fullOutput.trim().length === 0 ? "WARNING: the stage artifact is empty; search the companion transcript for this stage's work." : undefined;
|
|
@@ -73687,7 +74841,7 @@ async function run(def, inputs, opts = {}) {
|
|
|
73687
74841
|
if (!isWorkflowDefinition(def))
|
|
73688
74842
|
throw new Error(workflowDefinitionRequirementMessage("run(definition, inputs)", def));
|
|
73689
74843
|
const depth = opts.depth ?? 0;
|
|
73690
|
-
const maxDepth = opts.config?.maxDepth ??
|
|
74844
|
+
const maxDepth = opts.config?.maxDepth ?? 3;
|
|
73691
74845
|
if (depth >= maxDepth) {
|
|
73692
74846
|
return {
|
|
73693
74847
|
runId: opts.runId ?? crypto.randomUUID(),
|
|
@@ -75141,9 +76295,9 @@ var init_sdk_owner_release = __esm(() => {
|
|
|
75141
76295
|
|
|
75142
76296
|
// dist/builtin/workflows/src/durable/sdk-recovery-scope.ts
|
|
75143
76297
|
import { realpathSync as realpathSync5 } from "node:fs";
|
|
75144
|
-
import { resolve as
|
|
76298
|
+
import { resolve as resolve17 } from "node:path";
|
|
75145
76299
|
function canonicalDirectory(path) {
|
|
75146
|
-
const absolute =
|
|
76300
|
+
const absolute = resolve17(path);
|
|
75147
76301
|
try {
|
|
75148
76302
|
const canonical = realpathSync5.native(absolute);
|
|
75149
76303
|
return process.platform === "win32" ? canonical.toLowerCase() : canonical;
|
|
@@ -75880,9 +77034,7 @@ function runSnapshotsFromCheckpoints(checkpoints, rootRunId, rootRunName, fallba
|
|
|
75880
77034
|
}
|
|
75881
77035
|
for (const group of grouped.values())
|
|
75882
77036
|
group.sort(compareDraftSourceOrder);
|
|
75883
|
-
|
|
75884
|
-
retainReachableRunGroups(grouped, rootRunId);
|
|
75885
|
-
if (policy.failOnUnreachable && grouped.size !== groupedCount)
|
|
77037
|
+
if (!retainReachableRunGroups(grouped, rootRunId) && policy.failOnUnreachable)
|
|
75886
77038
|
return [];
|
|
75887
77039
|
if (!validateRunGroups(grouped, rootRunId, toolCheckpoints, checkpoints, policy.retainActive))
|
|
75888
77040
|
return [];
|
|
@@ -87496,7 +88648,7 @@ async function withDormantRunWrite(store, runId, write) {
|
|
|
87496
88648
|
|
|
87497
88649
|
// dist/builtin/workflows/src/extension/dormant-run-intercom.ts
|
|
87498
88650
|
var PENDING_STAGE_ROUTE_EVENT2 = "atomic:workflow-pending-stage-route";
|
|
87499
|
-
var
|
|
88651
|
+
var PENDING_STAGE_MESSAGE_EVENT3 = "atomic:workflow-pending-stage-message";
|
|
87500
88652
|
var PENDING_STAGE_ROUTE_REVOKED_EVENT = "atomic:workflow-pending-stage-route-revoked";
|
|
87501
88653
|
var DORMANT_RUN_SUBAGENT_HOLD_EVENT = "atomic:workflow-dormant-run-subagent-hold";
|
|
87502
88654
|
var DORMANT_RUN_RECHECK_MS = 15000;
|
|
@@ -87672,7 +88824,7 @@ function registerDormantRunIntercomRoutes(pi, activeStore, deps) {
|
|
|
87672
88824
|
...event.senderRegistrationName === undefined ? {} : { senderRegistrationName: event.senderRegistrationName },
|
|
87673
88825
|
...event.senderReturnAddress === undefined ? {} : { senderReturnAddress: event.senderReturnAddress }
|
|
87674
88826
|
};
|
|
87675
|
-
pi.events?.emit?.(
|
|
88827
|
+
pi.events?.emit?.(PENDING_STAGE_MESSAGE_EVENT3, forwarded);
|
|
87676
88828
|
if (forwarded.handled && forwarded.completion !== undefined)
|
|
87677
88829
|
return await forwarded.completion;
|
|
87678
88830
|
return { outcome: "refused", reason: "Session not found" };
|
|
@@ -87692,7 +88844,7 @@ function registerDormantRunIntercomRoutes(pi, activeStore, deps) {
|
|
|
87692
88844
|
}
|
|
87693
88845
|
return isLive(run.runId) ? await redispatch(event) : { outcome: "refused", reason: `Workflow run ${run.runId} is no longer resumable from this session` };
|
|
87694
88846
|
};
|
|
87695
|
-
const messages = pi.events?.on?.(
|
|
88847
|
+
const messages = pi.events?.on?.(PENDING_STAGE_MESSAGE_EVENT3, (payload) => {
|
|
87696
88848
|
if (disposed || !isDormantMessageEvent(payload) || payload.handled || payload.live === true)
|
|
87697
88849
|
return;
|
|
87698
88850
|
const target = parseWorkflowStageTarget(payload.target);
|
|
@@ -89610,8 +90762,8 @@ function mergeConfigs(base, override) {
|
|
|
89610
90762
|
};
|
|
89611
90763
|
}
|
|
89612
90764
|
var WORKFLOW_CONFIG_DEFAULTS = {
|
|
89613
|
-
maxDepth:
|
|
89614
|
-
defaultConcurrency:
|
|
90765
|
+
maxDepth: 3,
|
|
90766
|
+
defaultConcurrency: 1000,
|
|
89615
90767
|
persistRuns: true,
|
|
89616
90768
|
statusFile: false,
|
|
89617
90769
|
resumeInFlight: "never",
|
|
@@ -91378,7 +92530,7 @@ init_registry();
|
|
|
91378
92530
|
// dist/builtin/workflows/src/extension/discovery-loaders.ts
|
|
91379
92531
|
init_workflow_module_loader();
|
|
91380
92532
|
import { readdir as readdir2, stat as stat2 } from "node:fs/promises";
|
|
91381
|
-
import { extname, isAbsolute as isAbsolute13, join as join32, resolve as
|
|
92533
|
+
import { extname, isAbsolute as isAbsolute13, join as join32, resolve as resolve18 } from "node:path";
|
|
91382
92534
|
async function scanWorkflowDir(dir) {
|
|
91383
92535
|
try {
|
|
91384
92536
|
const entries = await readdir2(dir, { withFileTypes: true });
|
|
@@ -91422,7 +92574,7 @@ async function loadFromPaths(pathsOrMap, kind, baseCwd, diagnostics, getWorkflow
|
|
|
91422
92574
|
const all = [];
|
|
91423
92575
|
const entries = Array.isArray(pathsOrMap) ? pathsOrMap.map((p) => ({ rawPath: p })) : Object.entries(pathsOrMap).map(([name, p]) => ({ rawPath: p, configuredName: name }));
|
|
91424
92576
|
for (const { rawPath, configuredName } of entries) {
|
|
91425
|
-
const absPath = isAbsolute13(rawPath) ? rawPath :
|
|
92577
|
+
const absPath = isAbsolute13(rawPath) ? rawPath : resolve18(baseCwd, rawPath);
|
|
91426
92578
|
let pathStats;
|
|
91427
92579
|
try {
|
|
91428
92580
|
pathStats = await stat2(absPath);
|
|
@@ -98180,7 +99332,7 @@ var DEFAULT_PROMPT_GUIDANCE = [
|
|
|
98180
99332
|
- Bounded loops must create distinct tracked work for every iteration; never reopen an ancestor below its downstream work. Stacked implementation slices are unrolled, not looped: each slice gets a fresh child boundary and distinct tracked nodes, and no slice may add an edge back to the current frontier, itself, or an ancestor. If a retained session receives follow-up without new dependency work, keep it as non-topological activity metadata rather than adding a back-edge. Before launch, sketch nodes and dependencies for every branch and loop and reject any proposed parent edge to the node itself or an ancestor.
|
|
98181
99333
|
- Treat dynamic graph validation during execution, replay, and journal reconstruction as the authoritative cycle boundary. If runtime topology code changes, require incremental edge checks and reconstruction validation; prompt guidance and TypeScript types cannot replace those runtime checks.
|
|
98182
99334
|
- Every workflow invocation receives the stable control group \`workflow:<rootRunId>\`. Stages inherit it unless an intentional explicit group is authored. Named groups and \`group: true\` become invocation-owned subgroups under \`workflow:<rootRunId>/...\`, so repeated names cannot collide across runs. The invocation context may list and exactly send/ask live stages in its owned subgroups and queue send to known pending stages; this authority is directional and does not let subgroup siblings see or reach each other. \`intercom list\` also shows possible future targets, and the invocation context may queue sticky sends to their path patterns. Pending and pattern ask remain unsupported. A \`workflow:<rootRunId>/**\` send broadcasts to all live stages and remains sticky for future descendants until root termination. \`group: "default"\` is the explicit non-owned shared-group escape and does not receive pending invocation routing. Do not mint root group names manually.
|
|
98183
|
-
- Stages in the same invocation group are peers, not only steerable targets. A live stage can \`intercom list\` its siblings and \`send\`/\`ask\` them directly to debate a finding with evidence, hand off paths and reproductions, claim shared files or expensive steps, or learn what a sibling already verified; it can queue \`send\` to a known pending sibling and \`ask\` a completed sibling that retains a valid conversation, which reopens it for a post-mortem turn. Delegated subagents inside a stage inherit the same group. Named subgroups isolate their members from sibling subgroups.
|
|
99335
|
+
- Stages in the same invocation group are peers, not only steerable targets. A live stage can \`intercom list\` its siblings and \`send\`/\`ask\` them directly to debate a finding with evidence, hand off paths and reproductions, claim shared files or expensive steps, or learn what a sibling already verified; it can queue \`send\` to a known pending sibling and \`ask\` a completed sibling that retains a valid conversation, which reopens it for a post-mortem turn. Delegated subagents inside a stage inherit the same group. Named subgroups isolate their members from sibling subgroups. Default to peer-to-peer (swarm) exchange; the supervisor is not a message relay. There are no limits on communication between agents; it is encouraged whenever you need to. Stages proactively discover peers and broadcast, debate, claim, hand off, and ask each other directly, and each stage still returns its own structured result. Scale the swarm with the task: for very complex work or when the user asks for heavy parallelism, run hundreds or thousands of agents that self-organize into teams (named subgroups, per-stage subagent sets, child workflows) and coordinate peer-to-peer like an organization.
|
|
98184
99336
|
- Prefer \`ctx.tool(name, args, fn)\` for workflow-owned TypeScript operations with side effects, such as filesystem writes, network mutations, and external API actions. It creates a tracked, non-attachable durable graph node before invoking \`fn\`; it may be used before, between, after, or without model stages. A completed \`ctx.tool\` call is durably checkpointed with its serializable result, so resume replays that result without rerunning \`fn\`. For checks that may fail during a bounded repair loop, opt into \`failureMode: "return"\`, branch on the typed outcome, and pass only the needed error fields to the repair stage or artifact yourself; Atomic does not inject failure evidence into later prompts. Retries do not bound a hung callback, so every \`ctx.tool\` callback that starts a child process or performs network I/O MUST set an explicit positive finite \`timeoutMs\` and forward the supplied \`signal\` to cancellable work. A timeout applies per attempt, aborts that attempt's signal, and gives retries a fresh deadline and signal. Cancellation and storage faults still throw. Keep pure computation and side-effect-free transformations as ordinary TypeScript. Do not wrap agent-stage internals or every function call indiscriminately; this rule applies to side effects orchestrated directly by the workflow definition.
|
|
98185
99337
|
- Pass large stage context through files/artifacts and \`reads\` with an explicit \`Read the file at <path>...\` prompt rather than large \`previous\` payloads or injected session history.
|
|
98186
99338
|
- Wrap critical stage-prompt content in \`<keepContext>\` / \`</keepContext>\`. Tagged content survives compression verbatim regardless of the compression ratio, and every line of the span including the tag lines is protected. A long-running stage will be compacted, and compaction ranks lines individually: an objective is verbose and restated while the constraint that narrows it is usually one line, so the constraint is the cheaper deletion and what survives reads as broader permission than intended. Tag role constraints that bound a stage to part of the work, acceptance criteria and immutable contracts, explicit prohibitions, and identifiers a stage must not lose such as a target branch, worktree path, or run ID. Do not tag bulk context: protected lines count against the keep target rather than raising it, so a large protected span makes the surrounding transcript compress harder — tag the constraint, not the material it applies to.
|