@bastani/atomic 0.9.43 → 0.9.44-alpha.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +42 -0
- package/dist/builtin/intercom/CHANGELOG.md +22 -0
- package/dist/builtin/intercom/broker/broker.ts +31 -7
- package/dist/builtin/intercom/broker/client-message-validation.ts +6 -0
- package/dist/builtin/intercom/broker/client.ts +69 -0
- package/dist/builtin/intercom/broker/send-signature.ts +4 -1
- package/dist/builtin/intercom/index.bundle.mjs +309 -109
- package/dist/builtin/intercom/live-route-refusal.ts +20 -0
- package/dist/builtin/intercom/package.json +1 -1
- package/dist/builtin/intercom/skills/intercom/SKILL.md +11 -6
- package/dist/builtin/intercom/types.ts +28 -1
- package/dist/builtin/mcp/CHANGELOG.md +7 -0
- package/dist/builtin/mcp/package.json +1 -1
- package/dist/builtin/subagents/CHANGELOG.md +9 -0
- package/dist/builtin/subagents/README.md +8 -8
- package/dist/builtin/subagents/agents/worker.md +1 -1
- package/dist/builtin/subagents/package.json +1 -1
- package/dist/builtin/subagents/skills/agent-browser/SKILL.md +1 -1
- package/dist/builtin/subagents/skills/cua-driver/EMBEDDING.md +21 -2
- package/dist/builtin/subagents/skills/cua-driver/LINUX.md +2 -2
- package/dist/builtin/subagents/skills/cua-driver/MACOS.md +342 -10
- package/dist/builtin/subagents/skills/cua-driver/README.md +4 -3
- package/dist/builtin/subagents/skills/cua-driver/RUNTIME.md +3 -1
- package/dist/builtin/subagents/skills/cua-driver/SKILL.md +7 -4
- package/dist/builtin/subagents/skills/cua-driver/WINDOWS.md +6 -5
- package/dist/builtin/subagents/skills/cua-driver/WORKFLOW.md +15 -0
- package/dist/builtin/subagents/skills/herdr/SKILL.md +22 -10
- package/dist/builtin/subagents/skills/subagent/SKILL.md +8 -5
- package/dist/builtin/subagents/src/extension/index.bundle.mjs +6 -6
- package/dist/builtin/web-access/package.json +1 -1
- package/dist/builtin/workflows/CHANGELOG.md +20 -0
- package/dist/builtin/workflows/README.md +1 -1
- package/dist/builtin/workflows/builtin/{chunk-frqxd4d1.js → chunk-k8d7ehjx.js} +22 -0
- package/dist/builtin/workflows/builtin/{chunk-3s95f7ry.js → chunk-qxpymnaf.js} +1 -1
- package/dist/builtin/workflows/builtin/{chunk-edhf30ef.js → chunk-zjecnjxx.js} +1 -1
- package/dist/builtin/workflows/builtin/goal.js +2 -2
- package/dist/builtin/workflows/builtin/index.js +3 -3
- package/dist/builtin/workflows/builtin/ralph.js +2 -2
- package/dist/builtin/workflows/package.json +1 -1
- package/dist/builtin/workflows/src/extension/index.bundle.mjs +1844 -603
- package/dist/builtin/workflows/src/extension/local-worker-entry.mjs +1880 -605
- package/dist/builtin/workflows/src/index.js +1896 -655
- package/dist/builtin/workflows/src/local-worker-entry.mjs +1880 -605
- package/dist/cli/args.js +1 -1
- package/dist/cli/args.js.map +1 -1
- package/dist/core/agent-session-custom-message-commit.d.ts.map +1 -1
- package/dist/core/agent-session-custom-message-commit.js +59 -0
- package/dist/core/agent-session-custom-message-commit.js.map +1 -1
- package/dist/core/agent-session-methods.d.ts +1 -0
- package/dist/core/agent-session-methods.d.ts.map +1 -1
- package/dist/core/agent-session-methods.js.map +1 -1
- package/dist/core/agent-session-prompt.d.ts +4 -0
- package/dist/core/agent-session-prompt.d.ts.map +1 -1
- package/dist/core/agent-session-prompt.js +10 -2
- package/dist/core/agent-session-prompt.js.map +1 -1
- package/dist/core/agent-session.d.ts +2 -0
- package/dist/core/agent-session.d.ts.map +1 -1
- package/dist/core/agent-session.js +9 -0
- package/dist/core/agent-session.js.map +1 -1
- package/dist/core/builtin-packages.d.ts.map +1 -1
- package/dist/core/builtin-packages.js +13 -13
- package/dist/core/builtin-packages.js.map +1 -1
- package/dist/core/exec.d.ts.map +1 -1
- package/dist/core/exec.js +11 -2
- package/dist/core/exec.js.map +1 -1
- package/dist/core/package-manager-resolver.d.ts.map +1 -1
- package/dist/core/package-manager-resolver.js +25 -14
- package/dist/core/package-manager-resolver.js.map +1 -1
- package/dist/core/package-manager-types.d.ts +8 -0
- package/dist/core/package-manager-types.d.ts.map +1 -1
- package/dist/core/package-manager-types.js.map +1 -1
- package/dist/core/package-manager.d.ts +1 -1
- package/dist/core/package-manager.d.ts.map +1 -1
- package/dist/core/package-manager.js.map +1 -1
- package/dist/core/resource-loader-extension-warnings.d.ts +4 -1
- package/dist/core/resource-loader-extension-warnings.d.ts.map +1 -1
- package/dist/core/resource-loader-extension-warnings.js +15 -0
- package/dist/core/resource-loader-extension-warnings.js.map +1 -1
- package/dist/core/resource-loader-package-resources.d.ts.map +1 -1
- package/dist/core/resource-loader-package-resources.js +1 -0
- package/dist/core/resource-loader-package-resources.js.map +1 -1
- package/dist/core/resource-loader-reload.d.ts.map +1 -1
- package/dist/core/resource-loader-reload.js +5 -3
- package/dist/core/resource-loader-reload.js.map +1 -1
- package/dist/extensions/mcp/client/oauth/flow.d.ts.map +1 -1
- package/dist/extensions/mcp/client/oauth/flow.js +4 -1
- package/dist/extensions/mcp/client/oauth/flow.js.map +1 -1
- package/dist/extensions/mcp/client/oauth/types.d.ts +1 -0
- package/dist/extensions/mcp/client/oauth/types.d.ts.map +1 -1
- package/dist/extensions/mcp/client/oauth/types.js +8 -4
- package/dist/extensions/mcp/client/oauth/types.js.map +1 -1
- package/dist/extensions/mcp/client/transports/streamable-http.d.ts.map +1 -1
- package/dist/extensions/mcp/client/transports/streamable-http.js +2 -1
- package/dist/extensions/mcp/client/transports/streamable-http.js.map +1 -1
- package/dist/index.d.ts +1 -1
- package/dist/index.d.ts.map +1 -1
- package/dist/index.js.map +1 -1
- package/dist/main.d.ts.map +1 -1
- package/dist/main.js +20 -10
- package/dist/main.js.map +1 -1
- package/dist/modes/interactive/components/model-selector.d.ts.map +1 -1
- package/dist/modes/interactive/components/model-selector.js +4 -2
- package/dist/modes/interactive/components/model-selector.js.map +1 -1
- package/dist/modes/interactive/components/scoped-models-selector.d.ts.map +1 -1
- package/dist/modes/interactive/components/scoped-models-selector.js +4 -2
- package/dist/modes/interactive/components/scoped-models-selector.js.map +1 -1
- package/dist/modes/interactive/components/tool-execution.d.ts +6 -0
- package/dist/modes/interactive/components/tool-execution.d.ts.map +1 -1
- package/dist/modes/interactive/components/tool-execution.js +11 -1
- package/dist/modes/interactive/components/tool-execution.js.map +1 -1
- package/dist/modes/interactive/model-search.d.ts +10 -0
- package/dist/modes/interactive/model-search.d.ts.map +1 -1
- package/dist/modes/interactive/model-search.js +21 -0
- package/dist/modes/interactive/model-search.js.map +1 -1
- package/dist/package-manager-cli.d.ts.map +1 -1
- package/dist/package-manager-cli.js +5 -0
- package/dist/package-manager-cli.js.map +1 -1
- package/dist/utils/tools-manager.d.ts.map +1 -1
- package/dist/utils/tools-manager.js +30 -35
- package/dist/utils/tools-manager.js.map +1 -1
- package/docs/computer-use.md +5 -5
- package/docs/custom-provider/streaming.md +9 -14
- package/docs/environment-variables.md +1 -0
- package/docs/intercom/operations.md +1 -1
- package/docs/intercom.md +16 -2
- package/docs/packages.md +2 -0
- package/docs/settings.md +2 -2
- package/docs/skills.md +2 -2
- package/docs/subagents.md +47 -10
- package/docs/workflows/api-reference.md +1 -1
- package/docs/workflows/authoring.md +1 -1
- package/docs/workflows/operations.md +8 -6
- package/docs/workflows/reliable-design.md +23 -5
- package/examples/extensions/custom-provider-anthropic/index.ts +31 -31
- package/examples/extensions/todo.ts +7 -0
- package/npm-shrinkwrap.json +32 -32
- package/package.json +3 -3
|
@@ -1858,7 +1858,13 @@ function createStoreContext(state = createStoreState()) {
|
|
|
1858
1858
|
}
|
|
1859
1859
|
return cachedGraphSnapshot;
|
|
1860
1860
|
}
|
|
1861
|
+
let batchDepth = 0;
|
|
1862
|
+
let batchedNotification = false;
|
|
1861
1863
|
function notify() {
|
|
1864
|
+
if (batchDepth > 0) {
|
|
1865
|
+
batchedNotification = true;
|
|
1866
|
+
return;
|
|
1867
|
+
}
|
|
1862
1868
|
observation.capture();
|
|
1863
1869
|
for (const fn of state.invalidationListeners) {
|
|
1864
1870
|
try {
|
|
@@ -1878,6 +1884,18 @@ function createStoreContext(state = createStoreState()) {
|
|
|
1878
1884
|
state.version++;
|
|
1879
1885
|
notify();
|
|
1880
1886
|
}
|
|
1887
|
+
function batch(update) {
|
|
1888
|
+
batchDepth++;
|
|
1889
|
+
try {
|
|
1890
|
+
return update();
|
|
1891
|
+
} finally {
|
|
1892
|
+
batchDepth--;
|
|
1893
|
+
if (batchDepth === 0 && batchedNotification) {
|
|
1894
|
+
batchedNotification = false;
|
|
1895
|
+
notify();
|
|
1896
|
+
}
|
|
1897
|
+
}
|
|
1898
|
+
}
|
|
1881
1899
|
function findRun(runId) {
|
|
1882
1900
|
return state.runs.find((r) => r.id === runId);
|
|
1883
1901
|
}
|
|
@@ -1931,6 +1949,7 @@ function createStoreContext(state = createStoreState()) {
|
|
|
1931
1949
|
snapshot,
|
|
1932
1950
|
graphSnapshot,
|
|
1933
1951
|
notify,
|
|
1952
|
+
batch,
|
|
1934
1953
|
bumpAndNotify,
|
|
1935
1954
|
findRun,
|
|
1936
1955
|
findStage,
|
|
@@ -3867,6 +3886,9 @@ function createRunStoreMethods(context) {
|
|
|
3867
3886
|
subscribeInvalidation(fn) {
|
|
3868
3887
|
state.invalidationListeners.add(fn);
|
|
3869
3888
|
return () => state.invalidationListeners.delete(fn);
|
|
3889
|
+
},
|
|
3890
|
+
batch(update) {
|
|
3891
|
+
return context.batch(update);
|
|
3870
3892
|
}
|
|
3871
3893
|
};
|
|
3872
3894
|
}
|
|
@@ -5840,22 +5862,37 @@ function childScopedEvidenceExists(checkpoints, boundaryReplayKey, childRunId) {
|
|
|
5840
5862
|
}
|
|
5841
5863
|
function retainReachableRunGroups(grouped, rootRunId) {
|
|
5842
5864
|
const reachable = new Set([rootRunId]);
|
|
5843
|
-
const
|
|
5865
|
+
const settled = new Set;
|
|
5866
|
+
const pending = [{ runId: rootRunId, live: true }];
|
|
5844
5867
|
while (pending.length > 0) {
|
|
5845
|
-
const
|
|
5868
|
+
const { runId, live } = pending.pop();
|
|
5869
|
+
const drafts = grouped.get(runId);
|
|
5846
5870
|
if (drafts === undefined)
|
|
5847
5871
|
continue;
|
|
5848
5872
|
for (const draft of drafts) {
|
|
5849
|
-
const childRunId =
|
|
5850
|
-
if (childRunId === undefined ||
|
|
5873
|
+
const childRunId = workflowChildFromDraft(draft)?.runId ?? draft.topology?.boundary?.child.runId;
|
|
5874
|
+
if (childRunId === undefined || !grouped.has(childRunId))
|
|
5851
5875
|
continue;
|
|
5852
|
-
|
|
5853
|
-
|
|
5876
|
+
if (live && childRunIdFromDraft(draft) !== undefined) {
|
|
5877
|
+
if (reachable.has(childRunId))
|
|
5878
|
+
continue;
|
|
5879
|
+
reachable.add(childRunId);
|
|
5880
|
+
pending.push({ runId: childRunId, live: true });
|
|
5881
|
+
} else if (!settled.has(childRunId)) {
|
|
5882
|
+
settled.add(childRunId);
|
|
5883
|
+
pending.push({ runId: childRunId, live: false });
|
|
5884
|
+
}
|
|
5854
5885
|
}
|
|
5855
5886
|
}
|
|
5856
|
-
|
|
5857
|
-
|
|
5858
|
-
|
|
5887
|
+
let complete = true;
|
|
5888
|
+
for (const runId of grouped.keys()) {
|
|
5889
|
+
if (reachable.has(runId))
|
|
5890
|
+
continue;
|
|
5891
|
+
if (!settled.has(runId))
|
|
5892
|
+
complete = false;
|
|
5893
|
+
grouped.delete(runId);
|
|
5894
|
+
}
|
|
5895
|
+
return complete;
|
|
5859
5896
|
}
|
|
5860
5897
|
function childRunIdFromDraft(draft) {
|
|
5861
5898
|
const boundary = draft.topology?.boundary;
|
|
@@ -6771,6 +6808,13 @@ function removeProcessExitCleanup(cleanup) {
|
|
|
6771
6808
|
if (processExitCleanups.delete(cleanup) && processExitCleanups.size === 0)
|
|
6772
6809
|
process.off("exit", runProcessExitCleanups);
|
|
6773
6810
|
}
|
|
6811
|
+
function snapshotNeedsHistory(tail) {
|
|
6812
|
+
const event = tail.snapshot.event;
|
|
6813
|
+
if (event.type !== RUN_EVENT_TYPES.messageQueued)
|
|
6814
|
+
return false;
|
|
6815
|
+
const payload = event.payload;
|
|
6816
|
+
return hasPendingStageDeliveryDelta(payload.pendingStageMessages);
|
|
6817
|
+
}
|
|
6774
6818
|
function* invoke(action) {
|
|
6775
6819
|
try {
|
|
6776
6820
|
const result = action();
|
|
@@ -6886,8 +6930,11 @@ class JournalDurableBackend {
|
|
|
6886
6930
|
this.projection.applyWorkflowSnapshot(event.payload.handle);
|
|
6887
6931
|
this.prompts.set(entry.runId, new PromptReservationState(event.payload.handle.pendingPrompts, event.payload.prompts));
|
|
6888
6932
|
} else {
|
|
6889
|
-
const handle = event.payload;
|
|
6890
6933
|
const previous = this.projection.getWorkflow(entry.runId);
|
|
6934
|
+
const handle = event.type === RUN_EVENT_TYPES.messageQueued && hasPendingStageDeliveryDelta(event.payload.pendingStageMessages) ? {
|
|
6935
|
+
...event.payload,
|
|
6936
|
+
pendingStageMessages: decodePendingStageDeliveries(previous?.pendingStageMessages ?? [], event.payload.pendingStageMessages ?? [])
|
|
6937
|
+
} : event.payload;
|
|
6891
6938
|
this.projection.applyWorkflowSnapshot(handle);
|
|
6892
6939
|
if (event.type === RUN_EVENT_TYPES.runStarted || previous?.pendingPrompts !== handle.pendingPrompts)
|
|
6893
6940
|
this.prompts.delete(entry.runId);
|
|
@@ -7183,11 +7230,23 @@ class JournalDurableBackend {
|
|
|
7183
7230
|
for (const message of messages)
|
|
7184
7231
|
if (message.runId !== logicalRunId)
|
|
7185
7232
|
throw new JournalMessageScopeError(logicalRunId, message.runId);
|
|
7186
|
-
|
|
7233
|
+
const next = {
|
|
7187
7234
|
...handle,
|
|
7188
7235
|
pendingStageMessages: replacePendingStageMessagesForRun(handle.pendingStageMessages ?? [], logicalRunId, messages),
|
|
7189
7236
|
updatedAt: Math.max(Date.now(), handle.updatedAt + 1)
|
|
7190
|
-
}
|
|
7237
|
+
};
|
|
7238
|
+
const encoded = encodePendingStageDeliveries(handle.pendingStageMessages ?? [], next.pendingStageMessages ?? []);
|
|
7239
|
+
const base = this.sequences.get(runId);
|
|
7240
|
+
if (encoded !== undefined && base !== undefined) {
|
|
7241
|
+
try {
|
|
7242
|
+
yield* this.append(runId, { type: RUN_EVENT_TYPES.messageQueued, payload: { ...next, pendingStageMessages: encoded } }, { expectSeq: base });
|
|
7243
|
+
return true;
|
|
7244
|
+
} catch (error) {
|
|
7245
|
+
if (!(error instanceof JournalCasConflictError))
|
|
7246
|
+
throw error;
|
|
7247
|
+
}
|
|
7248
|
+
}
|
|
7249
|
+
yield* this.snapshot(next, RUN_EVENT_TYPES.messageQueued);
|
|
7191
7250
|
return true;
|
|
7192
7251
|
}
|
|
7193
7252
|
async archivePendingStageMessage(runId, message) {
|
|
@@ -7281,7 +7340,7 @@ class JournalDurableBackend {
|
|
|
7281
7340
|
if (summarized !== undefined && summarized.seq === version.seq && summarized.at === version.at)
|
|
7282
7341
|
return;
|
|
7283
7342
|
const tail = yield* invoke(() => journal.snapshotTail(runId));
|
|
7284
|
-
if (tail !== undefined)
|
|
7343
|
+
if (tail !== undefined && !snapshotNeedsHistory(tail))
|
|
7285
7344
|
return this.summarize(runId, tail);
|
|
7286
7345
|
if (summarized === undefined && version.seq === 0)
|
|
7287
7346
|
return;
|
|
@@ -7507,6 +7566,69 @@ class JournalDurableBackend {
|
|
|
7507
7566
|
}
|
|
7508
7567
|
}
|
|
7509
7568
|
}
|
|
7569
|
+
function identity(message) {
|
|
7570
|
+
return JSON.stringify([message.runId, message.id]);
|
|
7571
|
+
}
|
|
7572
|
+
function uniqueByIdentity(messages) {
|
|
7573
|
+
const result = new Map;
|
|
7574
|
+
const repeated = new Set;
|
|
7575
|
+
for (const message of messages) {
|
|
7576
|
+
const key = identity(message);
|
|
7577
|
+
if (result.has(key))
|
|
7578
|
+
repeated.add(key);
|
|
7579
|
+
result.set(key, message);
|
|
7580
|
+
}
|
|
7581
|
+
for (const key of repeated)
|
|
7582
|
+
result.delete(key);
|
|
7583
|
+
return result;
|
|
7584
|
+
}
|
|
7585
|
+
function isDeltaEncoded(message) {
|
|
7586
|
+
return message.deliveryDelta !== undefined;
|
|
7587
|
+
}
|
|
7588
|
+
function encodePendingStageDeliveries(previous, next) {
|
|
7589
|
+
const bases = uniqueByIdentity(previous);
|
|
7590
|
+
const nextIdentities = uniqueByIdentity(next);
|
|
7591
|
+
let encodedAny = false;
|
|
7592
|
+
const encoded = next.map((message) => {
|
|
7593
|
+
const base = bases.get(identity(message))?.deliveries;
|
|
7594
|
+
const deliveries = message.deliveries;
|
|
7595
|
+
if (base === undefined || deliveries === undefined || !nextIdentities.has(identity(message)))
|
|
7596
|
+
return message;
|
|
7597
|
+
const drop = [];
|
|
7598
|
+
let kept = 0;
|
|
7599
|
+
base.forEach((delivery, index) => {
|
|
7600
|
+
if (kept < deliveries.length && JSON.stringify(delivery) === JSON.stringify(deliveries[kept]))
|
|
7601
|
+
kept++;
|
|
7602
|
+
else
|
|
7603
|
+
drop.push(index);
|
|
7604
|
+
});
|
|
7605
|
+
const add = deliveries.slice(kept);
|
|
7606
|
+
if (drop.length + add.length >= deliveries.length)
|
|
7607
|
+
return message;
|
|
7608
|
+
encodedAny = true;
|
|
7609
|
+
const { deliveries: _full, ...rest } = message;
|
|
7610
|
+
return { ...rest, deliveryDelta: drop.length === 0 ? { add } : { drop, add } };
|
|
7611
|
+
});
|
|
7612
|
+
return encodedAny ? encoded : undefined;
|
|
7613
|
+
}
|
|
7614
|
+
function decodePendingStageDeliveries(previous, messages) {
|
|
7615
|
+
if (!messages.some(isDeltaEncoded))
|
|
7616
|
+
return messages;
|
|
7617
|
+
const bases = uniqueByIdentity(previous);
|
|
7618
|
+
return messages.map((message) => {
|
|
7619
|
+
if (!isDeltaEncoded(message))
|
|
7620
|
+
return message;
|
|
7621
|
+
const base = bases.get(identity(message))?.deliveries;
|
|
7622
|
+
if (base === undefined)
|
|
7623
|
+
throw new Error(`atomic-workflows: journal delivery change for ${identity(message)} has no base entry`);
|
|
7624
|
+
const { deliveryDelta, ...rest } = message;
|
|
7625
|
+
const dropped = new Set(deliveryDelta.drop ?? []);
|
|
7626
|
+
return { ...rest, deliveries: [...base.filter((_, index) => !dropped.has(index)), ...deliveryDelta.add] };
|
|
7627
|
+
});
|
|
7628
|
+
}
|
|
7629
|
+
function hasPendingStageDeliveryDelta(messages) {
|
|
7630
|
+
return messages?.some(isDeltaEncoded) === true;
|
|
7631
|
+
}
|
|
7510
7632
|
var PROCESS_HOLDER_ID, processExitCleanups, JournalOwnershipError, JournalBackendClosedError, JournalOperationAbortedError, JournalExecutionRetiredError, JournalMessageScopeError;
|
|
7511
7633
|
var init_journal_backend = __esm(() => {
|
|
7512
7634
|
init_workflow_instance_ownership_error();
|
|
@@ -15531,7 +15653,7 @@ class ConcurrencyLimiter {
|
|
|
15531
15653
|
}
|
|
15532
15654
|
}
|
|
15533
15655
|
function createRunLimiter(defaultConcurrency) {
|
|
15534
|
-
return new ConcurrencyLimiter(defaultConcurrency ??
|
|
15656
|
+
return new ConcurrencyLimiter(defaultConcurrency ?? 1000);
|
|
15535
15657
|
}
|
|
15536
15658
|
|
|
15537
15659
|
// dist/builtin/workflows/src/shared/possible-stages.ts
|
|
@@ -17023,7 +17145,7 @@ function kebabCase(name) {
|
|
|
17023
17145
|
function errorMessage2(error) {
|
|
17024
17146
|
return error instanceof Error ? error.message : String(error);
|
|
17025
17147
|
}
|
|
17026
|
-
var DEFAULT_MAX_DEPTH =
|
|
17148
|
+
var DEFAULT_MAX_DEPTH = 3, BUILTIN_BARREL_SPECIFIERS, BUILTIN_SUBPATH_PREFIXES, TRACKED_NODE_METHODS;
|
|
17027
17149
|
var init_possible_stages = __esm(() => {
|
|
17028
17150
|
BUILTIN_BARREL_SPECIFIERS = new Set(["@bastani/atomic/workflows/builtin"]);
|
|
17029
17151
|
BUILTIN_SUBPATH_PREFIXES = ["@bastani/atomic/workflows/builtin/"];
|
|
@@ -60578,6 +60700,44 @@ class GraphFrontierTracker {
|
|
|
60578
60700
|
}
|
|
60579
60701
|
}
|
|
60580
60702
|
|
|
60703
|
+
// dist/builtin/workflows/src/shared/external-run-routes.ts
|
|
60704
|
+
function claimExternalRunRoute(store, runId, owner) {
|
|
60705
|
+
let owners = registry.owners.get(store);
|
|
60706
|
+
if (owners === undefined) {
|
|
60707
|
+
owners = new Map;
|
|
60708
|
+
registry.owners.set(store, owners);
|
|
60709
|
+
}
|
|
60710
|
+
owners.set(runId, owner);
|
|
60711
|
+
return () => {
|
|
60712
|
+
const current = registry.owners.get(store);
|
|
60713
|
+
if (current?.get(runId) !== owner)
|
|
60714
|
+
return;
|
|
60715
|
+
current.delete(runId);
|
|
60716
|
+
if (current.size === 0)
|
|
60717
|
+
registry.owners.delete(store);
|
|
60718
|
+
};
|
|
60719
|
+
}
|
|
60720
|
+
function externalRunRouteOwner(store, runId) {
|
|
60721
|
+
return registry.owners.get(store)?.get(runId);
|
|
60722
|
+
}
|
|
60723
|
+
function registerPendingStageMessageRouter(store, router) {
|
|
60724
|
+
registry.routers.set(store, router);
|
|
60725
|
+
return () => {
|
|
60726
|
+
if (registry.routers.get(store) === router)
|
|
60727
|
+
registry.routers.delete(store);
|
|
60728
|
+
};
|
|
60729
|
+
}
|
|
60730
|
+
function pendingStageMessageRouter(store) {
|
|
60731
|
+
return registry.routers.get(store);
|
|
60732
|
+
}
|
|
60733
|
+
var registryKey, processState, registry;
|
|
60734
|
+
var init_external_run_routes = __esm(() => {
|
|
60735
|
+
registryKey = Symbol.for("atomic-workflows/external-run-routes@1");
|
|
60736
|
+
processState = globalThis;
|
|
60737
|
+
registry = processState[registryKey] ?? { owners: new WeakMap, routers: new WeakMap };
|
|
60738
|
+
processState[registryKey] = registry;
|
|
60739
|
+
});
|
|
60740
|
+
|
|
60581
60741
|
// dist/builtin/workflows/src/shared/pending-stage-route-capability.ts
|
|
60582
60742
|
import { randomUUID as randomUUID11 } from "node:crypto";
|
|
60583
60743
|
function capabilitiesFor(store) {
|
|
@@ -60602,12 +60762,12 @@ function rotateWorkflowPendingStageRouteCapability(store, runId) {
|
|
|
60602
60762
|
capabilitiesFor(store).set(runId, capability);
|
|
60603
60763
|
return capability;
|
|
60604
60764
|
}
|
|
60605
|
-
var routeCapabilitiesKey,
|
|
60765
|
+
var routeCapabilitiesKey, processState2, routeCapabilities;
|
|
60606
60766
|
var init_pending_stage_route_capability = __esm(() => {
|
|
60607
60767
|
routeCapabilitiesKey = Symbol.for("atomic-workflows/pending-stage-route-capabilities@1");
|
|
60608
|
-
|
|
60609
|
-
routeCapabilities =
|
|
60610
|
-
|
|
60768
|
+
processState2 = globalThis;
|
|
60769
|
+
routeCapabilities = processState2[routeCapabilitiesKey] ?? new WeakMap;
|
|
60770
|
+
processState2[routeCapabilitiesKey] = routeCapabilities;
|
|
60611
60771
|
});
|
|
60612
60772
|
|
|
60613
60773
|
// dist/builtin/workflows/src/shared/pending-stage-route-readiness.ts
|
|
@@ -60639,12 +60799,12 @@ function hasWorkflowPendingStageRouteOwner(store) {
|
|
|
60639
60799
|
function workflowPendingStageRouteReady(store, runId) {
|
|
60640
60800
|
return owners.get(store)?.ready(runId);
|
|
60641
60801
|
}
|
|
60642
|
-
var routeOwnersKey,
|
|
60802
|
+
var routeOwnersKey, processState3, owners;
|
|
60643
60803
|
var init_pending_stage_route_readiness = __esm(() => {
|
|
60644
60804
|
routeOwnersKey = Symbol.for("atomic-workflows/pending-stage-route-owners@1");
|
|
60645
|
-
|
|
60646
|
-
owners =
|
|
60647
|
-
|
|
60805
|
+
processState3 = globalThis;
|
|
60806
|
+
owners = processState3[routeOwnersKey] ?? new WeakMap;
|
|
60807
|
+
processState3[routeOwnersKey] = owners;
|
|
60648
60808
|
});
|
|
60649
60809
|
|
|
60650
60810
|
// dist/builtin/workflows/src/shared/sticky-stage-delivery-status.ts
|
|
@@ -60769,6 +60929,9 @@ function registerPendingStageIntercomBridge(pi, activeStore, ownsRun = () => tru
|
|
|
60769
60929
|
notificationId,
|
|
60770
60930
|
reason
|
|
60771
60931
|
};
|
|
60932
|
+
const external = externalRunRouteOwner(activeStore, entry.runId);
|
|
60933
|
+
if (external !== undefined)
|
|
60934
|
+
return external.notifyUndeliverable(payload);
|
|
60772
60935
|
pi.events?.emit?.(PENDING_STAGE_UNDELIVERABLE_EVENT, payload);
|
|
60773
60936
|
return payload.handled && await payload.completion === true;
|
|
60774
60937
|
};
|
|
@@ -60780,19 +60943,29 @@ function registerPendingStageIntercomBridge(pi, activeStore, ownsRun = () => tru
|
|
|
60780
60943
|
return;
|
|
60781
60944
|
const ownedRunIds = new Set;
|
|
60782
60945
|
const runs = activeStore.runs();
|
|
60946
|
+
const runById = new Map(runs.map((run) => [run.id, run]));
|
|
60947
|
+
const childrenByBoundary = new Map;
|
|
60948
|
+
for (const run of runs) {
|
|
60949
|
+
if (run.parentRunId === undefined || run.parentStageId === undefined)
|
|
60950
|
+
continue;
|
|
60951
|
+
const key = JSON.stringify([run.parentRunId, run.parentStageId]);
|
|
60952
|
+
const children = childrenByBoundary.get(key) ?? [];
|
|
60953
|
+
children.push(run);
|
|
60954
|
+
childrenByBoundary.set(key, children);
|
|
60955
|
+
}
|
|
60783
60956
|
const resolveBoundarySegments = createWorkflowBoundarySegmentsResolver(runs);
|
|
60784
60957
|
for (const run of runs) {
|
|
60785
|
-
if (!ownsRun(run.id))
|
|
60958
|
+
if (externalRunRouteOwner(activeStore, run.id) !== undefined || !ownsRun(run.id))
|
|
60786
60959
|
continue;
|
|
60787
|
-
const rootRunId =
|
|
60960
|
+
const rootRunId = reciprocalWorkflowRootRunId(runById, run.id);
|
|
60788
60961
|
if (rootRunId === undefined)
|
|
60789
60962
|
continue;
|
|
60790
60963
|
ownedRunIds.add(run.id);
|
|
60791
|
-
const parentRun =
|
|
60964
|
+
const parentRun = run.parentRunId === undefined ? undefined : runById.get(run.parentRunId);
|
|
60792
60965
|
const boundary = parentRun?.stages.find((stage) => stage.id === run.parentStageId);
|
|
60793
60966
|
const parent = parentRun === undefined || boundary === undefined ? undefined : {
|
|
60794
60967
|
runId: parentRun.id,
|
|
60795
|
-
stageKeys: [boundary.id, boundary.name].filter((key) =>
|
|
60968
|
+
stageKeys: [boundary.id, boundary.name].filter((key) => resolveIndexedChildRun(childrenByBoundary, parentRun, key)?.id === run.id)
|
|
60796
60969
|
};
|
|
60797
60970
|
const announcement = {
|
|
60798
60971
|
runId: run.id,
|
|
@@ -60933,16 +61106,22 @@ function registerPendingStageIntercomBridge(pi, activeStore, ownsRun = () => tru
|
|
|
60933
61106
|
payload.handled = true;
|
|
60934
61107
|
payload.completion = queueAndPersist(activeStore, resolvedEvent, workflowInvocationIntercomGroup(rootRunId), stage.pendingStageDeliveryAvailable === true);
|
|
60935
61108
|
};
|
|
60936
|
-
const
|
|
60937
|
-
const stickySubscription = pi.events?.on?.(STICKY_LIVE_DELIVERED_EVENT, (payload) => {
|
|
61109
|
+
const routeStickyLiveDelivered = (payload) => {
|
|
60938
61110
|
if (disposed || !isStickyLiveDeliveredEvent(payload) || payload.handled)
|
|
60939
61111
|
return;
|
|
60940
61112
|
payload.handled = true;
|
|
60941
61113
|
payload.completion = outsideWorkflowExecutions(() => recordConfirmedStickyDeliveries(activeStore, payload));
|
|
61114
|
+
};
|
|
61115
|
+
const subscription = pi.events?.on?.(PENDING_STAGE_MESSAGE_EVENT, (payload) => outsideWorkflowExecutions(() => routePendingStageMessage(payload)));
|
|
61116
|
+
const stickySubscription = pi.events?.on?.(STICKY_LIVE_DELIVERED_EVENT, routeStickyLiveDelivered);
|
|
61117
|
+
const disposeRouter = registerPendingStageMessageRouter(activeStore, {
|
|
61118
|
+
route: (payload) => outsideWorkflowExecutions(() => routePendingStageMessage(payload)),
|
|
61119
|
+
sticky: routeStickyLiveDelivered
|
|
60942
61120
|
});
|
|
60943
61121
|
const dispose = () => {
|
|
60944
61122
|
disposed = true;
|
|
60945
61123
|
disposeReadiness();
|
|
61124
|
+
disposeRouter();
|
|
60946
61125
|
routeCompletions.clear();
|
|
60947
61126
|
unsubscribeStore();
|
|
60948
61127
|
if (typeof subscription === "function")
|
|
@@ -61068,6 +61247,12 @@ function isMaterializedNonAgentTarget(runs, target) {
|
|
|
61068
61247
|
return false;
|
|
61069
61248
|
return nonAgentNodes(run).some((node) => node.id === stageKey || node.name === stageKey);
|
|
61070
61249
|
}
|
|
61250
|
+
function resolveIndexedChildRun(childrenByBoundary, parent, segment) {
|
|
61251
|
+
const boundariesById = parent.stages.filter((stage) => stage.id === segment);
|
|
61252
|
+
const boundaries = boundariesById.length > 0 ? boundariesById : parent.stages.filter((stage) => stage.name === segment);
|
|
61253
|
+
const children = boundaries.flatMap((boundary) => childrenByBoundary.get(JSON.stringify([parent.id, boundary.id])) ?? []);
|
|
61254
|
+
return children.length === 1 ? children[0] : undefined;
|
|
61255
|
+
}
|
|
61071
61256
|
function resolveChildRun(runs, parent, segment) {
|
|
61072
61257
|
const boundariesById = parent.stages.filter((stage) => stage.id === segment);
|
|
61073
61258
|
const boundaries = boundariesById.length > 0 ? boundariesById : parent.stages.filter((stage) => stage.name === segment);
|
|
@@ -61407,6 +61592,7 @@ var init_pending_stage_intercom = __esm(() => {
|
|
|
61407
61592
|
init_execution_context_boundary();
|
|
61408
61593
|
init_run_owner_backend();
|
|
61409
61594
|
init_store_backend();
|
|
61595
|
+
init_external_run_routes();
|
|
61410
61596
|
init_intercom_group();
|
|
61411
61597
|
init_pending_stage_route_capability();
|
|
61412
61598
|
init_pending_stage_route_readiness();
|
|
@@ -61546,7 +61732,8 @@ function buildSendSignature(to, options) {
|
|
|
61546
61732
|
attachments: canonicalizeAttachmentsForSendSignature(options.attachments) ?? [],
|
|
61547
61733
|
replyTo: options.replyTo ?? null,
|
|
61548
61734
|
expectsReply: options.expectsReply ?? false,
|
|
61549
|
-
replyError: options.replyError ?? null
|
|
61735
|
+
replyError: options.replyError ?? null,
|
|
61736
|
+
...options.delivery === undefined || options.delivery === "steer" ? {} : { delivery: options.delivery }
|
|
61550
61737
|
});
|
|
61551
61738
|
}
|
|
61552
61739
|
|
|
@@ -61732,6 +61919,8 @@ function isMessage(value2) {
|
|
|
61732
61919
|
return false;
|
|
61733
61920
|
if (message.expectsReply !== undefined && typeof message.expectsReply !== "boolean")
|
|
61734
61921
|
return false;
|
|
61922
|
+
if (message.delivery !== undefined && message.delivery !== "steer" && message.delivery !== "followUp" && message.delivery !== "interrupt")
|
|
61923
|
+
return false;
|
|
61735
61924
|
if (message.replyError !== undefined && typeof message.replyError !== "string")
|
|
61736
61925
|
return false;
|
|
61737
61926
|
if (message.source !== undefined && !isMessageSource(message.source))
|
|
@@ -61865,7 +62054,7 @@ function isWorkflowFutureStageRosterEntries(value2) {
|
|
|
61865
62054
|
function isWorkflowStageRosterEntries(value2) {
|
|
61866
62055
|
return Array.isArray(value2) && value2.every((entry) => typeof entry === "object" && entry !== null && entry.kind === "workflow-stage" && typeof entry.runId === "string" && typeof entry.stageId === "string" && typeof entry.stageName === "string" && typeof entry.target === "string" && (entry.lifecycle === "pending" || entry.lifecycle === "running") && typeof entry.group === "string" && (entry.sessionId === undefined || typeof entry.sessionId === "string"));
|
|
61867
62056
|
}
|
|
61868
|
-
var BROKER_SOCKET, BROKER_CONNECT_TIMEOUT_MS = 1e4, PENDING_STAGE_ROUTE_ACK_TIMEOUT_MS = 1e4, GROUP_REQUEST_TIMEOUT_MS = 5000, PRESENCE_ACK_TIMEOUT_MS = 5000, RESTORING_RECIPIENT_REQUEST_TIMEOUT_MS = 5000, HELD_SEND_GRACE_MS = 5000, IntercomClient;
|
|
62057
|
+
var BROKER_SOCKET, BROKER_CONNECT_TIMEOUT_MS = 1e4, PENDING_STAGE_ROUTE_ACK_TIMEOUT_MS = 1e4, GROUP_REQUEST_TIMEOUT_MS = 5000, PRESENCE_ACK_TIMEOUT_MS = 5000, RESTORING_RECIPIENT_REQUEST_TIMEOUT_MS = 5000, HELD_SEND_GRACE_MS = 5000, IntercomPendingStageRouteRefusedError, PENDING_STAGE_ROUTE_REFUSAL_CODES, IntercomClient;
|
|
61869
62058
|
var init_client = __esm(() => {
|
|
61870
62059
|
init_paths();
|
|
61871
62060
|
init_pending_send_registry();
|
|
@@ -61874,6 +62063,21 @@ var init_client = __esm(() => {
|
|
|
61874
62063
|
init_live_route_refusal();
|
|
61875
62064
|
init_retry_policy();
|
|
61876
62065
|
BROKER_SOCKET = getBrokerSocketPath();
|
|
62066
|
+
IntercomPendingStageRouteRefusedError = class IntercomPendingStageRouteRefusedError extends Error {
|
|
62067
|
+
connectionKept;
|
|
62068
|
+
code;
|
|
62069
|
+
constructor(reason, refusal = { connectionKept: false }) {
|
|
62070
|
+
super(reason);
|
|
62071
|
+
this.name = "IntercomPendingStageRouteRefusedError";
|
|
62072
|
+
this.connectionKept = refusal.connectionKept;
|
|
62073
|
+
this.code = refusal.code;
|
|
62074
|
+
}
|
|
62075
|
+
};
|
|
62076
|
+
PENDING_STAGE_ROUTE_REFUSAL_CODES = new Set([
|
|
62077
|
+
"not_authorized",
|
|
62078
|
+
"invalid_stage_roster",
|
|
62079
|
+
"invalid_possible_stage_roster"
|
|
62080
|
+
]);
|
|
61877
62081
|
IntercomClient = class IntercomClient extends EventEmitter {
|
|
61878
62082
|
returnAddress;
|
|
61879
62083
|
preferredSessionId;
|
|
@@ -61881,6 +62085,8 @@ var init_client = __esm(() => {
|
|
|
61881
62085
|
connectTimeoutMs;
|
|
61882
62086
|
brokerAcknowledgesReceipt = false;
|
|
61883
62087
|
brokerAcknowledgesPendingStageRoutes = false;
|
|
62088
|
+
multiplexesPendingStageRoutes;
|
|
62089
|
+
brokerRefusesPendingStageRoutesInPlace = false;
|
|
61884
62090
|
brokerAnswersStatus = false;
|
|
61885
62091
|
socket = null;
|
|
61886
62092
|
_sessionId = null;
|
|
@@ -61946,6 +62152,7 @@ var init_client = __esm(() => {
|
|
|
61946
62152
|
this.preferredSessionId = options.preferredSessionId;
|
|
61947
62153
|
this.acknowledgesReceipt = options.acknowledgesReceipt === true;
|
|
61948
62154
|
this.connectTimeoutMs = options.connectTimeoutMs ?? BROKER_CONNECT_TIMEOUT_MS;
|
|
62155
|
+
this.multiplexesPendingStageRoutes = options.multiplexesPendingStageRoutes === true;
|
|
61949
62156
|
}
|
|
61950
62157
|
failPending(error) {
|
|
61951
62158
|
this.pendingSends.rejectAll(error);
|
|
@@ -62058,6 +62265,7 @@ var init_client = __esm(() => {
|
|
|
62058
62265
|
this._supervisorSessionId = null;
|
|
62059
62266
|
this.brokerAcknowledgesReceipt = false;
|
|
62060
62267
|
this.brokerAcknowledgesPendingStageRoutes = false;
|
|
62268
|
+
this.brokerRefusesPendingStageRoutesInPlace = false;
|
|
62061
62269
|
this.brokerAnswersStatus = false;
|
|
62062
62270
|
this._messageSource = undefined;
|
|
62063
62271
|
this.disconnectError = null;
|
|
@@ -62136,7 +62344,7 @@ var init_client = __esm(() => {
|
|
|
62136
62344
|
}
|
|
62137
62345
|
switch (brokerMessage.type) {
|
|
62138
62346
|
case "registered": {
|
|
62139
|
-
if (typeof brokerMessage.sessionId !== "string" || brokerMessage.supervisorSessionId !== undefined && typeof brokerMessage.supervisorSessionId !== "string" || brokerMessage.acknowledgesReceipt !== undefined && brokerMessage.acknowledgesReceipt !== true || brokerMessage.acknowledgesPendingStageRoutes !== undefined && brokerMessage.acknowledgesPendingStageRoutes !== true || brokerMessage.brokerStatus !== undefined && brokerMessage.brokerStatus !== true) {
|
|
62347
|
+
if (typeof brokerMessage.sessionId !== "string" || brokerMessage.supervisorSessionId !== undefined && typeof brokerMessage.supervisorSessionId !== "string" || brokerMessage.acknowledgesReceipt !== undefined && brokerMessage.acknowledgesReceipt !== true || brokerMessage.acknowledgesPendingStageRoutes !== undefined && brokerMessage.acknowledgesPendingStageRoutes !== true || brokerMessage.refusesPendingStageRoutesInPlace !== undefined && brokerMessage.refusesPendingStageRoutesInPlace !== true || brokerMessage.brokerStatus !== undefined && brokerMessage.brokerStatus !== true) {
|
|
62140
62348
|
throw new Error("Invalid registered message");
|
|
62141
62349
|
}
|
|
62142
62350
|
if (this._sessionId !== null) {
|
|
@@ -62146,6 +62354,7 @@ var init_client = __esm(() => {
|
|
|
62146
62354
|
this._supervisorSessionId = typeof brokerMessage.supervisorSessionId === "string" ? brokerMessage.supervisorSessionId : null;
|
|
62147
62355
|
this.brokerAcknowledgesReceipt = this.acknowledgesReceipt && brokerMessage.acknowledgesReceipt === true;
|
|
62148
62356
|
this.brokerAcknowledgesPendingStageRoutes = brokerMessage.acknowledgesPendingStageRoutes === true;
|
|
62357
|
+
this.brokerRefusesPendingStageRoutesInPlace = brokerMessage.refusesPendingStageRoutesInPlace === true;
|
|
62149
62358
|
this.brokerAnswersStatus = brokerMessage.brokerStatus === true;
|
|
62150
62359
|
this.emit("_registered", { type: "registered", sessionId: brokerMessage.sessionId });
|
|
62151
62360
|
break;
|
|
@@ -62158,6 +62367,12 @@ var init_client = __esm(() => {
|
|
|
62158
62367
|
this.emit("_registration_failed", refusal);
|
|
62159
62368
|
break;
|
|
62160
62369
|
}
|
|
62370
|
+
const refusedRouteId = typeof brokerMessage.requestId === "string" ? brokerMessage.requestId : "";
|
|
62371
|
+
const refusedRoute = this.pendingStageRouteRegistrations.get(refusedRouteId);
|
|
62372
|
+
if (refusedRoute !== undefined) {
|
|
62373
|
+
this.pendingStageRouteRegistrations.delete(refusedRouteId);
|
|
62374
|
+
refusedRoute.reject(new IntercomPendingStageRouteRefusedError(brokerMessage.reason));
|
|
62375
|
+
}
|
|
62161
62376
|
this.disconnectError ??= refusal;
|
|
62162
62377
|
this.failPending(this.disconnectError);
|
|
62163
62378
|
this.socket?.destroy();
|
|
@@ -62283,6 +62498,21 @@ var init_client = __esm(() => {
|
|
|
62283
62498
|
this.emit("pending_stage_route_revoked", { runId });
|
|
62284
62499
|
break;
|
|
62285
62500
|
}
|
|
62501
|
+
case "pending_stage_route_refused": {
|
|
62502
|
+
const { requestId, runId, code, reason } = brokerMessage;
|
|
62503
|
+
if (typeof requestId !== "string" || typeof runId !== "string" || typeof reason !== "string" || typeof code !== "string") {
|
|
62504
|
+
throw new Error("Invalid pending-stage route refusal");
|
|
62505
|
+
}
|
|
62506
|
+
const pending = this.pendingStageRouteRegistrations.get(requestId);
|
|
62507
|
+
if (!pending)
|
|
62508
|
+
return;
|
|
62509
|
+
this.pendingStageRouteRegistrations.delete(requestId);
|
|
62510
|
+
pending.reject(new IntercomPendingStageRouteRefusedError(reason, {
|
|
62511
|
+
connectionKept: true,
|
|
62512
|
+
...PENDING_STAGE_ROUTE_REFUSAL_CODES.has(code) ? { code } : {}
|
|
62513
|
+
}));
|
|
62514
|
+
break;
|
|
62515
|
+
}
|
|
62286
62516
|
case "pending_stage_route_registered": {
|
|
62287
62517
|
const { requestId } = brokerMessage;
|
|
62288
62518
|
if (typeof requestId !== "string")
|
|
@@ -62786,7 +63016,8 @@ var init_client = __esm(() => {
|
|
|
62786
63016
|
stages,
|
|
62787
63017
|
possibleStages,
|
|
62788
63018
|
...parent === undefined ? {} : { parent },
|
|
62789
|
-
...dormant === true ? { dormant: true } : {}
|
|
63019
|
+
...dormant === true ? { dormant: true } : {},
|
|
63020
|
+
...this.multiplexesPendingStageRoutes && this.brokerRefusesPendingStageRoutesInPlace ? { refusalKeepsConnection: true } : {}
|
|
62790
63021
|
});
|
|
62791
63022
|
} catch (error) {
|
|
62792
63023
|
clearTimeout(timeout);
|
|
@@ -62899,6 +63130,7 @@ var init_client = __esm(() => {
|
|
|
62899
63130
|
timestamp: Date.now(),
|
|
62900
63131
|
replyTo: options.replyTo,
|
|
62901
63132
|
expectsReply: options.expectsReply,
|
|
63133
|
+
...options.delivery === undefined || options.delivery === "steer" ? {} : { delivery: options.delivery },
|
|
62902
63134
|
replyError: options.replyError,
|
|
62903
63135
|
source: this._messageSource,
|
|
62904
63136
|
content: { text: options.text, attachments: options.attachments }
|
|
@@ -63520,120 +63752,165 @@ var init_reconnect_backoff = __esm(() => {
|
|
|
63520
63752
|
BACKOFF_MS = [1000, 2000, 5000, 1e4, 30000];
|
|
63521
63753
|
});
|
|
63522
63754
|
|
|
63523
|
-
// dist/builtin/intercom/route-connect-gate.ts
|
|
63524
|
-
function createConcurrencyGate(limit) {
|
|
63525
|
-
let active = 0;
|
|
63526
|
-
const waiting = [];
|
|
63527
|
-
const release = () => {
|
|
63528
|
-
const next = waiting.shift();
|
|
63529
|
-
if (next === undefined)
|
|
63530
|
-
active -= 1;
|
|
63531
|
-
else
|
|
63532
|
-
next();
|
|
63533
|
-
};
|
|
63534
|
-
return {
|
|
63535
|
-
async run(operation) {
|
|
63536
|
-
if (active < limit)
|
|
63537
|
-
active += 1;
|
|
63538
|
-
else
|
|
63539
|
-
await new Promise((resolve) => waiting.push(resolve));
|
|
63540
|
-
try {
|
|
63541
|
-
return await operation();
|
|
63542
|
-
} finally {
|
|
63543
|
-
release();
|
|
63544
|
-
}
|
|
63545
|
-
}
|
|
63546
|
-
};
|
|
63547
|
-
}
|
|
63548
|
-
var ROUTE_CLIENT_CONNECT_CONCURRENCY = 4;
|
|
63549
|
-
|
|
63550
63755
|
// dist/builtin/intercom/workflow-route-owner.ts
|
|
63551
63756
|
import { randomUUID as randomUUID14 } from "node:crypto";
|
|
63552
63757
|
import { createEventBus, getBuiltinPackagePaths } from "@bastani/atomic";
|
|
63758
|
+
function closedError() {
|
|
63759
|
+
return new Error("Workflow Intercom route owner closed");
|
|
63760
|
+
}
|
|
63553
63761
|
function createWorkflowRouteOwner(cwd) {
|
|
63554
63762
|
const events = createEventBus();
|
|
63555
63763
|
const config = loadConfig();
|
|
63556
63764
|
const id = randomUUID14();
|
|
63557
63765
|
const routes = new Map;
|
|
63558
|
-
const
|
|
63766
|
+
const connections = new Map;
|
|
63767
|
+
const isCurrent = (state) => routes.get(state.route.runId) === state;
|
|
63768
|
+
const connectionKey = (state) => state.isolated ? state.route.runId : state.route.group;
|
|
63769
|
+
function connectionOf(state) {
|
|
63770
|
+
const key = connectionKey(state);
|
|
63771
|
+
let connection = connections.get(key);
|
|
63772
|
+
if (connection === undefined) {
|
|
63773
|
+
connection = { key, group: state.route.group, acknowledged: false, routes: new Map };
|
|
63774
|
+
connections.set(key, connection);
|
|
63775
|
+
}
|
|
63776
|
+
connection.routes.set(state.route.runId, state);
|
|
63777
|
+
return connection;
|
|
63778
|
+
}
|
|
63779
|
+
function detach(state) {
|
|
63780
|
+
const key = connectionKey(state);
|
|
63781
|
+
const connection = connections.get(key);
|
|
63782
|
+
connection?.routes.delete(state.route.runId);
|
|
63783
|
+
if (!state.isolated || connection === undefined)
|
|
63784
|
+
return;
|
|
63785
|
+
connections.delete(key);
|
|
63786
|
+
connection.client?.disconnect().catch(() => {});
|
|
63787
|
+
}
|
|
63788
|
+
function settle(state, registered) {
|
|
63789
|
+
state.registered = registered;
|
|
63790
|
+
if (!registered && isCurrent(state))
|
|
63791
|
+
return;
|
|
63792
|
+
for (const waiter of state.waiters)
|
|
63793
|
+
waiter(registered);
|
|
63794
|
+
state.waiters.clear();
|
|
63795
|
+
}
|
|
63559
63796
|
function scheduleRetry(state) {
|
|
63560
|
-
if (state
|
|
63797
|
+
if (!isCurrent(state) || state.retry !== undefined)
|
|
63561
63798
|
return;
|
|
63562
63799
|
state.retry = setTimeout(() => {
|
|
63563
63800
|
state.retry = undefined;
|
|
63564
|
-
publish(state, state.route).catch(() =>
|
|
63801
|
+
publish(state, state.route).catch(() => {});
|
|
63565
63802
|
}, jitteredReconnectDelayMs(state.attempt++));
|
|
63566
63803
|
state.retry.unref();
|
|
63567
63804
|
}
|
|
63568
|
-
|
|
63569
|
-
if (
|
|
63570
|
-
|
|
63571
|
-
|
|
63572
|
-
|
|
63573
|
-
|
|
63574
|
-
|
|
63575
|
-
|
|
63576
|
-
}
|
|
63577
|
-
|
|
63578
|
-
|
|
63579
|
-
|
|
63580
|
-
if (
|
|
63581
|
-
|
|
63582
|
-
|
|
63583
|
-
|
|
63584
|
-
|
|
63585
|
-
|
|
63586
|
-
|
|
63587
|
-
|
|
63588
|
-
|
|
63589
|
-
|
|
63590
|
-
|
|
63591
|
-
|
|
63592
|
-
if (
|
|
63593
|
-
|
|
63594
|
-
const
|
|
63595
|
-
|
|
63596
|
-
|
|
63597
|
-
|
|
63598
|
-
|
|
63599
|
-
|
|
63600
|
-
|
|
63601
|
-
|
|
63602
|
-
|
|
63603
|
-
|
|
63604
|
-
|
|
63605
|
-
|
|
63606
|
-
|
|
63607
|
-
|
|
63608
|
-
|
|
63609
|
-
|
|
63610
|
-
|
|
63611
|
-
|
|
63612
|
-
|
|
63805
|
+
function drop(connection, client) {
|
|
63806
|
+
if (connection.client === client) {
|
|
63807
|
+
connection.client = undefined;
|
|
63808
|
+
connection.acknowledged = false;
|
|
63809
|
+
for (const state of connection.routes.values()) {
|
|
63810
|
+
settle(state, false);
|
|
63811
|
+
scheduleRetry(state);
|
|
63812
|
+
}
|
|
63813
|
+
}
|
|
63814
|
+
client.disconnect().catch(() => {});
|
|
63815
|
+
}
|
|
63816
|
+
function connect(connection) {
|
|
63817
|
+
if (connection.client?.isConnected())
|
|
63818
|
+
return Promise.resolve(connection.client);
|
|
63819
|
+
connection.connecting ??= (async () => {
|
|
63820
|
+
const [extensionDir] = getBuiltinPackagePaths({
|
|
63821
|
+
workflows: false,
|
|
63822
|
+
subagents: false,
|
|
63823
|
+
mcp: false,
|
|
63824
|
+
"web-access": false
|
|
63825
|
+
});
|
|
63826
|
+
if (extensionDir === undefined)
|
|
63827
|
+
throw new Error("Intercom builtin package is unavailable");
|
|
63828
|
+
await spawnBrokerIfNeeded(config.brokerCommand, config.brokerArgs, undefined, extensionDir);
|
|
63829
|
+
if (connections.get(connection.key) !== connection)
|
|
63830
|
+
throw closedError();
|
|
63831
|
+
const client = new IntercomClient(`${id}:pending-stage-route:${connection.key}`, {
|
|
63832
|
+
multiplexesPendingStageRoutes: true
|
|
63833
|
+
});
|
|
63834
|
+
connection.client = client;
|
|
63835
|
+
connection.acknowledged = false;
|
|
63836
|
+
client.on("error", () => {});
|
|
63837
|
+
client.on("disconnected", () => drop(connection, client));
|
|
63838
|
+
client.on("pending_stage_message", (request) => {
|
|
63839
|
+
if (connection.client !== client)
|
|
63840
|
+
return;
|
|
63841
|
+
const event = { ...request, handled: false };
|
|
63842
|
+
events.emit("atomic:workflow-pending-stage-message", event);
|
|
63843
|
+
const completion = event.handled && event.completion !== undefined ? event.completion : Promise.resolve({ outcome: "refused", reason: "Session not found" });
|
|
63844
|
+
completion.catch((error) => ({ outcome: "refused", reason: error.message })).then((result) => client.respondPendingStageMessage(request.requestId, result)).catch(() => {});
|
|
63845
|
+
});
|
|
63846
|
+
client.on("sticky_live_delivered", (notice) => {
|
|
63847
|
+
if (connection.client !== client)
|
|
63848
|
+
return;
|
|
63849
|
+
events.emit("atomic:workflow-sticky-live-delivered", { ...notice, handled: false });
|
|
63850
|
+
});
|
|
63851
|
+
try {
|
|
63852
|
+
await client.connect({
|
|
63853
|
+
cwd,
|
|
63854
|
+
model: "workflow",
|
|
63855
|
+
pid: process.pid,
|
|
63856
|
+
startedAt: Date.now(),
|
|
63857
|
+
lastActivity: Date.now(),
|
|
63858
|
+
group: connection.group,
|
|
63859
|
+
groups: [connection.group],
|
|
63860
|
+
recipientPurpose: "control"
|
|
63861
|
+
});
|
|
63862
|
+
if (connections.get(connection.key) !== connection)
|
|
63863
|
+
throw closedError();
|
|
63864
|
+
} catch (error) {
|
|
63865
|
+
drop(connection, client);
|
|
63866
|
+
throw error;
|
|
63867
|
+
}
|
|
63868
|
+
for (const state of connection.routes.values()) {
|
|
63869
|
+
if (state.registered || state.retry === undefined)
|
|
63870
|
+
continue;
|
|
63871
|
+
clearTimeout(state.retry);
|
|
63872
|
+
state.retry = undefined;
|
|
63873
|
+
publish(state, state.route).catch(() => {});
|
|
63874
|
+
}
|
|
63875
|
+
return client;
|
|
63876
|
+
})().finally(() => {
|
|
63877
|
+
connection.connecting = undefined;
|
|
63613
63878
|
});
|
|
63614
|
-
return
|
|
63879
|
+
return connection.connecting;
|
|
63615
63880
|
}
|
|
63616
63881
|
function publish(state, route) {
|
|
63617
|
-
const completion = state.tail.catch(() => {}).then(
|
|
63618
|
-
if (state
|
|
63619
|
-
throw
|
|
63620
|
-
const
|
|
63882
|
+
const completion = state.tail.catch(() => {}).then(async () => {
|
|
63883
|
+
if (!isCurrent(state))
|
|
63884
|
+
throw closedError();
|
|
63885
|
+
const connection = connectionOf(state);
|
|
63886
|
+
const established = connection.acknowledged && connection.client?.isConnected() ? connection.client : undefined;
|
|
63887
|
+
let client;
|
|
63621
63888
|
try {
|
|
63622
|
-
|
|
63889
|
+
client = await connect(connection);
|
|
63623
63890
|
await client.registerPendingStageRouteAcknowledged(route.runId, route.group, route.capability, route.stages, route.possibleStages, route.parent);
|
|
63891
|
+
if (connection.client !== client || !isCurrent(state))
|
|
63892
|
+
return;
|
|
63893
|
+
connection.acknowledged = true;
|
|
63624
63894
|
state.attempt = 0;
|
|
63625
63895
|
clearTimeout(state.retry);
|
|
63626
63896
|
state.retry = undefined;
|
|
63897
|
+
settle(state, true);
|
|
63627
63898
|
} catch (error) {
|
|
63628
|
-
|
|
63629
|
-
|
|
63630
|
-
|
|
63631
|
-
await client?.disconnect();
|
|
63899
|
+
if (error instanceof IntercomPendingStageRouteRefusedError && error.connectionKept) {
|
|
63900
|
+
scheduleRetry(state);
|
|
63901
|
+
throw error;
|
|
63632
63902
|
}
|
|
63903
|
+
if (error instanceof IntercomPendingStageRouteRefusedError && !state.isolated && isCurrent(state)) {
|
|
63904
|
+
detach(state);
|
|
63905
|
+
state.isolated = true;
|
|
63906
|
+
connectionOf(state);
|
|
63907
|
+
}
|
|
63908
|
+
if (client !== undefined && (client !== established || !client.isConnected()))
|
|
63909
|
+
drop(connection, client);
|
|
63633
63910
|
scheduleRetry(state);
|
|
63634
63911
|
throw error;
|
|
63635
63912
|
}
|
|
63636
|
-
})
|
|
63913
|
+
});
|
|
63637
63914
|
state.tail = completion;
|
|
63638
63915
|
completion.catch(() => {});
|
|
63639
63916
|
return completion;
|
|
@@ -63642,10 +63919,15 @@ function createWorkflowRouteOwner(cwd) {
|
|
|
63642
63919
|
const route = payload;
|
|
63643
63920
|
let state = routes.get(route.runId);
|
|
63644
63921
|
if (state === undefined) {
|
|
63645
|
-
state = { route, tail: Promise.resolve(), attempt: 0,
|
|
63922
|
+
state = { route, tail: Promise.resolve(), attempt: 0, registered: false, isolated: false, waiters: new Set };
|
|
63646
63923
|
routes.set(route.runId, state);
|
|
63924
|
+
} else if (state.route.group !== route.group) {
|
|
63925
|
+
detach(state);
|
|
63926
|
+
state.isolated = false;
|
|
63927
|
+
settle(state, false);
|
|
63647
63928
|
}
|
|
63648
63929
|
state.route = route;
|
|
63930
|
+
connectionOf(state);
|
|
63649
63931
|
route.completion = publish(state, route);
|
|
63650
63932
|
});
|
|
63651
63933
|
events.on("atomic:workflow-pending-stage-undeliverable", (payload) => {
|
|
@@ -63656,7 +63938,7 @@ function createWorkflowRouteOwner(cwd) {
|
|
|
63656
63938
|
event.handled = true;
|
|
63657
63939
|
event.completion = (async () => {
|
|
63658
63940
|
await publish(state, state.route);
|
|
63659
|
-
const client = state
|
|
63941
|
+
const client = connections.get(connectionKey(state))?.client;
|
|
63660
63942
|
if (client === undefined || !client.isConnected())
|
|
63661
63943
|
return false;
|
|
63662
63944
|
const text = `Pending workflow stage could not receive intercom message: ${event.reason}`;
|
|
@@ -63666,18 +63948,33 @@ function createWorkflowRouteOwner(cwd) {
|
|
|
63666
63948
|
});
|
|
63667
63949
|
return {
|
|
63668
63950
|
events,
|
|
63951
|
+
registered(runId) {
|
|
63952
|
+
const state = routes.get(runId);
|
|
63953
|
+
if (state === undefined)
|
|
63954
|
+
return Promise.resolve(false);
|
|
63955
|
+
if (state.registered)
|
|
63956
|
+
return Promise.resolve(true);
|
|
63957
|
+
return new Promise((resolve) => state.waiters.add(resolve));
|
|
63958
|
+
},
|
|
63669
63959
|
async closeGroup(group) {
|
|
63670
63960
|
const closing = [...routes.values()].filter((state) => state.route.group === group);
|
|
63961
|
+
const closingConnections = [...connections.values()].filter((connection) => connection.group === group);
|
|
63962
|
+
for (const connection of closingConnections) {
|
|
63963
|
+
connections.delete(connection.key);
|
|
63964
|
+
connection.routes.clear();
|
|
63965
|
+
}
|
|
63671
63966
|
for (const state of closing) {
|
|
63672
|
-
state.closed = true;
|
|
63673
|
-
clearTimeout(state.retry);
|
|
63674
63967
|
routes.delete(state.route.runId);
|
|
63968
|
+
clearTimeout(state.retry);
|
|
63969
|
+
state.retry = undefined;
|
|
63970
|
+
settle(state, false);
|
|
63675
63971
|
}
|
|
63676
|
-
await Promise.all(
|
|
63677
|
-
await
|
|
63678
|
-
await
|
|
63679
|
-
await state.client?.disconnect();
|
|
63972
|
+
await Promise.all(closingConnections.map(async (connection) => {
|
|
63973
|
+
await connection.client?.disconnect();
|
|
63974
|
+
await connection.connecting?.catch(() => {});
|
|
63680
63975
|
}));
|
|
63976
|
+
await Promise.all(closing.map((state) => state.tail.catch(() => {})));
|
|
63977
|
+
await Promise.all(closingConnections.map((connection) => connection.client?.disconnect()));
|
|
63681
63978
|
}
|
|
63682
63979
|
};
|
|
63683
63980
|
}
|
|
@@ -63860,7 +64157,7 @@ var init_workflow_infrastructure_error = __esm(() => {
|
|
|
63860
64157
|
[/Local worker (?:process )?cleanup (?:was not confirmed|unconfirmed)/, "worker_cleanup_unconfirmed"],
|
|
63861
64158
|
[/Local worker retry limit/, "worker_retry_limit"],
|
|
63862
64159
|
[/Local worker (?:startup timed out|exited before readiness)/, "worker_startup_failed"],
|
|
63863
|
-
[/Worker exited before delivery acknowledgement/, "worker_lost"],
|
|
64160
|
+
[/Worker exited before delivery acknowledgement|Local worker pool coordinator exited/, "worker_lost"],
|
|
63864
64161
|
[/Stale work lease/, "stale_work_lease"],
|
|
63865
64162
|
[/Intercom runtime no longer active/, "intercom_unavailable"],
|
|
63866
64163
|
[/Storage unavailable/, "storage_unavailable"]
|
|
@@ -64159,7 +64456,7 @@ var init_definitions = __esm(() => {
|
|
|
64159
64456
|
// dist/builtin/workflows/src/durable/journal/pool-mutations.ts
|
|
64160
64457
|
import { StorageProvider as StorageProvider2 } from "@bastani/atomic/storage";
|
|
64161
64458
|
async function validatePoolMutation(tx, journal, root, work, mutation) {
|
|
64162
|
-
if (!["checkpoint", "session-create", "session-
|
|
64459
|
+
if (!["checkpoint", "session-create", "session-replace"].includes(mutation.kind))
|
|
64163
64460
|
throw new Error("Invalid pool mutation");
|
|
64164
64461
|
const checkpoint = mutation.checkpoint;
|
|
64165
64462
|
const prefix = `${work.scope}:`;
|
|
@@ -64227,10 +64524,18 @@ async function sessionStore(tx, prefix) {
|
|
|
64227
64524
|
return new StorageProvider2().openSessionStore({ kind: "database", database, prefix }, { agentDir: "", domain: "sessions" });
|
|
64228
64525
|
}
|
|
64229
64526
|
async function applyPoolSession(tx, prefix, workId, mutation) {
|
|
64527
|
+
const owners = `${prefix}session_owners`;
|
|
64528
|
+
if (mutation.kind === "session-append") {
|
|
64529
|
+
if (typeof mutation.sessionId !== "string" || mutation.sessionId === "")
|
|
64530
|
+
throw new Error("Pool sessions require database identity");
|
|
64531
|
+
const owner = await tx.get(`SELECT work FROM ${owners} WHERE session = ?`, mutation.sessionId);
|
|
64532
|
+
if (owner?.work !== workId)
|
|
64533
|
+
throw new Error("Pool session scope mismatch");
|
|
64534
|
+
return withSessionStore(tx, prefix, (store) => store.append(mutation.sessionId, mutation.entries, { expectSeq: mutation.expectSeq }));
|
|
64535
|
+
}
|
|
64230
64536
|
const { sessionId, replayKey } = mutation.checkpoint;
|
|
64231
64537
|
if (!sessionId || mutation.checkpoint.sessionFile !== undefined)
|
|
64232
64538
|
throw new Error("Pool sessions require database identity");
|
|
64233
|
-
const owners = `${prefix}session_owners`;
|
|
64234
64539
|
if (mutation.kind === "session-create") {
|
|
64235
64540
|
if (mutation.header.id !== sessionId)
|
|
64236
64541
|
throw new Error("Pool session identity mismatch");
|
|
@@ -64240,10 +64545,12 @@ async function applyPoolSession(tx, prefix, workId, mutation) {
|
|
|
64240
64545
|
if (owner?.work !== workId || owner.replay_key !== replayKey)
|
|
64241
64546
|
throw new Error("Pool session scope mismatch");
|
|
64242
64547
|
}
|
|
64548
|
+
return withSessionStore(tx, prefix, (store) => mutation.kind === "session-create" ? store.create(mutation.header, mutation.entries) : store.replace(sessionId, mutation.header, mutation.entries, { expectSeq: mutation.expectSeq }));
|
|
64549
|
+
}
|
|
64550
|
+
async function withSessionStore(tx, prefix, write) {
|
|
64243
64551
|
const store = await sessionStore(tx, prefix);
|
|
64244
64552
|
try {
|
|
64245
|
-
|
|
64246
|
-
return result.lastSeq;
|
|
64553
|
+
return (await write(store)).lastSeq;
|
|
64247
64554
|
} finally {
|
|
64248
64555
|
await store.close();
|
|
64249
64556
|
}
|
|
@@ -64291,8 +64598,11 @@ var init_pool_stage_usage = __esm(() => {
|
|
|
64291
64598
|
});
|
|
64292
64599
|
|
|
64293
64600
|
// dist/builtin/workflows/src/durable/journal/sql-pool-queue.ts
|
|
64294
|
-
import { randomUUID as randomUUID16 } from "node:crypto";
|
|
64601
|
+
import { createHash as createHash4, randomUUID as randomUUID16 } from "node:crypto";
|
|
64295
64602
|
import { assertStorageSchemaVersion as assertStorageSchemaVersion2 } from "@bastani/atomic/storage";
|
|
64603
|
+
function requestIdentity(request) {
|
|
64604
|
+
return createHash4("sha256").update(request).digest("hex");
|
|
64605
|
+
}
|
|
64296
64606
|
function fromRow(row) {
|
|
64297
64607
|
return {
|
|
64298
64608
|
...JSON.parse(row.descriptor),
|
|
@@ -64348,10 +64658,11 @@ class SqlPoolQueue {
|
|
|
64348
64658
|
const versions = await tx.all(`SELECT version FROM ${pool.meta}`);
|
|
64349
64659
|
if (versions.length > 1)
|
|
64350
64660
|
throw new Error("Corrupt pool schema metadata");
|
|
64351
|
-
|
|
64352
|
-
|
|
64353
|
-
|
|
64354
|
-
|
|
64661
|
+
const version = versions[0]?.version;
|
|
64662
|
+
if (version === undefined)
|
|
64663
|
+
await tx.run(`INSERT INTO ${pool.meta} VALUES (?)`, POOL_SCHEMA_VERSION);
|
|
64664
|
+
else if (version !== 1)
|
|
64665
|
+
assertStorageSchemaVersion2("journal", version, POOL_SCHEMA_VERSION);
|
|
64355
64666
|
await tx.exec(`
|
|
64356
64667
|
CREATE TABLE IF NOT EXISTS ${pool.roots}(root TEXT PRIMARY KEY REFERENCES ${journal.runs}(run_id) ON DELETE CASCADE, timeout INTEGER NOT NULL);
|
|
64357
64668
|
CREATE TABLE IF NOT EXISTS ${pool.work}(
|
|
@@ -64382,9 +64693,23 @@ class SqlPoolQueue {
|
|
|
64382
64693
|
state TEXT NOT NULL DEFAULT 'running', revision INTEGER NOT NULL DEFAULT 0
|
|
64383
64694
|
)`);
|
|
64384
64695
|
await tx.exec(poolUsageSchema(journal.runs));
|
|
64696
|
+
if (version === 1)
|
|
64697
|
+
await pool.migrateFromVersion1(tx);
|
|
64385
64698
|
});
|
|
64386
64699
|
return pool;
|
|
64387
64700
|
}
|
|
64701
|
+
async migrateFromVersion1(tx) {
|
|
64702
|
+
await tx.run(`DELETE FROM ${this.operations} WHERE substr(request, 1, 1) = '['`);
|
|
64703
|
+
for (let after = 0;; ) {
|
|
64704
|
+
const rows = await tx.all(`SELECT rowid, request FROM ${this.operations} WHERE rowid > ? ORDER BY rowid LIMIT 256`, after);
|
|
64705
|
+
if (rows.length === 0)
|
|
64706
|
+
break;
|
|
64707
|
+
for (const row of rows)
|
|
64708
|
+
await tx.run(`UPDATE ${this.operations} SET request = ? WHERE rowid = ?`, requestIdentity(String(row.request)), row.rowid);
|
|
64709
|
+
after = Number(rows.at(-1).rowid);
|
|
64710
|
+
}
|
|
64711
|
+
await tx.run(`UPDATE ${this.meta} SET version = ?`, POOL_SCHEMA_VERSION);
|
|
64712
|
+
}
|
|
64388
64713
|
async authority(tx, lease, timeout) {
|
|
64389
64714
|
const row = await tx.get(`SELECT l.*, r.status, p.timeout, ${STORE_NOW2} AS now FROM ${this.journal.leases} l JOIN ${this.journal.runs} r ON r.run_id = l.run_id LEFT JOIN ${this.roots} p ON p.root = l.run_id WHERE l.run_id = ?`, lease.runId);
|
|
64390
64715
|
if (row?.timeout === null && timeout === undefined)
|
|
@@ -64525,31 +64850,18 @@ class SqlPoolQueue {
|
|
|
64525
64850
|
throw new Error(STALE_WORK_LEASE);
|
|
64526
64851
|
return row;
|
|
64527
64852
|
}
|
|
64528
|
-
async renew(lease, token,
|
|
64853
|
+
async renew(lease, token, durationMs) {
|
|
64529
64854
|
positive(durationMs);
|
|
64530
64855
|
lease = snapshotLease(lease);
|
|
64531
64856
|
token = { ...token };
|
|
64532
|
-
validateText("key", operationId);
|
|
64533
|
-
const request = JSON.stringify([token.epoch, token.generation, token.worker, durationMs]);
|
|
64534
64857
|
return this.transaction(async (tx) => {
|
|
64535
64858
|
const { now } = await this.authority(tx, lease);
|
|
64536
64859
|
const row = await this.current(tx, lease, token);
|
|
64537
64860
|
if (row.epoch !== lease.epoch || row.expires_at < now)
|
|
64538
64861
|
throw new Error(STALE_WORK_LEASE);
|
|
64539
|
-
const
|
|
64540
|
-
|
|
64541
|
-
|
|
64542
|
-
throw new Error("Pool operation identity conflict");
|
|
64543
|
-
return JSON.parse(previous.receipt);
|
|
64544
|
-
}
|
|
64545
|
-
const receipt = {
|
|
64546
|
-
...fromRow(row),
|
|
64547
|
-
expiresAt: Math.max(Number(row.expires_at), expiry(now, durationMs)),
|
|
64548
|
-
renewedAt: now
|
|
64549
|
-
};
|
|
64550
|
-
await tx.run(`UPDATE ${this.work} SET expires_at = ?, renewed_at = ? WHERE id = ?`, receipt.expiresAt, now, row.id);
|
|
64551
|
-
await tx.run(`INSERT INTO ${this.operations} VALUES (?, ?, ?, ?, ?)`, row.id, token.generation, operationId, request, JSON.stringify(receipt));
|
|
64552
|
-
return receipt;
|
|
64862
|
+
const expiresAt = Math.max(Number(row.expires_at), expiry(now, durationMs));
|
|
64863
|
+
await tx.run(`UPDATE ${this.work} SET expires_at = ?, renewed_at = ? WHERE id = ?`, expiresAt, now, row.id);
|
|
64864
|
+
return { ...fromRow(row), expiresAt, renewedAt: now };
|
|
64553
64865
|
});
|
|
64554
64866
|
}
|
|
64555
64867
|
async commit(lease, token, operationId, mutation) {
|
|
@@ -64558,6 +64870,7 @@ class SqlPoolQueue {
|
|
|
64558
64870
|
validateText("key", operationId);
|
|
64559
64871
|
assertWorkflowSerializableObject(mutation, "Pool mutation");
|
|
64560
64872
|
const request = serializeJson({ epoch: token.epoch, worker: token.worker, mutation });
|
|
64873
|
+
const identity = requestIdentity(request);
|
|
64561
64874
|
mutation = JSON.parse(request).mutation;
|
|
64562
64875
|
return this.transaction(async (tx) => {
|
|
64563
64876
|
const { now, status } = await this.authority(tx, lease);
|
|
@@ -64566,7 +64879,7 @@ class SqlPoolQueue {
|
|
|
64566
64879
|
throw new Error(STALE_WORK_LEASE);
|
|
64567
64880
|
const previous = await tx.get(`SELECT * FROM ${this.operations} WHERE work = ? AND generation = ? AND operation = ?`, token.id, token.generation, operationId);
|
|
64568
64881
|
if (previous) {
|
|
64569
|
-
if (previous.request !==
|
|
64882
|
+
if (previous.request !== identity)
|
|
64570
64883
|
throw new Error("Pool operation identity conflict");
|
|
64571
64884
|
return JSON.parse(previous.receipt);
|
|
64572
64885
|
}
|
|
@@ -64574,34 +64887,41 @@ class SqlPoolQueue {
|
|
|
64574
64887
|
throw new Error(STALE_WORK_LEASE);
|
|
64575
64888
|
if (!await this.admitsAttemptWrites(tx, lease, status))
|
|
64576
64889
|
throw new Error("Pool root is not running");
|
|
64577
|
-
const
|
|
64578
|
-
await
|
|
64579
|
-
if (checkpoint.completedAt > Math.max(now, Date.now()) + MAX_WORKER_CLOCK_SKEW_MS)
|
|
64580
|
-
throw new Error("Pool checkpoint timestamp is in the future");
|
|
64581
|
-
assertWorkflowSerializableObject(checkpoint, "Pool checkpoint");
|
|
64582
|
-
const key = JSON.stringify([RUN_EVENT_TYPES.checkpointRecorded, checkpoint.kind, checkpoint.checkpointId]);
|
|
64583
|
-
const payload = serializeJson(checkpoint);
|
|
64584
|
-
const existing = await tx.get(`SELECT seq, at, payload FROM ${this.journal.events} WHERE run_id = ? AND key = ?`, lease.runId, key);
|
|
64585
|
-
if (existing && existing.payload !== payload)
|
|
64586
|
-
throw new Error("Pool checkpoint identity conflict");
|
|
64587
|
-
if (existing && mutation.kind !== "checkpoint")
|
|
64588
|
-
throw new Error("Pool session checkpoint identity conflict");
|
|
64589
|
-
const lastSeq = mutation.kind === "checkpoint" ? undefined : await applyPoolSession(tx, `${this.journal.runs}_pool_`, row.id, mutation);
|
|
64590
|
-
const latest = await tx.get(`SELECT COALESCE(MAX(seq), 0) AS seq FROM ${this.journal.events} WHERE run_id = ?`, lease.runId);
|
|
64591
|
-
const receipt = existing ? { seq: Number(existing.seq), at: Number(existing.at) } : {
|
|
64592
|
-
seq: journalCounter(lease.runId, "seq", Number(latest.seq) + 1, "overflow"),
|
|
64593
|
-
at: now,
|
|
64594
|
-
...lastSeq === undefined ? {} : { lastSeq }
|
|
64595
|
-
};
|
|
64596
|
-
if (!existing)
|
|
64597
|
-
await tx.run(`INSERT INTO ${this.journal.events} VALUES (?, ?, ?, ?, ?, ?, ?)`, lease.runId, receipt.seq, lease.epoch, key, RUN_EVENT_TYPES.checkpointRecorded, payload, now);
|
|
64598
|
-
const billed = poolCheckpointUsage(checkpoint);
|
|
64599
|
-
if (billed && !existing)
|
|
64600
|
-
await upsertPoolUsage(tx, this.journal.runs, row.id, token.generation, billed.key, billed.usage);
|
|
64601
|
-
await tx.run(`INSERT INTO ${this.operations} VALUES (?, ?, ?, ?, ?)`, token.id, token.generation, operationId, request, JSON.stringify(receipt));
|
|
64890
|
+
const receipt = mutation.kind === "session-append" ? await this.appendSession(tx, row, mutation, now) : await this.recordCheckpoint(tx, lease, token, row, mutation, now);
|
|
64891
|
+
await tx.run(`INSERT INTO ${this.operations} VALUES (?, ?, ?, ?, ?)`, token.id, token.generation, operationId, identity, JSON.stringify(receipt));
|
|
64602
64892
|
return receipt;
|
|
64603
64893
|
});
|
|
64604
64894
|
}
|
|
64895
|
+
async appendSession(tx, row, mutation, now) {
|
|
64896
|
+
return { at: now, lastSeq: await applyPoolSession(tx, `${this.journal.runs}_pool_`, row.id, mutation) };
|
|
64897
|
+
}
|
|
64898
|
+
async recordCheckpoint(tx, lease, token, row, mutation, now) {
|
|
64899
|
+
const checkpoint = mutation.checkpoint;
|
|
64900
|
+
await validatePoolMutation(tx, this.journal, lease.runId, fromRow(row), mutation);
|
|
64901
|
+
if (checkpoint.completedAt > Math.max(now, Date.now()) + MAX_WORKER_CLOCK_SKEW_MS)
|
|
64902
|
+
throw new Error("Pool checkpoint timestamp is in the future");
|
|
64903
|
+
assertWorkflowSerializableObject(checkpoint, "Pool checkpoint");
|
|
64904
|
+
const key = JSON.stringify([RUN_EVENT_TYPES.checkpointRecorded, checkpoint.kind, checkpoint.checkpointId]);
|
|
64905
|
+
const payload = serializeJson(checkpoint);
|
|
64906
|
+
const existing = await tx.get(`SELECT seq, at, payload FROM ${this.journal.events} WHERE run_id = ? AND key = ?`, lease.runId, key);
|
|
64907
|
+
if (existing && existing.payload !== payload)
|
|
64908
|
+
throw new Error("Pool checkpoint identity conflict");
|
|
64909
|
+
if (existing && mutation.kind !== "checkpoint")
|
|
64910
|
+
throw new Error("Pool session checkpoint identity conflict");
|
|
64911
|
+
const lastSeq = mutation.kind === "checkpoint" ? undefined : await applyPoolSession(tx, `${this.journal.runs}_pool_`, row.id, mutation);
|
|
64912
|
+
const latest = await tx.get(`SELECT COALESCE(MAX(seq), 0) AS seq FROM ${this.journal.events} WHERE run_id = ?`, lease.runId);
|
|
64913
|
+
const receipt = existing ? { seq: Number(existing.seq), at: Number(existing.at) } : {
|
|
64914
|
+
seq: journalCounter(lease.runId, "seq", Number(latest.seq) + 1, "overflow"),
|
|
64915
|
+
at: now,
|
|
64916
|
+
...lastSeq === undefined ? {} : { lastSeq }
|
|
64917
|
+
};
|
|
64918
|
+
if (!existing)
|
|
64919
|
+
await tx.run(`INSERT INTO ${this.journal.events} VALUES (?, ?, ?, ?, ?, ?, ?)`, lease.runId, receipt.seq, lease.epoch, key, RUN_EVENT_TYPES.checkpointRecorded, payload, now);
|
|
64920
|
+
const billed = poolCheckpointUsage(checkpoint);
|
|
64921
|
+
if (billed && !existing)
|
|
64922
|
+
await upsertPoolUsage(tx, this.journal.runs, row.id, token.generation, billed.key, billed.usage);
|
|
64923
|
+
return receipt;
|
|
64924
|
+
}
|
|
64605
64925
|
async complete(lease, token, result) {
|
|
64606
64926
|
lease = snapshotLease(lease);
|
|
64607
64927
|
token = { ...token };
|
|
@@ -64688,7 +65008,7 @@ function openSqlPoolQueue(journal) {
|
|
|
64688
65008
|
return Promise.reject(new Error("Worker pool requires a SQL run journal"));
|
|
64689
65009
|
return SqlPoolQueue.open(storage);
|
|
64690
65010
|
}
|
|
64691
|
-
var STORE_NOW2 = "CAST(unixepoch('subsec') * 1000 AS INTEGER)", MAX_WORKER_CLOCK_SKEW_MS = 1000, STALE_WORK_LEASE = "Stale work lease";
|
|
65011
|
+
var STORE_NOW2 = "CAST(unixepoch('subsec') * 1000 AS INTEGER)", MAX_WORKER_CLOCK_SKEW_MS = 1000, POOL_SCHEMA_VERSION = 2, STALE_WORK_LEASE = "Stale work lease";
|
|
64692
65012
|
var init_sql_pool_queue = __esm(() => {
|
|
64693
65013
|
init_serializable();
|
|
64694
65014
|
init_boundary_topology();
|
|
@@ -64700,6 +65020,15 @@ var init_sql_pool_queue = __esm(() => {
|
|
|
64700
65020
|
init_values();
|
|
64701
65021
|
});
|
|
64702
65022
|
|
|
65023
|
+
// dist/builtin/workflows/src/workers/coordinator-protocol.ts
|
|
65024
|
+
var POOL_COORDINATOR_ROLE = "pool-coordinator", POOL_SUMMARY_INTERVAL_MS = 250;
|
|
65025
|
+
|
|
65026
|
+
// dist/builtin/workflows/src/workers/host-options.ts
|
|
65027
|
+
var hostWorkerBindings;
|
|
65028
|
+
var init_host_options = __esm(() => {
|
|
65029
|
+
hostWorkerBindings = Symbol("host-worker-bindings");
|
|
65030
|
+
});
|
|
65031
|
+
|
|
64703
65032
|
// dist/builtin/workflows/src/durable/journal/sql-pool-usage.ts
|
|
64704
65033
|
function validate(report) {
|
|
64705
65034
|
validateText("key", report.key);
|
|
@@ -64770,10 +65099,844 @@ var init_sql_pool_usage = __esm(() => {
|
|
|
64770
65099
|
init_values();
|
|
64771
65100
|
});
|
|
64772
65101
|
|
|
64773
|
-
// dist/builtin/workflows/src/workers/
|
|
64774
|
-
|
|
64775
|
-
|
|
64776
|
-
|
|
65102
|
+
// dist/builtin/workflows/src/workers/pool-coordinator.ts
|
|
65103
|
+
import { fork } from "node:child_process";
|
|
65104
|
+
import { randomUUID as randomUUID17 } from "node:crypto";
|
|
65105
|
+
import { createWriteStream } from "node:fs";
|
|
65106
|
+
import { mkdir as mkdir6, rm as rm3, rmdir } from "node:fs/promises";
|
|
65107
|
+
import { dirname as dirname14, join as join29, resolve as resolve13 } from "node:path";
|
|
65108
|
+
import { setTimeout as delay2 } from "node:timers/promises";
|
|
65109
|
+
function workerLogDirectory(journalPath, root) {
|
|
65110
|
+
return join29(dirname14(resolve13(journalPath)), "worker-logs", encodeURIComponent(root).replaceAll(".", "%2E"));
|
|
65111
|
+
}
|
|
65112
|
+
function token(work) {
|
|
65113
|
+
return {
|
|
65114
|
+
id: work.id,
|
|
65115
|
+
generation: work.generation,
|
|
65116
|
+
worker: work.worker,
|
|
65117
|
+
epoch: work.epoch,
|
|
65118
|
+
renewedAt: work.renewedAt,
|
|
65119
|
+
expiresAt: work.expiresAt
|
|
65120
|
+
};
|
|
65121
|
+
}
|
|
65122
|
+
function outside(runId) {
|
|
65123
|
+
return new Error(`Pool stage publication is outside the work subtree: ${runId}`);
|
|
65124
|
+
}
|
|
65125
|
+
function isLiveStage(stage) {
|
|
65126
|
+
return (stage.sessionId !== undefined || stage.sessionFile !== undefined) && stage.status !== "completed" && stage.status !== "failed" && stage.status !== "skipped";
|
|
65127
|
+
}
|
|
65128
|
+
function projectRootRun(run) {
|
|
65129
|
+
return {
|
|
65130
|
+
id: run.id,
|
|
65131
|
+
name: run.name,
|
|
65132
|
+
status: run.status,
|
|
65133
|
+
startedAt: run.startedAt,
|
|
65134
|
+
stages: run.stages.map((stage) => ({
|
|
65135
|
+
id: stage.id,
|
|
65136
|
+
name: stage.name,
|
|
65137
|
+
status: stage.status,
|
|
65138
|
+
parentIds: [],
|
|
65139
|
+
toolEvents: [],
|
|
65140
|
+
...stage.replayKey === undefined ? {} : { replayKey: stage.replayKey },
|
|
65141
|
+
...stage.intercomGroup === undefined ? {} : { intercomGroup: stage.intercomGroup },
|
|
65142
|
+
...stage.workflowChildRun === undefined ? {} : { workflowChildRun: stage.workflowChildRun },
|
|
65143
|
+
...stage.workflowChild === undefined ? {} : { workflowChild: { ...stage.workflowChild, outputs: {} } }
|
|
65144
|
+
}))
|
|
65145
|
+
};
|
|
65146
|
+
}
|
|
65147
|
+
|
|
65148
|
+
class PooledRoutes {
|
|
65149
|
+
root;
|
|
65150
|
+
owner;
|
|
65151
|
+
mirror = createStore();
|
|
65152
|
+
owners = new Map;
|
|
65153
|
+
attempts = new Map;
|
|
65154
|
+
activated = false;
|
|
65155
|
+
bridge;
|
|
65156
|
+
constructor(root, owner) {
|
|
65157
|
+
this.root = root;
|
|
65158
|
+
this.owner = owner;
|
|
65159
|
+
}
|
|
65160
|
+
static async open(root, cwd, rootRun, forward) {
|
|
65161
|
+
await Promise.resolve().then(() => init_workflow_route_owner());
|
|
65162
|
+
const owner = createWorkflowRouteOwner(cwd);
|
|
65163
|
+
const routes = new PooledRoutes(root, owner);
|
|
65164
|
+
if (rootRun !== undefined)
|
|
65165
|
+
routes.updateRoot(rootRun);
|
|
65166
|
+
routes.bridge = registerPendingStageIntercomBridge({
|
|
65167
|
+
events: {
|
|
65168
|
+
emit: (event, payload) => owner.events.emit(event, payload),
|
|
65169
|
+
on: (event, listener) => event === PENDING_STAGE_MESSAGE_EVENT2 || event === STICKY_LIVE_DELIVERED_EVENT2 ? undefined : owner.events.on(event, listener)
|
|
65170
|
+
}
|
|
65171
|
+
}, routes.mirror, (runId) => routes.activated && routes.owners.has(runId));
|
|
65172
|
+
for (const [event, kind] of [
|
|
65173
|
+
[PENDING_STAGE_MESSAGE_EVENT2, "pending-stage-message"],
|
|
65174
|
+
[STICKY_LIVE_DELIVERED_EVENT2, "sticky-live-delivered"]
|
|
65175
|
+
])
|
|
65176
|
+
owner.events.on(event, (payload) => {
|
|
65177
|
+
const claimed = payload;
|
|
65178
|
+
if (claimed.handled !== false)
|
|
65179
|
+
return;
|
|
65180
|
+
claimed.handled = true;
|
|
65181
|
+
const { handled: _handled, completion: _completion, ...forwarded } = claimed;
|
|
65182
|
+
claimed.completion = forward({ kind, payload: forwarded });
|
|
65183
|
+
});
|
|
65184
|
+
return routes;
|
|
65185
|
+
}
|
|
65186
|
+
updateRoot(rootRun) {
|
|
65187
|
+
this.mirror.recordRunStart({ ...rootRun, inputs: {}, toolNodes: [], rootRunId: this.root });
|
|
65188
|
+
}
|
|
65189
|
+
accept(entries) {
|
|
65190
|
+
this.mirror.batch(() => {
|
|
65191
|
+
for (const { work, runs } of entries) {
|
|
65192
|
+
let attempt = this.attempts.get(work.id);
|
|
65193
|
+
if (attempt !== undefined && work.generation < attempt.generation)
|
|
65194
|
+
continue;
|
|
65195
|
+
if (attempt === undefined || attempt.generation !== work.generation) {
|
|
65196
|
+
if (attempt !== undefined)
|
|
65197
|
+
this.release(attempt);
|
|
65198
|
+
attempt = { generation: work.generation, runs: attempt?.runs ?? new Set, fenced: false };
|
|
65199
|
+
this.attempts.set(work.id, attempt);
|
|
65200
|
+
}
|
|
65201
|
+
if (attempt.fenced)
|
|
65202
|
+
continue;
|
|
65203
|
+
for (const run of runs) {
|
|
65204
|
+
attempt.runs.add(run.id);
|
|
65205
|
+
this.owners.set(run.id, work.id);
|
|
65206
|
+
const { pendingStageMessages: _messages, ...published } = run;
|
|
65207
|
+
this.mirror.recordRunStart({ ...published, rootRunId: this.root });
|
|
65208
|
+
if (run.stages.some((stage) => stage.pendingStageDeliveryAvailable === true))
|
|
65209
|
+
this.activated = true;
|
|
65210
|
+
}
|
|
65211
|
+
}
|
|
65212
|
+
});
|
|
65213
|
+
}
|
|
65214
|
+
fence(work) {
|
|
65215
|
+
const attempt = this.attempts.get(work.id);
|
|
65216
|
+
if (attempt === undefined || attempt.generation > work.generation)
|
|
65217
|
+
return;
|
|
65218
|
+
if (attempt.generation < work.generation) {
|
|
65219
|
+
this.release(attempt);
|
|
65220
|
+
this.attempts.set(work.id, { generation: work.generation, runs: attempt.runs, fenced: true });
|
|
65221
|
+
return;
|
|
65222
|
+
}
|
|
65223
|
+
this.release(attempt);
|
|
65224
|
+
attempt.fenced = true;
|
|
65225
|
+
}
|
|
65226
|
+
async route(work, runId) {
|
|
65227
|
+
this.requireOwned(work, runId);
|
|
65228
|
+
const readiness = workflowPendingStageRouteReady(this.mirror, runId);
|
|
65229
|
+
await readiness?.completion.catch(() => {});
|
|
65230
|
+
readiness?.assertCurrent();
|
|
65231
|
+
const waiting = new AbortController;
|
|
65232
|
+
await Promise.race([
|
|
65233
|
+
this.owner.registered(runId),
|
|
65234
|
+
delay2(ROUTE_REGISTRATION_WAIT_MS, undefined, { signal: waiting.signal, ref: false }).catch(() => {})
|
|
65235
|
+
]);
|
|
65236
|
+
waiting.abort();
|
|
65237
|
+
this.requireOwned(work, runId);
|
|
65238
|
+
return workflowPendingStageRouteCapability(this.mirror, runId);
|
|
65239
|
+
}
|
|
65240
|
+
async undeliverable(payload) {
|
|
65241
|
+
const event = { ...payload, handled: false };
|
|
65242
|
+
this.owner.events.emit(PENDING_STAGE_UNDELIVERABLE_EVENT2, event);
|
|
65243
|
+
return event.handled && await event.completion === true;
|
|
65244
|
+
}
|
|
65245
|
+
async close() {
|
|
65246
|
+
const bridge = this.bridge;
|
|
65247
|
+
if (bridge === undefined)
|
|
65248
|
+
return;
|
|
65249
|
+
this.bridge = undefined;
|
|
65250
|
+
try {
|
|
65251
|
+
await bridge.flush();
|
|
65252
|
+
} finally {
|
|
65253
|
+
bridge.refresh();
|
|
65254
|
+
bridge();
|
|
65255
|
+
await this.owner.closeGroup(workflowInvocationIntercomGroup(this.root));
|
|
65256
|
+
}
|
|
65257
|
+
}
|
|
65258
|
+
requireOwned(work, runId) {
|
|
65259
|
+
const attempt = this.attempts.get(work.id);
|
|
65260
|
+
if (attempt === undefined || attempt.generation !== work.generation || attempt.fenced)
|
|
65261
|
+
throw new Error(STALE_WORK_LEASE);
|
|
65262
|
+
if (!attempt.runs.has(runId))
|
|
65263
|
+
throw outside(runId);
|
|
65264
|
+
}
|
|
65265
|
+
release(attempt) {
|
|
65266
|
+
this.mirror.batch(() => {
|
|
65267
|
+
for (const id of attempt.runs) {
|
|
65268
|
+
rotateWorkflowPendingStageRouteCapability(this.mirror, id);
|
|
65269
|
+
const run = this.mirror.runs().find((candidate) => candidate.id === id);
|
|
65270
|
+
if (run?.stages.some(isLiveStage) !== true)
|
|
65271
|
+
continue;
|
|
65272
|
+
this.mirror.recordRunStart({
|
|
65273
|
+
...run,
|
|
65274
|
+
stages: run.stages.map((stage) => {
|
|
65275
|
+
if (!isLiveStage(stage))
|
|
65276
|
+
return stage;
|
|
65277
|
+
const { sessionId: _session, sessionFile: _file, ...rest } = stage;
|
|
65278
|
+
return { ...rest, attachable: false, attached: false };
|
|
65279
|
+
})
|
|
65280
|
+
});
|
|
65281
|
+
}
|
|
65282
|
+
});
|
|
65283
|
+
}
|
|
65284
|
+
}
|
|
65285
|
+
|
|
65286
|
+
class PoolCoordinator {
|
|
65287
|
+
port;
|
|
65288
|
+
children = new Set;
|
|
65289
|
+
incarnations = new Map;
|
|
65290
|
+
readyWorkers = new Set;
|
|
65291
|
+
logs = new Map;
|
|
65292
|
+
cleanups = new Set;
|
|
65293
|
+
controlAcks = new Map;
|
|
65294
|
+
lastHeard = new Map;
|
|
65295
|
+
pendingActivity = new Map;
|
|
65296
|
+
batches = new Map;
|
|
65297
|
+
publicationFailures = new Map;
|
|
65298
|
+
requests = new Map;
|
|
65299
|
+
deliveryRequests = new Map;
|
|
65300
|
+
inflight = new Set;
|
|
65301
|
+
lastTick = performance.now();
|
|
65302
|
+
stalledMs = 0;
|
|
65303
|
+
ticker;
|
|
65304
|
+
recoveryTimer;
|
|
65305
|
+
flushTimer;
|
|
65306
|
+
lastFlushAt = 0;
|
|
65307
|
+
batchSequence = 0;
|
|
65308
|
+
requestSequence = 0;
|
|
65309
|
+
usageChanged = false;
|
|
65310
|
+
budgetBlocked = false;
|
|
65311
|
+
desiredControl;
|
|
65312
|
+
root;
|
|
65313
|
+
lease;
|
|
65314
|
+
options;
|
|
65315
|
+
journal;
|
|
65316
|
+
queue;
|
|
65317
|
+
usageLedger;
|
|
65318
|
+
usageTotal;
|
|
65319
|
+
admission;
|
|
65320
|
+
routes;
|
|
65321
|
+
publishing = false;
|
|
65322
|
+
sources = [];
|
|
65323
|
+
logDirectory;
|
|
65324
|
+
starting;
|
|
65325
|
+
recovering;
|
|
65326
|
+
nextRecoveryAt = 0;
|
|
65327
|
+
cleanupFailure;
|
|
65328
|
+
closed = false;
|
|
65329
|
+
closing;
|
|
65330
|
+
replacements = 0;
|
|
65331
|
+
constructor(port) {
|
|
65332
|
+
this.port = port;
|
|
65333
|
+
port.onMessage((message) => this.receive(message));
|
|
65334
|
+
this.ticker = setInterval(() => this.observeResponsiveness(), TICK_INTERVAL_MS).unref();
|
|
65335
|
+
}
|
|
65336
|
+
async shutdown() {
|
|
65337
|
+
await this.closing?.catch(() => {});
|
|
65338
|
+
while (this.inflight.size > 0)
|
|
65339
|
+
await Promise.allSettled([...this.inflight]);
|
|
65340
|
+
await this.journal?.close().catch(() => {});
|
|
65341
|
+
}
|
|
65342
|
+
receive(message) {
|
|
65343
|
+
if (message.kind === "request-reply") {
|
|
65344
|
+
const pending = this.requests.get(message.id);
|
|
65345
|
+
this.requests.delete(message.id);
|
|
65346
|
+
if (message.error !== undefined)
|
|
65347
|
+
pending?.reject(new Error(message.error));
|
|
65348
|
+
else
|
|
65349
|
+
pending?.resolve(message.value);
|
|
65350
|
+
return;
|
|
65351
|
+
}
|
|
65352
|
+
if ("id" in message) {
|
|
65353
|
+
const command = message;
|
|
65354
|
+
const handled = this.execute(command).then((value2) => this.port.send({ kind: "reply", id: command.id, value: value2 }), (error) => this.port.send({ kind: "reply", id: command.id, error: error.message }));
|
|
65355
|
+
this.inflight.add(handled);
|
|
65356
|
+
handled.finally(() => this.inflight.delete(handled));
|
|
65357
|
+
return;
|
|
65358
|
+
}
|
|
65359
|
+
this.notice(message);
|
|
65360
|
+
}
|
|
65361
|
+
notice(notice) {
|
|
65362
|
+
if (notice.kind === "budget")
|
|
65363
|
+
this.budgetBlocked = notice.blocked;
|
|
65364
|
+
else if (notice.kind === "root")
|
|
65365
|
+
this.routes?.updateRoot(notice.rootRun);
|
|
65366
|
+
else {
|
|
65367
|
+
const batch = this.batches.get(notice.batch);
|
|
65368
|
+
this.batches.delete(notice.batch);
|
|
65369
|
+
const rejected = new Map(notice.rejected.map((entry) => [entry.workId, entry]));
|
|
65370
|
+
for (const entry of rejected.values())
|
|
65371
|
+
this.publicationFailures.set(entry.workId, { generation: entry.generation, error: entry.error });
|
|
65372
|
+
this.routes?.accept((batch?.entries ?? []).filter((entry) => rejected.get(entry.work.id)?.generation !== entry.work.generation));
|
|
65373
|
+
batch?.settle();
|
|
65374
|
+
}
|
|
65375
|
+
}
|
|
65376
|
+
async execute(command) {
|
|
65377
|
+
switch (command.kind) {
|
|
65378
|
+
case "prepare":
|
|
65379
|
+
return this.prepare(command.root, command.lease, command.options, command.control);
|
|
65380
|
+
case "control":
|
|
65381
|
+
return this.control(command.state);
|
|
65382
|
+
case "start":
|
|
65383
|
+
this.starting ??= this.start(command.sources, command.cwd, command.publish, command.rootRun);
|
|
65384
|
+
return this.starting;
|
|
65385
|
+
case "enqueue":
|
|
65386
|
+
return this.enqueue(command);
|
|
65387
|
+
case "transport":
|
|
65388
|
+
return this.transport(command.request, command);
|
|
65389
|
+
case "undeliverable":
|
|
65390
|
+
return await this.routes?.undeliverable(command.payload) ?? false;
|
|
65391
|
+
case "close":
|
|
65392
|
+
this.closing ??= this.close();
|
|
65393
|
+
return this.closing;
|
|
65394
|
+
}
|
|
65395
|
+
}
|
|
65396
|
+
async prepare(root, lease, options, control) {
|
|
65397
|
+
if (this.closed)
|
|
65398
|
+
throw new Error("Local worker pool is closed");
|
|
65399
|
+
if (this.root !== undefined && this.root !== root)
|
|
65400
|
+
throw new Error("Local worker pool belongs to another root");
|
|
65401
|
+
this.root = root;
|
|
65402
|
+
this.options = options;
|
|
65403
|
+
this.desiredControl ??= control;
|
|
65404
|
+
this.journal ??= await resolveRunJournal({ journal: { path: options.journalPath } });
|
|
65405
|
+
this.usageLedger ??= await openSqlPoolUsage(this.journal);
|
|
65406
|
+
this.usageTotal = await this.usageLedger.total(root);
|
|
65407
|
+
this.lease = lease;
|
|
65408
|
+
const queue = await openSqlPoolQueue(this.journal);
|
|
65409
|
+
await queue.initialize(lease, COORDINATOR_LEASE_MS);
|
|
65410
|
+
await queue.releasePreviousIncarnations(lease);
|
|
65411
|
+
for (;; ) {
|
|
65412
|
+
const intended = this.desiredControl ?? "running";
|
|
65413
|
+
await queue.setControl(lease, intended);
|
|
65414
|
+
if (intended === (this.desiredControl ?? "running"))
|
|
65415
|
+
break;
|
|
65416
|
+
}
|
|
65417
|
+
this.queue = queue;
|
|
65418
|
+
return this.usageTotal;
|
|
65419
|
+
}
|
|
65420
|
+
async control(state) {
|
|
65421
|
+
this.desiredControl = state;
|
|
65422
|
+
if (!this.queue || !this.lease)
|
|
65423
|
+
return;
|
|
65424
|
+
const control = await this.queue.setControl(this.lease, state);
|
|
65425
|
+
const deadline = Date.now() + CONTROL_ACKNOWLEDGEMENT_TIMEOUT_MS;
|
|
65426
|
+
while ([...this.readyWorkers].some((worker) => (this.controlAcks.get(worker) ?? -1) < control.revision)) {
|
|
65427
|
+
if (this.desiredControl !== state)
|
|
65428
|
+
return;
|
|
65429
|
+
if (Date.now() >= deadline) {
|
|
65430
|
+
if (state === "quit")
|
|
65431
|
+
break;
|
|
65432
|
+
throw new Error("Worker control acknowledgement unresolved");
|
|
65433
|
+
}
|
|
65434
|
+
await delay2(25);
|
|
65435
|
+
}
|
|
65436
|
+
this.flush();
|
|
65437
|
+
}
|
|
65438
|
+
async start(sources, cwd, publish, rootRun) {
|
|
65439
|
+
if (this.closed)
|
|
65440
|
+
throw new Error("Local worker pool is closed");
|
|
65441
|
+
const root = this.root;
|
|
65442
|
+
if (!this.queue || !this.lease || !this.options || root === undefined || this.lease.runId !== root)
|
|
65443
|
+
throw new Error("Local worker pool requires root preparation");
|
|
65444
|
+
this.sources = sources;
|
|
65445
|
+
this.publishing = publish;
|
|
65446
|
+
if (publish) {
|
|
65447
|
+
this.routes = await PooledRoutes.open(root, cwd, rootRun, (request) => this.request(request));
|
|
65448
|
+
if (this.closed) {
|
|
65449
|
+
await this.routes.close();
|
|
65450
|
+
throw new Error("Local worker pool is closed");
|
|
65451
|
+
}
|
|
65452
|
+
}
|
|
65453
|
+
if (this.options.admission)
|
|
65454
|
+
this.admission = await SqlResourceAdmission.open(this.journal, {
|
|
65455
|
+
...this.options.admission,
|
|
65456
|
+
root,
|
|
65457
|
+
holder: "coordinator"
|
|
65458
|
+
});
|
|
65459
|
+
this.logDirectory = workerLogDirectory(this.options.journalPath, root);
|
|
65460
|
+
await mkdir6(this.logDirectory, { recursive: true }).catch(() => {
|
|
65461
|
+
return;
|
|
65462
|
+
});
|
|
65463
|
+
this.recoveryTimer = setInterval(() => {
|
|
65464
|
+
this.recoverExpired().catch((error) => this.fail(error));
|
|
65465
|
+
}, RECOVERY_INTERVAL_MS).unref();
|
|
65466
|
+
const workers = Array.from({ length: this.options.workers }, () => this.spawn());
|
|
65467
|
+
try {
|
|
65468
|
+
await Promise.any(workers);
|
|
65469
|
+
} catch (error) {
|
|
65470
|
+
throw error.errors[0];
|
|
65471
|
+
}
|
|
65472
|
+
}
|
|
65473
|
+
async enqueue(command) {
|
|
65474
|
+
if (!this.queue || !this.lease || this.root === undefined)
|
|
65475
|
+
throw new Error("Local worker pool requires root preparation");
|
|
65476
|
+
if (command.rootRun !== undefined)
|
|
65477
|
+
this.routes?.updateRoot(command.rootRun);
|
|
65478
|
+
const work = await this.queue.enqueue(this.lease, command.descriptor, command.boundary);
|
|
65479
|
+
const result = await this.queue.result(this.root, work.id);
|
|
65480
|
+
return { work, ...result === undefined ? {} : { result } };
|
|
65481
|
+
}
|
|
65482
|
+
async transport(request, payload) {
|
|
65483
|
+
const target = this.deliveryRequests.get(request);
|
|
65484
|
+
if (target === undefined || !this.incarnations.has(target.slot.worker))
|
|
65485
|
+
throw new Error("Worker exited before delivery acknowledgement");
|
|
65486
|
+
const { slot, workerRequest } = target;
|
|
65487
|
+
await new Promise((resolve, reject) => {
|
|
65488
|
+
const id = randomUUID17();
|
|
65489
|
+
slot.deliveries.set(id, { resolve, reject });
|
|
65490
|
+
slot.child.send({ id: workerRequest, delivery: { id, from: payload.from, message: payload.message } }, (error) => {
|
|
65491
|
+
if (error) {
|
|
65492
|
+
slot.deliveries.delete(id);
|
|
65493
|
+
reject(error);
|
|
65494
|
+
}
|
|
65495
|
+
});
|
|
65496
|
+
});
|
|
65497
|
+
}
|
|
65498
|
+
request(body) {
|
|
65499
|
+
const id = ++this.requestSequence;
|
|
65500
|
+
return new Promise((resolve, reject) => {
|
|
65501
|
+
this.requests.set(id, { resolve, reject });
|
|
65502
|
+
this.port.send({ kind: "request", id, request: body });
|
|
65503
|
+
});
|
|
65504
|
+
}
|
|
65505
|
+
fail(error) {
|
|
65506
|
+
this.port.send({ kind: "failure", error: error.message });
|
|
65507
|
+
}
|
|
65508
|
+
publish(work, runs) {
|
|
65509
|
+
const failure = this.publicationFailures.get(work.id);
|
|
65510
|
+
if (failure?.generation === work.generation)
|
|
65511
|
+
throw new Error(failure.error);
|
|
65512
|
+
if (!this.publishing)
|
|
65513
|
+
return;
|
|
65514
|
+
this.pendingActivity.set(work.id, { work: token(work), runs });
|
|
65515
|
+
this.scheduleFlush();
|
|
65516
|
+
}
|
|
65517
|
+
scheduleFlush() {
|
|
65518
|
+
if (this.flushTimer !== undefined || this.closed)
|
|
65519
|
+
return;
|
|
65520
|
+
const wait = Math.max(0, this.lastFlushAt + POOL_SUMMARY_INTERVAL_MS - performance.now());
|
|
65521
|
+
this.flushTimer = setTimeout(() => {
|
|
65522
|
+
this.flushTimer = undefined;
|
|
65523
|
+
if (this.batches.size > 0) {
|
|
65524
|
+
this.lastFlushAt = performance.now();
|
|
65525
|
+
this.scheduleFlush();
|
|
65526
|
+
return;
|
|
65527
|
+
}
|
|
65528
|
+
this.flush();
|
|
65529
|
+
}, wait);
|
|
65530
|
+
}
|
|
65531
|
+
flush() {
|
|
65532
|
+
clearTimeout(this.flushTimer);
|
|
65533
|
+
this.flushTimer = undefined;
|
|
65534
|
+
if (this.pendingActivity.size === 0 && !this.usageChanged) {
|
|
65535
|
+
const last = [...this.batches.values()].at(-1);
|
|
65536
|
+
return last === undefined ? Promise.resolve() : new Promise((resolve) => this.chainSettle(last, resolve));
|
|
65537
|
+
}
|
|
65538
|
+
this.lastFlushAt = performance.now();
|
|
65539
|
+
const batch = ++this.batchSequence;
|
|
65540
|
+
const entries = [...this.pendingActivity.values()];
|
|
65541
|
+
this.pendingActivity.clear();
|
|
65542
|
+
const usage = this.usageChanged ? this.usageTotal : undefined;
|
|
65543
|
+
this.usageChanged = false;
|
|
65544
|
+
const settled = new Promise((resolve) => this.batches.set(batch, { entries, settle: resolve }));
|
|
65545
|
+
this.port.send({ kind: "activity", batch, entries, ...usage === undefined ? {} : { usage } });
|
|
65546
|
+
return settled;
|
|
65547
|
+
}
|
|
65548
|
+
chainSettle(batch, resolve) {
|
|
65549
|
+
const settle = batch.settle;
|
|
65550
|
+
batch.settle = () => {
|
|
65551
|
+
settle();
|
|
65552
|
+
resolve();
|
|
65553
|
+
};
|
|
65554
|
+
}
|
|
65555
|
+
noteUsage(usage) {
|
|
65556
|
+
this.usageTotal = usage;
|
|
65557
|
+
this.usageChanged = true;
|
|
65558
|
+
this.scheduleFlush();
|
|
65559
|
+
}
|
|
65560
|
+
async sendResult(workId) {
|
|
65561
|
+
const result = await this.queue.result(this.root, workId);
|
|
65562
|
+
if (result === undefined)
|
|
65563
|
+
return false;
|
|
65564
|
+
this.flush();
|
|
65565
|
+
this.port.send({
|
|
65566
|
+
kind: "result",
|
|
65567
|
+
workId,
|
|
65568
|
+
result,
|
|
65569
|
+
...this.usageTotal === undefined ? {} : { usage: this.usageTotal }
|
|
65570
|
+
});
|
|
65571
|
+
return true;
|
|
65572
|
+
}
|
|
65573
|
+
fence(work) {
|
|
65574
|
+
this.pendingActivity.delete(work.id);
|
|
65575
|
+
this.routes?.fence(work);
|
|
65576
|
+
this.port.send({ kind: "fence", work: { id: work.id, generation: work.generation } });
|
|
65577
|
+
}
|
|
65578
|
+
spawn() {
|
|
65579
|
+
if (this.closed)
|
|
65580
|
+
return Promise.reject(new Error("Local worker pool is closed"));
|
|
65581
|
+
const root = this.root;
|
|
65582
|
+
const options = this.options;
|
|
65583
|
+
const child = fork(options.workerEntry ?? process.argv[1], [
|
|
65584
|
+
JSON.stringify({
|
|
65585
|
+
journalPath: options.journalPath,
|
|
65586
|
+
root,
|
|
65587
|
+
definitions: this.sources,
|
|
65588
|
+
admission: options.admission
|
|
65589
|
+
})
|
|
65590
|
+
], { stdio: ["ignore", "ignore", "pipe", "ipc"], execArgv: [] });
|
|
65591
|
+
const disconnected = new Promise((resolve) => child.once("disconnect", () => resolve()));
|
|
65592
|
+
this.children.add(child);
|
|
65593
|
+
if (child.pid !== undefined)
|
|
65594
|
+
this.port.send({ kind: "spawned", pid: child.pid });
|
|
65595
|
+
const worker = randomUUID17();
|
|
65596
|
+
this.incarnations.set(worker, child);
|
|
65597
|
+
const slot = { worker, child, deliveries: new Map };
|
|
65598
|
+
let ready = false;
|
|
65599
|
+
let serial = Promise.resolve();
|
|
65600
|
+
const logPath = join29(this.logDirectory, `${worker}.log`);
|
|
65601
|
+
this.logs.set(child, logPath);
|
|
65602
|
+
let logFailure;
|
|
65603
|
+
const log = createWriteStream(logPath, { flags: "a" });
|
|
65604
|
+
log.on("error", (error) => {
|
|
65605
|
+
logFailure = error.message;
|
|
65606
|
+
});
|
|
65607
|
+
let stderr = "";
|
|
65608
|
+
child.stderr?.on("data", (chunk) => {
|
|
65609
|
+
stderr = (stderr + chunk.toString()).slice(-STDERR_TAIL_CHARACTERS);
|
|
65610
|
+
});
|
|
65611
|
+
child.stderr?.on("error", (error) => {
|
|
65612
|
+
logFailure ??= error.message;
|
|
65613
|
+
log.end();
|
|
65614
|
+
});
|
|
65615
|
+
child.stderr?.pipe(log);
|
|
65616
|
+
const diagnosed = (summary) => new Error(`${summary} (worker stderr log: ${logFailure === undefined ? logPath : `unavailable, ${logFailure}`})${stderr ? `: ${stderr}` : ""}`);
|
|
65617
|
+
const startupTimeoutMs = options.startupTimeoutMs;
|
|
65618
|
+
let timedOut = false;
|
|
65619
|
+
let terminated = false;
|
|
65620
|
+
const terminate = () => {
|
|
65621
|
+
if (terminated)
|
|
65622
|
+
return;
|
|
65623
|
+
terminated = true;
|
|
65624
|
+
child.kill("SIGKILL");
|
|
65625
|
+
};
|
|
65626
|
+
const replied = (error) => {
|
|
65627
|
+
if (error)
|
|
65628
|
+
terminate();
|
|
65629
|
+
};
|
|
65630
|
+
return new Promise((resolve, reject) => {
|
|
65631
|
+
const startup = setTimeout(() => {
|
|
65632
|
+
timedOut = true;
|
|
65633
|
+
terminate();
|
|
65634
|
+
}, startupTimeoutMs);
|
|
65635
|
+
child.on("error", (error) => {
|
|
65636
|
+
clearTimeout(startup);
|
|
65637
|
+
reject(error);
|
|
65638
|
+
terminate();
|
|
65639
|
+
});
|
|
65640
|
+
const closed = new Promise((resolve) => child.once("close", () => resolve()));
|
|
65641
|
+
child.once("exit", () => {
|
|
65642
|
+
Promise.race([closed, delay2(STDERR_DRAIN_GRACE_MS)]).then(() => reject(diagnosed(timedOut ? `Local worker startup timed out after ${startupTimeoutMs} ms` : "Local worker exited before readiness")));
|
|
65643
|
+
this.children.delete(child);
|
|
65644
|
+
this.logs.delete(child);
|
|
65645
|
+
if (child.pid !== undefined)
|
|
65646
|
+
this.port.send({ kind: "exited", pid: child.pid });
|
|
65647
|
+
clearTimeout(startup);
|
|
65648
|
+
this.incarnations.delete(worker);
|
|
65649
|
+
this.readyWorkers.delete(worker);
|
|
65650
|
+
this.controlAcks.delete(worker);
|
|
65651
|
+
this.lastHeard.delete(worker);
|
|
65652
|
+
if (slot.work)
|
|
65653
|
+
this.fence(slot.work);
|
|
65654
|
+
for (const delivery of slot.deliveries.values())
|
|
65655
|
+
delivery.reject(new Error("Worker exited before delivery acknowledgement"));
|
|
65656
|
+
slot.deliveries.clear();
|
|
65657
|
+
const cleanup = disconnected.then(() => serial).then(async () => {
|
|
65658
|
+
try {
|
|
65659
|
+
if (await this.admission?.releaseStoppedSessions(worker))
|
|
65660
|
+
throw new Error("Local worker process cleanup unconfirmed; permits quarantined");
|
|
65661
|
+
} catch (error) {
|
|
65662
|
+
this.cleanupFailure = error instanceof Error ? error : new Error(String(error));
|
|
65663
|
+
throw this.cleanupFailure;
|
|
65664
|
+
}
|
|
65665
|
+
}).then(async () => {
|
|
65666
|
+
if (this.closed || !ready || this.desiredControl === "quit")
|
|
65667
|
+
return;
|
|
65668
|
+
let work = slot.work;
|
|
65669
|
+
if (work && await this.sendResult(work.id)) {
|
|
65670
|
+
work = undefined;
|
|
65671
|
+
slot.work = undefined;
|
|
65672
|
+
this.replacements = 0;
|
|
65673
|
+
}
|
|
65674
|
+
if (this.closed)
|
|
65675
|
+
return;
|
|
65676
|
+
if (work)
|
|
65677
|
+
await this.queue.releaseExited(this.lease, work);
|
|
65678
|
+
const exhausted = work ? work.generation >= 3 : ++this.replacements > 3;
|
|
65679
|
+
if (exhausted)
|
|
65680
|
+
this.fail(diagnosed("Local worker retry limit"));
|
|
65681
|
+
else
|
|
65682
|
+
await this.spawn();
|
|
65683
|
+
}).catch((error) => {
|
|
65684
|
+
this.fail(error);
|
|
65685
|
+
});
|
|
65686
|
+
this.cleanups.add(cleanup);
|
|
65687
|
+
cleanup.finally(() => this.cleanups.delete(cleanup));
|
|
65688
|
+
});
|
|
65689
|
+
child.on("message", (message) => {
|
|
65690
|
+
this.observeResponsiveness();
|
|
65691
|
+
this.lastHeard.set(worker, { at: performance.now(), stalled: this.stalledMs });
|
|
65692
|
+
if (message.kind === "delivery-ack") {
|
|
65693
|
+
const pending = slot.deliveries.get(message.deliveryId);
|
|
65694
|
+
slot.deliveries.delete(message.deliveryId);
|
|
65695
|
+
if (message.error)
|
|
65696
|
+
pending?.reject(new Error(message.error));
|
|
65697
|
+
else
|
|
65698
|
+
pending?.resolve();
|
|
65699
|
+
return;
|
|
65700
|
+
}
|
|
65701
|
+
let routeReady;
|
|
65702
|
+
const handled = serial.then(async () => {
|
|
65703
|
+
if (message.kind === "ready") {
|
|
65704
|
+
if (terminated)
|
|
65705
|
+
return;
|
|
65706
|
+
ready = true;
|
|
65707
|
+
this.readyWorkers.add(worker);
|
|
65708
|
+
clearTimeout(startup);
|
|
65709
|
+
resolve();
|
|
65710
|
+
return;
|
|
65711
|
+
}
|
|
65712
|
+
await this.starting;
|
|
65713
|
+
const queue = this.queue;
|
|
65714
|
+
const lease = this.lease;
|
|
65715
|
+
if (message.kind === "control")
|
|
65716
|
+
return queue.control(lease);
|
|
65717
|
+
if (message.kind === "controlled") {
|
|
65718
|
+
const control = await queue.control(lease);
|
|
65719
|
+
if (message.revision > control.revision)
|
|
65720
|
+
throw new Error("Unknown control revision");
|
|
65721
|
+
this.controlAcks.set(worker, Math.max(this.controlAcks.get(worker) ?? -1, message.revision));
|
|
65722
|
+
return;
|
|
65723
|
+
}
|
|
65724
|
+
if (message.kind === "claim") {
|
|
65725
|
+
if (slot.work)
|
|
65726
|
+
throw new Error("Worker already holds an attempt");
|
|
65727
|
+
if (this.budgetBlocked)
|
|
65728
|
+
return;
|
|
65729
|
+
await this.recoverExpired();
|
|
65730
|
+
slot.work = await queue.claim(lease, worker, this.sources.map((source) => ({
|
|
65731
|
+
definition: `${source.path}#${source.exportName}`,
|
|
65732
|
+
digest: source.digest
|
|
65733
|
+
})), options.leaseMs);
|
|
65734
|
+
return slot.work;
|
|
65735
|
+
}
|
|
65736
|
+
let work = slot.work;
|
|
65737
|
+
if (!work)
|
|
65738
|
+
throw new Error("Worker has no active attempt");
|
|
65739
|
+
if (message.kind === "renew" || work.expiresAt - Date.now() < options.leaseMs / 2) {
|
|
65740
|
+
work = await queue.extend(lease, work, options.leaseMs);
|
|
65741
|
+
slot.work = work;
|
|
65742
|
+
}
|
|
65743
|
+
if (message.kind === "activity")
|
|
65744
|
+
return this.publish(work, message.runs);
|
|
65745
|
+
if (message.kind === "route") {
|
|
65746
|
+
if (this.routes === undefined)
|
|
65747
|
+
return;
|
|
65748
|
+
await this.flush();
|
|
65749
|
+
this.assertPublished(work);
|
|
65750
|
+
routeReady = this.routes.route(work, message.runId);
|
|
65751
|
+
return;
|
|
65752
|
+
}
|
|
65753
|
+
if (message.kind === "delivery-ready") {
|
|
65754
|
+
if (!this.publishing)
|
|
65755
|
+
return;
|
|
65756
|
+
await this.flush();
|
|
65757
|
+
this.assertPublished(work);
|
|
65758
|
+
return this.request({
|
|
65759
|
+
kind: "ready",
|
|
65760
|
+
work: token(work),
|
|
65761
|
+
runId: message.runId,
|
|
65762
|
+
stageId: message.stageId,
|
|
65763
|
+
stageName: message.stageName,
|
|
65764
|
+
...message.sessionId === undefined ? {} : { sessionId: message.sessionId }
|
|
65765
|
+
});
|
|
65766
|
+
}
|
|
65767
|
+
if (message.kind === "delivery") {
|
|
65768
|
+
if (!message.recipient)
|
|
65769
|
+
throw new Error("Worker delivery requires a session identity");
|
|
65770
|
+
if (!this.publishing)
|
|
65771
|
+
return;
|
|
65772
|
+
await this.flush();
|
|
65773
|
+
this.assertPublished(work);
|
|
65774
|
+
const id = ++this.requestSequence;
|
|
65775
|
+
this.deliveryRequests.set(id, { slot, workerRequest: message.id });
|
|
65776
|
+
try {
|
|
65777
|
+
return await new Promise((resolve, reject) => {
|
|
65778
|
+
this.requests.set(id, { resolve, reject });
|
|
65779
|
+
this.port.send({
|
|
65780
|
+
kind: "request",
|
|
65781
|
+
id,
|
|
65782
|
+
request: {
|
|
65783
|
+
kind: "deliver",
|
|
65784
|
+
work: token(work),
|
|
65785
|
+
runId: message.runId,
|
|
65786
|
+
stageId: message.stageId,
|
|
65787
|
+
stageName: message.stageName,
|
|
65788
|
+
recipient: message.recipient
|
|
65789
|
+
}
|
|
65790
|
+
});
|
|
65791
|
+
});
|
|
65792
|
+
} finally {
|
|
65793
|
+
this.deliveryRequests.delete(id);
|
|
65794
|
+
}
|
|
65795
|
+
}
|
|
65796
|
+
if (message.kind === "renew")
|
|
65797
|
+
return;
|
|
65798
|
+
if (message.kind === "commit") {
|
|
65799
|
+
const receipt = await queue.commit(lease, work, message.operation, message.mutation);
|
|
65800
|
+
this.noteUsage(await this.usageLedger.total(root));
|
|
65801
|
+
return receipt;
|
|
65802
|
+
}
|
|
65803
|
+
if (message.kind === "usage") {
|
|
65804
|
+
this.noteUsage(await this.usageLedger.report(lease, work, message.reports));
|
|
65805
|
+
return;
|
|
65806
|
+
}
|
|
65807
|
+
if (message.kind === "budget") {
|
|
65808
|
+
await queue.assertCurrent(lease, work);
|
|
65809
|
+
if (this.budgetBlocked)
|
|
65810
|
+
throw new Error("Root workflow budget exhausted");
|
|
65811
|
+
return queue.control(lease);
|
|
65812
|
+
}
|
|
65813
|
+
if (message.kind !== "complete")
|
|
65814
|
+
throw new Error("Invalid worker operation");
|
|
65815
|
+
await queue.complete(lease, work, message.result);
|
|
65816
|
+
this.replacements = 0;
|
|
65817
|
+
slot.work = undefined;
|
|
65818
|
+
await this.sendResult(work.id);
|
|
65819
|
+
});
|
|
65820
|
+
const response = handled.then((value2) => routeReady ?? value2);
|
|
65821
|
+
if (message.kind !== "delivery" && message.kind !== "delivery-ready")
|
|
65822
|
+
serial = handled.then(() => {}, () => {});
|
|
65823
|
+
response.then((value2) => child.send({ id: message.id, value: value2 }, replied), (error) => child.send({ id: message.id, error: error.message }, replied));
|
|
65824
|
+
});
|
|
65825
|
+
});
|
|
65826
|
+
}
|
|
65827
|
+
assertPublished(work) {
|
|
65828
|
+
const failure = this.publicationFailures.get(work.id);
|
|
65829
|
+
if (failure?.generation === work.generation)
|
|
65830
|
+
throw new Error(failure.error);
|
|
65831
|
+
}
|
|
65832
|
+
recoverExpired() {
|
|
65833
|
+
if (this.recovering)
|
|
65834
|
+
return this.recovering;
|
|
65835
|
+
const root = this.root;
|
|
65836
|
+
if (this.closed || root === undefined || !this.queue || performance.now() < this.nextRecoveryAt)
|
|
65837
|
+
return Promise.resolve();
|
|
65838
|
+
const queue = this.queue;
|
|
65839
|
+
const lease = this.lease;
|
|
65840
|
+
const leaseMs = this.options.leaseMs;
|
|
65841
|
+
this.recovering = (async () => {
|
|
65842
|
+
await new Promise((resolve) => setImmediate(resolve));
|
|
65843
|
+
await this.admission?.reconcile();
|
|
65844
|
+
const expiredBefore = Date.now();
|
|
65845
|
+
for (const work of await queue.inspect(root)) {
|
|
65846
|
+
if (work.state !== "leased" || work.expiresAt >= expiredBefore)
|
|
65847
|
+
continue;
|
|
65848
|
+
const child = work.worker === null ? undefined : this.incarnations.get(work.worker);
|
|
65849
|
+
if (child && work.worker !== null && this.isLive(work.worker, child)) {
|
|
65850
|
+
await queue.extend(lease, work, leaseMs).catch((error) => {
|
|
65851
|
+
if (error.message !== STALE_WORK_LEASE)
|
|
65852
|
+
throw error;
|
|
65853
|
+
});
|
|
65854
|
+
continue;
|
|
65855
|
+
}
|
|
65856
|
+
if (child && !await this.stop(child).then(() => true, () => false))
|
|
65857
|
+
continue;
|
|
65858
|
+
await queue.expire(lease, work);
|
|
65859
|
+
}
|
|
65860
|
+
})().finally(() => {
|
|
65861
|
+
this.recovering = undefined;
|
|
65862
|
+
this.nextRecoveryAt = performance.now() + RECOVERY_INTERVAL_MS;
|
|
65863
|
+
});
|
|
65864
|
+
return this.recovering;
|
|
65865
|
+
}
|
|
65866
|
+
observeResponsiveness() {
|
|
65867
|
+
const now = performance.now();
|
|
65868
|
+
const gap = now - this.lastTick;
|
|
65869
|
+
if (gap > STALL_THRESHOLD_MS)
|
|
65870
|
+
this.stalledMs += gap - TICK_INTERVAL_MS;
|
|
65871
|
+
this.lastTick = now;
|
|
65872
|
+
}
|
|
65873
|
+
isLive(worker, child) {
|
|
65874
|
+
this.observeResponsiveness();
|
|
65875
|
+
const heard = this.lastHeard.get(worker);
|
|
65876
|
+
return child.exitCode === null && child.signalCode === null && child.connected && heard !== undefined && performance.now() - heard.at - (this.stalledMs - heard.stalled) < this.options.leaseMs;
|
|
65877
|
+
}
|
|
65878
|
+
stop(child) {
|
|
65879
|
+
if (child.pid === undefined || child.exitCode !== null || child.signalCode !== null)
|
|
65880
|
+
return Promise.resolve();
|
|
65881
|
+
return new Promise((resolve, reject) => {
|
|
65882
|
+
const timeout = setTimeout(() => reject(new Error("Local worker cleanup was not confirmed")), 5000);
|
|
65883
|
+
child.once("exit", () => {
|
|
65884
|
+
clearTimeout(timeout);
|
|
65885
|
+
resolve();
|
|
65886
|
+
});
|
|
65887
|
+
child.kill("SIGKILL");
|
|
65888
|
+
});
|
|
65889
|
+
}
|
|
65890
|
+
async close() {
|
|
65891
|
+
this.closed = true;
|
|
65892
|
+
clearInterval(this.ticker);
|
|
65893
|
+
clearInterval(this.recoveryTimer);
|
|
65894
|
+
let failure;
|
|
65895
|
+
let stopping = [];
|
|
65896
|
+
try {
|
|
65897
|
+
await this.recovering?.catch(() => {});
|
|
65898
|
+
stopping = [...this.children].map((child) => ({ child, log: this.logs.get(child) }));
|
|
65899
|
+
await Promise.all(stopping.map(({ child }) => this.stop(child)));
|
|
65900
|
+
await Promise.all([...this.cleanups]);
|
|
65901
|
+
if (this.cleanupFailure)
|
|
65902
|
+
throw this.cleanupFailure;
|
|
65903
|
+
} catch (error) {
|
|
65904
|
+
failure = error;
|
|
65905
|
+
}
|
|
65906
|
+
if (failure === undefined) {
|
|
65907
|
+
const stopped = stopping.flatMap(({ child, log }) => log !== undefined && (child.exitCode !== null || child.signalCode !== null) ? [log] : []);
|
|
65908
|
+
await Promise.all(stopped.map((log) => rm3(log, { force: true }).catch(() => {})));
|
|
65909
|
+
if (this.logDirectory !== undefined)
|
|
65910
|
+
await rmdir(this.logDirectory).catch(() => {});
|
|
65911
|
+
}
|
|
65912
|
+
this.flush();
|
|
65913
|
+
try {
|
|
65914
|
+
await this.routes?.close();
|
|
65915
|
+
} catch (error) {
|
|
65916
|
+
failure ??= error;
|
|
65917
|
+
}
|
|
65918
|
+
for (const batch of this.batches.values())
|
|
65919
|
+
batch.settle();
|
|
65920
|
+
this.batches.clear();
|
|
65921
|
+
for (const pending of this.requests.values())
|
|
65922
|
+
pending.reject(new Error("Local worker pool is closed"));
|
|
65923
|
+
this.requests.clear();
|
|
65924
|
+
if (failure !== undefined)
|
|
65925
|
+
throw failure;
|
|
65926
|
+
}
|
|
65927
|
+
}
|
|
65928
|
+
var CONTROL_ACKNOWLEDGEMENT_TIMEOUT_MS = 5000, RECOVERY_INTERVAL_MS = 500, TICK_INTERVAL_MS = 100, STALL_THRESHOLD_MS = 500, COORDINATOR_LEASE_MS, STDERR_TAIL_CHARACTERS = 8192, STDERR_DRAIN_GRACE_MS = 1000, ROUTE_REGISTRATION_WAIT_MS = 120000, PENDING_STAGE_MESSAGE_EVENT2 = "atomic:workflow-pending-stage-message", STICKY_LIVE_DELIVERED_EVENT2 = "atomic:workflow-sticky-live-delivered", PENDING_STAGE_UNDELIVERABLE_EVENT2 = "atomic:workflow-pending-stage-undeliverable";
|
|
65929
|
+
var init_pool_coordinator = __esm(() => {
|
|
65930
|
+
init_resolution();
|
|
65931
|
+
init_sql_pool_queue();
|
|
65932
|
+
init_sql_pool_usage();
|
|
65933
|
+
init_pending_stage_intercom();
|
|
65934
|
+
init_intercom_group();
|
|
65935
|
+
init_pending_stage_route_capability();
|
|
65936
|
+
init_pending_stage_route_readiness();
|
|
65937
|
+
init_store();
|
|
65938
|
+
init_resource_admission();
|
|
65939
|
+
COORDINATOR_LEASE_MS = 3 * STAGE_SESSION_HEARTBEAT_INTERVAL_MS;
|
|
64777
65940
|
});
|
|
64778
65941
|
|
|
64779
65942
|
// dist/builtin/workflows/src/workers/protocol.ts
|
|
@@ -65023,21 +66186,26 @@ class PoolStageHandoff {
|
|
|
65023
66186
|
store;
|
|
65024
66187
|
queue;
|
|
65025
66188
|
lease;
|
|
66189
|
+
claimRun;
|
|
65026
66190
|
attempts = new Map;
|
|
65027
66191
|
owners = new Map;
|
|
65028
66192
|
publications = new Map;
|
|
65029
66193
|
drains = new Map;
|
|
65030
|
-
constructor(backend, store, queue, lease) {
|
|
66194
|
+
constructor(backend, store, queue, lease, claimRun) {
|
|
65031
66195
|
this.backend = backend;
|
|
65032
66196
|
this.store = store;
|
|
65033
66197
|
this.queue = queue;
|
|
65034
66198
|
this.lease = lease;
|
|
66199
|
+
this.claimRun = claimRun;
|
|
65035
66200
|
}
|
|
65036
66201
|
publish(work, runs) {
|
|
65037
66202
|
const previous = this.publications.get(work.id) ?? Promise.resolve();
|
|
65038
66203
|
const publication = previous.catch(() => {
|
|
65039
66204
|
return;
|
|
65040
|
-
}).then(() =>
|
|
66205
|
+
}).then(async () => {
|
|
66206
|
+
const commit = await this.prepare(work, runs);
|
|
66207
|
+
this.store.batch(commit);
|
|
66208
|
+
});
|
|
65041
66209
|
const settled = publication.catch(() => {
|
|
65042
66210
|
return;
|
|
65043
66211
|
});
|
|
@@ -65048,6 +66216,21 @@ class PoolStageHandoff {
|
|
|
65048
66216
|
});
|
|
65049
66217
|
return publication;
|
|
65050
66218
|
}
|
|
66219
|
+
async publishBatch(publications) {
|
|
66220
|
+
const prepared = [];
|
|
66221
|
+
for (const { work, runs } of publications)
|
|
66222
|
+
prepared.push(await this.prepare(work, runs).catch((error) => error));
|
|
66223
|
+
return this.store.batch(() => prepared.map((commit) => {
|
|
66224
|
+
if (commit instanceof Error)
|
|
66225
|
+
return commit;
|
|
66226
|
+
try {
|
|
66227
|
+
commit();
|
|
66228
|
+
return;
|
|
66229
|
+
} catch (error) {
|
|
66230
|
+
return error instanceof Error ? error : new Error(String(error));
|
|
66231
|
+
}
|
|
66232
|
+
}));
|
|
66233
|
+
}
|
|
65051
66234
|
async route(work, runId) {
|
|
65052
66235
|
this.requireOwned(await this.begin(work), runId);
|
|
65053
66236
|
const readiness = workflowPendingStageRouteReady(this.store, runId);
|
|
@@ -65098,7 +66281,7 @@ class PoolStageHandoff {
|
|
|
65098
66281
|
const result = await this.queue.result(root, work.id);
|
|
65099
66282
|
if (result === undefined)
|
|
65100
66283
|
return;
|
|
65101
|
-
await this.backend.
|
|
66284
|
+
await this.backend.catchUpWorkflow(root);
|
|
65102
66285
|
const terminals = new Map;
|
|
65103
66286
|
for (const checkpoint of this.backend.listCheckpoints(root)) {
|
|
65104
66287
|
const boundary = checkpoint.kind === "stage" ? checkpoint.topology?.boundary : undefined;
|
|
@@ -65108,6 +66291,9 @@ class PoolStageHandoff {
|
|
|
65108
66291
|
if (child?.runId === boundary.child.runId)
|
|
65109
66292
|
terminals.set(child.runId, { status: child.status, outputs: child.outputs });
|
|
65110
66293
|
}
|
|
66294
|
+
this.store.batch(() => this.settle(work, result, terminals));
|
|
66295
|
+
}
|
|
66296
|
+
settle(work, result, terminals) {
|
|
65111
66297
|
for (const run of this.store.runs()) {
|
|
65112
66298
|
if (run.endedAt !== undefined || this.owners.get(run.id) !== work.id)
|
|
65113
66299
|
continue;
|
|
@@ -65131,22 +66317,31 @@ class PoolStageHandoff {
|
|
|
65131
66317
|
if (current !== undefined && current.generation > work.generation)
|
|
65132
66318
|
return;
|
|
65133
66319
|
const attempt = current?.generation === work.generation ? current : this.start(work, current);
|
|
65134
|
-
this.release(attempt);
|
|
66320
|
+
this.store.batch(() => this.release(attempt));
|
|
65135
66321
|
attempt.fenced = true;
|
|
65136
66322
|
}
|
|
65137
|
-
async
|
|
66323
|
+
async prepare(work, runs) {
|
|
65138
66324
|
const subtree = this.subtree(work, runs);
|
|
65139
66325
|
const root = this.lease.runId;
|
|
65140
|
-
const
|
|
65141
|
-
|
|
65142
|
-
|
|
65143
|
-
|
|
65144
|
-
|
|
65145
|
-
|
|
65146
|
-
|
|
65147
|
-
|
|
65148
|
-
|
|
66326
|
+
const owned = (known) => {
|
|
66327
|
+
for (const run of subtree) {
|
|
66328
|
+
const existing = known.get(run.id);
|
|
66329
|
+
const owner = this.owners.get(run.id);
|
|
66330
|
+
if (owner !== undefined && owner !== work.id || owner === undefined && existing !== undefined && (existing.parentRunId !== run.parentRunId || existing.parentStageId !== run.parentStageId))
|
|
66331
|
+
throw outside2(run.id);
|
|
66332
|
+
}
|
|
66333
|
+
};
|
|
66334
|
+
const before = new Map(this.store.runs().map((run) => [run.id, run]));
|
|
66335
|
+
owned(before);
|
|
66336
|
+
if (subtree.some((run) => before.get(run.id) === undefined))
|
|
66337
|
+
await this.backend.catchUpWorkflow(root);
|
|
65149
66338
|
await this.queue.assertCurrent(this.lease, work);
|
|
66339
|
+
return () => this.commit(work, subtree, owned);
|
|
66340
|
+
}
|
|
66341
|
+
commit(work, subtree, owned) {
|
|
66342
|
+
const root = this.lease.runId;
|
|
66343
|
+
const known = new Map(this.store.runs().map((run) => [run.id, run]));
|
|
66344
|
+
owned(known);
|
|
65150
66345
|
const attempt = this.enter(work);
|
|
65151
66346
|
const execution = {
|
|
65152
66347
|
worker: work.worker ?? "",
|
|
@@ -65158,11 +66353,13 @@ class PoolStageHandoff {
|
|
|
65158
66353
|
...run,
|
|
65159
66354
|
rootRunId: root,
|
|
65160
66355
|
pendingStageMessages: [
|
|
65161
|
-
...
|
|
66356
|
+
...known.get(run.id)?.pendingStageMessages ?? pendingStageMessagesForDurableRun(this.backend, run.id, root)
|
|
65162
66357
|
],
|
|
65163
66358
|
stages: run.stages.map((stage) => ({ ...stage, workerExecution: execution }))
|
|
65164
66359
|
}));
|
|
65165
66360
|
const changed = published.some((run) => activity(known.get(run.id)) !== activity(run));
|
|
66361
|
+
for (const run of published)
|
|
66362
|
+
this.claimRun?.(run.id);
|
|
65166
66363
|
for (const run of published) {
|
|
65167
66364
|
attempt.runs.add(run.id);
|
|
65168
66365
|
this.owners.set(run.id, work.id);
|
|
@@ -65187,7 +66384,7 @@ class PoolStageHandoff {
|
|
|
65187
66384
|
const byId = new Map;
|
|
65188
66385
|
for (const run of runs) {
|
|
65189
66386
|
if (run.id === root || byId.has(run.id))
|
|
65190
|
-
throw
|
|
66387
|
+
throw outside2(run.id);
|
|
65191
66388
|
byId.set(run.id, run);
|
|
65192
66389
|
}
|
|
65193
66390
|
if (runs.length === 0)
|
|
@@ -65200,25 +66397,25 @@ class PoolStageHandoff {
|
|
|
65200
66397
|
const top = byId.get(work.childRunId);
|
|
65201
66398
|
if (top !== undefined) {
|
|
65202
66399
|
if (top.parentRunId !== root || top.parentStageId !== work.boundaryId)
|
|
65203
|
-
throw
|
|
66400
|
+
throw outside2(top.id);
|
|
65204
66401
|
const pending = [top];
|
|
65205
66402
|
for (let run = pending.shift();run !== undefined; run = pending.shift()) {
|
|
65206
66403
|
if (run.rootRunId !== undefined && run.rootRunId !== root)
|
|
65207
|
-
throw
|
|
66404
|
+
throw outside2(run.id);
|
|
65208
66405
|
ordered.push(run);
|
|
65209
66406
|
for (const child of runs) {
|
|
65210
66407
|
if (child.parentRunId !== run.id)
|
|
65211
66408
|
continue;
|
|
65212
66409
|
const parentStage = run.stages.find((stage) => stage.id === child.parentStageId);
|
|
65213
66410
|
if (authoritativeWorkflowChildRunId(parentStage) !== child.id)
|
|
65214
|
-
throw
|
|
66411
|
+
throw outside2(child.id);
|
|
65215
66412
|
pending.push(child);
|
|
65216
66413
|
}
|
|
65217
66414
|
}
|
|
65218
66415
|
}
|
|
65219
66416
|
const stray = runs.find((run) => !ordered.includes(run));
|
|
65220
66417
|
if (stray !== undefined)
|
|
65221
|
-
throw
|
|
66418
|
+
throw outside2(stray.id);
|
|
65222
66419
|
return ordered;
|
|
65223
66420
|
}
|
|
65224
66421
|
async begin(work) {
|
|
@@ -65273,7 +66470,7 @@ class PoolStageHandoff {
|
|
|
65273
66470
|
}
|
|
65274
66471
|
requireOwned(attempt, runId) {
|
|
65275
66472
|
if (!attempt.runs.has(runId))
|
|
65276
|
-
throw
|
|
66473
|
+
throw outside2(runId);
|
|
65277
66474
|
}
|
|
65278
66475
|
delivery(attempt, work, runId, stageId, stageName) {
|
|
65279
66476
|
const key = JSON.stringify([runId, stageId]);
|
|
@@ -65291,7 +66488,7 @@ class PoolStageHandoff {
|
|
|
65291
66488
|
return delivery;
|
|
65292
66489
|
}
|
|
65293
66490
|
}
|
|
65294
|
-
function
|
|
66491
|
+
function outside2(runId) {
|
|
65295
66492
|
return new Error(`Pool stage publication is outside the work subtree: ${runId}`);
|
|
65296
66493
|
}
|
|
65297
66494
|
function isLive(stage) {
|
|
@@ -65316,33 +66513,122 @@ var init_stage_handoff = __esm(() => {
|
|
|
65316
66513
|
});
|
|
65317
66514
|
|
|
65318
66515
|
// dist/builtin/workflows/src/workers/local-pool.ts
|
|
65319
|
-
import { fork } from "node:child_process";
|
|
65320
|
-
import {
|
|
65321
|
-
import {
|
|
65322
|
-
import { mkdir as mkdir6, rm as rm3, rmdir } from "node:fs/promises";
|
|
65323
|
-
import { dirname as dirname14, join as join29, resolve as resolve13 } from "node:path";
|
|
65324
|
-
import { setTimeout as delay2 } from "node:timers/promises";
|
|
66516
|
+
import { fork as fork2 } from "node:child_process";
|
|
66517
|
+
import { resolve as resolve14 } from "node:path";
|
|
66518
|
+
import { setTimeout as delay3 } from "node:timers/promises";
|
|
65325
66519
|
import { fileURLToPath as fileURLToPath6 } from "node:url";
|
|
65326
66520
|
import { isBunBinary as isBunBinary2 } from "@bastani/atomic";
|
|
65327
|
-
function
|
|
65328
|
-
return
|
|
66521
|
+
function defaultWorkerEntry() {
|
|
66522
|
+
return isBunBinary2 ? INTERNAL_WORKFLOW_WORKER_ARG : fileURLToPath6(new URL("./local-worker-entry.mjs", import.meta.url));
|
|
66523
|
+
}
|
|
66524
|
+
function launchCoordinatorProcess() {
|
|
66525
|
+
const child = fork2(defaultWorkerEntry(), [JSON.stringify({ role: POOL_COORDINATOR_ROLE })], {
|
|
66526
|
+
stdio: ["ignore", "ignore", "pipe", "ipc"],
|
|
66527
|
+
execArgv: []
|
|
66528
|
+
});
|
|
66529
|
+
let stderr = "";
|
|
66530
|
+
child.stderr?.on("data", (chunk) => {
|
|
66531
|
+
stderr = (stderr + chunk.toString()).slice(-COORDINATOR_STDERR_TAIL_CHARACTERS);
|
|
66532
|
+
});
|
|
66533
|
+
const listeners = [];
|
|
66534
|
+
let lostReason;
|
|
66535
|
+
const lose = (reason) => {
|
|
66536
|
+
if (lostReason !== undefined)
|
|
66537
|
+
return;
|
|
66538
|
+
lostReason = reason;
|
|
66539
|
+
for (const listener of listeners)
|
|
66540
|
+
listener(reason);
|
|
66541
|
+
};
|
|
66542
|
+
const exited = new Promise((resolve) => child.once("exit", () => resolve()));
|
|
66543
|
+
child.on("error", lose);
|
|
66544
|
+
child.once("exit", (code, signal) => lose(new Error(`Local worker pool coordinator exited (${signal ?? `code ${code}`})${stderr ? `: ${stderr.trim()}` : ""}`)));
|
|
66545
|
+
return {
|
|
66546
|
+
pid: child.pid,
|
|
66547
|
+
port: {
|
|
66548
|
+
send(message) {
|
|
66549
|
+
if (lostReason !== undefined || !child.connected)
|
|
66550
|
+
return;
|
|
66551
|
+
child.send(message, (error) => {
|
|
66552
|
+
if (error)
|
|
66553
|
+
lose(error);
|
|
66554
|
+
});
|
|
66555
|
+
},
|
|
66556
|
+
onMessage(listener) {
|
|
66557
|
+
child.on("message", (message) => listener(message));
|
|
66558
|
+
},
|
|
66559
|
+
onClose(listener) {
|
|
66560
|
+
if (lostReason !== undefined)
|
|
66561
|
+
listener(lostReason);
|
|
66562
|
+
else
|
|
66563
|
+
listeners.push(listener);
|
|
66564
|
+
},
|
|
66565
|
+
async close() {
|
|
66566
|
+
if (child.connected)
|
|
66567
|
+
child.disconnect();
|
|
66568
|
+
if (child.exitCode !== null || child.signalCode !== null)
|
|
66569
|
+
return;
|
|
66570
|
+
const timedOut = await Promise.race([exited.then(() => false), delay3(COORDINATOR_EXIT_GRACE_MS, true)]);
|
|
66571
|
+
if (timedOut) {
|
|
66572
|
+
child.kill("SIGKILL");
|
|
66573
|
+
await exited;
|
|
66574
|
+
}
|
|
66575
|
+
}
|
|
66576
|
+
}
|
|
66577
|
+
};
|
|
66578
|
+
}
|
|
66579
|
+
function launchCoordinatorInProcess() {
|
|
66580
|
+
const mainListeners = [];
|
|
66581
|
+
const coordinatorListeners = [];
|
|
66582
|
+
const copy = (message) => JSON.parse(JSON.stringify(message));
|
|
66583
|
+
const coordinator = new PoolCoordinator({
|
|
66584
|
+
send(message) {
|
|
66585
|
+
const copied = copy(message);
|
|
66586
|
+
setImmediate(() => {
|
|
66587
|
+
for (const listener of mainListeners)
|
|
66588
|
+
listener(copied);
|
|
66589
|
+
});
|
|
66590
|
+
},
|
|
66591
|
+
onMessage(listener) {
|
|
66592
|
+
coordinatorListeners.push(listener);
|
|
66593
|
+
},
|
|
66594
|
+
onClose() {},
|
|
66595
|
+
async close() {}
|
|
66596
|
+
});
|
|
66597
|
+
return {
|
|
66598
|
+
pid: process.pid,
|
|
66599
|
+
port: {
|
|
66600
|
+
send(message) {
|
|
66601
|
+
const copied = copy(message);
|
|
66602
|
+
setImmediate(() => {
|
|
66603
|
+
for (const listener of coordinatorListeners)
|
|
66604
|
+
listener(copied);
|
|
66605
|
+
});
|
|
66606
|
+
},
|
|
66607
|
+
onMessage(listener) {
|
|
66608
|
+
mainListeners.push(listener);
|
|
66609
|
+
},
|
|
66610
|
+
onClose() {},
|
|
66611
|
+
async close() {
|
|
66612
|
+
coordinator.shutdown();
|
|
66613
|
+
}
|
|
66614
|
+
}
|
|
66615
|
+
};
|
|
65329
66616
|
}
|
|
65330
66617
|
|
|
65331
66618
|
class LocalWorkerPool {
|
|
65332
66619
|
backend;
|
|
65333
66620
|
options;
|
|
65334
|
-
|
|
65335
|
-
|
|
65336
|
-
|
|
65337
|
-
|
|
65338
|
-
|
|
65339
|
-
|
|
65340
|
-
|
|
65341
|
-
|
|
65342
|
-
|
|
65343
|
-
|
|
65344
|
-
|
|
65345
|
-
rootStalledMs = 0;
|
|
66621
|
+
pids = new Set;
|
|
66622
|
+
coordinator;
|
|
66623
|
+
coordinatorLost;
|
|
66624
|
+
commands = new Map;
|
|
66625
|
+
commandSequence = 0;
|
|
66626
|
+
inbound = Promise.resolve();
|
|
66627
|
+
works = new Map;
|
|
66628
|
+
attempts = new Map;
|
|
66629
|
+
results = new Map;
|
|
66630
|
+
changed = Promise.withResolvers();
|
|
66631
|
+
waiting = 0;
|
|
65346
66632
|
ticker;
|
|
65347
66633
|
leaseMs;
|
|
65348
66634
|
desiredControl;
|
|
@@ -65350,18 +66636,17 @@ class LocalWorkerPool {
|
|
|
65350
66636
|
store;
|
|
65351
66637
|
handoff;
|
|
65352
66638
|
releaseRoutes;
|
|
66639
|
+
routeClaims = new Map;
|
|
65353
66640
|
sources = [];
|
|
65354
66641
|
queue;
|
|
65355
66642
|
lease;
|
|
65356
|
-
admission;
|
|
65357
|
-
usageLedger;
|
|
65358
66643
|
usageTotal;
|
|
65359
66644
|
rootBudget;
|
|
66645
|
+
budgetBlocked = false;
|
|
66646
|
+
rootSignature;
|
|
65360
66647
|
starting;
|
|
65361
|
-
cleanupFailure;
|
|
65362
66648
|
failure;
|
|
65363
66649
|
closed = false;
|
|
65364
|
-
replacements = 0;
|
|
65365
66650
|
releaseBinding;
|
|
65366
66651
|
constructor(backend, options) {
|
|
65367
66652
|
this.backend = backend;
|
|
@@ -65376,10 +66661,12 @@ class LocalWorkerPool {
|
|
|
65376
66661
|
this.releaseBinding = bindChildWorkerDispatcher(backend, this, options.rootId);
|
|
65377
66662
|
if (options.admission && options.rootId === undefined)
|
|
65378
66663
|
workflowResourceAdmission.set(backend, options.admission);
|
|
65379
|
-
this.ticker = setInterval(() => this.observeResponsiveness(), ROOT_TICK_INTERVAL_MS).unref();
|
|
65380
66664
|
}
|
|
65381
66665
|
get workerPids() {
|
|
65382
|
-
return [...this.
|
|
66666
|
+
return [...this.pids];
|
|
66667
|
+
}
|
|
66668
|
+
get coordinatorPid() {
|
|
66669
|
+
return this.coordinator?.pid;
|
|
65383
66670
|
}
|
|
65384
66671
|
usage() {
|
|
65385
66672
|
return this.usageTotal;
|
|
@@ -65387,47 +66674,39 @@ class LocalWorkerPool {
|
|
|
65387
66674
|
async prepare(root, store) {
|
|
65388
66675
|
if (this.closed)
|
|
65389
66676
|
throw new Error("Local worker pool is closed");
|
|
65390
|
-
if (!(this.backend.journal instanceof SqlRunJournal) ||
|
|
66677
|
+
if (!(this.backend.journal instanceof SqlRunJournal) || resolve14(this.options.journalPath) !== resolve14(this.backend.journal.path) || sqlPoolStorage(this.backend.journal)?.tables.runs !== "wf_runs")
|
|
65391
66678
|
throw new Error("Local workers require the same default-prefix SQLite journal");
|
|
65392
66679
|
if (this.root !== undefined && this.root !== root)
|
|
65393
66680
|
throw new Error("Local worker pool belongs to another root");
|
|
65394
66681
|
this.root = root;
|
|
65395
66682
|
if (store)
|
|
65396
66683
|
this.store = store;
|
|
65397
|
-
this.usageLedger ??= await openSqlPoolUsage(this.backend.journal);
|
|
65398
|
-
this.usageTotal = await this.usageLedger.total(root);
|
|
65399
66684
|
const state = await this.backend.journal.leaseState(root);
|
|
65400
66685
|
if (!state || state.holderId !== this.backend.holderId)
|
|
65401
66686
|
throw new Error("Local worker pool requires the root owner");
|
|
65402
66687
|
this.lease = { runId: root, holderId: state.holderId, epoch: state.epoch };
|
|
65403
|
-
const
|
|
65404
|
-
await
|
|
65405
|
-
|
|
65406
|
-
|
|
65407
|
-
|
|
65408
|
-
|
|
65409
|
-
|
|
65410
|
-
|
|
65411
|
-
|
|
65412
|
-
|
|
66688
|
+
const workerEntry = this.options.workerEntry ?? (isBunBinary2 || this.options.coordinator === "in-process" ? defaultWorkerEntry() : undefined);
|
|
66689
|
+
this.usageTotal = await this.command({
|
|
66690
|
+
kind: "prepare",
|
|
66691
|
+
root,
|
|
66692
|
+
lease: this.lease,
|
|
66693
|
+
options: {
|
|
66694
|
+
journalPath: this.options.journalPath,
|
|
66695
|
+
workers: this.options.workers,
|
|
66696
|
+
leaseMs: this.leaseMs,
|
|
66697
|
+
startupTimeoutMs: this.options.startupTimeoutMs ?? DEFAULT_STARTUP_TIMEOUT_MS,
|
|
66698
|
+
...this.options.admission === undefined ? {} : { admission: this.options.admission },
|
|
66699
|
+
...workerEntry === undefined ? {} : { workerEntry }
|
|
66700
|
+
},
|
|
66701
|
+
...this.desiredControl === undefined ? {} : { control: this.desiredControl }
|
|
66702
|
+
});
|
|
66703
|
+
this.queue ??= await openSqlPoolQueue(this.backend.journal);
|
|
65413
66704
|
}
|
|
65414
66705
|
async control(state) {
|
|
65415
66706
|
this.desiredControl = state;
|
|
65416
66707
|
try {
|
|
65417
|
-
if (
|
|
65418
|
-
|
|
65419
|
-
const control = await this.queue.setControl(this.lease, state);
|
|
65420
|
-
const deadline = Date.now() + CONTROL_ACKNOWLEDGEMENT_TIMEOUT_MS;
|
|
65421
|
-
while ([...this.readyWorkers].some((worker) => (this.controlAcks.get(worker) ?? -1) < control.revision)) {
|
|
65422
|
-
if (this.desiredControl !== state)
|
|
65423
|
-
return;
|
|
65424
|
-
if (Date.now() >= deadline) {
|
|
65425
|
-
if (state === "quit")
|
|
65426
|
-
break;
|
|
65427
|
-
throw new Error("Worker control acknowledgement unresolved");
|
|
65428
|
-
}
|
|
65429
|
-
await delay2(25);
|
|
65430
|
-
}
|
|
66708
|
+
if (this.coordinator !== undefined)
|
|
66709
|
+
await this.command({ kind: "control", state });
|
|
65431
66710
|
} finally {
|
|
65432
66711
|
if (state === "quit")
|
|
65433
66712
|
await this.close();
|
|
@@ -65473,290 +66752,61 @@ class LocalWorkerPool {
|
|
|
65473
66752
|
throw new Error(`Worker definition is not importable: ${binding.exportName}`);
|
|
65474
66753
|
return source;
|
|
65475
66754
|
}));
|
|
65476
|
-
|
|
65477
|
-
|
|
66755
|
+
const cwd = String(this.backend.getWorkflow(root)?.invocationCwd ?? process.cwd());
|
|
66756
|
+
const store = this.store;
|
|
66757
|
+
if (store) {
|
|
66758
|
+
this.releaseRoutes = await acquireStandaloneIntercom(store, root, cwd);
|
|
65478
66759
|
if (this.closed) {
|
|
65479
66760
|
await this.releaseRoutes();
|
|
65480
66761
|
this.releaseRoutes = undefined;
|
|
65481
66762
|
throw new Error("Local worker pool is closed");
|
|
65482
66763
|
}
|
|
65483
|
-
|
|
65484
|
-
|
|
65485
|
-
|
|
65486
|
-
this.
|
|
65487
|
-
|
|
65488
|
-
root,
|
|
65489
|
-
holder: "coordinator"
|
|
66764
|
+
const routeOwner = {
|
|
66765
|
+
notifyUndeliverable: async (payload) => !this.closed && await this.command({ kind: "undeliverable", payload }) === true
|
|
66766
|
+
};
|
|
66767
|
+
this.handoff = new PoolStageHandoff(this.backend, store, this.queueView(), this.lease, (runId) => {
|
|
66768
|
+
this.routeClaims.set(runId, claimExternalRunRoute(store, runId, routeOwner));
|
|
65490
66769
|
});
|
|
65491
|
-
this.logDirectory = workerLogDirectory(this.options.journalPath, root);
|
|
65492
|
-
await mkdir6(this.logDirectory, { recursive: true }).catch(() => {
|
|
65493
|
-
return;
|
|
65494
|
-
});
|
|
65495
|
-
const workers = Array.from({ length: this.options.workers }, () => this.spawn(root));
|
|
65496
|
-
try {
|
|
65497
|
-
await Promise.any(workers);
|
|
65498
|
-
} catch (error) {
|
|
65499
|
-
throw error.errors[0];
|
|
65500
66770
|
}
|
|
65501
|
-
|
|
65502
|
-
|
|
65503
|
-
|
|
65504
|
-
|
|
65505
|
-
|
|
65506
|
-
|
|
65507
|
-
journalPath: this.options.journalPath,
|
|
65508
|
-
root,
|
|
65509
|
-
definitions: this.sources,
|
|
65510
|
-
admission: this.options.admission
|
|
65511
|
-
})
|
|
65512
|
-
], { stdio: ["ignore", "ignore", "pipe", "ipc"], execArgv: [] });
|
|
65513
|
-
const disconnected = new Promise((resolve) => child.once("disconnect", () => resolve()));
|
|
65514
|
-
this.children.add(child);
|
|
65515
|
-
const worker = randomUUID17();
|
|
65516
|
-
this.incarnations.set(worker, child);
|
|
65517
|
-
let ready = false;
|
|
65518
|
-
let serial = Promise.resolve();
|
|
65519
|
-
let work;
|
|
65520
|
-
const deliveries = new Map;
|
|
65521
|
-
const logPath = join29(this.logDirectory, `${worker}.log`);
|
|
65522
|
-
this.logs.set(child, logPath);
|
|
65523
|
-
let logFailure;
|
|
65524
|
-
const log = createWriteStream(logPath, { flags: "a" });
|
|
65525
|
-
log.on("error", (error) => {
|
|
65526
|
-
logFailure = error.message;
|
|
65527
|
-
});
|
|
65528
|
-
let stderr = "";
|
|
65529
|
-
child.stderr?.on("data", (chunk) => {
|
|
65530
|
-
stderr = (stderr + chunk.toString()).slice(-STDERR_TAIL_CHARACTERS);
|
|
65531
|
-
});
|
|
65532
|
-
child.stderr?.on("error", (error) => {
|
|
65533
|
-
logFailure ??= error.message;
|
|
65534
|
-
log.end();
|
|
65535
|
-
});
|
|
65536
|
-
child.stderr?.pipe(log);
|
|
65537
|
-
const diagnosed = (summary) => new Error(`${summary} (worker stderr log: ${logFailure === undefined ? logPath : `unavailable, ${logFailure}`})${stderr ? `: ${stderr}` : ""}`);
|
|
65538
|
-
const startupTimeoutMs = this.options.startupTimeoutMs ?? DEFAULT_STARTUP_TIMEOUT_MS;
|
|
65539
|
-
let timedOut = false;
|
|
65540
|
-
let terminated = false;
|
|
65541
|
-
const terminate = () => {
|
|
65542
|
-
if (terminated)
|
|
65543
|
-
return;
|
|
65544
|
-
terminated = true;
|
|
65545
|
-
child.kill("SIGKILL");
|
|
65546
|
-
};
|
|
65547
|
-
const replied = (error) => {
|
|
65548
|
-
if (error)
|
|
65549
|
-
terminate();
|
|
65550
|
-
};
|
|
65551
|
-
return new Promise((resolve, reject) => {
|
|
65552
|
-
const startup = setTimeout(() => {
|
|
65553
|
-
timedOut = true;
|
|
65554
|
-
terminate();
|
|
65555
|
-
}, startupTimeoutMs);
|
|
65556
|
-
child.on("error", (error) => {
|
|
65557
|
-
clearTimeout(startup);
|
|
65558
|
-
reject(error);
|
|
65559
|
-
terminate();
|
|
65560
|
-
});
|
|
65561
|
-
const closed = new Promise((resolve) => child.once("close", () => resolve()));
|
|
65562
|
-
child.once("exit", () => {
|
|
65563
|
-
Promise.race([closed, delay2(STDERR_DRAIN_GRACE_MS)]).then(() => reject(diagnosed(timedOut ? `Local worker startup timed out after ${startupTimeoutMs} ms` : "Local worker exited before readiness")));
|
|
65564
|
-
this.children.delete(child);
|
|
65565
|
-
this.logs.delete(child);
|
|
65566
|
-
clearTimeout(startup);
|
|
65567
|
-
this.incarnations.delete(worker);
|
|
65568
|
-
this.readyWorkers.delete(worker);
|
|
65569
|
-
this.controlAcks.delete(worker);
|
|
65570
|
-
this.lastHeard.delete(worker);
|
|
65571
|
-
if (work)
|
|
65572
|
-
this.handoff?.fence(work);
|
|
65573
|
-
for (const delivery of deliveries.values())
|
|
65574
|
-
delivery.reject(new Error("Worker exited before delivery acknowledgement"));
|
|
65575
|
-
deliveries.clear();
|
|
65576
|
-
const cleanup = disconnected.then(() => serial).then(async () => {
|
|
65577
|
-
try {
|
|
65578
|
-
if (await this.admission?.releaseStoppedSessions(worker))
|
|
65579
|
-
throw new Error("Local worker process cleanup unconfirmed; permits quarantined");
|
|
65580
|
-
} catch (error) {
|
|
65581
|
-
this.cleanupFailure = error instanceof Error ? error : new Error(String(error));
|
|
65582
|
-
throw this.cleanupFailure;
|
|
65583
|
-
}
|
|
65584
|
-
}).then(async () => {
|
|
65585
|
-
if (this.closed || !ready || this.desiredControl === "quit")
|
|
65586
|
-
return;
|
|
65587
|
-
if (work && await this.queue.result(root, work.id)) {
|
|
65588
|
-
work = undefined;
|
|
65589
|
-
this.replacements = 0;
|
|
65590
|
-
}
|
|
65591
|
-
if (this.closed)
|
|
65592
|
-
return;
|
|
65593
|
-
if (work)
|
|
65594
|
-
await this.queue.releaseExited(this.lease, work);
|
|
65595
|
-
const exhausted = work ? work.generation >= 3 : ++this.replacements > 3;
|
|
65596
|
-
if (exhausted)
|
|
65597
|
-
this.failure = diagnosed("Local worker retry limit");
|
|
65598
|
-
else
|
|
65599
|
-
await this.spawn(root);
|
|
65600
|
-
}).catch((error) => {
|
|
65601
|
-
this.failure = error;
|
|
65602
|
-
});
|
|
65603
|
-
this.cleanups.add(cleanup);
|
|
65604
|
-
cleanup.finally(() => this.cleanups.delete(cleanup));
|
|
65605
|
-
});
|
|
65606
|
-
child.on("message", (message) => {
|
|
65607
|
-
this.observeResponsiveness();
|
|
65608
|
-
this.lastHeard.set(worker, { at: performance.now(), rootStalled: this.rootStalledMs });
|
|
65609
|
-
if (message.kind === "delivery-ack") {
|
|
65610
|
-
const pending = deliveries.get(message.deliveryId);
|
|
65611
|
-
deliveries.delete(message.deliveryId);
|
|
65612
|
-
if (message.error)
|
|
65613
|
-
pending?.reject(new Error(message.error));
|
|
65614
|
-
else
|
|
65615
|
-
pending?.resolve();
|
|
65616
|
-
return;
|
|
65617
|
-
}
|
|
65618
|
-
const response = serial.then(async () => {
|
|
65619
|
-
if (message.kind === "ready") {
|
|
65620
|
-
if (terminated)
|
|
65621
|
-
return;
|
|
65622
|
-
ready = true;
|
|
65623
|
-
this.readyWorkers.add(worker);
|
|
65624
|
-
clearTimeout(startup);
|
|
65625
|
-
resolve();
|
|
65626
|
-
return;
|
|
65627
|
-
}
|
|
65628
|
-
await this.starting;
|
|
65629
|
-
const queue = this.queue;
|
|
65630
|
-
const lease = this.lease;
|
|
65631
|
-
if (message.kind === "control")
|
|
65632
|
-
return queue.control(lease);
|
|
65633
|
-
if (message.kind === "controlled") {
|
|
65634
|
-
const control = await queue.control(lease);
|
|
65635
|
-
if (message.revision > control.revision)
|
|
65636
|
-
throw new Error("Unknown control revision");
|
|
65637
|
-
this.controlAcks.set(worker, Math.max(this.controlAcks.get(worker) ?? -1, message.revision));
|
|
65638
|
-
return;
|
|
65639
|
-
}
|
|
65640
|
-
if (message.kind === "claim") {
|
|
65641
|
-
if (work)
|
|
65642
|
-
throw new Error("Worker already holds an attempt");
|
|
65643
|
-
const decision = this.rootBudget?.checkpoint();
|
|
65644
|
-
if (decision?.kind === "exhausted" || decision?.kind === "wrap_up")
|
|
65645
|
-
return;
|
|
65646
|
-
await this.recoverExpired(root);
|
|
65647
|
-
work = await queue.claim(lease, worker, this.sources.map((source) => ({
|
|
65648
|
-
definition: `${source.path}#${source.exportName}`,
|
|
65649
|
-
digest: source.digest
|
|
65650
|
-
})), this.leaseMs);
|
|
65651
|
-
return work;
|
|
65652
|
-
}
|
|
65653
|
-
if (!work)
|
|
65654
|
-
throw new Error("Worker has no active attempt");
|
|
65655
|
-
if (message.kind === "renew" || work.expiresAt - Date.now() < this.leaseMs / 2)
|
|
65656
|
-
work = await queue.extend(lease, work, this.leaseMs);
|
|
65657
|
-
if (message.kind === "activity")
|
|
65658
|
-
return this.handoff?.publish(work, message.runs);
|
|
65659
|
-
if (message.kind === "route")
|
|
65660
|
-
return this.handoff?.route(work, message.runId);
|
|
65661
|
-
if (message.kind === "delivery-ready")
|
|
65662
|
-
return this.handoff?.ready(work, message.runId, message.stageId, message.stageName, message.sessionId);
|
|
65663
|
-
if (message.kind === "delivery") {
|
|
65664
|
-
if (!message.recipient)
|
|
65665
|
-
throw new Error("Worker delivery requires a session identity");
|
|
65666
|
-
return this.handoff?.deliver(work, message.runId, message.stageId, message.stageName, message.recipient, (from, payload) => new Promise((resolve, reject) => {
|
|
65667
|
-
const id = randomUUID17();
|
|
65668
|
-
deliveries.set(id, { resolve, reject });
|
|
65669
|
-
child.send({ id: message.id, delivery: { id, from, message: payload } }, (error) => {
|
|
65670
|
-
if (error) {
|
|
65671
|
-
deliveries.delete(id);
|
|
65672
|
-
reject(error);
|
|
65673
|
-
}
|
|
65674
|
-
});
|
|
65675
|
-
}));
|
|
65676
|
-
}
|
|
65677
|
-
if (message.kind === "renew")
|
|
65678
|
-
return;
|
|
65679
|
-
if (message.kind === "commit") {
|
|
65680
|
-
const receipt = await queue.commit(lease, work, message.operation, message.mutation);
|
|
65681
|
-
this.usageTotal = await this.usageLedger.total(root);
|
|
65682
|
-
return receipt;
|
|
65683
|
-
}
|
|
65684
|
-
if (message.kind === "usage") {
|
|
65685
|
-
this.usageTotal = await this.usageLedger.report(lease, work, message.reports);
|
|
65686
|
-
return;
|
|
65687
|
-
}
|
|
65688
|
-
if (message.kind === "budget") {
|
|
65689
|
-
await queue.assertCurrent(lease, work);
|
|
65690
|
-
const decision = this.rootBudget?.checkpoint();
|
|
65691
|
-
if (decision?.kind === "exhausted" || decision?.kind === "wrap_up")
|
|
65692
|
-
throw new Error("Root workflow budget exhausted");
|
|
65693
|
-
return queue.control(lease);
|
|
65694
|
-
}
|
|
65695
|
-
if (message.kind !== "complete")
|
|
65696
|
-
throw new Error("Invalid worker operation");
|
|
65697
|
-
await queue.complete(lease, work, message.result);
|
|
65698
|
-
this.replacements = 0;
|
|
65699
|
-
work = undefined;
|
|
65700
|
-
});
|
|
65701
|
-
if (message.kind !== "delivery" && message.kind !== "delivery-ready")
|
|
65702
|
-
serial = response.then(() => {}, () => {});
|
|
65703
|
-
response.then((value2) => child.send({ id: message.id, value: value2 }, replied), (error) => child.send({ id: message.id, error: error.message }, replied));
|
|
65704
|
-
});
|
|
66771
|
+
await this.command({
|
|
66772
|
+
kind: "start",
|
|
66773
|
+
sources: this.sources,
|
|
66774
|
+
cwd,
|
|
66775
|
+
publish: store !== undefined,
|
|
66776
|
+
...this.rootRun()
|
|
65705
66777
|
});
|
|
65706
66778
|
}
|
|
65707
|
-
|
|
65708
|
-
|
|
65709
|
-
|
|
65710
|
-
|
|
65711
|
-
|
|
65712
|
-
|
|
65713
|
-
|
|
65714
|
-
|
|
65715
|
-
|
|
65716
|
-
|
|
65717
|
-
if (work.state !== "leased" || work.expiresAt >= expiredBefore)
|
|
65718
|
-
continue;
|
|
65719
|
-
const child = work.worker === null ? undefined : this.incarnations.get(work.worker);
|
|
65720
|
-
if (child && work.worker !== null && this.isLive(work.worker, child)) {
|
|
65721
|
-
await this.queue.extend(this.lease, work, this.leaseMs).catch((error) => {
|
|
65722
|
-
if (error.message !== STALE_WORK_LEASE)
|
|
65723
|
-
throw error;
|
|
65724
|
-
});
|
|
65725
|
-
continue;
|
|
65726
|
-
}
|
|
65727
|
-
if (child && !await this.stop(child).then(() => true, () => false))
|
|
65728
|
-
continue;
|
|
65729
|
-
await this.queue.expire(this.lease, work);
|
|
65730
|
-
}
|
|
65731
|
-
})().finally(() => {
|
|
65732
|
-
this.recovering = undefined;
|
|
65733
|
-
this.nextRecoveryAt = performance.now() + RECOVERY_INTERVAL_MS;
|
|
65734
|
-
});
|
|
65735
|
-
return this.recovering;
|
|
66779
|
+
queueView() {
|
|
66780
|
+
return {
|
|
66781
|
+
assertCurrent: async (_lease, work) => {
|
|
66782
|
+
const current = this.attempts.get(work.id);
|
|
66783
|
+
if (current === undefined || current.fenced || current.generation !== work.generation || current.worker !== work.worker || current.epoch !== work.epoch)
|
|
66784
|
+
throw new Error(STALE_WORK_LEASE);
|
|
66785
|
+
},
|
|
66786
|
+
withCurrentAppend: (lease, work, assertSession, action) => this.queue.withCurrentAppend(lease, work, assertSession, action),
|
|
66787
|
+
result: async (root, id) => this.results.get(id) ?? this.queue.result(root, id)
|
|
66788
|
+
};
|
|
65736
66789
|
}
|
|
65737
|
-
|
|
65738
|
-
const
|
|
65739
|
-
|
|
65740
|
-
if (gap > ROOT_STALL_THRESHOLD_MS)
|
|
65741
|
-
this.rootStalledMs += gap - ROOT_TICK_INTERVAL_MS;
|
|
65742
|
-
this.lastTick = now;
|
|
66790
|
+
rootProjection() {
|
|
66791
|
+
const run = this.store?.runs().find((candidate) => candidate.id === this.root);
|
|
66792
|
+
return run === undefined ? undefined : projectRootRun(run);
|
|
65743
66793
|
}
|
|
65744
|
-
|
|
65745
|
-
this.
|
|
65746
|
-
|
|
65747
|
-
|
|
66794
|
+
rootRun() {
|
|
66795
|
+
const rootRun = this.rootProjection();
|
|
66796
|
+
if (rootRun === undefined)
|
|
66797
|
+
return {};
|
|
66798
|
+
this.rootSignature = JSON.stringify(rootRun);
|
|
66799
|
+
return { rootRun };
|
|
65748
66800
|
}
|
|
65749
|
-
|
|
65750
|
-
|
|
65751
|
-
|
|
65752
|
-
|
|
65753
|
-
|
|
65754
|
-
|
|
65755
|
-
|
|
65756
|
-
|
|
65757
|
-
|
|
65758
|
-
child.kill("SIGKILL");
|
|
65759
|
-
});
|
|
66801
|
+
refreshRootRun() {
|
|
66802
|
+
const rootRun = this.rootProjection();
|
|
66803
|
+
if (rootRun === undefined)
|
|
66804
|
+
return;
|
|
66805
|
+
const signature = JSON.stringify(rootRun);
|
|
66806
|
+
if (signature === this.rootSignature)
|
|
66807
|
+
return;
|
|
66808
|
+
this.rootSignature = signature;
|
|
66809
|
+
this.notice({ kind: "root", rootRun });
|
|
65760
66810
|
}
|
|
65761
66811
|
async run(definition, inputs, options) {
|
|
65762
66812
|
try {
|
|
@@ -65789,7 +66839,7 @@ class LocalWorkerPool {
|
|
|
65789
66839
|
const selectedSource = this.sources[index];
|
|
65790
66840
|
if (await definitionDigest(selectedSource.path) !== selectedSource.digest)
|
|
65791
66841
|
throw new Error("Worker definition compatibility error");
|
|
65792
|
-
await this.backend.
|
|
66842
|
+
await this.backend.catchUpWorkflow(root);
|
|
65793
66843
|
const boundary = this.backend.listCheckpoints(root).find((entry) => entry.kind === "stage" && entry.topology?.boundary?.event === "start" && entry.topology.stageId === options.parentRun.stageId);
|
|
65794
66844
|
if (boundary?.kind !== "stage")
|
|
65795
66845
|
throw new Error("Missing durable child boundary");
|
|
@@ -65800,39 +66850,232 @@ class LocalWorkerPool {
|
|
|
65800
66850
|
config: options.config,
|
|
65801
66851
|
executionMode: options.executionMode
|
|
65802
66852
|
}));
|
|
65803
|
-
|
|
65804
|
-
|
|
65805
|
-
|
|
65806
|
-
|
|
65807
|
-
|
|
65808
|
-
|
|
65809
|
-
|
|
65810
|
-
|
|
65811
|
-
|
|
65812
|
-
|
|
65813
|
-
|
|
65814
|
-
|
|
66853
|
+
this.evaluateBudget();
|
|
66854
|
+
const enqueued = await this.command({
|
|
66855
|
+
kind: "enqueue",
|
|
66856
|
+
descriptor: {
|
|
66857
|
+
boundaryId: options.parentRun.stageId,
|
|
66858
|
+
childRunId: options.runId,
|
|
66859
|
+
scope: options.durableScope.scopePrefix,
|
|
66860
|
+
definition: `${selectedSource.path}#${selectedSource.exportName}`,
|
|
66861
|
+
digest: selectedSource.digest,
|
|
66862
|
+
inputs,
|
|
66863
|
+
execution
|
|
66864
|
+
},
|
|
66865
|
+
boundary,
|
|
66866
|
+
...this.rootRun()
|
|
66867
|
+
});
|
|
66868
|
+
const work = enqueued.work;
|
|
66869
|
+
this.works.set(work.id, work);
|
|
66870
|
+
if (enqueued.result !== undefined && !this.results.has(work.id))
|
|
66871
|
+
this.results.set(work.id, enqueued.result);
|
|
66872
|
+
this.waiting++;
|
|
66873
|
+
this.ticker ??= setInterval(() => {
|
|
66874
|
+
this.evaluateBudget();
|
|
66875
|
+
this.signalChange();
|
|
66876
|
+
}, POOL_SUMMARY_INTERVAL_MS).unref();
|
|
66877
|
+
try {
|
|
66878
|
+
for (;; ) {
|
|
66879
|
+
options.signal?.throwIfAborted();
|
|
66880
|
+
if (this.closed) {
|
|
66881
|
+
if (this.failure)
|
|
66882
|
+
throw this.failure;
|
|
66883
|
+
if (this.desiredControl === "quit")
|
|
66884
|
+
throw new WorkflowGracefulQuitError(root, "worker pool teardown");
|
|
66885
|
+
throw new Error("Local worker pool is closed");
|
|
66886
|
+
}
|
|
66887
|
+
const result = this.results.get(work.id);
|
|
66888
|
+
if (result) {
|
|
66889
|
+
await this.inbound;
|
|
66890
|
+
await this.backend.catchUpWorkflow(root);
|
|
66891
|
+
await this.handoff?.complete(work);
|
|
66892
|
+
return result;
|
|
66893
|
+
}
|
|
65815
66894
|
if (this.failure)
|
|
65816
66895
|
throw this.failure;
|
|
65817
|
-
if (this.
|
|
65818
|
-
|
|
65819
|
-
|
|
66896
|
+
if (this.budgetBlocked)
|
|
66897
|
+
await this.rootBudget.stopAtBoundaryAsync(definition.name);
|
|
66898
|
+
await this.nextChange(options.signal);
|
|
65820
66899
|
}
|
|
65821
|
-
|
|
65822
|
-
if (
|
|
65823
|
-
|
|
65824
|
-
|
|
65825
|
-
return result;
|
|
66900
|
+
} finally {
|
|
66901
|
+
if (--this.waiting === 0) {
|
|
66902
|
+
clearInterval(this.ticker);
|
|
66903
|
+
this.ticker = undefined;
|
|
65826
66904
|
}
|
|
65827
|
-
if (this.failure)
|
|
65828
|
-
throw this.failure;
|
|
65829
|
-
const decision = this.rootBudget?.checkpoint();
|
|
65830
|
-
if (decision?.kind === "exhausted" || decision?.kind === "wrap_up")
|
|
65831
|
-
await this.rootBudget.stopAtBoundaryAsync(definition.name);
|
|
65832
|
-
await this.recoverExpired(root);
|
|
65833
|
-
await delay2(25, undefined, { signal: options.signal });
|
|
65834
66905
|
}
|
|
65835
66906
|
}
|
|
66907
|
+
evaluateBudget() {
|
|
66908
|
+
const decision = this.rootBudget?.checkpoint();
|
|
66909
|
+
const blocked = decision?.kind === "exhausted" || decision?.kind === "wrap_up";
|
|
66910
|
+
if (blocked === this.budgetBlocked)
|
|
66911
|
+
return;
|
|
66912
|
+
this.budgetBlocked = blocked;
|
|
66913
|
+
this.notice({ kind: "budget", blocked });
|
|
66914
|
+
}
|
|
66915
|
+
signalChange() {
|
|
66916
|
+
const changed = this.changed;
|
|
66917
|
+
this.changed = Promise.withResolvers();
|
|
66918
|
+
changed.resolve();
|
|
66919
|
+
}
|
|
66920
|
+
nextChange(signal) {
|
|
66921
|
+
const changed = this.changed.promise;
|
|
66922
|
+
if (signal === undefined)
|
|
66923
|
+
return changed;
|
|
66924
|
+
return new Promise((resolve, reject) => {
|
|
66925
|
+
const abort = () => reject(signal.reason);
|
|
66926
|
+
signal.addEventListener("abort", abort, { once: true });
|
|
66927
|
+
changed.then(() => {
|
|
66928
|
+
signal.removeEventListener("abort", abort);
|
|
66929
|
+
resolve();
|
|
66930
|
+
});
|
|
66931
|
+
});
|
|
66932
|
+
}
|
|
66933
|
+
connect() {
|
|
66934
|
+
if (this.coordinator !== undefined)
|
|
66935
|
+
return this.coordinator;
|
|
66936
|
+
const coordinator = this.options.coordinator === "in-process" ? launchCoordinatorInProcess() : launchCoordinatorProcess();
|
|
66937
|
+
this.coordinator = coordinator;
|
|
66938
|
+
coordinator.port.onMessage((event) => {
|
|
66939
|
+
this.inbound = this.inbound.then(() => this.receive(event)).catch(() => {});
|
|
66940
|
+
});
|
|
66941
|
+
coordinator.port.onClose((reason) => {
|
|
66942
|
+
this.coordinatorLost = reason;
|
|
66943
|
+
this.inbound = this.inbound.then(() => this.lost(reason));
|
|
66944
|
+
});
|
|
66945
|
+
return coordinator;
|
|
66946
|
+
}
|
|
66947
|
+
lost(reason) {
|
|
66948
|
+
this.pids.clear();
|
|
66949
|
+
if (!this.closed)
|
|
66950
|
+
this.failure ??= reason;
|
|
66951
|
+
for (const pending of this.commands.values())
|
|
66952
|
+
pending.reject(reason);
|
|
66953
|
+
this.commands.clear();
|
|
66954
|
+
this.signalChange();
|
|
66955
|
+
}
|
|
66956
|
+
command(body) {
|
|
66957
|
+
const coordinator = this.connect();
|
|
66958
|
+
if (this.coordinatorLost !== undefined)
|
|
66959
|
+
return Promise.reject(this.coordinatorLost);
|
|
66960
|
+
const id = ++this.commandSequence;
|
|
66961
|
+
return new Promise((resolve, reject) => {
|
|
66962
|
+
this.commands.set(id, { resolve, reject });
|
|
66963
|
+
coordinator.port.send({ ...body, id });
|
|
66964
|
+
});
|
|
66965
|
+
}
|
|
66966
|
+
notice(notice) {
|
|
66967
|
+
this.coordinator?.port.send(notice);
|
|
66968
|
+
}
|
|
66969
|
+
async receive(event) {
|
|
66970
|
+
switch (event.kind) {
|
|
66971
|
+
case "reply": {
|
|
66972
|
+
const pending = this.commands.get(event.id);
|
|
66973
|
+
this.commands.delete(event.id);
|
|
66974
|
+
if (event.error !== undefined)
|
|
66975
|
+
pending?.reject(new Error(event.error));
|
|
66976
|
+
else
|
|
66977
|
+
pending?.resolve(event.value);
|
|
66978
|
+
return;
|
|
66979
|
+
}
|
|
66980
|
+
case "activity":
|
|
66981
|
+
return this.applyActivity(event.batch, event.entries, event.usage);
|
|
66982
|
+
case "spawned":
|
|
66983
|
+
this.pids.add(event.pid);
|
|
66984
|
+
return;
|
|
66985
|
+
case "exited":
|
|
66986
|
+
this.pids.delete(event.pid);
|
|
66987
|
+
return;
|
|
66988
|
+
case "fence": {
|
|
66989
|
+
const current = this.attempts.get(event.work.id);
|
|
66990
|
+
if (current?.generation === event.work.generation)
|
|
66991
|
+
current.fenced = true;
|
|
66992
|
+
this.handoff?.fence(event.work);
|
|
66993
|
+
return;
|
|
66994
|
+
}
|
|
66995
|
+
case "result":
|
|
66996
|
+
if (event.usage !== undefined)
|
|
66997
|
+
this.usageTotal = event.usage;
|
|
66998
|
+
this.results.set(event.workId, event.result);
|
|
66999
|
+
this.signalChange();
|
|
67000
|
+
return;
|
|
67001
|
+
case "failure":
|
|
67002
|
+
this.failure ??= new Error(event.error);
|
|
67003
|
+
this.signalChange();
|
|
67004
|
+
return;
|
|
67005
|
+
case "request":
|
|
67006
|
+
this.serve(event.id, event.request);
|
|
67007
|
+
return;
|
|
67008
|
+
}
|
|
67009
|
+
}
|
|
67010
|
+
async applyActivity(batch, entries, usage) {
|
|
67011
|
+
if (usage !== undefined)
|
|
67012
|
+
this.usageTotal = usage;
|
|
67013
|
+
const rejected = [];
|
|
67014
|
+
const handoff = this.handoff;
|
|
67015
|
+
if (handoff !== undefined) {
|
|
67016
|
+
const publications = entries.flatMap((entry) => {
|
|
67017
|
+
const work = this.track(entry.work);
|
|
67018
|
+
if (work !== undefined)
|
|
67019
|
+
return [{ work, runs: entry.runs }];
|
|
67020
|
+
rejected.push({ workId: entry.work.id, generation: entry.work.generation, error: STALE_WORK_LEASE });
|
|
67021
|
+
return [];
|
|
67022
|
+
});
|
|
67023
|
+
const outcomes = await handoff.publishBatch(publications);
|
|
67024
|
+
outcomes.forEach((error, index) => {
|
|
67025
|
+
const { work } = publications[index];
|
|
67026
|
+
if (error !== undefined)
|
|
67027
|
+
rejected.push({ workId: work.id, generation: work.generation, error: error.message });
|
|
67028
|
+
});
|
|
67029
|
+
}
|
|
67030
|
+
this.notice({ kind: "published", batch, rejected });
|
|
67031
|
+
this.refreshRootRun();
|
|
67032
|
+
}
|
|
67033
|
+
track(attempt) {
|
|
67034
|
+
const work = this.works.get(attempt.id);
|
|
67035
|
+
if (work === undefined)
|
|
67036
|
+
return;
|
|
67037
|
+
const current = this.attempts.get(attempt.id);
|
|
67038
|
+
if (current === undefined || attempt.generation > current.generation)
|
|
67039
|
+
this.attempts.set(attempt.id, { ...attempt, fenced: false });
|
|
67040
|
+
return { ...work, ...attempt, state: "leased" };
|
|
67041
|
+
}
|
|
67042
|
+
async serve(id, request) {
|
|
67043
|
+
try {
|
|
67044
|
+
const value2 = await this.answer(id, request);
|
|
67045
|
+
this.coordinator?.port.send({ kind: "request-reply", id, ...value2 === undefined ? {} : { value: value2 } });
|
|
67046
|
+
} catch (error) {
|
|
67047
|
+
this.coordinator?.port.send({
|
|
67048
|
+
kind: "request-reply",
|
|
67049
|
+
id,
|
|
67050
|
+
error: error instanceof Error ? error.message : String(error)
|
|
67051
|
+
});
|
|
67052
|
+
}
|
|
67053
|
+
}
|
|
67054
|
+
async answer(id, request) {
|
|
67055
|
+
if (request.kind === "pending-stage-message" || request.kind === "sticky-live-delivered") {
|
|
67056
|
+
const router = this.store === undefined ? undefined : pendingStageMessageRouter(this.store);
|
|
67057
|
+
const event = {
|
|
67058
|
+
...request.payload,
|
|
67059
|
+
handled: false
|
|
67060
|
+
};
|
|
67061
|
+
if (request.kind === "pending-stage-message")
|
|
67062
|
+
router?.route(event);
|
|
67063
|
+
else
|
|
67064
|
+
router?.sticky(event);
|
|
67065
|
+
if (event.handled && event.completion !== undefined)
|
|
67066
|
+
return await event.completion;
|
|
67067
|
+
return request.kind === "pending-stage-message" ? { outcome: "refused", reason: "Session not found" } : undefined;
|
|
67068
|
+
}
|
|
67069
|
+
const handoff = this.handoff;
|
|
67070
|
+
const work = this.track(request.work);
|
|
67071
|
+
if (handoff === undefined || work === undefined)
|
|
67072
|
+
throw new Error(STALE_WORK_LEASE);
|
|
67073
|
+
if (request.kind === "ready")
|
|
67074
|
+
return handoff.ready(work, request.runId, request.stageId, request.stageName, request.sessionId);
|
|
67075
|
+
return handoff.deliver(work, request.runId, request.stageId, request.stageName, request.recipient, async (from, message) => {
|
|
67076
|
+
await this.command({ kind: "transport", request: id, from, message });
|
|
67077
|
+
});
|
|
67078
|
+
}
|
|
65836
67079
|
closing;
|
|
65837
67080
|
close() {
|
|
65838
67081
|
this.closing ??= this.dispose();
|
|
@@ -65841,27 +67084,28 @@ class LocalWorkerPool {
|
|
|
65841
67084
|
async dispose() {
|
|
65842
67085
|
this.closed = true;
|
|
65843
67086
|
clearInterval(this.ticker);
|
|
67087
|
+
this.ticker = undefined;
|
|
65844
67088
|
this.releaseBinding();
|
|
65845
67089
|
if (this.options.rootId === undefined)
|
|
65846
67090
|
workflowResourceAdmission.delete(this.backend);
|
|
67091
|
+
this.signalChange();
|
|
65847
67092
|
let failure;
|
|
65848
|
-
|
|
65849
|
-
|
|
65850
|
-
|
|
65851
|
-
|
|
65852
|
-
|
|
65853
|
-
|
|
65854
|
-
|
|
65855
|
-
|
|
65856
|
-
|
|
65857
|
-
|
|
65858
|
-
|
|
65859
|
-
|
|
65860
|
-
const stopped = stopping.flatMap(({ child, log }) => log !== undefined && (child.exitCode !== null || child.signalCode !== null) ? [log] : []);
|
|
65861
|
-
await Promise.all(stopped.map((log) => rm3(log, { force: true }).catch(() => {})));
|
|
65862
|
-
if (this.logDirectory !== undefined)
|
|
65863
|
-
await rmdir(this.logDirectory).catch(() => {});
|
|
67093
|
+
const coordinator = this.coordinator;
|
|
67094
|
+
if (coordinator !== undefined) {
|
|
67095
|
+
if (this.coordinatorLost === undefined) {
|
|
67096
|
+
try {
|
|
67097
|
+
await this.command({ kind: "close" });
|
|
67098
|
+
} catch (error) {
|
|
67099
|
+
failure = error;
|
|
67100
|
+
}
|
|
67101
|
+
}
|
|
67102
|
+
await this.inbound;
|
|
67103
|
+
await coordinator.port.close();
|
|
67104
|
+
this.pids.clear();
|
|
65864
67105
|
}
|
|
67106
|
+
for (const release of this.routeClaims.values())
|
|
67107
|
+
release();
|
|
67108
|
+
this.routeClaims.clear();
|
|
65865
67109
|
try {
|
|
65866
67110
|
await this.releaseRoutes?.();
|
|
65867
67111
|
} catch (error) {
|
|
@@ -65871,27 +67115,26 @@ class LocalWorkerPool {
|
|
|
65871
67115
|
throw failure;
|
|
65872
67116
|
}
|
|
65873
67117
|
}
|
|
65874
|
-
var INTERNAL_WORKFLOW_WORKER_ARG = "--internal-workflow-worker", DEFAULT_WORK_LEASE_MS = 30000, MINIMUM_WORK_LEASE_MS,
|
|
67118
|
+
var INTERNAL_WORKFLOW_WORKER_ARG = "--internal-workflow-worker", DEFAULT_WORK_LEASE_MS = 30000, MINIMUM_WORK_LEASE_MS, DEFAULT_STARTUP_TIMEOUT_MS = 30000, MAX_TIMER_DELAY_MS = 2147483647, COORDINATOR_STDERR_TAIL_CHARACTERS = 8192, COORDINATOR_EXIT_GRACE_MS = 5000;
|
|
65875
67119
|
var init_local_pool = __esm(() => {
|
|
65876
67120
|
init_internal();
|
|
65877
67121
|
init_sql_pool_queue();
|
|
65878
|
-
init_sql_pool_usage();
|
|
65879
67122
|
init_sql_run_journal();
|
|
65880
67123
|
init_workflow_tool_abort();
|
|
65881
67124
|
init_sdk_intercom();
|
|
67125
|
+
init_external_run_routes();
|
|
65882
67126
|
init_workflow_infrastructure_error();
|
|
65883
67127
|
init_admission_binding();
|
|
65884
67128
|
init_definitions();
|
|
65885
67129
|
init_dispatch();
|
|
65886
67130
|
init_host_options();
|
|
65887
|
-
|
|
67131
|
+
init_pool_coordinator();
|
|
65888
67132
|
init_stage_handoff();
|
|
65889
67133
|
MINIMUM_WORK_LEASE_MS = 2 * WORKER_HEARTBEAT_INTERVAL_MS;
|
|
65890
|
-
COORDINATOR_LEASE_MS = 3 * STAGE_SESSION_HEARTBEAT_INTERVAL_MS;
|
|
65891
67134
|
});
|
|
65892
67135
|
|
|
65893
67136
|
// dist/builtin/workflows/src/workers/public-pool.ts
|
|
65894
|
-
import { resolve as
|
|
67137
|
+
import { resolve as resolve15 } from "node:path";
|
|
65895
67138
|
import { isDeepStrictEqual as isDeepStrictEqual2 } from "node:util";
|
|
65896
67139
|
function withWorkerSettings(options, settings) {
|
|
65897
67140
|
const configured = settings?.workflows;
|
|
@@ -65940,11 +67183,11 @@ async function activatePublicPool(backend, options) {
|
|
|
65940
67183
|
...options.localWorkers,
|
|
65941
67184
|
startupTimeoutMs: undefined,
|
|
65942
67185
|
leaseMs: undefined,
|
|
65943
|
-
definitions: options.localWorkers.definitions.map((source) => ({ ...source, path:
|
|
67186
|
+
definitions: options.localWorkers.definitions.map((source) => ({ ...source, path: resolve15(cwd, source.path) }))
|
|
65944
67187
|
};
|
|
65945
67188
|
const admission = options.admission && {
|
|
65946
67189
|
...options.admission,
|
|
65947
|
-
...options.admission.userDatabase ? { userDatabase:
|
|
67190
|
+
...options.admission.userDatabase ? { userDatabase: resolve15(cwd, options.admission.userDatabase) } : {}
|
|
65948
67191
|
};
|
|
65949
67192
|
const sources = localWorkers && await Promise.all(localWorkers.definitions.map(async (source) => ({ ...source, digest: await definitionDigest(source.path) })));
|
|
65950
67193
|
const resources = JSON.parse(JSON.stringify({
|
|
@@ -72552,9 +73795,9 @@ var init_stage_runner_controller = __esm(() => {
|
|
|
72552
73795
|
});
|
|
72553
73796
|
|
|
72554
73797
|
// dist/builtin/workflows/src/runs/foreground/stage-runner-output.ts
|
|
72555
|
-
import { createHash as
|
|
73798
|
+
import { createHash as createHash5 } from "node:crypto";
|
|
72556
73799
|
import { mkdir as mkdir7, writeFile as writeFile10 } from "node:fs/promises";
|
|
72557
|
-
import { basename as basename5, dirname as dirname15, isAbsolute as isAbsolute11, join as join30, resolve as
|
|
73800
|
+
import { basename as basename5, dirname as dirname15, isAbsolute as isAbsolute11, join as join30, resolve as resolve16 } from "node:path";
|
|
72558
73801
|
function normalizeMaxOutput2(maxOutput) {
|
|
72559
73802
|
return {
|
|
72560
73803
|
bytes: maxOutput?.bytes ?? DEFAULT_MAX_OUTPUT_BYTES,
|
|
@@ -72616,7 +73859,7 @@ function formatByteSize(bytes) {
|
|
|
72616
73859
|
return `${value2.toFixed(1)} ${units[unitIndex]}`;
|
|
72617
73860
|
}
|
|
72618
73861
|
function transcriptPath(runId, outputPath) {
|
|
72619
|
-
const digest =
|
|
73862
|
+
const digest = createHash5("sha256").update(outputPath).digest("hex").slice(0, 16);
|
|
72620
73863
|
return join30(workflowArtifactRunPath(runId), "transcripts", `${digest}-${basename5(outputPath)}.transcript.md`);
|
|
72621
73864
|
}
|
|
72622
73865
|
function appendContent(lines, content) {
|
|
@@ -72662,7 +73905,7 @@ function renderTranscript(messages, fallbackOutput) {
|
|
|
72662
73905
|
`);
|
|
72663
73906
|
}
|
|
72664
73907
|
function savedOutputReference(outputPath, fullOutput, transcriptPathValue, transcriptText, transcriptAvailable, warning) {
|
|
72665
|
-
const absolutePath =
|
|
73908
|
+
const absolutePath = resolve16(outputPath);
|
|
72666
73909
|
const bytes = Buffer.byteLength(fullOutput, "utf8");
|
|
72667
73910
|
const lines = countLines(fullOutput);
|
|
72668
73911
|
const transcriptBytes = Buffer.byteLength(transcriptText, "utf8");
|
|
@@ -72677,8 +73920,8 @@ function resolveOutputPath(output, runtimeCwd, requestedCwd) {
|
|
|
72677
73920
|
return;
|
|
72678
73921
|
if (isAbsolute11(output))
|
|
72679
73922
|
return output;
|
|
72680
|
-
const baseCwd = requestedCwd === undefined ? runtimeCwd : isAbsolute11(requestedCwd) ? requestedCwd :
|
|
72681
|
-
return
|
|
73923
|
+
const baseCwd = requestedCwd === undefined ? runtimeCwd : isAbsolute11(requestedCwd) ? requestedCwd : resolve16(runtimeCwd, requestedCwd);
|
|
73924
|
+
return resolve16(baseCwd, output);
|
|
72682
73925
|
}
|
|
72683
73926
|
function degenerateWarning(fullOutput) {
|
|
72684
73927
|
return fullOutput.trim().length === 0 ? "WARNING: the stage artifact is empty; search the companion transcript for this stage's work." : undefined;
|
|
@@ -73687,7 +74930,7 @@ async function run(def, inputs, opts = {}) {
|
|
|
73687
74930
|
if (!isWorkflowDefinition(def))
|
|
73688
74931
|
throw new Error(workflowDefinitionRequirementMessage("run(definition, inputs)", def));
|
|
73689
74932
|
const depth = opts.depth ?? 0;
|
|
73690
|
-
const maxDepth = opts.config?.maxDepth ??
|
|
74933
|
+
const maxDepth = opts.config?.maxDepth ?? 3;
|
|
73691
74934
|
if (depth >= maxDepth) {
|
|
73692
74935
|
return {
|
|
73693
74936
|
runId: opts.runId ?? crypto.randomUUID(),
|
|
@@ -75141,9 +76384,9 @@ var init_sdk_owner_release = __esm(() => {
|
|
|
75141
76384
|
|
|
75142
76385
|
// dist/builtin/workflows/src/durable/sdk-recovery-scope.ts
|
|
75143
76386
|
import { realpathSync as realpathSync5 } from "node:fs";
|
|
75144
|
-
import { resolve as
|
|
76387
|
+
import { resolve as resolve17 } from "node:path";
|
|
75145
76388
|
function canonicalDirectory(path) {
|
|
75146
|
-
const absolute =
|
|
76389
|
+
const absolute = resolve17(path);
|
|
75147
76390
|
try {
|
|
75148
76391
|
const canonical = realpathSync5.native(absolute);
|
|
75149
76392
|
return process.platform === "win32" ? canonical.toLowerCase() : canonical;
|
|
@@ -75880,9 +77123,7 @@ function runSnapshotsFromCheckpoints(checkpoints, rootRunId, rootRunName, fallba
|
|
|
75880
77123
|
}
|
|
75881
77124
|
for (const group of grouped.values())
|
|
75882
77125
|
group.sort(compareDraftSourceOrder);
|
|
75883
|
-
|
|
75884
|
-
retainReachableRunGroups(grouped, rootRunId);
|
|
75885
|
-
if (policy.failOnUnreachable && grouped.size !== groupedCount)
|
|
77126
|
+
if (!retainReachableRunGroups(grouped, rootRunId) && policy.failOnUnreachable)
|
|
75886
77127
|
return [];
|
|
75887
77128
|
if (!validateRunGroups(grouped, rootRunId, toolCheckpoints, checkpoints, policy.retainActive))
|
|
75888
77129
|
return [];
|
|
@@ -87496,7 +88737,7 @@ async function withDormantRunWrite(store, runId, write) {
|
|
|
87496
88737
|
|
|
87497
88738
|
// dist/builtin/workflows/src/extension/dormant-run-intercom.ts
|
|
87498
88739
|
var PENDING_STAGE_ROUTE_EVENT2 = "atomic:workflow-pending-stage-route";
|
|
87499
|
-
var
|
|
88740
|
+
var PENDING_STAGE_MESSAGE_EVENT3 = "atomic:workflow-pending-stage-message";
|
|
87500
88741
|
var PENDING_STAGE_ROUTE_REVOKED_EVENT = "atomic:workflow-pending-stage-route-revoked";
|
|
87501
88742
|
var DORMANT_RUN_SUBAGENT_HOLD_EVENT = "atomic:workflow-dormant-run-subagent-hold";
|
|
87502
88743
|
var DORMANT_RUN_RECHECK_MS = 15000;
|
|
@@ -87672,7 +88913,7 @@ function registerDormantRunIntercomRoutes(pi, activeStore, deps) {
|
|
|
87672
88913
|
...event.senderRegistrationName === undefined ? {} : { senderRegistrationName: event.senderRegistrationName },
|
|
87673
88914
|
...event.senderReturnAddress === undefined ? {} : { senderReturnAddress: event.senderReturnAddress }
|
|
87674
88915
|
};
|
|
87675
|
-
pi.events?.emit?.(
|
|
88916
|
+
pi.events?.emit?.(PENDING_STAGE_MESSAGE_EVENT3, forwarded);
|
|
87676
88917
|
if (forwarded.handled && forwarded.completion !== undefined)
|
|
87677
88918
|
return await forwarded.completion;
|
|
87678
88919
|
return { outcome: "refused", reason: "Session not found" };
|
|
@@ -87692,7 +88933,7 @@ function registerDormantRunIntercomRoutes(pi, activeStore, deps) {
|
|
|
87692
88933
|
}
|
|
87693
88934
|
return isLive(run.runId) ? await redispatch(event) : { outcome: "refused", reason: `Workflow run ${run.runId} is no longer resumable from this session` };
|
|
87694
88935
|
};
|
|
87695
|
-
const messages = pi.events?.on?.(
|
|
88936
|
+
const messages = pi.events?.on?.(PENDING_STAGE_MESSAGE_EVENT3, (payload) => {
|
|
87696
88937
|
if (disposed || !isDormantMessageEvent(payload) || payload.handled || payload.live === true)
|
|
87697
88938
|
return;
|
|
87698
88939
|
const target = parseWorkflowStageTarget(payload.target);
|
|
@@ -89610,8 +90851,8 @@ function mergeConfigs(base, override) {
|
|
|
89610
90851
|
};
|
|
89611
90852
|
}
|
|
89612
90853
|
var WORKFLOW_CONFIG_DEFAULTS = {
|
|
89613
|
-
maxDepth:
|
|
89614
|
-
defaultConcurrency:
|
|
90854
|
+
maxDepth: 3,
|
|
90855
|
+
defaultConcurrency: 1000,
|
|
89615
90856
|
persistRuns: true,
|
|
89616
90857
|
statusFile: false,
|
|
89617
90858
|
resumeInFlight: "never",
|
|
@@ -91378,7 +92619,7 @@ init_registry();
|
|
|
91378
92619
|
// dist/builtin/workflows/src/extension/discovery-loaders.ts
|
|
91379
92620
|
init_workflow_module_loader();
|
|
91380
92621
|
import { readdir as readdir2, stat as stat2 } from "node:fs/promises";
|
|
91381
|
-
import { extname, isAbsolute as isAbsolute13, join as join32, resolve as
|
|
92622
|
+
import { extname, isAbsolute as isAbsolute13, join as join32, resolve as resolve18 } from "node:path";
|
|
91382
92623
|
async function scanWorkflowDir(dir) {
|
|
91383
92624
|
try {
|
|
91384
92625
|
const entries = await readdir2(dir, { withFileTypes: true });
|
|
@@ -91422,7 +92663,7 @@ async function loadFromPaths(pathsOrMap, kind, baseCwd, diagnostics, getWorkflow
|
|
|
91422
92663
|
const all = [];
|
|
91423
92664
|
const entries = Array.isArray(pathsOrMap) ? pathsOrMap.map((p) => ({ rawPath: p })) : Object.entries(pathsOrMap).map(([name, p]) => ({ rawPath: p, configuredName: name }));
|
|
91424
92665
|
for (const { rawPath, configuredName } of entries) {
|
|
91425
|
-
const absPath = isAbsolute13(rawPath) ? rawPath :
|
|
92666
|
+
const absPath = isAbsolute13(rawPath) ? rawPath : resolve18(baseCwd, rawPath);
|
|
91426
92667
|
let pathStats;
|
|
91427
92668
|
try {
|
|
91428
92669
|
pathStats = await stat2(absPath);
|
|
@@ -98180,7 +99421,7 @@ var DEFAULT_PROMPT_GUIDANCE = [
|
|
|
98180
99421
|
- Bounded loops must create distinct tracked work for every iteration; never reopen an ancestor below its downstream work. Stacked implementation slices are unrolled, not looped: each slice gets a fresh child boundary and distinct tracked nodes, and no slice may add an edge back to the current frontier, itself, or an ancestor. If a retained session receives follow-up without new dependency work, keep it as non-topological activity metadata rather than adding a back-edge. Before launch, sketch nodes and dependencies for every branch and loop and reject any proposed parent edge to the node itself or an ancestor.
|
|
98181
99422
|
- Treat dynamic graph validation during execution, replay, and journal reconstruction as the authoritative cycle boundary. If runtime topology code changes, require incremental edge checks and reconstruction validation; prompt guidance and TypeScript types cannot replace those runtime checks.
|
|
98182
99423
|
- Every workflow invocation receives the stable control group \`workflow:<rootRunId>\`. Stages inherit it unless an intentional explicit group is authored. Named groups and \`group: true\` become invocation-owned subgroups under \`workflow:<rootRunId>/...\`, so repeated names cannot collide across runs. The invocation context may list and exactly send/ask live stages in its owned subgroups and queue send to known pending stages; this authority is directional and does not let subgroup siblings see or reach each other. \`intercom list\` also shows possible future targets, and the invocation context may queue sticky sends to their path patterns. Pending and pattern ask remain unsupported. A \`workflow:<rootRunId>/**\` send broadcasts to all live stages and remains sticky for future descendants until root termination. \`group: "default"\` is the explicit non-owned shared-group escape and does not receive pending invocation routing. Do not mint root group names manually.
|
|
98183
|
-
- Stages in the same invocation group are peers, not only steerable targets. A live stage can \`intercom list\` its siblings and \`send\`/\`ask\` them directly to debate a finding with evidence, hand off paths and reproductions, claim shared files or expensive steps, or learn what a sibling already verified; it can queue \`send\` to a known pending sibling and \`ask\` a completed sibling that retains a valid conversation, which reopens it for a post-mortem turn. Delegated subagents inside a stage inherit the same group. Named subgroups isolate their members from sibling subgroups.
|
|
99424
|
+
- Stages in the same invocation group are peers, not only steerable targets. A live stage can \`intercom list\` its siblings and \`send\`/\`ask\` them directly to debate a finding with evidence, hand off paths and reproductions, claim shared files or expensive steps, or learn what a sibling already verified; it can queue \`send\` to a known pending sibling and \`ask\` a completed sibling that retains a valid conversation, which reopens it for a post-mortem turn. Delegated subagents inside a stage inherit the same group. Named subgroups isolate their members from sibling subgroups. Default to peer-to-peer (swarm) exchange; the supervisor is not a message relay. There are no limits on communication between agents; it is encouraged whenever you need to. Stages proactively discover peers and broadcast, debate, claim, hand off, and ask each other directly, and each stage still returns its own structured result. Scale the swarm with the task: for very complex work or when the user asks for heavy parallelism, run hundreds or thousands of agents that self-organize into teams (named subgroups, per-stage subagent sets, child workflows) and coordinate peer-to-peer like an organization.
|
|
98184
99425
|
- Prefer \`ctx.tool(name, args, fn)\` for workflow-owned TypeScript operations with side effects, such as filesystem writes, network mutations, and external API actions. It creates a tracked, non-attachable durable graph node before invoking \`fn\`; it may be used before, between, after, or without model stages. A completed \`ctx.tool\` call is durably checkpointed with its serializable result, so resume replays that result without rerunning \`fn\`. For checks that may fail during a bounded repair loop, opt into \`failureMode: "return"\`, branch on the typed outcome, and pass only the needed error fields to the repair stage or artifact yourself; Atomic does not inject failure evidence into later prompts. Retries do not bound a hung callback, so every \`ctx.tool\` callback that starts a child process or performs network I/O MUST set an explicit positive finite \`timeoutMs\` and forward the supplied \`signal\` to cancellable work. A timeout applies per attempt, aborts that attempt's signal, and gives retries a fresh deadline and signal. Cancellation and storage faults still throw. Keep pure computation and side-effect-free transformations as ordinary TypeScript. Do not wrap agent-stage internals or every function call indiscriminately; this rule applies to side effects orchestrated directly by the workflow definition.
|
|
98185
99426
|
- Pass large stage context through files/artifacts and \`reads\` with an explicit \`Read the file at <path>...\` prompt rather than large \`previous\` payloads or injected session history.
|
|
98186
99427
|
- Wrap critical stage-prompt content in \`<keepContext>\` / \`</keepContext>\`. Tagged content survives compression verbatim regardless of the compression ratio, and every line of the span including the tag lines is protected. A long-running stage will be compacted, and compaction ranks lines individually: an objective is verbose and restated while the constraint that narrows it is usually one line, so the constraint is the cheaper deletion and what survives reads as broader permission than intended. Tag role constraints that bound a stage to part of the work, acceptance criteria and immutable contracts, explicit prohibitions, and identifiers a stage must not lose such as a target branch, worktree path, or run ID. Do not tag bulk context: protected lines count against the keep target rather than raising it, so a large protected span makes the surrounding transcript compress harder — tag the constraint, not the material it applies to.
|