@bastani/atomic 0.9.54-alpha.1 → 0.9.55-alpha.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +12 -0
- package/dist/builtin/intercom/CHANGELOG.md +14 -0
- package/dist/builtin/intercom/README.md +2 -1
- package/dist/builtin/intercom/broker/announce-on-import.ts +5 -0
- package/dist/builtin/intercom/broker/broker.ts +28 -0
- package/dist/builtin/intercom/broker/client.ts +48 -3
- package/dist/builtin/intercom/broker/paths.ts +9 -0
- package/dist/builtin/intercom/broker/socket-writes.ts +5 -0
- package/dist/builtin/intercom/broker/spawn.ts +58 -6
- package/dist/builtin/intercom/broker/start-announcement.ts +77 -0
- package/dist/builtin/intercom/broker/startup-timeout.ts +23 -0
- package/dist/builtin/intercom/index.bundle.mjs +184 -73
- package/dist/builtin/intercom/package.json +1 -1
- package/dist/builtin/intercom/recoverable-disconnect.ts +38 -2
- package/dist/builtin/mcp/package.json +1 -1
- package/dist/builtin/subagents/package.json +1 -1
- package/dist/builtin/web-access/package.json +1 -1
- package/dist/builtin/workflows/CHANGELOG.md +26 -0
- package/dist/builtin/workflows/package.json +1 -1
- package/dist/builtin/workflows/src/durable/journal/resolution.d.ts +7 -0
- package/dist/builtin/workflows/src/extension/index.bundle.mjs +8458 -7874
- package/dist/builtin/workflows/src/extension/local-worker-entry.mjs +687 -111
- package/dist/builtin/workflows/src/index.js +834 -264
- package/dist/builtin/workflows/src/local-worker-entry.mjs +687 -111
- package/dist/builtin/workflows/src/workers/public-options.d.ts +8 -0
- package/dist/core/extension-startup-failure.d.ts.map +1 -1
- package/dist/core/extension-startup-failure.js +1 -0
- package/dist/core/extension-startup-failure.js.map +1 -1
- package/dist/core/storage/session/jsonl-session-store.d.ts +27 -0
- package/dist/core/storage/session/jsonl-session-store.d.ts.map +1 -1
- package/dist/core/storage/session/jsonl-session-store.js +99 -7
- package/dist/core/storage/session/jsonl-session-store.js.map +1 -1
- package/docs/settings.md +2 -2
- package/docs/storage.md +2 -0
- package/docs/workflows/api-reference.md +3 -3
- package/docs/workflows/operations.md +3 -3
- package/npm-shrinkwrap.json +32 -32
- package/package.json +3 -3
package/CHANGELOG.md
CHANGED
|
@@ -2,6 +2,18 @@
|
|
|
2
2
|
|
|
3
3
|
## [Unreleased]
|
|
4
4
|
|
|
5
|
+
## [0.9.55] - 2026-10-30
|
|
6
|
+
|
|
7
|
+
### Fixed
|
|
8
|
+
|
|
9
|
+
- Fixed a session write failing, or the next write waiting up to ten seconds, on Windows when the session lock directory could not be removed at once because another program, such as a virus scanner or file indexer, held it. The release is now retried, and a lock Atomic still cannot remove is cleared by the next write instead of waiting to go stale
|
|
10
|
+
|
|
11
|
+
## [0.9.55-alpha.1] - 2026-10-30
|
|
12
|
+
|
|
13
|
+
### Fixed
|
|
14
|
+
|
|
15
|
+
- Fixed a session write failing, or the next write waiting up to ten seconds, on Windows when the session lock directory could not be removed at once because another program, such as a virus scanner or file indexer, held it. The release is now retried, and a lock Atomic still cannot remove is cleared by the next write instead of waiting to go stale
|
|
16
|
+
|
|
5
17
|
## [0.9.54] - 2026-10-29
|
|
6
18
|
|
|
7
19
|
Fixes from upstream pi after 1.4.0: packages, Windows paths and start-up.
|
|
@@ -4,6 +4,20 @@ All notable changes to the `pi-intercom` extension will be documented in this fi
|
|
|
4
4
|
|
|
5
5
|
## [Unreleased]
|
|
6
6
|
|
|
7
|
+
## [0.9.55] - 2026-10-30
|
|
8
|
+
|
|
9
|
+
### Fixed
|
|
10
|
+
|
|
11
|
+
- Intercom no longer fails to start on a busy machine or with an empty jiti cache because the broker was still starting. A new broker now gets 30 seconds, not 5, to answer on its socket. That is sized from measured start-up: 0.5 s on an idle machine, 2.6 s with five processes per core, and 8.7 s, 15 s and 28 s with an empty jiti cache at five, nine and seventeen. `brokerStartupTimeoutMs` in the Intercom `config.json` changes it, and the `Broker failed to start within timeout` error now names that setting. On Windows, a broker that crashes during start-up is reported as soon as it exits, with its log, instead of after that whole budget: the broker writes its own pid to `broker.starting` in the Intercom directory as its first act, and the session that spawned it watches that process. Only a broker that is alive and not yet listening is waited for the full budget.
|
|
12
|
+
- A session that connects as the broker retires, whether idle for its window, shut down or replaced, no longer fails with `Connection closed before registration` or `connect ENOENT`. A broker that is closing answers a registration with `broker_retiring` instead of accepting a session it is about to drop, including one still waiting for its multi-node presence write when the shutdown begins. A connection that ended before the broker answered, with nothing answering on its socket any more, is a recoverable disconnect. The session spawns the replacement broker and registers again for up to ten seconds, so a child session's launch and a workflow stage's warm-up no longer fail on the race.
|
|
13
|
+
|
|
14
|
+
## [0.9.55-alpha.1] - 2026-10-30
|
|
15
|
+
|
|
16
|
+
### Fixed
|
|
17
|
+
|
|
18
|
+
- Intercom no longer fails to start on a busy machine or with an empty jiti cache because the broker was still starting. A new broker now gets 30 seconds, not 5, to answer on its socket. That is sized from measured start-up: 0.5 s on an idle machine, 2.6 s with five processes per core, and 8.7 s, 15 s and 28 s with an empty jiti cache at five, nine and seventeen. `brokerStartupTimeoutMs` in the Intercom `config.json` changes it, and the `Broker failed to start within timeout` error now names that setting. On Windows, a broker that crashes during start-up is reported as soon as it exits, with its log, instead of after that whole budget: the broker writes its own pid to `broker.starting` in the Intercom directory as its first act, and the session that spawned it watches that process. Only a broker that is alive and not yet listening is waited for the full budget.
|
|
19
|
+
- A session that connects as the broker retires, whether idle for its window, shut down or replaced, no longer fails with `Connection closed before registration` or `connect ENOENT`. A broker that is closing answers a registration with `broker_retiring` instead of accepting a session it is about to drop, including one still waiting for its multi-node presence write when the shutdown begins. A connection that ended before the broker answered, with nothing answering on its socket any more, is a recoverable disconnect. The session spawns the replacement broker and registers again for up to ten seconds, so a child session's launch and a workflow stage's warm-up no longer fail on the race.
|
|
20
|
+
|
|
7
21
|
## [0.9.48] - 2026-10-27
|
|
8
22
|
|
|
9
23
|
### Fixed
|
|
@@ -425,6 +425,7 @@ The default `npx --no-install tsx` pair is a compatibility sentinel: intercom re
|
|
|
425
425
|
|---------|---------|-------------|
|
|
426
426
|
| `brokerCommand` | `"npx"` | Command used to start the local broker process; the default sentinel is hardened internally to avoid PATH lookup |
|
|
427
427
|
| `brokerArgs` | `["--no-install", "tsx"]` | Arguments passed to `brokerCommand` before the broker script path |
|
|
428
|
+
| `brokerStartupTimeoutMs` | `30000` | How long a session waits for a newly spawned broker to answer on its socket, in milliseconds (at least `1000`). A broker loads its modules through jiti, which takes about 0.5 s on an idle machine and several seconds on a busy one or with an empty jiti cache (the first start after a reboot); raise it if Intercom start-up reports `Broker failed to start within timeout` on a very slow machine |
|
|
428
429
|
| `confirmSend` | false | Show a confirmation dialog before non-reply sends from an interactive session with UI |
|
|
429
430
|
| `replyHint` | true | Include reply instruction in incoming messages |
|
|
430
431
|
| `status` | — | Optional custom status suffix shown after the automatic lifecycle status, for example `thinking · researching` |
|
|
@@ -514,7 +515,7 @@ The broker caps the file from the inside, since nothing on the parent side can b
|
|
|
514
515
|
|
|
515
516
|
**Local IPC instead of TCP.** Same-machine only by design. `pi-intercom` uses Unix sockets on macOS/Linux and a named pipe on Windows, which keeps setup simple and avoids port management.
|
|
516
517
|
|
|
517
|
-
**Auto-spawn with file lock.** The broker starts on first connection and exits after 5 seconds idle. There is no daemon to manage. A spawn lock file, keyed by PID and timestamp, prevents duplicate brokers when multiple sessions start at once. A broker that starts while another broker already answers on the socket exits instead of taking the socket over; it replaces only a stale socket file. A broker whose socket path comes to point at another broker shuts down, and its sessions reconnect to the broker that owns the path. `atomic intercom restart` asks the broker to shut down over its socket (a `shutdown` request, accepted before registration like `broker_status`), so its sessions reconnect to a broker that reads the current storage selection.
|
|
518
|
+
**Auto-spawn with file lock.** The broker starts on first connection and exits after 5 seconds idle. There is no daemon to manage. A spawn lock file, keyed by PID and timestamp, prevents duplicate brokers when multiple sessions start at once. A broker that starts while another broker already answers on the socket exits instead of taking the socket over; it replaces only a stale socket file. A broker whose socket path comes to point at another broker shuts down, and its sessions reconnect to the broker that owns the path. A session that connects just as the broker retires does not fail: a broker that is closing refuses a registration with a `broker_retiring` reply, and a connection that is reset or ended before the broker answered, with nothing left answering on the socket, means the same. Either way the session spawns the replacement broker and registers again, retrying for up to ten seconds. `atomic intercom restart` asks the broker to shut down over its socket (a `shutdown` request, accepted before registration like `broker_status`), so its sessions reconnect to a broker that reads the current storage selection.
|
|
518
519
|
|
|
519
520
|
**`ask` stays client-side.** The broker still routes plain messages; it does not have a special request/response mode for `ask`. The client waits for a matching reply before it triggers a new turn, then returns that reply as the tool result. Reply hints make that flow practical by showing the recipient the exact `send` call to use. Separately, `list` / `sessions` now carry a `requestId` so a delayed session-list reply cannot be mistaken for a newer one.
|
|
520
521
|
|
|
@@ -0,0 +1,5 @@
|
|
|
1
|
+
import { announceBrokerStart } from "./start-announcement.js";
|
|
2
|
+
|
|
3
|
+
// Imported second by broker.ts, straight after the stderr cap and before any module that does real work, so the pid is
|
|
4
|
+
// announced before the broker's module graph loads: a crash while loading is then a crash the parent can see.
|
|
5
|
+
announceBrokerStart();
|
|
@@ -1,6 +1,8 @@
|
|
|
1
1
|
// MUST stay first: ESM evaluates static dependencies before the importer's body, so this is the
|
|
2
2
|
// only position from which the stderr cap covers the other modules' own initialization.
|
|
3
3
|
import "./bounded-stderr-install.js";
|
|
4
|
+
// MUST stay second: it announces the broker's pid before the rest of the module graph loads (see start-announcement.ts).
|
|
5
|
+
import "./announce-on-import.js";
|
|
4
6
|
import net from "net";
|
|
5
7
|
import { chmodSync, writeFileSync, unlinkSync, mkdirSync, readFileSync, statSync } from "fs";
|
|
6
8
|
import { dirname } from "path";
|
|
@@ -11,6 +13,7 @@ import {
|
|
|
11
13
|
flushHeldWrites,
|
|
12
14
|
type HeldWrite,
|
|
13
15
|
holdWrites,
|
|
16
|
+
isHoldingWrites,
|
|
14
17
|
releaseHeldWrites,
|
|
15
18
|
writeMessageIfOpen,
|
|
16
19
|
writeMessageWithOutcome,
|
|
@@ -19,6 +22,7 @@ import { boundWorkflowPossibleStageRow } from "../workflow-delivery-states.js";
|
|
|
19
22
|
import { isSocketAnswering } from "./socket-liveness.js";
|
|
20
23
|
import { acquireSocketReplacementLock } from "./socket-replacement-lock.js";
|
|
21
24
|
import { startBrokerWatchdog } from "./watchdog.js";
|
|
25
|
+
import { retractBrokerStartAnnouncement } from "./start-announcement.js";
|
|
22
26
|
import {
|
|
23
27
|
getBrokerDeliveredMessagesPath,
|
|
24
28
|
getBrokerPidPath,
|
|
@@ -85,6 +89,7 @@ import { type ReceiptSender, RecipientReceipts, UNCERTAIN_RECEIPT_REASON } from
|
|
|
85
89
|
import { matchStagePathSegments } from "../workflow-stage-path-matching.js";
|
|
86
90
|
import { DELIVERED_MESSAGE_TTL_MS } from "../retry-policy.js";
|
|
87
91
|
import { isAgentRecipient } from "../recipient-purpose.js";
|
|
92
|
+
import { BROKER_RETIRING_CODE, BROKER_RETIRING_REASON } from "../recoverable-disconnect.js";
|
|
88
93
|
import {
|
|
89
94
|
LIVE_WORKFLOW_STAGE_ROUTE_REFUSAL_REASONS,
|
|
90
95
|
type LiveWorkflowStageRouteRefusalCode,
|
|
@@ -102,6 +107,12 @@ const LISTEN_ATTEMPTS = 5;
|
|
|
102
107
|
const LISTEN_RETRY_DELAY_MS = 100;
|
|
103
108
|
const SOCKET_OWNERSHIP_CHECK_MS = 1_000;
|
|
104
109
|
const MULTI_NODE_CLOSE_BOUND_MS = 2_000;
|
|
110
|
+
/** What a registration is answered with once this broker is closing: the client spawns the replacement and registers again. */
|
|
111
|
+
const RETIRING_REFUSAL = {
|
|
112
|
+
type: "registration_failed",
|
|
113
|
+
reason: BROKER_RETIRING_REASON,
|
|
114
|
+
code: BROKER_RETIRING_CODE,
|
|
115
|
+
} as const;
|
|
105
116
|
const NO_SHARED_STORAGE_REASON = "no shared storage selection (top-level `storage` in global settings, or ATOMIC_STORAGE)";
|
|
106
117
|
/** The off reasons `resolveBrokerStateSource` reports without an error, as the log and `broker_status` state them. */
|
|
107
118
|
const MULTI_NODE_OFF_REASONS: Readonly<Record<string, string>> = {
|
|
@@ -393,6 +404,8 @@ class IntercomBroker {
|
|
|
393
404
|
if (process.platform !== "win32") this.boundSocketInode = statSync(SOCKET_PATH).ino;
|
|
394
405
|
releaseReplacementLock();
|
|
395
406
|
writeFileSync(PID_PATH, String(process.pid));
|
|
407
|
+
// The pid file takes over from the start-up announcement the moment the broker answers.
|
|
408
|
+
retractBrokerStartAnnouncement();
|
|
396
409
|
console.log(`Intercom broker started (pid: ${process.pid})`);
|
|
397
410
|
startBrokerWatchdog(SOCKET_PATH, this.instanceId, getBrokerWatchdogLogPath());
|
|
398
411
|
this.lastConnectionAt = performance.now();
|
|
@@ -1627,6 +1640,13 @@ class IntercomBroker {
|
|
|
1627
1640
|
|
|
1628
1641
|
switch (clientMessage.type) {
|
|
1629
1642
|
case "register": {
|
|
1643
|
+
if (this.shuttingDown) {
|
|
1644
|
+
// A client that connected while this broker was closing (a multi-node broker takes up to two seconds to give
|
|
1645
|
+
// its leases up) learns it lost the race and spawns the replacement, instead of finding its socket reset.
|
|
1646
|
+
writeMessageIfOpen(socket, RETIRING_REFUSAL);
|
|
1647
|
+
socket.end();
|
|
1648
|
+
return;
|
|
1649
|
+
}
|
|
1630
1650
|
if (!isSessionRegistration(clientMessage.session)) {
|
|
1631
1651
|
throw new Error("Invalid register message");
|
|
1632
1652
|
}
|
|
@@ -2539,6 +2559,14 @@ class IntercomBroker {
|
|
|
2539
2559
|
console.log("Broker shutting down");
|
|
2540
2560
|
|
|
2541
2561
|
for (const session of this.sessions.values()) {
|
|
2562
|
+
// A registration still waiting for its presence write has been taken but not answered, and its frames are held
|
|
2563
|
+
// behind the `registered` it is owed. It gets the same refusal a registration arriving now gets: the held
|
|
2564
|
+
// frames go (each confirmed write reports `false`, as for a session that is gone) and the refusal is written
|
|
2565
|
+
// before the socket ends, so its client registers with the replacement instead of finding the socket closed.
|
|
2566
|
+
if (isHoldingWrites(session.socket)) {
|
|
2567
|
+
discardHeldWrites(releaseHeldWrites(session.socket));
|
|
2568
|
+
writeMessageIfOpen(session.socket, RETIRING_REFUSAL);
|
|
2569
|
+
}
|
|
2542
2570
|
session.socket.end();
|
|
2543
2571
|
}
|
|
2544
2572
|
this.sessions.clear();
|
|
@@ -24,7 +24,13 @@ import { buildSendSignature, PendingSendRegistry, PendingSendTimeoutError } from
|
|
|
24
24
|
import { readSubagentMessageSource } from "../source-ownership.js";
|
|
25
25
|
import { isMessage, isSessionInfo } from "./client-message-validation.js";
|
|
26
26
|
import { normalizeGroups } from "../group.js";
|
|
27
|
-
import {
|
|
27
|
+
import {
|
|
28
|
+
BROKER_RETIRING_CODE,
|
|
29
|
+
IntercomBrokerRetiringError,
|
|
30
|
+
IntercomBrokerTimeoutError,
|
|
31
|
+
IntercomClientDisconnectedError,
|
|
32
|
+
} from "../recoverable-disconnect.js";
|
|
33
|
+
import { isSocketAnswering } from "./socket-liveness.js";
|
|
28
34
|
import { IntercomLiveRouteRefusedError, isLiveWorkflowStageRouteRefusalCode } from "../live-route-refusal.js";
|
|
29
35
|
import { SEND_RESPONSE_TIMEOUT_MS } from "../retry-policy.js";
|
|
30
36
|
|
|
@@ -64,6 +70,38 @@ function isBacklogRefusal(error: Error): boolean {
|
|
|
64
70
|
const code = (error as NodeJS.ErrnoException).code;
|
|
65
71
|
return code === "ECONNREFUSED" || code === "EAGAIN";
|
|
66
72
|
}
|
|
73
|
+
|
|
74
|
+
/** The broker closed the connection before it answered the registration. */
|
|
75
|
+
class ConnectionClosedBeforeRegistrationError extends Error {
|
|
76
|
+
constructor() {
|
|
77
|
+
super("Connection closed before registration");
|
|
78
|
+
this.name = "ConnectionClosedBeforeRegistrationError";
|
|
79
|
+
}
|
|
80
|
+
}
|
|
81
|
+
|
|
82
|
+
/**
|
|
83
|
+
* Whether a failure before the broker answered `register` is what a broker that retired leaves behind: the socket path
|
|
84
|
+
* is gone (`ENOENT`), or the connection was reset or ended without an answer. A refusal the kernel gives a full listen
|
|
85
|
+
* backlog (`ECONNREFUSED`, `EAGAIN`) is not: those are retried above and still reject as before.
|
|
86
|
+
*/
|
|
87
|
+
function isLostBrokerSignature(error: Error): boolean {
|
|
88
|
+
if (error instanceof ConnectionClosedBeforeRegistrationError) return true;
|
|
89
|
+
const code = (error as NodeJS.ErrnoException).code;
|
|
90
|
+
return code === "ENOENT" || code === "ECONNRESET" || code === "EPIPE";
|
|
91
|
+
}
|
|
92
|
+
|
|
93
|
+
/**
|
|
94
|
+
* A failure before registration, classified: when it has the signature of a lost broker and nothing answers on the
|
|
95
|
+
* broker socket any more, the broker retired first and the failure is recoverable by spawning its replacement. A broker
|
|
96
|
+
* that still answers closed the connection for its own reasons (a refused or malformed registration), so the failure
|
|
97
|
+
* stays what it was.
|
|
98
|
+
*/
|
|
99
|
+
async function classifyRegistrationFailure(error: Error): Promise<Error> {
|
|
100
|
+
if (!isLostBrokerSignature(error) || (await isSocketAnswering(BROKER_SOCKET))) return error;
|
|
101
|
+
return new IntercomBrokerRetiringError(`Intercom broker went away before it registered this session: ${error.message}`, {
|
|
102
|
+
cause: error,
|
|
103
|
+
});
|
|
104
|
+
}
|
|
67
105
|
/** Extra wait past a broker hold, so the broker's own outcome for a held send always arrives first (#3760). */
|
|
68
106
|
const HELD_SEND_GRACE_MS = 5000;
|
|
69
107
|
|
|
@@ -409,7 +447,12 @@ export class IntercomClient extends EventEmitter {
|
|
|
409
447
|
return;
|
|
410
448
|
} catch (error) {
|
|
411
449
|
if (attempt.signal.aborted) throw connectAborted();
|
|
412
|
-
if (!(error instanceof Error)
|
|
450
|
+
if (!(error instanceof Error)) throw error;
|
|
451
|
+
if (!isBacklogRefusal(error) || retry >= BROKER_BACKLOG_RETRIES) {
|
|
452
|
+
const classified = await classifyRegistrationFailure(error);
|
|
453
|
+
if (attempt.signal.aborted) throw connectAborted();
|
|
454
|
+
throw classified;
|
|
455
|
+
}
|
|
413
456
|
const backoff = BROKER_BACKLOG_RETRY_BASE_MS * 2 ** retry;
|
|
414
457
|
await abortableDelay(backoff * (0.5 + Math.random() / 2), attempt.signal);
|
|
415
458
|
}
|
|
@@ -496,7 +539,7 @@ export class IntercomClient extends EventEmitter {
|
|
|
496
539
|
this.emit("disconnected", disconnectError);
|
|
497
540
|
}
|
|
498
541
|
if (wasConnecting) {
|
|
499
|
-
reject(new
|
|
542
|
+
reject(new ConnectionClosedBeforeRegistrationError());
|
|
500
543
|
}
|
|
501
544
|
};
|
|
502
545
|
|
|
@@ -645,6 +688,8 @@ export class IntercomClient extends EventEmitter {
|
|
|
645
688
|
// reason as an ordinary, non-recoverable error.
|
|
646
689
|
const refusal = isLiveWorkflowStageRouteRefusalCode(brokerMessage.code)
|
|
647
690
|
? new IntercomLiveRouteRefusedError(brokerMessage.code, brokerMessage.reason)
|
|
691
|
+
: brokerMessage.code === BROKER_RETIRING_CODE
|
|
692
|
+
? new IntercomBrokerRetiringError(brokerMessage.reason)
|
|
648
693
|
: new Error(brokerMessage.reason);
|
|
649
694
|
if (this._sessionId === null) {
|
|
650
695
|
// Still registering: `connect()` owns the failure and its own cleanup.
|
|
@@ -75,6 +75,15 @@ export function getBrokerLogPath(agentDir: string = getAgentDir()): string {
|
|
|
75
75
|
return join(getIntercomDirPath(agentDir), "broker.log");
|
|
76
76
|
}
|
|
77
77
|
|
|
78
|
+
/**
|
|
79
|
+
* Where a broker announces its real pid as its first act, for a parent that cannot watch the process it spawned (the
|
|
80
|
+
* Windows launcher hides it); see `start-announcement.ts`. Written once at start-up and removed once the broker listens,
|
|
81
|
+
* never after the broker ends.
|
|
82
|
+
*/
|
|
83
|
+
export function getBrokerStartAnnouncementPath(agentDir: string = getAgentDir()): string {
|
|
84
|
+
return join(getIntercomDirPath(agentDir), "broker.starting");
|
|
85
|
+
}
|
|
86
|
+
|
|
78
87
|
export function getBrokerWatchdogLogPath(agentDir: string = getAgentDir()): string {
|
|
79
88
|
return join(getIntercomDirPath(agentDir), "broker-watchdog.log");
|
|
80
89
|
}
|
|
@@ -129,6 +129,11 @@ export function holdWrites(socket: net.Socket): void {
|
|
|
129
129
|
if (!heldWrites.has(socket)) heldWrites.set(socket, []);
|
|
130
130
|
}
|
|
131
131
|
|
|
132
|
+
/** Whether `socket`'s frames are being held, that is, its deferred `registered` has not been written. */
|
|
133
|
+
export function isHoldingWrites(socket: net.Socket): boolean {
|
|
134
|
+
return heldWrites.has(socket);
|
|
135
|
+
}
|
|
136
|
+
|
|
132
137
|
/** Stop holding frames for `socket`; returns the frames held, in the order they were written. */
|
|
133
138
|
export function releaseHeldWrites(socket: net.Socket): HeldWrite[] {
|
|
134
139
|
const held = heldWrites.get(socket) ?? [];
|
|
@@ -1,12 +1,21 @@
|
|
|
1
1
|
import { spawn } from "child_process";
|
|
2
|
+
import { randomUUID } from "crypto";
|
|
2
3
|
import { appendFileSync, chmodSync, closeSync, existsSync, mkdirSync, openSync, readFileSync, readSync, statSync, unlinkSync, writeFileSync } from "fs";
|
|
3
4
|
import { join, dirname } from "path";
|
|
4
5
|
import { fileURLToPath } from "url";
|
|
5
6
|
import { createRequire } from "module";
|
|
6
7
|
import { isSocketAnswering } from "./socket-liveness.js";
|
|
8
|
+
import { BROKER_READY_TIMEOUT_MS } from "./startup-timeout.js";
|
|
9
|
+
import {
|
|
10
|
+
BROKER_ANNOUNCE_PATH_ENV,
|
|
11
|
+
BROKER_ANNOUNCE_TOKEN_ENV,
|
|
12
|
+
type BrokerStartAnnouncement,
|
|
13
|
+
readBrokerStartAnnouncement,
|
|
14
|
+
} from "./start-announcement.js";
|
|
7
15
|
import { createChildProcessEnvironment, isBunBinary, processSharedStorageSelection } from "@bastani/atomic";
|
|
8
16
|
import {
|
|
9
17
|
BROKER_STORAGE_ENV,
|
|
18
|
+
getBrokerStartAnnouncementPath,
|
|
10
19
|
getBrokerLogPath,
|
|
11
20
|
getBrokerPidPath,
|
|
12
21
|
getBrokerSocketPath,
|
|
@@ -37,6 +46,7 @@ const BROKER_SOCKET = getBrokerSocketPath();
|
|
|
37
46
|
const BROKER_PID = getBrokerPidPath();
|
|
38
47
|
const BROKER_SPAWN_LOCK = getBrokerSpawnLockPath();
|
|
39
48
|
const BROKER_LOG = getBrokerLogPath();
|
|
49
|
+
const BROKER_START_ANNOUNCEMENT = getBrokerStartAnnouncementPath();
|
|
40
50
|
|
|
41
51
|
function ensurePrivateDirectory(path: string): void {
|
|
42
52
|
mkdirSync(path, { recursive: true, mode: 0o700 });
|
|
@@ -58,8 +68,7 @@ export const BROKER_POLL_INITIAL_INTERVAL_MS = 10;
|
|
|
58
68
|
/** Upper bound for the readiness poll interval, matching the historic flat poll. */
|
|
59
69
|
export const BROKER_POLL_MAX_INTERVAL_MS = 100;
|
|
60
70
|
|
|
61
|
-
|
|
62
|
-
export const BROKER_READY_TIMEOUT_MS = 5000;
|
|
71
|
+
export { BROKER_READY_TIMEOUT_MS };
|
|
63
72
|
|
|
64
73
|
/** Backstop only: a spawn lock normally ends when its broker is ready or the processes it names are gone. */
|
|
65
74
|
export const BROKER_SPAWN_LOCK_MAX_AGE_MS = 60_000;
|
|
@@ -175,6 +184,12 @@ export function getWindowsBrokerCommandLine(
|
|
|
175
184
|
* dropped instead: launching correctly without a captured log beats launching wrongly with one.
|
|
176
185
|
* A `%` reaches here only from a custom `brokerCommand`/`brokerArgs` or an install path that
|
|
177
186
|
* contains one, and the direct (non-launcher) Windows path still captures stderr normally.
|
|
187
|
+
*
|
|
188
|
+
* Nothing may follow the broker in this chain: after the broker's process ends, the resident `cmd.exe` must do no
|
|
189
|
+
* file I/O, because tests, `atomic intercom restart` and users remove the intercom directory the moment the broker
|
|
190
|
+
* is gone, and a file the chain creates there then fails that removal (the marker an earlier version appended made
|
|
191
|
+
* every Windows test that kills a broker and removes its directory fail with ENOTEMPTY). The broker's own start-up
|
|
192
|
+
* announcement (`start-announcement.ts`) is how the parent learns that a broker it cannot watch has exited.
|
|
178
193
|
*/
|
|
179
194
|
export function getWindowsStderrRedirectCommandLine(commandLine: string, logPath: string): string {
|
|
180
195
|
if (commandLine.includes("%")) return commandLine;
|
|
@@ -247,10 +262,12 @@ export function getBrokerLaunchSpec(
|
|
|
247
262
|
*
|
|
248
263
|
* The broker routes through the shared storage this process was launched with (`--storage`,
|
|
249
264
|
* `--storage-file` or `ATOMIC_STORAGE`), passed as `ATOMIC_STORAGE`; without one it reads settings.
|
|
265
|
+
* With `announcement` it also gets the file and token it announces its pid with (`start-announcement.ts`).
|
|
250
266
|
*/
|
|
251
267
|
export function getBrokerSpawnOptions(
|
|
252
268
|
extensionDir: string = EXTENSION_DIR,
|
|
253
269
|
stderr: number | "ignore" = "ignore",
|
|
270
|
+
announcement?: BrokerStartAnnouncement,
|
|
254
271
|
): {
|
|
255
272
|
detached: true;
|
|
256
273
|
stdio: ["ignore", "ignore", number | "ignore"];
|
|
@@ -266,6 +283,9 @@ export function getBrokerSpawnOptions(
|
|
|
266
283
|
env: createChildProcessEnvironment({
|
|
267
284
|
NODE_NO_WARNINGS: "1",
|
|
268
285
|
...(storage === undefined ? {} : { [BROKER_STORAGE_ENV]: storage }),
|
|
286
|
+
...(announcement === undefined
|
|
287
|
+
? {}
|
|
288
|
+
: { [BROKER_ANNOUNCE_PATH_ENV]: announcement.path, [BROKER_ANNOUNCE_TOKEN_ENV]: announcement.token }),
|
|
269
289
|
}),
|
|
270
290
|
windowsHide: true,
|
|
271
291
|
};
|
|
@@ -357,6 +377,17 @@ export async function spawnBrokerIfNeeded(
|
|
|
357
377
|
|
|
358
378
|
const brokerPath = resolveBrokerEntrypoint(extensionDir);
|
|
359
379
|
const launch = getBrokerLaunchSpec(brokerPath, brokerCommand, brokerArgs, extensionDir);
|
|
380
|
+
// The Windows launcher hides the broker's pid and exit, so there the broker announces its pid, which this watches.
|
|
381
|
+
// Elsewhere the child is the broker and its exit event says how it ended.
|
|
382
|
+
const announcement: BrokerStartAnnouncement | undefined =
|
|
383
|
+
launch.kind === "windows-launcher" ? { path: BROKER_START_ANNOUNCEMENT, token: randomUUID() } : undefined;
|
|
384
|
+
if (announcement !== undefined) {
|
|
385
|
+
try {
|
|
386
|
+
unlinkSync(announcement.path);
|
|
387
|
+
} catch {
|
|
388
|
+
// None left by an earlier broker, or not removable: its token is not this spawn's, so it is ignored.
|
|
389
|
+
}
|
|
390
|
+
}
|
|
360
391
|
if (launch.kind === "windows-launcher") {
|
|
361
392
|
writeWindowsHiddenLauncher(launch.launcherCommandLine, launch.launcherPath);
|
|
362
393
|
}
|
|
@@ -371,7 +402,7 @@ export async function spawnBrokerIfNeeded(
|
|
|
371
402
|
const logFd = launch.kind === "direct" ? openSync(BROKER_LOG, "a") : undefined;
|
|
372
403
|
let child: ReturnType<typeof spawn>;
|
|
373
404
|
try {
|
|
374
|
-
child = spawn(launch.command, launch.args, getBrokerSpawnOptions(extensionDir, logFd ?? "ignore"));
|
|
405
|
+
child = spawn(launch.command, launch.args, getBrokerSpawnOptions(extensionDir, logFd ?? "ignore", announcement));
|
|
375
406
|
} finally {
|
|
376
407
|
if (logFd !== undefined) closeSync(logFd);
|
|
377
408
|
}
|
|
@@ -411,7 +442,10 @@ export async function spawnBrokerIfNeeded(
|
|
|
411
442
|
|
|
412
443
|
child.once("error", onError);
|
|
413
444
|
child.once("exit", onExit);
|
|
414
|
-
waitForBroker(
|
|
445
|
+
waitForBroker(
|
|
446
|
+
readyTimeoutMs,
|
|
447
|
+
announcement === undefined ? undefined : () => readBrokerStartAnnouncement(announcement),
|
|
448
|
+
).then(() => {
|
|
415
449
|
cleanup();
|
|
416
450
|
resolve();
|
|
417
451
|
}, (error) => {
|
|
@@ -549,16 +583,34 @@ export function releaseSpawnLock(owner: string, lockPath: string = BROKER_SPAWN_
|
|
|
549
583
|
* Poll the broker socket with a short initial interval and bounded exponential backoff.
|
|
550
584
|
* Startup is tens of milliseconds, so the previous flat 100 ms sleep dominated it; the
|
|
551
585
|
* overall timeout semantics are unchanged.
|
|
586
|
+
*
|
|
587
|
+
* `announcedPid` reports the pid of a broker this spawner cannot watch directly (behind the Windows launcher), once the
|
|
588
|
+
* broker has announced it. A broker whose pid is gone and that does not answer has crashed, and the wait ends at once:
|
|
589
|
+
* only a broker that is alive and not yet listening is waited for the whole budget.
|
|
552
590
|
*/
|
|
553
|
-
export async function waitForBroker(
|
|
591
|
+
export async function waitForBroker(
|
|
592
|
+
timeoutMs = BROKER_READY_TIMEOUT_MS,
|
|
593
|
+
announcedPid?: () => number | undefined,
|
|
594
|
+
): Promise<void> {
|
|
554
595
|
const start = Date.now();
|
|
555
596
|
let interval = BROKER_POLL_INITIAL_INTERVAL_MS;
|
|
556
597
|
while (Date.now() - start < timeoutMs) {
|
|
557
598
|
if (await checkSocketConnectable()) {
|
|
558
599
|
return;
|
|
559
600
|
}
|
|
601
|
+
const pid = announcedPid?.();
|
|
602
|
+
if (pid !== undefined && !isProcessAlive(pid)) {
|
|
603
|
+
// A broker that finds a live broker on the socket yields and exits; that one answering is a success.
|
|
604
|
+
if (await checkSocketConnectable()) return;
|
|
605
|
+
throw new Error(`Intercom broker exited before startup (pid ${pid})\n${describeBrokerLog()}`);
|
|
606
|
+
}
|
|
560
607
|
await sleep(interval);
|
|
561
608
|
interval = Math.min(interval * 2, BROKER_POLL_MAX_INTERVAL_MS);
|
|
562
609
|
}
|
|
563
|
-
throw new Error(
|
|
610
|
+
throw new Error(
|
|
611
|
+
`Broker failed to start within timeout (${timeoutMs} ms)\n`
|
|
612
|
+
+ "The broker did not answer on its socket in time; it may still be starting, which a busy machine or an empty jiti "
|
|
613
|
+
+ "cache slows. Try again, or raise `brokerStartupTimeoutMs` in the Intercom config.json (default "
|
|
614
|
+
+ `${BROKER_READY_TIMEOUT_MS} ms).\n${describeBrokerLog()}`,
|
|
615
|
+
);
|
|
564
616
|
}
|
|
@@ -0,0 +1,77 @@
|
|
|
1
|
+
import { readFileSync, unlinkSync, writeFileSync } from "node:fs";
|
|
2
|
+
|
|
3
|
+
/**
|
|
4
|
+
* A spawned broker announces its real pid, so a parent that cannot watch the process it spawned can tell a broker that
|
|
5
|
+
* crashed during start-up from one that is still loading.
|
|
6
|
+
*
|
|
7
|
+
* On Windows the broker starts behind `wscript -> cmd -> node`: `wscript` returns at once and hides the pid, so without
|
|
8
|
+
* this the parent waits the whole start-up budget for a broker that died in its first second. The broker is the one
|
|
9
|
+
* process that knows its own pid, so it writes it (with the spawn's token) before it loads the rest of its modules, and
|
|
10
|
+
* the parent polls `process.kill(pid, 0)` on it. That is the whole mechanism, and deliberately nothing else: no process
|
|
11
|
+
* of the launcher chain writes anything once the broker has ended, because tests, `atomic intercom restart` and users
|
|
12
|
+
* remove the intercom directory as soon as the broker is gone, and a file created there at that moment fails the
|
|
13
|
+
* removal on Windows.
|
|
14
|
+
*
|
|
15
|
+
* The parent hands the broker the file and token through the environment, so a broker started any other way (a test,
|
|
16
|
+
* an older parent) announces nothing. The token ties a file to the spawn that asked for it: a broker that is retiring
|
|
17
|
+
* and a replacement that is starting never read each other's announcement as their own.
|
|
18
|
+
*/
|
|
19
|
+
export const BROKER_ANNOUNCE_PATH_ENV = "ATOMIC_INTERCOM_BROKER_ANNOUNCE_PATH";
|
|
20
|
+
export const BROKER_ANNOUNCE_TOKEN_ENV = "ATOMIC_INTERCOM_BROKER_ANNOUNCE_TOKEN";
|
|
21
|
+
|
|
22
|
+
/** The file and token one spawn asks its broker to announce itself with. */
|
|
23
|
+
export interface BrokerStartAnnouncement {
|
|
24
|
+
readonly path: string;
|
|
25
|
+
readonly token: string;
|
|
26
|
+
}
|
|
27
|
+
|
|
28
|
+
/** Tokens go in the file and the environment unquoted, so only plain characters are accepted. */
|
|
29
|
+
const TOKEN_PATTERN = /^[A-Za-z0-9-]+$/u;
|
|
30
|
+
|
|
31
|
+
/** The announcement the environment asks this process for, or undefined when it asks for none. */
|
|
32
|
+
function requested(env: NodeJS.ProcessEnv): BrokerStartAnnouncement | undefined {
|
|
33
|
+
const path = env[BROKER_ANNOUNCE_PATH_ENV];
|
|
34
|
+
const token = env[BROKER_ANNOUNCE_TOKEN_ENV];
|
|
35
|
+
if (!path || !token || !TOKEN_PATTERN.test(token)) return undefined;
|
|
36
|
+
return { path, token };
|
|
37
|
+
}
|
|
38
|
+
|
|
39
|
+
/** Broker side: write `<token> <pid>` where the parent asked. A failure to write must never keep the broker from starting. */
|
|
40
|
+
export function announceBrokerStart(env: NodeJS.ProcessEnv = process.env, pid: number = process.pid): void {
|
|
41
|
+
const announcement = requested(env);
|
|
42
|
+
if (announcement === undefined) return;
|
|
43
|
+
try {
|
|
44
|
+
writeFileSync(announcement.path, `${announcement.token} ${pid}\n`, { mode: 0o600 });
|
|
45
|
+
} catch {
|
|
46
|
+
// The parent then waits out its start-up budget, as it did before broker announcements existed.
|
|
47
|
+
}
|
|
48
|
+
}
|
|
49
|
+
|
|
50
|
+
/** Broker side: take the announcement back once the broker listens, so none outlives the broker's start-up. */
|
|
51
|
+
export function retractBrokerStartAnnouncement(env: NodeJS.ProcessEnv = process.env, pid: number = process.pid): void {
|
|
52
|
+
const announcement = requested(env);
|
|
53
|
+
if (announcement === undefined) return;
|
|
54
|
+
// Only the broker's own announcement: a stale file from another broker is the parent's to replace.
|
|
55
|
+
if (readBrokerStartAnnouncement(announcement) !== pid) return;
|
|
56
|
+
try {
|
|
57
|
+
unlinkSync(announcement.path);
|
|
58
|
+
} catch {
|
|
59
|
+
// Already gone, or held open by the parent's read; the next spawn replaces it.
|
|
60
|
+
}
|
|
61
|
+
}
|
|
62
|
+
|
|
63
|
+
/** Parent side: the pid this spawn's broker announced, or undefined until it has, or when the file is another spawn's. */
|
|
64
|
+
export function readBrokerStartAnnouncement(announcement: BrokerStartAnnouncement): number | undefined {
|
|
65
|
+
let text: string;
|
|
66
|
+
try {
|
|
67
|
+
text = readFileSync(announcement.path, "utf-8");
|
|
68
|
+
} catch {
|
|
69
|
+
return undefined;
|
|
70
|
+
}
|
|
71
|
+
// A complete line only: a write still in progress is read as nothing yet.
|
|
72
|
+
if (!text.endsWith("\n")) return undefined;
|
|
73
|
+
const [token, pid, ...rest] = text.trim().split(/\s+/u);
|
|
74
|
+
if (token !== announcement.token || rest.length > 0) return undefined;
|
|
75
|
+
const parsed = Number(pid);
|
|
76
|
+
return Number.isInteger(parsed) && parsed > 0 ? parsed : undefined;
|
|
77
|
+
}
|
|
@@ -0,0 +1,23 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* How long one spawn attempt waits for a new broker to answer on its socket, unless `brokerStartupTimeoutMs` in the
|
|
3
|
+
* Intercom config says otherwise.
|
|
4
|
+
*
|
|
5
|
+
* A Node install starts the broker from `broker/broker.ts` through jiti, so the wait is the broker's own start-up,
|
|
6
|
+
* which scales with the CPU share it gets. Measured from spawn to the first answered connection (Linux, one broker,
|
|
7
|
+
* pinned to a core shared with busy loops):
|
|
8
|
+
*
|
|
9
|
+
* | jiti cache | idle | 5 processes per core | 9 per core | 17 per core |
|
|
10
|
+
* | ---------- | ------ | -------------------- | ---------- | ----------- |
|
|
11
|
+
* | warm | 0.49 s | 2.6 s | 4.6 s | |
|
|
12
|
+
* | cold | 1.57 s | 8.7 s | 15.0 s | 27.7 s |
|
|
13
|
+
*
|
|
14
|
+
* jiti's cache lives in the temporary directory, so a cold cache is the first start after a reboot or on a
|
|
15
|
+
* throwaway CI machine. The previous 5 s covered only a warm cache on a busy host, so a loaded or freshly started
|
|
16
|
+
* machine failed Intercom start-up with a broker that was still starting. 30 s covers a cold cache at 17 processes
|
|
17
|
+
* per core. A broker that exits ends the wait at once (except behind the Windows launcher, which hides its pid), so a
|
|
18
|
+
* long budget delays only the report of a broker that stays alive without listening.
|
|
19
|
+
*/
|
|
20
|
+
export const BROKER_READY_TIMEOUT_MS = 30_000;
|
|
21
|
+
|
|
22
|
+
/** The smallest `brokerStartupTimeoutMs` the config accepts: below this a broker cannot listen on any host. */
|
|
23
|
+
export const MIN_BROKER_STARTUP_TIMEOUT_MS = 1_000;
|