rollbridge 0.1.58 → 0.1.60

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,20 @@
1
+ ### Fixed
2
+
3
+ - A refused owner mutation now names its holder. The refusal used to read
4
+ `Another owner mutation is already in progress` with no context, so an
5
+ operator could not tell which command held the fence, which process held it,
6
+ for how long, or what it was blocked on. The guardian now records the
7
+ mutation's operation, lease id, holding PID, start time and current blocker
8
+ (`waitingFor`, refinable while the mutation runs through
9
+ `reportOwnerMutationWait`) and reports them in every refusal, including the
10
+ replacement-prepare fence.
11
+
12
+ ### Changed
13
+
14
+ - The owner-mutation fence is bounded. A mutation that outruns its lease
15
+ (default 15 minutes, per-command override through `mutationTimeoutMs`) is
16
+ logged as `OWNER_MUTATION_LEASE_EXPIRED`, reported to the holder, and
17
+ released, so a stuck mutation can no longer lock out every later deploy. The
18
+ holder fails loudly when it next ends that mutation instead of silently
19
+ keeping the fence. Retained generation drains are unaffected: they are not
20
+ owner mutations and may still run for hours.
@@ -0,0 +1,32 @@
1
+ ### Added
2
+
3
+ - `processes[].dependsOn` names the same-release processes a process needs, and
4
+ Rollbridge now waits for each dependency's allocated port **before** spawning
5
+ the dependent, on `processes[].dependencyTimeoutMs` (default 30 seconds) — a
6
+ budget of its own that is never charged to `health.timeoutMs`. Each wait is
7
+ logged as `release process dependency wait` and `release process dependency
8
+ ready` with `attempts` and `elapsedMs`. `validate` rejects a dependency on
9
+ itself, an unknown process, a `singleton`, a process with no `port`, and a
10
+ process Rollbridge starts after the dependent, which also rules out cycles.
11
+
12
+ ### Fixed
13
+
14
+ - A release-scoped process no longer spends the proxied health deadline waiting
15
+ for a dependency. Measured on production 2026-11-06: `background-jobs-main`
16
+ took 11 817 ms to start listening (14:33:39.175 → 14:33:50.992) and the web
17
+ command's own `wait-port` hop then ate part of the same 30 000 ms that
18
+ `proxy.healthTimeoutMs` and `health.timeoutMs` allowed for the whole start
19
+ sequence, leaving the app about 16 seconds to boot before Rollbridge killed a
20
+ release that was still starting normally. The health gate now measures only
21
+ the proxied process's own boot and reports `bootMs`, `healthGateMs`, and
22
+ `waitedForDependenciesMs` in `release candidate health ready`.
23
+ - A failed release start is diagnostic. The health failure now reads
24
+ `Health check failed for <url>: <reason> (process <id>: 18 probes, 941ms
25
+ elapsed of the 900ms health deadline)` instead of naming only the connection
26
+ error, and `release startup failed` carries `deadlineMs`, `dependencyWaitMs`,
27
+ `elapsedMs`, `releaseElapsedMs`, `stage` (`dependency` or `health`), `probes`,
28
+ `failedProcessId`, `readyProcesses`, `unreadyProcesses`, and the failed
29
+ process's `lastOutput`/`lastOutputAt`. A dependency gate that expires fails
30
+ the deploy as `Process <id> could not start because dependency <id> was not
31
+ ready: …` on its own budget; the health gate is unchanged, so a genuinely
32
+ unhealthy release still fails and the incumbent keeps serving.
package/docs/config.md CHANGED
@@ -233,6 +233,8 @@ legacyTakeover: {
233
233
  | `env` | object of string → string | `{}` | Extra environment variables (values templated). Merged over the injected `ROLLBRIDGE_*` vars. |
234
234
  | `port` | number or `{from, to}` | unset | Port (or range) allocated per release. **Required for the `proxied` process.** A plain number `n` means the fixed port `n` (`{from: n, to: n}`). |
235
235
  | `health` | object or `false` | enabled with defaults | Health check for the `proxied` process; set `false` to disable (see below). |
236
+ | `dependsOn` | array of process ids | `[]` | Processes whose allocated port must accept a connection before Rollbridge starts this process (see [`processes[].dependsOn`](#processesdependson)). |
237
+ | `dependencyTimeoutMs` | positive number | `30000` | Budget for each `dependsOn` wait. Charged to its own clock, never to `health.timeoutMs`. |
236
238
  | `stopSignal` | signal name (e.g. `"SIGTERM"`, `"SIGINT"`, `"SIGQUIT"`) | `"SIGTERM"` | Signal sent to gracefully stop the process; after `gracefulStopMs` it is `SIGKILL`ed. Use a worker's quit signal so it finishes in-flight work before exiting. |
237
239
  | `nonBlockingDrain` | boolean | `false` | When a release is retired, drain this process **immediately** (in parallel with the proxied connection drain) instead of after it. Companion processes only — typically background workers (see below). |
238
240
  | `lifecycle` | object | no hooks | Command hooks run when gracefully stopping the process (see below). |
@@ -447,6 +449,43 @@ release). Set `health: false` to disable it.
447
449
  | `health.intervalMs` | number | `250` | Delay between probes. |
448
450
  | `health.startDelayMs` | non-negative number | `0` | Wait this long after the process starts before the first probe (runs before the `timeoutMs` window). |
449
451
 
452
+ ### `processes[].dependsOn`
453
+
454
+ A release-scoped process that talks to a same-release service usually waits for
455
+ that service inside its own command (`wait-port … && exec <app>`). That wait
456
+ runs after Rollbridge has started the process, so it is charged against
457
+ `health.timeoutMs`: a dependency that takes 12 seconds to bind leaves the app 18
458
+ seconds of a 30-second health deadline, and the deploy fails while the app is
459
+ still booting normally.
460
+
461
+ Declare the dependency instead, and Rollbridge waits for each `dependsOn`
462
+ process's allocated port **before** spawning the dependent, on a budget of its
463
+ own (`dependencyTimeoutMs`):
464
+
465
+ ```js
466
+ {
467
+ id: "web",
468
+ policy: "proxied",
469
+ dependsOn: ["background-jobs-main"],
470
+ dependencyTimeoutMs: 30000,
471
+ command: "exec node build/bin/velocious.js server --port {{port}}",
472
+ port: {from: 14500, to: 14599},
473
+ health: {path: "/ping", timeoutMs: 120000}
474
+ }
475
+ ```
476
+
477
+ `health.timeoutMs` then measures only the app's own boot. Both waits are logged
478
+ separately: `release process dependency wait` / `release process dependency
479
+ ready` (with `attempts` and `elapsedMs`) and `release candidate health ready`
480
+ (with `bootMs`, `healthGateMs`, and `waitedForDependenciesMs`).
481
+
482
+ `dependsOn` accepts ids of processes Rollbridge starts before this one —
483
+ persistent services, handoff services, companions, and the proxied process. A
484
+ dependency must have a `port` (there is nothing to wait for otherwise), may not
485
+ be a `singleton` (singleton ownership is not per-release), and may not be this
486
+ process itself. Dependencies that Rollbridge would start after this process are
487
+ rejected, which also rules out cycles.
488
+
450
489
  ## `compose[]`
451
490
 
452
491
  Optional Docker Compose services this app updates. Each entry is a unit Rollbridge
@@ -548,6 +587,7 @@ Rollbridge sets these in every managed process's environment (the process's own
548
587
  - `replicas` must be a positive integer; `replicas > 1` is allowed only on a `companion` process without a `port`. Process ids must not contain `#` (reserved for replica instance ids).
549
588
  - `lifecycle.activateCommand`/`quietCommand`/`drainCommand`/`stopCommand` must be non-empty strings when set, and `lifecycle.drainTimeoutMs` a non-negative number; `lifecycle.drainCommand` requires a positive `lifecycle.drainTimeoutMs`. `activateCommand` is allowed on at most one handoff service, requires that service's `quietCommand`, and requires `statePath` plus `ownerRecovery`. A `lifecycle.stopCommand` may not be combined with a custom `stopSignal` (the `stopCommand` runs instead of the signal, so the signal would be ignored).
550
589
  - `nonBlockingDrain` must be a boolean, and is allowed only on a `companion` process.
590
+ - `dependsOn` must be an array of configured process ids: not the process itself, not a `singleton`, not a process without a `port`, and not a process Rollbridge starts after it. `dependencyTimeoutMs` must be a positive number.
551
591
  - `statePath` must be a string when set.
552
592
  - `compose` must be a non-empty array of unit mappings. Each unit needs `id`, `service`, `projectDirectory`, and a valid `image` reference (a tag, or a `@sha256:` digest with 64 hex characters; no whitespace, no leading `-`).
553
593
  - Compose unit `id`s must be unique and must not collide with a process `id`.
@@ -54,14 +54,27 @@ Health check failed for http://127.0.0.1:18182/ping: HTTP 503
54
54
  new release never went live; the previous release stays active.
55
55
 
56
56
  **Diagnose.** The new release's `proxied` process didn't return a healthy
57
- response in time. Check its output with `rollbridge logs --process <id>` and its
58
- state/`exitCode` with `rollbridge status`. Common causes: the app doesn't listen
59
- on the templated `{{port}}`, the `health.path` returns a non-2xx status, or the
60
- app boots slower than `health.timeoutMs`.
57
+ response in time. The reason names what it spent that budget on —
58
+ `(process web: 18 probes, 941ms elapsed of the 900ms health deadline)` — and the
59
+ `release startup failed` event carries `deadlineMs`, `elapsedMs`,
60
+ `dependencyWaitMs`, `probes`, `stage` (`"dependency"` or `"health"`),
61
+ `failedProcessId`, `readyProcesses`/`unreadyProcesses`, and the process's
62
+ `lastOutput`. Check that process's output with `rollbridge logs --process <id>`
63
+ and its state/`exitCode` with `rollbridge status`. Common causes: the app doesn't
64
+ listen on the templated `{{port}}`, the `health.path` returns a non-2xx status, or
65
+ the app boots slower than `health.timeoutMs`.
66
+
67
+ If the boot itself is fine but the app waits for a same-release service inside
68
+ its own command, that wait is inside the health deadline. `stage: "dependency"`
69
+ plus a `release process dependency ready` event shows the wait Rollbridge did on
70
+ the dependency's own budget.
61
71
 
62
72
  **Fix.** Make the proxied command bind `{{port}}` and serve `health.path` with a
63
- 2xx status. For slow boots, raise `health.timeoutMs` or set `health.startDelayMs`
64
- so probing begins after the app is up.
73
+ 2xx status. For slow boots, raise `health.timeoutMs` to above the measured boot
74
+ time (or set `health.startDelayMs` so probing begins after the app is up). For a
75
+ same-release service dependency, declare `dependsOn` (see
76
+ [`processes[].dependsOn`](config.md#processesdependson)) so the wait runs on
77
+ `dependencyTimeoutMs` instead of eating the boot budget.
65
78
 
66
79
  ## Port conflicts / exhausted ranges
67
80
 
@@ -148,3 +161,27 @@ separate destructive `--takeover-owner` path. If
148
161
  `retirementError` is set,
149
162
  inspect the quiet-hook events. Rollbridge deliberately leaves that generation
150
163
  alive rather than signaling arbitrary PIDs or continuing its stop sequence.
164
+
165
+ ## A command is refused with "Another owner mutation is already in progress"
166
+
167
+ **Symptom.** `deploy`, `stop`, `restart`, `rollback` or
168
+ `recover-generation-transition` fails with
169
+ `Another owner mutation is already in progress: owner mutation "<operation>";
170
+ mutation <id>; held by process <pid>; since <ISO-8601>; <n>s elapsed; waiting
171
+ for <blocker>; refused <operation>`.
172
+
173
+ **Cause.** The committed daemon holds the guardian's owner-mutation fence for
174
+ one control mutation at a time, and a second mutation arrived while it was still
175
+ held.
176
+
177
+ **Fix.** Read the named holder before touching anything: the message states
178
+ which mutation holds the fence, its lease id, the PID of the process that
179
+ started it, when it started, how long it has run, and the blocker that holder
180
+ last reported (`waitingFor`, which a running mutation can refine through
181
+ `reportOwnerMutationWait`). That is the command to inspect — not the one that
182
+ was refused. A mutation that outruns its lease (15 minutes by default,
183
+ overridable per command with `mutationTimeoutMs`) is logged as
184
+ `OWNER_MUTATION_LEASE_EXPIRED`, reported to the holder, and released, so the
185
+ next mutation proceeds and the stuck one fails loudly when it next ends its
186
+ lease. A retained jobs generation is not an owner mutation and may still drain
187
+ for hours; only the fence is bounded.
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "rollbridge",
3
- "version": "0.1.58",
3
+ "version": "0.1.60",
4
4
  "description": "Zero-downtime process supervisor and local traffic switcher for deploy-managed apps.",
5
5
  "keywords": [
6
6
  "deploy",
package/src/config.js CHANGED
@@ -6,6 +6,7 @@ import {createRequire} from "node:module"
6
6
  import os from "node:os"
7
7
  import path from "node:path"
8
8
  import {pathToFileURL} from "node:url"
9
+ import {processStartOrder} from "./process-order.js"
9
10
 
10
11
  /**
11
12
  * @typedef {import("./json.js").JsonValue} JsonValue
@@ -17,7 +18,7 @@ import {pathToFileURL} from "node:url"
17
18
  * @typedef {{activateCommand?: string, activateTimeoutMs?: number, drainCommand?: string, drainTimeoutMs: number, quietCommand?: string, reactivateCommand?: string, stopCommand?: string}} LifecycleConfig
18
19
  * @typedef {number | "indefinite"} StopTimeoutMs
19
20
  * @typedef {"persistent" | "handoff"} ServiceDeployStrategy
20
- * @typedef {{cwd?: string, deployStrategy: ServiceDeployStrategy, env: Record<string, string>, gracefulStopMs: StopTimeoutMs, health?: HealthConfig, id: string, lifecycle: LifecycleConfig, memory?: MemoryConfig, nonBlockingDrain: boolean, outputLines: number, policy: ProcessPolicy, port?: PortRange, replicas: number, restart: RestartConfig, restartDelayMs: number, stopSignal: string, command: string}} ProcessConfig
21
+ * @typedef {{cwd?: string, dependencyTimeoutMs: number, dependsOn: string[], deployStrategy: ServiceDeployStrategy, env: Record<string, string>, gracefulStopMs: StopTimeoutMs, health?: HealthConfig, id: string, lifecycle: LifecycleConfig, memory?: MemoryConfig, nonBlockingDrain: boolean, outputLines: number, policy: ProcessPolicy, port?: PortRange, replicas: number, restart: RestartConfig, restartDelayMs: number, stopSignal: string, command: string}} ProcessConfig
21
22
  * @typedef {{group?: number | string, mode?: number, owner?: number | string, path: string}} ControlConfig
22
23
  * @typedef {{drainTimeoutMs: number, forceStopTimeoutMs: number, healthPath: string, healthTimeoutMs: number, host: string, port: number, upstreamHost: string}} ProxyConfig
23
24
  * @typedef {{includes: string[], name: string}} LegacyTakeoverProcessConfig
@@ -35,6 +36,13 @@ import {pathToFileURL} from "node:url"
35
36
 
36
37
  const PROCESS_POLICIES = new Set(["proxied", "companion", "singleton", "service"])
37
38
  const DEFAULT_CONFIG_FILENAMES = ["rollbridge.js"]
39
+ // Budget for one process waiting for a dependency's port, separate from the health budget.
40
+ // Measured on TensorBuzz production (server3, 2026-11-06): the candidate's
41
+ // background-jobs-main took 11817ms (14:33:39.175 start -> 14:33:50.992 listening) before
42
+ // the web process could reach its own boot, and the production command chain already waited
43
+ // 30000ms per hop. 30000ms keeps that dependency budget, so the health deadline measures
44
+ // only the app boot.
45
+ const DEFAULT_DEPENDENCY_TIMEOUT_MS = 30000
38
46
  const commonJsRequire = createRequire(import.meta.url)
39
47
 
40
48
  /**
@@ -232,7 +240,7 @@ function normalizeProcess(value, index, proxy, issues) {
232
240
  if (!isPlainObject(value)) {
233
241
  issues.push({fix: `Define processes[${index}] as a mapping with id, policy, and command.`, message: `processes[${index}] must be an object`})
234
242
 
235
- return {command: "", cwd: undefined, deployStrategy: "persistent", env: {}, gracefulStopMs: proxy.forceStopTimeoutMs, health: undefined, id: "", lifecycle: {drainTimeoutMs: 0}, memory: undefined, nonBlockingDrain: false, outputLines: 50, policy: "companion", port: undefined, replicas: 1, restart: defaultRestartConfig(), restartDelayMs: 1000, stopSignal: "SIGTERM"}
243
+ return {command: "", cwd: undefined, dependencyTimeoutMs: DEFAULT_DEPENDENCY_TIMEOUT_MS, dependsOn: [], deployStrategy: "persistent", env: {}, gracefulStopMs: proxy.forceStopTimeoutMs, health: undefined, id: "", lifecycle: {drainTimeoutMs: 0}, memory: undefined, nonBlockingDrain: false, outputLines: 50, policy: "companion", port: undefined, replicas: 1, restart: defaultRestartConfig(), restartDelayMs: 1000, stopSignal: "SIGTERM"}
236
244
  }
237
245
 
238
246
  const source = value
@@ -240,6 +248,8 @@ function normalizeProcess(value, index, proxy, issues) {
240
248
  return {
241
249
  command: normalizeString(source.command, `processes[${index}].command`, issues),
242
250
  cwd: source.cwd === undefined ? undefined : normalizeString(source.cwd, `processes[${index}].cwd`, issues),
251
+ dependencyTimeoutMs: normalizeDependencyTimeoutMs(source.dependencyTimeoutMs, `processes[${index}].dependencyTimeoutMs`, issues),
252
+ dependsOn: normalizeDependsOn(source.dependsOn, `processes[${index}].dependsOn`, issues),
243
253
  deployStrategy: normalizeDeployStrategy(source.deployStrategy, `processes[${index}].deployStrategy`, issues),
244
254
  env: normalizeEnv(source.env, `processes[${index}].env`, issues),
245
255
  gracefulStopMs: normalizeStopTimeout(source.gracefulStopMs, `processes[${index}].gracefulStopMs`, proxy.forceStopTimeoutMs, issues),
@@ -765,6 +775,34 @@ function normalizeDeployStrategy(value, key, issues) {
765
775
  return "persistent"
766
776
  }
767
777
 
778
+ /**
779
+ * @param {JsonValue} value - Raw dependency list.
780
+ * @param {string} key - Config key.
781
+ * @param {ConfigIssue[]} issues - Issue collector.
782
+ * @returns {string[]} Ids of the processes this process waits for before it starts.
783
+ */
784
+ function normalizeDependsOn(value, key, issues) {
785
+ return normalizeStringList(value, key, issues)
786
+ }
787
+
788
+ /**
789
+ * @param {JsonValue} value - Raw dependency wait budget.
790
+ * @param {string} key - Config key.
791
+ * @param {ConfigIssue[]} issues - Issue collector.
792
+ * @returns {number} Milliseconds this process may spend waiting for its dependency ports.
793
+ */
794
+ function normalizeDependencyTimeoutMs(value, key, issues) {
795
+ const timeoutMs = normalizeNumber(value, key, issues, {default: DEFAULT_DEPENDENCY_TIMEOUT_MS})
796
+
797
+ if (timeoutMs < 1) {
798
+ issues.push({fix: `Set ${key} to a positive number of milliseconds, e.g. 30000, or omit it for the default.`, message: `${key} must be a positive number`})
799
+
800
+ return DEFAULT_DEPENDENCY_TIMEOUT_MS
801
+ }
802
+
803
+ return timeoutMs
804
+ }
805
+
768
806
  /**
769
807
  * @param {JsonValue} value - Raw output retention value.
770
808
  * @param {string} key - Config key.
@@ -949,6 +987,57 @@ function validateProcessSet(processes, issues) {
949
987
 
950
988
  issues.push({fix: `Add a port range to the proxied process "${processConfig.id || "(unnamed)"}", e.g. port: {from: 18000, to: 18099}.`, message: `Proxied process ${processConfig.id || "(unnamed)"} must define a port range`})
951
989
  }
990
+
991
+ validateProcessDependencies(processes, issues)
992
+ }
993
+
994
+ /**
995
+ * Validates `dependsOn`: a dependency must name a ported process that Rollbridge starts
996
+ * before the dependent. Waiting is done on the dependency's allocated port, and an edge that
997
+ * points backwards in the start order can never be satisfied, which also rules out cycles.
998
+ * @param {ProcessConfig[]} processes - Normalized processes.
999
+ * @param {ConfigIssue[]} issues - Issue collector.
1000
+ * @returns {void}
1001
+ */
1002
+ function validateProcessDependencies(processes, issues) {
1003
+ const declared = new Map(processes.filter((processConfig) => processConfig.id).map((processConfig) => [processConfig.id, processConfig]))
1004
+ const startOrder = new Map(processStartOrder(processes, {includePersistentServices: true}).map((processConfig, index) => [processConfig.id, index]))
1005
+
1006
+ for (const processConfig of processes) {
1007
+ if (!processConfig.id || processConfig.dependsOn.length === 0) continue
1008
+
1009
+ const dependentIndex = startOrder.get(processConfig.id)
1010
+
1011
+ for (const dependencyId of processConfig.dependsOn) {
1012
+ if (dependencyId === processConfig.id) {
1013
+ issues.push({fix: `Remove "${processConfig.id}" from its own dependsOn.`, message: `Process "${processConfig.id}" cannot depend on itself`})
1014
+ continue
1015
+ }
1016
+
1017
+ const dependency = declared.get(dependencyId)
1018
+
1019
+ if (!dependency) {
1020
+ issues.push({fix: `Name a configured process in "${processConfig.id}".dependsOn; "${dependencyId}" is not one of: ${[...declared.keys()].join(", ")}.`, message: `Process "${processConfig.id}" depends on unknown process "${dependencyId}"`})
1021
+ continue
1022
+ }
1023
+
1024
+ if (dependency.policy === "singleton") {
1025
+ issues.push({fix: `Remove "${dependencyId}" from "${processConfig.id}".dependsOn; a singleton is not part of a release generation, so Rollbridge cannot gate a release start on it.`, message: `Process "${processConfig.id}" cannot depend on singleton "${dependencyId}"`})
1026
+ continue
1027
+ }
1028
+
1029
+ if (!dependency.port) {
1030
+ issues.push({fix: `Give "${dependencyId}" a port, or remove it from "${processConfig.id}".dependsOn; Rollbridge waits for a dependency on its allocated port.`, message: `Process "${dependencyId}" has no port, so "${processConfig.id}" cannot wait for it`})
1031
+ continue
1032
+ }
1033
+
1034
+ const dependencyIndex = startOrder.get(dependencyId)
1035
+
1036
+ if (dependencyIndex === undefined || dependentIndex === undefined || dependencyIndex > dependentIndex) {
1037
+ issues.push({fix: `Move "${dependencyId}" before "${processConfig.id}" in processes[]; Rollbridge starts persistent services, then handoff services, companions, and the proxied process.`, message: `Process "${processConfig.id}" depends on "${dependencyId}", which Rollbridge starts after it`})
1038
+ }
1039
+ }
1040
+ }
952
1041
  }
953
1042
 
954
1043
  /**
package/src/daemon.js CHANGED
@@ -146,7 +146,7 @@ export default class RollbridgeDaemon {
146
146
  this.logger("release generation transition recovery failed", {error: error instanceof Error ? error.message : String(error), releaseId: transition.candidateReleaseId})
147
147
  }
148
148
  return {}
149
- })
149
+ }, {waitingFor: "the retained release generations to settle"})
150
150
  } finally {
151
151
  this.generationTransitionRecovery = false
152
152
  }
@@ -1389,7 +1389,7 @@ export default class RollbridgeDaemon {
1389
1389
  releaseId: stringOrUndefined(data.releaseId),
1390
1390
  releasePath: requiredString(data.releasePath, "releasePath"),
1391
1391
  revision: stringOrUndefined(data.revision)
1392
- }))
1392
+ }), {waitingFor: "candidate activation and the retired generation handoff"})
1393
1393
  }
1394
1394
 
1395
1395
  if (commandName === "recover-generation-transition") {
@@ -1399,7 +1399,7 @@ export default class RollbridgeDaemon {
1399
1399
  releaseId: requiredString(data.releaseId, "releaseId"),
1400
1400
  releasePath: requiredString(data.releasePath, "releasePath"),
1401
1401
  revision: requiredString(data.revision, "revision")
1402
- }))
1402
+ }), {waitingFor: "the retained release generations to settle"})
1403
1403
  }
1404
1404
 
1405
1405
  if (commandName === "status") {
@@ -1414,18 +1414,18 @@ export default class RollbridgeDaemon {
1414
1414
  return await this.executeOwnerMutation("stop", async () => {
1415
1415
  await this.stopRelease(stringOrUndefined(data.releaseId))
1416
1416
  return this.status()
1417
- })
1417
+ }, {waitingFor: "the release's managed processes to stop"})
1418
1418
  }
1419
1419
 
1420
1420
  if (commandName === "restart") {
1421
1421
  return await this.executeOwnerMutation("restart", async () => await this.restartProcesses({
1422
1422
  policy: stringOrUndefined(data.policy),
1423
1423
  processId: stringOrUndefined(data.processId)
1424
- }))
1424
+ }), {waitingFor: "the managed process to stop and start again"})
1425
1425
  }
1426
1426
 
1427
1427
  if (commandName === "rollback") {
1428
- return await this.executeOwnerMutation("rollback", async () => await this.rollback({releaseId: stringOrUndefined(data.releaseId)}))
1428
+ return await this.executeOwnerMutation("rollback", async () => await this.rollback({releaseId: stringOrUndefined(data.releaseId)}), {waitingFor: "the previous release to activate"})
1429
1429
  }
1430
1430
 
1431
1431
  if (commandName === "shutdown") {
@@ -1489,11 +1489,13 @@ export default class RollbridgeDaemon {
1489
1489
  * Runs one owner mutation under the guardian's prepare/commit exclusion fence.
1490
1490
  * @param {string} operation - Mutation diagnostic name.
1491
1491
  * @param {() => Promise<Record<string, JsonValue>>} callback - Mutating operation.
1492
+ * @param {{waitingFor?: string}} [options] - What this mutation blocks on, so a refused
1493
+ * later mutation names the real blocker instead of just "in progress".
1492
1494
  * @returns {Promise<Record<string, JsonValue>>} Operation result.
1493
1495
  */
1494
- async executeOwnerMutation(operation, callback) {
1496
+ async executeOwnerMutation(operation, callback, {waitingFor} = {}) {
1495
1497
  if (!this.guardian) return await callback()
1496
- const mutationId = await this.guardian.beginOwnerMutation(operation)
1498
+ const mutationId = await this.guardian.beginOwnerMutation(operation, {ownerPid: process.pid, waitingFor})
1497
1499
  let result
1498
1500
  let operationError
1499
1501
 
@@ -323,10 +323,11 @@ export default class GuardianClient {
323
323
  /**
324
324
  * Acquires the committed owner's mutation fence.
325
325
  * @param {string} operation - Control mutation name.
326
+ * @param {{mutationTimeoutMs?: number, ownerPid?: number, waitingFor?: string}} [details] - Lease bound, holding PID and what the mutation blocks on.
326
327
  * @returns {Promise<string>} Mutation lease id.
327
328
  */
328
- async beginOwnerMutation(operation) {
329
- const result = /** @type {{mutationId: string}} */ (await this.request({command: "begin-owner-mutation", operation}))
329
+ async beginOwnerMutation(operation, {mutationTimeoutMs, ownerPid, waitingFor} = {}) {
330
+ const result = /** @type {{mutationId: string}} */ (await this.request({command: "begin-owner-mutation", mutationTimeoutMs, operation, ownerPid, waitingFor}))
330
331
 
331
332
  return result.mutationId
332
333
  }
@@ -336,6 +337,16 @@ export default class GuardianClient {
336
337
  await this.request({command: "end-owner-mutation", mutationId})
337
338
  }
338
339
 
340
+ /**
341
+ * Refines what a held mutation is waiting for, so a later refusal names the real blocker.
342
+ * @param {string} mutationId - Mutation lease id.
343
+ * @param {string} waitingFor - Current blocker.
344
+ * @returns {Promise<void>} Report acknowledged.
345
+ */
346
+ async reportOwnerMutationWait(mutationId, waitingFor) {
347
+ await this.request({command: "report-owner-mutation-wait", mutationId, waitingFor})
348
+ }
349
+
339
350
  /** @returns {Promise<{committedReplacementId: string | null, ownerClaimed: boolean, retirementFailed?: boolean, retirementPending?: boolean, retirementReady?: boolean}>} Transaction status. */
340
351
  async replacementStatus() {
341
352
  return /** @type {{committedReplacementId: string | null, ownerClaimed: boolean, retirementFailed?: boolean, retirementPending?: boolean, retirementReady?: boolean}} */ (await this.request({command: "replacement-status"}))
package/src/health.js CHANGED
@@ -1,35 +1,126 @@
1
1
  // @ts-check
2
2
 
3
+ import net from "node:net"
4
+
5
+ /**
6
+ * @typedef {import("./config.js").HealthConfig} HealthConfig
7
+ * @typedef {{attempts: number, deadlineMs: number, elapsedMs: number, processId: string, url: string}} HealthCheckDiagnostics
8
+ * @typedef {Error & {healthCheck?: HealthCheckDiagnostics}} HealthCheckError
9
+ */
10
+
11
+ /**
12
+ * @param {number} ms - Milliseconds.
13
+ * @returns {Promise<void>} Resolves after the delay.
14
+ */
15
+ function sleep(ms) {
16
+ return new Promise((resolve) => setTimeout(resolve, ms))
17
+ }
18
+
3
19
  /**
4
20
  * Waits for a process HTTP health endpoint.
21
+ *
22
+ * The failure names its own budget: `Health check failed ...: fetch failed` cannot tell a
23
+ * slow boot apart from a health deadline that was simply too short.
5
24
  * @param {object} args - Options.
6
25
  * @param {string} args.host - Host.
7
26
  * @param {number} args.port - Port.
8
- * @param {import("./config.js").HealthConfig} args.health - Health config.
9
- * @returns {Promise<void>} Resolves when healthy.
27
+ * @param {HealthConfig} args.health - Health config.
28
+ * @param {string} [args.processId] - Process the probe belongs to, named in the failure.
29
+ * @returns {Promise<{elapsedMs: number}>} Time spent reaching the first healthy response.
10
30
  */
11
- export async function waitForHealth({health, host, port}) {
31
+ export async function waitForHealth({health, host, port, processId = "proxied"}) {
32
+ const startedAtMs = Date.now()
33
+
12
34
  if (health.startDelayMs > 0) {
13
- await new Promise((resolve) => setTimeout(resolve, health.startDelayMs))
35
+ await sleep(health.startDelayMs)
14
36
  }
15
37
 
16
- const deadline = Date.now() + health.timeoutMs
38
+ const deadlineMs = health.timeoutMs
39
+ const deadline = Date.now() + deadlineMs
17
40
  const url = `http://${host}:${port}${health.path}`
41
+ let attempts = 0
18
42
  let lastError = "no attempts"
19
43
 
20
44
  while (Date.now() <= deadline) {
45
+ attempts += 1
46
+
21
47
  try {
22
48
  const response = await fetch(url)
23
49
 
24
- if (response.ok) return
50
+ if (response.ok) return {elapsedMs: Date.now() - startedAtMs}
25
51
 
26
52
  lastError = `HTTP ${response.status}`
27
53
  } catch (error) {
28
54
  lastError = error instanceof Error ? error.message : String(error)
29
55
  }
30
56
 
31
- await new Promise((resolve) => setTimeout(resolve, health.intervalMs))
57
+ await sleep(health.intervalMs)
32
58
  }
33
59
 
34
- throw new Error(`Health check failed for ${url}: ${lastError}`)
60
+ const elapsedMs = Date.now() - startedAtMs
61
+ /** @type {HealthCheckError} */
62
+ const failure = new Error(`Health check failed for ${url}: ${lastError} (process ${processId}: ${attempts} probes, ${elapsedMs}ms elapsed of the ${deadlineMs}ms health deadline)`)
63
+
64
+ failure.healthCheck = {attempts, deadlineMs, elapsedMs, processId, url}
65
+
66
+ throw failure
67
+ }
68
+
69
+ /**
70
+ * Waits until a TCP endpoint accepts a connection, on a budget of its own.
71
+ *
72
+ * This is the dependency leg of a release start: waiting for a process that must bind a
73
+ * port before another one can boot. Charged to the health deadline it measures the
74
+ * dependency instead of the app, so it is timed here on a separate budget.
75
+ * @param {object} args - Options.
76
+ * @param {string} args.host - Host to connect to.
77
+ * @param {number} args.intervalMs - Delay between connection attempts.
78
+ * @param {number} args.port - Port that must accept a connection.
79
+ * @param {number} args.timeoutMs - Budget for the wait.
80
+ * @returns {Promise<{attempts: number, elapsedMs: number}>} Time and attempts spent waiting.
81
+ */
82
+ export async function waitForListeningPort({host, intervalMs, port, timeoutMs}) {
83
+ const startedAtMs = Date.now()
84
+ const deadline = startedAtMs + timeoutMs
85
+ let attempts = 0
86
+
87
+ while (Date.now() <= deadline) {
88
+ attempts += 1
89
+
90
+ if (await tryConnect({deadline, host, intervalMs, port})) {
91
+ return {attempts, elapsedMs: Date.now() - startedAtMs}
92
+ }
93
+
94
+ await sleep(Math.max(1, Math.min(intervalMs, deadline - Date.now())))
95
+ }
96
+
97
+ throw new Error(`Dependency endpoint ${host}:${port} did not accept a connection within ${timeoutMs}ms (${attempts} attempts, ${Date.now() - startedAtMs}ms waited)`)
98
+ }
99
+
100
+ /**
101
+ * @param {object} args - Attempt options.
102
+ * @param {number} args.deadline - Absolute deadline that bounds this attempt.
103
+ * @param {string} args.host - Host to connect to.
104
+ * @param {number} args.intervalMs - Delay between connection attempts.
105
+ * @param {number} args.port - Port to connect to.
106
+ * @returns {Promise<boolean>} True when the endpoint accepted the connection.
107
+ */
108
+ function tryConnect({deadline, host, intervalMs, port}) {
109
+ return new Promise((resolve) => {
110
+ const socket = net.createConnection({host, port})
111
+ let settled = false
112
+
113
+ /** @param {boolean} connected - Whether the endpoint accepted the connection. */
114
+ const finish = (connected) => {
115
+ if (settled) return
116
+ settled = true
117
+ socket.destroy()
118
+ resolve(connected)
119
+ }
120
+
121
+ socket.setTimeout(Math.max(1, Math.min(intervalMs, deadline - Date.now())))
122
+ socket.once("connect", () => finish(true))
123
+ socket.once("error", () => finish(false))
124
+ socket.once("timeout", () => finish(false))
125
+ })
35
126
  }