rollbridge 0.1.70 → 0.1.71

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (63) hide show
  1. package/README.md +10 -63
  2. package/TODO.md +0 -13
  3. package/docs/cli.md +3 -128
  4. package/docs/config.md +6 -177
  5. package/docs/troubleshooting.md +6 -97
  6. package/docs/velocious.md +1 -1
  7. package/docs/workers.md +0 -11
  8. package/examples/tensorbuzz.com.js +2 -3
  9. package/package.json +1 -1
  10. package/src/cli.js +2 -68
  11. package/src/config.js +8 -347
  12. package/src/daemon.js +65 -387
  13. package/src/doctor.js +0 -74
  14. package/src/guardian-client.js +8 -67
  15. package/src/health.js +8 -99
  16. package/src/managed-process.js +16 -197
  17. package/src/process-guardian.js +22 -212
  18. package/src/process-memory.js +2 -62
  19. package/src/release-group.js +37 -435
  20. package/tensorbuzz.yml +1 -20
  21. package/test/completion.test.js +2 -5
  22. package/test/config-validation.test.js +5 -135
  23. package/test/guardian-client.test.js +5 -119
  24. package/test/health.test.js +1 -74
  25. package/test/managed-process.test.js +1 -260
  26. package/test/owner-recovery.test.js +21 -125
  27. package/test/owner-replacement.test.js +14 -175
  28. package/test/process-memory.test.js +1 -47
  29. package/test/release-group.test.js +2 -408
  30. package/test/rollbridge.test.js +72 -737
  31. package/test/shutdown-completion.test.js +0 -52
  32. package/test/support/process.js +1 -3
  33. package/changelog.d/20261025-guardian-re-register-dead-generation.md +0 -13
  34. package/changelog.d/20261026-stale-committed-generation-transition-recovery.md +0 -1
  35. package/changelog.d/20261104-compose-service-units.md +0 -21
  36. package/changelog.d/20261105-owner-mutation-lease-diagnostics.md +0 -20
  37. package/changelog.d/20261106-release-dependency-boot-budgets.md +0 -32
  38. package/changelog.d/20261109-path-filtered-builds.md +0 -11
  39. package/changelog.d/20261110-default-boot-budget-above-measured-startup.md +0 -26
  40. package/changelog.d/20261111-contention-sensitive-daemon-specs.md +0 -27
  41. package/changelog.d/20261113-bounded-nonblocking-retirement.md +0 -5
  42. package/changelog.d/20261115-bound-draining-release-groups.md +0 -62
  43. package/changelog.d/20261115-restore-deploy-refuses-held-generation.md +0 -27
  44. package/changelog.d/20261115-wedged-generation-escalation.md +0 -44
  45. package/changelog.d/20261116-bounded-explicit-stop-retirement.md +0 -30
  46. package/changelog.d/20261116-config-json-canonical-shape.md +0 -11
  47. package/changelog.d/20261116-drain-specs-gate-the-retirement-record.md +0 -29
  48. package/changelog.d/20261116-guardian-version-handoff.md +0 -37
  49. package/changelog.d/20261116-reclaim-dead-owner-mutation.md +0 -7
  50. package/changelog.d/20261117-activate-before-incumbent-retirement.md +0 -1
  51. package/changelog.d/20261117-activation-path-drain-deadline.md +0 -35
  52. package/changelog.d/20261117-stop-companion-after-retired-dependency.md +0 -7
  53. package/docs/compose-services.md +0 -222
  54. package/src/compose-unit.js +0 -725
  55. package/src/drain-policy.js +0 -252
  56. package/src/process-order.js +0 -26
  57. package/test/compose-config.test.js +0 -160
  58. package/test/compose-docker.test.js +0 -297
  59. package/test/compose-unit.test.js +0 -415
  60. package/test/drain-policy.test.js +0 -328
  61. package/test/fixtures/deferred-boot-app.js +0 -106
  62. package/test/fixtures/gated-app.js +0 -50
  63. package/test/tensorbuzz-config.test.js +0 -228
package/README.md CHANGED
@@ -57,7 +57,7 @@ export default {
57
57
  port: 8182,
58
58
  upstreamHost: "127.0.0.1",
59
59
  healthPath: "/ping",
60
- healthTimeoutMs: 120000,
60
+ healthTimeoutMs: 30000,
61
61
  drainTimeoutMs: 60000,
62
62
  forceStopTimeoutMs: 10000
63
63
  },
@@ -183,9 +183,8 @@ handoffs and exits only after the worker pool has drained.
183
183
 
184
184
  For a coordinator that starts quiescent, give the handoff jobs-main paired
185
185
  `lifecycle.activateCommand` and `quietCommand` hooks. After candidate health,
186
- Rollbridge activates the candidate, commits the active proxy target, and only
187
- then starts the old generation's bounded retirement asynchronously. The durable
188
- transition is
186
+ Rollbridge waits for old retirement, waits for candidate activation, and commits
187
+ the active proxy target without an awaited boundary. The durable transition is
189
188
  generation-scoped and resumable; failures remain visible and block unrelated
190
189
  deploys. Post-commit singleton replacement is also journaled and must complete
191
190
  before an exact retry reports success. If the active coordinator restarts,
@@ -256,14 +255,6 @@ supervisor. The upgrade fails before handoff and requires one explicit clean
256
255
  remain atomic. A genuinely pre-replacement guardian still uses the separately
257
256
  documented one-time disruptive compatibility bridge below.
258
257
 
259
- A retained guardian that reports a different rollbridge version from the daemon
260
- runtime is handed to a current-version coordinator through that same bridge, so
261
- a package upgrade does not need a supervisor restart: the coordinator owns the
262
- daemon and every process it starts, the retained guardian keeps the processes it
263
- already owns, and nothing waits for those to drain. `status.ownerTransition`
264
- reports `mode: "guardian-version-handoff"`. The refusal is the last resort and
265
- names both versions plus the exact `shutdown`/`ensure-daemon` commands.
266
-
267
258
  There is one explicit compatibility boundary: the first upgrade from a genuine
268
259
  pre-owner-replacement Rollbridge guardian and daemon cannot share its listeners
269
260
  on supported Node 20. The same boundary applies to a retained transition guardian
@@ -326,34 +317,6 @@ rendered when the process starts:
326
317
  Referencing a placeholder with no value (including an unset `{{env.<NAME>}}`)
327
318
  fails the process start with a clear error, so typos surface immediately.
328
319
 
329
- ### Docker Compose services
330
-
331
- A delivery that also updates Docker Compose services (a Socketduct hub, a gateway
332
- leg, a builder DinD) declares them in `compose`. Each unit is rolled with the
333
- same handshake a process deploy uses — quiesce, wait for in-flight work, recreate
334
- with the target image, gate on activation, roll back to the retained previous
335
- image on failure — instead of a raw `docker compose up -d --force-recreate`:
336
-
337
- ```js
338
- compose: [
339
- {
340
- id: "gateway",
341
- service: "gateway",
342
- projectDirectory: "/srv/socketduct/socketduct-builder-a-standard",
343
- image: "registry.diestoeckels.de/socketduct/gateway:2.4.1",
344
- stopGracePeriodMs: 45000,
345
- drain: {command: "{{composeArgs}} exec -T gateway appctl drain", timeoutMs: 120000},
346
- inFlight: {command: "{{composeArgs}} exec -T gateway appctl in-flight", intervalMs: 2000, timeoutMs: 900000},
347
- activation: {startDelayMs: 2000, timeoutMs: 120000, readyUrl: "http://127.0.0.1:7421/ready"}
348
- }
349
- ]
350
- ```
351
-
352
- A drain or in-flight timeout fails the roll and leaves the current container
353
- serving; a unit whose container publishes the Rollbridge proxy port is never
354
- rolled. See [`docs/compose-services.md`](docs/compose-services.md) for the
355
- stage-by-stage behavior and the fleet runbook.
356
-
357
320
  Configuration examples live in `examples/`, including
358
321
  `examples/tensorbuzz.com.js` for the current TensorBuzz backend deployment; see
359
322
  [`docs/tensorbuzz-runbook.md`](docs/tensorbuzz-runbook.md) for the matching
@@ -466,14 +429,14 @@ On `rollbridge deploy`, the required ordering is:
466
429
  1. starts any missing persistent service and the candidate's handoff services;
467
430
  2. starts the new release's `companion`s, then its `proxied` process, and
468
431
  health-checks the proxied process;
469
- 3. when `activateCommand` is configured, activates and acknowledges the
470
- candidate;
471
- 4. synchronously switches new traffic to the new release;
432
+ 3. when `activateCommand` is configured, retires and acknowledges the previous
433
+ jobs generation, then activates and acknowledges the candidate;
434
+ 4. synchronously switches new traffic to the new release (hook-free configs keep
435
+ the existing switch-then-retire behavior);
472
436
  5. replaces `singleton`s (stops the old one, then starts the new one);
473
- 6. starts the previous generation's bounded retirement asynchronously after the
474
- traffic switch; deploy success never waits for quiescence or drain, and
475
- Rollbridge supervises all retained drains in the background and reaps each
476
- generation only after its handoffs and workers end.
437
+ 6. returns success without waiting for the previous generation or its independent
438
+ HTTP/WebSocket drain; Rollbridge supervises all retained drains in the
439
+ background and reaps each generation only after its handoffs and workers end.
477
440
 
478
441
  If the new release fails to start or health-check, the previous release stays
479
442
  active and any service started during this deploy is rolled back.
@@ -662,22 +625,6 @@ rollbridge rollback --config rollbridge.js # the previous relea
662
625
  rollbridge rollback --config rollbridge.js --release-id v3
663
626
  ```
664
627
 
665
- Roll one declared Docker Compose service drain-first (quiesce it, wait for its
666
- in-flight work, recreate it with the target image, gate on activation, roll back
667
- to the retained previous image if activation fails), roll it back to that
668
- retained image, or read its live container and retained generations:
669
-
670
- ```bash
671
- rollbridge compose-roll --config rollbridge.js --unit gateway
672
- rollbridge compose-roll --config rollbridge.js --unit gateway --image registry.local/gateway:2.4.0
673
- rollbridge compose-rollback --config rollbridge.js --unit gateway
674
- rollbridge compose-status --config rollbridge.js --unit gateway
675
- ```
676
-
677
- Each prints the roll result JSON and exits non-zero when `status` is `failed`. A
678
- drain or in-flight timeout fails the roll **before** anything is recreated, so
679
- the current container keeps serving.
680
-
681
628
  Restart non-proxied processes in place — all of them, one by id, or a policy
682
629
  group (the proxied process is never restarted; use `deploy` for that):
683
630
 
package/TODO.md CHANGED
@@ -39,7 +39,6 @@ This roadmap tracks planned Rollbridge features and documentation. Rollbridge sh
39
39
  - [x] Implement opt-in durable guardian recovery for exact same-authority daemon process replacement without stopping retained generations.
40
40
  - [x] Implement guardian-fenced atomic incompatible owner/config/control-socket/package/runtime replacement without stopping retained generations.
41
41
  - [x] Add the authenticated one-time disruptive bridge for pre-replacement guardians; all protocol-capable replacements remain atomic.
42
- - [x] Activate and traffic-switch a healthy candidate before asynchronously starting bounded old-generation retirement.
43
42
  - [x] Replicas and stable worker indexes. (Supported on port-less `companion` processes; `proxied`/`singleton`/ported processes stay single.)
44
43
  - [x] Allow one process config to start multiple replicas (`replicas`, companion-only for now).
45
44
  - [x] Expose `ROLLBRIDGE_REPLICA_INDEX`, replica count, and per-replica template context (`{{replicaIndex}}`/`{{replicaCount}}`).
@@ -72,18 +71,6 @@ This roadmap tracks planned Rollbridge features and documentation. Rollbridge sh
72
71
  - [x] Validate unsupported lifecycle-hook combinations once worker lifecycle hooks land. (`lifecycle.drainCommand` requires a positive `drainTimeoutMs`; `nonBlockingDrain` is companion-only; a `lifecycle.stopCommand` may not be combined with a custom `stopSignal`, since the command runs instead of the signal.)
73
72
  - [x] Include example fixes in validation output.
74
73
 
75
- ## Compose service units
76
-
77
- - [x] Model a Docker Compose service as a Rollbridge-managed unit (`compose[]`): project directory, service name, target image, `stop_grace_period`, drain command and timeout, in-flight gate, and activation probe (Compose healthcheck plus optional ready command/URL).
78
- - [x] Roll one unit drain-first — quiesce, wait for accepted work to settle, recreate only that service, gate on activation — instead of a raw `docker compose up -d --force-recreate` (`rollbridge compose-roll`, `compose-rollback`, `compose-status`).
79
- - [x] Retain the replaced image in the generation ledger (`composeStatePath`) so the previous tag stays selectable and rollback is a first-class operation, not a manual compose invocation.
80
- - [x] Keep the proxied-entry-point guarantee: a unit whose container publishes the Rollbridge proxy port is refused, and a roll never runs while a release generation transition is unresolved.
81
- - [x] Define drain and in-flight timeout behaviour: both fail the roll before anything is recreated, so the current container keeps serving; an activation failure recreates the previous image and reports `rolled_back`.
82
- - [x] Operator runbook for the Socketduct hub, one gateway leg, and one builder DinD (`docs/compose-services.md`).
83
- - [ ] Ship the quiescence hook (`appctl drain`) and an in-flight probe on the hub, gateway, and builder DinD images. Until they exist, the in-flight gate is the only protection work already admitted has, and a roll is better than `up -d --force-recreate` but not a true drain.
84
- - [ ] Switch TensorBuzz's Socketduct provisioning path (`backend/src/services/docker-servers/socketduct-gateway-provisioning-service.js`) from raw `up -d --force-recreate` to `rollbridge compose-roll`. That is a TensorBuzz change, not a Rollbridge one.
85
- - [ ] Prove a drain-first roll on a real builder host. The current proof is a real Compose daemon (`test/compose-docker.test.js`, opt-in), not a production builder.
86
-
87
74
  ## Minor Features
88
75
 
89
76
  - [x] Add a control-socket permission option (`control.mode`) for shared deploy users.
package/docs/cli.md CHANGED
@@ -116,14 +116,12 @@ the candidate binds and validates its listeners, and an authenticated fenced
116
116
  transaction commits guardian authority and the final control socket. A lost
117
117
  control response is accepted only when the guardian confirms the exact committed
118
118
  transaction id. The first authenticated upgrade from a genuine pre-replacement
119
- guardian/daemon, from a retained prepare/stage guardian that explicitly lacks
120
- the retired-owner commit capability, or from a retained guardian whose rollbridge
121
- version differs from the candidate runtime, is the exception: `ensure-daemon` preserves its exact
119
+ guardian/daemon, or from a retained prepare/stage guardian that explicitly lacks
120
+ the retired-owner commit capability, is the sole exception: `ensure-daemon` preserves its exact
122
121
  guardian-owned processes but deliberately retires the old listeners before the
123
122
  Node 20 candidate binds, so existing proxy/control connections may close.
124
123
  Successful status JSON records this as `ownerTransition.disruptive: true` and
125
- `ownerTransition.mode: "legacy-first-upgrade"` for a protocol gap or
126
- `"guardian-version-handoff"` for a version drift. This bridge requires
124
+ `ownerTransition.mode: "legacy-first-upgrade"`. This one-time bridge requires
127
125
  the incumbent config identity unchanged; make config/socket changes in a second,
128
126
  atomic invocation. Auth, transport, malformed-response, arbitrary unknown-command,
129
127
  and authority failures do not qualify and fail before any deploy is sent.
@@ -165,25 +163,6 @@ for old workers, jobs, or HTTP/WebSocket connections to finish. Prints
165
163
  If the new release fails to start or health-check, the previous release stays
166
164
  active and the command errors.
167
165
 
168
- A deploy is also refused **before any of the candidate's processes start** when the
169
- handoff would push the number of concurrently draining release groups past
170
- [`releaseRetention.maxConcurrentDrainingReleaseGroups`](config.md#releaseretention) —
171
- the incumbent the handoff is about to retire counts too. The incumbent keeps serving,
172
- nothing of the candidate is started, and the error names every draining group with its
173
- `progressing`/`timedOut`/`wedged` classification and the bound it exceeded.
174
-
175
- The same pre-activation refusal applies when `--release-id` names a release Rollbridge
176
- still **retains** while that generation holds live work: draining with processes alive, or
177
- recorded drained while a process group outlived its tracked process
178
- (`processes[].heldProcessGroup`). Starting a second generation under that id re-registers
179
- the guardian process keys the retained generation still holds, so the deploy aborts
180
- partway through startup with `Guardian provenance mismatch for release:<id>:<key>` — the
181
- shape a failed-deploy restore produced twice on 2026-11-15 (`20261115195051` at 20:29:55Z
182
- and again at 20:41:26Z, 12 minutes apart). The refusal names the retained state, each held
183
- process, and the release the guardian reports as active, and leaves the retained generation
184
- untouched. A generation whose processes are genuinely gone stays re-deployable: the
185
- 2026-10-25 carve-out for re-registering a dead generation is unchanged.
186
-
187
166
  With an opt-in handoff-service `lifecycle.activateCommand`, deploy journals the
188
167
  exact transition, waits for old retirement acknowledgement, waits for candidate
189
168
  activation acknowledgement, and synchronously commits the active proxy target.
@@ -264,21 +243,6 @@ Errors when there is no previous release, the `--release-id` is not a retained
264
243
  release, or the target is already active. Only releases Rollbridge still retains
265
244
  (see [`releaseRetention`](config.md#releaseretention)) can be rolled back to.
266
245
 
267
- It also refuses a target whose generation is recorded as drained while its
268
- processes still hold live work — a process the retirement wait left alive, or a
269
- process group that outlived the tracked process (`processes[].heldProcessGroup`,
270
- see [`status`](#status)). Re-deploying that release id would re-register its
271
- guardian process keys under a new provenance while the old processes still hold
272
- them, and the guardian answers `Guardian provenance mismatch for
273
- release:<id>:<key>` after the deploy has already begun. The refusal is raised
274
- before anything is touched and names each held process together with the release
275
- the guardian reports as active, so the deploy tool reconciles its own `current`
276
- pointer with `rollbridge status` instead of restoring a generation that already
277
- drained. A target that is still **draining** is not refused here: stopping it is
278
- what makes its id reusable, and the deploy it then runs applies the same fence
279
- after that stop — so a stop that leaves the generation's process group alive is
280
- reported as the refusal above instead of colliding mid-startup.
281
-
282
246
  **Migration constraints.** Rollback only manages processes — it does **not**
283
247
  revert database migrations or other external state. The target release's on-disk
284
248
  directory must still exist, and its code must be compatible with the current
@@ -286,65 +250,6 @@ schema. Keep migrations backwards-compatible (the same rule that lets old and
286
250
  new releases overlap during a deploy) so rolling code back to a retained release
287
251
  stays safe.
288
252
 
289
- ## `compose-roll`
290
-
291
- ```
292
- rollbridge compose-roll [--config <path>] --unit <id> [--image <reference>]
293
- ```
294
-
295
- Rolls one declared Compose unit (see [`compose[]`](config.md#compose)) with the
296
- same handshake a process deploy uses: quiesce it, wait for the work it already
297
- accepted to settle, recreate the container with the target image, gate on
298
- activation, and roll back to the retained previous image when activation fails.
299
- `--image` overrides the unit's configured `image` for this roll.
300
-
301
- Prints the roll result JSON and exits non-zero when `status` is `failed`:
302
-
303
- ```json
304
- {"unitId": "gateway", "releaseId": "20261104104321888", "status": "rolled", "stage": "activation", "image": "registry.local/gateway:2.4.0", "previousImage": "registry.local/gateway:2.3.1", "containerId": "ba108432e759", "generations": [...], "rollback": null, "error": null}
305
- ```
306
-
307
- `stage` names the stage that ended the roll: `drain`, `in_flight`, `recreate`,
308
- `activation`, `rollback`, or `rollback_failed`. A `drain` or `in_flight` failure
309
- means the container was **never recreated** — the old one keeps serving. An
310
- `activation` failure recreates the previous image and reports `status:
311
- "rolled_back"` with `rollback.image`; if that rollback cannot activate either,
312
- the result is `stage: "rollback_failed"` and a non-zero exit.
313
-
314
- Errors (exit non-zero before any change) when the unit is not declared in the
315
- config, its image reference is unpinned or malformed, a release generation
316
- transition is unresolved (`resume the exact deploy first`), or its container
317
- publishes the Rollbridge proxy port — a Compose unit must never replace the
318
- proxied entry point in place.
319
-
320
- ## `compose-rollback`
321
-
322
- ```
323
- rollbridge compose-rollback [--config <path>] --unit <id> [--image <reference>]
324
- ```
325
-
326
- Rolls a Compose unit back to an earlier image through the same drain-first path:
327
- with no `--image` it selects the image retained before the newest activated one
328
- in `composeStatePath`, so the tag `compose-roll` replaced is the target. Prints
329
- the same result JSON as `compose-roll` and exits non-zero when `status` is
330
- `failed`. A successful explicit rollback reports `status: "rolled"` (the unit
331
- rolled to the image you named); an automatic rollback after a failed activation
332
- reports `status: "rolled_back"`.
333
-
334
- Errors when no earlier image is retained for the unit — pass `--image
335
- <reference>` to roll back explicitly to a tag Rollbridge has never rolled to.
336
-
337
- ## `compose-status`
338
-
339
- ```
340
- rollbridge compose-status [--config <path>] --unit <id>
341
- ```
342
-
343
- Prints the unit's live container (`containerId`, `state`, `healthStatus`,
344
- `image`, published `ports`) and every retained generation (`image`,
345
- `previousImage`, `status`, `releaseId`, `containerId`, `activatedAt`), plus the
346
- `ledgerPath` they are stored in. Read-only: it never changes the container.
347
-
348
253
  ## `status`
349
254
 
350
255
  ```
@@ -359,36 +264,6 @@ Memory-supervised processes also report `rssBytes`, `memoryRestarts`,
359
264
  `lastMemoryRestartAt`, and `children` (the process tree: each group member's
360
265
  `pid`, `command`, and `rssBytes`).
361
266
 
362
- Every process also reports `retirementTimedOut` — `true` once the supervisor has
363
- stopped waiting for it — and `heldProcessGroup` (`{pgid, liveMemberCount}`) when a
364
- bounded stop left the process group running after the tracked process exited. That
365
- group is the leak a `stopped` record otherwise hides: nothing in Rollbridge can
366
- signal or wait for it again, yet it still holds the generation's work. It is
367
- re-checked at most once every 5 seconds, and clears itself when its members leave.
368
-
369
- Each **draining** release reports a `drain` classification — `status`, the `reason`
370
- behind it, the `sampleWindowMs` it rests on, and `cpuDeltaMs`:
371
-
372
- - `progressing` — the generation still owns live connections inside
373
- `proxy.drainTimeoutMs`, a process is still inside its retirement budget, or its
374
- process-group CPU advanced across the sample window. A legitimate multi-hour drain reports this.
375
- - `timedOut` — the generation still owns live connections, but the configured drain budget has
376
- already expired: `drain budget 60000ms exceeded by 900000ms with 1 live connection still owned
377
- by the generation`. The connection wait was given up at `proxy.drainTimeoutMs`; the record says
378
- so instead of claiming progress for as long as one socket stays open. It is a report only —
379
- `timedOut` never authorises a signal, and escalation still requires `wedged`.
380
- - `wedged` — the drain can no longer progress: no live connection, every
381
- retirement budget expired, and either no runner child remains or the group's CPU
382
- is flat across a window of at least 20 seconds.
383
- - `unproven` — the evidence is inconclusive: a first observation, a window shorter
384
- than 20 seconds, or a host where process CPU cannot be read.
385
-
386
- `drain` is `undefined` for a release that is not draining. Gate on this
387
- classification rather than on a raw process count: a `progressing` group holding
388
- three live processes is a healthy drain, a `wedged` group holding one is a
389
- generation that will never finish. Observations are taken at most once per window
390
- per draining group, so repeated `status` reads stay cheap.
391
-
392
267
  `--no-logs` omits only the captured `logs` array from release, service, and
393
268
  singleton process statuses. Use it for bounded machine-readable lifecycle
394
269
  attestations that do not need process output. The equivalent control request is
package/docs/config.md CHANGED
@@ -44,8 +44,6 @@ restart.
44
44
  | Field | Type | Default | Description |
45
45
  | --- | --- | --- | --- |
46
46
  | `application` | string | basename of the config file's directory | Names the app; used in the default control-socket path and the `ROLLBRIDGE_APPLICATION` env var. |
47
- | `compose` | array | unset (no Compose units) | Docker Compose services Rollbridge rolls drain-first (see [`compose[]`](#compose)). |
48
- | `composeStatePath` | string | `rollbridge-<application>.compose.state.json` next to the config file | File that retains each Compose unit's image generations; relative paths resolve from the config file directory. |
49
47
  | `control` | object | — | Control-socket settings (see below). |
50
48
  | `legacyTakeover` | object | unset | Optional matchers for `rollbridge predeploy-cleanup` to stop pre-Rollbridge supervisors during first handover (see below). |
51
49
  | `ownerRecovery` | object | unset | Durable guardian recovery and atomic incompatible owner replacement; requires `statePath` (see below). |
@@ -77,9 +75,8 @@ to let a deploy group talk to the daemon.
77
75
  | `proxy.port` | number | `8182` | Stable port Nginx (or another front end) points at. |
78
76
  | `proxy.upstreamHost` | string | `proxy.host`, or `"127.0.0.1"` when `proxy.host` is `0.0.0.0`/`::` | Host Rollbridge uses for release health checks and proxy targets. |
79
77
  | `proxy.healthPath` | string | `"/ping"` | Default health-check path for proxied processes. |
80
- | `proxy.healthTimeoutMs` | number | `120000` | Default boot budget for proxied processes: how long a release may take to answer its health path for the first time. Measured on TensorBuzz production (server3, 2026-11-06): the web app was still initialising when a `30000` deadline expired, so two deploys were killed mid-boot; `120000` is the budget the same release publishes on today. Waiting for a `dependsOn` service is not charged to it. |
81
- | `proxy.drainTimeoutMs` | number | `60000` | How long to drain HTTP/WebSocket connections before stopping the retired proxied process. Expiry never stops an independently draining jobs generation. It is also the budget a `drain` classification measures elapsed drain against: a generation that still owns connections past it reports `timedOut` rather than `progressing` (see [`status` in the CLI guide](cli.md#status)). |
82
- | `proxy.nonBlockingDrainTimeoutMs` | number | `60000` | How long release retirement waits for an independently draining process before recording it as still draining and completing the release; `stop` and `rollback` hand the same bound to the processes they must not kill, so neither command can hold the owner-mutation fence for the whole drain. Expiry never sends `SIGKILL`; the process keeps its live work. |
78
+ | `proxy.healthTimeoutMs` | number | `30000` | Default health-check timeout for proxied processes. |
79
+ | `proxy.drainTimeoutMs` | number | `60000` | How long to drain HTTP/WebSocket connections before stopping the retired proxied process. Expiry never stops an independently draining jobs generation. |
83
80
  | `proxy.forceStopTimeoutMs` | number | `10000` | Default per-process graceful-stop timeout (`SIGTERM`, then `SIGKILL`). |
84
81
 
85
82
  ## `releaseRetention`
@@ -88,81 +85,10 @@ to let a deploy group talk to the daemon.
88
85
  | --- | --- | --- | --- |
89
86
  | `releaseRetention.keep` | non-negative integer | `10` | Number of most-recent **stopped** releases the daemon keeps in memory and reports in `status`. |
90
87
  | `releaseRetention.maxAgeMs` | non-negative number | `0` (disabled) | Also prune stopped releases older than this many milliseconds. |
91
- | `releaseRetention.maxConcurrentDrainingReleaseGroups` | non-negative integer | `12` | How many release groups may drain at once. A deploy that would push the count past this bound is **refused before activation** (see below); `0` disables the check. |
92
- | `releaseRetention.escalateWedgedGenerations` | boolean | `false` | Lets the generation supervisor `SIGKILL` a process group it has **proven** wedged and holding nothing. Off by default: the wedge is reported with its pids instead of terminated. See the escalation below. |
93
88
 
94
89
  Active and draining releases are never pruned. This governs Rollbridge's own
95
90
  release records; the deploy tool still owns on-disk release directories.
96
91
 
97
- ### The draining-generation bound
98
-
99
- Activating a new generation retires the incumbent, so every deploy adds one draining
100
- group. When a drain outlasts the deploy cadence, that backlog grows without limit: the
101
- measured TensorBuzz case had 16 releases draining in 18–114 minutes (median ~37 minutes)
102
- against deploys every 3–7 minutes, and each further deploy added another group that its own
103
- post-activation check then reported as a failure.
104
-
105
- `deploy` therefore counts the groups already draining plus the incumbent it is about to
106
- retire, and refuses before starting any process of the candidate:
107
-
108
- ```text
109
- Deploy refused before activation: 13 release groups already draining (20261115063805 wedged
110
- (1 live process past its retirement budget with no runner child remaining); …). Activating
111
- 20261115073717 would make 14, above the releaseRetention.maxConcurrentDrainingReleaseGroups
112
- bound of 12.
113
- ```
114
-
115
- The refusal is a deploy failure like any other: the incumbent keeps serving, nothing of the
116
- candidate is started, and the named reason says which groups are still holding the fleet and
117
- whether each is `progressing`, `timedOut`, or `wedged` (see [`status` in the CLI
118
- guide](cli.md#status)). A refused deploy is retried once a drain finishes; the bound is a
119
- policy, not a queue.
120
-
121
- ### The wedged-generation escalation
122
-
123
- A bounded retirement wait (see [`proxy.nonBlockingDrainTimeoutMs`](#proxy)) makes the release
124
- finish, but it deliberately leaves the wedged process alive: `gracefulStopMs: "indefinite"`
125
- exists precisely because a stop-timeout `SIGKILL` there aborts live build lanes (measured in
126
- STOE-1077: 67 builds, 27.3 build-hours). The residue is what then blocks the fleet — the
127
- process group still holds the generation, and a consumer that requires one live generation per
128
- role refuses every later deploy until an operator signals the exact PIDs.
129
-
130
- `releaseRetention.escalateWedgedGenerations: true` hands that decision back to the supervisor.
131
- It is **off by default**, because only the operator knows whether their worker's live work sits
132
- in forked runners (kill-safe to prove) or inside the process itself (not provable from procfs).
133
-
134
- The escalation is not a timer, a poller, or a watchdog. It runs at two moments that already
135
- exist: at the end of a retirement transition, bracketing the wait that transition already
136
- performed, and at the start of a `deploy`, reusing the liveness observation the admission bound
137
- already takes. A generation that stopped with a live process group keeps its observations, so
138
- the next deploy can prove the residue without sampling anything new.
139
-
140
- A `SIGKILL` is delivered only when **all four** proofs hold, each named in the log line:
141
-
142
- 1. The generation is classified `wedged` — it owns no live connection.
143
- 2. Every live process already expired its retirement budget.
144
- 3. **No runner child remains** in any tracked process group. This is the STOE-1077 boundary: a
145
- live lane is never aborted, and a wedge that still holds one is reported instead of killed.
146
- 4. The group's CPU is **flat across a full 20 s sample window**, so a group that is merely slow
147
- is never mistaken for a wedged one.
148
-
149
- Anything else reports. A refused wedge is logged with the pids and process groups it is holding
150
- — the information an operator needs in place of guessing — and the deploy proceeds to the
151
- admission bound above:
152
-
153
- ```text
154
- release wedged generation left alive {escalation: refused, pids: [4242], processIds: ["worker"],
155
- reason: "no escalation: 1 runner child is still live in the generation; SIGKILL would abort
156
- in-flight work", releaseId: "20261115164249", trigger: "deploy of 20261115202720"}
157
- ```
158
-
159
- When the proofs do hold, the supervisor terminates the group and says so:
160
-
161
- ```text
162
- process retirement escalated to SIGKILL {id: "worker", pid: 4242, reason: "release retirement: escalating: …"}
163
- release wedged generation escalated {escalation: SIGKILL, processIds: ["worker"], releaseId: "20261115164249"}
164
- ```
165
-
166
92
  ## `statePath`
167
93
 
168
94
  When set, the daemon persists a sanitized operational state snapshot — the active and
@@ -242,16 +168,12 @@ The first upgrade from an authenticated pre-replacement Rollbridge guardian and
242
168
  daemon uses an explicitly disruptive compatibility bridge because that legacy
243
169
  owner cannot transfer listeners on Node 20. A retained guardian that supports
244
170
  replacement prepare/stage but lacks the retired-owner commit command is explicitly
245
- classified and uses the same bridge, and so is a retained guardian whose rollbridge
246
- version differs from the daemon runtime: the candidate starts a coordinator of its
247
- own version, the retained guardian keeps the processes it already owns, and nothing
248
- waits for those to drain. Rollbridge attests the exact guardian and
171
+ classified and uses the same bridge. Rollbridge attests the exact guardian and
249
172
  daemon processes, sockets, runtime/config authority, and durable process
250
173
  registrations before retiring the legacy listeners. Managed process PIDs and
251
174
  release state remain supervised, but live proxy/control connections may close.
252
175
  The resulting status includes `ownerTransition: {disruptive: true, mode:
253
- "legacy-first-upgrade", ...}` for a protocol gap or `{disruptive: true, mode:
254
- "guardian-version-handoff", ...}` for a version drift. The bridge only accepts the incumbent config
176
+ "legacy-first-upgrade", ...}`. The bridge only accepts the incumbent config
255
177
  identity; retry config or socket changes after the protocol upgrade, when the
256
178
  normal atomic handoff applies. Other guardian/auth/transport/identity failures
257
179
  remain fail-closed.
@@ -309,8 +231,6 @@ legacyTakeover: {
309
231
  | `env` | object of string → string | `{}` | Extra environment variables (values templated). Merged over the injected `ROLLBRIDGE_*` vars. |
310
232
  | `port` | number or `{from, to}` | unset | Port (or range) allocated per release. **Required for the `proxied` process.** A plain number `n` means the fixed port `n` (`{from: n, to: n}`). |
311
233
  | `health` | object or `false` | enabled with defaults | Health check for the `proxied` process; set `false` to disable (see below). |
312
- | `dependsOn` | array of process ids | `[]` | Processes whose allocated port must accept a connection before Rollbridge starts this process (see [`processes[].dependsOn`](#processesdependson)). |
313
- | `dependencyTimeoutMs` | positive number | `30000` | Budget for each `dependsOn` wait. Charged to its own clock, never to `health.timeoutMs`. |
314
234
  | `stopSignal` | signal name (e.g. `"SIGTERM"`, `"SIGINT"`, `"SIGQUIT"`) | `"SIGTERM"` | Signal sent to gracefully stop the process; after `gracefulStopMs` it is `SIGKILL`ed. Use a worker's quit signal so it finishes in-flight work before exiting. |
315
235
  | `nonBlockingDrain` | boolean | `false` | When a release is retired, drain this process **immediately** (in parallel with the proxied connection drain) instead of after it. Companion processes only — typically background workers (see below). |
316
236
  | `lifecycle` | object | no hooks | Command hooks run when gracefully stopping the process (see below). |
@@ -521,97 +441,10 @@ release). Set `health: false` to disable it.
521
441
  | Field | Type | Default | Description |
522
442
  | --- | --- | --- | --- |
523
443
  | `health.path` | string | `proxy.healthPath` | HTTP path probed on the process's port. |
524
- | `health.timeoutMs` | number | `proxy.healthTimeoutMs` | Total time to wait for the first healthy response, measured from the process launch. Only the app's own boot is charged to it — a `dependsOn` wait runs on `dependencyTimeoutMs` before this window opens. |
444
+ | `health.timeoutMs` | number | `proxy.healthTimeoutMs` | Total time to wait for the first healthy response. |
525
445
  | `health.intervalMs` | number | `250` | Delay between probes. |
526
446
  | `health.startDelayMs` | non-negative number | `0` | Wait this long after the process starts before the first probe (runs before the `timeoutMs` window). |
527
447
 
528
- ### `processes[].dependsOn`
529
-
530
- A release-scoped process that talks to a same-release service usually waits for
531
- that service inside its own command (`wait-port … && exec <app>`). That wait
532
- runs after Rollbridge has started the process, so it is charged against
533
- `health.timeoutMs`: a dependency that takes 12 seconds to bind leaves the app 18
534
- seconds of a 30-second health deadline, and the deploy fails while the app is
535
- still booting normally.
536
-
537
- Declare the dependency instead, and Rollbridge waits for each `dependsOn`
538
- process's allocated port **before** spawning the dependent, on a budget of its
539
- own (`dependencyTimeoutMs`):
540
-
541
- ```js
542
- {
543
- id: "web",
544
- policy: "proxied",
545
- dependsOn: ["background-jobs-main"],
546
- dependencyTimeoutMs: 30000,
547
- command: "exec node build/bin/velocious.js server --port {{port}}",
548
- port: {from: 14500, to: 14599},
549
- health: {path: "/ping", timeoutMs: 120000}
550
- }
551
- ```
552
-
553
- `health.timeoutMs` then measures only the app's own boot. Both waits are logged
554
- separately: `release process dependency wait` / `release process dependency
555
- ready` (with `attempts` and `elapsedMs`) and `release candidate health ready`
556
- (with `bootMs`, `healthGateMs`, and `waitedForDependenciesMs`).
557
-
558
- `dependsOn` accepts ids of processes Rollbridge starts before this one —
559
- persistent services, handoff services, companions, and the proxied process. A
560
- dependency must have a `port` (there is nothing to wait for otherwise), may not
561
- be a `singleton` (singleton ownership is not per-release), and may not be this
562
- process itself. Dependencies that Rollbridge would start after this process are
563
- rejected, which also rules out cycles.
564
-
565
- ## `compose[]`
566
-
567
- Optional Docker Compose services this app updates. Each entry is a unit Rollbridge
568
- rolls drain-first — quiesce, wait for in-flight work, recreate with the target
569
- image, gate on activation, roll back on failure — instead of a raw
570
- `docker compose up -d --force-recreate`. See
571
- [`docs/compose-services.md`](compose-services.md) for the stage-by-stage behavior
572
- and the operator commands.
573
-
574
- | Field | Type | Default | Description |
575
- | --- | --- | --- | --- |
576
- | `compose[].id` | string | **required** | Names the unit for `compose-roll`/`compose-rollback`/`compose-status`. Unique across units and not equal to any process id. |
577
- | `compose[].service` | string | **required** | Service name inside the Compose file. |
578
- | `compose[].projectDirectory` | string | **required** | Directory holding the Compose file; every Compose command runs here. |
579
- | `compose[].image` | string | **required** | Target image reference for `compose-roll` (a tag or a `@sha256:` digest; never a bare image name). |
580
- | `compose[].composeFile` | string | `"compose.yml"` | Compose file relative to `projectDirectory`. |
581
- | `compose[].projectName` | string | unset (Compose default) | Passed as `--project-name`, so a project whose services come from several files stays addressable. |
582
- | `compose[].envFile` | string | unset | Passed as `--env-file`, relative to `projectDirectory`. |
583
- | `compose[].stopGracePeriodMs` | non-negative number | `10000` | Written into the roll's override as the service's `stop_grace_period`: how long Compose waits for the old container to exit. |
584
- | `compose[].recreateTimeoutMs` | non-negative number | `120000` | Timeout for the `compose up -d --no-deps --force-recreate` the roll runs. |
585
- | `compose[].drain.command` | string | unset | Hook that stops the unit admitting new work. Runs through a shell; unset means the unit has no quiescence hook and the roll proceeds to the in-flight gate. |
586
- | `compose[].drain.timeoutMs` | positive number | `60000` | Required when `drain.command` is set. Expiry fails the roll at `stage: "drain"` without recreating the container. |
587
- | `compose[].inFlight.command` | string | unset | Gate that exits 0 only when the work the unit already accepted has settled. Polled until it exits 0 or `inFlight.timeoutMs` expires. |
588
- | `compose[].inFlight.intervalMs` | positive number | `1000` | Delay between gate attempts. |
589
- | `compose[].inFlight.timeoutMs` | positive number | `0` (no gate) | Required when `inFlight.command` is set. Expiry fails the roll at `stage: "in_flight"` without recreating the container. |
590
- | `compose[].activation.healthcheck` | boolean | `true` | Require the recreated container's Compose healthcheck to report `healthy` (containers without a `HEALTHCHECK` pass). |
591
- | `compose[].activation.readyCommand` | string | unset | Additional readiness probe run through a shell; must exit 0. |
592
- | `compose[].activation.readyUrl` | string | unset | Additional readiness probe fetched over HTTP; any 2xx passes. |
593
- | `compose[].activation.startDelayMs` | non-negative number | `0` | Wait before the first probe (runs before the `timeoutMs` window). |
594
- | `compose[].activation.intervalMs` | positive number | `1000` | Delay between probes. |
595
- | `compose[].activation.timeoutMs` | positive number | `60000` | How long the activation gate may keep probing before the roll rolls back. |
596
- | `compose[].rollback.enabled` | boolean | `true` | Roll back to the retained previous image when activation fails. When `false`, a failed activation is reported as a failed roll. |
597
-
598
- `composeStatePath` (top-level) is where each unit's generations are retained, so
599
- `compose-rollback` has a previous tag to select. It defaults to
600
- `rollbridge-<application>.compose.state.json` next to the config file.
601
-
602
- Drain, in-flight, and readiness-command hooks support `{{...}}` placeholders:
603
-
604
- | Placeholder | Value |
605
- | --- | --- |
606
- | `{{application}}` | `application` |
607
- | `{{unitId}}` | This unit's `id`. |
608
- | `{{service}}` | This unit's `service`. |
609
- | `{{composeArgs}}` | The Compose global arguments for this unit (`docker compose`, its `--project-name`, `-f <file>`, `--env-file`), so a hook can run `{{composeArgs}} exec -T <service> ...`. |
610
- | `{{composeFile}}` | This unit's `composeFile`. |
611
- | `{{projectDirectory}}` | This unit's `projectDirectory`. |
612
- | `{{releaseId}}` | This roll's release id. |
613
- | `{{env.<NAME>}}` | A variable from the Rollbridge process environment. |
614
-
615
448
  ## Template variables
616
449
 
617
450
  `command`, `cwd`, and `env` values support `{{...}}` placeholders, rendered when
@@ -657,14 +490,10 @@ Rollbridge sets these in every managed process's environment (the process's own
657
490
  - `port` must be a positive port number or an ascending `{from, to}` range.
658
491
  - `control.mode` must be an octal mode between `0` and `0o777`.
659
492
  - `control.owner` and `control.group` must each be a non-negative integer id or a non-empty name (resolved at daemon start).
660
- - `outputLines` and `releaseRetention.keep` must be positive/non-negative integers; `health.startDelayMs`, `releaseRetention.maxAgeMs`, and `releaseRetention.maxConcurrentDrainingReleaseGroups` must be non-negative numbers.
493
+ - `outputLines` and `releaseRetention.keep` must be positive/non-negative integers; `health.startDelayMs` and `releaseRetention.maxAgeMs` must be non-negative numbers.
661
494
  - `restart.maxRestarts` must be a non-negative integer (omit it for unlimited restarts); `restart.backoffFactor` must be a number ≥ 1; `restart.windowMs` and `restart.maxDelayMs` must be non-negative numbers.
662
495
  - When `memory` is set, `memory.limitBytes` must be a positive integer, `memory.warnBytes` a non-negative integer, and `memory.checkIntervalMs` a positive number.
663
496
  - `replicas` must be a positive integer; `replicas > 1` is allowed only on a `companion` process without a `port`. Process ids must not contain `#` (reserved for replica instance ids).
664
497
  - `lifecycle.activateCommand`/`quietCommand`/`drainCommand`/`stopCommand` must be non-empty strings when set, and `lifecycle.drainTimeoutMs` a non-negative number; `lifecycle.drainCommand` requires a positive `lifecycle.drainTimeoutMs`. `activateCommand` is allowed on at most one handoff service, requires that service's `quietCommand`, and requires `statePath` plus `ownerRecovery`. A `lifecycle.stopCommand` may not be combined with a custom `stopSignal` (the `stopCommand` runs instead of the signal, so the signal would be ignored).
665
498
  - `nonBlockingDrain` must be a boolean, and is allowed only on a `companion` process.
666
- - `dependsOn` must be an array of configured process ids: not the process itself, not a `singleton`, not a process without a `port`, and not a process Rollbridge starts after it. `dependencyTimeoutMs` must be a positive number.
667
499
  - `statePath` must be a string when set.
668
- - `compose` must be a non-empty array of unit mappings. Each unit needs `id`, `service`, `projectDirectory`, and a valid `image` reference (a tag, or a `@sha256:` digest with 64 hex characters; no whitespace, no leading `-`).
669
- - Compose unit `id`s must be unique and must not collide with a process `id`.
670
- - `compose[].drain.command` requires a positive `compose[].drain.timeoutMs`; `compose[].inFlight.command` requires a positive `compose[].inFlight.timeoutMs`.