rollbridge 0.1.73 → 0.1.75
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +63 -10
- package/TODO.md +13 -0
- package/changelog.d/20261025-guardian-re-register-dead-generation.md +13 -0
- package/changelog.d/20261026-stale-committed-generation-transition-recovery.md +1 -0
- package/changelog.d/20261104-compose-service-units.md +21 -0
- package/changelog.d/20261105-owner-mutation-lease-diagnostics.md +20 -0
- package/changelog.d/20261106-release-dependency-boot-budgets.md +32 -0
- package/changelog.d/20261109-path-filtered-builds.md +11 -0
- package/changelog.d/20261110-default-boot-budget-above-measured-startup.md +26 -0
- package/changelog.d/20261111-contention-sensitive-daemon-specs.md +27 -0
- package/changelog.d/20261113-bounded-nonblocking-retirement.md +5 -0
- package/changelog.d/20261115-bound-draining-release-groups.md +62 -0
- package/changelog.d/20261115-restore-deploy-refuses-held-generation.md +27 -0
- package/changelog.d/20261115-wedged-generation-escalation.md +44 -0
- package/changelog.d/20261116-bounded-explicit-stop-retirement.md +30 -0
- package/changelog.d/20261116-config-json-canonical-shape.md +11 -0
- package/changelog.d/20261116-drain-specs-gate-the-retirement-record.md +29 -0
- package/changelog.d/20261116-guardian-version-handoff.md +37 -0
- package/changelog.d/20261116-reclaim-dead-owner-mutation.md +7 -0
- package/changelog.d/20261117-activate-before-incumbent-retirement.md +1 -0
- package/changelog.d/20261117-activation-path-drain-deadline.md +35 -0
- package/changelog.d/20261117-stop-companion-after-retired-dependency.md +7 -0
- package/changelog.d/20261118-bounded-control-client-close.md +4 -0
- package/changelog.d/20261118-incumbent-retirement-authority-guard.md +1 -0
- package/changelog.d/20261118-owner-retirement-no-guardian-bound.md +19 -0
- package/changelog.d/20261118-shutdown-retirement-bound-restored.md +31 -0
- package/docs/cli.md +128 -3
- package/docs/compose-services.md +222 -0
- package/docs/config.md +177 -6
- package/docs/troubleshooting.md +97 -6
- package/docs/velocious.md +1 -1
- package/docs/workers.md +11 -0
- package/examples/tensorbuzz.com.js +3 -2
- package/package.json +1 -1
- package/src/cli.js +69 -3
- package/src/compose-unit.js +725 -0
- package/src/config.js +346 -8
- package/src/control-client.js +32 -5
- package/src/daemon.js +448 -76
- package/src/doctor.js +74 -0
- package/src/drain-policy.js +252 -0
- package/src/guardian-client.js +92 -9
- package/src/health.js +99 -8
- package/src/managed-process.js +197 -16
- package/src/process-guardian.js +212 -22
- package/src/process-memory.js +62 -2
- package/src/process-order.js +26 -0
- package/src/release-group.js +452 -39
- package/tensorbuzz.yml +20 -1
- package/test/completion.test.js +5 -2
- package/test/compose-config.test.js +160 -0
- package/test/compose-docker.test.js +297 -0
- package/test/compose-unit.test.js +415 -0
- package/test/config-validation.test.js +135 -5
- package/test/control-protocol.test.js +22 -0
- package/test/drain-policy.test.js +328 -0
- package/test/fixtures/deferred-boot-app.js +106 -0
- package/test/fixtures/gated-app.js +50 -0
- package/test/guardian-client.test.js +149 -5
- package/test/health.test.js +74 -1
- package/test/managed-process.test.js +260 -1
- package/test/owner-recovery.test.js +151 -20
- package/test/owner-replacement.test.js +176 -15
- package/test/process-memory.test.js +47 -1
- package/test/release-group.test.js +447 -2
- package/test/rollbridge.test.js +774 -72
- package/test/shutdown-completion.test.js +119 -0
- package/test/shutdown-retirement-bound.test.js +162 -0
- package/test/support/process.js +3 -1
- package/test/tensorbuzz-config.test.js +228 -0
package/README.md
CHANGED
|
@@ -57,7 +57,7 @@ export default {
|
|
|
57
57
|
port: 8182,
|
|
58
58
|
upstreamHost: "127.0.0.1",
|
|
59
59
|
healthPath: "/ping",
|
|
60
|
-
healthTimeoutMs:
|
|
60
|
+
healthTimeoutMs: 120000,
|
|
61
61
|
drainTimeoutMs: 60000,
|
|
62
62
|
forceStopTimeoutMs: 10000
|
|
63
63
|
},
|
|
@@ -183,8 +183,9 @@ handoffs and exits only after the worker pool has drained.
|
|
|
183
183
|
|
|
184
184
|
For a coordinator that starts quiescent, give the handoff jobs-main paired
|
|
185
185
|
`lifecycle.activateCommand` and `quietCommand` hooks. After candidate health,
|
|
186
|
-
Rollbridge
|
|
187
|
-
|
|
186
|
+
Rollbridge activates the candidate, commits the active proxy target, and only
|
|
187
|
+
then starts the old generation's bounded retirement asynchronously. The durable
|
|
188
|
+
transition is
|
|
188
189
|
generation-scoped and resumable; failures remain visible and block unrelated
|
|
189
190
|
deploys. Post-commit singleton replacement is also journaled and must complete
|
|
190
191
|
before an exact retry reports success. If the active coordinator restarts,
|
|
@@ -255,6 +256,14 @@ supervisor. The upgrade fails before handoff and requires one explicit clean
|
|
|
255
256
|
remain atomic. A genuinely pre-replacement guardian still uses the separately
|
|
256
257
|
documented one-time disruptive compatibility bridge below.
|
|
257
258
|
|
|
259
|
+
A retained guardian that reports a different rollbridge version from the daemon
|
|
260
|
+
runtime is handed to a current-version coordinator through that same bridge, so
|
|
261
|
+
a package upgrade does not need a supervisor restart: the coordinator owns the
|
|
262
|
+
daemon and every process it starts, the retained guardian keeps the processes it
|
|
263
|
+
already owns, and nothing waits for those to drain. `status.ownerTransition`
|
|
264
|
+
reports `mode: "guardian-version-handoff"`. The refusal is the last resort and
|
|
265
|
+
names both versions plus the exact `shutdown`/`ensure-daemon` commands.
|
|
266
|
+
|
|
258
267
|
There is one explicit compatibility boundary: the first upgrade from a genuine
|
|
259
268
|
pre-owner-replacement Rollbridge guardian and daemon cannot share its listeners
|
|
260
269
|
on supported Node 20. The same boundary applies to a retained transition guardian
|
|
@@ -317,6 +326,34 @@ rendered when the process starts:
|
|
|
317
326
|
Referencing a placeholder with no value (including an unset `{{env.<NAME>}}`)
|
|
318
327
|
fails the process start with a clear error, so typos surface immediately.
|
|
319
328
|
|
|
329
|
+
### Docker Compose services
|
|
330
|
+
|
|
331
|
+
A delivery that also updates Docker Compose services (a Socketduct hub, a gateway
|
|
332
|
+
leg, a builder DinD) declares them in `compose`. Each unit is rolled with the
|
|
333
|
+
same handshake a process deploy uses — quiesce, wait for in-flight work, recreate
|
|
334
|
+
with the target image, gate on activation, roll back to the retained previous
|
|
335
|
+
image on failure — instead of a raw `docker compose up -d --force-recreate`:
|
|
336
|
+
|
|
337
|
+
```js
|
|
338
|
+
compose: [
|
|
339
|
+
{
|
|
340
|
+
id: "gateway",
|
|
341
|
+
service: "gateway",
|
|
342
|
+
projectDirectory: "/srv/socketduct/socketduct-builder-a-standard",
|
|
343
|
+
image: "registry.diestoeckels.de/socketduct/gateway:2.4.1",
|
|
344
|
+
stopGracePeriodMs: 45000,
|
|
345
|
+
drain: {command: "{{composeArgs}} exec -T gateway appctl drain", timeoutMs: 120000},
|
|
346
|
+
inFlight: {command: "{{composeArgs}} exec -T gateway appctl in-flight", intervalMs: 2000, timeoutMs: 900000},
|
|
347
|
+
activation: {startDelayMs: 2000, timeoutMs: 120000, readyUrl: "http://127.0.0.1:7421/ready"}
|
|
348
|
+
}
|
|
349
|
+
]
|
|
350
|
+
```
|
|
351
|
+
|
|
352
|
+
A drain or in-flight timeout fails the roll and leaves the current container
|
|
353
|
+
serving; a unit whose container publishes the Rollbridge proxy port is never
|
|
354
|
+
rolled. See [`docs/compose-services.md`](docs/compose-services.md) for the
|
|
355
|
+
stage-by-stage behavior and the fleet runbook.
|
|
356
|
+
|
|
320
357
|
Configuration examples live in `examples/`, including
|
|
321
358
|
`examples/tensorbuzz.com.js` for the current TensorBuzz backend deployment; see
|
|
322
359
|
[`docs/tensorbuzz-runbook.md`](docs/tensorbuzz-runbook.md) for the matching
|
|
@@ -429,14 +466,14 @@ On `rollbridge deploy`, the required ordering is:
|
|
|
429
466
|
1. starts any missing persistent service and the candidate's handoff services;
|
|
430
467
|
2. starts the new release's `companion`s, then its `proxied` process, and
|
|
431
468
|
health-checks the proxied process;
|
|
432
|
-
3. when `activateCommand` is configured,
|
|
433
|
-
|
|
434
|
-
4. synchronously switches new traffic to the new release
|
|
435
|
-
the existing switch-then-retire behavior);
|
|
469
|
+
3. when `activateCommand` is configured, activates and acknowledges the
|
|
470
|
+
candidate;
|
|
471
|
+
4. synchronously switches new traffic to the new release;
|
|
436
472
|
5. replaces `singleton`s (stops the old one, then starts the new one);
|
|
437
|
-
6.
|
|
438
|
-
|
|
439
|
-
|
|
473
|
+
6. starts the previous generation's bounded retirement asynchronously after the
|
|
474
|
+
traffic switch; deploy success never waits for quiescence or drain, and
|
|
475
|
+
Rollbridge supervises all retained drains in the background and reaps each
|
|
476
|
+
generation only after its handoffs and workers end.
|
|
440
477
|
|
|
441
478
|
If the new release fails to start or health-check, the previous release stays
|
|
442
479
|
active and any service started during this deploy is rolled back.
|
|
@@ -625,6 +662,22 @@ rollbridge rollback --config rollbridge.js # the previous relea
|
|
|
625
662
|
rollbridge rollback --config rollbridge.js --release-id v3
|
|
626
663
|
```
|
|
627
664
|
|
|
665
|
+
Roll one declared Docker Compose service drain-first (quiesce it, wait for its
|
|
666
|
+
in-flight work, recreate it with the target image, gate on activation, roll back
|
|
667
|
+
to the retained previous image if activation fails), roll it back to that
|
|
668
|
+
retained image, or read its live container and retained generations:
|
|
669
|
+
|
|
670
|
+
```bash
|
|
671
|
+
rollbridge compose-roll --config rollbridge.js --unit gateway
|
|
672
|
+
rollbridge compose-roll --config rollbridge.js --unit gateway --image registry.local/gateway:2.4.0
|
|
673
|
+
rollbridge compose-rollback --config rollbridge.js --unit gateway
|
|
674
|
+
rollbridge compose-status --config rollbridge.js --unit gateway
|
|
675
|
+
```
|
|
676
|
+
|
|
677
|
+
Each prints the roll result JSON and exits non-zero when `status` is `failed`. A
|
|
678
|
+
drain or in-flight timeout fails the roll **before** anything is recreated, so
|
|
679
|
+
the current container keeps serving.
|
|
680
|
+
|
|
628
681
|
Restart non-proxied processes in place — all of them, one by id, or a policy
|
|
629
682
|
group (the proxied process is never restarted; use `deploy` for that):
|
|
630
683
|
|
package/TODO.md
CHANGED
|
@@ -39,6 +39,7 @@ This roadmap tracks planned Rollbridge features and documentation. Rollbridge sh
|
|
|
39
39
|
- [x] Implement opt-in durable guardian recovery for exact same-authority daemon process replacement without stopping retained generations.
|
|
40
40
|
- [x] Implement guardian-fenced atomic incompatible owner/config/control-socket/package/runtime replacement without stopping retained generations.
|
|
41
41
|
- [x] Add the authenticated one-time disruptive bridge for pre-replacement guardians; all protocol-capable replacements remain atomic.
|
|
42
|
+
- [x] Activate and traffic-switch a healthy candidate before asynchronously starting bounded old-generation retirement.
|
|
42
43
|
- [x] Replicas and stable worker indexes. (Supported on port-less `companion` processes; `proxied`/`singleton`/ported processes stay single.)
|
|
43
44
|
- [x] Allow one process config to start multiple replicas (`replicas`, companion-only for now).
|
|
44
45
|
- [x] Expose `ROLLBRIDGE_REPLICA_INDEX`, replica count, and per-replica template context (`{{replicaIndex}}`/`{{replicaCount}}`).
|
|
@@ -71,6 +72,18 @@ This roadmap tracks planned Rollbridge features and documentation. Rollbridge sh
|
|
|
71
72
|
- [x] Validate unsupported lifecycle-hook combinations once worker lifecycle hooks land. (`lifecycle.drainCommand` requires a positive `drainTimeoutMs`; `nonBlockingDrain` is companion-only; a `lifecycle.stopCommand` may not be combined with a custom `stopSignal`, since the command runs instead of the signal.)
|
|
72
73
|
- [x] Include example fixes in validation output.
|
|
73
74
|
|
|
75
|
+
## Compose service units
|
|
76
|
+
|
|
77
|
+
- [x] Model a Docker Compose service as a Rollbridge-managed unit (`compose[]`): project directory, service name, target image, `stop_grace_period`, drain command and timeout, in-flight gate, and activation probe (Compose healthcheck plus optional ready command/URL).
|
|
78
|
+
- [x] Roll one unit drain-first — quiesce, wait for accepted work to settle, recreate only that service, gate on activation — instead of a raw `docker compose up -d --force-recreate` (`rollbridge compose-roll`, `compose-rollback`, `compose-status`).
|
|
79
|
+
- [x] Retain the replaced image in the generation ledger (`composeStatePath`) so the previous tag stays selectable and rollback is a first-class operation, not a manual compose invocation.
|
|
80
|
+
- [x] Keep the proxied-entry-point guarantee: a unit whose container publishes the Rollbridge proxy port is refused, and a roll never runs while a release generation transition is unresolved.
|
|
81
|
+
- [x] Define drain and in-flight timeout behaviour: both fail the roll before anything is recreated, so the current container keeps serving; an activation failure recreates the previous image and reports `rolled_back`.
|
|
82
|
+
- [x] Operator runbook for the Socketduct hub, one gateway leg, and one builder DinD (`docs/compose-services.md`).
|
|
83
|
+
- [ ] Ship the quiescence hook (`appctl drain`) and an in-flight probe on the hub, gateway, and builder DinD images. Until they exist, the in-flight gate is the only protection work already admitted has, and a roll is better than `up -d --force-recreate` but not a true drain.
|
|
84
|
+
- [ ] Switch TensorBuzz's Socketduct provisioning path (`backend/src/services/docker-servers/socketduct-gateway-provisioning-service.js`) from raw `up -d --force-recreate` to `rollbridge compose-roll`. That is a TensorBuzz change, not a Rollbridge one.
|
|
85
|
+
- [ ] Prove a drain-first roll on a real builder host. The current proof is a real Compose daemon (`test/compose-docker.test.js`, opt-in), not a production builder.
|
|
86
|
+
|
|
74
87
|
## Minor Features
|
|
75
88
|
|
|
76
89
|
- [x] Add a control-socket permission option (`control.mode`) for shared deploy users.
|
|
@@ -0,0 +1,13 @@
|
|
|
1
|
+
### Fixed
|
|
2
|
+
|
|
3
|
+
- A guardian provenance mismatch no longer fences a key whose registered child is
|
|
4
|
+
conclusively gone. When a deploy fails after the daemon committed and switched
|
|
5
|
+
traffic, the old release's processes are retired while the guardian still holds
|
|
6
|
+
their provenance record. Every later deploy reconstructed the same key from a new
|
|
7
|
+
release definition and was rejected with `Guardian provenance mismatch for
|
|
8
|
+
release:<id>:<key>` — including for a release with no live processes at all —
|
|
9
|
+
wedging every subsequent deploy (TensorBuzz production, STC-695). The mismatch
|
|
10
|
+
now only fails closed while the registered process is live (or has an unknown
|
|
11
|
+
state); a `stopped`/`failed` record with no pid is residue of a dead generation
|
|
12
|
+
and the committed owner may re-register the key under the new definition. Live
|
|
13
|
+
processes under a foreign provenance stay rejected.
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
Recover-generation-transition now recognizes a committed journal whose candidate is the live incumbent: when the recovery fence names the release that is actually running, the transition that published it is already complete, so recovery reports `already_recovered` instead of refusing the only fence that describes the authoritative release. A deploy that fails before its own transition is journaled (for example at candidate health) is now recoverable by rerunning the deploy, with no hand-edits to runtime state. The stale-fence guard keeps failing closed for every genuinely in-flight phase (`activating_candidate`, `retiring_previous`, `restoring_previous`, `retiring_failed_candidate`, `degraded_active`) and for committed journals whose fence names any other release.
|
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
### Added
|
|
2
|
+
|
|
3
|
+
- Docker Compose services declared in `compose[]` are now Rollbridge-managed units rolled
|
|
4
|
+
drain-first instead of a raw `docker compose up -d --force-recreate`. `rollbridge
|
|
5
|
+
compose-roll --unit <id> [--image <ref>]` quiesces the unit (`drain.command`), polls its
|
|
6
|
+
in-flight gate until the work it already accepted settles, recreates only that service
|
|
7
|
+
(`up -d --no-deps --force-recreate`) through a generated override pinning the target image and
|
|
8
|
+
`stop_grace_period`, then gates on activation (container running, Compose healthcheck, optional
|
|
9
|
+
`readyCommand`/`readyUrl) before recording the generation. `compose-rollback` reuses the same
|
|
10
|
+
path to select the retained previous tag, and `compose-status` reports the live container and
|
|
11
|
+
the retained generations. Each roll result names its `stage` and exits non-zero on `failed`.
|
|
12
|
+
- A drain or in-flight failure fails the roll **before** anything is recreated, so the current
|
|
13
|
+
container keeps serving; an activation failure recreates the previous image and reports
|
|
14
|
+
`rolled_back`, while a rollback that cannot activate reports `rollback_failed` instead of
|
|
15
|
+
success. Generations are retained in `composeStatePath`
|
|
16
|
+
(`rollbridge-<application>.compose.state.json` by default), keeping the replaced tag selectable.
|
|
17
|
+
- Compose rolls refuse to run while a release generation transition is unresolved, and refuse a
|
|
18
|
+
unit whose container publishes the Rollbridge proxy port — checked before the roll and again
|
|
19
|
+
inside the activation gate — so a Compose unit can never replace the proxied entry point in
|
|
20
|
+
place. `rollbridge doctor` preflights the Compose CLI, each unit's Compose file and env file,
|
|
21
|
+
and that its service is declared, and `validate` reports `compose` config issues with fixes.
|
|
@@ -0,0 +1,20 @@
|
|
|
1
|
+
### Fixed
|
|
2
|
+
|
|
3
|
+
- A refused owner mutation now names its holder. The refusal used to read
|
|
4
|
+
`Another owner mutation is already in progress` with no context, so an
|
|
5
|
+
operator could not tell which command held the fence, which process held it,
|
|
6
|
+
for how long, or what it was blocked on. The guardian now records the
|
|
7
|
+
mutation's operation, lease id, holding PID, start time and current blocker
|
|
8
|
+
(`waitingFor`, refinable while the mutation runs through
|
|
9
|
+
`reportOwnerMutationWait`) and reports them in every refusal, including the
|
|
10
|
+
replacement-prepare fence.
|
|
11
|
+
|
|
12
|
+
### Changed
|
|
13
|
+
|
|
14
|
+
- The owner-mutation fence is bounded. A mutation that outruns its lease
|
|
15
|
+
(default 15 minutes, per-command override through `mutationTimeoutMs`) is
|
|
16
|
+
logged as `OWNER_MUTATION_LEASE_EXPIRED`, reported to the holder, and
|
|
17
|
+
released, so a stuck mutation can no longer lock out every later deploy. The
|
|
18
|
+
holder fails loudly when it next ends that mutation instead of silently
|
|
19
|
+
keeping the fence. Retained generation drains are unaffected: they are not
|
|
20
|
+
owner mutations and may still run for hours.
|
|
@@ -0,0 +1,32 @@
|
|
|
1
|
+
### Added
|
|
2
|
+
|
|
3
|
+
- `processes[].dependsOn` names the same-release processes a process needs, and
|
|
4
|
+
Rollbridge now waits for each dependency's allocated port **before** spawning
|
|
5
|
+
the dependent, on `processes[].dependencyTimeoutMs` (default 30 seconds) — a
|
|
6
|
+
budget of its own that is never charged to `health.timeoutMs`. Each wait is
|
|
7
|
+
logged as `release process dependency wait` and `release process dependency
|
|
8
|
+
ready` with `attempts` and `elapsedMs`. `validate` rejects a dependency on
|
|
9
|
+
itself, an unknown process, a `singleton`, a process with no `port`, and a
|
|
10
|
+
process Rollbridge starts after the dependent, which also rules out cycles.
|
|
11
|
+
|
|
12
|
+
### Fixed
|
|
13
|
+
|
|
14
|
+
- A release-scoped process no longer spends the proxied health deadline waiting
|
|
15
|
+
for a dependency. Measured on production 2026-11-06: `background-jobs-main`
|
|
16
|
+
took 11 817 ms to start listening (14:33:39.175 → 14:33:50.992) and the web
|
|
17
|
+
command's own `wait-port` hop then ate part of the same 30 000 ms that
|
|
18
|
+
`proxy.healthTimeoutMs` and `health.timeoutMs` allowed for the whole start
|
|
19
|
+
sequence, leaving the app about 16 seconds to boot before Rollbridge killed a
|
|
20
|
+
release that was still starting normally. The health gate now measures only
|
|
21
|
+
the proxied process's own boot and reports `bootMs`, `healthGateMs`, and
|
|
22
|
+
`waitedForDependenciesMs` in `release candidate health ready`.
|
|
23
|
+
- A failed release start is diagnostic. The health failure now reads
|
|
24
|
+
`Health check failed for <url>: <reason> (process <id>: 18 probes, 941ms
|
|
25
|
+
elapsed of the 900ms health deadline)` instead of naming only the connection
|
|
26
|
+
error, and `release startup failed` carries `deadlineMs`, `dependencyWaitMs`,
|
|
27
|
+
`elapsedMs`, `releaseElapsedMs`, `stage` (`dependency` or `health`), `probes`,
|
|
28
|
+
`failedProcessId`, `readyProcesses`, `unreadyProcesses`, and the failed
|
|
29
|
+
process's `lastOutput`/`lastOutputAt`. A dependency gate that expires fails
|
|
30
|
+
the deploy as `Process <id> could not start because dependency <id> was not
|
|
31
|
+
ready: …` on its own budget; the health gate is unchanged, so a genuinely
|
|
32
|
+
unhealthy release still fails and the incumbent keeps serving.
|
|
@@ -0,0 +1,11 @@
|
|
|
1
|
+
### Added
|
|
2
|
+
|
|
3
|
+
- `tensorbuzz.yml` declares a `paths` filter for the CI lane: `"**"` with a quoted
|
|
4
|
+
`!**/*.md` negation, so a delivery that changes only Markdown is recorded as a
|
|
5
|
+
terminal `skipped` build instead of installing the dependency tree and running
|
|
6
|
+
`npm run all-checks`. 31 of the repository's 114 tracked paths are Markdown and
|
|
7
|
+
no gate input is among them — `test/tensorbuzz-config.test.js` asserts that
|
|
8
|
+
partition over every tracked path, that every negation is quoted, and that the
|
|
9
|
+
catch-all stays first so a later rule cannot re-admit what a negation carved out.
|
|
10
|
+
`tensorbuzz.yml` itself, an unavailable or capped changed-file list, and a manual
|
|
11
|
+
rebuild still run the lane.
|
|
@@ -0,0 +1,26 @@
|
|
|
1
|
+
### Changed
|
|
2
|
+
|
|
3
|
+
- `proxy.healthTimeoutMs` now defaults to **120000** instead of 30000, so a release
|
|
4
|
+
gets a boot budget above the measured production startup without every project
|
|
5
|
+
having to configure one. Measured on TensorBuzz production (server3) 2026-11-06:
|
|
6
|
+
release `20261106142602` started its `web` process at 14:33:39.205, was still
|
|
7
|
+
printing `attachment storage root migration` lines at 14:34:08.842, and was
|
|
8
|
+
SIGTERMed at 14:34:09.492 without ever logging a listening line — the app's own
|
|
9
|
+
boot alone exceeded the 30 000 ms deadline, after the dependency wait had already
|
|
10
|
+
consumed about 11 817 ms of it. Release `20261106144454` was killed 15 373 ms after
|
|
11
|
+
its `web` process started, before reaching those lines. 120 000 ms is the budget the
|
|
12
|
+
same release has published on since `config/rollbridge.production.js` set it on
|
|
13
|
+
2026-11-06.
|
|
14
|
+
- A process that configures no `health` block inherits that budget from
|
|
15
|
+
`proxy.healthTimeoutMs`; setting either level still wins over the default. The
|
|
16
|
+
documentation examples now show 120 000 rather than the value that failed.
|
|
17
|
+
|
|
18
|
+
### Fixed
|
|
19
|
+
|
|
20
|
+
- A release whose app needs more than 30 seconds to boot publishes while an
|
|
21
|
+
incumbent generation is live. `test/rollbridge.test.js` now deploys a candidate
|
|
22
|
+
that answers `/ping` 32 seconds after launch on the default budget and asserts
|
|
23
|
+
`release candidate health ready` records `bootMs`, `healthGateMs`, and
|
|
24
|
+
`deadlineMs: 120000`, with the incumbent still serving `/release` during the boot.
|
|
25
|
+
The health gate is not weakened: a candidate that never becomes healthy still
|
|
26
|
+
fails the deploy when its deadline expires and the incumbent keeps serving.
|
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
### Fixed
|
|
2
|
+
|
|
3
|
+
- `test/owner-replacement.test.js` "failed partial upgrade resumes an incumbent retired release
|
|
4
|
+
drain" no longer fails on a busy runner. Its whole-fixture budget was `5000ms` while the scenario
|
|
5
|
+
spawns six child processes (daemon, a `worker` and a proxied `web` for each of two releases, plus a
|
|
6
|
+
partial owner-replacement guardian) and then waits for the retired drain to resume, so its cost
|
|
7
|
+
tracked host contention instead of the behaviour under test. Measured in the same invocation:
|
|
8
|
+
1.457s at load average 8.3 and 7.105s at load average 24.5, both with the drain completing — the
|
|
9
|
+
cap sat between the two, which is why the spec reported `Error: Timed out after 5000ms` naming no
|
|
10
|
+
stage while nothing in the drain stalled. The budget is now `PARTIAL_DRAIN_FIXTURE_TIMEOUT_MS`
|
|
11
|
+
(20000ms, 2.8x the measured worst case) with the measurement cited at its declaration.
|
|
12
|
+
- The same spec now asserts the resumed drain in the retired release's own status record
|
|
13
|
+
(`drainStartedAt` and `stoppedAt`, ordered) instead of only the stdout line its wait is keyed on,
|
|
14
|
+
so a release cleared without draining its processes cannot pass. No product behavior changed and no
|
|
15
|
+
health gate was weakened: the waits stay event-gated, so a release that never resumes its drain
|
|
16
|
+
still fails.
|
|
17
|
+
- `test/rollbridge.test.js` "candidate activation quiesces the old jobs generation before a blocked
|
|
18
|
+
singleton replacement completes" no longer depends on inotify to reach its assertions. It awaited an
|
|
19
|
+
`fs.watch` event with no initial read and no fallback, so in a runtime whose filesystem delivers no
|
|
20
|
+
events it hung to the runner's ambient 60s default and named no stage. Measured here: a minimal
|
|
21
|
+
`fs.watch` receives 0 events for a file written by an external process, on overlayfs `/tmp` and on
|
|
22
|
+
ext3 `/data`, with `max_user_watches` at 1048576 — the container, not the product. The wait is now on
|
|
23
|
+
the durable marker the old generation writes (`SINGLETON_QUIET_MARKER_TIMEOUT_MS = 20000`, spec
|
|
24
|
+
measured at 1.740s), naming the marker path if it ever expires. The three assertions are unchanged
|
|
25
|
+
and now execute everywhere. The suite's other `fs.watch` helpers (`waitForState`,
|
|
26
|
+
`waitForLifecycleEvent`) already read the file before watching, which is why only this spec was
|
|
27
|
+
affected.
|
|
@@ -0,0 +1,5 @@
|
|
|
1
|
+
Retired releases now complete their transition after the configurable
|
|
2
|
+
`proxy.nonBlockingDrainTimeoutMs` wait even when a non-blocking process has an
|
|
3
|
+
indefinite graceful stop. Rollbridge records the process id and leaves that
|
|
4
|
+
process alive, preserving in-flight work while allowing the guardian to record
|
|
5
|
+
`release drained` and stop the handoff services.
|
|
@@ -0,0 +1,62 @@
|
|
|
1
|
+
### Added
|
|
2
|
+
|
|
3
|
+
- `releaseRetention.maxConcurrentDrainingReleaseGroups` (default **12**) bounds how many
|
|
4
|
+
release groups may drain at once, and `deploy` evaluates it **before the handoff**. Every
|
|
5
|
+
activation retires one more generation, so whenever deploys arrive faster than a generation
|
|
6
|
+
drains the backlog grows monotonically. Measured on TensorBuzz production 2026-11-14/15
|
|
7
|
+
(`shared/log/rollbridge.log`, `traffic switched` → `release drained` pairs for 16 releases):
|
|
8
|
+
drain duration 18 min … 114 min, median ≈37 min (`20261114232900` 23:31:24 → 01:25:53 is
|
|
9
|
+
114m29s), against 7 traffic switches in 61 minutes on 2026-11-14 — a cadence of 3–7 minutes.
|
|
10
|
+
A supervisor tolerating one draining group is unsatisfiable at that cadence: 37 / 3 = 12 is
|
|
11
|
+
the backlog the measured median produces at the fastest measured cadence. The count includes
|
|
12
|
+
the incumbent the handoff is about to retire, because that is the only version of the number
|
|
13
|
+
a pre-handoff decision can act on. A deploy over the bound is refused before any process of
|
|
14
|
+
the candidate starts, naming each draining group and its classification:
|
|
15
|
+
`Deploy refused before activation: 2 release groups already draining (20261115063805 wedged
|
|
16
|
+
(…); 20261114232900 progressing (2 live connections still owned by the generation)).
|
|
17
|
+
Activating 20261115073717 would make 3, above the
|
|
18
|
+
releaseRetention.maxConcurrentDrainingReleaseGroups bound of 2.` `0` disables the check.
|
|
19
|
+
Nothing is queued or deferred by a timer: the refusal is an ordinary deploy failure, and the
|
|
20
|
+
incumbent keeps serving.
|
|
21
|
+
- `status` now distinguishes a drain that is still progressing from one that is wedged. Each
|
|
22
|
+
draining release reports `drain: {cpuDeltaMs, reason, sampleWindowMs, status}` with
|
|
23
|
+
`status` one of `progressing`, `wedged`, or `unproven`, and each process reports
|
|
24
|
+
`retirementTimedOut` plus `heldProcessGroup: {liveMemberCount, pgid}`. A group is `wedged`
|
|
25
|
+
only when its drain can no longer progress — no live connection, every retirement budget
|
|
26
|
+
expired, and either no runner child remains or the group's cumulative CPU (procfs
|
|
27
|
+
`utime`+`stime`, defunct members included) is flat across a window of at least 20 000 ms.
|
|
28
|
+
Every inconclusive reading reports `progressing` or `unproven`, never `wedged`: the backlog
|
|
29
|
+
this fleet actually has is honest slow drain, measured as 3 live
|
|
30
|
+
`background-jobs-runner: 1 job` children with a `DockerServerDrainFillJob` `handed_off` for
|
|
31
|
+
30 minutes, whose last job completed 6 seconds before the guardian logged `release drained`.
|
|
32
|
+
Consumers gate on that classification instead of on a raw process count.
|
|
33
|
+
|
|
34
|
+
### Fixed
|
|
35
|
+
|
|
36
|
+
- A process group that outlives its tracked process is now recorded and reported. Rollbridge
|
|
37
|
+
spawns every process `detached: true`, so the tracked pid is the group leader; when the
|
|
38
|
+
leader exits on the stop signal while a member it launched keeps running, the tracked record
|
|
39
|
+
goes `stopped` with no pid while the group still holds the generation's work. `status`
|
|
40
|
+
reports it as `heldProcessGroup`, the drain classifier observes it by group id, and the
|
|
41
|
+
guardian's provenance fence (`childRecordIsGone`, now shared with the daemon instead of
|
|
42
|
+
inlined in `process-guardian.js`) counts it as not gone. The group id is recorded by the stop
|
|
43
|
+
path that already waited for the group to leave, not by `onExit`: that hook fires for every
|
|
44
|
+
retired process, and rediscovering the group there costs a procfs scan (~200 ms measured
|
|
45
|
+
here) on every exit, including the ordinary ones where the group left cleanly.
|
|
46
|
+
- `rollback` refuses a target whose generation is recorded as drained while its processes still
|
|
47
|
+
hold live work, naming each held process and the release the guardian reports as active.
|
|
48
|
+
Re-deploying that release id re-registered its guardian process keys under a new provenance
|
|
49
|
+
while the old processes still held them, and the deploy failed partway through with
|
|
50
|
+
`Guardian provenance mismatch for release:20261115060540:background-jobs-main` (TensorBuzz
|
|
51
|
+
production, run `408591ff-e72e-4351-b15a-7ec239a62f6b`, 2026-11-15) — leaving the deploy
|
|
52
|
+
tool's `current` pointer pinned at `20261115060540` while the guardian reported
|
|
53
|
+
`20261115073717` active. The refusal happens before anything is touched and names the
|
|
54
|
+
guardian's active release, so the pointer is reconciled from `rollbridge status` instead of
|
|
55
|
+
silently diverging.
|
|
56
|
+
- Drain liveness observations are bounded: a generation that still owns live connections is
|
|
57
|
+
classified without reading procfs at all, and a new observation is taken at most once per
|
|
58
|
+
20 000 ms window per draining group (held groups re-verify at most once per 5 000 ms).
|
|
59
|
+
Sampling on every `status` read was measured here at ~200 ms per group on a host with 9 530
|
|
60
|
+
processes, enough to starve a 1 000 ms connection drain in `test/rollbridge.test.js` "a
|
|
61
|
+
nonBlockingDrain worker stops immediately while its release is still draining". No timer,
|
|
62
|
+
poll, or sweeper was added: the bound is a staleness check evaluated when `status` is read.
|
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
### Fixed
|
|
2
|
+
|
|
3
|
+
- `deploy` refuses a `--release-id` it still retains while that generation holds live work,
|
|
4
|
+
before any process of the candidate starts. PR #12 fenced `rollback` only; the restore that
|
|
5
|
+
actually ran in TensorBuzz production was a deploy command naming the drained release id, and
|
|
6
|
+
it failed twice in twelve minutes (`shared/log/rollbridge.log`, server3, 2026-11-15):
|
|
7
|
+
`20:29:51Z traffic switched 20261115195051 → 20261115202720` then `20:29:55Z release startup
|
|
8
|
+
failed` / `deploy failed` with `Guardian provenance mismatch for
|
|
9
|
+
release:20261115195051:background-jobs-main`, and the identical pair at `20:41:21Z` /
|
|
10
|
+
`20:41:26Z` against `20261115203845`. Only one `release drained` landed in that window
|
|
11
|
+
(`20261115164249` at 20:37:03Z), and a 20:54Z process census found all three generations alive
|
|
12
|
+
at once — `20261115195051` (main + worker + 3 runners), `20261115202720` (main + worker + 4
|
|
13
|
+
runners), `20261115203845` (main + worker + 4 runners) — with `20261115195051`'s
|
|
14
|
+
`velocious background-jobs-main` still running as pid 2310504. The provenance key therefore
|
|
15
|
+
belonged to a live generation, so the 2026-10-25 carve-out that lets a dead generation's key be
|
|
16
|
+
re-registered never applied; the collision aborted the deploy after the handoff had already
|
|
17
|
+
moved traffic, leaving the deploy tool's `current` pointer at `releases/20261115195051` while
|
|
18
|
+
the guardian served `20261115203845`. The refusal is decided from the retained record with the
|
|
19
|
+
guardian's own `childRecordIsGone` fence, names the retained state, each held process (a held
|
|
20
|
+
process group included), and the release the guardian reports as active, and starts nothing:
|
|
21
|
+
`Deploy refused before activation: release "20261115195051" is retained in state draining while
|
|
22
|
+
1 process of that generation still hold live work (…). … The guardian's active release is
|
|
23
|
+
"20261115203845"; deploy a new release id or reconcile the deployment pointer with \`rollbridge
|
|
24
|
+
status\`.` Journaled generation-transition resume is exempt — it re-drives the transaction that
|
|
25
|
+
already owns the release id — and a rollback to a still-draining target still stops that
|
|
26
|
+
generation first, which is what makes its id reusable; the same fence then applies after the
|
|
27
|
+
stop.
|
|
@@ -0,0 +1,44 @@
|
|
|
1
|
+
### Added
|
|
2
|
+
|
|
3
|
+
- `releaseRetention.escalateWedgedGenerations` (default **false**) gives the generation
|
|
4
|
+
supervisor a bounded action for a wedge it can already classify, instead of leaving the
|
|
5
|
+
operator to signal PIDs by hand. STOE-1687 measured the gap on TensorBuzz production
|
|
6
|
+
2026-11-15: generation `20261115164249` claimed zero `handed_off` jobs and produced no
|
|
7
|
+
`release drained` record for about four hours until it exited on its own at 20:37:03 UTC.
|
|
8
|
+
While it lingered, `runtime:assert-single-jobs-generation` aborted deploys `20261115202720`
|
|
9
|
+
and `20261115203845` at `Hook after_runtime`, the second deploy's rollback failed too
|
|
10
|
+
(`Restoration of previous release '20261115195051' also failed … incumbent '20261115203845'
|
|
11
|
+
is authoritative`), and three jobs generations were live simultaneously at 20:42 UTC. The
|
|
12
|
+
prepared remedy was SIGTERM against the exact runner PIDs — an operator action, with no
|
|
13
|
+
bounded path to it from the supervisor.
|
|
14
|
+
- The escalation is **not a timer, poller, watchdog, or sweeper**. It runs at two moments that
|
|
15
|
+
already exist: as the last act of `drainAndStop`, bracketing the retirement wait with a
|
|
16
|
+
liveness observation taken when the transition starts and one taken when its stops return
|
|
17
|
+
(the gap is `proxy.nonBlockingDrainTimeoutMs` at minimum, so flatness is provable from the
|
|
18
|
+
transition itself), and at the start of `deploy`, reusing the observation the
|
|
19
|
+
draining-generation admission bound already takes. A generation that stopped with a live
|
|
20
|
+
process group now keeps its observations instead of dropping them, so the next deploy can
|
|
21
|
+
prove the residue without sampling anything new.
|
|
22
|
+
- A `SIGKILL` is delivered only when four proofs hold, each named in the log line: the
|
|
23
|
+
generation is classified `wedged` (no live connection), every live process already expired
|
|
24
|
+
its retirement budget, **no runner child remains** in any tracked process group, and the
|
|
25
|
+
group's CPU is flat across a full 20 000 ms window. The third is the STOE-1077 boundary —
|
|
26
|
+
67 builds and 27.3 build-hours lost in 7 days to a stop-timeout kill reaching live lanes —
|
|
27
|
+
and it is proven independently of the classifier: a wedge that still holds a live runner is
|
|
28
|
+
reported, never killed. `test/managed-process.test.js` "retirement timeout leaves an
|
|
29
|
+
indefinite worker alive without SIGKILL" stays green, and
|
|
30
|
+
`test/release-group.test.js` "a wedged generation that still holds a live runner child is
|
|
31
|
+
reported, never killed" is the control.
|
|
32
|
+
- A wedge the supervisor may not terminate is now loud rather than silent.
|
|
33
|
+
`release wedged generation left alive` names the pids, process ids, and held process groups
|
|
34
|
+
with the exact proof that is missing (`no escalation: 1 runner child is still live in the
|
|
35
|
+
generation; SIGKILL would abort in-flight work`), and
|
|
36
|
+
`release wedged generation escalated` plus `process retirement escalated to SIGKILL` record
|
|
37
|
+
the action taken. `ManagedProcess.escalateRetirement` re-checks its own two invariants before
|
|
38
|
+
signalling anything — the process must already be recorded past its retirement budget, and
|
|
39
|
+
its group must still have a member that can be signalled — and its confirmation wait is
|
|
40
|
+
bounded at 5 000 ms so the escalation cannot become the unbounded wait it replaced.
|
|
41
|
+
- Off by default, deliberately: `gracefulStopMs: "indefinite"` is a consumer's explicit
|
|
42
|
+
statement that a deploy must not kill that process, and procfs cannot distinguish a wedged
|
|
43
|
+
worker from one quietly finishing work that lives inside the process rather than in forked
|
|
44
|
+
runners. Enabling it is the operator's decision, per consumer.
|
|
@@ -0,0 +1,30 @@
|
|
|
1
|
+
### Fixed
|
|
2
|
+
|
|
3
|
+
- An explicit `rollbridge stop` (and the stop inside `rollback`) now gives the processes it stops
|
|
4
|
+
the same retirement bound the deploy path already used, `proxy.nonBlockingDrainTimeoutMs`
|
|
5
|
+
(default **60 000 ms**), and reports what it left alive instead of hanging. Measured on
|
|
6
|
+
TensorBuzz production 2026-11-16 (`shared/log/rollbridge.log`, `tensorbuzz-production-web-1`):
|
|
7
|
+
deploys were refused with `Another owner mutation is already in progress` from 07:35Z to 07:58Z,
|
|
8
|
+
and the `stop` of release `20261115203845` that began at 08:01:29Z was still running 30+ minutes
|
|
9
|
+
later. The phase that never completed is `ManagedProcess.performStop` step 3:
|
|
10
|
+
`waitForProcessGroupExit(pgid, undefined)` for the `background-jobs-worker`, reached from
|
|
11
|
+
`stopRelease` → `ReleaseGroup.stop()` with no options while that process carries
|
|
12
|
+
`gracefulStopMs: "indefinite"`. Because the command runs inside `executeOwnerMutation`, the wait
|
|
13
|
+
held the mutation fence for the whole drain. `ReleaseGroup.stop({retirementTimeoutMs})` now
|
|
14
|
+
bounds a stop for the processes the supervisor must not kill — a `nonBlockingDrain` process and
|
|
15
|
+
any process whose graceful window is `"indefinite"` (`ManagedProcess.hasUnboundedGracefulWindow()`)
|
|
16
|
+
— leaves them running, and logs `release stop gave up waiting for a retiring process` with the
|
|
17
|
+
process id, release id and budget. Shutdown and retired-generation drain are deliberately left
|
|
18
|
+
unbudgeted: those waits own the processes they stop.
|
|
19
|
+
- A retirement budget now **caps** a finite graceful window instead of only standing in for a
|
|
20
|
+
missing one. `performStop` previously used the graceful deadline whenever one existed and took
|
|
21
|
+
the retirement branch only when `gracefulDeadline === undefined`, so after TensorBuzz moved its
|
|
22
|
+
worker from `"indefinite"` to 4,5 h the explicit stop would still have held the control plane for
|
|
23
|
+
4,5 h — and then escalated to SIGKILL, killing the in-flight builds the consumer's
|
|
24
|
+
`gracefulStopMs` exists to protect. Measured against `origin/master` with the new spec: the stop
|
|
25
|
+
of a worker that never exits on its stop signal returns in **31,3 s** (the worker's own delay)
|
|
26
|
+
at base and in **2,9 s** with the fix, with `retirementTimedOut` recorded and no SIGKILL sent.
|
|
27
|
+
- With a `lifecycle.stopCommand`, the graceful window is now measured from the start of the stop
|
|
28
|
+
step rather than from after the hook returns. `hookTimeoutMs()` already bounds the hook by the
|
|
29
|
+
same window, so the step previously allowed hook + window in series (up to 2 × `gracefulStopMs`)
|
|
30
|
+
inside an owner mutation; the whole step is now bounded by the window.
|
|
@@ -0,0 +1,11 @@
|
|
|
1
|
+
## Fixed
|
|
2
|
+
|
|
3
|
+
- Make a normalized config identical to its own JSON form, so a release config
|
|
4
|
+
that crossed the guardian's owner-replacement transfer is still equal to the
|
|
5
|
+
config file it came from. Owner replacement publishes release configs over the
|
|
6
|
+
guardian's JSON socket, which drops undefined-valued properties; the candidate
|
|
7
|
+
daemon then held a config that no longer matched its own file, and the next
|
|
8
|
+
deploy aborted with "Config changes to control cannot be applied live" even
|
|
9
|
+
though the file was byte-identical and hashed to the same owner-config digest
|
|
10
|
+
(measured on tensorbuzz/production, 2026-11-16). The only remedy the tool
|
|
11
|
+
offered was a manual daemon restart.
|
|
@@ -0,0 +1,29 @@
|
|
|
1
|
+
### Fixed
|
|
2
|
+
|
|
3
|
+
- Four drain/retirement specs read the daemon's process record before the daemon had observed the
|
|
4
|
+
event they describe, and failed intermittently on a busy runner while passing locally. Measured
|
|
5
|
+
here in a `node:26` container against `origin/master` `c390da9` (2026-11-16): running the three
|
|
6
|
+
`test/rollbridge.test.js` specs on their own gave `an explicit stop returns within its retirement
|
|
7
|
+
budget…` 0/6 passing, `rollback refuses to restore a drained generation…` 2/6 and `a restore
|
|
8
|
+
deploy naming a drained generation…` 2/6, and `test/managed-process.test.js` "a retirement budget
|
|
9
|
+
caps a finite graceful window…" 0/4. The same invocations inside the whole file passed, which is
|
|
10
|
+
what made them look green locally and red in CI: every recent `rollbridge` master commit carries
|
|
11
|
+
a failed `TensorBuzz: build_1` naming one of these specs.
|
|
12
|
+
- `test/rollbridge.test.js` now gates on the settled record. A bounded retirement stop returns the
|
|
13
|
+
moment its budget expires — that is the point of the bound — while the shell wrapper it signalled
|
|
14
|
+
is still on its way out, so the record briefly reports the wrapper pid and `state: "quiesced"`,
|
|
15
|
+
and `ManagedProcess.currentHeldProcessGroup()` deliberately reports nothing until the tracked
|
|
16
|
+
process is `stopped`. Measured: `stopRelease` returned after 106 ms with
|
|
17
|
+
`{pid: 13640, retirementTimedOut: true, state: "quiesced"}` and the record settled to
|
|
18
|
+
`{heldProcessGroup: {liveMemberCount: 1, pgid: 13640}, state: "stopped"}` within the next 50 ms.
|
|
19
|
+
The new `retirementRecordSettled` helper waits for that observation before the assertions, so the
|
|
20
|
+
specs still prove the group outlived its tracked process instead of assuming it.
|
|
21
|
+
- `test/managed-process.test.js` "a retirement budget caps a finite graceful window instead of being
|
|
22
|
+
outvoted by it" signalled its fixture before the fixture had installed its `SIGTERM` disposition,
|
|
23
|
+
so the default disposition applied, the group exited, `waitForProcessGroupExit` returned true, and
|
|
24
|
+
`retirementTimedOut` stayed false — the same boot race the neighbouring specs already gate on. It
|
|
25
|
+
now announces readiness the same way (`require("node:fs").writeFileSync(readyPath, "ready")` after
|
|
26
|
+
the handler is installed) and waits for that marker before the stop.
|
|
27
|
+
- No product behaviour changed and no wait was lengthened: the gates are on durable observations
|
|
28
|
+
(a process record reaching `stopped`, a marker file the fixture writes), so a daemon that never
|
|
29
|
+
records the residue still fails.
|
|
@@ -0,0 +1,37 @@
|
|
|
1
|
+
### Fixed
|
|
2
|
+
|
|
3
|
+
- A rollbridge version bump now hands the retained guardian to a current-version coordinator
|
|
4
|
+
automatically instead of refusing until a human restarts a daemon or a supervisor. Measured on
|
|
5
|
+
TensorBuzz production (STOE-2040): the live guardian was pid 2423469, 19 days up, running
|
|
6
|
+
`…/releases/20261028121152/backend/node_modules/rollbridge/src/process-guardian.js` (rollbridge
|
|
7
|
+
0.1.55) while the daemon runtime was 0.1.61, and 45 deploys were refused with
|
|
8
|
+
`Another owner mutation is already in progress`. Nothing replaced the guardian: every daemon
|
|
9
|
+
start reuses `recovery.guardian` from the state file, so the version mismatch had no path out
|
|
10
|
+
at all.
|
|
11
|
+
- The retained guardian's version is read from its own executable (`guardianVersionFromProcess`)
|
|
12
|
+
before any replacement transaction is prepared, so the version probe does not move the
|
|
13
|
+
`capabilities` command ahead of `prepare-owner-replacement` where a capability-gated guardian
|
|
14
|
+
expects it. `test/owner-replacement.test.js` "replacement-capable guardian without daemon
|
|
15
|
+
recovery aborts before handoff and resumes incumbent drains" is the control for that ordering.
|
|
16
|
+
- A drifted guardian takes the authenticated upgrade bridge that already reconciles the one-time
|
|
17
|
+
legacy owner: the candidate starts a coordinator of its own version
|
|
18
|
+
(`<statePath>.guardian-<version>.sock`), the retained guardian keeps supervising every process
|
|
19
|
+
it already owns, and `status.ownerTransition` reports `mode: "guardian-version-handoff"` with
|
|
20
|
+
`disruptive: true`. Nothing waits for the retained generation to drain — the retained guardian
|
|
21
|
+
retires when its own processes exit. `test/owner-replacement.test.js`
|
|
22
|
+
"version-drifted retained guardian is handed to a current-version coordinator automatically"
|
|
23
|
+
asserts the retained worker pids stay live and reported by the retained guardian afterwards.
|
|
24
|
+
- Consequence of that design: every rollbridge version bump crosses the disruptive boundary once.
|
|
25
|
+
The incumbent daemon is `SIGKILL`ed, the supervised application processes keep their exact PIDs
|
|
26
|
+
under guardian supervision, and existing proxy/control connections may close. Replacements at
|
|
27
|
+
matching versions remain candidate-first and atomic.
|
|
28
|
+
- The refusal is now the last resort and names both versions plus the exact commands:
|
|
29
|
+
`The persistent Rollbridge guardian is version <guardian> while this daemon runs rollbridge
|
|
30
|
+
<daemon>; … Run "rollbridge shutdown --config <path>" and then "rollbridge ensure-daemon
|
|
31
|
+
--config <path>" to replace the supervisor with this Rollbridge installation.` It fires only
|
|
32
|
+
when no handoff can be authorised: no incumbent daemon to hand guardian ownership from, no
|
|
33
|
+
incumbent PID reported by `ensure-daemon`, an unattestable guardian version, an incumbent PID
|
|
34
|
+
that is not the attested owner, or a config change in the same invocation (the bridge keeps the
|
|
35
|
+
incumbent config identity unchanged). `test/owner-recovery.test.js`
|
|
36
|
+
"startup against a version-drifted guardian without an incumbent refuses and names both versions
|
|
37
|
+
and the repair command" asserts the refusal text and that the live generation is untouched.
|
|
@@ -0,0 +1,7 @@
|
|
|
1
|
+
## Reclaim owner-mutation leases held by dead daemons
|
|
2
|
+
|
|
3
|
+
The guardian now verifies the recorded mutation holder PID before refusing a new
|
|
4
|
+
owner mutation. Dead holders are reclaimed and reported as an
|
|
5
|
+
`owner-mutation-reclaimed` event, so a daemon death cannot leave the deploy fence
|
|
6
|
+
wedged until an operator kills the guardian. `rollbridge status` and `doctor` now
|
|
7
|
+
show the guardian version and active mutation lease diagnostics.
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
- Activate and switch traffic to a ready replacement generation before quiescing the incumbent. A stalled retirement can no longer stop the active web or jobs generation before its successor is serving traffic; retirement failures remain observable on the retired release after the handoff.
|
|
@@ -0,0 +1,35 @@
|
|
|
1
|
+
### Fixed
|
|
2
|
+
|
|
3
|
+
- A draining generation that still owns live connections is no longer reported `progressing`
|
|
4
|
+
without bound. The connection branch of `classifyDrainProgress` had no deadline, so a release
|
|
5
|
+
whose `proxy.drainTimeoutMs` wait had already expired kept claiming progress for as long as one
|
|
6
|
+
websocket stayed open. It now reports the new `timedOut` status, naming the budget and how far
|
|
7
|
+
past it the drain is: `drain budget 60000ms exceeded by 1000ms with 1 live connection still
|
|
8
|
+
owned by the generation`. Measured on TensorBuzz production 2026-11-17 (`shared/log/rollbridge.log`,
|
|
9
|
+
`tensorbuzz-production-web-1`): the incumbent retired by the failed deploy of `20261117063703`
|
|
10
|
+
began draining at `drainStartedAt 06:40:21.244Z` and was still read `draining` with
|
|
11
|
+
`drain.status: progressing`, `sampleWindowMs: 0` and one held websocket at 07:00Z and again at
|
|
12
|
+
07:30Z, 25+ min past `proxy.drainTimeoutMs: 60000`. `timedOut` is a report, never an escalation:
|
|
13
|
+
`decideWedgedEscalation` still acts only on a `wedged` classification, and
|
|
14
|
+
`test/drain-policy.test.js` "the drain deadline names the deadline and never escalates" holds
|
|
15
|
+
that boundary. No signal path and no scheduled sampler is added.
|
|
16
|
+
- A candidate retired by a failed pre-commit activation is now drained by the compensation that
|
|
17
|
+
retired it. `compensatePreCommitActivationFailure` started the candidate's `drainAndPrune` only
|
|
18
|
+
on its success path, after the checkpoint clear, so when incumbent restoration threw the retired
|
|
19
|
+
candidate was left `draining` with its processes alive: candidate `20261117063703` still held
|
|
20
|
+
`web` pid 3201928 when read at 07:30Z, 47 min after its compensation threw at 06:42:31.115Z, and
|
|
21
|
+
reached `release drained` only when a later compensation reached its success path at
|
|
22
|
+
08:47:00.159Z. A retained generation that still holds live processes is the state that refuses
|
|
23
|
+
every later deploy, and a failed candidate is never a restoration target. The drain now starts
|
|
24
|
+
where the candidate is recorded retired and stays in the background — the compensation does not
|
|
25
|
+
wait for it, so this cannot turn a failed activation into a blocking stop.
|
|
26
|
+
- New coverage for both halves: `test/release-group.test.js` "a draining generation that still owns
|
|
27
|
+
a connection past its drain budget is named timedOut, not left claiming progress" reads the
|
|
28
|
+
classification out of `status()` for a release whose `drainStartedAt` is 90 s old with one
|
|
29
|
+
connection still owned, and back to `progressing` for the same release inside budget.
|
|
30
|
+
`test/rollbridge.test.js` "a candidate retired by a failed activation is drained to stopped
|
|
31
|
+
instead of left holding live processes" fails activation, rejects the incumbent restoration, and
|
|
32
|
+
asserts the retired candidate reaches `stopped` with a `release drained` event — on the base tree
|
|
33
|
+
that wait times out because the drain was unreachable. "unresolved generation transition fences
|
|
34
|
+
stop, restart, and rollback mutations" additionally checks that the candidate its deploy retired
|
|
35
|
+
is stopped by its own compensation drain while the fence refuses every mutation.
|