rollbridge 0.1.72 → 0.1.74
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +63 -10
- package/TODO.md +13 -0
- package/changelog.d/20261025-guardian-re-register-dead-generation.md +13 -0
- package/changelog.d/20261026-stale-committed-generation-transition-recovery.md +1 -0
- package/changelog.d/20261104-compose-service-units.md +21 -0
- package/changelog.d/20261105-owner-mutation-lease-diagnostics.md +20 -0
- package/changelog.d/20261106-release-dependency-boot-budgets.md +32 -0
- package/changelog.d/20261109-path-filtered-builds.md +11 -0
- package/changelog.d/20261110-default-boot-budget-above-measured-startup.md +26 -0
- package/changelog.d/20261111-contention-sensitive-daemon-specs.md +27 -0
- package/changelog.d/20261113-bounded-nonblocking-retirement.md +5 -0
- package/changelog.d/20261115-bound-draining-release-groups.md +62 -0
- package/changelog.d/20261115-restore-deploy-refuses-held-generation.md +27 -0
- package/changelog.d/20261115-wedged-generation-escalation.md +44 -0
- package/changelog.d/20261116-bounded-explicit-stop-retirement.md +30 -0
- package/changelog.d/20261116-config-json-canonical-shape.md +11 -0
- package/changelog.d/20261116-drain-specs-gate-the-retirement-record.md +29 -0
- package/changelog.d/20261116-guardian-version-handoff.md +37 -0
- package/changelog.d/20261116-reclaim-dead-owner-mutation.md +7 -0
- package/changelog.d/20261117-activate-before-incumbent-retirement.md +1 -0
- package/changelog.d/20261117-activation-path-drain-deadline.md +35 -0
- package/changelog.d/20261117-stop-companion-after-retired-dependency.md +7 -0
- package/changelog.d/20261118-incumbent-retirement-authority-guard.md +1 -0
- package/changelog.d/20261118-owner-retirement-no-guardian-bound.md +19 -0
- package/changelog.d/20261118-shutdown-retirement-bound-restored.md +31 -0
- package/docs/cli.md +128 -3
- package/docs/compose-services.md +222 -0
- package/docs/config.md +177 -6
- package/docs/troubleshooting.md +97 -6
- package/docs/velocious.md +1 -1
- package/docs/workers.md +11 -0
- package/examples/tensorbuzz.com.js +3 -2
- package/package.json +1 -1
- package/src/cli.js +68 -2
- package/src/compose-unit.js +725 -0
- package/src/config.js +346 -8
- package/src/daemon.js +410 -73
- package/src/doctor.js +74 -0
- package/src/drain-policy.js +252 -0
- package/src/guardian-client.js +67 -8
- package/src/health.js +99 -8
- package/src/managed-process.js +197 -16
- package/src/process-guardian.js +212 -22
- package/src/process-memory.js +62 -2
- package/src/process-order.js +26 -0
- package/src/release-group.js +435 -37
- package/tensorbuzz.yml +20 -1
- package/test/completion.test.js +5 -2
- package/test/compose-config.test.js +160 -0
- package/test/compose-docker.test.js +297 -0
- package/test/compose-unit.test.js +415 -0
- package/test/config-validation.test.js +135 -5
- package/test/drain-policy.test.js +328 -0
- package/test/fixtures/deferred-boot-app.js +106 -0
- package/test/fixtures/gated-app.js +50 -0
- package/test/guardian-client.test.js +119 -5
- package/test/health.test.js +74 -1
- package/test/managed-process.test.js +260 -1
- package/test/owner-recovery.test.js +125 -21
- package/test/owner-replacement.test.js +175 -14
- package/test/process-memory.test.js +47 -1
- package/test/release-group.test.js +408 -2
- package/test/rollbridge.test.js +774 -72
- package/test/shutdown-completion.test.js +119 -0
- package/test/shutdown-retirement-bound.test.js +162 -0
- package/test/support/process.js +3 -1
- package/test/tensorbuzz-config.test.js +228 -0
|
@@ -0,0 +1,7 @@
|
|
|
1
|
+
# Stop companions after a retired dependency
|
|
2
|
+
|
|
3
|
+
Release companions that declare a same-generation service dependency now stop
|
|
4
|
+
after an unexpected exit when that dependency has been retired. They no longer
|
|
5
|
+
restart forever against the retired service's port. The TensorBuzz worker
|
|
6
|
+
example declares its jobs-main dependency and relies on the worker's lifecycle
|
|
7
|
+
handshake instead of a permanent port wait.
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
- Guard incumbent retirement on the candidate already being authoritative, and cover the handoff boundary where the replacement is carrying traffic while the old web and jobs generation is still alive.
|
|
@@ -0,0 +1,19 @@
|
|
|
1
|
+
### Fixed
|
|
2
|
+
|
|
3
|
+
- `performOwnerRetirement`'s no-guardian branch now bounds the stops it issues after releasing
|
|
4
|
+
listeners with `proxy.nonBlockingDrainTimeoutMs`, the same retirement budget `performShutdown`
|
|
5
|
+
(PR #28), `stopRelease` and `rollback` already use. Without a guardian nothing else retires
|
|
6
|
+
those processes, so an unbounded `stop()` on a companion whose graceful window is `"indefinite"`
|
|
7
|
+
left the generation `draining` forever: the release never reached `stopped`, its processes stayed
|
|
8
|
+
`quiesced` with no owner that could stop them, and the state that made the measured 2026-11-17
|
|
9
|
+
`rollbridge rollback` hang for 7 minutes — with the owner-mutation fence left held so
|
|
10
|
+
`restart --process web` was refused with `Another owner mutation is already in progress` — was
|
|
11
|
+
never cleared. The bound is "stop waiting", never "SIGKILL": the process is left alive and named
|
|
12
|
+
by `release stop gave up waiting for a retiring process` (STOE-1077).
|
|
13
|
+
- New coverage: `test/shutdown-completion.test.js` "retiring an owner without a guardian stops
|
|
14
|
+
waiting for its indefinite-window companion" retires an owner holding a companion that ignores
|
|
15
|
+
its stop signal, and asserts the retirement names the process it gave up on with its `processId`,
|
|
16
|
+
`releaseId` and `timeoutMs`, the release and that process both reach `stopped`, and the process
|
|
17
|
+
group is still alive afterwards. Red on the base tree, where the residue line never arrives.
|
|
18
|
+
No watchdog, sampler or scheduled retry is added: the deadline is the existing retirement budget,
|
|
19
|
+
applied at the call site that was missing it.
|
|
@@ -0,0 +1,31 @@
|
|
|
1
|
+
### Fixed
|
|
2
|
+
|
|
3
|
+
- `rollbridge shutdown` now bounds **every** stop it issues with `proxy.nonBlockingDrainTimeoutMs`
|
|
4
|
+
(default **60 000 ms**) — the releases, the starting releases, the singletons and the daemon
|
|
5
|
+
services. The bound is "stop waiting", never "SIGKILL": a `nonBlockingDrain` process and any
|
|
6
|
+
process whose graceful window is `"indefinite"` is left draining and reported with
|
|
7
|
+
`release stop gave up waiting for a retiring process`, exactly as an explicit `rollbridge stop`
|
|
8
|
+
already behaves (STOE-2015).
|
|
9
|
+
- The shutdown half of that bound had been landed in PR #21 (`a85b605`, 2026-11-17 22:08Z) and was
|
|
10
|
+
**dropped from `master` by the merge `b1db998`** on Mika Mesh's `STOE-1861` branch: `master`
|
|
11
|
+
`28cba85` carries 1 bounded `release.stop({retirementTimeoutMs})` (the `stopRelease` path) and 0
|
|
12
|
+
in `performShutdown`, while `a85b605` carried 3 plus a bounded `guardian.shutdown(…)`. The spec
|
|
13
|
+
PR #21 added survived, so `master` is red on its own test: `test/shutdown-completion.test.js`
|
|
14
|
+
"shutdown returns after the non-blocking drain budget instead of waiting for a worker" fails at
|
|
15
|
+
`origin/master` (9 passed, 1 failed, the test outliving its own watchdog) and passes 10/10 with
|
|
16
|
+
this change.
|
|
17
|
+
- Singletons and daemon services were never bounded by any version of this fix — `performShutdown`
|
|
18
|
+
called `processInstance.stop()` with no options for `this.singletons` and `this.services`, so one
|
|
19
|
+
process with `gracefulStopMs: "indefinite"` kept the daemon alive after its control socket was
|
|
20
|
+
already unlinked: alive, unreachable, and still holding the generation it had stopped. STOE-2275
|
|
21
|
+
measured that shape on TensorBuzz production 2026-11-17 (`tensorbuzz-production-web-1`):
|
|
22
|
+
`kill -TERM 3231962` at 21:43:5xZ, the active release `20261117181640`'s `background-jobs-main`
|
|
23
|
+
and `web` exited code 0 at 21:44:14Z, the daemon never exited, `/tmp/rollbridge-tensorbuzz.sock`
|
|
24
|
+
was gone, the proxy on 4500 produced 4,800 `connect ECONNREFUSED 127.0.0.1:14502` entries, and
|
|
25
|
+
`rollbridge rollback` then hung for 7 minutes with the owner-mutation fence left held.
|
|
26
|
+
- New spec `test/shutdown-retirement-bound.test.js` pins both shapes: a release companion and a
|
|
27
|
+
daemon service that ignore their stop signal. Each asserts the shutdown returns inside the budget,
|
|
28
|
+
the process group is left alive rather than killed, the control socket and state file are actually
|
|
29
|
+
removed, and — for the release shape — that a new owner can bind the same control endpoint
|
|
30
|
+
immediately afterwards. Red at `origin/master` (`0 passed, 2 failed, 2 total`, both sides report
|
|
31
|
+
`wedged`), green with the fix.
|
package/docs/cli.md
CHANGED
|
@@ -116,12 +116,14 @@ the candidate binds and validates its listeners, and an authenticated fenced
|
|
|
116
116
|
transaction commits guardian authority and the final control socket. A lost
|
|
117
117
|
control response is accepted only when the guardian confirms the exact committed
|
|
118
118
|
transaction id. The first authenticated upgrade from a genuine pre-replacement
|
|
119
|
-
guardian/daemon,
|
|
120
|
-
the retired-owner commit capability,
|
|
119
|
+
guardian/daemon, from a retained prepare/stage guardian that explicitly lacks
|
|
120
|
+
the retired-owner commit capability, or from a retained guardian whose rollbridge
|
|
121
|
+
version differs from the candidate runtime, is the exception: `ensure-daemon` preserves its exact
|
|
121
122
|
guardian-owned processes but deliberately retires the old listeners before the
|
|
122
123
|
Node 20 candidate binds, so existing proxy/control connections may close.
|
|
123
124
|
Successful status JSON records this as `ownerTransition.disruptive: true` and
|
|
124
|
-
`ownerTransition.mode: "legacy-first-upgrade"
|
|
125
|
+
`ownerTransition.mode: "legacy-first-upgrade"` for a protocol gap or
|
|
126
|
+
`"guardian-version-handoff"` for a version drift. This bridge requires
|
|
125
127
|
the incumbent config identity unchanged; make config/socket changes in a second,
|
|
126
128
|
atomic invocation. Auth, transport, malformed-response, arbitrary unknown-command,
|
|
127
129
|
and authority failures do not qualify and fail before any deploy is sent.
|
|
@@ -163,6 +165,25 @@ for old workers, jobs, or HTTP/WebSocket connections to finish. Prints
|
|
|
163
165
|
If the new release fails to start or health-check, the previous release stays
|
|
164
166
|
active and the command errors.
|
|
165
167
|
|
|
168
|
+
A deploy is also refused **before any of the candidate's processes start** when the
|
|
169
|
+
handoff would push the number of concurrently draining release groups past
|
|
170
|
+
[`releaseRetention.maxConcurrentDrainingReleaseGroups`](config.md#releaseretention) —
|
|
171
|
+
the incumbent the handoff is about to retire counts too. The incumbent keeps serving,
|
|
172
|
+
nothing of the candidate is started, and the error names every draining group with its
|
|
173
|
+
`progressing`/`timedOut`/`wedged` classification and the bound it exceeded.
|
|
174
|
+
|
|
175
|
+
The same pre-activation refusal applies when `--release-id` names a release Rollbridge
|
|
176
|
+
still **retains** while that generation holds live work: draining with processes alive, or
|
|
177
|
+
recorded drained while a process group outlived its tracked process
|
|
178
|
+
(`processes[].heldProcessGroup`). Starting a second generation under that id re-registers
|
|
179
|
+
the guardian process keys the retained generation still holds, so the deploy aborts
|
|
180
|
+
partway through startup with `Guardian provenance mismatch for release:<id>:<key>` — the
|
|
181
|
+
shape a failed-deploy restore produced twice on 2026-11-15 (`20261115195051` at 20:29:55Z
|
|
182
|
+
and again at 20:41:26Z, 12 minutes apart). The refusal names the retained state, each held
|
|
183
|
+
process, and the release the guardian reports as active, and leaves the retained generation
|
|
184
|
+
untouched. A generation whose processes are genuinely gone stays re-deployable: the
|
|
185
|
+
2026-10-25 carve-out for re-registering a dead generation is unchanged.
|
|
186
|
+
|
|
166
187
|
With an opt-in handoff-service `lifecycle.activateCommand`, deploy journals the
|
|
167
188
|
exact transition, waits for old retirement acknowledgement, waits for candidate
|
|
168
189
|
activation acknowledgement, and synchronously commits the active proxy target.
|
|
@@ -243,6 +264,21 @@ Errors when there is no previous release, the `--release-id` is not a retained
|
|
|
243
264
|
release, or the target is already active. Only releases Rollbridge still retains
|
|
244
265
|
(see [`releaseRetention`](config.md#releaseretention)) can be rolled back to.
|
|
245
266
|
|
|
267
|
+
It also refuses a target whose generation is recorded as drained while its
|
|
268
|
+
processes still hold live work — a process the retirement wait left alive, or a
|
|
269
|
+
process group that outlived the tracked process (`processes[].heldProcessGroup`,
|
|
270
|
+
see [`status`](#status)). Re-deploying that release id would re-register its
|
|
271
|
+
guardian process keys under a new provenance while the old processes still hold
|
|
272
|
+
them, and the guardian answers `Guardian provenance mismatch for
|
|
273
|
+
release:<id>:<key>` after the deploy has already begun. The refusal is raised
|
|
274
|
+
before anything is touched and names each held process together with the release
|
|
275
|
+
the guardian reports as active, so the deploy tool reconciles its own `current`
|
|
276
|
+
pointer with `rollbridge status` instead of restoring a generation that already
|
|
277
|
+
drained. A target that is still **draining** is not refused here: stopping it is
|
|
278
|
+
what makes its id reusable, and the deploy it then runs applies the same fence
|
|
279
|
+
after that stop — so a stop that leaves the generation's process group alive is
|
|
280
|
+
reported as the refusal above instead of colliding mid-startup.
|
|
281
|
+
|
|
246
282
|
**Migration constraints.** Rollback only manages processes — it does **not**
|
|
247
283
|
revert database migrations or other external state. The target release's on-disk
|
|
248
284
|
directory must still exist, and its code must be compatible with the current
|
|
@@ -250,6 +286,65 @@ schema. Keep migrations backwards-compatible (the same rule that lets old and
|
|
|
250
286
|
new releases overlap during a deploy) so rolling code back to a retained release
|
|
251
287
|
stays safe.
|
|
252
288
|
|
|
289
|
+
## `compose-roll`
|
|
290
|
+
|
|
291
|
+
```
|
|
292
|
+
rollbridge compose-roll [--config <path>] --unit <id> [--image <reference>]
|
|
293
|
+
```
|
|
294
|
+
|
|
295
|
+
Rolls one declared Compose unit (see [`compose[]`](config.md#compose)) with the
|
|
296
|
+
same handshake a process deploy uses: quiesce it, wait for the work it already
|
|
297
|
+
accepted to settle, recreate the container with the target image, gate on
|
|
298
|
+
activation, and roll back to the retained previous image when activation fails.
|
|
299
|
+
`--image` overrides the unit's configured `image` for this roll.
|
|
300
|
+
|
|
301
|
+
Prints the roll result JSON and exits non-zero when `status` is `failed`:
|
|
302
|
+
|
|
303
|
+
```json
|
|
304
|
+
{"unitId": "gateway", "releaseId": "20261104104321888", "status": "rolled", "stage": "activation", "image": "registry.local/gateway:2.4.0", "previousImage": "registry.local/gateway:2.3.1", "containerId": "ba108432e759", "generations": [...], "rollback": null, "error": null}
|
|
305
|
+
```
|
|
306
|
+
|
|
307
|
+
`stage` names the stage that ended the roll: `drain`, `in_flight`, `recreate`,
|
|
308
|
+
`activation`, `rollback`, or `rollback_failed`. A `drain` or `in_flight` failure
|
|
309
|
+
means the container was **never recreated** — the old one keeps serving. An
|
|
310
|
+
`activation` failure recreates the previous image and reports `status:
|
|
311
|
+
"rolled_back"` with `rollback.image`; if that rollback cannot activate either,
|
|
312
|
+
the result is `stage: "rollback_failed"` and a non-zero exit.
|
|
313
|
+
|
|
314
|
+
Errors (exit non-zero before any change) when the unit is not declared in the
|
|
315
|
+
config, its image reference is unpinned or malformed, a release generation
|
|
316
|
+
transition is unresolved (`resume the exact deploy first`), or its container
|
|
317
|
+
publishes the Rollbridge proxy port — a Compose unit must never replace the
|
|
318
|
+
proxied entry point in place.
|
|
319
|
+
|
|
320
|
+
## `compose-rollback`
|
|
321
|
+
|
|
322
|
+
```
|
|
323
|
+
rollbridge compose-rollback [--config <path>] --unit <id> [--image <reference>]
|
|
324
|
+
```
|
|
325
|
+
|
|
326
|
+
Rolls a Compose unit back to an earlier image through the same drain-first path:
|
|
327
|
+
with no `--image` it selects the image retained before the newest activated one
|
|
328
|
+
in `composeStatePath`, so the tag `compose-roll` replaced is the target. Prints
|
|
329
|
+
the same result JSON as `compose-roll` and exits non-zero when `status` is
|
|
330
|
+
`failed`. A successful explicit rollback reports `status: "rolled"` (the unit
|
|
331
|
+
rolled to the image you named); an automatic rollback after a failed activation
|
|
332
|
+
reports `status: "rolled_back"`.
|
|
333
|
+
|
|
334
|
+
Errors when no earlier image is retained for the unit — pass `--image
|
|
335
|
+
<reference>` to roll back explicitly to a tag Rollbridge has never rolled to.
|
|
336
|
+
|
|
337
|
+
## `compose-status`
|
|
338
|
+
|
|
339
|
+
```
|
|
340
|
+
rollbridge compose-status [--config <path>] --unit <id>
|
|
341
|
+
```
|
|
342
|
+
|
|
343
|
+
Prints the unit's live container (`containerId`, `state`, `healthStatus`,
|
|
344
|
+
`image`, published `ports`) and every retained generation (`image`,
|
|
345
|
+
`previousImage`, `status`, `releaseId`, `containerId`, `activatedAt`), plus the
|
|
346
|
+
`ledgerPath` they are stored in. Read-only: it never changes the container.
|
|
347
|
+
|
|
253
348
|
## `status`
|
|
254
349
|
|
|
255
350
|
```
|
|
@@ -264,6 +359,36 @@ Memory-supervised processes also report `rssBytes`, `memoryRestarts`,
|
|
|
264
359
|
`lastMemoryRestartAt`, and `children` (the process tree: each group member's
|
|
265
360
|
`pid`, `command`, and `rssBytes`).
|
|
266
361
|
|
|
362
|
+
Every process also reports `retirementTimedOut` — `true` once the supervisor has
|
|
363
|
+
stopped waiting for it — and `heldProcessGroup` (`{pgid, liveMemberCount}`) when a
|
|
364
|
+
bounded stop left the process group running after the tracked process exited. That
|
|
365
|
+
group is the leak a `stopped` record otherwise hides: nothing in Rollbridge can
|
|
366
|
+
signal or wait for it again, yet it still holds the generation's work. It is
|
|
367
|
+
re-checked at most once every 5 seconds, and clears itself when its members leave.
|
|
368
|
+
|
|
369
|
+
Each **draining** release reports a `drain` classification — `status`, the `reason`
|
|
370
|
+
behind it, the `sampleWindowMs` it rests on, and `cpuDeltaMs`:
|
|
371
|
+
|
|
372
|
+
- `progressing` — the generation still owns live connections inside
|
|
373
|
+
`proxy.drainTimeoutMs`, a process is still inside its retirement budget, or its
|
|
374
|
+
process-group CPU advanced across the sample window. A legitimate multi-hour drain reports this.
|
|
375
|
+
- `timedOut` — the generation still owns live connections, but the configured drain budget has
|
|
376
|
+
already expired: `drain budget 60000ms exceeded by 900000ms with 1 live connection still owned
|
|
377
|
+
by the generation`. The connection wait was given up at `proxy.drainTimeoutMs`; the record says
|
|
378
|
+
so instead of claiming progress for as long as one socket stays open. It is a report only —
|
|
379
|
+
`timedOut` never authorises a signal, and escalation still requires `wedged`.
|
|
380
|
+
- `wedged` — the drain can no longer progress: no live connection, every
|
|
381
|
+
retirement budget expired, and either no runner child remains or the group's CPU
|
|
382
|
+
is flat across a window of at least 20 seconds.
|
|
383
|
+
- `unproven` — the evidence is inconclusive: a first observation, a window shorter
|
|
384
|
+
than 20 seconds, or a host where process CPU cannot be read.
|
|
385
|
+
|
|
386
|
+
`drain` is `undefined` for a release that is not draining. Gate on this
|
|
387
|
+
classification rather than on a raw process count: a `progressing` group holding
|
|
388
|
+
three live processes is a healthy drain, a `wedged` group holding one is a
|
|
389
|
+
generation that will never finish. Observations are taken at most once per window
|
|
390
|
+
per draining group, so repeated `status` reads stay cheap.
|
|
391
|
+
|
|
267
392
|
`--no-logs` omits only the captured `logs` array from release, service, and
|
|
268
393
|
singleton process statuses. Use it for bounded machine-readable lifecycle
|
|
269
394
|
attestations that do not need process output. The equivalent control request is
|
|
@@ -0,0 +1,222 @@
|
|
|
1
|
+
# Compose service units
|
|
2
|
+
|
|
3
|
+
A Compose unit is a Docker Compose service that Rollbridge updates with the same handshake it
|
|
4
|
+
already uses for its Node processes: quiesce the unit, wait for the work it already accepted to
|
|
5
|
+
finish, recreate the container with the new image, gate on activation, and roll back to the
|
|
6
|
+
previous image when activation fails.
|
|
7
|
+
|
|
8
|
+
Before this, every Compose service in the same delivery as a Rollbridge-managed app was updated
|
|
9
|
+
with a raw `docker compose up -d --force-recreate <service>`. That command has no drain (in-flight
|
|
10
|
+
work dies with the container), no activation gate (a broken image becomes the live service), and
|
|
11
|
+
no rollback (the previous tag is whatever the operator last typed). On the Socketduct gateway legs
|
|
12
|
+
and the builder DinDs that cost is visible: in-flight builds die, and a gateway leg that is
|
|
13
|
+
half-rolled leaves its Docker server `online: false` with `storageHealthReason: transport_failure`.
|
|
14
|
+
|
|
15
|
+
## The roll, stage by stage
|
|
16
|
+
|
|
17
|
+
`rollbridge compose-roll --unit <id>` runs these stages in order. Each stage that fails stops the
|
|
18
|
+
roll and says which stage failed.
|
|
19
|
+
|
|
20
|
+
1. **Preflight.** The target image reference is checked, and the roll refuses if a Rollbridge
|
|
21
|
+
release generation transition is unresolved (`Cannot roll Compose unit ... while release
|
|
22
|
+
generation transition <id> is unresolved at <phase>; resume the exact deploy first`). It also
|
|
23
|
+
refuses if the unit's container publishes the Rollbridge proxy port — a Compose unit must never
|
|
24
|
+
replace the proxied entry point in place.
|
|
25
|
+
2. **Drain.** `drain.command` runs. This is the hook that tells the service to stop admitting new
|
|
26
|
+
work. If it exits non-zero, the roll fails at `stage: "drain"` and the container is **not**
|
|
27
|
+
recreated.
|
|
28
|
+
3. **In-flight wait.** `inFlight.command` is polled until it exits 0 (the unit's accepted work has
|
|
29
|
+
settled) or `inFlight.timeoutMs` expires. Expiry fails the roll at `stage: "in_flight"` and the
|
|
30
|
+
container is **not** recreated — Rollbridge never recreates anyway because the drain timed out.
|
|
31
|
+
4. **Recreate.** `docker compose ... up -d --no-deps --force-recreate <service>` runs with a
|
|
32
|
+
generated override file that pins the target image and the unit's `stop_grace_period`. The
|
|
33
|
+
override is written next to the Compose file, used once, and deleted; the operator's Compose
|
|
34
|
+
file is never rewritten. `--no-deps` keeps the roll to this one service.
|
|
35
|
+
5. **Activation gate.** The new container must be `running`, pass its Compose healthcheck (when it
|
|
36
|
+
has one), not front the proxy port, and pass `activation.readyCommand` and/or
|
|
37
|
+
`activation.readyUrl` if configured. Polling runs until `activation.timeoutMs`.
|
|
38
|
+
6. **Rollback.** If activation fails and a previous image is retained, the previous image is
|
|
39
|
+
recreated through the same path and gated the same way. Success reports `status: "rolled_back"`
|
|
40
|
+
with `rollback.image`; if the rollback itself cannot activate, the roll reports
|
|
41
|
+
`stage: "rollback_failed"` and exits non-zero instead of pretending the unit is healthy.
|
|
42
|
+
|
|
43
|
+
Every activated or rolled-back generation is appended to the unit's ledger
|
|
44
|
+
(`composeStatePath`), keeping the image it replaced. That retained tag is what
|
|
45
|
+
`compose-rollback` selects.
|
|
46
|
+
|
|
47
|
+
## Commands
|
|
48
|
+
|
|
49
|
+
```bash
|
|
50
|
+
rollbridge compose-roll --config rollbridge.js --unit gateway
|
|
51
|
+
rollbridge compose-roll --config rollbridge.js --unit gateway --image registry.local/gateway:2.4.0
|
|
52
|
+
rollbridge compose-rollback --config rollbridge.js --unit gateway
|
|
53
|
+
rollbridge compose-status --config rollbridge.js --unit gateway
|
|
54
|
+
```
|
|
55
|
+
|
|
56
|
+
- `compose-roll` targets the unit's configured `image` unless `--image` names another reference.
|
|
57
|
+
- `compose-rollback` selects the retained previous image from the ledger unless `--image` names one.
|
|
58
|
+
- Both print the roll result JSON and exit non-zero when `status` is `failed`, so deploy tooling
|
|
59
|
+
can gate on them.
|
|
60
|
+
- `compose-status` prints the live container (id, state, health, image, published ports) and the
|
|
61
|
+
retained generations.
|
|
62
|
+
|
|
63
|
+
Result shape:
|
|
64
|
+
|
|
65
|
+
```json
|
|
66
|
+
{
|
|
67
|
+
"unitId": "gateway",
|
|
68
|
+
"releaseId": "20261104104321888",
|
|
69
|
+
"status": "rolled",
|
|
70
|
+
"stage": "activation",
|
|
71
|
+
"image": "registry.local/gateway:2.4.0",
|
|
72
|
+
"previousImage": "registry.local/gateway:2.3.1",
|
|
73
|
+
"containerId": "ba108432e759",
|
|
74
|
+
"generations": [{"image": "registry.local/gateway:2.4.0", "previousImage": "registry.local/gateway:2.3.1", "status": "activated", "releaseId": "20261104104321888", "containerId": "ba108432e759", "activatedAt": "2026-11-04T10:43:27.258Z", "unitId": "gateway"}],
|
|
75
|
+
"rollback": null,
|
|
76
|
+
"error": null
|
|
77
|
+
}
|
|
78
|
+
```
|
|
79
|
+
|
|
80
|
+
`status` is `rolled`, `rolled_back`, or `failed`; `stage` is `drain`, `in_flight`, `recreate`,
|
|
81
|
+
`activation`, `rollback`, or `rollback_failed`.
|
|
82
|
+
|
|
83
|
+
## Fleet runbook
|
|
84
|
+
|
|
85
|
+
These replace `docker compose up -d --force-recreate <service>` for the three Compose services in
|
|
86
|
+
the fleet. In each case the Compose file stays where it is; Rollbridge only pins the image for the
|
|
87
|
+
roll. Paths, project names, and image names below are illustrative — take the real ones from that
|
|
88
|
+
host's Compose file, and remember a Compose unit can only be rolled to an image the daemon on that
|
|
89
|
+
host can already resolve.
|
|
90
|
+
|
|
91
|
+
### Socketduct hub
|
|
92
|
+
|
|
93
|
+
```js
|
|
94
|
+
// /srv/socketduct-hub/rollbridge.js
|
|
95
|
+
export default {
|
|
96
|
+
application: "socketduct-hub",
|
|
97
|
+
proxy: {host: "127.0.0.1", port: 8182},
|
|
98
|
+
processes: [/* the proxied hub process, unchanged */],
|
|
99
|
+
compose: [{
|
|
100
|
+
id: "hub",
|
|
101
|
+
service: "hub",
|
|
102
|
+
projectDirectory: "/srv/socketduct-hub",
|
|
103
|
+
composeFile: "compose.yml",
|
|
104
|
+
projectName: "socketduct-hub",
|
|
105
|
+
image: "registry.diestoeckels.de/socketduct/hub:1.9.2",
|
|
106
|
+
stopGracePeriodMs: 30000,
|
|
107
|
+
drain: {command: "{{composeArgs}} exec -T hub appctl drain", timeoutMs: 60000},
|
|
108
|
+
inFlight: {command: "{{composeArgs}} exec -T hub appctl in-flight", intervalMs: 2000, timeoutMs: 300000},
|
|
109
|
+
activation: {startDelayMs: 2000, timeoutMs: 90000, readyUrl: "http://127.0.0.1:7420/ready"}
|
|
110
|
+
}]
|
|
111
|
+
}
|
|
112
|
+
```
|
|
113
|
+
|
|
114
|
+
```bash
|
|
115
|
+
rollbridge doctor --config /srv/socketduct-hub/rollbridge.js # preflights the compose CLI, the file, and the service
|
|
116
|
+
rollbridge compose-roll --config /srv/socketduct-hub/rollbridge.js --unit hub --image registry.diestoeckels.de/socketduct/hub:1.9.3
|
|
117
|
+
rollbridge compose-status --config /srv/socketduct-hub/rollbridge.js --unit hub
|
|
118
|
+
```
|
|
119
|
+
|
|
120
|
+
### One Socketduct gateway leg
|
|
121
|
+
|
|
122
|
+
A gateway leg is the per-host `socketduct-<host>-<class>` gateway. Rolling one leg must not touch
|
|
123
|
+
the other legs, so the unit names only its own project directory and service:
|
|
124
|
+
|
|
125
|
+
```js
|
|
126
|
+
// /srv/socketduct/socketduct-builder-a-standard/rollbridge.js
|
|
127
|
+
export default {
|
|
128
|
+
application: "socketduct-builder-a-standard",
|
|
129
|
+
proxy: {host: "127.0.0.1", port: 8182},
|
|
130
|
+
processes: [/* the proxied gateway process, unchanged */],
|
|
131
|
+
compose: [{
|
|
132
|
+
id: "gateway",
|
|
133
|
+
service: "gateway",
|
|
134
|
+
projectDirectory: "/srv/socketduct/socketduct-builder-a-standard",
|
|
135
|
+
composeFile: "compose.yml",
|
|
136
|
+
projectName: "socketduct-builder-a-standard",
|
|
137
|
+
image: "registry.diestoeckels.de/socketduct/gateway:2.4.1",
|
|
138
|
+
stopGracePeriodMs: 45000,
|
|
139
|
+
drain: {command: "{{composeArgs}} exec -T gateway appctl drain", timeoutMs: 120000},
|
|
140
|
+
inFlight: {command: "{{composeArgs}} exec -T gateway appctl in-flight", intervalMs: 2000, timeoutMs: 900000},
|
|
141
|
+
activation: {startDelayMs: 2000, timeoutMs: 120000, readyUrl: "http://127.0.0.1:7421/ready"}
|
|
142
|
+
}]
|
|
143
|
+
}
|
|
144
|
+
```
|
|
145
|
+
|
|
146
|
+
```bash
|
|
147
|
+
rollbridge compose-roll --config /srv/socketduct/socketduct-builder-a-standard/rollbridge.js --unit gateway
|
|
148
|
+
rollbridge compose-rollback --config /srv/socketduct/socketduct-builder-a-standard/rollbridge.js --unit gateway
|
|
149
|
+
```
|
|
150
|
+
|
|
151
|
+
The long `inFlight.timeoutMs` is deliberate: a gateway leg that is mid-handoff should finish it.
|
|
152
|
+
If it never settles, the roll fails and the leg keeps serving the old image — which is the
|
|
153
|
+
behaviour that keeps the Docker server `online: true` instead of leaving it
|
|
154
|
+
`storageHealthReason: transport_failure`.
|
|
155
|
+
|
|
156
|
+
Gateway images are pinned by digest with `pull_policy: never`, so they cannot be updated by
|
|
157
|
+
pulling: `compose[].image` names the new `@sha256:` digest (or a tag verified present on that
|
|
158
|
+
machine), and the image has to be loaded on the host before the roll. A roll never touches the
|
|
159
|
+
other services in the project — `--no-deps` means an unsatisfiable sibling service can no longer
|
|
160
|
+
stop this gateway from being recreated.
|
|
161
|
+
|
|
162
|
+
### One builder DinD
|
|
163
|
+
|
|
164
|
+
The builder DinD keeps its established identities exactly: Compose project
|
|
165
|
+
`peakflow_builder`, service `docker-server`, container `peakflow_builder-docker-server-1`.
|
|
166
|
+
|
|
167
|
+
```js
|
|
168
|
+
// /srv/peakflow-builder/rollbridge.js
|
|
169
|
+
export default {
|
|
170
|
+
application: "peakflow-builder",
|
|
171
|
+
proxy: {host: "127.0.0.1", port: 8182},
|
|
172
|
+
processes: [/* the proxied builder API process, unchanged */],
|
|
173
|
+
compose: [{
|
|
174
|
+
id: "builder-dind",
|
|
175
|
+
service: "docker-server",
|
|
176
|
+
projectDirectory: "/srv/peakflow-builder",
|
|
177
|
+
composeFile: "compose.yml",
|
|
178
|
+
projectName: "peakflow_builder",
|
|
179
|
+
image: "ghcr.io/kaspernj/peakflow-builder:amd64",
|
|
180
|
+
stopGracePeriodMs: 60000,
|
|
181
|
+
inFlight: {command: "node /srv/peakflow-builder/scripts/quiescence-probe.js", intervalMs: 5000, timeoutMs: 1800000},
|
|
182
|
+
activation: {startDelayMs: 5000, timeoutMs: 180000, readyCommand: "docker inspect -f '{{.State.Running}}' peakflow_builder-docker-server-1 | grep -q true"}
|
|
183
|
+
}]
|
|
184
|
+
}
|
|
185
|
+
```
|
|
186
|
+
|
|
187
|
+
```bash
|
|
188
|
+
rollbridge compose-roll --config /srv/peakflow-builder/rollbridge.js --unit builder-dind
|
|
189
|
+
```
|
|
190
|
+
|
|
191
|
+
A DinD that is mid-build is the case the drain exists for: `inFlight.timeoutMs` is the budget for
|
|
192
|
+
the builds already admitted, and expiry keeps the old DinD serving rather than killing them. This
|
|
193
|
+
example sets no `drain.command` because the builder DinD has no quiescence hook to call today —
|
|
194
|
+
until it exposes one, the in-flight gate is the only protection an admitted build has, and the
|
|
195
|
+
roll is strictly better than `up -d --force-recreate` but not yet a true drain.
|
|
196
|
+
|
|
197
|
+
## What is proven, and what is not
|
|
198
|
+
|
|
199
|
+
- **Proven by tests** (`test/compose-unit.test.js`, `test/compose-config.test.js`): drain-first
|
|
200
|
+
ordering, drain/in-flight failure never recreating, a `docker compose up` the daemon refuses
|
|
201
|
+
(for example a `pull_policy: never` gateway digest that is not loaded on that host) failing the
|
|
202
|
+
roll at `stage: "recreate"` with the previous container still running, its image still selected
|
|
203
|
+
and the ledger untouched, activation failure rolling back to the retained tag, a failed rollback
|
|
204
|
+
reported as `rollback_failed`, `compose-rollback` selecting the retained tag, refusal to front
|
|
205
|
+
the proxy port, refusal during an unresolved generation transition, and the config validation
|
|
206
|
+
rules.
|
|
207
|
+
- **Proven against a real Docker Compose daemon** (`test/compose-docker.test.js`, opt-in with
|
|
208
|
+
`ROLLBRIDGE_COMPOSE_DOCKER_TEST=1`): a real Compose service is rolled drain-first (the container
|
|
209
|
+
records `drain` before `settled`, and the new image starts only after that), the new image is
|
|
210
|
+
live afterwards, the previous tag is retained in the ledger, and rolling to an image whose
|
|
211
|
+
healthcheck never passes leaves the previous container serving and answering its readiness
|
|
212
|
+
endpoint. It also proves the proxy-port refusal against a real published port.
|
|
213
|
+
- **Not implemented here**: the roll only runs whatever command a unit names; the hooks themselves
|
|
214
|
+
live in the service images. The Socketduct hub and gateway images now ship `appctl drain` and
|
|
215
|
+
`appctl in-flight` with the exit contract this document gates on (`0` the condition holds,
|
|
216
|
+
`1` it does not hold yet, `2` the probe cannot answer), so the example commands above name
|
|
217
|
+
something those images ship. The builder DinD still has no drain hook — its gate is the
|
|
218
|
+
in-flight probe in `kaspernj/tensorbuzz-builder` (`scripts/quiescence-probe.js`), which reports
|
|
219
|
+
whether a build is mid-flight.
|
|
220
|
+
- **Not changed here**: TensorBuzz's Socketduct provisioning path still issues raw
|
|
221
|
+
`docker compose up -d --force-recreate` when it provisions gateway legs. Switching that backend
|
|
222
|
+
to call `rollbridge compose-roll` is a TensorBuzz change, tracked separately from this one.
|