rollbridge 0.1.56 → 0.1.59
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +44 -0
- package/TODO.md +12 -0
- package/changelog.d/20261013-recover-dead-guardian-state.md +1 -0
- package/changelog.d/20261025-guardian-re-register-dead-generation.md +13 -0
- package/changelog.d/20261026-stale-committed-generation-transition-recovery.md +1 -0
- package/changelog.d/20261104-compose-service-units.md +21 -0
- package/changelog.d/20261105-owner-mutation-lease-diagnostics.md +20 -0
- package/docs/cli.md +70 -1
- package/docs/compose-services.md +220 -0
- package/docs/config.md +55 -0
- package/docs/troubleshooting.md +24 -0
- package/package.json +1 -1
- package/src/cli.js +82 -4
- package/src/compose-unit.js +725 -0
- package/src/config.js +201 -2
- package/src/daemon.js +23 -8
- package/src/doctor.js +67 -0
- package/src/guardian-client.js +13 -2
- package/src/process-guardian.js +137 -17
- package/src/recover.js +128 -8
- package/test/completion.test.js +5 -2
- package/test/compose-config.test.js +158 -0
- package/test/compose-docker.test.js +297 -0
- package/test/compose-unit.test.js +415 -0
- package/test/fixtures/gated-app.js +50 -0
- package/test/guardian-client.test.js +70 -0
- package/test/recover.test.js +119 -0
- package/test/rollbridge.test.js +106 -6
package/README.md
CHANGED
|
@@ -317,6 +317,34 @@ rendered when the process starts:
|
|
|
317
317
|
Referencing a placeholder with no value (including an unset `{{env.<NAME>}}`)
|
|
318
318
|
fails the process start with a clear error, so typos surface immediately.
|
|
319
319
|
|
|
320
|
+
### Docker Compose services
|
|
321
|
+
|
|
322
|
+
A delivery that also updates Docker Compose services (a Socketduct hub, a gateway
|
|
323
|
+
leg, a builder DinD) declares them in `compose`. Each unit is rolled with the
|
|
324
|
+
same handshake a process deploy uses — quiesce, wait for in-flight work, recreate
|
|
325
|
+
with the target image, gate on activation, roll back to the retained previous
|
|
326
|
+
image on failure — instead of a raw `docker compose up -d --force-recreate`:
|
|
327
|
+
|
|
328
|
+
```js
|
|
329
|
+
compose: [
|
|
330
|
+
{
|
|
331
|
+
id: "gateway",
|
|
332
|
+
service: "gateway",
|
|
333
|
+
projectDirectory: "/srv/socketduct/socketduct-builder-a-standard",
|
|
334
|
+
image: "registry.diestoeckels.de/socketduct/gateway:2.4.1",
|
|
335
|
+
stopGracePeriodMs: 45000,
|
|
336
|
+
drain: {command: "{{composeArgs}} exec -T gateway appctl drain", timeoutMs: 120000},
|
|
337
|
+
inFlight: {command: "{{composeArgs}} exec -T gateway appctl in-flight", intervalMs: 2000, timeoutMs: 900000},
|
|
338
|
+
activation: {startDelayMs: 2000, timeoutMs: 120000, readyUrl: "http://127.0.0.1:7421/ready"}
|
|
339
|
+
}
|
|
340
|
+
]
|
|
341
|
+
```
|
|
342
|
+
|
|
343
|
+
A drain or in-flight timeout fails the roll and leaves the current container
|
|
344
|
+
serving; a unit whose container publishes the Rollbridge proxy port is never
|
|
345
|
+
rolled. See [`docs/compose-services.md`](docs/compose-services.md) for the
|
|
346
|
+
stage-by-stage behavior and the fleet runbook.
|
|
347
|
+
|
|
320
348
|
Configuration examples live in `examples/`, including
|
|
321
349
|
`examples/tensorbuzz.com.js` for the current TensorBuzz backend deployment; see
|
|
322
350
|
[`docs/tensorbuzz-runbook.md`](docs/tensorbuzz-runbook.md) for the matching
|
|
@@ -625,6 +653,22 @@ rollbridge rollback --config rollbridge.js # the previous relea
|
|
|
625
653
|
rollbridge rollback --config rollbridge.js --release-id v3
|
|
626
654
|
```
|
|
627
655
|
|
|
656
|
+
Roll one declared Docker Compose service drain-first (quiesce it, wait for its
|
|
657
|
+
in-flight work, recreate it with the target image, gate on activation, roll back
|
|
658
|
+
to the retained previous image if activation fails), roll it back to that
|
|
659
|
+
retained image, or read its live container and retained generations:
|
|
660
|
+
|
|
661
|
+
```bash
|
|
662
|
+
rollbridge compose-roll --config rollbridge.js --unit gateway
|
|
663
|
+
rollbridge compose-roll --config rollbridge.js --unit gateway --image registry.local/gateway:2.4.0
|
|
664
|
+
rollbridge compose-rollback --config rollbridge.js --unit gateway
|
|
665
|
+
rollbridge compose-status --config rollbridge.js --unit gateway
|
|
666
|
+
```
|
|
667
|
+
|
|
668
|
+
Each prints the roll result JSON and exits non-zero when `status` is `failed`. A
|
|
669
|
+
drain or in-flight timeout fails the roll **before** anything is recreated, so
|
|
670
|
+
the current container keeps serving.
|
|
671
|
+
|
|
628
672
|
Restart non-proxied processes in place — all of them, one by id, or a policy
|
|
629
673
|
group (the proxied process is never restarted; use `deploy` for that):
|
|
630
674
|
|
package/TODO.md
CHANGED
|
@@ -71,6 +71,18 @@ This roadmap tracks planned Rollbridge features and documentation. Rollbridge sh
|
|
|
71
71
|
- [x] Validate unsupported lifecycle-hook combinations once worker lifecycle hooks land. (`lifecycle.drainCommand` requires a positive `drainTimeoutMs`; `nonBlockingDrain` is companion-only; a `lifecycle.stopCommand` may not be combined with a custom `stopSignal`, since the command runs instead of the signal.)
|
|
72
72
|
- [x] Include example fixes in validation output.
|
|
73
73
|
|
|
74
|
+
## Compose service units
|
|
75
|
+
|
|
76
|
+
- [x] Model a Docker Compose service as a Rollbridge-managed unit (`compose[]`): project directory, service name, target image, `stop_grace_period`, drain command and timeout, in-flight gate, and activation probe (Compose healthcheck plus optional ready command/URL).
|
|
77
|
+
- [x] Roll one unit drain-first — quiesce, wait for accepted work to settle, recreate only that service, gate on activation — instead of a raw `docker compose up -d --force-recreate` (`rollbridge compose-roll`, `compose-rollback`, `compose-status`).
|
|
78
|
+
- [x] Retain the replaced image in the generation ledger (`composeStatePath`) so the previous tag stays selectable and rollback is a first-class operation, not a manual compose invocation.
|
|
79
|
+
- [x] Keep the proxied-entry-point guarantee: a unit whose container publishes the Rollbridge proxy port is refused, and a roll never runs while a release generation transition is unresolved.
|
|
80
|
+
- [x] Define drain and in-flight timeout behaviour: both fail the roll before anything is recreated, so the current container keeps serving; an activation failure recreates the previous image and reports `rolled_back`.
|
|
81
|
+
- [x] Operator runbook for the Socketduct hub, one gateway leg, and one builder DinD (`docs/compose-services.md`).
|
|
82
|
+
- [ ] Ship the quiescence hook (`appctl drain`) and an in-flight probe on the hub, gateway, and builder DinD images. Until they exist, the in-flight gate is the only protection work already admitted has, and a roll is better than `up -d --force-recreate` but not a true drain.
|
|
83
|
+
- [ ] Switch TensorBuzz's Socketduct provisioning path (`backend/src/services/docker-servers/socketduct-gateway-provisioning-service.js`) from raw `up -d --force-recreate` to `rollbridge compose-roll`. That is a TensorBuzz change, not a Rollbridge one.
|
|
84
|
+
- [ ] Prove a drain-first roll on a real builder host. The current proof is a real Compose daemon (`test/compose-docker.test.js`, opt-in), not a production builder.
|
|
85
|
+
|
|
74
86
|
## Minor Features
|
|
75
87
|
|
|
76
88
|
- [x] Add a control-socket permission option (`control.mode`) for shared deploy users.
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
Recover now recognizes a persisted state that conclusively names a dead guardian — its recorded pid is gone and nothing is listening on its Unix socket — and, once every orphaned process is confirmed stopped, archives the state under a timestamped evidence name (bytes preserved verbatim) while moving the dead socket metadata aside, instead of clearing it. A later deploy can then boot a fresh generation through the ordinary recovery path. This automates the manual state-level recovery for the documented dead-guardian dead-end (host precedent archived 2026-08-30 and 2026-09-21; TensorBuzz production runbook `docs/production-rollbridge-recovery.md`), where every boot path previously failed with no automatic recovery. Any ambiguity in the provenance check keeps the existing fail-closed behavior.
|
|
@@ -0,0 +1,13 @@
|
|
|
1
|
+
### Fixed
|
|
2
|
+
|
|
3
|
+
- A guardian provenance mismatch no longer fences a key whose registered child is
|
|
4
|
+
conclusively gone. When a deploy fails after the daemon committed and switched
|
|
5
|
+
traffic, the old release's processes are retired while the guardian still holds
|
|
6
|
+
their provenance record. Every later deploy reconstructed the same key from a new
|
|
7
|
+
release definition and was rejected with `Guardian provenance mismatch for
|
|
8
|
+
release:<id>:<key>` — including for a release with no live processes at all —
|
|
9
|
+
wedging every subsequent deploy (TensorBuzz production, STC-695). The mismatch
|
|
10
|
+
now only fails closed while the registered process is live (or has an unknown
|
|
11
|
+
state); a `stopped`/`failed` record with no pid is residue of a dead generation
|
|
12
|
+
and the committed owner may re-register the key under the new definition. Live
|
|
13
|
+
processes under a foreign provenance stay rejected.
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
Recover-generation-transition now recognizes a committed journal whose candidate is the live incumbent: when the recovery fence names the release that is actually running, the transition that published it is already complete, so recovery reports `already_recovered` instead of refusing the only fence that describes the authoritative release. A deploy that fails before its own transition is journaled (for example at candidate health) is now recoverable by rerunning the deploy, with no hand-edits to runtime state. The stale-fence guard keeps failing closed for every genuinely in-flight phase (`activating_candidate`, `retiring_previous`, `restoring_previous`, `retiring_failed_candidate`, `degraded_active`) and for committed journals whose fence names any other release.
|
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
### Added
|
|
2
|
+
|
|
3
|
+
- Docker Compose services declared in `compose[]` are now Rollbridge-managed units rolled
|
|
4
|
+
drain-first instead of a raw `docker compose up -d --force-recreate`. `rollbridge
|
|
5
|
+
compose-roll --unit <id> [--image <ref>]` quiesces the unit (`drain.command`), polls its
|
|
6
|
+
in-flight gate until the work it already accepted settles, recreates only that service
|
|
7
|
+
(`up -d --no-deps --force-recreate`) through a generated override pinning the target image and
|
|
8
|
+
`stop_grace_period`, then gates on activation (container running, Compose healthcheck, optional
|
|
9
|
+
`readyCommand`/`readyUrl) before recording the generation. `compose-rollback` reuses the same
|
|
10
|
+
path to select the retained previous tag, and `compose-status` reports the live container and
|
|
11
|
+
the retained generations. Each roll result names its `stage` and exits non-zero on `failed`.
|
|
12
|
+
- A drain or in-flight failure fails the roll **before** anything is recreated, so the current
|
|
13
|
+
container keeps serving; an activation failure recreates the previous image and reports
|
|
14
|
+
`rolled_back`, while a rollback that cannot activate reports `rollback_failed` instead of
|
|
15
|
+
success. Generations are retained in `composeStatePath`
|
|
16
|
+
(`rollbridge-<application>.compose.state.json` by default), keeping the replaced tag selectable.
|
|
17
|
+
- Compose rolls refuse to run while a release generation transition is unresolved, and refuse a
|
|
18
|
+
unit whose container publishes the Rollbridge proxy port — checked before the roll and again
|
|
19
|
+
inside the activation gate — so a Compose unit can never replace the proxied entry point in
|
|
20
|
+
place. `rollbridge doctor` preflights the Compose CLI, each unit's Compose file and env file,
|
|
21
|
+
and that its service is declared, and `validate` reports `compose` config issues with fixes.
|
|
@@ -0,0 +1,20 @@
|
|
|
1
|
+
### Fixed
|
|
2
|
+
|
|
3
|
+
- A refused owner mutation now names its holder. The refusal used to read
|
|
4
|
+
`Another owner mutation is already in progress` with no context, so an
|
|
5
|
+
operator could not tell which command held the fence, which process held it,
|
|
6
|
+
for how long, or what it was blocked on. The guardian now records the
|
|
7
|
+
mutation's operation, lease id, holding PID, start time and current blocker
|
|
8
|
+
(`waitingFor`, refinable while the mutation runs through
|
|
9
|
+
`reportOwnerMutationWait`) and reports them in every refusal, including the
|
|
10
|
+
replacement-prepare fence.
|
|
11
|
+
|
|
12
|
+
### Changed
|
|
13
|
+
|
|
14
|
+
- The owner-mutation fence is bounded. A mutation that outruns its lease
|
|
15
|
+
(default 15 minutes, per-command override through `mutationTimeoutMs`) is
|
|
16
|
+
logged as `OWNER_MUTATION_LEASE_EXPIRED`, reported to the holder, and
|
|
17
|
+
released, so a stuck mutation can no longer lock out every later deploy. The
|
|
18
|
+
holder fails loudly when it next ends that mutation instead of silently
|
|
19
|
+
keeping the fence. Retained generation drains are unaffected: they are not
|
|
20
|
+
owner mutations and may still run for hours.
|
package/docs/cli.md
CHANGED
|
@@ -250,6 +250,65 @@ schema. Keep migrations backwards-compatible (the same rule that lets old and
|
|
|
250
250
|
new releases overlap during a deploy) so rolling code back to a retained release
|
|
251
251
|
stays safe.
|
|
252
252
|
|
|
253
|
+
## `compose-roll`
|
|
254
|
+
|
|
255
|
+
```
|
|
256
|
+
rollbridge compose-roll [--config <path>] --unit <id> [--image <reference>]
|
|
257
|
+
```
|
|
258
|
+
|
|
259
|
+
Rolls one declared Compose unit (see [`compose[]`](config.md#compose)) with the
|
|
260
|
+
same handshake a process deploy uses: quiesce it, wait for the work it already
|
|
261
|
+
accepted to settle, recreate the container with the target image, gate on
|
|
262
|
+
activation, and roll back to the retained previous image when activation fails.
|
|
263
|
+
`--image` overrides the unit's configured `image` for this roll.
|
|
264
|
+
|
|
265
|
+
Prints the roll result JSON and exits non-zero when `status` is `failed`:
|
|
266
|
+
|
|
267
|
+
```json
|
|
268
|
+
{"unitId": "gateway", "releaseId": "20261104104321888", "status": "rolled", "stage": "activation", "image": "registry.local/gateway:2.4.0", "previousImage": "registry.local/gateway:2.3.1", "containerId": "ba108432e759", "generations": [...], "rollback": null, "error": null}
|
|
269
|
+
```
|
|
270
|
+
|
|
271
|
+
`stage` names the stage that ended the roll: `drain`, `in_flight`, `recreate`,
|
|
272
|
+
`activation`, `rollback`, or `rollback_failed`. A `drain` or `in_flight` failure
|
|
273
|
+
means the container was **never recreated** — the old one keeps serving. An
|
|
274
|
+
`activation` failure recreates the previous image and reports `status:
|
|
275
|
+
"rolled_back"` with `rollback.image`; if that rollback cannot activate either,
|
|
276
|
+
the result is `stage: "rollback_failed"` and a non-zero exit.
|
|
277
|
+
|
|
278
|
+
Errors (exit non-zero before any change) when the unit is not declared in the
|
|
279
|
+
config, its image reference is unpinned or malformed, a release generation
|
|
280
|
+
transition is unresolved (`resume the exact deploy first`), or its container
|
|
281
|
+
publishes the Rollbridge proxy port — a Compose unit must never replace the
|
|
282
|
+
proxied entry point in place.
|
|
283
|
+
|
|
284
|
+
## `compose-rollback`
|
|
285
|
+
|
|
286
|
+
```
|
|
287
|
+
rollbridge compose-rollback [--config <path>] --unit <id> [--image <reference>]
|
|
288
|
+
```
|
|
289
|
+
|
|
290
|
+
Rolls a Compose unit back to an earlier image through the same drain-first path:
|
|
291
|
+
with no `--image` it selects the image retained before the newest activated one
|
|
292
|
+
in `composeStatePath`, so the tag `compose-roll` replaced is the target. Prints
|
|
293
|
+
the same result JSON as `compose-roll` and exits non-zero when `status` is
|
|
294
|
+
`failed`. A successful explicit rollback reports `status: "rolled"` (the unit
|
|
295
|
+
rolled to the image you named); an automatic rollback after a failed activation
|
|
296
|
+
reports `status: "rolled_back"`.
|
|
297
|
+
|
|
298
|
+
Errors when no earlier image is retained for the unit — pass `--image
|
|
299
|
+
<reference>` to roll back explicitly to a tag Rollbridge has never rolled to.
|
|
300
|
+
|
|
301
|
+
## `compose-status`
|
|
302
|
+
|
|
303
|
+
```
|
|
304
|
+
rollbridge compose-status [--config <path>] --unit <id>
|
|
305
|
+
```
|
|
306
|
+
|
|
307
|
+
Prints the unit's live container (`containerId`, `state`, `healthStatus`,
|
|
308
|
+
`image`, published `ports`) and every retained generation (`image`,
|
|
309
|
+
`previousImage`, `status`, `releaseId`, `containerId`, `activatedAt`), plus the
|
|
310
|
+
`ledgerPath` they are stored in. Read-only: it never changes the container.
|
|
311
|
+
|
|
253
312
|
## `status`
|
|
254
313
|
|
|
255
314
|
```
|
|
@@ -368,7 +427,7 @@ Cleans up orphaned managed processes left by a **crashed** daemon. It reads the
|
|
|
368
427
|
persisted state ([`statePath`](config.md#statepath)) and finds managed processes
|
|
369
428
|
whose pids are still alive. Without `--force` it only **lists** them (a dry run);
|
|
370
429
|
with `--force` it stops each one's process group (`SIGTERM`, then `SIGKILL` after
|
|
371
|
-
`proxy.forceStopTimeoutMs`) and
|
|
430
|
+
`proxy.forceStopTimeoutMs`) and removes the stale state file (see below).
|
|
372
431
|
|
|
373
432
|
Run it **before** restarting the daemon after a crash. It refuses to run while a
|
|
374
433
|
daemon (or another process) holds the control socket — those pids belong to a
|
|
@@ -381,6 +440,16 @@ file is **kept** so you can investigate and re-run `recover`, and the command
|
|
|
381
440
|
exits non-zero. Requires `statePath`; also exits non-zero when it is unset or a
|
|
382
441
|
daemon is running.
|
|
383
442
|
|
|
443
|
+
Normally the state file is cleared once every orphan is stopped. When it
|
|
444
|
+
conclusively names a **dead guardian** — the recorded guardian pid is gone and
|
|
445
|
+
nothing is listening on its Unix socket — `--force` **archives** it instead,
|
|
446
|
+
under a timestamped `*.dead-guardian-<YYYYMMDDTHHmmZ>.json` evidence name, and
|
|
447
|
+
moves the dead guardian socket metadata aside. Such a state dead-ends every boot
|
|
448
|
+
path; archiving preserves the bytes as evidence while letting the next deploy
|
|
449
|
+
boot a fresh generation. The dry run and the report expose this as
|
|
450
|
+
`deadGuardianState`, `archived`, and `archivePath`; a live guardian listener or a
|
|
451
|
+
live recorded pid blocks the archive (fail closed).
|
|
452
|
+
|
|
384
453
|
## `shutdown`
|
|
385
454
|
|
|
386
455
|
```
|
|
@@ -0,0 +1,220 @@
|
|
|
1
|
+
# Compose service units
|
|
2
|
+
|
|
3
|
+
A Compose unit is a Docker Compose service that Rollbridge updates with the same handshake it
|
|
4
|
+
already uses for its Node processes: quiesce the unit, wait for the work it already accepted to
|
|
5
|
+
finish, recreate the container with the new image, gate on activation, and roll back to the
|
|
6
|
+
previous image when activation fails.
|
|
7
|
+
|
|
8
|
+
Before this, every Compose service in the same delivery as a Rollbridge-managed app was updated
|
|
9
|
+
with a raw `docker compose up -d --force-recreate <service>`. That command has no drain (in-flight
|
|
10
|
+
work dies with the container), no activation gate (a broken image becomes the live service), and
|
|
11
|
+
no rollback (the previous tag is whatever the operator last typed). On the Socketduct gateway legs
|
|
12
|
+
and the builder DinDs that cost is visible: in-flight builds die, and a gateway leg that is
|
|
13
|
+
half-rolled leaves its Docker server `online: false` with `storageHealthReason: transport_failure`.
|
|
14
|
+
|
|
15
|
+
## The roll, stage by stage
|
|
16
|
+
|
|
17
|
+
`rollbridge compose-roll --unit <id>` runs these stages in order. Each stage that fails stops the
|
|
18
|
+
roll and says which stage failed.
|
|
19
|
+
|
|
20
|
+
1. **Preflight.** The target image reference is checked, and the roll refuses if a Rollbridge
|
|
21
|
+
release generation transition is unresolved (`Cannot roll Compose unit ... while release
|
|
22
|
+
generation transition <id> is unresolved at <phase>; resume the exact deploy first`). It also
|
|
23
|
+
refuses if the unit's container publishes the Rollbridge proxy port — a Compose unit must never
|
|
24
|
+
replace the proxied entry point in place.
|
|
25
|
+
2. **Drain.** `drain.command` runs. This is the hook that tells the service to stop admitting new
|
|
26
|
+
work. If it exits non-zero, the roll fails at `stage: "drain"` and the container is **not**
|
|
27
|
+
recreated.
|
|
28
|
+
3. **In-flight wait.** `inFlight.command` is polled until it exits 0 (the unit's accepted work has
|
|
29
|
+
settled) or `inFlight.timeoutMs` expires. Expiry fails the roll at `stage: "in_flight"` and the
|
|
30
|
+
container is **not** recreated — Rollbridge never recreates anyway because the drain timed out.
|
|
31
|
+
4. **Recreate.** `docker compose ... up -d --no-deps --force-recreate <service>` runs with a
|
|
32
|
+
generated override file that pins the target image and the unit's `stop_grace_period`. The
|
|
33
|
+
override is written next to the Compose file, used once, and deleted; the operator's Compose
|
|
34
|
+
file is never rewritten. `--no-deps` keeps the roll to this one service.
|
|
35
|
+
5. **Activation gate.** The new container must be `running`, pass its Compose healthcheck (when it
|
|
36
|
+
has one), not front the proxy port, and pass `activation.readyCommand` and/or
|
|
37
|
+
`activation.readyUrl` if configured. Polling runs until `activation.timeoutMs`.
|
|
38
|
+
6. **Rollback.** If activation fails and a previous image is retained, the previous image is
|
|
39
|
+
recreated through the same path and gated the same way. Success reports `status: "rolled_back"`
|
|
40
|
+
with `rollback.image`; if the rollback itself cannot activate, the roll reports
|
|
41
|
+
`stage: "rollback_failed"` and exits non-zero instead of pretending the unit is healthy.
|
|
42
|
+
|
|
43
|
+
Every activated or rolled-back generation is appended to the unit's ledger
|
|
44
|
+
(`composeStatePath`), keeping the image it replaced. That retained tag is what
|
|
45
|
+
`compose-rollback` selects.
|
|
46
|
+
|
|
47
|
+
## Commands
|
|
48
|
+
|
|
49
|
+
```bash
|
|
50
|
+
rollbridge compose-roll --config rollbridge.js --unit gateway
|
|
51
|
+
rollbridge compose-roll --config rollbridge.js --unit gateway --image registry.local/gateway:2.4.0
|
|
52
|
+
rollbridge compose-rollback --config rollbridge.js --unit gateway
|
|
53
|
+
rollbridge compose-status --config rollbridge.js --unit gateway
|
|
54
|
+
```
|
|
55
|
+
|
|
56
|
+
- `compose-roll` targets the unit's configured `image` unless `--image` names another reference.
|
|
57
|
+
- `compose-rollback` selects the retained previous image from the ledger unless `--image` names one.
|
|
58
|
+
- Both print the roll result JSON and exit non-zero when `status` is `failed`, so deploy tooling
|
|
59
|
+
can gate on them.
|
|
60
|
+
- `compose-status` prints the live container (id, state, health, image, published ports) and the
|
|
61
|
+
retained generations.
|
|
62
|
+
|
|
63
|
+
Result shape:
|
|
64
|
+
|
|
65
|
+
```json
|
|
66
|
+
{
|
|
67
|
+
"unitId": "gateway",
|
|
68
|
+
"releaseId": "20261104104321888",
|
|
69
|
+
"status": "rolled",
|
|
70
|
+
"stage": "activation",
|
|
71
|
+
"image": "registry.local/gateway:2.4.0",
|
|
72
|
+
"previousImage": "registry.local/gateway:2.3.1",
|
|
73
|
+
"containerId": "ba108432e759",
|
|
74
|
+
"generations": [{"image": "registry.local/gateway:2.4.0", "previousImage": "registry.local/gateway:2.3.1", "status": "activated", "releaseId": "20261104104321888", "containerId": "ba108432e759", "activatedAt": "2026-11-04T10:43:27.258Z", "unitId": "gateway"}],
|
|
75
|
+
"rollback": null,
|
|
76
|
+
"error": null
|
|
77
|
+
}
|
|
78
|
+
```
|
|
79
|
+
|
|
80
|
+
`status` is `rolled`, `rolled_back`, or `failed`; `stage` is `drain`, `in_flight`, `recreate`,
|
|
81
|
+
`activation`, `rollback`, or `rollback_failed`.
|
|
82
|
+
|
|
83
|
+
## Fleet runbook
|
|
84
|
+
|
|
85
|
+
These replace `docker compose up -d --force-recreate <service>` for the three Compose services in
|
|
86
|
+
the fleet. In each case the Compose file stays where it is; Rollbridge only pins the image for the
|
|
87
|
+
roll. Paths, project names, and image names below are illustrative — take the real ones from that
|
|
88
|
+
host's Compose file, and remember a Compose unit can only be rolled to an image the daemon on that
|
|
89
|
+
host can already resolve.
|
|
90
|
+
|
|
91
|
+
### Socketduct hub
|
|
92
|
+
|
|
93
|
+
```js
|
|
94
|
+
// /srv/socketduct-hub/rollbridge.js
|
|
95
|
+
export default {
|
|
96
|
+
application: "socketduct-hub",
|
|
97
|
+
proxy: {host: "127.0.0.1", port: 8182},
|
|
98
|
+
processes: [/* the proxied hub process, unchanged */],
|
|
99
|
+
compose: [{
|
|
100
|
+
id: "hub",
|
|
101
|
+
service: "hub",
|
|
102
|
+
projectDirectory: "/srv/socketduct-hub",
|
|
103
|
+
composeFile: "compose.yml",
|
|
104
|
+
projectName: "socketduct-hub",
|
|
105
|
+
image: "registry.diestoeckels.de/socketduct/hub:1.9.2",
|
|
106
|
+
stopGracePeriodMs: 30000,
|
|
107
|
+
drain: {command: "{{composeArgs}} exec -T hub appctl drain", timeoutMs: 60000},
|
|
108
|
+
inFlight: {command: "{{composeArgs}} exec -T hub appctl in-flight", intervalMs: 2000, timeoutMs: 300000},
|
|
109
|
+
activation: {startDelayMs: 2000, timeoutMs: 90000, readyUrl: "http://127.0.0.1:7420/ready"}
|
|
110
|
+
}]
|
|
111
|
+
}
|
|
112
|
+
```
|
|
113
|
+
|
|
114
|
+
```bash
|
|
115
|
+
rollbridge doctor --config /srv/socketduct-hub/rollbridge.js # preflights the compose CLI, the file, and the service
|
|
116
|
+
rollbridge compose-roll --config /srv/socketduct-hub/rollbridge.js --unit hub --image registry.diestoeckels.de/socketduct/hub:1.9.3
|
|
117
|
+
rollbridge compose-status --config /srv/socketduct-hub/rollbridge.js --unit hub
|
|
118
|
+
```
|
|
119
|
+
|
|
120
|
+
### One Socketduct gateway leg
|
|
121
|
+
|
|
122
|
+
A gateway leg is the per-host `socketduct-<host>-<class>` gateway. Rolling one leg must not touch
|
|
123
|
+
the other legs, so the unit names only its own project directory and service:
|
|
124
|
+
|
|
125
|
+
```js
|
|
126
|
+
// /srv/socketduct/socketduct-builder-a-standard/rollbridge.js
|
|
127
|
+
export default {
|
|
128
|
+
application: "socketduct-builder-a-standard",
|
|
129
|
+
proxy: {host: "127.0.0.1", port: 8182},
|
|
130
|
+
processes: [/* the proxied gateway process, unchanged */],
|
|
131
|
+
compose: [{
|
|
132
|
+
id: "gateway",
|
|
133
|
+
service: "gateway",
|
|
134
|
+
projectDirectory: "/srv/socketduct/socketduct-builder-a-standard",
|
|
135
|
+
composeFile: "compose.yml",
|
|
136
|
+
projectName: "socketduct-builder-a-standard",
|
|
137
|
+
image: "registry.diestoeckels.de/socketduct/gateway:2.4.1",
|
|
138
|
+
stopGracePeriodMs: 45000,
|
|
139
|
+
drain: {command: "{{composeArgs}} exec -T gateway appctl drain", timeoutMs: 120000},
|
|
140
|
+
inFlight: {command: "{{composeArgs}} exec -T gateway appctl in-flight", intervalMs: 2000, timeoutMs: 900000},
|
|
141
|
+
activation: {startDelayMs: 2000, timeoutMs: 120000, readyUrl: "http://127.0.0.1:7421/ready"}
|
|
142
|
+
}]
|
|
143
|
+
}
|
|
144
|
+
```
|
|
145
|
+
|
|
146
|
+
```bash
|
|
147
|
+
rollbridge compose-roll --config /srv/socketduct/socketduct-builder-a-standard/rollbridge.js --unit gateway
|
|
148
|
+
rollbridge compose-rollback --config /srv/socketduct/socketduct-builder-a-standard/rollbridge.js --unit gateway
|
|
149
|
+
```
|
|
150
|
+
|
|
151
|
+
The long `inFlight.timeoutMs` is deliberate: a gateway leg that is mid-handoff should finish it.
|
|
152
|
+
If it never settles, the roll fails and the leg keeps serving the old image — which is the
|
|
153
|
+
behaviour that keeps the Docker server `online: true` instead of leaving it
|
|
154
|
+
`storageHealthReason: transport_failure`.
|
|
155
|
+
|
|
156
|
+
Gateway images are pinned by digest with `pull_policy: never`, so they cannot be updated by
|
|
157
|
+
pulling: `compose[].image` names the new `@sha256:` digest (or a tag verified present on that
|
|
158
|
+
machine), and the image has to be loaded on the host before the roll. A roll never touches the
|
|
159
|
+
other services in the project — `--no-deps` means an unsatisfiable sibling service can no longer
|
|
160
|
+
stop this gateway from being recreated.
|
|
161
|
+
|
|
162
|
+
### One builder DinD
|
|
163
|
+
|
|
164
|
+
The builder DinD keeps its established identities exactly: Compose project
|
|
165
|
+
`peakflow_builder`, service `docker-server`, container `peakflow_builder-docker-server-1`.
|
|
166
|
+
|
|
167
|
+
```js
|
|
168
|
+
// /srv/peakflow-builder/rollbridge.js
|
|
169
|
+
export default {
|
|
170
|
+
application: "peakflow-builder",
|
|
171
|
+
proxy: {host: "127.0.0.1", port: 8182},
|
|
172
|
+
processes: [/* the proxied builder API process, unchanged */],
|
|
173
|
+
compose: [{
|
|
174
|
+
id: "builder-dind",
|
|
175
|
+
service: "docker-server",
|
|
176
|
+
projectDirectory: "/srv/peakflow-builder",
|
|
177
|
+
composeFile: "compose.yml",
|
|
178
|
+
projectName: "peakflow_builder",
|
|
179
|
+
image: "ghcr.io/kaspernj/peakflow-builder:amd64",
|
|
180
|
+
stopGracePeriodMs: 60000,
|
|
181
|
+
inFlight: {command: "test -z \"$(docker exec peakflow_builder-docker-server-1 sh -c 'ls /var/lib/docker/buildkit 2>/dev/null')\"", intervalMs: 5000, timeoutMs: 1800000},
|
|
182
|
+
activation: {startDelayMs: 5000, timeoutMs: 180000, readyCommand: "docker inspect -f '{{.State.Running}}' peakflow_builder-docker-server-1 | grep -q true"}
|
|
183
|
+
}]
|
|
184
|
+
}
|
|
185
|
+
```
|
|
186
|
+
|
|
187
|
+
```bash
|
|
188
|
+
rollbridge compose-roll --config /srv/peakflow-builder/rollbridge.js --unit builder-dind
|
|
189
|
+
```
|
|
190
|
+
|
|
191
|
+
A DinD that is mid-build is the case the drain exists for: `inFlight.timeoutMs` is the budget for
|
|
192
|
+
the builds already admitted, and expiry keeps the old DinD serving rather than killing them. This
|
|
193
|
+
example sets no `drain.command` because the builder DinD has no quiescence hook to call today —
|
|
194
|
+
until it exposes one, the in-flight gate is the only protection an admitted build has, and the
|
|
195
|
+
roll is strictly better than `up -d --force-recreate` but not yet a true drain.
|
|
196
|
+
|
|
197
|
+
## What is proven, and what is not
|
|
198
|
+
|
|
199
|
+
- **Proven by tests** (`test/compose-unit.test.js`, `test/compose-config.test.js`): drain-first
|
|
200
|
+
ordering, drain/in-flight failure never recreating, a `docker compose up` the daemon refuses
|
|
201
|
+
(for example a `pull_policy: never` gateway digest that is not loaded on that host) failing the
|
|
202
|
+
roll at `stage: "recreate"` with the previous container still running, its image still selected
|
|
203
|
+
and the ledger untouched, activation failure rolling back to the retained tag, a failed rollback
|
|
204
|
+
reported as `rollback_failed`, `compose-rollback` selecting the retained tag, refusal to front
|
|
205
|
+
the proxy port, refusal during an unresolved generation transition, and the config validation
|
|
206
|
+
rules.
|
|
207
|
+
- **Proven against a real Docker Compose daemon** (`test/compose-docker.test.js`, opt-in with
|
|
208
|
+
`ROLLBRIDGE_COMPOSE_DOCKER_TEST=1`): a real Compose service is rolled drain-first (the container
|
|
209
|
+
records `drain` before `settled`, and the new image starts only after that), the new image is
|
|
210
|
+
live afterwards, the previous tag is retained in the ledger, and rolling to an image whose
|
|
211
|
+
healthcheck never passes leaves the previous container serving and answering its readiness
|
|
212
|
+
endpoint. It also proves the proxy-port refusal against a real published port.
|
|
213
|
+
- **Not implemented here**: the drain and in-flight hooks only work if the service exposes a
|
|
214
|
+
quiescence command and an in-flight probe. The Socketduct hub/gateway images and the builder
|
|
215
|
+
DinD need those hooks (or an equivalent probe) before their units can be declared; the example
|
|
216
|
+
`appctl drain` / `appctl in-flight` commands above name the contract, not something those images
|
|
217
|
+
ship today.
|
|
218
|
+
- **Not changed here**: TensorBuzz's Socketduct provisioning path still issues raw
|
|
219
|
+
`docker compose up -d --force-recreate` when it provisions gateway legs. Switching that backend
|
|
220
|
+
to call `rollbridge compose-roll` is a TensorBuzz change, tracked separately from this one.
|
package/docs/config.md
CHANGED
|
@@ -44,6 +44,8 @@ restart.
|
|
|
44
44
|
| Field | Type | Default | Description |
|
|
45
45
|
| --- | --- | --- | --- |
|
|
46
46
|
| `application` | string | basename of the config file's directory | Names the app; used in the default control-socket path and the `ROLLBRIDGE_APPLICATION` env var. |
|
|
47
|
+
| `compose` | array | unset (no Compose units) | Docker Compose services Rollbridge rolls drain-first (see [`compose[]`](#compose)). |
|
|
48
|
+
| `composeStatePath` | string | `rollbridge-<application>.compose.state.json` next to the config file | File that retains each Compose unit's image generations; relative paths resolve from the config file directory. |
|
|
47
49
|
| `control` | object | — | Control-socket settings (see below). |
|
|
48
50
|
| `legacyTakeover` | object | unset | Optional matchers for `rollbridge predeploy-cleanup` to stop pre-Rollbridge supervisors during first handover (see below). |
|
|
49
51
|
| `ownerRecovery` | object | unset | Durable guardian recovery and atomic incompatible owner replacement; requires `statePath` (see below). |
|
|
@@ -445,6 +447,56 @@ release). Set `health: false` to disable it.
|
|
|
445
447
|
| `health.intervalMs` | number | `250` | Delay between probes. |
|
|
446
448
|
| `health.startDelayMs` | non-negative number | `0` | Wait this long after the process starts before the first probe (runs before the `timeoutMs` window). |
|
|
447
449
|
|
|
450
|
+
## `compose[]`
|
|
451
|
+
|
|
452
|
+
Optional Docker Compose services this app updates. Each entry is a unit Rollbridge
|
|
453
|
+
rolls drain-first — quiesce, wait for in-flight work, recreate with the target
|
|
454
|
+
image, gate on activation, roll back on failure — instead of a raw
|
|
455
|
+
`docker compose up -d --force-recreate`. See
|
|
456
|
+
[`docs/compose-services.md`](compose-services.md) for the stage-by-stage behavior
|
|
457
|
+
and the operator commands.
|
|
458
|
+
|
|
459
|
+
| Field | Type | Default | Description |
|
|
460
|
+
| --- | --- | --- | --- |
|
|
461
|
+
| `compose[].id` | string | **required** | Names the unit for `compose-roll`/`compose-rollback`/`compose-status`. Unique across units and not equal to any process id. |
|
|
462
|
+
| `compose[].service` | string | **required** | Service name inside the Compose file. |
|
|
463
|
+
| `compose[].projectDirectory` | string | **required** | Directory holding the Compose file; every Compose command runs here. |
|
|
464
|
+
| `compose[].image` | string | **required** | Target image reference for `compose-roll` (a tag or a `@sha256:` digest; never a bare image name). |
|
|
465
|
+
| `compose[].composeFile` | string | `"compose.yml"` | Compose file relative to `projectDirectory`. |
|
|
466
|
+
| `compose[].projectName` | string | unset (Compose default) | Passed as `--project-name`, so a project whose services come from several files stays addressable. |
|
|
467
|
+
| `compose[].envFile` | string | unset | Passed as `--env-file`, relative to `projectDirectory`. |
|
|
468
|
+
| `compose[].stopGracePeriodMs` | non-negative number | `10000` | Written into the roll's override as the service's `stop_grace_period`: how long Compose waits for the old container to exit. |
|
|
469
|
+
| `compose[].recreateTimeoutMs` | non-negative number | `120000` | Timeout for the `compose up -d --no-deps --force-recreate` the roll runs. |
|
|
470
|
+
| `compose[].drain.command` | string | unset | Hook that stops the unit admitting new work. Runs through a shell; unset means the unit has no quiescence hook and the roll proceeds to the in-flight gate. |
|
|
471
|
+
| `compose[].drain.timeoutMs` | positive number | `60000` | Required when `drain.command` is set. Expiry fails the roll at `stage: "drain"` without recreating the container. |
|
|
472
|
+
| `compose[].inFlight.command` | string | unset | Gate that exits 0 only when the work the unit already accepted has settled. Polled until it exits 0 or `inFlight.timeoutMs` expires. |
|
|
473
|
+
| `compose[].inFlight.intervalMs` | positive number | `1000` | Delay between gate attempts. |
|
|
474
|
+
| `compose[].inFlight.timeoutMs` | positive number | `0` (no gate) | Required when `inFlight.command` is set. Expiry fails the roll at `stage: "in_flight"` without recreating the container. |
|
|
475
|
+
| `compose[].activation.healthcheck` | boolean | `true` | Require the recreated container's Compose healthcheck to report `healthy` (containers without a `HEALTHCHECK` pass). |
|
|
476
|
+
| `compose[].activation.readyCommand` | string | unset | Additional readiness probe run through a shell; must exit 0. |
|
|
477
|
+
| `compose[].activation.readyUrl` | string | unset | Additional readiness probe fetched over HTTP; any 2xx passes. |
|
|
478
|
+
| `compose[].activation.startDelayMs` | non-negative number | `0` | Wait before the first probe (runs before the `timeoutMs` window). |
|
|
479
|
+
| `compose[].activation.intervalMs` | positive number | `1000` | Delay between probes. |
|
|
480
|
+
| `compose[].activation.timeoutMs` | positive number | `60000` | How long the activation gate may keep probing before the roll rolls back. |
|
|
481
|
+
| `compose[].rollback.enabled` | boolean | `true` | Roll back to the retained previous image when activation fails. When `false`, a failed activation is reported as a failed roll. |
|
|
482
|
+
|
|
483
|
+
`composeStatePath` (top-level) is where each unit's generations are retained, so
|
|
484
|
+
`compose-rollback` has a previous tag to select. It defaults to
|
|
485
|
+
`rollbridge-<application>.compose.state.json` next to the config file.
|
|
486
|
+
|
|
487
|
+
Drain, in-flight, and readiness-command hooks support `{{...}}` placeholders:
|
|
488
|
+
|
|
489
|
+
| Placeholder | Value |
|
|
490
|
+
| --- | --- |
|
|
491
|
+
| `{{application}}` | `application` |
|
|
492
|
+
| `{{unitId}}` | This unit's `id`. |
|
|
493
|
+
| `{{service}}` | This unit's `service`. |
|
|
494
|
+
| `{{composeArgs}}` | The Compose global arguments for this unit (`docker compose`, its `--project-name`, `-f <file>`, `--env-file`), so a hook can run `{{composeArgs}} exec -T <service> ...`. |
|
|
495
|
+
| `{{composeFile}}` | This unit's `composeFile`. |
|
|
496
|
+
| `{{projectDirectory}}` | This unit's `projectDirectory`. |
|
|
497
|
+
| `{{releaseId}}` | This roll's release id. |
|
|
498
|
+
| `{{env.<NAME>}}` | A variable from the Rollbridge process environment. |
|
|
499
|
+
|
|
448
500
|
## Template variables
|
|
449
501
|
|
|
450
502
|
`command`, `cwd`, and `env` values support `{{...}}` placeholders, rendered when
|
|
@@ -497,3 +549,6 @@ Rollbridge sets these in every managed process's environment (the process's own
|
|
|
497
549
|
- `lifecycle.activateCommand`/`quietCommand`/`drainCommand`/`stopCommand` must be non-empty strings when set, and `lifecycle.drainTimeoutMs` a non-negative number; `lifecycle.drainCommand` requires a positive `lifecycle.drainTimeoutMs`. `activateCommand` is allowed on at most one handoff service, requires that service's `quietCommand`, and requires `statePath` plus `ownerRecovery`. A `lifecycle.stopCommand` may not be combined with a custom `stopSignal` (the `stopCommand` runs instead of the signal, so the signal would be ignored).
|
|
498
550
|
- `nonBlockingDrain` must be a boolean, and is allowed only on a `companion` process.
|
|
499
551
|
- `statePath` must be a string when set.
|
|
552
|
+
- `compose` must be a non-empty array of unit mappings. Each unit needs `id`, `service`, `projectDirectory`, and a valid `image` reference (a tag, or a `@sha256:` digest with 64 hex characters; no whitespace, no leading `-`).
|
|
553
|
+
- Compose unit `id`s must be unique and must not collide with a process `id`.
|
|
554
|
+
- `compose[].drain.command` requires a positive `compose[].drain.timeoutMs`; `compose[].inFlight.command` requires a positive `compose[].inFlight.timeoutMs`.
|
package/docs/troubleshooting.md
CHANGED
|
@@ -148,3 +148,27 @@ separate destructive `--takeover-owner` path. If
|
|
|
148
148
|
`retirementError` is set,
|
|
149
149
|
inspect the quiet-hook events. Rollbridge deliberately leaves that generation
|
|
150
150
|
alive rather than signaling arbitrary PIDs or continuing its stop sequence.
|
|
151
|
+
|
|
152
|
+
## A command is refused with "Another owner mutation is already in progress"
|
|
153
|
+
|
|
154
|
+
**Symptom.** `deploy`, `stop`, `restart`, `rollback` or
|
|
155
|
+
`recover-generation-transition` fails with
|
|
156
|
+
`Another owner mutation is already in progress: owner mutation "<operation>";
|
|
157
|
+
mutation <id>; held by process <pid>; since <ISO-8601>; <n>s elapsed; waiting
|
|
158
|
+
for <blocker>; refused <operation>`.
|
|
159
|
+
|
|
160
|
+
**Cause.** The committed daemon holds the guardian's owner-mutation fence for
|
|
161
|
+
one control mutation at a time, and a second mutation arrived while it was still
|
|
162
|
+
held.
|
|
163
|
+
|
|
164
|
+
**Fix.** Read the named holder before touching anything: the message states
|
|
165
|
+
which mutation holds the fence, its lease id, the PID of the process that
|
|
166
|
+
started it, when it started, how long it has run, and the blocker that holder
|
|
167
|
+
last reported (`waitingFor`, which a running mutation can refine through
|
|
168
|
+
`reportOwnerMutationWait`). That is the command to inspect — not the one that
|
|
169
|
+
was refused. A mutation that outruns its lease (15 minutes by default,
|
|
170
|
+
overridable per command with `mutationTimeoutMs`) is logged as
|
|
171
|
+
`OWNER_MUTATION_LEASE_EXPIRED`, reported to the holder, and released, so the
|
|
172
|
+
next mutation proceeds and the stuck one fails loudly when it next ends its
|
|
173
|
+
lease. A retained jobs generation is not an owner mutation and may still drain
|
|
174
|
+
for hours; only the fence is bounded.
|