rollbridge 0.1.70 → 0.1.71
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +10 -63
- package/TODO.md +0 -13
- package/docs/cli.md +3 -128
- package/docs/config.md +6 -177
- package/docs/troubleshooting.md +6 -97
- package/docs/velocious.md +1 -1
- package/docs/workers.md +0 -11
- package/examples/tensorbuzz.com.js +2 -3
- package/package.json +1 -1
- package/src/cli.js +2 -68
- package/src/config.js +8 -347
- package/src/daemon.js +65 -387
- package/src/doctor.js +0 -74
- package/src/guardian-client.js +8 -67
- package/src/health.js +8 -99
- package/src/managed-process.js +16 -197
- package/src/process-guardian.js +22 -212
- package/src/process-memory.js +2 -62
- package/src/release-group.js +37 -435
- package/tensorbuzz.yml +1 -20
- package/test/completion.test.js +2 -5
- package/test/config-validation.test.js +5 -135
- package/test/guardian-client.test.js +5 -119
- package/test/health.test.js +1 -74
- package/test/managed-process.test.js +1 -260
- package/test/owner-recovery.test.js +21 -125
- package/test/owner-replacement.test.js +14 -175
- package/test/process-memory.test.js +1 -47
- package/test/release-group.test.js +2 -408
- package/test/rollbridge.test.js +72 -737
- package/test/shutdown-completion.test.js +0 -52
- package/test/support/process.js +1 -3
- package/changelog.d/20261025-guardian-re-register-dead-generation.md +0 -13
- package/changelog.d/20261026-stale-committed-generation-transition-recovery.md +0 -1
- package/changelog.d/20261104-compose-service-units.md +0 -21
- package/changelog.d/20261105-owner-mutation-lease-diagnostics.md +0 -20
- package/changelog.d/20261106-release-dependency-boot-budgets.md +0 -32
- package/changelog.d/20261109-path-filtered-builds.md +0 -11
- package/changelog.d/20261110-default-boot-budget-above-measured-startup.md +0 -26
- package/changelog.d/20261111-contention-sensitive-daemon-specs.md +0 -27
- package/changelog.d/20261113-bounded-nonblocking-retirement.md +0 -5
- package/changelog.d/20261115-bound-draining-release-groups.md +0 -62
- package/changelog.d/20261115-restore-deploy-refuses-held-generation.md +0 -27
- package/changelog.d/20261115-wedged-generation-escalation.md +0 -44
- package/changelog.d/20261116-bounded-explicit-stop-retirement.md +0 -30
- package/changelog.d/20261116-config-json-canonical-shape.md +0 -11
- package/changelog.d/20261116-drain-specs-gate-the-retirement-record.md +0 -29
- package/changelog.d/20261116-guardian-version-handoff.md +0 -37
- package/changelog.d/20261116-reclaim-dead-owner-mutation.md +0 -7
- package/changelog.d/20261117-activate-before-incumbent-retirement.md +0 -1
- package/changelog.d/20261117-activation-path-drain-deadline.md +0 -35
- package/changelog.d/20261117-stop-companion-after-retired-dependency.md +0 -7
- package/docs/compose-services.md +0 -222
- package/src/compose-unit.js +0 -725
- package/src/drain-policy.js +0 -252
- package/src/process-order.js +0 -26
- package/test/compose-config.test.js +0 -160
- package/test/compose-docker.test.js +0 -297
- package/test/compose-unit.test.js +0 -415
- package/test/drain-policy.test.js +0 -328
- package/test/fixtures/deferred-boot-app.js +0 -106
- package/test/fixtures/gated-app.js +0 -50
- package/test/tensorbuzz-config.test.js +0 -228
package/docs/compose-services.md
DELETED
|
@@ -1,222 +0,0 @@
|
|
|
1
|
-
# Compose service units
|
|
2
|
-
|
|
3
|
-
A Compose unit is a Docker Compose service that Rollbridge updates with the same handshake it
|
|
4
|
-
already uses for its Node processes: quiesce the unit, wait for the work it already accepted to
|
|
5
|
-
finish, recreate the container with the new image, gate on activation, and roll back to the
|
|
6
|
-
previous image when activation fails.
|
|
7
|
-
|
|
8
|
-
Before this, every Compose service in the same delivery as a Rollbridge-managed app was updated
|
|
9
|
-
with a raw `docker compose up -d --force-recreate <service>`. That command has no drain (in-flight
|
|
10
|
-
work dies with the container), no activation gate (a broken image becomes the live service), and
|
|
11
|
-
no rollback (the previous tag is whatever the operator last typed). On the Socketduct gateway legs
|
|
12
|
-
and the builder DinDs that cost is visible: in-flight builds die, and a gateway leg that is
|
|
13
|
-
half-rolled leaves its Docker server `online: false` with `storageHealthReason: transport_failure`.
|
|
14
|
-
|
|
15
|
-
## The roll, stage by stage
|
|
16
|
-
|
|
17
|
-
`rollbridge compose-roll --unit <id>` runs these stages in order. Each stage that fails stops the
|
|
18
|
-
roll and says which stage failed.
|
|
19
|
-
|
|
20
|
-
1. **Preflight.** The target image reference is checked, and the roll refuses if a Rollbridge
|
|
21
|
-
release generation transition is unresolved (`Cannot roll Compose unit ... while release
|
|
22
|
-
generation transition <id> is unresolved at <phase>; resume the exact deploy first`). It also
|
|
23
|
-
refuses if the unit's container publishes the Rollbridge proxy port — a Compose unit must never
|
|
24
|
-
replace the proxied entry point in place.
|
|
25
|
-
2. **Drain.** `drain.command` runs. This is the hook that tells the service to stop admitting new
|
|
26
|
-
work. If it exits non-zero, the roll fails at `stage: "drain"` and the container is **not**
|
|
27
|
-
recreated.
|
|
28
|
-
3. **In-flight wait.** `inFlight.command` is polled until it exits 0 (the unit's accepted work has
|
|
29
|
-
settled) or `inFlight.timeoutMs` expires. Expiry fails the roll at `stage: "in_flight"` and the
|
|
30
|
-
container is **not** recreated — Rollbridge never recreates anyway because the drain timed out.
|
|
31
|
-
4. **Recreate.** `docker compose ... up -d --no-deps --force-recreate <service>` runs with a
|
|
32
|
-
generated override file that pins the target image and the unit's `stop_grace_period`. The
|
|
33
|
-
override is written next to the Compose file, used once, and deleted; the operator's Compose
|
|
34
|
-
file is never rewritten. `--no-deps` keeps the roll to this one service.
|
|
35
|
-
5. **Activation gate.** The new container must be `running`, pass its Compose healthcheck (when it
|
|
36
|
-
has one), not front the proxy port, and pass `activation.readyCommand` and/or
|
|
37
|
-
`activation.readyUrl` if configured. Polling runs until `activation.timeoutMs`.
|
|
38
|
-
6. **Rollback.** If activation fails and a previous image is retained, the previous image is
|
|
39
|
-
recreated through the same path and gated the same way. Success reports `status: "rolled_back"`
|
|
40
|
-
with `rollback.image`; if the rollback itself cannot activate, the roll reports
|
|
41
|
-
`stage: "rollback_failed"` and exits non-zero instead of pretending the unit is healthy.
|
|
42
|
-
|
|
43
|
-
Every activated or rolled-back generation is appended to the unit's ledger
|
|
44
|
-
(`composeStatePath`), keeping the image it replaced. That retained tag is what
|
|
45
|
-
`compose-rollback` selects.
|
|
46
|
-
|
|
47
|
-
## Commands
|
|
48
|
-
|
|
49
|
-
```bash
|
|
50
|
-
rollbridge compose-roll --config rollbridge.js --unit gateway
|
|
51
|
-
rollbridge compose-roll --config rollbridge.js --unit gateway --image registry.local/gateway:2.4.0
|
|
52
|
-
rollbridge compose-rollback --config rollbridge.js --unit gateway
|
|
53
|
-
rollbridge compose-status --config rollbridge.js --unit gateway
|
|
54
|
-
```
|
|
55
|
-
|
|
56
|
-
- `compose-roll` targets the unit's configured `image` unless `--image` names another reference.
|
|
57
|
-
- `compose-rollback` selects the retained previous image from the ledger unless `--image` names one.
|
|
58
|
-
- Both print the roll result JSON and exit non-zero when `status` is `failed`, so deploy tooling
|
|
59
|
-
can gate on them.
|
|
60
|
-
- `compose-status` prints the live container (id, state, health, image, published ports) and the
|
|
61
|
-
retained generations.
|
|
62
|
-
|
|
63
|
-
Result shape:
|
|
64
|
-
|
|
65
|
-
```json
|
|
66
|
-
{
|
|
67
|
-
"unitId": "gateway",
|
|
68
|
-
"releaseId": "20261104104321888",
|
|
69
|
-
"status": "rolled",
|
|
70
|
-
"stage": "activation",
|
|
71
|
-
"image": "registry.local/gateway:2.4.0",
|
|
72
|
-
"previousImage": "registry.local/gateway:2.3.1",
|
|
73
|
-
"containerId": "ba108432e759",
|
|
74
|
-
"generations": [{"image": "registry.local/gateway:2.4.0", "previousImage": "registry.local/gateway:2.3.1", "status": "activated", "releaseId": "20261104104321888", "containerId": "ba108432e759", "activatedAt": "2026-11-04T10:43:27.258Z", "unitId": "gateway"}],
|
|
75
|
-
"rollback": null,
|
|
76
|
-
"error": null
|
|
77
|
-
}
|
|
78
|
-
```
|
|
79
|
-
|
|
80
|
-
`status` is `rolled`, `rolled_back`, or `failed`; `stage` is `drain`, `in_flight`, `recreate`,
|
|
81
|
-
`activation`, `rollback`, or `rollback_failed`.
|
|
82
|
-
|
|
83
|
-
## Fleet runbook
|
|
84
|
-
|
|
85
|
-
These replace `docker compose up -d --force-recreate <service>` for the three Compose services in
|
|
86
|
-
the fleet. In each case the Compose file stays where it is; Rollbridge only pins the image for the
|
|
87
|
-
roll. Paths, project names, and image names below are illustrative — take the real ones from that
|
|
88
|
-
host's Compose file, and remember a Compose unit can only be rolled to an image the daemon on that
|
|
89
|
-
host can already resolve.
|
|
90
|
-
|
|
91
|
-
### Socketduct hub
|
|
92
|
-
|
|
93
|
-
```js
|
|
94
|
-
// /srv/socketduct-hub/rollbridge.js
|
|
95
|
-
export default {
|
|
96
|
-
application: "socketduct-hub",
|
|
97
|
-
proxy: {host: "127.0.0.1", port: 8182},
|
|
98
|
-
processes: [/* the proxied hub process, unchanged */],
|
|
99
|
-
compose: [{
|
|
100
|
-
id: "hub",
|
|
101
|
-
service: "hub",
|
|
102
|
-
projectDirectory: "/srv/socketduct-hub",
|
|
103
|
-
composeFile: "compose.yml",
|
|
104
|
-
projectName: "socketduct-hub",
|
|
105
|
-
image: "registry.diestoeckels.de/socketduct/hub:1.9.2",
|
|
106
|
-
stopGracePeriodMs: 30000,
|
|
107
|
-
drain: {command: "{{composeArgs}} exec -T hub appctl drain", timeoutMs: 60000},
|
|
108
|
-
inFlight: {command: "{{composeArgs}} exec -T hub appctl in-flight", intervalMs: 2000, timeoutMs: 300000},
|
|
109
|
-
activation: {startDelayMs: 2000, timeoutMs: 90000, readyUrl: "http://127.0.0.1:7420/ready"}
|
|
110
|
-
}]
|
|
111
|
-
}
|
|
112
|
-
```
|
|
113
|
-
|
|
114
|
-
```bash
|
|
115
|
-
rollbridge doctor --config /srv/socketduct-hub/rollbridge.js # preflights the compose CLI, the file, and the service
|
|
116
|
-
rollbridge compose-roll --config /srv/socketduct-hub/rollbridge.js --unit hub --image registry.diestoeckels.de/socketduct/hub:1.9.3
|
|
117
|
-
rollbridge compose-status --config /srv/socketduct-hub/rollbridge.js --unit hub
|
|
118
|
-
```
|
|
119
|
-
|
|
120
|
-
### One Socketduct gateway leg
|
|
121
|
-
|
|
122
|
-
A gateway leg is the per-host `socketduct-<host>-<class>` gateway. Rolling one leg must not touch
|
|
123
|
-
the other legs, so the unit names only its own project directory and service:
|
|
124
|
-
|
|
125
|
-
```js
|
|
126
|
-
// /srv/socketduct/socketduct-builder-a-standard/rollbridge.js
|
|
127
|
-
export default {
|
|
128
|
-
application: "socketduct-builder-a-standard",
|
|
129
|
-
proxy: {host: "127.0.0.1", port: 8182},
|
|
130
|
-
processes: [/* the proxied gateway process, unchanged */],
|
|
131
|
-
compose: [{
|
|
132
|
-
id: "gateway",
|
|
133
|
-
service: "gateway",
|
|
134
|
-
projectDirectory: "/srv/socketduct/socketduct-builder-a-standard",
|
|
135
|
-
composeFile: "compose.yml",
|
|
136
|
-
projectName: "socketduct-builder-a-standard",
|
|
137
|
-
image: "registry.diestoeckels.de/socketduct/gateway:2.4.1",
|
|
138
|
-
stopGracePeriodMs: 45000,
|
|
139
|
-
drain: {command: "{{composeArgs}} exec -T gateway appctl drain", timeoutMs: 120000},
|
|
140
|
-
inFlight: {command: "{{composeArgs}} exec -T gateway appctl in-flight", intervalMs: 2000, timeoutMs: 900000},
|
|
141
|
-
activation: {startDelayMs: 2000, timeoutMs: 120000, readyUrl: "http://127.0.0.1:7421/ready"}
|
|
142
|
-
}]
|
|
143
|
-
}
|
|
144
|
-
```
|
|
145
|
-
|
|
146
|
-
```bash
|
|
147
|
-
rollbridge compose-roll --config /srv/socketduct/socketduct-builder-a-standard/rollbridge.js --unit gateway
|
|
148
|
-
rollbridge compose-rollback --config /srv/socketduct/socketduct-builder-a-standard/rollbridge.js --unit gateway
|
|
149
|
-
```
|
|
150
|
-
|
|
151
|
-
The long `inFlight.timeoutMs` is deliberate: a gateway leg that is mid-handoff should finish it.
|
|
152
|
-
If it never settles, the roll fails and the leg keeps serving the old image — which is the
|
|
153
|
-
behaviour that keeps the Docker server `online: true` instead of leaving it
|
|
154
|
-
`storageHealthReason: transport_failure`.
|
|
155
|
-
|
|
156
|
-
Gateway images are pinned by digest with `pull_policy: never`, so they cannot be updated by
|
|
157
|
-
pulling: `compose[].image` names the new `@sha256:` digest (or a tag verified present on that
|
|
158
|
-
machine), and the image has to be loaded on the host before the roll. A roll never touches the
|
|
159
|
-
other services in the project — `--no-deps` means an unsatisfiable sibling service can no longer
|
|
160
|
-
stop this gateway from being recreated.
|
|
161
|
-
|
|
162
|
-
### One builder DinD
|
|
163
|
-
|
|
164
|
-
The builder DinD keeps its established identities exactly: Compose project
|
|
165
|
-
`peakflow_builder`, service `docker-server`, container `peakflow_builder-docker-server-1`.
|
|
166
|
-
|
|
167
|
-
```js
|
|
168
|
-
// /srv/peakflow-builder/rollbridge.js
|
|
169
|
-
export default {
|
|
170
|
-
application: "peakflow-builder",
|
|
171
|
-
proxy: {host: "127.0.0.1", port: 8182},
|
|
172
|
-
processes: [/* the proxied builder API process, unchanged */],
|
|
173
|
-
compose: [{
|
|
174
|
-
id: "builder-dind",
|
|
175
|
-
service: "docker-server",
|
|
176
|
-
projectDirectory: "/srv/peakflow-builder",
|
|
177
|
-
composeFile: "compose.yml",
|
|
178
|
-
projectName: "peakflow_builder",
|
|
179
|
-
image: "ghcr.io/kaspernj/peakflow-builder:amd64",
|
|
180
|
-
stopGracePeriodMs: 60000,
|
|
181
|
-
inFlight: {command: "node /srv/peakflow-builder/scripts/quiescence-probe.js", intervalMs: 5000, timeoutMs: 1800000},
|
|
182
|
-
activation: {startDelayMs: 5000, timeoutMs: 180000, readyCommand: "docker inspect -f '{{.State.Running}}' peakflow_builder-docker-server-1 | grep -q true"}
|
|
183
|
-
}]
|
|
184
|
-
}
|
|
185
|
-
```
|
|
186
|
-
|
|
187
|
-
```bash
|
|
188
|
-
rollbridge compose-roll --config /srv/peakflow-builder/rollbridge.js --unit builder-dind
|
|
189
|
-
```
|
|
190
|
-
|
|
191
|
-
A DinD that is mid-build is the case the drain exists for: `inFlight.timeoutMs` is the budget for
|
|
192
|
-
the builds already admitted, and expiry keeps the old DinD serving rather than killing them. This
|
|
193
|
-
example sets no `drain.command` because the builder DinD has no quiescence hook to call today —
|
|
194
|
-
until it exposes one, the in-flight gate is the only protection an admitted build has, and the
|
|
195
|
-
roll is strictly better than `up -d --force-recreate` but not yet a true drain.
|
|
196
|
-
|
|
197
|
-
## What is proven, and what is not
|
|
198
|
-
|
|
199
|
-
- **Proven by tests** (`test/compose-unit.test.js`, `test/compose-config.test.js`): drain-first
|
|
200
|
-
ordering, drain/in-flight failure never recreating, a `docker compose up` the daemon refuses
|
|
201
|
-
(for example a `pull_policy: never` gateway digest that is not loaded on that host) failing the
|
|
202
|
-
roll at `stage: "recreate"` with the previous container still running, its image still selected
|
|
203
|
-
and the ledger untouched, activation failure rolling back to the retained tag, a failed rollback
|
|
204
|
-
reported as `rollback_failed`, `compose-rollback` selecting the retained tag, refusal to front
|
|
205
|
-
the proxy port, refusal during an unresolved generation transition, and the config validation
|
|
206
|
-
rules.
|
|
207
|
-
- **Proven against a real Docker Compose daemon** (`test/compose-docker.test.js`, opt-in with
|
|
208
|
-
`ROLLBRIDGE_COMPOSE_DOCKER_TEST=1`): a real Compose service is rolled drain-first (the container
|
|
209
|
-
records `drain` before `settled`, and the new image starts only after that), the new image is
|
|
210
|
-
live afterwards, the previous tag is retained in the ledger, and rolling to an image whose
|
|
211
|
-
healthcheck never passes leaves the previous container serving and answering its readiness
|
|
212
|
-
endpoint. It also proves the proxy-port refusal against a real published port.
|
|
213
|
-
- **Not implemented here**: the roll only runs whatever command a unit names; the hooks themselves
|
|
214
|
-
live in the service images. The Socketduct hub and gateway images now ship `appctl drain` and
|
|
215
|
-
`appctl in-flight` with the exit contract this document gates on (`0` the condition holds,
|
|
216
|
-
`1` it does not hold yet, `2` the probe cannot answer), so the example commands above name
|
|
217
|
-
something those images ship. The builder DinD still has no drain hook — its gate is the
|
|
218
|
-
in-flight probe in `kaspernj/tensorbuzz-builder` (`scripts/quiescence-probe.js`), which reports
|
|
219
|
-
whether a build is mid-flight.
|
|
220
|
-
- **Not changed here**: TensorBuzz's Socketduct provisioning path still issues raw
|
|
221
|
-
`docker compose up -d --force-recreate` when it provisions gateway legs. Switching that backend
|
|
222
|
-
to call `rollbridge compose-roll` is a TensorBuzz change, tracked separately from this one.
|