rollbridge 0.1.68 → 0.1.70

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -183,8 +183,9 @@ handoffs and exits only after the worker pool has drained.
183
183
 
184
184
  For a coordinator that starts quiescent, give the handoff jobs-main paired
185
185
  `lifecycle.activateCommand` and `quietCommand` hooks. After candidate health,
186
- Rollbridge waits for old retirement, waits for candidate activation, and commits
187
- the active proxy target without an awaited boundary. The durable transition is
186
+ Rollbridge activates the candidate, commits the active proxy target, and only
187
+ then starts the old generation's bounded retirement asynchronously. The durable
188
+ transition is
188
189
  generation-scoped and resumable; failures remain visible and block unrelated
189
190
  deploys. Post-commit singleton replacement is also journaled and must complete
190
191
  before an exact retry reports success. If the active coordinator restarts,
@@ -255,6 +256,14 @@ supervisor. The upgrade fails before handoff and requires one explicit clean
255
256
  remain atomic. A genuinely pre-replacement guardian still uses the separately
256
257
  documented one-time disruptive compatibility bridge below.
257
258
 
259
+ A retained guardian that reports a different rollbridge version from the daemon
260
+ runtime is handed to a current-version coordinator through that same bridge, so
261
+ a package upgrade does not need a supervisor restart: the coordinator owns the
262
+ daemon and every process it starts, the retained guardian keeps the processes it
263
+ already owns, and nothing waits for those to drain. `status.ownerTransition`
264
+ reports `mode: "guardian-version-handoff"`. The refusal is the last resort and
265
+ names both versions plus the exact `shutdown`/`ensure-daemon` commands.
266
+
258
267
  There is one explicit compatibility boundary: the first upgrade from a genuine
259
268
  pre-owner-replacement Rollbridge guardian and daemon cannot share its listeners
260
269
  on supported Node 20. The same boundary applies to a retained transition guardian
@@ -457,14 +466,14 @@ On `rollbridge deploy`, the required ordering is:
457
466
  1. starts any missing persistent service and the candidate's handoff services;
458
467
  2. starts the new release's `companion`s, then its `proxied` process, and
459
468
  health-checks the proxied process;
460
- 3. when `activateCommand` is configured, retires and acknowledges the previous
461
- jobs generation, then activates and acknowledges the candidate;
462
- 4. synchronously switches new traffic to the new release (hook-free configs keep
463
- the existing switch-then-retire behavior);
469
+ 3. when `activateCommand` is configured, activates and acknowledges the
470
+ candidate;
471
+ 4. synchronously switches new traffic to the new release;
464
472
  5. replaces `singleton`s (stops the old one, then starts the new one);
465
- 6. returns success without waiting for the previous generation or its independent
466
- HTTP/WebSocket drain; Rollbridge supervises all retained drains in the
467
- background and reaps each generation only after its handoffs and workers end.
473
+ 6. starts the previous generation's bounded retirement asynchronously after the
474
+ traffic switch; deploy success never waits for quiescence or drain, and
475
+ Rollbridge supervises all retained drains in the background and reaps each
476
+ generation only after its handoffs and workers end.
468
477
 
469
478
  If the new release fails to start or health-check, the previous release stays
470
479
  active and any service started during this deploy is rolled back.
package/TODO.md CHANGED
@@ -39,6 +39,7 @@ This roadmap tracks planned Rollbridge features and documentation. Rollbridge sh
39
39
  - [x] Implement opt-in durable guardian recovery for exact same-authority daemon process replacement without stopping retained generations.
40
40
  - [x] Implement guardian-fenced atomic incompatible owner/config/control-socket/package/runtime replacement without stopping retained generations.
41
41
  - [x] Add the authenticated one-time disruptive bridge for pre-replacement guardians; all protocol-capable replacements remain atomic.
42
+ - [x] Activate and traffic-switch a healthy candidate before asynchronously starting bounded old-generation retirement.
42
43
  - [x] Replicas and stable worker indexes. (Supported on port-less `companion` processes; `proxied`/`singleton`/ported processes stay single.)
43
44
  - [x] Allow one process config to start multiple replicas (`replicas`, companion-only for now).
44
45
  - [x] Expose `ROLLBRIDGE_REPLICA_INDEX`, replica count, and per-replica template context (`{{replicaIndex}}`/`{{replicaCount}}`).
@@ -0,0 +1,37 @@
1
+ ### Fixed
2
+
3
+ - A rollbridge version bump now hands the retained guardian to a current-version coordinator
4
+ automatically instead of refusing until a human restarts a daemon or a supervisor. Measured on
5
+ TensorBuzz production (STOE-2040): the live guardian was pid 2423469, 19 days up, running
6
+ `…/releases/20261028121152/backend/node_modules/rollbridge/src/process-guardian.js` (rollbridge
7
+ 0.1.55) while the daemon runtime was 0.1.61, and 45 deploys were refused with
8
+ `Another owner mutation is already in progress`. Nothing replaced the guardian: every daemon
9
+ start reuses `recovery.guardian` from the state file, so the version mismatch had no path out
10
+ at all.
11
+ - The retained guardian's version is read from its own executable (`guardianVersionFromProcess`)
12
+ before any replacement transaction is prepared, so the version probe does not move the
13
+ `capabilities` command ahead of `prepare-owner-replacement` where a capability-gated guardian
14
+ expects it. `test/owner-replacement.test.js` "replacement-capable guardian without daemon
15
+ recovery aborts before handoff and resumes incumbent drains" is the control for that ordering.
16
+ - A drifted guardian takes the authenticated upgrade bridge that already reconciles the one-time
17
+ legacy owner: the candidate starts a coordinator of its own version
18
+ (`<statePath>.guardian-<version>.sock`), the retained guardian keeps supervising every process
19
+ it already owns, and `status.ownerTransition` reports `mode: "guardian-version-handoff"` with
20
+ `disruptive: true`. Nothing waits for the retained generation to drain — the retained guardian
21
+ retires when its own processes exit. `test/owner-replacement.test.js`
22
+ "version-drifted retained guardian is handed to a current-version coordinator automatically"
23
+ asserts the retained worker pids stay live and reported by the retained guardian afterwards.
24
+ - Consequence of that design: every rollbridge version bump crosses the disruptive boundary once.
25
+ The incumbent daemon is `SIGKILL`ed, the supervised application processes keep their exact PIDs
26
+ under guardian supervision, and existing proxy/control connections may close. Replacements at
27
+ matching versions remain candidate-first and atomic.
28
+ - The refusal is now the last resort and names both versions plus the exact commands:
29
+ `The persistent Rollbridge guardian is version <guardian> while this daemon runs rollbridge
30
+ <daemon>; … Run "rollbridge shutdown --config <path>" and then "rollbridge ensure-daemon
31
+ --config <path>" to replace the supervisor with this Rollbridge installation.` It fires only
32
+ when no handoff can be authorised: no incumbent daemon to hand guardian ownership from, no
33
+ incumbent PID reported by `ensure-daemon`, an unattestable guardian version, an incumbent PID
34
+ that is not the attested owner, or a config change in the same invocation (the bridge keeps the
35
+ incumbent config identity unchanged). `test/owner-recovery.test.js`
36
+ "startup against a version-drifted guardian without an incumbent refuses and names both versions
37
+ and the repair command" asserts the refusal text and that the live generation is untouched.
@@ -0,0 +1,7 @@
1
+ ## Reclaim owner-mutation leases held by dead daemons
2
+
3
+ The guardian now verifies the recorded mutation holder PID before refusing a new
4
+ owner mutation. Dead holders are reclaimed and reported as an
5
+ `owner-mutation-reclaimed` event, so a daemon death cannot leave the deploy fence
6
+ wedged until an operator kills the guardian. `rollbridge status` and `doctor` now
7
+ show the guardian version and active mutation lease diagnostics.
@@ -0,0 +1 @@
1
+ - Activate and switch traffic to a ready replacement generation before quiescing the incumbent. A stalled retirement can no longer stop the active web or jobs generation before its successor is serving traffic; retirement failures remain observable on the retired release after the handoff.
@@ -0,0 +1,35 @@
1
+ ### Fixed
2
+
3
+ - A draining generation that still owns live connections is no longer reported `progressing`
4
+ without bound. The connection branch of `classifyDrainProgress` had no deadline, so a release
5
+ whose `proxy.drainTimeoutMs` wait had already expired kept claiming progress for as long as one
6
+ websocket stayed open. It now reports the new `timedOut` status, naming the budget and how far
7
+ past it the drain is: `drain budget 60000ms exceeded by 1000ms with 1 live connection still
8
+ owned by the generation`. Measured on TensorBuzz production 2026-11-17 (`shared/log/rollbridge.log`,
9
+ `tensorbuzz-production-web-1`): the incumbent retired by the failed deploy of `20261117063703`
10
+ began draining at `drainStartedAt 06:40:21.244Z` and was still read `draining` with
11
+ `drain.status: progressing`, `sampleWindowMs: 0` and one held websocket at 07:00Z and again at
12
+ 07:30Z, 25+ min past `proxy.drainTimeoutMs: 60000`. `timedOut` is a report, never an escalation:
13
+ `decideWedgedEscalation` still acts only on a `wedged` classification, and
14
+ `test/drain-policy.test.js` "the drain deadline names the deadline and never escalates" holds
15
+ that boundary. No signal path and no scheduled sampler is added.
16
+ - A candidate retired by a failed pre-commit activation is now drained by the compensation that
17
+ retired it. `compensatePreCommitActivationFailure` started the candidate's `drainAndPrune` only
18
+ on its success path, after the checkpoint clear, so when incumbent restoration threw the retired
19
+ candidate was left `draining` with its processes alive: candidate `20261117063703` still held
20
+ `web` pid 3201928 when read at 07:30Z, 47 min after its compensation threw at 06:42:31.115Z, and
21
+ reached `release drained` only when a later compensation reached its success path at
22
+ 08:47:00.159Z. A retained generation that still holds live processes is the state that refuses
23
+ every later deploy, and a failed candidate is never a restoration target. The drain now starts
24
+ where the candidate is recorded retired and stays in the background — the compensation does not
25
+ wait for it, so this cannot turn a failed activation into a blocking stop.
26
+ - New coverage for both halves: `test/release-group.test.js` "a draining generation that still owns
27
+ a connection past its drain budget is named timedOut, not left claiming progress" reads the
28
+ classification out of `status()` for a release whose `drainStartedAt` is 90 s old with one
29
+ connection still owned, and back to `progressing` for the same release inside budget.
30
+ `test/rollbridge.test.js` "a candidate retired by a failed activation is drained to stopped
31
+ instead of left holding live processes" fails activation, rejects the incumbent restoration, and
32
+ asserts the retired candidate reaches `stopped` with a `release drained` event — on the base tree
33
+ that wait times out because the drain was unreachable. "unresolved generation transition fences
34
+ stop, restart, and rollback mutations" additionally checks that the candidate its deploy retired
35
+ is stopped by its own compensation drain while the fence refuses every mutation.
@@ -0,0 +1,7 @@
1
+ # Stop companions after a retired dependency
2
+
3
+ Release companions that declare a same-generation service dependency now stop
4
+ after an unexpected exit when that dependency has been retired. They no longer
5
+ restart forever against the retired service's port. The TensorBuzz worker
6
+ example declares its jobs-main dependency and relies on the worker's lifecycle
7
+ handshake instead of a permanent port wait.
package/docs/cli.md CHANGED
@@ -116,12 +116,14 @@ the candidate binds and validates its listeners, and an authenticated fenced
116
116
  transaction commits guardian authority and the final control socket. A lost
117
117
  control response is accepted only when the guardian confirms the exact committed
118
118
  transaction id. The first authenticated upgrade from a genuine pre-replacement
119
- guardian/daemon, or from a retained prepare/stage guardian that explicitly lacks
120
- the retired-owner commit capability, is the sole exception: `ensure-daemon` preserves its exact
119
+ guardian/daemon, from a retained prepare/stage guardian that explicitly lacks
120
+ the retired-owner commit capability, or from a retained guardian whose rollbridge
121
+ version differs from the candidate runtime, is the exception: `ensure-daemon` preserves its exact
121
122
  guardian-owned processes but deliberately retires the old listeners before the
122
123
  Node 20 candidate binds, so existing proxy/control connections may close.
123
124
  Successful status JSON records this as `ownerTransition.disruptive: true` and
124
- `ownerTransition.mode: "legacy-first-upgrade"`. This one-time bridge requires
125
+ `ownerTransition.mode: "legacy-first-upgrade"` for a protocol gap or
126
+ `"guardian-version-handoff"` for a version drift. This bridge requires
125
127
  the incumbent config identity unchanged; make config/socket changes in a second,
126
128
  atomic invocation. Auth, transport, malformed-response, arbitrary unknown-command,
127
129
  and authority failures do not qualify and fail before any deploy is sent.
@@ -168,7 +170,7 @@ handoff would push the number of concurrently draining release groups past
168
170
  [`releaseRetention.maxConcurrentDrainingReleaseGroups`](config.md#releaseretention) —
169
171
  the incumbent the handoff is about to retire counts too. The incumbent keeps serving,
170
172
  nothing of the candidate is started, and the error names every draining group with its
171
- `progressing`/`wedged` classification and the bound it exceeded.
173
+ `progressing`/`timedOut`/`wedged` classification and the bound it exceeded.
172
174
 
173
175
  The same pre-activation refusal applies when `--release-id` names a release Rollbridge
174
176
  still **retains** while that generation holds live work: draining with processes alive, or
@@ -367,9 +369,14 @@ re-checked at most once every 5 seconds, and clears itself when its members leav
367
369
  Each **draining** release reports a `drain` classification — `status`, the `reason`
368
370
  behind it, the `sampleWindowMs` it rests on, and `cpuDeltaMs`:
369
371
 
370
- - `progressing` — the generation still owns live connections, a process is still
371
- inside its retirement budget, or its process-group CPU advanced across the
372
- sample window. A legitimate multi-hour drain reports this.
372
+ - `progressing` — the generation still owns live connections inside
373
+ `proxy.drainTimeoutMs`, a process is still inside its retirement budget, or its
374
+ process-group CPU advanced across the sample window. A legitimate multi-hour drain reports this.
375
+ - `timedOut` — the generation still owns live connections, but the configured drain budget has
376
+ already expired: `drain budget 60000ms exceeded by 900000ms with 1 live connection still owned
377
+ by the generation`. The connection wait was given up at `proxy.drainTimeoutMs`; the record says
378
+ so instead of claiming progress for as long as one socket stays open. It is a report only —
379
+ `timedOut` never authorises a signal, and escalation still requires `wedged`.
373
380
  - `wedged` — the drain can no longer progress: no live connection, every
374
381
  retirement budget expired, and either no runner child remains or the group's CPU
375
382
  is flat across a window of at least 20 seconds.
package/docs/config.md CHANGED
@@ -78,7 +78,7 @@ to let a deploy group talk to the daemon.
78
78
  | `proxy.upstreamHost` | string | `proxy.host`, or `"127.0.0.1"` when `proxy.host` is `0.0.0.0`/`::` | Host Rollbridge uses for release health checks and proxy targets. |
79
79
  | `proxy.healthPath` | string | `"/ping"` | Default health-check path for proxied processes. |
80
80
  | `proxy.healthTimeoutMs` | number | `120000` | Default boot budget for proxied processes: how long a release may take to answer its health path for the first time. Measured on TensorBuzz production (server3, 2026-11-06): the web app was still initialising when a `30000` deadline expired, so two deploys were killed mid-boot; `120000` is the budget the same release publishes on today. Waiting for a `dependsOn` service is not charged to it. |
81
- | `proxy.drainTimeoutMs` | number | `60000` | How long to drain HTTP/WebSocket connections before stopping the retired proxied process. Expiry never stops an independently draining jobs generation. |
81
+ | `proxy.drainTimeoutMs` | number | `60000` | How long to drain HTTP/WebSocket connections before stopping the retired proxied process. Expiry never stops an independently draining jobs generation. It is also the budget a `drain` classification measures elapsed drain against: a generation that still owns connections past it reports `timedOut` rather than `progressing` (see [`status` in the CLI guide](cli.md#status)). |
82
82
  | `proxy.nonBlockingDrainTimeoutMs` | number | `60000` | How long release retirement waits for an independently draining process before recording it as still draining and completing the release; `stop` and `rollback` hand the same bound to the processes they must not kill, so neither command can hold the owner-mutation fence for the whole drain. Expiry never sends `SIGKILL`; the process keeps its live work. |
83
83
  | `proxy.forceStopTimeoutMs` | number | `10000` | Default per-process graceful-stop timeout (`SIGTERM`, then `SIGKILL`). |
84
84
 
@@ -114,7 +114,7 @@ bound of 12.
114
114
 
115
115
  The refusal is a deploy failure like any other: the incumbent keeps serving, nothing of the
116
116
  candidate is started, and the named reason says which groups are still holding the fleet and
117
- whether each is `progressing` or `wedged` (see [`status` in the CLI
117
+ whether each is `progressing`, `timedOut`, or `wedged` (see [`status` in the CLI
118
118
  guide](cli.md#status)). A refused deploy is retried once a drain finishes; the bound is a
119
119
  policy, not a queue.
120
120
 
@@ -242,12 +242,16 @@ The first upgrade from an authenticated pre-replacement Rollbridge guardian and
242
242
  daemon uses an explicitly disruptive compatibility bridge because that legacy
243
243
  owner cannot transfer listeners on Node 20. A retained guardian that supports
244
244
  replacement prepare/stage but lacks the retired-owner commit command is explicitly
245
- classified and uses the same bridge. Rollbridge attests the exact guardian and
245
+ classified and uses the same bridge, and so is a retained guardian whose rollbridge
246
+ version differs from the daemon runtime: the candidate starts a coordinator of its
247
+ own version, the retained guardian keeps the processes it already owns, and nothing
248
+ waits for those to drain. Rollbridge attests the exact guardian and
246
249
  daemon processes, sockets, runtime/config authority, and durable process
247
250
  registrations before retiring the legacy listeners. Managed process PIDs and
248
251
  release state remain supervised, but live proxy/control connections may close.
249
252
  The resulting status includes `ownerTransition: {disruptive: true, mode:
250
- "legacy-first-upgrade", ...}`. The bridge only accepts the incumbent config
253
+ "legacy-first-upgrade", ...}` for a protocol gap or `{disruptive: true, mode:
254
+ "guardian-version-handoff", ...}` for a version drift. The bridge only accepts the incumbent config
251
255
  identity; retry config or socket changes after the protocol upgrade, when the
252
256
  normal atomic handoff applies. Other guardian/auth/transport/identity failures
253
257
  remain fail-closed.
@@ -27,6 +27,33 @@ preparation itself fails, check permissions for
27
27
  `/tmp/rollbridge-<user-id>-<application-hash>-runtime`) before retrying. The directory
28
28
  must be private to the invoking user.
29
29
 
30
+ ## Retained guardian on an older rollbridge version
31
+
32
+ **Symptom.** `ensure-daemon` reports `The persistent Rollbridge guardian is version <x> while
33
+ this daemon runs rollbridge <y>` and confirms that the deploy was not sent.
34
+
35
+ **Cause.** The daemon reuses `recovery.guardian` from the state file on every start, so a
36
+ supervisor left running from an earlier release outlives the package it came from. A version
37
+ mismatch is normally reconciled without an operator: `ensure-daemon` starts a coordinator of its
38
+ own version over the retained guardian, the retained guardian keeps supervising the processes it
39
+ already owns, and nothing waits for them to drain. `status.ownerTransition` then reports
40
+ `mode: "guardian-version-handoff"` with `disruptive: true`; the incumbent daemon is retired while
41
+ its supervised application processes keep their exact PIDs.
42
+
43
+ **Fix.** The refusal is the last resort and names both versions plus the exact commands. Run them
44
+ in that order:
45
+
46
+ ```sh
47
+ rollbridge shutdown --config /etc/rollbridge/app.cjs
48
+ rollbridge ensure-daemon --config /etc/rollbridge/app.cjs
49
+ ```
50
+
51
+ It fires only when no handoff can be authorised: no incumbent daemon to hand guardian ownership
52
+ from, no incumbent PID reported by `ensure-daemon`, a guardian version that cannot be attested, an
53
+ incumbent PID that is not the attested owner, or a config change in the same invocation — the
54
+ bridge keeps the incumbent config identity unchanged, so apply a config or socket change in a
55
+ second invocation once the supervisor is current.
56
+
30
57
  If startup instead reports `Rollbridge daemon candidate <pid> exited before
31
58
  readiness`, the ensuring CLI observed that exact child exit before it could
32
59
  attest status. The diagnostic includes the exit code or signal and spawned
package/docs/workers.md CHANGED
@@ -26,6 +26,7 @@ fixed port such as `7330`.
26
26
  {
27
27
  id: "background-jobs-worker",
28
28
  policy: "companion",
29
+ dependsOn: ["background-jobs-main"],
29
30
  env: {VELOCIOUS_BACKGROUND_JOBS_PORT: "{{ports.background-jobs-main}}"},
30
31
  command: "npx velocious background-jobs-worker",
31
32
  replicas: 4,
@@ -60,6 +60,7 @@ export default {
60
60
  {
61
61
  id: "background-jobs-worker",
62
62
  policy: "companion",
63
+ dependsOn: ["background-jobs-main"],
63
64
  cwd: "{{releasePath}}/backend",
64
65
  env: {
65
66
  NODE_ENV: "production",
@@ -67,7 +68,7 @@ export default {
67
68
  VELOCIOUS_BEACON_PORT: "{{ports.beacon}}",
68
69
  VELOCIOUS_BACKGROUND_JOBS_PORT: "{{ports.background-jobs-main}}"
69
70
  },
70
- command: "wait-for-it 127.0.0.1:{{ports.beacon}} --strict -- wait-for-it 127.0.0.1:{{ports.background-jobs-main}} --strict -- npx velocious background-jobs-worker",
71
+ command: "wait-for-it 127.0.0.1:{{ports.beacon}} --strict -- npx velocious background-jobs-worker",
71
72
  lifecycle: {
72
73
  quietCommand: "appctl jobs-worker-retire --pid $ROLLBRIDGE_PID",
73
74
  reactivateCommand: "appctl jobs-worker-reactivate --pid $ROLLBRIDGE_PID"
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "rollbridge",
3
- "version": "0.1.68",
3
+ "version": "0.1.70",
4
4
  "description": "Zero-downtime process supervisor and local traffic switcher for deploy-managed apps.",
5
5
  "keywords": [
6
6
  "deploy",