rollbridge 0.1.27 → 0.1.29
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/AGENTS.md +14 -0
- package/README.md +69 -14
- package/TODO.md +5 -2
- package/changelog.d/20260828-atomic-owner-replacement.md +22 -0
- package/changelog.d/20260828-durable-owner-recovery.md +7 -0
- package/changelog.d/20260828-same-owner-jobs-generations.md +6 -0
- package/docs/cli.md +43 -21
- package/docs/config.md +71 -18
- package/docs/logging.md +8 -3
- package/docs/tensorbuzz-runbook.md +7 -6
- package/docs/troubleshooting.md +28 -12
- package/docs/velocious.md +11 -4
- package/docs/workers.md +8 -2
- package/examples/tensorbuzz.com.js +12 -4
- package/package.json +1 -1
- package/src/cli.js +209 -36
- package/src/config.js +8 -2
- package/src/control-client.js +118 -1
- package/src/daemon.js +921 -45
- package/src/guardian-client.js +429 -0
- package/src/managed-process.js +45 -15
- package/src/process-guardian.js +588 -0
- package/src/release-group.js +190 -15
- package/src/state-store.js +1 -1
- package/test/config-validation.test.js +22 -0
- package/test/fixtures/pre-split3-daemon-runner.js +30 -0
- package/test/fixtures/pre-split3-daemon.js +1336 -0
- package/test/fixtures/pre-split3-guardian-client.js +293 -0
- package/test/fixtures/pre-split3-process-guardian.js +292 -0
- package/test/fixtures/service-app.js +32 -2
- package/test/guardian-client.test.js +304 -0
- package/test/owner-recovery.test.js +858 -0
- package/test/owner-replacement.test.js +772 -0
- package/test/release-runtime-retention.test.js +1 -1
- package/test/rollbridge.test.js +178 -5
- package/test/shutdown-completion.test.js +1 -1
- package/test/state-store.test.js +12 -0
package/AGENTS.md
CHANGED
|
@@ -40,6 +40,20 @@ implemented. Do not claim production compliance when source/config still uses a
|
|
|
40
40
|
fixed jobs-main, worker adoption by a new main, destructive orphan recovery, or
|
|
41
41
|
synchronous cleanup.
|
|
42
42
|
|
|
43
|
+
Current same-authority behavior quiesces configured handoff services after
|
|
44
|
+
candidate activation, retains concurrent generations, reports live release
|
|
45
|
+
references, and can opt into `ownerRecovery` so a durable process guardian
|
|
46
|
+
preserves and reconstructs active/draining generations after daemon process
|
|
47
|
+
exit. With `ownerRecovery` and the same `statePath` transaction anchor,
|
|
48
|
+
`ensure-daemon` also replaces incompatible config, control-socket, package, and
|
|
49
|
+
runtime owners through a guardian-fenced candidate-first handoff while retaining
|
|
50
|
+
active and draining generations. The first authenticated upgrade from a genuine
|
|
51
|
+
pre-replacement guardian/daemon is a documented disruptive compatibility bridge:
|
|
52
|
+
it preserves exact supervised processes and state, but may close proxy/control
|
|
53
|
+
connections. Every protocol-capable replacement after that bridge is atomic.
|
|
54
|
+
`--takeover-owner` remains a destructive
|
|
55
|
+
external-supervisor migration path, not that atomic handoff.
|
|
56
|
+
|
|
43
57
|
## Validation and publication
|
|
44
58
|
|
|
45
59
|
The project is ESM JavaScript with JSDoc type checking. Package scripts are the
|
package/README.md
CHANGED
|
@@ -83,6 +83,7 @@ export default {
|
|
|
83
83
|
deployStrategy: "handoff",
|
|
84
84
|
cwd: "{{releasePath}}",
|
|
85
85
|
command: "env VELOCIOUS_BACKGROUND_JOBS_PORT={{port}} npx velocious background-jobs-main",
|
|
86
|
+
lifecycle: {quietCommand: "appctl jobs-main-retire --pid $ROLLBRIDGE_PID"},
|
|
86
87
|
port: {from: 7331, to: 7399}
|
|
87
88
|
},
|
|
88
89
|
{
|
|
@@ -97,6 +98,9 @@ export default {
|
|
|
97
98
|
}
|
|
98
99
|
```
|
|
99
100
|
|
|
101
|
+
`appctl jobs-main-retire` is illustrative; replace it with a reviewed command
|
|
102
|
+
that quiesces the real jobs-main without exiting it.
|
|
103
|
+
|
|
100
104
|
Each process retains its most recent stdout/stderr lines and reports them in
|
|
101
105
|
`status`. Set `outputLines` (a positive integer, default 50) per process to keep
|
|
102
106
|
more or fewer lines for chatty or quiet processes.
|
|
@@ -169,6 +173,13 @@ handoffs as soon as its release retires, independently of the proxied connection
|
|
|
169
173
|
drain. Its release-scoped handoff jobs-main remains running to supervise existing
|
|
170
174
|
handoffs and exits only after the worker pool has drained.
|
|
171
175
|
|
|
176
|
+
Give the handoff jobs-main a `lifecycle.quietCommand` that stops schedules,
|
|
177
|
+
dispatch, and new handoffs without terminating the main. After candidate health
|
|
178
|
+
and traffic activation, Rollbridge waits only for this bounded quiescence step,
|
|
179
|
+
then returns while the old main and workers drain together. A failed quiet hook
|
|
180
|
+
is reported as `retirementError`; Rollbridge leaves that generation alive for
|
|
181
|
+
diagnosis instead of silently continuing to stop it.
|
|
182
|
+
|
|
172
183
|
See [`docs/workers.md`](docs/workers.md) for the full release-generation
|
|
173
184
|
deployment pattern: a handoff `background-jobs-main`, its companion worker pool,
|
|
174
185
|
independent quiescence, and durable supervision while retained generations drain.
|
|
@@ -184,13 +195,13 @@ owns cleaning up on-disk release directories.
|
|
|
184
195
|
releaseRetention: {keep: 5, maxAgeMs: 86400000}
|
|
185
196
|
```
|
|
186
197
|
|
|
187
|
-
Set `statePath` to have the daemon persist
|
|
198
|
+
Set `statePath` to have the daemon persist sanitized recovery state to a file
|
|
188
199
|
(active/draining releases, process pids, counters, sanitized recent events).
|
|
189
200
|
Commands, environment mappings, child command lines, and captured process output
|
|
190
201
|
are deliberately excluded; those diagnostics remain available through the live
|
|
191
|
-
status/log/event APIs.
|
|
192
|
-
reports managed processes still alive from a daemon that
|
|
193
|
-
— advisory orphan detection.
|
|
202
|
+
status/log/event APIs. Without `ownerRecovery`, the next startup reads any
|
|
203
|
+
leftover file and reports managed processes still alive from a daemon that
|
|
204
|
+
didn't shut down cleanly — advisory orphan detection. In that mode, after a crash run
|
|
194
205
|
`rollbridge recover` to list those leftovers and `rollbridge recover --force` to
|
|
195
206
|
stop them before restarting the daemon. A clean `shutdown` removes the file. See
|
|
196
207
|
[`docs/config.md`](docs/config.md#statepath).
|
|
@@ -199,6 +210,41 @@ stop them before restarting the daemon. A clean `shutdown` removes the file. See
|
|
|
199
210
|
statePath: "/var/lib/rollbridge/ticket-server.state.json"
|
|
200
211
|
```
|
|
201
212
|
|
|
213
|
+
For durable daemon process recovery and atomic owner replacement, opt into
|
|
214
|
+
`ownerRecovery`. Rollbridge
|
|
215
|
+
then runs a private local process guardian which remains the OS supervisor for
|
|
216
|
+
managed processes if the control daemon exits unexpectedly. A replacement using
|
|
217
|
+
the exact same normalized config/runtime reconnects within `reconnectGraceMs`,
|
|
218
|
+
reconstructs active and draining generations and their ports, and fences
|
|
219
|
+
concurrent replacements. `ensure-daemon` can also prepare a requested
|
|
220
|
+
config/control-socket/package/runtime owner, prove it healthy, and atomically
|
|
221
|
+
transfer guardian authority while every retained generation keeps its exact
|
|
222
|
+
release reference and drains asynchronously. The old `statePath` is the durable
|
|
223
|
+
transaction anchor and cannot change during this handoff. The `0600` state file
|
|
224
|
+
contains the guardian capability; protect its directory accordingly.
|
|
225
|
+
Prepared transactions fence owner mutations and compare a monotonic guardian
|
|
226
|
+
state revision at staging. Existing HTTP/WebSocket connections remain owned by
|
|
227
|
+
the retired listener process, while their counts transfer to the new daemon so
|
|
228
|
+
later deploys continue to honor the original drain boundary.
|
|
229
|
+
|
|
230
|
+
There is one explicit compatibility boundary: the first upgrade from a genuine
|
|
231
|
+
pre-owner-replacement Rollbridge guardian and daemon cannot share its listeners
|
|
232
|
+
on supported Node 20. After authenticating the guardian, exact daemon PID/socket,
|
|
233
|
+
runtime authority, and durable owned-process state, `ensure-daemon` performs a
|
|
234
|
+
one-time **disruptive** bridge. Existing proxy/control connections may close,
|
|
235
|
+
the retained processes keep their exact PIDs under guardian supervision, and
|
|
236
|
+
`status.ownerTransition` reports `mode: "legacy-first-upgrade"` with
|
|
237
|
+
`disruptive: true`. The bridge requires the existing config identity unchanged;
|
|
238
|
+
apply config/socket changes in a subsequent invocation, which uses the atomic
|
|
239
|
+
protocol. Unknown commands, auth/transport failures, malformed responses, and
|
|
240
|
+
identity mismatches fail closed without entering this bridge. Every replacement
|
|
241
|
+
after this protocol upgrade remains candidate-first and atomic.
|
|
242
|
+
|
|
243
|
+
```js
|
|
244
|
+
statePath: "/var/lib/rollbridge/ticket-server.state.json",
|
|
245
|
+
ownerRecovery: {reconnectGraceMs: 30000}
|
|
246
|
+
```
|
|
247
|
+
|
|
202
248
|
During the first migration from an old supervisor, set `legacyTakeover` and run
|
|
203
249
|
`rollbridge predeploy-cleanup --release-path <path>` before `rollbridge deploy`.
|
|
204
250
|
Rollbridge will only stop configured legacy processes when no reusable active
|
|
@@ -241,7 +287,7 @@ rendered when the process starts:
|
|
|
241
287
|
Referencing a placeholder with no value (including an unset `{{env.<NAME>}}`)
|
|
242
288
|
fails the process start with a clear error, so typos surface immediately.
|
|
243
289
|
|
|
244
|
-
|
|
290
|
+
Configuration examples live in `examples/`, including
|
|
245
291
|
`examples/tensorbuzz.com.js` for the current TensorBuzz backend deployment; see
|
|
246
292
|
[`docs/tensorbuzz-runbook.md`](docs/tensorbuzz-runbook.md) for the matching
|
|
247
293
|
production runbook (ports, deploy ordering, rollback constraints, and day-to-day
|
|
@@ -338,6 +384,7 @@ candidate coordinator.
|
|
|
338
384
|
deployStrategy: "handoff",
|
|
339
385
|
cwd: "{{releasePath}}",
|
|
340
386
|
command: "npx velocious background-jobs-main",
|
|
387
|
+
lifecycle: {quietCommand: "appctl jobs-main-retire --pid $ROLLBRIDGE_PID"},
|
|
341
388
|
port: {from: 7331, to: 7399}
|
|
342
389
|
}
|
|
343
390
|
```
|
|
@@ -350,7 +397,7 @@ On `rollbridge deploy`, the required ordering is:
|
|
|
350
397
|
2. starts the new release's `companion`s, then its `proxied` process, and
|
|
351
398
|
health-checks the proxied process;
|
|
352
399
|
3. switches new traffic to the new release;
|
|
353
|
-
4.
|
|
400
|
+
4. quiesces the previous jobs-main and worker pool as one retired generation;
|
|
354
401
|
5. replaces `singleton`s (stops the old one, then starts the new one);
|
|
355
402
|
6. returns success without waiting for the previous generation or its independent
|
|
356
403
|
HTTP/WebSocket drain; Rollbridge supervises all retained drains in the
|
|
@@ -359,6 +406,12 @@ On `rollbridge deploy`, the required ordering is:
|
|
|
359
406
|
If the new release fails to start or health-check, the previous release stays
|
|
360
407
|
active and any service started during this deploy is rolled back.
|
|
361
408
|
|
|
409
|
+
`status.releaseReferences` lists the id and path of every active or draining
|
|
410
|
+
release. A reference disappears only when that release is fully stopped. With
|
|
411
|
+
`ownerRecovery`, those references and generations survive both same-authority
|
|
412
|
+
daemon recovery and guardian-fenced incompatible config/control-socket/package/
|
|
413
|
+
runtime replacement through `ensure-daemon`.
|
|
414
|
+
|
|
362
415
|
## Commands
|
|
363
416
|
|
|
364
417
|
`--config` is optional for every command. When omitted, Rollbridge looks for
|
|
@@ -439,19 +492,21 @@ rollbridge daemon --config /srv/ticket-server/rollbridge.js \
|
|
|
439
492
|
External supervisors that need to replace a foreground owner without waiting
|
|
440
493
|
for retained generations to drain add `--takeover-owner`. The candidate starts
|
|
441
494
|
and health-checks the exact release first. Only then does it retire the accepted
|
|
442
|
-
owner's proxy/control listeners.
|
|
443
|
-
|
|
444
|
-
|
|
445
|
-
|
|
446
|
-
|
|
495
|
+
owner's proxy/control listeners. Candidate bootstrap failure leaves the accepted
|
|
496
|
+
owner untouched. Current takeover does not preserve retained jobs generations
|
|
497
|
+
or transfer listeners atomically, so it is not a zero-downtime package, config,
|
|
498
|
+
or socket upgrade mechanism. Use `ownerRecovery` plus `ensure-daemon` for the
|
|
499
|
+
guardian-fenced atomic replacement contract. This is opt-in; ordinary daemon
|
|
500
|
+
bootstrap and `shutdown` keep their existing behavior.
|
|
447
501
|
|
|
448
502
|
The four bootstrap inputs are all-or-nothing and use absolute config/release
|
|
449
503
|
paths. Rollbridge binds its proxy, activates the release through the normal
|
|
450
504
|
deploy path, then exposes the control socket and stays foreground. A failed
|
|
451
505
|
activation stops only processes started by that attempt and exits non-zero;
|
|
452
|
-
persisted processes from a previous daemon are reported
|
|
453
|
-
|
|
454
|
-
|
|
506
|
+
without `ownerRecovery`, persisted processes from a previous daemon are reported
|
|
507
|
+
as advisory orphans and are never recovered or killed implicitly. With
|
|
508
|
+
`ownerRecovery`, bootstrap reconnects only to the matching guardian and
|
|
509
|
+
reconstructs its provenanced generations before exposing control.
|
|
455
510
|
|
|
456
511
|
External supervisors may add `--boot-attestation` with exactly `sha256:` plus
|
|
457
512
|
64 lowercase hexadecimal characters. After successful activation, `rollbridge
|
package/TODO.md
CHANGED
|
@@ -35,7 +35,10 @@ This roadmap tracks planned Rollbridge features and documentation. Rollbridge sh
|
|
|
35
35
|
- [x] Support signal-only lifecycle steps for workers that can quiet on a Unix signal. (Per-process `stopSignal`; sent before the `SIGKILL`-after-`gracefulStopMs` fallback.)
|
|
36
36
|
- [x] Add a non-blocking drain mode so a worker can quiesce at release retirement independently of the HTTP/WebSocket connection drain (`nonBlockingDrain`). This control alone does not provide durable retired-generation supervision.
|
|
37
37
|
- [x] Document the required Velocious release-generation contract (`docs/velocious.md` and `docs/workers.md`) without treating documentation as proof that the runtime implements it.
|
|
38
|
-
- [
|
|
38
|
+
- [x] Implement same-owner jobs-main retirement, concurrent generations, independent drains, and live release-reference reporting.
|
|
39
|
+
- [x] Implement opt-in durable guardian recovery for exact same-authority daemon process replacement without stopping retained generations.
|
|
40
|
+
- [x] Implement guardian-fenced atomic incompatible owner/config/control-socket/package/runtime replacement without stopping retained generations.
|
|
41
|
+
- [x] Add the authenticated one-time disruptive bridge for pre-replacement guardians; all protocol-capable replacements remain atomic.
|
|
39
42
|
- [x] Replicas and stable worker indexes. (Supported on port-less `companion` processes; `proxied`/`singleton`/ported processes stay single.)
|
|
40
43
|
- [x] Allow one process config to start multiple replicas (`replicas`, companion-only for now).
|
|
41
44
|
- [x] Expose `ROLLBRIDGE_REPLICA_INDEX`, replica count, and per-replica template context (`{{replicaIndex}}`/`{{replicaCount}}`).
|
|
@@ -43,7 +46,7 @@ This roadmap tracks planned Rollbridge features and documentation. Rollbridge sh
|
|
|
43
46
|
- [x] Preserve readable status output for replica groups (each instance shown as `<id>#<index>`).
|
|
44
47
|
- [x] Persistent daemon state and recovery.
|
|
45
48
|
- [x] Persist active release, draining releases, process metadata, counters, and recent events (opt-in `statePath`; atomic snapshot on change + periodic).
|
|
46
|
-
- [x] Reconnect
|
|
49
|
+
- [x] Reconnect to guardian-owned child processes when `ownerRecovery` is enabled; otherwise `status.orphans` retains advisory legacy behavior.
|
|
47
50
|
- [x] Detect and report orphaned Rollbridge-managed processes. (On startup, reports persisted process pids that are still alive; advisory, see `statePath`.)
|
|
48
51
|
- [x] Add a recovery mode for safe startup after daemon crash or machine reboot. (`rollbridge recover` lists orphaned processes from the persisted state and, with `--force`, stops them and clears the state; refuses while a daemon is running.)
|
|
49
52
|
- [x] Rollback support.
|
|
@@ -0,0 +1,22 @@
|
|
|
1
|
+
### Added
|
|
2
|
+
|
|
3
|
+
- Atomically replace incompatible config, control-socket, package, and runtime
|
|
4
|
+
owners through `ensure-daemon` when durable `ownerRecovery` is configured.
|
|
5
|
+
The guardian fences candidates, transfers exact active and draining release
|
|
6
|
+
supervision only after candidate readiness, and converges after an ambiguous
|
|
7
|
+
commit response or a crash at the committed-state boundary.
|
|
8
|
+
- Bridge the first authenticated pre-replacement guardian/daemon upgrade while
|
|
9
|
+
preserving exact supervised process PIDs and release state. This one-time
|
|
10
|
+
compatibility transition is explicitly disruptive: existing proxy/control
|
|
11
|
+
connections may close and status records `legacy-first-upgrade`. Subsequent
|
|
12
|
+
protocol-capable owner replacements remain atomic, including on Node 20.
|
|
13
|
+
- Fence post-prepare owner-state mutations with a guardian revision check and
|
|
14
|
+
carry incumbent listener connection counts until the old HTTP/WebSocket
|
|
15
|
+
sockets drain, so later deploys cannot stop a still-connected process early.
|
|
16
|
+
- Reserve every live generation's allocated endpoints before its processes bind
|
|
17
|
+
and until that generation truly stops, and reconstruct those reservations on
|
|
18
|
+
owner recovery so overlapping jobs generations cannot reuse an endpoint.
|
|
19
|
+
- Keep an exact newly spawned daemon candidate referenced until its status and
|
|
20
|
+
PID authority are verified. `ensure-daemon` now reports an exact early child
|
|
21
|
+
exit instead of leaving its top-level operation unsettled or waiting for a
|
|
22
|
+
generic readiness timeout; the healthy daemon is detached only afterward.
|
|
@@ -0,0 +1,7 @@
|
|
|
1
|
+
### Added
|
|
2
|
+
|
|
3
|
+
- Add opt-in `ownerRecovery` with a durable authenticated process guardian so an
|
|
4
|
+
exact same-authority daemon replacement reconstructs active and draining
|
|
5
|
+
generations, ports, lifecycle supervision, and release references after an
|
|
6
|
+
unexpected daemon exit. Concurrent replacements are fenced and mismatched or
|
|
7
|
+
partial recovery state fails closed.
|
|
@@ -0,0 +1,6 @@
|
|
|
1
|
+
## Fixed
|
|
2
|
+
|
|
3
|
+
- Quiesce release-scoped handoff services after healthy candidate activation,
|
|
4
|
+
retain them with their original workers during asynchronous drain, and report
|
|
5
|
+
active/draining release references. Cross-owner recovery and atomic owner
|
|
6
|
+
upgrades remain future work.
|
package/docs/cli.md
CHANGED
|
@@ -50,9 +50,9 @@ or number). Supplying only some bootstrap options, or an invalid value, exits
|
|
|
50
50
|
non-zero before listeners start. Activation failure emits a structured
|
|
51
51
|
`bootstrap activation failed` event, cleans up processes owned by that attempt,
|
|
52
52
|
and exits non-zero without exposing the control socket or inventing an active
|
|
53
|
-
release. `
|
|
54
|
-
|
|
55
|
-
|
|
53
|
+
release. With `ownerRecovery`, daemon startup first claims the matching guardian
|
|
54
|
+
and reconstructs its active/draining generations. Without it, `statePath`
|
|
55
|
+
entries remain advisory orphans for explicit recovery.
|
|
56
56
|
|
|
57
57
|
`--boot-attestation` is an optional, non-secret opaque ownership token for an
|
|
58
58
|
external supervisor. Its canonical format is exactly `sha256:` followed by 64
|
|
@@ -73,9 +73,10 @@ preserves nor transfers retained-generation supervision to the replacement.
|
|
|
73
73
|
The replacement can bind before those stops finish. A bootstrap failure occurs
|
|
74
74
|
before retirement, so the previously accepted owner remains available.
|
|
75
75
|
|
|
76
|
-
|
|
77
|
-
|
|
78
|
-
|
|
76
|
+
`ownerRecovery` covers unexpected process exit under the exact same authority.
|
|
77
|
+
Guardian-fenced incompatible config/package/runtime/control-socket handoff is
|
|
78
|
+
provided by `ensure-daemon`, not `--takeover-owner`; it preserves supervision
|
|
79
|
+
without handing old workers to a new jobs-main.
|
|
79
80
|
|
|
80
81
|
## `ensure-daemon`
|
|
81
82
|
|
|
@@ -95,15 +96,29 @@ Before starting a detached daemon, Rollbridge atomically copies its runtime code
|
|
|
95
96
|
and production dependency closure into a content-addressed directory outside
|
|
96
97
|
the invoking release. This keeps the long-lived daemon valid when deploy
|
|
97
98
|
retention removes that release. A responsive daemon is reused only when its
|
|
98
|
-
runtime
|
|
99
|
-
|
|
100
|
-
|
|
99
|
+
runtime and normalized config authority match. With `ownerRecovery` and the same
|
|
100
|
+
`statePath`, an incompatible config/control-socket/package/runtime owner is
|
|
101
|
+
replaced candidate-first: the guardian retains active and draining generations,
|
|
102
|
+
the candidate binds and validates its listeners, and an authenticated fenced
|
|
103
|
+
transaction commits guardian authority and the final control socket. A lost
|
|
104
|
+
control response is accepted only when the guardian confirms the exact committed
|
|
105
|
+
transaction id. The first authenticated upgrade from a genuine pre-replacement
|
|
106
|
+
guardian/daemon is the sole exception: `ensure-daemon` preserves its exact
|
|
107
|
+
guardian-owned processes but deliberately retires the old listeners before the
|
|
108
|
+
Node 20 candidate binds, so existing proxy/control connections may close.
|
|
109
|
+
Successful status JSON records this as `ownerTransition.disruptive: true` and
|
|
110
|
+
`ownerTransition.mode: "legacy-first-upgrade"`. This one-time bridge requires
|
|
111
|
+
the incumbent config identity unchanged; make config/socket changes in a second,
|
|
112
|
+
atomic invocation. Auth, transport, malformed-response, arbitrary unknown-command,
|
|
113
|
+
and authority failures do not qualify and fail before any deploy is sent.
|
|
101
114
|
|
|
102
115
|
- `--daemon-log-path <path>` — file the detached daemon's stdout/stderr is
|
|
103
116
|
appended to. Default: `/tmp/rollbridge-<application>.log`. See
|
|
104
117
|
[`logging.md`](logging.md) for the log format and rotation guidance.
|
|
105
118
|
- `--daemon-pid-path <path>` — file the detached daemon's PID is written to.
|
|
106
|
-
Default: `/tmp/rollbridge-<application>.pid`.
|
|
119
|
+
Default: `/tmp/rollbridge-<application>.pid`. During replacement, the file
|
|
120
|
+
continues to name the incumbent until the reachable winner reports and
|
|
121
|
+
publishes its exact `daemonPid`.
|
|
107
122
|
- `--daemon-runtime-path <path>` — parent directory for content-addressed daemon
|
|
108
123
|
runtime snapshots. Default:
|
|
109
124
|
`/tmp/rollbridge-<user-id>-<application-hash>-runtime`. The directory must be owned
|
|
@@ -134,17 +149,24 @@ for old workers, jobs, or HTTP/WebSocket connections to finish. Prints
|
|
|
134
149
|
If the new release fails to start or health-check, the previous release stays
|
|
135
150
|
active and the command errors.
|
|
136
151
|
|
|
137
|
-
After
|
|
138
|
-
|
|
139
|
-
|
|
140
|
-
|
|
141
|
-
|
|
142
|
-
|
|
143
|
-
|
|
144
|
-
|
|
145
|
-
|
|
146
|
-
|
|
147
|
-
|
|
152
|
+
After activation, deploy waits for bounded quiet hooks of old handoff services
|
|
153
|
+
and `nonBlockingDrain` companions, then returns without waiting for their drains.
|
|
154
|
+
A failure is logged and exposed as `retirementError`; the generation stays alive.
|
|
155
|
+
The successful activation response also includes
|
|
156
|
+
`retirement: {status: "quiescence_failed", releaseId, error}` so callers cannot
|
|
157
|
+
mistake the retirement failure for an unqualified transition.
|
|
158
|
+
`status.releaseReferences` lists `{releaseId, releasePath}` for every active or
|
|
159
|
+
draining release and excludes fully stopped history.
|
|
160
|
+
|
|
161
|
+
After candidate activation, `Daemon.deploy()` begins old-generation retirement
|
|
162
|
+
and asynchronous drain before awaiting singleton replacement. A singleton
|
|
163
|
+
replacement failure can therefore return non-zero while the candidate remains
|
|
164
|
+
active, but it cannot leave the old jobs generation dispatching.
|
|
165
|
+
|
|
166
|
+
With `ownerRecovery`, active and draining generations remain guardian-supervised
|
|
167
|
+
across unexpected same-authority daemon exit and reconstruct on replacement;
|
|
168
|
+
they also transfer intact through an incompatible `ensure-daemon` owner handoff.
|
|
169
|
+
Without it, surviving PIDs remain advisory orphans for `recover --force`.
|
|
148
170
|
|
|
149
171
|
Before each deploy, the daemon reloads the config path it was started with.
|
|
150
172
|
Compatible process and lifecycle changes apply to the new release and govern
|
package/docs/config.md
CHANGED
|
@@ -46,6 +46,7 @@ restart.
|
|
|
46
46
|
| `application` | string | basename of the config file's directory | Names the app; used in the default control-socket path and the `ROLLBRIDGE_APPLICATION` env var. |
|
|
47
47
|
| `control` | object | — | Control-socket settings (see below). |
|
|
48
48
|
| `legacyTakeover` | object | unset | Optional matchers for `rollbridge predeploy-cleanup` to stop pre-Rollbridge supervisors during first handover (see below). |
|
|
49
|
+
| `ownerRecovery` | object | unset | Durable guardian recovery and atomic incompatible owner replacement; requires `statePath` (see below). |
|
|
49
50
|
| `proxy` | object | **required** | Proxy listener and shared defaults (see below). |
|
|
50
51
|
| `processes` | array | **required** | Managed processes (see below). Exactly one must be `proxied`. |
|
|
51
52
|
| `releaseRetention` | object | — | How many stopped releases the daemon retains (see below). |
|
|
@@ -90,7 +91,7 @@ release records; the deploy tool still owns on-disk release directories.
|
|
|
90
91
|
|
|
91
92
|
## `statePath`
|
|
92
93
|
|
|
93
|
-
When set, the daemon persists a
|
|
94
|
+
When set, the daemon persists a sanitized operational state snapshot — the active and
|
|
94
95
|
draining releases, each managed process's recovery metadata (including pid),
|
|
95
96
|
restart counters, and recent structured events — to this file (atomically, on
|
|
96
97
|
changes and every few seconds). Process commands, working directories,
|
|
@@ -98,13 +99,13 @@ environment mappings, child command lines, and retained stdout/stderr are never
|
|
|
98
99
|
persisted. They remain available from the live `status`, `logs`, and `events`
|
|
99
100
|
APIs while the daemon is running. On a clean `shutdown` the file is removed.
|
|
100
101
|
|
|
101
|
-
|
|
102
|
-
processes whose pids are still alive — likely orphans from a daemon that
|
|
103
|
-
without shutting down cleanly — in its log
|
|
102
|
+
Without `ownerRecovery`, the **next startup** reads any leftover file and reports
|
|
103
|
+
managed processes whose pids are still alive — likely orphans from a daemon that
|
|
104
|
+
crashed without shutting down cleanly — in its log/event history and the
|
|
104
105
|
`orphans` array of [`rollbridge status`](cli.md#status). This is **advisory**:
|
|
105
|
-
|
|
106
|
-
automatically
|
|
107
|
-
|
|
106
|
+
that mode cannot re-adopt detached children, so it does not stop them
|
|
107
|
+
automatically. A recycled pid can be a false positive, so treat a report as a
|
|
108
|
+
prompt to investigate. Use
|
|
108
109
|
[`rollbridge recover`](cli.md#recover) to list and (with `--force`) stop those
|
|
109
110
|
orphans after a crash.
|
|
110
111
|
|
|
@@ -114,6 +115,54 @@ statePath: "/var/lib/rollbridge/ticket-server.state.json"
|
|
|
114
115
|
|
|
115
116
|
Leave `statePath` unset to disable persistence (the default).
|
|
116
117
|
|
|
118
|
+
Set `ownerRecovery` to opt into same-authority process-exit recovery and atomic
|
|
119
|
+
incompatible owner replacement:
|
|
120
|
+
|
|
121
|
+
```js
|
|
122
|
+
statePath: "/var/lib/rollbridge/ticket-server.state.json",
|
|
123
|
+
ownerRecovery: {reconnectGraceMs: 30000}
|
|
124
|
+
```
|
|
125
|
+
|
|
126
|
+
The private guardian socket is derived from `statePath`; the atomic state file is
|
|
127
|
+
written mode `0600` and contains its authentication capability. The guardian
|
|
128
|
+
owns managed child processes, restart policy, lifecycle hooks, and exit events.
|
|
129
|
+
After an unexpected daemon exit, an exact config/runtime replacement claims the
|
|
130
|
+
guardian during `reconnectGraceMs`, restores active and draining releases with
|
|
131
|
+
their allocated ports, and resumes proxy/control ownership. Concurrent matching
|
|
132
|
+
starts are fenced: one claims ownership and losers attest that winner. A config
|
|
133
|
+
identity mismatch or partial state fails closed without rewriting the snapshot.
|
|
134
|
+
Owner disconnection alone never reclaims accepted work or transfers workers:
|
|
135
|
+
guardian-owned processes and their generation-local connections continue during
|
|
136
|
+
the grace, so the replacement reconnects to supervision rather than duplicating
|
|
137
|
+
execution.
|
|
138
|
+
|
|
139
|
+
For a responsive incompatible owner, `ensure-daemon` prepares the requested
|
|
140
|
+
durable runtime, restores exact active and draining generation definitions from
|
|
141
|
+
the authenticated guardian, starts the candidate listeners, and then commits a
|
|
142
|
+
single fenced guardian/control-socket handoff. Config identity, process topology,
|
|
143
|
+
control path, and package/runtime identity may change; `statePath` remains the
|
|
144
|
+
unchanged transaction anchor. Failures before commit leave the old owner serving.
|
|
145
|
+
After commit, drains resume under their original release configs and never block
|
|
146
|
+
the command. A monotonic guardian revision rejects a stale candidate if owner
|
|
147
|
+
state changes after prepare. Listener-owned HTTP/WebSocket counts remain fenced
|
|
148
|
+
to their releases until the retired listener reports them drained, preventing a
|
|
149
|
+
later deploy from stopping the connected process early. Compatible per-deploy
|
|
150
|
+
config reloads remain unchanged.
|
|
151
|
+
|
|
152
|
+
The first upgrade from an authenticated pre-replacement Rollbridge guardian and
|
|
153
|
+
daemon uses an explicitly disruptive compatibility bridge because that legacy
|
|
154
|
+
owner cannot transfer listeners on Node 20. Rollbridge attests the exact guardian
|
|
155
|
+
and daemon processes, sockets, runtime/config authority, and durable process
|
|
156
|
+
registrations before retiring the legacy listeners. Managed process PIDs and
|
|
157
|
+
release state remain supervised, but live proxy/control connections may close.
|
|
158
|
+
The resulting status includes `ownerTransition: {disruptive: true, mode:
|
|
159
|
+
"legacy-first-upgrade", ...}`. The bridge only accepts the incumbent config
|
|
160
|
+
identity; retry config or socket changes after the protocol upgrade, when the
|
|
161
|
+
normal atomic handoff applies. Other guardian/auth/transport/identity failures
|
|
162
|
+
remain fail-closed.
|
|
163
|
+
|
|
164
|
+
Without `ownerRecovery`, `statePath` retains the advisory orphan behavior above.
|
|
165
|
+
|
|
117
166
|
## `legacyTakeover`
|
|
118
167
|
|
|
119
168
|
`legacyTakeover` lets deploy scripts run `rollbridge predeploy-cleanup` during
|
|
@@ -206,10 +255,14 @@ range** so old and new instances can run at the same time:
|
|
|
206
255
|
policy: "service",
|
|
207
256
|
deployStrategy: "handoff",
|
|
208
257
|
command: "npx velocious background-jobs-main",
|
|
258
|
+
lifecycle: {quietCommand: "appctl jobs-main-retire --pid $ROLLBRIDGE_PID"},
|
|
209
259
|
port: {from: 7331, to: 7399}
|
|
210
260
|
}
|
|
211
261
|
```
|
|
212
262
|
|
|
263
|
+
The `appctl` command is illustrative; the application must provide a reviewed
|
|
264
|
+
equivalent that quiesces admission without terminating jobs-main.
|
|
265
|
+
|
|
213
266
|
Reference it from same-release processes with `{{ports.background-jobs-main}}`.
|
|
214
267
|
During a deploy, old workers keep the old port and new workers get the new port.
|
|
215
268
|
For background jobs, the required compliant architecture makes the handoff
|
|
@@ -218,14 +271,14 @@ retirement must quiesce the old jobs-main's scheduling, dispatch, and new worker
|
|
|
218
271
|
handoffs while keeping it with its workers until their accepted work settles.
|
|
219
272
|
Workers are not adopted by the new service.
|
|
220
273
|
|
|
221
|
-
|
|
222
|
-
|
|
223
|
-
|
|
224
|
-
|
|
225
|
-
|
|
226
|
-
|
|
227
|
-
|
|
228
|
-
|
|
274
|
+
Configure `lifecycle.quietCommand` on the handoff service to stop schedules,
|
|
275
|
+
dispatch, and new handoffs without exiting. Immediately after activation,
|
|
276
|
+
Rollbridge quiesces it with `nonBlockingDrain` companions, then returns without
|
|
277
|
+
waiting for their drain. A failed hook leaves the generation alive, records
|
|
278
|
+
`retirementError`, and emits `release retirement quiescence failed`. Durable
|
|
279
|
+
same-authority recovery and guardian-fenced incompatible
|
|
280
|
+
config/control-socket/package/runtime replacement are available with
|
|
281
|
+
`ownerRecovery`.
|
|
229
282
|
|
|
230
283
|
### `processes[].lifecycle`
|
|
231
284
|
|
|
@@ -276,9 +329,9 @@ proxied process (a job worker on a shared queue). Rollbridge starts that
|
|
|
276
329
|
companion's configured stop sequence **as soon as the release is retired**, in
|
|
277
330
|
parallel with the connection drain, rather than after it. Its quiet command or
|
|
278
331
|
signal must make the worker stop accepting new handoffs. The asynchronous
|
|
279
|
-
release drain continues after the deploy response.
|
|
280
|
-
|
|
281
|
-
|
|
332
|
+
release drain continues after the deploy response. Rollbridge simultaneously
|
|
333
|
+
quiesces same-release handoff services, so an old main stops new dispatch while
|
|
334
|
+
remaining available to its old workers.
|
|
282
335
|
|
|
283
336
|
```js
|
|
284
337
|
{id: "worker", policy: "companion", command: "…", nonBlockingDrain: true, stopSignal: "SIGINT", gracefulStopMs: "indefinite"}
|
package/docs/logging.md
CHANGED
|
@@ -40,9 +40,14 @@ When `statePath` is configured, its recovery snapshot is intentionally not a
|
|
|
40
40
|
log archive: process commands, environment mappings, child command lines, and
|
|
41
41
|
captured stdout/stderr are excluded. Use the live APIs above or the configured
|
|
42
42
|
daemon log for those diagnostics, and protect that log according to the
|
|
43
|
-
sensitivity of application output.
|
|
44
|
-
|
|
45
|
-
|
|
43
|
+
sensitivity of application output. With `ownerRecovery`, the mode-`0600`
|
|
44
|
+
snapshot also contains the private guardian capability and its directory must be
|
|
45
|
+
protected accordingly.
|
|
46
|
+
|
|
47
|
+
Without `ownerRecovery`, both in-memory views clear when the daemon restarts.
|
|
48
|
+
With it, guardian-held process output remains available after reconnection while
|
|
49
|
+
the replacement daemon begins a new event history. The configured log file
|
|
50
|
+
persists in either mode.
|
|
46
51
|
|
|
47
52
|
## Rotation
|
|
48
53
|
|
|
@@ -69,12 +69,13 @@ paths, and process references. Do not treat them as generic orphans to force-sto
|
|
|
69
69
|
merely because a supervisor restarted. Cleanup becomes eligible only after the
|
|
70
70
|
last retained process exits.
|
|
71
71
|
|
|
72
|
-
|
|
73
|
-
|
|
74
|
-
|
|
75
|
-
|
|
76
|
-
|
|
77
|
-
generations.
|
|
72
|
+
Rollbridge meets the same-authority daemon-exit portion when `ownerRecovery` is
|
|
73
|
+
configured: its guardian retains provenanced processes and a replacement
|
|
74
|
+
reconstructs concurrent generations, endpoints, and `releaseReferences`.
|
|
75
|
+
With the same `statePath`, `ensure-daemon` also performs guardian-fenced
|
|
76
|
+
incompatible config/control-socket/package/runtime replacement without stopping
|
|
77
|
+
those generations. Without that opt-in, surviving PIDs remain advisory orphans.
|
|
78
|
+
`--takeover-owner` remains a separate destructive migration path.
|
|
78
79
|
|
|
79
80
|
## Operator checks
|
|
80
81
|
|
package/docs/troubleshooting.md
CHANGED
|
@@ -6,18 +6,30 @@
|
|
|
6
6
|
running daemon has a legacy or mismatched runtime and confirms that the deploy
|
|
7
7
|
was not sent.
|
|
8
8
|
|
|
9
|
-
**Cause.** A daemon already owns the stable proxy/control socket, but
|
|
10
|
-
|
|
11
|
-
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
|
|
9
|
+
**Cause.** A daemon already owns the stable proxy/control socket, but Rollbridge
|
|
10
|
+
cannot authenticate and attest an allowed owner transition.
|
|
11
|
+
|
|
12
|
+
**Fix.** With `ownerRecovery`, a genuine pre-owner-replacement Rollbridge daemon
|
|
13
|
+
and guardian can cross the documented one-time disruptive bridge automatically.
|
|
14
|
+
Its existing proxy/control connections may close; successful status reports
|
|
15
|
+
`ownerTransition.mode: "legacy-first-upgrade"`. Keep the incumbent config
|
|
16
|
+
identity unchanged for that first invocation, then apply config/socket changes
|
|
17
|
+
through the now-atomic replacement protocol. Do not treat arbitrary `Unknown
|
|
18
|
+
command`, authentication, transport, malformed-response, or identity errors as
|
|
19
|
+
legacy evidence: those intentionally fail closed and require repairing the
|
|
20
|
+
reported authority or local socket/state problem. Without `ownerRecovery`, plan
|
|
21
|
+
an explicit supervised restart during a safe handoff. If durable runtime
|
|
22
|
+
preparation itself fails, check permissions for
|
|
17
23
|
`--daemon-runtime-path` (default
|
|
18
24
|
`/tmp/rollbridge-<user-id>-<application-hash>-runtime`) before retrying. The directory
|
|
19
25
|
must be private to the invoking user.
|
|
20
26
|
|
|
27
|
+
If startup instead reports `Rollbridge daemon candidate <pid> exited before
|
|
28
|
+
readiness`, the ensuring CLI observed that exact child exit before it could
|
|
29
|
+
attest status. The diagnostic includes the exit code or signal and spawned
|
|
30
|
+
arguments. Inspect the configured `--daemon-log-path` for that PID's startup
|
|
31
|
+
failure; this is distinct from a control-socket readiness timeout.
|
|
32
|
+
|
|
21
33
|
Start with these three commands — they diagnose most problems without guessing:
|
|
22
34
|
|
|
23
35
|
- `rollbridge validate` — config errors, with an example fix for each.
|
|
@@ -123,7 +135,11 @@ end close idle WebSockets on deploy). In the documented compliant jobs topology,
|
|
|
123
135
|
jobs companions use `nonBlockingDrain: true`, so timeout expiry affects only the
|
|
124
136
|
web side and must not stop a still-draining jobs generation.
|
|
125
137
|
|
|
126
|
-
|
|
127
|
-
|
|
128
|
-
|
|
129
|
-
|
|
138
|
+
`status.releaseReferences` reports active and draining releases until full stop;
|
|
139
|
+
Rampway still owns enforcement against on-disk cleanup. With `ownerRecovery`,
|
|
140
|
+
references reconstruct across same-authority daemon process replacement and
|
|
141
|
+
transfer across an incompatible `ensure-daemon` owner handoff. They do not
|
|
142
|
+
transfer through the separate destructive `--takeover-owner` path. If
|
|
143
|
+
`retirementError` is set,
|
|
144
|
+
inspect the quiet-hook events. Rollbridge deliberately leaves that generation
|
|
145
|
+
alive rather than signaling arbitrary PIDs or continuing its stop sequence.
|