rollbridge 0.1.27 → 0.1.29

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (37) hide show
  1. package/AGENTS.md +14 -0
  2. package/README.md +69 -14
  3. package/TODO.md +5 -2
  4. package/changelog.d/20260828-atomic-owner-replacement.md +22 -0
  5. package/changelog.d/20260828-durable-owner-recovery.md +7 -0
  6. package/changelog.d/20260828-same-owner-jobs-generations.md +6 -0
  7. package/docs/cli.md +43 -21
  8. package/docs/config.md +71 -18
  9. package/docs/logging.md +8 -3
  10. package/docs/tensorbuzz-runbook.md +7 -6
  11. package/docs/troubleshooting.md +28 -12
  12. package/docs/velocious.md +11 -4
  13. package/docs/workers.md +8 -2
  14. package/examples/tensorbuzz.com.js +12 -4
  15. package/package.json +1 -1
  16. package/src/cli.js +209 -36
  17. package/src/config.js +8 -2
  18. package/src/control-client.js +118 -1
  19. package/src/daemon.js +921 -45
  20. package/src/guardian-client.js +429 -0
  21. package/src/managed-process.js +45 -15
  22. package/src/process-guardian.js +588 -0
  23. package/src/release-group.js +190 -15
  24. package/src/state-store.js +1 -1
  25. package/test/config-validation.test.js +22 -0
  26. package/test/fixtures/pre-split3-daemon-runner.js +30 -0
  27. package/test/fixtures/pre-split3-daemon.js +1336 -0
  28. package/test/fixtures/pre-split3-guardian-client.js +293 -0
  29. package/test/fixtures/pre-split3-process-guardian.js +292 -0
  30. package/test/fixtures/service-app.js +32 -2
  31. package/test/guardian-client.test.js +304 -0
  32. package/test/owner-recovery.test.js +858 -0
  33. package/test/owner-replacement.test.js +772 -0
  34. package/test/release-runtime-retention.test.js +1 -1
  35. package/test/rollbridge.test.js +178 -5
  36. package/test/shutdown-completion.test.js +1 -1
  37. package/test/state-store.test.js +12 -0
package/AGENTS.md CHANGED
@@ -40,6 +40,20 @@ implemented. Do not claim production compliance when source/config still uses a
40
40
  fixed jobs-main, worker adoption by a new main, destructive orphan recovery, or
41
41
  synchronous cleanup.
42
42
 
43
+ Current same-authority behavior quiesces configured handoff services after
44
+ candidate activation, retains concurrent generations, reports live release
45
+ references, and can opt into `ownerRecovery` so a durable process guardian
46
+ preserves and reconstructs active/draining generations after daemon process
47
+ exit. With `ownerRecovery` and the same `statePath` transaction anchor,
48
+ `ensure-daemon` also replaces incompatible config, control-socket, package, and
49
+ runtime owners through a guardian-fenced candidate-first handoff while retaining
50
+ active and draining generations. The first authenticated upgrade from a genuine
51
+ pre-replacement guardian/daemon is a documented disruptive compatibility bridge:
52
+ it preserves exact supervised processes and state, but may close proxy/control
53
+ connections. Every protocol-capable replacement after that bridge is atomic.
54
+ `--takeover-owner` remains a destructive
55
+ external-supervisor migration path, not that atomic handoff.
56
+
43
57
  ## Validation and publication
44
58
 
45
59
  The project is ESM JavaScript with JSDoc type checking. Package scripts are the
package/README.md CHANGED
@@ -83,6 +83,7 @@ export default {
83
83
  deployStrategy: "handoff",
84
84
  cwd: "{{releasePath}}",
85
85
  command: "env VELOCIOUS_BACKGROUND_JOBS_PORT={{port}} npx velocious background-jobs-main",
86
+ lifecycle: {quietCommand: "appctl jobs-main-retire --pid $ROLLBRIDGE_PID"},
86
87
  port: {from: 7331, to: 7399}
87
88
  },
88
89
  {
@@ -97,6 +98,9 @@ export default {
97
98
  }
98
99
  ```
99
100
 
101
+ `appctl jobs-main-retire` is illustrative; replace it with a reviewed command
102
+ that quiesces the real jobs-main without exiting it.
103
+
100
104
  Each process retains its most recent stdout/stderr lines and reports them in
101
105
  `status`. Set `outputLines` (a positive integer, default 50) per process to keep
102
106
  more or fewer lines for chatty or quiet processes.
@@ -169,6 +173,13 @@ handoffs as soon as its release retires, independently of the proxied connection
169
173
  drain. Its release-scoped handoff jobs-main remains running to supervise existing
170
174
  handoffs and exits only after the worker pool has drained.
171
175
 
176
+ Give the handoff jobs-main a `lifecycle.quietCommand` that stops schedules,
177
+ dispatch, and new handoffs without terminating the main. After candidate health
178
+ and traffic activation, Rollbridge waits only for this bounded quiescence step,
179
+ then returns while the old main and workers drain together. A failed quiet hook
180
+ is reported as `retirementError`; Rollbridge leaves that generation alive for
181
+ diagnosis instead of silently continuing to stop it.
182
+
172
183
  See [`docs/workers.md`](docs/workers.md) for the full release-generation
173
184
  deployment pattern: a handoff `background-jobs-main`, its companion worker pool,
174
185
  independent quiescence, and durable supervision while retained generations drain.
@@ -184,13 +195,13 @@ owns cleaning up on-disk release directories.
184
195
  releaseRetention: {keep: 5, maxAgeMs: 86400000}
185
196
  ```
186
197
 
187
- Set `statePath` to have the daemon persist secret-safe recovery state to a file
198
+ Set `statePath` to have the daemon persist sanitized recovery state to a file
188
199
  (active/draining releases, process pids, counters, sanitized recent events).
189
200
  Commands, environment mappings, child command lines, and captured process output
190
201
  are deliberately excluded; those diagnostics remain available through the live
191
- status/log/event APIs. On the next startup Rollbridge reads any leftover file and
192
- reports managed processes still alive from a daemon that didn't shut down cleanly
193
- — advisory orphan detection. After a crash, run
202
+ status/log/event APIs. Without `ownerRecovery`, the next startup reads any
203
+ leftover file and reports managed processes still alive from a daemon that
204
+ didn't shut down cleanly — advisory orphan detection. In that mode, after a crash run
194
205
  `rollbridge recover` to list those leftovers and `rollbridge recover --force` to
195
206
  stop them before restarting the daemon. A clean `shutdown` removes the file. See
196
207
  [`docs/config.md`](docs/config.md#statepath).
@@ -199,6 +210,41 @@ stop them before restarting the daemon. A clean `shutdown` removes the file. See
199
210
  statePath: "/var/lib/rollbridge/ticket-server.state.json"
200
211
  ```
201
212
 
213
+ For durable daemon process recovery and atomic owner replacement, opt into
214
+ `ownerRecovery`. Rollbridge
215
+ then runs a private local process guardian which remains the OS supervisor for
216
+ managed processes if the control daemon exits unexpectedly. A replacement using
217
+ the exact same normalized config/runtime reconnects within `reconnectGraceMs`,
218
+ reconstructs active and draining generations and their ports, and fences
219
+ concurrent replacements. `ensure-daemon` can also prepare a requested
220
+ config/control-socket/package/runtime owner, prove it healthy, and atomically
221
+ transfer guardian authority while every retained generation keeps its exact
222
+ release reference and drains asynchronously. The old `statePath` is the durable
223
+ transaction anchor and cannot change during this handoff. The `0600` state file
224
+ contains the guardian capability; protect its directory accordingly.
225
+ Prepared transactions fence owner mutations and compare a monotonic guardian
226
+ state revision at staging. Existing HTTP/WebSocket connections remain owned by
227
+ the retired listener process, while their counts transfer to the new daemon so
228
+ later deploys continue to honor the original drain boundary.
229
+
230
+ There is one explicit compatibility boundary: the first upgrade from a genuine
231
+ pre-owner-replacement Rollbridge guardian and daemon cannot share its listeners
232
+ on supported Node 20. After authenticating the guardian, exact daemon PID/socket,
233
+ runtime authority, and durable owned-process state, `ensure-daemon` performs a
234
+ one-time **disruptive** bridge. Existing proxy/control connections may close,
235
+ the retained processes keep their exact PIDs under guardian supervision, and
236
+ `status.ownerTransition` reports `mode: "legacy-first-upgrade"` with
237
+ `disruptive: true`. The bridge requires the existing config identity unchanged;
238
+ apply config/socket changes in a subsequent invocation, which uses the atomic
239
+ protocol. Unknown commands, auth/transport failures, malformed responses, and
240
+ identity mismatches fail closed without entering this bridge. Every replacement
241
+ after this protocol upgrade remains candidate-first and atomic.
242
+
243
+ ```js
244
+ statePath: "/var/lib/rollbridge/ticket-server.state.json",
245
+ ownerRecovery: {reconnectGraceMs: 30000}
246
+ ```
247
+
202
248
  During the first migration from an old supervisor, set `legacyTakeover` and run
203
249
  `rollbridge predeploy-cleanup --release-path <path>` before `rollbridge deploy`.
204
250
  Rollbridge will only stop configured legacy processes when no reusable active
@@ -241,7 +287,7 @@ rendered when the process starts:
241
287
  Referencing a placeholder with no value (including an unset `{{env.<NAME>}}`)
242
288
  fails the process start with a clear error, so typos surface immediately.
243
289
 
244
- Production-ready examples live in `examples/`, including
290
+ Configuration examples live in `examples/`, including
245
291
  `examples/tensorbuzz.com.js` for the current TensorBuzz backend deployment; see
246
292
  [`docs/tensorbuzz-runbook.md`](docs/tensorbuzz-runbook.md) for the matching
247
293
  production runbook (ports, deploy ordering, rollback constraints, and day-to-day
@@ -338,6 +384,7 @@ candidate coordinator.
338
384
  deployStrategy: "handoff",
339
385
  cwd: "{{releasePath}}",
340
386
  command: "npx velocious background-jobs-main",
387
+ lifecycle: {quietCommand: "appctl jobs-main-retire --pid $ROLLBRIDGE_PID"},
341
388
  port: {from: 7331, to: 7399}
342
389
  }
343
390
  ```
@@ -350,7 +397,7 @@ On `rollbridge deploy`, the required ordering is:
350
397
  2. starts the new release's `companion`s, then its `proxied` process, and
351
398
  health-checks the proxied process;
352
399
  3. switches new traffic to the new release;
353
- 4. marks the previous jobs-main and worker pool retired as one generation;
400
+ 4. quiesces the previous jobs-main and worker pool as one retired generation;
354
401
  5. replaces `singleton`s (stops the old one, then starts the new one);
355
402
  6. returns success without waiting for the previous generation or its independent
356
403
  HTTP/WebSocket drain; Rollbridge supervises all retained drains in the
@@ -359,6 +406,12 @@ On `rollbridge deploy`, the required ordering is:
359
406
  If the new release fails to start or health-check, the previous release stays
360
407
  active and any service started during this deploy is rolled back.
361
408
 
409
+ `status.releaseReferences` lists the id and path of every active or draining
410
+ release. A reference disappears only when that release is fully stopped. With
411
+ `ownerRecovery`, those references and generations survive both same-authority
412
+ daemon recovery and guardian-fenced incompatible config/control-socket/package/
413
+ runtime replacement through `ensure-daemon`.
414
+
362
415
  ## Commands
363
416
 
364
417
  `--config` is optional for every command. When omitted, Rollbridge looks for
@@ -439,19 +492,21 @@ rollbridge daemon --config /srv/ticket-server/rollbridge.js \
439
492
  External supervisors that need to replace a foreground owner without waiting
440
493
  for retained generations to drain add `--takeover-owner`. The candidate starts
441
494
  and health-checks the exact release first. Only then does it retire the accepted
442
- owner's proxy/control listeners. Durable supervision of old jobs-main/worker
443
- generations must be preserved or transferred while the attested replacement
444
- binds the stable listeners; replacement does not mean full synchronous shutdown.
445
- Candidate bootstrap failure leaves the accepted owner untouched. This is opt-in;
446
- ordinary daemon bootstrap and `shutdown` keep their existing behavior.
495
+ owner's proxy/control listeners. Candidate bootstrap failure leaves the accepted
496
+ owner untouched. Current takeover does not preserve retained jobs generations
497
+ or transfer listeners atomically, so it is not a zero-downtime package, config,
498
+ or socket upgrade mechanism. Use `ownerRecovery` plus `ensure-daemon` for the
499
+ guardian-fenced atomic replacement contract. This is opt-in; ordinary daemon
500
+ bootstrap and `shutdown` keep their existing behavior.
447
501
 
448
502
  The four bootstrap inputs are all-or-nothing and use absolute config/release
449
503
  paths. Rollbridge binds its proxy, activates the release through the normal
450
504
  deploy path, then exposes the control socket and stays foreground. A failed
451
505
  activation stops only processes started by that attempt and exits non-zero;
452
- persisted processes from a previous daemon are reported as orphans and are never
453
- recovered or killed implicitly; their live PID records remain in `statePath` for
454
- explicit recovery.
506
+ without `ownerRecovery`, persisted processes from a previous daemon are reported
507
+ as advisory orphans and are never recovered or killed implicitly. With
508
+ `ownerRecovery`, bootstrap reconnects only to the matching guardian and
509
+ reconstructs its provenanced generations before exposing control.
455
510
 
456
511
  External supervisors may add `--boot-attestation` with exactly `sha256:` plus
457
512
  64 lowercase hexadecimal characters. After successful activation, `rollbridge
package/TODO.md CHANGED
@@ -35,7 +35,10 @@ This roadmap tracks planned Rollbridge features and documentation. Rollbridge sh
35
35
  - [x] Support signal-only lifecycle steps for workers that can quiet on a Unix signal. (Per-process `stopSignal`; sent before the `SIGKILL`-after-`gracefulStopMs` fallback.)
36
36
  - [x] Add a non-blocking drain mode so a worker can quiesce at release retirement independently of the HTTP/WebSocket connection drain (`nonBlockingDrain`). This control alone does not provide durable retired-generation supervision.
37
37
  - [x] Document the required Velocious release-generation contract (`docs/velocious.md` and `docs/workers.md`) without treating documentation as proof that the runtime implements it.
38
- - [ ] Implement and verify durable release-scoped jobs-main retirement: owned-handoff supervision, recovery across daemon/host replacement, multiple concurrent retired generations, and release-reference reporting for cleanup pins.
38
+ - [x] Implement same-owner jobs-main retirement, concurrent generations, independent drains, and live release-reference reporting.
39
+ - [x] Implement opt-in durable guardian recovery for exact same-authority daemon process replacement without stopping retained generations.
40
+ - [x] Implement guardian-fenced atomic incompatible owner/config/control-socket/package/runtime replacement without stopping retained generations.
41
+ - [x] Add the authenticated one-time disruptive bridge for pre-replacement guardians; all protocol-capable replacements remain atomic.
39
42
  - [x] Replicas and stable worker indexes. (Supported on port-less `companion` processes; `proxied`/`singleton`/ported processes stay single.)
40
43
  - [x] Allow one process config to start multiple replicas (`replicas`, companion-only for now).
41
44
  - [x] Expose `ROLLBRIDGE_REPLICA_INDEX`, replica count, and per-replica template context (`{{replicaIndex}}`/`{{replicaCount}}`).
@@ -43,7 +46,7 @@ This roadmap tracks planned Rollbridge features and documentation. Rollbridge sh
43
46
  - [x] Preserve readable status output for replica groups (each instance shown as `<id>#<index>`).
44
47
  - [x] Persistent daemon state and recovery.
45
48
  - [x] Persist active release, draining releases, process metadata, counters, and recent events (opt-in `statePath`; atomic snapshot on change + periodic).
46
- - [x] Reconnect status to still-running child processes after daemon restart where possible. (Feasible subset: `status` now includes an `orphans` array still-alive managed processes from the prior daemon's persisted state, re-checked each call. Full re-management/stdout-exit re-attach stays infeasible; the daemon reports them and `rollbridge recover` stops them.)
49
+ - [x] Reconnect to guardian-owned child processes when `ownerRecovery` is enabled; otherwise `status.orphans` retains advisory legacy behavior.
47
50
  - [x] Detect and report orphaned Rollbridge-managed processes. (On startup, reports persisted process pids that are still alive; advisory, see `statePath`.)
48
51
  - [x] Add a recovery mode for safe startup after daemon crash or machine reboot. (`rollbridge recover` lists orphaned processes from the persisted state and, with `--force`, stops them and clears the state; refuses while a daemon is running.)
49
52
  - [x] Rollback support.
@@ -0,0 +1,22 @@
1
+ ### Added
2
+
3
+ - Atomically replace incompatible config, control-socket, package, and runtime
4
+ owners through `ensure-daemon` when durable `ownerRecovery` is configured.
5
+ The guardian fences candidates, transfers exact active and draining release
6
+ supervision only after candidate readiness, and converges after an ambiguous
7
+ commit response or a crash at the committed-state boundary.
8
+ - Bridge the first authenticated pre-replacement guardian/daemon upgrade while
9
+ preserving exact supervised process PIDs and release state. This one-time
10
+ compatibility transition is explicitly disruptive: existing proxy/control
11
+ connections may close and status records `legacy-first-upgrade`. Subsequent
12
+ protocol-capable owner replacements remain atomic, including on Node 20.
13
+ - Fence post-prepare owner-state mutations with a guardian revision check and
14
+ carry incumbent listener connection counts until the old HTTP/WebSocket
15
+ sockets drain, so later deploys cannot stop a still-connected process early.
16
+ - Reserve every live generation's allocated endpoints before its processes bind
17
+ and until that generation truly stops, and reconstruct those reservations on
18
+ owner recovery so overlapping jobs generations cannot reuse an endpoint.
19
+ - Keep an exact newly spawned daemon candidate referenced until its status and
20
+ PID authority are verified. `ensure-daemon` now reports an exact early child
21
+ exit instead of leaving its top-level operation unsettled or waiting for a
22
+ generic readiness timeout; the healthy daemon is detached only afterward.
@@ -0,0 +1,7 @@
1
+ ### Added
2
+
3
+ - Add opt-in `ownerRecovery` with a durable authenticated process guardian so an
4
+ exact same-authority daemon replacement reconstructs active and draining
5
+ generations, ports, lifecycle supervision, and release references after an
6
+ unexpected daemon exit. Concurrent replacements are fenced and mismatched or
7
+ partial recovery state fails closed.
@@ -0,0 +1,6 @@
1
+ ## Fixed
2
+
3
+ - Quiesce release-scoped handoff services after healthy candidate activation,
4
+ retain them with their original workers during asynchronous drain, and report
5
+ active/draining release references. Cross-owner recovery and atomic owner
6
+ upgrades remain future work.
package/docs/cli.md CHANGED
@@ -50,9 +50,9 @@ or number). Supplying only some bootstrap options, or an invalid value, exits
50
50
  non-zero before listeners start. Activation failure emits a structured
51
51
  `bootstrap activation failed` event, cleans up processes owned by that attempt,
52
52
  and exits non-zero without exposing the control socket or inventing an active
53
- release. `statePath` entries from a previous daemon remain advisory orphans:
54
- bootstrap never runs recovery and never signals those processes, and retains
55
- their live PID records in `statePath` for explicit recovery.
53
+ release. With `ownerRecovery`, daemon startup first claims the matching guardian
54
+ and reconstructs its active/draining generations. Without it, `statePath`
55
+ entries remain advisory orphans for explicit recovery.
56
56
 
57
57
  `--boot-attestation` is an optional, non-secret opaque ownership token for an
58
58
  external supervisor. Its canonical format is exactly `sha256:` followed by 64
@@ -73,9 +73,10 @@ preserves nor transfers retained-generation supervision to the replacement.
73
73
  The replacement can bind before those stops finish. A bootstrap failure occurs
74
74
  before retirement, so the previously accepted owner remains available.
75
75
 
76
- A compliant future owner handoff must instead preserve or transfer durable
77
- supervision of retained generations without handing old workers to a new
78
- jobs-main. Current `--takeover-owner` does not provide that behavior.
76
+ `ownerRecovery` covers unexpected process exit under the exact same authority.
77
+ Guardian-fenced incompatible config/package/runtime/control-socket handoff is
78
+ provided by `ensure-daemon`, not `--takeover-owner`; it preserves supervision
79
+ without handing old workers to a new jobs-main.
79
80
 
80
81
  ## `ensure-daemon`
81
82
 
@@ -95,15 +96,29 @@ Before starting a detached daemon, Rollbridge atomically copies its runtime code
95
96
  and production dependency closure into a content-addressed directory outside
96
97
  the invoking release. This keeps the long-lived daemon valid when deploy
97
98
  retention removes that release. A responsive daemon is reused only when its
98
- runtime identity matches the invoking Rollbridge installation; a legacy or
99
- mismatched daemon causes the command to fail before any deploy is sent. Stop and
100
- restart such a daemon explicitly during a safe maintenance handoff.
99
+ runtime and normalized config authority match. With `ownerRecovery` and the same
100
+ `statePath`, an incompatible config/control-socket/package/runtime owner is
101
+ replaced candidate-first: the guardian retains active and draining generations,
102
+ the candidate binds and validates its listeners, and an authenticated fenced
103
+ transaction commits guardian authority and the final control socket. A lost
104
+ control response is accepted only when the guardian confirms the exact committed
105
+ transaction id. The first authenticated upgrade from a genuine pre-replacement
106
+ guardian/daemon is the sole exception: `ensure-daemon` preserves its exact
107
+ guardian-owned processes but deliberately retires the old listeners before the
108
+ Node 20 candidate binds, so existing proxy/control connections may close.
109
+ Successful status JSON records this as `ownerTransition.disruptive: true` and
110
+ `ownerTransition.mode: "legacy-first-upgrade"`. This one-time bridge requires
111
+ the incumbent config identity unchanged; make config/socket changes in a second,
112
+ atomic invocation. Auth, transport, malformed-response, arbitrary unknown-command,
113
+ and authority failures do not qualify and fail before any deploy is sent.
101
114
 
102
115
  - `--daemon-log-path <path>` — file the detached daemon's stdout/stderr is
103
116
  appended to. Default: `/tmp/rollbridge-<application>.log`. See
104
117
  [`logging.md`](logging.md) for the log format and rotation guidance.
105
118
  - `--daemon-pid-path <path>` — file the detached daemon's PID is written to.
106
- Default: `/tmp/rollbridge-<application>.pid`.
119
+ Default: `/tmp/rollbridge-<application>.pid`. During replacement, the file
120
+ continues to name the incumbent until the reachable winner reports and
121
+ publishes its exact `daemonPid`.
107
122
  - `--daemon-runtime-path <path>` — parent directory for content-addressed daemon
108
123
  runtime snapshots. Default:
109
124
  `/tmp/rollbridge-<user-id>-<application-hash>-runtime`. The directory must be owned
@@ -134,17 +149,24 @@ for old workers, jobs, or HTTP/WebSocket connections to finish. Prints
134
149
  If the new release fails to start or health-check, the previous release stays
135
150
  active and the command errors.
136
151
 
137
- After candidate activation, `Daemon.deploy()` synchronously waits for singleton
138
- replacement before starting `drainAndPrune` and returning. A replacement failure
139
- can therefore return a non-zero result while the candidate remains active, and a
140
- slow replacement delays both the response and retirement of the old release.
141
-
142
- This is a process-lifetime non-blocking drain, not durable supervision across a
143
- daemon or host restart. It continues across later deploys only while the same
144
- daemon remains alive. After a restart, surviving PIDs from persisted state are
145
- advisory orphans that Rollbridge cannot re-adopt; explicit `recover --force`
146
- stops them. Restart-surviving retained-generation ownership and recovery remain
147
- required future behavior.
152
+ After activation, deploy waits for bounded quiet hooks of old handoff services
153
+ and `nonBlockingDrain` companions, then returns without waiting for their drains.
154
+ A failure is logged and exposed as `retirementError`; the generation stays alive.
155
+ The successful activation response also includes
156
+ `retirement: {status: "quiescence_failed", releaseId, error}` so callers cannot
157
+ mistake the retirement failure for an unqualified transition.
158
+ `status.releaseReferences` lists `{releaseId, releasePath}` for every active or
159
+ draining release and excludes fully stopped history.
160
+
161
+ After candidate activation, `Daemon.deploy()` begins old-generation retirement
162
+ and asynchronous drain before awaiting singleton replacement. A singleton
163
+ replacement failure can therefore return non-zero while the candidate remains
164
+ active, but it cannot leave the old jobs generation dispatching.
165
+
166
+ With `ownerRecovery`, active and draining generations remain guardian-supervised
167
+ across unexpected same-authority daemon exit and reconstruct on replacement;
168
+ they also transfer intact through an incompatible `ensure-daemon` owner handoff.
169
+ Without it, surviving PIDs remain advisory orphans for `recover --force`.
148
170
 
149
171
  Before each deploy, the daemon reloads the config path it was started with.
150
172
  Compatible process and lifecycle changes apply to the new release and govern
package/docs/config.md CHANGED
@@ -46,6 +46,7 @@ restart.
46
46
  | `application` | string | basename of the config file's directory | Names the app; used in the default control-socket path and the `ROLLBRIDGE_APPLICATION` env var. |
47
47
  | `control` | object | — | Control-socket settings (see below). |
48
48
  | `legacyTakeover` | object | unset | Optional matchers for `rollbridge predeploy-cleanup` to stop pre-Rollbridge supervisors during first handover (see below). |
49
+ | `ownerRecovery` | object | unset | Durable guardian recovery and atomic incompatible owner replacement; requires `statePath` (see below). |
49
50
  | `proxy` | object | **required** | Proxy listener and shared defaults (see below). |
50
51
  | `processes` | array | **required** | Managed processes (see below). Exactly one must be `proxied`. |
51
52
  | `releaseRetention` | object | — | How many stopped releases the daemon retains (see below). |
@@ -90,7 +91,7 @@ release records; the deploy tool still owns on-disk release directories.
90
91
 
91
92
  ## `statePath`
92
93
 
93
- When set, the daemon persists a secret-safe state snapshot — the active and
94
+ When set, the daemon persists a sanitized operational state snapshot — the active and
94
95
  draining releases, each managed process's recovery metadata (including pid),
95
96
  restart counters, and recent structured events — to this file (atomically, on
96
97
  changes and every few seconds). Process commands, working directories,
@@ -98,13 +99,13 @@ environment mappings, child command lines, and retained stdout/stderr are never
98
99
  persisted. They remain available from the live `status`, `logs`, and `events`
99
100
  APIs while the daemon is running. On a clean `shutdown` the file is removed.
100
101
 
101
- On the **next startup**, the daemon reads any leftover file and reports managed
102
- processes whose pids are still alive — likely orphans from a daemon that crashed
103
- without shutting down cleanly — in its log and event history, and in the
102
+ Without `ownerRecovery`, the **next startup** reads any leftover file and reports
103
+ managed processes whose pids are still alive — likely orphans from a daemon that
104
+ crashed without shutting down cleanly — in its log/event history and the
104
105
  `orphans` array of [`rollbridge status`](cli.md#status). This is **advisory**:
105
- Rollbridge cannot re-adopt detached children, so it does not stop them
106
- automatically; the operator verifies and stops the leftovers. A recycled pid can
107
- be a false positive, so treat a report as a prompt to investigate. Use
106
+ that mode cannot re-adopt detached children, so it does not stop them
107
+ automatically. A recycled pid can be a false positive, so treat a report as a
108
+ prompt to investigate. Use
108
109
  [`rollbridge recover`](cli.md#recover) to list and (with `--force`) stop those
109
110
  orphans after a crash.
110
111
 
@@ -114,6 +115,54 @@ statePath: "/var/lib/rollbridge/ticket-server.state.json"
114
115
 
115
116
  Leave `statePath` unset to disable persistence (the default).
116
117
 
118
+ Set `ownerRecovery` to opt into same-authority process-exit recovery and atomic
119
+ incompatible owner replacement:
120
+
121
+ ```js
122
+ statePath: "/var/lib/rollbridge/ticket-server.state.json",
123
+ ownerRecovery: {reconnectGraceMs: 30000}
124
+ ```
125
+
126
+ The private guardian socket is derived from `statePath`; the atomic state file is
127
+ written mode `0600` and contains its authentication capability. The guardian
128
+ owns managed child processes, restart policy, lifecycle hooks, and exit events.
129
+ After an unexpected daemon exit, an exact config/runtime replacement claims the
130
+ guardian during `reconnectGraceMs`, restores active and draining releases with
131
+ their allocated ports, and resumes proxy/control ownership. Concurrent matching
132
+ starts are fenced: one claims ownership and losers attest that winner. A config
133
+ identity mismatch or partial state fails closed without rewriting the snapshot.
134
+ Owner disconnection alone never reclaims accepted work or transfers workers:
135
+ guardian-owned processes and their generation-local connections continue during
136
+ the grace, so the replacement reconnects to supervision rather than duplicating
137
+ execution.
138
+
139
+ For a responsive incompatible owner, `ensure-daemon` prepares the requested
140
+ durable runtime, restores exact active and draining generation definitions from
141
+ the authenticated guardian, starts the candidate listeners, and then commits a
142
+ single fenced guardian/control-socket handoff. Config identity, process topology,
143
+ control path, and package/runtime identity may change; `statePath` remains the
144
+ unchanged transaction anchor. Failures before commit leave the old owner serving.
145
+ After commit, drains resume under their original release configs and never block
146
+ the command. A monotonic guardian revision rejects a stale candidate if owner
147
+ state changes after prepare. Listener-owned HTTP/WebSocket counts remain fenced
148
+ to their releases until the retired listener reports them drained, preventing a
149
+ later deploy from stopping the connected process early. Compatible per-deploy
150
+ config reloads remain unchanged.
151
+
152
+ The first upgrade from an authenticated pre-replacement Rollbridge guardian and
153
+ daemon uses an explicitly disruptive compatibility bridge because that legacy
154
+ owner cannot transfer listeners on Node 20. Rollbridge attests the exact guardian
155
+ and daemon processes, sockets, runtime/config authority, and durable process
156
+ registrations before retiring the legacy listeners. Managed process PIDs and
157
+ release state remain supervised, but live proxy/control connections may close.
158
+ The resulting status includes `ownerTransition: {disruptive: true, mode:
159
+ "legacy-first-upgrade", ...}`. The bridge only accepts the incumbent config
160
+ identity; retry config or socket changes after the protocol upgrade, when the
161
+ normal atomic handoff applies. Other guardian/auth/transport/identity failures
162
+ remain fail-closed.
163
+
164
+ Without `ownerRecovery`, `statePath` retains the advisory orphan behavior above.
165
+
117
166
  ## `legacyTakeover`
118
167
 
119
168
  `legacyTakeover` lets deploy scripts run `rollbridge predeploy-cleanup` during
@@ -206,10 +255,14 @@ range** so old and new instances can run at the same time:
206
255
  policy: "service",
207
256
  deployStrategy: "handoff",
208
257
  command: "npx velocious background-jobs-main",
258
+ lifecycle: {quietCommand: "appctl jobs-main-retire --pid $ROLLBRIDGE_PID"},
209
259
  port: {from: 7331, to: 7399}
210
260
  }
211
261
  ```
212
262
 
263
+ The `appctl` command is illustrative; the application must provide a reviewed
264
+ equivalent that quiesces admission without terminating jobs-main.
265
+
213
266
  Reference it from same-release processes with `{{ports.background-jobs-main}}`.
214
267
  During a deploy, old workers keep the old port and new workers get the new port.
215
268
  For background jobs, the required compliant architecture makes the handoff
@@ -218,14 +271,14 @@ retirement must quiesce the old jobs-main's scheduling, dispatch, and new worker
218
271
  handoffs while keeping it with its workers until their accepted work settles.
219
272
  Workers are not adopted by the new service.
220
273
 
221
- That immediate old-main quiescence is **required future compliance behavior**,
222
- not current `deployStrategy: "handoff"` behavior. Today the release group starts
223
- `stop()` for `nonBlockingDrain` companions at retirement but sends no retirement
224
- or quiescence notice to the handoff service. It waits for the connection drain,
225
- stops other dependent processes, waits for the non-blocking companion stops, and
226
- only then stops the handoff service. Consequently, old and new jobs-main
227
- instances can overlap scheduling and dispatch ownership. Do not treat the
228
- configuration above alone as compliance with the background-jobs contract.
274
+ Configure `lifecycle.quietCommand` on the handoff service to stop schedules,
275
+ dispatch, and new handoffs without exiting. Immediately after activation,
276
+ Rollbridge quiesces it with `nonBlockingDrain` companions, then returns without
277
+ waiting for their drain. A failed hook leaves the generation alive, records
278
+ `retirementError`, and emits `release retirement quiescence failed`. Durable
279
+ same-authority recovery and guardian-fenced incompatible
280
+ config/control-socket/package/runtime replacement are available with
281
+ `ownerRecovery`.
229
282
 
230
283
  ### `processes[].lifecycle`
231
284
 
@@ -276,9 +329,9 @@ proxied process (a job worker on a shared queue). Rollbridge starts that
276
329
  companion's configured stop sequence **as soon as the release is retired**, in
277
330
  parallel with the connection drain, rather than after it. Its quiet command or
278
331
  signal must make the worker stop accepting new handoffs. The asynchronous
279
- release drain continues after the deploy response. As described above, current
280
- Rollbridge does not simultaneously quiesce the handoff service, so this setting
281
- alone does not prevent overlapping jobs-main scheduling or dispatch ownership.
332
+ release drain continues after the deploy response. Rollbridge simultaneously
333
+ quiesces same-release handoff services, so an old main stops new dispatch while
334
+ remaining available to its old workers.
282
335
 
283
336
  ```js
284
337
  {id: "worker", policy: "companion", command: "…", nonBlockingDrain: true, stopSignal: "SIGINT", gracefulStopMs: "indefinite"}
package/docs/logging.md CHANGED
@@ -40,9 +40,14 @@ When `statePath` is configured, its recovery snapshot is intentionally not a
40
40
  log archive: process commands, environment mappings, child command lines, and
41
41
  captured stdout/stderr are excluded. Use the live APIs above or the configured
42
42
  daemon log for those diagnostics, and protect that log according to the
43
- sensitivity of application output.
44
-
45
- Both are cleared when the daemon restarts; the log file persists.
43
+ sensitivity of application output. With `ownerRecovery`, the mode-`0600`
44
+ snapshot also contains the private guardian capability and its directory must be
45
+ protected accordingly.
46
+
47
+ Without `ownerRecovery`, both in-memory views clear when the daemon restarts.
48
+ With it, guardian-held process output remains available after reconnection while
49
+ the replacement daemon begins a new event history. The configured log file
50
+ persists in either mode.
46
51
 
47
52
  ## Rotation
48
53
 
@@ -69,12 +69,13 @@ paths, and process references. Do not treat them as generic orphans to force-sto
69
69
  merely because a supervisor restarted. Cleanup becomes eligible only after the
70
70
  last retained process exits.
71
71
 
72
- Current Rollbridge does not yet meet those recovery and owner-handoff
73
- requirements. Its non-blocking release drains last only for the current daemon's
74
- lifetime; after restart it reports surviving PIDs as advisory, non-adoptable
75
- orphans, and forced recovery stops them. `--takeover-owner` quiesces and starts
76
- asynchronous stops for every managed process instead of transferring retained
77
- generations.
72
+ Rollbridge meets the same-authority daemon-exit portion when `ownerRecovery` is
73
+ configured: its guardian retains provenanced processes and a replacement
74
+ reconstructs concurrent generations, endpoints, and `releaseReferences`.
75
+ With the same `statePath`, `ensure-daemon` also performs guardian-fenced
76
+ incompatible config/control-socket/package/runtime replacement without stopping
77
+ those generations. Without that opt-in, surviving PIDs remain advisory orphans.
78
+ `--takeover-owner` remains a separate destructive migration path.
78
79
 
79
80
  ## Operator checks
80
81
 
@@ -6,18 +6,30 @@
6
6
  running daemon has a legacy or mismatched runtime and confirms that the deploy
7
7
  was not sent.
8
8
 
9
- **Cause.** A daemon already owns the stable proxy/control socket, but it cannot
10
- attest to the same immutable Rollbridge runtime as the CLI preparing the deploy.
11
- Rollbridge does not silently restart it because rebinding the proxy could cause
12
- downtime or abandon managed processes.
13
-
14
- **Fix.** Keep the current release active, explicitly stop and restart the daemon
15
- with the intended Rollbridge installation during a safe handoff, then retry the
16
- deploy. If durable runtime preparation itself fails, check permissions for
9
+ **Cause.** A daemon already owns the stable proxy/control socket, but Rollbridge
10
+ cannot authenticate and attest an allowed owner transition.
11
+
12
+ **Fix.** With `ownerRecovery`, a genuine pre-owner-replacement Rollbridge daemon
13
+ and guardian can cross the documented one-time disruptive bridge automatically.
14
+ Its existing proxy/control connections may close; successful status reports
15
+ `ownerTransition.mode: "legacy-first-upgrade"`. Keep the incumbent config
16
+ identity unchanged for that first invocation, then apply config/socket changes
17
+ through the now-atomic replacement protocol. Do not treat arbitrary `Unknown
18
+ command`, authentication, transport, malformed-response, or identity errors as
19
+ legacy evidence: those intentionally fail closed and require repairing the
20
+ reported authority or local socket/state problem. Without `ownerRecovery`, plan
21
+ an explicit supervised restart during a safe handoff. If durable runtime
22
+ preparation itself fails, check permissions for
17
23
  `--daemon-runtime-path` (default
18
24
  `/tmp/rollbridge-<user-id>-<application-hash>-runtime`) before retrying. The directory
19
25
  must be private to the invoking user.
20
26
 
27
+ If startup instead reports `Rollbridge daemon candidate <pid> exited before
28
+ readiness`, the ensuring CLI observed that exact child exit before it could
29
+ attest status. The diagnostic includes the exit code or signal and spawned
30
+ arguments. Inspect the configured `--daemon-log-path` for that PID's startup
31
+ failure; this is distinct from a control-socket readiness timeout.
32
+
21
33
  Start with these three commands — they diagnose most problems without guessing:
22
34
 
23
35
  - `rollbridge validate` — config errors, with an example fix for each.
@@ -123,7 +135,11 @@ end close idle WebSockets on deploy). In the documented compliant jobs topology,
123
135
  jobs companions use `nonBlockingDrain: true`, so timeout expiry affects only the
124
136
  web side and must not stop a still-draining jobs generation.
125
137
 
126
- Reporting release references to Rampway and pinning release directories against
127
- on-disk cleanup are required future behavior, not implemented today. Current
128
- Rollbridge `status` and `releaseRetention` govern only its in-memory release
129
- records and do not fence Rampway cleanup.
138
+ `status.releaseReferences` reports active and draining releases until full stop;
139
+ Rampway still owns enforcement against on-disk cleanup. With `ownerRecovery`,
140
+ references reconstruct across same-authority daemon process replacement and
141
+ transfer across an incompatible `ensure-daemon` owner handoff. They do not
142
+ transfer through the separate destructive `--takeover-owner` path. If
143
+ `retirementError` is set,
144
+ inspect the quiet-hook events. Rollbridge deliberately leaves that generation
145
+ alive rather than signaling arbitrary PIDs or continuing its stop sequence.