rollbridge 0.1.65 → 0.1.67

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,27 @@
1
+ ### Fixed
2
+
3
+ - `deploy` refuses a `--release-id` it still retains while that generation holds live work,
4
+ before any process of the candidate starts. PR #12 fenced `rollback` only; the restore that
5
+ actually ran in TensorBuzz production was a deploy command naming the drained release id, and
6
+ it failed twice in twelve minutes (`shared/log/rollbridge.log`, server3, 2026-11-15):
7
+ `20:29:51Z traffic switched 20261115195051 → 20261115202720` then `20:29:55Z release startup
8
+ failed` / `deploy failed` with `Guardian provenance mismatch for
9
+ release:20261115195051:background-jobs-main`, and the identical pair at `20:41:21Z` /
10
+ `20:41:26Z` against `20261115203845`. Only one `release drained` landed in that window
11
+ (`20261115164249` at 20:37:03Z), and a 20:54Z process census found all three generations alive
12
+ at once — `20261115195051` (main + worker + 3 runners), `20261115202720` (main + worker + 4
13
+ runners), `20261115203845` (main + worker + 4 runners) — with `20261115195051`'s
14
+ `velocious background-jobs-main` still running as pid 2310504. The provenance key therefore
15
+ belonged to a live generation, so the 2026-10-25 carve-out that lets a dead generation's key be
16
+ re-registered never applied; the collision aborted the deploy after the handoff had already
17
+ moved traffic, leaving the deploy tool's `current` pointer at `releases/20261115195051` while
18
+ the guardian served `20261115203845`. The refusal is decided from the retained record with the
19
+ guardian's own `childRecordIsGone` fence, names the retained state, each held process (a held
20
+ process group included), and the release the guardian reports as active, and starts nothing:
21
+ `Deploy refused before activation: release "20261115195051" is retained in state draining while
22
+ 1 process of that generation still hold live work (…). … The guardian's active release is
23
+ "20261115203845"; deploy a new release id or reconcile the deployment pointer with \`rollbridge
24
+ status\`.` Journaled generation-transition resume is exempt — it re-drives the transaction that
25
+ already owns the release id — and a rollback to a still-draining target still stops that
26
+ generation first, which is what makes its id reusable; the same fence then applies after the
27
+ stop.
@@ -0,0 +1,44 @@
1
+ ### Added
2
+
3
+ - `releaseRetention.escalateWedgedGenerations` (default **false**) gives the generation
4
+ supervisor a bounded action for a wedge it can already classify, instead of leaving the
5
+ operator to signal PIDs by hand. STOE-1687 measured the gap on TensorBuzz production
6
+ 2026-11-15: generation `20261115164249` claimed zero `handed_off` jobs and produced no
7
+ `release drained` record for about four hours until it exited on its own at 20:37:03 UTC.
8
+ While it lingered, `runtime:assert-single-jobs-generation` aborted deploys `20261115202720`
9
+ and `20261115203845` at `Hook after_runtime`, the second deploy's rollback failed too
10
+ (`Restoration of previous release '20261115195051' also failed … incumbent '20261115203845'
11
+ is authoritative`), and three jobs generations were live simultaneously at 20:42 UTC. The
12
+ prepared remedy was SIGTERM against the exact runner PIDs — an operator action, with no
13
+ bounded path to it from the supervisor.
14
+ - The escalation is **not a timer, poller, watchdog, or sweeper**. It runs at two moments that
15
+ already exist: as the last act of `drainAndStop`, bracketing the retirement wait with a
16
+ liveness observation taken when the transition starts and one taken when its stops return
17
+ (the gap is `proxy.nonBlockingDrainTimeoutMs` at minimum, so flatness is provable from the
18
+ transition itself), and at the start of `deploy`, reusing the observation the
19
+ draining-generation admission bound already takes. A generation that stopped with a live
20
+ process group now keeps its observations instead of dropping them, so the next deploy can
21
+ prove the residue without sampling anything new.
22
+ - A `SIGKILL` is delivered only when four proofs hold, each named in the log line: the
23
+ generation is classified `wedged` (no live connection), every live process already expired
24
+ its retirement budget, **no runner child remains** in any tracked process group, and the
25
+ group's CPU is flat across a full 20 000 ms window. The third is the STOE-1077 boundary —
26
+ 67 builds and 27.3 build-hours lost in 7 days to a stop-timeout kill reaching live lanes —
27
+ and it is proven independently of the classifier: a wedge that still holds a live runner is
28
+ reported, never killed. `test/managed-process.test.js` "retirement timeout leaves an
29
+ indefinite worker alive without SIGKILL" stays green, and
30
+ `test/release-group.test.js` "a wedged generation that still holds a live runner child is
31
+ reported, never killed" is the control.
32
+ - A wedge the supervisor may not terminate is now loud rather than silent.
33
+ `release wedged generation left alive` names the pids, process ids, and held process groups
34
+ with the exact proof that is missing (`no escalation: 1 runner child is still live in the
35
+ generation; SIGKILL would abort in-flight work`), and
36
+ `release wedged generation escalated` plus `process retirement escalated to SIGKILL` record
37
+ the action taken. `ManagedProcess.escalateRetirement` re-checks its own two invariants before
38
+ signalling anything — the process must already be recorded past its retirement budget, and
39
+ its group must still have a member that can be signalled — and its confirmation wait is
40
+ bounded at 5 000 ms so the escalation cannot become the unbounded wait it replaced.
41
+ - Off by default, deliberately: `gracefulStopMs: "indefinite"` is a consumer's explicit
42
+ statement that a deploy must not kill that process, and procfs cannot distinguish a wedged
43
+ worker from one quietly finishing work that lives inside the process rather than in forked
44
+ runners. Enabling it is the operator's decision, per consumer.
@@ -0,0 +1,30 @@
1
+ ### Fixed
2
+
3
+ - An explicit `rollbridge stop` (and the stop inside `rollback`) now gives the processes it stops
4
+ the same retirement bound the deploy path already used, `proxy.nonBlockingDrainTimeoutMs`
5
+ (default **60 000 ms**), and reports what it left alive instead of hanging. Measured on
6
+ TensorBuzz production 2026-11-16 (`shared/log/rollbridge.log`, `tensorbuzz-production-web-1`):
7
+ deploys were refused with `Another owner mutation is already in progress` from 07:35Z to 07:58Z,
8
+ and the `stop` of release `20261115203845` that began at 08:01:29Z was still running 30+ minutes
9
+ later. The phase that never completed is `ManagedProcess.performStop` step 3:
10
+ `waitForProcessGroupExit(pgid, undefined)` for the `background-jobs-worker`, reached from
11
+ `stopRelease` → `ReleaseGroup.stop()` with no options while that process carries
12
+ `gracefulStopMs: "indefinite"`. Because the command runs inside `executeOwnerMutation`, the wait
13
+ held the mutation fence for the whole drain. `ReleaseGroup.stop({retirementTimeoutMs})` now
14
+ bounds a stop for the processes the supervisor must not kill — a `nonBlockingDrain` process and
15
+ any process whose graceful window is `"indefinite"` (`ManagedProcess.hasUnboundedGracefulWindow()`)
16
+ — leaves them running, and logs `release stop gave up waiting for a retiring process` with the
17
+ process id, release id and budget. Shutdown and retired-generation drain are deliberately left
18
+ unbudgeted: those waits own the processes they stop.
19
+ - A retirement budget now **caps** a finite graceful window instead of only standing in for a
20
+ missing one. `performStop` previously used the graceful deadline whenever one existed and took
21
+ the retirement branch only when `gracefulDeadline === undefined`, so after TensorBuzz moved its
22
+ worker from `"indefinite"` to 4,5 h the explicit stop would still have held the control plane for
23
+ 4,5 h — and then escalated to SIGKILL, killing the in-flight builds the consumer's
24
+ `gracefulStopMs` exists to protect. Measured against `origin/master` with the new spec: the stop
25
+ of a worker that never exits on its stop signal returns in **31,3 s** (the worker's own delay)
26
+ at base and in **2,9 s** with the fix, with `retirementTimedOut` recorded and no SIGKILL sent.
27
+ - With a `lifecycle.stopCommand`, the graceful window is now measured from the start of the stop
28
+ step rather than from after the hook returns. `hookTimeoutMs()` already bounds the hook by the
29
+ same window, so the step previously allowed hook + window in series (up to 2 × `gracefulStopMs`)
30
+ inside an owner mutation; the whole step is now bounded by the window.
package/docs/cli.md CHANGED
@@ -170,6 +170,18 @@ the incumbent the handoff is about to retire counts too. The incumbent keeps ser
170
170
  nothing of the candidate is started, and the error names every draining group with its
171
171
  `progressing`/`wedged` classification and the bound it exceeded.
172
172
 
173
+ The same pre-activation refusal applies when `--release-id` names a release Rollbridge
174
+ still **retains** while that generation holds live work: draining with processes alive, or
175
+ recorded drained while a process group outlived its tracked process
176
+ (`processes[].heldProcessGroup`). Starting a second generation under that id re-registers
177
+ the guardian process keys the retained generation still holds, so the deploy aborts
178
+ partway through startup with `Guardian provenance mismatch for release:<id>:<key>` — the
179
+ shape a failed-deploy restore produced twice on 2026-11-15 (`20261115195051` at 20:29:55Z
180
+ and again at 20:41:26Z, 12 minutes apart). The refusal names the retained state, each held
181
+ process, and the release the guardian reports as active, and leaves the retained generation
182
+ untouched. A generation whose processes are genuinely gone stays re-deployable: the
183
+ 2026-10-25 carve-out for re-registering a dead generation is unchanged.
184
+
173
185
  With an opt-in handoff-service `lifecycle.activateCommand`, deploy journals the
174
186
  exact transition, waits for old retirement acknowledgement, waits for candidate
175
187
  activation acknowledgement, and synchronously commits the active proxy target.
@@ -260,7 +272,10 @@ release:<id>:<key>` after the deploy has already begun. The refusal is raised
260
272
  before anything is touched and names each held process together with the release
261
273
  the guardian reports as active, so the deploy tool reconciles its own `current`
262
274
  pointer with `rollbridge status` instead of restoring a generation that already
263
- drained.
275
+ drained. A target that is still **draining** is not refused here: stopping it is
276
+ what makes its id reusable, and the deploy it then runs applies the same fence
277
+ after that stop — so a stop that leaves the generation's process group alive is
278
+ reported as the refusal above instead of colliding mid-startup.
264
279
 
265
280
  **Migration constraints.** Rollback only manages processes — it does **not**
266
281
  revert database migrations or other external state. The target release's on-disk
package/docs/config.md CHANGED
@@ -79,7 +79,7 @@ to let a deploy group talk to the daemon.
79
79
  | `proxy.healthPath` | string | `"/ping"` | Default health-check path for proxied processes. |
80
80
  | `proxy.healthTimeoutMs` | number | `120000` | Default boot budget for proxied processes: how long a release may take to answer its health path for the first time. Measured on TensorBuzz production (server3, 2026-11-06): the web app was still initialising when a `30000` deadline expired, so two deploys were killed mid-boot; `120000` is the budget the same release publishes on today. Waiting for a `dependsOn` service is not charged to it. |
81
81
  | `proxy.drainTimeoutMs` | number | `60000` | How long to drain HTTP/WebSocket connections before stopping the retired proxied process. Expiry never stops an independently draining jobs generation. |
82
- | `proxy.nonBlockingDrainTimeoutMs` | number | `60000` | How long release retirement waits for an independently draining process before recording it as still draining and completing the release. Expiry never sends `SIGKILL`; the process keeps its live work. |
82
+ | `proxy.nonBlockingDrainTimeoutMs` | number | `60000` | How long release retirement waits for an independently draining process before recording it as still draining and completing the release; `stop` and `rollback` hand the same bound to the processes they must not kill, so neither command can hold the owner-mutation fence for the whole drain. Expiry never sends `SIGKILL`; the process keeps its live work. |
83
83
  | `proxy.forceStopTimeoutMs` | number | `10000` | Default per-process graceful-stop timeout (`SIGTERM`, then `SIGKILL`). |
84
84
 
85
85
  ## `releaseRetention`
@@ -89,6 +89,7 @@ to let a deploy group talk to the daemon.
89
89
  | `releaseRetention.keep` | non-negative integer | `10` | Number of most-recent **stopped** releases the daemon keeps in memory and reports in `status`. |
90
90
  | `releaseRetention.maxAgeMs` | non-negative number | `0` (disabled) | Also prune stopped releases older than this many milliseconds. |
91
91
  | `releaseRetention.maxConcurrentDrainingReleaseGroups` | non-negative integer | `12` | How many release groups may drain at once. A deploy that would push the count past this bound is **refused before activation** (see below); `0` disables the check. |
92
+ | `releaseRetention.escalateWedgedGenerations` | boolean | `false` | Lets the generation supervisor `SIGKILL` a process group it has **proven** wedged and holding nothing. Off by default: the wedge is reported with its pids instead of terminated. See the escalation below. |
92
93
 
93
94
  Active and draining releases are never pruned. This governs Rollbridge's own
94
95
  release records; the deploy tool still owns on-disk release directories.
@@ -117,6 +118,51 @@ whether each is `progressing` or `wedged` (see [`status` in the CLI
117
118
  guide](cli.md#status)). A refused deploy is retried once a drain finishes; the bound is a
118
119
  policy, not a queue.
119
120
 
121
+ ### The wedged-generation escalation
122
+
123
+ A bounded retirement wait (see [`proxy.nonBlockingDrainTimeoutMs`](#proxy)) makes the release
124
+ finish, but it deliberately leaves the wedged process alive: `gracefulStopMs: "indefinite"`
125
+ exists precisely because a stop-timeout `SIGKILL` there aborts live build lanes (measured in
126
+ STOE-1077: 67 builds, 27.3 build-hours). The residue is what then blocks the fleet — the
127
+ process group still holds the generation, and a consumer that requires one live generation per
128
+ role refuses every later deploy until an operator signals the exact PIDs.
129
+
130
+ `releaseRetention.escalateWedgedGenerations: true` hands that decision back to the supervisor.
131
+ It is **off by default**, because only the operator knows whether their worker's live work sits
132
+ in forked runners (kill-safe to prove) or inside the process itself (not provable from procfs).
133
+
134
+ The escalation is not a timer, a poller, or a watchdog. It runs at two moments that already
135
+ exist: at the end of a retirement transition, bracketing the wait that transition already
136
+ performed, and at the start of a `deploy`, reusing the liveness observation the admission bound
137
+ already takes. A generation that stopped with a live process group keeps its observations, so
138
+ the next deploy can prove the residue without sampling anything new.
139
+
140
+ A `SIGKILL` is delivered only when **all four** proofs hold, each named in the log line:
141
+
142
+ 1. The generation is classified `wedged` — it owns no live connection.
143
+ 2. Every live process already expired its retirement budget.
144
+ 3. **No runner child remains** in any tracked process group. This is the STOE-1077 boundary: a
145
+ live lane is never aborted, and a wedge that still holds one is reported instead of killed.
146
+ 4. The group's CPU is **flat across a full 20 s sample window**, so a group that is merely slow
147
+ is never mistaken for a wedged one.
148
+
149
+ Anything else reports. A refused wedge is logged with the pids and process groups it is holding
150
+ — the information an operator needs in place of guessing — and the deploy proceeds to the
151
+ admission bound above:
152
+
153
+ ```text
154
+ release wedged generation left alive {escalation: refused, pids: [4242], processIds: ["worker"],
155
+ reason: "no escalation: 1 runner child is still live in the generation; SIGKILL would abort
156
+ in-flight work", releaseId: "20261115164249", trigger: "deploy of 20261115202720"}
157
+ ```
158
+
159
+ When the proofs do hold, the supervisor terminates the group and says so:
160
+
161
+ ```text
162
+ process retirement escalated to SIGKILL {id: "worker", pid: 4242, reason: "release retirement: escalating: …"}
163
+ release wedged generation escalated {escalation: SIGKILL, processIds: ["worker"], releaseId: "20261115164249"}
164
+ ```
165
+
120
166
  ## `statePath`
121
167
 
122
168
  When set, the daemon persists a sanitized operational state snapshot — the active and
@@ -165,6 +165,30 @@ separate destructive `--takeover-owner` path. If
165
165
  inspect the quiet-hook events. Rollbridge deliberately leaves that generation
166
166
  alive rather than signaling arbitrary PIDs or continuing its stop sequence.
167
167
 
168
+ **A drain recorded but a process group is still alive.** `release drained`
169
+ appears, the release reports `state: "stopped"`, yet `processes[].pid` or
170
+ `processes[].heldProcessGroup` is still reported and later deploys keep naming
171
+ this generation `wedged`. That is a wedged non-blocking drain: its retirement
172
+ budget expired, the release finished retiring, and the process group it left
173
+ behind still holds the generation.
174
+
175
+ **Diagnose.** Read `drain.reason` for the classification and the evidence behind
176
+ it, and `heldProcessGroup: {liveMemberCount, pgid}` for the group that outlived
177
+ its tracked process. `escalation: refused` in a
178
+ `release wedged generation left alive` line names why the supervisor declined to
179
+ terminate it — usually a runner child that is still live, which is exactly the
180
+ case a `SIGKILL` must never reach.
181
+
182
+ **Fix.** With `releaseRetention.escalateWedgedGenerations: true`, the next
183
+ deploy terminates the group when all four proofs hold (no live connection, every
184
+ retirement budget expired, no runner child remaining, CPU flat across a full
185
+ 20 000 ms window) and logs `release wedged generation escalated`. It does nothing
186
+ when any proof is missing, so enabling it does not turn a slow drain into a kill.
187
+ Without it, the log line is the deliverable: it names the pids and process groups
188
+ to signal by hand. A wedge that persists across many deploys with live runner
189
+ children is a drain that cannot finish, not a supervisor fault — see
190
+ [`docs/config.md`](config.md#the-wedged-generation-escalation).
191
+
168
192
  ## A command is refused with "Another owner mutation is already in progress"
169
193
 
170
194
  **Symptom.** `deploy`, `stop`, `restart`, `rollback` or
package/docs/workers.md CHANGED
@@ -106,5 +106,15 @@ Returned work may then run in the active generation. Normal retirement does not
106
106
  simulate a disconnect and does not make a new jobs-main adopt the old worker's
107
107
  handoffs.
108
108
 
109
+ The supervisor's answer to a drain that cannot finish is separate from these
110
+ timeouts. `releaseRetention.escalateWedgedGenerations` (off by default) lets the
111
+ generation supervisor terminate a process group it has proven wedged and holding
112
+ nothing — no live connection, every retirement budget expired, no runner child
113
+ remaining, CPU flat across a full sample window — at the end of a retirement
114
+ transition or at the next deploy. It never waits for a drain and never kills a
115
+ process that still holds a live runner; a wedge it may not terminate is reported
116
+ with its pids instead. See
117
+ [`docs/config.md`](config.md#the-wedged-generation-escalation).
118
+
109
119
  See [`docs/velocious.md`](velocious.md) for the complete topology and
110
120
  [`docs/config.md`](config.md#processesdeploystrategy) for handoff-service fields.
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "rollbridge",
3
- "version": "0.1.65",
3
+ "version": "0.1.67",
4
4
  "description": "Zero-downtime process supervisor and local traffic switcher for deploy-managed apps.",
5
5
  "keywords": [
6
6
  "deploy",
package/src/config.js CHANGED
@@ -23,7 +23,7 @@ import {processStartOrder} from "./process-order.js"
23
23
  * @typedef {{drainTimeoutMs: number, forceStopTimeoutMs: number, healthPath: string, healthTimeoutMs: number, host: string, nonBlockingDrainTimeoutMs: number, port: number, upstreamHost: string}} ProxyConfig
24
24
  * @typedef {{includes: string[], name: string}} LegacyTakeoverProcessConfig
25
25
  * @typedef {{forceStopTimeoutMs: number, processes: LegacyTakeoverProcessConfig[], screens: string[]}} LegacyTakeoverConfig
26
- * @typedef {{keep: number, maxAgeMs: number, maxConcurrentDrainingReleaseGroups: number}} ReleaseRetentionConfig
26
+ * @typedef {{escalateWedgedGenerations: boolean, keep: number, maxAgeMs: number, maxConcurrentDrainingReleaseGroups: number}} ReleaseRetentionConfig
27
27
  * @typedef {{reconnectGraceMs: number}} OwnerRecoveryConfig
28
28
  * @typedef {{command: string | undefined, timeoutMs: number}} ComposeDrainConfig
29
29
  * @typedef {{command: string | undefined, intervalMs: number, timeoutMs: number}} ComposeInFlightConfig
@@ -888,6 +888,7 @@ function normalizeReleaseRetention(source, issues) {
888
888
  const maxConcurrentDrainingReleaseGroups = normalizeNumber(source.maxConcurrentDrainingReleaseGroups, "releaseRetention.maxConcurrentDrainingReleaseGroups", issues, {default: DEFAULT_MAX_CONCURRENT_DRAINING_RELEASE_GROUPS})
889
889
 
890
890
  return {
891
+ escalateWedgedGenerations: normalizeBoolean(source.escalateWedgedGenerations, "releaseRetention.escalateWedgedGenerations", issues, false),
891
892
  keep: nonNegativeOrDefault(keep, "releaseRetention.keep", issues, 10, true),
892
893
  maxAgeMs: nonNegativeOrDefault(maxAgeMs, "releaseRetention.maxAgeMs", issues, 0, false),
893
894
  maxConcurrentDrainingReleaseGroups: nonNegativeOrDefault(maxConcurrentDrainingReleaseGroups, "releaseRetention.maxConcurrentDrainingReleaseGroups", issues, DEFAULT_MAX_CONCURRENT_DRAINING_RELEASE_GROUPS, true)
package/src/daemon.js CHANGED
@@ -33,6 +33,7 @@ const STATE_PERSIST_INTERVAL_MS = 5000
33
33
  * @typedef {Omit<import("./release-group.js").ReleaseStatus, "processes"> & {processes: ManagedProcessStatusWithoutLogs[]}} ReleaseStatusWithoutLogs
34
34
  * @typedef {Omit<ProcessStatus, "process"> & {process: ManagedProcessStatusWithoutLogs}} ProcessStatusWithoutLogs
35
35
  * @typedef {Omit<DaemonStatus, "releases" | "services" | "singletons"> & {releases: ReleaseStatusWithoutLogs[], services: ProcessStatusWithoutLogs[], singletons: ProcessStatusWithoutLogs[]}} DaemonStatusWithoutLogs
36
+ * @typedef {{count: number, hold: string[], processes: {heldProcessGroupPgid: number | null, pid: number | null, processId: string, state: import("./managed-process.js").ManagedProcessState}[], state: import("./release-group.js").ReleaseState}} HeldGenerationHold
36
37
  * @typedef {{configDigest: string, format: number, guardian: {pid?: number, socketPath: string, token: string}, reconnectGraceMs: number}} OwnerRecoveryMetadata
37
38
  * @typedef {DaemonStatus & {recovery: OwnerRecoveryMetadata, serviceReleaseIds?: Record<string, string>, singletonReleaseIds?: Record<string, string>}} OwnerRecoverySnapshot
38
39
  * @typedef {{authority: JsonValue, config: import("./config.js").RollbridgeConfig, listenerConnectionSources?: Record<string, Record<string, {http: number, websocket: number}>>, listenerSourceId?: string, recovery?: {command: JsonValue, reconnectGraceMs: number, startupTimeoutMs: number}, releaseConfigs?: Record<string, import("./config.js").RollbridgeConfig>, retiredListenerHandoff?: number, serviceReleaseIds?: Record<string, string>, singletonReleaseIds?: Record<string, string>, snapshot: OwnerRecoverySnapshot}} PrivateOwnerState
@@ -1633,6 +1634,35 @@ export default class RollbridgeDaemon {
1633
1634
  await this.updateGenerationTransition("restoring_committed")
1634
1635
  return await this.resumeGenerationTransition()
1635
1636
  }
1637
+ await this.escalateWedgedGenerations(nextConfig, newReleaseId)
1638
+ // A release id that is still retained by a generation holding live work cannot be started
1639
+ // a second time: the candidate's processes re-register the guardian keys that the retained
1640
+ // generation's own processes still hold, so the deploy aborts partway through `start()`
1641
+ // with a bare `Guardian provenance mismatch` — after the caller has committed to this
1642
+ // revision and, for a restore, after the deployment pointer has already moved. Refusing
1643
+ // from the retained record before any process starts keeps the incumbent serving, names the
1644
+ // release the guardian reports as active, and leaves the retained generation intact for
1645
+ // `rollbridge status` to reconcile against. Journaled transition resume above is exempt:
1646
+ // it re-drives the transaction that already owns this release id.
1647
+ //
1648
+ // The escalation above runs first on purpose: it is what clears the residue that made this
1649
+ // guard refuse the measured 2026-11-15 and 2026-11-16 deploys. A wedge that escalation can
1650
+ // prove is kill-safe stops holding its keys and the handoff proceeds; a generation that
1651
+ // still holds live work is not killed and is named here instead of aborting mid-`start()`.
1652
+ const heldGeneration = this.retainedGenerationHoldingLiveWork(newReleaseId)
1653
+
1654
+ if (heldGeneration) {
1655
+ const activeReleaseId = this.activeRelease?.releaseId ?? null
1656
+
1657
+ this.logger("deploy refused: retained generation still holds live processes", {
1658
+ activeReleaseId,
1659
+ heldProcesses: heldGeneration.processes,
1660
+ releaseId: newReleaseId,
1661
+ retainedState: heldGeneration.state
1662
+ })
1663
+
1664
+ throw new Error(`Deploy refused before activation: release "${newReleaseId}" is retained in state ${heldGeneration.state} while ${heldGeneration.count} process${heldGeneration.count === 1 ? "" : "es"} of that generation still hold live work (${heldGeneration.hold.join("; ")}). Activating that release id again would re-register those processes under a new provenance and abort with a guardian provenance mismatch. The guardian's active release is ${activeReleaseId ? `"${activeReleaseId}"` : "none"}; deploy a new release id or reconcile the deployment pointer with \`rollbridge status\`.`)
1665
+ }
1636
1666
  const admission = this.evaluateGenerationAdmission(newReleaseId, nextConfig)
1637
1667
 
1638
1668
  if (!admission.allowed) {
@@ -1729,6 +1759,50 @@ export default class RollbridgeDaemon {
1729
1759
  })
1730
1760
  }
1731
1761
 
1762
+ /**
1763
+ * Reports whether a retained release group still holds live work that a re-activation would collide with.
1764
+ *
1765
+ * `stopped` is not the same claim as `drained`: the guardian's provenance fence only lets a
1766
+ * process key be re-registered once the record that holds it is gone. A generation recorded
1767
+ * as drained while its process group is still alive — the shape an indefinite worker drain
1768
+ * produces — still holds its keys, so its release id is not available for a restore.
1769
+ * @param {string} releaseId - Release id a handoff would activate.
1770
+ * @returns {HeldGenerationHold | undefined} The live hold, or undefined when the id is free.
1771
+ */
1772
+ retainedGenerationHoldingLiveWork(releaseId) {
1773
+ const retained = this.releases.get(releaseId)
1774
+
1775
+ if (!retained || retained === this.activeRelease) return undefined
1776
+
1777
+ return heldGenerationHold(retained)
1778
+ }
1779
+
1780
+ /**
1781
+ * Gives every retiring generation the escalation decision its own retirement left open,
1782
+ * before the handoff is judged.
1783
+ *
1784
+ * A deploy is the event that makes a wedged generation expensive: until something tries to
1785
+ * activate a successor, a retired release holding a live process group costs nothing
1786
+ * observable. Acting here means the observation that proves the wedge and the action on it
1787
+ * come from the same read the admission bound already performs, so no sampler has to run
1788
+ * alongside the daemon. Generations that already recorded `release drained` are included:
1789
+ * their retirement finished, and the process group they left behind is the residue that
1790
+ * blocked the measured 2026-11-15 deploys.
1791
+ * @param {import("./config.js").RollbridgeConfig} config - Config governing the handoff.
1792
+ * @param {string} candidateReleaseId - Release the handoff would activate.
1793
+ * @returns {Promise<void>} Resolves once every retiring generation has been decided.
1794
+ */
1795
+ async escalateWedgedGenerations(config, candidateReleaseId) {
1796
+ if (!config.releaseRetention.escalateWedgedGenerations) return
1797
+
1798
+ for (const release of this.releases.values()) {
1799
+ if (release.releaseId === candidateReleaseId) continue
1800
+ if (release.state !== "draining" && release.state !== "stopped" && release.state !== "failed") continue
1801
+
1802
+ await release.escalateWedgedGeneration({reason: `deploy of ${candidateReleaseId}`})
1803
+ }
1804
+ }
1805
+
1732
1806
  /**
1733
1807
  * Continues one exact durable generation transition without an internal retry loop.
1734
1808
  * @returns {Promise<Record<string, JsonValue>>} Deploy result after commit.
@@ -2457,43 +2531,30 @@ export default class RollbridgeDaemon {
2457
2531
  // process group that outlived the tracked process and is invisible as a pid — together
2458
2532
  // with the release the guardian actually reports as active, so the deployment tool can
2459
2533
  // reconcile its own current pointer instead of restoring a generation that already drained.
2460
- const heldProcesses = target.state === "stopped"
2461
- ? target.status().processes.filter((processStatus) => !childRecordIsGone(processStatus))
2462
- : []
2534
+ // A target that is still draining is not refused here: stopping it is what makes its id
2535
+ // reusable, and the deploy below applies the same fence after that stop.
2536
+ const heldGeneration = target.state === "stopped" ? heldGenerationHold(target) : undefined
2463
2537
 
2464
- if (heldProcesses.length > 0) {
2538
+ if (heldGeneration) {
2465
2539
  const activeReleaseId = this.activeRelease?.releaseId ?? null
2466
- const held = heldProcesses.map((processStatus) => {
2467
- if (processStatus.heldProcessGroup) {
2468
- const {liveMemberCount, pgid} = processStatus.heldProcessGroup
2469
- const members = liveMemberCount === undefined ? "" : ` with ${liveMemberCount} live member(s)`
2470
-
2471
- return `${processStatus.id}: process group ${pgid}${members} still alive after the tracked process exited`
2472
- }
2473
-
2474
- return `${processStatus.id}: state ${processStatus.state}, pid ${processStatus.pid ?? "unknown"}`
2475
- }).join("; ")
2476
2540
 
2477
2541
  this.logger("rollback refused: drained generation still holds live processes", {
2478
2542
  activeReleaseId,
2479
- heldProcesses: heldProcesses.map((processStatus) => ({
2480
- heldProcessGroupPgid: processStatus.heldProcessGroup?.pgid ?? null,
2481
- pid: processStatus.pid ?? null,
2482
- processId: processStatus.id,
2483
- state: processStatus.state
2484
- })),
2543
+ heldProcesses: heldGeneration.processes,
2485
2544
  releaseId: target.releaseId,
2486
2545
  revision: target.revision
2487
2546
  })
2488
2547
 
2489
- throw new Error(`Release "${target.releaseId}" is recorded as drained while ${heldProcesses.length} process${heldProcesses.length === 1 ? "" : "es"} of that generation still hold live work (${held}). Restoring it would re-register those processes under a new provenance and fail with a guardian provenance mismatch. The guardian's active release is ${activeReleaseId ? `"${activeReleaseId}"` : "none"}; reconcile the deployment pointer with \`rollbridge status\` before restoring.`)
2548
+ throw new Error(`Release "${target.releaseId}" is recorded as drained while ${heldGeneration.count} process${heldGeneration.count === 1 ? "" : "es"} of that generation still hold live work (${heldGeneration.hold.join("; ")}). Restoring it would re-register those processes under a new provenance and fail with a guardian provenance mismatch. The guardian's active release is ${activeReleaseId ? `"${activeReleaseId}"` : "none"}; reconcile the deployment pointer with \`rollbridge status\` before restoring.`)
2490
2549
  }
2491
2550
 
2492
2551
  // The target may still be draining a prior deploy (live processes). Stop it before the
2493
2552
  // deploy below re-uses its id in this.releases, otherwise the still-running instance
2494
2553
  // would be dropped from status/pruning/shutdown and could be orphaned.
2554
+ // Rollback is an owner mutation too, so this stop carries the same retirement bound as
2555
+ // `stop`: a process the supervisor must not kill must not be able to hold the fence.
2495
2556
  if (target.state !== "stopped" && target.state !== "failed") {
2496
- await target.stop()
2557
+ await target.stop({retirementTimeoutMs: this.config.proxy.nonBlockingDrainTimeoutMs})
2497
2558
  }
2498
2559
 
2499
2560
  this.logger("rollback starting", {releaseId: target.releaseId, releasePath: target.releasePath})
@@ -2714,7 +2775,7 @@ export default class RollbridgeDaemon {
2714
2775
 
2715
2776
  /**
2716
2777
  * @param {string | undefined} releaseId - Release id, or active release when omitted.
2717
- * @returns {Promise<void>} Resolves when stopped.
2778
+ * @returns {Promise<void>} Resolves when stopped, or when retirement stops waiting.
2718
2779
  */
2719
2780
  async stopRelease(releaseId) {
2720
2781
  this.assertNoUnresolvedGenerationTransition("stop")
@@ -2723,7 +2784,10 @@ export default class RollbridgeDaemon {
2723
2784
  if (!release) throw new Error(`Release not found: ${releaseId || "active"}`)
2724
2785
  if (release === this.activeRelease) this.activeRelease = undefined
2725
2786
 
2726
- await release.stop()
2787
+ // This command runs inside the owner mutation, so an unbudgeted wait for a process the
2788
+ // supervisor must not kill holds the fence for the whole drain and every later deploy is
2789
+ // refused. The retirement bound is the same one the deploy path already uses.
2790
+ await release.stop({retirementTimeoutMs: this.config.proxy.nonBlockingDrainTimeoutMs})
2727
2791
  this.logger("release stopped", {releaseId: release.releaseId})
2728
2792
  this.pruneStoppedReleases()
2729
2793
  this.persistState()
@@ -3155,6 +3219,42 @@ function statusIncludeLogs(value) {
3155
3219
  return value
3156
3220
  }
3157
3221
 
3222
+ /**
3223
+ * Names the live work a retained release group still holds, using the guardian's own fence.
3224
+ *
3225
+ * A process counts as holding work when the guardian would refuse to hand its key to a new
3226
+ * provenance: a running or stopping process, or a record already `stopped` whose process group
3227
+ * outlived it and is invisible as a pid.
3228
+ * @param {ReleaseGroup} release - Retained generation to inspect.
3229
+ * @returns {HeldGenerationHold | undefined} The hold and its log payload, or undefined when every process is gone.
3230
+ */
3231
+ function heldGenerationHold(release) {
3232
+ const heldProcesses = release.status().processes.filter((processStatus) => !childRecordIsGone(processStatus))
3233
+
3234
+ if (heldProcesses.length === 0) return undefined
3235
+
3236
+ return {
3237
+ count: heldProcesses.length,
3238
+ hold: heldProcesses.map((processStatus) => {
3239
+ if (processStatus.heldProcessGroup) {
3240
+ const {liveMemberCount, pgid} = processStatus.heldProcessGroup
3241
+ const members = liveMemberCount === undefined ? "" : ` with ${liveMemberCount} live member(s)`
3242
+
3243
+ return `${processStatus.id}: process group ${pgid}${members} still alive after the tracked process exited`
3244
+ }
3245
+
3246
+ return `${processStatus.id}: state ${processStatus.state}, pid ${processStatus.pid ?? "unknown"}`
3247
+ }),
3248
+ processes: heldProcesses.map((processStatus) => ({
3249
+ heldProcessGroupPgid: processStatus.heldProcessGroup?.pgid ?? null,
3250
+ pid: processStatus.pid ?? null,
3251
+ processId: processStatus.id,
3252
+ state: processStatus.state
3253
+ })),
3254
+ state: release.state
3255
+ }
3256
+ }
3257
+
3158
3258
  /**
3159
3259
  * @param {PrivateOwnerState} transfer - Authenticated private incumbent state.
3160
3260
  * @param {GenerationTransition} transition - Unresolved generation transition.
@@ -7,6 +7,8 @@
7
7
  * @typedef {{cpuDeltaMs: number | undefined, reason: string, sampleWindowMs: number, status: DrainProgress}} DrainClassification
8
8
  * @typedef {{releaseId: string, reason: string, status: DrainProgress}} DrainGroupReport
9
9
  * @typedef {{allowed: boolean, draining: DrainGroupReport[], reason: string | undefined, wouldBeDrainingCount: number}} GenerationAdmission
10
+ * @typedef {{pgid: number, processId: string}} EscalationTarget
11
+ * @typedef {{decision: "escalate" | "report", killSafe: boolean, reason: string, targets: EscalationTarget[]}} EscalationDecision
10
12
  */
11
13
 
12
14
  /**
@@ -100,6 +102,83 @@ export function admitGenerationRelease({candidateReleaseId, draining, incumbentW
100
102
  return {allowed: false, draining, reason, wouldBeDrainingCount}
101
103
  }
102
104
 
105
+ /**
106
+ * Decides whether a proven wedge may be terminated by the generation supervisor.
107
+ *
108
+ * A wedge claim is not a kill licence. The claim says a drain cannot progress any more; a
109
+ * SIGKILL is only safe when it also cannot destroy work. That needs four separate proofs: the
110
+ * generation is classified wedged (so it owns no live connection), every live process is
111
+ * already past its retirement budget, no runner child remains in any tracked process group —
112
+ * STOE-1077 measured 67 builds and 27.3 build-hours lost to exactly this signal reaching live
113
+ * lanes — and the group's CPU is proven flat across a full sample window, which rules out a
114
+ * group that is merely slow. Anything else reports, with the missing proof named.
115
+ * @param {object} args - Decision inputs.
116
+ * @param {DrainClassification} args.classification - The classification the wedge claim rests on.
117
+ * @param {DrainLivenessSample} args.currentSample - Newest liveness observation, the one the decision acts on.
118
+ * @param {number} args.minSampleWindowMs - Shortest window that can prove flat CPU.
119
+ * @returns {EscalationDecision} Escalate with named targets, or report the missing proof.
120
+ */
121
+ export function decideWedgedEscalation({classification, currentSample, minSampleWindowMs}) {
122
+ if (classification.status !== "wedged") {
123
+ return reportEscalation(`no escalation: drain is ${classification.status} (${classification.reason})`)
124
+ }
125
+
126
+ const live = currentSample.processes.filter((process) => process.pid !== undefined || process.heldProcessGroupPgid !== undefined)
127
+
128
+ if (live.length === 0) {
129
+ return reportEscalation("no escalation: the generation holds no live process")
130
+ }
131
+
132
+ const insideBudget = live.filter((process) => !process.retirementTimedOut)
133
+
134
+ if (insideBudget.length > 0) {
135
+ return reportEscalation(`no escalation: process ${insideBudget[0].id} has not expired its retirement budget`)
136
+ }
137
+
138
+ const liveChildCount = live.reduce((total, process) => total + (process.liveChildCount ?? 0), 0)
139
+
140
+ if (liveChildCount > 0) {
141
+ return reportEscalation(`no escalation: ${liveChildCount} runner ${liveChildCount === 1 ? "child is" : "children are"} still live in the generation; SIGKILL would abort in-flight work`)
142
+ }
143
+
144
+ if (classification.sampleWindowMs < minSampleWindowMs) {
145
+ return reportEscalation(`no escalation: liveness window ${classification.sampleWindowMs}ms is below the ${minSampleWindowMs}ms minimum`)
146
+ }
147
+
148
+ if (classification.cpuDeltaMs !== 0) {
149
+ const evidence = classification.cpuDeltaMs === undefined
150
+ ? "process CPU is not measurable on this host"
151
+ : `${classification.cpuDeltaMs}ms of CPU moved across ${classification.sampleWindowMs}ms`
152
+
153
+ return reportEscalation(`no escalation: CPU flatness is not proven (${evidence})`)
154
+ }
155
+
156
+ const targets = live.flatMap((process) => {
157
+ const pgid = process.heldProcessGroupPgid ?? process.pid
158
+
159
+ return pgid === undefined ? [] : [{pgid, processId: process.id}]
160
+ })
161
+
162
+ if (targets.length === 0) {
163
+ return reportEscalation("no escalation: no process group id is addressable")
164
+ }
165
+
166
+ return {
167
+ decision: "escalate",
168
+ killSafe: true,
169
+ reason: `escalating: ${live.length} live process${live.length === 1 ? "" : "es"} past its retirement budget, no runner child remaining, CPU flat across ${classification.sampleWindowMs}ms (${classification.reason})`,
170
+ targets
171
+ }
172
+ }
173
+
174
+ /**
175
+ * @param {string} reason - The proof the escalation is missing.
176
+ * @returns {EscalationDecision} Report-only decision.
177
+ */
178
+ function reportEscalation(reason) {
179
+ return {decision: "report", killSafe: false, reason, targets: []}
180
+ }
181
+
103
182
  /**
104
183
  * Names the process groups that outlived their tracked process.
105
184
  *