rollbridge 0.1.66 → 0.1.67
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/changelog.d/20261115-wedged-generation-escalation.md +44 -0
- package/changelog.d/20261116-bounded-explicit-stop-retirement.md +30 -0
- package/docs/config.md +47 -1
- package/docs/troubleshooting.md +24 -0
- package/docs/workers.md +10 -0
- package/package.json +1 -1
- package/src/config.js +2 -1
- package/src/daemon.js +40 -3
- package/src/drain-policy.js +79 -0
- package/src/managed-process.js +71 -6
- package/src/release-group.js +146 -22
- package/test/config-validation.test.js +5 -4
- package/test/drain-policy.test.js +102 -1
- package/test/managed-process.test.js +205 -1
- package/test/release-group.test.js +196 -0
- package/test/rollbridge.test.js +100 -0
|
@@ -0,0 +1,44 @@
|
|
|
1
|
+
### Added
|
|
2
|
+
|
|
3
|
+
- `releaseRetention.escalateWedgedGenerations` (default **false**) gives the generation
|
|
4
|
+
supervisor a bounded action for a wedge it can already classify, instead of leaving the
|
|
5
|
+
operator to signal PIDs by hand. STOE-1687 measured the gap on TensorBuzz production
|
|
6
|
+
2026-11-15: generation `20261115164249` claimed zero `handed_off` jobs and produced no
|
|
7
|
+
`release drained` record for about four hours until it exited on its own at 20:37:03 UTC.
|
|
8
|
+
While it lingered, `runtime:assert-single-jobs-generation` aborted deploys `20261115202720`
|
|
9
|
+
and `20261115203845` at `Hook after_runtime`, the second deploy's rollback failed too
|
|
10
|
+
(`Restoration of previous release '20261115195051' also failed … incumbent '20261115203845'
|
|
11
|
+
is authoritative`), and three jobs generations were live simultaneously at 20:42 UTC. The
|
|
12
|
+
prepared remedy was SIGTERM against the exact runner PIDs — an operator action, with no
|
|
13
|
+
bounded path to it from the supervisor.
|
|
14
|
+
- The escalation is **not a timer, poller, watchdog, or sweeper**. It runs at two moments that
|
|
15
|
+
already exist: as the last act of `drainAndStop`, bracketing the retirement wait with a
|
|
16
|
+
liveness observation taken when the transition starts and one taken when its stops return
|
|
17
|
+
(the gap is `proxy.nonBlockingDrainTimeoutMs` at minimum, so flatness is provable from the
|
|
18
|
+
transition itself), and at the start of `deploy`, reusing the observation the
|
|
19
|
+
draining-generation admission bound already takes. A generation that stopped with a live
|
|
20
|
+
process group now keeps its observations instead of dropping them, so the next deploy can
|
|
21
|
+
prove the residue without sampling anything new.
|
|
22
|
+
- A `SIGKILL` is delivered only when four proofs hold, each named in the log line: the
|
|
23
|
+
generation is classified `wedged` (no live connection), every live process already expired
|
|
24
|
+
its retirement budget, **no runner child remains** in any tracked process group, and the
|
|
25
|
+
group's CPU is flat across a full 20 000 ms window. The third is the STOE-1077 boundary —
|
|
26
|
+
67 builds and 27.3 build-hours lost in 7 days to a stop-timeout kill reaching live lanes —
|
|
27
|
+
and it is proven independently of the classifier: a wedge that still holds a live runner is
|
|
28
|
+
reported, never killed. `test/managed-process.test.js` "retirement timeout leaves an
|
|
29
|
+
indefinite worker alive without SIGKILL" stays green, and
|
|
30
|
+
`test/release-group.test.js` "a wedged generation that still holds a live runner child is
|
|
31
|
+
reported, never killed" is the control.
|
|
32
|
+
- A wedge the supervisor may not terminate is now loud rather than silent.
|
|
33
|
+
`release wedged generation left alive` names the pids, process ids, and held process groups
|
|
34
|
+
with the exact proof that is missing (`no escalation: 1 runner child is still live in the
|
|
35
|
+
generation; SIGKILL would abort in-flight work`), and
|
|
36
|
+
`release wedged generation escalated` plus `process retirement escalated to SIGKILL` record
|
|
37
|
+
the action taken. `ManagedProcess.escalateRetirement` re-checks its own two invariants before
|
|
38
|
+
signalling anything — the process must already be recorded past its retirement budget, and
|
|
39
|
+
its group must still have a member that can be signalled — and its confirmation wait is
|
|
40
|
+
bounded at 5 000 ms so the escalation cannot become the unbounded wait it replaced.
|
|
41
|
+
- Off by default, deliberately: `gracefulStopMs: "indefinite"` is a consumer's explicit
|
|
42
|
+
statement that a deploy must not kill that process, and procfs cannot distinguish a wedged
|
|
43
|
+
worker from one quietly finishing work that lives inside the process rather than in forked
|
|
44
|
+
runners. Enabling it is the operator's decision, per consumer.
|
|
@@ -0,0 +1,30 @@
|
|
|
1
|
+
### Fixed
|
|
2
|
+
|
|
3
|
+
- An explicit `rollbridge stop` (and the stop inside `rollback`) now gives the processes it stops
|
|
4
|
+
the same retirement bound the deploy path already used, `proxy.nonBlockingDrainTimeoutMs`
|
|
5
|
+
(default **60 000 ms**), and reports what it left alive instead of hanging. Measured on
|
|
6
|
+
TensorBuzz production 2026-11-16 (`shared/log/rollbridge.log`, `tensorbuzz-production-web-1`):
|
|
7
|
+
deploys were refused with `Another owner mutation is already in progress` from 07:35Z to 07:58Z,
|
|
8
|
+
and the `stop` of release `20261115203845` that began at 08:01:29Z was still running 30+ minutes
|
|
9
|
+
later. The phase that never completed is `ManagedProcess.performStop` step 3:
|
|
10
|
+
`waitForProcessGroupExit(pgid, undefined)` for the `background-jobs-worker`, reached from
|
|
11
|
+
`stopRelease` → `ReleaseGroup.stop()` with no options while that process carries
|
|
12
|
+
`gracefulStopMs: "indefinite"`. Because the command runs inside `executeOwnerMutation`, the wait
|
|
13
|
+
held the mutation fence for the whole drain. `ReleaseGroup.stop({retirementTimeoutMs})` now
|
|
14
|
+
bounds a stop for the processes the supervisor must not kill — a `nonBlockingDrain` process and
|
|
15
|
+
any process whose graceful window is `"indefinite"` (`ManagedProcess.hasUnboundedGracefulWindow()`)
|
|
16
|
+
— leaves them running, and logs `release stop gave up waiting for a retiring process` with the
|
|
17
|
+
process id, release id and budget. Shutdown and retired-generation drain are deliberately left
|
|
18
|
+
unbudgeted: those waits own the processes they stop.
|
|
19
|
+
- A retirement budget now **caps** a finite graceful window instead of only standing in for a
|
|
20
|
+
missing one. `performStop` previously used the graceful deadline whenever one existed and took
|
|
21
|
+
the retirement branch only when `gracefulDeadline === undefined`, so after TensorBuzz moved its
|
|
22
|
+
worker from `"indefinite"` to 4,5 h the explicit stop would still have held the control plane for
|
|
23
|
+
4,5 h — and then escalated to SIGKILL, killing the in-flight builds the consumer's
|
|
24
|
+
`gracefulStopMs` exists to protect. Measured against `origin/master` with the new spec: the stop
|
|
25
|
+
of a worker that never exits on its stop signal returns in **31,3 s** (the worker's own delay)
|
|
26
|
+
at base and in **2,9 s** with the fix, with `retirementTimedOut` recorded and no SIGKILL sent.
|
|
27
|
+
- With a `lifecycle.stopCommand`, the graceful window is now measured from the start of the stop
|
|
28
|
+
step rather than from after the hook returns. `hookTimeoutMs()` already bounds the hook by the
|
|
29
|
+
same window, so the step previously allowed hook + window in series (up to 2 × `gracefulStopMs`)
|
|
30
|
+
inside an owner mutation; the whole step is now bounded by the window.
|
package/docs/config.md
CHANGED
|
@@ -79,7 +79,7 @@ to let a deploy group talk to the daemon.
|
|
|
79
79
|
| `proxy.healthPath` | string | `"/ping"` | Default health-check path for proxied processes. |
|
|
80
80
|
| `proxy.healthTimeoutMs` | number | `120000` | Default boot budget for proxied processes: how long a release may take to answer its health path for the first time. Measured on TensorBuzz production (server3, 2026-11-06): the web app was still initialising when a `30000` deadline expired, so two deploys were killed mid-boot; `120000` is the budget the same release publishes on today. Waiting for a `dependsOn` service is not charged to it. |
|
|
81
81
|
| `proxy.drainTimeoutMs` | number | `60000` | How long to drain HTTP/WebSocket connections before stopping the retired proxied process. Expiry never stops an independently draining jobs generation. |
|
|
82
|
-
| `proxy.nonBlockingDrainTimeoutMs` | number | `60000` | How long release retirement waits for an independently draining process before recording it as still draining and completing the release. Expiry never sends `SIGKILL`; the process keeps its live work. |
|
|
82
|
+
| `proxy.nonBlockingDrainTimeoutMs` | number | `60000` | How long release retirement waits for an independently draining process before recording it as still draining and completing the release; `stop` and `rollback` hand the same bound to the processes they must not kill, so neither command can hold the owner-mutation fence for the whole drain. Expiry never sends `SIGKILL`; the process keeps its live work. |
|
|
83
83
|
| `proxy.forceStopTimeoutMs` | number | `10000` | Default per-process graceful-stop timeout (`SIGTERM`, then `SIGKILL`). |
|
|
84
84
|
|
|
85
85
|
## `releaseRetention`
|
|
@@ -89,6 +89,7 @@ to let a deploy group talk to the daemon.
|
|
|
89
89
|
| `releaseRetention.keep` | non-negative integer | `10` | Number of most-recent **stopped** releases the daemon keeps in memory and reports in `status`. |
|
|
90
90
|
| `releaseRetention.maxAgeMs` | non-negative number | `0` (disabled) | Also prune stopped releases older than this many milliseconds. |
|
|
91
91
|
| `releaseRetention.maxConcurrentDrainingReleaseGroups` | non-negative integer | `12` | How many release groups may drain at once. A deploy that would push the count past this bound is **refused before activation** (see below); `0` disables the check. |
|
|
92
|
+
| `releaseRetention.escalateWedgedGenerations` | boolean | `false` | Lets the generation supervisor `SIGKILL` a process group it has **proven** wedged and holding nothing. Off by default: the wedge is reported with its pids instead of terminated. See the escalation below. |
|
|
92
93
|
|
|
93
94
|
Active and draining releases are never pruned. This governs Rollbridge's own
|
|
94
95
|
release records; the deploy tool still owns on-disk release directories.
|
|
@@ -117,6 +118,51 @@ whether each is `progressing` or `wedged` (see [`status` in the CLI
|
|
|
117
118
|
guide](cli.md#status)). A refused deploy is retried once a drain finishes; the bound is a
|
|
118
119
|
policy, not a queue.
|
|
119
120
|
|
|
121
|
+
### The wedged-generation escalation
|
|
122
|
+
|
|
123
|
+
A bounded retirement wait (see [`proxy.nonBlockingDrainTimeoutMs`](#proxy)) makes the release
|
|
124
|
+
finish, but it deliberately leaves the wedged process alive: `gracefulStopMs: "indefinite"`
|
|
125
|
+
exists precisely because a stop-timeout `SIGKILL` there aborts live build lanes (measured in
|
|
126
|
+
STOE-1077: 67 builds, 27.3 build-hours). The residue is what then blocks the fleet — the
|
|
127
|
+
process group still holds the generation, and a consumer that requires one live generation per
|
|
128
|
+
role refuses every later deploy until an operator signals the exact PIDs.
|
|
129
|
+
|
|
130
|
+
`releaseRetention.escalateWedgedGenerations: true` hands that decision back to the supervisor.
|
|
131
|
+
It is **off by default**, because only the operator knows whether their worker's live work sits
|
|
132
|
+
in forked runners (kill-safe to prove) or inside the process itself (not provable from procfs).
|
|
133
|
+
|
|
134
|
+
The escalation is not a timer, a poller, or a watchdog. It runs at two moments that already
|
|
135
|
+
exist: at the end of a retirement transition, bracketing the wait that transition already
|
|
136
|
+
performed, and at the start of a `deploy`, reusing the liveness observation the admission bound
|
|
137
|
+
already takes. A generation that stopped with a live process group keeps its observations, so
|
|
138
|
+
the next deploy can prove the residue without sampling anything new.
|
|
139
|
+
|
|
140
|
+
A `SIGKILL` is delivered only when **all four** proofs hold, each named in the log line:
|
|
141
|
+
|
|
142
|
+
1. The generation is classified `wedged` — it owns no live connection.
|
|
143
|
+
2. Every live process already expired its retirement budget.
|
|
144
|
+
3. **No runner child remains** in any tracked process group. This is the STOE-1077 boundary: a
|
|
145
|
+
live lane is never aborted, and a wedge that still holds one is reported instead of killed.
|
|
146
|
+
4. The group's CPU is **flat across a full 20 s sample window**, so a group that is merely slow
|
|
147
|
+
is never mistaken for a wedged one.
|
|
148
|
+
|
|
149
|
+
Anything else reports. A refused wedge is logged with the pids and process groups it is holding
|
|
150
|
+
— the information an operator needs in place of guessing — and the deploy proceeds to the
|
|
151
|
+
admission bound above:
|
|
152
|
+
|
|
153
|
+
```text
|
|
154
|
+
release wedged generation left alive {escalation: refused, pids: [4242], processIds: ["worker"],
|
|
155
|
+
reason: "no escalation: 1 runner child is still live in the generation; SIGKILL would abort
|
|
156
|
+
in-flight work", releaseId: "20261115164249", trigger: "deploy of 20261115202720"}
|
|
157
|
+
```
|
|
158
|
+
|
|
159
|
+
When the proofs do hold, the supervisor terminates the group and says so:
|
|
160
|
+
|
|
161
|
+
```text
|
|
162
|
+
process retirement escalated to SIGKILL {id: "worker", pid: 4242, reason: "release retirement: escalating: …"}
|
|
163
|
+
release wedged generation escalated {escalation: SIGKILL, processIds: ["worker"], releaseId: "20261115164249"}
|
|
164
|
+
```
|
|
165
|
+
|
|
120
166
|
## `statePath`
|
|
121
167
|
|
|
122
168
|
When set, the daemon persists a sanitized operational state snapshot — the active and
|
package/docs/troubleshooting.md
CHANGED
|
@@ -165,6 +165,30 @@ separate destructive `--takeover-owner` path. If
|
|
|
165
165
|
inspect the quiet-hook events. Rollbridge deliberately leaves that generation
|
|
166
166
|
alive rather than signaling arbitrary PIDs or continuing its stop sequence.
|
|
167
167
|
|
|
168
|
+
**A drain recorded but a process group is still alive.** `release drained`
|
|
169
|
+
appears, the release reports `state: "stopped"`, yet `processes[].pid` or
|
|
170
|
+
`processes[].heldProcessGroup` is still reported and later deploys keep naming
|
|
171
|
+
this generation `wedged`. That is a wedged non-blocking drain: its retirement
|
|
172
|
+
budget expired, the release finished retiring, and the process group it left
|
|
173
|
+
behind still holds the generation.
|
|
174
|
+
|
|
175
|
+
**Diagnose.** Read `drain.reason` for the classification and the evidence behind
|
|
176
|
+
it, and `heldProcessGroup: {liveMemberCount, pgid}` for the group that outlived
|
|
177
|
+
its tracked process. `escalation: refused` in a
|
|
178
|
+
`release wedged generation left alive` line names why the supervisor declined to
|
|
179
|
+
terminate it — usually a runner child that is still live, which is exactly the
|
|
180
|
+
case a `SIGKILL` must never reach.
|
|
181
|
+
|
|
182
|
+
**Fix.** With `releaseRetention.escalateWedgedGenerations: true`, the next
|
|
183
|
+
deploy terminates the group when all four proofs hold (no live connection, every
|
|
184
|
+
retirement budget expired, no runner child remaining, CPU flat across a full
|
|
185
|
+
20 000 ms window) and logs `release wedged generation escalated`. It does nothing
|
|
186
|
+
when any proof is missing, so enabling it does not turn a slow drain into a kill.
|
|
187
|
+
Without it, the log line is the deliverable: it names the pids and process groups
|
|
188
|
+
to signal by hand. A wedge that persists across many deploys with live runner
|
|
189
|
+
children is a drain that cannot finish, not a supervisor fault — see
|
|
190
|
+
[`docs/config.md`](config.md#the-wedged-generation-escalation).
|
|
191
|
+
|
|
168
192
|
## A command is refused with "Another owner mutation is already in progress"
|
|
169
193
|
|
|
170
194
|
**Symptom.** `deploy`, `stop`, `restart`, `rollback` or
|
package/docs/workers.md
CHANGED
|
@@ -106,5 +106,15 @@ Returned work may then run in the active generation. Normal retirement does not
|
|
|
106
106
|
simulate a disconnect and does not make a new jobs-main adopt the old worker's
|
|
107
107
|
handoffs.
|
|
108
108
|
|
|
109
|
+
The supervisor's answer to a drain that cannot finish is separate from these
|
|
110
|
+
timeouts. `releaseRetention.escalateWedgedGenerations` (off by default) lets the
|
|
111
|
+
generation supervisor terminate a process group it has proven wedged and holding
|
|
112
|
+
nothing — no live connection, every retirement budget expired, no runner child
|
|
113
|
+
remaining, CPU flat across a full sample window — at the end of a retirement
|
|
114
|
+
transition or at the next deploy. It never waits for a drain and never kills a
|
|
115
|
+
process that still holds a live runner; a wedge it may not terminate is reported
|
|
116
|
+
with its pids instead. See
|
|
117
|
+
[`docs/config.md`](config.md#the-wedged-generation-escalation).
|
|
118
|
+
|
|
109
119
|
See [`docs/velocious.md`](velocious.md) for the complete topology and
|
|
110
120
|
[`docs/config.md`](config.md#processesdeploystrategy) for handoff-service fields.
|
package/package.json
CHANGED
package/src/config.js
CHANGED
|
@@ -23,7 +23,7 @@ import {processStartOrder} from "./process-order.js"
|
|
|
23
23
|
* @typedef {{drainTimeoutMs: number, forceStopTimeoutMs: number, healthPath: string, healthTimeoutMs: number, host: string, nonBlockingDrainTimeoutMs: number, port: number, upstreamHost: string}} ProxyConfig
|
|
24
24
|
* @typedef {{includes: string[], name: string}} LegacyTakeoverProcessConfig
|
|
25
25
|
* @typedef {{forceStopTimeoutMs: number, processes: LegacyTakeoverProcessConfig[], screens: string[]}} LegacyTakeoverConfig
|
|
26
|
-
* @typedef {{keep: number, maxAgeMs: number, maxConcurrentDrainingReleaseGroups: number}} ReleaseRetentionConfig
|
|
26
|
+
* @typedef {{escalateWedgedGenerations: boolean, keep: number, maxAgeMs: number, maxConcurrentDrainingReleaseGroups: number}} ReleaseRetentionConfig
|
|
27
27
|
* @typedef {{reconnectGraceMs: number}} OwnerRecoveryConfig
|
|
28
28
|
* @typedef {{command: string | undefined, timeoutMs: number}} ComposeDrainConfig
|
|
29
29
|
* @typedef {{command: string | undefined, intervalMs: number, timeoutMs: number}} ComposeInFlightConfig
|
|
@@ -888,6 +888,7 @@ function normalizeReleaseRetention(source, issues) {
|
|
|
888
888
|
const maxConcurrentDrainingReleaseGroups = normalizeNumber(source.maxConcurrentDrainingReleaseGroups, "releaseRetention.maxConcurrentDrainingReleaseGroups", issues, {default: DEFAULT_MAX_CONCURRENT_DRAINING_RELEASE_GROUPS})
|
|
889
889
|
|
|
890
890
|
return {
|
|
891
|
+
escalateWedgedGenerations: normalizeBoolean(source.escalateWedgedGenerations, "releaseRetention.escalateWedgedGenerations", issues, false),
|
|
891
892
|
keep: nonNegativeOrDefault(keep, "releaseRetention.keep", issues, 10, true),
|
|
892
893
|
maxAgeMs: nonNegativeOrDefault(maxAgeMs, "releaseRetention.maxAgeMs", issues, 0, false),
|
|
893
894
|
maxConcurrentDrainingReleaseGroups: nonNegativeOrDefault(maxConcurrentDrainingReleaseGroups, "releaseRetention.maxConcurrentDrainingReleaseGroups", issues, DEFAULT_MAX_CONCURRENT_DRAINING_RELEASE_GROUPS, true)
|
package/src/daemon.js
CHANGED
|
@@ -1634,6 +1634,7 @@ export default class RollbridgeDaemon {
|
|
|
1634
1634
|
await this.updateGenerationTransition("restoring_committed")
|
|
1635
1635
|
return await this.resumeGenerationTransition()
|
|
1636
1636
|
}
|
|
1637
|
+
await this.escalateWedgedGenerations(nextConfig, newReleaseId)
|
|
1637
1638
|
// A release id that is still retained by a generation holding live work cannot be started
|
|
1638
1639
|
// a second time: the candidate's processes re-register the guardian keys that the retained
|
|
1639
1640
|
// generation's own processes still hold, so the deploy aborts partway through `start()`
|
|
@@ -1643,6 +1644,11 @@ export default class RollbridgeDaemon {
|
|
|
1643
1644
|
// release the guardian reports as active, and leaves the retained generation intact for
|
|
1644
1645
|
// `rollbridge status` to reconcile against. Journaled transition resume above is exempt:
|
|
1645
1646
|
// it re-drives the transaction that already owns this release id.
|
|
1647
|
+
//
|
|
1648
|
+
// The escalation above runs first on purpose: it is what clears the residue that made this
|
|
1649
|
+
// guard refuse the measured 2026-11-15 and 2026-11-16 deploys. A wedge that escalation can
|
|
1650
|
+
// prove is kill-safe stops holding its keys and the handoff proceeds; a generation that
|
|
1651
|
+
// still holds live work is not killed and is named here instead of aborting mid-`start()`.
|
|
1646
1652
|
const heldGeneration = this.retainedGenerationHoldingLiveWork(newReleaseId)
|
|
1647
1653
|
|
|
1648
1654
|
if (heldGeneration) {
|
|
@@ -1771,6 +1777,32 @@ export default class RollbridgeDaemon {
|
|
|
1771
1777
|
return heldGenerationHold(retained)
|
|
1772
1778
|
}
|
|
1773
1779
|
|
|
1780
|
+
/**
|
|
1781
|
+
* Gives every retiring generation the escalation decision its own retirement left open,
|
|
1782
|
+
* before the handoff is judged.
|
|
1783
|
+
*
|
|
1784
|
+
* A deploy is the event that makes a wedged generation expensive: until something tries to
|
|
1785
|
+
* activate a successor, a retired release holding a live process group costs nothing
|
|
1786
|
+
* observable. Acting here means the observation that proves the wedge and the action on it
|
|
1787
|
+
* come from the same read the admission bound already performs, so no sampler has to run
|
|
1788
|
+
* alongside the daemon. Generations that already recorded `release drained` are included:
|
|
1789
|
+
* their retirement finished, and the process group they left behind is the residue that
|
|
1790
|
+
* blocked the measured 2026-11-15 deploys.
|
|
1791
|
+
* @param {import("./config.js").RollbridgeConfig} config - Config governing the handoff.
|
|
1792
|
+
* @param {string} candidateReleaseId - Release the handoff would activate.
|
|
1793
|
+
* @returns {Promise<void>} Resolves once every retiring generation has been decided.
|
|
1794
|
+
*/
|
|
1795
|
+
async escalateWedgedGenerations(config, candidateReleaseId) {
|
|
1796
|
+
if (!config.releaseRetention.escalateWedgedGenerations) return
|
|
1797
|
+
|
|
1798
|
+
for (const release of this.releases.values()) {
|
|
1799
|
+
if (release.releaseId === candidateReleaseId) continue
|
|
1800
|
+
if (release.state !== "draining" && release.state !== "stopped" && release.state !== "failed") continue
|
|
1801
|
+
|
|
1802
|
+
await release.escalateWedgedGeneration({reason: `deploy of ${candidateReleaseId}`})
|
|
1803
|
+
}
|
|
1804
|
+
}
|
|
1805
|
+
|
|
1774
1806
|
/**
|
|
1775
1807
|
* Continues one exact durable generation transition without an internal retry loop.
|
|
1776
1808
|
* @returns {Promise<Record<string, JsonValue>>} Deploy result after commit.
|
|
@@ -2519,8 +2551,10 @@ export default class RollbridgeDaemon {
|
|
|
2519
2551
|
// The target may still be draining a prior deploy (live processes). Stop it before the
|
|
2520
2552
|
// deploy below re-uses its id in this.releases, otherwise the still-running instance
|
|
2521
2553
|
// would be dropped from status/pruning/shutdown and could be orphaned.
|
|
2554
|
+
// Rollback is an owner mutation too, so this stop carries the same retirement bound as
|
|
2555
|
+
// `stop`: a process the supervisor must not kill must not be able to hold the fence.
|
|
2522
2556
|
if (target.state !== "stopped" && target.state !== "failed") {
|
|
2523
|
-
await target.stop()
|
|
2557
|
+
await target.stop({retirementTimeoutMs: this.config.proxy.nonBlockingDrainTimeoutMs})
|
|
2524
2558
|
}
|
|
2525
2559
|
|
|
2526
2560
|
this.logger("rollback starting", {releaseId: target.releaseId, releasePath: target.releasePath})
|
|
@@ -2741,7 +2775,7 @@ export default class RollbridgeDaemon {
|
|
|
2741
2775
|
|
|
2742
2776
|
/**
|
|
2743
2777
|
* @param {string | undefined} releaseId - Release id, or active release when omitted.
|
|
2744
|
-
* @returns {Promise<void>} Resolves when stopped.
|
|
2778
|
+
* @returns {Promise<void>} Resolves when stopped, or when retirement stops waiting.
|
|
2745
2779
|
*/
|
|
2746
2780
|
async stopRelease(releaseId) {
|
|
2747
2781
|
this.assertNoUnresolvedGenerationTransition("stop")
|
|
@@ -2750,7 +2784,10 @@ export default class RollbridgeDaemon {
|
|
|
2750
2784
|
if (!release) throw new Error(`Release not found: ${releaseId || "active"}`)
|
|
2751
2785
|
if (release === this.activeRelease) this.activeRelease = undefined
|
|
2752
2786
|
|
|
2753
|
-
|
|
2787
|
+
// This command runs inside the owner mutation, so an unbudgeted wait for a process the
|
|
2788
|
+
// supervisor must not kill holds the fence for the whole drain and every later deploy is
|
|
2789
|
+
// refused. The retirement bound is the same one the deploy path already uses.
|
|
2790
|
+
await release.stop({retirementTimeoutMs: this.config.proxy.nonBlockingDrainTimeoutMs})
|
|
2754
2791
|
this.logger("release stopped", {releaseId: release.releaseId})
|
|
2755
2792
|
this.pruneStoppedReleases()
|
|
2756
2793
|
this.persistState()
|
package/src/drain-policy.js
CHANGED
|
@@ -7,6 +7,8 @@
|
|
|
7
7
|
* @typedef {{cpuDeltaMs: number | undefined, reason: string, sampleWindowMs: number, status: DrainProgress}} DrainClassification
|
|
8
8
|
* @typedef {{releaseId: string, reason: string, status: DrainProgress}} DrainGroupReport
|
|
9
9
|
* @typedef {{allowed: boolean, draining: DrainGroupReport[], reason: string | undefined, wouldBeDrainingCount: number}} GenerationAdmission
|
|
10
|
+
* @typedef {{pgid: number, processId: string}} EscalationTarget
|
|
11
|
+
* @typedef {{decision: "escalate" | "report", killSafe: boolean, reason: string, targets: EscalationTarget[]}} EscalationDecision
|
|
10
12
|
*/
|
|
11
13
|
|
|
12
14
|
/**
|
|
@@ -100,6 +102,83 @@ export function admitGenerationRelease({candidateReleaseId, draining, incumbentW
|
|
|
100
102
|
return {allowed: false, draining, reason, wouldBeDrainingCount}
|
|
101
103
|
}
|
|
102
104
|
|
|
105
|
+
/**
|
|
106
|
+
* Decides whether a proven wedge may be terminated by the generation supervisor.
|
|
107
|
+
*
|
|
108
|
+
* A wedge claim is not a kill licence. The claim says a drain cannot progress any more; a
|
|
109
|
+
* SIGKILL is only safe when it also cannot destroy work. That needs four separate proofs: the
|
|
110
|
+
* generation is classified wedged (so it owns no live connection), every live process is
|
|
111
|
+
* already past its retirement budget, no runner child remains in any tracked process group —
|
|
112
|
+
* STOE-1077 measured 67 builds and 27.3 build-hours lost to exactly this signal reaching live
|
|
113
|
+
* lanes — and the group's CPU is proven flat across a full sample window, which rules out a
|
|
114
|
+
* group that is merely slow. Anything else reports, with the missing proof named.
|
|
115
|
+
* @param {object} args - Decision inputs.
|
|
116
|
+
* @param {DrainClassification} args.classification - The classification the wedge claim rests on.
|
|
117
|
+
* @param {DrainLivenessSample} args.currentSample - Newest liveness observation, the one the decision acts on.
|
|
118
|
+
* @param {number} args.minSampleWindowMs - Shortest window that can prove flat CPU.
|
|
119
|
+
* @returns {EscalationDecision} Escalate with named targets, or report the missing proof.
|
|
120
|
+
*/
|
|
121
|
+
export function decideWedgedEscalation({classification, currentSample, minSampleWindowMs}) {
|
|
122
|
+
if (classification.status !== "wedged") {
|
|
123
|
+
return reportEscalation(`no escalation: drain is ${classification.status} (${classification.reason})`)
|
|
124
|
+
}
|
|
125
|
+
|
|
126
|
+
const live = currentSample.processes.filter((process) => process.pid !== undefined || process.heldProcessGroupPgid !== undefined)
|
|
127
|
+
|
|
128
|
+
if (live.length === 0) {
|
|
129
|
+
return reportEscalation("no escalation: the generation holds no live process")
|
|
130
|
+
}
|
|
131
|
+
|
|
132
|
+
const insideBudget = live.filter((process) => !process.retirementTimedOut)
|
|
133
|
+
|
|
134
|
+
if (insideBudget.length > 0) {
|
|
135
|
+
return reportEscalation(`no escalation: process ${insideBudget[0].id} has not expired its retirement budget`)
|
|
136
|
+
}
|
|
137
|
+
|
|
138
|
+
const liveChildCount = live.reduce((total, process) => total + (process.liveChildCount ?? 0), 0)
|
|
139
|
+
|
|
140
|
+
if (liveChildCount > 0) {
|
|
141
|
+
return reportEscalation(`no escalation: ${liveChildCount} runner ${liveChildCount === 1 ? "child is" : "children are"} still live in the generation; SIGKILL would abort in-flight work`)
|
|
142
|
+
}
|
|
143
|
+
|
|
144
|
+
if (classification.sampleWindowMs < minSampleWindowMs) {
|
|
145
|
+
return reportEscalation(`no escalation: liveness window ${classification.sampleWindowMs}ms is below the ${minSampleWindowMs}ms minimum`)
|
|
146
|
+
}
|
|
147
|
+
|
|
148
|
+
if (classification.cpuDeltaMs !== 0) {
|
|
149
|
+
const evidence = classification.cpuDeltaMs === undefined
|
|
150
|
+
? "process CPU is not measurable on this host"
|
|
151
|
+
: `${classification.cpuDeltaMs}ms of CPU moved across ${classification.sampleWindowMs}ms`
|
|
152
|
+
|
|
153
|
+
return reportEscalation(`no escalation: CPU flatness is not proven (${evidence})`)
|
|
154
|
+
}
|
|
155
|
+
|
|
156
|
+
const targets = live.flatMap((process) => {
|
|
157
|
+
const pgid = process.heldProcessGroupPgid ?? process.pid
|
|
158
|
+
|
|
159
|
+
return pgid === undefined ? [] : [{pgid, processId: process.id}]
|
|
160
|
+
})
|
|
161
|
+
|
|
162
|
+
if (targets.length === 0) {
|
|
163
|
+
return reportEscalation("no escalation: no process group id is addressable")
|
|
164
|
+
}
|
|
165
|
+
|
|
166
|
+
return {
|
|
167
|
+
decision: "escalate",
|
|
168
|
+
killSafe: true,
|
|
169
|
+
reason: `escalating: ${live.length} live process${live.length === 1 ? "" : "es"} past its retirement budget, no runner child remaining, CPU flat across ${classification.sampleWindowMs}ms (${classification.reason})`,
|
|
170
|
+
targets
|
|
171
|
+
}
|
|
172
|
+
}
|
|
173
|
+
|
|
174
|
+
/**
|
|
175
|
+
* @param {string} reason - The proof the escalation is missing.
|
|
176
|
+
* @returns {EscalationDecision} Report-only decision.
|
|
177
|
+
*/
|
|
178
|
+
function reportEscalation(reason) {
|
|
179
|
+
return {decision: "report", killSafe: false, reason, targets: []}
|
|
180
|
+
}
|
|
181
|
+
|
|
103
182
|
/**
|
|
104
183
|
* Names the process groups that outlived their tracked process.
|
|
105
184
|
*
|
package/src/managed-process.js
CHANGED
|
@@ -9,6 +9,10 @@ const MAX_BUFFERED_OUTPUT_CHARACTERS = 64 * 1024
|
|
|
9
9
|
// Reading a process group means scanning procfs, and status() is read continuously. A held
|
|
10
10
|
// group is re-verified on this interval instead of on every read.
|
|
11
11
|
const HELD_GROUP_RECHECK_MS = 5000
|
|
12
|
+
// How long an escalated SIGKILL is given to clear the group before the group is recorded as
|
|
13
|
+
// outliving it again. Bounded so the escalation itself can never become the unbounded wait it
|
|
14
|
+
// replaced.
|
|
15
|
+
const ESCALATION_KILL_CONFIRM_MS = 5000
|
|
12
16
|
|
|
13
17
|
/**
|
|
14
18
|
* @typedef {import("./json.js").JsonValue} JsonValue
|
|
@@ -525,20 +529,25 @@ export default class ManagedProcess extends EventEmitter {
|
|
|
525
529
|
if (this.processGroupExists(pgid)) {
|
|
526
530
|
const timeoutMs = options.timeoutMs ?? this.stopTimeoutMs
|
|
527
531
|
const retirementDeadline = options.retirementTimeoutMs === undefined ? undefined : Date.now() + options.retirementTimeoutMs
|
|
528
|
-
|
|
532
|
+
const gracefulDeadline = timeoutMs === "indefinite" ? undefined : Date.now() + timeoutMs
|
|
533
|
+
// A retirement budget is the caller's decision that this process must not be waited for and
|
|
534
|
+
// must not be killed, so it caps the graceful window rather than only standing in for a
|
|
535
|
+
// missing one. Read the other way round, a finite `gracefulStopMs` outvotes the retirement
|
|
536
|
+
// bound and the wait runs for the whole window: TensorBuzz production moved its worker from
|
|
537
|
+
// `"indefinite"` to 4,5 h and the stop would still have held the control plane for 4,5 h.
|
|
538
|
+
const retirementBound = retirementDeadline !== undefined && (gracefulDeadline === undefined || retirementDeadline < gracefulDeadline)
|
|
539
|
+
const waitDeadline = retirementBound ? retirementDeadline : gracefulDeadline
|
|
529
540
|
|
|
530
541
|
if (stopCommand) {
|
|
531
542
|
await this.runHook(stopCommand, hookTimeoutMs, "stop command", pgid)
|
|
532
|
-
gracefulDeadline = timeoutMs === "indefinite" ? undefined : Date.now() + timeoutMs
|
|
533
543
|
} else {
|
|
534
544
|
this.intentionalStopSignal = /** @type {ProcessExitSignal} */ (this.stopSignal)
|
|
535
|
-
|
|
536
|
-
await this.signalProcessGroup(this.stopSignal, pgid, gracefulDeadline ?? retirementDeadline)
|
|
545
|
+
await this.signalProcessGroup(this.stopSignal, pgid, waitDeadline)
|
|
537
546
|
}
|
|
538
547
|
|
|
539
|
-
const exited = await this.waitForProcessGroupExit(pgid,
|
|
548
|
+
const exited = await this.waitForProcessGroupExit(pgid, waitDeadline)
|
|
540
549
|
|
|
541
|
-
if (!exited &&
|
|
550
|
+
if (!exited && retirementBound) {
|
|
542
551
|
this.retirementTimedOut = true
|
|
543
552
|
this.markGroupOutlivedStop(pgid)
|
|
544
553
|
this.logger("process retirement wait timed out; leaving process alive", {id: this.id, pid: pgid, timeoutMs: options.retirementTimeoutMs})
|
|
@@ -560,6 +569,49 @@ export default class ManagedProcess extends EventEmitter {
|
|
|
560
569
|
this.state = "stopped"
|
|
561
570
|
}
|
|
562
571
|
|
|
572
|
+
/**
|
|
573
|
+
* Delivers the SIGKILL that retirement proved it must not wait for.
|
|
574
|
+
*
|
|
575
|
+
* Retirement deliberately leaves a non-blocking process alive: STOE-1077 measured what a
|
|
576
|
+
* stop-timeout kill costs when the process still holds live lanes (67 builds, 27.3
|
|
577
|
+
* build-hours). That is right while the process can still drain and wrong once it cannot — a
|
|
578
|
+
* wedged generation keeps its release open, keeps its port reservation, and makes every
|
|
579
|
+
* later deploy wait on it until an operator signals the exact PIDs by hand. This is the
|
|
580
|
+
* acting half of that decision, and it re-checks the two invariants this class owns before
|
|
581
|
+
* sending anything: the process must already be recorded as past its retirement budget, and
|
|
582
|
+
* its group must still have a member that can be signalled.
|
|
583
|
+
* @param {{reason: string}} options - Named evidence behind the escalation.
|
|
584
|
+
* @returns {Promise<boolean>} True when a live process group was signalled.
|
|
585
|
+
*/
|
|
586
|
+
async escalateRetirement({reason}) {
|
|
587
|
+
const pgid = this.pid ?? this.groupOutlivedStop
|
|
588
|
+
|
|
589
|
+
if (pgid === undefined) return false
|
|
590
|
+
|
|
591
|
+
if (!this.retirementTimedOut) {
|
|
592
|
+
this.logger("retirement escalation refused; process is not past its retirement budget", {id: this.id, pid: pgid})
|
|
593
|
+
|
|
594
|
+
return false
|
|
595
|
+
}
|
|
596
|
+
|
|
597
|
+
if (!this.processGroupExists(pgid)) {
|
|
598
|
+
this.groupOutlivedStop = undefined
|
|
599
|
+
this.heldProcessGroupCount = undefined
|
|
600
|
+
|
|
601
|
+
return false
|
|
602
|
+
}
|
|
603
|
+
|
|
604
|
+
this.logger("process retirement escalated to SIGKILL", {id: this.id, pid: pgid, reason})
|
|
605
|
+
|
|
606
|
+
const killDeadline = Date.now() + ESCALATION_KILL_CONFIRM_MS
|
|
607
|
+
|
|
608
|
+
await this.signalProcessGroup("SIGKILL", pgid, killDeadline)
|
|
609
|
+
|
|
610
|
+
if (!await this.waitForProcessGroupExit(pgid, killDeadline)) this.markGroupOutlivedStop(pgid)
|
|
611
|
+
|
|
612
|
+
return true
|
|
613
|
+
}
|
|
614
|
+
|
|
563
615
|
/** @returns {Promise<void>} Stops restarts and waits until the process has stopped accepting new work. */
|
|
564
616
|
async quiesce() {
|
|
565
617
|
if (this.quiescePromise) return await this.quiescePromise
|
|
@@ -702,6 +754,19 @@ export default class ManagedProcess extends EventEmitter {
|
|
|
702
754
|
return this.stopTimeoutMs
|
|
703
755
|
}
|
|
704
756
|
|
|
757
|
+
/**
|
|
758
|
+
* Reports whether this process's graceful window has no upper bound.
|
|
759
|
+
*
|
|
760
|
+
* `"indefinite"` is the consumer's statement that the supervisor must never kill this process;
|
|
761
|
+
* it is not a statement that a caller may wait for it forever. A stop that has no retirement
|
|
762
|
+
* budget for such a process can only hang, and when that stop runs inside an owner mutation it
|
|
763
|
+
* locks out every later mutation (STOE-2015).
|
|
764
|
+
* @returns {boolean} True when no deadline can ever end the wait.
|
|
765
|
+
*/
|
|
766
|
+
hasUnboundedGracefulWindow() {
|
|
767
|
+
return this.stopTimeoutMs === "indefinite"
|
|
768
|
+
}
|
|
769
|
+
|
|
705
770
|
/**
|
|
706
771
|
* Runs a lifecycle hook command, bounded by a timeout so a hung hook can never block stop().
|
|
707
772
|
* Failures are logged and returned so generation retirement can surface a failed quiet hook;
|