rollbridge 0.1.65 → 0.1.66
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
### Fixed
|
|
2
|
+
|
|
3
|
+
- `deploy` refuses a `--release-id` it still retains while that generation holds live work,
|
|
4
|
+
before any process of the candidate starts. PR #12 fenced `rollback` only; the restore that
|
|
5
|
+
actually ran in TensorBuzz production was a deploy command naming the drained release id, and
|
|
6
|
+
it failed twice in twelve minutes (`shared/log/rollbridge.log`, server3, 2026-11-15):
|
|
7
|
+
`20:29:51Z traffic switched 20261115195051 → 20261115202720` then `20:29:55Z release startup
|
|
8
|
+
failed` / `deploy failed` with `Guardian provenance mismatch for
|
|
9
|
+
release:20261115195051:background-jobs-main`, and the identical pair at `20:41:21Z` /
|
|
10
|
+
`20:41:26Z` against `20261115203845`. Only one `release drained` landed in that window
|
|
11
|
+
(`20261115164249` at 20:37:03Z), and a 20:54Z process census found all three generations alive
|
|
12
|
+
at once — `20261115195051` (main + worker + 3 runners), `20261115202720` (main + worker + 4
|
|
13
|
+
runners), `20261115203845` (main + worker + 4 runners) — with `20261115195051`'s
|
|
14
|
+
`velocious background-jobs-main` still running as pid 2310504. The provenance key therefore
|
|
15
|
+
belonged to a live generation, so the 2026-10-25 carve-out that lets a dead generation's key be
|
|
16
|
+
re-registered never applied; the collision aborted the deploy after the handoff had already
|
|
17
|
+
moved traffic, leaving the deploy tool's `current` pointer at `releases/20261115195051` while
|
|
18
|
+
the guardian served `20261115203845`. The refusal is decided from the retained record with the
|
|
19
|
+
guardian's own `childRecordIsGone` fence, names the retained state, each held process (a held
|
|
20
|
+
process group included), and the release the guardian reports as active, and starts nothing:
|
|
21
|
+
`Deploy refused before activation: release "20261115195051" is retained in state draining while
|
|
22
|
+
1 process of that generation still hold live work (…). … The guardian's active release is
|
|
23
|
+
"20261115203845"; deploy a new release id or reconcile the deployment pointer with \`rollbridge
|
|
24
|
+
status\`.` Journaled generation-transition resume is exempt — it re-drives the transaction that
|
|
25
|
+
already owns the release id — and a rollback to a still-draining target still stops that
|
|
26
|
+
generation first, which is what makes its id reusable; the same fence then applies after the
|
|
27
|
+
stop.
|
package/docs/cli.md
CHANGED
|
@@ -170,6 +170,18 @@ the incumbent the handoff is about to retire counts too. The incumbent keeps ser
|
|
|
170
170
|
nothing of the candidate is started, and the error names every draining group with its
|
|
171
171
|
`progressing`/`wedged` classification and the bound it exceeded.
|
|
172
172
|
|
|
173
|
+
The same pre-activation refusal applies when `--release-id` names a release Rollbridge
|
|
174
|
+
still **retains** while that generation holds live work: draining with processes alive, or
|
|
175
|
+
recorded drained while a process group outlived its tracked process
|
|
176
|
+
(`processes[].heldProcessGroup`). Starting a second generation under that id re-registers
|
|
177
|
+
the guardian process keys the retained generation still holds, so the deploy aborts
|
|
178
|
+
partway through startup with `Guardian provenance mismatch for release:<id>:<key>` — the
|
|
179
|
+
shape a failed-deploy restore produced twice on 2026-11-15 (`20261115195051` at 20:29:55Z
|
|
180
|
+
and again at 20:41:26Z, 12 minutes apart). The refusal names the retained state, each held
|
|
181
|
+
process, and the release the guardian reports as active, and leaves the retained generation
|
|
182
|
+
untouched. A generation whose processes are genuinely gone stays re-deployable: the
|
|
183
|
+
2026-10-25 carve-out for re-registering a dead generation is unchanged.
|
|
184
|
+
|
|
173
185
|
With an opt-in handoff-service `lifecycle.activateCommand`, deploy journals the
|
|
174
186
|
exact transition, waits for old retirement acknowledgement, waits for candidate
|
|
175
187
|
activation acknowledgement, and synchronously commits the active proxy target.
|
|
@@ -260,7 +272,10 @@ release:<id>:<key>` after the deploy has already begun. The refusal is raised
|
|
|
260
272
|
before anything is touched and names each held process together with the release
|
|
261
273
|
the guardian reports as active, so the deploy tool reconciles its own `current`
|
|
262
274
|
pointer with `rollbridge status` instead of restoring a generation that already
|
|
263
|
-
drained.
|
|
275
|
+
drained. A target that is still **draining** is not refused here: stopping it is
|
|
276
|
+
what makes its id reusable, and the deploy it then runs applies the same fence
|
|
277
|
+
after that stop — so a stop that leaves the generation's process group alive is
|
|
278
|
+
reported as the refusal above instead of colliding mid-startup.
|
|
264
279
|
|
|
265
280
|
**Migration constraints.** Rollback only manages processes — it does **not**
|
|
266
281
|
revert database migrations or other external state. The target release's on-disk
|
package/package.json
CHANGED
package/src/daemon.js
CHANGED
|
@@ -33,6 +33,7 @@ const STATE_PERSIST_INTERVAL_MS = 5000
|
|
|
33
33
|
* @typedef {Omit<import("./release-group.js").ReleaseStatus, "processes"> & {processes: ManagedProcessStatusWithoutLogs[]}} ReleaseStatusWithoutLogs
|
|
34
34
|
* @typedef {Omit<ProcessStatus, "process"> & {process: ManagedProcessStatusWithoutLogs}} ProcessStatusWithoutLogs
|
|
35
35
|
* @typedef {Omit<DaemonStatus, "releases" | "services" | "singletons"> & {releases: ReleaseStatusWithoutLogs[], services: ProcessStatusWithoutLogs[], singletons: ProcessStatusWithoutLogs[]}} DaemonStatusWithoutLogs
|
|
36
|
+
* @typedef {{count: number, hold: string[], processes: {heldProcessGroupPgid: number | null, pid: number | null, processId: string, state: import("./managed-process.js").ManagedProcessState}[], state: import("./release-group.js").ReleaseState}} HeldGenerationHold
|
|
36
37
|
* @typedef {{configDigest: string, format: number, guardian: {pid?: number, socketPath: string, token: string}, reconnectGraceMs: number}} OwnerRecoveryMetadata
|
|
37
38
|
* @typedef {DaemonStatus & {recovery: OwnerRecoveryMetadata, serviceReleaseIds?: Record<string, string>, singletonReleaseIds?: Record<string, string>}} OwnerRecoverySnapshot
|
|
38
39
|
* @typedef {{authority: JsonValue, config: import("./config.js").RollbridgeConfig, listenerConnectionSources?: Record<string, Record<string, {http: number, websocket: number}>>, listenerSourceId?: string, recovery?: {command: JsonValue, reconnectGraceMs: number, startupTimeoutMs: number}, releaseConfigs?: Record<string, import("./config.js").RollbridgeConfig>, retiredListenerHandoff?: number, serviceReleaseIds?: Record<string, string>, singletonReleaseIds?: Record<string, string>, snapshot: OwnerRecoverySnapshot}} PrivateOwnerState
|
|
@@ -1633,6 +1634,29 @@ export default class RollbridgeDaemon {
|
|
|
1633
1634
|
await this.updateGenerationTransition("restoring_committed")
|
|
1634
1635
|
return await this.resumeGenerationTransition()
|
|
1635
1636
|
}
|
|
1637
|
+
// A release id that is still retained by a generation holding live work cannot be started
|
|
1638
|
+
// a second time: the candidate's processes re-register the guardian keys that the retained
|
|
1639
|
+
// generation's own processes still hold, so the deploy aborts partway through `start()`
|
|
1640
|
+
// with a bare `Guardian provenance mismatch` — after the caller has committed to this
|
|
1641
|
+
// revision and, for a restore, after the deployment pointer has already moved. Refusing
|
|
1642
|
+
// from the retained record before any process starts keeps the incumbent serving, names the
|
|
1643
|
+
// release the guardian reports as active, and leaves the retained generation intact for
|
|
1644
|
+
// `rollbridge status` to reconcile against. Journaled transition resume above is exempt:
|
|
1645
|
+
// it re-drives the transaction that already owns this release id.
|
|
1646
|
+
const heldGeneration = this.retainedGenerationHoldingLiveWork(newReleaseId)
|
|
1647
|
+
|
|
1648
|
+
if (heldGeneration) {
|
|
1649
|
+
const activeReleaseId = this.activeRelease?.releaseId ?? null
|
|
1650
|
+
|
|
1651
|
+
this.logger("deploy refused: retained generation still holds live processes", {
|
|
1652
|
+
activeReleaseId,
|
|
1653
|
+
heldProcesses: heldGeneration.processes,
|
|
1654
|
+
releaseId: newReleaseId,
|
|
1655
|
+
retainedState: heldGeneration.state
|
|
1656
|
+
})
|
|
1657
|
+
|
|
1658
|
+
throw new Error(`Deploy refused before activation: release "${newReleaseId}" is retained in state ${heldGeneration.state} while ${heldGeneration.count} process${heldGeneration.count === 1 ? "" : "es"} of that generation still hold live work (${heldGeneration.hold.join("; ")}). Activating that release id again would re-register those processes under a new provenance and abort with a guardian provenance mismatch. The guardian's active release is ${activeReleaseId ? `"${activeReleaseId}"` : "none"}; deploy a new release id or reconcile the deployment pointer with \`rollbridge status\`.`)
|
|
1659
|
+
}
|
|
1636
1660
|
const admission = this.evaluateGenerationAdmission(newReleaseId, nextConfig)
|
|
1637
1661
|
|
|
1638
1662
|
if (!admission.allowed) {
|
|
@@ -1729,6 +1753,24 @@ export default class RollbridgeDaemon {
|
|
|
1729
1753
|
})
|
|
1730
1754
|
}
|
|
1731
1755
|
|
|
1756
|
+
/**
|
|
1757
|
+
* Reports whether a retained release group still holds live work that a re-activation would collide with.
|
|
1758
|
+
*
|
|
1759
|
+
* `stopped` is not the same claim as `drained`: the guardian's provenance fence only lets a
|
|
1760
|
+
* process key be re-registered once the record that holds it is gone. A generation recorded
|
|
1761
|
+
* as drained while its process group is still alive — the shape an indefinite worker drain
|
|
1762
|
+
* produces — still holds its keys, so its release id is not available for a restore.
|
|
1763
|
+
* @param {string} releaseId - Release id a handoff would activate.
|
|
1764
|
+
* @returns {HeldGenerationHold | undefined} The live hold, or undefined when the id is free.
|
|
1765
|
+
*/
|
|
1766
|
+
retainedGenerationHoldingLiveWork(releaseId) {
|
|
1767
|
+
const retained = this.releases.get(releaseId)
|
|
1768
|
+
|
|
1769
|
+
if (!retained || retained === this.activeRelease) return undefined
|
|
1770
|
+
|
|
1771
|
+
return heldGenerationHold(retained)
|
|
1772
|
+
}
|
|
1773
|
+
|
|
1732
1774
|
/**
|
|
1733
1775
|
* Continues one exact durable generation transition without an internal retry loop.
|
|
1734
1776
|
* @returns {Promise<Record<string, JsonValue>>} Deploy result after commit.
|
|
@@ -2457,36 +2499,21 @@ export default class RollbridgeDaemon {
|
|
|
2457
2499
|
// process group that outlived the tracked process and is invisible as a pid — together
|
|
2458
2500
|
// with the release the guardian actually reports as active, so the deployment tool can
|
|
2459
2501
|
// reconcile its own current pointer instead of restoring a generation that already drained.
|
|
2460
|
-
|
|
2461
|
-
|
|
2462
|
-
|
|
2502
|
+
// A target that is still draining is not refused here: stopping it is what makes its id
|
|
2503
|
+
// reusable, and the deploy below applies the same fence after that stop.
|
|
2504
|
+
const heldGeneration = target.state === "stopped" ? heldGenerationHold(target) : undefined
|
|
2463
2505
|
|
|
2464
|
-
if (
|
|
2506
|
+
if (heldGeneration) {
|
|
2465
2507
|
const activeReleaseId = this.activeRelease?.releaseId ?? null
|
|
2466
|
-
const held = heldProcesses.map((processStatus) => {
|
|
2467
|
-
if (processStatus.heldProcessGroup) {
|
|
2468
|
-
const {liveMemberCount, pgid} = processStatus.heldProcessGroup
|
|
2469
|
-
const members = liveMemberCount === undefined ? "" : ` with ${liveMemberCount} live member(s)`
|
|
2470
|
-
|
|
2471
|
-
return `${processStatus.id}: process group ${pgid}${members} still alive after the tracked process exited`
|
|
2472
|
-
}
|
|
2473
|
-
|
|
2474
|
-
return `${processStatus.id}: state ${processStatus.state}, pid ${processStatus.pid ?? "unknown"}`
|
|
2475
|
-
}).join("; ")
|
|
2476
2508
|
|
|
2477
2509
|
this.logger("rollback refused: drained generation still holds live processes", {
|
|
2478
2510
|
activeReleaseId,
|
|
2479
|
-
heldProcesses:
|
|
2480
|
-
heldProcessGroupPgid: processStatus.heldProcessGroup?.pgid ?? null,
|
|
2481
|
-
pid: processStatus.pid ?? null,
|
|
2482
|
-
processId: processStatus.id,
|
|
2483
|
-
state: processStatus.state
|
|
2484
|
-
})),
|
|
2511
|
+
heldProcesses: heldGeneration.processes,
|
|
2485
2512
|
releaseId: target.releaseId,
|
|
2486
2513
|
revision: target.revision
|
|
2487
2514
|
})
|
|
2488
2515
|
|
|
2489
|
-
throw new Error(`Release "${target.releaseId}" is recorded as drained while ${
|
|
2516
|
+
throw new Error(`Release "${target.releaseId}" is recorded as drained while ${heldGeneration.count} process${heldGeneration.count === 1 ? "" : "es"} of that generation still hold live work (${heldGeneration.hold.join("; ")}). Restoring it would re-register those processes under a new provenance and fail with a guardian provenance mismatch. The guardian's active release is ${activeReleaseId ? `"${activeReleaseId}"` : "none"}; reconcile the deployment pointer with \`rollbridge status\` before restoring.`)
|
|
2490
2517
|
}
|
|
2491
2518
|
|
|
2492
2519
|
// The target may still be draining a prior deploy (live processes). Stop it before the
|
|
@@ -3155,6 +3182,42 @@ function statusIncludeLogs(value) {
|
|
|
3155
3182
|
return value
|
|
3156
3183
|
}
|
|
3157
3184
|
|
|
3185
|
+
/**
|
|
3186
|
+
* Names the live work a retained release group still holds, using the guardian's own fence.
|
|
3187
|
+
*
|
|
3188
|
+
* A process counts as holding work when the guardian would refuse to hand its key to a new
|
|
3189
|
+
* provenance: a running or stopping process, or a record already `stopped` whose process group
|
|
3190
|
+
* outlived it and is invisible as a pid.
|
|
3191
|
+
* @param {ReleaseGroup} release - Retained generation to inspect.
|
|
3192
|
+
* @returns {HeldGenerationHold | undefined} The hold and its log payload, or undefined when every process is gone.
|
|
3193
|
+
*/
|
|
3194
|
+
function heldGenerationHold(release) {
|
|
3195
|
+
const heldProcesses = release.status().processes.filter((processStatus) => !childRecordIsGone(processStatus))
|
|
3196
|
+
|
|
3197
|
+
if (heldProcesses.length === 0) return undefined
|
|
3198
|
+
|
|
3199
|
+
return {
|
|
3200
|
+
count: heldProcesses.length,
|
|
3201
|
+
hold: heldProcesses.map((processStatus) => {
|
|
3202
|
+
if (processStatus.heldProcessGroup) {
|
|
3203
|
+
const {liveMemberCount, pgid} = processStatus.heldProcessGroup
|
|
3204
|
+
const members = liveMemberCount === undefined ? "" : ` with ${liveMemberCount} live member(s)`
|
|
3205
|
+
|
|
3206
|
+
return `${processStatus.id}: process group ${pgid}${members} still alive after the tracked process exited`
|
|
3207
|
+
}
|
|
3208
|
+
|
|
3209
|
+
return `${processStatus.id}: state ${processStatus.state}, pid ${processStatus.pid ?? "unknown"}`
|
|
3210
|
+
}),
|
|
3211
|
+
processes: heldProcesses.map((processStatus) => ({
|
|
3212
|
+
heldProcessGroupPgid: processStatus.heldProcessGroup?.pgid ?? null,
|
|
3213
|
+
pid: processStatus.pid ?? null,
|
|
3214
|
+
processId: processStatus.id,
|
|
3215
|
+
state: processStatus.state
|
|
3216
|
+
})),
|
|
3217
|
+
state: release.state
|
|
3218
|
+
}
|
|
3219
|
+
}
|
|
3220
|
+
|
|
3158
3221
|
/**
|
|
3159
3222
|
* @param {PrivateOwnerState} transfer - Authenticated private incumbent state.
|
|
3160
3223
|
* @param {GenerationTransition} transition - Unresolved generation transition.
|
package/test/rollbridge.test.js
CHANGED
|
@@ -2127,6 +2127,79 @@ test("rollback refuses to restore a drained generation that still holds live pro
|
|
|
2127
2127
|
}
|
|
2128
2128
|
})
|
|
2129
2129
|
|
|
2130
|
+
test("deploy refuses to activate a release id whose draining generation still holds live processes", async () => {
|
|
2131
|
+
const fixture = await createFixture()
|
|
2132
|
+
const daemon = await startDaemon(fixture.config)
|
|
2133
|
+
/** @type {WebSocket | undefined} */
|
|
2134
|
+
let socket
|
|
2135
|
+
|
|
2136
|
+
try {
|
|
2137
|
+
await daemon.deploy({releaseId: "v1", releasePath: fixture.root, revision: "v1"})
|
|
2138
|
+
|
|
2139
|
+
// A held WebSocket keeps v1 draining after v2 takes traffic over, so v1's generation is
|
|
2140
|
+
// still alive when a restore names its id — the measured 2026-11-15 20:29Z and 20:41Z
|
|
2141
|
+
// shape, where the re-activation aborted with a bare guardian provenance mismatch.
|
|
2142
|
+
socket = await openWebSocket(daemon)
|
|
2143
|
+
await daemon.deploy({releaseId: "v2", releasePath: fixture.root, revision: "v2"})
|
|
2144
|
+
|
|
2145
|
+
const draining = statusRelease(daemon, "v1")
|
|
2146
|
+
const heldPid = draining.processes.find((processStatus) => processStatus.id === "web")?.pid
|
|
2147
|
+
|
|
2148
|
+
if (!heldPid) throw new Error("the draining generation should still hold its web process")
|
|
2149
|
+
|
|
2150
|
+
await expect(daemon.deploy({releaseId: "v1", releasePath: fixture.root, revision: "v1"})).rejects.toThrow(/Deploy refused before activation: release "v1" is retained in state draining while 1 process of that generation still hold live work \(web: state running, pid \d+\)\. Activating that release id again would re-register those processes under a new provenance and abort with a guardian provenance mismatch\. The guardian's active release is "v2"; deploy a new release id or reconcile the deployment pointer with `rollbridge status`\./)
|
|
2151
|
+
|
|
2152
|
+
const afterRefusal = daemon.status()
|
|
2153
|
+
|
|
2154
|
+
// Refusing before activation leaves one generation per release id: nothing was started,
|
|
2155
|
+
// the incumbent keeps serving, and the draining generation is still the one status reports.
|
|
2156
|
+
expect(afterRefusal.activeReleaseId).toBe("v2")
|
|
2157
|
+
expect(afterRefusal.releases.filter((release) => release.releaseId === "v1").length).toBe(1)
|
|
2158
|
+
expect(statusRelease(daemon, "v1").processes.find((processStatus) => processStatus.id === "web")?.pid).toBe(heldPid)
|
|
2159
|
+
expect(await fetchText(daemon, "/release")).toBe("v2")
|
|
2160
|
+
} finally {
|
|
2161
|
+
if (socket) socket.close()
|
|
2162
|
+
await daemon.shutdown()
|
|
2163
|
+
await fs.rm(fixture.root, {force: true, recursive: true})
|
|
2164
|
+
}
|
|
2165
|
+
})
|
|
2166
|
+
|
|
2167
|
+
test("a restore deploy naming a drained generation that outlived its tracked process is refused by name", async () => {
|
|
2168
|
+
const fixture = await createFixture({nonBlockingDrainWorker: true, workerStopDelayMs: 6000})
|
|
2169
|
+
const config = normalizeConfig({
|
|
2170
|
+
...fixture.config,
|
|
2171
|
+
processes: fixture.config.processes.map((processConfig) => processConfig.id === "worker"
|
|
2172
|
+
? {...processConfig, gracefulStopMs: "indefinite"}
|
|
2173
|
+
: processConfig),
|
|
2174
|
+
proxy: {...fixture.config.proxy, nonBlockingDrainTimeoutMs: 200}
|
|
2175
|
+
})
|
|
2176
|
+
const daemon = await startDaemon(config)
|
|
2177
|
+
|
|
2178
|
+
try {
|
|
2179
|
+
await daemon.deploy({releaseId: "v1", releasePath: fixture.root, revision: "v1"})
|
|
2180
|
+
await daemon.deploy({releaseId: "v2", releasePath: fixture.root, revision: "v2"})
|
|
2181
|
+
|
|
2182
|
+
// v1 is recorded as drained while its worker keeps its in-flight work in the group its
|
|
2183
|
+
// shell wrapper led. The 2026-10-25 carve-out that lets a dead generation's process key be
|
|
2184
|
+
// re-registered does not apply here: the group is alive, so the key is still held.
|
|
2185
|
+
await waitFor(() => statusRelease(daemon, "v1").state === "stopped")
|
|
2186
|
+
|
|
2187
|
+
const worker = statusRelease(daemon, "v1").processes.find((processStatus) => processStatus.id === "worker")
|
|
2188
|
+
|
|
2189
|
+
if (!worker?.heldProcessGroup) {
|
|
2190
|
+
throw new Error(`the drained generation should still hold the worker's process group, got ${JSON.stringify(worker && {pid: worker.pid, retirementTimedOut: worker.retirementTimedOut, state: worker.state})}`)
|
|
2191
|
+
}
|
|
2192
|
+
|
|
2193
|
+
await expect(daemon.deploy({releaseId: "v1", releasePath: fixture.root, revision: "v1"})).rejects.toThrow(/Deploy refused before activation: release "v1" is retained in state stopped while 1 process of that generation still hold live work \(worker: process group \d+ with \d+ live member\(s\) still alive after the tracked process exited\)\. Activating that release id again would re-register those processes under a new provenance and abort with a guardian provenance mismatch\. The guardian's active release is "v2"/)
|
|
2194
|
+
expect(daemon.status().activeReleaseId).toBe("v2")
|
|
2195
|
+
expect(daemon.status().releases.filter((release) => release.releaseId === "v1").length).toBe(1)
|
|
2196
|
+
expect(await fetchText(daemon, "/release")).toBe("v2")
|
|
2197
|
+
} finally {
|
|
2198
|
+
await daemon.shutdown()
|
|
2199
|
+
await fs.rm(fixture.root, {force: true, recursive: true})
|
|
2200
|
+
}
|
|
2201
|
+
})
|
|
2202
|
+
|
|
2130
2203
|
test("the rollback control command switches traffic over the socket", async () => {
|
|
2131
2204
|
const fixture = await createFixture()
|
|
2132
2205
|
const daemon = await startDaemon(fixture.config)
|