rollbridge 0.1.65 → 0.1.66

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,27 @@
1
+ ### Fixed
2
+
3
+ - `deploy` refuses a `--release-id` it still retains while that generation holds live work,
4
+ before any process of the candidate starts. PR #12 fenced `rollback` only; the restore that
5
+ actually ran in TensorBuzz production was a deploy command naming the drained release id, and
6
+ it failed twice in twelve minutes (`shared/log/rollbridge.log`, server3, 2026-11-15):
7
+ `20:29:51Z traffic switched 20261115195051 → 20261115202720` then `20:29:55Z release startup
8
+ failed` / `deploy failed` with `Guardian provenance mismatch for
9
+ release:20261115195051:background-jobs-main`, and the identical pair at `20:41:21Z` /
10
+ `20:41:26Z` against `20261115203845`. Only one `release drained` landed in that window
11
+ (`20261115164249` at 20:37:03Z), and a 20:54Z process census found all three generations alive
12
+ at once — `20261115195051` (main + worker + 3 runners), `20261115202720` (main + worker + 4
13
+ runners), `20261115203845` (main + worker + 4 runners) — with `20261115195051`'s
14
+ `velocious background-jobs-main` still running as pid 2310504. The provenance key therefore
15
+ belonged to a live generation, so the 2026-10-25 carve-out that lets a dead generation's key be
16
+ re-registered never applied; the collision aborted the deploy after the handoff had already
17
+ moved traffic, leaving the deploy tool's `current` pointer at `releases/20261115195051` while
18
+ the guardian served `20261115203845`. The refusal is decided from the retained record with the
19
+ guardian's own `childRecordIsGone` fence, names the retained state, each held process (a held
20
+ process group included), and the release the guardian reports as active, and starts nothing:
21
+ `Deploy refused before activation: release "20261115195051" is retained in state draining while
22
+ 1 process of that generation still hold live work (…). … The guardian's active release is
23
+ "20261115203845"; deploy a new release id or reconcile the deployment pointer with \`rollbridge
24
+ status\`.` Journaled generation-transition resume is exempt — it re-drives the transaction that
25
+ already owns the release id — and a rollback to a still-draining target still stops that
26
+ generation first, which is what makes its id reusable; the same fence then applies after the
27
+ stop.
package/docs/cli.md CHANGED
@@ -170,6 +170,18 @@ the incumbent the handoff is about to retire counts too. The incumbent keeps ser
170
170
  nothing of the candidate is started, and the error names every draining group with its
171
171
  `progressing`/`wedged` classification and the bound it exceeded.
172
172
 
173
+ The same pre-activation refusal applies when `--release-id` names a release Rollbridge
174
+ still **retains** while that generation holds live work: draining with processes alive, or
175
+ recorded drained while a process group outlived its tracked process
176
+ (`processes[].heldProcessGroup`). Starting a second generation under that id re-registers
177
+ the guardian process keys the retained generation still holds, so the deploy aborts
178
+ partway through startup with `Guardian provenance mismatch for release:<id>:<key>` — the
179
+ shape a failed-deploy restore produced twice on 2026-11-15 (`20261115195051` at 20:29:55Z
180
+ and again at 20:41:26Z, 12 minutes apart). The refusal names the retained state, each held
181
+ process, and the release the guardian reports as active, and leaves the retained generation
182
+ untouched. A generation whose processes are genuinely gone stays re-deployable: the
183
+ 2026-10-25 carve-out for re-registering a dead generation is unchanged.
184
+
173
185
  With an opt-in handoff-service `lifecycle.activateCommand`, deploy journals the
174
186
  exact transition, waits for old retirement acknowledgement, waits for candidate
175
187
  activation acknowledgement, and synchronously commits the active proxy target.
@@ -260,7 +272,10 @@ release:<id>:<key>` after the deploy has already begun. The refusal is raised
260
272
  before anything is touched and names each held process together with the release
261
273
  the guardian reports as active, so the deploy tool reconciles its own `current`
262
274
  pointer with `rollbridge status` instead of restoring a generation that already
263
- drained.
275
+ drained. A target that is still **draining** is not refused here: stopping it is
276
+ what makes its id reusable, and the deploy it then runs applies the same fence
277
+ after that stop — so a stop that leaves the generation's process group alive is
278
+ reported as the refusal above instead of colliding mid-startup.
264
279
 
265
280
  **Migration constraints.** Rollback only manages processes — it does **not**
266
281
  revert database migrations or other external state. The target release's on-disk
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "rollbridge",
3
- "version": "0.1.65",
3
+ "version": "0.1.66",
4
4
  "description": "Zero-downtime process supervisor and local traffic switcher for deploy-managed apps.",
5
5
  "keywords": [
6
6
  "deploy",
package/src/daemon.js CHANGED
@@ -33,6 +33,7 @@ const STATE_PERSIST_INTERVAL_MS = 5000
33
33
  * @typedef {Omit<import("./release-group.js").ReleaseStatus, "processes"> & {processes: ManagedProcessStatusWithoutLogs[]}} ReleaseStatusWithoutLogs
34
34
  * @typedef {Omit<ProcessStatus, "process"> & {process: ManagedProcessStatusWithoutLogs}} ProcessStatusWithoutLogs
35
35
  * @typedef {Omit<DaemonStatus, "releases" | "services" | "singletons"> & {releases: ReleaseStatusWithoutLogs[], services: ProcessStatusWithoutLogs[], singletons: ProcessStatusWithoutLogs[]}} DaemonStatusWithoutLogs
36
+ * @typedef {{count: number, hold: string[], processes: {heldProcessGroupPgid: number | null, pid: number | null, processId: string, state: import("./managed-process.js").ManagedProcessState}[], state: import("./release-group.js").ReleaseState}} HeldGenerationHold
36
37
  * @typedef {{configDigest: string, format: number, guardian: {pid?: number, socketPath: string, token: string}, reconnectGraceMs: number}} OwnerRecoveryMetadata
37
38
  * @typedef {DaemonStatus & {recovery: OwnerRecoveryMetadata, serviceReleaseIds?: Record<string, string>, singletonReleaseIds?: Record<string, string>}} OwnerRecoverySnapshot
38
39
  * @typedef {{authority: JsonValue, config: import("./config.js").RollbridgeConfig, listenerConnectionSources?: Record<string, Record<string, {http: number, websocket: number}>>, listenerSourceId?: string, recovery?: {command: JsonValue, reconnectGraceMs: number, startupTimeoutMs: number}, releaseConfigs?: Record<string, import("./config.js").RollbridgeConfig>, retiredListenerHandoff?: number, serviceReleaseIds?: Record<string, string>, singletonReleaseIds?: Record<string, string>, snapshot: OwnerRecoverySnapshot}} PrivateOwnerState
@@ -1633,6 +1634,29 @@ export default class RollbridgeDaemon {
1633
1634
  await this.updateGenerationTransition("restoring_committed")
1634
1635
  return await this.resumeGenerationTransition()
1635
1636
  }
1637
+ // A release id that is still retained by a generation holding live work cannot be started
1638
+ // a second time: the candidate's processes re-register the guardian keys that the retained
1639
+ // generation's own processes still hold, so the deploy aborts partway through `start()`
1640
+ // with a bare `Guardian provenance mismatch` — after the caller has committed to this
1641
+ // revision and, for a restore, after the deployment pointer has already moved. Refusing
1642
+ // from the retained record before any process starts keeps the incumbent serving, names the
1643
+ // release the guardian reports as active, and leaves the retained generation intact for
1644
+ // `rollbridge status` to reconcile against. Journaled transition resume above is exempt:
1645
+ // it re-drives the transaction that already owns this release id.
1646
+ const heldGeneration = this.retainedGenerationHoldingLiveWork(newReleaseId)
1647
+
1648
+ if (heldGeneration) {
1649
+ const activeReleaseId = this.activeRelease?.releaseId ?? null
1650
+
1651
+ this.logger("deploy refused: retained generation still holds live processes", {
1652
+ activeReleaseId,
1653
+ heldProcesses: heldGeneration.processes,
1654
+ releaseId: newReleaseId,
1655
+ retainedState: heldGeneration.state
1656
+ })
1657
+
1658
+ throw new Error(`Deploy refused before activation: release "${newReleaseId}" is retained in state ${heldGeneration.state} while ${heldGeneration.count} process${heldGeneration.count === 1 ? "" : "es"} of that generation still hold live work (${heldGeneration.hold.join("; ")}). Activating that release id again would re-register those processes under a new provenance and abort with a guardian provenance mismatch. The guardian's active release is ${activeReleaseId ? `"${activeReleaseId}"` : "none"}; deploy a new release id or reconcile the deployment pointer with \`rollbridge status\`.`)
1659
+ }
1636
1660
  const admission = this.evaluateGenerationAdmission(newReleaseId, nextConfig)
1637
1661
 
1638
1662
  if (!admission.allowed) {
@@ -1729,6 +1753,24 @@ export default class RollbridgeDaemon {
1729
1753
  })
1730
1754
  }
1731
1755
 
1756
+ /**
1757
+ * Reports whether a retained release group still holds live work that a re-activation would collide with.
1758
+ *
1759
+ * `stopped` is not the same claim as `drained`: the guardian's provenance fence only lets a
1760
+ * process key be re-registered once the record that holds it is gone. A generation recorded
1761
+ * as drained while its process group is still alive — the shape an indefinite worker drain
1762
+ * produces — still holds its keys, so its release id is not available for a restore.
1763
+ * @param {string} releaseId - Release id a handoff would activate.
1764
+ * @returns {HeldGenerationHold | undefined} The live hold, or undefined when the id is free.
1765
+ */
1766
+ retainedGenerationHoldingLiveWork(releaseId) {
1767
+ const retained = this.releases.get(releaseId)
1768
+
1769
+ if (!retained || retained === this.activeRelease) return undefined
1770
+
1771
+ return heldGenerationHold(retained)
1772
+ }
1773
+
1732
1774
  /**
1733
1775
  * Continues one exact durable generation transition without an internal retry loop.
1734
1776
  * @returns {Promise<Record<string, JsonValue>>} Deploy result after commit.
@@ -2457,36 +2499,21 @@ export default class RollbridgeDaemon {
2457
2499
  // process group that outlived the tracked process and is invisible as a pid — together
2458
2500
  // with the release the guardian actually reports as active, so the deployment tool can
2459
2501
  // reconcile its own current pointer instead of restoring a generation that already drained.
2460
- const heldProcesses = target.state === "stopped"
2461
- ? target.status().processes.filter((processStatus) => !childRecordIsGone(processStatus))
2462
- : []
2502
+ // A target that is still draining is not refused here: stopping it is what makes its id
2503
+ // reusable, and the deploy below applies the same fence after that stop.
2504
+ const heldGeneration = target.state === "stopped" ? heldGenerationHold(target) : undefined
2463
2505
 
2464
- if (heldProcesses.length > 0) {
2506
+ if (heldGeneration) {
2465
2507
  const activeReleaseId = this.activeRelease?.releaseId ?? null
2466
- const held = heldProcesses.map((processStatus) => {
2467
- if (processStatus.heldProcessGroup) {
2468
- const {liveMemberCount, pgid} = processStatus.heldProcessGroup
2469
- const members = liveMemberCount === undefined ? "" : ` with ${liveMemberCount} live member(s)`
2470
-
2471
- return `${processStatus.id}: process group ${pgid}${members} still alive after the tracked process exited`
2472
- }
2473
-
2474
- return `${processStatus.id}: state ${processStatus.state}, pid ${processStatus.pid ?? "unknown"}`
2475
- }).join("; ")
2476
2508
 
2477
2509
  this.logger("rollback refused: drained generation still holds live processes", {
2478
2510
  activeReleaseId,
2479
- heldProcesses: heldProcesses.map((processStatus) => ({
2480
- heldProcessGroupPgid: processStatus.heldProcessGroup?.pgid ?? null,
2481
- pid: processStatus.pid ?? null,
2482
- processId: processStatus.id,
2483
- state: processStatus.state
2484
- })),
2511
+ heldProcesses: heldGeneration.processes,
2485
2512
  releaseId: target.releaseId,
2486
2513
  revision: target.revision
2487
2514
  })
2488
2515
 
2489
- throw new Error(`Release "${target.releaseId}" is recorded as drained while ${heldProcesses.length} process${heldProcesses.length === 1 ? "" : "es"} of that generation still hold live work (${held}). Restoring it would re-register those processes under a new provenance and fail with a guardian provenance mismatch. The guardian's active release is ${activeReleaseId ? `"${activeReleaseId}"` : "none"}; reconcile the deployment pointer with \`rollbridge status\` before restoring.`)
2516
+ throw new Error(`Release "${target.releaseId}" is recorded as drained while ${heldGeneration.count} process${heldGeneration.count === 1 ? "" : "es"} of that generation still hold live work (${heldGeneration.hold.join("; ")}). Restoring it would re-register those processes under a new provenance and fail with a guardian provenance mismatch. The guardian's active release is ${activeReleaseId ? `"${activeReleaseId}"` : "none"}; reconcile the deployment pointer with \`rollbridge status\` before restoring.`)
2490
2517
  }
2491
2518
 
2492
2519
  // The target may still be draining a prior deploy (live processes). Stop it before the
@@ -3155,6 +3182,42 @@ function statusIncludeLogs(value) {
3155
3182
  return value
3156
3183
  }
3157
3184
 
3185
+ /**
3186
+ * Names the live work a retained release group still holds, using the guardian's own fence.
3187
+ *
3188
+ * A process counts as holding work when the guardian would refuse to hand its key to a new
3189
+ * provenance: a running or stopping process, or a record already `stopped` whose process group
3190
+ * outlived it and is invisible as a pid.
3191
+ * @param {ReleaseGroup} release - Retained generation to inspect.
3192
+ * @returns {HeldGenerationHold | undefined} The hold and its log payload, or undefined when every process is gone.
3193
+ */
3194
+ function heldGenerationHold(release) {
3195
+ const heldProcesses = release.status().processes.filter((processStatus) => !childRecordIsGone(processStatus))
3196
+
3197
+ if (heldProcesses.length === 0) return undefined
3198
+
3199
+ return {
3200
+ count: heldProcesses.length,
3201
+ hold: heldProcesses.map((processStatus) => {
3202
+ if (processStatus.heldProcessGroup) {
3203
+ const {liveMemberCount, pgid} = processStatus.heldProcessGroup
3204
+ const members = liveMemberCount === undefined ? "" : ` with ${liveMemberCount} live member(s)`
3205
+
3206
+ return `${processStatus.id}: process group ${pgid}${members} still alive after the tracked process exited`
3207
+ }
3208
+
3209
+ return `${processStatus.id}: state ${processStatus.state}, pid ${processStatus.pid ?? "unknown"}`
3210
+ }),
3211
+ processes: heldProcesses.map((processStatus) => ({
3212
+ heldProcessGroupPgid: processStatus.heldProcessGroup?.pgid ?? null,
3213
+ pid: processStatus.pid ?? null,
3214
+ processId: processStatus.id,
3215
+ state: processStatus.state
3216
+ })),
3217
+ state: release.state
3218
+ }
3219
+ }
3220
+
3158
3221
  /**
3159
3222
  * @param {PrivateOwnerState} transfer - Authenticated private incumbent state.
3160
3223
  * @param {GenerationTransition} transition - Unresolved generation transition.
@@ -2127,6 +2127,79 @@ test("rollback refuses to restore a drained generation that still holds live pro
2127
2127
  }
2128
2128
  })
2129
2129
 
2130
+ test("deploy refuses to activate a release id whose draining generation still holds live processes", async () => {
2131
+ const fixture = await createFixture()
2132
+ const daemon = await startDaemon(fixture.config)
2133
+ /** @type {WebSocket | undefined} */
2134
+ let socket
2135
+
2136
+ try {
2137
+ await daemon.deploy({releaseId: "v1", releasePath: fixture.root, revision: "v1"})
2138
+
2139
+ // A held WebSocket keeps v1 draining after v2 takes traffic over, so v1's generation is
2140
+ // still alive when a restore names its id — the measured 2026-11-15 20:29Z and 20:41Z
2141
+ // shape, where the re-activation aborted with a bare guardian provenance mismatch.
2142
+ socket = await openWebSocket(daemon)
2143
+ await daemon.deploy({releaseId: "v2", releasePath: fixture.root, revision: "v2"})
2144
+
2145
+ const draining = statusRelease(daemon, "v1")
2146
+ const heldPid = draining.processes.find((processStatus) => processStatus.id === "web")?.pid
2147
+
2148
+ if (!heldPid) throw new Error("the draining generation should still hold its web process")
2149
+
2150
+ await expect(daemon.deploy({releaseId: "v1", releasePath: fixture.root, revision: "v1"})).rejects.toThrow(/Deploy refused before activation: release "v1" is retained in state draining while 1 process of that generation still hold live work \(web: state running, pid \d+\)\. Activating that release id again would re-register those processes under a new provenance and abort with a guardian provenance mismatch\. The guardian's active release is "v2"; deploy a new release id or reconcile the deployment pointer with `rollbridge status`\./)
2151
+
2152
+ const afterRefusal = daemon.status()
2153
+
2154
+ // Refusing before activation leaves one generation per release id: nothing was started,
2155
+ // the incumbent keeps serving, and the draining generation is still the one status reports.
2156
+ expect(afterRefusal.activeReleaseId).toBe("v2")
2157
+ expect(afterRefusal.releases.filter((release) => release.releaseId === "v1").length).toBe(1)
2158
+ expect(statusRelease(daemon, "v1").processes.find((processStatus) => processStatus.id === "web")?.pid).toBe(heldPid)
2159
+ expect(await fetchText(daemon, "/release")).toBe("v2")
2160
+ } finally {
2161
+ if (socket) socket.close()
2162
+ await daemon.shutdown()
2163
+ await fs.rm(fixture.root, {force: true, recursive: true})
2164
+ }
2165
+ })
2166
+
2167
+ test("a restore deploy naming a drained generation that outlived its tracked process is refused by name", async () => {
2168
+ const fixture = await createFixture({nonBlockingDrainWorker: true, workerStopDelayMs: 6000})
2169
+ const config = normalizeConfig({
2170
+ ...fixture.config,
2171
+ processes: fixture.config.processes.map((processConfig) => processConfig.id === "worker"
2172
+ ? {...processConfig, gracefulStopMs: "indefinite"}
2173
+ : processConfig),
2174
+ proxy: {...fixture.config.proxy, nonBlockingDrainTimeoutMs: 200}
2175
+ })
2176
+ const daemon = await startDaemon(config)
2177
+
2178
+ try {
2179
+ await daemon.deploy({releaseId: "v1", releasePath: fixture.root, revision: "v1"})
2180
+ await daemon.deploy({releaseId: "v2", releasePath: fixture.root, revision: "v2"})
2181
+
2182
+ // v1 is recorded as drained while its worker keeps its in-flight work in the group its
2183
+ // shell wrapper led. The 2026-10-25 carve-out that lets a dead generation's process key be
2184
+ // re-registered does not apply here: the group is alive, so the key is still held.
2185
+ await waitFor(() => statusRelease(daemon, "v1").state === "stopped")
2186
+
2187
+ const worker = statusRelease(daemon, "v1").processes.find((processStatus) => processStatus.id === "worker")
2188
+
2189
+ if (!worker?.heldProcessGroup) {
2190
+ throw new Error(`the drained generation should still hold the worker's process group, got ${JSON.stringify(worker && {pid: worker.pid, retirementTimedOut: worker.retirementTimedOut, state: worker.state})}`)
2191
+ }
2192
+
2193
+ await expect(daemon.deploy({releaseId: "v1", releasePath: fixture.root, revision: "v1"})).rejects.toThrow(/Deploy refused before activation: release "v1" is retained in state stopped while 1 process of that generation still hold live work \(worker: process group \d+ with \d+ live member\(s\) still alive after the tracked process exited\)\. Activating that release id again would re-register those processes under a new provenance and abort with a guardian provenance mismatch\. The guardian's active release is "v2"/)
2194
+ expect(daemon.status().activeReleaseId).toBe("v2")
2195
+ expect(daemon.status().releases.filter((release) => release.releaseId === "v1").length).toBe(1)
2196
+ expect(await fetchText(daemon, "/release")).toBe("v2")
2197
+ } finally {
2198
+ await daemon.shutdown()
2199
+ await fs.rm(fixture.root, {force: true, recursive: true})
2200
+ }
2201
+ })
2202
+
2130
2203
  test("the rollback control command switches traffic over the socket", async () => {
2131
2204
  const fixture = await createFixture()
2132
2205
  const daemon = await startDaemon(fixture.config)