rollbridge 0.1.23 → 0.1.25

package/docs/workers.md

@@ -1,117 +1,95 @@
-# Background-job worker deployment
+# Background-job generation deployment
 
-This guide covers deploying background-job workers (or any non-HTTP worker pool)
-with Rollbridge so that in-flight jobs finish across a deploy. It uses features
-that exist today; the command-based lifecycle hooks mentioned at the end are
-still on the roadmap.
+This is the required Rollbridge lifecycle for a background-jobs runtime. A
+generation is release-scoped and contains its own `background-jobs-main` plus
+its worker pool. It is not just a set of workers attached to one persistent
+coordinator.
 
-## Run workers as a `companion`
+## Process topology
 
-Give each worker the `companion` policy. Companions are **release-scoped**: every
-release starts its own workers running that release's code, and a release's
-workers are stopped only when that release is retired (drained) after a newer
-release takes over. They start **before** the `proxied` web process, so they're
-ready before traffic switches.
+Configure jobs-main as a handoff `service` on a multi-port range and the workers
+as same-release `companion`s. Beacon may remain a shared persistent service on a
+fixed port such as `7330`.
 
 ```js
 {
-  id: "worker",
-  policy: "companion",
-  cwd: "{{releasePath}}",
-  command: "npx velocious background-jobs-worker"
-}
-```
-
-## Scale the pool with `replicas`
-
-Set `replicas` to run several identical workers (a port-less companion only).
-Each instance runs as `worker#0`, `worker#1`, … and gets
-`ROLLBRIDGE_REPLICA_INDEX` / `ROLLBRIDGE_REPLICA_COUNT` (and `{{replicaIndex}}` /
-`{{replicaCount}}`), so an instance can claim a distinct shard, queue, or lock:
-
-```js
-{id: "worker", policy: "companion", command: "npx velocious background-jobs-worker", replicas: 4}
-```
-
-Restart the pool with `rollbridge restart --process worker` (all replicas) or a
-single instance with `rollbridge restart --process worker#0`.
-
-## Finish in-flight jobs on stop (`stopSignal` + `gracefulStopMs`)
-
-When Rollbridge stops a worker — during a deploy's drain, a `rollbridge restart`,
-or shutdown — it sends the worker's **`stopSignal`** (default `SIGTERM`), waits up
-to **`gracefulStopMs`**, then `SIGKILL`s it if it hasn't exited. That window is
-the worker's chance to finish its current job and exit cleanly.
-
-- Set `stopSignal` to the signal your worker quiets/drains on. Many job runners
-  finish the current job and exit on `SIGTERM` (the default); some use `SIGINT`
-  or `SIGQUIT`. Use the one your worker treats as "drain and exit".
-- Set `gracefulStopMs` to at least your longest job's duration, so a job in
-  progress is not cut off by the `SIGKILL` fallback. Use `"indefinite"` only for
-  workers that are safe to leave draining until they exit on their own.
-
-```js
+  id: "background-jobs-main",
+  policy: "service",
+  deployStrategy: "handoff",
+  command: "npx velocious background-jobs-main",
+  port: {from: 7331, to: 7399}
+},
 {
-  id: "worker",
+  id: "background-jobs-worker",
   policy: "companion",
+  env: {VELOCIOUS_BACKGROUND_JOBS_PORT: "{{ports.background-jobs-main}}"},
   command: "npx velocious background-jobs-worker",
   replicas: 4,
-  stopSignal: "SIGTERM",
+  nonBlockingDrain: true,
   gracefulStopMs: "indefinite"
 }
 ```
 
-## What happens across a deploy
-
-1. The new release's workers start (running the **new** code) before traffic
-   switches to the new web process.
-2. Both old and new workers run while the previous release drains, so **both
-   code versions consume the shared queue at once.** Keep job code
-   backwards-compatible across a deploy — the same rule as database migrations.
-3. When the previous release is retired (its HTTP/WebSocket connections close or
-   `proxy.drainTimeoutMs` elapses), its workers are stopped: `stopSignal`, then
-   `SIGKILL` after `gracefulStopMs`.
-
-Because old workers are retired on the release's **connection** drain (not on
-their own job queue draining), a job still running when the release is retired
-gets only the `gracefulStopMs` window to finish, unless `gracefulStopMs` is
-`"indefinite"`. Keep jobs **idempotent and safe to retry** so a job interrupted
-at a finite `SIGKILL` fallback can run again.
-
-## Command-based lifecycle hooks
-
-For workers that quiesce or drain via a command rather than a single signal, set
-a `lifecycle` block. When Rollbridge gracefully stops the worker it runs
-`quietCommand` (stop accepting new work), then drains (`drainCommand`, or waits up
-to `drainTimeoutMs` for the worker to exit), then `stopCommand` or `stopSignal`,
-then `SIGKILL` after `gracefulStopMs`. Each hook gets `ROLLBRIDGE_PID` and is
-bounded by a timeout, so a slow hook can't wedge a deploy.
-
-```js
-{
-  id: "worker",
-  policy: "companion",
-  command: "npx velocious background-jobs-worker",
-  replicas: 4,
-  lifecycle: {quietCommand: "kill -TSTP -$ROLLBRIDGE_PID", drainTimeoutMs: 60000}
-}
-```
-
-See [`docs/config.md`](config.md#processeslifecycle) for the hook reference.
-
-## Non-blocking drain
-
-By default a retired release's workers are stopped only after the proxied
-process's connections have drained. Set `nonBlockingDrain: true` on a worker
-companion whose work is independent of the web process (a job worker on a shared
-queue) to start its graceful stop **immediately** when the release is retired —
-in parallel with the connection drain. The new release's workers handle new work
-while the old workers finish their in-flight jobs:
-
-```js
-{id: "worker", policy: "companion", command: "…", nonBlockingDrain: true, gracefulStopMs: "indefinite"}
-```
-
-See [`docs/config.md`](config.md) for `stopSignal`, `replicas`, and
-`gracefulStopMs`, and [`docs/velocious.md`](velocious.md) for a full Velocious
-deployment (Beacon, jobs-main, workers, web) example.
+Each worker receives its generation's jobs-main port. Old workers keep that port
+for their entire lifetime; normal deploy draining never hands them to, or lets
+them reconnect to, the new jobs-main. `replicas` scales the pool as
+`background-jobs-worker#0`, `#1`, and so on.
+
+## Deploy and retirement sequence
+
+1. Before activation, Rollbridge starts the candidate release's jobs-main and
+   complete worker pool, then starts and health-checks the candidate web process.
+2. Activation switches new web traffic and makes the candidate jobs generation
+   active.
+3. The previous jobs generation retires as one unit. Its jobs-main stops schedule
+   ownership, new queue dispatch, and new ordinary worker handoffs. Its workers
+   stop accepting handoffs.
+4. The old jobs-main stays running with its old workers. It continues owning
+   their connections and heartbeats, lease fencing, terminal-report acceptance
+   and acknowledgement, and durable store transitions. The old worker/reporting
+   side durably retries terminal reports, tracks outstanding report promises,
+   enforces per-job execution timeouts, and owns and reaps child runners.
+5. Work returned or retried to the shared queue becomes eligible for the new
+   active generation. The retired main never dispatches it again.
+6. The old main and workers remain one release generation until every accepted
+   handoff settles. Only then, after every old worker drains and exits, may the
+   old jobs-main exit. Rollbridge then reaps the generation and reports that its
+   release reference ended so Rampway can release the retention pin.
+
+Old and new generations may overlap for hours, each running its own release code
+and jobs-main endpoint. Multiple retired generations may drain concurrently.
+
+## Deploy completion is independent
+
+The deploy succeeds when the candidate release is activated and healthy. The
+command and deploy lock do not wait for old jobs generations, workers, jobs,
+HTTP/WebSocket connections, or other retained services to finish. The required
+supervisor contract durably retains generations after the command returns and
+across later deploys and supervisor/host recovery. Every referenced release
+directory must be reported to Rampway and stays pinned against cleanup until the
+last retained process exits. Current Rollbridge provides asynchronous draining
+only for the lifetime of the current daemon; see the current-behavior notes in
+[`docs/config.md`](config.md#processesdeploystrategy) and
+[`docs/cli.md`](cli.md#deploy).
+
+HTTP/WebSocket drain and jobs drain are independent. Set `nonBlockingDrain: true`
+so workers stop accepting new handoffs when retirement starts rather than after
+the connection drain. Closing or timing out HTTP connections must never kill a
+still-draining jobs-main or worker pool.
+
+## Timeouts and failures
+
+`stopSignal`, `lifecycle`, and `gracefulStopMs` remain useful process-stop tools,
+and the jobs framework should enforce per-job timeouts for genuinely hung work.
+They are not the primary deployment solution. A legitimate multi-hour job makes
+a multi-hour generation drain valid; do not turn a normal worker-shutdown timeout
+into the deploy deadline.
+
+If a worker connection is actually lost, its old jobs-main applies lease fencing
+and the durable store transitions that make work eligible to return or retry.
+Returned work may then run in the active generation. Normal retirement does not
+simulate a disconnect and does not make a new jobs-main adopt the old worker's
+handoffs.
+
+See [`docs/velocious.md`](velocious.md) for the complete topology and
+[`docs/config.md`](config.md#processesdeploystrategy) for handoff-service fields.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "rollbridge",
-  "version": "0.1.23",
+  "version": "0.1.25",
   "description": "Zero-downtime process supervisor and local traffic switcher for deploy-managed apps.",
   "keywords": [
     "deploy",

package/src/daemon.js

@@ -882,12 +882,13 @@ export default class RollbridgeDaemon {
     }
 
     await captureShutdownError(cleanupErrors, "proxy close", async () => this.proxy.close())
-    const stopResults = await Promise.allSettled([
-      ...[...this.services.values()].map((processInstance) => processInstance.stop()),
+    const dependentStopResults = await Promise.allSettled([
       ...[...this.singletons.values()].map((processInstance) => processInstance.stop()),
       ...[...this.startingReleases].map((release) => release.stop()),
       ...[...this.releases.values()].map((release) => release.stop())
     ])
+    const serviceStopResults = await Promise.allSettled([...this.services.values()].map((processInstance) => processInstance.stop()))
+    const stopResults = [...dependentStopResults, ...serviceStopResults]
     await captureShutdownError(cleanupErrors, "proxy server close", () => this.closeServer(this.proxyServer))
 
     // Wait for any in-flight write first so it can't recreate or overwrite the final state (no

package/test/shutdown-completion.test.js

@@ -115,6 +115,60 @@ test("shutdown response waits for endpoint and owned-process cleanup before imme
   }
 })
 
+test("shutdown keeps daemon services alive until release-owned dependents stop", async () => {
+  const root = await fs.mkdtemp(path.join(os.tmpdir(), "rollbridge-shutdown-service-order-"))
+  const socketPath = path.join(root, "control.sock")
+  const processCommand = `${JSON.stringify(process.execPath)} -e ${JSON.stringify("setInterval(() => {}, 1000)")}`
+  const config = buildConfig(socketPath, {
+    companion: {command: processCommand, gracefulStopMs: "indefinite", id: "worker", policy: "companion"},
+    service: {command: processCommand, id: "coordinator", policy: "service"}
+  })
+  const daemon = new RollbridgeDaemon({config, logger: () => {}})
+  /** @type {() => void} */
+  let releaseWorker = () => {}
+  const workerGate = new Promise((resolve) => { releaseWorker = () => resolve(undefined) })
+  let serviceStopStarted = false
+
+  try {
+    await daemon.start()
+    await daemon.deploy({releaseId: "v1", releasePath: root, revision: "v1"})
+
+    const release = daemon.activeRelease
+    const coordinator = daemon.services.get("coordinator")
+
+    assert.ok(release)
+    assert.ok(coordinator)
+
+    const originalReleaseStop = release.stop.bind(release)
+    const originalCoordinatorStop = coordinator.stop.bind(coordinator)
+    let signalReleaseStopStarted = () => {}
+    const releaseStopStarted = new Promise((resolve) => { signalReleaseStopStarted = () => resolve(undefined) })
+
+    release.stop = async () => {
+      signalReleaseStopStarted()
+      await workerGate
+      await originalReleaseStop()
+    }
+    coordinator.stop = async () => {
+      serviceStopStarted = true
+      await originalCoordinatorStop()
+    }
+
+    const shutdown = daemon.shutdown()
+
+    await releaseStopStarted
+    const serviceStoppedWhileWorkerWasDraining = serviceStopStarted
+    releaseWorker()
+    await shutdown
+
+    assert.equal(serviceStoppedWhileWorkerWasDraining, false, "a worker must retain access to daemon services throughout its drain")
+  } finally {
+    releaseWorker()
+    await daemon.shutdown()
+    await fs.rm(root, {force: true, recursive: true})
+  }
+})
+
 test("external-owner retirement releases listeners before a long-draining companion exits", async () => {
   const root = await fs.mkdtemp(path.join(os.tmpdir(), "rollbridge-owner-retirement-"))
   const socketPath = path.join(root, "rollbridge.sock")
@@ -314,13 +368,13 @@ test("shutdown of an already-stopped endpoint fails explicitly", async () => {
 
 /**
  * @param {string} socketPath - Control socket path.
- * @param {{companion?: Record<string, import("../src/json.js").JsonValue>}} [options] - Optional companion process.
+ * @param {{companion?: Record<string, import("../src/json.js").JsonValue>, service?: Record<string, import("../src/json.js").JsonValue>}} [options] - Optional dependent processes.
  * @returns {import("../src/config.js").RollbridgeConfig} Normalized config.
  */
-function buildConfig(socketPath, {companion} = {}) {
+function buildConfig(socketPath, {companion, service} = {}) {
   return normalizeConfig({
     ...rawConfig(socketPath),
-    ...(companion ? {processes: [companion, ...rawConfig(socketPath).processes]} : {})
+    ...((companion || service) ? {processes: [...(service ? [service] : []), ...(companion ? [companion] : []), ...rawConfig(socketPath).processes]} : {})
   })
 }