rollbridge 0.1.24 → 0.1.26

package/AGENTS.md

@@ -0,0 +1,55 @@
+# Rollbridge contributor guidance
+
+Rollbridge is the process supervisor and local traffic switcher. Keep deployment
+tool concerns outside this repository: Rampway owns activation transactions,
+deploy locks, release-retention metadata, and on-disk cleanup; Rollbridge owns
+process lifecycle, retained generations, ports, and recovery state.
+
+## Background-jobs lifecycle invariant
+
+- One runtime generation is one release-scoped `background-jobs-main` plus its
+  worker pool. Start the complete candidate generation before activation.
+- After activation, retire the old generation as one unit. Its main stops
+  schedules, new dispatch, and new ordinary worker handoffs; its workers stop
+  accepting handoffs. The old main remains running with those workers and owns
+  their connections, lease fencing, report acceptance and acknowledgement, and
+  durable store transitions. The worker/reporting side durably retries terminal
+  reports, tracks outstanding report promises, enforces per-job execution
+  timeouts, and owns and reaps child runners. Main and workers remain one release
+  generation until every accepted handoff settles.
+- Returned or retried work may be dispatched by the new active generation. A
+  retired main never dispatches it again. Old workers never reconnect or hand
+  over to the new main during a normal deploy.
+- Jobs generations may overlap for hours on release-scoped endpoints. Beacon may
+  remain shared on `7330`; jobs-main must use a per-release port range.
+- Deploy success is candidate activation and health. Deploy completion must not
+  wait for retired jobs generations, workers, jobs, HTTP/WebSocket connections,
+  or other retained services. HTTP and jobs drains are independent; an HTTP
+  drain finishing or timing out must not stop a live jobs generation.
+- Persist and recover retired-generation ownership across later deploys and
+  supervisor/host recovery. Multiple generations may drain concurrently. Report
+  release references so the deployment tool can keep them pinned until no
+  retained process uses them.
+- Runtime-owner replacement transfers or preserves this durable supervision and
+  returns once the replacement is healthy. It is never a synchronous full
+  shutdown. Per-job timeouts remain valid, but a normal worker-shutdown timeout
+  is not the deploy solution; legitimate multi-hour drains are valid.
+
+Documentation must distinguish required architecture from behavior not yet
+implemented. Do not claim production compliance when source/config still uses a
+fixed jobs-main, worker adoption by a new main, destructive orphan recovery, or
+synchronous cleanup.
+
+## Validation and publication
+
+The project is ESM JavaScript with JSDoc type checking. Package scripts are the
+source of truth: `npm run typecheck`, `npm run lint`, `npm test`, and the combined
+`npm run all-checks`. Run focused checks for the files changed; documentation-only
+work requires at least `git diff --check` plus any existing repository-owned
+Markdown/link check, without installing dependencies.
+
+Work on a feature branch and open a normal pull request against `master`; never
+push feature work directly to `master`. Releases are maintainer-only and use
+`npm run release:patch` from an up-to-date clean default branch after
+`npm run all-checks`; do not edit package versions or publish during ordinary PR
+work. See `docs/releasing.md` for the complete release checklist.

package/README.md

@@ -2,7 +2,22 @@
 
 Rollbridge is a Node.js process supervisor and local traffic switcher for zero-downtime deploys.
 
-Nginx points at one stable Rollbridge proxy port. Deploy tooling asks Rollbridge to start a new release, health-check it, switch new traffic to it, and drain old HTTP/WebSocket connections before stopping the previous release.
+Nginx points at one stable Rollbridge proxy port. Deploy tooling asks Rollbridge to start a new release, health-check it, and switch new traffic to it. Retirement then continues asynchronously: HTTP/WebSocket connections and retained process generations drain independently after the deploy command returns.
+
+> **Required jobs-generation contract:** a release-scoped background-jobs
+> runtime is its own `background-jobs-main` plus worker pool. Rollbridge starts a
+> complete candidate generation before activation. After activation the retired
+> main stops schedules, new dispatch, and new worker handoffs, but remains with
+> its old workers to supervise every handoff it already made until all jobs and
+> workers settle. Old workers never move to the new main. Generations may overlap
+> for hours on separate ports, and neither deploy completion nor HTTP drain
+> completion waits for or kills them. See [Background-job worker
+> deployment](docs/workers.md) and the [Velocious deployment
+> guide](docs/velocious.md).
+
+This is the required architecture, not evidence that every released runtime or
+consumer config already implements durable recovery and release-reference
+reporting; verify source and config before claiming compliance.
 
 ## Install
 
@@ -47,23 +62,28 @@ export default {
   processes: [
     {
       id: "beacon",
-      policy: "companion",
+      policy: "service",
       cwd: "{{releasePath}}",
       command: "env VELOCIOUS_BEACON_PORT={{port}} npx velocious beacon",
-      port: {from: 17330, to: 17399}
+      port: 7330
     },
     {
       id: "background-jobs-worker",
       policy: "companion",
       cwd: "{{releasePath}}",
+      env: {VELOCIOUS_BACKGROUND_JOBS_PORT: "{{ports.background-jobs-main}}"},
       command: "npx velocious background-jobs-worker",
+      nonBlockingDrain: true,
+      gracefulStopMs: "indefinite",
       outputLines: 200
     },
     {
       id: "background-jobs-main",
       policy: "service",
+      deployStrategy: "handoff",
       cwd: "{{releasePath}}",
-      command: "npx velocious background-jobs-main"
+      command: "env VELOCIOUS_BACKGROUND_JOBS_PORT={{port}} npx velocious background-jobs-main",
+      port: {from: 7331, to: 7399}
     },
     {
       id: "web",
@@ -118,10 +138,11 @@ See [`docs/config.md`](docs/config.md#processesmemory).
 memory: {limitBytes: 536870912, warnBytes: 402653184, checkIntervalMs: 5000}
 ```
 
-Set a process's `stopSignal` (default `"SIGTERM"`) to the signal it quiets on, so
-a worker finishes its in-flight work before exiting. Rollbridge sends `stopSignal`
-to gracefully stop the process and `SIGKILL`s it only if it hasn't exited within
-`gracefulStopMs`. For example, a job worker that drains on `SIGINT`:
+Set a process's `stopSignal` (default `"SIGTERM"`) to the signal it quiets on.
+This is a process-level stop mechanism, not the primary deploy-completion
+mechanism for a release-scoped jobs generation. A retired generation may remain
+for hours, and its old jobs-main must stay available until its workers finish.
+For example, a generic worker that drains on `SIGINT`:
 
 ```js
 {id: "worker", policy: "companion", command: "…", stopSignal: "SIGINT", gracefulStopMs: 60000}
@@ -138,18 +159,19 @@ queue. See [`docs/config.md`](docs/config.md#processesreplicas).
 {id: "worker", policy: "companion", command: "npx velocious background-jobs-worker", replicas: 4}
 ```
 
-For workers that quiesce or drain via a command, set a `lifecycle` block —
+For generic workers that quiesce or drain via a command, set a `lifecycle` block —
 Rollbridge runs `quietCommand`, then drains (`drainCommand`/`drainTimeoutMs`),
 then `stopCommand`/`stopSignal`, then `SIGKILL` after `gracefulStopMs` when
 gracefully stopping the process. Each hook is bounded so it can't wedge a stop.
 
-Set `nonBlockingDrain: true` on a worker companion to start its graceful stop the
-moment its release is retired — in parallel with the proxied connection drain,
-not after it — so new workers handle new work while the old workers finish theirs.
+Set `nonBlockingDrain: true` on a jobs worker companion so it stops accepting new
+handoffs as soon as its release retires, independently of the proxied connection
+drain. Its release-scoped handoff jobs-main remains running to supervise existing
+handoffs and exits only after the worker pool has drained.
 
-See [`docs/workers.md`](docs/workers.md) for the full safe background-job worker
-deployment pattern — companion policy, `replicas`, and finishing in-flight jobs
-on deploy with `stopSignal`/`lifecycle` + `gracefulStopMs`.
+See [`docs/workers.md`](docs/workers.md) for the full release-generation
+deployment pattern: a handoff `background-jobs-main`, its companion worker pool,
+independent quiescence, and durable supervision while retained generations drain.
 
 Set `releaseRetention` to bound how many stopped (drained) releases the daemon
 keeps in memory and reports in `status`. `keep` (default `10`) retains the most
@@ -270,8 +292,9 @@ A release-scoped helper (for example a background worker bound to one release).
 It starts **before** the proxied process in the same release, so release-local
 dependencies are ready before the health check, and it is auto-restarted while
 its release is active. Each release gets its own companions; a release's
-companions stop when that release is drained and retired after a newer release
-takes over.
+companions normally stop when that release is drained and retired after a newer
+release takes over. A jobs worker configured with `nonBlockingDrain` instead
+quiesces at retirement and drains independently of HTTP/WebSocket connections.
 
 ```js
 {
@@ -279,7 +302,8 @@ takes over.
   policy: "companion",
   cwd: "{{releasePath}}",
   command: "npx velocious background-jobs-worker",
-  gracefulStopMs: 60000
+  nonBlockingDrain: true,
+  gracefulStopMs: "indefinite"
 }
 ```
 
@@ -301,36 +325,36 @@ new copies simultaneously during a deploy would be unsafe.
 
 ### `service`
 
-A daemon-wide broker that should outlive individual releases — for example
-Velocious Beacon or `background-jobs-main`. Rollbridge starts it once (before
-release processes that depend on it), keeps it running across deploys, and gives
-it a stable port that does not change between releases. After each successful
-deploy its restart template is refreshed to the latest release, so if it crashes
-it restarts from the newest good release. It keeps restarting until the daemon
-shuts down.
+A service can be daemon-wide and persistent, or release-scoped with
+`deployStrategy: "handoff"`. Velocious Beacon is normally persistent on a stable
+port. `background-jobs-main` is not: it must be a handoff service with one port
+per release so old workers keep their old coordinator while new workers use the
+candidate coordinator.
 
 ```js
 {
   id: "background-jobs-main",
   policy: "service",
+  deployStrategy: "handoff",
   cwd: "{{releasePath}}",
   command: "npx velocious background-jobs-main",
-  port: 7331
+  port: {from: 7331, to: 7399}
 }
 ```
 
 ### Deploy ordering
 
-On `rollbridge deploy`, Rollbridge:
+On `rollbridge deploy`, the required ordering is:
 
-1. starts any `service` that is not already running;
+1. starts any missing persistent service and the candidate's handoff services;
 2. starts the new release's `companion`s, then its `proxied` process, and
    health-checks the proxied process;
 3. switches new traffic to the new release;
-4. refreshes each `service`'s restart template to the new release;
+4. marks the previous jobs-main and worker pool retired as one generation;
 5. replaces `singleton`s (stops the old one, then starts the new one);
-6. drains the previous release's connections, then stops its `proxied` and
-   `companion` processes.
+6. returns success without waiting for the previous generation or its independent
+   HTTP/WebSocket drain; Rollbridge supervises all retained drains in the
+   background and reaps each generation only after its handoffs and workers end.
 
 If the new release fails to start or health-check, the previous release stays
 active and any service started during this deploy is rolled back.
@@ -413,13 +437,13 @@ rollbridge daemon --config /srv/ticket-server/rollbridge.js \
 ```
 
 External supervisors that need to replace a foreground owner without waiting
-for its workers to drain add `--takeover-owner`. The candidate starts and
-health-checks the exact release first. Only then does it retire the accepted
-owner's proxy/control listeners; the retired daemon keeps its already-accepted
-workers until their ordinary drain completes while the attested replacement
-binds the stable listeners. Candidate bootstrap failure leaves the accepted
-owner untouched. This is opt-in; ordinary daemon bootstrap and `shutdown` keep
-their existing behavior.
+for retained generations to drain add `--takeover-owner`. The candidate starts
+and health-checks the exact release first. Only then does it retire the accepted
+owner's proxy/control listeners. Durable supervision of old jobs-main/worker
+generations must be preserved or transferred while the attested replacement
+binds the stable listeners; replacement does not mean full synchronous shutdown.
+Candidate bootstrap failure leaves the accepted owner untouched. This is opt-in;
+ordinary daemon bootstrap and `shutdown` keep their existing behavior.
 
 The four bootstrap inputs are all-or-nothing and use absolute config/release
 paths. Rollbridge binds its proxy, activates the release through the normal
@@ -600,7 +624,7 @@ daemon holds its log file open, so logrotate needs `copytruncate`.
 
 ## Deployment Notes
 
-Run migrations before `rollbridge deploy`, and keep migrations backwards-compatible while old and new web releases overlap. For stable local brokers such as Velocious Beacon or `background-jobs-main`, use `service` when the process should survive deploys and restart from the latest successful release if it crashes.
+Run migrations before `rollbridge deploy`, and keep migrations backwards-compatible while old and new web and jobs generations overlap. Velocious Beacon may be a persistent fixed-port `service`; configure `background-jobs-main` as a release-scoped handoff service on a port range. A normal deploy may leave several retired generations draining concurrently.
 
 See [`docs/deploy-recipes.md`](docs/deploy-recipes.md) for ready-to-use shell, CI, and Capistrano recipes that drive Rollbridge through its CLI, and [`docs/troubleshooting.md`](docs/troubleshooting.md) for diagnosing health-check failures, port conflicts, stale sockets, crash loops, and stuck draining releases.
 

package/TODO.md

@@ -30,11 +30,12 @@ This roadmap tracks planned Rollbridge features and documentation. Rollbridge sh
   - [x] Distinguish crash restarts, deploy replacements, manual restarts, and memory restarts in status/events. (Per-process `lastStartReason` + a `reason` on the `process started` event; the `memory` reason is wired and fires once memory supervision restarts a process.)
   - [x] Add a `restart` CLI command for a single process, a policy group, or all non-proxied workers.
   - [x] Keep restart behavior safe for job workers by using lifecycle hooks before termination. (Manual restart, memory restart, and deploy-drain stops all run the `lifecycle` hooks via `stop()`.)
-- [x] Graceful job-worker lifecycle.
+- [x] Graceful process-stop controls for job workers.
   - [x] Add generic lifecycle hooks such as `quietCommand`, `drainCommand`, `drainTimeoutMs`, and `stopCommand` (per-process `lifecycle`).
   - [x] Support signal-only lifecycle steps for workers that can quiet on a Unix signal. (Per-process `stopSignal`; sent before the `SIGKILL`-after-`gracefulStopMs` fallback.)
-  - [x] Add a non-blocking drain mode so new workers can start while old workers finish running jobs (per-process `nonBlockingDrain`; drains the worker in parallel with the connection drain).
-  - [x] Document a Velocious background-jobs-worker recipe once the lifecycle contract is implemented (`docs/velocious.md` → Worker recipe).
+  - [x] Add a non-blocking drain mode so a worker can quiesce at release retirement independently of the HTTP/WebSocket connection drain (`nonBlockingDrain`). This control alone does not provide durable retired-generation supervision.
+  - [x] Document the required Velocious release-generation contract (`docs/velocious.md` and `docs/workers.md`) without treating documentation as proof that the runtime implements it.
+  - [ ] Implement and verify durable release-scoped jobs-main retirement: owned-handoff supervision, recovery across daemon/host replacement, multiple concurrent retired generations, and release-reference reporting for cleanup pins.
 - [x] Replicas and stable worker indexes. (Supported on port-less `companion` processes; `proxied`/`singleton`/ported processes stay single.)
   - [x] Allow one process config to start multiple replicas (`replicas`, companion-only for now).
   - [x] Expose `ROLLBRIDGE_REPLICA_INDEX`, replica count, and per-replica template context (`{{replicaIndex}}`/`{{replicaCount}}`).
@@ -91,7 +92,7 @@ This roadmap tracks planned Rollbridge features and documentation. Rollbridge sh
 - [x] Write a CLI reference for `daemon`, `ensure-daemon`, `deploy`, `status`, `stop`, `shutdown`, and future commands (`docs/cli.md`).
 - [x] Expand process policy docs with deployment examples for `proxied`, `companion`, `singleton`, and `service`.
 - [x] Document memory checks and auto-restart behavior after the feature lands (`docs/config.md` → `processes[].memory`).
-- [x] Document safe background-job deployment patterns (`docs/workers.md`: companion + `replicas` + `stopSignal` + `gracefulStopMs`, old/new worker overlap).
+- [x] Document the required background-job generation pattern (`docs/workers.md`: release-scoped jobs-main + worker pool, independent quiescence, durable retained supervision, and deploy completion that does not wait for drains).
 - [x] Document worker lifecycle hooks (`docs/config.md` → `processes[].lifecycle`, `docs/workers.md`).
 - [x] Add a Velocious deployment guide with Beacon, background-jobs-main, background-jobs-worker, and web process examples (`docs/velocious.md`).
 - [x] Add an Nginx guide with WebSocket headers, timeouts, and common failure modes (`docs/nginx.md`).

package/docs/cli.md

@@ -66,10 +66,16 @@ and waits for control-socket deployments.
 
 `--takeover-owner` requires the complete bootstrap tuple. It bootstraps and
 health-checks the replacement before sending the current daemon the private
-retirement command. Retirement stops stable listeners and new work promptly,
-but does not wait for owned workers' normal drain before the replacement binds
-the proxy and control socket. A bootstrap failure occurs before retirement, so
-the previously accepted owner remains available.
+retirement command. The current `performOwnerRetirement` path quiesces every
+service, singleton, starting release, and retained release, releases the stable
+listeners, and starts asynchronous `stop()` calls for all of them. It neither
+preserves nor transfers retained-generation supervision to the replacement.
+The replacement can bind before those stops finish. A bootstrap failure occurs
+before retirement, so the previously accepted owner remains available.
+
+A compliant future owner handoff must instead preserve or transfer durable
+supervision of retained generations without handing old workers to a new
+jobs-main. Current `--takeover-owner` does not provide that behavior.
 
 ## `ensure-daemon`
 
@@ -120,12 +126,26 @@ rollbridge deploy --release-path <path>
                   [--daemon-start-timeout-ms <ms>]
 ```
 
-Starts the prepared release, health-checks the proxied process, switches new
-traffic to it, then drains and stops the previous release. Prints
+Starts the complete prepared release generation, health-checks the proxied
+process, and switches new traffic to it. The current daemon then starts
+retirement of the previous release asynchronously, so the command does not wait
+for old workers, jobs, or HTTP/WebSocket connections to finish. Prints
 `{"status": "success", "activeReleaseId": "...", "previousReleaseId": "..."}`.
 If the new release fails to start or health-check, the previous release stays
 active and the command errors.
 
+After candidate activation, `Daemon.deploy()` synchronously waits for singleton
+replacement before starting `drainAndPrune` and returning. A replacement failure
+can therefore return a non-zero result while the candidate remains active, and a
+slow replacement delays both the response and retirement of the old release.
+
+This is a process-lifetime non-blocking drain, not durable supervision across a
+daemon or host restart. It continues across later deploys only while the same
+daemon remains alive. After a restart, surviving PIDs from persisted state are
+advisory orphans that Rollbridge cannot re-adopt; explicit `recover --force`
+stops them. Restart-surviving retained-generation ownership and recovery remain
+required future behavior.
+
 Before each deploy, the daemon reloads the config path it was started with.
 Compatible process and lifecycle changes apply to the new release and govern
 how the previous release retires, including updated `nonBlockingDrain`,

package/docs/config.md

@@ -75,7 +75,7 @@ to let a deploy group talk to the daemon.
 | `proxy.upstreamHost` | string | `proxy.host`, or `"127.0.0.1"` when `proxy.host` is `0.0.0.0`/`::` | Host Rollbridge uses for release health checks and proxy targets. |
 | `proxy.healthPath` | string | `"/ping"` | Default health-check path for proxied processes. |
 | `proxy.healthTimeoutMs` | number | `30000` | Default health-check timeout for proxied processes. |
-| `proxy.drainTimeoutMs` | number | `60000` | How long to drain open connections from a retired release before stopping it. |
+| `proxy.drainTimeoutMs` | number | `60000` | How long to drain HTTP/WebSocket connections before stopping the retired proxied process. Expiry never stops an independently draining jobs generation. |
 | `proxy.forceStopTimeoutMs` | number | `10000` | Default per-process graceful-stop timeout (`SIGTERM`, then `SIGKILL`). |
 
 ## `releaseRetention`
@@ -212,6 +212,20 @@ range** so old and new instances can run at the same time:
 
 Reference it from same-release processes with `{{ports.background-jobs-main}}`.
 During a deploy, old workers keep the old port and new workers get the new port.
+For background jobs, the required compliant architecture makes the handoff
+service and its workers one release generation. Immediately after activation,
+retirement must quiesce the old jobs-main's scheduling, dispatch, and new worker
+handoffs while keeping it with its workers until their accepted work settles.
+Workers are not adopted by the new service.
+
+That immediate old-main quiescence is **required future compliance behavior**,
+not current `deployStrategy: "handoff"` behavior. Today the release group starts
+`stop()` for `nonBlockingDrain` companions at retirement but sends no retirement
+or quiescence notice to the handoff service. It waits for the connection drain,
+stops other dependent processes, waits for the non-blocking companion stops, and
+only then stops the handoff service. Consequently, old and new jobs-main
+instances can overlap scheduling and dispatch ownership. Do not treat the
+configuration above alone as compliance with the background-jobs contract.
 
 ### `processes[].lifecycle`
 
@@ -220,6 +234,11 @@ deploy's drain, a `rollbridge restart`, a memory restart, or shutdown. They let
 job worker quiesce and finish in-flight work before it is terminated. Omit
 `lifecycle` for the default behavior (just `stopSignal` then `SIGKILL`).
 
+These hooks describe an individual process stop. They do not define when a
+deploy completes and must not impose a short normal-drain deadline on a jobs
+generation. Per-job timeouts remain the correct bound for genuinely hung jobs;
+legitimate hours-long generation drains are valid.
+
 | Field | Type | Default | Description |
 | --- | --- | --- | --- |
 | `lifecycle.quietCommand` | string | unset | Run first to tell the process to stop accepting new work. |
@@ -253,15 +272,16 @@ That keeps a worker alive in case the draining web process still depends on it
 but it also holds a background worker open for the whole connection drain.
 
 Set `nonBlockingDrain: true` on a `companion` whose work is independent of the
-proxied process (a job worker on a shared queue). Its graceful stop — `lifecycle`
-hooks, or `stopSignal` then `SIGKILL` after `gracefulStopMs` — then starts **as
-soon as the release is retired**, in parallel with the connection drain, rather
-than after it. The new release's workers (started before traffic switches) handle
-new work while the retired release's workers finish their in-flight jobs. The
-whole drain stays non-blocking — the deploy returns immediately.
+proxied process (a job worker on a shared queue). Rollbridge starts that
+companion's configured stop sequence **as soon as the release is retired**, in
+parallel with the connection drain, rather than after it. Its quiet command or
+signal must make the worker stop accepting new handoffs. The asynchronous
+release drain continues after the deploy response. As described above, current
+Rollbridge does not simultaneously quiesce the handoff service, so this setting
+alone does not prevent overlapping jobs-main scheduling or dispatch ownership.
 
 ```js
-{id: "worker", policy: "companion", command: "…", nonBlockingDrain: true, stopSignal: "SIGINT", gracefulStopMs: 60000}
+{id: "worker", policy: "companion", command: "…", nonBlockingDrain: true, stopSignal: "SIGINT", gracefulStopMs: "indefinite"}
 ```
 
 ### `processes[].restart`

package/docs/nginx.md

@@ -67,10 +67,11 @@ Related Rollbridge timeouts (configured in `rollbridge.js`, not Nginx):
 
 - `proxy.healthTimeoutMs` gates how long a new release has to become healthy
   before a deploy aborts — it does not affect request timeouts.
-- `proxy.drainTimeoutMs` is how long Rollbridge keeps an old release alive for
-  in-flight connections during a deploy. Keep Nginx's `proxy_read_timeout` for
-  WebSocket locations comfortably above it so the front end doesn't cut
-  connections Rollbridge is still draining.
+- `proxy.drainTimeoutMs` is how long Rollbridge keeps an old proxied web process
+  alive for in-flight connections during a deploy. Keep Nginx's
+  `proxy_read_timeout` for WebSocket locations comfortably above it so the front
+  end doesn't cut connections Rollbridge is still draining. Retained jobs
+  generations drain independently and are not stopped when this timeout expires.
 
 ## Forwarded headers
 

package/docs/tensorbuzz-runbook.md

@@ -1,129 +1,95 @@
-# TensorBuzz production runbook
+# TensorBuzz Rollbridge runbook
 
-Operating the TensorBuzz backend under Rollbridge. The production config lives at
-[`examples/tensorbuzz.com.js`](../examples/tensorbuzz.com.js); this runbook
-assumes it is deployed to a stable path (`/etc/rollbridge/tensorbuzz.com.js`
-below) and the daemon runs as a systemd service (see
-[Running under systemd](../README.md#running-under-systemd)). For the general
-Velocious topology and the worker recipe, see [`docs/velocious.md`](velocious.md).
+This runbook defines the required TensorBuzz backend topology. The example at
+[`examples/tensorbuzz.com.js`](../examples/tensorbuzz.com.js) is illustrative;
+verify the deployed consumer config and Rollbridge implementation before
+asserting production compliance.
 
-## Ports
+## Ports and generations
 
-| Port | Process | Notes |
+| Port | Process | Contract |
 | --- | --- | --- |
-| `4500` | Rollbridge proxy | The stable public port. **Nginx proxies the backend host to `127.0.0.1:4500`** — never to a release's web port. |
-| `7330` | `beacon` (`service`) | Fixed; the shared broker every release connects to. |
-| `7331` | `background-jobs-main` (`service`) | Fixed; the job coordinator. |
-| `14500`–`14599` | `web` (`proxied`) | One port per release, allocated per deploy; Rollbridge forwards `4500` here. |
-| (none) | `background-jobs-worker` (`companion`) | A per-release worker; no listening port. |
-
-Control socket: `/tmp/rollbridge-tensorbuzz.sock`.
-
-## Process topology
-
-- **`beacon`** and **`background-jobs-main`** are `service`s: one daemon-wide
-  instance each, on their fixed ports, surviving deploys.
-- **`background-jobs-worker`** is a `companion`: a fresh worker per release,
-  running that release's code, with `gracefulStopMs: 60000` so an in-flight job
-  finishes before `SIGKILL`.
-- **`web`** is the one `proxied` process, health-checked at `/ping` before
-  traffic switches.
-
-Each process waits for its dependencies with `wait-for-it` (`beacon` →
-`background-jobs-main` → `worker`/`web`), so nothing starts talking to Beacon or
-the job coordinator before they listen.
-
-## External services
-
-Rollbridge manages **only the four processes above**. Everything else the
-Velocious app depends on — the database and any other backing services — is
-**provisioned and operated outside Rollbridge**: Rollbridge does not start, stop,
-health-check, or know about them. Configure those connections through the app's
-own environment/config. When such a dependency is down, the `web` process's
-`/ping` health check is what gates a deploy (a release that can't reach its
-database won't pass health and won't go live).
-
-## Deploying
-
-Drive deploys through the CLI (see [`docs/deploy-recipes.md`](deploy-recipes.md)).
-Run **backwards-compatible** migrations before switching traffic, because the old
-and new releases overlap during the drain:
-
-```bash
-release_path=/srv/tensorbuzz/releases/<timestamp>     # prepared by your pipeline
-(cd "$release_path/backend" && npx velocious db:migrate)
-
-rollbridge deploy \
-  --ensure-daemon \
-  --config /etc/rollbridge/tensorbuzz.com.js \
-  --release-path "$release_path" \
-  --revision "$(git -C "$release_path/backend" rev-parse HEAD)"
-```
-
-### Deploy ordering
-
-On `rollbridge deploy`, Rollbridge:
-
-1. starts any missing `service` (`beacon`, `background-jobs-main`);
-2. starts the new release's `background-jobs-worker`, then its `web` process, and
-   health-checks `web` on its `{{port}}`/`/ping`;
-3. switches new traffic to the new `web`;
-4. refreshes the services' restart templates to the new release;
-5. drains the previous release's connections, then stops its `web` and worker.
-
-If the new release fails to start or health-check, **the previous release stays
-active** and the command exits non-zero — so a failed deploy never takes the site
-down.
-
-## Rollback
-
-```bash
-rollbridge rollback --config /etc/rollbridge/tensorbuzz.com.js
-# or a specific retained release:
-rollbridge rollback --config /etc/rollbridge/tensorbuzz.com.js --release-id <id>
-```
-
-Rollback re-runs the deploy flow on a retained release, health-checks it, and
-switches traffic back. Constraints:
-
-- **Migrations are not reverted.** Rollback only manages processes; if a release
-  bumped the schema, rolling code back requires that the old code still works
-  against the new schema — keep migrations backwards-compatible (the same rule as
-  deploys).
-- The target release's on-disk directory must still exist (don't prune it from
-  disk before you might roll back to it).
-- Only releases Rollbridge still retains (`releaseRetention`) can be targeted.
-
-## Day-to-day operations
-
-```bash
-C=/etc/rollbridge/tensorbuzz.com.js
-
-rollbridge status  --config "$C"                 # active release, ports, per-process state
-rollbridge logs    --config "$C" --process web   # recent stdout/stderr of a process
-rollbridge events  --config "$C"                 # deploys, switches, crashes, restarts
-rollbridge doctor  --config "$C"                 # pre-flight: socket, proxy port, state
-rollbridge restart --config "$C" --process background-jobs-worker   # bounce the worker
-```
-
-Restarting `beacon` or `background-jobs-main` bounces a shared broker and briefly
-disrupts everything that depends on it; prefer `deploy`/`rollback` for code
-changes. See [`docs/troubleshooting.md`](troubleshooting.md) for health-check
-failures, port conflicts, stale sockets, crash loops, and stuck draining
-releases.
-
-## Crash recovery
-
-Set [`statePath`](config.md#statepath) in the config to have the daemon persist
-its state. After a daemon crash or reboot, `rollbridge doctor` reports any
-**orphaned** processes still alive from the previous daemon. To clean them up
-before restarting the daemon, run `rollbridge recover` (a dry run that lists
-them), then `rollbridge recover --force` to stop them:
-
-```bash
-rollbridge recover --config /etc/rollbridge/tensorbuzz.com.js          # list leftovers
-rollbridge recover --config /etc/rollbridge/tensorbuzz.com.js --force  # stop them
-```
-
-A machine reboot kills every process, so there are usually no orphans afterward —
-the daemon just starts fresh.
+| `4500` | Rollbridge proxy | Stable public upstream for Nginx. |
+| `7330` | Beacon | Shared persistent service; fixed port is allowed. |
+| `7331`–`7399` | `background-jobs-main` | One allocated endpoint per release generation; never one fixed persistent coordinator. |
+| `14500`–`14599` | web | One proxied endpoint per release. |
+| none | workers | Same-release companions connected only to their generation's jobs-main. |
+
+Each jobs generation contains one jobs-main and its complete worker pool, all
+running the same release code. Several old generations may continue draining
+while a newer generation is active.
+
+## Required deploy order
+
+1. Prepare the candidate release and run backwards-compatible migrations.
+2. Start the candidate jobs-main on a new port, then its worker pool and web
+   process. Health-check web before activation.
+3. Activate the candidate release and switch new traffic.
+4. Retire the previous jobs-main and workers as one generation. Jobs-main stops
+   schedule ownership, new dispatch, and new handoffs; workers stop accepting
+   handoffs.
+5. Return deploy success and release the deploy lock. Do not wait for old jobs,
+   workers, jobs-main, HTTP/WebSocket connections, or other retained services.
+
+The retired jobs-main remains running on its old endpoint with its old workers.
+For accepted handoffs it owns worker connections and heartbeats, lease fencing,
+terminal-report acceptance and acknowledgement, and durable store transitions.
+The old worker/reporting side durably retries terminal reports, tracks
+outstanding report promises, enforces per-job execution timeouts, and owns and
+reaps child runners. Returned or retried work becomes eligible for the new active
+generation and is never redispatched by the retired main. Old workers do not
+reconnect to or transfer their handoffs to the new main.
+
+Old main and workers remain one release generation until every accepted handoff
+settles. Only then, after every old worker exits, may jobs-main exit and
+Rollbridge reap the generation. The referenced release directory stays pinned
+against Rampway cleanup until that point.
+
+## Independent drains
+
+Set the worker companion to `nonBlockingDrain: true` so it quiesces when its
+generation retires. HTTP/WebSocket connection drain continues independently.
+Finishing or timing out the HTTP drain must never stop a still-draining jobs
+generation. A legitimate multi-hour job and generation drain are valid.
+
+Per-job timeouts remain the backstop for genuinely hung jobs. Do not use a short
+worker-shutdown or supervisor timeout as the primary deploy solution; deployment
+has already completed after candidate activation and health.
+
+## Runtime-owner and recovery requirements
+
+Required compliant behavior durably supervises every retired generation after
+the deploy command returns, across later deploys and supervisor/host recovery.
+Runtime-owner or version handoff must preserve or transfer that supervision and
+return once the replacement is healthy. It must not perform full synchronous
+shutdown, kill retained generations, or make the new jobs-main adopt old workers.
+
+Recovery must reconstruct retained generation ownership, endpoints, release
+paths, and process references. Do not treat them as generic orphans to force-stop
+merely because a supervisor restarted. Cleanup becomes eligible only after the
+last retained process exits.
+
+Current Rollbridge does not yet meet those recovery and owner-handoff
+requirements. Its non-blocking release drains last only for the current daemon's
+lifetime; after restart it reports surviving PIDs as advisory, non-adoptable
+orphans, and forced recovery stops them. `--takeover-owner` quiesces and starts
+asynchronous stops for every managed process instead of transferring retained
+generations.
+
+## Operator checks
+
+Use `rollbridge status`, logs, and events to confirm:
+
+- the active jobs generation uses the active release and a unique jobs-main port;
+- every retired generation retains its own jobs-main, workers, endpoint, and
+  pinned release path;
+- the deploy command has returned while long drains continue;
+- no retired jobs-main is dispatching new ordinary queued work;
+- HTTP drain completion or timeout did not stop a jobs generation; and
+- completed generations are reaped and Rampway is told their release references
+  ended so it can release the pins.
+
+Do not restart Beacon or a jobs-main casually. Never use `shutdown`, forced
+orphan recovery, or process signals as a substitute for the normal retained-
+generation lifecycle. See [`docs/velocious.md`](velocious.md) and
+[`docs/workers.md`](workers.md) for the architecture details.

package/docs/troubleshooting.md

@@ -109,12 +109,21 @@ stays active.
 in `state: "draining"` with non-zero `connections` (often `websocket`).
 
 **Diagnose.** Long-lived connections (WebSockets, SSE, streaming responses) keep
-the retired release alive until they close or `proxy.drainTimeoutMs` elapses.
-`status` shows the release's `connections.http`/`connections.websocket` and
-`drainStartedAt`.
-
-**Fix.** Draining ends automatically when those connections close, or after
-`proxy.drainTimeoutMs` (then the release is stopped regardless). Lower
-`proxy.drainTimeoutMs` to force-stop sooner, or make clients reconnect (for
-example, have the front end close idle WebSockets on deploy). Once stopped, the
-release is pruned per `releaseRetention`.
+the retired proxied web process alive until they close or
+`proxy.drainTimeoutMs` elapses. `status` shows the release's
+`connections.http`/`connections.websocket` and `drainStartedAt`. A retained jobs
+generation has an independent lifecycle and may remain after the web drain ends.
+
+**Fix.** The connection drain ends automatically when those connections close,
+or after `proxy.drainTimeoutMs`. Rollbridge then stops the retired proxied web
+process and other connection-dependent processes, including ordinary companions
+with `nonBlockingDrain: false`. Lower the timeout only to shorten
+HTTP/WebSocket retention, or make clients reconnect (for example, have the front
+end close idle WebSockets on deploy). In the documented compliant jobs topology,
+jobs companions use `nonBlockingDrain: true`, so timeout expiry affects only the
+web side and must not stop a still-draining jobs generation.
+
+Reporting release references to Rampway and pinning release directories against
+on-disk cleanup are required future behavior, not implemented today. Current
+Rollbridge `status` and `releaseRetention` govern only its in-memory release
+records and do not fence Rampway cleanup.

package/docs/velocious.md

@@ -1,37 +1,35 @@
 # Velocious deployment guide
 
-A Velocious backend typically runs four kinds of process: **Beacon** (the
-message broker other processes connect to), **background-jobs-main** (the job
-coordinator), **background-jobs-worker** (runs the jobs), and the **web/API**
-server. This guide maps each to a Rollbridge process policy, shows a complete
-`rollbridge.js`, and explains startup ordering and what happens on a deploy.
+A Velocious backend normally runs Beacon, `background-jobs-main`, a
+`background-jobs-worker` pool, and the web/API server. For deploy lifecycle
+purposes, jobs-main and its workers are one release-scoped **jobs generation**.
+This topology is required; a single persistent fixed-port jobs-main is not a
+safe coordinator for workers that may drain across releases.
 
-A production version of this config lives at
-[`examples/tensorbuzz.com.js`](../examples/tensorbuzz.com.js).
+This page states the architecture contract. It does not by itself assert that a
+particular Rollbridge release or consumer production config implements durable
+retired-generation recovery; verify source and config before claiming compliance.
 
 ## Process mapping
 
-| Velocious process | Policy | Why |
+| Velocious process | Rollbridge policy | Lifecycle |
 | --- | --- | --- |
-| `beacon` | `service` | A shared broker the other processes connect to. It should survive deploys and keep a **stable port**, so workers and the web process always reach the same Beacon. |
-| `background-jobs-main` | `service` with `deployStrategy: "handoff"` | The job coordinator. Run it as a handoff service so each release's workers and web process use a same-release coordinator while old releases drain (see [Choosing the jobs-main policy](#choosing-the-jobs-main-policy)). |
-| `background-jobs-worker` | `companion` | Release-scoped: one set of workers per active release, started before the web process and running that release's code. |
-| `web` | `proxied` | Receives external HTTP/WebSocket traffic, is health-checked before traffic switches, and is drained on the next deploy. Exactly one process is `proxied`. |
-
-See [README → Process Policies](../README.md#process-policies) for the full
-semantics of each policy and [`docs/config.md`](config.md) for every field.
+| `beacon` | persistent `service` | Shared broker; it may remain daemon-wide on fixed port `7330`. |
+| `background-jobs-main` | `service` with `deployStrategy: "handoff"` | One endpoint per release; owns worker connections, lease fencing, report acceptance/acknowledgement, and durable store transitions. |
+| `background-jobs-worker` | `companion` with `nonBlockingDrain: true` | Release-scoped pool; executes accepted work, owns child runners and execution timeouts, and durably retries terminal reports while tracking their promises. |
+| `web` | `proxied` | Health-gated active HTTP/WebSocket target with a per-release port. |
 
 ## Example `rollbridge.js`
 
 ```js
-// rollbridge.js
 export default {
   application: "tensorbuzz",
   control: {path: "/tmp/rollbridge-tensorbuzz.sock"},
+  statePath: "/var/lib/rollbridge/tensorbuzz.json",
 
   proxy: {
     host: "127.0.0.1",
-    port: 4500,          // the stable port Nginx points at
+    port: 4500,
     healthPath: "/ping",
     healthTimeoutMs: 30000,
     drainTimeoutMs: 60000,
@@ -39,7 +37,6 @@ export default {
   },
 
   processes: [
-    // Shared broker — one daemon-wide instance on a stable port.
     {
       id: "beacon",
       policy: "service",
@@ -48,8 +45,6 @@ export default {
       command: "npx velocious beacon",
       port: 7330
     },
-
-    // Job coordinator — one release-scoped service instance per deploy.
     {
       id: "background-jobs-main",
       policy: "service",
@@ -63,23 +58,20 @@ export default {
       command: "wait-for-it 127.0.0.1:{{ports.beacon}} --strict -- npx velocious background-jobs-main",
       port: {from: 7331, to: 7399}
     },
-
-    // Workers — one set per release; raise gracefulStopMs to let in-flight
-    // jobs finish during a deploy.
     {
       id: "background-jobs-worker",
       policy: "companion",
+      nonBlockingDrain: true,
       cwd: "{{releasePath}}/backend",
       env: {
         NODE_ENV: "production",
         VELOCIOUS_BEACON_PORT: "{{ports.beacon}}",
         VELOCIOUS_BACKGROUND_JOBS_PORT: "{{ports.background-jobs-main}}"
       },
-      command: "wait-for-it 127.0.0.1:{{ports.beacon}} --strict -- wait-for-it 127.0.0.1:{{ports.background-jobs-main}} --strict -- npx velocious background-jobs-worker",
+      command: "wait-for-it 127.0.0.1:{{ports.background-jobs-main}} --strict -- npx velocious background-jobs-worker",
+      replicas: 4,
       gracefulStopMs: "indefinite"
     },
-
-    // Web/API — the one proxied process.
     {
       id: "web",
       policy: "proxied",
@@ -89,7 +81,7 @@ export default {
         VELOCIOUS_BEACON_PORT: "{{ports.beacon}}",
         VELOCIOUS_BACKGROUND_JOBS_PORT: "{{ports.background-jobs-main}}"
       },
-      command: "wait-for-it 127.0.0.1:{{ports.beacon}} --strict -- wait-for-it 127.0.0.1:{{ports.background-jobs-main}} --strict -- npx velocious server --host 127.0.0.1 --port {{port}}",
+      command: "wait-for-it 127.0.0.1:{{ports.background-jobs-main}} --strict -- npx velocious server --host 127.0.0.1 --port {{port}}",
       port: {from: 14500, to: 14599},
       health: {path: "/ping", timeoutMs: 30000, intervalMs: 500}
     }
@@ -97,146 +89,77 @@ export default {
 }
 ```
 
-## Wiring processes together
-
-Beacon gets a **fixed** port (`7330`) because it is a persistent `service` — a
-stable port lets every release's processes find the shared broker.
-`background-jobs-main` gets a **range** (`{from: 7331, to: 7399}`) because it is a
-handoff service: Rollbridge allocates a new port per release so old workers keep
-talking to the old coordinator while new workers and web use the new one. The
-proxied `web` process also gets a **range** (`{from: 14500, to: 14599}`) so old
-and new web releases can run side by side during the drain.
-
-Cross-reference ports with `{{ports.<id>}}` and pass them to Velocious through
-`env`. Rollbridge also injects `ROLLBRIDGE_<ID>_PORT` for every process (e.g.
-`ROLLBRIDGE_BACKGROUND_JOBS_MAIN_PORT`), so you can read ports from the
-environment instead of templating if you prefer — see
-[`docs/config.md`](config.md#injected-environment-variables).
-
-### Startup ordering
-
-Only the `proxied` process is health-checked, so dependent processes must wait
-for their dependencies themselves. Two mechanisms combine:
-
-1. **Policy ordering.** On each deploy Rollbridge starts handoff `service`s
-   first, then the release's `companion`s, then the `proxied` process (see
-   [README → Deploy ordering](../README.md#deploy-ordering)).
-2. **Readiness gating.** `wait-for-it 127.0.0.1:{{ports.beacon}} --strict -- …`
-   blocks the command until Beacon's port accepts connections, so
-   `background-jobs-main`, the worker, and `web` don't start talking to Beacon
-   before it is listening. `wait-for-it` is a small standalone script (install it
-   on the host); any equivalent port-wait works.
-
-## Deploying
-
-Drive deploys through the Rollbridge CLI — Rollbridge ships no deploy-tool
-plugins (see [`docs/deploy-recipes.md`](deploy-recipes.md) for shell/CI/Capistrano
-recipes). The minimal step after a release directory is prepared:
-
-```bash
-release_path=/srv/tensorbuzz/releases/20260523120000  # prepared by your pipeline
-
-# Run backwards-compatible migrations BEFORE switching traffic: the old and new
-# web releases overlap during the drain.
-(cd "$release_path/backend" && npx velocious db:migrate)
-
-rollbridge deploy \
-  --ensure-daemon \
-  --config /etc/rollbridge/rollbridge.js \
-  --release-path "$release_path" \
-  --revision "$(git -C "$release_path/backend" rev-parse HEAD)"
-```
-
-`rollbridge deploy` starts the new release's worker and web process,
-health-checks `web` on its `{{port}}`/`/ping`, switches traffic, then drains and
-stops the previous release. It exits non-zero (leaving the previous release
-active) if the new release fails to start or health-check, so a failed deploy
-never promotes a broken release.
-
-## Background jobs across a deploy
-
-The worker is a `companion`, so each release runs its own workers:
-
-- On deploy, the **new** release's workers start (running the new code) before
-  traffic switches; the **old** release's workers are stopped when that release
-  is drained and retired — the worker's `stopSignal`, then `SIGKILL` after
-  `gracefulStopMs`.
-- Set `stopSignal` to the signal your worker drains on and `gracefulStopMs` to at
-  least your longest in-flight job. Use `gracefulStopMs: "indefinite"` when the
-  worker can safely drain until it exits on its own. Set `replicas` to run a pool
-  of workers.
-
-See [`docs/workers.md`](workers.md) for the full safe background-job deployment
-pattern (companion + `replicas` + `stopSignal`/`lifecycle` hooks +
-`gracefulStopMs`), the old/new worker overlap, and `nonBlockingDrain` to start the
-old workers' drain immediately when a release is retired.
-
-### Worker recipe
-
-A complete `background-jobs-worker` entry that runs a pool and finishes in-flight
-jobs across a deploy:
-
-```js
-{
-  id: "background-jobs-worker",
-  policy: "companion",
-  cwd: "{{releasePath}}/backend",
-  env: {
-    NODE_ENV: "production",
-    VELOCIOUS_ENV: "production",
-    VELOCIOUS_BEACON_PORT: "{{ports.beacon}}",
-    VELOCIOUS_BACKGROUND_JOBS_PORT: "{{ports.background-jobs-main}}"
-  },
-  command: "wait-for-it 127.0.0.1:{{ports.beacon}} --strict -- wait-for-it 127.0.0.1:{{ports.background-jobs-main}} --strict -- npx velocious background-jobs-worker",
-  replicas: 4,
-  gracefulStopMs: "indefinite"
-}
-```
-
-- `replicas: 4` runs four worker instances (`background-jobs-worker#0` … `#3`),
-  each with `ROLLBRIDGE_REPLICA_INDEX`/`ROLLBRIDGE_REPLICA_COUNT` if you shard work.
-- On deploy the new release's workers start before traffic switches; the old
-  release's workers receive `SIGTERM` (the default `stopSignal`) when the old
-  release is retired, then wait to exit. With `gracefulStopMs: "indefinite"`,
-  Rollbridge does not send a `SIGKILL` fallback.
-
-If your worker quiesces on a command or a non-default signal, add a `lifecycle`
-block — Rollbridge runs `quietCommand`, drains for up to `drainTimeoutMs`, then
-stops. For example, send a quiet signal to the worker's process group before the
-drain:
-
-```js
-lifecycle: {quietCommand: "kill -TSTP -$ROLLBRIDGE_PID", drainTimeoutMs: 60000}
-```
-
-### Choosing the jobs-main policy
-
-`background-jobs-main` coordinates workers, so choose its lifecycle deliberately:
-
-- **`service` with `deployStrategy: "handoff"`** — starts one coordinator per
-  release on a port from a range. New workers and web get the new release's port;
-  old workers keep the old release's port while they drain. This is the safest
-  default when the coordinator should run the same code version as its workers.
-- **`service` with the default `deployStrategy: "persistent"`** — keeps one
-  daemon-wide coordinator on a stable port. Workers from every release talk to the
-  same coordinator, but it keeps running the release it was started from and only
-  adopts the latest template if it restarts later.
-- **`singleton`** — stops the old instance and then starts the new one on each
-  deploy, so it always runs the latest release's code and two copies never
-  overlap. The trade-off: a brief coordination gap while it restarts.
-
-Beacon is a broker rather than code that changes per release, so `service` is
-almost always right for it.
-
-## Verifying
-
-After a deploy, `rollbridge status` should show `beacon` as a long-lived service
-with an unchanged port, `background-jobs-main` as the active release's handoff
-service, one `background-jobs-worker` for the active release, and the `web`
-process `proxied` with its connection counts. Use
-[`rollbridge logs --process <id>`](cli.md) to read recent output from any
-process, and [`docs/troubleshooting.md`](troubleshooting.md) for health-check,
-port, and draining problems.
-
-For the front end, point Nginx at the stable `proxy.port` (here `4500`), never at
-a release's web port — see [`docs/nginx.md`](nginx.md).
+Beacon keeps its fixed port because it is intentionally shared. Jobs-main uses a
+range because every release gets its own coordinator. Same-release
+`{{ports.background-jobs-main}}` expansion ensures that old workers retain the
+old endpoint while candidate workers use the candidate endpoint.
+
+## Deploy and activation
+
+Run backwards-compatible migrations before activation, then invoke
+`rollbridge deploy` with the prepared release. Rollbridge starts the candidate
+jobs-main, its complete worker pool, and the web process before health gating and
+activation. A candidate startup or health failure leaves the previous release
+active.
+
+After successful activation, the deploy returns without waiting for any retired
+generation or HTTP/WebSocket connection to finish. The old and new release code
+may therefore overlap for hours. Keep schema, queue payloads, and external side
+effects compatible across that window.
+
+## Retired jobs-generation contract
+
+Retire jobs-main and its workers as one unit:
+
+- jobs-main relinquishes recurring schedule ownership and stops dispatching
+  queued work or making new worker handoffs;
+- workers stop advertising or accepting new handoffs;
+- jobs-main remains running on the old endpoint and owns worker connections and
+  heartbeats, lease fencing, terminal-report acceptance and acknowledgement, and
+  durable store transitions for its accepted handoffs;
+- the old worker/reporting side durably retries terminal reports, tracks
+  outstanding report promises, enforces per-job execution timeouts, and owns and
+  reaps child runners;
+- a job returned or retried to the shared queue becomes eligible for the new
+  active generation, and the retired main never dispatches it again;
+- old workers never reconnect to or transfer their handoffs to the new jobs-main;
+- old main and workers remain one release generation until all accepted work
+  settles; jobs-main exits only after that and after all workers drain and exit,
+  after which Rollbridge may reap the generation.
+
+HTTP/WebSocket and jobs drains are independent. `proxy.drainTimeoutMs` bounds the
+connection drain only; reaching it must not stop a still-draining jobs generation.
+`nonBlockingDrain: true` starts worker quiescence at retirement rather than after
+the HTTP drain.
+
+The process supervisor must retain multiple old generations concurrently,
+persist their ownership across later deploys and supervisor/host recovery, and
+report every referenced release directory so Rampway can pin it against cleanup.
+A runtime owner/version handoff preserves or transfers that supervision and
+returns after the replacement is healthy; it is not a full synchronous shutdown.
+
+These are target requirements. Current Rollbridge drains releases
+asynchronously only while the same daemon remains alive, cannot re-adopt
+surviving PIDs after restart, and stops rather than transfers all managed
+processes during `--takeover-owner`; see [`docs/cli.md`](cli.md#daemon).
+
+## Timeouts
+
+Velocious per-job timeouts remain responsible for genuinely hung work. Rollbridge
+stop signals, lifecycle hooks, and graceful-stop bounds remain emergency/process
+controls. Do not use a short normal worker-shutdown timeout to make deployment
+complete: deployment is already complete after healthy activation, and a
+legitimate hours-long job makes an hours-long generation drain valid.
+
+## Verification
+
+After a deploy, status must be able to show the active generation and every
+retired generation still draining, including each jobs-main endpoint, worker
+pool, release path, and retention reference. Beacon may keep `7330`; active and
+retired jobs-main instances must use different ports. Confirm that the deploy
+command has returned even while retained generations remain and that an HTTP
+drain timeout does not terminate them.
+
+See [`docs/workers.md`](workers.md) for the focused lifecycle,
+[`docs/config.md`](config.md) for configuration fields, and
+[`docs/tensorbuzz-runbook.md`](tensorbuzz-runbook.md) for the consumer runbook.

package/docs/workers.md

@@ -1,117 +1,95 @@
-# Background-job worker deployment
+# Background-job generation deployment
 
-This guide covers deploying background-job workers (or any non-HTTP worker pool)
-with Rollbridge so that in-flight jobs finish across a deploy. It uses features
-that exist today; the command-based lifecycle hooks mentioned at the end are
-still on the roadmap.
+This is the required Rollbridge lifecycle for a background-jobs runtime. A
+generation is release-scoped and contains its own `background-jobs-main` plus
+its worker pool. It is not just a set of workers attached to one persistent
+coordinator.
 
-## Run workers as a `companion`
+## Process topology
 
-Give each worker the `companion` policy. Companions are **release-scoped**: every
-release starts its own workers running that release's code, and a release's
-workers are stopped only when that release is retired (drained) after a newer
-release takes over. They start **before** the `proxied` web process, so they're
-ready before traffic switches.
+Configure jobs-main as a handoff `service` on a multi-port range and the workers
+as same-release `companion`s. Beacon may remain a shared persistent service on a
+fixed port such as `7330`.
 
 ```js
 {
-  id: "worker",
-  policy: "companion",
-  cwd: "{{releasePath}}",
-  command: "npx velocious background-jobs-worker"
-}
-```
-
-## Scale the pool with `replicas`
-
-Set `replicas` to run several identical workers (a port-less companion only).
-Each instance runs as `worker#0`, `worker#1`, … and gets
-`ROLLBRIDGE_REPLICA_INDEX` / `ROLLBRIDGE_REPLICA_COUNT` (and `{{replicaIndex}}` /
-`{{replicaCount}}`), so an instance can claim a distinct shard, queue, or lock:
-
-```js
-{id: "worker", policy: "companion", command: "npx velocious background-jobs-worker", replicas: 4}
-```
-
-Restart the pool with `rollbridge restart --process worker` (all replicas) or a
-single instance with `rollbridge restart --process worker#0`.
-
-## Finish in-flight jobs on stop (`stopSignal` + `gracefulStopMs`)
-
-When Rollbridge stops a worker — during a deploy's drain, a `rollbridge restart`,
-or shutdown — it sends the worker's **`stopSignal`** (default `SIGTERM`), waits up
-to **`gracefulStopMs`**, then `SIGKILL`s it if it hasn't exited. That window is
-the worker's chance to finish its current job and exit cleanly.
-
-- Set `stopSignal` to the signal your worker quiets/drains on. Many job runners
-  finish the current job and exit on `SIGTERM` (the default); some use `SIGINT`
-  or `SIGQUIT`. Use the one your worker treats as "drain and exit".
-- Set `gracefulStopMs` to at least your longest job's duration, so a job in
-  progress is not cut off by the `SIGKILL` fallback. Use `"indefinite"` only for
-  workers that are safe to leave draining until they exit on their own.
-
-```js
+  id: "background-jobs-main",
+  policy: "service",
+  deployStrategy: "handoff",
+  command: "npx velocious background-jobs-main",
+  port: {from: 7331, to: 7399}
+},
 {
-  id: "worker",
+  id: "background-jobs-worker",
   policy: "companion",
+  env: {VELOCIOUS_BACKGROUND_JOBS_PORT: "{{ports.background-jobs-main}}"},
   command: "npx velocious background-jobs-worker",
   replicas: 4,
-  stopSignal: "SIGTERM",
+  nonBlockingDrain: true,
   gracefulStopMs: "indefinite"
 }
 ```
 
-## What happens across a deploy
-
-1. The new release's workers start (running the **new** code) before traffic
-   switches to the new web process.
-2. Both old and new workers run while the previous release drains, so **both
-   code versions consume the shared queue at once.** Keep job code
-   backwards-compatible across a deploy — the same rule as database migrations.
-3. When the previous release is retired (its HTTP/WebSocket connections close or
-   `proxy.drainTimeoutMs` elapses), its workers are stopped: `stopSignal`, then
-   `SIGKILL` after `gracefulStopMs`.
-
-Because old workers are retired on the release's **connection** drain (not on
-their own job queue draining), a job still running when the release is retired
-gets only the `gracefulStopMs` window to finish, unless `gracefulStopMs` is
-`"indefinite"`. Keep jobs **idempotent and safe to retry** so a job interrupted
-at a finite `SIGKILL` fallback can run again.
-
-## Command-based lifecycle hooks
-
-For workers that quiesce or drain via a command rather than a single signal, set
-a `lifecycle` block. When Rollbridge gracefully stops the worker it runs
-`quietCommand` (stop accepting new work), then drains (`drainCommand`, or waits up
-to `drainTimeoutMs` for the worker to exit), then `stopCommand` or `stopSignal`,
-then `SIGKILL` after `gracefulStopMs`. Each hook gets `ROLLBRIDGE_PID` and is
-bounded by a timeout, so a slow hook can't wedge a deploy.
-
-```js
-{
-  id: "worker",
-  policy: "companion",
-  command: "npx velocious background-jobs-worker",
-  replicas: 4,
-  lifecycle: {quietCommand: "kill -TSTP -$ROLLBRIDGE_PID", drainTimeoutMs: 60000}
-}
-```
-
-See [`docs/config.md`](config.md#processeslifecycle) for the hook reference.
-
-## Non-blocking drain
-
-By default a retired release's workers are stopped only after the proxied
-process's connections have drained. Set `nonBlockingDrain: true` on a worker
-companion whose work is independent of the web process (a job worker on a shared
-queue) to start its graceful stop **immediately** when the release is retired —
-in parallel with the connection drain. The new release's workers handle new work
-while the old workers finish their in-flight jobs:
-
-```js
-{id: "worker", policy: "companion", command: "…", nonBlockingDrain: true, gracefulStopMs: "indefinite"}
-```
-
-See [`docs/config.md`](config.md) for `stopSignal`, `replicas`, and
-`gracefulStopMs`, and [`docs/velocious.md`](velocious.md) for a full Velocious
-deployment (Beacon, jobs-main, workers, web) example.
+Each worker receives its generation's jobs-main port. Old workers keep that port
+for their entire lifetime; normal deploy draining never hands them to, or lets
+them reconnect to, the new jobs-main. `replicas` scales the pool as
+`background-jobs-worker#0`, `#1`, and so on.
+
+## Deploy and retirement sequence
+
+1. Before activation, Rollbridge starts the candidate release's jobs-main and
+   complete worker pool, then starts and health-checks the candidate web process.
+2. Activation switches new web traffic and makes the candidate jobs generation
+   active.
+3. The previous jobs generation retires as one unit. Its jobs-main stops schedule
+   ownership, new queue dispatch, and new ordinary worker handoffs. Its workers
+   stop accepting handoffs.
+4. The old jobs-main stays running with its old workers. It continues owning
+   their connections and heartbeats, lease fencing, terminal-report acceptance
+   and acknowledgement, and durable store transitions. The old worker/reporting
+   side durably retries terminal reports, tracks outstanding report promises,
+   enforces per-job execution timeouts, and owns and reaps child runners.
+5. Work returned or retried to the shared queue becomes eligible for the new
+   active generation. The retired main never dispatches it again.
+6. The old main and workers remain one release generation until every accepted
+   handoff settles. Only then, after every old worker drains and exits, may the
+   old jobs-main exit. Rollbridge then reaps the generation and reports that its
+   release reference ended so Rampway can release the retention pin.
+
+Old and new generations may overlap for hours, each running its own release code
+and jobs-main endpoint. Multiple retired generations may drain concurrently.
+
+## Deploy completion is independent
+
+The deploy succeeds when the candidate release is activated and healthy. The
+command and deploy lock do not wait for old jobs generations, workers, jobs,
+HTTP/WebSocket connections, or other retained services to finish. The required
+supervisor contract durably retains generations after the command returns and
+across later deploys and supervisor/host recovery. Every referenced release
+directory must be reported to Rampway and stays pinned against cleanup until the
+last retained process exits. Current Rollbridge provides asynchronous draining
+only for the lifetime of the current daemon; see the current-behavior notes in
+[`docs/config.md`](config.md#processesdeploystrategy) and
+[`docs/cli.md`](cli.md#deploy).
+
+HTTP/WebSocket drain and jobs drain are independent. Set `nonBlockingDrain: true`
+so workers stop accepting new handoffs when retirement starts rather than after
+the connection drain. Closing or timing out HTTP connections must never kill a
+still-draining jobs-main or worker pool.
+
+## Timeouts and failures
+
+`stopSignal`, `lifecycle`, and `gracefulStopMs` remain useful process-stop tools,
+and the jobs framework should enforce per-job timeouts for genuinely hung work.
+They are not the primary deployment solution. A legitimate multi-hour job makes
+a multi-hour generation drain valid; do not turn a normal worker-shutdown timeout
+into the deploy deadline.
+
+If a worker connection is actually lost, its old jobs-main applies lease fencing
+and the durable store transitions that make work eligible to return or retry.
+Returned work may then run in the active generation. Normal retirement does not
+simulate a disconnect and does not make a new jobs-main adopt the old worker's
+handoffs.
+
+See [`docs/velocious.md`](velocious.md) for the complete topology and
+[`docs/config.md`](config.md#processesdeploystrategy) for handoff-service fields.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "rollbridge",
-  "version": "0.1.24",
+  "version": "0.1.26",
   "description": "Zero-downtime process supervisor and local traffic switcher for deploy-managed apps.",
   "keywords": [
     "deploy",