rollbridge 0.1.23 → 0.1.25

package/AGENTS.md

@@ -0,0 +1,55 @@
+# Rollbridge contributor guidance
+
+Rollbridge is the process supervisor and local traffic switcher. Keep deployment
+tool concerns outside this repository: Rampway owns activation transactions,
+deploy locks, release-retention metadata, and on-disk cleanup; Rollbridge owns
+process lifecycle, retained generations, ports, and recovery state.
+
+## Background-jobs lifecycle invariant
+
+- One runtime generation is one release-scoped `background-jobs-main` plus its
+  worker pool. Start the complete candidate generation before activation.
+- After activation, retire the old generation as one unit. Its main stops
+  schedules, new dispatch, and new ordinary worker handoffs; its workers stop
+  accepting handoffs. The old main remains running with those workers and owns
+  their connections, lease fencing, report acceptance and acknowledgement, and
+  durable store transitions. The worker/reporting side durably retries terminal
+  reports, tracks outstanding report promises, enforces per-job execution
+  timeouts, and owns and reaps child runners. Main and workers remain one release
+  generation until every accepted handoff settles.
+- Returned or retried work may be dispatched by the new active generation. A
+  retired main never dispatches it again. Old workers never reconnect or hand
+  over to the new main during a normal deploy.
+- Jobs generations may overlap for hours on release-scoped endpoints. Beacon may
+  remain shared on `7330`; jobs-main must use a per-release port range.
+- Deploy success is candidate activation and health. Deploy completion must not
+  wait for retired jobs generations, workers, jobs, HTTP/WebSocket connections,
+  or other retained services. HTTP and jobs drains are independent; an HTTP
+  drain finishing or timing out must not stop a live jobs generation.
+- Persist and recover retired-generation ownership across later deploys and
+  supervisor/host recovery. Multiple generations may drain concurrently. Report
+  release references so the deployment tool can keep them pinned until no
+  retained process uses them.
+- Runtime-owner replacement transfers or preserves this durable supervision and
+  returns once the replacement is healthy. It is never a synchronous full
+  shutdown. Per-job timeouts remain valid, but a normal worker-shutdown timeout
+  is not the deploy solution; legitimate multi-hour drains are valid.
+
+Documentation must distinguish required architecture from behavior not yet
+implemented. Do not claim production compliance when source/config still uses a
+fixed jobs-main, worker adoption by a new main, destructive orphan recovery, or
+synchronous cleanup.
+
+## Validation and publication
+
+The project is ESM JavaScript with JSDoc type checking. Package scripts are the
+source of truth: `npm run typecheck`, `npm run lint`, `npm test`, and the combined
+`npm run all-checks`. Run focused checks for the files changed; documentation-only
+work requires at least `git diff --check` plus any existing repository-owned
+Markdown/link check, without installing dependencies.
+
+Work on a feature branch and open a normal pull request against `master`; never
+push feature work directly to `master`. Releases are maintainer-only and use
+`npm run release:patch` from an up-to-date clean default branch after
+`npm run all-checks`; do not edit package versions or publish during ordinary PR
+work. See `docs/releasing.md` for the complete release checklist.

package/README.md

@@ -2,7 +2,22 @@
 
 Rollbridge is a Node.js process supervisor and local traffic switcher for zero-downtime deploys.
 
-Nginx points at one stable Rollbridge proxy port. Deploy tooling asks Rollbridge to start a new release, health-check it, switch new traffic to it, and drain old HTTP/WebSocket connections before stopping the previous release.
+Nginx points at one stable Rollbridge proxy port. Deploy tooling asks Rollbridge to start a new release, health-check it, and switch new traffic to it. Retirement then continues asynchronously: HTTP/WebSocket connections and retained process generations drain independently after the deploy command returns.
+
+> **Required jobs-generation contract:** a release-scoped background-jobs
+> runtime is its own `background-jobs-main` plus worker pool. Rollbridge starts a
+> complete candidate generation before activation. After activation the retired
+> main stops schedules, new dispatch, and new worker handoffs, but remains with
+> its old workers to supervise every handoff it already made until all jobs and
+> workers settle. Old workers never move to the new main. Generations may overlap
+> for hours on separate ports, and neither deploy completion nor HTTP drain
+> completion waits for or kills them. See [Background-job worker
+> deployment](docs/workers.md) and the [Velocious deployment
+> guide](docs/velocious.md).
+
+This is the required architecture, not evidence that every released runtime or
+consumer config already implements durable recovery and release-reference
+reporting; verify source and config before claiming compliance.
 
 ## Install
 
@@ -47,23 +62,28 @@ export default {
   processes: [
     {
       id: "beacon",
-      policy: "companion",
+      policy: "service",
       cwd: "{{releasePath}}",
       command: "env VELOCIOUS_BEACON_PORT={{port}} npx velocious beacon",
-      port: {from: 17330, to: 17399}
+      port: 7330
     },
     {
       id: "background-jobs-worker",
       policy: "companion",
       cwd: "{{releasePath}}",
+      env: {VELOCIOUS_BACKGROUND_JOBS_PORT: "{{ports.background-jobs-main}}"},
       command: "npx velocious background-jobs-worker",
+      nonBlockingDrain: true,
+      gracefulStopMs: "indefinite",
       outputLines: 200
     },
     {
       id: "background-jobs-main",
       policy: "service",
+      deployStrategy: "handoff",
       cwd: "{{releasePath}}",
-      command: "npx velocious background-jobs-main"
+      command: "env VELOCIOUS_BACKGROUND_JOBS_PORT={{port}} npx velocious background-jobs-main",
+      port: {from: 7331, to: 7399}
     },
     {
       id: "web",
@@ -118,10 +138,11 @@ See [`docs/config.md`](docs/config.md#processesmemory).
 memory: {limitBytes: 536870912, warnBytes: 402653184, checkIntervalMs: 5000}
 ```
 
-Set a process's `stopSignal` (default `"SIGTERM"`) to the signal it quiets on, so
-a worker finishes its in-flight work before exiting. Rollbridge sends `stopSignal`
-to gracefully stop the process and `SIGKILL`s it only if it hasn't exited within
-`gracefulStopMs`. For example, a job worker that drains on `SIGINT`:
+Set a process's `stopSignal` (default `"SIGTERM"`) to the signal it quiets on.
+This is a process-level stop mechanism, not the primary deploy-completion
+mechanism for a release-scoped jobs generation. A retired generation may remain
+for hours, and its old jobs-main must stay available until its workers finish.
+For example, a generic worker that drains on `SIGINT`:
 
 ```js
 {id: "worker", policy: "companion", command: "…", stopSignal: "SIGINT", gracefulStopMs: 60000}
@@ -138,18 +159,19 @@ queue. See [`docs/config.md`](docs/config.md#processesreplicas).
 {id: "worker", policy: "companion", command: "npx velocious background-jobs-worker", replicas: 4}
 ```
 
-For workers that quiesce or drain via a command, set a `lifecycle` block —
+For generic workers that quiesce or drain via a command, set a `lifecycle` block —
 Rollbridge runs `quietCommand`, then drains (`drainCommand`/`drainTimeoutMs`),
 then `stopCommand`/`stopSignal`, then `SIGKILL` after `gracefulStopMs` when
 gracefully stopping the process. Each hook is bounded so it can't wedge a stop.
 
-Set `nonBlockingDrain: true` on a worker companion to start its graceful stop the
-moment its release is retired — in parallel with the proxied connection drain,
-not after it — so new workers handle new work while the old workers finish theirs.
+Set `nonBlockingDrain: true` on a jobs worker companion so it stops accepting new
+handoffs as soon as its release retires, independently of the proxied connection
+drain. Its release-scoped handoff jobs-main remains running to supervise existing
+handoffs and exits only after the worker pool has drained.
 
-See [`docs/workers.md`](docs/workers.md) for the full safe background-job worker
-deployment pattern — companion policy, `replicas`, and finishing in-flight jobs
-on deploy with `stopSignal`/`lifecycle` + `gracefulStopMs`.
+See [`docs/workers.md`](docs/workers.md) for the full release-generation
+deployment pattern: a handoff `background-jobs-main`, its companion worker pool,
+independent quiescence, and durable supervision while retained generations drain.
 
 Set `releaseRetention` to bound how many stopped (drained) releases the daemon
 keeps in memory and reports in `status`. `keep` (default `10`) retains the most
@@ -270,8 +292,9 @@ A release-scoped helper (for example a background worker bound to one release).
 It starts **before** the proxied process in the same release, so release-local
 dependencies are ready before the health check, and it is auto-restarted while
 its release is active. Each release gets its own companions; a release's
-companions stop when that release is drained and retired after a newer release
-takes over.
+companions normally stop when that release is drained and retired after a newer
+release takes over. A jobs worker configured with `nonBlockingDrain` instead
+quiesces at retirement and drains independently of HTTP/WebSocket connections.
 
 ```js
 {
@@ -279,7 +302,8 @@ takes over.
   policy: "companion",
   cwd: "{{releasePath}}",
   command: "npx velocious background-jobs-worker",
-  gracefulStopMs: 60000
+  nonBlockingDrain: true,
+  gracefulStopMs: "indefinite"
 }
 ```
 
@@ -301,36 +325,36 @@ new copies simultaneously during a deploy would be unsafe.
 
 ### `service`
 
-A daemon-wide broker that should outlive individual releases — for example
-Velocious Beacon or `background-jobs-main`. Rollbridge starts it once (before
-release processes that depend on it), keeps it running across deploys, and gives
-it a stable port that does not change between releases. After each successful
-deploy its restart template is refreshed to the latest release, so if it crashes
-it restarts from the newest good release. It keeps restarting until the daemon
-shuts down.
+A service can be daemon-wide and persistent, or release-scoped with
+`deployStrategy: "handoff"`. Velocious Beacon is normally persistent on a stable
+port. `background-jobs-main` is not: it must be a handoff service with one port
+per release so old workers keep their old coordinator while new workers use the
+candidate coordinator.
 
 ```js
 {
   id: "background-jobs-main",
   policy: "service",
+  deployStrategy: "handoff",
   cwd: "{{releasePath}}",
   command: "npx velocious background-jobs-main",
-  port: 7331
+  port: {from: 7331, to: 7399}
 }
 ```
 
 ### Deploy ordering
 
-On `rollbridge deploy`, Rollbridge:
+On `rollbridge deploy`, the required ordering is:
 
-1. starts any `service` that is not already running;
+1. starts any missing persistent service and the candidate's handoff services;
 2. starts the new release's `companion`s, then its `proxied` process, and
    health-checks the proxied process;
 3. switches new traffic to the new release;
-4. refreshes each `service`'s restart template to the new release;
+4. marks the previous jobs-main and worker pool retired as one generation;
 5. replaces `singleton`s (stops the old one, then starts the new one);
-6. drains the previous release's connections, then stops its `proxied` and
-   `companion` processes.
+6. returns success without waiting for the previous generation or its independent
+   HTTP/WebSocket drain; Rollbridge supervises all retained drains in the
+   background and reaps each generation only after its handoffs and workers end.
 
 If the new release fails to start or health-check, the previous release stays
 active and any service started during this deploy is rolled back.
@@ -413,13 +437,13 @@ rollbridge daemon --config /srv/ticket-server/rollbridge.js \
 ```
 
 External supervisors that need to replace a foreground owner without waiting
-for its workers to drain add `--takeover-owner`. The candidate starts and
-health-checks the exact release first. Only then does it retire the accepted
-owner's proxy/control listeners; the retired daemon keeps its already-accepted
-workers until their ordinary drain completes while the attested replacement
-binds the stable listeners. Candidate bootstrap failure leaves the accepted
-owner untouched. This is opt-in; ordinary daemon bootstrap and `shutdown` keep
-their existing behavior.
+for retained generations to drain add `--takeover-owner`. The candidate starts
+and health-checks the exact release first. Only then does it retire the accepted
+owner's proxy/control listeners. Durable supervision of old jobs-main/worker
+generations must be preserved or transferred while the attested replacement
+binds the stable listeners; replacement does not mean full synchronous shutdown.
+Candidate bootstrap failure leaves the accepted owner untouched. This is opt-in;
+ordinary daemon bootstrap and `shutdown` keep their existing behavior.
 
 The four bootstrap inputs are all-or-nothing and use absolute config/release
 paths. Rollbridge binds its proxy, activates the release through the normal
@@ -600,7 +624,7 @@ daemon holds its log file open, so logrotate needs `copytruncate`.
 
 ## Deployment Notes
 
-Run migrations before `rollbridge deploy`, and keep migrations backwards-compatible while old and new web releases overlap. For stable local brokers such as Velocious Beacon or `background-jobs-main`, use `service` when the process should survive deploys and restart from the latest successful release if it crashes.
+Run migrations before `rollbridge deploy`, and keep migrations backwards-compatible while old and new web and jobs generations overlap. Velocious Beacon may be a persistent fixed-port `service`; configure `background-jobs-main` as a release-scoped handoff service on a port range. A normal deploy may leave several retired generations draining concurrently.
 
 See [`docs/deploy-recipes.md`](docs/deploy-recipes.md) for ready-to-use shell, CI, and Capistrano recipes that drive Rollbridge through its CLI, and [`docs/troubleshooting.md`](docs/troubleshooting.md) for diagnosing health-check failures, port conflicts, stale sockets, crash loops, and stuck draining releases.
 

package/TODO.md

@@ -30,11 +30,12 @@ This roadmap tracks planned Rollbridge features and documentation. Rollbridge sh
   - [x] Distinguish crash restarts, deploy replacements, manual restarts, and memory restarts in status/events. (Per-process `lastStartReason` + a `reason` on the `process started` event; the `memory` reason is wired and fires once memory supervision restarts a process.)
   - [x] Add a `restart` CLI command for a single process, a policy group, or all non-proxied workers.
   - [x] Keep restart behavior safe for job workers by using lifecycle hooks before termination. (Manual restart, memory restart, and deploy-drain stops all run the `lifecycle` hooks via `stop()`.)
-- [x] Graceful job-worker lifecycle.
+- [x] Graceful process-stop controls for job workers.
   - [x] Add generic lifecycle hooks such as `quietCommand`, `drainCommand`, `drainTimeoutMs`, and `stopCommand` (per-process `lifecycle`).
   - [x] Support signal-only lifecycle steps for workers that can quiet on a Unix signal. (Per-process `stopSignal`; sent before the `SIGKILL`-after-`gracefulStopMs` fallback.)
-  - [x] Add a non-blocking drain mode so new workers can start while old workers finish running jobs (per-process `nonBlockingDrain`; drains the worker in parallel with the connection drain).
-  - [x] Document a Velocious background-jobs-worker recipe once the lifecycle contract is implemented (`docs/velocious.md` → Worker recipe).
+  - [x] Add a non-blocking drain mode so a worker can quiesce at release retirement independently of the HTTP/WebSocket connection drain (`nonBlockingDrain`). This control alone does not provide durable retired-generation supervision.
+  - [x] Document the required Velocious release-generation contract (`docs/velocious.md` and `docs/workers.md`) without treating documentation as proof that the runtime implements it.
+  - [ ] Implement and verify durable release-scoped jobs-main retirement: owned-handoff supervision, recovery across daemon/host replacement, multiple concurrent retired generations, and release-reference reporting for cleanup pins.
 - [x] Replicas and stable worker indexes. (Supported on port-less `companion` processes; `proxied`/`singleton`/ported processes stay single.)
   - [x] Allow one process config to start multiple replicas (`replicas`, companion-only for now).
   - [x] Expose `ROLLBRIDGE_REPLICA_INDEX`, replica count, and per-replica template context (`{{replicaIndex}}`/`{{replicaCount}}`).
@@ -91,7 +92,7 @@ This roadmap tracks planned Rollbridge features and documentation. Rollbridge sh
 - [x] Write a CLI reference for `daemon`, `ensure-daemon`, `deploy`, `status`, `stop`, `shutdown`, and future commands (`docs/cli.md`).
 - [x] Expand process policy docs with deployment examples for `proxied`, `companion`, `singleton`, and `service`.
 - [x] Document memory checks and auto-restart behavior after the feature lands (`docs/config.md` → `processes[].memory`).
-- [x] Document safe background-job deployment patterns (`docs/workers.md`: companion + `replicas` + `stopSignal` + `gracefulStopMs`, old/new worker overlap).
+- [x] Document the required background-job generation pattern (`docs/workers.md`: release-scoped jobs-main + worker pool, independent quiescence, durable retained supervision, and deploy completion that does not wait for drains).
 - [x] Document worker lifecycle hooks (`docs/config.md` → `processes[].lifecycle`, `docs/workers.md`).
 - [x] Add a Velocious deployment guide with Beacon, background-jobs-main, background-jobs-worker, and web process examples (`docs/velocious.md`).
 - [x] Add an Nginx guide with WebSocket headers, timeouts, and common failure modes (`docs/nginx.md`).

package/docs/cli.md

@@ -66,10 +66,16 @@ and waits for control-socket deployments.
 
 `--takeover-owner` requires the complete bootstrap tuple. It bootstraps and
 health-checks the replacement before sending the current daemon the private
-retirement command. Retirement stops stable listeners and new work promptly,
-but does not wait for owned workers' normal drain before the replacement binds
-the proxy and control socket. A bootstrap failure occurs before retirement, so
-the previously accepted owner remains available.
+retirement command. The current `performOwnerRetirement` path quiesces every
+service, singleton, starting release, and retained release, releases the stable
+listeners, and starts asynchronous `stop()` calls for all of them. It neither
+preserves nor transfers retained-generation supervision to the replacement.
+The replacement can bind before those stops finish. A bootstrap failure occurs
+before retirement, so the previously accepted owner remains available.
+
+A compliant future owner handoff must instead preserve or transfer durable
+supervision of retained generations without handing old workers to a new
+jobs-main. Current `--takeover-owner` does not provide that behavior.
 
 ## `ensure-daemon`
 
@@ -120,12 +126,26 @@ rollbridge deploy --release-path <path>
                   [--daemon-start-timeout-ms <ms>]
 ```
 
-Starts the prepared release, health-checks the proxied process, switches new
-traffic to it, then drains and stops the previous release. Prints
+Starts the complete prepared release generation, health-checks the proxied
+process, and switches new traffic to it. The current daemon then starts
+retirement of the previous release asynchronously, so the command does not wait
+for old workers, jobs, or HTTP/WebSocket connections to finish. Prints
 `{"status": "success", "activeReleaseId": "...", "previousReleaseId": "..."}`.
 If the new release fails to start or health-check, the previous release stays
 active and the command errors.
 
+After candidate activation, `Daemon.deploy()` synchronously waits for singleton
+replacement before starting `drainAndPrune` and returning. A replacement failure
+can therefore return a non-zero result while the candidate remains active, and a
+slow replacement delays both the response and retirement of the old release.
+
+This is a process-lifetime non-blocking drain, not durable supervision across a
+daemon or host restart. It continues across later deploys only while the same
+daemon remains alive. After a restart, surviving PIDs from persisted state are
+advisory orphans that Rollbridge cannot re-adopt; explicit `recover --force`
+stops them. Restart-surviving retained-generation ownership and recovery remain
+required future behavior.
+
 Before each deploy, the daemon reloads the config path it was started with.
 Compatible process and lifecycle changes apply to the new release and govern
 how the previous release retires, including updated `nonBlockingDrain`,

package/docs/config.md

@@ -75,7 +75,7 @@ to let a deploy group talk to the daemon.
 | `proxy.upstreamHost` | string | `proxy.host`, or `"127.0.0.1"` when `proxy.host` is `0.0.0.0`/`::` | Host Rollbridge uses for release health checks and proxy targets. |
 | `proxy.healthPath` | string | `"/ping"` | Default health-check path for proxied processes. |
 | `proxy.healthTimeoutMs` | number | `30000` | Default health-check timeout for proxied processes. |
-| `proxy.drainTimeoutMs` | number | `60000` | How long to drain open connections from a retired release before stopping it. |
+| `proxy.drainTimeoutMs` | number | `60000` | How long to drain HTTP/WebSocket connections before stopping the retired proxied process. Expiry never stops an independently draining jobs generation. |
 | `proxy.forceStopTimeoutMs` | number | `10000` | Default per-process graceful-stop timeout (`SIGTERM`, then `SIGKILL`). |
 
 ## `releaseRetention`
@@ -212,6 +212,20 @@ range** so old and new instances can run at the same time:
 
 Reference it from same-release processes with `{{ports.background-jobs-main}}`.
 During a deploy, old workers keep the old port and new workers get the new port.
+For background jobs, the required compliant architecture makes the handoff
+service and its workers one release generation. Immediately after activation,
+retirement must quiesce the old jobs-main's scheduling, dispatch, and new worker
+handoffs while keeping it with its workers until their accepted work settles.
+Workers are not adopted by the new service.
+
+That immediate old-main quiescence is **required future compliance behavior**,
+not current `deployStrategy: "handoff"` behavior. Today the release group starts
+`stop()` for `nonBlockingDrain` companions at retirement but sends no retirement
+or quiescence notice to the handoff service. It waits for the connection drain,
+stops other dependent processes, waits for the non-blocking companion stops, and
+only then stops the handoff service. Consequently, old and new jobs-main
+instances can overlap scheduling and dispatch ownership. Do not treat the
+configuration above alone as compliance with the background-jobs contract.
 
 ### `processes[].lifecycle`
 
@@ -220,6 +234,11 @@ deploy's drain, a `rollbridge restart`, a memory restart, or shutdown. They let
 job worker quiesce and finish in-flight work before it is terminated. Omit
 `lifecycle` for the default behavior (just `stopSignal` then `SIGKILL`).
 
+These hooks describe an individual process stop. They do not define when a
+deploy completes and must not impose a short normal-drain deadline on a jobs
+generation. Per-job timeouts remain the correct bound for genuinely hung jobs;
+legitimate hours-long generation drains are valid.
+
 | Field | Type | Default | Description |
 | --- | --- | --- | --- |
 | `lifecycle.quietCommand` | string | unset | Run first to tell the process to stop accepting new work. |
@@ -253,15 +272,16 @@ That keeps a worker alive in case the draining web process still depends on it
 but it also holds a background worker open for the whole connection drain.
 
 Set `nonBlockingDrain: true` on a `companion` whose work is independent of the
-proxied process (a job worker on a shared queue). Its graceful stop — `lifecycle`
-hooks, or `stopSignal` then `SIGKILL` after `gracefulStopMs` — then starts **as
-soon as the release is retired**, in parallel with the connection drain, rather
-than after it. The new release's workers (started before traffic switches) handle
-new work while the retired release's workers finish their in-flight jobs. The
-whole drain stays non-blocking — the deploy returns immediately.
+proxied process (a job worker on a shared queue). Rollbridge starts that
+companion's configured stop sequence **as soon as the release is retired**, in
+parallel with the connection drain, rather than after it. Its quiet command or
+signal must make the worker stop accepting new handoffs. The asynchronous
+release drain continues after the deploy response. As described above, current
+Rollbridge does not simultaneously quiesce the handoff service, so this setting
+alone does not prevent overlapping jobs-main scheduling or dispatch ownership.
 
 ```js
-{id: "worker", policy: "companion", command: "…", nonBlockingDrain: true, stopSignal: "SIGINT", gracefulStopMs: 60000}
+{id: "worker", policy: "companion", command: "…", nonBlockingDrain: true, stopSignal: "SIGINT", gracefulStopMs: "indefinite"}
 ```
 
 ### `processes[].restart`

package/docs/nginx.md

@@ -67,10 +67,11 @@ Related Rollbridge timeouts (configured in `rollbridge.js`, not Nginx):
 
 - `proxy.healthTimeoutMs` gates how long a new release has to become healthy
   before a deploy aborts — it does not affect request timeouts.
-- `proxy.drainTimeoutMs` is how long Rollbridge keeps an old release alive for
-  in-flight connections during a deploy. Keep Nginx's `proxy_read_timeout` for
-  WebSocket locations comfortably above it so the front end doesn't cut
-  connections Rollbridge is still draining.
+- `proxy.drainTimeoutMs` is how long Rollbridge keeps an old proxied web process
+  alive for in-flight connections during a deploy. Keep Nginx's
+  `proxy_read_timeout` for WebSocket locations comfortably above it so the front
+  end doesn't cut connections Rollbridge is still draining. Retained jobs
+  generations drain independently and are not stopped when this timeout expires.
 
 ## Forwarded headers