rollbridge 0.1.23 → 0.1.25

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (13) hide show
  1. package/AGENTS.md +55 -0
  2. package/README.md +63 -39
  3. package/TODO.md +5 -4
  4. package/docs/cli.md +26 -6
  5. package/docs/config.md +28 -8
  6. package/docs/nginx.md +5 -4
  7. package/docs/tensorbuzz-runbook.md +91 -125
  8. package/docs/troubleshooting.md +18 -9
  9. package/docs/velocious.md +93 -170
  10. package/docs/workers.md +81 -103
  11. package/package.json +1 -1
  12. package/src/daemon.js +3 -2
  13. package/test/shutdown-completion.test.js +57 -3
package/docs/tensorbuzz-runbook.md CHANGED
@@ -1,129 +1,95 @@
1
- # TensorBuzz production runbook
1
+ # TensorBuzz Rollbridge runbook
2
2
 
3
- Operating the TensorBuzz backend under Rollbridge. The production config lives at
4
- [`examples/tensorbuzz.com.js`](../examples/tensorbuzz.com.js); this runbook
5
- assumes it is deployed to a stable path (`/etc/rollbridge/tensorbuzz.com.js`
6
- below) and the daemon runs as a systemd service (see
7
- [Running under systemd](../README.md#running-under-systemd)). For the general
8
- Velocious topology and the worker recipe, see [`docs/velocious.md`](velocious.md).
3
+ This runbook defines the required TensorBuzz backend topology. The example at
4
+ [`examples/tensorbuzz.com.js`](../examples/tensorbuzz.com.js) is illustrative;
5
+ verify the deployed consumer config and Rollbridge implementation before
6
+ asserting production compliance.
9
7
 
10
- ## Ports
8
+ ## Ports and generations
11
9
 
12
- | Port | Process | Notes |
10
+ | Port | Process | Contract |
13
11
  | --- | --- | --- |
14
- | `4500` | Rollbridge proxy | The stable public port. **Nginx proxies the backend host to `127.0.0.1:4500`** — never to a release's web port. |
15
- | `7330` | `beacon` (`service`) | Fixed; the shared broker every release connects to. |
16
- | `7331` | `background-jobs-main` (`service`) | Fixed; the job coordinator. |
17
- | `14500`–`14599` | `web` (`proxied`) | One port per release, allocated per deploy; Rollbridge forwards `4500` here. |
18
- | (none) | `background-jobs-worker` (`companion`) | A per-release worker; no listening port. |
19
-
20
- Control socket: `/tmp/rollbridge-tensorbuzz.sock`.
21
-
22
- ## Process topology
23
-
24
- - **`beacon`** and **`background-jobs-main`** are `service`s: one daemon-wide
25
- instance each, on their fixed ports, surviving deploys.
26
- - **`background-jobs-worker`** is a `companion`: a fresh worker per release,
27
- running that release's code, with `gracefulStopMs: 60000` so an in-flight job
28
- finishes before `SIGKILL`.
29
- - **`web`** is the one `proxied` process, health-checked at `/ping` before
30
- traffic switches.
31
-
32
- Each process waits for its dependencies with `wait-for-it` (`beacon` →
33
- `background-jobs-main` → `worker`/`web`), so nothing starts talking to Beacon or
34
- the job coordinator before they listen.
35
-
36
- ## External services
37
-
38
- Rollbridge manages **only the four processes above**. Everything else the
39
- Velocious app depends on — the database and any other backing services — is
40
- **provisioned and operated outside Rollbridge**: Rollbridge does not start, stop,
41
- health-check, or know about them. Configure those connections through the app's
42
- own environment/config. When such a dependency is down, the `web` process's
43
- `/ping` health check is what gates a deploy (a release that can't reach its
44
- database won't pass health and won't go live).
45
-
46
- ## Deploying
47
-
48
- Drive deploys through the CLI (see [`docs/deploy-recipes.md`](deploy-recipes.md)).
49
- Run **backwards-compatible** migrations before switching traffic, because the old
50
- and new releases overlap during the drain:
51
-
52
- ```bash
53
- release_path=/srv/tensorbuzz/releases/<timestamp> # prepared by your pipeline
54
- (cd "$release_path/backend" && npx velocious db:migrate)
55
-
56
- rollbridge deploy \
57
- --ensure-daemon \
58
- --config /etc/rollbridge/tensorbuzz.com.js \
59
- --release-path "$release_path" \
60
- --revision "$(git -C "$release_path/backend" rev-parse HEAD)"
61
- ```
62
-
63
- ### Deploy ordering
64
-
65
- On `rollbridge deploy`, Rollbridge:
66
-
67
- 1. starts any missing `service` (`beacon`, `background-jobs-main`);
68
- 2. starts the new release's `background-jobs-worker`, then its `web` process, and
69
- health-checks `web` on its `{{port}}`/`/ping`;
70
- 3. switches new traffic to the new `web`;
71
- 4. refreshes the services' restart templates to the new release;
72
- 5. drains the previous release's connections, then stops its `web` and worker.
73
-
74
- If the new release fails to start or health-check, **the previous release stays
75
- active** and the command exits non-zero — so a failed deploy never takes the site
76
- down.
77
-
78
- ## Rollback
79
-
80
- ```bash
81
- rollbridge rollback --config /etc/rollbridge/tensorbuzz.com.js
82
- # or a specific retained release:
83
- rollbridge rollback --config /etc/rollbridge/tensorbuzz.com.js --release-id <id>
84
- ```
85
-
86
- Rollback re-runs the deploy flow on a retained release, health-checks it, and
87
- switches traffic back. Constraints:
88
-
89
- - **Migrations are not reverted.** Rollback only manages processes; if a release
90
- bumped the schema, rolling code back requires that the old code still works
91
- against the new schema — keep migrations backwards-compatible (the same rule as
92
- deploys).
93
- - The target release's on-disk directory must still exist (don't prune it from
94
- disk before you might roll back to it).
95
- - Only releases Rollbridge still retains (`releaseRetention`) can be targeted.
96
-
97
- ## Day-to-day operations
98
-
99
- ```bash
100
- C=/etc/rollbridge/tensorbuzz.com.js
101
-
102
- rollbridge status --config "$C" # active release, ports, per-process state
103
- rollbridge logs --config "$C" --process web # recent stdout/stderr of a process
104
- rollbridge events --config "$C" # deploys, switches, crashes, restarts
105
- rollbridge doctor --config "$C" # pre-flight: socket, proxy port, state
106
- rollbridge restart --config "$C" --process background-jobs-worker # bounce the worker
107
- ```
108
-
109
- Restarting `beacon` or `background-jobs-main` bounces a shared broker and briefly
110
- disrupts everything that depends on it; prefer `deploy`/`rollback` for code
111
- changes. See [`docs/troubleshooting.md`](troubleshooting.md) for health-check
112
- failures, port conflicts, stale sockets, crash loops, and stuck draining
113
- releases.
114
-
115
- ## Crash recovery
116
-
117
- Set [`statePath`](config.md#statepath) in the config to have the daemon persist
118
- its state. After a daemon crash or reboot, `rollbridge doctor` reports any
119
- **orphaned** processes still alive from the previous daemon. To clean them up
120
- before restarting the daemon, run `rollbridge recover` (a dry run that lists
121
- them), then `rollbridge recover --force` to stop them:
122
-
123
- ```bash
124
- rollbridge recover --config /etc/rollbridge/tensorbuzz.com.js # list leftovers
125
- rollbridge recover --config /etc/rollbridge/tensorbuzz.com.js --force # stop them
126
- ```
127
-
128
- A machine reboot kills every process, so there are usually no orphans afterward —
129
- the daemon just starts fresh.
12
+ | `4500` | Rollbridge proxy | Stable public upstream for Nginx. |
13
+ | `7330` | Beacon | Shared persistent service; fixed port is allowed. |
14
+ | `7331`–`7399` | `background-jobs-main` | One allocated endpoint per release generation; never one fixed persistent coordinator. |
15
+ | `14500`–`14599` | web | One proxied endpoint per release. |
16
+ | none | workers | Same-release companions connected only to their generation's jobs-main. |
17
+
18
+ Each jobs generation contains one jobs-main and its complete worker pool, all
19
+ running the same release code. Several old generations may continue draining
20
+ while a newer generation is active.
21
+
22
+ ## Required deploy order
23
+
24
+ 1. Prepare the candidate release and run backwards-compatible migrations.
25
+ 2. Start the candidate jobs-main on a new port, then its worker pool and web
26
+ process. Health-check web before activation.
27
+ 3. Activate the candidate release and switch new traffic.
28
+ 4. Retire the previous jobs-main and workers as one generation. Jobs-main stops
29
+ schedule ownership, new dispatch, and new handoffs; workers stop accepting
30
+ handoffs.
31
+ 5. Return deploy success and release the deploy lock. Do not wait for old jobs,
32
+ workers, jobs-main, HTTP/WebSocket connections, or other retained services.
33
+
34
+ The retired jobs-main remains running on its old endpoint with its old workers.
35
+ For accepted handoffs it owns worker connections and heartbeats, lease fencing,
36
+ terminal-report acceptance and acknowledgement, and durable store transitions.
37
+ The old worker/reporting side durably retries terminal reports, tracks
38
+ outstanding report promises, enforces per-job execution timeouts, and owns and
39
+ reaps child runners. Returned or retried work becomes eligible for the new active
40
+ generation and is never redispatched by the retired main. Old workers do not
41
+ reconnect to or transfer their handoffs to the new main.
42
+
43
+ Old main and workers remain one release generation until every accepted handoff
44
+ settles. Only then, after every old worker exits, may jobs-main exit and
45
+ Rollbridge reap the generation. The referenced release directory stays pinned
46
+ against Rampway cleanup until that point.
47
+
48
+ ## Independent drains
49
+
50
+ Set the worker companion to `nonBlockingDrain: true` so it quiesces when its
51
+ generation retires. HTTP/WebSocket connection drain continues independently.
52
+ Finishing or timing out the HTTP drain must never stop a still-draining jobs
53
+ generation. A legitimate multi-hour job and generation drain are valid.
54
+
55
+ Per-job timeouts remain the backstop for genuinely hung jobs. Do not use a short
56
+ worker-shutdown or supervisor timeout as the primary deploy solution; deployment
57
+ has already completed after candidate activation and health.
58
+
59
+ ## Runtime-owner and recovery requirements
60
+
61
+ Required compliant behavior durably supervises every retired generation after
62
+ the deploy command returns, across later deploys and supervisor/host recovery.
63
+ Runtime-owner or version handoff must preserve or transfer that supervision and
64
+ return once the replacement is healthy. It must not perform full synchronous
65
+ shutdown, kill retained generations, or make the new jobs-main adopt old workers.
66
+
67
+ Recovery must reconstruct retained generation ownership, endpoints, release
68
+ paths, and process references. Do not treat them as generic orphans to force-stop
69
+ merely because a supervisor restarted. Cleanup becomes eligible only after the
70
+ last retained process exits.
71
+
72
+ Current Rollbridge does not yet meet those recovery and owner-handoff
73
+ requirements. Its non-blocking release drains last only for the current daemon's
74
+ lifetime; after restart it reports surviving PIDs as advisory, non-adoptable
75
+ orphans, and forced recovery stops them. `--takeover-owner` quiesces and starts
76
+ asynchronous stops for every managed process instead of transferring retained
77
+ generations.
78
+
79
+ ## Operator checks
80
+
81
+ Use `rollbridge status`, logs, and events to confirm:
82
+
83
+ - the active jobs generation uses the active release and a unique jobs-main port;
84
+ - every retired generation retains its own jobs-main, workers, endpoint, and
85
+ pinned release path;
86
+ - the deploy command has returned while long drains continue;
87
+ - no retired jobs-main is dispatching new ordinary queued work;
88
+ - HTTP drain completion or timeout did not stop a jobs generation; and
89
+ - completed generations are reaped and Rampway is told their release references
90
+ ended so it can release the pins.
91
+
92
+ Do not restart Beacon or a jobs-main casually. Never use `shutdown`, forced
93
+ orphan recovery, or process signals as a substitute for the normal retained-
94
+ generation lifecycle. See [`docs/velocious.md`](velocious.md) and
95
+ [`docs/workers.md`](workers.md) for the architecture details.
package/docs/troubleshooting.md CHANGED
@@ -109,12 +109,21 @@ stays active.
109
109
  in `state: "draining"` with non-zero `connections` (often `websocket`).
110
110
 
111
111
  **Diagnose.** Long-lived connections (WebSockets, SSE, streaming responses) keep
112
- the retired release alive until they close or `proxy.drainTimeoutMs` elapses.
113
- `status` shows the release's `connections.http`/`connections.websocket` and
114
- `drainStartedAt`.
115
-
116
- **Fix.** Draining ends automatically when those connections close, or after
117
- `proxy.drainTimeoutMs` (then the release is stopped regardless). Lower
118
- `proxy.drainTimeoutMs` to force-stop sooner, or make clients reconnect (for
119
- example, have the front end close idle WebSockets on deploy). Once stopped, the
120
- release is pruned per `releaseRetention`.
112
+ the retired proxied web process alive until they close or
113
+ `proxy.drainTimeoutMs` elapses. `status` shows the release's
114
+ `connections.http`/`connections.websocket` and `drainStartedAt`. A retained jobs
115
+ generation has an independent lifecycle and may remain after the web drain ends.
116
+
117
+ **Fix.** The connection drain ends automatically when those connections close,
118
+ or after `proxy.drainTimeoutMs`. Rollbridge then stops the retired proxied web
119
+ process and other connection-dependent processes, including ordinary companions
120
+ with `nonBlockingDrain: false`. Lower the timeout only to shorten
121
+ HTTP/WebSocket retention, or make clients reconnect (for example, have the front
122
+ end close idle WebSockets on deploy). In the documented compliant jobs topology,
123
+ jobs companions use `nonBlockingDrain: true`, so timeout expiry affects only the
124
+ web side and must not stop a still-draining jobs generation.
125
+
126
+ Reporting release references to Rampway and pinning release directories against
127
+ on-disk cleanup are required future behavior, not implemented today. Current
128
+ Rollbridge `status` and `releaseRetention` govern only its in-memory release
129
+ records and do not fence Rampway cleanup.
package/docs/velocious.md CHANGED
@@ -1,37 +1,35 @@
1
1
  # Velocious deployment guide
2
2
 
3
- A Velocious backend typically runs four kinds of process: **Beacon** (the
4
- message broker other processes connect to), **background-jobs-main** (the job
5
- coordinator), **background-jobs-worker** (runs the jobs), and the **web/API**
6
- server. This guide maps each to a Rollbridge process policy, shows a complete
7
- `rollbridge.js`, and explains startup ordering and what happens on a deploy.
3
+ A Velocious backend normally runs Beacon, `background-jobs-main`, a
4
+ `background-jobs-worker` pool, and the web/API server. For deploy lifecycle
5
+ purposes, jobs-main and its workers are one release-scoped **jobs generation**.
6
+ This topology is required; a single persistent fixed-port jobs-main is not a
7
+ safe coordinator for workers that may drain across releases.
8
8
 
9
- A production version of this config lives at
10
- [`examples/tensorbuzz.com.js`](../examples/tensorbuzz.com.js).
9
+ This page states the architecture contract. It does not by itself assert that a
10
+ particular Rollbridge release or consumer production config implements durable
11
+ retired-generation recovery; verify source and config before claiming compliance.
11
12
 
12
13
  ## Process mapping
13
14
 
14
- | Velocious process | Policy | Why |
15
+ | Velocious process | Rollbridge policy | Lifecycle |
15
16
  | --- | --- | --- |
16
- | `beacon` | `service` | A shared broker the other processes connect to. It should survive deploys and keep a **stable port**, so workers and the web process always reach the same Beacon. |
17
- | `background-jobs-main` | `service` with `deployStrategy: "handoff"` | The job coordinator. Run it as a handoff service so each release's workers and web process use a same-release coordinator while old releases drain (see [Choosing the jobs-main policy](#choosing-the-jobs-main-policy)). |
18
- | `background-jobs-worker` | `companion` | Release-scoped: one set of workers per active release, started before the web process and running that release's code. |
19
- | `web` | `proxied` | Receives external HTTP/WebSocket traffic, is health-checked before traffic switches, and is drained on the next deploy. Exactly one process is `proxied`. |
20
-
21
- See [README → Process Policies](../README.md#process-policies) for the full
22
- semantics of each policy and [`docs/config.md`](config.md) for every field.
17
+ | `beacon` | persistent `service` | Shared broker; it may remain daemon-wide on fixed port `7330`. |
18
+ | `background-jobs-main` | `service` with `deployStrategy: "handoff"` | One endpoint per release; owns worker connections, lease fencing, report acceptance/acknowledgement, and durable store transitions. |
19
+ | `background-jobs-worker` | `companion` with `nonBlockingDrain: true` | Release-scoped pool; executes accepted work, owns child runners and execution timeouts, and durably retries terminal reports while tracking their promises. |
20
+ | `web` | `proxied` | Health-gated active HTTP/WebSocket target with a per-release port. |
23
21
 
24
22
  ## Example `rollbridge.js`
25
23
 
26
24
  ```js
27
- // rollbridge.js
28
25
  export default {
29
26
  application: "tensorbuzz",
30
27
  control: {path: "/tmp/rollbridge-tensorbuzz.sock"},
28
+ statePath: "/var/lib/rollbridge/tensorbuzz.json",
31
29
 
32
30
  proxy: {
33
31
  host: "127.0.0.1",
34
- port: 4500, // the stable port Nginx points at
32
+ port: 4500,
35
33
  healthPath: "/ping",
36
34
  healthTimeoutMs: 30000,
37
35
  drainTimeoutMs: 60000,
@@ -39,7 +37,6 @@ export default {
39
37
  },
40
38
 
41
39
  processes: [
42
- // Shared broker — one daemon-wide instance on a stable port.
43
40
  {
44
41
  id: "beacon",
45
42
  policy: "service",
@@ -48,8 +45,6 @@ export default {
48
45
  command: "npx velocious beacon",
49
46
  port: 7330
50
47
  },
51
-
52
- // Job coordinator — one release-scoped service instance per deploy.
53
48
  {
54
49
  id: "background-jobs-main",
55
50
  policy: "service",
@@ -63,23 +58,20 @@ export default {
63
58
  command: "wait-for-it 127.0.0.1:{{ports.beacon}} --strict -- npx velocious background-jobs-main",
64
59
  port: {from: 7331, to: 7399}
65
60
  },
66
-
67
- // Workers — one set per release; raise gracefulStopMs to let in-flight
68
- // jobs finish during a deploy.
69
61
  {
70
62
  id: "background-jobs-worker",
71
63
  policy: "companion",
64
+ nonBlockingDrain: true,
72
65
  cwd: "{{releasePath}}/backend",
73
66
  env: {
74
67
  NODE_ENV: "production",
75
68
  VELOCIOUS_BEACON_PORT: "{{ports.beacon}}",
76
69
  VELOCIOUS_BACKGROUND_JOBS_PORT: "{{ports.background-jobs-main}}"
77
70
  },
78
- command: "wait-for-it 127.0.0.1:{{ports.beacon}} --strict -- wait-for-it 127.0.0.1:{{ports.background-jobs-main}} --strict -- npx velocious background-jobs-worker",
71
+ command: "wait-for-it 127.0.0.1:{{ports.background-jobs-main}} --strict -- npx velocious background-jobs-worker",
72
+ replicas: 4,
79
73
  gracefulStopMs: "indefinite"
80
74
  },
81
-
82
- // Web/API — the one proxied process.
83
75
  {
84
76
  id: "web",
85
77
  policy: "proxied",
@@ -89,7 +81,7 @@ export default {
89
81
  VELOCIOUS_BEACON_PORT: "{{ports.beacon}}",
90
82
  VELOCIOUS_BACKGROUND_JOBS_PORT: "{{ports.background-jobs-main}}"
91
83
  },
92
- command: "wait-for-it 127.0.0.1:{{ports.beacon}} --strict -- wait-for-it 127.0.0.1:{{ports.background-jobs-main}} --strict -- npx velocious server --host 127.0.0.1 --port {{port}}",
84
+ command: "wait-for-it 127.0.0.1:{{ports.background-jobs-main}} --strict -- npx velocious server --host 127.0.0.1 --port {{port}}",
93
85
  port: {from: 14500, to: 14599},
94
86
  health: {path: "/ping", timeoutMs: 30000, intervalMs: 500}
95
87
  }
@@ -97,146 +89,77 @@ export default {
97
89
  }
98
90
  ```
99
91
 
100
- ## Wiring processes together
101
-
102
- Beacon gets a **fixed** port (`7330`) because it is a persistent `service` — a
103
- stable port lets every release's processes find the shared broker.
104
- `background-jobs-main` gets a **range** (`{from: 7331, to: 7399}`) because it is a
105
- handoff service: Rollbridge allocates a new port per release so old workers keep
106
- talking to the old coordinator while new workers and web use the new one. The
107
- proxied `web` process also gets a **range** (`{from: 14500, to: 14599}`) so old
108
- and new web releases can run side by side during the drain.
109
-
110
- Cross-reference ports with `{{ports.<id>}}` and pass them to Velocious through
111
- `env`. Rollbridge also injects `ROLLBRIDGE_<ID>_PORT` for every process (e.g.
112
- `ROLLBRIDGE_BACKGROUND_JOBS_MAIN_PORT`), so you can read ports from the
113
- environment instead of templating if you prefer — see
114
- [`docs/config.md`](config.md#injected-environment-variables).
115
-
116
- ### Startup ordering
117
-
118
- Only the `proxied` process is health-checked, so dependent processes must wait
119
- for their dependencies themselves. Two mechanisms combine:
120
-
121
- 1. **Policy ordering.** On each deploy Rollbridge starts handoff `service`s
122
- first, then the release's `companion`s, then the `proxied` process (see
123
- [README → Deploy ordering](../README.md#deploy-ordering)).
124
- 2. **Readiness gating.** `wait-for-it 127.0.0.1:{{ports.beacon}} --strict -- …`
125
- blocks the command until Beacon's port accepts connections, so
126
- `background-jobs-main`, the worker, and `web` don't start talking to Beacon
127
- before it is listening. `wait-for-it` is a small standalone script (install it
128
- on the host); any equivalent port-wait works.
129
-
130
- ## Deploying
131
-
132
- Drive deploys through the Rollbridge CLI — Rollbridge ships no deploy-tool
133
- plugins (see [`docs/deploy-recipes.md`](deploy-recipes.md) for shell/CI/Capistrano
134
- recipes). The minimal step after a release directory is prepared:
135
-
136
- ```bash
137
- release_path=/srv/tensorbuzz/releases/20260523120000 # prepared by your pipeline
138
-
139
- # Run backwards-compatible migrations BEFORE switching traffic: the old and new
140
- # web releases overlap during the drain.
141
- (cd "$release_path/backend" && npx velocious db:migrate)
142
-
143
- rollbridge deploy \
144
- --ensure-daemon \
145
- --config /etc/rollbridge/rollbridge.js \
146
- --release-path "$release_path" \
147
- --revision "$(git -C "$release_path/backend" rev-parse HEAD)"
148
- ```
149
-
150
- `rollbridge deploy` starts the new release's worker and web process,
151
- health-checks `web` on its `{{port}}`/`/ping`, switches traffic, then drains and
152
- stops the previous release. It exits non-zero (leaving the previous release
153
- active) if the new release fails to start or health-check, so a failed deploy
154
- never promotes a broken release.
155
-
156
- ## Background jobs across a deploy
157
-
158
- The worker is a `companion`, so each release runs its own workers:
159
-
160
- - On deploy, the **new** release's workers start (running the new code) before
161
- traffic switches; the **old** release's workers are stopped when that release
162
- is drained and retired — the worker's `stopSignal`, then `SIGKILL` after
163
- `gracefulStopMs`.
164
- - Set `stopSignal` to the signal your worker drains on and `gracefulStopMs` to at
165
- least your longest in-flight job. Use `gracefulStopMs: "indefinite"` when the
166
- worker can safely drain until it exits on its own. Set `replicas` to run a pool
167
- of workers.
168
-
169
- See [`docs/workers.md`](workers.md) for the full safe background-job deployment
170
- pattern (companion + `replicas` + `stopSignal`/`lifecycle` hooks +
171
- `gracefulStopMs`), the old/new worker overlap, and `nonBlockingDrain` to start the
172
- old workers' drain immediately when a release is retired.
173
-
174
- ### Worker recipe
175
-
176
- A complete `background-jobs-worker` entry that runs a pool and finishes in-flight
177
- jobs across a deploy:
178
-
179
- ```js
180
- {
181
- id: "background-jobs-worker",
182
- policy: "companion",
183
- cwd: "{{releasePath}}/backend",
184
- env: {
185
- NODE_ENV: "production",
186
- VELOCIOUS_ENV: "production",
187
- VELOCIOUS_BEACON_PORT: "{{ports.beacon}}",
188
- VELOCIOUS_BACKGROUND_JOBS_PORT: "{{ports.background-jobs-main}}"
189
- },
190
- command: "wait-for-it 127.0.0.1:{{ports.beacon}} --strict -- wait-for-it 127.0.0.1:{{ports.background-jobs-main}} --strict -- npx velocious background-jobs-worker",
191
- replicas: 4,
192
- gracefulStopMs: "indefinite"
193
- }
194
- ```
195
-
196
- - `replicas: 4` runs four worker instances (`background-jobs-worker#0` … `#3`),
197
- each with `ROLLBRIDGE_REPLICA_INDEX`/`ROLLBRIDGE_REPLICA_COUNT` if you shard work.
198
- - On deploy the new release's workers start before traffic switches; the old
199
- release's workers receive `SIGTERM` (the default `stopSignal`) when the old
200
- release is retired, then wait to exit. With `gracefulStopMs: "indefinite"`,
201
- Rollbridge does not send a `SIGKILL` fallback.
202
-
203
- If your worker quiesces on a command or a non-default signal, add a `lifecycle`
204
- block — Rollbridge runs `quietCommand`, drains for up to `drainTimeoutMs`, then
205
- stops. For example, send a quiet signal to the worker's process group before the
206
- drain:
207
-
208
- ```js
209
- lifecycle: {quietCommand: "kill -TSTP -$ROLLBRIDGE_PID", drainTimeoutMs: 60000}
210
- ```
211
-
212
- ### Choosing the jobs-main policy
213
-
214
- `background-jobs-main` coordinates workers, so choose its lifecycle deliberately:
215
-
216
- - **`service` with `deployStrategy: "handoff"`** — starts one coordinator per
217
- release on a port from a range. New workers and web get the new release's port;
218
- old workers keep the old release's port while they drain. This is the safest
219
- default when the coordinator should run the same code version as its workers.
220
- - **`service` with the default `deployStrategy: "persistent"`** — keeps one
221
- daemon-wide coordinator on a stable port. Workers from every release talk to the
222
- same coordinator, but it keeps running the release it was started from and only
223
- adopts the latest template if it restarts later.
224
- - **`singleton`** — stops the old instance and then starts the new one on each
225
- deploy, so it always runs the latest release's code and two copies never
226
- overlap. The trade-off: a brief coordination gap while it restarts.
227
-
228
- Beacon is a broker rather than code that changes per release, so `service` is
229
- almost always right for it.
230
-
231
- ## Verifying
232
-
233
- After a deploy, `rollbridge status` should show `beacon` as a long-lived service
234
- with an unchanged port, `background-jobs-main` as the active release's handoff
235
- service, one `background-jobs-worker` for the active release, and the `web`
236
- process `proxied` with its connection counts. Use
237
- [`rollbridge logs --process <id>`](cli.md) to read recent output from any
238
- process, and [`docs/troubleshooting.md`](troubleshooting.md) for health-check,
239
- port, and draining problems.
240
-
241
- For the front end, point Nginx at the stable `proxy.port` (here `4500`), never at
242
- a release's web port — see [`docs/nginx.md`](nginx.md).
92
+ Beacon keeps its fixed port because it is intentionally shared. Jobs-main uses a
93
+ range because every release gets its own coordinator. Same-release
94
+ `{{ports.background-jobs-main}}` expansion ensures that old workers retain the
95
+ old endpoint while candidate workers use the candidate endpoint.
96
+
97
+ ## Deploy and activation
98
+
99
+ Run backwards-compatible migrations before activation, then invoke
100
+ `rollbridge deploy` with the prepared release. Rollbridge starts the candidate
101
+ jobs-main, its complete worker pool, and the web process before health gating and
102
+ activation. A candidate startup or health failure leaves the previous release
103
+ active.
104
+
105
+ After successful activation, the deploy returns without waiting for any retired
106
+ generation or HTTP/WebSocket connection to finish. The old and new release code
107
+ may therefore overlap for hours. Keep schema, queue payloads, and external side
108
+ effects compatible across that window.
109
+
110
+ ## Retired jobs-generation contract
111
+
112
+ Retire jobs-main and its workers as one unit:
113
+
114
+ - jobs-main relinquishes recurring schedule ownership and stops dispatching
115
+ queued work or making new worker handoffs;
116
+ - workers stop advertising or accepting new handoffs;
117
+ - jobs-main remains running on the old endpoint and owns worker connections and
118
+ heartbeats, lease fencing, terminal-report acceptance and acknowledgement, and
119
+ durable store transitions for its accepted handoffs;
120
+ - the old worker/reporting side durably retries terminal reports, tracks
121
+ outstanding report promises, enforces per-job execution timeouts, and owns and
122
+ reaps child runners;
123
+ - a job returned or retried to the shared queue becomes eligible for the new
124
+ active generation, and the retired main never dispatches it again;
125
+ - old workers never reconnect to or transfer their handoffs to the new jobs-main;
126
+ - old main and workers remain one release generation until all accepted work
127
+ settles; jobs-main exits only after that and after all workers drain and exit,
128
+ after which Rollbridge may reap the generation.
129
+
130
+ HTTP/WebSocket and jobs drains are independent. `proxy.drainTimeoutMs` bounds the
131
+ connection drain only; reaching it must not stop a still-draining jobs generation.
132
+ `nonBlockingDrain: true` starts worker quiescence at retirement rather than after
133
+ the HTTP drain.
134
+
135
+ The process supervisor must retain multiple old generations concurrently,
136
+ persist their ownership across later deploys and supervisor/host recovery, and
137
+ report every referenced release directory so Rampway can pin it against cleanup.
138
+ A runtime owner/version handoff preserves or transfers that supervision and
139
+ returns after the replacement is healthy; it is not a full synchronous shutdown.
140
+
141
+ These are target requirements. Current Rollbridge drains releases
142
+ asynchronously only while the same daemon remains alive, cannot re-adopt
143
+ surviving PIDs after restart, and stops rather than transfers all managed
144
+ processes during `--takeover-owner`; see [`docs/cli.md`](cli.md#daemon).
145
+
146
+ ## Timeouts
147
+
148
+ Velocious per-job timeouts remain responsible for genuinely hung work. Rollbridge
149
+ stop signals, lifecycle hooks, and graceful-stop bounds remain emergency/process
150
+ controls. Do not use a short normal worker-shutdown timeout to make deployment
151
+ complete: deployment is already complete after healthy activation, and a
152
+ legitimate hours-long job makes an hours-long generation drain valid.
153
+
154
+ ## Verification
155
+
156
+ After a deploy, status must be able to show the active generation and every
157
+ retired generation still draining, including each jobs-main endpoint, worker
158
+ pool, release path, and retention reference. Beacon may keep `7330`; active and
159
+ retired jobs-main instances must use different ports. Confirm that the deploy
160
+ command has returned even while retained generations remain and that an HTTP
161
+ drain timeout does not terminate them.
162
+
163
+ See [`docs/workers.md`](workers.md) for the focused lifecycle,
164
+ [`docs/config.md`](config.md) for configuration fields, and
165
+ [`docs/tensorbuzz-runbook.md`](tensorbuzz-runbook.md) for the consumer runbook.