pgbus 0.13.0 → 0.13.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: '0589d2d6e69eb8b2ba5863cb8c8e2779bca29528030410508e1930fcfbd0d774'
-  data.tar.gz: c491442cd7b1d2bdccc5ff95153dc31abb424111906e17872b6b112830a1cde7
+  metadata.gz: 295404bb7b5b6d289a2bfd4f10d9e5bc6df442f8d949454cf1dd78adf620a25f
+  data.tar.gz: 2ee9c8e2c40f47bb900e10d2ef4c55b5e21e3cd4d997c614c628b696ac759183
 SHA512:
-  metadata.gz: 63e8480dcd7efb70bd7933cba734c23d7baa6f9e50ff658b575720da5ccc53e43fde1e4aca0fc7c30b6715448d040c1bb2fff472f5ac1686ae3e62853beb0bdd
-  data.tar.gz: 186da9576934f14264e9d9989c1a035d1c725e8283d1a735692da90f1610f4c538776b558a526d0c536466f92e405ffcbf99fd5bf229f9bf9b92dd1e0d7ec876
+  metadata.gz: 184484168fd458dbdb09b98d12ae5d906ebb510a05abc626fd3812ee4cfebb5cd575c090b8014b9524a9aa19f65e99d1737df50b191ce79367e77858f72c5024
+  data.tar.gz: 92170d5077128d507968989f8d9d789bdc9afa7b4ccf92ecdc3b39b357a82608de39a83b4e9f2f899078466714bfb30f3f0c357a9dc977f35c9c21254c8b7dbd

data/CHANGELOG.md

@@ -2,6 +2,14 @@
 
 ### Added
 
+- **Health-checked rolling restarts for the job container (issue #386).** ⚠️ **Behavior change on the standalone `/readyz`.** The supervisor's `health_port` server previously answered `/readyz` with the cluster-wide HealthAnalyzer verdict — so during a rolling deploy a freshly-booted container could pass an orchestrator's health gate on the strength of the *old* container's still-heartbeating workers, and the old container (with all its capacity) was stopped before the new one had forked a single child. The standalone `/readyz` is now **container-local**: 200 only when *this* supervisor verified its connection, bootstrapped queues, forked every configured child, and all of them are currently alive — with 503 bodies `BOOTING` (pre-boot), `DEGRADED` (a child died and is waiting out crash-restart backoff — precisely the state a deploy gate must fail on, keeping the old container running), and `DRAINING` (stop signal received). No database access on the probe path; the supervisor publishes an immutable snapshot per monitor pass and the accept thread reads it. The Rails-mounted `Pgbus::Web::HealthApp` keeps the cluster-wide verdict unchanged. Alongside it: **`pgbus-health`**, a shipped executable probe for docker `HEALTHCHECK` blocks (plain Ruby + stdlib sockets, loads neither Bundler nor the gem — cheap at 1–5s intervals, works in curl-less images; exit 0/1/2 = healthy/unhealthy/usage), and a README "Rolling restarts (Kamal, docker)" guide covering the healthcheck block, stop-timeout alignment, overlap-window duplicate-supervisor safety, and the `read_ct`-vs-deploy-kill DLQ caveat. Refs #386.
+
+### Changed
+
+- **Shutdown budgets are now alignable end-to-end (issue #386).** New `config.shutdown_timeout` bounds how long the supervisor waits for children after forwarding TERM before escalating to SIGKILL — previously a hardcoded 30s, which silently SIGKILLed workers mid-drain the moment `drain_timeout` was raised past it. Default derives `drain_timeout + 5` so the deadline tracks the drain window automatically; an explicit value below `drain_timeout` logs a boot warning. `Consumer#shutdown`'s pool wait (its only drain bound) now follows `config.drain_timeout` instead of a hardcoded 30s, and `Worker#shutdown`'s post-drain residual wait drops from a second full 30s window to 5s — the drain loop already waited `drain_timeout`, and a job still running has proven it won't finish. Rule of thumb: orchestrator stop grace period > `shutdown_timeout` > `drain_timeout`. Refs #386.
+
+- **Streams: one LISTEN connection per web host — `streams_listen_scope` (issue #382).** ⚠️ **Default behavior change.** Previously every Puma worker lazily opened its own dedicated streams LISTEN connection on first SSE use, so a web host pinned one direct connection per worker. Under the new default (`streams_listen_scope = :master`) the `pgbus_streams` Puma plugin runs a **MasterHub** in the preforking master: ONE `Web::Streamer::Listener` on the refcounted union of every worker's stream channels, fanning wakes — **including ephemeral payloads** — out to workers over a Unix domain socket with length-prefixed frames (`Streamer::HubProtocol`). Workers connect lazily (nothing is inherited across fork) and the synchronous `ensure_listening` ack contract is preserved cross-process: a sub is registered before LISTEN executes and acked only after, so the no-lost-broadcast guarantee holds. Backpressure follows the streams rules: durable wakes are droppable at a per-worker cap (they self-heal via `read_after`), **ephemeral wakes are never dropped** — a worker that stops draining is evicted, which triggers its own fallback. **Fallback is per-worker listeners, not loss**: whenever the hub is absent or dies (no `preload_app!`, single-mode Puma, crash, eviction) each worker's `FailoverListener` swaps in a real per-worker `Listener` and re-LISTENs its recorded subscriptions — connection footprint balloons back to pre-#382 levels (census-visible) but no broadcast semantics change; the worker stays local until it recycles. Measured (local PG, n=50): the master→worker hop is noise-level free — single-broadcast SSE roundtrip p50 16.00ms via the hub vs 16.93ms per-worker. **`:master` effectively requires `preload_app!`** (the hub waits for the app's pgbus initializer; without it the deadline expires quietly and workers stay per-worker). **Rollback:** `config.streams_listen_scope = :process`. Refs #382, builds on the #381 patterns.
+
 - **Host-level shared LISTEN: `worker_notify_scope` — the supervisor now owns ONE direct LISTEN connection for the whole host (issue #381).** ⚠️ **Default behavior change.** Previously every worker fork and every consumer fork opened its own dedicated LISTEN connection (`NotifyListener`), so a host's direct-connection footprint scaled with fork count — on transaction-pool PgBouncer platforms those connections come out of the scarcest slice of `max_connections`, and a 5-capsule + 2-consumer host pinned 7. Under the new default (`config.worker_notify_scope = :supervisor`) the supervisor runs a single `NotifyHub`: one `NotifyListener` on the union of every capsule's and consumer's queue channels (wildcards via the shared resolver, consumer sets via the registry), fanning wakes out to forks over per-fork pipes (`W` wake / `H` healthy / `P` degraded bytes; a fork whose pipe reports degraded or reaches EOF falls back to fast polling exactly like a failed local listener). Footprint drops to **1 direct LISTEN connection per job host**, verified by integration test: routing is per-fork (an insert wakes only the forks reading that queue, wildcard capsules unconditionally), and `pg_terminate_backend` on the shared connection is survived — reconnect, re-LISTEN, wakes flow again. **Rollback:** `config.worker_notify_scope = :fork` restores the previous per-fork listeners byte-for-byte. Dedicated LISTEN connections are now census-tagged `application_name=pgbus-listen` so `pg_stat_activity` can count them. Refs #381.
 - **`pgbus doctor`: new "Connection budget" check (issue #381).** Prints how many direct LISTEN connections the current config pins — 1 per host under `:supervisor` scope, capsules + consumers under `:fork` (honoring `config.roles`), plus a "+1 per web-server process (streams)" clause — so operators can do pooler capacity math from the doctor output alone. Informational, always `:ok`. Refs #381.
 - **Benchmarks: `rake bench:notify_wake` and `rake bench:notify_chaos` (issue #381).** Wake-path latency (send → wake, p50/p95/p99, direct vs hub-mediated), empty-read cost, LISTEN connection census, and failure-mode measurements (killed LISTEN backend, wedged fork, FD churn, fan-out cost). Refs #381.
@@ -14,6 +22,8 @@
 
 ### Fixed
 
+- **`idempotent!` handlers no longer silently drop an execution when the consumer is killed mid-handler — the idempotency claim is now two-phase (issue #385).** `EventBus::Handler#process!` inserted the `pgbus_processed_events` claim row *before* `handle(event)` ran, so a SIGKILL between claim and completion (supervisor watchdog, container runtime after its stop timeout, OOM — i.e. every container-replacement deploy's worst case) turned the redelivered message into a skip: the mechanism meant to dedup *duplicate* executions converted a crash *during* the one permitted execution into at-most-once. The claim now lands with a NULL `completed_at` (*pending*) and is stamped completed only after `handle` returns; on redelivery a pending claim **re-runs** the handler while a completed claim still skips. The in-memory dedup cache only ever records completed executions, and a `handle` that raises leaves the claim pending so VT redelivery retries it. The only double-execution window is a still-alive handler running past its visibility timeout — the same at-least-once window every non-idempotent handler already has. **Upgrading:** new installs get the column from `pgbus:install`; existing installs run `rails generate pgbus:add_processed_event_completion` (also picked up by `pgbus:update`; `--database` supported) — legacy rows are backfilled as completed so history is not retroactively re-run. Until the migration runs, an upgraded gem detects the missing column once and falls back to the legacy single-phase claim with a warning naming the generator. Refs #385.
+
 - **SIGTERM no longer segfaults the process: both LISTEN listeners stopped closing their PG connection from the stopping thread (issue #375).** `Process::NotifyListener#stop` (supervisor thread, on SIGTERM) closed the listener's dedicated `PG::Connection` to interrupt the blocking `wait_for_notify`, but left `@conn` pointing at it — so the listener thread unwound into `run_loop`'s `ensure` and ran `safe_unlisten_all`, exec'ing `UNLISTEN` on a connection the other thread was concurrently freeing. `PG::Connection#close` is `PQfinish`: it frees the PGconn **and its OpenSSL objects**, so `PQsendQuery` walked into freed TLS state and the whole worker died with `[BUG] Segmentation fault` — reproducibly, on essentially every container-replacement deploy against a TLS Postgres, sometimes in several forked workers at once. `rescue PG::Error` cannot catch a C-level SEGV, so the "safe" in `safe_unlisten_all` never held. **The connection is now single-owner**: the listener thread is the only thread that may exec, wait, or close on it, from build through teardown. `#stop` signals by clearing `@running` and joining — nothing more — and the listener thread closes its own connection in `run_loop`'s `ensure`. The stop is observed within one `wait_for_notify` timeout (`health_check_ms`, **250 ms** in the default configuration since workers/consumers derive it from `polling_interval`), so `#stop`'s join budget is now `health_check_ms + 5s` instead of a flat 5 s — a flat timeout could expire before a listener with a large `health_check_ms` had even one chance to observe the stop. The teardown `UNLISTEN` round-trip is **dropped entirely** rather than merely made safe: it ran immediately before the close, and closing a session deregisters every `LISTEN` server-side, so it bought nothing at any time. The identical defect in `Web::Streamer::Listener` — same close-from-`#stop`, same `UNLISTEN`-in-`ensure`, and there without even a mutex around `@conn` — is fixed the same way, so the dashboard's Puma worker stops crashing on the same deploy; its listener thread now also closes its own connection (previously only `#stop` did). Cost: shutdown can take up to one health-check cycle longer per listener — measured against a real PostgreSQL at the default `health_check_ms` of 250: **~207 ms on a fully idle queue, ~5 ms when the queue has any NOTIFY traffic** (a notification returns the wait immediately, and the loop then sees the cleared flag). 40 start/stop cycles under concurrent NOTIFY load leaked zero `LISTEN` backends. Thanks to the reporter for the crash dumps and the root-cause analysis. Refs #375.
 - **The health verdict no longer emits false STALLED reports — `max_read_ct` was never populated, and the wedge signal counted queues no capsule drains (issue #367).** Two correctness defects in `Pgbus::MCP::HealthAnalyzer` (surfaced through `pgbus doctor` / `pgbus_health` / the MCP health tool). **(1)** The `all_unread?` wedge check read `:max_read_ct`, but the metrics query never selected it — so the guard always degenerated to "never claimed" and a busy-but-healthy queue caught mid-burst produced a STALLED verdict with a factually wrong "read_ct=0 (never claimed)" reason. The metrics query now exposes a per-queue **`visible_unread_length`** (`count(*) WHERE vt <= NOW() AND read_ct = 0`) and the analyzer keys the wedge off *visible, never-claimed* messages. Counting per visible message (not `max(read_ct)` over the whole table) means one retried message left in-queue after its backoff, or an in-flight message claimed by a peer, can no longer veto the signal for a pile of genuinely-unclaimed jobs. **(2)** The verdict reasoned about **every** non-DLQ/non-stream queue against the global worker fleet — but a worker can only claim from queues its capsule subscribes to, so "M workers alive but never claimed" was vacuous for a queue nobody drains (ad-hoc queues, unregistered stream queues — see #366). The analyzer now intersects the STALLED backlog with `Web::DataSource#drained_queue_names` (each configured capsule's queues, priority `_pN` sub-tables expanded via the client's queue strategy, unioned with EventBus handler queues; `nil` for a `*` wildcard = drains everything, fail-open on error). Queues nobody drains get their own DEGRADED signal — *"N queue(s) hold messages but no capsule is configured to drain them"* — instead of being folded into the worker-wedge verdict. A heart-beating-but-`:stalled` worker is still reported STALLED regardless of which queue holds the backlog. Reason strings now truncate to the first 10 queue names with `(+N more)` so a flagged fleet of hundreds of queues doesn't produce a multi-KB log line. Refs #367, #366.
 - **`allowed_global_id_models` now actually guards ActiveJob arguments, not only EventBus payloads (issue #368).** The doctor warned in production that `nil` means "allow-all GlobalID arguments", but the allowlist was only enforced in `Serializer.locate_global_id` — reached from EventBus `_global_id` payloads — while the ordinary job path (`Executor` → `ActiveJob::Base.deserialize` → Rails' unrestricted `GlobalID::Locator`) never checked it. Operators who set an allowlist after following the doctor had a false sense of security; the common `SomeJob.perform_later(record)` pattern was unguarded. Job deserialization now goes through `Serializer.deserialize_job_data`, which walks `_aj_globalid` keys (including nested arrays/hashes) and reuses the same gate as EventBus when the allowlist is set; `nil` remains zero-cost allow-all. Rejected models raise `Pgbus::SerializationError` and are treated as a normal job failure. Apps with ActiveStorage attachments should include `ActiveStorage::Blob` (and related models) on the allowlist. Docs + doctor copy updated. Refs #368.

data/README.md

@@ -221,6 +221,20 @@ Pgbus::EventBus::Registry.instance.subscribe(
 )
 ```
 
+`idempotent!` uses a **two-phase claim**: a *pending* row in
+`pgbus_processed_events` is inserted before `handle` runs, and only stamped
+`completed_at` after `handle` returns. Deduplication applies to **completed**
+executions only — if the consumer process is killed mid-handler (deploy,
+OOM, supervisor watchdog), the redelivered message finds the pending claim
+and **re-runs the handler** instead of silently skipping it. The semantics
+are at-least-once with dedup of completed executions: the only
+double-execution window is a handler still running past its visibility
+timeout — the same window every non-idempotent handler already has.
+Installs created before this feature need the upgrade migration:
+`rails generate pgbus:add_processed_event_completion` (supports
+`--database`); until it runs, idempotent handlers fall back to the old
+single-phase claim and log a warning.
+
 ### 4. Start workers
 
 ```bash
@@ -1103,14 +1117,18 @@ For the **HTTP** transport, point the client at the mounted URL with a streamabl
 
 ### Health endpoints (liveness / readiness)
 
-For orchestrators like Kubernetes, Pgbus exposes two HTTP probes: `/livez` (is the serving process up?) and `/readyz` (are queues draining, or is a worker silently wedged?). `/readyz` runs the same `OK` / `DEGRADED` / `STALLED` verdict as the MCP `pgbus_health` tool — `STALLED` (visible backlog while workers heart-beat but don't claim) fails readiness.
+For orchestrators like Kubernetes, Pgbus exposes two HTTP probes: `/livez` (is the serving process up?) and `/readyz`. Readiness means different things in the two places the probes are served:
+
+- **Mounted in Rails** (`Pgbus::Web::HealthApp`): `/readyz` runs the cluster-wide `OK` / `DEGRADED` / `STALLED` verdict, same as the MCP `pgbus_health` tool — `STALLED` (visible backlog while workers heart-beat but don't claim) fails readiness.
+- **Standalone from the supervisor** (`health_port`): `/readyz` is **container-local** — did *this* supervisor finish booting, and are all the children *it* forked alive? That is the signal a rolling deploy's health gate needs; the cluster verdict would let a brand-new container pass on the strength of the *old* container's workers.
 
 | Path | Method | 200 | 503 | Touches DB |
 |---|---|---|---|---|
 | `/livez` | GET | always (`ok`) | never | no |
-| `/readyz` | GET | verdict `OK` or `DEGRADED` | verdict `STALLED`, or DB unreachable (`{"status":"ERROR"}`) | yes |
+| `/readyz` (mounted) | GET | verdict `OK` or `DEGRADED` | verdict `STALLED`, or DB unreachable (`{"status":"ERROR"}`) | yes |
+| `/readyz` (supervisor) | GET | `OK` — booted, all children live | `BOOTING`, `DEGRADED` (child down), `DRAINING` (stopping) | no |
 
-Unknown paths return `404`; non-`GET` methods return `405`. The `/readyz` body is the verdict JSON, so a probe failure is self-describing in the pod's event log.
+Unknown paths return `404`; non-`GET` methods return `405`. The `/readyz` body is JSON, so a probe failure is self-describing in the pod's event log.
 
 #### Mount in your Rails app
 
@@ -1151,6 +1169,61 @@ readinessProbe:
   httpGet: { path: /readyz, port: 9394 }
 ```
 
+The supervisor's `/readyz` answers from its own state, never the database:
+
+```json
+{ "status": "OK", "expected": 3, "live": 3 }
+```
+
+- `BOOTING` (503) until the connection is verified, queues are bootstrapped, and every configured child has been forked. `expected` is stamped at that instant.
+- `OK` (200) while all expected children are in the fork table. A clean worker recycle never dips the count — the snapshot refreshes after reap-and-restart each monitor pass.
+- `DEGRADED` (503) when a child died and is waiting out crash-restart backoff. During a rolling deploy this is the desired failure mode: a crash-looping replacement never goes ready, so the old container keeps running.
+- `DRAINING` (503) the moment a stop signal arrives.
+
+#### `pgbus-health`: container HEALTHCHECK probe
+
+`pgbus-health` ships with the gem: a dependency-free probe (plain Ruby + stdlib sockets — no Bundler, no Rails, nothing else loaded) that GETs `127.0.0.1:<port>/readyz` and exits `0` on 200, `1` on anything else, `2` on usage errors. Cheap enough for a 1–5s `HEALTHCHECK` interval, and it works in images without curl:
+
+```bash
+pgbus-health --port 9394              # or PGBUS_HEALTH_PORT=9394 pgbus-health
+pgbus-health --port 9394 --path /livez --timeout 2
+```
+
+### Rolling restarts (Kamal, docker)
+
+Kamal distributions with per-role health checks (for example the [`dash` branch](https://github.com/mhenrixon/kamal)) can rolling-restart a non-proxied job role: start the new container, poll its docker `HEALTHCHECK` until healthy, and only then `docker stop` the old one. Wire the pgbus container into that gate:
+
+```yaml
+# config/deploy.yml
+servers:
+  job:
+    hosts: [...]
+    cmd: bin/pgbus start
+    healthcheck:
+      cmd: bin/pgbus-health --port 9394
+      interval: 5s
+      start_period: 30s     # cover Rails boot + queue bootstrap
+    stop_timeout: 45        # must exceed pgbus shutdown_timeout (see below)
+env:
+  clear:
+    PGBUS_HEALTH_PORT: 9394
+```
+
+(`bundle binstubs pgbus` generates `bin/pgbus-health`; adjust the path if your image invokes gem executables differently.)
+
+**The shutdown timeline.** On `docker stop`, SIGTERM reaches the supervisor and readiness flips to `DRAINING`; children stop claiming work and drain in-flight jobs for up to `drain_timeout` (default 30s); the supervisor waits `shutdown_timeout` (default `drain_timeout + 5`) before SIGKILLing stragglers. Align the three knobs outside-in:
+
+```text
+orchestrator stop_timeout  >  pgbus shutdown_timeout  >  pgbus drain_timeout
+        45s                        35s (derived)                30s
+```
+
+If the orchestrator's stop grace period is *shorter* than `shutdown_timeout`, docker SIGKILLs the whole tree mid-drain and the graceful path never gets to finish. Raising `drain_timeout` raises the derived `shutdown_timeout` automatically; raise `stop_timeout` to match.
+
+**The overlap window is safe by construction.** Between "new container healthy" and "old container stopped", two supervisors run against the same database. Nothing double-fires: queue claims use `FOR UPDATE SKIP LOCKED`, `single_active_consumer` queues arbitrate via session-level advisory locks (released the instant a killed process's connection dies), two live recurring schedulers dedup on the `(task_key, run_at)` unique record, and dispatcher maintenance is idempotent. "One scheduler per deployment" is a steady-state rule; a deploy window may briefly violate it without consequence.
+
+**What a hard kill still costs.** Jobs killed past the drain window are redelivered after their visibility timeout (at-least-once holds) — but PGMQ's `read_ct` increments exactly like a logical failure, so a long-running job that straddles *repeated* deploy kills can be pushed to the DLQ without its code ever raising. `zombie_detection` logs exactly this pattern (`read_ct > 1` with no recorded failure). Keep jobs shorter than `drain_timeout`, or raise it (and `stop_timeout`) for queues that can't be. For `idempotent!` event handlers there is a separate crash-window caveat tracked in [#385](https://github.com/mhenrixon/pgbus/issues/385).
+
 ### Boot diagnostics banner
 
 `Supervisor#run` logs a one-block banner right after the heartbeat starts and before queues bootstrap, so a misconfigured deployment states its actual settings instead of forcing an operator to attach a console. Every line is `"[Pgbus] boot:"`-prefixed and renders cleanly under both the `:text` and `:json` log formatters:
@@ -1876,7 +1949,7 @@ A single preflight command that answers "is this environment healthy enough to r
 | Broadcast queue | — | Turbo broadcasts share the default queue in production, or `streams_broadcast_queue` is set but no worker capsule drains it |
 | Primary affinity | — | Job connection is on a read-only replica (`pg_is_in_recovery`) — a read/write-splitting pooler may be stalling jobs |
 | Dedicated connections | Streamer LISTEN and/or worker notify dedicated path cannot connect | — |
-| Connection budget | — (informational: prints how many direct LISTEN connections the current config pins — 1 per host under `worker_notify_scope: :supervisor`, one per fork under `:fork`, plus 1 per web process when streams are enabled) | — |
+| Connection budget | — (informational: prints how many direct LISTEN connections the current config pins — 1 per host under `worker_notify_scope: :supervisor`, one per fork under `:fork`; streams add 1 per web host under `streams_listen_scope: :master` or 1 per web process under `:process`) | — |
 
 ```bash
 pgbus doctor       # prints the report; exit 1 unless every check passed
@@ -2097,6 +2170,7 @@ Curated headline options for the README. The full operator reference (with types
 | `zombie_detection` | `true` | Detect and reclaim work from crashed workers |
 | `read_timeout` | `30` | Seconds before a single PGMQ read is bounded (libpq `statement_timeout` + `tcp_user_timeout` on a dedicated connection; nil disables) |
 | `drain_timeout` | `30` | Seconds to wait for in-flight jobs during graceful shutdown before abandoning them |
+| `shutdown_timeout` | `drain_timeout + 5` | Seconds the supervisor waits for children after TERM before SIGKILL; an orchestrator's stop grace period must exceed it |
 | `stall_threshold` | `300` | Seconds without progress before a worker is considered stalled |
 | `priority_levels` | `nil` | Number of priority sub-queues (nil = disabled, 2-10) |
 | `default_priority` | `1` | Default priority for jobs without explicit priority |

data/Rakefile

@@ -25,7 +25,7 @@ namespace :bench do
   # no-DB unit suite that bench:all runs in CI.
   db_benches = %w[connection_pool_bench integration_bench streams_bench streams_read_pool_bench
                   execution_modes_bench pool_swap_bench pool_autoscale_bench job_burst_bench
-                  notify_wake_bench notify_chaos_bench].freeze
+                  notify_wake_bench notify_chaos_bench streams_hub_bench].freeze
   # The unit suite is every *_bench.rb that doesn't need a database, derived
   # from the directory so a new unit bench is picked up automatically (kept in
   # sync with bench:one, which globs the same files).
@@ -95,6 +95,11 @@ namespace :bench do
     ruby "benchmarks/notify_chaos_bench.rb"
   end
 
+  desc "Run streams master-hub latency benchmark (#382 hop cost + census; requires PGBUS_DATABASE_URL)"
+  task :streams_hub do
+    ruby "benchmarks/streams_hub_bench.rb"
+  end
+
   desc "Run a single benchmark: rake bench:one[client_bench]"
   task :one, [:name] do |_t, args|
     name = args[:name] or abort "Usage: rake bench:one[serialization_bench|client_bench|...]"

data/app/models/pgbus/processed_event.rb

@@ -5,5 +5,50 @@ module Pgbus
     self.table_name = "pgbus_processed_events"
 
     scope :expired, ->(before) { where("processed_at < ?", before) }
+
+    @completion_column_mutex = Mutex.new
+
+    class << self
+      # Whether pgbus_processed_events has the completed_at column that backs
+      # the two-phase idempotency claim (issue #385). Detected once per process
+      # (memoized under a mutex) so the schema probe never lands on the
+      # per-event hot path. An upgraded gem running against a not-yet-migrated
+      # table gets `false` plus a one-time warning pointing at the upgrade
+      # generator — Handler then falls back to the legacy single-phase claim.
+      #
+      # A detection error (e.g. the database is briefly unreachable) is NOT
+      # memoized: it propagates to the caller — where the event's normal
+      # failure path leaves the message for VT redelivery — and the next
+      # delivery probes again.
+      def completion_column?
+        detected = @completion_column
+        return detected unless detected.nil?
+
+        @completion_column_mutex.synchronize do
+          @completion_column = detect_completion_column if @completion_column.nil?
+          @completion_column
+        end
+      end
+
+      # Test seam: clear the memoized detection so specs can exercise both
+      # schema shapes in one process.
+      def reset_completion_column_check!
+        @completion_column_mutex.synchronize { @completion_column = nil }
+      end
+
+      private
+
+      def detect_completion_column
+        supported = column_names.include?("completed_at")
+        unless supported
+          Pgbus.logger.warn do
+            "[Pgbus] pgbus_processed_events is missing the completed_at column; idempotent handlers " \
+              "fall back to single-phase claims (a crash mid-handler can skip work on redelivery). " \
+              "Run `rails generate pgbus:add_processed_event_completion` and migrate."
+          end
+        end
+        supported
+      end
+    end
   end
 end

data/exe/pgbus-health

@@ -0,0 +1,9 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+# Container HEALTHCHECK probe (issue #386). Deliberately loads ONLY the
+# probe file — never the pgbus gem, Bundler, or Rails — because a docker
+# HEALTHCHECK runs this every few seconds.
+require_relative "../lib/pgbus/health_probe"
+
+exit Pgbus::HealthProbe.run(ARGV)

data/lib/generators/pgbus/add_processed_event_completion_generator.rb

@@ -0,0 +1,45 @@
+# frozen_string_literal: true
+
+require "rails/generators"
+require "rails/generators/active_record"
+require_relative "migration_path"
+
+module Pgbus
+  module Generators
+    class AddProcessedEventCompletionGenerator < Rails::Generators::Base
+      include ActiveRecord::Generators::Migration
+      include MigrationPath
+
+      source_root File.expand_path("templates", __dir__)
+
+      desc "Add completed_at to pgbus_processed_events for two-phase idempotency claims " \
+           "(crash mid-handler re-runs instead of silently skipping)"
+
+      class_option :database,
+                   type: :string,
+                   default: nil,
+                   desc: "Use a separate database for pgbus tables (e.g. --database=pgbus)"
+
+      def create_migration_file
+        migration_template "add_processed_event_completion.rb.erb",
+                           File.join(pgbus_migrate_path, "add_pgbus_processed_event_completion.rb")
+      end
+
+      def display_post_install
+        say ""
+        say "Pgbus two-phase idempotency claim migration installed!", :green
+        say ""
+        say "Next steps:"
+        say "  1. Run: rails db:migrate#{migrate_command_suffix}"
+        say "  2. Restart pgbus: bin/pgbus start"
+        say ""
+      end
+
+      private
+
+      def migration_version
+        "[#{ActiveRecord::Migration.current_version}]"
+      end
+    end
+  end
+end

data/lib/generators/pgbus/templates/add_processed_event_completion.rb.erb

@@ -0,0 +1,19 @@
+class AddPgbusProcessedEventCompletion < ActiveRecord::Migration<%= migration_version %>
+  def up
+    add_column :pgbus_processed_events, :completed_at, :datetime
+
+    # Backfill legacy rows as completed: a row written by the single-phase
+    # code means handle() was at least started — treating it as completed
+    # preserves the old skip behavior instead of retroactively re-running
+    # history on the next delivery of an old event.
+    execute <<~SQL
+      UPDATE pgbus_processed_events
+      SET completed_at = processed_at
+      WHERE completed_at IS NULL
+    SQL
+  end
+
+  def down
+    remove_column :pgbus_processed_events, :completed_at
+  end
+end

data/lib/generators/pgbus/templates/migration.rb.erb

@@ -30,6 +30,9 @@ class CreatePgbusTables < ActiveRecord::Migration<%= migration_version %>
       t.string :event_id, null: false
       t.string :handler_class, null: false
       t.datetime :processed_at, null: false, default: -> { "CURRENT_TIMESTAMP" }
+      # Two-phase idempotency claim: NULL = claimed but not finished (a crash
+      # mid-handler re-runs on redelivery); set = completed, dedup applies.
+      t.datetime :completed_at
     end
 
     add_index :pgbus_processed_events, [:event_id, :handler_class],

data/lib/pgbus/configuration.rb

@@ -35,6 +35,16 @@ module Pgbus
     # wait, so recycling/deploy never wedges on a permanently-stuck job.
     attr_accessor :stall_threshold, :read_timeout, :drain_timeout
 
+    # shutdown_timeout bounds how long the supervisor waits for its children
+    # after forwarding TERM before escalating to SIGKILL. nil (default) derives
+    # drain_timeout + SHUTDOWN_TIMEOUT_MARGIN, so raising drain_timeout keeps
+    # the supervisor's deadline above the workers' drain window. An orchestrator
+    # stop grace period (Kamal stop_timeout, Kubernetes terminationGracePeriod)
+    # should exceed this value, or docker SIGKILLs the whole tree first.
+    attr_writer :shutdown_timeout
+
+    SHUTDOWN_TIMEOUT_MARGIN = 5
+
     # Dispatcher settings
     attr_accessor :dispatch_interval
 
@@ -238,6 +248,7 @@ module Pgbus
       @stall_threshold = 90
       @read_timeout = 30
       @drain_timeout = 30
+      @shutdown_timeout = nil
 
       @dispatch_interval = 1.0
 
@@ -270,6 +281,7 @@ module Pgbus
 
       @worker_notify_wakeup = nil
       @worker_notify_scope = :supervisor
+      @streams_listen_scope = :master
       @worker_notify_host = nil
       @worker_notify_port = nil
       @worker_notify_database_url = nil
@@ -632,6 +644,34 @@ module Pgbus
       @doctor_on_boot = coerced
     end
 
+    # Where the streams LISTEN connection lives (issue #382):
+    #   :master (default) — ONE shared listener in the preforking web master
+    #     (MasterHub); workers connect lazily over a Unix socket and fall back
+    #     to a per-worker listener whenever the hub is absent or dies.
+    #   :process — one listener per web process: the pre-0.13 behavior, and
+    #     the automatic behavior on single-mode / non-preforking servers.
+    attr_reader :streams_listen_scope
+
+    VALID_STREAMS_LISTEN_SCOPES = %i[master process].freeze
+
+    def streams_listen_scope=(scope)
+      coerced = case scope
+                when Symbol then scope
+                when String then scope.to_sym
+                else
+                  raise Pgbus::ConfigurationError,
+                        "Invalid streams_listen_scope type: #{scope.class}. " \
+                        "Must be :master (one shared LISTEN connection per web host) or :process (one per worker)"
+                end
+      unless VALID_STREAMS_LISTEN_SCOPES.include?(coerced)
+        raise Pgbus::ConfigurationError,
+              "Invalid streams_listen_scope: #{coerced.inspect}. " \
+              "Must be :master (one shared LISTEN connection per web host) or :process (one per worker)"
+      end
+
+      @streams_listen_scope = coerced
+    end
+
     VALID_WORKER_NOTIFY_SCOPES = %i[supervisor fork].freeze
 
     # Validated at assignment time like the other enum options. A String is
@@ -683,6 +723,8 @@ module Pgbus
       end
       raise Pgbus::ConfigurationError, "drain_timeout must be > 0" unless drain_timeout.is_a?(Numeric) && drain_timeout.positive?
 
+      validate_shutdown_timeout!
+
       unless stats_flush_size.is_a?(Integer) && stats_flush_size.positive?
         raise Pgbus::ConfigurationError, "stats_flush_size must be a positive integer"
       end
@@ -736,6 +778,30 @@ module Pgbus
       self
     end
 
+    # An explicit shutdown_timeout must be a positive number; nil keeps the
+    # derived drain_timeout + margin default. A value below drain_timeout is
+    # legal but self-defeating (the supervisor SIGKILLs workers mid-drain), so
+    # it warns instead of raising.
+    def validate_shutdown_timeout!
+      explicit = @shutdown_timeout
+      # Finite real only: Float::INFINITY would blow up Supervisor#shutdown's
+      # `Time.now + shutdown_timeout` before any child cleanup ran, and a
+      # Complex would crash `positive?` — reject both here, at boot.
+      valid = explicit.is_a?(Numeric) && explicit.real? && explicit.finite? && explicit.positive?
+      unless explicit.nil? || valid
+        raise Pgbus::ConfigurationError,
+              "shutdown_timeout must be a positive finite number or nil " \
+              "(defaults to drain_timeout + #{SHUTDOWN_TIMEOUT_MARGIN})"
+      end
+
+      return unless explicit && explicit < drain_timeout
+
+      Pgbus.logger.warn do
+        "[Pgbus] shutdown_timeout (#{explicit}s) is below drain_timeout (#{drain_timeout}s) — " \
+          "the supervisor will SIGKILL workers before their drain window ends"
+      end
+    end
+
     # Pre-1.0 surface-freeze: reject malformed values for core job-path keys at
     # boot rather than failing deep in a worker/dispatcher/poller/scheduler
     # thread, per-enqueue, or by silently corrupting queue names / leaving the
@@ -1208,6 +1274,12 @@ module Pgbus
     # because only one runs at a time per reactor thread.
     ASYNC_POOL_CONNECTIONS = 3
 
+    # Resolved supervisor SIGKILL deadline: the explicit value when set,
+    # otherwise drain_timeout + SHUTDOWN_TIMEOUT_MARGIN (see attr_writer docs).
+    def shutdown_timeout
+      @shutdown_timeout || (drain_timeout + SHUTDOWN_TIMEOUT_MARGIN)
+    end
+
     def resolved_pool_size
       return pool_size if pool_size
 

data/lib/pgbus/doctor.rb

@@ -394,12 +394,23 @@ module Pgbus
         consumers: consumers, con_plural: consumers == 1 ? "" : "s",
         share: count == 1 && @config.worker_notify_scope == :supervisor ? " share it" : ""
       )
-      detail += " + 1 per web-server process (streams)" if @config.streams_enabled
+      detail += streams_budget_clause if @config.streams_enabled
       Check.new(name: "Connection budget", status: :ok, detail: detail)
     rescue StandardError => e
       Check.new(name: "Connection budget", status: :warn, detail: "#{e.class}: #{e.message}")
     end
 
+    # Streams add their own LISTEN footprint on web hosts: one per host with
+    # the master hub (#382, the default — workers fall back per-worker only
+    # during a hub outage), one per web process under :process scope.
+    def streams_budget_clause
+      if @config.streams_listen_scope == :master
+        " + 1 per web host (streams master hub; per-worker fallback during a hub outage costs 1 per web process)"
+      else
+        " + 1 per web-server process (streams)"
+      end
+    end
+
     # Open one dedicated connection the way the runtime does, verify it
     # answers, close it. Returns nil on success, "label: error" on failure.
     def probe_dedicated_connection(label, opts)

data/lib/pgbus/event_bus/handler.rb

@@ -45,6 +45,7 @@ module Pgbus
         Instrumentation.instrument("pgbus.event_processed", instrument_payload) do
           handle(event)
         end
+        complete_claim!(event.event_id) if self.class.idempotent?
         :handled
       rescue StandardError => e
         instrument(
@@ -100,13 +101,25 @@ module Pgbus
         ActiveSupport::Notifications.instrument(event_name, payload)
       end
 
-      # Atomically claim idempotency: INSERT ... ON CONFLICT DO NOTHING.
-      # Returns true if this handler claimed the event (row was inserted),
-      # false if another handler already processed it (conflict, no insert).
+      # Two-phase idempotency claim (issue #385). Phase 1: atomically claim
+      # via INSERT ... ON CONFLICT DO NOTHING with completed_at NULL — a
+      # *pending* claim. Returns true when this delivery should run handle:
       #
-      # Uses an in-memory dedup cache to skip the DB for recently-seen events.
+      #   - insert won → fresh claim
+      #   - insert lost, completed_at NULL → a prior attempt claimed but was
+      #     killed before finishing (SIGKILL mid-handler); re-run so the crash
+      #     doesn't silently drop the execution. Safe: PGMQ's VT means the
+      #     prior holder is dead or wedged past its timeout — the same
+      #     at-least-once window every non-idempotent handler has.
+      #
+      # Returns false (skip) only for a *completed* execution. Phase 2 is
+      # complete_claim! after handle returns; only completed executions enter
+      # the in-memory dedup cache.
+      #
+      # Legacy fallback: without the completed_at column (upgraded gem,
+      # not-yet-migrated table) this degrades to the old single-phase claim.
       def claim_idempotency?(event_id)
-        cache_key = "#{event_id}:#{self.class.name}"
+        cache_key = dedup_key(event_id)
         return false if self.class.dedup_cache.seen?(cache_key)
 
         result = ProcessedEvent.insert(
@@ -114,9 +127,38 @@ module Pgbus
           unique_by: %i[event_id handler_class]
         )
 
-        claimed = result.rows.any?
+        unless ProcessedEvent.completion_column?
+          self.class.dedup_cache.mark!(cache_key)
+          return result.rows.any?
+        end
+
+        return true if result.rows.any?
+
+        completed_at = ProcessedEvent
+                       .where(event_id: event_id, handler_class: self.class.name)
+                       .pick(:completed_at)
+        return true if completed_at.nil? # pending claim (or purged row) → re-run
+
         self.class.dedup_cache.mark!(cache_key)
-        claimed
+        false
+      end
+
+      # Phase 2: stamp the claim completed and only then admit it to the
+      # dedup cache. Skipped on legacy schemas (single-phase claims are
+      # already cached at claim time). If this write fails, process!'s rescue
+      # re-raises, the consumer leaves the message for VT redelivery, and the
+      # still-pending claim re-runs — at-least-once, never a silent drop.
+      def complete_claim!(event_id)
+        return unless ProcessedEvent.completion_column?
+
+        ProcessedEvent
+          .where(event_id: event_id, handler_class: self.class.name)
+          .update_all(completed_at: Time.now.utc)
+        self.class.dedup_cache.mark!(dedup_key(event_id))
+      end
+
+      def dedup_key(event_id)
+        "#{event_id}:#{self.class.name}"
       end
     end
   end

data/lib/pgbus/generators/migration_detector.rb

@@ -77,6 +77,7 @@ module Pgbus
         add_outbox: "pgbus:add_outbox",
         add_recurring: "pgbus:add_recurring",
         add_failed_events_index: "pgbus:add_failed_events_index",
+        add_processed_event_completion: "pgbus:add_processed_event_completion",
         tune_autovacuum: "pgbus:tune_autovacuum",
         tune_fillfactor: "pgbus:tune_fillfactor"
       }.freeze
@@ -95,6 +96,7 @@ module Pgbus
         add_outbox: "outbox entries table (transactional outbox)",
         add_recurring: "recurring tasks + executions tables",
         add_failed_events_index: "unique index on pgbus_failed_events (queue_name, msg_id)",
+        add_processed_event_completion: "completed_at on pgbus_processed_events (two-phase idempotency claim)",
         tune_autovacuum: "autovacuum tuning for PGMQ queue and archive tables",
         tune_fillfactor: "fillfactor=70 on PGMQ queue tables (reduces page density during update churn)"
       }.freeze
@@ -120,6 +122,7 @@ module Pgbus
           *outbox_migrations,
           *recurring_migrations,
           *failed_events_index_migrations,
+          *processed_event_completion_migrations,
           *autovacuum_migrations,
           *fillfactor_migrations
         ]
@@ -210,6 +213,16 @@ module Pgbus
         [:add_failed_events_index]
       end
 
+      # completed_at backs the two-phase idempotency claim (issue #385).
+      # Without it, idempotent handlers fall back to single-phase claims and
+      # a crash mid-handler silently drops the execution on redelivery.
+      def processed_event_completion_migrations
+        return [] unless table_exists?("pgbus_processed_events")
+        return [] if column_names("pgbus_processed_events").include?("completed_at")
+
+        [:add_processed_event_completion]
+      end
+
       # Autovacuum tuning: check if any PGMQ queue table already has
       # custom autovacuum settings applied. If not, queue the migration.
       def autovacuum_migrations