pgbus 0.13.1 → 0.13.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 8c955f8fb2ac4e66fb2db388505ee83341aa6dec7c00f606162f7fba7c29cb75
-  data.tar.gz: a31e43c6f48cb10609d73dc507e9d177d25a299ae25c6f5d5a39a045018c32e8
+  metadata.gz: 295404bb7b5b6d289a2bfd4f10d9e5bc6df442f8d949454cf1dd78adf620a25f
+  data.tar.gz: 2ee9c8e2c40f47bb900e10d2ef4c55b5e21e3cd4d997c614c628b696ac759183
 SHA512:
-  metadata.gz: 7cd3fba720a5ab724fe4f0b5b89168cb2df4ba68e0a5a35cca725093724f8f2b2109c379a2263442111367001a52d2eda440eef8e1c2469653d2022d2a92d148
-  data.tar.gz: 4a4e5e1c2c229c599541e61ac6ad5602dfbfcf057b346b22b0271c10a48752c41075c7166b43dd55ece7ef4a30d79ec3fd6f5cac5b898b2578ecb90c73bcc2a8
+  metadata.gz: 184484168fd458dbdb09b98d12ae5d906ebb510a05abc626fd3812ee4cfebb5cd575c090b8014b9524a9aa19f65e99d1737df50b191ce79367e77858f72c5024
+  data.tar.gz: 92170d5077128d507968989f8d9d789bdc9afa7b4ccf92ecdc3b39b357a82608de39a83b4e9f2f899078466714bfb30f3f0c357a9dc977f35c9c21254c8b7dbd

data/CHANGELOG.md

@@ -2,6 +2,12 @@
 
 ### Added
 
+- **Health-checked rolling restarts for the job container (issue #386).** ⚠️ **Behavior change on the standalone `/readyz`.** The supervisor's `health_port` server previously answered `/readyz` with the cluster-wide HealthAnalyzer verdict — so during a rolling deploy a freshly-booted container could pass an orchestrator's health gate on the strength of the *old* container's still-heartbeating workers, and the old container (with all its capacity) was stopped before the new one had forked a single child. The standalone `/readyz` is now **container-local**: 200 only when *this* supervisor verified its connection, bootstrapped queues, forked every configured child, and all of them are currently alive — with 503 bodies `BOOTING` (pre-boot), `DEGRADED` (a child died and is waiting out crash-restart backoff — precisely the state a deploy gate must fail on, keeping the old container running), and `DRAINING` (stop signal received). No database access on the probe path; the supervisor publishes an immutable snapshot per monitor pass and the accept thread reads it. The Rails-mounted `Pgbus::Web::HealthApp` keeps the cluster-wide verdict unchanged. Alongside it: **`pgbus-health`**, a shipped executable probe for docker `HEALTHCHECK` blocks (plain Ruby + stdlib sockets, loads neither Bundler nor the gem — cheap at 1–5s intervals, works in curl-less images; exit 0/1/2 = healthy/unhealthy/usage), and a README "Rolling restarts (Kamal, docker)" guide covering the healthcheck block, stop-timeout alignment, overlap-window duplicate-supervisor safety, and the `read_ct`-vs-deploy-kill DLQ caveat. Refs #386.
+
+### Changed
+
+- **Shutdown budgets are now alignable end-to-end (issue #386).** New `config.shutdown_timeout` bounds how long the supervisor waits for children after forwarding TERM before escalating to SIGKILL — previously a hardcoded 30s, which silently SIGKILLed workers mid-drain the moment `drain_timeout` was raised past it. Default derives `drain_timeout + 5` so the deadline tracks the drain window automatically; an explicit value below `drain_timeout` logs a boot warning. `Consumer#shutdown`'s pool wait (its only drain bound) now follows `config.drain_timeout` instead of a hardcoded 30s, and `Worker#shutdown`'s post-drain residual wait drops from a second full 30s window to 5s — the drain loop already waited `drain_timeout`, and a job still running has proven it won't finish. Rule of thumb: orchestrator stop grace period > `shutdown_timeout` > `drain_timeout`. Refs #386.
+
 - **Streams: one LISTEN connection per web host — `streams_listen_scope` (issue #382).** ⚠️ **Default behavior change.** Previously every Puma worker lazily opened its own dedicated streams LISTEN connection on first SSE use, so a web host pinned one direct connection per worker. Under the new default (`streams_listen_scope = :master`) the `pgbus_streams` Puma plugin runs a **MasterHub** in the preforking master: ONE `Web::Streamer::Listener` on the refcounted union of every worker's stream channels, fanning wakes — **including ephemeral payloads** — out to workers over a Unix domain socket with length-prefixed frames (`Streamer::HubProtocol`). Workers connect lazily (nothing is inherited across fork) and the synchronous `ensure_listening` ack contract is preserved cross-process: a sub is registered before LISTEN executes and acked only after, so the no-lost-broadcast guarantee holds. Backpressure follows the streams rules: durable wakes are droppable at a per-worker cap (they self-heal via `read_after`), **ephemeral wakes are never dropped** — a worker that stops draining is evicted, which triggers its own fallback. **Fallback is per-worker listeners, not loss**: whenever the hub is absent or dies (no `preload_app!`, single-mode Puma, crash, eviction) each worker's `FailoverListener` swaps in a real per-worker `Listener` and re-LISTENs its recorded subscriptions — connection footprint balloons back to pre-#382 levels (census-visible) but no broadcast semantics change; the worker stays local until it recycles. Measured (local PG, n=50): the master→worker hop is noise-level free — single-broadcast SSE roundtrip p50 16.00ms via the hub vs 16.93ms per-worker. **`:master` effectively requires `preload_app!`** (the hub waits for the app's pgbus initializer; without it the deadline expires quietly and workers stay per-worker). **Rollback:** `config.streams_listen_scope = :process`. Refs #382, builds on the #381 patterns.
 
 - **Host-level shared LISTEN: `worker_notify_scope` — the supervisor now owns ONE direct LISTEN connection for the whole host (issue #381).** ⚠️ **Default behavior change.** Previously every worker fork and every consumer fork opened its own dedicated LISTEN connection (`NotifyListener`), so a host's direct-connection footprint scaled with fork count — on transaction-pool PgBouncer platforms those connections come out of the scarcest slice of `max_connections`, and a 5-capsule + 2-consumer host pinned 7. Under the new default (`config.worker_notify_scope = :supervisor`) the supervisor runs a single `NotifyHub`: one `NotifyListener` on the union of every capsule's and consumer's queue channels (wildcards via the shared resolver, consumer sets via the registry), fanning wakes out to forks over per-fork pipes (`W` wake / `H` healthy / `P` degraded bytes; a fork whose pipe reports degraded or reaches EOF falls back to fast polling exactly like a failed local listener). Footprint drops to **1 direct LISTEN connection per job host**, verified by integration test: routing is per-fork (an insert wakes only the forks reading that queue, wildcard capsules unconditionally), and `pg_terminate_backend` on the shared connection is survived — reconnect, re-LISTEN, wakes flow again. **Rollback:** `config.worker_notify_scope = :fork` restores the previous per-fork listeners byte-for-byte. Dedicated LISTEN connections are now census-tagged `application_name=pgbus-listen` so `pg_stat_activity` can count them. Refs #381.
@@ -16,6 +22,8 @@
 
 ### Fixed
 
+- **`idempotent!` handlers no longer silently drop an execution when the consumer is killed mid-handler — the idempotency claim is now two-phase (issue #385).** `EventBus::Handler#process!` inserted the `pgbus_processed_events` claim row *before* `handle(event)` ran, so a SIGKILL between claim and completion (supervisor watchdog, container runtime after its stop timeout, OOM — i.e. every container-replacement deploy's worst case) turned the redelivered message into a skip: the mechanism meant to dedup *duplicate* executions converted a crash *during* the one permitted execution into at-most-once. The claim now lands with a NULL `completed_at` (*pending*) and is stamped completed only after `handle` returns; on redelivery a pending claim **re-runs** the handler while a completed claim still skips. The in-memory dedup cache only ever records completed executions, and a `handle` that raises leaves the claim pending so VT redelivery retries it. The only double-execution window is a still-alive handler running past its visibility timeout — the same at-least-once window every non-idempotent handler already has. **Upgrading:** new installs get the column from `pgbus:install`; existing installs run `rails generate pgbus:add_processed_event_completion` (also picked up by `pgbus:update`; `--database` supported) — legacy rows are backfilled as completed so history is not retroactively re-run. Until the migration runs, an upgraded gem detects the missing column once and falls back to the legacy single-phase claim with a warning naming the generator. Refs #385.
+
 - **SIGTERM no longer segfaults the process: both LISTEN listeners stopped closing their PG connection from the stopping thread (issue #375).** `Process::NotifyListener#stop` (supervisor thread, on SIGTERM) closed the listener's dedicated `PG::Connection` to interrupt the blocking `wait_for_notify`, but left `@conn` pointing at it — so the listener thread unwound into `run_loop`'s `ensure` and ran `safe_unlisten_all`, exec'ing `UNLISTEN` on a connection the other thread was concurrently freeing. `PG::Connection#close` is `PQfinish`: it frees the PGconn **and its OpenSSL objects**, so `PQsendQuery` walked into freed TLS state and the whole worker died with `[BUG] Segmentation fault` — reproducibly, on essentially every container-replacement deploy against a TLS Postgres, sometimes in several forked workers at once. `rescue PG::Error` cannot catch a C-level SEGV, so the "safe" in `safe_unlisten_all` never held. **The connection is now single-owner**: the listener thread is the only thread that may exec, wait, or close on it, from build through teardown. `#stop` signals by clearing `@running` and joining — nothing more — and the listener thread closes its own connection in `run_loop`'s `ensure`. The stop is observed within one `wait_for_notify` timeout (`health_check_ms`, **250 ms** in the default configuration since workers/consumers derive it from `polling_interval`), so `#stop`'s join budget is now `health_check_ms + 5s` instead of a flat 5 s — a flat timeout could expire before a listener with a large `health_check_ms` had even one chance to observe the stop. The teardown `UNLISTEN` round-trip is **dropped entirely** rather than merely made safe: it ran immediately before the close, and closing a session deregisters every `LISTEN` server-side, so it bought nothing at any time. The identical defect in `Web::Streamer::Listener` — same close-from-`#stop`, same `UNLISTEN`-in-`ensure`, and there without even a mutex around `@conn` — is fixed the same way, so the dashboard's Puma worker stops crashing on the same deploy; its listener thread now also closes its own connection (previously only `#stop` did). Cost: shutdown can take up to one health-check cycle longer per listener — measured against a real PostgreSQL at the default `health_check_ms` of 250: **~207 ms on a fully idle queue, ~5 ms when the queue has any NOTIFY traffic** (a notification returns the wait immediately, and the loop then sees the cleared flag). 40 start/stop cycles under concurrent NOTIFY load leaked zero `LISTEN` backends. Thanks to the reporter for the crash dumps and the root-cause analysis. Refs #375.
 - **The health verdict no longer emits false STALLED reports — `max_read_ct` was never populated, and the wedge signal counted queues no capsule drains (issue #367).** Two correctness defects in `Pgbus::MCP::HealthAnalyzer` (surfaced through `pgbus doctor` / `pgbus_health` / the MCP health tool). **(1)** The `all_unread?` wedge check read `:max_read_ct`, but the metrics query never selected it — so the guard always degenerated to "never claimed" and a busy-but-healthy queue caught mid-burst produced a STALLED verdict with a factually wrong "read_ct=0 (never claimed)" reason. The metrics query now exposes a per-queue **`visible_unread_length`** (`count(*) WHERE vt <= NOW() AND read_ct = 0`) and the analyzer keys the wedge off *visible, never-claimed* messages. Counting per visible message (not `max(read_ct)` over the whole table) means one retried message left in-queue after its backoff, or an in-flight message claimed by a peer, can no longer veto the signal for a pile of genuinely-unclaimed jobs. **(2)** The verdict reasoned about **every** non-DLQ/non-stream queue against the global worker fleet — but a worker can only claim from queues its capsule subscribes to, so "M workers alive but never claimed" was vacuous for a queue nobody drains (ad-hoc queues, unregistered stream queues — see #366). The analyzer now intersects the STALLED backlog with `Web::DataSource#drained_queue_names` (each configured capsule's queues, priority `_pN` sub-tables expanded via the client's queue strategy, unioned with EventBus handler queues; `nil` for a `*` wildcard = drains everything, fail-open on error). Queues nobody drains get their own DEGRADED signal — *"N queue(s) hold messages but no capsule is configured to drain them"* — instead of being folded into the worker-wedge verdict. A heart-beating-but-`:stalled` worker is still reported STALLED regardless of which queue holds the backlog. Reason strings now truncate to the first 10 queue names with `(+N more)` so a flagged fleet of hundreds of queues doesn't produce a multi-KB log line. Refs #367, #366.
 - **`allowed_global_id_models` now actually guards ActiveJob arguments, not only EventBus payloads (issue #368).** The doctor warned in production that `nil` means "allow-all GlobalID arguments", but the allowlist was only enforced in `Serializer.locate_global_id` — reached from EventBus `_global_id` payloads — while the ordinary job path (`Executor` → `ActiveJob::Base.deserialize` → Rails' unrestricted `GlobalID::Locator`) never checked it. Operators who set an allowlist after following the doctor had a false sense of security; the common `SomeJob.perform_later(record)` pattern was unguarded. Job deserialization now goes through `Serializer.deserialize_job_data`, which walks `_aj_globalid` keys (including nested arrays/hashes) and reuses the same gate as EventBus when the allowlist is set; `nil` remains zero-cost allow-all. Rejected models raise `Pgbus::SerializationError` and are treated as a normal job failure. Apps with ActiveStorage attachments should include `ActiveStorage::Blob` (and related models) on the allowlist. Docs + doctor copy updated. Refs #368.

data/README.md

@@ -221,6 +221,20 @@ Pgbus::EventBus::Registry.instance.subscribe(
 )
 ```
 
+`idempotent!` uses a **two-phase claim**: a *pending* row in
+`pgbus_processed_events` is inserted before `handle` runs, and only stamped
+`completed_at` after `handle` returns. Deduplication applies to **completed**
+executions only — if the consumer process is killed mid-handler (deploy,
+OOM, supervisor watchdog), the redelivered message finds the pending claim
+and **re-runs the handler** instead of silently skipping it. The semantics
+are at-least-once with dedup of completed executions: the only
+double-execution window is a handler still running past its visibility
+timeout — the same window every non-idempotent handler already has.
+Installs created before this feature need the upgrade migration:
+`rails generate pgbus:add_processed_event_completion` (supports
+`--database`); until it runs, idempotent handlers fall back to the old
+single-phase claim and log a warning.
+
 ### 4. Start workers
 
 ```bash
@@ -1103,14 +1117,18 @@ For the **HTTP** transport, point the client at the mounted URL with a streamabl
 
 ### Health endpoints (liveness / readiness)
 
-For orchestrators like Kubernetes, Pgbus exposes two HTTP probes: `/livez` (is the serving process up?) and `/readyz` (are queues draining, or is a worker silently wedged?). `/readyz` runs the same `OK` / `DEGRADED` / `STALLED` verdict as the MCP `pgbus_health` tool — `STALLED` (visible backlog while workers heart-beat but don't claim) fails readiness.
+For orchestrators like Kubernetes, Pgbus exposes two HTTP probes: `/livez` (is the serving process up?) and `/readyz`. Readiness means different things in the two places the probes are served:
+
+- **Mounted in Rails** (`Pgbus::Web::HealthApp`): `/readyz` runs the cluster-wide `OK` / `DEGRADED` / `STALLED` verdict, same as the MCP `pgbus_health` tool — `STALLED` (visible backlog while workers heart-beat but don't claim) fails readiness.
+- **Standalone from the supervisor** (`health_port`): `/readyz` is **container-local** — did *this* supervisor finish booting, and are all the children *it* forked alive? That is the signal a rolling deploy's health gate needs; the cluster verdict would let a brand-new container pass on the strength of the *old* container's workers.
 
 | Path | Method | 200 | 503 | Touches DB |
 |---|---|---|---|---|
 | `/livez` | GET | always (`ok`) | never | no |
-| `/readyz` | GET | verdict `OK` or `DEGRADED` | verdict `STALLED`, or DB unreachable (`{"status":"ERROR"}`) | yes |
+| `/readyz` (mounted) | GET | verdict `OK` or `DEGRADED` | verdict `STALLED`, or DB unreachable (`{"status":"ERROR"}`) | yes |
+| `/readyz` (supervisor) | GET | `OK` — booted, all children live | `BOOTING`, `DEGRADED` (child down), `DRAINING` (stopping) | no |
 
-Unknown paths return `404`; non-`GET` methods return `405`. The `/readyz` body is the verdict JSON, so a probe failure is self-describing in the pod's event log.
+Unknown paths return `404`; non-`GET` methods return `405`. The `/readyz` body is JSON, so a probe failure is self-describing in the pod's event log.
 
 #### Mount in your Rails app
 
@@ -1151,6 +1169,61 @@ readinessProbe:
   httpGet: { path: /readyz, port: 9394 }
 ```
 
+The supervisor's `/readyz` answers from its own state, never the database:
+
+```json
+{ "status": "OK", "expected": 3, "live": 3 }
+```
+
+- `BOOTING` (503) until the connection is verified, queues are bootstrapped, and every configured child has been forked. `expected` is stamped at that instant.
+- `OK` (200) while all expected children are in the fork table. A clean worker recycle never dips the count — the snapshot refreshes after reap-and-restart each monitor pass.
+- `DEGRADED` (503) when a child died and is waiting out crash-restart backoff. During a rolling deploy this is the desired failure mode: a crash-looping replacement never goes ready, so the old container keeps running.
+- `DRAINING` (503) the moment a stop signal arrives.
+
+#### `pgbus-health`: container HEALTHCHECK probe
+
+`pgbus-health` ships with the gem: a dependency-free probe (plain Ruby + stdlib sockets — no Bundler, no Rails, nothing else loaded) that GETs `127.0.0.1:<port>/readyz` and exits `0` on 200, `1` on anything else, `2` on usage errors. Cheap enough for a 1–5s `HEALTHCHECK` interval, and it works in images without curl:
+
+```bash
+pgbus-health --port 9394              # or PGBUS_HEALTH_PORT=9394 pgbus-health
+pgbus-health --port 9394 --path /livez --timeout 2
+```
+
+### Rolling restarts (Kamal, docker)
+
+Kamal distributions with per-role health checks (for example the [`dash` branch](https://github.com/mhenrixon/kamal)) can rolling-restart a non-proxied job role: start the new container, poll its docker `HEALTHCHECK` until healthy, and only then `docker stop` the old one. Wire the pgbus container into that gate:
+
+```yaml
+# config/deploy.yml
+servers:
+  job:
+    hosts: [...]
+    cmd: bin/pgbus start
+    healthcheck:
+      cmd: bin/pgbus-health --port 9394
+      interval: 5s
+      start_period: 30s     # cover Rails boot + queue bootstrap
+    stop_timeout: 45        # must exceed pgbus shutdown_timeout (see below)
+env:
+  clear:
+    PGBUS_HEALTH_PORT: 9394
+```
+
+(`bundle binstubs pgbus` generates `bin/pgbus-health`; adjust the path if your image invokes gem executables differently.)
+
+**The shutdown timeline.** On `docker stop`, SIGTERM reaches the supervisor and readiness flips to `DRAINING`; children stop claiming work and drain in-flight jobs for up to `drain_timeout` (default 30s); the supervisor waits `shutdown_timeout` (default `drain_timeout + 5`) before SIGKILLing stragglers. Align the three knobs outside-in:
+
+```text
+orchestrator stop_timeout  >  pgbus shutdown_timeout  >  pgbus drain_timeout
+        45s                        35s (derived)                30s
+```
+
+If the orchestrator's stop grace period is *shorter* than `shutdown_timeout`, docker SIGKILLs the whole tree mid-drain and the graceful path never gets to finish. Raising `drain_timeout` raises the derived `shutdown_timeout` automatically; raise `stop_timeout` to match.
+
+**The overlap window is safe by construction.** Between "new container healthy" and "old container stopped", two supervisors run against the same database. Nothing double-fires: queue claims use `FOR UPDATE SKIP LOCKED`, `single_active_consumer` queues arbitrate via session-level advisory locks (released the instant a killed process's connection dies), two live recurring schedulers dedup on the `(task_key, run_at)` unique record, and dispatcher maintenance is idempotent. "One scheduler per deployment" is a steady-state rule; a deploy window may briefly violate it without consequence.
+
+**What a hard kill still costs.** Jobs killed past the drain window are redelivered after their visibility timeout (at-least-once holds) — but PGMQ's `read_ct` increments exactly like a logical failure, so a long-running job that straddles *repeated* deploy kills can be pushed to the DLQ without its code ever raising. `zombie_detection` logs exactly this pattern (`read_ct > 1` with no recorded failure). Keep jobs shorter than `drain_timeout`, or raise it (and `stop_timeout`) for queues that can't be. For `idempotent!` event handlers there is a separate crash-window caveat tracked in [#385](https://github.com/mhenrixon/pgbus/issues/385).
+
 ### Boot diagnostics banner
 
 `Supervisor#run` logs a one-block banner right after the heartbeat starts and before queues bootstrap, so a misconfigured deployment states its actual settings instead of forcing an operator to attach a console. Every line is `"[Pgbus] boot:"`-prefixed and renders cleanly under both the `:text` and `:json` log formatters:
@@ -2097,6 +2170,7 @@ Curated headline options for the README. The full operator reference (with types
 | `zombie_detection` | `true` | Detect and reclaim work from crashed workers |
 | `read_timeout` | `30` | Seconds before a single PGMQ read is bounded (libpq `statement_timeout` + `tcp_user_timeout` on a dedicated connection; nil disables) |
 | `drain_timeout` | `30` | Seconds to wait for in-flight jobs during graceful shutdown before abandoning them |
+| `shutdown_timeout` | `drain_timeout + 5` | Seconds the supervisor waits for children after TERM before SIGKILL; an orchestrator's stop grace period must exceed it |
 | `stall_threshold` | `300` | Seconds without progress before a worker is considered stalled |
 | `priority_levels` | `nil` | Number of priority sub-queues (nil = disabled, 2-10) |
 | `default_priority` | `1` | Default priority for jobs without explicit priority |

data/app/models/pgbus/processed_event.rb

@@ -5,5 +5,50 @@ module Pgbus
     self.table_name = "pgbus_processed_events"
 
     scope :expired, ->(before) { where("processed_at < ?", before) }
+
+    @completion_column_mutex = Mutex.new
+
+    class << self
+      # Whether pgbus_processed_events has the completed_at column that backs
+      # the two-phase idempotency claim (issue #385). Detected once per process
+      # (memoized under a mutex) so the schema probe never lands on the
+      # per-event hot path. An upgraded gem running against a not-yet-migrated
+      # table gets `false` plus a one-time warning pointing at the upgrade
+      # generator — Handler then falls back to the legacy single-phase claim.
+      #
+      # A detection error (e.g. the database is briefly unreachable) is NOT
+      # memoized: it propagates to the caller — where the event's normal
+      # failure path leaves the message for VT redelivery — and the next
+      # delivery probes again.
+      def completion_column?
+        detected = @completion_column
+        return detected unless detected.nil?
+
+        @completion_column_mutex.synchronize do
+          @completion_column = detect_completion_column if @completion_column.nil?
+          @completion_column
+        end
+      end
+
+      # Test seam: clear the memoized detection so specs can exercise both
+      # schema shapes in one process.
+      def reset_completion_column_check!
+        @completion_column_mutex.synchronize { @completion_column = nil }
+      end
+
+      private
+
+      def detect_completion_column
+        supported = column_names.include?("completed_at")
+        unless supported
+          Pgbus.logger.warn do
+            "[Pgbus] pgbus_processed_events is missing the completed_at column; idempotent handlers " \
+              "fall back to single-phase claims (a crash mid-handler can skip work on redelivery). " \
+              "Run `rails generate pgbus:add_processed_event_completion` and migrate."
+          end
+        end
+        supported
+      end
+    end
   end
 end

data/exe/pgbus-health

@@ -0,0 +1,9 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+# Container HEALTHCHECK probe (issue #386). Deliberately loads ONLY the
+# probe file — never the pgbus gem, Bundler, or Rails — because a docker
+# HEALTHCHECK runs this every few seconds.
+require_relative "../lib/pgbus/health_probe"
+
+exit Pgbus::HealthProbe.run(ARGV)

data/lib/generators/pgbus/add_processed_event_completion_generator.rb

@@ -0,0 +1,45 @@
+# frozen_string_literal: true
+
+require "rails/generators"
+require "rails/generators/active_record"
+require_relative "migration_path"
+
+module Pgbus
+  module Generators
+    class AddProcessedEventCompletionGenerator < Rails::Generators::Base
+      include ActiveRecord::Generators::Migration
+      include MigrationPath
+
+      source_root File.expand_path("templates", __dir__)
+
+      desc "Add completed_at to pgbus_processed_events for two-phase idempotency claims " \
+           "(crash mid-handler re-runs instead of silently skipping)"
+
+      class_option :database,
+                   type: :string,
+                   default: nil,
+                   desc: "Use a separate database for pgbus tables (e.g. --database=pgbus)"
+
+      def create_migration_file
+        migration_template "add_processed_event_completion.rb.erb",
+                           File.join(pgbus_migrate_path, "add_pgbus_processed_event_completion.rb")
+      end
+
+      def display_post_install
+        say ""
+        say "Pgbus two-phase idempotency claim migration installed!", :green
+        say ""
+        say "Next steps:"
+        say "  1. Run: rails db:migrate#{migrate_command_suffix}"
+        say "  2. Restart pgbus: bin/pgbus start"
+        say ""
+      end
+
+      private
+
+      def migration_version
+        "[#{ActiveRecord::Migration.current_version}]"
+      end
+    end
+  end
+end

data/lib/generators/pgbus/templates/add_processed_event_completion.rb.erb

@@ -0,0 +1,19 @@
+class AddPgbusProcessedEventCompletion < ActiveRecord::Migration<%= migration_version %>
+  def up
+    add_column :pgbus_processed_events, :completed_at, :datetime
+
+    # Backfill legacy rows as completed: a row written by the single-phase
+    # code means handle() was at least started — treating it as completed
+    # preserves the old skip behavior instead of retroactively re-running
+    # history on the next delivery of an old event.
+    execute <<~SQL
+      UPDATE pgbus_processed_events
+      SET completed_at = processed_at
+      WHERE completed_at IS NULL
+    SQL
+  end
+
+  def down
+    remove_column :pgbus_processed_events, :completed_at
+  end
+end

data/lib/generators/pgbus/templates/migration.rb.erb

@@ -30,6 +30,9 @@ class CreatePgbusTables < ActiveRecord::Migration<%= migration_version %>
       t.string :event_id, null: false
       t.string :handler_class, null: false
       t.datetime :processed_at, null: false, default: -> { "CURRENT_TIMESTAMP" }
+      # Two-phase idempotency claim: NULL = claimed but not finished (a crash
+      # mid-handler re-runs on redelivery); set = completed, dedup applies.
+      t.datetime :completed_at
     end
 
     add_index :pgbus_processed_events, [:event_id, :handler_class],

data/lib/pgbus/configuration.rb

@@ -35,6 +35,16 @@ module Pgbus
     # wait, so recycling/deploy never wedges on a permanently-stuck job.
     attr_accessor :stall_threshold, :read_timeout, :drain_timeout
 
+    # shutdown_timeout bounds how long the supervisor waits for its children
+    # after forwarding TERM before escalating to SIGKILL. nil (default) derives
+    # drain_timeout + SHUTDOWN_TIMEOUT_MARGIN, so raising drain_timeout keeps
+    # the supervisor's deadline above the workers' drain window. An orchestrator
+    # stop grace period (Kamal stop_timeout, Kubernetes terminationGracePeriod)
+    # should exceed this value, or docker SIGKILLs the whole tree first.
+    attr_writer :shutdown_timeout
+
+    SHUTDOWN_TIMEOUT_MARGIN = 5
+
     # Dispatcher settings
     attr_accessor :dispatch_interval
 
@@ -238,6 +248,7 @@ module Pgbus
       @stall_threshold = 90
       @read_timeout = 30
       @drain_timeout = 30
+      @shutdown_timeout = nil
 
       @dispatch_interval = 1.0
 
@@ -712,6 +723,8 @@ module Pgbus
       end
       raise Pgbus::ConfigurationError, "drain_timeout must be > 0" unless drain_timeout.is_a?(Numeric) && drain_timeout.positive?
 
+      validate_shutdown_timeout!
+
       unless stats_flush_size.is_a?(Integer) && stats_flush_size.positive?
         raise Pgbus::ConfigurationError, "stats_flush_size must be a positive integer"
       end
@@ -765,6 +778,30 @@ module Pgbus
       self
     end
 
+    # An explicit shutdown_timeout must be a positive number; nil keeps the
+    # derived drain_timeout + margin default. A value below drain_timeout is
+    # legal but self-defeating (the supervisor SIGKILLs workers mid-drain), so
+    # it warns instead of raising.
+    def validate_shutdown_timeout!
+      explicit = @shutdown_timeout
+      # Finite real only: Float::INFINITY would blow up Supervisor#shutdown's
+      # `Time.now + shutdown_timeout` before any child cleanup ran, and a
+      # Complex would crash `positive?` — reject both here, at boot.
+      valid = explicit.is_a?(Numeric) && explicit.real? && explicit.finite? && explicit.positive?
+      unless explicit.nil? || valid
+        raise Pgbus::ConfigurationError,
+              "shutdown_timeout must be a positive finite number or nil " \
+              "(defaults to drain_timeout + #{SHUTDOWN_TIMEOUT_MARGIN})"
+      end
+
+      return unless explicit && explicit < drain_timeout
+
+      Pgbus.logger.warn do
+        "[Pgbus] shutdown_timeout (#{explicit}s) is below drain_timeout (#{drain_timeout}s) — " \
+          "the supervisor will SIGKILL workers before their drain window ends"
+      end
+    end
+
     # Pre-1.0 surface-freeze: reject malformed values for core job-path keys at
     # boot rather than failing deep in a worker/dispatcher/poller/scheduler
     # thread, per-enqueue, or by silently corrupting queue names / leaving the
@@ -1237,6 +1274,12 @@ module Pgbus
     # because only one runs at a time per reactor thread.
     ASYNC_POOL_CONNECTIONS = 3
 
+    # Resolved supervisor SIGKILL deadline: the explicit value when set,
+    # otherwise drain_timeout + SHUTDOWN_TIMEOUT_MARGIN (see attr_writer docs).
+    def shutdown_timeout
+      @shutdown_timeout || (drain_timeout + SHUTDOWN_TIMEOUT_MARGIN)
+    end
+
     def resolved_pool_size
       return pool_size if pool_size
 

data/lib/pgbus/event_bus/handler.rb

@@ -45,6 +45,7 @@ module Pgbus
         Instrumentation.instrument("pgbus.event_processed", instrument_payload) do
           handle(event)
         end
+        complete_claim!(event.event_id) if self.class.idempotent?
         :handled
       rescue StandardError => e
         instrument(
@@ -100,13 +101,25 @@ module Pgbus
         ActiveSupport::Notifications.instrument(event_name, payload)
       end
 
-      # Atomically claim idempotency: INSERT ... ON CONFLICT DO NOTHING.
-      # Returns true if this handler claimed the event (row was inserted),
-      # false if another handler already processed it (conflict, no insert).
+      # Two-phase idempotency claim (issue #385). Phase 1: atomically claim
+      # via INSERT ... ON CONFLICT DO NOTHING with completed_at NULL — a
+      # *pending* claim. Returns true when this delivery should run handle:
       #
-      # Uses an in-memory dedup cache to skip the DB for recently-seen events.
+      #   - insert won → fresh claim
+      #   - insert lost, completed_at NULL → a prior attempt claimed but was
+      #     killed before finishing (SIGKILL mid-handler); re-run so the crash
+      #     doesn't silently drop the execution. Safe: PGMQ's VT means the
+      #     prior holder is dead or wedged past its timeout — the same
+      #     at-least-once window every non-idempotent handler has.
+      #
+      # Returns false (skip) only for a *completed* execution. Phase 2 is
+      # complete_claim! after handle returns; only completed executions enter
+      # the in-memory dedup cache.
+      #
+      # Legacy fallback: without the completed_at column (upgraded gem,
+      # not-yet-migrated table) this degrades to the old single-phase claim.
       def claim_idempotency?(event_id)
-        cache_key = "#{event_id}:#{self.class.name}"
+        cache_key = dedup_key(event_id)
         return false if self.class.dedup_cache.seen?(cache_key)
 
         result = ProcessedEvent.insert(
@@ -114,9 +127,38 @@ module Pgbus
           unique_by: %i[event_id handler_class]
         )
 
-        claimed = result.rows.any?
+        unless ProcessedEvent.completion_column?
+          self.class.dedup_cache.mark!(cache_key)
+          return result.rows.any?
+        end
+
+        return true if result.rows.any?
+
+        completed_at = ProcessedEvent
+                       .where(event_id: event_id, handler_class: self.class.name)
+                       .pick(:completed_at)
+        return true if completed_at.nil? # pending claim (or purged row) → re-run
+
         self.class.dedup_cache.mark!(cache_key)
-        claimed
+        false
+      end
+
+      # Phase 2: stamp the claim completed and only then admit it to the
+      # dedup cache. Skipped on legacy schemas (single-phase claims are
+      # already cached at claim time). If this write fails, process!'s rescue
+      # re-raises, the consumer leaves the message for VT redelivery, and the
+      # still-pending claim re-runs — at-least-once, never a silent drop.
+      def complete_claim!(event_id)
+        return unless ProcessedEvent.completion_column?
+
+        ProcessedEvent
+          .where(event_id: event_id, handler_class: self.class.name)
+          .update_all(completed_at: Time.now.utc)
+        self.class.dedup_cache.mark!(dedup_key(event_id))
+      end
+
+      def dedup_key(event_id)
+        "#{event_id}:#{self.class.name}"
       end
     end
   end

data/lib/pgbus/generators/migration_detector.rb

@@ -77,6 +77,7 @@ module Pgbus
         add_outbox: "pgbus:add_outbox",
         add_recurring: "pgbus:add_recurring",
         add_failed_events_index: "pgbus:add_failed_events_index",
+        add_processed_event_completion: "pgbus:add_processed_event_completion",
         tune_autovacuum: "pgbus:tune_autovacuum",
         tune_fillfactor: "pgbus:tune_fillfactor"
       }.freeze
@@ -95,6 +96,7 @@ module Pgbus
         add_outbox: "outbox entries table (transactional outbox)",
         add_recurring: "recurring tasks + executions tables",
         add_failed_events_index: "unique index on pgbus_failed_events (queue_name, msg_id)",
+        add_processed_event_completion: "completed_at on pgbus_processed_events (two-phase idempotency claim)",
         tune_autovacuum: "autovacuum tuning for PGMQ queue and archive tables",
         tune_fillfactor: "fillfactor=70 on PGMQ queue tables (reduces page density during update churn)"
       }.freeze
@@ -120,6 +122,7 @@ module Pgbus
           *outbox_migrations,
           *recurring_migrations,
           *failed_events_index_migrations,
+          *processed_event_completion_migrations,
           *autovacuum_migrations,
           *fillfactor_migrations
         ]
@@ -210,6 +213,16 @@ module Pgbus
         [:add_failed_events_index]
       end
 
+      # completed_at backs the two-phase idempotency claim (issue #385).
+      # Without it, idempotent handlers fall back to single-phase claims and
+      # a crash mid-handler silently drops the execution on redelivery.
+      def processed_event_completion_migrations
+        return [] unless table_exists?("pgbus_processed_events")
+        return [] if column_names("pgbus_processed_events").include?("completed_at")
+
+        [:add_processed_event_completion]
+      end
+
       # Autovacuum tuning: check if any PGMQ queue table already has
       # custom autovacuum settings applied. If not, queue the migration.
       def autovacuum_migrations

data/lib/pgbus/health_probe.rb

@@ -0,0 +1,132 @@
+# frozen_string_literal: true
+
+require "socket"
+
+module Pgbus
+  # Dependency-free readiness probe for container HEALTHCHECKs (issue #386).
+  #
+  # exe/pgbus-health loads this file via require_relative and nothing else:
+  # a docker HEALTHCHECK runs the probe every few seconds, so it must never
+  # drag in Bundler, Zeitwerk, Rails, or the rest of the gem. Only Ruby's
+  # bundled socket library is allowed here.
+  #
+  #   healthcheck:
+  #     cmd: bin/pgbus-health            # port from PGBUS_HEALTH_PORT
+  #     cmd: bin/pgbus-health --port 9394 --path /livez
+  #
+  # Exit codes: 0 healthy (HTTP 2xx), 1 unhealthy (non-2xx, refused, timeout),
+  # 2 usage error (no/invalid port).
+  class HealthProbe
+    EXIT_OK = 0
+    EXIT_UNHEALTHY = 1
+    EXIT_USAGE = 2
+
+    DEFAULT_PATH = "/readyz"
+    DEFAULT_TIMEOUT = 2.0
+    HOST = "127.0.0.1"
+
+    USAGE = "usage: pgbus-health [--port PORT] [--path PATH] [--timeout SECONDS]\n       " \
+            "port falls back to the PGBUS_HEALTH_PORT environment variable\n"
+
+    def self.run(argv, env: ENV, out: $stdout, err: $stderr)
+      new(argv, env: env, out: out, err: err).run
+    end
+
+    def initialize(argv, env: ENV, out: $stdout, err: $stderr)
+      @out = out
+      @err = err
+      @path = DEFAULT_PATH
+      @timeout = DEFAULT_TIMEOUT
+      @port = env["PGBUS_HEALTH_PORT"]
+      @usage_error = false
+      parse(argv)
+    end
+
+    def run
+      return usage_failure if @usage_error
+
+      port = Integer(@port, exception: false)
+      # Out-of-range ports would reach Socket.tcp and raise SocketError — a
+      # backtrace where a HEALTHCHECK needs a deterministic exit code.
+      return usage_failure unless port&.between?(1, 65_535)
+
+      probe(port)
+    end
+
+    private
+
+    # Hand-rolled flag parsing: three flags do not justify optparse in a
+    # script whose reason to exist is loading nothing.
+    def parse(argv)
+      args = argv.dup
+      until args.empty?
+        flag = args.shift
+        value = args.shift
+        return @usage_error = true if value.nil?
+
+        case flag
+        when "--port" then @port = value
+        when "--path" then @path = value
+        when "--timeout"
+          # A typo'd timeout must be a usage error, not `to_f`'s silent 0.0 —
+          # a zero deadline reports the container unhealthy on every probe.
+          timeout = Float(value, exception: false)
+          return @usage_error = true unless timeout&.positive?
+
+          @timeout = timeout
+        else
+          return @usage_error = true
+        end
+      end
+    end
+
+    def usage_failure
+      @err.write(USAGE)
+      EXIT_USAGE
+    end
+
+    def probe(port)
+      status = http_status(port)
+      healthy = status&.between?(200, 299)
+      @out.write("pgbus-health: #{@path} -> #{status || "no response"}\n")
+      healthy ? EXIT_OK : EXIT_UNHEALTHY
+    rescue SystemCallError, IOError, SocketError => e
+      @err.write("pgbus-health: #{@path} -> #{e.class}: #{e.message}\n")
+      EXIT_UNHEALTHY
+    end
+
+    # Minimal HTTP/1.0 exchange: send the request, read just the status line.
+    # The deadline covers connect and read together.
+    def http_status(port)
+      deadline = monotonic_now + @timeout
+      Socket.tcp(HOST, port, connect_timeout: @timeout) do |sock|
+        sock.write("GET #{@path} HTTP/1.0\r\nHost: #{HOST}\r\nConnection: close\r\n\r\n")
+        line = read_status_line(sock, deadline)
+        code = line&.split(" ", 3)&.fetch(1, nil)
+        Integer(code, exception: false)
+      end
+    end
+
+    def read_status_line(sock, deadline)
+      buffer = +""
+      until buffer.include?("\n")
+        remaining = deadline - monotonic_now
+        return nil if remaining <= 0 || !sock.wait_readable(remaining)
+
+        chunk = sock.read_nonblock(1024, exception: false)
+        return nil if chunk.nil? # EOF before a full status line
+        next if chunk == :wait_readable # spurious wakeup — re-wait on the deadline
+
+        buffer << chunk
+      end
+      buffer[/\A[^\r\n]*/]
+    end
+
+    # ::Process, not Process — inside the Pgbus namespace the bare constant
+    # resolves to Pgbus::Process (the process model), which is also why this
+    # file must never be renamed into that namespace.
+    def monotonic_now
+      ::Process.clock_gettime(::Process::CLOCK_MONOTONIC)
+    end
+  end
+end

data/lib/pgbus/process/consumer.rb

@@ -480,7 +480,10 @@ module Pgbus
       def shutdown
         stop_wake_source
         @pool.shutdown
-        @pool.wait_for_termination(30)
+        # The consumer has no quiesce-gated drain loop like Worker's, so this
+        # wait IS its drain window — bound it by the same knob workers use
+        # instead of a hardcoded 30s (issue #386).
+        @pool.wait_for_termination(config.drain_timeout)
         @stat_buffer&.stop
         @heartbeat&.stop
         restore_signals

data/lib/pgbus/process/readiness_snapshot.rb

@@ -0,0 +1,30 @@
+# frozen_string_literal: true
+
+module Pgbus
+  module Process
+    # Immutable container-local readiness state, published by the supervisor
+    # (one atomic swap per monitor pass) and read by the standalone health
+    # server's accept thread — the immutability is what makes the cross-thread
+    # handoff safe without a lock (issue #386).
+    #
+    # `expected` is the child count forked by boot_processes; `live` is the
+    # current fork-table size. A child sitting in crash-restart backoff keeps
+    # `live < expected`, which is exactly the signal a rolling deploy's health
+    # gate needs to fail on: the replacement container never goes ready, and
+    # the orchestrator keeps the old container running.
+    ReadinessSnapshot = Data.define(:booted, :shutting_down, :expected, :live) do
+      def ready?
+        booted && !shutting_down && live >= expected
+      end
+
+      # DRAINING wins over BOOTING: a supervisor told to stop mid-boot is
+      # leaving, not arriving, and must never look like it will become ready.
+      def status
+        return "DRAINING" if shutting_down
+        return "BOOTING" unless booted
+
+        ready? ? "OK" : "DEGRADED"
+      end
+    end
+  end
+end

data/lib/pgbus/process/supervisor.rb

@@ -44,6 +44,16 @@ module Pgbus
         @pending_restarts = pending_restarts
         @crash_counts = Hash.new(0)
         @notify_hub = notify_hub
+        @intended_children = 0
+        @readiness = Concurrent::AtomicReference.new(
+          ReadinessSnapshot.new(booted: false, shutting_down: shutting_down, expected: 0, live: forks.size)
+        )
+      end
+
+      # The current container-local readiness state. Safe to call from any
+      # thread (the health server's accept thread reads it per probe).
+      def readiness_snapshot
+        @readiness.get
       end
 
       def shutting_down?
@@ -97,6 +107,7 @@ module Pgbus
         start_notify_hub
 
         boot_processes
+        mark_booted
         monitor_loop
       ensure
         shutdown
@@ -105,17 +116,51 @@ module Pgbus
       def graceful_shutdown
         Pgbus.logger.info { "[Pgbus] Supervisor: graceful shutdown requested" }
         @shutting_down = true
+        refresh_readiness
         signal_children("TERM")
       end
 
       def immediate_shutdown
         Pgbus.logger.warn { "[Pgbus] Supervisor: immediate shutdown requested" }
         @shutting_down = true
+        refresh_readiness
         signal_children("QUIT")
       end
 
       private
 
+      # Boot is complete: connection verified, queues bootstrapped, every
+      # configured child fork ATTEMPTED. The baseline is the larger of the
+      # intended-attempt count and the fork-table size: a boot-time fork
+      # failure (EAGAIN/ENOMEM, logged-and-swallowed in fork_*) leaves
+      # intended > live, so the readiness gate reports DEGRADED instead of
+      # blessing a container that is missing workers. Roles that legitimately
+      # declined to boot (scheduler with no recurring tasks) never reach a
+      # fork_* method and are counted by neither side.
+      def mark_booted
+        @booted = true
+        @expected_children = [@intended_children, @forks.size].max
+        refresh_readiness
+      end
+
+      # Count a child the configuration intends this boot to run. Called at
+      # the top of every fork_* method — before the fork can fail — and only
+      # pre-boot, so restart_child's re-forks never inflate the baseline.
+      def note_intended_child
+        @intended_children += 1 unless @booted
+      end
+
+      # Publish a fresh snapshot; the swapped-in Data is immutable, so the
+      # health server's accept thread always reads a consistent state.
+      def refresh_readiness
+        @readiness.set(
+          ReadinessSnapshot.new(
+            booted: !!@booted, shutting_down: @shutting_down,
+            expected: @expected_children || 0, live: @forks.size
+          )
+        )
+      end
+
       # Log a single boot diagnostics banner: the settings that actually
       # determine whether this deployment works. One consecutive block of
       # "[Pgbus] boot:"-prefixed info lines so it reads cleanly under both the
@@ -253,6 +298,7 @@ module Pgbus
       end
 
       def fork_worker(worker_config, slot: nil)
+        note_intended_child
         queues = worker_config[:queues] || [config.default_queue]
         threads = worker_config[:threads] || 5
         single_active = worker_config[:single_active_consumer] || false
@@ -337,6 +383,7 @@ module Pgbus
       end
 
       def fork_dispatcher
+        note_intended_child
         pid = fork do
           restore_signals
           setup_child_process
@@ -364,6 +411,7 @@ module Pgbus
       end
 
       def fork_scheduler
+        note_intended_child
         pid = fork do
           restore_signals
           setup_child_process
@@ -427,6 +475,7 @@ module Pgbus
       end
 
       def fork_consumer(consumer_config, slot: nil)
+        note_intended_child
         # Array() so a consumer entry without :topics can't NoMethodError the
         # supervisor on the topics.join log lines below.
         topics = Array(consumer_config[:topics])
@@ -504,6 +553,7 @@ module Pgbus
       end
 
       def fork_outbox_poller
+        note_intended_child
         pid = fork do
           restore_signals
           setup_child_process
@@ -537,6 +587,9 @@ module Pgbus
             # refresh, and fork status broadcast (issue #381).
             @notify_hub&.tick
           end
+          # After reap + restarts so a clean recycle (reaped and re-forked in
+          # the same pass) never dips the published live count (issue #386).
+          refresh_readiness
           interruptible_sleep(FORK_WAIT)
         end
       end
@@ -821,7 +874,12 @@ module Pgbus
       def start_health_server
         return unless config.health_port
 
-        @health_server = Pgbus::Web::HealthServer.new(port: config.health_port, bind: config.health_bind)
+        # The standalone server answers /readyz from THIS supervisor's
+        # container-local snapshot — a rolling deploy's health gate must
+        # measure the new container, not the fleet-wide verdict a sibling
+        # container's workers can satisfy (issue #386).
+        app = Pgbus::Web::HealthApp.new(local_readiness: -> { readiness_snapshot })
+        @health_server = Pgbus::Web::HealthServer.new(port: config.health_port, bind: config.health_bind, app: app)
         @health_server.start
       end
 
@@ -876,8 +934,11 @@ module Pgbus
       end
 
       def shutdown
-        # Wait for all children with timeout
-        deadline = Time.now + 30
+        # Wait for children to drain and exit, bounded by config.shutdown_timeout
+        # (default drain_timeout + 5) so raising the drain window can never
+        # mean SIGKILLing workers mid-drain. An orchestrator's stop grace
+        # period should exceed this value (issue #386).
+        deadline = Time.now + config.shutdown_timeout
 
         until @forks.empty? || Time.now > deadline
           reap_children

data/lib/pgbus/process/worker.rb

@@ -155,6 +155,12 @@ module Pgbus
       NOTIFY_RETRY_BASE_SECONDS = 5
       NOTIFY_RETRY_MAX_SECONDS = 300
 
+      # Residual pool wait in #shutdown, AFTER the drain loop already spent up
+      # to config.drain_timeout on in-flight jobs. Short by design: a job still
+      # running has proven it won't finish, and this wait competes with the
+      # supervisor's shutdown_timeout deadline (issue #386).
+      POOL_TERMINATION_WAIT = 5
+
       def run
         setup_signals
         start_heartbeat
@@ -175,9 +181,9 @@ module Pgbus
 
           break if @lifecycle.stopped?
           # quiesced? (all slots free), not idle? (any slot free) — exiting
-          # with work still in flight abandons those jobs to the 30s
-          # wait_for_termination timeout in shutdown. Bounded by
-          # config.drain_timeout so a stuck job can't wedge the loop forever.
+          # with work still in flight abandons those jobs to shutdown's short
+          # POOL_TERMINATION_WAIT residual. Bounded by config.drain_timeout so
+          # a stuck job can't wedge the loop forever.
           break if @lifecycle.draining? && (@pool.quiesced? || drain_deadline_exceeded?)
 
           claim_and_execute if @lifecycle.can_process?
@@ -792,7 +798,11 @@ module Pgbus
         Pgbus.logger.info { "[Pgbus] Worker draining thread pool..." }
         stop_wake_source
         @pool.shutdown
-        @pool.wait_for_termination(30)
+        # Residual wait only: the drain loop already waited up to
+        # config.drain_timeout for in-flight jobs. A job still running here has
+        # proven it won't finish; waiting another full window would push the
+        # worker past the supervisor's shutdown_timeout deadline (issue #386).
+        @pool.wait_for_termination(POOL_TERMINATION_WAIT)
         @stat_buffer&.stop
         @queue_lock&.unlock_all
         @heartbeat&.stop

data/lib/pgbus/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Pgbus
-  VERSION = "0.13.1"
+  VERSION = "0.13.2"
 end

data/lib/pgbus/web/health_app.rb

@@ -49,8 +49,15 @@ module Pgbus
       # @param data_source [Pgbus::Web::DataSource, nil] read layer for /readyz.
       #   nil (the default) builds a fresh DataSource per readiness check, which
       #   avoids serving stale metrics from a long-lived app's memoized instance.
-      def initialize(data_source: nil)
+      # @param local_readiness [#call, nil] when set, /readyz answers from this
+      #   callable's {Process::ReadinessSnapshot} instead of the cluster-wide
+      #   analyzer — the supervisor's standalone HealthServer passes its own
+      #   snapshot so a rolling deploy's health gate measures THIS container,
+      #   not the fleet (issue #386). The Rails-mounted app leaves it nil and
+      #   keeps the cluster verdict.
+      def initialize(data_source: nil, local_readiness: nil)
         @data_source = data_source
+        @local_readiness = local_readiness
       end
 
       def call(env)
@@ -69,6 +76,8 @@ module Pgbus
       end
 
       def readyz
+        return local_readyz if @local_readiness
+
         # HealthAnalyzer lives in the MCP namespace, which is excluded from
         # Zeitwerk (its *tools* subclass the optional `mcp` gem). The analyzer
         # itself has no gem dependency, so require just that one file — the
@@ -83,6 +92,19 @@ module Pgbus
         [503, JSON_HEADERS.dup, [{ status: "ERROR", error: e.message }.to_json]]
       end
 
+      # Container-local readiness: no database, no analyzer — just the
+      # supervisor's published snapshot. The error path mirrors the cluster
+      # readyz: 503 ERROR, logged, never swallowed.
+      def local_readyz
+        snapshot = @local_readiness.call
+        status = snapshot.ready? ? 200 : 503
+        body = { status: snapshot.status, expected: snapshot.expected, live: snapshot.live }
+        [status, JSON_HEADERS.dup, [body.to_json]]
+      rescue StandardError => e
+        Pgbus.logger.error { "[Pgbus::Web::HealthApp] local readiness check failed: #{e.class}: #{e.message}" }
+        [503, JSON_HEADERS.dup, [{ status: "ERROR", error: e.message }.to_json]]
+      end
+
       # Reuse an injected DataSource (tests, an app that wants one shared
       # instance); otherwise build a fresh one each check so per-instance
       # memoization can never serve stale queue/process metrics.

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: pgbus
 version: !ruby/object:Gem::Version
-  version: 0.13.1
+  version: 0.13.2
 platform: ruby
 authors:
 - Mikael Henriksson
@@ -114,6 +114,7 @@ email:
 - mikael@mhenrixon.com
 executables:
 - pgbus
+- pgbus-health
 extensions: []
 extra_rdoc_files: []
 files:
@@ -207,6 +208,7 @@ files:
 - config/locales/sv.yml
 - config/routes.rb
 - exe/pgbus
+- exe/pgbus-health
 - lib/active_job/queue_adapters/pgbus_adapter.rb
 - lib/generators/pgbus/add_failed_events_index_generator.rb
 - lib/generators/pgbus/add_job_stats_generator.rb
@@ -214,6 +216,7 @@ files:
 - lib/generators/pgbus/add_job_stats_queue_index_generator.rb
 - lib/generators/pgbus/add_outbox_generator.rb
 - lib/generators/pgbus/add_presence_generator.rb
+- lib/generators/pgbus/add_processed_event_completion_generator.rb
 - lib/generators/pgbus/add_queue_states_generator.rb
 - lib/generators/pgbus/add_recurring_generator.rb
 - lib/generators/pgbus/add_stream_queues_generator.rb
@@ -228,6 +231,7 @@ files:
 - lib/generators/pgbus/templates/add_job_stats_queue_index.rb.erb
 - lib/generators/pgbus/templates/add_outbox.rb.erb
 - lib/generators/pgbus/templates/add_presence.rb.erb
+- lib/generators/pgbus/templates/add_processed_event_completion.rb.erb
 - lib/generators/pgbus/templates/add_queue_states.rb.erb
 - lib/generators/pgbus/templates/add_recurring_tables.rb.erb
 - lib/generators/pgbus/templates/add_stream_queues.rb.erb
@@ -282,6 +286,7 @@ files:
 - lib/pgbus/failed_event_recorder.rb
 - lib/pgbus/generators/database_target_detector.rb
 - lib/pgbus/generators/migration_detector.rb
+- lib/pgbus/health_probe.rb
 - lib/pgbus/instrumentation.rb
 - lib/pgbus/integrations/appsignal.rb
 - lib/pgbus/integrations/appsignal/dashboard.json
@@ -333,6 +338,7 @@ files:
 - lib/pgbus/process/notify_probe.rb
 - lib/pgbus/process/primary_validator.rb
 - lib/pgbus/process/queue_lock.rb
+- lib/pgbus/process/readiness_snapshot.rb
 - lib/pgbus/process/signal_handler.rb
 - lib/pgbus/process/supervisor.rb
 - lib/pgbus/process/wake_pipe.rb