pgbus 0.11.4 → 0.12.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: c43ef6f3c3efe57de0f647a0811f83bcf98f08e49c63e2da3d6b9cae1c4d3f16
-  data.tar.gz: edb510a56a9410f8cbf127504c3e31ae459561620a7ef62c7011c97f665908ae
+  metadata.gz: a966ed340b2cd097f82b926f90647a0d481df35da76bb4ef06f4e3c13e3d5692
+  data.tar.gz: 779124d08216918b02a162f2a1263c4115cd7a7fb9404dc0055dc940da28793f
 SHA512:
-  metadata.gz: 21f9209e70c028b51c2794ac2a73d71f55b06e0a1e23fede2c7a074c27c732bf88959a859c2f9f79de8484cd4cb51cc4c876fc801ab6981fc05541926312dd34
-  data.tar.gz: f15cf91037ccee58333e0e74648db8336acf40511ef495a608428ff839e340e5c6ddd06a89cde323d2a6f08ea5b2f5abfbd9717acfbd5ed3d2e3023a1b47f84b
+  metadata.gz: b25519a1d9ea9e1e2700b959a5ebcf03adf8efbcb130001f1e962067e0def2546b1644e199d24d2b849b29ba67f15bc83eef5f41506ffde3d6ea5d79da752af7
+  data.tar.gz: f0ec03ccc2b075b90f6b87cc5eb1315d5d3834b2cdd7c9c19fc851f350196f6860b941ec0566aac0315f531bf29356f9a1f94b68e852aa497eb642833ddc3800

data/CHANGELOG.md

@@ -1,5 +1,11 @@
 ## [Unreleased]
 
+### Fixed
+
+- **`streams_pool_database_url` / `streams_pool_host` / `streams_pool_port` — route the streams PGMQ pool independently of the streamer's LISTEN connection (issue #358).** 0.12.0 started building the dedicated streams pool from `streams_connection_options` (correct for a separate streams database), but that method is also how pooler-bypass installs pin the LISTEN connection to the **direct** Postgres port (`streams_port = 5432`, the documented "workers go through PgBouncer, streamer goes direct" pattern) — so upgrading silently moved up to `streams_pool_size` connections **per process** onto the direct port, whose `max_connections` ceiling on managed Postgres is typically low. A modest fleet exhausts it (`FATAL: remaining connection slots are reserved for roles with the SUPERUSER attribute`): `StreamApp` then 500s every SSE connect and durable broadcasts fail at publish, intermittently, because the pools are lazy. Only LISTEN actually needs the direct port (it dies at transaction-pool COMMIT boundaries); the pool's broadcast INSERTs and replay reads are plain pooler-safe SQL. The new `streams_pool_*` triple routes the POOL independently — applied to the base options exactly like the `streams_*` and `worker_notify_*` groups: set `streams_pool_port` back to the pooled port and only the LISTEN pins remain direct. Default `nil` = follows `streams_connection_options`, byte-identical to 0.12.0, so separate-streams-DB installs are unchanged. Refs #358.
+- **The health verdict no longer reads durable stream queues as a wedged fleet (issue #359).** The Process-liveness signal (`pgbus doctor`, `pgbus_health`, the MCP health tool) treats *visible messages with `read_ct=0` while workers are alive* as the silent-worker-wedge signature — but durable stream delivery is a non-consuming peek, so **every** stream queue matches it permanently by design. On a streams-heavy install the verdict screamed STALLED listing hundreds of healthy stream queues, burying real wedges in noise. `HealthAnalyzer` now excludes queues registered in the `pgbus_stream_queues` registry from the operational set (exactly like DLQs) — out of the STALLED/DEGRADED reasons and the backlog totals — via a new `Web::DataSource#stream_queue_names` (loaded fresh per verdict; degrades to an empty set on pre-registry installs). Same bug class as #308/#309, same registry cure. Refs #359.
+- **The `upgrade_pgmq` migration no longer silently kills NOTIFY-gated wakeups fleet-wide (issue #360).** The generated migration drops every pgmq function **with CASCADE** — which also drops the `trigger_notify_queue_insert_listeners` trigger from every existing queue table (it depends on the dropped `pgmq.notify_queue_listeners()`), and `install_sql` re-created the functions but never the per-queue triggers. Result: after every PGMQ schema upgrade, NOTIFY wakeups (#174) silently died and workers fell back to `polling_interval` polling — nothing errored; latency degraded and DB poll load rose. Trigger re-install only happened in `ensure_single_queue` (per-process memoized), so on the common deploy ordering where the job supervisor boots *before* migrations run, the supervisor installed triggers at boot and the migration wiped them minutes later — with nothing left to re-install them until a full process restart. New `PgmqSchema.reinstall_notify_triggers_sql` replays `pgmq.enable_notify_insert` for every row in `pgmq.notify_insert_throttle` (a table, preserved by the upgrade, recording exactly which queues had notify and at what throttle; the function is idempotent), and the migration template runs it after `install_sql` — restoring every trigger at its original interval. No-op on vendored versions without the notify feature. **Already ran an upgrade migration?** Run `Pgbus::PgmqSchema.reinstall_notify_triggers_sql` once (e.g. from a console: `ActiveRecord::Base.connection.execute(...)`) or restart the fleet; `pgbus doctor`'s LISTEN/NOTIFY check confirms the repair. Refs #360.
+
 ### Breaking Changes
 
 - **`config.validate!` now rejects malformed values for 12 core config keys at boot instead of failing deep in a worker or silently misbehaving (issue #335).** Before the 1.0 surface freeze, these headline job-path keys had **zero** validation while streams keys had 14 checks — so a bad value surfaced as a crash inside a worker/dispatcher/poller/scheduler thread, per-enqueue, or by silently corrupting every derived queue name / leaving the dashboard open. `Pgbus.configure` (eager-validating by default) now raises `Pgbus::ConfigurationError` naming the key for: the worker-recycling trio `max_jobs_per_worker`/`max_memory_mb`/`max_worker_lifetime` (positive number or `nil` to disable); the intervals `dispatch_interval`/`outbox_poll_interval`/`recurring_schedule_interval` (positive number); `outbox_batch_size` (positive integer); `default_priority` (non-negative integer — `0` is a valid level); `queue_prefix`/`default_queue` (non-empty String — an empty prefix silently produced malformed queue names); `web_auth` (a callable or `nil` — a non-callable silently left the dashboard open); `error_reporters` (an Array); and `connects_to` (a Hash or `nil` — previously a malformed value raised a raw `TypeError` deep in engine boot). This only rejects configuration that was already broken; if a `Pgbus.configure` block set one of these to an invalid value it will now fail loudly at boot (the intended pre-freeze tightening). Set `config.eager_validation = false` to defer. Refs #335, #282.
@@ -10,6 +16,8 @@
 
 ### Added
 
+- **`Pgbus::Client#reload` — operator escape hatch that drops every pooled PGMQ connection (job pool AND the live streams pool) and lets the pools rebuild lazily on next checkout (issue #354).** Built on pgmq-ruby 0.7.1's `PGMQ::Client#reload`; use it to recover connections libpq still reports as `CONNECTION_OK` but that are in fact wedged (e.g. after a wall-clock interrupt cut a query mid-flight), which pgmq-ruby's checkout health check cannot detect. Unlike `#close` the pools stay usable, and connections checked out by other threads mid-reload are unaffected. The streams half goes through the `ResizablePool` wrapper, so it always targets the live (possibly hot-swapped, #323) client. No-op with a warning on the shared-AR Proc path — those pool slots wrap ActiveRecord's own raw connection, and pgbus won't close a socket it doesn't own. Refs #354.
+- **PGMQ: vendored schema v1.12.0.** Adds `pgmq.read_grouped_head_with_poll` — a polling wrapper over `read_grouped_head` (reads the heads of N FIFO groups, waiting up to `max_poll_seconds` for messages); the 1.11.1 → 1.12.0 upstream delta is otherwise comments/whitespace only, so nothing pgbus calls through `Pgbus::Client` changes. Fresh embedded installs get 1.12.0; existing installs: `rake pgbus:pgmq:status`, then `rails generate pgbus:upgrade_pgmq` + `rails db:migrate` (`rails db:migrate:pgbus` on a separate-database install). No `Pgbus::Client` API is added for the new function (the locked pgmq-ruby does not expose it; pgbus has no consumer). Refs #351.
 - **Doctor check 10: "Dedicated connections" — the preflight now opens the streamer/notify-listener connections exactly the way the runtime does (issue #352).** The streamer's LISTEN connection and the worker `NotifyListener` bypass the Client's pgmq pools entirely (they own a single raw `PG` connection each), so every existing doctor check — all of which probe through `Pgbus::Client` — passed while both dedicated paths were broken. The new check builds each configured dedicated connection via `Pgbus::DedicatedConnection` (streams when `streams_enabled`, notify when `worker_notify_wakeup?`), runs `SELECT 1`, and closes it, failing with the underlying PG error and the affected path label. Not strict-fatal (same reasoning as the Database check: a transient DB blip at boot must not lockstep-abort a fleet), but with `doctor_on_boot` it turns "every SSE request 500s after deploy" into a named failing check in the boot report. Refs #352.
 - **`pgbus dashboard` — print the vendored AppSignal dashboards as import-ready JSON.** The gem ships four AppSignal dashboard definitions (the main one submitted to appsignal/public_config#80 plus the health/streams/throughput extras), but they're stored in the automated-dashboard format — a `metric_keys` trigger wrapping the `dashboard` object — which AppSignal's "Import dashboard" dialog doesn't accept, so trying one on a real app meant hand-unwrapping JSON. `pgbus dashboard [main|health|streams|throughput]` prints the inner dashboard object ready to paste into the import dialog (`pgbus dashboard health | pbcopy`); `pgbus dashboard --list` enumerates the available names and titles.
 - **`config.doctor_on_boot` (+ `pgbus start --doctor` / `--doctor-strict`) — run the doctor preflight inside the booting supervisor, one Rails boot instead of two (issue #347).** The recommended deploy preflight was `bin/pgbus doctor || true` in the container entrypoint before `bin/pgbus start` — but both commands `require config/environment`, so the job container boots the full Rails app **twice** on every deploy (cost scaling with app boot time), and running doctor pre-supervisor makes the `Process liveness` check false-fail (no workers exist yet), which is exactly why the entrypoint has to swallow the exit code with `|| true` — losing any chance to gate on a genuinely-fatal finding. Now `config.doctor_on_boot = :report` (or `pgbus start --doctor`) runs the checks in the already-booted supervisor, after the DB is verified reachable and queues are bootstrapped but **before any worker is forked**: one boot, and the worker-dependent `Process liveness` check is skipped (it has no workers to observe yet and would false-fail on stale prior-generation rows during a redeploy). `config.doctor_on_boot = :strict` (or `--doctor-strict`) additionally **refuses to boot** — raising before forking anything, so the `ensure shutdown` path tears down the heartbeat/health server and no child starts — but *only* on a genuinely-fatal, non-transient check: a `Configuration` failure (a real config bug) or an **absent** PGMQ schema. It deliberately does **not** abort on `Queues`/`Database` failures — those are the transients the lenient queue bootstrap is built to ride out (children crash-and-backoff until the DB recovers), and `verify_connection!` already gated a hard-down DB moments earlier; making them strict-fatal would take down a whole fleet's cold boot in lockstep on a momentarily-saturated primary. Default `nil`/`false` = off (byte-identical to today; the supervisor constructs no `Doctor` and does zero extra work). The standalone `pgbus doctor` command and its exit-code semantics are unchanged. Refs #347.
@@ -36,6 +44,8 @@
 
 ### Fixed
 
+- **`ensures_uniqueness` / `limits_concurrency` declared on a base class now reach every subclass — previously the declaration was silently inert for them (issue #357).** Both macros stored their config in a class-level ivar (`@pgbus_uniqueness` / `@pgbus_concurrency`) and every lookup read `active_job.class.pgbus_uniqueness` without walking ancestors — class-level ivars are not inherited in Ruby, so `class RecurringBase < ApplicationJob; ensures_uniqueness strategy: :until_executed; end` gave subclasses NO uniqueness key, no scheduler overlap protection, and no #333 guard, with no warning that the natural reading of the code was false. The readers now fall back to the nearest ancestor's declaration (a class's own declaration still beats an inherited one). Coupled fix, required to make inheritance safe rather than harmful: no default key proc is stored anymore — the old `key || ->(*) { name }` captured the DECLARING class, so an inheritance fix alone would have collapsed every subclass into ONE shared key (the base class's name), and with `on_conflict: :discard` sibling jobs would silently discard each other's enqueues — strictly worse than inert. The class-name default is now resolved from the ENQUEUED job's class at all three call sites (`Uniqueness.resolve_key`, `Concurrency.resolve_key`, and the recurring scheduler's `resolve_uniqueness_key`), and `config[:key]` is `nil` when no explicit `key:` was given. Consumer note: concrete-class declarations resolve to exactly the same keys as before, but a base-class declaration that was silently inert now ACTIVATES per-subclass uniqueness/concurrency on upgrade — audit any base-class declarations (the macro finally does what it says). In the scheduler path the no-key default is additionally qualified with the recurring task's `args:` (`SyncJob:["site_a"]`) so two tasks pointing at the same job class with different arguments don't share one lock — the scheduler's analogue of the #333 raise-at-enqueue guard, which a scheduler loop can't use — and a failure to resolve ANY uniqueness key — default or user-supplied key proc — now fails closed (skips the tick, logs, and reaches `error_reporters`) instead of degrading to nil and enqueuing WITHOUT a lock, which silently disabled the very protection the proc configured. Both macros also now reject a non-nil, non-callable `key:` (e.g. `key: false`) at definition time with the documented ArgumentError — previously `false` slipped the truthiness guard and surfaced as a `NoMethodError` at enqueue (uniqueness) or silently acted as the class-name default (concurrency). Measured (benchmark-ips, enqueue-path budget per docs/performance.md): the shared `Support.call_key_proc` dispatch is ~17% faster than the old inline dispatch (136ns vs 160ns/op), the class-name default is 3.7× faster than the old stored-proc call (23ns vs 86ns/op), and the inherited-config ancestor walk adds ~15ns/op. Refs #357, #333.
+- **The Ruby-Timeout read fallback no longer leaves a wedged `CONNECTION_OK` connection in the pgmq pool to re-hang the next read (issue #354).** On the one path where libpq can't bound a hung socket (dedicated connection on non-Linux hosts or libpq < 12), the last-resort `Timeout.timeout` interrupts the read via `Thread#raise` — and libpq may leave the pooled `PG::Connection` reporting `CONNECTION_OK` while it will in fact re-hang on reuse, invisible to pgmq-ruby's checkout health check (it isn't `CONNECTION_BAD`). This was a documented KNOWN LIMITATION waiting on a public pool-reload upstream (cf. mensfeld/pgmq-ruby#94); pgmq-ruby 0.7.1 shipped it, so the fallback now raises an internal `ReadTimeoutError` subclass and reloads the job pool before re-raising — the poisoned connection (already checked back in by `connection_pool`'s ensure) is dropped and the pool rebuilds lazily. A clean server-side `statement_timeout` cancel still never triggers a reload (the connection is healthy), and a reload failure is logged without masking the timeout. Requires pgmq-ruby >= 0.7.1 (dependency bumped to `~> 0.7.1`). Refs #354.
 - **`connection_guc_mode = :session` no longer kills every SSE stream and NOTIFY wakeup with `PG::Error: invalid connection option "variables"` (issue #352).** In `:session` mode, `forward_connection_variables` deliberately leaves the database.yml `variables:` hash on the connection options for the *caller* to strip and apply via post-connect `SET` (a transaction-mode pooler rejects the libpq `options` startup param — the reason `:session` mode exists). `Pgbus::Client#wrap_session_gucs` honors that contract on both pgmq pools, but the two dedicated raw-connect paths — the streamer's `build_raw_pg_connection` and the worker `NotifyListener` — passed the hash straight to `PG.connect`, which rejects the non-libpq `:variables` key. Result: the first stream request per process raised, `StreamApp` rescued it into a 500 (so **every** `/pgbus/streams/…` request failed from then on), and NOTIFY-gated wakeups died with workers silently falling back to polling. Both paths now connect through the new `Pgbus::DedicatedConnection.connect`, which strips `:variables` and applies each GUC via post-connect `SET` — dedicated connections *keep* the operator's GUCs (`statement_timeout`, `timezone`, …), matching the pooled paths, and the mechanism works on both a direct port and a session-mode pooler. A guard spec confines raw `PG.connect` call sites to `Client` and `DedicatedConnection`, so the next dedicated-connection path can't silently reintroduce the bypass. Refs #352.
 - **AppSignal main dashboard: job and event lines now split by `job_class` and `routing_key`.** Applied the upstream review suggestions committed on appsignal/public_config#80 to the vendored `dashboard.json`: the "Job status per queue" panel gains a `job_class` wildcard tag filter and label (`%job_class% - %queue% - %status%`), and "Event handler status" gains `routing_key` (`%handler% - %status% - %routing_key%`). Without the wildcard tag entry a `%placeholder%` in the line label never resolves, so lines from different job classes / routing keys were collapsed together. The subscriber already emitted both tags; only the dashboard definition lagged.
 - **Migration generators now route to the pgbus database when `connects_to` is configured, even without an explicit `--database` (issue #344).** `rails g pgbus:add_stream_queues` (and every sibling generator via `Generators::MigrationPath`) keyed the separate-database decision purely on the `--database` flag. An app that had already set `config.connects_to = { database: { writing: :pgbus } }` (so its pgbus migrations live in `db/pgbus_migrate/` per database.yml's `migrations_paths:`) still got the migration written to the **primary** DB's `db/migrate/` on a bare invocation — where it then ran against the wrong database — and the "Next steps" output told the operator to run `rails db:migrate` instead of `rails db:migrate:pgbus`. `MigrationPath` now auto-detects the target database from `connects_to` (via the existing `DatabaseTargetDetector`: runtime config → initializer scan → application.rb scan) when `--database` is absent; an explicit `--database` still wins. The detected name drives both the migration path (resolving `migrations_paths` for that database, falling back to the `db/pgbus_migrate` convention) and the `db:migrate:<name>` suffix in every generator's post-install output. An app with no separate database is unaffected. Refs #344.

data/lib/generators/pgbus/templates/upgrade_pgmq.rb.erb

@@ -8,6 +8,13 @@ class UpgradePgmqToV<%= target_version_slug.camelize %> < ActiveRecord::Migratio
     # This uses the vendored SQL which doesn't require the pgmq extension.
     execute Pgbus::PgmqSchema.install_sql("<%= target_version %>")
 
+    # Re-install the per-queue NOTIFY insert triggers the function drop
+    # cascaded away (pgmq.notify_insert_throttle records which queues had
+    # them and at what throttle; enable_notify_insert is idempotent).
+    # Without this, NOTIFY-gated wakeups silently die and workers fall back
+    # to polling until each queue is re-ensured by a process restart.
+    execute Pgbus::PgmqSchema.reinstall_notify_triggers_sql
+
     # Record the upgrade
     execute <<~SQL
       CREATE TABLE IF NOT EXISTS pgbus_pgmq_schema_versions (

data/lib/pgbus/client/resizable_pool.rb

@@ -101,6 +101,15 @@ module Pgbus
         end
       end
 
+      # Drop every pooled connection on the CURRENT streams client and let it
+      # rebuild lazily on next checkout (pgmq-ruby >= 0.7.1) — see
+      # Client#reload (issue #354). Takes the swap mutex so a reload cannot
+      # race a concurrent swap/close into reloading a pool that is being
+      # retired; reads stay lock-free.
+      def reload
+        @swap_mutex.synchronize { @ref.get.pgmq.reload }
+      end
+
       def stats_snapshot
         @last || SwapStats.new(swap_count: 0, last_drain_seconds: 0.0, last_conns_closed: 0,
                                last_from_size: nil, last_to_size: nil, last_drained: nil)

data/lib/pgbus/client.rb

@@ -89,15 +89,17 @@ module Pgbus
         # persistent connection instead of a fresh PG.connect per call. Its own
         # PGMQ::Client → its own connection_pool, sized independently of worker
         # thread counts.
-        # Build the streams pool from streams_connection_options (which defaults
-        # to connection_options but honors streams_database_url/host/port for a
-        # separate/direct streams DB — issue #315), bounds-applied, and tagged
-        # with a per-process application_name so the autoscaler can count peer
-        # processes from pg_stat_activity (issue #323 P1/P2). Snapshot it so a
-        # hot-swap rebuilds a byte-identical pool at a new size.
+        # Build the streams pool from streams_pool_connection_options (defaults
+        # to streams_connection_options so a separate streams DB carries the
+        # pool with it — issue #315 — but overridable via streams_pool_* so a
+        # pooler-bypass install keeps the pool off the direct port — issue
+        # #358), bounds-applied, and tagged with a per-process application_name
+        # so the autoscaler can count peer processes from pg_stat_activity
+        # (issue #323 P1/P2). Snapshot it so a hot-swap rebuilds a
+        # byte-identical pool at a new size.
         @streams_conn_opts = wrap_session_gucs(
           tag_application_name(
-            apply_connection_bounds(config.streams_connection_options)
+            apply_connection_bounds(config.streams_pool_connection_options)
           )
         )
         @streams_pgmq = PGMQ::Client.new(@streams_conn_opts, pool_size: config.streams_pool_size,
@@ -794,6 +796,32 @@ module Pgbus
       @streams_pool.stats_snapshot
     end
 
+    # Operator escape hatch (issue #354): drop every pooled PGMQ connection —
+    # job pool AND the live streams pool — and let the pools rebuild lazily on
+    # next checkout (pgmq-ruby >= 0.7.1). Use to recover connections libpq
+    # still reports as CONNECTION_OK but that are in fact wedged (e.g. after a
+    # wall-clock interrupt cut a query mid-flight), which pgmq-ruby's checkout
+    # health check cannot detect. Unlike #close, the pools stay usable.
+    # Connections checked out by other threads mid-reload are unaffected.
+    #
+    # No-op (returns false) on the shared-AR Proc path: those pool slots wrap
+    # ActiveRecord's own raw connection — reloading would close AR's socket
+    # out from under the application. Returns true after a reload.
+    def reload # rubocop:disable Naming/PredicateMethod -- command that reports whether it acted, like #ping
+      if @shared_connection
+        Pgbus.logger.warn do
+          "[Pgbus::Client] reload skipped: pgbus is sharing ActiveRecord's connection " \
+            "(Proc connection_options) and won't close a socket it doesn't own. " \
+            "Manage that connection through ActiveRecord instead."
+        end
+        return false
+      end
+
+      @pgmq.reload
+      @streams_pool.reload
+      true
+    end
+
     private
 
     # Human-readable label for which config knob supplied the connection
@@ -1146,6 +1174,18 @@ module Pgbus
     READ_TIMEOUT_SLACK = 5
     private_constant :READ_TIMEOUT_SLACK
 
+    # Raised (instead of plain ReadTimeoutError) when the Ruby Timeout last
+    # resort fires — i.e. Thread#raise interrupted a read mid-flight on a
+    # socket libpq couldn't bound (issue #354). IS-A Pgbus::ReadTimeoutError,
+    # so the public contract is unchanged; the distinct class lets
+    # with_read_timeout tell "wedged socket — reload the pool" (this) apart
+    # from "clean server-side statement_timeout cancel — connection healthy"
+    # (plain ReadTimeoutError), where reloading would churn a healthy pool on
+    # every slow query. Internal signal carried on the unwind path itself (no
+    # thread-local/ivar state around a Thread#raise interrupt); application
+    # code should rescue Pgbus::ReadTimeoutError.
+    class WedgedReadTimeout < Pgbus::ReadTimeoutError; end
+
     # Bound a read and surface a timeout as Pgbus::ReadTimeoutError. Prefer
     # libpq-native bounds baked into the connection; the Ruby Timeout is a
     # narrow, last-resort fallback used only where libpq cannot bound a hung
@@ -1179,12 +1219,19 @@ module Pgbus
     #      which AR passes straight through to the connection. #initialize logs a
     #      one-time hint when read_timeout is set on a Proc connection.
     #
-    #      KNOWN LIMITATION: when (3) fires on a genuinely hung socket, libpq may
-    #      leave the pooled PG::Connection reporting CONNECTION_OK while it will
-    #      in fact re-hang on reuse, and pgmq-ruby's health check won't discard
-    #      it (it isn't CONNECTION_BAD). The proper fix is a public pool-reload on
-    #      pgmq-ruby (follow-up, cf. mensfeld/pgmq-ruby#94); until then it's
-    #      documented and confined to the non-Linux dedicated path.
+    #      WEDGED-SOCKET RECOVERY (issue #354): when (3) fires on a genuinely
+    #      hung socket, libpq may leave the pooled PG::Connection reporting
+    #      CONNECTION_OK while it will in fact re-hang on reuse, and pgmq-ruby's
+    #      checkout health check won't discard it (it isn't CONNECTION_BAD). So
+    #      the Timeout raises WedgedReadTimeout (a ReadTimeoutError subclass)
+    #      and the rescue below drops every pooled connection via @pgmq.reload
+    #      (pgmq-ruby >= 0.7.1) — the pool rebuilds lazily on next checkout.
+    #      By the time the rescue runs, Thread#raise has unwound the read and
+    #      connection_pool's ensure has checked the poisoned connection back in
+    #      as idle, so reload does discard it. A small window remains where
+    #      another thread checks it out first; that thread's own read bound /
+    #      stale-retry covers it — reload narrows the window, it doesn't need
+    #      to close it atomically.
     #
     # MUST wrap only the bare `@pgmq.read*` call, inside both `synchronized` and
     # `with_stale_connection_retry`, so the Timeout clock starts only after the
@@ -1203,10 +1250,26 @@ module Pgbus
       return mapping_statement_timeout(&block) unless timeout&.positive?
 
       # rubocop:disable Pgbus/NoRubyTimeout -- deliberate last-resort bound; see above
-      Timeout.timeout(timeout + READ_TIMEOUT_SLACK, Pgbus::ReadTimeoutError) do
+      Timeout.timeout(timeout + READ_TIMEOUT_SLACK, WedgedReadTimeout) do
         mapping_statement_timeout(&block)
       end
       # rubocop:enable Pgbus/NoRubyTimeout
+    rescue WedgedReadTimeout
+      reload_pool_after_wedged_timeout
+      raise
+    end
+
+    # Best-effort job-pool reload after the Ruby Timeout fallback interrupted a
+    # read (see with_read_timeout). A reload failure is logged, never raised —
+    # the caller is already unwinding with ReadTimeoutError, which is the
+    # actionable error; masking it with a secondary pool failure would hide
+    # which read timed out.
+    def reload_pool_after_wedged_timeout
+      @pgmq.reload
+    rescue StandardError => e
+      Pgbus.logger.warn do
+        "[Pgbus::Client] pool reload after wedged read timeout failed: #{e.class}: #{e.message}"
+      end
     end
 
     # True when libpq's connection-baked read bounds (statement_timeout +

data/lib/pgbus/concurrency.rb

@@ -16,25 +16,30 @@ module Pgbus
       #
       # Options:
       #   to:          Maximum concurrent jobs for the same key (required)
-      #   key:         Proc receiving job arguments, returns a string key. Default: job class name.
+      #   key:         Proc receiving job arguments, returns a string key.
+      #                Default: the ENQUEUED job's class name (resolved at
+      #                resolve time, so an inherited declaration keys each
+      #                subclass separately — issue #357).
       #   duration:    Safety expiry for semaphore (default: 15 minutes)
       #   on_conflict: What to do when limit is reached — :block, :discard, or :raise (default: :block)
       def limits_concurrency(to:, key: nil, duration: 15 * 60, on_conflict: :block) # rubocop:disable Naming/MethodParameterName
         raise ArgumentError, "to: must be a positive integer" unless to.is_a?(Integer) && to.positive?
         raise ArgumentError, "on_conflict must be :block, :discard, or :raise" unless %i[block discard raise].include?(on_conflict)
         raise ArgumentError, "duration must be a positive number" unless duration.is_a?(Numeric) && duration.positive?
-        raise ArgumentError, "key must be callable (Proc or lambda)" if key && !key.respond_to?(:call)
+        raise ArgumentError, "key must be callable (Proc or lambda)" if !key.nil? && !key.respond_to?(:call)
 
         @pgbus_concurrency = {
           limit: to,
-          key: key || ->(*) { name },
+          key: key,
           duration: duration,
           on_conflict: on_conflict
         }.freeze
       end
 
+      # The nearest declaration in the ancestor chain wins — same inheritance
+      # contract as Uniqueness#pgbus_uniqueness (issue #357).
       def pgbus_concurrency
-        @pgbus_concurrency
+        @pgbus_concurrency || (superclass.pgbus_concurrency if superclass.respond_to?(:pgbus_concurrency))
       end
     end
 
@@ -47,13 +52,11 @@ module Pgbus
         config = active_job.class.pgbus_concurrency
         return nil unless config
 
-        args = active_job.arguments
-        last = args.last
-        if last.is_a?(Hash) && last.each_key.all?(Symbol)
-          config[:key].call(*args[...-1], **last)
-        else
-          config[:key].call(*args)
-        end
+        # Class-name default, resolved from the ENQUEUED job's class so an
+        # inherited declaration keys each subclass separately (#357).
+        return active_job.class.name unless config[:key]
+
+        Support.call_key_proc(config[:key], active_job.arguments)
       end
 
       # Inject the resolved concurrency key into the job's serialized payload.

data/lib/pgbus/configuration.rb

@@ -167,6 +167,7 @@ module Pgbus
                   :streams_durable_patterns, :streams_broadcast_queue,
                   :streams_presence_patterns, :streams_presence_member,
                   :streams_host, :streams_port, :streams_database_url,
+                  :streams_pool_host, :streams_pool_port, :streams_pool_database_url,
                   :streams_pool_size, :streams_pool_timeout,
                   :streams_fanout_write_deadline_ms, :streams_dispatch_queue_limit,
                   :streams_writer_threads, :streams_writer_buffer_limit,
@@ -338,6 +339,22 @@ module Pgbus
       # thread-safe and streams keep using the single serialized connection.
       @streams_pool_size = 5
       @streams_pool_timeout = 5
+      # Streams-POOL-only connection overrides (issue #358). The pool above
+      # defaults to wherever the streamer connects (streams_host/port/
+      # database_url), which is right for a separate streams database — but
+      # wrong for the pooler-bypass pattern, where streams_port pins the
+      # LISTEN connection to the direct Postgres port purely because LISTEN
+      # dies in transaction pooling. The pool's traffic (broadcast INSERTs,
+      # replay reads) is plain pooler-safe SQL, and on a direct port with a
+      # low max_connections ceiling every process's streams_pool_size
+      # connections eat scarce direct slots. Set any of these to route the
+      # POOL independently of the LISTEN connection:
+      #
+      #   c.streams_port = 5432        # LISTEN bypasses the pooler
+      #   c.streams_pool_port = 6432   # the pool stays on the pooler
+      @streams_pool_host = nil
+      @streams_pool_port = nil
+      @streams_pool_database_url = nil
       # Self-tuning streams-pool autoscaling (issue #323). Opt-in, default off.
       # When true, a per-web-process control loop grows the dedicated streams
       # pool into a FAIR SHARE of live Postgres connection headroom under
@@ -1214,6 +1231,33 @@ module Pgbus
       override_connection_options(url: streams_database_url, host: streams_host, port: streams_port)
     end
 
+    # Connection options for the dedicated streams PGMQ pool — durable-broadcast
+    # publish INSERTs and the dispatcher's replay reads (issue #315). Defaults
+    # to `streams_connection_options`, so a genuinely separate streams database
+    # carries the pool with it. When any of `streams_pool_database_url` /
+    # `streams_pool_host` / `streams_pool_port` is set, the pool is routed
+    # independently — applied to the BASE options, exactly like the other two
+    # override groups.
+    #
+    # Why this exists (issue #358): only the streamer's LISTEN connection needs
+    # to bypass a transaction-mode pooler (LISTEN dies at COMMIT boundaries);
+    # the pool's INSERT/SELECT traffic is pooler-safe. Without this split, a
+    # `streams_port` direct-port pin drags streams_pool_size connections per
+    # process onto the direct port's low max_connections ceiling:
+    #
+    #   c.streams_port = 5432        # LISTEN bypasses the pooler
+    #   c.streams_pool_port = 6432   # the pool stays on the pooler
+    #
+    # Precedence: streams_pool_database_url > streams_pool_host/port over the
+    # base > streams_connection_options.
+    def streams_pool_connection_options
+      return streams_connection_options unless streams_pool_database_url || streams_pool_host || streams_pool_port
+
+      override_connection_options(
+        url: streams_pool_database_url, host: streams_pool_host, port: streams_pool_port
+      )
+    end
+
     # Connection options for the Worker's dedicated NotifyListener connection.
     # Mirrors streams_connection_options: defaults to the base connection_options,
     # overridable via worker_notify_database_url / worker_notify_host /

data/lib/pgbus/mcp/health_analyzer.rb

@@ -32,13 +32,19 @@ module Pgbus
       def verdict
         queues          = @data_source.queues_with_metrics
         processes       = @data_source.processes
+        stream_names    = @data_source.stream_queue_names
         health, health_error = safe_queue_health
 
-        # Partition queues once: non-DLQ (the operational set) and the subset
-        # of those with visible, claimable backlog. Paused queues are removed
-        # from the STALLED backlog (an intentional pause is not the wedge —
-        # it's reported under DEGRADED), but kept in `non_dlq` for the summary.
-        non_dlq = queues.reject { |q| dlq?(q) }
+        # Partition queues once: the operational set excludes DLQs and
+        # registered stream queues, and the backlog is the subset with
+        # visible, claimable messages. Stream queues are excluded like DLQs
+        # (issue #359): stream delivery is a non-consuming peek, so their
+        # messages are permanently visible with read_ct=0 — exactly the wedge
+        # signature — and counting them makes the verdict scream STALLED on
+        # every streams-heavy install. Paused queues are removed from the
+        # STALLED backlog (an intentional pause is not the wedge — it's
+        # reported under DEGRADED), but kept in `non_dlq` for the summary.
+        non_dlq = queues.reject { |q| dlq?(q) || stream_names.include?(q[:name].to_s) }
         backlog = non_dlq.select { |q| q[:queue_visible_length].to_i.positive? }
         active_backlog = backlog.reject { |q| q[:paused] }