pgbus 0.9.11 → 0.10.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 56605f0cc86326e66b73d13d9090bbcdc522b9ee6e571dd19f62f02aadce3dac
-  data.tar.gz: e53deb0e75aec959f4c0bf54d2d969d52899d57c59ca59e0441f989d897745b2
+  metadata.gz: d927ac10d70c354f2624eb9495c0e21d6913a3f6a074e94705e32ac69a5f51db
+  data.tar.gz: 36c12b3e6cb620f9443bde0a3cdc0137072076ee38e40381d5a3cc3960504596
 SHA512:
-  metadata.gz: 3bdbd54ebc2c5d414d5e85212adee43bea5645d4422397e1b5c41e41a63b19fb7695cbd32da7096c3455c3ff6f8a4afab2a7ad510d443d9b5cc561abdd01043a
-  data.tar.gz: 4cb472bbaad4c5052f732cfe452c07fc725ac739afceca8cd4e58a05a9785950e64b1efe050f6d97a245cd94fe0b85218ee1f616b753d552e113a498aa135289
+  metadata.gz: c1381157be5c95baab7d9fa9097d18d212c74a3262887ddb31629c54c2cab42c0ca8d699f7da6565a8649fd41d5f4eda63b5b77660cd28cfb8a4516eb04d852d
+  data.tar.gz: c36c9c2012f2ccf315e3102bbfc61cbddf3462134782687670bcdebf72f7bae7d639b2b0824b2fd1339ef9836d0b68e7a106169148d324a486652c1770e14665

data/CHANGELOG.md

@@ -7,6 +7,7 @@
 
 ### Added
 
+- **Connection-pooler safety (`config.require_primary`, `config.connection_guc_mode`) — pgbus is now correct-and-loud behind a transaction/read-write-splitting pooler instead of silently stalling (issue #332).** Three linked fixes. (1) **Primary affinity:** a read/write-splitting pooler (pgdog/pgcat) can route pgmq's VOLATILE `read`/`archive`/`delete` to a read replica, where workers read nothing (`read_ct=0`) and jobs stop with a **healthy heartbeat** — the supervisor watchdog keys on loop advancement, not read progress, so nothing surfaced the stall. The job read path had zero primary-affinity defense (only the LISTEN/streamer paths validated the primary via `PrimaryValidator`). Setting `config.require_primary = true` (default **false** — a single-primary deployment is byte-identical) makes `verify_connection!` reject a connection that lands on a replica (`pg_is_in_recovery() => t`) at supervisor boot, raising a clear `ConfigurationError` instead of forking children that read nothing. (2) **GUC forwarding:** pgbus's AR-config extraction dropped database.yml's `variables:` block, so `client_min_messages` (and any GUC) never reached pgmq's dedicated raw connections (NOTICE flooding that three consuming apps hand-patched via `connection_params`). GUCs are now forwarded, on both the AR-extracted and explicit-`connection_params` paths, by `config.connection_guc_mode`: `:options` (default) bakes them into the libpq `options` STARTUP param (byte-compatible with today for the read-timeout `statement_timeout`, now also carrying `variables:`), while `:session` applies them via post-connect `SET` on a fresh connection instead — for a transaction-mode PgBouncer that **rejects** the `options` startup param with a FATAL. `pgbus doctor` gains a ninth check ("Primary affinity") that warns (never fails) when the job connection is currently on a replica, naming the direct-port remediation. DB-gated integration tests prove both GUC modes round-trip a forwarded GUC and that `require_primary` verifies on a real primary. Refs #332.
 - **Job-burst tuning guidance + gate benchmark (`rake bench:job_burst`), and a decision NOT to autoscale the job pool (issue #323 phase 3).** Following the issue's measure-first mandate, `benchmarks/job_burst_bench.rb` sweeps worker thread count under a job burst in two connection regimes (pool matched to threads vs. pool fixed small). Measured finding: a job holds a pgmq connection only for the `read_batch` + `archive` round-trip (not the job body), so **the DB connection pool is the hard burst ceiling, not the thread count** — in the fixed-pool regime `peak_busy` never exceeds `pool_size` no matter how many threads run (extra threads just queue on checkout, latency climbs, throughput plateaus), while matching the pool to the thread count scales throughput ~linearly (≈8× from 2→16). The conclusion — issue #323's explicitly sanctioned outcome — is that **elastic job threads on a fixed connection pool cannot help**, so pgbus does *not* autoscale the job execution pool; the fix for job bursts is **static headroom**: raise `threads` (and let `pool_size` follow, or keep it ≥ `threads`). Documented in the Performance & tuning docs. The streams-pool autoscaling below stays the elastic story where it *does* pay off (the streams pool's connection cost genuinely scales with its size). Refs #323.
 - **Fan-out writer throughput gate benchmark (`writer_burst_bench.rb`), and a decision NOT to make the writer pool elastic (issue #323 phase 1).** Following the issue's measure-first mandate, `benchmarks/writer_burst_bench.rb` drives a sustained fan-out burst (a fleet of connections × many wakes) through the `OutboundPump` while sweeping `streams_writer_threads`, across fast/typical socket-write profiles. Measured finding: throughput scales ~linearly with writer count (≈6× from 2→16) — but that is generic parallelism (N workers finish N independent blocking writes ~N× faster), which argues for the **existing static `streams_writer_threads` knob**, not for elastic writers: a fixed-N sweep can't demonstrate the *spikiness* that would make a static count wasteful enough to justify dynamic resize, and the genuinely harmful case (a slow/congested client) is already **shed** by the #320/#322 fan-out deadline + mark-dead path rather than absorbed by scaling writers. Given the elastic-writer rewrite is the issue's highest-risk phase (it would touch the ordering/cursor invariant #321's whole review exists to protect — mis-sizing risks out-of-order SSE frames), the conclusion — issue #323's sanctioned outcome — is that pgbus does **not** autoscale the writer pool; the fan-out-throughput knob is **static `streams_writer_threads`, sized for your peak fan-out fleet**. Documented in the Performance & tuning docs. Refs #323.
 - **Self-tuning elastic streams-pool autoscaling (`config.streams_pool_autoscale`, opt-in, default off).** The dedicated streams pool (durable-broadcast publish + dispatcher replay reads) is a fixed `streams_pool_size` (default 5); under a genuine burst of SSE clients it saturates, and a saturated pool *serialises* replay reads (it doesn't error — the checkout waits; see #324), slowing fan-out. Setting `config.streams_pool_autoscale = true` runs a **periodic maintenance check** (default every 5 minutes, `config.streams_pool_autoscale_interval`) that **grows the streams pool into a fair share of live Postgres connection headroom** while it's saturated and **shrinks it back to `streams_pool_size`** when the burst passes — with no connection-count target to tune: every threshold derives from live `max_connections`. The check adds **no extra connection and no extra thread** — it's a lightweight `pg_stat_activity` query that runs on the streamer's *existing idle LISTEN connection* (a pghero-`capture_query_stats`-style periodic task, run on the listener thread in its health-check window). It counts peer stream processes by a per-process `application_name` tag (`config.streams_application_name`, default `"pgbus_streams"`), and each process grows only into its own fair share of free connections (`free / peers`, claiming ¼, bounded by a per-grow step and half the remaining headroom) so N forked processes provably can't collectively exhaust the database — even in a synchronized cold-boot herd. If the database runs *critically* low on free connections, every process **emergency-shrinks to baseline immediately**, overriding the saturation signal (protecting the DB wins). `config.streams_pool_max` is an **optional** hard per-process ceiling (`nil` = the dynamic fair share is the cap). No per-sample hysteresis or cooldown is needed — the slow cadence is itself the debounce: one grow (or shrink) step per check, and a sustained burst converges over a few checks. Built on the #325 hot-swap primitive; the decision object is a pure function (no thread, no connection), DB-free unit-tested (scripted headroom, incl. convergence-across-checks and emergency-shrink), with a DB-gated integration test proving real grow/shrink/emergency and a `rake bench:pool_autoscale` benchmark (check cost ~0.3 ms; grow-under-saturation `3→7→10`). **Two prerequisite fixes ship with it:** streams-pool connections are now tagged with a per-process `application_name`, and the streams pool is now built from `streams_connection_options` (honoring `streams_database_url`/host/port) rather than the job-database options — the latter a latent wrong-DB bug in the existing streams pool that only surfaced with a separate streams database. Default-off = byte-identical (the Listener runs no maintenance). A pure-*publisher* worker process (no streamer, only `send_stream_message`) autoscales too: each publish opportunistically triggers the same check, throttled to at most once per `streams_pool_autoscale_interval` across all publisher threads (a lock-free compare-and-set claims the window). The publish thread does only that CAS and then hands the work to a background single-worker executor — it never runs the `pg_stat_activity` query or the resize inline (off the hot path), and a quiet process spawns no thread. The query runs through the *job* pool (so it never competes with the streams pool it grows), and the publisher decision is **grow-only**: it grows under its own publish pressure and emergency-shrinks on DB exhaustion, but leaves normal idle shrink to the streamer/consumer that sees the sustained-idle picture. Fail-soft throughout, so a broadcast never fails on a telemetry query. For steady load, a larger static `streams_pool_size` remains the simpler choice — autoscaling earns its keep under genuinely bursty streams traffic. `PoolAutoscaler` lives in `Pgbus::Streams::` (it is a streams-pool controller, not web-specific). Refs #323.
@@ -26,6 +27,7 @@
 
 ### Fixed
 
+- **A pooler-deaf NOTIFY listener no longer raises the worker poll floor to 15s — worse latency than a plain no-NOTIFY deployment (issue #332).** pgbus already detects a transaction-mode pooler / replica LISTEN connection with a self-NOTIFY probe (`NotifyProbe`) and logs a loud, actionable error, but the probe's boolean result was **discarded**: a live-but-deaf listener still reported `running? == true`, so `Worker#wake_timeout` treated it as a healthy NOTIFY source and raised the empty-fetch poll floor to `NOTIFY_FALLBACK_POLL_SECONDS` (15s) — behind a pooler the loop would then sleep up to 15s between polls while jobs sat ready. `NotifyListener` now records the probe result and exposes `delivering?`; `wake_timeout` treats a non-delivering listener as absent and keeps polling at the fast `polling_interval` until a direct-port connection is configured. The streamer path needed no change — its listener already waits on the short `streams_listen_health_check_ms` and has no equivalent 15s ceiling. Refs #332.
 - **The dispatcher no longer logs a scary `Concurrency cleanup failed: ... terminating connection due to administrator command` WARN when an idle DB connection is killed between maintenance cycles.** The dispatcher is a long-lived loop that queries the DB on coarse intervals (concurrency cleanup every 300s, most tasks hourly) and, until now, kept its pooled ActiveRecord connection leased across the idle sleep. When the server terminated that idle backend — routine in **local development** (a Postgres restart, `rails db:migrate` calling `pg_terminate_backend` on idle backends, foreman/overmind restarting the DB) — the next cycle reused the dead socket and *every* maintenance task raised `PG::ConnectionBad` with PostgreSQL's `terminating connection due to administrator command` text, surfaced as an alarming WARN even though the process was healthy and self-heals on the next tick. Two changes: (1) `run_maintenance` now returns every leased AR connection to the pool after each cycle (`clear_active_connections!`, mirroring the Streamer's existing `release_ar_connections`), so an idle backend is never reused — the next cycle checks out afresh and AR's `verify!`-on-checkout transparently discards a dead socket and reconnects; (2) every maintenance task's failure log now routes through `log_maintenance_failure`, which recognizes the terminated-idle-connection signal and logs a calm INFO ("reconnecting on the next cycle") instead of a WARN. Non-connection failures still WARN unchanged, and maintenance backoff is unaffected (a self-healing cycle resets the failure streak on the next successful reconnect).
 - **`priority_levels > 1` no longer silently breaks durable streams.** With job priority enabled, `QueueFactory.for` selects `PriorityStrategy`, and `Client#send_message` routes *every* send — including durable stream broadcasts (which pass no `priority:`) — to a `_p<default_priority>` sub-queue. But the streamer LISTENs and replays exclusively on the **bare** `pgmq.q_<stream>` queue (`read_after`, `notify_stream`), so the broadcast never reached the browser; on a fresh DB, `ensure_stream_queue` also raised `PG::UndefinedTable` because `ensure_queue` created only `_p0.._pN`, never the bare table the NOTIFY trigger and archive index attach to. Durable broadcasts now go through a dedicated `Client#send_stream_message` that always targets the bare queue, and `ensure_stream_queue` creates the bare queue directly via `ensure_single_queue` — both independent of the priority strategy, mirroring the bare-name read path. Streams are delivered by a non-consuming peek; priority sub-queues are meaningless for them. Ephemeral broadcasts (the default mode) were never affected. Refs #310.
 - **Stream maintenance and wildcard workers now correctly identify stream queues (finishes the #306/#307 orphan-sweep fix and closes a durable-stream data-loss bug).** `streams_queue_prefix` (default `pgbus_stream`) was consulted only by the dispatcher's `prune_stream_archives` and `sweep_orphan_streams`, but no code ever *named* a queue with it — real stream queues are `#{queue_prefix}_<name>` (`pgbus_<name>`). So both filters matched **zero** real stream queues: per-stream retention never ran, and the age-based orphan drop shipped in #307 was inert. Worse, a *job* queue whose logical name started with `stream_` (physical `pgbus_stream_x`) *did* match and could be dropped. All stream detection now goes through the new `pgbus_stream_queues` registry (see Added): `prune_stream_archives` and `sweep_orphan_streams` iterate registered streams (reviving #306/#307), `compact_archives` skips registered streams so each stream archive is compacted by exactly one path at its per-stream retention (not the 7-day job default), and — closing #309 — a `queues: ['*']` worker's `resolve_wildcard_queues` now excludes registered stream queues, so wildcard workers no longer claim durable broadcasts, fail to deserialize them, and DLQ-move them out of the stream's replay history. `streams_queue_prefix` is retained for backward compatibility but is now inert (documented). Refs #308, #309.

data/lib/pgbus/client.rb

@@ -79,7 +79,7 @@ module Pgbus
         # Both raise clean PG errors — no Ruby Timeout, no Thread#raise. Only
         # safe on this dedicated-connection branch — never on the shared-AR Proc
         # path, where statement_timeout would leak into application queries.
-        conn_opts = apply_connection_bounds(conn_opts)
+        conn_opts = wrap_session_gucs(apply_connection_bounds(conn_opts))
         @pgmq = PGMQ::Client.new(conn_opts, pool_size: config.resolved_pool_size, pool_timeout: config.pool_timeout)
         @pgmq_mutex = nil
         # Dedicated streams pool (issue #315): isolates the durable-stream
@@ -95,8 +95,10 @@ module Pgbus
         # with a per-process application_name so the autoscaler can count peer
         # processes from pg_stat_activity (issue #323 P1/P2). Snapshot it so a
         # hot-swap rebuilds a byte-identical pool at a new size.
-        @streams_conn_opts = tag_application_name(
-          apply_connection_bounds(config.streams_connection_options)
+        @streams_conn_opts = wrap_session_gucs(
+          tag_application_name(
+            apply_connection_bounds(config.streams_connection_options)
+          )
         )
         @streams_pgmq = PGMQ::Client.new(@streams_conn_opts, pool_size: config.streams_pool_size,
                                                              pool_timeout: config.streams_pool_timeout)
@@ -163,9 +165,22 @@ module Pgbus
     # carries the underlying error plus which config source was in use.
     def verify_connection!
       synchronized do
-        @pgmq.with_connection { |conn| conn.exec("SELECT 1") }
+        @pgmq.with_connection do |conn|
+          conn.exec("SELECT 1")
+          # When require_primary is set, reject a connection that landed on a
+          # read-only replica at boot rather than letting a read/write-splitting
+          # pooler silently route pgmq's VOLATILE read/archive to a standby,
+          # where workers read nothing and jobs stop with a healthy heartbeat
+          # (issue #332). Off by default, so a single-primary deployment is
+          # unaffected.
+          Process::PrimaryValidator.validate_primary!(conn) if config.require_primary
+        end
       end
       true
+    rescue Process::ReplicaConnectionError => e
+      raise ConfigurationError,
+            "Database connection via #{connection_source} landed on a read-only replica " \
+            "(require_primary is set): #{e.message}"
     rescue PGMQ::Errors::ConnectionError, PG::Error => e
       raise ConfigurationError, "Database connection failed via #{connection_source}: #{e.message}"
     end
@@ -182,6 +197,17 @@ module Pgbus
       true
     end
 
+    # Whether the job connection currently lands on a read-only replica
+    # (pg_is_in_recovery() => t). Used by the doctor to warn about a
+    # read/write-splitting pooler that could route pgmq's VOLATILE read/archive
+    # to a standby, silently stalling job processing (issue #332). Raw PG error
+    # propagates so the caller can render the reason.
+    def in_recovery?
+      with_raw_connection do |conn|
+        conn.exec(Process::PrimaryValidator::RECOVERY_QUERY).getvalue(0, 0) == "t"
+      end
+    end
+
     # The logical queue names pgbus expects to exist based on the configuration
     # (default queue + worker capsules + recurring tasks). Public wrapper around
     # collect_configured_queues so the doctor can diff configured-vs-existing
@@ -890,7 +916,14 @@ module Pgbus
                PG.connect(opts)
              when Hash
                owned = true
-               PG.connect(**opts)
+               # :variables is a database.yml convention, not a libpq keyword —
+               # strip it before PG.connect and apply the GUCs via SET so this
+               # raw bootstrap/DDL connection matches the pooled connections
+               # (issue #332). Empty/absent variables is a plain connect.
+               variables = opts[:variables]
+               conn = PG.connect(**opts.except(:variables))
+               variables&.each { |name, value| conn.exec("SET #{name} = '#{value}'") }
+               conn
              else
                raise ConfigurationError, "Cannot resolve raw PG connection from #{opts.class}"
              end
@@ -1299,6 +1332,31 @@ module Pgbus
       end
     end
 
+    # In :session GUC mode, a Hash conn_opts carrying database.yml `:variables`
+    # must apply those GUCs via post-connect `SET` rather than the libpq
+    # `options` STARTUP param (which a transaction-mode PgBouncer rejects). We
+    # can't pass `:variables` to PG.connect (not a libpq keyword), so wrap the
+    # opts in a fresh-connect factory Proc: pgmq-ruby natively accepts a callable
+    # per pool slot (pgmq connection.rb), and it must return a UNIQUE
+    # PG::Connection each call (pgmq guards against a shared object). Applies to
+    # a Hash with `:variables` only — a String URL or no variables passes through
+    # unchanged, as does :options mode (where forward_connection_variables
+    # already baked the GUCs into `options`). See issue #332.
+    def wrap_session_gucs(conn_opts)
+      return conn_opts unless config.connection_guc_mode == :session
+      return conn_opts unless conn_opts.is_a?(Hash)
+
+      variables = conn_opts[:variables]
+      return conn_opts if variables.nil? || variables.empty?
+
+      pg_opts = conn_opts.except(:variables)
+      lambda do
+        conn = PG.connect(pg_opts)
+        variables.each { |name, value| conn.exec("SET #{name} = '#{value}'") }
+        conn
+      end
+    end
+
     def apply_connection_bounds(conn_opts)
       timeout = config.read_timeout
       return conn_opts unless timeout&.positive?

data/lib/pgbus/configuration.rb

@@ -102,6 +102,27 @@ module Pgbus
     # Requires a matching entry in config/database.yml under the "pgbus" key.
     attr_accessor :connects_to
 
+    # Reject a job-pool connection that landed on a read-only replica
+    # (pg_is_in_recovery() => t). Default false — a correctly-configured
+    # single-primary deployment sees no change. Set true when a read/write
+    # splitting pooler (pgdog/pgcat) could route pgmq's VOLATILE read/archive
+    # to a replica, which otherwise makes workers silently read nothing.
+    # See issue #332.
+    attr_accessor :require_primary
+
+    # How pgbus forwards database.yml GUCs (`variables:` such as
+    # client_min_messages, and the read-timeout statement_timeout) onto its
+    # dedicated pgmq connections:
+    #   :options (default) — bake them into the libpq `options` STARTUP param
+    #                        (-c key=value). Works everywhere EXCEPT a
+    #                        transaction-mode PgBouncer, which rejects the
+    #                        `options` startup param with a FATAL.
+    #   :session           — apply them via post-connect `SET` statements on a
+    #                        fresh connection instead, so a transaction-mode
+    #                        pooler accepts them. Dedicated-connection path only.
+    # See issue #332.
+    attr_reader :connection_guc_mode
+
     # Zombie message detection — logs a warning when a message is redelivered
     # (read_ct > 1) without any prior failure recorded in pgbus_failed_events.
     attr_accessor :zombie_detection
@@ -248,6 +269,8 @@ module Pgbus
       @stats_flush_interval = StatBuffer::DEFAULT_FLUSH_INTERVAL
 
       @connects_to = nil
+      @require_primary = false
+      @connection_guc_mode = :options
 
       @web_auth = nil
       @web_refresh_interval = 5000
@@ -510,6 +533,18 @@ module Pgbus
       @group_mode = coerced
     end
 
+    VALID_CONNECTION_GUC_MODES = %i[options session].freeze
+
+    def connection_guc_mode=(mode)
+      mode = mode.to_sym
+      unless VALID_CONNECTION_GUC_MODES.include?(mode)
+        raise Pgbus::ConfigurationError,
+              "Invalid connection_guc_mode: #{mode}. Must be one of: #{VALID_CONNECTION_GUC_MODES.join(", ")}"
+      end
+
+      @connection_guc_mode = mode
+    end
+
     VALID_PGMQ_SCHEMA_MODES = %i[auto extension embedded].freeze
 
     def pgmq_schema_mode=(mode)
@@ -594,6 +629,8 @@ module Pgbus
         raise Pgbus::ConfigurationError, "insights_default_minutes must be a positive integer"
       end
 
+      raise Pgbus::ConfigurationError, "require_primary must be true or false" unless [true, false].include?(require_primary)
+
       validate_streams!
       validate_metrics_backend!
 
@@ -1039,7 +1076,17 @@ module Pgbus
       if database_url
         database_url
       elsif connection_params
-        connection_params
+        # An explicit connection_params Hash may carry a database.yml-style
+        # `:variables` block; forward it the same way the AR-extracted path does
+        # so client_min_messages etc. actually reach pgmq's connections and a
+        # non-libpq :variables key never reaches PG.connect (issue #332). A
+        # non-Hash connection_params (e.g. a Proc returning a raw connection)
+        # passes through untouched.
+        if connection_params.is_a?(Hash)
+          forward_connection_variables(connection_params.except(:variables), connection_params[:variables])
+        else
+          connection_params
+        end
       elsif defined?(ActiveRecord::Base)
         # Extract connection config from ActiveRecord so pgmq-ruby creates its
         # own dedicated PG connections. Sharing AR's raw_connection via a Proc
@@ -1326,13 +1373,15 @@ module Pgbus
       # Rails 7.1+ db_config.configuration_hash returns the full config
       config_hash = db_config.configuration_hash
 
-      {
+      base = {
         host: config_hash[:host] || "localhost",
         port: (config_hash[:port] || 5432).to_i,
         dbname: config_hash[:database],
         user: config_hash[:username],
         password: config_hash[:password]
       }.compact
+
+      forward_connection_variables(base, config_hash[:variables])
     rescue StandardError => e
       # Fallback to Proc path if AR config extraction fails (e.g., adapter
       # doesn't expose standard config keys). Log a warning since this path
@@ -1348,5 +1397,28 @@ module Pgbus
         -> { ActiveRecord::Base.connection.raw_connection }
       end
     end
+
+    # database.yml's `variables:` block (e.g. client_min_messages) is a Rails
+    # convention, not a libpq keyword — pgmq-ruby passes the hash straight to
+    # PG.connect, which would ignore/reject a `:variables` key. So carry the GUCs
+    # forward by the mechanism the operator's pooler tolerates:
+    #   :options — bake them into the libpq `options` STARTUP param
+    #              (-c key=value), appended to any existing options. A
+    #              transaction-mode PgBouncer rejects this param.
+    #   :session — leave the raw `:variables` hash on the returned options so the
+    #              Client applies them via post-connect `SET` (pooler-safe).
+    # Empty/absent variables is a no-op on both paths. See issue #332.
+    def forward_connection_variables(base, variables)
+      return base if variables.nil? || variables.empty?
+
+      case connection_guc_mode
+      when :session
+        base.merge(variables: variables)
+      else
+        gucs = variables.map { |k, v| "-c #{k}=#{v}" }.join(" ")
+        options = [base[:options], gucs].compact.reject(&:empty?).join(" ")
+        base.merge(options: options)
+      end
+    end
   end
 end

data/lib/pgbus/doctor.rb

@@ -5,7 +5,7 @@ require "pgbus/mcp/health_analyzer"
 
 module Pgbus
   # Preflight diagnostics for a pgbus deployment — the single command that
-  # answers "is this environment healthy enough to run?". Runs eight checks and
+  # answers "is this environment healthy enough to run?". Runs nine checks and
   # returns a machine-readable result plus a human report, so `pgbus doctor`
   # and `rake pgbus:doctor` can gate a deploy or CI run (exit 0 on success,
   # 1 on any failure).
@@ -42,7 +42,8 @@ module Pgbus
         check_notify,
         check_processes,
         check_allowed_global_id_models,
-        check_broadcast_queue
+        check_broadcast_queue,
+        check_primary
       ].map(&:to_h)
     end
 
@@ -248,6 +249,29 @@ module Pgbus
       Check.new(name: "Broadcast queue", status: :fail, detail: "#{e.class}: #{e.message}")
     end
 
+    # 9. Primary affinity — pooler safety (issue #332). A read/write-splitting
+    # pooler (pgdog/pgcat) can route pgmq's VOLATILE read/archive to a read
+    # replica, where workers read nothing and jobs stop with a healthy
+    # heartbeat. If the job connection currently lands on a replica
+    # (pg_is_in_recovery() => t), warn with the direct-port remediation. A
+    # warning, never a failure: a deliberate replica-read setup is rare but
+    # possible, and require_primary is the enforcement knob for those who want a
+    # hard stop.
+    def check_primary
+      if @client.in_recovery?
+        return Check.new(name: "Primary affinity", status: :warn,
+                         detail: "job connection is on a read-only replica (pg_is_in_recovery() => t) — " \
+                                 "a read/write-splitting pooler may be routing pgmq reads to a standby, " \
+                                 "silently stalling jobs. Point the connection at the DIRECT primary port " \
+                                 "(worker_notify_* / streams_* overrides) and set require_primary to reject " \
+                                 "a replica at boot")
+      end
+
+      Check.new(name: "Primary affinity", status: :ok, detail: "on primary")
+    rescue StandardError => e
+      Check.new(name: "Primary affinity", status: :warn, detail: "could not determine (#{e.class}: #{e.message})")
+    end
+
     # True when some configured worker capsule drains the given queue — either
     # by naming it explicitly or via a "*" wildcard (which drains every queue).
     def worker_drains?(queue)

data/lib/pgbus/process/notify_listener.rb

@@ -49,12 +49,25 @@ module Pgbus
         @running = false
         @thread = nil
         @conn = nil
+        # Optimistic until the start-time self-probe runs: assume NOTIFY delivery
+        # works so a not-yet-probed listener isn't mistaken for a pooler-deaf one.
+        @delivering = true
       end
 
       def listening_to
         @state_mutex.synchronize { @listening_to.dup }
       end
 
+      # Whether the start-time self-probe confirmed this connection can actually
+      # receive a NOTIFY. False when a transaction-mode pooler or replica
+      # silently drops LISTEN: the thread is still alive (running? == true) but
+      # will never wake the loop. The Worker/Consumer consult this so a
+      # live-but-deaf listener is treated as absent for wake-timeout purposes —
+      # fast polling, not the 15s NOTIFY ceiling (issue #332).
+      def delivering?
+        @state_mutex.synchronize { @delivering }
+      end
+
       def start
         @state_mutex.synchronize do
           return self if @running
@@ -116,8 +129,11 @@ module Pgbus
         # One-shot delivery self-probe on the initial connection only. A pooler
         # or replica that silently breaks LISTEN/NOTIFY is surfaced here with an
         # actionable error; the listener still runs and degrades to polling.
-        # Reconnects skip the probe to stay cheap.
-        NotifyProbe.probe_notify_delivery!(conn, logger: @logger)
+        # Reconnects skip the probe to stay cheap. Record the result so the
+        # owning worker can drop back to fast polling instead of the 15s NOTIFY
+        # ceiling when this connection can't actually deliver (issue #332).
+        delivering = NotifyProbe.probe_notify_delivery!(conn, logger: @logger)
+        @state_mutex.synchronize { @delivering = delivering }
         @state_mutex.synchronize { @conn = conn }
         drain_commands
 

data/lib/pgbus/process/worker.rb

@@ -605,13 +605,20 @@ module Pgbus
       def wake_timeout
         # A dead listener (running? false) will never wake the loop, so treat
         # it as absent and keep polling at the short interval until
-        # ensure_notify_listener restarts it. Only a live listener earns the
-        # long NOTIFY-mode ceiling.
-        return effective_polling_interval unless notify_wakeup? && @notify_listener&.running?
+        # ensure_notify_listener restarts it. A live-but-deaf listener
+        # (delivering? false — a transaction-mode pooler or replica that drops
+        # LISTEN) is the same story: it will never wake us, so don't raise the
+        # poll floor to the 15s ceiling behind it (issue #332). Only a live,
+        # delivering listener earns the long NOTIFY-mode ceiling.
+        return effective_polling_interval unless notify_wakeup? && listener_delivering?
 
         [effective_polling_interval, config.polling_interval, NOTIFY_FALLBACK_POLL_SECONDS].max
       end
 
+      def listener_delivering?
+        @notify_listener&.running? && @notify_listener.delivering?
+      end
+
       def effective_polling_interval
         return config.polling_interval if @consumer_priority.zero?
 

data/lib/pgbus/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Pgbus
-  VERSION = "0.9.11"
+  VERSION = "0.10.0"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: pgbus
 version: !ruby/object:Gem::Version
-  version: 0.9.11
+  version: 0.10.0
 platform: ruby
 authors:
 - Mikael Henriksson