pgbus 0.11.4 → 0.12.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: c43ef6f3c3efe57de0f647a0811f83bcf98f08e49c63e2da3d6b9cae1c4d3f16
-  data.tar.gz: edb510a56a9410f8cbf127504c3e31ae459561620a7ef62c7011c97f665908ae
+  metadata.gz: f2e4a0eacc67a1d3edf2fa7aae4ffd835cf03ae275f016aa92c2f490923897cd
+  data.tar.gz: defeb677222dc6ac50c755225e3da9b57752667e03569af747f1f8a26974b95d
 SHA512:
-  metadata.gz: 21f9209e70c028b51c2794ac2a73d71f55b06e0a1e23fede2c7a074c27c732bf88959a859c2f9f79de8484cd4cb51cc4c876fc801ab6981fc05541926312dd34
-  data.tar.gz: f15cf91037ccee58333e0e74648db8336acf40511ef495a608428ff839e340e5c6ddd06a89cde323d2a6f08ea5b2f5abfbd9717acfbd5ed3d2e3023a1b47f84b
+  metadata.gz: 2a9b8e1d6efa1106a5f56325d48bae1bcc650916154e54ccffa675fa9a5fde887f412536ca6d40d56e907ec804de66b3b1ca2000c5c66142889d9f8ff07dce3d
+  data.tar.gz: 0f5da2baedef6ad70d304e1d91339a6f7613687c642d40bb7b52fd508d198fde5840a22b2f42f2a0bbf3429aea7b9f8f1c71d20d63e35dbaa29b787b43fb4601

data/CHANGELOG.md

@@ -10,6 +10,8 @@
 
 ### Added
 
+- **`Pgbus::Client#reload` — operator escape hatch that drops every pooled PGMQ connection (job pool AND the live streams pool) and lets the pools rebuild lazily on next checkout (issue #354).** Built on pgmq-ruby 0.7.1's `PGMQ::Client#reload`; use it to recover connections libpq still reports as `CONNECTION_OK` but that are in fact wedged (e.g. after a wall-clock interrupt cut a query mid-flight), which pgmq-ruby's checkout health check cannot detect. Unlike `#close` the pools stay usable, and connections checked out by other threads mid-reload are unaffected. The streams half goes through the `ResizablePool` wrapper, so it always targets the live (possibly hot-swapped, #323) client. No-op with a warning on the shared-AR Proc path — those pool slots wrap ActiveRecord's own raw connection, and pgbus won't close a socket it doesn't own. Refs #354.
+- **PGMQ: vendored schema v1.12.0.** Adds `pgmq.read_grouped_head_with_poll` — a polling wrapper over `read_grouped_head` (reads the heads of N FIFO groups, waiting up to `max_poll_seconds` for messages); the 1.11.1 → 1.12.0 upstream delta is otherwise comments/whitespace only, so nothing pgbus calls through `Pgbus::Client` changes. Fresh embedded installs get 1.12.0; existing installs: `rake pgbus:pgmq:status`, then `rails generate pgbus:upgrade_pgmq` + `rails db:migrate` (`rails db:migrate:pgbus` on a separate-database install). No `Pgbus::Client` API is added for the new function (the locked pgmq-ruby does not expose it; pgbus has no consumer). Refs #351.
 - **Doctor check 10: "Dedicated connections" — the preflight now opens the streamer/notify-listener connections exactly the way the runtime does (issue #352).** The streamer's LISTEN connection and the worker `NotifyListener` bypass the Client's pgmq pools entirely (they own a single raw `PG` connection each), so every existing doctor check — all of which probe through `Pgbus::Client` — passed while both dedicated paths were broken. The new check builds each configured dedicated connection via `Pgbus::DedicatedConnection` (streams when `streams_enabled`, notify when `worker_notify_wakeup?`), runs `SELECT 1`, and closes it, failing with the underlying PG error and the affected path label. Not strict-fatal (same reasoning as the Database check: a transient DB blip at boot must not lockstep-abort a fleet), but with `doctor_on_boot` it turns "every SSE request 500s after deploy" into a named failing check in the boot report. Refs #352.
 - **`pgbus dashboard` — print the vendored AppSignal dashboards as import-ready JSON.** The gem ships four AppSignal dashboard definitions (the main one submitted to appsignal/public_config#80 plus the health/streams/throughput extras), but they're stored in the automated-dashboard format — a `metric_keys` trigger wrapping the `dashboard` object — which AppSignal's "Import dashboard" dialog doesn't accept, so trying one on a real app meant hand-unwrapping JSON. `pgbus dashboard [main|health|streams|throughput]` prints the inner dashboard object ready to paste into the import dialog (`pgbus dashboard health | pbcopy`); `pgbus dashboard --list` enumerates the available names and titles.
 - **`config.doctor_on_boot` (+ `pgbus start --doctor` / `--doctor-strict`) — run the doctor preflight inside the booting supervisor, one Rails boot instead of two (issue #347).** The recommended deploy preflight was `bin/pgbus doctor || true` in the container entrypoint before `bin/pgbus start` — but both commands `require config/environment`, so the job container boots the full Rails app **twice** on every deploy (cost scaling with app boot time), and running doctor pre-supervisor makes the `Process liveness` check false-fail (no workers exist yet), which is exactly why the entrypoint has to swallow the exit code with `|| true` — losing any chance to gate on a genuinely-fatal finding. Now `config.doctor_on_boot = :report` (or `pgbus start --doctor`) runs the checks in the already-booted supervisor, after the DB is verified reachable and queues are bootstrapped but **before any worker is forked**: one boot, and the worker-dependent `Process liveness` check is skipped (it has no workers to observe yet and would false-fail on stale prior-generation rows during a redeploy). `config.doctor_on_boot = :strict` (or `--doctor-strict`) additionally **refuses to boot** — raising before forking anything, so the `ensure shutdown` path tears down the heartbeat/health server and no child starts — but *only* on a genuinely-fatal, non-transient check: a `Configuration` failure (a real config bug) or an **absent** PGMQ schema. It deliberately does **not** abort on `Queues`/`Database` failures — those are the transients the lenient queue bootstrap is built to ride out (children crash-and-backoff until the DB recovers), and `verify_connection!` already gated a hard-down DB moments earlier; making them strict-fatal would take down a whole fleet's cold boot in lockstep on a momentarily-saturated primary. Default `nil`/`false` = off (byte-identical to today; the supervisor constructs no `Doctor` and does zero extra work). The standalone `pgbus doctor` command and its exit-code semantics are unchanged. Refs #347.
@@ -36,6 +38,7 @@
 
 ### Fixed
 
+- **The Ruby-Timeout read fallback no longer leaves a wedged `CONNECTION_OK` connection in the pgmq pool to re-hang the next read (issue #354).** On the one path where libpq can't bound a hung socket (dedicated connection on non-Linux hosts or libpq < 12), the last-resort `Timeout.timeout` interrupts the read via `Thread#raise` — and libpq may leave the pooled `PG::Connection` reporting `CONNECTION_OK` while it will in fact re-hang on reuse, invisible to pgmq-ruby's checkout health check (it isn't `CONNECTION_BAD`). This was a documented KNOWN LIMITATION waiting on a public pool-reload upstream (cf. mensfeld/pgmq-ruby#94); pgmq-ruby 0.7.1 shipped it, so the fallback now raises an internal `ReadTimeoutError` subclass and reloads the job pool before re-raising — the poisoned connection (already checked back in by `connection_pool`'s ensure) is dropped and the pool rebuilds lazily. A clean server-side `statement_timeout` cancel still never triggers a reload (the connection is healthy), and a reload failure is logged without masking the timeout. Requires pgmq-ruby >= 0.7.1 (dependency bumped to `~> 0.7.1`). Refs #354.
 - **`connection_guc_mode = :session` no longer kills every SSE stream and NOTIFY wakeup with `PG::Error: invalid connection option "variables"` (issue #352).** In `:session` mode, `forward_connection_variables` deliberately leaves the database.yml `variables:` hash on the connection options for the *caller* to strip and apply via post-connect `SET` (a transaction-mode pooler rejects the libpq `options` startup param — the reason `:session` mode exists). `Pgbus::Client#wrap_session_gucs` honors that contract on both pgmq pools, but the two dedicated raw-connect paths — the streamer's `build_raw_pg_connection` and the worker `NotifyListener` — passed the hash straight to `PG.connect`, which rejects the non-libpq `:variables` key. Result: the first stream request per process raised, `StreamApp` rescued it into a 500 (so **every** `/pgbus/streams/…` request failed from then on), and NOTIFY-gated wakeups died with workers silently falling back to polling. Both paths now connect through the new `Pgbus::DedicatedConnection.connect`, which strips `:variables` and applies each GUC via post-connect `SET` — dedicated connections *keep* the operator's GUCs (`statement_timeout`, `timezone`, …), matching the pooled paths, and the mechanism works on both a direct port and a session-mode pooler. A guard spec confines raw `PG.connect` call sites to `Client` and `DedicatedConnection`, so the next dedicated-connection path can't silently reintroduce the bypass. Refs #352.
 - **AppSignal main dashboard: job and event lines now split by `job_class` and `routing_key`.** Applied the upstream review suggestions committed on appsignal/public_config#80 to the vendored `dashboard.json`: the "Job status per queue" panel gains a `job_class` wildcard tag filter and label (`%job_class% - %queue% - %status%`), and "Event handler status" gains `routing_key` (`%handler% - %status% - %routing_key%`). Without the wildcard tag entry a `%placeholder%` in the line label never resolves, so lines from different job classes / routing keys were collapsed together. The subscriber already emitted both tags; only the dashboard definition lagged.
 - **Migration generators now route to the pgbus database when `connects_to` is configured, even without an explicit `--database` (issue #344).** `rails g pgbus:add_stream_queues` (and every sibling generator via `Generators::MigrationPath`) keyed the separate-database decision purely on the `--database` flag. An app that had already set `config.connects_to = { database: { writing: :pgbus } }` (so its pgbus migrations live in `db/pgbus_migrate/` per database.yml's `migrations_paths:`) still got the migration written to the **primary** DB's `db/migrate/` on a bare invocation — where it then ran against the wrong database — and the "Next steps" output told the operator to run `rails db:migrate` instead of `rails db:migrate:pgbus`. `MigrationPath` now auto-detects the target database from `connects_to` (via the existing `DatabaseTargetDetector`: runtime config → initializer scan → application.rb scan) when `--database` is absent; an explicit `--database` still wins. The detected name drives both the migration path (resolving `migrations_paths` for that database, falling back to the `db/pgbus_migrate` convention) and the `db:migrate:<name>` suffix in every generator's post-install output. An app with no separate database is unaffected. Refs #344.

data/lib/pgbus/client/resizable_pool.rb

@@ -101,6 +101,15 @@ module Pgbus
         end
       end
 
+      # Drop every pooled connection on the CURRENT streams client and let it
+      # rebuild lazily on next checkout (pgmq-ruby >= 0.7.1) — see
+      # Client#reload (issue #354). Takes the swap mutex so a reload cannot
+      # race a concurrent swap/close into reloading a pool that is being
+      # retired; reads stay lock-free.
+      def reload
+        @swap_mutex.synchronize { @ref.get.pgmq.reload }
+      end
+
       def stats_snapshot
         @last || SwapStats.new(swap_count: 0, last_drain_seconds: 0.0, last_conns_closed: 0,
                                last_from_size: nil, last_to_size: nil, last_drained: nil)

data/lib/pgbus/client.rb

@@ -794,6 +794,32 @@ module Pgbus
       @streams_pool.stats_snapshot
     end
 
+    # Operator escape hatch (issue #354): drop every pooled PGMQ connection —
+    # job pool AND the live streams pool — and let the pools rebuild lazily on
+    # next checkout (pgmq-ruby >= 0.7.1). Use to recover connections libpq
+    # still reports as CONNECTION_OK but that are in fact wedged (e.g. after a
+    # wall-clock interrupt cut a query mid-flight), which pgmq-ruby's checkout
+    # health check cannot detect. Unlike #close, the pools stay usable.
+    # Connections checked out by other threads mid-reload are unaffected.
+    #
+    # No-op (returns false) on the shared-AR Proc path: those pool slots wrap
+    # ActiveRecord's own raw connection — reloading would close AR's socket
+    # out from under the application. Returns true after a reload.
+    def reload # rubocop:disable Naming/PredicateMethod -- command that reports whether it acted, like #ping
+      if @shared_connection
+        Pgbus.logger.warn do
+          "[Pgbus::Client] reload skipped: pgbus is sharing ActiveRecord's connection " \
+            "(Proc connection_options) and won't close a socket it doesn't own. " \
+            "Manage that connection through ActiveRecord instead."
+        end
+        return false
+      end
+
+      @pgmq.reload
+      @streams_pool.reload
+      true
+    end
+
     private
 
     # Human-readable label for which config knob supplied the connection
@@ -1146,6 +1172,18 @@ module Pgbus
     READ_TIMEOUT_SLACK = 5
     private_constant :READ_TIMEOUT_SLACK
 
+    # Raised (instead of plain ReadTimeoutError) when the Ruby Timeout last
+    # resort fires — i.e. Thread#raise interrupted a read mid-flight on a
+    # socket libpq couldn't bound (issue #354). IS-A Pgbus::ReadTimeoutError,
+    # so the public contract is unchanged; the distinct class lets
+    # with_read_timeout tell "wedged socket — reload the pool" (this) apart
+    # from "clean server-side statement_timeout cancel — connection healthy"
+    # (plain ReadTimeoutError), where reloading would churn a healthy pool on
+    # every slow query. Internal signal carried on the unwind path itself (no
+    # thread-local/ivar state around a Thread#raise interrupt); application
+    # code should rescue Pgbus::ReadTimeoutError.
+    class WedgedReadTimeout < Pgbus::ReadTimeoutError; end
+
     # Bound a read and surface a timeout as Pgbus::ReadTimeoutError. Prefer
     # libpq-native bounds baked into the connection; the Ruby Timeout is a
     # narrow, last-resort fallback used only where libpq cannot bound a hung
@@ -1179,12 +1217,19 @@ module Pgbus
     #      which AR passes straight through to the connection. #initialize logs a
     #      one-time hint when read_timeout is set on a Proc connection.
     #
-    #      KNOWN LIMITATION: when (3) fires on a genuinely hung socket, libpq may
-    #      leave the pooled PG::Connection reporting CONNECTION_OK while it will
-    #      in fact re-hang on reuse, and pgmq-ruby's health check won't discard
-    #      it (it isn't CONNECTION_BAD). The proper fix is a public pool-reload on
-    #      pgmq-ruby (follow-up, cf. mensfeld/pgmq-ruby#94); until then it's
-    #      documented and confined to the non-Linux dedicated path.
+    #      WEDGED-SOCKET RECOVERY (issue #354): when (3) fires on a genuinely
+    #      hung socket, libpq may leave the pooled PG::Connection reporting
+    #      CONNECTION_OK while it will in fact re-hang on reuse, and pgmq-ruby's
+    #      checkout health check won't discard it (it isn't CONNECTION_BAD). So
+    #      the Timeout raises WedgedReadTimeout (a ReadTimeoutError subclass)
+    #      and the rescue below drops every pooled connection via @pgmq.reload
+    #      (pgmq-ruby >= 0.7.1) — the pool rebuilds lazily on next checkout.
+    #      By the time the rescue runs, Thread#raise has unwound the read and
+    #      connection_pool's ensure has checked the poisoned connection back in
+    #      as idle, so reload does discard it. A small window remains where
+    #      another thread checks it out first; that thread's own read bound /
+    #      stale-retry covers it — reload narrows the window, it doesn't need
+    #      to close it atomically.
     #
     # MUST wrap only the bare `@pgmq.read*` call, inside both `synchronized` and
     # `with_stale_connection_retry`, so the Timeout clock starts only after the
@@ -1203,10 +1248,26 @@ module Pgbus
       return mapping_statement_timeout(&block) unless timeout&.positive?
 
       # rubocop:disable Pgbus/NoRubyTimeout -- deliberate last-resort bound; see above
-      Timeout.timeout(timeout + READ_TIMEOUT_SLACK, Pgbus::ReadTimeoutError) do
+      Timeout.timeout(timeout + READ_TIMEOUT_SLACK, WedgedReadTimeout) do
         mapping_statement_timeout(&block)
       end
       # rubocop:enable Pgbus/NoRubyTimeout
+    rescue WedgedReadTimeout
+      reload_pool_after_wedged_timeout
+      raise
+    end
+
+    # Best-effort job-pool reload after the Ruby Timeout fallback interrupted a
+    # read (see with_read_timeout). A reload failure is logged, never raised —
+    # the caller is already unwinding with ReadTimeoutError, which is the
+    # actionable error; masking it with a secondary pool failure would hide
+    # which read timed out.
+    def reload_pool_after_wedged_timeout
+      @pgmq.reload
+    rescue StandardError => e
+      Pgbus.logger.warn do
+        "[Pgbus::Client] pool reload after wedged read timeout failed: #{e.class}: #{e.message}"
+      end
     end
 
     # True when libpq's connection-baked read bounds (statement_timeout +