pgbus 0.23.0 → 0.23.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: f3456584745e787627f4a24ccf2bc9ee4d9e6703c8e507aa696d1390660c032f
-  data.tar.gz: a9abc1e458a6f06e4ebd9bf96dae36e3a93a880c338374e53c62a80ab2a6da0d
+  metadata.gz: 9d5cf17ed290df4dfd9ab65b43e594a67c2050ed0b4c9b6b3d19c886f9a7191c
+  data.tar.gz: 902f6b6c806ef2861d42fed64a9d0e1283c4165c2df44333a517d8c45f5c55cf
 SHA512:
-  metadata.gz: a63b3ca0d61efffa0cdd9545eb911aaffd3732f60f646b0d9ff6a3711a7ac83332c13e1d3d9703ecc6dde35e0b3fc0e51ebe53f37ff6c8b6bceaad924f77a423
-  data.tar.gz: 6cca1439453ad78ef45824a1ffbd29891c2451295af842c0efb2c38abdcd58db5e09ba6980d84a6721e9e7bca9c533cbdd0ad76ca29f07c607e7d2c9ab858e75
+  metadata.gz: 749638cf39ec81aeb595b5b8104dae598253d02278ee68b22da1f7edd949434c2691d5415b02acecc69c486ef4c487dbae241dd96b80d7ee8ffba723e1b9b029
+  data.tar.gz: c1c134bc2d1347cfe26348b1742a05fd44eba7cb567574690418c05ff53cb0af359f1695a099ef99f0ea05f56ab621f9a0b69e6bbb10a5c63dd098f3cc64354c

data/CHANGELOG.md

@@ -1,5 +1,11 @@
 ## [Unreleased]
 
+## [0.23.1] - 2026-11-04
+
+### Fixed
+
+- **A stale-connection retry drops the pooled connections before its last attempt, so a connection libpq left mid-reset no longer fails every call until the process restarts (issue #645).** A `PG::Connection#reset` that `Thread#raise` interrupted while it was polling (a request-timeout middleware, a forced shutdown) leaves the pooled connection in `CONNECTION_STARTED`: neither `finished?` nor `CONNECTION_BAD`, so pgmq-ruby's checkout health check keeps handing it out and libpq refuses every send with `PQsendQueryParams no connection to the server`. `with_stale_connection_retry` matched that as stale and retried twice on the same socket, so every enqueue and read from that process failed (seen for 15+ minutes after a network partition, on one web worker, until it was restarted). The first retry is unchanged: it goes through pgmq-ruby's health check alone, which replaces a socket the server closed without touching the rest of the pool. When that retry fails too, the client drops the idle connections of both pools (the job pool and the streams pool, as `Client#reload` does) before the last one, and logs one warning, `Dropping pooled connections before the last stale-connection retry (attempt 2/2): <error>`. Connections other threads hold are discarded when they come back, as on the #354 wedged-read path. Dedicated connections only: on the shared ActiveRecord connection (a `connection_params` Proc) the pools are never reloaded, and nothing extra is logged. Each pool reloads on its own, so one reload that raises does not stop the other; a failure is logged at warn and the retry still runs; the caller still sees the stale `ConnectionError` if the retry fails. A non-stale `ConnectionError` (a pool timeout, a completion whose result was lost) is neither retried nor reloaded, as before. Refs #645.
+
 ## [0.23.0] - 2026-11-02
 
 ### Added

data/README.md

@@ -25,6 +25,7 @@ PostgreSQL-native job processing and event bus for Rails, built on [PGMQ](https:
   - [Circuit breaker and queue pause/resume](#circuit-breaker-and-queue-pauseresume)
   - [Client-level circuit breaker (database-down)](#client-level-circuit-breaker-database-down)
   - [Read timeouts (libpq-native)](#read-timeouts-libpq-native)
+  - [Stale connection retries](#stale-connection-retries)
   - [Prefetch flow control](#prefetch-flow-control)
   - [Read-ahead](#read-ahead)
   - [Worker recycling](#worker-recycling)
@@ -550,6 +551,10 @@ production:
 
 A custom RuboCop cop, `Pgbus/NoRubyTimeout`, bans `Timeout.timeout` project-wide to prevent this class of bug from regressing.
 
+### Stale connection retries
+
+A `PGMQ::Errors::ConnectionError` whose message says the pooled connection was dead before it was used (`no connection to the server`, `connection is closed`, a PgBouncer-closed socket and the like) is retried twice, after about 0.1 s and 0.5 s. Errors that may have reached the server (`server closed the connection unexpectedly`) and pool timeouts are never retried. The first retry relies on pgmq-ruby's checkout health check, which replaces a closed connection. On a dedicated connection (`database_url` or `connection_params`), before the last retry, pgbus drops the idle connections of both pools (job and streams) and logs `Dropping pooled connections before the last stale-connection retry`. This recovers a connection the health check keeps handing out, such as one a `Thread#raise` left half-way through a reset (`CONNECTION_STARTED`). A read interrupted by the Ruby `Timeout` fallback above reloads the job pool the same way. `Pgbus.client.reload` drops the idle connections of both pools by hand. None of these reload anything on the shared ActiveRecord `Proc` connection, whose socket pgbus does not own.
+
 ### Prefetch flow control
 
 Cap the number of in-flight (claimed but unfinished) messages per worker:

data/lib/pgbus/client.rb

@@ -1876,6 +1876,16 @@ module Pgbus
     # restart or a brief failover — that outlasts the first immediate retry
     # still gets a second, backed-off chance rather than failing an enqueue
     # the caller may never retry.
+    #
+    # The last retry runs on freshly reloaded pools (issue #645). A reset that
+    # Thread#raise interrupted mid-poll leaves the pooled PG::Connection in
+    # CONNECTION_STARTED: neither finished? nor CONNECTION_BAD, so pgmq-ruby's
+    # checkout health check keeps handing it out and libpq refuses every send
+    # ("no connection to the server"). The first retry still goes through that
+    # health check alone, which replaces a socket the server closed without
+    # touching the rest of the pool; only when it fails too are both pools'
+    # idle connections dropped (reload_pools_before_last_stale_retry), as the
+    # #354 wedged-read path does.
     STALE_RETRY_ATTEMPTS = 2
     private_constant :STALE_RETRY_ATTEMPTS
 
@@ -2007,6 +2017,31 @@ module Pgbus
       end
     end
 
+    # Drop both pools' idle connections before the last stale retry (issue
+    # #645; see STALE_RETRY_ATTEMPTS), so it checks out a fresh connection
+    # instead of the one pgmq-ruby's health check kept handing back. Skipped
+    # silently on the shared-AR Proc path: those slots wrap ActiveRecord's own
+    # socket (#reload refuses there with a warning that would be noise on every
+    # retry). Each pool reloads on its own, so one failing does not keep the
+    # other's connections. A reload failure is logged, never raised: the stale
+    # ConnectionError the retry may raise next is the actionable error.
+    def reload_pools_before_last_stale_retry(attempt, error)
+      return if @shared_connection
+
+      Pgbus.logger.warn do
+        "[Pgbus::Client] Dropping pooled connections before the last stale-connection retry " \
+          "(attempt #{attempt}/#{STALE_RETRY_ATTEMPTS}): #{error.message}"
+      end
+      { "job" => @pgmq, "streams" => @streams_pool }.each do |name, pool|
+        pool.reload
+      rescue StandardError => e
+        Pgbus.logger.warn do
+          "[Pgbus::Client] #{name} pool reload before the last stale-connection retry failed: " \
+            "#{e.class}: #{e.message}"
+        end
+      end
+    end
+
     # True when libpq's connection-baked read bounds (statement_timeout +
     # tcp_user_timeout + keepalives) fully cover both a slow query AND a
     # dead/hung socket, so with_read_timeout can skip the Ruby Timeout entirely.
@@ -2293,6 +2328,7 @@ module Pgbus
           "[Pgbus::Client] Retrying after stale pgmq connection " \
             "(attempt #{attempts}/#{STALE_RETRY_ATTEMPTS}): #{e.message}"
         end
+        reload_pools_before_last_stale_retry(attempts, e) if attempts == STALE_RETRY_ATTEMPTS
         retry
       end
     end

data/lib/pgbus/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Pgbus
-  VERSION = "0.23.0"
+  VERSION = "0.23.1"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: pgbus
 version: !ruby/object:Gem::Version
-  version: 0.23.0
+  version: 0.23.1
 platform: ruby
 authors:
 - Mikael Henriksson