pgbus 0.16.1 → 0.16.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 869b9dc59dc3fdab2396f6dcfa8e1fa04372abd23cdd527e0a07078d2d93a98d
-  data.tar.gz: 9fa6807b78c09b3ef397b98013165e866233ca8bceb6d660a3f12fe5a5b696d8
+  metadata.gz: bee03fec1c58fc9cda375da9c6aa4009f13091b5a3f042ae7de6e33d0a53fddb
+  data.tar.gz: 36da6db6f6786a67c9d0d8894501b31d783ca88011695683ecdf7da17aeaf749
 SHA512:
-  metadata.gz: 519e747379272929b6add403ca50e3b7b9a64df5b326a73b8372c24dd87d5edea58076fc9d93e95bd78de1df5e46482f095da283a67bf093aceae659ca895218
-  data.tar.gz: 5f32250830c195bb1e35422fa4ff504ff3e95cd0ba8afdd8cb9b239f18e1f3fc860339bf7fa5ca2284438413e2b8dfbae6e530b922418df1f5464641cd9f0b4a
+  metadata.gz: 133f075c1832d10fcfeb49eb937fe274f8009f03ce940970aef123a11c24129429122316212a02e42e4b40186d4c181b179bb9fa1237073cc5709da27edace4d
+  data.tar.gz: a52c6d64d27c696cb5287cdcccd03876a082eeb0106fd2c1978b8e95accf0f90f5474e99e9117ccf2039ffdf3ec5bdc9ee48279fc8bac59c98207ab39c3073c3

data/CHANGELOG.md

@@ -2,6 +2,8 @@
 
 ### Fixed
 
+- **Concurrent first-broadcast notify-insert setup no longer pages on a Postgres deadlock / lock-wait (the residual of #403).** Two processes racing the first durable Turbo/Pgbus stream broadcast can both see "notify trigger not current" and both run `pgmq.enable_notify_insert` → `DROP TRIGGER` under `AccessExclusiveLock`. Postgres deadlocks, or hits `lock_timeout`, or — on pooled connections where `statement_timeout` < `lock_timeout` — `canceling statement due to statement timeout` **while locking**. PGMQ wraps those as `PGMQ::Errors::ConnectionError`. 0.16.1 already skips DROP when the trigger is current and retries stale sockets via `with_stale_connection_retry`, but it did not retry these lock races, so the residual still paged (getzazu/app#3817; AppSignal Zazu ma-prod #546 / za-prod #530). `ensure_stream_queue` now wraps the notify-setup path in a dedicated `with_notify_lock_retry` (up to 3 attempts, short backoff **outside** `@pgmq_mutex`): deadlock, lock-not-available, lock-timeout, and statement-timeout **with** lock-wait context retry; a bare statement timeout and permission errors still fail fast, while a missing-queue `ConnectionError` uses the existing one-time queue-recreation path without lock retries. `with_stale_connection_retry` is unchanged — that helper stays idle-socket / no-SQL-sent only. Consumers that prepended an app-level retry around `ensure_stream_queue` (getzazu/app#3827) can drop that prepend after upgrading.
+
 - **A client that connects to the health port and never sends a request line no longer wedges every later probe (issue #455).** `Pgbus::Web::HealthServer` serves every probe from one accept-loop thread and read the request line with an unbounded `client.gets("\r\n", MAX_REQUEST_LINE)`, so a silent connection parked that read forever: docker's `--health-cmd` (`bin/pgbus-health`) then timed out on every attempt and the container stayed unhealthy for the rest of its life while the supervisor and its workers were fine. Each accepted connection now carries a 1-second `IO#timeout` — deliberately under `HealthProbe::DEFAULT_TIMEOUT` (2s), so one wedged client can't push the next probe past its own deadline — and the resulting `IO::TimeoutError` (an `IOError`) falls into the existing per-connection `rescue StandardError`, which logs it, drops the connection and frees the loop. `#stop` no longer leaks the thread either: a loop parked on an already-accepted client never sees the listening socket close, so `stop` kills the thread when it outlives the join and joins again afterwards (`Thread#kill` is asynchronous), instead of returning "stopped" while it is still running. Only reachable from inside the container on the `health_bind` 127.0.0.1 default — a wedged health gate, not a security issue. Refs #455.
 - **The unbound-lock reaper no longer counts a dead-lettered copy as a live message, so a `:until_executed` job that once dead-lettered can be enqueued again.** `Client#uniqueness_keys_present` scanned every queue in `pgmq.meta` for the payload's `pgbus_uniqueness_key` — including `*_dlq` queues. A dead-lettered copy keeps the original payload (key included) but is not in flight: the executor already released the lock when it moved the message. For a legacy `pending`/`msg_id=0` row (pre-#418 enqueue, or a bind that failed) the DLQ copy made the key look "still here" forever, the reaper kept the row, and an `on_conflict: :discard` job — a heartbeat, typically — was discarded on every enqueue until someone purged the DLQ by hand. Seen in production as three heartbeats silently dead for nine days behind a 468-message DLQ. Dead-letter queues are now skipped in the scan; bound-lock probes (`message_exists?`) were never affected because they resolve the logical queue's physical tables only.
 - **`Pgbus::Testing.disabled!` can no longer turn a Capybara teardown race into a hung test process (issue #443).** The `streams_test_mode` stub closed immediately, so a page on a stream-bearing layout had its `EventSource` reconnect every ~3s for the whole example; a reconnect landing after `disabled!` had switched test mode off — a config-level `after` hook runs *before* `Capybara.reset_sessions!` — took the real path and started a live `Streamer` (listener/dispatcher/heartbeat threads + a LISTEN connection) inside the RSpec process. Its orphaned threads then shared the test's pinned AR connection and CI died at the job timeout with only `message type 0x5a arrived from server while idle`. Three changes: the stub now emits `retry: 86400000` so the browser does not reconnect at all; `Streamer::Instance#shutdown!` snapshots every component's threads (`#threads` on Listener/Dispatcher/Heartbeat/OutboundPump/HubClient/FailoverListener), logs one error naming those still alive after their bounded joins, and returns that list (`Streamer.reset!` forwards it, `nil` when nothing was live); and `Testing.disabled!` raises `Pgbus::Testing::StreamerLeakError` — pointing at `config.append_after` — when threads leaked, while a live streamer that shut down cleanly is only logged. README and docs now recommend `config.append_after` for the teardown hook.

data/lib/pgbus/client/ensure_stream_queue.rb

@@ -78,22 +78,33 @@ module Pgbus
       # peer process's memo. The memoized path then skips `pgmq.create` and
       # `enable_notify_insert` raises "Queue does not exist".
       def ensure_stream_queue_tables(full_name, stream_name)
-        ensure_single_queue(full_name)
-
-        # PGMQ's default NOTIFY throttle is 250ms — meant to coalesce
-        # high-frequency worker queue inserts. Streams are latency-
-        # sensitive and need every broadcast to fire a NOTIFY, even
-        # when several are batched within a single millisecond.
-        # Override the throttle to 0 specifically for stream queues.
-        # Use the idempotent path to avoid deadlocks when multiple
-        # processes race to set up the same stream queue.
-        synchronized { enable_notify_if_needed(full_name, 0) }
+        # Lock-retry wraps one setup attempt (create-path 250ms notify
+        # plus the stream 0ms override). Missing-queue recovery stays
+        # *outside* that budget so a later lock error cannot re-run
+        # forget+recreate up to ATTEMPTS times. The recovery attempt
+        # gets its own lock-retry — the DROP TRIGGER race can still
+        # happen after we recreate. Sleeps run after synchronized
+        # releases @pgmq_mutex. Do not fold these errors into
+        # with_stale_connection_retry (idle-socket only).
+        setup_stream_queue_tables(full_name, stream_name)
       rescue PGMQ::Errors::ConnectionError => e
         raise unless missing_pgmq_queue_error?(e)
 
         forget_stream_queue_memo!(full_name, stream_name)
-        ensure_single_queue(full_name)
-        synchronized { enable_notify_if_needed(full_name, 0) }
+        setup_stream_queue_tables(full_name, stream_name)
+      end
+
+      def setup_stream_queue_tables(full_name, stream_name)
+        with_notify_lock_retry(stream_name) do
+          ensure_single_queue(full_name)
+
+          # PGMQ's default NOTIFY throttle is 250ms — meant to coalesce
+          # high-frequency worker queue inserts. Streams are latency-
+          # sensitive and need every broadcast to fire a NOTIFY, even
+          # when several are batched within a single millisecond.
+          # Override the throttle to 0 specifically for stream queues.
+          synchronized { enable_notify_if_needed(full_name, 0) }
+        end
       end
 
       def forget_stream_queue_memo!(full_name, stream_name)

data/lib/pgbus/client/notify_lock_retry.rb

@@ -0,0 +1,114 @@
+# frozen_string_literal: true
+
+module Pgbus
+  class Client
+    # Retries Postgres deadlocks / lock timeouts around stream-queue
+    # notify-insert setup.
+    #
+    # 0.16.1 already skips `pgmq.enable_notify_insert` when the notify
+    # trigger is current (#403 treats a concurrent CREATE as success), but
+    # two processes can still both see "not current" and deadlock — or hit
+    # lock_timeout / statement_timeout-while-locking — on `DROP TRIGGER`
+    # under AccessExclusiveLock. PGMQ wraps those as
+    # `PGMQ::Errors::ConnectionError`, which `with_stale_connection_retry`
+    # must not treat as stale (that helper is idle-socket / no-SQL-sent
+    # only).
+    #
+    # Sleep here — in the rescue, *outside* the yielded block — so the
+    # backoff never runs while `synchronized` holds `@pgmq_mutex`. Callers
+    # wrap the full stream-queue tables attempt (create-path 250ms notify
+    # plus the stream 0ms override), not hold the mutex across this method.
+    # Missing-queue errors stay outside this policy (one-time recreation
+    # in `ensure_stream_queue_tables`); permission errors fail fast.
+    # On the shared-AR path a lock error inside the caller's transaction
+    # aborts that transaction — retrying on the same connection would
+    # raise "current transaction is aborted" and hide the lock error, so
+    # we re-raise the original instead.
+    module NotifyLockRetry
+      ATTEMPTS = 3
+      DELAYS = [0.05, 0.15, 0.35].freeze
+
+      # Also match statement-timeout lock waits: pooled connections often
+      # keep lock_timeout above statement_timeout, so a DROP TRIGGER waiter
+      # dies as PG::QueryCanceled ("canceling statement due to statement
+      # timeout / while locking"), not LockWaitTimeout. Bare statement
+      # timeout (no lock context) is deterministic — do not retry.
+      LOCK_FAILURE_PATTERN = /
+        deadlock\ detected
+        |lock\ not\ available
+        |canceling\ statement\ due\ to\ lock\ timeout
+      /ix
+      STATEMENT_TIMEOUT = /canceling statement due to statement timeout/i
+      LOCK_WAIT_CONTEXT = /while locking/i
+
+      def self.retryable?(error)
+        current = error
+        depth = 0
+        while current && depth < 8
+          return true if lock_class?(current) || lock_message?(current)
+
+          current = current.cause
+          depth += 1
+        end
+        false
+      end
+
+      def self.lock_class?(error)
+        return false if error.nil?
+
+        (defined?(::PG::TRDeadlockDetected) && error.is_a?(::PG::TRDeadlockDetected)) ||
+          (defined?(::PG::LockNotAvailable) && error.is_a?(::PG::LockNotAvailable)) ||
+          (defined?(::ActiveRecord::Deadlocked) && error.is_a?(::ActiveRecord::Deadlocked)) ||
+          (defined?(::ActiveRecord::LockWaitTimeout) && error.is_a?(::ActiveRecord::LockWaitTimeout))
+      end
+      private_class_method :lock_class?
+
+      def self.lock_message?(error)
+        return false if error.nil?
+
+        message = error.message.to_s
+        return true if LOCK_FAILURE_PATTERN.match?(message)
+
+        STATEMENT_TIMEOUT.match?(message) && LOCK_WAIT_CONTEXT.match?(message)
+      end
+      private_class_method :lock_message?
+
+      private
+
+      def with_notify_lock_retry(stream_name)
+        attempts = 0
+        begin
+          yield
+        rescue StandardError => e
+          attempts += 1
+          raise unless attempts < ATTEMPTS && NotifyLockRetry.retryable?(e)
+          # Shared-AR connection already inside (or aborted by) the caller's
+          # transaction: a retry on the same socket cannot recover and would
+          # mask this lock error with "current transaction is aborted".
+          # Dedicated-pool and idle after_commit paths still retry.
+          raise if notify_lock_retry_blocked_by_caller_transaction?
+
+          # Sleep here — in the rescue, *outside* the yielded block — so the
+          # backoff never runs while @pgmq_mutex is held: on the shared-
+          # connection path the mutex lives inside `synchronized` within the
+          # yielded block, and the raise unwinds out of it (releasing the
+          # mutex) before we get here. See DELAYS. Clamp the index to the
+          # last delay so a future ATTEMPTS > DELAYS.size never sleeps nil.
+          sleep DELAYS[[attempts - 1, DELAYS.size - 1].min]
+
+          Pgbus.logger.warn do
+            "[Pgbus::Client] Retrying stream-queue notify setup after a Postgres lock failure " \
+              "(attempt #{attempts}/#{ATTEMPTS} stream=#{stream_name}): #{e.message}"
+          end
+          retry
+        end
+      end
+
+      def notify_lock_retry_blocked_by_caller_transaction?
+        queue_ddl_rides_caller_transaction?
+      rescue StandardError
+        false
+      end
+    end
+  end
+end

data/lib/pgbus/client.rb

@@ -4,6 +4,7 @@ require "json"
 require "socket"
 require "timeout"
 require_relative "client/read_after"
+require_relative "client/notify_lock_retry"
 require_relative "client/ensure_stream_queue"
 require_relative "client/notify_stream"
 require_relative "client/connection_health"
@@ -13,6 +14,7 @@ module Pgbus
   class Client
     include ReadAfter
     include FairRead
+    include NotifyLockRetry
     include EnsureStreamQueue
     include NotifyStream
 

data/lib/pgbus/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Pgbus
-  VERSION = "0.16.1"
+  VERSION = "0.16.2"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: pgbus
 version: !ruby/object:Gem::Version
-  version: 0.16.1
+  version: 0.16.2
 platform: ruby
 authors:
 - Mikael Henriksson
@@ -271,6 +271,7 @@ files:
 - lib/pgbus/client/connection_health.rb
 - lib/pgbus/client/ensure_stream_queue.rb
 - lib/pgbus/client/fair_read.rb
+- lib/pgbus/client/notify_lock_retry.rb
 - lib/pgbus/client/notify_stream.rb
 - lib/pgbus/client/read_after.rb
 - lib/pgbus/client/resizable_pool.rb