pgbus 0.12.3 → 0.12.4

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: b79edc460be04508034f7c4b34ba889f33493ab88878b3a272cf23d7f75d43ee
-  data.tar.gz: 9a000c729dfa32c5e3e6bb8b8df751fe75e159ff0a2160afee104b201e4f78e1
+  metadata.gz: 916f349cd5f3740aca0e462acdadeb3ad7322b0ce8f0363fa78d2328ac3c7c37
+  data.tar.gz: 13100607efcb25262f7d4708355d600009aff7f66e961d197d00fe2013ab9a88
 SHA512:
-  metadata.gz: 3ca04a2bb7436c7fc08f953b5fa19cc5130bb36de9d678287ffc49768bd6e3f142266a547e4ca852377b0fd66ce15cf18f1eac34857f75365dd62f9c7c10a754
-  data.tar.gz: 101a052d947241a6e35b80093846c20bfaec42454e40767073ad6cde8e6f64e09f3ae1ef879bd4686482c8b5e355d4770ddcba7ed61672465e7867b9382925f4
+  metadata.gz: 325b359be3de92b27add05e70ac9437b813df2e464ad1ceca2fbd3fe2114c4a52471e7e5f23564765e74d6c8330068cf0d9ab9ccaddd9443d98ab69088a1ee34
+  data.tar.gz: 5fb4ee93c03a463cddcfa490a438ae6d267fc12a23c633a7ee82f45754fde6232a23f5da2772df45f4653efbee4018b7b336adb79f50bf4e1ccc0438c4718971

data/CHANGELOG.md

@@ -6,6 +6,7 @@
 
 ### Fixed
 
+- **SIGTERM no longer segfaults the process: both LISTEN listeners stopped closing their PG connection from the stopping thread (issue #375).** `Process::NotifyListener#stop` (supervisor thread, on SIGTERM) closed the listener's dedicated `PG::Connection` to interrupt the blocking `wait_for_notify`, but left `@conn` pointing at it — so the listener thread unwound into `run_loop`'s `ensure` and ran `safe_unlisten_all`, exec'ing `UNLISTEN` on a connection the other thread was concurrently freeing. `PG::Connection#close` is `PQfinish`: it frees the PGconn **and its OpenSSL objects**, so `PQsendQuery` walked into freed TLS state and the whole worker died with `[BUG] Segmentation fault` — reproducibly, on essentially every container-replacement deploy against a TLS Postgres, sometimes in several forked workers at once. `rescue PG::Error` cannot catch a C-level SEGV, so the "safe" in `safe_unlisten_all` never held. **The connection is now single-owner**: the listener thread is the only thread that may exec, wait, or close on it, from build through teardown. `#stop` signals by clearing `@running` and joining — nothing more — and the listener thread closes its own connection in `run_loop`'s `ensure`. The stop is observed within one `wait_for_notify` timeout (`health_check_ms`, **250 ms** in the default configuration since workers/consumers derive it from `polling_interval`), so `#stop`'s join budget is now `health_check_ms + 5s` instead of a flat 5 s — a flat timeout could expire before a listener with a large `health_check_ms` had even one chance to observe the stop. The teardown `UNLISTEN` round-trip is **dropped entirely** rather than merely made safe: it ran immediately before the close, and closing a session deregisters every `LISTEN` server-side, so it bought nothing at any time. The identical defect in `Web::Streamer::Listener` — same close-from-`#stop`, same `UNLISTEN`-in-`ensure`, and there without even a mutex around `@conn` — is fixed the same way, so the dashboard's Puma worker stops crashing on the same deploy; its listener thread now also closes its own connection (previously only `#stop` did). Cost: shutdown can take up to one health-check cycle longer per listener — measured against a real PostgreSQL at the default `health_check_ms` of 250: **~207 ms on a fully idle queue, ~5 ms when the queue has any NOTIFY traffic** (a notification returns the wait immediately, and the loop then sees the cleared flag). 40 start/stop cycles under concurrent NOTIFY load leaked zero `LISTEN` backends. Thanks to the reporter for the crash dumps and the root-cause analysis. Refs #375.
 - **The health verdict no longer emits false STALLED reports — `max_read_ct` was never populated, and the wedge signal counted queues no capsule drains (issue #367).** Two correctness defects in `Pgbus::MCP::HealthAnalyzer` (surfaced through `pgbus doctor` / `pgbus_health` / the MCP health tool). **(1)** The `all_unread?` wedge check read `:max_read_ct`, but the metrics query never selected it — so the guard always degenerated to "never claimed" and a busy-but-healthy queue caught mid-burst produced a STALLED verdict with a factually wrong "read_ct=0 (never claimed)" reason. The metrics query now exposes a per-queue **`visible_unread_length`** (`count(*) WHERE vt <= NOW() AND read_ct = 0`) and the analyzer keys the wedge off *visible, never-claimed* messages. Counting per visible message (not `max(read_ct)` over the whole table) means one retried message left in-queue after its backoff, or an in-flight message claimed by a peer, can no longer veto the signal for a pile of genuinely-unclaimed jobs. **(2)** The verdict reasoned about **every** non-DLQ/non-stream queue against the global worker fleet — but a worker can only claim from queues its capsule subscribes to, so "M workers alive but never claimed" was vacuous for a queue nobody drains (ad-hoc queues, unregistered stream queues — see #366). The analyzer now intersects the STALLED backlog with `Web::DataSource#drained_queue_names` (each configured capsule's queues, priority `_pN` sub-tables expanded via the client's queue strategy, unioned with EventBus handler queues; `nil` for a `*` wildcard = drains everything, fail-open on error). Queues nobody drains get their own DEGRADED signal — *"N queue(s) hold messages but no capsule is configured to drain them"* — instead of being folded into the worker-wedge verdict. A heart-beating-but-`:stalled` worker is still reported STALLED regardless of which queue holds the backlog. Reason strings now truncate to the first 10 queue names with `(+N more)` so a flagged fleet of hundreds of queues doesn't produce a multi-KB log line. Refs #367, #366.
 - **`allowed_global_id_models` now actually guards ActiveJob arguments, not only EventBus payloads (issue #368).** The doctor warned in production that `nil` means "allow-all GlobalID arguments", but the allowlist was only enforced in `Serializer.locate_global_id` — reached from EventBus `_global_id` payloads — while the ordinary job path (`Executor` → `ActiveJob::Base.deserialize` → Rails' unrestricted `GlobalID::Locator`) never checked it. Operators who set an allowlist after following the doctor had a false sense of security; the common `SomeJob.perform_later(record)` pattern was unguarded. Job deserialization now goes through `Serializer.deserialize_job_data`, which walks `_aj_globalid` keys (including nested arrays/hashes) and reuses the same gate as EventBus when the allowlist is set; `nil` remains zero-cost allow-all. Rejected models raise `Pgbus::SerializationError` and are treated as a normal job failure. Apps with ActiveStorage attachments should include `ActiveStorage::Blob` (and related models) on the allowlist. Docs + doctor copy updated. Refs #368.
 - **Dormant pre-registry stream queues no longer false-STALL the doctor or escape the orphan sweep (issue #366).** Stream queues created before the `pgbus_stream_queues` migration only register on their next broadcast — completed checkout-style flows, ended chats, and one-shot progress streams never broadcast again, so they stay unregistered forever. Health/doctor then treated their permanent `read_ct=0` visible backlog as a worker wedge (STALLED on ~hundreds of healthy stream queues), and the orphan sweep skipped them because it only iterated the registry. Detection is fingerprint-based (the archive `a_<queue>_msg_id_idx` index that only `ensure_stream_queue` creates — no name heuristics). `StreamQueue.known_names` = registry ∪ fingerprints, used by health exclusion, orphan sweep / stream-archive prune, and wildcard workers; `rake pgbus:streams:backfill_registry` persists the missing rows; doctor surfaces a DEGRADED hint pointing at the backfill when unregistered fingerprints remain; `add_stream_queues` generator post-install mentions the task. Refs #366.

data/lib/pgbus/process/notify_listener.rb

@@ -30,12 +30,24 @@ module Pgbus
     # blocking IO call where the mutex MUST NOT be held), so wait_once reads
     # the connection out of the mutex first and operates on a local. Reconnect
     # publishes the new connection + channel set under the mutex.
+    #
+    # The mutex only makes the ivar READ safe. PG::Connection itself is not
+    # thread-safe, so the connection is single-owner: the listener thread is
+    # the ONLY thread that may exec, wait, or close on it, from build through
+    # teardown. #stop signals by clearing @running and joining — it never
+    # touches the connection, because #close is PQfinish and freeing the
+    # PGconn under a concurrent libpq call is a process-killing SEGV, not a
+    # rescuable PG::Error (issue #375).
     class NotifyListener
       CHANNEL_PREFIX = "pgmq.q_"
       CHANNEL_SUFFIX = ".INSERT"
 
       RECONNECT_BACKOFF_SECONDS = 0.5
 
+      # Grace added to one health-check cycle when #stop joins the listener
+      # thread. See #stop_join_timeout.
+      STOP_JOIN_GRACE_SECONDS = 5
+
       def initialize(physical_queues:, on_wake:, connection_options:,
                      health_check_ms: 1000, logger: Pgbus.logger)
         @physical_queues = Array(physical_queues)
@@ -80,22 +92,20 @@ module Pgbus
       end
 
       def stop
-        conn_to_close = nil
         @state_mutex.synchronize do
           return self unless @running
 
           @running = false
-          conn_to_close = @conn
         end
         @commands << [:stop]
-        # Interrupt the blocking wait by closing the socket; the rescue in
-        # wait_once sees @running == false and exits cleanly.
-        begin
-          conn_to_close&.close if conn_to_close.respond_to?(:close)
-        rescue StandardError
-          nil
-        end
-        @thread&.join(5)
+        # Deliberately does NOT touch @conn. PG::Connection is not thread-safe
+        # and #close is PQfinish: it frees the PGconn and its OpenSSL objects
+        # out from under whatever libpq call the listener thread is making on
+        # the same connection. That is a use-after-free — a process-killing
+        # SEGV, not a rescuable PG::Error (issue #375). The listener thread is
+        # the sole owner of @conn for its entire life and closes it in
+        # run_loop's ensure; clearing @running above is the whole stop signal.
+        @thread&.join(stop_join_timeout)
         @thread = nil
         self
       end
@@ -153,9 +163,26 @@ module Pgbus
         # Clear @running so #start can spawn a fresh thread after a fatal exit
         # (e.g. build_connection raising at boot). Without this, the dead
         # thread's @running stays true and #start returns early forever.
-        @state_mutex.synchronize { @running = false }
-        safe_unlisten_all
-        safe_close
+        #
+        # The LISTEN set is dropped as bookkeeping only — no UNLISTEN
+        # round-trip. We close on the next line and closing the session
+        # deregisters every LISTEN server-side, so the exec bought nothing
+        # while being the statement that raced #stop into a SEGV (issue #375).
+        #
+        # Capturing @conn in the SAME critical section that clears @running is
+        # what makes restart safe. #start is public and guarded only by
+        # @running, so a caller watching running? may spawn a fresh thread the
+        # instant it flips. If teardown read @conn in a later critical section
+        # it could pick up the NEW thread's connection and PQfinish it mid-use
+        # — the same use-after-free, reached through restart instead of #stop.
+        conn = @state_mutex.synchronize do
+          @running = false
+          @listening_to.clear
+          c = @conn
+          @conn = nil
+          c
+        end
+        close_quietly(conn)
       end
 
       def wait_once
@@ -166,7 +193,10 @@ module Pgbus
         got_notify = conn.wait_for_notify(timeout_s) do |_channel, _pid, _payload|
           @on_wake.call
         end
-        run_health_check(conn) unless got_notify
+        # Skip the keepalive when a stop landed during the wait: the loop is
+        # about to exit and close this connection anyway, so the round-trip
+        # would only add latency to shutdown.
+        run_health_check(conn) if !got_notify && running?
       rescue IOError, PG::Error => e
         return unless running?
 
@@ -270,14 +300,14 @@ module Pgbus
         end
       end
 
-      def safe_unlisten_all
-        channels, conn = @state_mutex.synchronize { [@listening_to.to_a, @conn] }
-        channels.each do |channel|
-          conn&.exec(%(UNLISTEN "#{channel}"))
-        rescue PG::Error
-          nil
-        end
-        @state_mutex.synchronize { @listening_to.clear }
+      # How long #stop waits for the listener thread to notice the cleared
+      # @running and finish teardown. The thread can be parked in
+      # wait_for_notify for one full health-check cycle before it re-checks the
+      # flag, so the budget is that cycle plus grace — a flat timeout would
+      # expire before a listener with a large health_check_ms had even one
+      # chance to observe the stop.
+      def stop_join_timeout
+        (@health_check_ms / 1000.0) + STOP_JOIN_GRACE_SECONDS
       end
 
       def safe_close
@@ -289,8 +319,10 @@ module Pgbus
         close_quietly(conn)
       end
 
-      # Close an unpublished PG::Connection (one that never made it into @conn,
-      # e.g. a half-built reconnect attempt where LISTEN raised). Best-effort.
+      # Close a PG::Connection we are done with — a half-built reconnect
+      # attempt that never made it into @conn, or the connection captured out
+      # of @conn during teardown. Always called on the listener thread, which
+      # owns the connection. Best-effort.
       def close_quietly(conn)
         conn&.close if conn.respond_to?(:close)
       rescue StandardError

data/lib/pgbus/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Pgbus
-  VERSION = "0.12.3"
+  VERSION = "0.12.4"
 end

data/lib/pgbus/web/streamer/listener.rb

@@ -15,7 +15,12 @@ module Pgbus
       #     only running `wait_for_notify` — all LISTEN/UNLISTEN SQL goes
       #     through a command queue that the listener thread drains between
       #     notifies
-      #   - #stop joins the thread cleanly
+      #   - #stop clears @running and joins; it never touches the connection
+      #   - the connection is SINGLE-OWNER: the listener thread is the only
+      #     thread that may exec, wait, or close on it, from construction
+      #     through teardown. PG::Connection is not thread-safe and #close is
+      #     PQfinish — freeing the PGconn under a concurrent libpq call is a
+      #     process-killing SEGV, not a rescuable PG::Error (issue #375)
       #
       # Health check: `wait_for_notify(timeout)` returns nil on timeout. When
       # it does, the listener runs `SELECT 1` as a TCP keepalive. If that
@@ -41,6 +46,10 @@ module Pgbus
 
         RECONNECT_BACKOFF_SECONDS = 0.5
 
+        # Grace added to one health-check cycle when #stop joins the listener
+        # thread. See #stop_join_timeout.
+        STOP_JOIN_GRACE_SECONDS = 5
+
         attr_reader :listening_to
 
         # @param connection_factory [#call] builds a FRESH PG connection on each
@@ -98,18 +107,16 @@ module Pgbus
 
           @running = false
           @commands << [:stop]
-          # Interrupt the blocking wait_for_notify by closing the PG
-          # connection. Without this, the listener thread would sit
-          # inside wait_for_notify until a NOTIFY arrived, which may
-          # never happen. Closing the socket raises PG::Error inside
-          # wait_for_notify; our rescue clause sees @running == false
-          # on the next iteration and exits cleanly.
-          begin
-            @conn.close if @conn.respond_to?(:close)
-          rescue StandardError
-            # best effort — connection may already be gone
-          end
-          @thread&.join(5)
+          # Deliberately does NOT touch @conn. PG::Connection is not
+          # thread-safe and #close is PQfinish: it frees the PGconn and its
+          # OpenSSL objects out from under whatever libpq call the listener
+          # thread is making on the same connection. That is a use-after-free —
+          # a process-killing SEGV, not a rescuable PG::Error (issue #375).
+          # The listener thread is the sole owner of @conn and closes it in
+          # run_loop's ensure; clearing @running above is the whole stop signal.
+          # It is observed within one wait_for_notify timeout (health_check_ms),
+          # which is what the join budget below is sized against.
+          @thread&.join(stop_join_timeout)
           @thread = nil
           self
         end
@@ -148,13 +155,18 @@ module Pgbus
 
             timeout_s = @health_check_ms / 1000.0
             begin
-              @conn.wait_for_notify(timeout_s) do |channel, _pid, payload|
+              got_notify = @conn.wait_for_notify(timeout_s) do |channel, _pid, payload|
                 handle_notify(channel, payload)
-              end || run_health_check
+              end
+              # Skip the keepalive when a stop landed during the wait: the loop
+              # is about to exit and close this connection anyway, so the
+              # round-trip would only add latency to shutdown.
+              run_health_check if !got_notify && @running
             rescue IOError => e
-              # #stop closes the PG connection to interrupt
-              # wait_for_notify, which raises IOError ("stream closed
-              # in another thread"). This is expected — exit cleanly.
+              # #stop no longer closes the connection to interrupt the wait
+              # (issue #375), so an IOError here is a genuine socket failure,
+              # not the expected shutdown signal. Reconnect unless we're
+              # stopping, in which case exit cleanly.
               break unless @running
 
               @logger.warn { "[Pgbus::Streamer::Listener] IO error (#{e.class}: #{e.message}) — reconnecting" }
@@ -167,7 +179,15 @@ module Pgbus
             end
           end
         ensure
-          safe_unlisten_all
+          # Bookkeeping only — no UNLISTEN round-trip. We close the connection
+          # on the next line and closing the session deregisters every LISTEN
+          # server-side, so the exec bought nothing while being the statement
+          # that raced #stop's PQfinish into a SEGV (issue #375).
+          @listening_to.clear
+          # The listener thread owns this connection, so the listener thread is
+          # the one that closes it — #stop only clears @running.
+          close_quietly(@conn)
+          @conn = nil
         end
 
         def drain_commands
@@ -204,6 +224,16 @@ module Pgbus
           (@health_check_ms / 1000.0) + 1.0
         end
 
+        # How long #stop waits for the listener thread to notice the cleared
+        # @running and finish teardown. The thread can be parked in
+        # wait_for_notify for one full health-check cycle before it re-checks
+        # the flag, so the budget is that cycle plus grace — a flat timeout
+        # would expire before a listener with a large health_check_ms had even
+        # one chance to observe the stop.
+        def stop_join_timeout
+          (@health_check_ms / 1000.0) + STOP_JOIN_GRACE_SECONDS
+        end
+
         def do_listen(queue_name)
           channel = channel_for(queue_name)
           return if @listening_to.include?(channel)
@@ -322,17 +352,6 @@ module Pgbus
           nil
         end
 
-        def safe_unlisten_all
-          @listening_to.each do |channel|
-            # @conn may be nil if #stop interrupted the reconnect loop between
-            # closing the old connection and publishing a new one.
-            @conn&.exec(%(UNLISTEN "#{channel}"))
-          rescue PG::Error
-            # connection may be dead; nothing we can do
-          end
-          @listening_to.clear
-        end
-
         def channel_for(queue_name)
           "#{CHANNEL_PREFIX}#{queue_name}#{CHANNEL_SUFFIX}"
         end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: pgbus
 version: !ruby/object:Gem::Version
-  version: 0.12.3
+  version: 0.12.4
 platform: ruby
 authors:
 - Mikael Henriksson