pgbus 0.16.5 → 0.16.6

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 4382e9b750a7e3538c2ba640aa476bb914cd1c1deba00fda429510ea1dcdc851
-  data.tar.gz: bc8bff2b3b08ccc6de7d67c677a8797be4d0a1b9f65019a9728d0531ad5a830a
+  metadata.gz: cddde0da7d83fc84f54404d10bcdbfccb06f25af74379b560ac4f9bb892f0041
+  data.tar.gz: 44eba7a5a2bbe87659f388c7fb10c5a7b7577a1f045bb8d48c527cfbf9266df5
 SHA512:
-  metadata.gz: 9630731c5e8b4ef6bb9db80e1e950ad6abee7748e49868fb613645b0d16469f477a6b0fe819427135a6bdf85589fb518e6204f80a3a20810939ed1c2038238a7
-  data.tar.gz: b4c5b23b4e4513b5447985ab96b1578a3a1931fda6b3b4700338b71d7d822a989cda6498670a31e7ef30848f69f4075b23a2f018764072f51b6e91ae258d8445
+  metadata.gz: 3bb680972106246ea3a16265730eb8b0567944e80323cbca8a9f68f7b7302b8afa2087cfee74e0fa99b695f038fd5e2104141971096ac8d76b236437ce147e60
+  data.tar.gz: cf822271a70b7ec8bcbbca0591838179678113c2b19849a87b4442cf5e0ccd3b2bac1d7a92e8820596c839ff1005c82d0c0ab45d47ddad2df1bae8094b0a9208

data/CHANGELOG.md

@@ -2,12 +2,17 @@
 
 ### Added
 
+- **`coalesce:` now works on the `Turbo::StreamsChannel` broadcast path.** `Pgbus::Streams::Stream#broadcast` has accepted `coalesce:` since #171, but the Turbo patch forwarded only four options (`durable:`, `exclude:`, `visible_to:`, `event:`), so nothing that broadcasts through `Turbo::StreamsChannel.broadcast_*_to` — every `Turbo::Broadcastable` model, and phlex-reactive's `Streamable.broadcast_to` — could debounce. A background job settling a 177-row collection therefore pushed 177 idempotent replaces of the same count badge at every peer. Any targeted helper now takes `coalesce:` (`@order.broadcast_replace_to :account, coalesce: 100`), as does a direct channel call (`Turbo::StreamsChannel.broadcast_replace_to(@account, target: "order-count", coalesce: true)`), with the same semantics as the direct API: opt-in, per `(stream, target)`, last-write-wins, `true` = the 50ms default window, an integer = the window in ms. Two things the option's plumbing needed beyond a fifth thread-local: the coalescing key, which dedupes on `(stream, target)` while `broadcast_stream_to` only ever sees the rendered `content:` — it is now resolved from the frame's own `target:`/`targets:` through the same `ActionView::RecordIdentifier.dom_id` conversion Turbo applies, so a record target keys on `order_7` rather than on a per-instance `#to_s`; and extraction of the pgbus options at the channel itself, since a direct `Turbo::StreamsChannel` call never passes through `Turbo::Broadcastable` and so previously leaked `durable: true` (and now `coalesce:`) into Turbo's renderer. `broadcast_refresh_to` and `broadcast_render_to` carry no target and so cannot coalesce — they raise the existing actionable "requires target:" error rather than guessing a key — and the `_later_to` variants remain out of scope, as the thread-local cannot follow into a job. The wire is byte-identical when `coalesce:` is absent. Closes #465.
+
 - **Concurrency is now visible: a section on the Locks page, three gauges, a home stat card and an MCP tool.** `limits_concurrency … on_conflict: :block` became durable in #460 — a parked job is never dropped and a key never runs past `to:` — but nothing showed it, so an operator whose per-record pipeline looked "stuck" had no way to see that the key's slot was held, how many jobs were parked behind it, or how long the oldest had waited, short of SQL. `pgbus_semaphores` and `pgbus_blocked_executions` appeared on no page, in no gauge, in no tool. One new read, `Web::DataSource#concurrency_stats`, now feeds four surfaces: the Locks page's **Concurrency** section (parked jobs, oldest wait, slots held, keys at limit, then a row per key with value / limit, lease state, parked count and oldest wait, auto-refreshing like the other dashboard panels); a **Parked jobs** card on the dashboard home; three unlabelled gauges on both `/pgbus/api/metrics` and the AppSignal probe (`pgbus_concurrency_blocked_executions`, `pgbus_concurrency_blocked_oldest_age_seconds`, `pgbus_concurrency_slots_held` — keys are per-record, so a per-key label would be an unbounded series); and a read-only `pgbus_concurrency` MCP tool returning the same data. The key rows come from a FULL OUTER JOIN, not a LEFT one: a key can have parked jobs and no semaphore (its holder died and the sweep removed the row) or a semaphore and nothing parked, and both shapes matter to an operator. Each row carries two guarded escape hatches. **Release** drops the semaphore and promotes through `Concurrency::BlockedExecution.promote_next`, so it takes each slot through the same guarded upsert an enqueue uses and can never reintroduce the over-admission #460 closed; when the lease is still fresh the confirm says so, because a job is then probably still running and releasing lets another start beside it. **Discard parked** drops a single key's parked jobs, naming the exact count in the confirm, and resolves the bookkeeping those jobs will never resolve themselves — a parked batch child is marked failed so its batch stops waiting and `on_failure` fires (#413), and an `:until_executed` uniqueness lock is released rather than orphaned (#423); each discard logs a warning naming the job and emits `pgbus.blocked_execution_discarded`. There is deliberately no cross-key "discard all parked": not losing work is the whole point of `:block`. Nothing is added to the enqueue or execute path. Refs #461.
 
 - **The batches list and a batch's progress panel now refresh themselves.** Batches was the only dashboard list whose `turbo-frame` was missing `data-auto-refresh`, and the batch detail page had no frame at all, so watching a batch drain meant reloading by hand. Both now poll on the existing `web_refresh_interval` (5s, pauses while you interact) like every other list — the batch progress panel is served by `GET /batches/:id?frame=progress`. This reuses the dashboard's turbo-frame polling rather than the SSE stack in `lib/pgbus/web/stream_app.rb`: that is the host-app-facing stream feature, and pointing it at the dashboard would open a PG `LISTEN` connection per viewer to move four integers.
 
 ### Fixed
 
+- **A dropped ActiveRecord socket no longer pages the host app for an event that was handled successfully.** EventBus consumers are long-lived threads holding a leased AR connection, and a pooler restart, an admin disconnect or a brief failover can kill that socket at any point. Rails reconnects most statements transparently, but `Relation#update_all` is marked `allow_retry: false` — and that is exactly the phase-2 claim stamp in `Handler#complete_claim!`, the one AR write that happens *after* `handle` has already returned. A drop there surfaced as `ActiveRecord::ConnectionFailed` ("PQconsumeInput() SSL error: unexpected eof while reading") on a message whose work was done, so the host app's exception tracker paged for a false failure and PGMQ redelivered the event for a re-run. `EventBus::StaleConnectionRetry` now reconnects this thread's lease (via `ConnectionPool#active_connection?`, the accessor that exists across the supported Rails range) and repeats the stamp once; a second drop still raises, leaving the message to VT redelivery. Only the leased connections are reconnected — `clear_all_connections!` would yank sockets out from under sibling consumers in the same process, turning one recoverable drop into many. The retryable patterns are deliberately *broader* than `Client::STALE_CONNECTION_PATTERNS` and for the opposite reason: that list excludes mid-flight drops because a half-committed enqueue would duplicate a message, whereas this only ever repeats an idempotent `SET completed_at = <now>`. Phase 1 (`claim_idempotency?`) is deliberately not wrapped — its INSERT may have committed before the socket died, and on a legacy schema the retry's empty `result.rows` would read as "another consumer owns this claim", turning a recoverable drop into a silently skipped event.
+
+
 - **The AppSignal probe's gauges and every MCP tool response were frozen at their first read.** `Web::DataSource` memoizes `queues_with_metrics` for the life of the instance — correct for the dashboard, which builds one instance per request, but the AppSignal `Probe::Runner` holds a single DataSource for the life of the process and `MCP::Server.build` injects one into the server context, so both replayed the first read forever: queue depth in AppSignal never moved after the first minute, and `pgbus_queues` returned the same snapshot for every call in a session. `DataSource#reset_cache!` drops the memos, and both long-lived consumers call it — the probe once per iteration, `MCP::BaseTool.data_source_from` once per tool call. Found while reviewing the concurrency aggregates added in the same release, which memoize the same way.
 
 - **`limits_concurrency … on_conflict: :block` now constrains a job without ever losing it, and never runs more than `to:` at once.** The same bug class solid_queue closed in rails/solid_queue#712 and #761, checked against pgbus's implementation. Four defects, each with a regression spec:

data/lib/pgbus/event_bus/handler.rb

@@ -158,12 +158,26 @@ module Pgbus
       # already cached at claim time). If this write fails, process!'s rescue
       # re-raises, the consumer leaves the message for VT redelivery, and the
       # still-pending claim re-runs — at-least-once, never a silent drop.
+      #
+      # Wrapped in StaleConnectionRetry because this is the one AR write that
+      # happens AFTER handle() has already succeeded: a socket dropped here
+      # costs the host app a paging exception for work that was in fact done.
+      # The stamp is an idempotent `SET completed_at = <now>`, so repeating a
+      # statement that may already have committed is safe.
+      #
+      # Phase 1 (claim_idempotency?) is deliberately NOT wrapped. Its INSERT
+      # may have committed before the socket died, and on a legacy schema the
+      # retry's empty `result.rows` would read as "someone else owns this
+      # claim" and return false — turning a recoverable drop into a silently
+      # skipped event. VT redelivery is the correct recovery there.
       def complete_claim!(event_id)
         return unless ProcessedEvent.completion_column?
 
-        ProcessedEvent
-          .where(event_id: event_id, handler_class: self.class.name)
-          .update_all(completed_at: Time.now.utc)
+        StaleConnectionRetry.call(context: self.class.name) do
+          ProcessedEvent
+            .where(event_id: event_id, handler_class: self.class.name)
+            .update_all(completed_at: Time.now.utc)
+        end
         self.class.dedup_cache.mark!(dedup_key(event_id))
       end
 

data/lib/pgbus/event_bus/stale_connection_retry.rb

@@ -0,0 +1,85 @@
+# frozen_string_literal: true
+
+module Pgbus
+  module EventBus
+    # Reconnect + one retry for an ActiveRecord socket that died underneath a
+    # long-lived consumer thread.
+    #
+    # A consumer holds its leased connection for the life of the message, and a
+    # pooler restart (PgBouncer `server_idle_timeout`), an admin disconnect or a
+    # brief failover can kill that socket at any point. Rails reconnects most
+    # statements transparently, but `Relation#update_all` is marked
+    # `allow_retry: false`, so the drop propagates — and the only call that
+    # matters here, the phase-2 claim stamp, is exactly an `update_all`. The
+    # host app saw a paging exception on a message whose handler had in fact
+    # succeeded.
+    #
+    # Deliberately narrower than `Client::STALE_CONNECTION_PATTERNS`, and for
+    # the opposite reason. That list excludes mid-flight drops because a
+    # half-committed *enqueue* would duplicate a message on retry. This one
+    # only ever wraps an idempotent `UPDATE ... SET completed_at = <now>`, so a
+    # statement that may already have committed is safe to repeat — which is
+    # what makes the mid-flight shapes retryable here and not there.
+    module StaleConnectionRetry
+      # The socket dying under a statement, not a server refusing us. A refused
+      # or timed-out connection is an outage: retrying it in-process buys
+      # nothing and hides the outage behind a doubled statement timeout.
+      #
+      # "ssl syscall error" is libpq's wording when the peer vanished without a
+      # TLS close_notify — the same drop as "unexpected eof while reading",
+      # reported from the syscall layer instead. It is in
+      # Client::STALE_CONNECTION_PATTERNS for the same reason.
+      TRANSIENT_DROP = /
+        PQconsumeInput|
+        server\ closed\ the\ connection\ unexpectedly|
+        unexpected\ eof\ while\ reading|
+        ssl\ syscall\ error
+      /ix
+
+      module_function
+
+      # Runs the block, and on a transient drop reconnects this thread's lease
+      # and runs it exactly once more. A second drop raises: the socket is not
+      # coming back inside this attempt, and PGMQ's visibility timeout is the
+      # right recovery — it redelivers to a healthy consumer.
+      #
+      # context — an identifier for the log line (the handler class name).
+      def call(context: nil, &block)
+        block.call
+      rescue ActiveRecord::ConnectionFailed => e
+        raise unless transient_drop?(e)
+
+        reconnect_leased!
+        Pgbus.logger.warn do
+          "[Pgbus::EventBus] Retrying after stale ActiveRecord connection drop" \
+            "#{" (#{context})" if context}: #{e.message}"
+        end
+        block.call
+      end
+
+      def transient_drop?(error)
+        return false unless error.is_a?(ActiveRecord::ConnectionFailed)
+
+        [error.message, error.cause&.message].compact.any? { |message| TRANSIENT_DROP.match?(message) }
+      end
+
+      # Only the connections this thread has actually leased.
+      # `clear_all_connections!` would yank sockets out from under sibling
+      # consumers sharing the process, turning one recoverable drop into many.
+      # `active_connection?` returns the lease or nil, and is the accessor this
+      # gem uses everywhere (Streams#current_open_transaction documents why we
+      # never reach for `ActiveRecord::Base.connection`). Its `active_connection`
+      # alias is `:nodoc:` and does not exist at all before Rails 7.2 — below
+      # our floor, so calling it would raise NoMethodError on the one path that
+      # exists to recover from an error.
+      def reconnect_leased!
+        ActiveRecord::Base.connection_handler.each_connection_pool(:all) do |pool|
+          connection = pool.active_connection?
+          next unless connection
+
+          connection.reconnect!
+        end
+      end
+    end
+  end
+end

data/lib/pgbus/streams/broadcast_opts.rb

@@ -0,0 +1,70 @@
+# frozen_string_literal: true
+
+module Pgbus
+  module Streams
+    # The thread-local channel that carries pgbus-specific broadcast options
+    # from a `Turbo::Broadcastable` / `Turbo::StreamsChannel` call site down to
+    # `TurboBroadcastable#broadcast_stream_to`, which is the only place with a
+    # `Pgbus::Streams::Stream` to hand them to.
+    #
+    # turbo-rails' broadcast helpers funnel everything through
+    # `broadcast_stream_to(*streamables, content:)` — a signature with nowhere
+    # to put `durable:`, `exclude:`, `visible_to:`, `event:` or `coalesce:`.
+    # Rather than fork turbo's whole helper surface, the patches pull those
+    # kwargs out before calling `super` and stash them here for the duration of
+    # the broadcast.
+    #
+    # Both patches (`BroadcastableOverride` on the model concern and
+    # `TurboBroadcastable` on the channel) share this module so there is one
+    # definition of which keys exist and how they are saved/restored.
+    module BroadcastOpts
+      # Options a caller may pass to any Turbo broadcast helper. They are
+      # deleted from the kwargs so they never reach turbo-rails' renderer.
+      KEYS = %i[durable exclude visible_to event coalesce].freeze
+
+      # `coalesce_target` is not a caller-facing kwarg — it is derived from the
+      # broadcast's own `target:`/`targets:` by the channel patch, because
+      # coalescing keys on `(stream, target)` and `broadcast_stream_to` never
+      # sees the target otherwise.
+      THREAD_LOCALS = {
+        durable: :pgbus_broadcast_durable,
+        exclude: :pgbus_broadcast_exclude,
+        visible_to: :pgbus_broadcast_visible_to,
+        event: :pgbus_broadcast_event,
+        coalesce: :pgbus_broadcast_coalesce,
+        coalesce_target: :pgbus_broadcast_coalesce_target
+      }.freeze
+
+      # Removes the pgbus options from `kwargs` (mutating it) and returns them.
+      # Keys the caller did not pass are absent from the result, so an outer
+      # `BroadcastOpts.with` block's values survive an inner broadcast that
+      # doesn't set them.
+      def self.extract!(kwargs)
+        KEYS.each_with_object({}) do |key, opts|
+          opts[key] = kwargs.delete(key) if kwargs.key?(key)
+        end
+      end
+
+      # Sets the thread-locals for the given options for the duration of the
+      # block, restoring the previous values afterwards (nesting-safe), even on
+      # error. Only the keys passed are touched.
+      def self.with(**opts)
+        previous = {}
+
+        opts.each do |key, value|
+          tl_key = THREAD_LOCALS.fetch(key)
+          previous[tl_key] = Thread.current[tl_key]
+          Thread.current[tl_key] = value
+        end
+
+        yield
+      ensure
+        previous.each { |tl_key, value| Thread.current[tl_key] = value }
+      end
+
+      def self.[](key)
+        Thread.current[THREAD_LOCALS.fetch(key)]
+      end
+    end
+  end
+end

data/lib/pgbus/streams/broadcastable_override.rb

@@ -40,7 +40,7 @@ module Pgbus
       # don't understand. We pull them out of kwargs (so they never reach
       # turbo-rails' renderer) and thread them to broadcast_stream_to via
       # thread-locals, mirroring the original durable: shim.
-      PGBUS_BROADCAST_OPTS = %i[durable exclude visible_to event].freeze
+      PGBUS_BROADCAST_OPTS = BroadcastOpts::KEYS
 
       BROADCAST_METHODS.each do |method_name|
         define_method(method_name) do |*streamables, **kwargs|
@@ -128,32 +128,19 @@ module Pgbus
       private
 
       def extract_pgbus_broadcast_opts(kwargs)
-        PGBUS_BROADCAST_OPTS.each_with_object({}) do |key, opts|
-          opts[key] = kwargs.delete(key) if kwargs.key?(key)
-        end
+        BroadcastOpts.extract!(kwargs)
       end
 
       # Set the pgbus broadcast thread-locals for the duration of the block,
       # restoring previous values afterwards (nested/concurrent-safe). Only
       # keys actually passed are touched, so unrelated outer broadcasts keep
       # their values.
-      def with_pgbus_broadcast_opts(durable: :__unset__, exclude: :__unset__, visible_to: :__unset__, event: :__unset__)
-        previous = {}
-        set = lambda do |tl_key, value|
-          next if value == :__unset__
-
-          previous[tl_key] = Thread.current[tl_key]
-          Thread.current[tl_key] = value
-        end
-
-        set.call(:pgbus_broadcast_durable, durable)
-        set.call(:pgbus_broadcast_exclude, exclude)
-        set.call(:pgbus_broadcast_visible_to, visible_to)
-        set.call(:pgbus_broadcast_event, event)
+      def with_pgbus_broadcast_opts(durable: :__unset__, exclude: :__unset__, visible_to: :__unset__,
+                                    event: :__unset__, coalesce: :__unset__, &)
+        opts = { durable: durable, exclude: exclude, visible_to: visible_to, event: event, coalesce: coalesce }
+        opts.reject! { |_key, value| value == :__unset__ }
 
-        yield
-      ensure
-        previous.each { |tl_key, value| Thread.current[tl_key] = value }
+        BroadcastOpts.with(**opts, &)
       end
     end
   end

data/lib/pgbus/streams/turbo_broadcastable.rb

@@ -34,9 +34,57 @@ module Pgbus
     # long as `Turbo.signed_stream_verifier_key` is set, which the Rails
     # app is already responsible for).
     module TurboBroadcastable
+      # Every targeted broadcast helper (`broadcast_replace_to`,
+      # `broadcast_append_to`, `broadcast_remove_to`, …) funnels through
+      # `broadcast_action_to`, so this is the one place that sees both the
+      # caller's kwargs and the `target:`/`targets:` the frame will carry.
+      #
+      # It does two things the model-level `BroadcastableOverride` can't:
+      #
+      # 1. Extracts the pgbus options from a *direct* channel call —
+      #    `Turbo::StreamsChannel.broadcast_replace_to(..., coalesce: true)` —
+      #    which never passes through `Turbo::Broadcastable` at all (this is
+      #    the path phlex-reactive's `Streamable.broadcast_to` takes).
+      # 2. Records the coalescing key. `coalesce:` dedupes on
+      #    `(stream, target)`, but `broadcast_stream_to` only receives the
+      #    rendered `content:` — the target is gone by then. We resolve it the
+      #    same way turbo will (`convert_to_turbo_stream_dom_id`), so a record
+      #    target keys on its stable dom_id rather than its object id.
+      #
+      # Resolving the key is skipped entirely unless coalescing was actually
+      # requested, so the uncoalesced path stays byte-identical and free.
+      def broadcast_action_to(*streamables, action:, target: nil, targets: nil, **rendering)
+        opts = BroadcastOpts.extract!(rendering)
+
+        coalesce = opts.key?(:coalesce) ? opts[:coalesce] : BroadcastOpts[:coalesce]
+        opts[:coalesce_target] = pgbus_coalesce_target(target, targets) if coalesce
+
+        BroadcastOpts.with(**opts) do
+          super(*streamables, action: action, target: target, targets: targets, **rendering)
+        end
+      end
+
+      # The two helpers that don't pass through `broadcast_action_to`. They
+      # carry no target, so they can't coalesce (`Stream#broadcast` raises an
+      # actionable error, same as `Pgbus.stream(x).broadcast(coalesce:)`
+      # without one) — but they still need their pgbus options pulled out of
+      # the kwargs, or a direct channel call leaks `durable: true` into
+      # turbo's renderer and it ends up as an HTML attribute.
+      def broadcast_refresh_to(*streamables, **attributes)
+        BroadcastOpts.with(**BroadcastOpts.extract!(attributes)) do
+          super(*streamables, **attributes)
+        end
+      end
+
+      def broadcast_render_to(*streamables, **rendering)
+        BroadcastOpts.with(**BroadcastOpts.extract!(rendering)) do
+          super(*streamables, **rendering)
+        end
+      end
+
       def broadcast_stream_to(*streamables, content:)
         name = stream_name_from(streamables)
-        override = Thread.current[:pgbus_broadcast_durable]
+        override = BroadcastOpts[:durable]
         # When no explicit thread-local override is present, let the config
         # resolver decide: it checks `streams_durable_patterns` first (exact
         # string or regex match), then falls back to
@@ -46,11 +94,24 @@ module Pgbus
         durable = override.nil? ? Pgbus.configuration.stream_durable?(name) : override
         Pgbus.stream(name, durable: durable).broadcast(
           content,
-          exclude: Thread.current[:pgbus_broadcast_exclude],
-          visible_to: Thread.current[:pgbus_broadcast_visible_to],
-          event: Thread.current[:pgbus_broadcast_event]
+          exclude: BroadcastOpts[:exclude],
+          visible_to: BroadcastOpts[:visible_to],
+          event: BroadcastOpts[:event],
+          coalesce: BroadcastOpts[:coalesce],
+          target: BroadcastOpts[:coalesce_target]
         )
       end
+
+      private
+
+      # The coalescing key, resolved exactly as turbo resolves the rendered
+      # `target=`/`targets=` attribute. `targets:` (a CSS selector) is the
+      # fallback turbo itself uses when `target:` is absent; we drop the `#`
+      # selector prefix because the key is never rendered — it only has to be
+      # stable and distinct.
+      def pgbus_coalesce_target(target, targets)
+        convert_to_turbo_stream_dom_id(target) || convert_to_turbo_stream_dom_id(targets)
+      end
     end
 
     # Apply the patch to Turbo::StreamsChannel's singleton class. Idempotent:

data/lib/pgbus/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Pgbus
-  VERSION = "0.16.5"
+  VERSION = "0.16.6"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: pgbus
 version: !ruby/object:Gem::Version
-  version: 0.16.5
+  version: 0.16.6
 platform: ruby
 authors:
 - Mikael Henriksson
@@ -294,6 +294,7 @@ files:
 - lib/pgbus/event_bus/handler.rb
 - lib/pgbus/event_bus/publisher.rb
 - lib/pgbus/event_bus/registry.rb
+- lib/pgbus/event_bus/stale_connection_retry.rb
 - lib/pgbus/event_bus/subscriber.rb
 - lib/pgbus/execution_pools.rb
 - lib/pgbus/execution_pools/async_pool.rb
@@ -375,6 +376,7 @@ files:
 - lib/pgbus/serializer.rb
 - lib/pgbus/stat_buffer.rb
 - lib/pgbus/streams.rb
+- lib/pgbus/streams/broadcast_opts.rb
 - lib/pgbus/streams/broadcastable_override.rb
 - lib/pgbus/streams/coalescer.rb
 - lib/pgbus/streams/cursor.rb