solid_queue-listen_notify 0.5.1 → 0.5.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 02e14b868585c5b513fd8c1e16222d75af15f95fb1a8775171505c4825b971a9
-  data.tar.gz: 16868e921aff4dd98118f4617da5b3007b6ad346e001231d34a8cb00f611d2de
+  metadata.gz: 8e7cb2a646ad717d865cea7a0806f9e1606bcc8a7b1c75aefdcf0211a280b4dc
+  data.tar.gz: c973ecd38ed1cc6aac33ebf297ba5d7c8adbf9d6379973ebc9685cb4e131e6a6
 SHA512:
-  metadata.gz: df252638c88f888e052bde5e4ac262b28cc83830a095e445a24937d816758122a8373ca474c3ae28541b0c152864034b0eeee32b09525089b9d6da99b46ef221
-  data.tar.gz: 468f5eed99503c2690e37f107d16afed17ef2c5df279f8f411d2d2e987d5363fe80467b81beb867e7d5304f49ae51d97de0348e7533c667a65ba3e9f37c03dfd
+  metadata.gz: b1970e7d67ad65596a81e9edd4af43d8d38775296e9b97b4372c8eafb27a02206cd37ea86a82d0d5753fff7d12551e55b25f32b5343db914c16d17e8c14c498d
+  data.tar.gz: cdca29cbc3ad90d3337e8bd942b553612cd61951c75723fb10ff9fc3024acc6aec19bbc558b1e1b972fd493b29f2ce952a2153fd1e9d651491500ca8712e0b50

data/CHANGELOG.md

@@ -7,6 +7,16 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.5.2] - 2026-07-28
+
+### Fixed
+
+- A raising `ActiveSupport::Notifications` subscriber can no longer strand a worker at the raised polling interval mid-registration, or run between a notification and the worker wake-ups it triggers.
+
+### Changed
+
+- Internal: the registry is now the single owner of per-worker state (queue snapshot and replaced polling interval); the `notify` event's duration no longer includes the dispatch itself (payload unchanged).
+
 ## [0.5.1] - 2026-07-28
 
 ### Fixed
@@ -19,7 +29,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 
 
-[Unreleased]: https://github.com/cmer/solid_queue-listen_notify/compare/v0.5.1...HEAD
+[Unreleased]: https://github.com/cmer/solid_queue-listen_notify/compare/v0.5.2...HEAD
+[0.5.2]: https://github.com/cmer/solid_queue-listen_notify/compare/v0.5.1...v0.5.2
 [0.5.1]: https://github.com/cmer/solid_queue-listen_notify/compare/v0.5.0...v0.5.1
 [0.5.0]: https://github.com/cmer/solid_queue-listen_notify/compare/v0.1.0...v0.5.0
 [0.1.0]: https://github.com/cmer/solid_queue-listen_notify/releases/tag/v0.1.0

data/CLAUDE.md

@@ -36,7 +36,7 @@ bundle exec appraisal generate                                     # after editi
 
 `lib/solid_queue/listen_notify.rb` is the facade: config (`mattr_accessor`, mirroring solid_queue's style; copied from `config.solid_queue_listen_notify` by the Railtie), `register`/`deregister` (called from the lifecycle hooks; fully rescued — this gem must never make a worker look broken), the per-pid memoized `operational?`, and `after_fork`.
 
-Flow at worker boot: `register(worker)` → `operational?` runs **Preflight** once per process → if operational, raise the worker's `polling_interval` to `fallback_polling_interval` (only ever raise, record the original) → `Registry.instance.register(worker)` starts the **Listener** on first registration.
+Flow at worker boot: `register(worker)` → `operational?` runs **Preflight** once per process → if operational, raise the worker's `polling_interval` to `fallback_polling_interval` (only ever raise) → `Registry.instance.register(worker, restore_interval:)` records the replaced interval with the worker and starts the **Listener** on first registration. The registry is the single owner of per-worker state (queue snapshot + replaced interval); on a fatal listener death it hands the `[worker, original_interval]` pairs back to the facade, which restores and wakes them.
 
 - **`Preflight`** (`preflight.rb`) — decides operational-or-not; never raises; each failure prints an actionable WARN/ERROR banner (loud failure is a product requirement, not a nicety). Checks in order: enabled → channel 1–63 bytes → adapter is PG → trigger installed (auto-installed by default when missing) → trigger function body contains the exact `pg_notify('<channel>', ...)` call → **cross-connection self-test**: LISTEN on the listener's connection path, NOTIFY from a pooled queue-DB connection. Cross-connection is what detects both PgBouncer transaction pooling and a `listen_database` pointing at the wrong database.
 - **`Registry`** (`registry.rb`) — per-process singleton multiplexing one listener to N workers (async mode runs N workers as threads in one process). Locking discipline is documented in its header comment and must be preserved: the mutex is never held across `wake_up`, `listener.stop`, or anything blocking; listeners are detached under the lock and stopped outside it. Cleanup is crash-tolerant, never hook-dependent (`on_worker_stop` is skipped on SIGQUIT `exit!`): dead workers are reaped on register and on the keepalive tick.

data/lib/solid_queue/listen_notify/connection_provider.rb

@@ -17,6 +17,27 @@ module SolidQueue
     # Everything ActiveRecord-shaped in here is resolved inside method bodies:
     # requiring this file must not require ActiveRecord.
     class ConnectionProvider
+      # Best-effort teardown for a connection this provider handed out: nobody
+      # else can close it (it was removed from its pool), and it may already be
+      # broken, so both statements are attempted and both failures swallowed.
+      def self.release(connection, channel:)
+        return if connection.nil?
+
+        begin
+          connection.execute("UNLISTEN #{connection.quote_column_name(channel)}")
+        rescue StandardError
+          nil
+        end
+
+        begin
+          connection.disconnect!
+        rescue StandardError
+          nil
+        end
+
+        nil
+      end
+
       def initialize
         @mutex = Mutex.new
         @pools = {}
@@ -30,10 +51,6 @@ module SolidQueue
         end
       end
 
-      def to_proc
-        method(:call).to_proc
-      end
-
       # Drops the memoized isolated pools. The connections handed out so far are
       # not affected: they were removed from those pools and belong to whoever
       # asked for them.

data/lib/solid_queue/listen_notify/listener.rb

@@ -1,5 +1,7 @@
 # frozen_string_literal: true
 
+require_relative "connection_provider"
+
 module SolidQueue
   module ListenNotify
     # One background thread owning one dedicated Postgres connection: LISTENs on
@@ -220,24 +222,18 @@ module SolidQueue
           end
         end
 
+        # Dispatches first, instruments after: the ordering is what guarantees
+        # that no notifications subscriber — however it subscribed — can stand
+        # between a notification and the wake-ups it owes.
         def handle_notification(payload)
           @consecutive_failures = 0
           queue_name = payload.to_s
-          dispatched = false
 
-          instrument(:notify, queue_name: queue_name) do |event|
-            dispatched = true
-            counts = registry.dispatch(queue_name)
-            event.merge!(counts) if counts.is_a?(Hash)
-          end
+          counts = registry.dispatch(queue_name)
 
-          # ActiveSupport runs a plain `subscribe` block AFTER the instrumented
-          # block, so a subscriber that raises has not stopped the fan-out above
-          # from happening. The rarer `subscribe(name, object_responding_to_start)`
-          # form does run before it, though, and a notification dropped because
-          # somebody's metrics object raised would be a job left waiting for the
-          # next poll.
-          registry.dispatch(queue_name) unless dispatched
+          event_payload = { queue_name: queue_name }
+          event_payload.merge!(counts) if counts.is_a?(Hash)
+          instrument(:notify, **event_payload)
         end
 
         def tick_keepalive
@@ -319,24 +315,10 @@ module SolidQueue
           nil
         end
 
-        # Best effort on both statements: this runs on connections we already
-        # know may be broken.
         def safe_disconnect
           connection = @connection
           @connection = nil
-          return if connection.nil?
-
-          begin
-            connection.execute("UNLISTEN #{connection.quote_column_name(channel)}")
-          rescue StandardError
-            nil
-          end
-
-          begin
-            connection.disconnect!
-          rescue StandardError
-            nil
-          end
+          ConnectionProvider.release(connection, channel: channel)
         end
 
         # Sleeps in wait_timeout-sized slices so that #stop is never delayed by

data/lib/solid_queue/listen_notify/preflight.rb

@@ -1,6 +1,7 @@
 # frozen_string_literal: true
 
 require_relative "trigger_installer"
+require_relative "connection_provider"
 
 module SolidQueue
   module ListenNotify
@@ -176,7 +177,7 @@ module SolidQueue
             end
           ensure
             # Nobody else can: this connection was removed from its pool.
-            disconnect(connection)
+            ConnectionProvider.release(connection, channel: channel)
           end
         end
 
@@ -190,20 +191,6 @@ module SolidQueue
           end
         end
 
-        def disconnect(connection)
-          begin
-            connection.execute("UNLISTEN #{connection.quote_column_name(channel)}")
-          rescue StandardError
-            nil
-          end
-
-          begin
-            connection.disconnect!
-          rescue StandardError
-            nil
-          end
-        end
-
         # Two statements, one connection: a transaction-mode pooler will very
         # likely serve them from two different backends. Only ever used to
         # enrich the self-test failure, never to decide anything.

data/lib/solid_queue/listen_notify/registry.rb

@@ -20,7 +20,15 @@ module SolidQueue
     #   is what guarantees "at most one listener per non-empty transition" when
     #   several worker threads register concurrently.
     # * #dispatch runs on the listener thread and performs zero database work.
+    #
+    # The registry is the single owner of per-worker state: the queue snapshot
+    # used to route notifications, and the polling interval the module replaced
+    # at registration time — kept so that a listener which dies for good can
+    # hand every worker back to be put where it was found. One map, one lock,
+    # one fork-reset path.
     class Registry
+      # Everything this process knows about a registered worker.
+      Entry = Struct.new(:queues, :restore_interval)
       class << self
         # A class-level ivar rather than a constant, because the fork hook has to
         # be able to REPLACE it: fork() copies only the calling thread, and this
@@ -70,17 +78,23 @@ module SolidQueue
       def initialize(listener_factory: nil)
         @listener_factory = listener_factory || method(:build_default_listener)
         @mutex = Mutex.new
-        @workers = {}
+        @workers = empty_workers
         @listener = nil
         @pid = ::Process.pid
       end
 
-      def register(worker)
+      # `restore_interval` is the polling interval the module replaced when it
+      # raised this worker's — nil when nothing was changed. It is recorded here,
+      # with the worker, so that the crash path has one place to look.
+      def register(worker, restore_interval: nil)
         guard_fork!
 
         @mutex.synchronize do
           sweep_dead_locked
-          @workers[worker] ||= snapshot_queues(worker)
+          entry = (@workers[worker] ||= Entry.new(snapshot_queues(worker), nil))
+          # ||=, not =: a re-registration must never overwrite the interval the
+          # worker originally had with one this gem already raised.
+          entry.restore_interval ||= restore_interval
           start_listener_locked
         end
 
@@ -118,7 +132,13 @@ module SolidQueue
           next false unless @listener.equal?(listener)
 
           @listener = nil
-          stranded = @workers.keys
+          # Hand each interval over exactly once: a second crash after a revival
+          # must not "restore" workers that are already back on their own.
+          stranded = @workers.map do |worker, entry|
+            pair = [ worker, entry.restore_interval ]
+            entry.restore_interval = nil
+            pair
+          end
           true
         end
 
@@ -141,8 +161,8 @@ module SolidQueue
         skipped_saturated = 0
         unreachable = nil
 
-        pairs.each do |worker, queues|
-          next unless QueueMatcher.matches?(queues, queue_name)
+        pairs.each do |worker, entry|
+          next unless QueueMatcher.matches?(entry.queues, queue_name)
 
           if !wake_saturated && pool_saturated?(worker)
             skipped_saturated += 1
@@ -202,7 +222,7 @@ module SolidQueue
         # single-threaded here: replace it instead of trying to take it.
         @mutex = Mutex.new
         @listener = nil
-        @workers = {}
+        @workers = empty_workers
         @pid = ::Process.pid
 
         ListenNotify.instrument(:fork_detected, parent_pid: parent_pid, pid: @pid)
@@ -217,7 +237,7 @@ module SolidQueue
         listener = @mutex.synchronize do
           current = @listener
           @listener = nil
-          @workers = {}
+          @workers = empty_workers
           current
         end
 
@@ -250,7 +270,14 @@ module SolidQueue
 
         # Caller holds @mutex.
         def sweep_dead_locked
-          @workers.delete_if { |worker, _queues| !worker_alive?(worker) }
+          @workers.delete_if { |worker, _entry| !worker_alive?(worker) }
+        end
+
+        # Keyed by identity: a Solid Queue worker defines no #hash or #eql?, and
+        # relying on the default object identity implicitly would be relying on
+        # something nobody wrote down.
+        def empty_workers
+          {}.compare_by_identity
         end
 
         def reap(workers)

data/lib/solid_queue/listen_notify/version.rb

@@ -2,6 +2,6 @@
 
 module SolidQueue
   module ListenNotify
-    VERSION = "0.5.1"
+    VERSION = "0.5.2"
   end
 end

data/lib/solid_queue/listen_notify.rb

@@ -60,12 +60,6 @@ module SolidQueue
     @@preflight_result = nil
     @@fallback_logger = nil
 
-    # worker => the polling interval it had before we raised it. Kept so that a
-    # listener which dies for good can put every worker back where it found it,
-    # rather than leaving it polling every 10 seconds with nothing to wake it.
-    @@overrides_lock = Mutex.new
-    @@overrides = {}.compare_by_identity
-
     # `SOLID_QUEUE_LISTEN_NOTIFY_ENABLED=false` is documented as a kill switch
     # that needs no deploy, and an application that sets `enabled` explicitly —
     # to either value — would otherwise overwrite the mattr default and take
@@ -107,8 +101,15 @@ module SolidQueue
       return nil unless operational?
 
       restore_to = raise_polling_interval(worker)
-      Registry.instance.register(worker)
-      remember_override(worker, restore_to)
+      Registry.instance.register(worker, restore_interval: restore_to)
+
+      # Instrumented only once the registry has recorded the override, so that
+      # a subscriber raising here finds the bookkeeping already consistent — it
+      # can never strand a raised interval that nobody remembers.
+      if restore_to
+        instrument :override_polling_interval,
+          worker_name: worker_name(worker), from: restore_to, to: fallback_polling_interval.to_f
+      end
       nil
     rescue StandardError, ScriptError => e
       # A worker left polling every 10 seconds with nothing to wake it is the one
@@ -121,12 +122,11 @@ module SolidQueue
     # on_worker_stop is not guaranteed to run at all (a supervisor that exceeds
     # its shutdown timeout exits without callbacks), so this is a convenience,
     # never the thing cleanup depends on.
+    #
+    # Never instantiate the registry just to deregister: this hook also runs in
+    # processes where nothing ever registered, and building a registry there
+    # would be building one that only exists to be empty.
     def deregister(worker)
-      forget_override(worker)
-
-      # Never instantiate the registry just to deregister: this hook also runs in
-      # processes where nothing ever registered, and building a registry there
-      # would be building one that only exists to be empty.
       return nil unless Registry.instantiated?
 
       Registry.instance.deregister(worker)
@@ -136,7 +136,9 @@ module SolidQueue
     end
 
     # Called by the registry when the listener thread died for good — a fatal,
-    # non-connection error, which no reconnect will fix.
+    # non-connection error, which no reconnect will fix. `stranded` is the
+    # registry's list of [worker, original_interval] pairs; the interval is nil
+    # for workers whose interval was never raised.
     #
     # The gem raised these workers' polling intervals on the strength of a
     # promise it can no longer keep, and a worker left polling every 10 seconds
@@ -144,8 +146,8 @@ module SolidQueue
     # gem. So the promise is withdrawn: every interval we raised goes back, every
     # worker is woken so it re-reads it rather than finishing its current sleep,
     # and the whole thing is said out loud.
-    def listener_crashed(error, workers)
-      restored = Array(workers).count { |worker| restore_override(worker) }
+    def listener_crashed(error, stranded)
+      restored = Array(stranded).count { |worker, interval| restore_override(worker, interval) }
       announce_listener_death(error, restored)
       nil
     rescue StandardError, ScriptError => e
@@ -184,18 +186,16 @@ module SolidQueue
     # belongs to the preflight lock: `operational?` holds it across connecting to
     # Postgres and a self-test that waits up to two seconds, and a child that
     # inherited it locked would hang the first worker to register — permanently,
-    # inside a lifecycle hook, with no error and no output. Replacing three
-    # mutexes in a process that is still single-threaded costs nothing, and it
-    # means the gem is not relying on an interpreter implementation detail to
-    # avoid its worst failure mode.
+    # inside a lifecycle hook, with no error and no output. Replacing a mutex in
+    # a process that is still single-threaded costs nothing, and it means the
+    # gem is not relying on an interpreter implementation detail to avoid its
+    # worst failure mode.
     #
-    # Order matters: locks first, then the guards that use them. The preflight
+    # Order matters: the lock first, then the guards that use it. The preflight
     # VERDICT needs no reset (it is stamped with the pid that reached it, so a
     # child recomputes its own).
     def after_fork
       @@preflight_lock = Mutex.new
-      @@overrides_lock = Mutex.new
-      @@overrides = {}.compare_by_identity
 
       connection_provider.forget_pools! if connection_provider.respond_to?(:forget_pools!)
       Registry.after_fork!
@@ -228,7 +228,6 @@ module SolidQueue
     # preflight to run again after reconfiguring.
     def reset!
       @@preflight_lock.synchronize { @@preflight_result = nil }
-      @@overrides_lock.synchronize { @@overrides = {}.compare_by_identity }
       connection_provider.reset! if connection_provider.respond_to?(:reset!)
       nil
     end
@@ -292,6 +291,8 @@ module SolidQueue
       # all only because operational? proved notifications arrive.
       #
       # Returns the interval that was replaced, or nil when nothing changed.
+      # Deliberately does NOT instrument: the caller does, once the registry
+      # has recorded what happened here.
       def raise_polling_interval(worker)
         target = fallback_polling_interval&.to_f
         return nil if target.nil?
@@ -301,7 +302,6 @@ module SolidQueue
         return nil if current.nil? || target <= current.to_f
 
         worker.polling_interval = target
-        instrument :override_polling_interval, worker_name: worker_name(worker), from: current, to: target
         current
       end
 
@@ -313,27 +313,10 @@ module SolidQueue
         nil
       end
 
-      # Overridden intervals ---------------------------------------------------
-      #
-      # Keyed by identity: a Solid Queue worker defines no #hash or #eql?, but
-      # relying on the default object identity implicitly would be relying on
-      # something nobody wrote down.
-
-      def remember_override(worker, previous)
-        return if previous.nil?
-
-        @@overrides_lock.synchronize { @@overrides[worker] = previous }
-      end
-
-      def forget_override(worker)
-        @@overrides_lock.synchronize { @@overrides.delete(worker) }
-      end
-
-      # Puts one worker back on its own interval and wakes it, so that it
-      # re-reads the interval now rather than at the end of the sleep it is
-      # already in. Returns whether anything was actually restored.
-      def restore_override(worker)
-        previous = @@overrides_lock.synchronize { @@overrides.delete(worker) }
+      # Puts one worker back on the interval the registry remembered for it and
+      # wakes it, so that it re-reads the interval now rather than at the end of
+      # the sleep it is already in. Returns whether anything was restored.
+      def restore_override(worker, previous)
         return false if previous.nil?
 
         current = current_polling_interval(worker)

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: solid_queue-listen_notify
 version: !ruby/object:Gem::Version
-  version: 0.5.1
+  version: 0.5.2
 platform: ruby
 authors:
 - Carl Mercier