pgbus 0.16.6 → 0.16.7

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: cddde0da7d83fc84f54404d10bcdbfccb06f25af74379b560ac4f9bb892f0041
-  data.tar.gz: 44eba7a5a2bbe87659f388c7fb10c5a7b7577a1f045bb8d48c527cfbf9266df5
+  metadata.gz: 82a6db71dd099a9b4fa24cb0a278184a3aed8bbee7acfb3f3cc6ef6b55064038
+  data.tar.gz: ad7034ee934aee91afdb4a6278200f8178caf0da609d49e57bc6bbcf27d3b7b5
 SHA512:
-  metadata.gz: 3bb680972106246ea3a16265730eb8b0567944e80323cbca8a9f68f7b7302b8afa2087cfee74e0fa99b695f038fd5e2104141971096ac8d76b236437ce147e60
-  data.tar.gz: cf822271a70b7ec8bcbbca0591838179678113c2b19849a87b4442cf5e0ccd3b2bac1d7a92e8820596c839ff1005c82d0c0ab45d47ddad2df1bae8094b0a9208
+  metadata.gz: 86977d3f705b8e3732211956582334f6e49a60e426af1082fdf585a175c9a189c9dc9ec83d5ceeaac2f0be7dcee40c1414e55a99225fcc90ac955537d8b4f01c
+  data.tar.gz: 3af2ae55db88774c89f3082b70e5456201c9a85ffd994d43beb197fe0f52d282209fc5de57db66c44de10afe34b37d37fa5404522eabb97d4eb2e91cbfc80da7

data/CHANGELOG.md

@@ -10,6 +10,10 @@
 
 ### Fixed
 
+- **A still-running event handler is no longer re-run as if it had crashed, and a slow one is no longer redelivered mid-run.** Two halves of one hole, follow-up to #469. First: `Process::Consumer` had no equivalent of `ActiveJob::Executor#with_visibility_heartbeat`, so an event message's visibility timeout was never re-armed — a handler slower than `visibility_timeout` (30s by default) was redelivered *while it was still running*, `read_ct` climbed on every redelivery, and after `max_retries` the event was dead-lettered without the handler ever raising. Workers have been protected from this since the heartbeat landed; event consumers never were. The consumer now tracks each message through `VisibilityHeartbeat` for exactly as long as its handlers run, releasing the entry before the archive so a beat can never re-arm a message that is already gone. Second: `Handler#claim_idempotency?` read a `pgbus_processed_events` row with `completed_at IS NULL` as proof that the previous holder had been killed mid-handler, and re-ran. That state equally describes a handler that is simply still running — on another thread, another fork or another host — so a second delivery (a genuine redelivery, or a duplicate envelope, whose independent visibility timeouts no heartbeat can serialize) executed the handler *concurrently with* the live one, which is precisely the double-execution `idempotent!` exists to prevent. The claim now carries liveness rather than only a claim instant: `EventBus::ClaimBeat` refreshes every in-flight claim's `processed_at` from the same beat that re-arms the message's visibility timeout, so message visibility and claim liveness go quiet together when a process dies. A pending claim silent for longer than twice the heartbeat interval is abandoned and re-runs, exactly as before; a fresher one is owned and the delivery skips, deferring to the holder — which either completes (nothing is lost) or fails, leaving its own message for visibility-timeout redelivery to recover. A skip is also no longer silent: `pgbus.event_skipped` carries the reason (`:completed`, `:cached` or `:owned`), the claim's age in seconds and the delivery's `read_ct`, and the metrics subscriber counts it as `pgbus_event_count` with `status: "skipped"` and the reason as a tag. No migration, no new column, and no change on a table that has not run `pgbus:add_processed_event_completion` — a single-phase claim has no pending state and therefore no ownership question. Closes #470.
+
+- **An event is no longer handed to every handler whose pattern matches it, so a wildcard handler ran once per subscriber on the topic.** Each subscriber gets its own queue (`Subscriber#setup!` creates and binds one), so a topic with N matching subscribers puts N copies of every event on the bus — but `Consumer#handle_message` resolved handlers with `Registry#handlers_for(routing_key)`, a pattern match that ignored which queue the message had just been read from, and ran every match. Each of the N copies therefore fanned out to all N handlers: **every handler ran N times per event**, with four `"#"` subscribers meaning four invocations of each handler for every event on the bus. `idempotent!` was the only thing hiding it, and it hides it imperfectly by design: the two-phase claim (#385) deliberately *re-runs* when the existing claim is still pending, on the theory that the prior holder crashed — but a pending claim is also exactly what a handler still running on another host looks like. So one event could have host A win the claim and start a 700ms handler while host B, reading a different subscriber's copy of the same event a few hundred milliseconds later, dispatched to that same handler, lost the claim insert, read `completed_at IS NULL` as "crashed, re-run", and executed it concurrently. In one host app that meant a "task completed" record written twice and the user emailed twice, with a single row in `pgbus_processed_events`, `read_ct = 1` on every archived copy, and the second execution's side effects timestamped inside the first's window — dispatch, not redelivery. A non-idempotent handler simply ran N times with no guard at all. `handlers_for` now takes a required `queue_name:` and selects on ownership *and* pattern, so a message read from queue Q goes to Q's owner(s) alone and each handler runs exactly once per event; duplicate execution again requires a real redelivery, which is the at-least-once contract the docs describe. The keyword is required rather than defaulted precisely so a keyword-less call cannot silently restore the fan-out — the two callers that genuinely want the pattern view (the `Testing.inline!` publish path and `Testing::EventStore#drain!`, neither of which has a queue, since the event never reaches PGMQ) moved to an explicitly named `Registry#subscribers_matching`, where per-subscriber delivery counts already match what owner-only dispatch now produces in production. Two handlers registered against the same explicit `queue_name:` still both run on that queue's delivery. The pattern check is kept alongside the ownership check because a routing key the owner's pattern does not match means a stale `pgmq.topic_bindings` row, and a stale binding must not run the handler. A message on a queue no subscriber in this process owns is still archived rather than looped through visibility-timeout redelivery into the DLQ — that was already the behavior when `handlers_for` returned `[]` — but it is no longer silent: it logs a warning naming the queue and routing key, rate-limited to once per queue per process so a permanently stale binding cannot flood the log, and emits `pgbus.event_unrouted` on every occurrence so the real rate stays visible in metrics. The pending-claim re-run in `Handler#claim_idempotency?` is deliberately untouched here; it is a separate question now that dispatch no longer manufactures the concurrency it was misreading. Closes #469.
+
 - **A dropped ActiveRecord socket no longer pages the host app for an event that was handled successfully.** EventBus consumers are long-lived threads holding a leased AR connection, and a pooler restart, an admin disconnect or a brief failover can kill that socket at any point. Rails reconnects most statements transparently, but `Relation#update_all` is marked `allow_retry: false` — and that is exactly the phase-2 claim stamp in `Handler#complete_claim!`, the one AR write that happens *after* `handle` has already returned. A drop there surfaced as `ActiveRecord::ConnectionFailed` ("PQconsumeInput() SSL error: unexpected eof while reading") on a message whose work was done, so the host app's exception tracker paged for a false failure and PGMQ redelivered the event for a re-run. `EventBus::StaleConnectionRetry` now reconnects this thread's lease (via `ConnectionPool#active_connection?`, the accessor that exists across the supported Rails range) and repeats the stamp once; a second drop still raises, leaving the message to VT redelivery. Only the leased connections are reconnected — `clear_all_connections!` would yank sockets out from under sibling consumers in the same process, turning one recoverable drop into many. The retryable patterns are deliberately *broader* than `Client::STALE_CONNECTION_PATTERNS` and for the opposite reason: that list excludes mid-flight drops because a half-committed enqueue would duplicate a message, whereas this only ever repeats an idempotent `SET completed_at = <now>`. Phase 1 (`claim_idempotency?`) is deliberately not wrapped — its INSERT may have committed before the socket died, and on a legacy schema the retry's empty `result.rows` would read as "another consumer owns this claim", turning a recoverable drop into a silently skipped event.
 
 

data/README.md

@@ -221,6 +221,14 @@ Pgbus::EventBus::Registry.instance.subscribe(
 )
 ```
 
+Each subscriber gets its own queue, and a queue is consumed only by the
+handler(s) registered against it: one published event matching N subscribers
+becomes N deliveries, one per handler, so every handler runs exactly once per
+event. (Two handlers sharing an explicit `queue_name:` both run on that queue's
+delivery.) A message on a queue no subscriber in this process owns — a stale
+topic binding left by a renamed or removed handler — is archived, logged once
+per queue, and reported as `pgbus.event_unrouted`.
+
 `idempotent!` uses a **two-phase claim**: a *pending* row in
 `pgbus_processed_events` is inserted before `handle` runs, and only stamped
 `completed_at` after `handle` returns. Deduplication applies to **completed**

data/lib/pgbus/event_bus/claim_beat.rb

@@ -0,0 +1,80 @@
+# frozen_string_literal: true
+
+module Pgbus
+  module EventBus
+    # Liveness for the idempotency claims in flight for one PGMQ message.
+    #
+    # A two-phase claim (issue #385) is a `pgbus_processed_events` row with
+    # `completed_at IS NULL`. That state says "claimed, not finished" — it does
+    # NOT say whether the claimer is dead or simply still running (issue #470).
+    # Handler resolves the ambiguity by age: a claim whose `processed_at` has
+    # gone quiet for longer than the ownership window is abandoned, anything
+    # fresher is owned. For that age to mean "silence" rather than
+    # "time since the claim was taken", something has to keep stamping it while
+    # the handler runs. This is that something.
+    #
+    # One beat per message, created by Process::Consumer and handed to every
+    # handler dispatched for it. A handler registers its claim for exactly the
+    # duration of `handle` and the consumer's VisibilityHeartbeat `on_beat` hook
+    # drives #touch! on the same cadence that re-arms the message's visibility
+    # timeout — so the claim and the message go quiet together when the process
+    # dies, and both stay fresh while it lives.
+    #
+    # #touch! runs on the heartbeat ticker thread while #register / #release run
+    # on a pool thread, hence the mutex.
+    class ClaimBeat
+      def initialize
+        @mutex = Mutex.new
+        @claims = []
+      end
+
+      def register(event_id, handler_class)
+        claim = [event_id, handler_class]
+        @mutex.synchronize { @claims << claim unless @claims.include?(claim) }
+        self
+      end
+
+      def release(event_id, handler_class)
+        @mutex.synchronize { @claims.delete([event_id, handler_class]) }
+        self
+      end
+
+      def size
+        @mutex.synchronize { @claims.size }
+      end
+
+      def empty?
+        size.zero?
+      end
+
+      # Refresh every registered claim's liveness stamp. Returns the number of
+      # claims touched. No-op on a legacy schema: without `completed_at` there
+      # are no pending claims to keep alive, and Handler's single-phase
+      # fallback never consults the age.
+      #
+      # A claim that fails to update is logged and skipped rather than raised:
+      # this runs inside the visibility heartbeat's beat, and one unwritable
+      # row must not cost every other in-flight message its VT extension.
+      def touch!
+        return 0 unless ProcessedEvent.completion_column?
+
+        now = Time.now.utc
+        @mutex.synchronize { @claims.dup }.count { |event_id, handler_class| touch(event_id, handler_class, now) }
+      end
+
+      private
+
+      def touch(event_id, handler_class, now)
+        ProcessedEvent
+          .where(event_id: event_id, handler_class: handler_class, completed_at: nil)
+          .update_all(processed_at: now)
+        true
+      rescue StandardError => e
+        Pgbus.logger.warn do
+          "[Pgbus] Could not refresh idempotency claim #{handler_class}/#{event_id}: #{e.class}: #{e.message}"
+        end
+        false
+      end
+    end
+  end
+end

data/lib/pgbus/event_bus/handler.rb

@@ -17,8 +17,20 @@ module Pgbus
         end
       end
 
-      def process(message)
-        with_rails_executor { process!(message) }
+      # Outcome of the two-phase claim. `age` is how long the losing delivery
+      # found the existing claim to have been silent, in seconds (nil unless
+      # the claim was pending).
+      ClaimResult = Data.define(:status, :age) do
+        def granted?
+          status == :claimed
+        end
+      end
+
+      # @param claim_beat [ClaimBeat, nil] the message's claim-liveness beat,
+      #   supplied by Process::Consumer. Absent for a hand-rolled caller: the
+      #   handler still runs, its claim simply ages from the claim instant.
+      def process(message, claim_beat: nil)
+        with_rails_executor { process!(message, claim_beat) }
       end
 
       def handle(event)
@@ -27,12 +39,18 @@ module Pgbus
 
       private
 
-      def process!(message)
+      def process!(message, claim_beat = nil)
         raw = JSON.parse(message.message)
         event = build_event(raw)
         routing_key = raw.dig("headers", "routing_key") || raw["routing_key"]
 
-        return :skipped if self.class.idempotent? && !claim_idempotency?(event.event_id)
+        if self.class.idempotent?
+          claim = claim_idempotency(event.event_id)
+          unless claim.granted?
+            instrument_skip(claim, event, message, routing_key)
+            return :skipped
+          end
+        end
 
         instrument_payload = {
           event_id: event.event_id,
@@ -42,11 +60,13 @@ module Pgbus
           read_ct: message.read_ct.to_i,
           msg_id: message.msg_id.to_i
         }
-        Instrumentation.instrument("pgbus.event_processed", instrument_payload) do
-          # Publisher's Current attributes (issue #431) are set for the handler
-          # and reverted after (CurrentAttributes#set semantics); the Rails
-          # executor wrap above additionally resets at completion.
-          Pgbus::CurrentAttributes.restore(event.context) { handle(event) }
+        with_claim_beat(claim_beat, event.event_id) do
+          Instrumentation.instrument("pgbus.event_processed", instrument_payload) do
+            # Publisher's Current attributes (issue #431) are set for the handler
+            # and reverted after (CurrentAttributes#set semantics); the Rails
+            # executor wrap above additionally resets at completion.
+            Pgbus::CurrentAttributes.restore(event.context) { handle(event) }
+          end
         end
         complete_claim!(event.event_id) if self.class.idempotent?
         :handled
@@ -64,7 +84,7 @@ module Pgbus
 
       # Mirrors Pgbus::ActiveJob::Executor#execute_job: wrap the handler
       # invocation in Rails.application.executor (or the reloader in dev)
-      # so AR connections leased by `claim_idempotency?` and `handle` are
+      # so AR connections leased by `claim_idempotency` and `handle` are
       # released back to the pool when this method returns. Without the
       # wrap, every consumed event leaks one AR connection on the consumer
       # thread — in dev that wedges `clear_reloadable_connections!`,
@@ -113,24 +133,32 @@ module Pgbus
 
       # Two-phase idempotency claim (issue #385). Phase 1: atomically claim
       # via INSERT ... ON CONFLICT DO NOTHING with completed_at NULL — a
-      # *pending* claim. Returns true when this delivery should run handle:
+      # *pending* claim. Returns a ClaimResult whose status is one of:
       #
-      #   - insert won → fresh claim
-      #   - insert lost, completed_at NULL → a prior attempt claimed but was
-      #     killed before finishing (SIGKILL mid-handler); re-run so the crash
-      #     doesn't silently drop the execution. Safe: PGMQ's VT means the
-      #     prior holder is dead or wedged past its timeout — the same
-      #     at-least-once window every non-idempotent handler has.
+      #   :claimed   — insert won (fresh claim), or the row was purged between
+      #                the losing insert and the read, or an existing pending
+      #                claim has gone silent for longer than the ownership
+      #                window: the holder is dead by the heartbeat's own
+      #                definition, so re-run rather than silently drop the
+      #                execution a SIGKILL interrupted.
+      #   :completed — the execution already finished. Skip.
+      #   :owned     — pending, and its liveness stamp is fresh: the holder is
+      #                still running (issue #470). Skip — running `handle`
+      #                concurrently with the holder is exactly the
+      #                double-execution `idempotent!` promises not to do. The
+      #                holder either completes (nothing lost) or fails, leaving
+      #                its own message for VT redelivery to recover.
+      #   :cached    — a completed execution already in this process's memory.
       #
-      # Returns false (skip) only for a *completed* execution. Phase 2 is
-      # complete_claim! after handle returns; only completed executions enter
-      # the in-memory dedup cache.
+      # Phase 2 is complete_claim! after handle returns; only completed
+      # executions enter the in-memory dedup cache.
       #
       # Legacy fallback: without the completed_at column (upgraded gem,
-      # not-yet-migrated table) this degrades to the old single-phase claim.
-      def claim_idempotency?(event_id)
+      # not-yet-migrated table) this degrades to the old single-phase claim,
+      # which has no pending state and therefore no ownership question.
+      def claim_idempotency(event_id)
         cache_key = dedup_key(event_id)
-        return false if self.class.dedup_cache.seen?(cache_key)
+        return ClaimResult.new(status: :cached, age: nil) if self.class.dedup_cache.seen?(cache_key)
 
         result = ProcessedEvent.insert(
           { event_id: event_id, handler_class: self.class.name, processed_at: Time.now.utc },
@@ -139,18 +167,79 @@ module Pgbus
 
         unless ProcessedEvent.completion_column?
           self.class.dedup_cache.mark!(cache_key)
-          return result.rows.any?
+          return ClaimResult.new(status: result.rows.any? ? :claimed : :completed, age: nil)
         end
 
-        return true if result.rows.any?
+        return ClaimResult.new(status: :claimed, age: nil) if result.rows.any?
+
+        inspect_existing_claim(event_id, cache_key)
+      end
+
+      # The insert lost, so a row exists (or existed). `pick` returns nil for
+      # the whole row when it has since been purged — not a pending claim,
+      # nothing is running, so claim it.
+      def inspect_existing_claim(event_id, cache_key)
+        completed_at, processed_at = ProcessedEvent
+                                     .where(event_id: event_id, handler_class: self.class.name)
+                                     .pick(:completed_at, :processed_at)
+
+        if completed_at
+          self.class.dedup_cache.mark!(cache_key)
+          return ClaimResult.new(status: :completed, age: nil)
+        end
+
+        return ClaimResult.new(status: :claimed, age: nil) if processed_at.nil?
+
+        age = Time.now.utc - processed_at.to_time.utc
+        return ClaimResult.new(status: :owned, age: age) if age < claim_ownership_window
+
+        ClaimResult.new(status: :claimed, age: age)
+      end
+
+      # How long a pending claim may stay silent before its holder counts as
+      # dead. ClaimBeat refreshes a live claim from the visibility heartbeat,
+      # which lands every extension inside [interval, 1.5 * interval] of the
+      # previous one — two intervals leaves margin for a late beat without
+      # stretching the window past the visibility timeout it rides on.
+      #
+      # With the heartbeat disabled a claim is never refreshed, so the window
+      # degrades to "roughly the first two thirds of one visibility timeout
+      # after the claim" — a redelivery, which cannot arrive before the VT has
+      # lapsed, still re-runs exactly as it did before issue #470.
+      def claim_ownership_window
+        Pgbus.configuration.effective_visibility_heartbeat_interval * 2
+      end
 
-        completed_at = ProcessedEvent
-                       .where(event_id: event_id, handler_class: self.class.name)
-                       .pick(:completed_at)
-        return true if completed_at.nil? # pending claim (or purged row) → re-run
+      # Register this claim with the message's beat for exactly the duration of
+      # handle: before it, there is nothing to keep alive; after it,
+      # complete_claim! owns the row and a beat touching processed_at would
+      # race the completion stamp.
+      def with_claim_beat(claim_beat, event_id)
+        return yield unless claim_beat && self.class.idempotent? && ProcessedEvent.completion_column?
+
+        claim_beat.register(event_id, self.class.name)
+        begin
+          yield
+        ensure
+          claim_beat.release(event_id, self.class.name)
+        end
+      end
 
-        self.class.dedup_cache.mark!(cache_key)
-        false
+      # A skip used to be silent, which made an over-eager re-run (issue #470)
+      # invisible in production: nothing distinguished "deduplicated" from
+      # "deferred to a live holder". The claim age and read_ct are what tell
+      # an operator which one happened.
+      def instrument_skip(claim, event, message, routing_key)
+        Instrumentation.instrument(
+          "pgbus.event_skipped",
+          event_id: event.event_id,
+          handler: self.class.name,
+          routing_key: routing_key,
+          reason: claim.status,
+          claim_age: claim.age,
+          read_ct: message.read_ct.to_i,
+          msg_id: message.msg_id.to_i
+        )
       end
 
       # Phase 2: stamp the claim completed and only then admit it to the
@@ -165,11 +254,11 @@ module Pgbus
       # The stamp is an idempotent `SET completed_at = <now>`, so repeating a
       # statement that may already have committed is safe.
       #
-      # Phase 1 (claim_idempotency?) is deliberately NOT wrapped. Its INSERT
+      # Phase 1 (claim_idempotency) is deliberately NOT wrapped. Its INSERT
       # may have committed before the socket died, and on a legacy schema the
       # retry's empty `result.rows` would read as "someone else owns this
-      # claim" and return false — turning a recoverable drop into a silently
-      # skipped event. VT redelivery is the correct recovery there.
+      # claim" and report :completed — turning a recoverable drop into a
+      # silently skipped event. VT redelivery is the correct recovery there.
       def complete_claim!(event_id)
         return unless ProcessedEvent.completion_column?
 

data/lib/pgbus/event_bus/publisher.rb

@@ -25,7 +25,7 @@ module Pgbus
           Pgbus::Testing.store.push_event(event)
 
           if Pgbus::Testing.inline? && delay.to_i <= 0
-            Pgbus::EventBus::Registry.instance.handlers_for(routing_key).each do |subscriber|
+            Pgbus::EventBus::Registry.instance.subscribers_matching(routing_key).each do |subscriber|
               Pgbus::CurrentAttributes.restore(event.context) { subscriber.handler_class.new.handle(event) }
             end
           end

data/lib/pgbus/event_bus/registry.rb

@@ -62,7 +62,30 @@ module Pgbus
         end
       end
 
-      def handlers_for(routing_key)
+      # Subscribers a message read from +queue_name+ must be dispatched to
+      # (issue #469). Every subscriber gets its own queue, so a topic with N
+      # matching subscribers produces N queue copies of each event; selecting by
+      # pattern alone fanned every copy out to every match, running each handler
+      # N times per event — and, across hosts, concurrently. Ownership is the
+      # primary filter; the pattern check still applies because a routing key
+      # the owner's pattern does not match means a stale pgmq.topic_bindings
+      # row, and a stale binding must not run the handler.
+      #
+      # +queue_name+ is required on purpose: a keyword-less call would silently
+      # restore the fan-out this closed. Callers that genuinely want the
+      # pattern view (the Testing inline/drain paths, which never touch a
+      # queue) use #subscribers_matching.
+      def handlers_for(routing_key, queue_name:)
+        @subscribers.select do |s|
+          s.queue_name == queue_name && matches?(s.pattern, routing_key)
+        end
+      end
+
+      # Pattern-only selection, with no queue in play. Used by the Testing
+      # inline/drain paths, where the event never reaches PGMQ and each matching
+      # subscriber is invoked exactly once — the same per-subscriber delivery
+      # count owner-only dispatch produces in production.
+      def subscribers_matching(routing_key)
         @subscribers.select { |s| matches?(s.pattern, routing_key) }
       end
 

data/lib/pgbus/instrumentation.rb

@@ -20,6 +20,10 @@ module Pgbus
   #                                  payload: queue, job_class, msg_id, vt, extensions
   #   pgbus.event_processed        — event handler succeeded
   #   pgbus.event_failed           — event handler raised; carries :exception_object
+  #   pgbus.event_unrouted         — a consumer read an event from a queue no
+  #                                  subscriber in this process owns (stale topic
+  #                                  binding); the message is archived
+  #                                  payload: queue_name, routing_key
   #   pgbus.stream.broadcast       — stream broadcast (sync or deferred)
   #   pgbus.outbox.publish         — outbox row created
   #   pgbus.recurring.enqueue      — scheduler enqueued a due recurring task

data/lib/pgbus/metrics/subscriber.rb

@@ -33,6 +33,7 @@ module Pgbus
             subscribe("pgbus.job_visibility_extended") { |event| on_job_visibility_extended(event) },
             subscribe("pgbus.event_processed") { |event| on_event_processed(event) },
             subscribe("pgbus.event_failed") { |event| on_event_failed(event) },
+            subscribe("pgbus.event_skipped") { |event| on_event_skipped(event) },
             subscribe("pgbus.client.send_message") { |event| on_send_message(event) },
             subscribe("pgbus.client.send_batch") { |event| on_send_batch(event) },
             subscribe("pgbus.client.read_batch") { |event| on_read_batch(event) },
@@ -139,6 +140,19 @@ module Pgbus
           )
         end
 
+        # An idempotent handler that did not run this delivery. `reason` is what
+        # makes the skip readable: :completed/:cached is deduplication working,
+        # :owned means a second delivery arrived while the holder was still
+        # running and was deferred to it (issue #470).
+        def on_event_skipped(event)
+          payload = event.payload
+          backend.increment(
+            "#{METRIC_PREFIX}event_count", 1,
+            compact(handler: payload[:handler], routing_key: payload[:routing_key],
+                    status: "skipped", reason: payload[:reason])
+          )
+        end
+
         # ── Client (PGMQ wrapper) ─────────────────────────────────────────
 
         def on_send_message(event)

data/lib/pgbus/process/consumer.rb

@@ -78,6 +78,11 @@ module Pgbus
         )
         @registry = EventBus::Registry.instance
         @circuit_breaker = Pgbus::CircuitBreaker.new(config: config)
+        # Queues already warned about for an unroutable message (issue #469).
+        # handle_message runs on the execution pool, so this is touched from
+        # several threads — Concurrent::Set makes add? the atomic
+        # test-and-set the once-per-queue guarantee needs.
+        @unrouted_queues = Concurrent::Set.new
         # stat_buffer: :default means "build one iff config.stats_enabled";
         # passing an explicit value (including nil) overrides that for tests.
         @stat_buffer =
@@ -222,10 +227,12 @@ module Pgbus
         raw = JSON.parse(message.message)
         routing_key = raw.dig("headers", "routing_key") || raw["routing_key"]
 
-        handlers = @registry.handlers_for(routing_key || "")
-        handlers.each do |subscriber|
-          handler = subscriber.handler_class.new
-          handler.process(message)
+        handlers = @registry.handlers_for(routing_key || "", queue_name: queue_name)
+
+        if handlers.empty?
+          report_unrouted(queue_name, routing_key)
+        else
+          dispatch(handlers, message, queue_name)
         end
 
         Pgbus.client.archive_message(queue_name, message.msg_id.to_i)
@@ -247,6 +254,65 @@ module Pgbus
         @jobs_processed.increment
       end
 
+      # Run every handler that owns this message, with the message's visibility
+      # timeout held open for as long as they take (issue #470).
+      #
+      # Without this the consumer had no equivalent of
+      # ActiveJob::Executor#with_visibility_heartbeat: an event handler slower
+      # than config.visibility_timeout (30s by default) was redelivered *while
+      # still running*, a second consumer read the same envelope, and the
+      # holder's pending idempotency claim was indistinguishable from a claim
+      # left by a crash — so the handler ran twice, concurrently.
+      #
+      # The beat also refreshes the claims the handlers register, which is what
+      # lets Handler tell "holder still running" from "holder died": message
+      # visibility and claim liveness go quiet together when this process does.
+      #
+      # Tracking ends before the caller archives — a beat must never re-arm the
+      # VT of a message that is already gone (same rule as the executor's).
+      def dispatch(handlers, message, queue_name)
+        claim_beat = EventBus::ClaimBeat.new
+
+        VisibilityHeartbeat.track(
+          client: Pgbus.client,
+          queue_name: queue_name,
+          msg_id: message.msg_id.to_i,
+          job_class: "EventConsumer",
+          config: config,
+          on_beat: -> { claim_beat.touch! }
+        ) do
+          handlers.each { |subscriber| subscriber.handler_class.new.process(message, claim_beat: claim_beat) }
+        end
+      end
+
+      # No subscriber in this process owns +queue_name+ (a stale
+      # pgmq.topic_bindings row left by a renamed or removed handler, another
+      # app bound to the same bus), or the owner's pattern no longer matches the
+      # routing key. The message is archived by the caller either way — looping
+      # it through VT redelivery would only walk it into the DLQ — but that used
+      # to happen in total silence. The warning is rate-limited to once per
+      # queue per process so a permanently stale binding cannot flood the log;
+      # the instrumentation fires on every message, so the real rate stays
+      # visible in metrics (issue #469).
+      def report_unrouted(queue_name, routing_key)
+        first_for_queue = @unrouted_queues.add?(queue_name)
+
+        if first_for_queue
+          Pgbus.logger.warn do
+            "[Pgbus] Consumer read an unroutable event from queue #{queue_name} " \
+              "(routing_key=#{routing_key.inspect}): no subscriber in this process owns that queue. " \
+              "Archiving. This usually means a stale topic binding — a handler was renamed or removed " \
+              "without unbinding its queue. Further unrouted messages on this queue are not logged."
+          end
+        end
+
+        Pgbus::Instrumentation.instrument(
+          "pgbus.event_unrouted",
+          queue_name: queue_name,
+          routing_key: routing_key
+        )
+      end
+
       # Record a job stat for the handled message, mirroring the shape the
       # executor pushes (Executor#record_stat) so consumer and worker throughput
       # land in the same pgbus_job_stats table. No-op unless stats are enabled.
@@ -521,6 +587,7 @@ module Pgbus
         # wait IS its drain window — bound it by the same knob workers use
         # instead of a hardcoded 30s (issue #386).
         @pool.wait_for_termination(config.drain_timeout)
+        VisibilityHeartbeat.stop
         @stat_buffer&.stop
         @heartbeat&.stop
         restore_signals

data/lib/pgbus/testing.rb

@@ -58,7 +58,7 @@ module Pgbus
           event = @mutex.synchronize { @events.first }
           break unless event
 
-          Pgbus::EventBus::Registry.instance.handlers_for(event.routing_key).each do |subscriber|
+          Pgbus::EventBus::Registry.instance.subscribers_matching(event.routing_key).each do |subscriber|
             # Restore the publisher's Current (issue #431) like the consumer does.
             Pgbus::CurrentAttributes.restore(event.context) { subscriber.handler_class.new.handle(event) }
           end

data/lib/pgbus/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Pgbus
-  VERSION = "0.16.6"
+  VERSION = "0.16.7"
 end

data/lib/pgbus/visibility_heartbeat.rb

@@ -25,9 +25,15 @@ module Pgbus
   # out with `pgbus_visibility_heartbeat false`.
   module VisibilityHeartbeat
     # `concurrency` is `[key, duration]` for a concurrency-limited job, else
-    # nil — one member, so the struct stays inside an 80-byte slot.
+    # nil. `on_beat` is an optional callable run on every extension — the event
+    # consumer uses it to refresh its handlers' idempotency claims (issue #470).
+    #
+    # The ninth member takes the struct out of the 80-byte slot it used to fit
+    # (measured: 80 → 160). That is paid at most once per in-flight message, so
+    # the whole table is bounded by the execution pool's capacity — a handful of
+    # entries per process, not one per enqueued job.
     Entry = Struct.new(:client, :queue_name, :prefixed, :msg_id, :job_class, :extended_at, :extensions,
-                       :concurrency, keyword_init: true)
+                       :concurrency, :on_beat, keyword_init: true)
 
     # Per-job opt-out, included on ActiveJob::Base by the engine:
     #
@@ -59,13 +65,14 @@ module Pgbus
       # @param config [Pgbus::Configuration]
       # @param concurrency [Array(String, Numeric), nil] semaphore key to keep alive alongside
       #   the message and how far to push its expiry on each beat
+      # @param on_beat [#call, nil] run after each extension; its failures are contained
       def track(client:, queue_name:, msg_id:, prefixed: true, job_class: nil, config: Pgbus.configuration,
-                concurrency: nil)
+                concurrency: nil, on_beat: nil)
         return yield unless config.visibility_heartbeat
 
         entry = Entry.new(client: client, queue_name: queue_name, prefixed: prefixed, msg_id: msg_id.to_i,
                           job_class: job_class, extended_at: monotonic_now, extensions: 0,
-                          concurrency: concurrency)
+                          concurrency: concurrency, on_beat: on_beat)
         register(entry, config)
         begin
           yield
@@ -130,6 +137,7 @@ module Pgbus
         entry.extended_at = now
         entry.extensions += 1
         touch_semaphore(entry)
+        run_on_beat(entry)
         Instrumentation.instrument(
           "pgbus.job_visibility_extended",
           queue: entry.queue_name, job_class: entry.job_class, msg_id: entry.msg_id, vt: vt,
@@ -186,6 +194,17 @@ module Pgbus
         end
       end
 
+      # Same containment as touch_semaphore: a lease the beat keeps alive
+      # alongside the message must never cost the message its extension.
+      def run_on_beat(entry)
+        entry.on_beat&.call
+      rescue StandardError => e
+        Pgbus.logger.warn do
+          "[Pgbus::VisibilityHeartbeat] on_beat hook failed for msg_id=#{entry.msg_id} " \
+            "queue=#{entry.queue_name}: #{e.class}: #{e.message}"
+        end
+      end
+
       def forget_parent_entries!
         return if @pid == ::Process.pid
 

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: pgbus
 version: !ruby/object:Gem::Version
-  version: 0.16.6
+  version: 0.16.7
 platform: ruby
 authors:
 - Mikael Henriksson
@@ -291,6 +291,7 @@ files:
 - lib/pgbus/engine.rb
 - lib/pgbus/error_reporter.rb
 - lib/pgbus/event.rb
+- lib/pgbus/event_bus/claim_beat.rb
 - lib/pgbus/event_bus/handler.rb
 - lib/pgbus/event_bus/publisher.rb
 - lib/pgbus/event_bus/registry.rb