pgbus 0.16.3 → 0.16.4

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 5ccb9204900296bfd53c5845faa89a91eec0eddedd7073818af147ef135daa82
-  data.tar.gz: db8ee8c7cbfe2ac34700bce9479736f19cf658a0a728a20a4a8a5988978363fd
+  metadata.gz: ddf8b7dde9919fc1ad8de39311bacac6ad8c50fadeac059035c5c0ff9b5c30c6
+  data.tar.gz: 719cec84b8486fac601755a84ea457f926b5b1b5faed8121e61a33a9c52b4972
 SHA512:
-  metadata.gz: 305bac06a4cdf80af363d4cdc242165e02319f9c9b060c98c44db31af833f4bb055f625b79cac14853ab8382a6458f6142b3f9543ddef55f329581bed112c0bb
-  data.tar.gz: '009aea5c6f176d5b19e87dbd0375e5d9624cd82aa48ace058af26107df57903c0896159d4541bd66615e06973abed5cebc1e7f0230ef935a34df25cba2864e2d'
+  metadata.gz: 9fe2d3c3dd637bb9a6af1b4e72092f5acfedd8237804e6d8316cc38f084a573aeef4ef943db0da616e1f108f6afd2cb367b1163f29cef7cb5550f7d20adf7074
+  data.tar.gz: 646442c236b49fd47a37b5f3e7e988e5bcd3a8263419e7d952b526f5f194debed2821813864fbf05635c96925c579f947e259a3385347bf1f8026b142d9e9953

data/CHANGELOG.md

@@ -6,6 +6,31 @@
 
 ### Fixed
 
+- **`limits_concurrency … on_conflict: :block` now constrains a job without ever losing it, and never runs more than `to:` at once.** The same bug class solid_queue closed in rails/solid_queue#712 and #761, checked against pgbus's implementation. Four defects, each with a regression spec:
+  - **A parked job was silently deleted once it had waited longer than `duration`.** The dispatcher's sweep `delete_all`-ed blocked rows past their `expires_at` (a debug log, nothing else), and `release_next!` refused to promote one — so with `to: 1` and a slow holder, the N-th parked enqueue was simply gone. Blocked rows no longer expire; the only way out of `pgbus_blocked_executions` is promotion. `expires_at` now orders the sweep, nothing more.
+  - **A semaphore expired under a job that was still running, and two runs of one key overlapped.** `duration` capped run time: once it passed, the sweep deleted the semaphore and promoted the next parked job beside the still-running holder — and that promotion took no slot at all, so a third enqueue then created a fresh semaphore and ran too. The visibility heartbeat now re-arms the semaphore alongside the message (`Semaphore.touch`), so `duration` bounds heartbeat silence, not run time; and every promotion — completion-time or sweep — takes its slot through the same `value < max_value` upsert an enqueue uses, so it can never push a key past `to:`.
+  - **A job parked while the holder was finishing was stranded until the semaphore expired.** The enqueue checked the semaphore and inserted the blocked row as two autocommit statements; a holder finishing in between found no parked row, released the slot, and nobody promoted the row it had just missed (solid_queue #456/#712). The adapter now checks and parks in one transaction — the upsert holds the semaphore row lock until commit even when it returns `:blocked` — and the completion signal decrements under that same lock before it looks for parked rows, so it waits for the in-flight park and then sees it.
+  - **A duplicate delivery released the slot twice.** A worker whose heartbeat lapsed had its message redelivered; when both copies finished, both signalled, and a `to: 1` key ran 2, then 3 (solid_queue #761). `pgmq.archive` returning `false` — the message was already archived by the other copy — is now the exact-once claim: that execution returns `:duplicate` and skips the concurrency, batch and uniqueness signals.
+  
+  Underneath, the parked payload was **double-encoded**: `BlockedExecution.insert` handed the jsonb column an already-serialized string, so every row was a jsonb *string*, not an object. Promotion worked only because the client passes a String body through untouched; everything that read the column as a document did the wrong thing silently — the job-class lookup (so a promoted job always took the default limit), `scheduled_at` (a parked `perform_later(wait:)` job promoted immediately), `Batch.backfill_execution` (a parked batch job never got its execution row) and the batch sweep's `payload->>'job_id'` (a parked batch job looked orphaned and was un-counted, undoing #423). Rows are now stored as objects; the sweep rewrites rows written before this release in place (`repair_double_encoded!`, idempotent) and `release_next!` reads both shapes. No migration.
+
+  Six narrower holes in the same guarantee, found in review:
+  - **A scheduled job lost its slot while still waiting in the queue.** The slot is taken at enqueue but its lease is only renewed once a worker picks the message up, so a `perform_later(wait: 1.hour)` job under a 15-minute `duration` was presumed dead mid-wait and a second job promoted for its key. A scheduled job's lease now covers its delay. The residual — a queue backed up longer than `duration` before any worker gets to the message — is documented rather than papered over.
+  - **A `duration` shorter than the heartbeat interval expired before the first beat.** `duration` is now floored at twice the effective heartbeat interval, so a lease can never lapse before the beat that would have renewed it.
+  - **The enqueue sent the message before its slot committed.** PGMQ has its own connection, so the send can never join the AR transaction; a commit failure after a successful send left the message live with the slot rolled back, and the next enqueue ran beside it. The send now happens after the commit, so the only crash window leaves a slot held with no message — under-admitting until the sweep reclaims it, never over-admitting. A send that raises hands its slot back immediately.
+  - **A batch backfill failure could undo a promotion.** `Batch.backfill_execution` ran inside the transaction `Semaphore.signal` opens, so a database error there poisoned it: the commit failed, the parked row came back and the slot was released while the promoted message was already live — the job ran twice. The backfill now has its own savepoint.
+  - **An ambiguous archive retry was read as a duplicate.** `archive_from` retries once on a connection error; if the first archive had actually committed and only its reply was lost, the retry reported "already archived" and the executor skipped every completion signal, stranding the slot, the batch execution and the uniqueness lock. It now distinguishes `:ambiguous` from `:already_archived` and claims the former. A genuine duplicate also releases its own `:while_executing` lock, conditioned on the message id so it can never drop a successor's.
+  - **The sweep's key scan could starve.** It took the thousand oldest parked keys; a thousand keys whose slots were held sat at the head of that window forever, so a key behind them whose holder had died was never serviced. The scan now selects only keys that could take a slot right now.
+
+  And five more from a second review pass:
+  - **An ambiguous send no longer undoes the bookkeeping for a message that may be live.** A connection error can reach the enqueuer after the produce has committed. Releasing the slot then admits a second job beside that message, and rolling the uniqueness lock back leaves it unguarded — so a database-shaped failure raised from inside the produce now keeps the slot, the lock and the batch count. Each is then reclaimed by its own recovery path if the message really was never written: the lease expiry plus the dispatcher's concurrency sweep for the slot, the dispatcher's unbound-lock reaper for the uniqueness row, and `Batch::Sweep`'s orphan pass for the batch count. Anything raised before the produce is called — taking the slot, parking the job — is not ambiguous and still rolls straight back, as is a non-database failure such as a serialization or argument error. The classifier is deliberately an over-approximation in one place: the client's pre-produce queue setup runs inside the same call, so its connection errors are treated as ambiguous too, erring toward state the reapers can reclaim.
+  - **`release_if_bound!` is scoped by queue.** PGMQ message ids are per-queue sequences, so matching a lock on `msg_id` alone could delete a successor holding the same key on another queue.
+  - **Heartbeat touches are floored too.** The floor was applied when taking a slot but not when renewing one, so a `duration` below the heartbeat interval still let a lease lapse mid-run.
+  - **A promoted scheduled job's lease covers its remaining delay**, matching the direct-enqueue path — a parked `wait:` job promoted while its scheduled time was still far out got a lease that expired before the message was ever visible.
+  - The concurrency page names the one window a lease cannot cover (enqueue to first heartbeat) and how to size `duration` against it.
+
+  `Concurrency::BlockedExecution.expire_stale` is gone; `Concurrency::Semaphore.signal(key, client:)` replaces the executor's promote-then-release pair; `Concurrency::BlockedExecution.promote_pending(client:)` is the sweep; `Executor#execute` returns `:duplicate` when another worker archived its message. Docs: the concurrency page now spells out the `:block` guarantees, what `duration` means and the one window it cannot cover.
+
 - **The batch progress bar rendered at zero width — because it rendered with no colour at all.** `app/frontend/pgbus/style.css` is a committed Tailwind build artifact that nothing rebuilds automatically; it was last compiled in #47, before the Batches tab existed. Every Tailwind class added to a view since then resolved to no CSS rule, so the progress fill on `/batches` and `/batches/:id` sized itself correctly from its inline `width:` and then painted nothing — a batch at 483/807 looked stuck at 0% no matter how often you reloaded. 33 classes were missing in total, including the bar's own `bg-green-500` / `bg-amber-500` fill and `bg-gray-200` track, the batches-table bar geometry (`h-2`, `w-24`), and `sm:grid-cols-4`, `hover:underline`, `gap-2`, `z-10`, `break-all`, `cursor-not-allowed` elsewhere on the dashboard. The artifact is rebuilt, `tailwind.css` now names its sources explicitly with `@source` (automatic detection would also walk the nested `docs/` app), `rake frontend:css` makes the build repeatable and version-pinned from `bun.lock`, and a new spec fails the suite whenever a class used in a view is absent from the artifact — the drift can no longer ship silently.
 
 - **Concurrent first-broadcast notify-insert setup no longer pages on a Postgres deadlock / lock-wait (the residual of #403).** Two processes racing the first durable Turbo/Pgbus stream broadcast can both see "notify trigger not current" and both run `pgmq.enable_notify_insert` → `DROP TRIGGER` under `AccessExclusiveLock`. Postgres deadlocks, or hits `lock_timeout`, or — on pooled connections where `statement_timeout` < `lock_timeout` — `canceling statement due to statement timeout` **while locking**. PGMQ wraps those as `PGMQ::Errors::ConnectionError`. 0.16.1 already skips DROP when the trigger is current and retries stale sockets via `with_stale_connection_retry`, but it did not retry these lock races, so the residual still paged (getzazu/app#3817; AppSignal Zazu ma-prod #546 / za-prod #530). `ensure_stream_queue` now wraps the notify-setup path in a dedicated `with_notify_lock_retry` (up to 3 attempts, short backoff **outside** `@pgmq_mutex`): deadlock, lock-not-available, lock-timeout, and statement-timeout **with** lock-wait context retry; a bare statement timeout and permission errors still fail fast, while a missing-queue `ConnectionError` uses the existing one-time queue-recreation path without lock retries. `with_stale_connection_retry` is unchanged — that helper stays idle-socket / no-SQL-sent only. Consumers that prepended an app-level retry around `ensure_stream_queue` (getzazu/app#3827) can drop that prepend after upgrading.

data/app/models/pgbus/blocked_execution.rb

@@ -5,19 +5,17 @@ module Pgbus
     self.table_name = "pgbus_blocked_executions"
 
     scope :for_key, ->(key) { where(concurrency_key: key) }
-    scope :expired, ->(now = Time.current) { where("expires_at < ?", now) }
 
-    # Atomic dequeue: DELETE the highest-priority non-expired row with FOR UPDATE SKIP LOCKED.
-    # Returns { queue_name:, payload: } or nil.
+    # Atomic dequeue: DELETE the highest-priority row with FOR UPDATE SKIP LOCKED.
+    # A parked job never ages out — `expires_at` orders the sweep, nothing more.
+    # Returns { queue_name:, payload:, priority: } or nil.
     def self.release_next!(concurrency_key)
-      now = Time.current
       result = connection.exec_query(
         <<~SQL,
           DELETE FROM pgbus_blocked_executions
           WHERE id = (
             SELECT id FROM pgbus_blocked_executions
             WHERE concurrency_key = $1
-              AND expires_at >= $2
             ORDER BY priority ASC, created_at ASC
             LIMIT 1
             FOR UPDATE SKIP LOCKED
@@ -25,16 +23,44 @@ module Pgbus
           RETURNING queue_name, payload, priority
         SQL
         "Pgbus Blocked Release",
-        [concurrency_key, now]
+        [concurrency_key]
       )
 
       row = result.first
       return nil unless row
 
       payload = row["payload"]
-      payload = JSON.parse(payload) if payload.is_a?(String)
+      # exec_query returns jsonb as text; a row written before the
+      # double-encoding fix parses to a String holding the real document.
+      2.times { payload = JSON.parse(payload) if payload.is_a?(String) }
 
       { queue_name: row["queue_name"], payload: payload, priority: row["priority"] }
     end
+
+    # Rewrite rows parked before the double-encoding fix (a jsonb string
+    # holding the document) as the document itself, so SQL readers such as
+    # the batch sweep's `payload->>'job_id'` see them. Idempotent; the sweep
+    # runs it before every promotion pass.
+    def self.repair_double_encoded!
+      where("jsonb_typeof(payload) = 'string'").update_all("payload = (payload #>> '{}')::jsonb")
+    end
+
+    # Keys with parked jobs that could take a slot right now — no semaphore
+    # row, or one with room — longest-waiting first.
+    #
+    # Filtering in SQL rather than skipping in Ruby is what keeps the scan's
+    # cap honest: a plain oldest-first list would fill with keys whose slots
+    # are held (their holders promote for themselves on completion anyway)
+    # and starve a key behind them whose holder died.
+    def self.promotable_keys(limit: 1000)
+      joins(<<~SQL)
+        LEFT JOIN pgbus_semaphores ON pgbus_semaphores.key = pgbus_blocked_executions.concurrency_key
+      SQL
+        .where("pgbus_semaphores.key IS NULL OR pgbus_semaphores.value < pgbus_semaphores.max_value")
+        .group(:concurrency_key)
+        .order(Arel.sql("MIN(pgbus_blocked_executions.created_at)"))
+        .limit(limit)
+        .pluck(:concurrency_key)
+    end
   end
 end

data/app/models/pgbus/semaphore.rb

@@ -7,16 +7,21 @@ module Pgbus
     scope :expired, ->(now = Time.current) { where("expires_at < ? OR value <= 0", now) }
 
     # Atomic conditional UPSERT. Returns :acquired or :blocked.
+    #
+    # A nil max_value means "keep the limit this row already records" — used
+    # when promoting a parked job whose class no longer resolves, so the
+    # promotion is judged against the real limit rather than a guessed 1.
+    # On a fresh row there is no recorded limit, so nil falls back to 1.
     def self.acquire!(key, max_value, expires_at)
       result = connection.exec_query(
         <<~SQL,
           INSERT INTO pgbus_semaphores (key, value, max_value, expires_at)
-          VALUES ($1, 1, $2, $3)
+          VALUES ($1, 1, COALESCE($2, 1), $3)
           ON CONFLICT (key) DO UPDATE
             SET value = pgbus_semaphores.value + 1,
-                max_value = EXCLUDED.max_value,
+                max_value = COALESCE($2, pgbus_semaphores.max_value),
                 expires_at = GREATEST(pgbus_semaphores.expires_at, EXCLUDED.expires_at)
-            WHERE pgbus_semaphores.value < EXCLUDED.max_value
+            WHERE pgbus_semaphores.value < COALESCE($2, pgbus_semaphores.max_value)
           RETURNING value
         SQL
         "Pgbus Semaphore Acquire",

data/app/models/pgbus/uniqueness_key.rb

@@ -70,6 +70,22 @@ module Pgbus
       )
     end
 
+    # Release a lock only while it still points at this message. Used by the
+    # executor when it finds its message already archived by another worker:
+    # the :while_executing lock it took belongs to this attempt and has to go
+    # back, but an unconditional key-only DELETE could drop a successor that
+    # has since acquired the same key.
+    # PGMQ message ids are per-queue sequences, so a msg_id alone is not an
+    # identity: the same number addresses a different message on every other
+    # queue. The queue is part of the match.
+    def self.release_if_bound!(lock_key, queue_name:, msg_id:)
+      Thread.current[:pgbus_uniqueness_created_at]&.delete(lock_key)
+      connection.exec_delete(
+        "DELETE FROM #{table_name} WHERE lock_key = $1 AND queue_name = $2 AND msg_id = $3",
+        "UniquenessKey Release If Bound", [lock_key, queue_name.to_s, msg_id.to_i]
+      )
+    end
+
     # Check if a key is currently locked.
     def self.locked?(lock_key)
       result = connection.select_value(

data/lib/pgbus/active_job/adapter.rb

@@ -70,17 +70,40 @@ module Pgbus
         priority = active_job.try(:priority)
         msg_id = nil
         blocked = false
+        # Only a failure raised from inside the produce can be ambiguous.
+        # Everything before it — taking the slot, parking the job — proves no
+        # message exists, however database-shaped the error looks. Never
+        # reset: once the send returns, msg_id is set and the rescue routes
+        # on that instead.
+        sending = false
 
         if key && concurrency
-          result = Concurrency::Semaphore.acquire(key, concurrency[:limit], concurrency[:duration])
+          # The check and the park commit together, under the semaphore row
+          # lock the upsert holds even when it reports :blocked — so a holder
+          # signalling right now waits and then sees the parked row instead
+          # of stranding it (rails/solid_queue#712).
+          acquired = false
+          Pgbus::Semaphore.transaction(requires_new: true) do
+            acquired = Concurrency::Semaphore.acquire(key, concurrency[:limit], slot_lease(concurrency, delay)) ==
+                       :acquired
+            blocked = handle_conflict(concurrency, active_job, key, queue, payload_hash, priority: priority) unless
+              acquired
+          end
 
-          if result == :acquired
-            msg_id = Pgbus.client.send_message(queue, payload_hash, delay: delay, priority: priority)
+          if acquired
+            # Deliberately AFTER the commit. PGMQ has its own connection, so
+            # the send can never join this transaction; sending first would
+            # mean a failed commit leaves the message live with the slot
+            # rolled back, and the next enqueue runs beside it. This way the
+            # only crash window leaves a slot held with no message — the
+            # sweep reclaims it, and until then the key is under-admitted,
+            # never over-admitted.
+            sending = true
+            msg_id = send_holding_slot(key, queue, payload_hash, delay: delay, priority: priority)
             active_job.provider_job_id = msg_id
-          else
-            blocked = handle_conflict(concurrency, active_job, key, queue, payload_hash, priority: priority)
           end
         else
+          sending = true
           msg_id = Pgbus.client.send_message(queue, payload_hash, delay: delay, priority: priority)
           active_job.provider_job_id = msg_id
         end
@@ -97,18 +120,69 @@ module Pgbus
         Thread.current[:pgbus_acquired_uniqueness_key] = nil
         active_job
       rescue StandardError => e
-        if msg_id.nil?
+        # An ambiguous send is treated exactly like a live message: the
+        # produce may have committed with only its reply lost, and undoing
+        # the bookkeeping for a message that is in fact live is the worse
+        # error — it would leave that job running with no uniqueness lock and
+        # uncounted by its batch. A batch left waiting for a job that never
+        # existed is recovered by the stalled-batch sweep; a batch that
+        # finishes early and fires its callback is not recoverable. The
+        # `sending` guard keeps that reprieve to the produce itself: a
+        # database error from the slot upsert or the park is not ambiguous.
+        if msg_id.nil? && !(sending && ambiguous_delivery?(e))
           rollback_acquired_uniqueness_lock
           uncount_batch_job(payload_hash)
         else
-          # Message is live: drop the thread-local so a later discard on this
-          # thread cannot release that job's uniqueness lock, but do not
-          # DELETE the pgbus_uniqueness_keys row.
+          # Drop the thread-local so a later discard on this thread cannot
+          # release that job's uniqueness lock, but do not DELETE the
+          # pgbus_uniqueness_keys row.
           Thread.current[:pgbus_acquired_uniqueness_key] = nil
         end
         raise e
       end
 
+      # True when a failure raised from inside the produce reached the
+      # database, so the message may have been written even though the reply
+      # did not come back. Deliberately an over-approximation: the client's
+      # pre-produce queue setup is inside the same call, so its connection
+      # errors are counted as ambiguous too. That errs toward keeping a lock
+      # and a batch count the reapers can reclaim, rather than dropping
+      # bookkeeping for a message that turns out to be live.
+      def ambiguous_delivery?(error)
+        (defined?(PGMQ::Errors::ConnectionError) && error.is_a?(PGMQ::Errors::ConnectionError)) ||
+          (defined?(PG::Error) && error.is_a?(PG::Error))
+      end
+
+      # A slot is leased for `duration` of silence, but the visibility
+      # heartbeat that renews it only starts once a worker picks the message
+      # up. A scheduled job waits in PGMQ until then, so its lease has to
+      # cover the delay as well or the sweep expires it mid-wait and promotes
+      # a second job for the same key.
+      def slot_lease(concurrency, delay)
+        Concurrency.effective_duration(concurrency[:duration]) + delay.to_i
+      end
+
+      # Hand the slot back only when the failure proves nothing was
+      # produced. On an ambiguous outcome the message may well be live, and
+      # releasing would admit a second job beside it — so the hold stands
+      # and the lease expiry plus the dispatcher's sweep reclaim it.
+      def send_holding_slot(key, queue, payload_hash, delay:, priority:)
+        Pgbus.client.send_message(queue, payload_hash, delay: delay, priority: priority)
+      rescue StandardError => e
+        if ambiguous_delivery?(e)
+          Pgbus.logger.warn do
+            "[Pgbus] Send outcome unknown for #{key}; holding its concurrency slot until the lease expires: #{e.message}"
+          end
+        else
+          begin
+            Concurrency::Semaphore.release(key)
+          rescue StandardError => release_error
+            Pgbus.logger.warn { "[Pgbus] Could not release concurrency slot after failed send: #{release_error.message}" }
+          end
+        end
+        raise
+      end
+
       def physical_queue(queue, priority)
         Pgbus.client.target_queue(queue, priority)
       end

data/lib/pgbus/active_job/executor.rb

@@ -96,13 +96,24 @@ module Pgbus
           # `batch` (and `batch.enqueue` for open batches) work inside a job.
           assign_batch_id(job, payload)
           Pgbus.logger.debug { "[Pgbus::Executor] running #{tag} job_class=#{job_class}" }
-          with_visibility_heartbeat(job, queue_name, msg_id, source_queue) { execute_job(job) }
+          with_visibility_heartbeat(job, queue_name, msg_id, source_queue, payload) { execute_job(job) }
           # retry_on re-enqueues from inside perform_now and returns normally:
           # this attempt is done (archive it) but the job is not — the retry
           # message carries the batch tag and signals on its own outcome.
           retried = Batch.retry_reenqueued?(payload["job_id"])
           Pgbus.logger.debug { "[Pgbus::Executor] perform_returned #{tag} job_class=#{job_class}" }
-          archive_from(queue_name, msg_id, source_queue: source_queue)
+          # Archiving is the exact-once claim on this execution.
+          # :already_archived means another worker archived the message (our
+          # heartbeat lapsed and it was redelivered): that worker owns the
+          # completion signals, and signalling again here would release the
+          # concurrency slot a second time (rails/solid_queue#761).
+          if archive_from(queue_name, msg_id, source_queue: source_queue) == :already_archived
+            Pgbus.logger.warn do
+              "[Pgbus::Executor] already archived elsewhere, skipping signals #{tag} job_class=#{job_class}"
+            end
+            release_duplicate_execution_lock(uniqueness_key, uniqueness_strategy, queue_name, msg_id)
+            return :duplicate
+          end
           Pgbus.logger.debug { "[Pgbus::Executor] archived #{tag} job_class=#{job_class}" }
           job_succeeded = true
           release_uniqueness_lock(uniqueness_key)
@@ -177,14 +188,18 @@ module Pgbus
       # Keep the message invisible while perform runs (see VisibilityHeartbeat).
       # Wraps only the perform: the heartbeat must be gone before archive or
       # the retry backoff touches the same message's VT.
-      def with_visibility_heartbeat(job, queue_name, msg_id, source_queue, &)
+      def with_visibility_heartbeat(job, queue_name, msg_id, source_queue, payload, &)
         klass = job.class
         per_job = klass.respond_to?(:pgbus_visibility_heartbeat_enabled) ? klass.pgbus_visibility_heartbeat_enabled : nil
         return yield if per_job == false
 
+        # The heartbeat also keeps the job's semaphore alive, so `duration`
+        # bounds silence, not run time.
+        concurrency_key = Concurrency.extract_key(payload)
         VisibilityHeartbeat.track(
           client: client, queue_name: source_queue || queue_name, prefixed: source_queue.nil?,
-          msg_id: msg_id, job_class: klass.name, config: config, &
+          msg_id: msg_id, job_class: klass.name, config: config,
+          concurrency: concurrency_key && [concurrency_key, Concurrency.config_for(klass)[:duration]], &
         )
       end
 
@@ -322,12 +337,10 @@ module Pgbus
         key = Concurrency.extract_key(payload)
         return unless key
 
-        # Atomic permit handoff: try to promote a blocked job first.
-        # promote_next wraps delete + enqueue in a transaction so neither is lost.
-        # If promoted, the slot stays occupied (no release needed).
-        # Only release the semaphore if there's nothing to promote.
-        promoted = Concurrency::BlockedExecution.promote_next(key, client: client)
-        Concurrency::Semaphore.release(key) unless promoted
+        # Release the slot and hand it to the next parked job in one
+        # transaction; the semaphore row lock orders this against a
+        # concurrent enqueue that is parking a job.
+        Concurrency::Semaphore.signal(key, client: client)
       rescue StandardError => e
         Pgbus.logger.warn { "[Pgbus] Concurrency signal failed: #{e.message}" }
       end
@@ -356,14 +369,26 @@ module Pgbus
       # that succeeded but failed to archive redelivers after VT expiry and
       # runs twice. If the retry also fails, fall through to the normal
       # failure path (recorded failure + VT-based redelivery).
+      # Returns :archived, :already_archived, or :ambiguous.
+      #
+      # A `false` reported after our OWN retry is ambiguous, not a duplicate:
+      # the first archive may well have committed with only its reply lost to
+      # the connection error that triggered the retry. Claiming it is the
+      # safe reading — the alternative strands this job's concurrency slot,
+      # batch and uniqueness state on every such blip, whereas a genuine
+      # duplicate would additionally require another worker to have taken and
+      # finished the same message inside the retry window.
       def archive_from(queue_name, msg_id, source_queue: nil)
         attempts = 0
         begin
-          if source_queue
-            client.archive_message(source_queue, msg_id, prefixed: false)
-          else
-            client.archive_message(queue_name, msg_id)
-          end
+          archived = if source_queue
+                       client.archive_message(source_queue, msg_id, prefixed: false)
+                     else
+                       client.archive_message(queue_name, msg_id)
+                     end
+          return :archived unless archived == false
+
+          attempts.positive? ? :ambiguous : :already_archived
         rescue StandardError => e
           attempts += 1
           raise unless attempts == 1 && connection_error?(e)
@@ -375,6 +400,21 @@ module Pgbus
         end
       end
 
+      # A :while_executing lock is bound to THIS message, so an execution
+      # that loses the archive race still has to hand its own lock back —
+      # conditionally, so it can never delete a successor's row. An
+      # :until_executed lock belongs to the job as a whole and is released by
+      # the worker that actually archived it.
+      def release_duplicate_execution_lock(uniqueness_key, uniqueness_strategy, queue_name, msg_id)
+        return unless uniqueness_key && uniqueness_strategy == :while_executing
+
+        # Same queue the execution lock was acquired under, so the match
+        # cannot land on another queue's message of the same id.
+        UniquenessKey.release_if_bound!(uniqueness_key, queue_name: queue_name, msg_id: msg_id)
+      rescue StandardError => e
+        Pgbus.logger.warn { "[Pgbus] Duplicate-execution lock release failed: #{e.message}" }
+      end
+
       def connection_error?(error)
         defined?(PGMQ::Errors::ConnectionError) && error.is_a?(PGMQ::Errors::ConnectionError)
       end

data/lib/pgbus/concurrency/blocked_execution.rb

@@ -7,11 +7,20 @@ module Pgbus
     module BlockedExecution
       class << self
         # Insert a blocked execution for a job that hit the concurrency limit.
+        #
+        # `expires_at` is a re-check hint for the sweep's ordering only — a
+        # parked job is never deleted for being old. The only way out of the
+        # table is promotion.
+        #
+        # The payload goes in as a Hash: the jsonb attribute serializes it
+        # once. Handing it a pre-serialized String stored a JSON *string*
+        # (double-encoded), which every document reader of the column —
+        # `payload->>'job_id'`, the job-class lookup, `scheduled_at` — misread.
         def insert(concurrency_key:, queue_name:, payload:, duration:, priority: 0)
           Pgbus::BlockedExecution.create!(
             concurrency_key: concurrency_key,
             queue_name: queue_name,
-            payload: JSON.generate(payload),
+            payload: payload,
             priority: priority,
             expires_at: Time.current + duration
           )
@@ -23,42 +32,53 @@ module Pgbus
           Pgbus::BlockedExecution.release_next!(concurrency_key)
         end
 
-        # Atomically promote the next blocked execution: delete the row and enqueue
-        # the job in a single transaction. Returns true if a job was promoted, false
-        # otherwise. This avoids losing a blocked row if enqueue fails.
+        # Atomically promote the next blocked execution: delete the row, take a
+        # semaphore slot for it and enqueue the job in a single transaction.
+        # Returns true if a job was promoted, false otherwise.
+        #
+        # The slot is taken through the same guarded upsert an enqueue uses,
+        # so a promotion can never push the key past its limit; when no slot
+        # is free the savepoint rolls back and the row stays parked. Runs as a
+        # savepoint so `Semaphore.signal` can wrap it with the release.
         def promote_next(concurrency_key, client:, delay: 0)
           released = nil
           msg_id = nil
-          Pgbus::BlockedExecution.transaction do
+          Pgbus::BlockedExecution.transaction(requires_new: true) do
             released = release_next(concurrency_key)
             raise ActiveRecord::Rollback unless released
 
+            # Resolve the delay first: a parked scheduled job goes back into
+            # PGMQ invisible, and nothing renews its lease until a worker
+            # picks it up, so the wait has to be part of the lease.
             actual_delay = resolve_delay(released[:payload], delay)
+            raise ActiveRecord::Rollback unless slot_taken?(concurrency_key, released[:payload], actual_delay)
+
             # Carry the enqueuer's priority through: under priority routing it
             # picks the _pN sub-queue, not just the release order (issue #423).
             msg_id = client.send_message(released[:queue_name], released[:payload],
                                          delay: actual_delay, priority: released[:priority])
           end
 
-          if released && msg_id
-            begin
-              Batch.backfill_execution(released[:payload], msg_id,
-                                       client.target_queue(released[:queue_name], released[:priority]))
-            rescue StandardError => e
-              Pgbus.logger.warn { "[Pgbus] Batch execution backfill failed after promote: #{e.message}" }
-            end
-          end
+          return false unless released && msg_id
 
-          !!released
+          backfill(released, msg_id, client)
+          true
         rescue StandardError => e
           Pgbus.logger.warn { "[Pgbus] Promote blocked execution failed for #{concurrency_key}: #{e.message}" }
           false
         end
 
-        # Delete blocked executions that have expired.
-        # Returns the count of deleted rows.
-        def expire_stale
-          Pgbus::BlockedExecution.expired(Time.current).delete_all
+        # Sweep: promote every parked job that can take a slot right now.
+        # Covers what the completion-time signal cannot — a holder that died
+        # (its semaphore expired and was swept) or a promote that failed.
+        # Returns the number of jobs promoted.
+        def promote_pending(client:, per_key: 100)
+          Pgbus::BlockedExecution.repair_double_encoded!
+          Pgbus::BlockedExecution.promotable_keys.sum do |key|
+            promoted = 0
+            promoted += 1 while promoted < per_key && promote_next(key, client: client)
+            promoted
+          end
         end
 
         # Count blocked executions for a given key. Useful for testing/monitoring.
@@ -68,6 +88,27 @@ module Pgbus
 
         private
 
+        # The batch execution row is bookkeeping, not the promotion. Give it
+        # its own savepoint: `Semaphore.signal` calls promote_next inside a
+        # transaction, and a database error out here would poison that
+        # transaction — the commit then fails, un-deleting the parked row and
+        # un-taking the slot while the message is already live, so the job
+        # runs a second time.
+        def backfill(released, msg_id, client)
+          Pgbus::BlockedExecution.transaction(requires_new: true) do
+            Batch.backfill_execution(released[:payload], msg_id,
+                                     client.target_queue(released[:queue_name], released[:priority]))
+          end
+        rescue StandardError => e
+          Pgbus.logger.warn { "[Pgbus] Batch execution backfill failed after promote: #{e.message}" }
+        end
+
+        def slot_taken?(concurrency_key, payload, delay = 0)
+          config = Concurrency.config_for_payload(payload)
+          expires_at = Time.current + Concurrency.effective_duration(config[:duration]) + delay.to_i
+          Pgbus::Semaphore.acquire!(concurrency_key, config[:limit], expires_at) == :acquired
+        end
+
         def resolve_delay(payload, default_delay)
           scheduled_at = payload["scheduled_at"]
           return default_delay unless scheduled_at

data/lib/pgbus/concurrency/semaphore.rb

@@ -6,6 +6,12 @@ module Pgbus
       class << self
         # Attempt to acquire a slot in the semaphore for the given key.
         # Returns :acquired if a slot was available, :blocked if the limit is reached.
+        #
+        # The upsert takes the semaphore row lock until the surrounding
+        # transaction commits — even when it returns :blocked. The adapter
+        # relies on that: parking the job inside the same transaction means a
+        # holder that signals concurrently waits on `release` below and then
+        # sees the parked row (rails/solid_queue#712).
         def acquire(key, max_value, duration)
           expires_at = Time.current + duration
           Pgbus::Semaphore.acquire!(key, max_value, expires_at)
@@ -16,6 +22,34 @@ module Pgbus
           Pgbus::Semaphore.where(key: key).update_all("value = GREATEST(value - 1, 0)")
         end
 
+        # A job holding a slot is done: give the slot back and hand it to the
+        # next parked job for this key, in one transaction. The decrement locks
+        # the semaphore row first, so an enqueue that is parking a job right
+        # now commits before the promote looks for parked rows.
+        def signal(key, client:)
+          Pgbus::Semaphore.transaction do
+            release(key)
+            BlockedExecution.promote_next(key, client: client)
+          end
+        end
+
+        # Push the semaphore's expiry out while a holder is still running.
+        # Driven by the visibility heartbeat, so `duration` is the longest a
+        # holder may go silent before its slot is presumed dead — not a cap on
+        # how long a job may run.
+        def touch(key, duration)
+          # Floored here as well as at acquire: renewing for a raw duration
+          # shorter than the gap to the next beat would let the lease lapse
+          # mid-run and the sweep promote beside a running job.
+          # Start the lease once the connection is in hand: waiting on a busy
+          # pool would otherwise be charged against the renewal.
+          floored = Concurrency.effective_duration(duration)
+          Pgbus::Semaphore.connection_pool.with_connection do
+            Pgbus::Semaphore.where(key: key)
+                            .update_all(["expires_at = GREATEST(expires_at, ?)", Time.current + floored])
+          end
+        end
+
         # Delete semaphores that have expired (safety net for crashed workers).
         # Returns an array of hashes with expired keys.
         # Uses DELETE ... RETURNING for atomicity (no race between pluck and delete).

data/lib/pgbus/concurrency.rb

@@ -7,6 +7,10 @@ module Pgbus
     extend ActiveSupport::Concern
 
     METADATA_KEY = "pgbus_concurrency_key"
+    # How long a slot holder may go silent (no visibility heartbeat) before
+    # its slot is presumed dead. Also the limit/duration used to promote a
+    # parked job whose class no longer resolves.
+    DEFAULT_DURATION = 15 * 60
 
     class_methods do
       # Limit concurrent execution of jobs with the same key.
@@ -20,9 +24,12 @@ module Pgbus
       #                Default: the ENQUEUED job's class name (resolved at
       #                resolve time, so an inherited declaration keys each
       #                subclass separately — issue #357).
-      #   duration:    Safety expiry for semaphore (default: 15 minutes)
+      #   duration:    How long a running holder may go without a visibility
+      #                heartbeat before its slot is presumed dead and swept
+      #                (default: 15 minutes). Not a cap on run time: the
+      #                heartbeat keeps the semaphore alive while the job runs.
       #   on_conflict: What to do when limit is reached — :block, :discard, or :raise (default: :block)
-      def limits_concurrency(to:, key: nil, duration: 15 * 60, on_conflict: :block) # rubocop:disable Naming/MethodParameterName
+      def limits_concurrency(to:, key: nil, duration: DEFAULT_DURATION, on_conflict: :block) # rubocop:disable Naming/MethodParameterName
         raise ArgumentError, "to: must be a positive integer" unless to.is_a?(Integer) && to.positive?
         raise ArgumentError, "on_conflict must be :block, :discard, or :raise" unless %i[block discard raise].include?(on_conflict)
         raise ArgumentError, "duration must be a positive number" unless duration.is_a?(Numeric) && duration.positive?
@@ -71,6 +78,33 @@ module Pgbus
       def extract_key(payload)
         payload[METADATA_KEY]
       end
+
+      # Limit and duration for a job class. A class with no concurrency
+      # config — or one that no longer resolves — gets `limit: nil`, meaning
+      # "whatever limit the semaphore row already records". Forcing 1 here
+      # would refuse every promotion for a `to: 3` key that still holds two
+      # slots, so its parked jobs could never reach the executor (which is
+      # what dead-letters a missing class).
+      def config_for(job_class)
+        config = job_class.respond_to?(:pgbus_concurrency) && job_class.pgbus_concurrency
+        return { limit: nil, duration: DEFAULT_DURATION } unless config
+
+        { limit: config[:limit], duration: config[:duration] }
+      end
+
+      # A slot's lease is only renewed by the visibility heartbeat, which
+      # first beats one interval after the job starts. A `duration` shorter
+      # than that would expire before the first touch and let the sweep
+      # promote beside a running job, so it is floored at two intervals.
+      def effective_duration(duration, config: Pgbus.configuration)
+        [duration, config.effective_visibility_heartbeat_interval * 2].max
+      end
+
+      def config_for_payload(payload)
+        config_for(Object.const_get(payload["job_class"].to_s))
+      rescue NameError
+        config_for(nil)
+      end
     end
   end
 end

data/lib/pgbus/process/dispatcher.rb

@@ -355,25 +355,18 @@ module Pgbus
       end
 
       def cleanup_concurrency
-        expired_keys = Concurrency::Semaphore.expire_stale
-        expired_keys.each do |row|
-          release_blocked_for_key(row["key"])
-        end
+        expired = Concurrency::Semaphore.expire_stale
+        Pgbus.logger.debug { "[Pgbus] Swept #{expired.size} expired semaphores" } if expired.any?
 
-        orphaned = Concurrency::BlockedExecution.expire_stale
-        Pgbus.logger.debug { "[Pgbus] Expired #{orphaned} orphaned blocked executions" } if orphaned.positive?
+        # A parked job is never dropped: the sweep only promotes what can take
+        # a slot now — behind a swept semaphore or a promote that failed.
+        promoted = Concurrency::BlockedExecution.promote_pending(client: Pgbus.client)
+        Pgbus.logger.debug { "[Pgbus] Promoted #{promoted} blocked executions" } if promoted.positive?
       rescue StandardError => e
         log_maintenance_failure("Concurrency cleanup", e)
         e
       end
 
-      def release_blocked_for_key(key)
-        promoted = Concurrency::BlockedExecution.promote_next(key, client: Pgbus.client)
-        Pgbus.logger.debug { "[Pgbus] Released blocked execution for key: #{key}" } if promoted
-      rescue StandardError => e
-        log_maintenance_failure("Releasing blocked execution for #{key}", e)
-      end
-
       def cleanup_batches
         retention = config.batch_retention
         return unless retention&.positive?

data/lib/pgbus/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Pgbus
-  VERSION = "0.16.3"
+  VERSION = "0.16.4"
 end

data/lib/pgbus/visibility_heartbeat.rb

@@ -24,8 +24,10 @@ module Pgbus
   # cadence with `config.visibility_heartbeat_interval`, or opt a job class
   # out with `pgbus_visibility_heartbeat false`.
   module VisibilityHeartbeat
+    # `concurrency` is `[key, duration]` for a concurrency-limited job, else
+    # nil — one member, so the struct stays inside an 80-byte slot.
     Entry = Struct.new(:client, :queue_name, :prefixed, :msg_id, :job_class, :extended_at, :extensions,
-                       keyword_init: true)
+                       :concurrency, keyword_init: true)
 
     # Per-job opt-out, included on ActiveJob::Base by the engine:
     #
@@ -55,11 +57,15 @@ module Pgbus
       # @param prefixed [Boolean] whether queue_name still needs the prefix
       # @param job_class [String, nil] for logging and instrumentation
       # @param config [Pgbus::Configuration]
-      def track(client:, queue_name:, msg_id:, prefixed: true, job_class: nil, config: Pgbus.configuration)
+      # @param concurrency [Array(String, Numeric), nil] semaphore key to keep alive alongside
+      #   the message and how far to push its expiry on each beat
+      def track(client:, queue_name:, msg_id:, prefixed: true, job_class: nil, config: Pgbus.configuration,
+                concurrency: nil)
         return yield unless config.visibility_heartbeat
 
         entry = Entry.new(client: client, queue_name: queue_name, prefixed: prefixed, msg_id: msg_id.to_i,
-                          job_class: job_class, extended_at: monotonic_now, extensions: 0)
+                          job_class: job_class, extended_at: monotonic_now, extensions: 0,
+                          concurrency: concurrency)
         register(entry, config)
         begin
           yield
@@ -123,6 +129,7 @@ module Pgbus
         entry.client.set_visibility_timeout(entry.queue_name, entry.msg_id, vt: vt, prefixed: entry.prefixed)
         entry.extended_at = now
         entry.extensions += 1
+        touch_semaphore(entry)
         Instrumentation.instrument(
           "pgbus.job_visibility_extended",
           queue: entry.queue_name, job_class: entry.job_class, msg_id: entry.msg_id, vt: vt,
@@ -165,6 +172,20 @@ module Pgbus
 
       # Entries registered before a fork belong to the parent's jobs; the
       # thread did not survive the fork either.
+      # A live holder keeps its semaphore from being swept, so `duration`
+      # bounds heartbeat silence rather than run time. Its own rescue: a
+      # failed touch must not cost the message its visibility extension.
+      def touch_semaphore(entry)
+        key, duration = entry.concurrency
+        return unless key
+
+        Concurrency::Semaphore.touch(key, duration || Concurrency::DEFAULT_DURATION)
+      rescue StandardError => e
+        Pgbus.logger.warn do
+          "[Pgbus::VisibilityHeartbeat] could not touch semaphore #{key}: #{e.class}: #{e.message}"
+        end
+      end
+
       def forget_parent_entries!
         return if @pid == ::Process.pid
 

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: pgbus
 version: !ruby/object:Gem::Version
-  version: 0.16.3
+  version: 0.16.4
 platform: ruby
 authors:
 - Mikael Henriksson