pgbus 0.21.0 → 0.21.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 1c3c6921f17cfc002f7177240a3d1b2c202a78c7127e8f0da75a75ebcaf365d1
-  data.tar.gz: b73396771876a230b1b62b6b30dceb87329be7259810c5346fd4a3fc78f59991
+  metadata.gz: c2dece606e269b4f406a3ba3d786facbdd53aedd675fda239803c5136e241678
+  data.tar.gz: 8498b9396aa4577f18b5e2e7793e82e9a67ce896b7495f061d51bd28e56112a1
 SHA512:
-  metadata.gz: 5667ec2a6ebd1ef9e09f5f89818e69784267b3aa0c13a0b5cef2782b346af2361c106454fbb58d67a688b48f451483327fa4f3edc0a1d9a82664aa599af52454
-  data.tar.gz: 83d7a44001fe09db593caacc788307198361279c023b7ed2ed8397fdec17fb353fa44087d9a973857951835d6cb1206314ade872a99d85586988561be2477a85
+  metadata.gz: 7032434023b3253d81e3c6668a4db3481a500a1421cce80fe94a45f235a244be23da1b0e25ec3657acbaaee4ba11c17e56891d4c401010f22340ae5635f0f55a
+  data.tar.gz: 76e5a0d5dd1343274848660e7ba2b6389b14c955551117b1b9a0821a994960ece32a46e6bedfccd4f2e9ddae3326aa37e3f11cfd3bea70edf78a76c3c6e3961f

data/CHANGELOG.md

@@ -1,5 +1,24 @@
 ## [Unreleased]
 
+### Added
+
+- **`config.boot_timeout` (default 120 seconds): the supervisor kills a child that never checks in (issue #587).** A fork checks in when it registers its `pgbus_processes` row on this host or writes its first liveness byte. One that has done neither `boot_timeout` seconds after the fork is logged at ERROR, sent `TTIN`, given 2 seconds, then SIGKILLed, and the normal restart path forks it again. A boot-deadline kill always counts as a crash, whatever the child's uptime, so a boot that hangs every time is re-forked with the crash-loop backoff (1 s doubling to 60 s) instead of every `boot_timeout`. Applies to workers, consumers, the dispatcher, the scheduler and the outbox poller. `nil` turns it off; anything else must be a positive number, or `validate!` raises `ConfigurationError`. When the watchdog's read of the process table fails, the boot deadline waits for a pass that can read it. Refs #587.
+- **A process whose registration in `pgbus_processes` failed at start retries it on each heartbeat (issue #587).** It used to stay without a row for good after one failed `create!` (a database blip at boot); a dispatcher, scheduler or outbox poller in that state would now be killed by the boot deadline even though it runs. Refs #587.
+- **`kill -TTIN <child pid>` logs every thread of a pgbus child with its backtrace at ERROR; the child keeps running (issue #587).** The watchdog sends it before every SIGKILL (boot timeout or stalled loop), so a hang is diagnosable from the log. Best effort: a child blocked in native code holding the GVL does not answer. Refs #587.
+- **The supervisor's standalone `/readyz` body carries `forked` (issue #587).** `forked` is the fork-table size (what `live` used to be); `live` now counts the children that have checked in. `Pgbus::Process::ReadinessSnapshot` gains `forked:`, defaulting to `live`. Refs #587.
+
+### Changed
+
+- **`pgbus_failed_events` gains a `source_queue` column; run `rails generate pgbus:update` (or `pgbus:add_failed_events_source_queue`) and migrate (issue #583).** The migration adds the column, creates the unique index `idx_pgbus_failed_events_queue_msg_source` on `(queue_name, msg_id, source_queue)` and drops `idx_pgbus_failed_events_queue_msg` on `(queue_name, msg_id)`. Until it runs, pgbus logs one warning per process and records failures as before (one row per logical queue and msg_id); a running process checks for the column again every minute, so it needs no restart after the migration. Fresh `pgbus:install` creates the new shape. `pgbus:update` no longer queues `pgbus:add_failed_events_index`, which would put the old index back, and queues the new migration again if the old index is present; `pgbus:add_failed_events_index` run by name adds nothing once the new index exists. `Pgbus::FailedEventRecorder.record!`, `.exists?`, `.last_error` and `.clear!` take a required `source_queue:` keyword, the physical table the message was read from. Refs #583.
+
+### Fixed
+
+- **A child that hangs before its first liveness signal is killed and forked again, and `/readyz` no longer counts it as live (issue #587).** A fork stuck between `fork` and its first check-in (in an initializer, the eager load or a connection) never exited, so nothing reaped it, and the watchdog skipped it because no liveness channel had spoken; the container-local `/readyz` counted it as live because it was in the fork table, so a rolling deploy passed its health gate with a dead capsule. The boot deadline above now kills it, and `/readyz` stays at 503 `DEGRADED` (`live < expected`) until every child has checked in. A dispatcher, scheduler or outbox poller checks in through its process row, which the watchdog reads every 10 seconds, so `/readyz` turns 200 up to 10 seconds after the last one registers, and stays at 503 while the database is unreachable at boot (those children cannot be seen to be alive). The watchdog reads only this host's process rows last beaten since the fork, so a row another container wrote under the same pid, or one a SIGKILLed predecessor left behind (a container restarted in place reuses its hostname and pids), no longer vouches for a local child or hides a stalled loop. Refs #587.
+- **A failed job's retry backoff applies under priority routing (issue #582).** With `priority_levels > 1` the executor set the backoff's visibility timeout on the logical queue's base table, which priority routing never creates; the call raised, was logged at debug, and the job came back after the read's visibility timeout instead of its backoff delay. The backoff now goes to the `_pN` sub-queue the worker read the message from, as the visibility heartbeat and the archive already did. Refs #582.
+- **Two priority sub-queues' failures with the same msg_id keep a row each (issue #583).** Each `_pN` sub-queue numbers its messages from its own sequence, and `pgbus_failed_events` was unique on the logical queue and the msg_id, so a second job failing with the same msg_id in another sub-queue overwrote the first one's error and kept its payload: the dashboard showed one failed job where two had failed, Retry and Discard acted on the first job's message and deleted the only row, and the first job's success cleared the second one's failure. Rows now record the physical table (`source_queue`) and are unique on `(queue_name, msg_id, source_queue)`. The executor, the event consumer and the worker's zombie check read, write and clear only the row of the table the message came from. On the Jobs page a failed row joins only its own table's message and shows as an orphan once that message is gone, Retry, Discard and Discard all look the message up in the table the row names, and Discard releases the job's uniqueness lock bounded by when that table's message was sent, not by a sibling sub-queue's. A row written before the migration has no `source_queue` and is treated as before: it can belong to any table of its queue. Refs #583.
+
+- **The dashboard's Retry, Discard and Discard all reach a failed job's message under priority routing (issue #577).** A failed-job row records the logical queue only, and with `priority_levels > 1` the message sits in a `_pN` sub-queue, never in the base table. Retry reset the visibility timeout on `pgmq.q_pgbus_<queue>`, which priority routing never creates, and answered "Could not retry job."; Discard and Discard all archived from that same table, so they reported the job discarded while its message stayed queued and ran again. Each sub-queue also numbers its messages from its own sequence, so the msg_id alone could name another job's message in a sibling sub-queue. The dashboard now finds the sub-queue whose message has the row's msg_id and carries the job's `job_id` (`Pgbus::Client#locate_message`, which returns that physical table or nil, and `#locate_messages` for many at once, one query per table), resets or archives the message there, and leaves a same-numbered message elsewhere alone. Discard all locates a queue's messages together and archives them in one batch per table. When no sub-queue holds it, Retry enqueues a fresh copy as before. Without priority routing there is one table and nothing changes, except that a message reusing the msg_id under another `job_id` is no longer reset or archived in the job's place. Refs #577.
+
 ## [0.21.0] - 2026-10-19
 
 ### Added
@@ -11,11 +30,15 @@
 ### Changed
 
 - **`actor:` and `reason:` are reserved keywords of `Pgbus::Workflow.start!` (issue #571).** Like `idempotency_key:`, they are taken for the audit trail and no longer reach the run's `arguments`. A workflow that receives an argument literally named `actor` or `reason` through `start!` renames it, e.g. `start!(subject, requested_by: "alice")` read as `arguments[:requested_by]`. Runs already started keep their stored arguments, and `rerun!` copies them as stored. Refs #571.
+- **`Pgbus::Client#message_exists?`, `#message_in_queue?`, `#message_archived?`, `#message_with_job_id?` and `#dead_letter_physical_name` take a logical queue name unless you pass `prefixed: false` (issue #569).** They used to guess that a name starting with `"#{queue_prefix}_"` was already physical. **If you call one of them with a physical table name, add `prefixed: false`;** without it the name is prefixed again. Under priority routing two of them also answer differently for a logical name: `message_archived?` probes every `_pN` archive (it probed a base table that priority routing never creates, and answered nil), and `dead_letter_physical_name` keeps a logical name's own `_pN` suffix (`orders_p2` → `…_orders_p2_dlq`, the queue `move_to_dead_letter` writes), stripping it only from a physical sub-queue name passed with `prefixed: false`.
+- **The batch completion callback is `on_complete`.** `Pgbus::Batch.new(on_complete:)`, the builder's `b.on_complete` and the `#on_complete` reader replace `on_finish`, which keeps working as a deprecated alias until 1.0 and logs one warning per process. The database columns stay `on_finish_class` / `on_finish_job` until the 1.0 schema cleanup, so batches created before upgrading still fire, and the `pgbus.batch_finished` instrumentation event keeps its name. The dashboard's batch page labels the callback "On Complete". Refs #538.
+
 
 ### Fixed
 
 - **The workflows one-ongoing index predicate dumps the same `schema.rb` line from a migrated and from a schema-loaded database (issue #573).** `pgbus:add_workflows` and the fresh-install migration declared `idx_pgbus_workflows_one_ongoing` with `where: "status IN ('ready','performing','waiting','paused')"`. `status` is a `varchar`, so Postgres stored the list as `varchar` literals cast to `text[]`; `db:schema:load` parses that dumped string into a different expression, and each later dump flipped the line depending on which kind of database it came from. Both templates now write `status::text IN (…)`, whose stored form replays to itself. The index means the same either way, so existing installs need nothing. An app that wants a stable dump can recreate the index in a migration of its own: `remove_index :pgbus_workflows, name: "idx_pgbus_workflows_one_ongoing"`, then `add_index :pgbus_workflows, [:workflow_class, :subject_type, :subject_id], unique: true, where: "status::text IN ('ready','performing','waiting','paused')", name: "idx_pgbus_workflows_one_ongoing"` (both run in the migration's transaction, so the constraint never lapses; `remove_index` takes an ACCESS EXCLUSIVE lock, so `pgbus_workflows` is locked against reads and writes (workers, workflow starts and the dashboard wait) until the migration commits), and dump the schema again. Refs #573.
 - **A guard that ends a `wait_for:` step after a reattempt consumes the signal the step holds (issue #572).** A woken attempt that called `reattempt!` (or failed into a pause and was resumed) kept its signal `claimed` for the next attempt, whose row carries no `signal_id` until its gate runs. When `cancel_if` or `skip_if` ended that next attempt (or a guard raised and its `on_exception` policy skipped or cancelled, or a deploy had declared the step a `removed_step` meanwhile), the signal stayed `claimed` forever. The attempt now records the held signal before its outcome, so it is consumed when the step ends and kept when it reattempts or pauses, as for any attempt. Refs #572.
+- **A queue whose logical name starts with the prefix is probed in its own table (issue #569).** `Pgbus::Client#message_exists?`, `#message_in_queue?`, `#message_archived?`, `#message_with_job_id?` and `#dead_letter_physical_name` decided whether a name was already physical by `name.start_with?("#{queue_prefix}_")`, so a logical `pgbus_mailers` (table `pgbus_pgbus_mailers`) was probed as `pgmq.q_pgbus_mailers`. The answer was "unknown", or another queue's answer when that table existed. The dispatcher kept an orphaned uniqueness lock under such a queue forever. The dashboard's failed-event Retry enqueued a copy next to the original message. The batch sweep kept an orphan execution row forever, or un-counted a dead-lettered job. The name's spelling no longer decides anything: these methods take a logical name, expanded through `config.queue_name` and the queue strategy, unless the caller passes `prefixed: false` for a physical table name. This is the `prefixed:` convention `delete_message`, `archive_message`, `set_visibility_timeout`, `purge_queue` and `drop_queue` already follow. `message_archived?` now also expands a logical name to every `_pN` archive under priority routing, as `message_exists?` already did; before, it probed a base table that does not exist there. The batch sweep now passes `prefixed: false` for a backfilled execution row (it holds the table its message went to) and the logical name for an orphan row. The caller-facing change is under Changed.
 
 ## [0.20.1] - 2026-10-18
 

data/lib/generators/pgbus/add_failed_events_source_queue_generator.rb

@@ -0,0 +1,44 @@
+# frozen_string_literal: true
+
+require "rails/generators"
+require "rails/generators/active_record"
+require_relative "migration_path"
+
+module Pgbus
+  module Generators
+    class AddFailedEventsSourceQueueGenerator < Rails::Generators::Base
+      include ActiveRecord::Generators::Migration
+      include MigrationPath
+
+      source_root File.expand_path("templates", __dir__)
+
+      desc "Add source_queue to pgbus_failed_events so each priority sub-queue's failure keeps its own row"
+
+      class_option :database,
+                   type: :string,
+                   default: nil,
+                   desc: "Use a separate database for pgbus tables (e.g. --database=pgbus)"
+
+      def create_migration_file
+        migration_template "add_failed_events_source_queue.rb.erb",
+                           File.join(pgbus_migrate_path, "add_pgbus_failed_events_source_queue.rb")
+      end
+
+      def display_post_install
+        say ""
+        say "Pgbus failed events source_queue added!", :green
+        say ""
+        say "Next steps:"
+        say "  1. Run: rails db:migrate#{migrate_command_suffix}"
+        say "  2. Restart pgbus processes so they record failures per physical queue"
+        say ""
+      end
+
+      private
+
+      def migration_version
+        "[#{ActiveRecord::Migration.current_version}]"
+      end
+    end
+  end
+end

data/lib/generators/pgbus/templates/add_failed_events_source_queue.rb.erb

@@ -0,0 +1,27 @@
+class AddPgbusFailedEventsSourceQueue < ActiveRecord::Migration<%= migration_version %>
+  # Under priority routing each _pN sub-queue numbers its messages from its
+  # own sequence, so two jobs can fail with the same msg_id under one logical
+  # queue. source_queue names the physical table the message was read from,
+  # and the failure upsert is keyed on it. Rows written before this keep
+  # NULL there: pgbus reads them as belonging to any table of their queue.
+  def up
+    add_column :pgbus_failed_events, :source_queue, :string, if_not_exists: true
+    add_index :pgbus_failed_events, %i[queue_name msg_id source_queue],
+      unique: true, name: "idx_pgbus_failed_events_queue_msg_source", if_not_exists: true
+    remove_index :pgbus_failed_events, name: "idx_pgbus_failed_events_queue_msg", if_exists: true
+  end
+
+  # (queue_name, msg_id) can repeat once source_queue is in use: the latest
+  # row of each pair is kept so the old unique index can be built again.
+  def down
+    execute <<~SQL
+      DELETE FROM pgbus_failed_events older
+      USING pgbus_failed_events newer
+      WHERE older.queue_name = newer.queue_name AND older.msg_id = newer.msg_id AND older.id < newer.id
+    SQL
+    add_index :pgbus_failed_events, %i[queue_name msg_id],
+      unique: true, name: "idx_pgbus_failed_events_queue_msg", if_not_exists: true
+    remove_index :pgbus_failed_events, name: "idx_pgbus_failed_events_queue_msg_source", if_exists: true
+    remove_column :pgbus_failed_events, :source_queue, if_exists: true
+  end
+end

data/lib/generators/pgbus/templates/add_failed_events_unique_index.rb.erb

@@ -1,5 +1,10 @@
 class AddPgbusFailedEventsUniqueIndex < ActiveRecord::Migration<%= migration_version %>
   def change
+    # Superseded by pgbus:add_failed_events_source_queue's (queue_name, msg_id,
+    # source_queue) index: this one would make two priority sub-queues'
+    # failures with the same msg_id share one row again.
+    return if index_name_exists?(:pgbus_failed_events, "idx_pgbus_failed_events_queue_msg_source")
+
     add_index :pgbus_failed_events, [:queue_name, :msg_id],
       unique: true, name: "idx_pgbus_failed_events_queue_msg",
       if_not_exists: true

data/lib/generators/pgbus/templates/migration.rb.erb

@@ -53,9 +53,12 @@ class CreatePgbusTables < ActiveRecord::Migration<%= migration_version %>
     add_index :pgbus_processes, :last_heartbeat_at,
       name: "idx_pgbus_processes_heartbeat"
 
-    # Failed events log (for dashboard visibility)
+    # Failed events log (for dashboard visibility). queue_name is the logical
+    # queue, source_queue the physical table the message was read from: under
+    # priority routing each _pN sub-queue numbers its messages on its own.
     create_table :pgbus_failed_events do |t|
       t.string :queue_name, null: false
+      t.string :source_queue
       t.bigint :msg_id
       t.jsonb :payload
       t.jsonb :headers
@@ -68,8 +71,8 @@ class CreatePgbusTables < ActiveRecord::Migration<%= migration_version %>
 
     add_index :pgbus_failed_events, :queue_name, name: "idx_pgbus_failed_events_queue"
     add_index :pgbus_failed_events, :failed_at, name: "idx_pgbus_failed_events_time"
-    add_index :pgbus_failed_events, [:queue_name, :msg_id],
-      unique: true, name: "idx_pgbus_failed_events_queue_msg"
+    add_index :pgbus_failed_events, [:queue_name, :msg_id, :source_queue],
+      unique: true, name: "idx_pgbus_failed_events_queue_msg_source"
 
     # Concurrency semaphores (counting locks for job concurrency limits)
     create_table :pgbus_semaphores do |t|

data/lib/pgbus/active_job/executor.rb

@@ -28,9 +28,11 @@ module Pgbus
         if read_count > config.max_retries
           # Read the last error before clear! deletes the only row that has
           # it: the DLQ copy carries it in its headers (issue #495).
-          last_error = FailedEventRecorder.last_error(queue_name: queue_name, msg_id: message.msg_id.to_i)
+          last_error = FailedEventRecorder.last_error(queue_name: queue_name, msg_id: message.msg_id.to_i,
+                                                      source_queue: physical_queue(queue_name, source_queue))
           handle_dead_letter(message, queue_name, payload, source_queue: source_queue, last_error: last_error)
-          FailedEventRecorder.clear!(queue_name: queue_name, msg_id: message.msg_id.to_i)
+          FailedEventRecorder.clear!(queue_name: queue_name, msg_id: message.msg_id.to_i,
+                                     source_queue: physical_queue(queue_name, source_queue))
           signal_concurrency(payload)
           signal_batch_discarded(payload)
           release_uniqueness_lock(Uniqueness.extract_key(payload), queue_name, message)
@@ -129,7 +131,10 @@ module Pgbus
           # failed attempt, so a first delivery has no row: skip the per-job
           # DELETE round trip (issue #484). A stale row from a dropped-and-
           # recreated queue reusing this msg_id is the dispatcher's to sweep.
-          FailedEventRecorder.clear!(queue_name: queue_name, msg_id: msg_id) if read_count > 1
+          if read_count > 1
+            FailedEventRecorder.clear!(queue_name: queue_name, msg_id: msg_id,
+                                       source_queue: physical_queue(queue_name, source_queue))
+          end
         end
 
         instrument("pgbus.job_completed", queue: queue_name, job_class: job_class)
@@ -145,7 +150,7 @@ module Pgbus
         # Before this, a fiber interruption between perform_now and archive_from
         # silently lost control flow — no failed event row, no job_failed
         # notification, uniqueness lock held until VT expired. See issue #126.
-        handle_failure(message, queue_name, e, payload: payload)
+        handle_failure(message, queue_name, e, payload: payload, source_queue: source_queue)
         # A failed :while_executing attempt is no longer executing — release
         # so the retry (same message, after VT) can acquire. :until_executed
         # keeps its lock until success/DLQ by design (#126, #333).
@@ -188,6 +193,14 @@ module Pgbus
         "msg_id=#{message.msg_id} queue=#{queue_name} read_ct=#{message.read_ct}"
       end
 
+      # The table the message was read from: the _pN sub-queue under priority
+      # routing, else the logical queue's own. Each table numbers its messages
+      # from its own sequence, so a failed-event row is keyed on it (issue
+      # #583). Built only on the failure and redelivery paths.
+      def physical_queue(queue_name, source_queue)
+        source_queue || config.queue_name(queue_name)
+      end
+
       def assign_batch_id(job, payload)
         batch_id = payload[Batch::METADATA_KEY]
         return unless batch_id && job.respond_to?(:batch_id=)
@@ -298,7 +311,7 @@ module Pgbus
         [((Time.now.utc - enqueued_at) * 1000).round, 0].max
       end
 
-      def handle_failure(message, queue_name, error, payload: nil)
+      def handle_failure(message, queue_name, error, payload: nil, source_queue: nil)
         ctx = { action: "execute_job", queue: queue_name, job_class: payload&.dig("job_class"),
                 msg_id: message.msg_id.to_i, read_ct: message.read_ct.to_i }
         ErrorReporter.report(error, ctx)
@@ -311,20 +324,23 @@ module Pgbus
         FailedEventRecorder.record!(
           queue_name: queue_name,
           msg_id: message.msg_id.to_i,
+          source_queue: physical_queue(queue_name, source_queue),
           payload: payload || message.message,
           headers: message.respond_to?(:headers) ? message.headers : nil,
           error: error,
           retry_count: [message.read_ct.to_i - 1, 0].max
         )
 
-        apply_retry_backoff(message, queue_name, payload)
+        apply_retry_backoff(message, queue_name, payload, source_queue: source_queue)
       end
 
       # Extend the message's visibility timeout with exponential backoff
       # so retries aren't all bunched at the default flat VT interval.
       # Skipped on the first read (read_ct=1) — that's the initial
-      # attempt, not a retry.
-      def apply_retry_backoff(message, queue_name, payload)
+      # attempt, not a retry. Under priority routing the message sits in the
+      # _pN sub-queue the worker read it from; the logical name resolves to a
+      # base table that is never created (issue #582).
+      def apply_retry_backoff(message, queue_name, payload, source_queue: nil)
         attempt = message.read_ct.to_i - 1
         return if attempt < 1
 
@@ -335,7 +351,11 @@ module Pgbus
                   RetryBackoff.compute_delay(attempt: attempt)
                 end
 
-        client.set_visibility_timeout(queue_name, message.msg_id.to_i, vt: delay)
+        if source_queue
+          client.set_visibility_timeout(source_queue, message.msg_id.to_i, vt: delay, prefixed: false)
+        else
+          client.set_visibility_timeout(queue_name, message.msg_id.to_i, vt: delay)
+        end
       rescue StandardError => e
         Pgbus.logger.debug { "[Pgbus] Retry backoff VT update failed: #{e.message}" }
       end

data/lib/pgbus/batch/sweep.rb

@@ -38,14 +38,17 @@ module Pgbus
           swept
         end
 
+        # A row with a msg_id was backfilled with the physical table its message
+        # was sent to (the `_pN` sub-queue under priority routing), so every
+        # probe takes that name as is.
         def classify_stale(row, client)
-          in_queue = client.message_in_queue?(row.queue_name, msg_id: row.msg_id)
+          in_queue = client.message_in_queue?(row.queue_name, msg_id: row.msg_id, prefixed: false)
           return :still_present if in_queue != false
 
-          dlq = client.dead_letter_physical_name(row.queue_name)
-          return :failed if client.message_with_job_id?(dlq, job_id: row.job_id) == true
+          dlq = client.dead_letter_physical_name(row.queue_name, prefixed: false)
+          return :failed if client.message_with_job_id?(dlq, job_id: row.job_id, prefixed: false) == true
 
-          return :completed if client.message_archived?(row.queue_name, msg_id: row.msg_id) == true
+          return :completed if client.message_archived?(row.queue_name, msg_id: row.msg_id, prefixed: false) == true
 
           Pgbus.logger.warn do
             "[Pgbus] Batch execution #{row.job_id} (batch #{row.batch_id}) has no queue, " \
@@ -84,6 +87,8 @@ module Pgbus
           swept
         end
 
+        # A row without a msg_id still holds the logical queue name it was
+        # inserted with (the job's queue_name); the backfill was never applied.
         def classify_orphan(row, client)
           return :live if row.queue_name.nil?
 
@@ -91,7 +96,7 @@ module Pgbus
           return :live if in_queue != false
 
           dlq = client.dead_letter_physical_name(row.queue_name)
-          return :failed if client.message_with_job_id?(dlq, job_id: row.job_id) == true
+          return :failed if client.message_with_job_id?(dlq, job_id: row.job_id, prefixed: false) == true
 
           :orphan
         end

data/lib/pgbus/client.rb

@@ -720,16 +720,24 @@ module Pgbus
     # reaper to determine if a uniqueness lock with msg_id=0 (placeholder)
     # still has a corresponding queue message.
     #
-    # +queue_name+ may be either a logical name (e.g. "default") or an already
-    # prefixed physical name (e.g. "pgbus_default"). The client normalizes both.
+    # +queue_name+ is a logical name (e.g. "default"), expanded through the
+    # queue strategy, unless the caller passes +prefixed: false+ for a physical
+    # table name (e.g. "pgbus_default_p1"), used as is. The name's spelling
+    # decides nothing: a logical "pgbus_mailers" lives in pgbus_pgbus_mailers
+    # (issue #569).
     #
     # Returns:
     #   true  — the message definitely exists in the queue
     #   false — the message definitely does not exist
     #   nil   — could not determine (e.g. queue table missing or unknown error).
     #           Callers MUST treat nil as "exists" for safety.
-    def message_in_queue?(queue_name, msg_id:)
-      message_exists?(queue_name, msg_id: msg_id)
+    #
+    # Under priority routing a logical name probes every `_pN` table, and each
+    # table has its own msg_id sequence: a bare +msg_id+ may match a different
+    # message in a sibling sub-queue. A caller that knows the sub-queue passes
+    # its physical name with +prefixed: false+.
+    def message_in_queue?(queue_name, msg_id:, prefixed: true)
+      message_exists?(queue_name, msg_id: msg_id, prefixed: prefixed)
     end
 
     # Look up a queue row by ActiveJob job_id in the JSON payload. Used by the
@@ -738,10 +746,10 @@ module Pgbus
     #
     # Same tri-state as message_exists?: true / false / nil (unknown). A
     # logical name expands to every physical table the strategy owns (the
-    # `_pN` sub-queues under priority routing); an already-prefixed physical
-    # name probes itself only.
-    def message_with_job_id?(queue_name, job_id:)
-      tables = lookup_physical_queue_names(queue_name).map { |name| QueueNameValidator.sanitize!(name) }
+    # `_pN` sub-queues under priority routing); a physical name passed with
+    # +prefixed: false+ probes itself only.
+    def message_with_job_id?(queue_name, job_id:, prefixed: true)
+      tables = lookup_physical_queue_names(queue_name, prefixed: prefixed).map { |name| QueueNameValidator.sanitize!(name) }
       determined = false
 
       synchronized do
@@ -758,43 +766,85 @@ module Pgbus
       determined ? false : nil # rubocop:disable Style/ReturnNilInPredicateMethodDefinition -- tri-state: nil means unknown
     end
 
-    # DLQ companion of a logical or already-physical queue name. Does not
-    # re-prefix a name that resolve_full_queue_name already expanded.
-    def dead_letter_physical_name(queue_name)
-      full = resolve_full_queue_name(queue_name)
+    # The physical table that holds message +msg_id+, or nil when none does.
+    # See locate_messages.
+    def locate_message(queue_name, msg_id:, job_id: nil, prefixed: true)
+      locate_messages(queue_name, [[msg_id, job_id]], prefixed: prefixed)[msg_id.to_i]
+    end
+
+    # { msg_id => physical table } for each [msg_id, job_id] pair whose
+    # message is queued, one query per table on one connection. A logical
+    # name expands to every table the strategy owns, and under priority
+    # routing each `_pN` sub-queue numbers its messages from its own
+    # sequence: only a message carrying the pair's job_id matches, or any
+    # message with that msg_id when job_id is nil, the first table winning
+    # (issue #577). A physical name with +prefixed: false+ is probed alone.
+    # A table that does not exist holds nothing; other errors raise.
+    def locate_messages(queue_name, messages, prefixed: true)
+      wanted = messages.to_h { |msg_id, job_id| [msg_id.to_i, job_id&.to_s] }
+      tables = lookup_physical_queue_names(queue_name, prefixed: prefixed)
+      located = {}
+
+      synchronized do
+        with_raw_connection do |conn|
+          tables.each do |name|
+            pending = wanted.keys - located.keys
+            break if pending.empty?
+
+            queued_job_ids(conn, QueueNameValidator.sanitize!(name), pending).each do |row|
+              msg_id = row["msg_id"].to_i
+              job_id = wanted[msg_id]
+              located[msg_id] ||= name if job_id.nil? || row["job_id"] == job_id
+            end
+          end
+        end
+      end
+
+      located
+    end
+
+    # DLQ companion of a logical queue name, or of a physical table name passed
+    # with +prefixed: false+.
+    def dead_letter_physical_name(queue_name, prefixed: true)
+      return config.dead_letter_queue_name(queue_name) if prefixed
+
       # Priority sub-queues share the logical queue's DLQ (`pgbus_default_dlq`),
-      # not a per-priority `…_pN_dlq`. Only strip when the strategy actually
-      # creates `_pN` tables — a logical queue named `orders_p0` must keep it.
-      base = @queue_strategy.priority? ? full.sub(/_p\d+\z/, "") : full
+      # not a per-priority `…_pN_dlq`. Only a physical name can be a sub-queue,
+      # and only when the strategy actually creates `_pN` tables: a logical
+      # queue named `orders_p0` keeps its suffix.
+      name = queue_name.to_s
+      base = @queue_strategy.priority? ? name.sub(/_p\d+\z/, "") : name
       "#{base}#{Pgbus::DEAD_LETTER_SUFFIX}"
     end
 
     # Same tri-state as message_exists?: true / false / nil (unknown).
-    def message_archived?(queue_name, msg_id:)
-      full_name = resolve_full_queue_name(queue_name)
-      sanitized = QueueNameValidator.sanitize!(full_name)
+    # A logical name probes the archive of every physical table the strategy
+    # owns, like message_exists?; a physical name (prefixed: false) its own.
+    # A bare msg_id is unique per table only (see message_in_queue?).
+    def message_archived?(queue_name, msg_id:, prefixed: true)
+      tables = lookup_physical_queue_names(queue_name, prefixed: prefixed).map { |name| QueueNameValidator.sanitize!(name) }
+      determined = false
 
       synchronized do
         with_raw_connection do |conn|
-          msg_id_archived?(conn, sanitized, msg_id.to_i)
+          tables.each do |sanitized|
+            archived = probe_archive_presence(conn, sanitized, msg_id.to_i)
+            return true if archived == true
+
+            determined = true unless archived == :missing
+          end
         end
       end
-    rescue ActiveRecord::StatementInvalid => e
-      raise unless undefined_table_error?(e)
-
-      nil
-    rescue StandardError => e
-      raise unless defined?(PG::UndefinedTable) && e.is_a?(PG::UndefinedTable)
 
-      nil
+      determined ? false : nil # rubocop:disable Style/ReturnNilInPredicateMethodDefinition -- tri-state: nil means unknown
     end
 
-    def message_exists?(queue_name, msg_id: nil, uniqueness_key: nil)
+    def message_exists?(queue_name, msg_id: nil, uniqueness_key: nil, prefixed: true)
       has_msg_id = !msg_id.nil?
       has_uniqueness_key = !uniqueness_key.nil?
       raise ArgumentError, "pass msg_id, uniqueness_key, or both" unless has_msg_id || has_uniqueness_key
 
-      tables = lookup_physical_queue_names(queue_name).map { |name| QueueNameValidator.sanitize!(name) }
+      tables = lookup_physical_queue_names(queue_name, prefixed: prefixed).map { |name| QueueNameValidator.sanitize!(name) }
       determined = false
 
       synchronized do
@@ -1092,21 +1142,10 @@ module Pgbus
       end
     end
 
-    # Accept either a logical name ("default") or an already-prefixed
-    # physical name ("pgbus_default") and return the physical name.
-    # Coerces symbols to strings so callers can pass either form.
-    def resolve_full_queue_name(queue_name)
-      name = queue_name.to_s
-      prefix = "#{config.queue_prefix}_"
-      name.start_with?(prefix) ? name : config.queue_name(name)
-    end
-
     # Logical names expand through QueueFactory so a priority queue's _pN
-    # tables are included; already-prefixed physical names stay as-is.
-    def lookup_physical_queue_names(queue_name)
-      name = queue_name.to_s
-      prefix = "#{config.queue_prefix}_"
-      name.start_with?(prefix) ? [name] : @queue_strategy.physical_queue_names(name)
+    # tables are included; a physical name (prefixed: false) stays as is.
+    def lookup_physical_queue_names(queue_name, prefixed:)
+      prefixed ? @queue_strategy.physical_queue_names(queue_name.to_s) : [queue_name.to_s]
     end
 
     def probe_queue_presence(conn, sanitized, msg_id:, uniqueness_key:)
@@ -1125,6 +1164,18 @@ module Pgbus
       :missing
     end
 
+    def probe_archive_presence(conn, sanitized, msg_id)
+      msg_id_archived?(conn, sanitized, msg_id)
+    rescue ActiveRecord::StatementInvalid => e
+      raise unless undefined_table_error?(e)
+
+      :missing
+    rescue StandardError => e
+      raise unless defined?(PG::UndefinedTable) && e.is_a?(PG::UndefinedTable)
+
+      :missing
+    end
+
     def probe_job_id_presence(conn, sanitized, job_id)
       job_id_present?(conn, sanitized, job_id)
     rescue ActiveRecord::StatementInvalid => e
@@ -1137,6 +1188,22 @@ module Pgbus
       :missing
     end
 
+    # msg_ids are integers, so the array literal is safe.
+    def queued_job_ids(conn, sanitized, msg_ids)
+      conn.exec_params(
+        "SELECT msg_id, message::jsonb ->> 'job_id' AS job_id FROM pgmq.q_#{sanitized} WHERE msg_id = ANY($1::bigint[])",
+        ["{#{msg_ids.join(",")}}"]
+      )
+    rescue ActiveRecord::StatementInvalid => e
+      raise unless undefined_table_error?(e)
+
+      []
+    rescue StandardError => e
+      raise unless defined?(PG::UndefinedTable) && e.is_a?(PG::UndefinedTable)
+
+      []
+    end
+
     def scan_uniqueness_keys(conn, sanitized, keys, found)
       placeholders = keys.each_index.map { |i| "$#{i + 1}" }.join(", ")
       result = conn.exec_params(

data/lib/pgbus/configuration.rb

@@ -54,6 +54,13 @@ module Pgbus
     # wait, so recycling/deploy never wedges on a permanently-stuck job.
     attr_accessor :stall_threshold, :read_timeout, :drain_timeout
 
+    # Boot deadline (default 120s): a forked child that has not checked in
+    # this long after fork (no pgbus_processes row on this host, no liveness
+    # byte) is asked for a thread dump (TTIN), then SIGKILLed and re-forked
+    # by the normal restart path. nil disables. Separate from stall_threshold
+    # because a cold Rails boot can legitimately outlast a loop-liveness window.
+    attr_accessor :boot_timeout
+
     # immediate_shutdown_grace (default 2s): on QUIT, a worker or consumer
     # stops taking work and gives the jobs already running this long to finish
     # before it kills their threads. Killed jobs' ActiveRecord connections are
@@ -312,6 +319,7 @@ module Pgbus
       @max_worker_lifetime = nil
 
       @stall_threshold = 90
+      @boot_timeout = 120
       @read_timeout = 30
       @drain_timeout = 30
       @immediate_shutdown_grace = 2
@@ -918,6 +926,9 @@ module Pgbus
       unless stall_threshold.nil? || (stall_threshold.is_a?(Numeric) && stall_threshold.positive?)
         raise Pgbus::ConfigurationError, "stall_threshold must be a positive number or nil to disable"
       end
+      unless boot_timeout.nil? || (boot_timeout.is_a?(Numeric) && boot_timeout.positive?)
+        raise Pgbus::ConfigurationError, "boot_timeout must be a positive number or nil to disable"
+      end
       unless read_timeout.nil? || (read_timeout.is_a?(Numeric) && read_timeout.positive?)
         raise Pgbus::ConfigurationError, "read_timeout must be a positive number or nil to disable"
       end