pgbus 0.21.1 → 0.21.3

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 59ff2a591041f47789d41f241c5496534dbba5314ebb37a0c2fe41286f611587
-  data.tar.gz: 9f8b3c5a4f88330b96cd6ecab5d4dfe18580d48765b3aa50873e3f54150f63b1
+  metadata.gz: afa142ad44c901764a77528d7e387f708321353b5323c6dafcf23577cd27556e
+  data.tar.gz: 3bdcb708e9547c70115ccc9babd430d038a8b0ffb483faf615842a5a96cbbdff
 SHA512:
-  metadata.gz: 76c28daff8340ec68e0cf5b0b7f9a69ad48b1d024d65ff9af37fbc734167a8b710afca0b1fce9163783b43e022f0d8e6d4d465219aa91282b52eb69d770e969b
-  data.tar.gz: 8f2b9d1f6c4f899c64a9dd0fa1918b64e158799e6172bb0e42cf7beac6ca191675e5df9c31e90eaa62c4f1714c6c55eb317612e8d7b6047ab3e069d5bbab6046
+  metadata.gz: 3204bb1c094c7765ca32942b76857774e6696070b3b1787057b06da5f185d55de3069397bb1deb1dbf7221daf22eb064dddcb7a7906471e40fddafd5cf14cb38
+  data.tar.gz: 28c7d2ff21d9c8f97a2db27694c1e21184c9549d5af9cbebbef7244e3690d14b13aa546f79d006eb14104189ca3246d1416a6bb1679d59cfe80353d59511e6e5

data/CHANGELOG.md

@@ -1,7 +1,24 @@
 ## [Unreleased]
 
+### Added
+
+- **`config.boot_timeout` (default 120 seconds): the supervisor kills a child that never checks in (issue #587).** A fork checks in when it registers its `pgbus_processes` row on this host or writes its first liveness byte. One that has done neither `boot_timeout` seconds after the fork is logged at ERROR, sent `TTIN`, given 2 seconds, then SIGKILLed, and the normal restart path forks it again. A boot-deadline kill always counts as a crash, whatever the child's uptime, so a boot that hangs every time is re-forked with the crash-loop backoff (1 s doubling to 60 s) instead of every `boot_timeout`. Applies to workers, consumers, the dispatcher, the scheduler and the outbox poller. `nil` turns it off; anything else must be a positive number, or `validate!` raises `ConfigurationError`. When the watchdog's read of the process table fails, the boot deadline waits for a pass that can read it. Refs #587.
+- **A process whose registration in `pgbus_processes` failed at start retries it on each heartbeat (issue #587).** It used to stay without a row for good after one failed `create!` (a database blip at boot); a dispatcher, scheduler or outbox poller in that state would now be killed by the boot deadline even though it runs. Refs #587.
+- **`kill -TTIN <child pid>` logs every thread of a pgbus child with its backtrace at ERROR; the child keeps running (issue #587).** The watchdog sends it before every SIGKILL (boot timeout or stalled loop), so a hang is diagnosable from the log. Best effort: a child blocked in native code holding the GVL does not answer. Refs #587.
+- **`config.health_unclaimed_grace` (default 60 seconds) and the per-queue metric `oldest_unread_claimable_age_sec` (issue #589).** The grace is how long a visible, never-read (`read_ct = 0`) message must have been claimable before the health verdict counts it as "never claimed while draining processes are alive". It takes a number of seconds or an `ActiveSupport::Duration`; `0` restores the old rule (any such message counts), and `validate!` raises `ConfigurationError` on nil, a negative or a non-finite value. `Web::DataSource#queues_with_metrics` and `#queue_detail` report `oldest_unread_claimable_age_sec`, the seconds since the oldest visible `read_ct = 0` message became claimable (`min(vt)`, so a delayed job's age starts when its wait expires and a retried message is left out), nil when there is none. The verdict's `summary` carries it as the oldest over the drained queues, and the STALLED reason names it (`never claimed, oldest claimable for 74s`). `Pgbus::MCP::HealthAnalyzer.new` takes an optional `grace:`. Refs #589.
+- **The supervisor's standalone `/readyz` body carries `forked` (issue #587).** `forked` is the fork-table size (what `live` used to be); `live` now counts the children that have checked in. `Pgbus::Process::ReadinessSnapshot` gains `forked:`, defaulting to `live`. Refs #587.
+
+### Changed
+
+- **`pgbus_failed_events` gains a `source_queue` column; run `rails generate pgbus:update` (or `pgbus:add_failed_events_source_queue`) and migrate (issue #583).** The migration adds the column, creates the unique index `idx_pgbus_failed_events_queue_msg_source` on `(queue_name, msg_id, source_queue)` and drops `idx_pgbus_failed_events_queue_msg` on `(queue_name, msg_id)`. Until it runs, pgbus logs one warning per process and records failures as before (one row per logical queue and msg_id); a running process checks for the column again every minute, so it needs no restart after the migration. Fresh `pgbus:install` creates the new shape. `pgbus:update` no longer queues `pgbus:add_failed_events_index`, which would put the old index back, and queues the new migration again if the old index is present; `pgbus:add_failed_events_index` run by name adds nothing once the new index exists. `Pgbus::FailedEventRecorder.record!`, `.exists?`, `.last_error` and `.clear!` take a required `source_queue:` keyword, the physical table the message was read from. Refs #583.
+
 ### Fixed
 
+- **`/readyz` and `pgbus_health` no longer flap to `STALLED` on a healthy fleet (issue #589).** The verdict reported `STALLED` (and the mounted `/readyz` 503) whenever a drained queue held one visible, never-read message while a worker or consumer was alive, however briefly it had been claimable: every enqueue and every delayed job whose wait just expired is such a message until a worker's next poll, so a busy install answered 503 on a large share of checks. That branch now counts a queue only once its oldest such message has been claimable for `health_unclaimed_grace` seconds; a wedged worker still reads `STALLED` after the grace. A queue hash without the age (an older or stubbed data source) still counts at once. The worker-reported stalled branch (heart-beating, claim loop not advancing) is unchanged. Refs #589.
+- **A child that hangs before its first liveness signal is killed and forked again, and `/readyz` no longer counts it as live (issue #587).** A fork stuck between `fork` and its first check-in (in an initializer, the eager load or a connection) never exited, so nothing reaped it, and the watchdog skipped it because no liveness channel had spoken; the container-local `/readyz` counted it as live because it was in the fork table, so a rolling deploy passed its health gate with a dead capsule. The boot deadline above now kills it, and `/readyz` stays at 503 `DEGRADED` (`live < expected`) until every child has checked in. A dispatcher, scheduler or outbox poller checks in through its process row, which the watchdog reads every 10 seconds, so `/readyz` turns 200 up to 10 seconds after the last one registers, and stays at 503 while the database is unreachable at boot (those children cannot be seen to be alive). The watchdog reads only this host's process rows last beaten since the fork, so a row another container wrote under the same pid, or one a SIGKILLed predecessor left behind (a container restarted in place reuses its hostname and pids), no longer vouches for a local child or hides a stalled loop. Refs #587.
+- **A failed job's retry backoff applies under priority routing (issue #582).** With `priority_levels > 1` the executor set the backoff's visibility timeout on the logical queue's base table, which priority routing never creates; the call raised, was logged at debug, and the job came back after the read's visibility timeout instead of its backoff delay. The backoff now goes to the `_pN` sub-queue the worker read the message from, as the visibility heartbeat and the archive already did. Refs #582.
+- **Two priority sub-queues' failures with the same msg_id keep a row each (issue #583).** Each `_pN` sub-queue numbers its messages from its own sequence, and `pgbus_failed_events` was unique on the logical queue and the msg_id, so a second job failing with the same msg_id in another sub-queue overwrote the first one's error and kept its payload: the dashboard showed one failed job where two had failed, Retry and Discard acted on the first job's message and deleted the only row, and the first job's success cleared the second one's failure. Rows now record the physical table (`source_queue`) and are unique on `(queue_name, msg_id, source_queue)`. The executor, the event consumer and the worker's zombie check read, write and clear only the row of the table the message came from. On the Jobs page a failed row joins only its own table's message and shows as an orphan once that message is gone, Retry, Discard and Discard all look the message up in the table the row names, and Discard releases the job's uniqueness lock bounded by when that table's message was sent, not by a sibling sub-queue's. A row written before the migration has no `source_queue` and is treated as before: it can belong to any table of its queue. Refs #583.
+
 - **The dashboard's Retry, Discard and Discard all reach a failed job's message under priority routing (issue #577).** A failed-job row records the logical queue only, and with `priority_levels > 1` the message sits in a `_pN` sub-queue, never in the base table. Retry reset the visibility timeout on `pgmq.q_pgbus_<queue>`, which priority routing never creates, and answered "Could not retry job."; Discard and Discard all archived from that same table, so they reported the job discarded while its message stayed queued and ran again. Each sub-queue also numbers its messages from its own sequence, so the msg_id alone could name another job's message in a sibling sub-queue. The dashboard now finds the sub-queue whose message has the row's msg_id and carries the job's `job_id` (`Pgbus::Client#locate_message`, which returns that physical table or nil, and `#locate_messages` for many at once, one query per table), resets or archives the message there, and leaves a same-numbered message elsewhere alone. Discard all locates a queue's messages together and archives them in one batch per table. When no sub-queue holds it, Retry enqueues a fresh copy as before. Without priority routing there is one table and nothing changes, except that a message reusing the msg_id under another `job_id` is no longer reset or archived in the job's place. Refs #577.
 
 ## [0.21.0] - 2026-10-19

data/README.md

@@ -1260,7 +1260,7 @@ All tools below are read-only — none mutates state, and there is no raw-SQL pa
 
 | Tool | Purpose |
 |------|---------|
-| `pgbus_health` | One-call verdict: `OK` / `DEGRADED` / `STALLED`. STALLED is the silent-worker-wedge signal (visible backlog while workers heart-beat but don't claim). Suitable for automated alerting. |
+| `pgbus_health` | One-call verdict: `OK` / `DEGRADED` / `STALLED`. STALLED is the silent-worker-wedge signal: a worker reports its claim loop stalled, or a visible backlog older than `health_unclaimed_grace` (default 60s) while workers heart-beat but don't claim. The summary carries `oldest_unread_claimable_age_sec`. Suitable for automated alerting. |
 | `pgbus_queues` | All queues: depth, visible count, oldest-message age, paused state. |
 | `pgbus_queue_detail` | Per-queue metrics + paused state + table health (dead tuples, bloat, vacuum age). |
 | `pgbus_processes` | Every process with kind, pid, heartbeat age, and `healthy`/`stale`/`stalled` status. |
@@ -1348,7 +1348,7 @@ For the **HTTP** transport, point the client at the mounted URL with a streamabl
 
 For orchestrators like Kubernetes, Pgbus exposes two HTTP probes: `/livez` (is the serving process up?) and `/readyz`. Readiness means different things in the two places the probes are served:
 
-- **Mounted in Rails** (`Pgbus::Web::HealthApp`): `/readyz` runs the cluster-wide `OK` / `DEGRADED` / `STALLED` verdict, same as the MCP `pgbus_health` tool — `STALLED` (visible backlog while workers heart-beat but don't claim) fails readiness.
+- **Mounted in Rails** (`Pgbus::Web::HealthApp`): `/readyz` runs the cluster-wide `OK` / `DEGRADED` / `STALLED` verdict, same as the MCP `pgbus_health` tool — `STALLED` (a never-claimed backlog older than `health_unclaimed_grace` while workers heart-beat but don't claim) fails readiness.
 - **Standalone from the supervisor** (`health_port`): `/readyz` is **container-local** — did *this* supervisor finish booting, and are all the children *it* forked alive? That is the signal a rolling deploy's health gate needs; the cluster verdict would let a brand-new container pass on the strength of the *old* container's workers.
 
 | Path | Method | 200 | 503 | Touches DB |
@@ -2560,6 +2560,7 @@ Curated headline options for the README. The full operator reference (with types
 | `statsd_port` | `8125` | StatsD UDP port (used when `metrics_backend = :statsd`) |
 | `health_port` | `nil` | Port for standalone HTTP liveness/readiness probes served by the supervisor; nil disables |
 | `health_bind` | `"127.0.0.1"` | Bind address for the standalone health server |
+| `health_unclaimed_grace` | `60` | Seconds a visible, never-read message must have been claimable before the health verdict reports it as never claimed (`STALLED`); `0` counts any such message |
 | `pgmq_schema_mode` | `:auto` | PGMQ schema install mode (`:auto`, `:extension`, `:embedded`) |
 | `doctor_on_boot` | `nil` | Run doctor inside the booting supervisor: `nil`/`false` = off, `:report` = log and boot, `:strict` = refuse to boot on a fatal check. Also via `--doctor` / `--doctor-strict`. |
 | `eager_validation` | `true` | Run `Configuration#validate!` automatically after the `Pgbus.configure` block yields; an invalid value raises `Pgbus::ConfigurationError` at boot. Set `false` to suppress and validate manually. |

data/lib/generators/pgbus/add_failed_events_source_queue_generator.rb

@@ -0,0 +1,44 @@
+# frozen_string_literal: true
+
+require "rails/generators"
+require "rails/generators/active_record"
+require_relative "migration_path"
+
+module Pgbus
+  module Generators
+    class AddFailedEventsSourceQueueGenerator < Rails::Generators::Base
+      include ActiveRecord::Generators::Migration
+      include MigrationPath
+
+      source_root File.expand_path("templates", __dir__)
+
+      desc "Add source_queue to pgbus_failed_events so each priority sub-queue's failure keeps its own row"
+
+      class_option :database,
+                   type: :string,
+                   default: nil,
+                   desc: "Use a separate database for pgbus tables (e.g. --database=pgbus)"
+
+      def create_migration_file
+        migration_template "add_failed_events_source_queue.rb.erb",
+                           File.join(pgbus_migrate_path, "add_pgbus_failed_events_source_queue.rb")
+      end
+
+      def display_post_install
+        say ""
+        say "Pgbus failed events source_queue added!", :green
+        say ""
+        say "Next steps:"
+        say "  1. Run: rails db:migrate#{migrate_command_suffix}"
+        say "  2. Restart pgbus processes so they record failures per physical queue"
+        say ""
+      end
+
+      private
+
+      def migration_version
+        "[#{ActiveRecord::Migration.current_version}]"
+      end
+    end
+  end
+end

data/lib/generators/pgbus/templates/add_failed_events_source_queue.rb.erb

@@ -0,0 +1,27 @@
+class AddPgbusFailedEventsSourceQueue < ActiveRecord::Migration<%= migration_version %>
+  # Under priority routing each _pN sub-queue numbers its messages from its
+  # own sequence, so two jobs can fail with the same msg_id under one logical
+  # queue. source_queue names the physical table the message was read from,
+  # and the failure upsert is keyed on it. Rows written before this keep
+  # NULL there: pgbus reads them as belonging to any table of their queue.
+  def up
+    add_column :pgbus_failed_events, :source_queue, :string, if_not_exists: true
+    add_index :pgbus_failed_events, %i[queue_name msg_id source_queue],
+      unique: true, name: "idx_pgbus_failed_events_queue_msg_source", if_not_exists: true
+    remove_index :pgbus_failed_events, name: "idx_pgbus_failed_events_queue_msg", if_exists: true
+  end
+
+  # (queue_name, msg_id) can repeat once source_queue is in use: the latest
+  # row of each pair is kept so the old unique index can be built again.
+  def down
+    execute <<~SQL
+      DELETE FROM pgbus_failed_events older
+      USING pgbus_failed_events newer
+      WHERE older.queue_name = newer.queue_name AND older.msg_id = newer.msg_id AND older.id < newer.id
+    SQL
+    add_index :pgbus_failed_events, %i[queue_name msg_id],
+      unique: true, name: "idx_pgbus_failed_events_queue_msg", if_not_exists: true
+    remove_index :pgbus_failed_events, name: "idx_pgbus_failed_events_queue_msg_source", if_exists: true
+    remove_column :pgbus_failed_events, :source_queue, if_exists: true
+  end
+end

data/lib/generators/pgbus/templates/add_failed_events_unique_index.rb.erb

@@ -1,5 +1,10 @@
 class AddPgbusFailedEventsUniqueIndex < ActiveRecord::Migration<%= migration_version %>
   def change
+    # Superseded by pgbus:add_failed_events_source_queue's (queue_name, msg_id,
+    # source_queue) index: this one would make two priority sub-queues'
+    # failures with the same msg_id share one row again.
+    return if index_name_exists?(:pgbus_failed_events, "idx_pgbus_failed_events_queue_msg_source")
+
     add_index :pgbus_failed_events, [:queue_name, :msg_id],
       unique: true, name: "idx_pgbus_failed_events_queue_msg",
       if_not_exists: true

data/lib/generators/pgbus/templates/migration.rb.erb

@@ -53,9 +53,12 @@ class CreatePgbusTables < ActiveRecord::Migration<%= migration_version %>
     add_index :pgbus_processes, :last_heartbeat_at,
       name: "idx_pgbus_processes_heartbeat"
 
-    # Failed events log (for dashboard visibility)
+    # Failed events log (for dashboard visibility). queue_name is the logical
+    # queue, source_queue the physical table the message was read from: under
+    # priority routing each _pN sub-queue numbers its messages on its own.
     create_table :pgbus_failed_events do |t|
       t.string :queue_name, null: false
+      t.string :source_queue
       t.bigint :msg_id
       t.jsonb :payload
       t.jsonb :headers
@@ -68,8 +71,8 @@ class CreatePgbusTables < ActiveRecord::Migration<%= migration_version %>
 
     add_index :pgbus_failed_events, :queue_name, name: "idx_pgbus_failed_events_queue"
     add_index :pgbus_failed_events, :failed_at, name: "idx_pgbus_failed_events_time"
-    add_index :pgbus_failed_events, [:queue_name, :msg_id],
-      unique: true, name: "idx_pgbus_failed_events_queue_msg"
+    add_index :pgbus_failed_events, [:queue_name, :msg_id, :source_queue],
+      unique: true, name: "idx_pgbus_failed_events_queue_msg_source"
 
     # Concurrency semaphores (counting locks for job concurrency limits)
     create_table :pgbus_semaphores do |t|

data/lib/pgbus/active_job/executor.rb

@@ -28,9 +28,11 @@ module Pgbus
         if read_count > config.max_retries
           # Read the last error before clear! deletes the only row that has
           # it: the DLQ copy carries it in its headers (issue #495).
-          last_error = FailedEventRecorder.last_error(queue_name: queue_name, msg_id: message.msg_id.to_i)
+          last_error = FailedEventRecorder.last_error(queue_name: queue_name, msg_id: message.msg_id.to_i,
+                                                      source_queue: physical_queue(queue_name, source_queue))
           handle_dead_letter(message, queue_name, payload, source_queue: source_queue, last_error: last_error)
-          FailedEventRecorder.clear!(queue_name: queue_name, msg_id: message.msg_id.to_i)
+          FailedEventRecorder.clear!(queue_name: queue_name, msg_id: message.msg_id.to_i,
+                                     source_queue: physical_queue(queue_name, source_queue))
           signal_concurrency(payload)
           signal_batch_discarded(payload)
           release_uniqueness_lock(Uniqueness.extract_key(payload), queue_name, message)
@@ -129,7 +131,10 @@ module Pgbus
           # failed attempt, so a first delivery has no row: skip the per-job
           # DELETE round trip (issue #484). A stale row from a dropped-and-
           # recreated queue reusing this msg_id is the dispatcher's to sweep.
-          FailedEventRecorder.clear!(queue_name: queue_name, msg_id: msg_id) if read_count > 1
+          if read_count > 1
+            FailedEventRecorder.clear!(queue_name: queue_name, msg_id: msg_id,
+                                       source_queue: physical_queue(queue_name, source_queue))
+          end
         end
 
         instrument("pgbus.job_completed", queue: queue_name, job_class: job_class)
@@ -145,7 +150,7 @@ module Pgbus
         # Before this, a fiber interruption between perform_now and archive_from
         # silently lost control flow — no failed event row, no job_failed
         # notification, uniqueness lock held until VT expired. See issue #126.
-        handle_failure(message, queue_name, e, payload: payload)
+        handle_failure(message, queue_name, e, payload: payload, source_queue: source_queue)
         # A failed :while_executing attempt is no longer executing — release
         # so the retry (same message, after VT) can acquire. :until_executed
         # keeps its lock until success/DLQ by design (#126, #333).
@@ -188,6 +193,14 @@ module Pgbus
         "msg_id=#{message.msg_id} queue=#{queue_name} read_ct=#{message.read_ct}"
       end
 
+      # The table the message was read from: the _pN sub-queue under priority
+      # routing, else the logical queue's own. Each table numbers its messages
+      # from its own sequence, so a failed-event row is keyed on it (issue
+      # #583). Built only on the failure and redelivery paths.
+      def physical_queue(queue_name, source_queue)
+        source_queue || config.queue_name(queue_name)
+      end
+
       def assign_batch_id(job, payload)
         batch_id = payload[Batch::METADATA_KEY]
         return unless batch_id && job.respond_to?(:batch_id=)
@@ -298,7 +311,7 @@ module Pgbus
         [((Time.now.utc - enqueued_at) * 1000).round, 0].max
       end
 
-      def handle_failure(message, queue_name, error, payload: nil)
+      def handle_failure(message, queue_name, error, payload: nil, source_queue: nil)
         ctx = { action: "execute_job", queue: queue_name, job_class: payload&.dig("job_class"),
                 msg_id: message.msg_id.to_i, read_ct: message.read_ct.to_i }
         ErrorReporter.report(error, ctx)
@@ -311,20 +324,23 @@ module Pgbus
         FailedEventRecorder.record!(
           queue_name: queue_name,
           msg_id: message.msg_id.to_i,
+          source_queue: physical_queue(queue_name, source_queue),
           payload: payload || message.message,
           headers: message.respond_to?(:headers) ? message.headers : nil,
           error: error,
           retry_count: [message.read_ct.to_i - 1, 0].max
         )
 
-        apply_retry_backoff(message, queue_name, payload)
+        apply_retry_backoff(message, queue_name, payload, source_queue: source_queue)
       end
 
       # Extend the message's visibility timeout with exponential backoff
       # so retries aren't all bunched at the default flat VT interval.
       # Skipped on the first read (read_ct=1) — that's the initial
-      # attempt, not a retry.
-      def apply_retry_backoff(message, queue_name, payload)
+      # attempt, not a retry. Under priority routing the message sits in the
+      # _pN sub-queue the worker read it from; the logical name resolves to a
+      # base table that is never created (issue #582).
+      def apply_retry_backoff(message, queue_name, payload, source_queue: nil)
         attempt = message.read_ct.to_i - 1
         return if attempt < 1
 
@@ -335,7 +351,11 @@ module Pgbus
                   RetryBackoff.compute_delay(attempt: attempt)
                 end
 
-        client.set_visibility_timeout(queue_name, message.msg_id.to_i, vt: delay)
+        if source_queue
+          client.set_visibility_timeout(source_queue, message.msg_id.to_i, vt: delay, prefixed: false)
+        else
+          client.set_visibility_timeout(queue_name, message.msg_id.to_i, vt: delay)
+        end
       rescue StandardError => e
         Pgbus.logger.debug { "[Pgbus] Retry backoff VT update failed: #{e.message}" }
       end

data/lib/pgbus/configuration.rb

@@ -54,6 +54,13 @@ module Pgbus
     # wait, so recycling/deploy never wedges on a permanently-stuck job.
     attr_accessor :stall_threshold, :read_timeout, :drain_timeout
 
+    # Boot deadline (default 120s): a forked child that has not checked in
+    # this long after fork (no pgbus_processes row on this host, no liveness
+    # byte) is asked for a thread dump (TTIN), then SIGKILLed and re-forked
+    # by the normal restart path. nil disables. Separate from stall_threshold
+    # because a cold Rails boot can legitimately outlast a loop-liveness window.
+    attr_accessor :boot_timeout
+
     # immediate_shutdown_grace (default 2s): on QUIT, a worker or consumer
     # stops taking work and gives the jobs already running this long to finish
     # before it kills their threads. Killed jobs' ActiveRecord connections are
@@ -234,6 +241,14 @@ module Pgbus
     # plain TCPServer bound to `health_bind` (default localhost).
     attr_accessor :health_port, :health_bind
 
+    # Seconds a visible, never-read (read_ct = 0) message must have been
+    # claimable before the health verdict counts it as "never claimed while
+    # draining processes are alive" (issue #589). Below it a worker's next
+    # poll is still coming, so a fresh enqueue or a delayed job whose wait just
+    # expired no longer reads STALLED. 0 restores the pre-0.22 rule (any
+    # visible read_ct = 0 row counts). Default 60.
+    attr_reader :health_unclaimed_grace
+
     # Streams (turbo-rails replacement, SSE-based)
     attr_accessor :streams_enabled, :streams_path, :streams_signed_name_secret,
                   :streams_default_retention, :streams_retention, :streams_heartbeat_interval, :streams_peer_close_interval,
@@ -312,6 +327,7 @@ module Pgbus
       @max_worker_lifetime = nil
 
       @stall_threshold = 90
+      @boot_timeout = 120
       @read_timeout = 30
       @drain_timeout = 30
       @immediate_shutdown_grace = 2
@@ -411,6 +427,7 @@ module Pgbus
       # HTTP health endpoints — nil port disables the standalone server.
       @health_port = nil
       @health_bind = "127.0.0.1"
+      @health_unclaimed_grace = 60
 
       @streams_enabled = true
       @streams_path = nil
@@ -918,6 +935,9 @@ module Pgbus
       unless stall_threshold.nil? || (stall_threshold.is_a?(Numeric) && stall_threshold.positive?)
         raise Pgbus::ConfigurationError, "stall_threshold must be a positive number or nil to disable"
       end
+      unless boot_timeout.nil? || (boot_timeout.is_a?(Numeric) && boot_timeout.positive?)
+        raise Pgbus::ConfigurationError, "boot_timeout must be a positive number or nil to disable"
+      end
       unless read_timeout.nil? || (read_timeout.is_a?(Numeric) && read_timeout.positive?)
         raise Pgbus::ConfigurationError, "read_timeout must be a positive number or nil to disable"
       end
@@ -940,6 +960,8 @@ module Pgbus
 
       raise Pgbus::ConfigurationError, "health_bind must be a non-empty String" unless health_bind.is_a?(String) && !health_bind.empty?
 
+      validate_health_unclaimed_grace!
+
       raise Pgbus::ConfigurationError, "statsd_host must be a non-empty String" unless statsd_host.is_a?(String) && !statsd_host.empty?
 
       unless statsd_port.is_a?(Integer) && statsd_port.between?(1, 65_535)
@@ -1528,6 +1550,14 @@ module Pgbus
       @recurring_execution_retention = coerce_duration!(value, :recurring_execution_retention)
     end
 
+    # Not coerce_duration!: 0 is valid here (the pre-0.22 rule), and nil is not
+    # a "disabled" sentinel. A Duration becomes integer seconds; anything else
+    # is kept for validate! to judge.
+    def health_unclaimed_grace=(value)
+      duration = defined?(ActiveSupport::Duration) && value.is_a?(ActiveSupport::Duration)
+      @health_unclaimed_grace = duration ? value.to_i : value
+    end
+
     # recurring_tasks_file (singular) is deprecated in favor of the plural
     # recurring_tasks_files. The engine still writes the singular internally to
     # record which default file was loaded, so it stays a plain accessor; the
@@ -1856,6 +1886,14 @@ module Pgbus
       raise Pgbus::ConfigurationError, "immediate_shutdown_grace must be a finite number >= 0"
     end
 
+    # A finite real number >= 0; nil is not "off" (0 is).
+    def validate_health_unclaimed_grace!
+      grace = health_unclaimed_grace
+      return if grace.is_a?(Numeric) && grace.real? && grace.finite? && grace >= 0
+
+      raise Pgbus::ConfigurationError, "health_unclaimed_grace must be a finite number of seconds >= 0, got #{grace.inspect}"
+    end
+
     def validate_positive_duration!(numeric, name)
       raise Pgbus::ConfigurationError, "#{name} must be a positive number, got #{numeric}" unless numeric.positive?
 

data/lib/pgbus/failed_event_recorder.rb

@@ -4,45 +4,49 @@ require "time"
 
 module Pgbus
   # Records job failures to pgbus_failed_events for dashboard visibility.
-  # Uses upsert (INSERT ON CONFLICT UPDATE) keyed on (queue_name, msg_id)
-  # so each message has at most one failed_event row tracking its latest error.
+  # Uses upsert (INSERT ON CONFLICT UPDATE) keyed on the message, so each
+  # message has at most one failed_event row tracking its latest error.
+  #
+  # A message is (queue_name, msg_id, source_queue): queue_name is the logical
+  # queue, source_queue the physical table the message was read from. Under
+  # priority routing each `_pN` sub-queue numbers its messages from its own
+  # sequence, so the logical name and msg_id alone can name two jobs (issue
+  # #583). A row written before the source_queue column existed has NULL there
+  # and is read as belonging to any table of its queue. Until the column is
+  # migrated, every query keeps its (queue_name, msg_id) shape.
   class FailedEventRecorder
+    # How long a "column missing" answer stands before the schema is probed
+    # again, so a process started before the migration picks the column up.
+    SOURCE_QUEUE_RECHECK_SECONDS = 60
+
+    @source_queue_column_mutex = Mutex.new
+
     class << self
-      def record!(queue_name:, msg_id:, payload:, headers:, error:, retry_count:)
-        connection.exec_query(
-          <<~SQL.squish,
-            INSERT INTO pgbus_failed_events
-              (queue_name, msg_id, payload, headers, error_class, error_message, backtrace, retry_count, failed_at)
-            VALUES ($1, $2, $3, $4, $5, $6, $7, $8, CURRENT_TIMESTAMP)
-            ON CONFLICT (queue_name, msg_id) DO UPDATE SET
-              error_class = EXCLUDED.error_class,
-              error_message = EXCLUDED.error_message,
-              backtrace = EXCLUDED.backtrace,
-              retry_count = EXCLUDED.retry_count,
-              failed_at = EXCLUDED.failed_at
-          SQL
-          "FailedEvent Record",
-          [
-            queue_name,
-            msg_id.to_i,
-            payload.is_a?(String) ? payload : JSON.generate(payload),
-            headers.is_a?(String) ? headers : headers&.then { |h| JSON.generate(h) },
-            error.class.name,
-            error.message.to_s.truncate(10_000),
-            error.backtrace&.first(30)&.join("\n"),
-            retry_count
-          ]
-        )
+      def record!(queue_name:, msg_id:, source_queue:, payload:, headers:, error:, retry_count:)
+        details = [
+          payload.is_a?(String) ? payload : JSON.generate(payload),
+          headers.is_a?(String) ? headers : headers&.then { |h| JSON.generate(h) },
+          error.class.name,
+          error.message.to_s.truncate(10_000),
+          error.backtrace&.first(30)&.join("\n"),
+          retry_count
+        ]
+        legacy = !source_queue_column?
+        if legacy
+          connection.exec_query(LEGACY_RECORD_SQL, "FailedEvent Record", [queue_name, msg_id.to_i, *details])
+        else
+          connection.exec_query(RECORD_SQL, "FailedEvent Record", [queue_name, source_queue, msg_id.to_i, *details])
+        end
       rescue StandardError => e
+        # A process started before the migration would keep sending the old
+        # ON CONFLICT target, whose index the migration drops: probe again.
+        reset_source_queue_column_check! if legacy
         ErrorReporter.report(e, { action: "record_failed_event", queue: queue_name, msg_id: msg_id })
       end
 
-      def exists?(queue_name:, msg_id:)
-        result = connection.select_value(
-          "SELECT 1 FROM pgbus_failed_events WHERE queue_name = $1 AND msg_id = $2 LIMIT 1",
-          "FailedEvent Exists",
-          [queue_name, msg_id.to_i]
-        )
+      def exists?(queue_name:, msg_id:, source_queue:)
+        sql, binds = message_scope(queue_name, msg_id, source_queue)
+        result = connection.select_value("SELECT 1 FROM pgbus_failed_events WHERE #{sql} LIMIT 1", "FailedEvent Exists", binds)
         !result.nil?
       rescue StandardError => e
         Pgbus.logger.debug { "[Pgbus] FailedEvent exists? check failed: #{e.class}: #{e.message}" }
@@ -53,12 +57,15 @@ module Pgbus
       # dead-letters the message and clears the row (issue #495). failed_at
       # comes back as an ISO 8601 UTC string whatever the adapter decodes it
       # to. A failed read returns nil: it must never stop a dead-letter.
-      def last_error(queue_name:, msg_id:)
+      # The message's own row wins over one written before source_queue.
+      def last_error(queue_name:, msg_id:, source_queue:)
+        sql, binds = message_scope(queue_name, msg_id, source_queue)
+        order = binds.size > 2 ? " ORDER BY source_queue IS NULL, failed_at DESC" : ""
         row = connection.select_one(
           "SELECT error_class, error_message, backtrace, retry_count, failed_at " \
-          "FROM pgbus_failed_events WHERE queue_name = $1 AND msg_id = $2 LIMIT 1",
+          "FROM pgbus_failed_events WHERE #{sql}#{order} LIMIT 1",
           "FailedEvent Last Error",
-          [queue_name, msg_id.to_i]
+          binds
         )
         return unless row
 
@@ -91,12 +98,11 @@ module Pgbus
         end
       end
 
-      def clear!(queue_name:, msg_id:)
-        connection.exec_delete(
-          "DELETE FROM pgbus_failed_events WHERE queue_name = $1 AND msg_id = $2",
-          "FailedEvent Clear",
-          [queue_name, msg_id.to_i]
-        )
+      # Deletes the message's own row, and a row written before source_queue
+      # existed: a sibling sub-queue's row with the same msg_id stays.
+      def clear!(queue_name:, msg_id:, source_queue:)
+        sql, binds = message_scope(queue_name, msg_id, source_queue)
+        connection.exec_delete("DELETE FROM pgbus_failed_events WHERE #{sql}", "FailedEvent Clear", binds)
       rescue StandardError => e
         # ERROR-level: a failed clear leaves a stale row in the dashboard
         # AFTER the job actually succeeded — confusing and load-bearing
@@ -107,8 +113,69 @@ module Pgbus
         end
       end
 
+      # Whether pgbus_failed_events has the source_queue column (issue #583).
+      # Probed like ProcessedEvent.completion_column?: a gem upgraded before
+      # its migration gets false, a one-time warning, and the (queue_name,
+      # msg_id) queries. A present column is remembered for good; a missing
+      # one for SOURCE_QUEUE_RECHECK_SECONDS. A probe that raises is not
+      # memoized.
+      def source_queue_column?
+        # Double-checked: the unlocked read only skips the lock while the
+        # answer stands; it is written under @source_queue_column_mutex. The
+        # state is true, nil (never probed) or the monotonic time a missing
+        # column's answer runs out.
+        known = source_queue_column_answer(@source_queue_column) # rubocop:disable ThreadSafety/ClassInstanceVariable
+        return known unless known.nil?
+
+        @source_queue_column_mutex.synchronize do
+          known = source_queue_column_answer(@source_queue_column)
+          next known unless known.nil?
+
+          supported = detect_source_queue_column(warn: @source_queue_column.nil?)
+          @source_queue_column = supported || (monotonic_now + SOURCE_QUEUE_RECHECK_SECONDS)
+          supported
+        end
+      end
+
+      # Clears the memoized probe: specs exercise both schema shapes, and a
+      # failed legacy write re-probes after the migration lands.
+      def reset_source_queue_column_check!
+        @source_queue_column_mutex.synchronize { @source_queue_column = nil }
+      end
+
       private
 
+      # WHERE fragment and binds naming one message's rows.
+      def message_scope(queue_name, msg_id, source_queue)
+        return ["queue_name = $1 AND msg_id = $2", [queue_name, msg_id.to_i]] unless source_queue_column?
+
+        ["queue_name = $1 AND msg_id = $2 AND (source_queue = $3 OR source_queue IS NULL)",
+         [queue_name, msg_id.to_i, source_queue]]
+      end
+
+      # true, false while a missing column's answer stands, nil to probe.
+      def source_queue_column_answer(state)
+        return true if state == true
+
+        false if state && monotonic_now < state
+      end
+
+      def monotonic_now
+        ::Process.clock_gettime(::Process::CLOCK_MONOTONIC)
+      end
+
+      def detect_source_queue_column(warn:)
+        supported = connection.column_exists?(:pgbus_failed_events, :source_queue)
+        if warn && !supported
+          Pgbus.logger.warn do
+            "[Pgbus] pgbus_failed_events is missing the source_queue column; under priority routing two " \
+              "sub-queues' failures with the same msg_id share one row. " \
+              "Run `rails generate pgbus:add_failed_events_source_queue` (or pgbus:update) and migrate."
+          end
+        end
+        supported
+      end
+
       def iso_utc(value)
         time = value.is_a?(String) ? Time.parse(value) : value
         time&.utc&.iso8601(6)
@@ -124,5 +191,31 @@ module Pgbus
         end
       end
     end
+
+    UPSERT_ERROR_COLUMNS = <<~SQL.gsub(/\s+/, " ").strip.freeze
+      DO UPDATE SET
+        error_class = EXCLUDED.error_class,
+        error_message = EXCLUDED.error_message,
+        backtrace = EXCLUDED.backtrace,
+        retry_count = EXCLUDED.retry_count,
+        failed_at = EXCLUDED.failed_at
+    SQL
+    private_constant :UPSERT_ERROR_COLUMNS
+
+    RECORD_SQL = <<~SQL.gsub(/\s+/, " ").strip.freeze
+      INSERT INTO pgbus_failed_events
+        (queue_name, source_queue, msg_id, payload, headers, error_class, error_message, backtrace, retry_count, failed_at)
+      VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, CURRENT_TIMESTAMP)
+      ON CONFLICT (queue_name, msg_id, source_queue) #{UPSERT_ERROR_COLUMNS}
+    SQL
+    private_constant :RECORD_SQL
+
+    LEGACY_RECORD_SQL = <<~SQL.gsub(/\s+/, " ").strip.freeze
+      INSERT INTO pgbus_failed_events
+        (queue_name, msg_id, payload, headers, error_class, error_message, backtrace, retry_count, failed_at)
+      VALUES ($1, $2, $3, $4, $5, $6, $7, $8, CURRENT_TIMESTAMP)
+      ON CONFLICT (queue_name, msg_id) #{UPSERT_ERROR_COLUMNS}
+    SQL
+    private_constant :LEGACY_RECORD_SQL
   end
 end