pgbus 0.21.2 → 0.21.3

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: c2dece606e269b4f406a3ba3d786facbdd53aedd675fda239803c5136e241678
-  data.tar.gz: 8498b9396aa4577f18b5e2e7793e82e9a67ce896b7495f061d51bd28e56112a1
+  metadata.gz: afa142ad44c901764a77528d7e387f708321353b5323c6dafcf23577cd27556e
+  data.tar.gz: 3bdcb708e9547c70115ccc9babd430d038a8b0ffb483faf615842a5a96cbbdff
 SHA512:
-  metadata.gz: 7032434023b3253d81e3c6668a4db3481a500a1421cce80fe94a45f235a244be23da1b0e25ec3657acbaaee4ba11c17e56891d4c401010f22340ae5635f0f55a
-  data.tar.gz: 76e5a0d5dd1343274848660e7ba2b6389b14c955551117b1b9a0821a994960ece32a46e6bedfccd4f2e9ddae3326aa37e3f11cfd3bea70edf78a76c3c6e3961f
+  metadata.gz: 3204bb1c094c7765ca32942b76857774e6696070b3b1787057b06da5f185d55de3069397bb1deb1dbf7221daf22eb064dddcb7a7906471e40fddafd5cf14cb38
+  data.tar.gz: 28c7d2ff21d9c8f97a2db27694c1e21184c9549d5af9cbebbef7244e3690d14b13aa546f79d006eb14104189ca3246d1416a6bb1679d59cfe80353d59511e6e5

data/CHANGELOG.md

@@ -5,6 +5,7 @@
 - **`config.boot_timeout` (default 120 seconds): the supervisor kills a child that never checks in (issue #587).** A fork checks in when it registers its `pgbus_processes` row on this host or writes its first liveness byte. One that has done neither `boot_timeout` seconds after the fork is logged at ERROR, sent `TTIN`, given 2 seconds, then SIGKILLed, and the normal restart path forks it again. A boot-deadline kill always counts as a crash, whatever the child's uptime, so a boot that hangs every time is re-forked with the crash-loop backoff (1 s doubling to 60 s) instead of every `boot_timeout`. Applies to workers, consumers, the dispatcher, the scheduler and the outbox poller. `nil` turns it off; anything else must be a positive number, or `validate!` raises `ConfigurationError`. When the watchdog's read of the process table fails, the boot deadline waits for a pass that can read it. Refs #587.
 - **A process whose registration in `pgbus_processes` failed at start retries it on each heartbeat (issue #587).** It used to stay without a row for good after one failed `create!` (a database blip at boot); a dispatcher, scheduler or outbox poller in that state would now be killed by the boot deadline even though it runs. Refs #587.
 - **`kill -TTIN <child pid>` logs every thread of a pgbus child with its backtrace at ERROR; the child keeps running (issue #587).** The watchdog sends it before every SIGKILL (boot timeout or stalled loop), so a hang is diagnosable from the log. Best effort: a child blocked in native code holding the GVL does not answer. Refs #587.
+- **`config.health_unclaimed_grace` (default 60 seconds) and the per-queue metric `oldest_unread_claimable_age_sec` (issue #589).** The grace is how long a visible, never-read (`read_ct = 0`) message must have been claimable before the health verdict counts it as "never claimed while draining processes are alive". It takes a number of seconds or an `ActiveSupport::Duration`; `0` restores the old rule (any such message counts), and `validate!` raises `ConfigurationError` on nil, a negative or a non-finite value. `Web::DataSource#queues_with_metrics` and `#queue_detail` report `oldest_unread_claimable_age_sec`, the seconds since the oldest visible `read_ct = 0` message became claimable (`min(vt)`, so a delayed job's age starts when its wait expires and a retried message is left out), nil when there is none. The verdict's `summary` carries it as the oldest over the drained queues, and the STALLED reason names it (`never claimed, oldest claimable for 74s`). `Pgbus::MCP::HealthAnalyzer.new` takes an optional `grace:`. Refs #589.
 - **The supervisor's standalone `/readyz` body carries `forked` (issue #587).** `forked` is the fork-table size (what `live` used to be); `live` now counts the children that have checked in. `Pgbus::Process::ReadinessSnapshot` gains `forked:`, defaulting to `live`. Refs #587.
 
 ### Changed
@@ -13,6 +14,7 @@
 
 ### Fixed
 
+- **`/readyz` and `pgbus_health` no longer flap to `STALLED` on a healthy fleet (issue #589).** The verdict reported `STALLED` (and the mounted `/readyz` 503) whenever a drained queue held one visible, never-read message while a worker or consumer was alive, however briefly it had been claimable: every enqueue and every delayed job whose wait just expired is such a message until a worker's next poll, so a busy install answered 503 on a large share of checks. That branch now counts a queue only once its oldest such message has been claimable for `health_unclaimed_grace` seconds; a wedged worker still reads `STALLED` after the grace. A queue hash without the age (an older or stubbed data source) still counts at once. The worker-reported stalled branch (heart-beating, claim loop not advancing) is unchanged. Refs #589.
 - **A child that hangs before its first liveness signal is killed and forked again, and `/readyz` no longer counts it as live (issue #587).** A fork stuck between `fork` and its first check-in (in an initializer, the eager load or a connection) never exited, so nothing reaped it, and the watchdog skipped it because no liveness channel had spoken; the container-local `/readyz` counted it as live because it was in the fork table, so a rolling deploy passed its health gate with a dead capsule. The boot deadline above now kills it, and `/readyz` stays at 503 `DEGRADED` (`live < expected`) until every child has checked in. A dispatcher, scheduler or outbox poller checks in through its process row, which the watchdog reads every 10 seconds, so `/readyz` turns 200 up to 10 seconds after the last one registers, and stays at 503 while the database is unreachable at boot (those children cannot be seen to be alive). The watchdog reads only this host's process rows last beaten since the fork, so a row another container wrote under the same pid, or one a SIGKILLed predecessor left behind (a container restarted in place reuses its hostname and pids), no longer vouches for a local child or hides a stalled loop. Refs #587.
 - **A failed job's retry backoff applies under priority routing (issue #582).** With `priority_levels > 1` the executor set the backoff's visibility timeout on the logical queue's base table, which priority routing never creates; the call raised, was logged at debug, and the job came back after the read's visibility timeout instead of its backoff delay. The backoff now goes to the `_pN` sub-queue the worker read the message from, as the visibility heartbeat and the archive already did. Refs #582.
 - **Two priority sub-queues' failures with the same msg_id keep a row each (issue #583).** Each `_pN` sub-queue numbers its messages from its own sequence, and `pgbus_failed_events` was unique on the logical queue and the msg_id, so a second job failing with the same msg_id in another sub-queue overwrote the first one's error and kept its payload: the dashboard showed one failed job where two had failed, Retry and Discard acted on the first job's message and deleted the only row, and the first job's success cleared the second one's failure. Rows now record the physical table (`source_queue`) and are unique on `(queue_name, msg_id, source_queue)`. The executor, the event consumer and the worker's zombie check read, write and clear only the row of the table the message came from. On the Jobs page a failed row joins only its own table's message and shows as an orphan once that message is gone, Retry, Discard and Discard all look the message up in the table the row names, and Discard releases the job's uniqueness lock bounded by when that table's message was sent, not by a sibling sub-queue's. A row written before the migration has no `source_queue` and is treated as before: it can belong to any table of its queue. Refs #583.

data/README.md

@@ -1260,7 +1260,7 @@ All tools below are read-only — none mutates state, and there is no raw-SQL pa
 
 | Tool | Purpose |
 |------|---------|
-| `pgbus_health` | One-call verdict: `OK` / `DEGRADED` / `STALLED`. STALLED is the silent-worker-wedge signal (visible backlog while workers heart-beat but don't claim). Suitable for automated alerting. |
+| `pgbus_health` | One-call verdict: `OK` / `DEGRADED` / `STALLED`. STALLED is the silent-worker-wedge signal: a worker reports its claim loop stalled, or a visible backlog older than `health_unclaimed_grace` (default 60s) while workers heart-beat but don't claim. The summary carries `oldest_unread_claimable_age_sec`. Suitable for automated alerting. |
 | `pgbus_queues` | All queues: depth, visible count, oldest-message age, paused state. |
 | `pgbus_queue_detail` | Per-queue metrics + paused state + table health (dead tuples, bloat, vacuum age). |
 | `pgbus_processes` | Every process with kind, pid, heartbeat age, and `healthy`/`stale`/`stalled` status. |
@@ -1348,7 +1348,7 @@ For the **HTTP** transport, point the client at the mounted URL with a streamabl
 
 For orchestrators like Kubernetes, Pgbus exposes two HTTP probes: `/livez` (is the serving process up?) and `/readyz`. Readiness means different things in the two places the probes are served:
 
-- **Mounted in Rails** (`Pgbus::Web::HealthApp`): `/readyz` runs the cluster-wide `OK` / `DEGRADED` / `STALLED` verdict, same as the MCP `pgbus_health` tool — `STALLED` (visible backlog while workers heart-beat but don't claim) fails readiness.
+- **Mounted in Rails** (`Pgbus::Web::HealthApp`): `/readyz` runs the cluster-wide `OK` / `DEGRADED` / `STALLED` verdict, same as the MCP `pgbus_health` tool — `STALLED` (a never-claimed backlog older than `health_unclaimed_grace` while workers heart-beat but don't claim) fails readiness.
 - **Standalone from the supervisor** (`health_port`): `/readyz` is **container-local** — did *this* supervisor finish booting, and are all the children *it* forked alive? That is the signal a rolling deploy's health gate needs; the cluster verdict would let a brand-new container pass on the strength of the *old* container's workers.
 
 | Path | Method | 200 | 503 | Touches DB |
@@ -2560,6 +2560,7 @@ Curated headline options for the README. The full operator reference (with types
 | `statsd_port` | `8125` | StatsD UDP port (used when `metrics_backend = :statsd`) |
 | `health_port` | `nil` | Port for standalone HTTP liveness/readiness probes served by the supervisor; nil disables |
 | `health_bind` | `"127.0.0.1"` | Bind address for the standalone health server |
+| `health_unclaimed_grace` | `60` | Seconds a visible, never-read message must have been claimable before the health verdict reports it as never claimed (`STALLED`); `0` counts any such message |
 | `pgmq_schema_mode` | `:auto` | PGMQ schema install mode (`:auto`, `:extension`, `:embedded`) |
 | `doctor_on_boot` | `nil` | Run doctor inside the booting supervisor: `nil`/`false` = off, `:report` = log and boot, `:strict` = refuse to boot on a fatal check. Also via `--doctor` / `--doctor-strict`. |
 | `eager_validation` | `true` | Run `Configuration#validate!` automatically after the `Pgbus.configure` block yields; an invalid value raises `Pgbus::ConfigurationError` at boot. Set `false` to suppress and validate manually. |

data/lib/pgbus/configuration.rb

@@ -241,6 +241,14 @@ module Pgbus
     # plain TCPServer bound to `health_bind` (default localhost).
     attr_accessor :health_port, :health_bind
 
+    # Seconds a visible, never-read (read_ct = 0) message must have been
+    # claimable before the health verdict counts it as "never claimed while
+    # draining processes are alive" (issue #589). Below it a worker's next
+    # poll is still coming, so a fresh enqueue or a delayed job whose wait just
+    # expired no longer reads STALLED. 0 restores the pre-0.22 rule (any
+    # visible read_ct = 0 row counts). Default 60.
+    attr_reader :health_unclaimed_grace
+
     # Streams (turbo-rails replacement, SSE-based)
     attr_accessor :streams_enabled, :streams_path, :streams_signed_name_secret,
                   :streams_default_retention, :streams_retention, :streams_heartbeat_interval, :streams_peer_close_interval,
@@ -419,6 +427,7 @@ module Pgbus
       # HTTP health endpoints — nil port disables the standalone server.
       @health_port = nil
       @health_bind = "127.0.0.1"
+      @health_unclaimed_grace = 60
 
       @streams_enabled = true
       @streams_path = nil
@@ -951,6 +960,8 @@ module Pgbus
 
       raise Pgbus::ConfigurationError, "health_bind must be a non-empty String" unless health_bind.is_a?(String) && !health_bind.empty?
 
+      validate_health_unclaimed_grace!
+
       raise Pgbus::ConfigurationError, "statsd_host must be a non-empty String" unless statsd_host.is_a?(String) && !statsd_host.empty?
 
       unless statsd_port.is_a?(Integer) && statsd_port.between?(1, 65_535)
@@ -1539,6 +1550,14 @@ module Pgbus
       @recurring_execution_retention = coerce_duration!(value, :recurring_execution_retention)
     end
 
+    # Not coerce_duration!: 0 is valid here (the pre-0.22 rule), and nil is not
+    # a "disabled" sentinel. A Duration becomes integer seconds; anything else
+    # is kept for validate! to judge.
+    def health_unclaimed_grace=(value)
+      duration = defined?(ActiveSupport::Duration) && value.is_a?(ActiveSupport::Duration)
+      @health_unclaimed_grace = duration ? value.to_i : value
+    end
+
     # recurring_tasks_file (singular) is deprecated in favor of the plural
     # recurring_tasks_files. The engine still writes the singular internally to
     # record which default file was loaded, so it stays a plain accessor; the
@@ -1867,6 +1886,14 @@ module Pgbus
       raise Pgbus::ConfigurationError, "immediate_shutdown_grace must be a finite number >= 0"
     end
 
+    # A finite real number >= 0; nil is not "off" (0 is).
+    def validate_health_unclaimed_grace!
+      grace = health_unclaimed_grace
+      return if grace.is_a?(Numeric) && grace.real? && grace.finite? && grace >= 0
+
+      raise Pgbus::ConfigurationError, "health_unclaimed_grace must be a finite number of seconds >= 0, got #{grace.inspect}"
+    end
+
     def validate_positive_duration!(numeric, name)
       raise Pgbus::ConfigurationError, "#{name} must be a positive number, got #{numeric}" unless numeric.positive?
 

data/lib/pgbus/mcp/health_analyzer.rb

@@ -35,8 +35,12 @@ module Pgbus
       # a multi-KB log line (issue #367 minor).
       NAME_LIMIT = 10
 
-      def initialize(data_source)
+      # +grace+ overrides Pgbus.configuration.health_unclaimed_grace (issue #589):
+      # seconds a visible read_ct=0 message must have been claimable before it
+      # counts as "never claimed". nil reads the config when the verdict runs.
+      def initialize(data_source, grace: nil)
         @data_source = data_source
+        @grace = grace
       end
 
       # Returns a machine-readable verdict hash suitable for both interactive
@@ -81,7 +85,9 @@ module Pgbus
           status: status,
           reasons: stalled + degraded,
           checked_at: Time.now.utc.iso8601,
-          summary: build_summary(queues, non_dlq, processes, health)
+          summary: build_summary(queues, non_dlq, processes, health).merge(
+            oldest_unread_claimable_age_sec: oldest_unread_age(drained)
+          )
         }
       end
 
@@ -139,7 +145,8 @@ module Pgbus
                      "while #{active_backlog.size} queue(s) have visible backlog: #{backlog_names(active_backlog)}"
         elsif live_drainers.any? && drained_backlog.any? && any_unread?(drained_backlog)
           unread = drained_backlog.select { |q| unread?(q) }
-          reasons << "#{unread.size} queue(s) have visible messages with read_ct=0 (never claimed) " \
+          reasons << "#{unread.size} queue(s) have visible messages with read_ct=0 " \
+                     "(never claimed#{unread_age_clause(unread)}) " \
                      "while #{live_drainers.size} draining process(es) are alive: #{backlog_names(unread)}"
         end
         reasons
@@ -211,10 +218,39 @@ module Pgbus
       #
       # When metrics don't expose the count (an old hash / a stubbed test) we
       # conservatively treat the queue as unread so we never miss the wedge.
+      #
+      # Issue #589: the oldest such row must also have been claimable for at
+      # least the grace. A fresh enqueue, or a delayed job whose wait just
+      # expired, is visible with read_ct=0 until a worker's next poll, so
+      # without the grace a healthy fleet read STALLED (and /readyz 503) on a
+      # large share of checks. An unknown age (key missing or nil) stays
+      # conservative and counts.
       def unread?(queue)
         return true unless queue.key?(:visible_unread_length)
+        return false unless queue[:visible_unread_length].to_i.positive?
+
+        age = unread_age(queue)
+        age.nil? || age >= grace
+      end
+
+      def grace
+        @grace || Pgbus.configuration.health_unclaimed_grace
+      end
+
+      def unread_age(queue)
+        queue[:oldest_unread_claimable_age_sec]
+      end
+
+      # Oldest known unread claimable age over +queues+, nil when none is known.
+      def oldest_unread_age(queues)
+        queues.filter_map { |q| unread_age(q) }.max
+      end
 
-        queue[:visible_unread_length].to_i.positive?
+      # ", oldest claimable for 74s" so an operator can tell a wedge from a
+      # spike; empty when no flagged queue reports an age.
+      def unread_age_clause(queues)
+        age = oldest_unread_age(queues)
+        age ? ", oldest claimable for #{age}s" : ""
       end
 
       # Join backlog queue names for a reason string, truncated so a flagged

data/lib/pgbus/mcp/tools/health_tool.rb

@@ -13,8 +13,10 @@ module Pgbus
         description <<~DESC
           Top-level pgbus health verdict. Returns OK, DEGRADED, or STALLED with
           the specific reasons. STALLED means the silent-wedge condition:
-          queues have visible backlog while workers are heart-beating but not
-          claiming (or a worker's claim loop has stalled). DEGRADED covers
+          queues hold a visible, never-claimed backlog older than
+          health_unclaimed_grace (default 60s) while workers are heart-beating
+          but not claiming (or a worker's claim loop has stalled); the summary's
+          oldest_unread_claimable_age_sec tells a wedge from a spike. DEGRADED covers
           stale processes, paused queues holding a backlog, dead-letter
           buildup, or a long-running transaction pinning the MVCC horizon. Use
           this first when triaging — it summarizes the whole system in one call

data/lib/pgbus/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Pgbus
-  VERSION = "0.21.2"
+  VERSION = "0.21.3"
 end

data/lib/pgbus/web/data_source.rb

@@ -1340,7 +1340,8 @@ module Pgbus
               (SELECT CASE WHEN is_called THEN last_value ELSE 0 END FROM pgmq.#{seq_name}) AS total_messages,
               (SELECT max(read_ct) FROM pgmq.#{qtable}) AS max_read_ct,
               (SELECT count(*) FROM pgmq.#{qtable} WHERE vt <= NOW() AND read_ct = 0) AS visible_unread_length,
-              (SELECT EXTRACT(epoch FROM (NOW() - min(vt)))::int FROM pgmq.#{qtable} WHERE vt <= NOW()) AS oldest_claimable_age_sec
+              (SELECT EXTRACT(epoch FROM (NOW() - min(vt)))::int FROM pgmq.#{qtable} WHERE vt <= NOW()) AS oldest_claimable_age_sec,
+              (SELECT EXTRACT(epoch FROM (NOW() - min(vt)))::int FROM pgmq.#{qtable} WHERE vt <= NOW() AND read_ct = 0) AS oldest_unread_claimable_age_sec
           SQL
         rescue StandardError => e
           Pgbus.logger.debug { "[Pgbus::Web] Skipping queue metrics for #{name}: #{e.message}" }
@@ -1362,7 +1363,8 @@ module Pgbus
             newest_msg_age_sec: row["newest_msg_age_sec"]&.to_i,
             total_messages: row["total_messages"].to_i,
             max_read_ct: row["max_read_ct"]&.to_i,
-            visible_unread_length: row["visible_unread_length"].to_i
+            visible_unread_length: row["visible_unread_length"].to_i,
+            oldest_unread_claimable_age_sec: row["oldest_unread_claimable_age_sec"]&.to_i
           }
         end
       rescue StandardError => e
@@ -1382,6 +1384,7 @@ module Pgbus
               EXTRACT(epoch FROM (NOW() - max(enqueued_at)))::int AS newest_msg_age_sec,
               EXTRACT(epoch FROM (NOW() - min(enqueued_at)))::int AS oldest_msg_age_sec,
               EXTRACT(epoch FROM (NOW() - min(vt) FILTER (WHERE vt <= NOW())))::int AS oldest_claimable_age_sec,
+              EXTRACT(epoch FROM (NOW() - min(vt) FILTER (WHERE vt <= NOW() AND read_ct = 0)))::int AS oldest_unread_claimable_age_sec,
               max(read_ct) AS max_read_ct,
               count(CASE WHEN vt <= NOW() AND read_ct = 0 THEN 1 END) AS visible_unread_length
             FROM pgmq.#{qtable}
@@ -1398,6 +1401,7 @@ module Pgbus
             q_summary.oldest_claimable_age_sec,
             q_summary.max_read_ct,
             q_summary.visible_unread_length,
+            q_summary.oldest_unread_claimable_age_sec,
             all_metrics.total_messages
           FROM q_summary, all_metrics
         SQL
@@ -1414,7 +1418,8 @@ module Pgbus
           newest_msg_age_sec: row["newest_msg_age_sec"]&.to_i,
           total_messages: row["total_messages"].to_i,
           max_read_ct: row["max_read_ct"]&.to_i,
-          visible_unread_length: row["visible_unread_length"].to_i
+          visible_unread_length: row["visible_unread_length"].to_i,
+          oldest_unread_claimable_age_sec: row["oldest_unread_claimable_age_sec"]&.to_i
         }
       rescue StandardError => e
         Pgbus.logger.error { "[Pgbus::Web] Error fetching metrics for #{queue_name}: #{e.class}: #{e.message}" }

data/sig/pgbus/configuration.rbs

@@ -85,6 +85,8 @@ module Pgbus
     def health_bind: () -> String
     def health_bind=: (untyped) -> untyped
     def health_port: () -> untyped
+    def health_unclaimed_grace: () -> untyped
+    def health_unclaimed_grace=: (untyped value) -> untyped
     def health_port=: (untyped) -> untyped
     def idempotency_ttl: () -> Numeric?
     def idempotency_ttl=: (untyped value) -> untyped

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: pgbus
 version: !ruby/object:Gem::Version
-  version: 0.21.2
+  version: 0.21.3
 platform: ruby
 authors:
 - Mikael Henriksson