pgbus 0.13.1 → 0.13.3

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 8c955f8fb2ac4e66fb2db388505ee83341aa6dec7c00f606162f7fba7c29cb75
-  data.tar.gz: a31e43c6f48cb10609d73dc507e9d177d25a299ae25c6f5d5a39a045018c32e8
+  metadata.gz: 2e9af048ef1aa2d0fecab62f5471ca9ab6c3e2595353c7eb1a13cde57c37564a
+  data.tar.gz: 29a35e789733ac394136eaea8fc2b7f2e709617ceee818dd2b2752d9b017d45d
 SHA512:
-  metadata.gz: 7cd3fba720a5ab724fe4f0b5b89168cb2df4ba68e0a5a35cca725093724f8f2b2109c379a2263442111367001a52d2eda440eef8e1c2469653d2022d2a92d148
-  data.tar.gz: 4a4e5e1c2c229c599541e61ac6ad5602dfbfcf057b346b22b0271c10a48752c41075c7166b43dd55ece7ef4a30d79ec3fd6f5cac5b898b2578ecb90c73bcc2a8
+  metadata.gz: 392a67ed233feb5ffe0a98cf597efe3962b9e7c81d07a7e351f6c4e8aab8c03b3edc8e55204c3f0d0a4b82287b7f523cada5ca79e0684752f50c8f3272c9bdc0
+  data.tar.gz: 95b48a0ae66aaa239da08fa3c235a412241ddcfacea8384d84a66ca298308678062dc0192592b07484596fe12e27145a71474d5547b5046370bfca2785ae0f03

data/CHANGELOG.md

@@ -1,7 +1,17 @@
 ## [Unreleased]
 
+### Fixed
+
+- **Queue-age metrics no longer count vt-parked (scheduled/retrying) messages — one delayed job stops reading as a degraded queue (issue #389).** ⚠️ **Behavior change on the AppSignal `pgbus_queue_latency` gauge.** pgmq's `oldest_msg_age_sec` is computed from `enqueued_at` and ignores `vt`, but a job enqueued with `wait:` or parked on a long retry backoff lives in the queue table with a future `vt` — that *is* the delayed-delivery mechanism. So a single parked message made the age metric grow at wall-clock rate for hours on an otherwise drained queue, and any latency alert thresholding on it fired continuously ("oldest message is 17045s old" on a healthy queue with depth 1, `read_ct` 0). Every metrics surface now also exposes **`oldest_claimable_age_sec`** — `now() - min(vt)` over rows with `vt <= now()`, i.e. the age of the oldest message actually *eligible for pickup*: an immediately-enqueued message contributes from enqueue time (matching the old number on a plain backlog), a scheduled/backoff-parked message contributes nothing until due, an in-flight message (vt pushed forward) is excluded, and nil means "no claimable backlog" even when the table is non-empty. Surfaces: `Web::DataSource` (dashboard, JSON API, MCP `pgbus_queues` tool), a new Prometheus gauge `pgbus_queue_oldest_claimable_age_seconds`, a new AppSignal gauge `pgbus_queue_oldest_claimable_age_seconds`, `Pgbus::Client#oldest_claimable_ages` (raw-SQL reader, since pgmq's `metrics_result` type is frozen upstream), and a CLAIMABLE column in `pgbus queues`. The AppSignal **`pgbus_queue_latency` gauge now derives from the claimable age** and always emits — `(claimable_age || 0) * 1000`, 0 = no claimable backlog — so existing latency alerts stop false-firing with no dashboard changes; the raw `pgbus_queue_oldest_message_age_seconds` gauge keeps its enqueue-time semantics everywhere. The dashboard queue tables additionally split depth into **Parked** (`depth − visible`) and show the claimable age in place of the raw age, so a queue holding only backoff retries reads visibly healthy. Refs #389.
+
 ### Added
 
+- **Health-checked rolling restarts for the job container (issue #386).** ⚠️ **Behavior change on the standalone `/readyz`.** The supervisor's `health_port` server previously answered `/readyz` with the cluster-wide HealthAnalyzer verdict — so during a rolling deploy a freshly-booted container could pass an orchestrator's health gate on the strength of the *old* container's still-heartbeating workers, and the old container (with all its capacity) was stopped before the new one had forked a single child. The standalone `/readyz` is now **container-local**: 200 only when *this* supervisor verified its connection, bootstrapped queues, forked every configured child, and all of them are currently alive — with 503 bodies `BOOTING` (pre-boot), `DEGRADED` (a child died and is waiting out crash-restart backoff — precisely the state a deploy gate must fail on, keeping the old container running), and `DRAINING` (stop signal received). No database access on the probe path; the supervisor publishes an immutable snapshot per monitor pass and the accept thread reads it. The Rails-mounted `Pgbus::Web::HealthApp` keeps the cluster-wide verdict unchanged. Alongside it: **`pgbus-health`**, a shipped executable probe for docker `HEALTHCHECK` blocks (plain Ruby + stdlib sockets, loads neither Bundler nor the gem — cheap at 1–5s intervals, works in curl-less images; exit 0/1/2 = healthy/unhealthy/usage), and a README "Rolling restarts (Kamal, docker)" guide covering the healthcheck block, stop-timeout alignment, overlap-window duplicate-supervisor safety, and the `read_ct`-vs-deploy-kill DLQ caveat. Refs #386.
+
+### Changed
+
+- **Shutdown budgets are now alignable end-to-end (issue #386).** New `config.shutdown_timeout` bounds how long the supervisor waits for children after forwarding TERM before escalating to SIGKILL — previously a hardcoded 30s, which silently SIGKILLed workers mid-drain the moment `drain_timeout` was raised past it. Default derives `drain_timeout + 5` so the deadline tracks the drain window automatically; an explicit value below `drain_timeout` logs a boot warning. `Consumer#shutdown`'s pool wait (its only drain bound) now follows `config.drain_timeout` instead of a hardcoded 30s, and `Worker#shutdown`'s post-drain residual wait drops from a second full 30s window to 5s — the drain loop already waited `drain_timeout`, and a job still running has proven it won't finish. Rule of thumb: orchestrator stop grace period > `shutdown_timeout` > `drain_timeout`. Refs #386.
+
 - **Streams: one LISTEN connection per web host — `streams_listen_scope` (issue #382).** ⚠️ **Default behavior change.** Previously every Puma worker lazily opened its own dedicated streams LISTEN connection on first SSE use, so a web host pinned one direct connection per worker. Under the new default (`streams_listen_scope = :master`) the `pgbus_streams` Puma plugin runs a **MasterHub** in the preforking master: ONE `Web::Streamer::Listener` on the refcounted union of every worker's stream channels, fanning wakes — **including ephemeral payloads** — out to workers over a Unix domain socket with length-prefixed frames (`Streamer::HubProtocol`). Workers connect lazily (nothing is inherited across fork) and the synchronous `ensure_listening` ack contract is preserved cross-process: a sub is registered before LISTEN executes and acked only after, so the no-lost-broadcast guarantee holds. Backpressure follows the streams rules: durable wakes are droppable at a per-worker cap (they self-heal via `read_after`), **ephemeral wakes are never dropped** — a worker that stops draining is evicted, which triggers its own fallback. **Fallback is per-worker listeners, not loss**: whenever the hub is absent or dies (no `preload_app!`, single-mode Puma, crash, eviction) each worker's `FailoverListener` swaps in a real per-worker `Listener` and re-LISTENs its recorded subscriptions — connection footprint balloons back to pre-#382 levels (census-visible) but no broadcast semantics change; the worker stays local until it recycles. Measured (local PG, n=50): the master→worker hop is noise-level free — single-broadcast SSE roundtrip p50 16.00ms via the hub vs 16.93ms per-worker. **`:master` effectively requires `preload_app!`** (the hub waits for the app's pgbus initializer; without it the deadline expires quietly and workers stay per-worker). **Rollback:** `config.streams_listen_scope = :process`. Refs #382, builds on the #381 patterns.
 
 - **Host-level shared LISTEN: `worker_notify_scope` — the supervisor now owns ONE direct LISTEN connection for the whole host (issue #381).** ⚠️ **Default behavior change.** Previously every worker fork and every consumer fork opened its own dedicated LISTEN connection (`NotifyListener`), so a host's direct-connection footprint scaled with fork count — on transaction-pool PgBouncer platforms those connections come out of the scarcest slice of `max_connections`, and a 5-capsule + 2-consumer host pinned 7. Under the new default (`config.worker_notify_scope = :supervisor`) the supervisor runs a single `NotifyHub`: one `NotifyListener` on the union of every capsule's and consumer's queue channels (wildcards via the shared resolver, consumer sets via the registry), fanning wakes out to forks over per-fork pipes (`W` wake / `H` healthy / `P` degraded bytes; a fork whose pipe reports degraded or reaches EOF falls back to fast polling exactly like a failed local listener). Footprint drops to **1 direct LISTEN connection per job host**, verified by integration test: routing is per-fork (an insert wakes only the forks reading that queue, wildcard capsules unconditionally), and `pg_terminate_backend` on the shared connection is survived — reconnect, re-LISTEN, wakes flow again. **Rollback:** `config.worker_notify_scope = :fork` restores the previous per-fork listeners byte-for-byte. Dedicated LISTEN connections are now census-tagged `application_name=pgbus-listen` so `pg_stat_activity` can count them. Refs #381.
@@ -16,6 +26,8 @@
 
 ### Fixed
 
+- **`idempotent!` handlers no longer silently drop an execution when the consumer is killed mid-handler — the idempotency claim is now two-phase (issue #385).** `EventBus::Handler#process!` inserted the `pgbus_processed_events` claim row *before* `handle(event)` ran, so a SIGKILL between claim and completion (supervisor watchdog, container runtime after its stop timeout, OOM — i.e. every container-replacement deploy's worst case) turned the redelivered message into a skip: the mechanism meant to dedup *duplicate* executions converted a crash *during* the one permitted execution into at-most-once. The claim now lands with a NULL `completed_at` (*pending*) and is stamped completed only after `handle` returns; on redelivery a pending claim **re-runs** the handler while a completed claim still skips. The in-memory dedup cache only ever records completed executions, and a `handle` that raises leaves the claim pending so VT redelivery retries it. The only double-execution window is a still-alive handler running past its visibility timeout — the same at-least-once window every non-idempotent handler already has. **Upgrading:** new installs get the column from `pgbus:install`; existing installs run `rails generate pgbus:add_processed_event_completion` (also picked up by `pgbus:update`; `--database` supported) — legacy rows are backfilled as completed so history is not retroactively re-run. Until the migration runs, an upgraded gem detects the missing column once and falls back to the legacy single-phase claim with a warning naming the generator. Refs #385.
+
 - **SIGTERM no longer segfaults the process: both LISTEN listeners stopped closing their PG connection from the stopping thread (issue #375).** `Process::NotifyListener#stop` (supervisor thread, on SIGTERM) closed the listener's dedicated `PG::Connection` to interrupt the blocking `wait_for_notify`, but left `@conn` pointing at it — so the listener thread unwound into `run_loop`'s `ensure` and ran `safe_unlisten_all`, exec'ing `UNLISTEN` on a connection the other thread was concurrently freeing. `PG::Connection#close` is `PQfinish`: it frees the PGconn **and its OpenSSL objects**, so `PQsendQuery` walked into freed TLS state and the whole worker died with `[BUG] Segmentation fault` — reproducibly, on essentially every container-replacement deploy against a TLS Postgres, sometimes in several forked workers at once. `rescue PG::Error` cannot catch a C-level SEGV, so the "safe" in `safe_unlisten_all` never held. **The connection is now single-owner**: the listener thread is the only thread that may exec, wait, or close on it, from build through teardown. `#stop` signals by clearing `@running` and joining — nothing more — and the listener thread closes its own connection in `run_loop`'s `ensure`. The stop is observed within one `wait_for_notify` timeout (`health_check_ms`, **250 ms** in the default configuration since workers/consumers derive it from `polling_interval`), so `#stop`'s join budget is now `health_check_ms + 5s` instead of a flat 5 s — a flat timeout could expire before a listener with a large `health_check_ms` had even one chance to observe the stop. The teardown `UNLISTEN` round-trip is **dropped entirely** rather than merely made safe: it ran immediately before the close, and closing a session deregisters every `LISTEN` server-side, so it bought nothing at any time. The identical defect in `Web::Streamer::Listener` — same close-from-`#stop`, same `UNLISTEN`-in-`ensure`, and there without even a mutex around `@conn` — is fixed the same way, so the dashboard's Puma worker stops crashing on the same deploy; its listener thread now also closes its own connection (previously only `#stop` did). Cost: shutdown can take up to one health-check cycle longer per listener — measured against a real PostgreSQL at the default `health_check_ms` of 250: **~207 ms on a fully idle queue, ~5 ms when the queue has any NOTIFY traffic** (a notification returns the wait immediately, and the loop then sees the cleared flag). 40 start/stop cycles under concurrent NOTIFY load leaked zero `LISTEN` backends. Thanks to the reporter for the crash dumps and the root-cause analysis. Refs #375.
 - **The health verdict no longer emits false STALLED reports — `max_read_ct` was never populated, and the wedge signal counted queues no capsule drains (issue #367).** Two correctness defects in `Pgbus::MCP::HealthAnalyzer` (surfaced through `pgbus doctor` / `pgbus_health` / the MCP health tool). **(1)** The `all_unread?` wedge check read `:max_read_ct`, but the metrics query never selected it — so the guard always degenerated to "never claimed" and a busy-but-healthy queue caught mid-burst produced a STALLED verdict with a factually wrong "read_ct=0 (never claimed)" reason. The metrics query now exposes a per-queue **`visible_unread_length`** (`count(*) WHERE vt <= NOW() AND read_ct = 0`) and the analyzer keys the wedge off *visible, never-claimed* messages. Counting per visible message (not `max(read_ct)` over the whole table) means one retried message left in-queue after its backoff, or an in-flight message claimed by a peer, can no longer veto the signal for a pile of genuinely-unclaimed jobs. **(2)** The verdict reasoned about **every** non-DLQ/non-stream queue against the global worker fleet — but a worker can only claim from queues its capsule subscribes to, so "M workers alive but never claimed" was vacuous for a queue nobody drains (ad-hoc queues, unregistered stream queues — see #366). The analyzer now intersects the STALLED backlog with `Web::DataSource#drained_queue_names` (each configured capsule's queues, priority `_pN` sub-tables expanded via the client's queue strategy, unioned with EventBus handler queues; `nil` for a `*` wildcard = drains everything, fail-open on error). Queues nobody drains get their own DEGRADED signal — *"N queue(s) hold messages but no capsule is configured to drain them"* — instead of being folded into the worker-wedge verdict. A heart-beating-but-`:stalled` worker is still reported STALLED regardless of which queue holds the backlog. Reason strings now truncate to the first 10 queue names with `(+N more)` so a flagged fleet of hundreds of queues doesn't produce a multi-KB log line. Refs #367, #366.
 - **`allowed_global_id_models` now actually guards ActiveJob arguments, not only EventBus payloads (issue #368).** The doctor warned in production that `nil` means "allow-all GlobalID arguments", but the allowlist was only enforced in `Serializer.locate_global_id` — reached from EventBus `_global_id` payloads — while the ordinary job path (`Executor` → `ActiveJob::Base.deserialize` → Rails' unrestricted `GlobalID::Locator`) never checked it. Operators who set an allowlist after following the doctor had a false sense of security; the common `SomeJob.perform_later(record)` pattern was unguarded. Job deserialization now goes through `Serializer.deserialize_job_data`, which walks `_aj_globalid` keys (including nested arrays/hashes) and reuses the same gate as EventBus when the allowlist is set; `nil` remains zero-cost allow-all. Rejected models raise `Pgbus::SerializationError` and are treated as a normal job failure. Apps with ActiveStorage attachments should include `ActiveStorage::Blob` (and related models) on the allowlist. Docs + doctor copy updated. Refs #368.

data/README.md

@@ -221,6 +221,20 @@ Pgbus::EventBus::Registry.instance.subscribe(
 )
 ```
 
+`idempotent!` uses a **two-phase claim**: a *pending* row in
+`pgbus_processed_events` is inserted before `handle` runs, and only stamped
+`completed_at` after `handle` returns. Deduplication applies to **completed**
+executions only — if the consumer process is killed mid-handler (deploy,
+OOM, supervisor watchdog), the redelivered message finds the pending claim
+and **re-runs the handler** instead of silently skipping it. The semantics
+are at-least-once with dedup of completed executions: the only
+double-execution window is a handler still running past its visibility
+timeout — the same window every non-idempotent handler already has.
+Installs created before this feature need the upgrade migration:
+`rails generate pgbus:add_processed_event_completion` (supports
+`--database`); until it runs, idempotent handlers fall back to the old
+single-phase claim and log a warning.
+
 ### 4. Start workers
 
 ```bash
@@ -977,6 +991,7 @@ When `config.metrics_enabled = true` (default), the dashboard exposes Prometheus
 
 | Metric | Description |
 |--------|-------------|
+| `pgbus_queue_oldest_claimable_age_seconds` | Age of the oldest message eligible for pickup (visibility timeout elapsed) — safe to alert on: scheduled/backoff-parked messages don't count until due; the series is omitted entirely when no claimable backlog exists (while the raw `pgbus_queue_oldest_message_age_seconds` gauge may still report a parked message's age) |
 | `pgbus_table_dead_tuples` | Dead tuple count per PGMQ table |
 | `pgbus_table_live_tuples` | Live tuple count per PGMQ table |
 | `pgbus_table_bloat_ratio` | Dead / (dead + live) per table |
@@ -1103,14 +1118,18 @@ For the **HTTP** transport, point the client at the mounted URL with a streamabl
 
 ### Health endpoints (liveness / readiness)
 
-For orchestrators like Kubernetes, Pgbus exposes two HTTP probes: `/livez` (is the serving process up?) and `/readyz` (are queues draining, or is a worker silently wedged?). `/readyz` runs the same `OK` / `DEGRADED` / `STALLED` verdict as the MCP `pgbus_health` tool — `STALLED` (visible backlog while workers heart-beat but don't claim) fails readiness.
+For orchestrators like Kubernetes, Pgbus exposes two HTTP probes: `/livez` (is the serving process up?) and `/readyz`. Readiness means different things in the two places the probes are served:
+
+- **Mounted in Rails** (`Pgbus::Web::HealthApp`): `/readyz` runs the cluster-wide `OK` / `DEGRADED` / `STALLED` verdict, same as the MCP `pgbus_health` tool — `STALLED` (visible backlog while workers heart-beat but don't claim) fails readiness.
+- **Standalone from the supervisor** (`health_port`): `/readyz` is **container-local** — did *this* supervisor finish booting, and are all the children *it* forked alive? That is the signal a rolling deploy's health gate needs; the cluster verdict would let a brand-new container pass on the strength of the *old* container's workers.
 
 | Path | Method | 200 | 503 | Touches DB |
 |---|---|---|---|---|
 | `/livez` | GET | always (`ok`) | never | no |
-| `/readyz` | GET | verdict `OK` or `DEGRADED` | verdict `STALLED`, or DB unreachable (`{"status":"ERROR"}`) | yes |
+| `/readyz` (mounted) | GET | verdict `OK` or `DEGRADED` | verdict `STALLED`, or DB unreachable (`{"status":"ERROR"}`) | yes |
+| `/readyz` (supervisor) | GET | `OK` — booted, all children live | `BOOTING`, `DEGRADED` (child down), `DRAINING` (stopping) | no |
 
-Unknown paths return `404`; non-`GET` methods return `405`. The `/readyz` body is the verdict JSON, so a probe failure is self-describing in the pod's event log.
+Unknown paths return `404`; non-`GET` methods return `405`. The `/readyz` body is JSON, so a probe failure is self-describing in the pod's event log.
 
 #### Mount in your Rails app
 
@@ -1151,6 +1170,61 @@ readinessProbe:
   httpGet: { path: /readyz, port: 9394 }
 ```
 
+The supervisor's `/readyz` answers from its own state, never the database:
+
+```json
+{ "status": "OK", "expected": 3, "live": 3 }
+```
+
+- `BOOTING` (503) until the connection is verified, queues are bootstrapped, and every configured child has been forked. `expected` is stamped at that instant.
+- `OK` (200) while all expected children are in the fork table. A clean worker recycle never dips the count — the snapshot refreshes after reap-and-restart each monitor pass.
+- `DEGRADED` (503) when a child died and is waiting out crash-restart backoff. During a rolling deploy this is the desired failure mode: a crash-looping replacement never goes ready, so the old container keeps running.
+- `DRAINING` (503) the moment a stop signal arrives.
+
+#### `pgbus-health`: container HEALTHCHECK probe
+
+`pgbus-health` ships with the gem: a dependency-free probe (plain Ruby + stdlib sockets — no Bundler, no Rails, nothing else loaded) that GETs `127.0.0.1:<port>/readyz` and exits `0` on 200, `1` on anything else, `2` on usage errors. Cheap enough for a 1–5s `HEALTHCHECK` interval, and it works in images without curl:
+
+```bash
+pgbus-health --port 9394              # or PGBUS_HEALTH_PORT=9394 pgbus-health
+pgbus-health --port 9394 --path /livez --timeout 2
+```
+
+### Rolling restarts (Kamal, docker)
+
+Kamal distributions with per-role health checks (for example the [`dash` branch](https://github.com/mhenrixon/kamal)) can rolling-restart a non-proxied job role: start the new container, poll its docker `HEALTHCHECK` until healthy, and only then `docker stop` the old one. Wire the pgbus container into that gate:
+
+```yaml
+# config/deploy.yml
+servers:
+  job:
+    hosts: [...]
+    cmd: bin/pgbus start
+    healthcheck:
+      cmd: bin/pgbus-health --port 9394
+      interval: 5s
+      start_period: 30s     # cover Rails boot + queue bootstrap
+    stop_timeout: 45        # must exceed pgbus shutdown_timeout (see below)
+env:
+  clear:
+    PGBUS_HEALTH_PORT: 9394
+```
+
+(`bundle binstubs pgbus` generates `bin/pgbus-health`; adjust the path if your image invokes gem executables differently.)
+
+**The shutdown timeline.** On `docker stop`, SIGTERM reaches the supervisor and readiness flips to `DRAINING`; children stop claiming work and drain in-flight jobs for up to `drain_timeout` (default 30s); the supervisor waits `shutdown_timeout` (default `drain_timeout + 5`) before SIGKILLing stragglers. Align the three knobs outside-in:
+
+```text
+orchestrator stop_timeout  >  pgbus shutdown_timeout  >  pgbus drain_timeout
+        45s                        35s (derived)                30s
+```
+
+If the orchestrator's stop grace period is *shorter* than `shutdown_timeout`, docker SIGKILLs the whole tree mid-drain and the graceful path never gets to finish. Raising `drain_timeout` raises the derived `shutdown_timeout` automatically; raise `stop_timeout` to match.
+
+**The overlap window is safe by construction.** Between "new container healthy" and "old container stopped", two supervisors run against the same database. Nothing double-fires: queue claims use `FOR UPDATE SKIP LOCKED`, `single_active_consumer` queues arbitrate via session-level advisory locks (released the instant a killed process's connection dies), two live recurring schedulers dedup on the `(task_key, run_at)` unique record, and dispatcher maintenance is idempotent. "One scheduler per deployment" is a steady-state rule; a deploy window may briefly violate it without consequence.
+
+**What a hard kill still costs.** Jobs killed past the drain window are redelivered after their visibility timeout (at-least-once holds) — but PGMQ's `read_ct` increments exactly like a logical failure, so a long-running job that straddles *repeated* deploy kills can be pushed to the DLQ without its code ever raising. `zombie_detection` logs exactly this pattern (`read_ct > 1` with no recorded failure). Keep jobs shorter than `drain_timeout`, or raise it (and `stop_timeout`) for queues that can't be. For `idempotent!` event handlers there is a separate crash-window caveat tracked in [#385](https://github.com/mhenrixon/pgbus/issues/385).
+
 ### Boot diagnostics banner
 
 `Supervisor#run` logs a one-block banner right after the heartbeat starts and before queues bootstrap, so a misconfigured deployment states its actual settings instead of forcing an operator to attach a console. Every line is `"[Pgbus] boot:"`-prefixed and renders cleanly under both the `:text` and `:json` log formatters:
@@ -2097,6 +2171,7 @@ Curated headline options for the README. The full operator reference (with types
 | `zombie_detection` | `true` | Detect and reclaim work from crashed workers |
 | `read_timeout` | `30` | Seconds before a single PGMQ read is bounded (libpq `statement_timeout` + `tcp_user_timeout` on a dedicated connection; nil disables) |
 | `drain_timeout` | `30` | Seconds to wait for in-flight jobs during graceful shutdown before abandoning them |
+| `shutdown_timeout` | `drain_timeout + 5` | Seconds the supervisor waits for children after TERM before SIGKILL; an orchestrator's stop grace period must exceed it |
 | `stall_threshold` | `300` | Seconds without progress before a worker is considered stalled |
 | `priority_levels` | `nil` | Number of priority sub-queues (nil = disabled, 2-10) |
 | `default_priority` | `1` | Default priority for jobs without explicit priority |

data/app/models/pgbus/processed_event.rb

@@ -5,5 +5,50 @@ module Pgbus
     self.table_name = "pgbus_processed_events"
 
     scope :expired, ->(before) { where("processed_at < ?", before) }
+
+    @completion_column_mutex = Mutex.new
+
+    class << self
+      # Whether pgbus_processed_events has the completed_at column that backs
+      # the two-phase idempotency claim (issue #385). Detected once per process
+      # (memoized under a mutex) so the schema probe never lands on the
+      # per-event hot path. An upgraded gem running against a not-yet-migrated
+      # table gets `false` plus a one-time warning pointing at the upgrade
+      # generator — Handler then falls back to the legacy single-phase claim.
+      #
+      # A detection error (e.g. the database is briefly unreachable) is NOT
+      # memoized: it propagates to the caller — where the event's normal
+      # failure path leaves the message for VT redelivery — and the next
+      # delivery probes again.
+      def completion_column?
+        detected = @completion_column
+        return detected unless detected.nil?
+
+        @completion_column_mutex.synchronize do
+          @completion_column = detect_completion_column if @completion_column.nil?
+          @completion_column
+        end
+      end
+
+      # Test seam: clear the memoized detection so specs can exercise both
+      # schema shapes in one process.
+      def reset_completion_column_check!
+        @completion_column_mutex.synchronize { @completion_column = nil }
+      end
+
+      private
+
+      def detect_completion_column
+        supported = column_names.include?("completed_at")
+        unless supported
+          Pgbus.logger.warn do
+            "[Pgbus] pgbus_processed_events is missing the completed_at column; idempotent handlers " \
+              "fall back to single-phase claims (a crash mid-handler can skip work on redelivery). " \
+              "Run `rails generate pgbus:add_processed_event_completion` and migrate."
+          end
+        end
+        supported
+      end
+    end
   end
 end

data/app/views/pgbus/dashboard/_queues_table.html.erb

@@ -12,7 +12,8 @@
             <th class="px-4 py-3 text-left text-xs font-medium uppercase text-gray-500"><%= t("pgbus.dashboard.queues_table.headers.queue") %></th>
             <th class="px-4 py-3 text-right text-xs font-medium uppercase text-gray-500"><%= t("pgbus.dashboard.queues_table.headers.depth") %></th>
             <th class="px-4 py-3 text-right text-xs font-medium uppercase text-gray-500"><%= t("pgbus.dashboard.queues_table.headers.visible") %></th>
-            <th class="px-4 py-3 text-right text-xs font-medium uppercase text-gray-500"><%= t("pgbus.dashboard.queues_table.headers.oldest") %></th>
+            <th class="px-4 py-3 text-right text-xs font-medium uppercase text-gray-500"><%= t("pgbus.dashboard.queues_table.headers.parked") %></th>
+            <th class="px-4 py-3 text-right text-xs font-medium uppercase text-gray-500"><%= t("pgbus.dashboard.queues_table.headers.oldest_claimable") %></th>
             <th class="px-4 py-3 text-right text-xs font-medium uppercase text-gray-500"><%= t("pgbus.dashboard.queues_table.headers.total") %></th>
           </tr>
         </thead>
@@ -25,12 +26,13 @@
               </td>
               <td data-label="Depth" class="px-4 py-3 text-sm text-right text-gray-700"><%= pgbus_number(q[:queue_length]) %></td>
               <td data-label="Visible" class="px-4 py-3 text-sm text-right text-gray-700"><%= pgbus_number(q[:queue_visible_length]) %></td>
-              <td data-label="Oldest" class="px-4 py-3 text-sm text-right text-gray-500"><%= q[:oldest_msg_age_sec] || "—" %></td>
+              <td data-label="Parked" class="px-4 py-3 text-sm text-right text-gray-500"><%= pgbus_number(q[:parked_length]) %></td>
+              <td data-label="Oldest claimable" class="px-4 py-3 text-sm text-right text-gray-500"><%= q[:oldest_claimable_age_sec] || "—" %></td>
               <td data-label="Total" class="px-4 py-3 text-sm text-right text-gray-500"><%= pgbus_number(q[:total_messages]) %></td>
             </tr>
           <% end %>
           <% if @queues.empty? %>
-            <tr><td colspan="5" class="px-4 py-8 text-center text-sm text-gray-400"><%= t("pgbus.dashboard.queues_table.empty") %></td></tr>
+            <tr><td colspan="6" class="px-4 py-8 text-center text-sm text-gray-400"><%= t("pgbus.dashboard.queues_table.empty") %></td></tr>
           <% end %>
         </tbody>
       </table>

data/app/views/pgbus/queues/_queues_list.html.erb

@@ -6,7 +6,8 @@
           <th class="px-4 py-3 text-left text-xs font-medium uppercase text-gray-500"><%= t("pgbus.queues.queues_list.headers.queue") %></th>
           <th class="px-4 py-3 text-right text-xs font-medium uppercase text-gray-500"><%= t("pgbus.queues.queues_list.headers.depth") %></th>
           <th class="px-4 py-3 text-right text-xs font-medium uppercase text-gray-500"><%= t("pgbus.queues.queues_list.headers.visible") %></th>
-          <th class="px-4 py-3 text-right text-xs font-medium uppercase text-gray-500"><%= t("pgbus.queues.queues_list.headers.oldest") %></th>
+          <th class="px-4 py-3 text-right text-xs font-medium uppercase text-gray-500"><%= t("pgbus.queues.queues_list.headers.parked") %></th>
+          <th class="px-4 py-3 text-right text-xs font-medium uppercase text-gray-500"><%= t("pgbus.queues.queues_list.headers.oldest_claimable") %></th>
           <th class="px-4 py-3 text-right text-xs font-medium uppercase text-gray-500"><%= t("pgbus.queues.queues_list.headers.newest") %></th>
           <th class="px-4 py-3 text-right text-xs font-medium uppercase text-gray-500"><%= t("pgbus.queues.queues_list.headers.total_ever") %></th>
           <th class="px-4 py-3 text-right text-xs font-medium uppercase text-gray-500"><%= t("pgbus.queues.queues_list.headers.actions") %></th>
@@ -24,7 +25,8 @@
             </td>
             <td data-label="Depth" class="px-4 py-3 text-sm text-right font-mono text-gray-700"><%= pgbus_number(q[:queue_length]) %></td>
             <td data-label="Visible" class="px-4 py-3 text-sm text-right font-mono text-gray-700"><%= pgbus_number(q[:queue_visible_length]) %></td>
-            <td data-label="Oldest" class="px-4 py-3 text-sm text-right text-gray-500"><%= q[:oldest_msg_age_sec] || "—" %></td>
+            <td data-label="Parked" class="px-4 py-3 text-sm text-right font-mono text-gray-500"><%= pgbus_number(q[:parked_length]) %></td>
+            <td data-label="Oldest claimable" class="px-4 py-3 text-sm text-right text-gray-500"><%= q[:oldest_claimable_age_sec] || "—" %></td>
             <td data-label="Newest" class="px-4 py-3 text-sm text-right text-gray-500"><%= q[:newest_msg_age_sec] || "—" %></td>
             <td data-label="Total" class="px-4 py-3 text-sm text-right text-gray-500"><%= pgbus_number(q[:total_messages]) %></td>
             <td data-label="Actions" class="px-4 py-3 text-sm text-right space-x-2">
@@ -51,7 +53,7 @@
           </tr>
         <% end %>
         <% if @queues.empty? %>
-          <tr><td colspan="7" class="px-4 py-8 text-center text-sm text-gray-400"><%= t("pgbus.queues.queues_list.empty") %></td></tr>
+          <tr><td colspan="8" class="px-4 py-8 text-center text-sm text-gray-400"><%= t("pgbus.queues.queues_list.empty") %></td></tr>
         <% end %>
       </tbody>
     </table>

data/app/views/pgbus/queues/show.html.erb

@@ -13,6 +13,9 @@
       <p class="text-sm text-gray-500 mt-1">
         <%= t("pgbus.queues.show.depth") %> <span class="font-mono"><%= @queue[:queue_length] %></span> |
         <%= t("pgbus.queues.show.visible") %> <span class="font-mono"><%= @queue[:queue_visible_length] %></span> |
+        <%= t("pgbus.queues.show.parked") %> <span class="font-mono"><%= @queue[:parked_length] %></span> |
+        <%= t("pgbus.queues.show.oldest") %> <span class="font-mono"><%= @queue[:oldest_msg_age_sec] || "—" %></span> |
+        <%= t("pgbus.queues.show.oldest_claimable") %> <span class="font-mono"><%= @queue[:oldest_claimable_age_sec] || "—" %></span> |
         <%= t("pgbus.queues.show.total") %> <span class="font-mono"><%= pgbus_number(@queue[:total_messages]) %></span>
       </p>
     <% end %>

data/config/locales/da.yml

@@ -65,7 +65,8 @@ da:
         empty: Ingen køer fundet
         headers:
           depth: Dybde
-          oldest: Ældste (s)
+          oldest_claimable: Ældste tilgængelig (s)
+          parked: Parkeret
           queue: Kø
           total: Total
           visible: Synlig
@@ -475,7 +476,8 @@ da:
           actions: Handlinger
           depth: Dybde
           newest: Nyeste (s)
-          oldest: Ældste (s)
+          oldest_claimable: Ældste tilgængelig (s)
+          parked: Parkeret
           queue: Kø
           total_ever: Total nogensinde
           visible: Synlig
@@ -516,6 +518,9 @@ da:
           scheduled: 'Planlagt:'
           timezone: 'Tidszone:'
           visible_at: 'Synlig fra:'
+        oldest: 'Ældste:'
+        oldest_claimable: 'Ældste tilgængelig:'
+        parked: 'Parkeret:'
         pause: Pause
         pause_confirm: Pause behandling?
         purge_confirm: Rens alle beskeder?

data/config/locales/de.yml

@@ -65,7 +65,8 @@ de:
         empty: Keine Warteschlangen gefunden
         headers:
           depth: Tiefe
-          oldest: Älteste (s)
+          oldest_claimable: Älteste verfügbar (s)
+          parked: Geparkt
           queue: Warteschlange
           total: Gesamt
           visible: Sichtbar
@@ -475,7 +476,8 @@ de:
           actions: Aktionen
           depth: Tiefe
           newest: Neueste (s)
-          oldest: Älteste (s)
+          oldest_claimable: Älteste verfügbar (s)
+          parked: Geparkt
           queue: Warteschlange
           total_ever: Insgesamt jemals
           visible: Sichtbar
@@ -516,6 +518,9 @@ de:
           scheduled: 'Geplant:'
           timezone: 'Zeitzone:'
           visible_at: 'Sichtbar ab:'
+        oldest: 'Älteste:'
+        oldest_claimable: 'Älteste verfügbar:'
+        parked: 'Geparkt:'
         pause: Pause
         pause_confirm: Verarbeitung pausieren?
         purge_confirm: Alle Nachrichten löschen?

data/config/locales/en.yml

@@ -65,7 +65,8 @@ en:
         empty: No queues found
         headers:
           depth: Depth
-          oldest: Oldest (s)
+          oldest_claimable: Oldest claimable (s)
+          parked: Parked
           queue: Queue
           total: Total
           visible: Visible
@@ -475,7 +476,8 @@ en:
           actions: Actions
           depth: Depth
           newest: Newest (s)
-          oldest: Oldest (s)
+          oldest_claimable: Oldest claimable (s)
+          parked: Parked
           queue: Queue
           total_ever: Total Ever
           visible: Visible
@@ -516,6 +518,9 @@ en:
           scheduled: 'Scheduled at:'
           timezone: 'Timezone:'
           visible_at: 'Visible at:'
+        oldest: 'Oldest:'
+        oldest_claimable: 'Oldest claimable:'
+        parked: 'Parked:'
         pause: Pause
         pause_confirm: Pause processing?
         purge_confirm: Purge all messages?

data/config/locales/es.yml

@@ -65,7 +65,8 @@ es:
         empty: No se encontraron colas
         headers:
           depth: Profundidad
-          oldest: El más antiguo (s)
+          oldest_claimable: Más antiguo disponible (s)
+          parked: Aparcados
           queue: Cola
           total: Total
           visible: Visible
@@ -475,7 +476,8 @@ es:
           actions: Acciones
           depth: Profundidad
           newest: Más nuevo (s)
-          oldest: Más antiguo (s)
+          oldest_claimable: Más antiguo disponible (s)
+          parked: Aparcados
           queue: Cola
           total_ever: Total acumulado
           visible: Visible
@@ -516,6 +518,9 @@ es:
           scheduled: 'Programado:'
           timezone: 'Zona horaria:'
           visible_at: 'Visible en:'
+        oldest: 'Más antiguo:'
+        oldest_claimable: 'Más antiguo disponible:'
+        parked: 'Aparcados:'
         pause: Pausar
         pause_confirm: "¿Pausar el procesamiento?"
         purge_confirm: "¿Purgar todos los mensajes?"

data/config/locales/fi.yml

@@ -65,7 +65,8 @@ fi:
         empty: Jonot eivät löytyneet
         headers:
           depth: Syvyys
-          oldest: Vanhin (s)
+          oldest_claimable: Vanhin saatavilla (s)
+          parked: Pysäköidyt
           queue: Jono
           total: Yhteensä
           visible: Näkyvissä
@@ -475,7 +476,8 @@ fi:
           actions: Toiminnot
           depth: Syvyys
           newest: Uusimmat (s)
-          oldest: Vanhimmat (s)
+          oldest_claimable: Vanhin saatavilla (s)
+          parked: Pysäköidyt
           queue: Jono
           total_ever: Yhteensä koskaan
           visible: Näkyvissä
@@ -516,6 +518,9 @@ fi:
           scheduled: 'Aikataulutettu:'
           timezone: 'Aikavyöhyke:'
           visible_at: 'Näkyvissä:'
+        oldest: 'Vanhin:'
+        oldest_claimable: 'Vanhin saatavilla:'
+        parked: 'Pysäköidyt:'
         pause: Tauko
         pause_confirm: Keskeytetäänkö käsittely?
         purge_confirm: Tyhjennetäänkö kaikki viestit?

data/config/locales/fr.yml

@@ -65,7 +65,8 @@ fr:
         empty: Aucune file d'attente trouvée
         headers:
           depth: Profondeur
-          oldest: Le plus ancien (s)
+          oldest_claimable: Plus ancien disponible (s)
+          parked: Différés
           queue: File d'attente
           total: Total
           visible: Visible
@@ -475,7 +476,8 @@ fr:
           actions: Actions
           depth: Profondeur
           newest: Le plus récent (s)
-          oldest: Le plus ancien (s)
+          oldest_claimable: Plus ancien disponible (s)
+          parked: Différés
           queue: File d'attente
           total_ever: Total jamais
           visible: Visible
@@ -516,6 +518,9 @@ fr:
           scheduled: 'Planifié :'
           timezone: 'Fuseau horaire :'
           visible_at: 'Visible à :'
+        oldest: 'Plus ancien :'
+        oldest_claimable: 'Plus ancien disponible :'
+        parked: 'Différés :'
         pause: Pause
         pause_confirm: Mettre en pause le traitement ?
         purge_confirm: Purger tous les messages ?

data/config/locales/it.yml

@@ -65,7 +65,8 @@ it:
         empty: Nessuna coda trovata
         headers:
           depth: Profondità
-          oldest: Più vecchio (s)
+          oldest_claimable: Più vecchio disponibile (s)
+          parked: Posticipati
           queue: Coda
           total: Totale
           visible: Visibile
@@ -475,7 +476,8 @@ it:
           actions: Azioni
           depth: Profondità
           newest: Più recente (s)
-          oldest: Più vecchio (s)
+          oldest_claimable: Più vecchio disponibile (s)
+          parked: Posticipati
           queue: Coda
           total_ever: Totale mai
           visible: Visibile
@@ -516,6 +518,9 @@ it:
           scheduled: 'Programmato:'
           timezone: 'Fuso orario:'
           visible_at: 'Visibile alle:'
+        oldest: 'Più vecchio:'
+        oldest_claimable: 'Più vecchio disponibile:'
+        parked: 'Posticipati:'
         pause: Pausa
         pause_confirm: Mettere in pausa l'elaborazione?
         purge_confirm: Eliminare tutti i messaggi?

data/config/locales/ja.yml

@@ -65,7 +65,8 @@ ja:
         empty: キューが見つかりません
         headers:
           depth: 深さ
-          oldest: 最古 (秒)
+          oldest_claimable: 取得可能な最古 (秒)
+          parked: 待機中
           queue: キュー
           total: 合計
           visible: 表示中
@@ -475,7 +476,8 @@ ja:
           actions: アクション
           depth: 深さ
           newest: 最新 (秒)
-          oldest: 最古 (秒)
+          oldest_claimable: 取得可能な最古 (秒)
+          parked: 待機中
           queue: キュー
           total_ever: 合計数
           visible: 表示中
@@ -516,6 +518,9 @@ ja:
           scheduled: スケジュール済み：
           timezone: タイムゾーン：
           visible_at: 表示可能日時：
+        oldest: '最古:'
+        oldest_claimable: '取得可能な最古:'
+        parked: '待機中:'
         pause: 一時停止
         pause_confirm: 処理を一時停止しますか？
         purge_confirm: すべてのメッセージを削除しますか？

data/config/locales/nb.yml

@@ -65,7 +65,8 @@ nb:
         empty: Ingen køer funnet
         headers:
           depth: Dybde
-          oldest: Eldste (s)
+          oldest_claimable: Eldste tilgjengelig (s)
+          parked: Parkert
           queue: Kø
           total: Totalt
           visible: Synlig
@@ -475,7 +476,8 @@ nb:
           actions: Handlinger
           depth: Dybde
           newest: Nyeste (s)
-          oldest: Eldste (s)
+          oldest_claimable: Eldste tilgjengelig (s)
+          parked: Parkert
           queue: Kø
           total_ever: Totalt noensinne
           visible: Synlig
@@ -516,6 +518,9 @@ nb:
           scheduled: 'Planlagt:'
           timezone: 'Tidssone:'
           visible_at: 'Synlig fra:'
+        oldest: 'Eldste:'
+        oldest_claimable: 'Eldste tilgjengelig:'
+        parked: 'Parkert:'
         pause: Pause
         pause_confirm: Pause behandling?
         purge_confirm: Rens alle meldinger?

data/config/locales/nl.yml

@@ -65,7 +65,8 @@ nl:
         empty: Geen wachtrijen gevonden
         headers:
           depth: Diepte
-          oldest: Oudste (s)
+          oldest_claimable: Oudste beschikbaar (s)
+          parked: Geparkeerd
           queue: Wachtrij
           total: Totaal
           visible: Zichtbaar
@@ -475,7 +476,8 @@ nl:
           actions: Acties
           depth: Diepte
           newest: Nieuwste (s)
-          oldest: Oudste (s)
+          oldest_claimable: Oudste beschikbaar (s)
+          parked: Geparkeerd
           queue: Wachtrij
           total_ever: Totaal ooit
           visible: Zichtbaar
@@ -516,6 +518,9 @@ nl:
           scheduled: 'Gepland:'
           timezone: 'Tijdzone:'
           visible_at: 'Zichtbaar op:'
+        oldest: 'Oudste:'
+        oldest_claimable: 'Oudste beschikbaar:'
+        parked: 'Geparkeerd:'
         pause: Pauzeren
         pause_confirm: Verwerking pauzeren?
         purge_confirm: Alle berichten verwijderen?

data/config/locales/pt.yml

@@ -65,7 +65,8 @@ pt:
         empty: Nenhuma fila encontrada
         headers:
           depth: Profundidade
-          oldest: Mais antigo (s)
+          oldest_claimable: Mais antigo disponível (s)
+          parked: Adiados
           queue: Fila
           total: Total
           visible: Visível
@@ -475,7 +476,8 @@ pt:
           actions: Ações
           depth: Profundidade
           newest: Mais novo (s)
-          oldest: Mais antigo (s)
+          oldest_claimable: Mais antigo disponível (s)
+          parked: Adiados
           queue: Fila
           total_ever: Total de todos os tempos
           visible: Visível
@@ -516,6 +518,9 @@ pt:
           scheduled: 'Agendado:'
           timezone: 'Fuso horário:'
           visible_at: 'Visível em:'
+        oldest: 'Mais antigo:'
+        oldest_claimable: 'Mais antigo disponível:'
+        parked: 'Adiados:'
         pause: Pausar
         pause_confirm: Pausar processamento?
         purge_confirm: Limpar todas as mensagens?