pgbus 0.9.7 → 0.9.8

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: a8282d1403685f1543b1df1394cc6199e9f9182f697db151c4d6cd22f20018b9
-  data.tar.gz: 83ff9056787cd40b38a8724a55cb2e23b670f446a37f9612c38a561ea1120c37
+  metadata.gz: 53f1c781915d9915275b46471ce97583b88051ea0d4e4232abee09cd9398d589
+  data.tar.gz: df3849a4e41849bbc679582047f442efecbb280672aa2ec9cad431665d6d50aa
 SHA512:
-  metadata.gz: 4095c65e96196ba5cb87d4e1df86c4066d5534610ff210c919f4b1f4d26d97005025add8d4ac97a8f153a2a340f0fa20cf66fdca79f8695bd30b537a899eb404
-  data.tar.gz: cba33628e71e9670d991abaf75c34ac53152e3a721734a13dca8bed262bcc2bda931f0721767395829c3f4e415245564d11f7e927608699f9bba76cb29485089
+  metadata.gz: 13b391c7779c00963eb8fa029ace02bcfc9cd80f9090385ad4a6c8c8ed6b1968137772cfd70f2cf4a8f321b1d5997418c46cdf547e7184b55caf51639a5e96b0
+  data.tar.gz: 23e94ef632af35f22d92bbbbbd0ce75bde0b6a46168ef5aa598076f03f6db97b6c21552b46af1883ee1358de259580bc0ff72ab7944d52200d55334ce73d823d

data/CHANGELOG.md

@@ -2,6 +2,32 @@
 
 ### Added
 
+- **Docs: a documentation site at [pgbus.zoolutions.llc](https://pgbus.zoolutions.llc).** A nested, stateless [docs-kit](https://github.com/mhenrixon/docs-kit) Rails app under `docs/` (the same monorepo pattern as docs-kit's own dogfood site and phlex-reactive) turns the single 1,786-line README into a browsable site: 25 pages across six groups (Getting started, Guide, Operations, Testing, Migrate, Reference), five hand-built theme-aware SVG flow diagrams (architecture, message lifecycle, event fanout, outbox, streams), full-text search, per-page Markdown twins, `/llms.txt`/`/llms-full.txt`, and a live read-only MCP endpoint (`POST /mcp`). Recurring tasks — previously implemented but undocumented — now have a dedicated page written from the source. The Configuration reference is generated from a data constant guarded by a drift spec that checks it against the real `Pgbus::Configuration` accessors in both directions, so a renamed or added option fails the build. The gem package is unchanged: `spec.files` globs only `app/ config/ exe/ lib/` + root files, so nothing under `docs/` ships in the gem. The three `docs/switch_from_*.md` migration guides now point to their site pages. Deployed via Kamal + the shared docs-kit reusable workflow; the docs site has its own CI job (`.github/workflows/docs-ci.yml`) scoped to `docs/**`. Refs #266.
+
+- **Observability: per-worker throughput rates persisted in heartbeat metadata.** Each worker already tracked live throughput in a `RateCounter` (incremented on `:dequeued`/`:processed`/`:failed` and snapshotted every heartbeat via the worker's `on_beat` hook), but those rates never left the process: `Worker#stats` was in-process only and `Heartbeat#beat` persisted just `last_heartbeat_at` plus `loop_tick_at`. The dashboard's Processes page could show only static boot-time metadata (queues/threads/pid) — operators had no cluster-wide view of per-worker throughput. `Pgbus::Process::Heartbeat#initialize` now accepts a `metadata_supplier:` (a callable returning a Hash, default `nil`); `#beat` calls it **after** `@on_beat` (so the rate counter's `snapshot!` has already refreshed the rates) and merges the result into the persisted `metadata` alongside `loop_tick_at`, all inside the existing `rescue StandardError` so a raising supplier logs a warning and the timer survives. `Worker#start_heartbeat` wires a supplier returning `{ "rates" => {processed, failed, dequeued}, "jobs_processed", "jobs_failed", "in_flight" }`. Heartbeats with no supplier (supervisor, dispatcher, consumer, scheduler, outbox poller, streamer) are untouched — `beat_metadata` returns `nil` for them and leaves the column exactly as before. No SQL changes: `Web::DataSource#format_process` already passes `metadata` through, keeping the no-raw-SQL-outside-`Web::DataSource` rule intact. The Processes table renders the rates human-readable via a new `pgbus_worker_rates` helper (e.g. `12.4/s processed · 0.2/s failed`, zero rates omitted), with i18n keys added under `pgbus.processes.processes_table.rates` across all 12 locales; the raw `rates`/counter keys are hidden from the generic metadata dump. Refs #221.
+
+- **Observability: HTTP liveness and readiness endpoints for orchestrators.** Pgbus had no HTTP health surface: the `OK`/`DEGRADED`/`STALLED` verdict in `Pgbus::MCP::HealthAnalyzer` was reachable only through the MCP JSON-RPC tool or the auth-gated dashboard, and the supervisor process (which forks and watches workers) had no HTTP surface at all — so a Kubernetes kubelet could not distinguish "supervisor alive" from "workers silently wedged". New `Pgbus::Web::HealthApp` (`lib/pgbus/web/health_app.rb`) is a plain Rack app exposing `GET /livez` (→ `200 text/plain "ok"`, unconditionally, **no** database access — the serving process is up) and `GET /readyz` (builds a `Web::DataSource`, runs `HealthAnalyzer#verdict`: `OK`/`DEGRADED` → `200`, `STALLED` → `503`, body = the verdict JSON; any `StandardError` such as an unreachable DB → `503 {"status":"ERROR"}`, logged via `Pgbus.logger` and never swallowed). Unknown paths return `404`; non-`GET` returns `405`. `DEGRADED` deliberately stays ready — only the silent-wedge `STALLED` signal fails readiness. Mount it in a web pod: `mount Pgbus::Web::HealthApp.new => "/pgbus/health"`. For worker pods that run the supervisor (no Puma to mount into), set `config.health_port` (Integer, default `nil` = disabled) and `config.health_bind` (default `"127.0.0.1"`) and `Supervisor#run` serves both paths standalone via `Pgbus::Web::HealthServer` (`lib/pgbus/web/health_server.rb`) — a single accept-loop thread over a `TCPServer` that parses only the HTTP request line, synthesizes a minimal Rack env, and dispatches to the same `HealthApp#call`, so routing and the verdict→status mapping live in exactly one place. The analyzer is required lazily (it has no `mcp`-gem dependency), so the standalone server works without the optional `mcp` gem installed. Refs #218.
+
+- **Observability: generic metrics adapter with Prometheus and StatsD backends.** `lib/pgbus/instrumentation.rb` documents 16 `pgbus.*` `ActiveSupport::Notifications` events, but the only shipped consumer was the AppSignal integration (`lib/pgbus/integrations/appsignal/subscriber.rb` subscribes 13 of them) — teams on Prometheus, Datadog, or plain StatsD got no metrics without hand-writing subscribers. New `Pgbus::Metrics` namespace (Zeitwerk-managed under `lib/pgbus/metrics/`, unlike the explicitly-required `integrations/`) adds a vendor-neutral second consumer: `Metrics::Backend` (interface — `increment(name, value = 1, tags = {})`, `gauge(name, value, tags = {})`, `histogram(name, value, tags = {})` — plus a `Null` no-op subclass); `Metrics::Subscriber` (mirrors the AppSignal subscriber's 13 subscriptions and identical `pgbus_`-prefixed metric names/tags — `pgbus_queue_job_count` with `status`, `pgbus_job_duration_ms`, `pgbus_messages_sent`/`pgbus_messages_read`, `pgbus_event_count`/`pgbus_event_duration_ms`, `pgbus_outbox_published`, `pgbus_recurring_enqueued`, `pgbus_worker_recycled`, `pgbus_stream_broadcast_count` — forwarding to the configured backend, every handler wrapped in a `safely` rescue+log so a raising backend never propagates into the producer thread); `Metrics::Backends::Prometheus` (dependency-free, Mutex-guarded in-process registry, histograms as sum+count summaries, text-exposition rendering with label escaping); `Metrics::PrometheusExporter` (a self-contained Rack app rendering exposition format v0.0.4, mountable like `StreamApp`, `503` when the configured backend is not Prometheus); and `Metrics::Backends::Statsd` (`UDPSocket` datagrams — `|c`/`|g`/`|ms` with DogStatsD `|#tag:value` tags, no gem dependency). Config keys in `lib/pgbus/configuration.rb`: `metrics_backend` (`nil` default | `:prometheus` | `:statsd` | a `Backend` instance — validated eagerly in `validate!`), `statsd_host` (`"127.0.0.1"`), `statsd_port` (`8125`); `ConfigLoader` auto-recognizes them. The subscriber installs from a new `pgbus.metrics` engine initializer that mirrors the AppSignal one — `next if metrics_backend.nil?` so the default installs no subscription (zero overhead). AppSignal is untouched; both integrations can run simultaneously. README documents configuration, the emitted metrics, and mounting the exporter. Refs #219.
+
+- **DX: `pgbus dlq` — dead-letter management from the CLI.** Dead-letter inspect/drain operations existed only behind the mounted web dashboard (`Web::DataSource#dlq_messages`/`#dlq_message_detail`/`#retry_dlq_message`/`#discard_dlq_message`/`#retry_all_dlq`/`#discard_all_dlq`), so headless deployments and incident runbooks could not inspect or drain a DLQ without ad-hoc SQL — which project rules forbid. New `Pgbus::CLI::DLQ` (`lib/pgbus/cli/dlq.rb`), dispatched from `CLI.start`, adds five subcommands that route through `Web::DataSource` so retry semantics stay identical to the dashboard (origin-queue re-enqueue by stripping `DEAD_LETTER_SUFFIX`, transactional produce+delete, lock release on discard) with zero raw SQL and no direct PGMQ calls: `pgbus dlq list [--page N] [--per-page N]` (table of msg_id, DLQ queue, origin queue, read_ct, enqueued_at plus a total-count footer — never payloads), `pgbus dlq show MSG_ID` (message metadata with the payload passed through `Web::PayloadFilter.filter_json` so the dashboard's sensitive-data filtering applies identically), `pgbus dlq retry MSG_ID` (resolve the DLQ queue via `dlq_message_detail`, then `retry_dlq_message`), `pgbus dlq retry-all` (prints the re-enqueued count), and `pgbus dlq purge MSG_ID` / `pgbus dlq purge --all --yes` (`--all` without `--yes` makes no changes and exits 1). An unknown msg_id and an unknown subcommand both exit 1. Refs #216.
+
+- **DX: boot diagnostics banner at supervisor start.** `Supervisor#run` logged only `"[Pgbus] Supervisor starting pid=…"` — the settings that decide whether a deployment actually works (which database it connected to, LISTEN/NOTIFY vs polling, the resolved pool size, which roles and capsules boot) were never stated, so a misconfigured deployment that sat idle forced operators to attach a console and inspect `Pgbus.configuration` by hand. `Supervisor#run` now emits a one-block banner (a run of `"[Pgbus] boot:"`-prefixed `info` lines that render cleanly under both the `:text` and `:json` log formatters) right after `start_heartbeat` and before `bootstrap_queues`: `Pgbus::VERSION`; the connection target reduced to `host/dbname` (never the password) across all three `connection_options` forms — `database_url` string, `connection_params` hash, and AR-derived hash; `config.resolved_pool_size`; `config.pgmq_schema_mode` and the installed PGMQ version (best-effort via `Client#pgmq_schema_version`, `unknown` on any error); `config.listen_notify` and `config.worker_notify_wakeup?`; the roles that will actually boot honoring `config.roles`; and one line per worker capsule (name or `anonymous`, queues, threads, execution mode via `config.execution_mode_for`) and per event consumer (topics, threads). Every DB-dependent field is wrapped so a transient failure degrades that field to `unknown` — the banner can never abort boot. Refs #213.
+
+- **DX: configuration is now validated eagerly at boot.** `Configuration#validate!` had no call site in `lib/` — `Pgbus.configure` just yielded and `ConfigLoader.apply` just assigned keys, so an invalid value like `visibility_timeout = 0` sat dormant until a worker code path consumed it, failing at runtime far from the misconfiguration (the opposite of the fail-loud-at-boot behavior `roles=` and the duration setters already had). `Pgbus.configure` now runs `configuration.validate!` after the yield, and `ConfigLoader.apply` runs `config.validate!` after its key loop — and since both `ConfigLoader.load` branches route through `apply` and the engine's `pgbus.configure` initializer loads `config/pgbus.yml` through it, a bad YAML value now aborts Rails boot with an `ArgumentError` naming the offending key. `validate!` stays DB-free (pure value checks), so eager validation adds no boot-time DB dependency; defaults are valid, so partial and sequential `configure` blocks pass. A new `attr_accessor :eager_validation` (default `true`) is the escape hatch for exotic setups that intentionally hold a transiently-invalid config: set `eager_validation = false` before or inside a `configure` block (or as a key in `pgbus.yml`) to suppress the automatic call — both call sites check the flag after applying settings, and explicit `validate!` still works. **Backward-incompatible in one direction:** an invalid-but-previously-unread config now raises at boot instead of silently later; that is the intended change, and `eager_validation = false` restores the old behavior. Refs #215.
+
+- **DX: `pgbus doctor` — a single preflight command for deploy/CI gating.** Nothing answered "is this pgbus environment healthy enough to run?" in one place: config validity was never checked at boot (`Configuration#validate!` was never called), DB reachability only surfaced when the supervisor forked children that crash-looped, PGMQ version lived in `rake pgbus:pgmq:status`, and the process-liveness verdict was reachable only through the MCP stdio server (which needs the optional `mcp` gem). New `Pgbus::Doctor#run` runs six checks — each a `{ name:, status: :ok|:warn|:fail, detail: }` — and `#success?`: (1) `Configuration#validate!`, (2) DB connectivity via a new public `Client#ping` (`SELECT 1` through `with_raw_connection`), (3) PGMQ schema presence and installed-vs-vendored version via a new `Client#pgmq_schema_version` (the same tracking-table read `pgbus:pgmq:status` uses, kept inside the Client so no raw SQL escapes it), (4) queue existence — configured queues (via `Client#configured_queues`, prefixed through `config.queue_name`) diffed against `Client#list_queues`, (5) LISTEN/NOTIFY liveness per configured queue via a new `Client#notify_enabled?`, and (6) process liveness via `Pgbus::MCP::HealthAnalyzer` (required directly — it only needs a `Web::DataSource`, not the `mcp` gem). Doctor never touches PGMQ/PostgreSQL directly and never raises: a broken environment turns every probe into a `:fail` result rather than a crash. `#report` prints one line per check plus a resolved-config summary (queue_prefix, default_queue, pgmq_schema_mode, resolved_pool_size, roles, capsules) with passwords redacted from `database_url`/`connection_params`. Wired into `pgbus doctor` (exit 1 unless `success?`, so it gates a deploy or CI run) and `rake pgbus:doctor`. Refs #212.
+
+- **Process: exponential cooldown on the worker queue evict/restore cycle.** When every queue a worker owned had been evicted — the queue table permanently deleted — `restore_evicted_queues` reinstated the initial queue set immediately, both from `evict_missing_queues` and again at the top of every `fetch_messages`. The worker then oscillated forever: fetch → undefined-table error → evict → restore → fetch → error…, emitting an error+warn pair every loop tick (default 0.1s), spamming logs and hammering the database with doomed reads. Restore is now gated by an exponential cooldown: the first restore after a full eviction waits `RESTORE_COOLDOWN_BASE` (30s), and each consecutive failed restore doubles the wait up to `RESTORE_COOLDOWN_MAX` (300s), tracked by a monotonic `@last_evicted_at` window and a `@restore_streak` counter. While the cooldown is pending, `fetch_messages` leaves `@queues` empty (the caller idles on the existing empty-`active_queues` path) and logs exactly one deferral `warn` per window ("all queues evicted; next restore attempt in Ns") instead of an error pair per tick. A fetch that completes without an undefined-queue error resets the streak to 0, so a recreated queue is reinstated promptly after the next eviction. Wildcard workers are untouched — both restore call sites remain guarded by `!@wildcard`. Refs #209.
+
+- **Process model: Consumer brought to parity with Worker — recycling and NOTIFY wake-up.** `Pgbus::Process::Consumer` had neither mechanism: its loop never checked `max_jobs_per_worker` / `max_memory_mb` / `max_worker_lifetime`, so a long-lived consumer fork grew memory unboundedly (the exact solid_queue problem recycling exists to fix), and its empty-read path always slept `polling_interval` even when a matching event had just arrived. Consumer now mirrors Worker: a per-iteration `check_recycle` trips once when any configured limit is exceeded (`@jobs_processed`, an `AtomicFixnum` incremented in an `ensure` on every handled message — success, DLQ-routed, and rescued failure alike, mirroring `Worker#process_message` so `max_jobs` recycling still trips on a poison/all-failing queue; a monotonic `@started_at_monotonic`; and the shared, TTL-cached `Pgbus::Process::MemoryUsage`), logs the reason, emits a `pgbus.consumer.recycle` instrumentation event, and sets `@shutting_down` so the loop drains the pool and exits cleanly — the clean exit earns an immediate supervisor restart, replacing the fork before its memory grows. For wake-up, the two `interruptible_sleep(polling_interval)` calls are replaced with `@wake_signal.wait(timeout: wake_timeout)`, and (when `config.worker_notify_wakeup?`) a `Pgbus::Process::NotifyListener` over the physical names of the consumer's subscribed queues fires `WakeSignal#notify!` the moment a row is inserted, collapsing idle latency from a full poll interval to near-instant. A live listener raises the empty-read wait to a `NOTIFY_FALLBACK_POLL_SECONDS` (15s) safety-net ceiling; a listener that fails to start or whose thread dies logs and falls back to polling, with `ensure_notify_listener` self-healing it on a 5s→300s exponential backoff. With no recycle config and notify wakeup off, behavior is unchanged. All PGMQ access stays behind `Pgbus::Client`. Refs #205.
+
+- **Connections: connection-pool utilization surfaced, and pool-exhaustion errors made actionable.** The PGMQ connection pool was invisible: pgmq-ruby exposes `size`/`available` counters, but pgbus never read them, so the first sign of an undersized or leaking pool was an opaque `PGMQ::Errors::ConnectionError: Connection pool timeout` after `pool_timeout` seconds — with no pool state and no gauge to alert on *before* saturation. New `Pgbus::Client#pool_stats` returns `{ size:, available:, pool_timeout: }` (pgmq-ruby's `stats` merged with the configured timeout) and works on both the dedicated-pool path and the shared-`Proc` path (`size` is 1 there); it is purely observational and rescues internally to `{}` so reading the pool can never break job processing. The worker heartbeat now emits a `pgbus.client.pool` instrumentation event once per beat (heartbeat-only, never on a per-job hot path) carrying that payload, and the AppSignal minutely probe reports `pgbus_pool_size` / `pgbus_pool_available` gauges tagged by hostname (the pool is per-process, unlike the cluster-wide queue/summary gauges). Finally, a bare `Connection pool timeout` error is re-raised as the *same* `PGMQ::Errors::ConnectionError` class (so `with_stale_connection_retry` semantics are unchanged and pool timeouts remain non-retryable) but with the live pool state and an actionable hint appended — "raise `Pgbus.configuration.pool_size` or reduce worker threads" — so the first signal of saturation is diagnosable. Refs #200.
+
+- **Connections: client-level circuit breaker for database-down conditions.** Resilience was previously per-queue only: `Pgbus::CircuitBreaker` records failures from job *execution* and persists its pause state in the database — useless when the database itself is down (its `check_paused` rescues and returns `false`, so nothing trips). Meanwhile every worker's `fetch_messages` rescued each `PGMQ::Errors::ConnectionError`, reported it to `ErrorReporter`, and re-polled every `polling_interval`, so a whole fleet hammered a dead database and flooded the error tracker for the entire outage. A new in-memory, process-local latch — `Pgbus::Client::ConnectionHealth`, owned by `Pgbus::Client` — trips after `OPEN_THRESHOLD` (5) consecutive `PGMQ::Errors::ConnectionError` across any operation. Once open, the read paths (`read_message`, `read_batch`, `read_multi`, `read_batch_prioritized`, `read_grouped*`, `read_with_poll`) fail fast with a new `Pgbus::ConnectionCircuitOpenError` *without checking out a pool connection*; a single half-open probe is admitted after a monotonic backoff window (1s base, doubling per re-open, capped at 60s), whose success closes and resets the breaker and whose failure re-opens it with a doubled window. Enqueues (`send_message`/`send_batch`) are deliberately never short-circuited — callers must see enqueue failures. `Worker#fetch_messages` rescues `ConnectionCircuitOpenError` before its generic rescue and idles the poll with no `ErrorReporter` call, so an outage produces two log lines total (a `warn` on open, an `info` on close) instead of one per worker per poll. The latch is `Mutex`-guarded (single shared state, atomic transitions); a half-open probe that fails with *any* error — including a `Pgbus::ReadTimeoutError` from a hung socket, which the read paths raise from inside the guard — re-opens the breaker rather than wedging it half-open, and a straggler read admitted while closed cannot re-close a breaker another thread just tripped. Refs #197.
+
+- **Connections: LISTEN/NOTIFY delivery self-probe at listener start.** A worker or streamer pointed at a transaction-mode pooler (PgBouncer drops `LISTEN` at transaction boundaries) or a read-only replica (`pg_notify()` raises) previously "connected successfully" but never received a wake-up, silently degrading to slow fallback polling / stalled SSE with no explanation. Each listener now runs a one-shot self-probe right after building its dedicated `PG::Connection`: `LISTEN` a unique per-process channel, `SELECT pg_notify()` to itself, and wait up to 2s for delivery. On failure it logs a single actionable `error` naming the direct-connection overrides to set (`worker_notify_database_url` / `worker_notify_host` / `worker_notify_port` for workers; `streams_database_url` / `streams_host` / `streams_port` for the streamer), best-effort `UNLISTEN`s the probe channel, and returns — the worker/streamer still start and degrade gracefully rather than crash. The probe runs on the initial connection only; reconnects skip it to stay cheap. New `Pgbus::Process::NotifyProbe`. Refs #191 (epic #190).
+
 - **Streams: optional publish-side coalescing for high-frequency broadcasts.** A chatty reactive component (live cursor, typing indicator, progress bar) can fan out many small broadcasts per second. Pass `coalesce:` (a window in milliseconds, or `true` for the 50ms default) with `target:` to `broadcast`/`broadcast_render` to batch per `(stream, target)` and publish only the *latest* frame within the window — superseded frames never hit the bus (no PGMQ insert, no NOTIFY, no fan-out). Last-write-wins, so it is opt-in and only safe for idempotent `replace`/`update` of a stable target (exactly the high-frequency case). The new `Pgbus::Streams::Coalescer` is a process-wide, thread-safe, trailing-edge-with-max-wait debounce: the first submit per window schedules the flush (bounding latency to one window) and later submits only overwrite the buffered payload; the flush re-enters the normal broadcast path, so a coalesced frame is just a deferred ordinary broadcast that still composes with `visible_to`/`exclude`/`event`/`durable`. Refs #171.
 
 - **Streams: typed SSE event names on broadcasts.** A broadcast can set the SSE `event:` field — `Pgbus.stream(name).broadcast(html, event: "presence")` (and `broadcast_render(..., event:)`) — while keeping the payload a Turbo Stream, so clients route on the typed name instead of sniffing the HTML. The default (`nil` or `"turbo-stream"`) is omitted from the JSONB payload to avoid redundancy, but is still set on the SSE frame's `event:` line (the connection adapter falls back to `turbo-stream`), so default consumers still get the standard `message`/turbo-stream path. The event flows through the JSONB payload → `StreamEnvelope.event` → the SSE frame's `event:` line (both the Puma and Falcon connection adapters). On the client, `<pgbus-stream-source>` dispatches a typed broadcast as a generic `pgbus:event` (`{ event, data, msgId }`) and a named `pgbus:<event>` (`{ data, msgId }`) for `addEventListener` ergonomics; declare typed event names via the element's `listen-events` attribute so they survive the EventSource reconnect path. Refs #170.
@@ -16,6 +42,12 @@
 
 - **Streams: `Pgbus.stream_key` is idempotent for an already-built key, plus `Pgbus.stream_key!`.** A single `String` argument is now treated as a pre-built pgbus stream key and returned unchanged (after the queue-name budget check) instead of tripping the colon-separator guard. This lets a consumer hold one `stream_key` value and pass it to both `turbo_stream_from` and the broadcaster without `stream_key("chat:lobby")` raising `ArgumentError`. The guard still fires for the genuinely ambiguous multi-fragment join (`stream_key("a:b", :c)`), and `Symbol`/record fragments with colons are still rejected (a colon there never came from `stream_key`). `Pgbus.stream_key!(key)` accepts a pre-built key explicitly (String required, budget enforced). Refs #167.
 
+- **Observability: StatBuffer flushes on drain entry and its flush thresholds are now configurable.** `Pgbus::StatBuffer` accumulated up to `DEFAULT_FLUSH_SIZE = 100` entries / `DEFAULT_FLUSH_INTERVAL = 5`s of job stats in memory, flushed only by the worker's periodic `flush_if_due` tick and the drain-loop `stop`. When the supervisor watchdog `SIGKILL`s a stalled worker — which cannot be trapped — everything buffered since the last flush was silently lost, and the drain phase's up-to-30s wait widened that window. `Worker#graceful_shutdown` and `Worker#check_recycle` now call `@stat_buffer&.flush` immediately after `transition_to(:draining)`, so entering drain on either a graceful `TERM` or a recycle threshold (`max_jobs`/`max_memory`/`max_lifetime`) persists the buffer before the watchdog can intervene. Both run on the main loop thread (signals are dispatched via `process_signals`, never in trap context), so the DB write is safe; `flush` is thread-safe and no-ops on an empty buffer and when stats are disabled (`@stat_buffer` is nil). New config keys `stats_flush_size` (default `100`, positive integer) and `stats_flush_interval` (default `5`, positive number) on `Configuration` are validated in `validate!`, auto-recognized by `ConfigLoader`, and passed into `StatBuffer.new` by the worker — tune them down for a tighter loss window on high-throughput deployments or up to reduce insert frequency. The residual loss window (≤ `stats_flush_interval` seconds / `stats_flush_size` entries lost only on an un-trappable `SIGKILL`, affecting insights accuracy only — never a job payload) is documented in the README stats section. Refs #220.
+
+### Changed
+
+- **DX: worker and event-consumer config keys are normalized once, at assignment.** Worker/consumer config hashes arrived with symbol keys (the Ruby DSL and `capsule`) or string keys (YAML through `ConfigLoader`), so every read site did a dual `w[:x] || w["x"]` lookup — and any new option read without the string fallback silently ignored a YAML-provided value, a recurring bug class. `Configuration#workers=` now maps each `Array`-form entry through a private `normalize_entry` (`transform_keys(&:to_sym)`; entries are flat hashes, no deep recursion) and raises `ArgumentError` for a non-Hash entry; `event_consumers` becomes an `attr_reader` plus a matching `event_consumers=` setter (nil passes through). The string/DSL path already yielded symbols, so it is unaffected. With both setters normalizing, the string fallbacks were deleted from every downstream site — `supervisor` (`fork_worker`, `fork_consumer`), `client` (`collect_configured_queues`), `configuration` (`execution_mode_for`, `validate!`, `capsule_name`, `validate_no_queue_overlap!`, `sum_thread_counts`), `doctor` (`capsule_summary`), and `cli` (`apply_capsule_filter`) — which now use symbol access only. YAML `workers:`/`event_consumers:` sections still boot with correct queues/threads/modes because `ConfigLoader` routes them through the same setters; the CLI `--queues`/`--capsule` overrides route through `workers=` too, so they get normalization for free. Out of scope: the config-converter generator (operates on raw YAML before `Configuration`) and the DB `metadata` JSON read in `consumer_priority` (Postgres always yields string keys there, so no symbol fallback exists to remove). Refs #214.
+
 ### Breaking Changes
 
 - **Queue names must be alphanumeric and underscores only.** Queue names containing dashes (e.g., `my-app-queue`) will now raise `ArgumentError`. Rename to underscored form (e.g., `my_app_queue`) before upgrading. This restriction prevents SQL injection via PGMQ queue identifiers, which are interpolated into table names and cannot be parameterized.
@@ -31,8 +63,14 @@
 
 ### Fixed
 
+- **The Turbo broadcast path now honors `streams_durable_patterns`.** `Pgbus::Streams::TurboBroadcastable#broadcast_stream_to` — the patch every `Turbo::Broadcastable` call (`broadcasts_to`, `broadcast_replace_to`, …) and phlex-reactive's `broadcast_*_to` funnel through — resolved durability from the thread-local override or `streams_default_broadcast_mode` only, then always passed an explicit `durable:` to `Pgbus.stream`. That short-circuited `Pgbus.stream`'s own resolver (`durable.nil? ? stream_durable?(name) : durable`), so `streams_durable_patterns` was dead configuration for the entire broadcast funnel: a matching pattern never promoted a broadcast to durable, and large component renders (> 8000 bytes) that were supposed to route through PGMQ instead went ephemeral through `ActionCable` and died with `PG::ProgramLimitExceeded: payload string too long`. `broadcast_stream_to` now resolves `override.nil? ? Pgbus.configuration.stream_durable?(name) : override`, letting the pattern resolver (which already falls back to `streams_default_broadcast_mode`) participate whenever no explicit thread-local override is present. Apps without `streams_durable_patterns` are unaffected; the thread-local override (set via `broadcastable_override`'s `durable:` kwarg) still wins over a matching pattern. Refs #267.
+- **Reads are now bounded by libpq-native timeouts instead of Ruby `Timeout`.** `Pgbus::Client#with_read_timeout` wrapped every read in `Timeout.timeout(config.read_timeout, Pgbus::ReadTimeoutError)`, which interrupts via `Thread#raise`. That can fire at an arbitrary point — including mid-libpq call — and leave a pooled `PG::Connection` in a state that re-hangs or returns wrong results on reuse; on the shared-AR path it does so on a connection ActiveRecord also queries. On the dedicated-connection path (`database_url` / `connection_params`), `Client#initialize` now bakes two libpq-native bounds into the connection options: (1) a server-side `statement_timeout` (`options=-c statement_timeout=<read_timeout>ms`) so Postgres cleanly cancels an overrunning query — surfaced by pgmq-ruby as `PGMQ::Errors::ConnectionError` ("canceling statement due to statement timeout") and re-raised as `Pgbus::ReadTimeoutError`; and (2) client-side `tcp_user_timeout` + `keepalives` (sized at `read_timeout + 5s`, and only when the linked libpq is ≥ 12 — older libpq rejects the `tcp_user_timeout` conninfo keyword and would fail the connection) so a dead/hung socket makes libpq raise `PG::ConnectionBad` synchronously — no `Thread#raise`, no buffer corruption — which the circuit breaker and stale-connection retry already handle. A caller-supplied `:options` connection param (e.g. `-c search_path=…`) is preserved — the `statement_timeout` flag is appended, not overwritten. On Linux with `read_timeout` set and libpq ≥ 12, these fully cover both a slow query and a hung socket, so Ruby `Timeout` is never wired in (detected at init via `Socket.const_defined?(:TCP_USER_TIMEOUT)` and `PG.library_version`). Ruby `Timeout` remains only as a narrow last resort on a *dedicated* connection where libpq can't bound the socket — non-Linux hosts (`tcp_user_timeout`/`keepalives` are no-ops there) or libpq < 12. The shared-AR `Proc` path gets neither a baked-in bound nor the Ruby `Timeout` (pgbus doesn't own that socket); instead `Client#initialize` logs a one-time hint to configure the same libpq timeouts in `database.yml` (`variables: { statement_timeout: <ms> }` plus `tcp_user_timeout`/`keepalives`), which ActiveRecord passes straight through. A new custom RuboCop cop `Pgbus/NoRubyTimeout` bans `Timeout.timeout` project-wide (no autocorrect) to prevent regressions. Note: on the dedicated path writes gain the same connection-wide `statement_timeout` (acceptable — an enqueue that can't complete within `read_timeout` is already failing). No per-call hot-path change (`client_bench` unchanged; the added cost is a one-time transform in `initialize`). Refs #198.
+- **Stream archive pruning no longer silently no-ops.** `Dispatcher#prune_stream_archives` read `config.archive_compaction_batch_size`, an accessor that was culled from `Configuration` into the `Pgbus::Process::Dispatcher::ARCHIVE_COMPACTION_BATCH_SIZE` constant in the 0.x config-cleanup refactor (#82). Because `prune_stream_archives` was introduced *after* that refactor and copied the pre-cull pattern, every invocation raised `NoMethodError`, which the method's outer `rescue StandardError` swallowed — so per-stream archive tables were never pruned and grew unbounded regardless of `streams_retention`/`streams_default_retention`. It now uses the constant, matching `#compact_archives`. Refs #204.
+- **Long dispatcher maintenance passes are now interruptible during shutdown.** `graceful_shutdown`/`immediate_shutdown` only set `@shutting_down`, which the run loop checked between maintenance passes but never inside them. With many queues or large bloated tables, a `TERM` arriving mid-pass blocked graceful shutdown until the entire pass finished — overrunning deploy windows until the supervisor escalated to `KILL`. The per-queue loops in `compact_archives`, `prune_stream_archives`, and `sweep_orphan_streams` now break before the next queue when a shutdown begins, and `TableMaintenance.run_maintenance` accepts a `stop_check:` callable (the dispatcher passes `-> { @shutting_down }`) checked before each table. An interrupted task simply re-runs next interval; retry-on-failure semantics are unchanged. Refs #204.
 - **EventBus handler invocations now run inside `Rails.application.executor` (or `.reloader` in dev).** `Pgbus::EventBus::Handler#process` previously ran outside any Rails executor wrapper, so `claim_idempotency?`'s `ProcessedEvent.insert` (and any `handle` body that touches AR) leased a connection that was never returned to the pool. With a small AR pool (Rails dev default with `RAILS_MAX_THREADS=3` is 6 connections), a handful of consumed events leaked the entire pool. The next request that triggers `clear_reloadable_connections!` (any reloadable change in dev) hung in `with_exclusively_acquired_all_connections`, surfacing as a confusing `Rack::Timeout` with a trace ending in `MonitorMixin#wait_for_cond`. Mirrors the executor-wrap pattern already used by `Pgbus::ActiveJob::Executor#execute_job`. No-op when Rails isn't loaded.
 - **Defensive retry on stale pooled pgmq connections in the enqueue path.** `Pgbus::Client#send_message`, `#send_batch`, and `#publish_to_topic` now retry once when `@pgmq.produce*` raises `PGMQ::Errors::ConnectionError` with a message indicating the pooled `PG::Connection` was killed beneath pgmq-ruby — typically by PgBouncer hitting `server_idle_timeout` / `client_idle_timeout`, an admin disconnect, or a TCP RST. Observed in production as `PQsocket() can't get socket descriptor` on the first produce following an idle window. pgmq-ruby's `auto_reconnect` recovers on the *next* pool checkout, so a single retry is sufficient; non-stale errors (pool timeout, misconfiguration, unreachable database) still propagate unchanged. Upstream pgmq-ruby fix for the underlying misclassification is in-flight at mensfeld/pgmq-ruby#94.
+- **Stale-connection retry now backs off and attempts twice.** The single immediate retry above recovers one dead pooled socket, but during a PgBouncer restart or a brief failover window the immediate second attempt lands in the same unavailable window and the error propagated — failing an enqueue the caller may never retry. `Pgbus::Client#with_stale_connection_retry` now retries a matched stale error up to `STALE_RETRY_ATTEMPTS` (2) times with a short backoff between attempts (`STALE_RETRY_DELAYS = [0.1, 0.5]`), giving a transient window time to clear. Safety is unchanged: the pattern list still matches only pre-flight/idle-socket errors (no SQL was sent), so retrying non-idempotent ops stays safe. The backoff runs error-path only — never on the success path — and the sleep is issued outside `synchronized`, so `@pgmq_mutex` is never held during a delay on the shared-connection path. No happy-path throughput or allocation change (`client_bench` identical before/after). Refs #196.
+- **Stalled dispatcher and scheduler are no longer reported healthy on the processes page.** `Web::DataSource#derive_process_status` evaluated the loop beacon only for workers, returning `:healthy` for every other kind, and neither the dispatcher (`{ pid }`) nor the scheduler (`{ pid, tasks }`) heartbeat carried a beacon. Because heartbeats run on an independent `Concurrent::TimerTask` thread, a dispatcher wedged inside `run_maintenance` (e.g. a stuck vacuum in `run_table_maintenance`) or a scheduler stuck in `tick` kept heartbeating and showed healthy forever while doing no work. Both processes now stamp a wall-clock loop beacon (`@loop_tick_at`, a `Concurrent::AtomicReference`) at the top of each main-loop iteration and pass `loop_tick_supplier:` to their heartbeat, mirroring `Worker#stamp_loop_tick`; the heartbeat persists it as `metadata.loop_tick_at` on every beat. `derive_process_status` now evaluates the beacon for `"dispatcher"` and `"scheduler"` too, with kind-aware thresholds that account for each loop's sleep — workers keep `config.stall_threshold` (90s), dispatcher uses `stall_threshold + config.dispatch_interval`, scheduler uses `stall_threshold + config.recurring_schedule_interval` — so a beacon older than the threshold renders the `:stalled` badge. A missing beacon (an older process during a rolling deploy) stays `:healthy`, and `stall_threshold = nil` disables the check as before. Worker stall detection is unchanged. Refs #222.
 
 ## [0.5.1] - 2026-04-08
 

data/README.md

@@ -4,6 +4,8 @@ PostgreSQL-native job processing and event bus for Rails, built on [PGMQ](https:
 
 **Why Pgbus?** If you already run PostgreSQL, you don't need Redis for background jobs. Pgbus gives you ActiveJob integration, AMQP-style topic routing, dead letter queues, worker memory management, and a live dashboard -- all backed by your existing database.
 
+📖 **Documentation:** [pgbus.zoolutions.llc](https://pgbus.zoolutions.llc) — guides, flow diagrams, and a full configuration reference. (This README stays the canonical GitHub reference.)
+
 [![Ruby](https://github.com/mhenrixon/pgbus/actions/workflows/main.yml/badge.svg)](https://github.com/mhenrixon/pgbus/actions/workflows/main.yml)
 
 ## Table of contents
@@ -36,6 +38,7 @@ PostgreSQL-native job processing and event bus for Rails, built on [PGMQ](https:
   - [Error reporting](#error-reporting)
   - [Structured logging](#structured-logging)
   - [Queue health monitoring](#queue-health-monitoring)
+  - [Health endpoints (liveness / readiness)](#health-endpoints-liveness--readiness)
 - [Real-time broadcasts](#real-time-broadcasts-turbo-streams-replacement)
 - [Testing](#testing)
   - [RSpec setup](#rspec-setup)
@@ -757,9 +760,58 @@ Three importable AppSignal dashboards ship with the gem:
 
 Import via the AppSignal dashboard UI ("New dashboard" → "Import JSON") or the AppSignal API.
 
+### Metrics adapter (Prometheus / StatsD)
+
+AppSignal is one consumer of pgbus's `ActiveSupport::Notifications` events. For teams on Prometheus, Datadog, or plain StatsD, the built-in **metrics adapter** consumes the same events and forwards them to a backend — no hand-written subscribers. It is **off by default** (`metrics_backend = nil` installs nothing, zero overhead) and runs independently of AppSignal, so both can be active at once.
+
+```ruby
+Pgbus.configure do |c|
+  c.metrics_backend = :prometheus   # in-process registry, scraped via the exporter
+  # or
+  c.metrics_backend = :statsd       # UDP datagrams (DogStatsD dialect)
+  c.statsd_host = "127.0.0.1"       # default
+  c.statsd_port = 8125              # default
+end
+```
+
+You can also assign a custom backend instance — any object subclassing `Pgbus::Metrics::Backend` (implementing `increment`, `gauge`, `histogram`):
+
+```ruby
+c.metrics_backend = MyOpenTelemetryBackend.new
+```
+
+**Metrics emitted** (all `pgbus_`-prefixed, low-cardinality tags only):
+
+| Metric | Type | Tags |
+|--------|------|------|
+| `pgbus_queue_job_count` | counter | `queue`, `job_class`, `status` (`processed`/`failed`/`dead_lettered`) |
+| `pgbus_job_duration_ms` | histogram | `queue`, `job_class` |
+| `pgbus_event_count` | counter | `handler`, `routing_key`, `status` |
+| `pgbus_event_duration_ms` | histogram | `handler`, `routing_key` |
+| `pgbus_messages_sent` / `pgbus_messages_read` | counter | `queue` |
+| `pgbus_stream_broadcast_count` | counter | `stream`, `deferred` |
+| `pgbus_outbox_published` | counter | `kind` |
+| `pgbus_recurring_enqueued` | counter | `task`, `class_name` |
+| `pgbus_worker_recycled` | counter | `reason` |
+
+A backend that raises (registry bug, StatsD socket down) is logged and swallowed — a metrics failure never propagates into the thread that emitted the event.
+
+#### Mounting the Prometheus exporter
+
+The `:prometheus` backend is an in-process registry; expose it for scraping by mounting `Pgbus::Metrics::PrometheusExporter` — a self-contained Rack app that renders text exposition format (v0.0.4):
+
+```ruby
+# config/routes.rb
+mount Pgbus::Metrics::PrometheusExporter.new => "/metrics"
+```
+
+With no argument the exporter reads `config.metrics_backend`, so a single `config.metrics_backend = :prometheus` wires both the subscriber and the exporter to the same registry. The app is plain Rack, so it also runs under any standalone Rack server (e.g. a one-line `config.ru`) — point Prometheus at `GET /metrics`.
+
+> The exporter returns **503** if `metrics_backend` is not a Prometheus backend (e.g. it's `:statsd` or `nil`), since there is no in-process registry to render.
+
 #### Custom subscriptions
 
-The integration is built on `ActiveSupport::Notifications`. If you want to push pgbus telemetry into a different APM (Datadog, New Relic, OpenTelemetry), subscribe directly:
+The metrics adapter above covers Prometheus and StatsD. For anything else (New Relic, OpenTelemetry, a bespoke sink), the events are built on `ActiveSupport::Notifications` — subscribe directly:
 
 ```ruby
 ActiveSupport::Notifications.subscribe(/^pgbus\./) do |name, start, finish, _id, payload|
@@ -953,6 +1005,56 @@ For the **HTTP** transport, point the client at the mounted URL with a streamabl
 }
 ```
 
+### Health endpoints (liveness / readiness)
+
+For orchestrators like Kubernetes, Pgbus exposes two HTTP probes: `/livez` (is the serving process up?) and `/readyz` (are queues draining, or is a worker silently wedged?). `/readyz` runs the same `OK` / `DEGRADED` / `STALLED` verdict as the MCP `pgbus_health` tool — `STALLED` (visible backlog while workers heart-beat but don't claim) fails readiness.
+
+| Path | Method | 200 | 503 | Touches DB |
+|---|---|---|---|---|
+| `/livez` | GET | always (`ok`) | never | no |
+| `/readyz` | GET | verdict `OK` or `DEGRADED` | verdict `STALLED`, or DB unreachable (`{"status":"ERROR"}`) | yes |
+
+Unknown paths return `404`; non-`GET` methods return `405`. The `/readyz` body is the verdict JSON, so a probe failure is self-describing in the pod's event log.
+
+#### Mount in your Rails app
+
+`Pgbus::Web::HealthApp` is a plain Rack app — mount it wherever your web pods already serve HTTP. It needs no auth (it exposes only aggregate health, never payloads) but keep it on an internal network:
+
+```ruby
+# config/routes.rb
+mount Pgbus::Web::HealthApp.new => "/pgbus/health"
+```
+
+```yaml
+# kubelet probes (Deployment spec)
+livenessProbe:
+  httpGet: { path: /pgbus/health/livez, port: 3000 }
+  periodSeconds: 10
+readinessProbe:
+  httpGet: { path: /pgbus/health/readyz, port: 3000 }
+  periodSeconds: 10
+  failureThreshold: 3
+```
+
+#### Standalone from the supervisor
+
+Worker pods run `bin/pgbus` (the supervisor), not Puma — so there is no Rails server to mount into. Set `health_port` and the supervisor serves both paths itself over a tiny TCP server (no Rails, no dashboard), letting a kubelet probe the process that actually forks and watches workers:
+
+```ruby
+Pgbus.configure do |c|
+  c.health_port = 9394        # nil (default) = disabled
+  c.health_bind = "0.0.0.0"   # default "127.0.0.1"
+end
+```
+
+```yaml
+# probe the supervisor pod directly
+livenessProbe:
+  httpGet: { path: /livez, port: 9394 }
+readinessProbe:
+  httpGet: { path: /readyz, port: 9394 }
+```
+
 ## Real-time broadcasts (turbo-streams replacement)
 
 Pgbus ships a drop-in replacement for turbo-rails' `turbo_stream_from` helper that fixes several well-known ActionCable correctness bugs by using PGMQ message IDs as a replay cursor. Same API as turbo-rails. No Redis. No ActionCable. No lost messages on reconnect.
@@ -1528,6 +1630,17 @@ rails generate pgbus:add_job_stats --database=pgbus
 
 Stats collection is enabled by default (`config.stats_enabled = true`). Old stats are cleaned up by the dispatcher based on `config.stats_retention` (default: 30 days). If the migration hasn't been run yet, stat recording is silently skipped.
 
+Stats are buffered in memory and bulk-inserted rather than written one row per job. Each worker flushes its buffer when it fills to `config.stats_flush_size` entries (default: `100`), when `config.stats_flush_interval` seconds have elapsed (default: `5`), and immediately when the worker begins draining (graceful `TERM` shutdown or a recycle threshold). Tune the thresholds down for high-throughput deployments that want a tighter loss window, or up to reduce insert frequency:
+
+```ruby
+Pgbus.configure do |c|
+  c.stats_flush_size = 500     # flush after 500 buffered stats
+  c.stats_flush_interval = 2   # ...or every 2 seconds, whichever comes first
+end
+```
+
+**Loss window:** stats are advisory (dashboard/insights only), never job payloads. Buffered entries are flushed on graceful shutdown and drain entry, so a clean stop loses nothing. If the supervisor watchdog `SIGKILL`s a stalled worker — which cannot be trapped — up to `stats_flush_interval` seconds / `stats_flush_size` entries accumulated since the last flush are lost. This affects insights accuracy only; no job is dropped.
+
 ### Database tables
 
 Pgbus uses these tables (created via PGMQ and migrations):
@@ -1632,6 +1745,8 @@ PostgreSQL + PGMQ
 | `web_live_updates` | `true` | Enable Turbo Frames auto-refresh on dashboard |
 | `stats_enabled` | `true` | Record job execution stats for insights dashboard |
 | `stats_retention` | `30.days` | How long to keep job stats. Accepts seconds, Duration, or `nil` to disable cleanup |
+| `stats_flush_size` | `100` | Buffered stat entries per worker before a bulk insert flush. Positive integer. Lower = tighter SIGKILL loss window. |
+| `stats_flush_interval` | `5` | Seconds between periodic stat buffer flushes. Positive number. |
 | `streams_test_mode` | `false` | Return a stub SSE response without hijack or background threads. Auto-enabled by `Pgbus::Testing.fake!`/`.inline!`. See [SSE streams in tests](#sse-streams-in-tests). |
 | `streams_stats_enabled` | `false` | Record stream broadcast/connect/disconnect stats (opt-in, can be high volume) |
 | `streams_path` | `nil` | Custom URL path for the SSE endpoint (nil = auto-detected from engine mount) |
@@ -1639,6 +1754,9 @@ PostgreSQL + PGMQ
 | `error_reporters` | `[]` | Array of callables invoked on caught exceptions. Each receives `(exception, context_hash)`. |
 | `log_format` | `:text` | Log formatter (`:text` or `:json`). Sets `logger.formatter` automatically. |
 | `metrics_enabled` | `true` | Enable Prometheus-compatible metrics on the dashboard |
+| `metrics_backend` | `nil` | Generic metrics adapter: `nil` (off), `:prometheus`, `:statsd`, or a `Pgbus::Metrics::Backend` instance |
+| `statsd_host` | `"127.0.0.1"` | StatsD UDP host (used when `metrics_backend = :statsd`) |
+| `statsd_port` | `8125` | StatsD UDP port (used when `metrics_backend = :statsd`) |
 
 ## Development
 

data/Rakefile

@@ -8,7 +8,16 @@ end
 
 require "rubocop/rake_task"
 
-RuboCop::RakeTask.new
+# Lint only the gem's own source — NOT the nested docs/ site. docs/ is a separate
+# consuming app with its own bundle and .rubocop.yml (it inherit_gems
+# rubocop-rails-omakase and requires docs_kit/rubocop, both absent from the gem's
+# bundle). RuboCop loads a directory's .rubocop.yml while scanning it — before
+# AllCops/Exclude applies — so a bare run discovers docs/.rubocop.yml and crashes
+# on the unresolvable gem inheritance. Passing explicit paths stops the
+# discovery. docs/ lints itself in the Docs site workflow. (Same fix as docs-kit.)
+RuboCop::RakeTask.new do |task|
+  task.patterns = %w[app benchmarks config lib spec Gemfile Rakefile pgbus.gemspec]
+end
 
 namespace :bench do
   bench_dir = "benchmarks"

data/app/helpers/pgbus/application_helper.rb

@@ -29,6 +29,43 @@ module Pgbus
       end
     end
 
+    # Order the rate keys are rendered in, and their i18n label suffixes.
+    WORKER_RATE_KEYS = %w[processed failed dequeued].freeze
+
+    # Metadata keys rendered specially elsewhere (throughput badge, health
+    # status) and therefore excluded from the generic key/value dump.
+    WORKER_INTERNAL_METADATA_KEYS = %w[rates jobs_processed jobs_failed in_flight loop_tick_at].freeze
+
+    # The subset of a process's metadata to render as generic key/value badges:
+    # everything except the throughput/health keys that have dedicated rendering.
+    def pgbus_display_metadata(metadata)
+      return {} unless metadata.is_a?(Hash)
+
+      metadata.reject { |k, _| WORKER_INTERNAL_METADATA_KEYS.include?(k.to_s) }
+    end
+
+    # Render a worker's per-second throughput rates (from heartbeat metadata)
+    # as a compact human-readable string, e.g. "12.4/s processed · 0.2/s failed".
+    # Zero rates are omitted; returns nil when there are no non-zero rates so
+    # callers can fall back to the raw metadata rendering.
+    def pgbus_worker_rates(metadata)
+      return nil unless metadata.is_a?(Hash)
+
+      rates = metadata["rates"]
+      return nil unless rates.is_a?(Hash)
+
+      parts = WORKER_RATE_KEYS.filter_map do |key|
+        value = rates[key].to_f
+        next if value.zero?
+
+        label = I18n.t("pgbus.processes.processes_table.rates.#{key}", default: key)
+        "#{value}/s #{label}"
+      end
+      return nil if parts.empty?
+
+      parts.join(" · ")
+    end
+
     def pgbus_status_badge(healthy_or_status)
       status = case healthy_or_status
                when true then :healthy

data/app/views/pgbus/processes/_processes_table.html.erb

@@ -21,7 +21,10 @@
             <td data-label="Last Heartbeat" class="px-4 py-3 text-sm text-gray-500"><%= pgbus_time_ago(p[:last_heartbeat_at]) %></td>
             <td data-label="Metadata" class="px-4 py-3 text-sm text-gray-500 font-mono text-xs max-w-xs truncate">
               <% if p[:metadata].is_a?(Hash) %>
-                <% p[:metadata].each do |k, v| %>
+                <% if (rates = pgbus_worker_rates(p[:metadata])) %>
+                  <span class="inline-flex items-center rounded bg-green-100 dark:bg-green-900/30 text-green-800 dark:text-green-300 px-1.5 py-0.5 text-xs mr-1 font-medium"><%= rates %></span>
+                <% end %>
+                <% pgbus_display_metadata(p[:metadata]).each do |k, v| %>
                   <span class="inline-flex items-center rounded bg-gray-100 px-1.5 py-0.5 text-xs mr-1"><%= k %>: <%= v %></span>
                 <% end %>
               <% end %>

data/config/locales/da.yml

@@ -458,6 +458,10 @@ da:
           metadata: Metadata
           pid: PID
           status: Status
+        rates:
+          dequeued: hentet
+          failed: mislykkedes
+          processed: behandlet
     queues:
       destroy:
         success: Køen %{name} slettet.

data/config/locales/de.yml

@@ -458,6 +458,10 @@ de:
           metadata: Metadaten
           pid: PID
           status: Status
+        rates:
+          dequeued: entnommen
+          failed: fehlgeschlagen
+          processed: verarbeitet
     queues:
       destroy:
         success: Warteschlange %{name} gelöscht.

data/config/locales/en.yml

@@ -458,6 +458,10 @@ en:
           metadata: Metadata
           pid: PID
           status: Status
+        rates:
+          dequeued: dequeued
+          failed: failed
+          processed: processed
     queues:
       destroy:
         success: Queue %{name} deleted.

data/config/locales/es.yml

@@ -458,6 +458,10 @@ es:
           metadata: Metadatos
           pid: PID
           status: Estado
+        rates:
+          dequeued: extraídos
+          failed: fallidos
+          processed: procesados
     queues:
       destroy:
         success: Cola %{name} eliminada.

data/config/locales/fi.yml

@@ -458,6 +458,10 @@ fi:
           metadata: Metatiedot
           pid: PID
           status: Tila
+        rates:
+          dequeued: haettu
+          failed: epäonnistui
+          processed: käsitelty
     queues:
       destroy:
         success: Jono %{name} poistettu.

data/config/locales/fr.yml

@@ -458,6 +458,10 @@ fr:
           metadata: Métadonnées
           pid: PID
           status: Statut
+        rates:
+          dequeued: retirés
+          failed: échoués
+          processed: traités
     queues:
       destroy:
         success: File d'attente %{name} supprimée.

data/config/locales/it.yml

@@ -458,6 +458,10 @@ it:
           metadata: Metadati
           pid: PID
           status: Stato
+        rates:
+          dequeued: prelevati
+          failed: falliti
+          processed: elaborati
     queues:
       destroy:
         success: Coda %{name} eliminata.

data/config/locales/ja.yml

@@ -458,6 +458,10 @@ ja:
           metadata: メタデータ
           pid: PID
           status: ステータス
+        rates:
+          dequeued: 取得
+          failed: 失敗
+          processed: 処理
     queues:
       destroy:
         success: キュー %{name} を削除しました。

data/config/locales/nb.yml

@@ -458,6 +458,10 @@ nb:
           metadata: Metadata
           pid: PID
           status: Status
+        rates:
+          dequeued: hentet
+          failed: mislyktes
+          processed: behandlet
     queues:
       destroy:
         success: Køen %{name} slettet.

data/config/locales/nl.yml

@@ -458,6 +458,10 @@ nl:
           metadata: Metadata
           pid: PID
           status: Status
+        rates:
+          dequeued: opgehaald
+          failed: mislukt
+          processed: verwerkt
     queues:
       destroy:
         success: Wachtrij %{name} verwijderd.

data/config/locales/pt.yml

@@ -458,6 +458,10 @@ pt:
           metadata: Metadados
           pid: PID
           status: Status
+        rates:
+          dequeued: retirados
+          failed: falharam
+          processed: processados
     queues:
       destroy:
         success: Fila %{name} excluída.

data/config/locales/sv.yml

@@ -458,6 +458,10 @@ sv:
           metadata: Metadata
           pid: PID
           status: Status
+        rates:
+          dequeued: hämtade
+          failed: misslyckade
+          processed: bearbetade
     queues:
       destroy:
         success: Kön %{name} borttagen.

data/lib/pgbus/active_job/executor.rb

@@ -296,14 +296,35 @@ module Pgbus
         Pgbus.logger.warn { "[Pgbus] Batch discard signal failed: #{e.message}" }
       end
 
+      # Archiving is idempotent — archiving an already-archived message is a
+      # no-op — so a connection error here is safe to retry once, unlike sends
+      # where a retry could duplicate the message. Without the retry, a job
+      # that succeeded but failed to archive redelivers after VT expiry and
+      # runs twice. If the retry also fails, fall through to the normal
+      # failure path (recorded failure + VT-based redelivery).
       def archive_from(queue_name, msg_id, source_queue: nil)
-        if source_queue
-          client.archive_message(source_queue, msg_id, prefixed: false)
-        else
-          client.archive_message(queue_name, msg_id)
+        attempts = 0
+        begin
+          if source_queue
+            client.archive_message(source_queue, msg_id, prefixed: false)
+          else
+            client.archive_message(queue_name, msg_id)
+          end
+        rescue StandardError => e
+          attempts += 1
+          raise unless attempts == 1 && connection_error?(e)
+
+          Pgbus.logger.warn do
+            "[Pgbus::Executor] Archive failed on connection error, retrying once: #{e.message}"
+          end
+          retry
         end
       end
 
+      def connection_error?(error)
+        defined?(PGMQ::Errors::ConnectionError) && error.is_a?(PGMQ::Errors::ConnectionError)
+      end
+
       def handle_dead_letter(message, queue_name, payload, source_queue: nil)
         Pgbus.logger.warn do
           job_class = payload["job_class"] || "unknown"