pgbus 0.9.7 → 0.9.9

data/README.md

@@ -4,6 +4,8 @@ PostgreSQL-native job processing and event bus for Rails, built on [PGMQ](https:
 
 **Why Pgbus?** If you already run PostgreSQL, you don't need Redis for background jobs. Pgbus gives you ActiveJob integration, AMQP-style topic routing, dead letter queues, worker memory management, and a live dashboard -- all backed by your existing database.
 
+📖 **Documentation:** [pgbus.zoolutions.llc](https://pgbus.zoolutions.llc) — guides, flow diagrams, and a full configuration reference. (This README stays the canonical GitHub reference.)
+
 [![Ruby](https://github.com/mhenrixon/pgbus/actions/workflows/main.yml/badge.svg)](https://github.com/mhenrixon/pgbus/actions/workflows/main.yml)
 
 ## Table of contents
@@ -21,6 +23,8 @@ PostgreSQL-native job processing and event bus for Rails, built on [PGMQ](https:
   - [Job uniqueness](#job-uniqueness)
   - [Concurrency controls](#concurrency-controls)
   - [Circuit breaker and queue pause/resume](#circuit-breaker-and-queue-pauseresume)
+  - [Client-level circuit breaker (database-down)](#client-level-circuit-breaker-database-down)
+  - [Read timeouts (libpq-native)](#read-timeouts-libpq-native)
   - [Prefetch flow control](#prefetch-flow-control)
   - [Worker recycling](#worker-recycling)
   - [Retry backoff](#retry-backoff)
@@ -34,8 +38,11 @@ PostgreSQL-native job processing and event bus for Rails, built on [PGMQ](https:
   - [Archive compaction](#archive-compaction)
 - [Observability](#observability)
   - [Error reporting](#error-reporting)
+  - [Connection pool metrics](#connection-pool-metrics)
   - [Structured logging](#structured-logging)
   - [Queue health monitoring](#queue-health-monitoring)
+  - [Health endpoints (liveness / readiness)](#health-endpoints-liveness--readiness)
+  - [Boot diagnostics banner](#boot-diagnostics-banner)
 - [Real-time broadcasts](#real-time-broadcasts-turbo-streams-replacement)
 - [Testing](#testing)
   - [RSpec setup](#rspec-setup)
@@ -129,12 +136,16 @@ end
 
 The capsule string DSL is the shortest form for the common case. Use `c.capsule` when you need named capsules with advanced options like `single_active_consumer` or `consumer_priority`. See [Routing and ordering](#routing-and-ordering) for the full set.
 
+Configuration is validated eagerly: `Pgbus.configure` runs `Configuration#validate!` right after your block yields, so an invalid value (`visibility_timeout = 0`, for example) raises `Pgbus::ConfigurationError` at boot instead of failing later inside a worker. Set `c.eager_validation = false` for the rare setup that intentionally holds a transiently-invalid config across sequential `configure` calls.
+
 > **Upgrading from an older pgbus?** Run `rails generate pgbus:update`. It does two things in one pass:
 >
 > - Converts any legacy `config/pgbus.yml` to a Ruby initializer at `config/initializers/pgbus.rb` (skipped if the initializer already exists).
 > - Inspects your live database and adds any missing pgbus migrations to `db/migrate` (or `db/pgbus_migrate` if you use `connects_to`). The generator detects your separate-database config automatically from `Pgbus.configuration.connects_to` or by scanning the initializer / `config/application.rb`, so you don't have to re-specify `--database=pgbus` every time.
 >
 > Useful flags: `--dry-run` (print the plan without creating files), `--skip-config`, `--skip-migrations`, `--quiet`. Running it on a database with no pgbus tables at all will redirect you to `pgbus:install` instead of stacking individual add_* migrations.
+>
+> For the full step-by-step procedure (including the vendored PGMQ schema check and post-upgrade verification with `pgbus doctor`) plus per-version breaking changes, see [Upgrading pgbus](https://pgbus.zoolutions.llc/docs/upgrading-pgbus).
 
 ### 2. Use as ActiveJob backend
 
@@ -167,19 +178,23 @@ Publish events with AMQP-style topic routing:
 
 ```ruby
 # Publish an event
-Pgbus::EventBus::Publisher.publish(
+Pgbus.publish(
   "orders.created",
   { order_id: order.id, total: order.total }
 )
 
 # Publish with delay
-Pgbus::EventBus::Publisher.publish_later(
+Pgbus.publish_later(
   "invoices.due",
   { invoice_id: invoice.id },
   delay: 30.days
 )
 ```
 
+`Pgbus.publish` / `Pgbus.publish_later` are top-level shortcuts for
+`Pgbus::EventBus::Publisher.publish` / `.publish_later` (symmetric with
+`Pgbus.stream`). The long form still works.
+
 Subscribe with handlers:
 
 ```ruby
@@ -294,36 +309,33 @@ end
 
 #### Lock lifecycle
 
-The lock is **never released by a timer**. It is held as long as the job exists in the system:
+The lock is **never released by a timer**. It is held as long as the job's message exists in the queue:
 
 ```text
-Enqueue ──→ pgbus_job_locks (state: queued, owner_pid: nil)
-                  │
-  Worker picks up job
+Enqueue ──→ pgbus_uniqueness_keys (lock_key, queue_name, msg_id)
                   │
-                  ▼
-           claim_for_execution! (state: executing, owner_pid: PID)
+  Worker picks up and runs the job
                   │
           ┌───────┴───────┐
           ▼               ▼
       Success           Crash
-      release!        (lock orphaned)
-      (row deleted)       │
+      release!        (lock orphaned —
+      (row deleted)    message still tracked
+                        by lock_key/queue_name/msg_id)
+                          │
                           ▼
                     Reaper checks:
-                    Is owner_pid in pgbus_processes
-                    with fresh heartbeat?
+                    Does the referenced message
+                    still exist in the PGMQ queue?
                           │
                     ┌─────┴─────┐
-                    No          Yes
+                   Gone       Present
                     ▼            ▼
                 release!      (keep lock,
-                (orphaned)     job is running)
+                (orphaned)     job still in flight)
 ```
 
-**Crash recovery** works through the reaper (runs every 5 minutes in the dispatcher). It cross-references `owner_pid` in `pgbus_job_locks` against `pgbus_processes` heartbeats. If the owning worker has no fresh heartbeat, the lock is orphaned and released — the PGMQ message's visibility timeout will expire and the job will be retried by another worker.
-
-A last-resort TTL (default 24 hours) handles the case where the entire pgbus supervisor is dead and the reaper itself can't run.
+**Crash recovery** works through the reaper (runs periodically in the dispatcher's `cleanup_job_locks`). It checks whether the message referenced by each lock (`queue_name` + `msg_id`) still exists in the PGMQ queue via `Client#message_exists?`. If the message is gone (delivered, expired, or the queue was truncated) and the lock is older than `2 * visibility_timeout` — to avoid racing a lock whose `send_message` hasn't committed yet — the lock is released. A lock backed by a message still in the queue is never touched, even if it looks old, so a recurring job that fails and retries can hold its lock for hours.
 
 #### Uniqueness vs concurrency controls
 
@@ -345,8 +357,8 @@ A last-resort TTL (default 24 hours) handles the case where the entire pgbus sup
 #### Setup
 
 ```bash
-rails generate pgbus:add_job_locks                  # Add the migration
-rails generate pgbus:add_job_locks --database=pgbus # For separate database
+rails generate pgbus:add_uniqueness_keys                   # Add the migration
+rails generate pgbus:add_uniqueness_keys --database=pgbus   # For separate database
 ```
 
 ### Concurrency controls
@@ -453,6 +465,49 @@ rails generate pgbus:add_queue_states           # Add the queue_states migration
 rails generate pgbus:add_queue_states --database=pgbus  # For separate database
 ```
 
+### Client-level circuit breaker (database-down)
+
+The circuit breaker above (`Pgbus::CircuitBreaker`) is **per-queue** and persists its pause state in the database — useless when the database itself is down, since its `check_paused` rescues and returns `false`, tripping nothing. `Pgbus::Client::ConnectionHealth` is a **separate, in-memory, process-local** latch owned by `Pgbus::Client` for exactly that failure mode. Do not confuse the two:
+
+| | `Pgbus::CircuitBreaker` | `Client::ConnectionHealth` |
+|---|---|---|
+| Scope | Per queue | Per client (whole process) |
+| Trips on | Job execution failures | Consecutive `PGMQ::Errors::ConnectionError` |
+| State lives in | `pgbus_queue_states` (DB) | In-process memory (Mutex-guarded) |
+| Survives restart | Yes | No — resets on process start |
+| Purpose | Isolate a queue whose job code keeps failing | Stop hammering a database that is down |
+| Configuration | `circuit_breaker_enabled` + constants | None — constants only |
+
+`ConnectionHealth` trips open after 5 consecutive connection errors across *any* operation. Once open, the read paths (`read_message`, `read_batch`, `read_multi`, `read_batch_prioritized`, `read_grouped*`, `read_with_poll`) fail fast with a new `Pgbus::ConnectionCircuitOpenError` **without checking out a pool connection**. A single half-open probe is admitted after a monotonic backoff (1s base, doubling per re-open, capped at 60s); its success closes and resets the breaker, its failure re-opens it with a doubled window. Enqueues (`send_message`/`send_batch`) are **never** short-circuited — callers must see enqueue failures. `Worker#fetch_messages` rescues `ConnectionCircuitOpenError` before its generic rescue and idles the poll with no `ErrorReporter` call, so a whole-fleet outage produces two log lines total (a `warn` on open, an `info` on close) instead of one per worker per poll.
+
+There is no configuration for this breaker — like `Pgbus::CircuitBreaker`'s thresholds, the values rarely need tuning.
+
+### Read timeouts (libpq-native)
+
+`config.read_timeout` (default `30` seconds) caps how long a single PGMQ read can block. Reads used to be bounded with Ruby `Timeout.timeout`, which interrupts via `Thread#raise` — that can fire mid-libpq-call and leave a pooled connection corrupted for the next checkout. On a **dedicated connection** (`database_url` or `connection_params`), pgbus now bakes two libpq-native bounds into the connection at boot instead:
+
+| Bound | How | Effect |
+|---|---|---|
+| Server-side | `statement_timeout` via `options=-c statement_timeout=<read_timeout>ms` | Postgres cleanly cancels an overrunning query, surfaced as `Pgbus::ReadTimeoutError` |
+| Client-side | `tcp_user_timeout` + `keepalives` (sized `read_timeout + 5s`) | A dead/hung socket makes libpq raise `PG::ConnectionBad` synchronously — no `Thread#raise`, no buffer corruption |
+
+The client-side bound only applies on Linux with libpq ≥ 12 (older libpq rejects the `tcp_user_timeout` conninfo keyword; non-Linux hosts no-op it) — detected automatically via `Socket.const_defined?(:TCP_USER_TIMEOUT)` and `PG.library_version`, no configuration needed. Ruby `Timeout` remains only as a narrow last-resort fallback on a dedicated connection where libpq can't bound the socket (non-Linux hosts, or libpq < 12).
+
+**The shared-AR `Proc` connection path** (`-> { ActiveRecord::Base.connection.raw_connection }`) gets neither bound automatically — pgbus doesn't own that socket. Configure the same libpq timeouts yourself in `database.yml`; ActiveRecord passes them straight through:
+
+```yaml
+# config/database.yml
+production:
+  primary:
+    <<: *default
+    variables:
+      statement_timeout: 30000   # ms — match config.read_timeout
+    # tcp_user_timeout / keepalives: set via the connection URL or driver/OS
+    # defaults; ActiveRecord passes libpq conninfo options straight through.
+```
+
+A custom RuboCop cop, `Pgbus/NoRubyTimeout`, bans `Timeout.timeout` project-wide to prevent this class of bug from regressing.
+
 ### Prefetch flow control
 
 Cap the number of in-flight (claimed but unfinished) messages per worker:
@@ -728,6 +783,25 @@ Reporters are wired into all critical rescue paths: job execution failures, work
 
 `ErrorReporter.report` is guaranteed to never raise — if a reporter or the logger itself throws, the error is swallowed silently. This preserves fault-tolerance invariants at every rescue site.
 
+### Error hierarchy
+
+Every operational error Pgbus raises descends from `Pgbus::Error`, so a single rescue catches them all:
+
+```ruby
+begin
+  Pgbus.configure { |c| c.visibility_timeout = 0 }
+rescue Pgbus::Error => e
+  # ConfigurationError, EnqueueError, ExecutionPoolError, SerializationError,
+  # QueueNotFoundError, SchemaNotReady, ... all land here
+end
+```
+
+The named subclasses (a partial list): `Pgbus::ConfigurationError` (`validate!` and config setters), `Pgbus::SerializationError` (payload / GlobalID rejection), `Pgbus::EnqueueError` (batch enqueue integrity), `Pgbus::ExecutionPoolError` (pool shutting down / at capacity), `Pgbus::QueueNotFoundError`, `Pgbus::DeadLetterError`, `Pgbus::JobNotUnique`, `Pgbus::SchemaNotReady`, `Pgbus::ReadTimeoutError`, and `Pgbus::ConnectionCircuitOpenError`.
+
+**One deliberate exception to the rule:** errors that reject a malformed *argument shape* stay `ArgumentError` subclasses, because that's what `ArgumentError` means — `Pgbus::Streams::StreamNameTooLong`, `Pgbus::Streams::Cursor::InvalidCursor`, and `Pgbus::Configuration::CapsuleDSL::ParseError`. Rescue those with `ArgumentError` (or the specific class), not `Pgbus::Error`.
+
+If you upgraded from 0.9.x and previously did `rescue ArgumentError` around `Pgbus.configure`, switch it to `rescue Pgbus::Error` — config validation now raises `Pgbus::ConfigurationError`, which is not an `ArgumentError`.
+
 ### AppSignal integration
 
 When the `appsignal` gem is loaded in your app, Pgbus auto-installs a subscriber and a minutely probe that report into AppSignal:
@@ -757,9 +831,59 @@ Three importable AppSignal dashboards ship with the gem:
 
 Import via the AppSignal dashboard UI ("New dashboard" → "Import JSON") or the AppSignal API.
 
+### Metrics adapter (Prometheus / StatsD)
+
+AppSignal is one consumer of pgbus's `ActiveSupport::Notifications` events. For teams on Prometheus, Datadog, or plain StatsD, the built-in **metrics adapter** consumes the same events and forwards them to a backend — no hand-written subscribers. It is **off by default** (`metrics_backend = nil` installs nothing, zero overhead) and runs independently of AppSignal, so both can be active at once.
+
+```ruby
+Pgbus.configure do |c|
+  c.metrics_backend = :prometheus   # in-process registry, scraped via the exporter
+  # or
+  c.metrics_backend = :statsd       # UDP datagrams (DogStatsD dialect)
+  c.statsd_host = "127.0.0.1"       # default
+  c.statsd_port = 8125              # default
+end
+```
+
+You can also assign a custom backend instance — any object subclassing `Pgbus::Metrics::Backend` (implementing `increment`, `gauge`, `histogram`):
+
+```ruby
+c.metrics_backend = MyOpenTelemetryBackend.new
+```
+
+**Metrics emitted** (all `pgbus_`-prefixed, low-cardinality tags only):
+
+| Metric | Type | Tags |
+|--------|------|------|
+| `pgbus_queue_job_count` | counter | `queue`, `job_class`, `status` (`processed`/`failed`/`dead_lettered`) |
+| `pgbus_job_duration_ms` | histogram | `queue`, `job_class` |
+| `pgbus_event_count` | counter | `handler`, `routing_key`, `status` |
+| `pgbus_event_duration_ms` | histogram | `handler`, `routing_key` |
+| `pgbus_messages_sent` / `pgbus_messages_read` | counter | `queue` |
+| `pgbus_stream_broadcast_count` | counter | `stream`, `deferred` |
+| `pgbus_outbox_published` | counter | `kind` |
+| `pgbus_recurring_enqueued` | counter | `task`, `class_name` |
+| `pgbus_worker_recycled` | counter | `reason`, `kind` (`worker`/`consumer`) |
+| `pgbus_pool_size` / `pgbus_pool_available` | gauge | `hostname` |
+
+A backend that raises (registry bug, StatsD socket down) is logged and swallowed — a metrics failure never propagates into the thread that emitted the event.
+
+#### Mounting the Prometheus exporter
+
+The `:prometheus` backend is an in-process registry; expose it for scraping by mounting `Pgbus::Metrics::PrometheusExporter` — a self-contained Rack app that renders text exposition format (v0.0.4):
+
+```ruby
+# config/routes.rb
+mount Pgbus::Metrics::PrometheusExporter.new => "/metrics"
+```
+
+With no argument the exporter reads `config.metrics_backend`, so a single `config.metrics_backend = :prometheus` wires both the subscriber and the exporter to the same registry. The app is plain Rack, so it also runs under any standalone Rack server (e.g. a one-line `config.ru`) — point Prometheus at `GET /metrics`.
+
+> The exporter returns **503** if `metrics_backend` is not a Prometheus backend (e.g. it's `:statsd` or `nil`), since there is no in-process registry to render.
+
 #### Custom subscriptions
 
-The integration is built on `ActiveSupport::Notifications`. If you want to push pgbus telemetry into a different APM (Datadog, New Relic, OpenTelemetry), subscribe directly:
+The metrics adapter above covers Prometheus and StatsD. For anything else (New Relic, OpenTelemetry, a bespoke sink), the events are built on `ActiveSupport::Notifications` — subscribe directly:
 
 ```ruby
 ActiveSupport::Notifications.subscribe(/^pgbus\./) do |name, start, finish, _id, payload|
@@ -768,7 +892,24 @@ ActiveSupport::Notifications.subscribe(/^pgbus\./) do |name, start, finish, _id,
 end
 ```
 
-Events emitted: `pgbus.executor.execute`, `pgbus.job_completed`, `pgbus.job_failed`, `pgbus.job_dead_lettered`, `pgbus.event_processed`, `pgbus.event_failed`, `pgbus.client.send_message`, `pgbus.client.send_batch`, `pgbus.client.read_batch`, `pgbus.stream.broadcast`, `pgbus.outbox.publish`, `pgbus.recurring.enqueue`, `pgbus.worker.recycle`. Payload keys are documented in `lib/pgbus/instrumentation.rb`.
+Metrics-relevant events emitted (the subset the metrics subscriber maps; see `lib/pgbus/instrumentation.rb` for the full `pgbus.*` catalog): `pgbus.executor.execute`, `pgbus.job_completed`, `pgbus.job_failed`, `pgbus.job_dead_lettered`, `pgbus.event_processed`, `pgbus.event_failed`, `pgbus.client.send_message`, `pgbus.client.send_batch`, `pgbus.client.read_batch`, `pgbus.client.pool`, `pgbus.stream.broadcast`, `pgbus.outbox.publish`, `pgbus.recurring.enqueue`, `pgbus.worker.recycle`, `pgbus.consumer.recycle`. Payload keys are documented in `lib/pgbus/instrumentation.rb`.
+
+### Connection pool metrics
+
+The PGMQ connection pool was previously invisible — pgmq-ruby exposes `size`/`available` counters, but nothing read them, so the first sign of an undersized or leaking pool was an opaque `PGMQ::Errors::ConnectionError: Connection pool timeout`. `Pgbus::Client#pool_stats` surfaces the live counters:
+
+```ruby
+Pgbus.client.pool_stats
+# => { size: 10, available: 3, pool_timeout: 5 }
+```
+
+It's purely observational (rescues internally to `{}`), so reading the pool can never break job processing, and it works on both the dedicated-pool path and the shared-`Proc` connection path (`size` reports `1` there). The worker heartbeat emits a `pgbus.client.pool` instrumentation event once per beat (never on a per-job hot path) carrying that payload, and the AppSignal minutely probe reports `pgbus_pool_size` / `pgbus_pool_available` gauges tagged by hostname (the pool is per-process, unlike the cluster-wide queue/summary gauges).
+
+A pool-timeout error is re-raised as the same `PGMQ::Errors::ConnectionError` class (so `with_stale_connection_retry` semantics and non-retryability are unchanged) with the live pool state and an actionable hint appended:
+
+```
+Connection pool timeout (pool {size: 10, available: 0, pool_timeout: 5}) — raise Pgbus.configuration.pool_size or reduce worker threads
+```
 
 ### Structured logging
 
@@ -953,6 +1094,74 @@ For the **HTTP** transport, point the client at the mounted URL with a streamabl
 }
 ```
 
+### Health endpoints (liveness / readiness)
+
+For orchestrators like Kubernetes, Pgbus exposes two HTTP probes: `/livez` (is the serving process up?) and `/readyz` (are queues draining, or is a worker silently wedged?). `/readyz` runs the same `OK` / `DEGRADED` / `STALLED` verdict as the MCP `pgbus_health` tool — `STALLED` (visible backlog while workers heart-beat but don't claim) fails readiness.
+
+| Path | Method | 200 | 503 | Touches DB |
+|---|---|---|---|---|
+| `/livez` | GET | always (`ok`) | never | no |
+| `/readyz` | GET | verdict `OK` or `DEGRADED` | verdict `STALLED`, or DB unreachable (`{"status":"ERROR"}`) | yes |
+
+Unknown paths return `404`; non-`GET` methods return `405`. The `/readyz` body is the verdict JSON, so a probe failure is self-describing in the pod's event log.
+
+#### Mount in your Rails app
+
+`Pgbus::Web::HealthApp` is a plain Rack app — mount it wherever your web pods already serve HTTP. It needs no auth (it exposes only aggregate health, never payloads) but keep it on an internal network:
+
+```ruby
+# config/routes.rb
+mount Pgbus::Web::HealthApp.new => "/pgbus/health"
+```
+
+```yaml
+# kubelet probes (Deployment spec)
+livenessProbe:
+  httpGet: { path: /pgbus/health/livez, port: 3000 }
+  periodSeconds: 10
+readinessProbe:
+  httpGet: { path: /pgbus/health/readyz, port: 3000 }
+  periodSeconds: 10
+  failureThreshold: 3
+```
+
+#### Standalone from the supervisor
+
+Worker pods run `bin/pgbus` (the supervisor), not Puma — so there is no Rails server to mount into. Set `health_port` and the supervisor serves both paths itself over a tiny TCP server (no Rails, no dashboard), letting a kubelet probe the process that actually forks and watches workers:
+
+```ruby
+Pgbus.configure do |c|
+  c.health_port = 9394        # nil (default) = disabled
+  c.health_bind = "0.0.0.0"   # default "127.0.0.1"
+end
+```
+
+```yaml
+# probe the supervisor pod directly
+livenessProbe:
+  httpGet: { path: /livez, port: 9394 }
+readinessProbe:
+  httpGet: { path: /readyz, port: 9394 }
+```
+
+### Boot diagnostics banner
+
+`Supervisor#run` logs a one-block banner right after the heartbeat starts and before queues bootstrap, so a misconfigured deployment states its actual settings instead of forcing an operator to attach a console. Every line is `"[Pgbus] boot:"`-prefixed and renders cleanly under both the `:text` and `:json` log formatters:
+
+```
+[Pgbus] boot: pgbus 0.9.8 pid=42317
+[Pgbus] boot: connection=host/dbname pool=12
+[Pgbus] boot: pgmq_schema_mode=auto pgmq_version=1.4.0
+[Pgbus] boot: listen_notify=true worker_notify_wakeup=true
+[Pgbus] boot: roles=workers,dispatcher,scheduler
+[Pgbus] boot: capsule=critical queues=critical threads=5 mode=threads
+[Pgbus] boot: capsule=default queues=default,mailers threads=10 mode=threads
+```
+
+It states: the pgbus version; the connection target reduced to `host/dbname` (never the password) across all three `connection_options` forms (`database_url` string, `connection_params` hash, AR-derived hash); the resolved pool size; `pgmq_schema_mode` and the installed PGMQ version (best-effort — `unknown` on any error); `listen_notify` and `worker_notify_wakeup?`; the roles that will actually boot (honoring `config.roles`); and one line per worker capsule (name, queues, threads, execution mode) and per event consumer (topics, threads).
+
+Every DB-dependent field is wrapped so a transient failure degrades that field to `unknown` — the banner can never abort boot.
+
 ## Real-time broadcasts (turbo-streams replacement)
 
 Pgbus ships a drop-in replacement for turbo-rails' `turbo_stream_from` helper that fixes several well-known ActionCable correctness bugs by using PGMQ message IDs as a replay cursor. Same API as turbo-rails. No Redis. No ActionCable. No lost messages on reconnect.
@@ -1070,6 +1279,20 @@ Pgbus.stream_key(chat, :messages)  # => "ai_chat_a3f8c1e9d2b47610_messages"
 
 The budget is computed from `config.queue_prefix` at call time so prefix overrides adjust automatically. If a stream name exceeds the budget, `Pgbus::Streams::StreamNameTooLong` is raised immediately with the offending name, computed budget, and a pointer to `Pgbus.stream_key` — before PGMQ is ever touched.
 
+#### `stream_key` idempotency
+
+A single `String` argument to `Pgbus.stream_key` is treated as an already-built pgbus stream key and returned unchanged (after the queue-name budget check), instead of tripping the colon-separator guard. This lets a consumer hold one `stream_key` value and pass it to both `pgbus_stream_from` and the broadcaster without a second call raising `ArgumentError`:
+
+```ruby
+key = Pgbus.stream_key(chat, :messages)  # => "ai_chat_a3f8c1e9d2b47610:messages"
+
+# Both calls accept the same pre-built key without raising:
+pgbus_stream_from(key)
+Pgbus.stream(key).broadcast(html)
+```
+
+`Pgbus.stream_key!(key)` accepts a pre-built key explicitly (`String` required, budget still enforced) for call sites that want to be explicit that no re-keying should happen. The guard still fires for the genuinely ambiguous multi-fragment join (`stream_key("a:b", :c)` — colliding with `stream_key("a", "b:c")`), and a `Symbol`/record fragment containing a colon is still rejected (a colon there never came from `stream_key`).
+
 ### Transactional broadcasts
 
 **This is the feature no other Rails real-time stack can offer.** A broadcast issued inside an open ActiveRecord transaction is deferred until the transaction commits. If it rolls back, the broadcast silently drops — clients never see the change that the database never persisted.
@@ -1108,6 +1331,29 @@ The replay cap is applied server-side: the helper computes `since_id = max(0, cu
 
 How much history is actually available depends on the stream's retention setting (`streams_retention` or `streams_default_retention`, both in seconds). A chat stream configured with `streams_retention = { /^chat_/ => 7.days }` will replay up to seven days of history with `replay: :all`; a notification stream with the 5-minute default will only go back five minutes.
 
+### `msg_id` reconciliation for optimistic UI
+
+Every delivered frame carries its monotonic PGMQ `msg_id` as the SSE `id:` line — the same watermark that powers reconnect replay. `<pgbus-stream-source>` surfaces it to the client two ways: the standard `message` `MessageEvent` sets `lastEventId` to the msg_id (Turbo ignores it; a reactive runtime listening for `message` reads the revision with no pgbus-specific API), and a `pgbus:message` `CustomEvent` carries `{ msgId, data }` (`msgId` is a `Number` when numeric; a negative value marks an ephemeral frame that bypassed PGMQ rather than a durable, archived one).
+
+**The reconciliation recipe:** track the highest applied `msgId` per render target; when a frame arrives, skip the morph if you've already applied a newer revision for that target. This stops a late echo — a broadcast that was in flight when a newer one landed — from clobbering a newer optimistic edit:
+
+```js
+const appliedRevision = new Map() // target -> highest applied msgId
+
+document.addEventListener("pgbus:message", (event) => {
+  const { msgId, data } = event.detail
+  const target = extractTargetFrom(data) // however your markup encodes it
+
+  const highest = appliedRevision.get(target) ?? -Infinity
+  if (msgId != null && msgId < highest) return // stale — skip the morph
+
+  appliedRevision.set(target, msgId)
+  applyMorph(target, data)
+})
+```
+
+This complements `exclude:` (below): `exclude:` handles the actor (never receives its own echo at all); msg_id reconciliation handles out-of-order delivery for everyone else.
+
 ### Server-side audience filtering
 
 Some broadcasts shouldn't reach every subscriber on a stream. Pgbus supports per-connection filtering via a registry of named predicates evaluated against each connection's authorize-hook context:
@@ -1145,6 +1391,95 @@ Failure semantics:
 
 The filter registry is process-local. Each Puma worker (or Falcon reactor) has its own copy populated at boot. Filter predicates run **on the subscriber side** — the predicate itself can't be serialized through PGMQ, so the broadcast carries only the label name.
 
+### Actor-echo suppression (`exclude:`)
+
+An actor who just triggered a change already applied it via the HTTP response of their own action. If the resulting broadcast reaches their own SSE connection too, it double-applies — re-running animations or clobbering an optimistic edit. Pass `exclude:` with a connection id to skip delivery to that one connection; everyone else still gets the broadcast:
+
+```ruby
+Pgbus.stream(room).broadcast(html, exclude: connection_id)
+```
+
+Every server-minted SSE connection exposes its id to the page: right after the open handshake, pgbus sends a `pgbus:connected` frame carrying the connection id, and `<pgbus-stream-source>` captures it onto its `connection-id` attribute and re-dispatches it as a `pgbus:connected` event (also present in `pgbus:open`'s detail). The page reads that id and sends it back as the `X-Pgbus-Connection` header on the action request that triggers the broadcast:
+
+```js
+document.addEventListener("pgbus:connected", (event) => {
+  document.querySelector("meta[name='pgbus-connection-id']")
+    ?.setAttribute("content", event.detail.connectionId)
+})
+
+// On the next fetch/form submit:
+fetch(url, {
+  method: "POST",
+  headers: { "X-Pgbus-Connection": connectionIdMetaTag() }
+})
+```
+
+```ruby
+def create
+  @message = @room.messages.create!(message_params)
+  @room.broadcast_append_to(:messages, exclude: request.headers["X-Pgbus-Connection"])
+end
+```
+
+A nil or blank `exclude:` is a no-op — the common path for background jobs and other server-initiated broadcasts with no originating connection. Reuses the existing per-connection delivery (Filters) path.
+
+### `broadcast_render` — render and broadcast in one call
+
+`Stream#broadcast_render` renders a Phlex component, a ViewComponent, or a pre-rendered HTML string into a complete `<turbo-stream>` action tag and broadcasts it atomically — removing the off-request render + tag-building boilerplate (and the easy-to-get-wrong view context) from every call site:
+
+```ruby
+Pgbus.stream("chat", room).broadcast_render(
+  renderable: Chat::Message.new(chat_message: msg),
+  action: :append,
+  target: "chat-messages-#{room}",
+  exclude: connection_id   # composes with #exclude
+)
+```
+
+`action` defaults to `:replace`; `target:` is required. The renderable is resolved via `String` → `#call` (Phlex) → `#render_in` (ViewComponent; called with a `nil` view context since there's no controller off-request) → `#to_s`. Content-less actions (`remove`) emit no `<template>` wrapper. `exclude:`, `visible_to:`, `durable:`, `event:`, and `coalesce:` all forward to `#broadcast` unchanged, so actor-echo suppression and audience filtering compose. A component that needs URL helpers or a full view context should be rendered by the app and the resulting string passed as `renderable:`.
+
+### Typed SSE event names (`event:`)
+
+A broadcast can set the SSE `event:` field while keeping the payload a Turbo Stream, so clients route on a typed name instead of sniffing the HTML:
+
+```ruby
+Pgbus.stream(name).broadcast(html, event: "presence")
+Pgbus.stream(name).broadcast_render(renderable: component, target: "cursor", event: "reactive")
+```
+
+The default (`nil` or `"turbo-stream"`) is omitted from the JSONB payload to avoid redundancy, but is still set on the SSE frame's `event:` line (falling back to `turbo-stream`), so default consumers still get the standard `message`/turbo-stream path.
+
+On the client, `<pgbus-stream-source>` dispatches a typed broadcast two ways: a generic `pgbus:event` (`{ event, data, msgId }`) for one listener that handles every typed event, and a named `pgbus:<event>` (`{ data, msgId }`) for `addEventListener("pgbus:presence", …)` ergonomics:
+
+```js
+document.addEventListener("pgbus:presence", (event) => {
+  const { data, msgId } = event.detail
+  // ...
+})
+```
+
+Native `EventSource` (the reconnect path) only invokes listeners registered by name, so declare every typed event name you use on the element's `listen-events` attribute (comma- or space-separated) — otherwise a typed broadcast is silently dropped after a reconnect, even though it worked on the first connection (which uses `fetch()` and routes any event generically):
+
+```erb
+<%= pgbus_stream_from @room, "listen-events": "presence reactive" %>
+```
+
+### `coalesce:` — publish-side debounce
+
+A chatty component — a live cursor, a typing indicator, a progress bar — can fan out many small broadcasts per second. Pass `coalesce:` (a window in milliseconds, or `true` for the 50ms default) together with `target:` to batch broadcasts per `(stream, target)` and publish only the *latest* frame within the window:
+
+```ruby
+Pgbus.stream(name).broadcast_render(
+  renderable: CursorPosition.new(x:, y:),
+  target: "cursor-#{user_id}",
+  coalesce: true   # or coalesce: 100 for a 100ms window
+)
+```
+
+Superseded frames never hit the bus at all — no PGMQ insert, no NOTIFY, no fan-out. This is last-write-wins, so it's only safe for **idempotent replace/update of a stable target** (exactly the high-frequency case above) — never for actions where every intermediate frame matters (an `append` to a running log, for instance).
+
+Semantics: the first submit for a `(stream, target)` schedules the flush one window later; every subsequent submit within that window only overwrites the buffered payload. Latency is bounded to one window (trailing-edge-with-max-wait, not a resettable debounce). The flush re-enters the normal broadcast path, so a coalesced frame still composes with `visible_to:`, `exclude:`, `event:`, and `durable:`. Coalescing is process-wide and in-memory — behind multiple Puma workers or Falcon processes, each process debounces its own submissions independently.
+
 ### Presence
 
 Pgbus tracks who is currently subscribed to a stream via a `pgbus_presence_members` table. This is the standard "X people are in this room" feature that chat apps and collaboration tools need:
@@ -1193,12 +1528,35 @@ Pgbus.stream(@room).presence.sweep!(older_than: 60.seconds.ago)
 
 The sweep uses `DELETE ... RETURNING` so multiple workers running it concurrently won't double-emit leave events.
 
-**Deliberately left to the application**:
+**Deliberately left to the application (manual API)**:
 
-- Join/leave is explicit, not connection-driven. The controller decides who is "present" — a connected SSE client is not always a present user (think tab-in-background, multi-tab dedup).
+- Join/leave is explicit, not connection-driven, unless you opt into connection-driven presence below. The controller decides who is "present" — a connected SSE client is not always a present user (think tab-in-background, multi-tab dedup).
 - The stale-member sweep is manual. Run it from a cron, an ActiveJob, or your existing heartbeat — pgbus does not assume one over the others.
 - The DOM markup for join/leave is whatever your `join`/`leave` block returns. Pgbus does not impose a fixed presence schema on `<pgbus-stream-source>`.
 
+#### Connection-driven presence (opt-in)
+
+Streams matching `config.streams_presence_patterns` (an exact string or a `Regexp`, mirroring `streams_durable_patterns`) automatically join a member when an SSE connection opens, leave when it closes, and refresh `last_seen_at` on every keepalive heartbeat tick — no explicit `join`/`leave`/sweeper calls required:
+
+```ruby
+Pgbus.configure do |c|
+  c.streams_presence_patterns = [/^room:/, "lobby"]
+end
+```
+
+Identity comes from the connection's authorize-hook context (the value your `StreamApp` `authorize:` callable returns). The built-in extractor handles the common shapes without any configuration: a `Hash` with `:member_id` (or `:id`) and optional `:metadata`, or any object responding to `#id` (e.g. a `User` model). For anything else, provide a custom extractor — a `->(context) { { id:, metadata: } }` callable returning `nil` for a context with no derivable identity (anonymous connections are simply skipped, not an error):
+
+```ruby
+Pgbus.configure do |c|
+  c.streams_presence_patterns = [/^room:/]
+  c.streams_presence_member = ->(user) {
+    { id: user.id, metadata: { name: user.name, avatar: user.avatar_url } } if user
+  }
+end
+```
+
+Membership work runs on the dispatcher thread (which already releases AR connections each pass); the heartbeat posts a batched touch per tick. Presence failures are logged and swallowed so a presence-table hiccup can't knock a live SSE connection out of the registry.
+
 ### Stream stats (opt-in)
 
 Pgbus can record one row in `pgbus_stream_stats` per broadcast, connect, and disconnect so the `/pgbus/insights` dashboard shows stream throughput alongside job throughput. This is **disabled by default** because stream event volume can dwarf job volume in chat-style apps — enable it deliberately when you want the observability.
@@ -1460,10 +1818,48 @@ Day-to-day running of Pgbus: starting and stopping processes, observing what is
 pgbus start     # Start supervisor with workers + dispatcher + scheduler
 pgbus status    # Show running processes
 pgbus queues    # List queues with depth/metrics
+pgbus dlq       # Inspect and drain dead-letter queues (see below)
+pgbus doctor    # Preflight diagnostics; exits 1 on any failed check (see below)
+pgbus mcp       # Start the read-only MCP diagnostic server over stdio
 pgbus version   # Print version
 pgbus help      # Show help
 ```
 
+#### pgbus doctor
+
+A single preflight command that answers "is this environment healthy enough to run?" — useful as a deploy or CI gate. It runs six checks and never raises; a broken environment turns every probe into a failed/warned check instead of a crash:
+
+| Check | Fails (`:fail`) when | Warns (`:warn`) when |
+|---|---|---|
+| Configuration | `Configuration#validate!` raises | — |
+| Database | Unreachable (`SELECT 1` via `Client#ping`) | — |
+| PGMQ schema | Schema not installed | Installed but untracked, or behind the vendored version |
+| Queues | A configured queue has no PGMQ table | — |
+| LISTEN/NOTIFY | — | A configured queue is missing its insert trigger (falls back to polling) |
+| Process liveness | Verdict is `STALLED` | Verdict is `DEGRADED` |
+
+```bash
+pgbus doctor       # prints the report; exit 1 unless every check passed
+rake pgbus:doctor  # identical checks, for a Rake-based deploy pipeline
+```
+
+The report ends with a resolved-config summary (queue prefix, pool size, roles, capsules) with any password redacted from `database_url` / `connection_params`. Warnings do not fail the exit code — only a `:fail` result does.
+
+#### pgbus dlq
+
+Dead-letter inspect/drain operations from a headless deployment or incident runbook, routed through `Web::DataSource` so retry/discard semantics are identical to the dashboard (origin-queue re-enqueue, transactional produce+delete, lock release on discard) with zero raw SQL and no direct PGMQ calls:
+
+| Subcommand | Does |
+|---|---|
+| `pgbus dlq list [--page N] [--per-page N]` | Table of msg_id / DLQ queue / origin queue / read_ct / enqueued_at, plus a total count. Never prints payloads. |
+| `pgbus dlq show MSG_ID` | Message metadata plus the payload, filtered through the dashboard's sensitive-data filter. |
+| `pgbus dlq retry MSG_ID` | Re-enqueue one message to its origin queue. |
+| `pgbus dlq retry-all` | Re-enqueue every dead-letter message; prints the count. |
+| `pgbus dlq purge MSG_ID` | Discard a single message. |
+| `pgbus dlq purge --all --yes` | Discard every dead-letter message. Omitting `--yes` makes no changes and exits 1. |
+
+An unknown `msg_id` or an unknown subcommand exits 1.
+
 #### Role flags (split deployments)
 
 By default, `pgbus start` boots every role in one supervisor (workers, dispatcher, scheduler, event consumers, outbox poller). For containerized deployments where each role lives in a separate process, use the role flags:
@@ -1495,7 +1891,7 @@ The dashboard is a mountable Rails engine at `/pgbus` with:
 - **Queues** — per-queue metrics, purge/pause/resume/delete actions
 - **Jobs** — enqueued and failed jobs, retry/discard actions
 - **Dead letter** — DLQ messages with retry/discard, bulk actions
-- **Processes** — active workers/dispatcher/consumers with heartbeat status
+- **Processes** — active workers/dispatcher/consumers with heartbeat status **and per-worker throughput** (e.g. `12.4/s processed · 0.2/s failed`)
 - **Events** — registered subscribers and processed events
 - **Outbox** — transactional outbox entries pending publication
 - **Locks** — active job uniqueness locks with state (queued/executing), owner PID@hostname, age
@@ -1503,6 +1899,8 @@ The dashboard is a mountable Rails engine at `/pgbus` with:
 
 All tables use Turbo Frames for periodic auto-refresh without page reloads. Destructive actions use styled confirmation dialogs (not browser `confirm()`), and flash messages appear as auto-dismissing toast notifications.
 
+Each worker snapshots its live in-process rate counter (dequeued/processed/failed) into its heartbeat metadata on every beat, so the Processes panel shows a cluster-wide, near-real-time throughput view with no extra query. Zero rates are omitted from the rendered string. Non-worker processes (dispatcher, scheduler, consumers) carry no rate data and render only their static boot metadata, unchanged.
+
 #### Queue management
 
 The queues page lets you manage PGMQ queues directly:
@@ -1528,6 +1926,17 @@ rails generate pgbus:add_job_stats --database=pgbus
 
 Stats collection is enabled by default (`config.stats_enabled = true`). Old stats are cleaned up by the dispatcher based on `config.stats_retention` (default: 30 days). If the migration hasn't been run yet, stat recording is silently skipped.
 
+Stats are buffered in memory and bulk-inserted rather than written one row per job. Each worker flushes its buffer when it fills to `config.stats_flush_size` entries (default: `100`), when `config.stats_flush_interval` seconds have elapsed (default: `5`), and immediately when the worker begins draining (graceful `TERM` shutdown or a recycle threshold). Tune the thresholds down for high-throughput deployments that want a tighter loss window, or up to reduce insert frequency:
+
+```ruby
+Pgbus.configure do |c|
+  c.stats_flush_size = 500     # flush after 500 buffered stats
+  c.stats_flush_interval = 2   # ...or every 2 seconds, whichever comes first
+end
+```
+
+**Loss window:** stats are advisory (dashboard/insights only), never job payloads. Buffered entries are flushed on graceful shutdown and drain entry, so a clean stop loses nothing. If the supervisor watchdog `SIGKILL`s a stalled worker — which cannot be trapped — up to `stats_flush_interval` seconds / `stats_flush_size` entries accumulated since the last flush are lost. This affects insights accuracy only; no job is dropped.
+
 ### Database tables
 
 Pgbus uses these tables (created via PGMQ and migrations):
@@ -1542,7 +1951,7 @@ Pgbus uses these tables (created via PGMQ and migrations):
 | `pgbus_semaphores` | Concurrency control counting semaphores |
 | `pgbus_blocked_executions` | Jobs waiting for a concurrency semaphore slot |
 | `pgbus_batches` | Batch tracking with job counters and callback config |
-| `pgbus_job_locks` | Job uniqueness locks (state, owner_pid, reaper correlation) |
+| `pgbus_uniqueness_keys` | Job uniqueness locks (lock_key, queue_name, msg_id) |
 | `pgbus_job_stats` | Job execution metrics (class, queue, status, duration) |
 | `pgbus_queue_states` | Queue pause/resume and circuit breaker state |
 | `pgbus_outbox_entries` | Transactional outbox entries pending publication |
@@ -1617,6 +2026,7 @@ PostgreSQL + PGMQ
 | `prefetch_limit` | `nil` | Max in-flight messages per worker (nil = unlimited) |
 | `dispatch_interval` | `1.0` | Seconds between dispatcher maintenance ticks |
 | `circuit_breaker_enabled` | `true` | Enable auto-pause on consecutive failures (threshold and backoff are tuned via `Pgbus::CircuitBreaker` constants) |
+| `read_timeout` | `30` | Seconds before a single PGMQ read is bounded (libpq `statement_timeout` + `tcp_user_timeout` on a dedicated connection; nil disables) |
 | `priority_levels` | `nil` | Number of priority sub-queues (nil = disabled, 2-10) |
 | `default_priority` | `1` | Default priority for jobs without explicit priority |
 | `archive_retention` | `7.days` | How long to keep archived messages. Accepts seconds, Duration, or `nil` to disable cleanup |
@@ -1632,6 +2042,8 @@ PostgreSQL + PGMQ
 | `web_live_updates` | `true` | Enable Turbo Frames auto-refresh on dashboard |
 | `stats_enabled` | `true` | Record job execution stats for insights dashboard |
 | `stats_retention` | `30.days` | How long to keep job stats. Accepts seconds, Duration, or `nil` to disable cleanup |
+| `stats_flush_size` | `100` | Buffered stat entries per worker before a bulk insert flush. Positive integer. Lower = tighter SIGKILL loss window. |
+| `stats_flush_interval` | `5` | Seconds between periodic stat buffer flushes. Positive number. |
 | `streams_test_mode` | `false` | Return a stub SSE response without hijack or background threads. Auto-enabled by `Pgbus::Testing.fake!`/`.inline!`. See [SSE streams in tests](#sse-streams-in-tests). |
 | `streams_stats_enabled` | `false` | Record stream broadcast/connect/disconnect stats (opt-in, can be high volume) |
 | `streams_path` | `nil` | Custom URL path for the SSE endpoint (nil = auto-detected from engine mount) |
@@ -1639,6 +2051,12 @@ PostgreSQL + PGMQ
 | `error_reporters` | `[]` | Array of callables invoked on caught exceptions. Each receives `(exception, context_hash)`. |
 | `log_format` | `:text` | Log formatter (`:text` or `:json`). Sets `logger.formatter` automatically. |
 | `metrics_enabled` | `true` | Enable Prometheus-compatible metrics on the dashboard |
+| `metrics_backend` | `nil` | Generic metrics adapter: `nil` (off), `:prometheus`, `:statsd`, or a `Pgbus::Metrics::Backend` instance |
+| `statsd_host` | `"127.0.0.1"` | StatsD UDP host (used when `metrics_backend = :statsd`) |
+| `statsd_port` | `8125` | StatsD UDP port (used when `metrics_backend = :statsd`) |
+| `health_port` | `nil` | Port for standalone HTTP liveness/readiness probes served by the supervisor; nil disables |
+| `health_bind` | `"127.0.0.1"` | Bind address for the standalone health server |
+| `eager_validation` | `true` | Run `Configuration#validate!` automatically after `Pgbus.configure` / `ConfigLoader.apply`; an invalid value raises `Pgbus::ConfigurationError` at boot. Set `false` to suppress and validate manually. |
 
 ## Development
 
@@ -1665,6 +2083,17 @@ PGBUS_DATABASE_URL=postgres://user@host/db bundle exec rake bench:streams
 
 The harness measures single-broadcast roundtrip latency, burst throughput, fanout to many clients, and concurrent connect under thundering herd. See `benchmarks/streams_bench.rb`.
 
+### Dependency watch
+
+A nightly GitHub Action (`.github/workflows/dependency-watch.yml`) compares the
+vendored PGMQ SQL (`lib/pgbus/pgmq_schema/pgmq_v*.sql`) and the locked
+`pgmq-ruby` gem version against their upstream releases, opening a GitHub issue
+on drift (deduplicated by exact title, so repeat runs are a no-op). It never
+auto-vendors SQL or auto-bumps the gem -- it only notifies. Set the optional
+`SLACK_WEBHOOK_URL` repository secret to also post drift notifications to
+Slack; the workflow logs and skips that step gracefully when the secret is
+unset.
+
 ## License
 
 The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).