pgbus 0.17.0 → 0.17.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 11b7ad58d91fe1eb9c0388008907523c77324f286f27a0e97a94353e4a8577b0
-  data.tar.gz: '0707593bb7c826f48f9a84db9a666057aaed3407b0d1cb559007f57caf080be2'
+  metadata.gz: 4fd9db0886380c5daee4231c3b3ae18740de1d75b7e763d159e0a55511e40fda
+  data.tar.gz: 77e1ce68911f70dc2aa2d515cb1f621932d49942acda00672c7c3c8418c7a0a8
 SHA512:
-  metadata.gz: 0f1422dea206dbeaac10b1fd7dfa74da2338c007cffb48d7a2b5429bf3a94e6f3e78b5795832d0a2f45346fe4350d4c7355d16b773247420894ea5234c50db85
-  data.tar.gz: 6d30c3583b7e6b5fa9dc89cf42115f57b182dea18b2d363f1ec413d6096ba32b76d75f008dcf7a2f25de3baf9ceba7e17f1edbb382b96e2aeb23bf52a8824af2
+  metadata.gz: 4ad2af9f7cdb993bf5f6577e2ab94cd4f3df8833e813b32e75751e694e0d7bb188ac49fd3ae11fdb46c9070a35b674579f756472519dae46cda8efc55f40afea
+  data.tar.gz: c9fd9452c8bff020cc581bb2fcd5e3cf1980d6c51b1ddf694b2bc5aeb20b29552c678b3ce3deb85f24b994b0a35e6cceba9b5cb15cb480d72bdd3b49f0a83afa

data/CHANGELOG.md

@@ -2,6 +2,26 @@
 
 ### Added
 
+- **The home page and Insights explain themselves.** The home page (`/pgbus`) showed numbers without words ("Failed / DLQ 5 / 3", a Throughput card that was always `0.0` on the web, Queue Health percentages coloured against thresholds it never stated). It now opens with a health verdict strip — the OK / DEGRADED / STALLED verdict and reasons `/readyz` and the MCP compute (`HealthAnalyzer`), with links to the page that explains them. Eight cards each say what their number means and link to the page that explains it: Queues (paused, dead-letter, holding jobs nothing drains), Enqueued (claimable now vs scheduled, running or retrying), Processes (healthy · stalled · stale), Recurring, Retrying (the Jobs page's Retrying count, when the next attempt runs, how many are due now, the newest error), Dead-lettered (the newest message's reason, read as the DLQ page reads it), Throughput (jobs/min over the last 5 minutes from job stats, or "needs job stats") and Parked. The queues table gains a Status column in words (the queue page's summary line: paused and why, nothing drains it, what is waiting). Queue Health states each threshold ("12% on pgmq.q_pgbus_events — vacuum due (warns above 10%, critical above 20%)"; the MVCC card names both its own 30 s / 60 s warnings and the verdict's 300 s) and its table always renders, with an empty state. Recent failures show the job and attempt and link to the failure. New `Web::DataSource#dashboard_signals` and `#queue_status_lines` (in `DataSource::DashboardSignals`) and the pure presenters `Web::HealthVerdict` and `Web::InsightsCharts`; `summary_stats`, `/pgbus/api/stats`, `/pgbus/api/metrics` and the AppSignal probe are unchanged. On Insights each chart has a caption sentence (also the chart's accessible name) and a `<details>` data table rendered on the server (at most 60 rows: long ranges roll minutes up), empty series show an empty state, and the page draws from series it embeds instead of fetching `/pgbus/api/insights` a second time (that endpoint is unchanged). Closes #492.
+
+- **The Processes page says what each process is doing, why its status is what it is, and what happens next.** `/pgbus/processes` showed kind · host · pid · a Healthy/Stalled/Stale badge · "45s ago" · a dump of raw metadata chips; the thresholds behind the badge lived only in Ruby, and nothing said what the process was doing or what would happen to it. Each row now has Process · Status · Activity · Recycle · Heartbeat cells: the Status cell says why ("Loop last ticked 2m ago (limit 1m 30s)", "No heartbeat for 7m 0s (limit 5m 0s)", "Heartbeat 5s ago (limit 5m 0s) · stall detection off") and what happens next ("the supervisor's watchdog restarts it", "no supervisor to restart it", "probably dead; removed at the dispatcher's next sweep (within 5m 0s)", "no live dispatcher to remove it", or what stops advancing when the dispatcher or scheduler stalls); Activity says what it drains or consumes, busy threads (or async slots), buffered claims, jobs/s and failures/s, what a dispatcher, scheduler or outbox poller does, and what a supervisor supervises; Recycle states progress against each configured limit ("1,240 / 10,000 jobs · 310 / 512 MB · up 42m 0s / 1h 0m"); Heartbeat adds uptime. Children are listed under their supervisor ("under supervisor 100"), grouped only when that supervisor's row is on the same host, so a standalone `pgbus worker` is never claimed by a stranger. Clicking a process expands its start time and the metadata nothing else renders. The home page's Processes panel puts the same sentence under each badge. Status derivation moved from `DataSource` into the pure presenter `Web::ProcessState` (with `Web::ProcessActivity` for the activity and recycle words), so the badge and its sentence come from the same moment and limits; `DataSource#processes` keeps its row shape and adds `supervisor_pid` and `started_at`. **New heartbeat keys**, written once per 60 s beat and never per job: every process `started_at`; forked kinds `supervisor_pid`; workers and consumers the pool's `capacity`/`busy`, `buffered`, `memory_mb` (an RSS read through `MemoryUsage`'s 5 s cache — a `ps` subprocess per minute on macOS) and the recycle limits the process resolved (a capsule's own when set); consumers gain a metadata supplier with their handled count. No schema change. Because `capacity`/`busy` are now persisted, `pgbus_worker_pool_capacity` and `pgbus_worker_pool_busy` on `/pgbus/api/metrics` emit for the first time. A process from before this release shows "—" where a fact is missing. Closes #493.
+
+- **The Events page says what state each event is in, why, and what happens next.** `/pgbus/events` had three tables that explained nothing: a pending list that faked columns inside one cell and showed a raw `vt` and `read_ct`, a processed list of `event_id · handler · processed_at`, and a subscriber list. It now has one pending-events list with real cells (ID · Event · Handler · State · Attempts · Next run / Reason · Actions), state tabs with counts (All / Ready / Scheduled / Handling / Retrying), a pager, auto-refresh and the Jobs page's row expansion. Every row names the handler its queue belongs to (the first subscriber registered on that queue) and says in words why it waits (events ahead, queue paused, no running consumer subscribes to its pattern, no healthy consumer, lease expired), that it is being handled (claimed when, lease expires when), or which attempt failed with which error and when the next one is due (or that the next read dead-letters it). Mark Handled, Discard, Reroute, Edit & Retry and Discard Selected keep working; an orphaned failed row (message gone) is discarded through its failed event. The processed-events audit says whether each claim completed, is being handled (claim refreshed recently) or went silent (the next delivery re-runs the handler), pages on its own `processed_page`, and offers **Replay** for a completed claim whose handler is still registered and whose archived message still carries a routing key its pattern matches. The subscriber list says whether a healthy consumer's topics cover each pattern and links to that handler's pending events. Internally the list is the Jobs list's SQL scoped to the handler queues: `DataSource#job_rows` / `#job_state_counts` take optional `queues:` and `exclude:` (additive; without them the SQL is unchanged), and `DataSource::EventList` + the pure presenter `Web::EventState` add the event words. `EventBus::Handler.claim_ownership_window` is now a class method. Closes #494.
+
+- **The dead-letter queue says why every message died.** A job that exhausted `max_retries` was moved to `<queue>_dlq` and, in the same breath, `FailedEventRecorder.clear!` deleted the only row that knew what went wrong, so `/pgbus/dlq` could show only the payload and a "Reads" column counting reads *in the DLQ*. The mover (the ActiveJob executor or the event consumer) now reads the last recorded error before the move and writes a `pgbus_dead_letter` block into the DLQ copy's PGMQ headers (`Pgbus::DeadLetterHeader`): reason (`max_retries_exceeded`), source (`worker`/`consumer`), the physical source queue (a priority sub-queue included), attempts, `max_retries`, `dead_lettered_at`, and the last error's class, message (1,000 chars), backtrace head (10 lines) and the attempt it came from. Existing headers (trace ids, `x-pgmq-group`) are kept. `Client#move_to_dead_letter` takes `headers:`. The DLQ page is a real table (ID · Job · Source queue · Died · Attempts · Reason · Actions) whose Reason cell says in words what killed the message; the row expansion shows the full error, the backtrace head and which later attempts recorded nothing; chips filter by DLQ (`?dlq=`) with counts and clicking an error class filters by it (`?error_class=`); `/pgbus/dlq/:id` opens with a "Why it died" card. Retrying from the DLQ strips the block (a live message must not claim to be dead) and carries `pgbus_dlq_retries`, so a second death reads "retried from the DLQ once before". Messages dead-lettered before this release read "Reason not recorded (dead-lettered before pgbus 0.18.0)" and keep working everywhere. `pgbus_dlq`/`pgbus_dlq_detail` (MCP) gain a `dead_letter` summary (error message and backtrace only when payloads are allowed); `pgbus dlq list` gains a REASON column and `pgbus dlq show` prints died, attempts, source, reason and error. `pgbus.job_dead_lettered` carries `error:` (the class). No schema change. The executor's success path is untouched: `executor_bench` main vs branch, same machine back to back, 64.6k vs 66.7k i/s (noise) and 6,448 B / 68 objects per job on both. Event handler errors appear in the block once the consumer records them (#494). Closes #495.
+
+- **docs: a "Sizing capsules" page on the docs site** (Operations, `/docs/sizing-capsules`). It covers when to give a queue threads, fibers, `processes:` or a capsule of its own: how to tell a CPU-bound queue (symptoms, plus the `gvl_wait` baseline from `rake bench:worker_profile`), a decision table, use cases with before/after configs (rendering fan-out, a memory-heavy export, duplicated capsules, a bulk sweep starving latency-sensitive work, and the counter-example of work in an external binary), the memory and connection math for a job host, and a checklist for moving off the workarounds. The README's "Threads vs processes" section and the docs site's Running workers page link to it. Closes #505.
+
+- **Capsules take `processes: N` for CPU-bound queues, and their own recycle limits.** A capsule was one forked process with `threads:` threads; threads share one GVL, so a `threads: 8` capsule running CPU-bound jobs (rendering, reports, PDFs) used about one core, and the only ways to more cores were several supervisors per host or sharding the queue. `c.capsule :render, queues: %w[render], threads: 1, processes: 4` now forks four identical workers reading the same queues. Each fork has its own slot — restart backoff, liveness pipe, `pgbus_processes` row, and a wake pipe when the supervisor-scope NotifyHub is active — so a crashed fork restarts alone; readiness counts every fork; the boot banner prints `processes=N`; worker heartbeat metadata gains `capsule` and `process` (`"2/4"`), which the dashboard's Processes page shows; and `pgbus doctor`'s connection budget counts each fork under `worker_notify_scope: :fork`. The default is 1, so every existing config boots unchanged, and the queue-overlap rule is untouched (the forks belong to one capsule). A capsule may also set `max_memory_mb:`, `max_jobs_per_worker:` and `max_worker_lifetime:`, which recycle only that capsule's workers and fall back to the global values when unset; all are validated at boot. README and the docs site gain a "Threads vs processes" section with the memory and connection math. Not in this change: `processes: :auto`, GVL-contention observability, and a string-DSL syntax for `processes`. Closes #503.
+
+- **feat(web): every growing dashboard list has a pager, every button goes through one helper, and mobile card labels follow the locale.** Outbox, Batches, both Locks tables and Recurring tasks now render the shared pager. Before, Batches silently dropped the 101st batch, Locks and concurrency keys stopped at 100, Recurring tasks loaded every row, and Outbox fetched pages but had no links to them. Totals come from a bounded count (`Web::DataSource#list_count`, a `LIMIT 10_001` probe; past the cap the pager says "10,000+"). The two Locks tables page independently on `page` and `keys_page`. Framed lists advance the URL, so the auto-refresh stays on the page you opened. The DataSource readers take `page:`/`per_page:` with defaults that keep today's output. The `pgbus_locks`, `pgbus_concurrency` and `pgbus_recurring` MCP tools accept optional `page`/`per_page` (max 100). Omitting them returns the same rows as before; the response now also carries `page`, `per_page`, `total` and `has_more`. `Pgbus::ButtonHelper` (`pgbus_button_to`, `pgbus_link_to`, `pgbus_button_tag`, `pgbus_button_classes`) replaces about 56 hand-written class strings. Each variant has light and dark colours, a focus ring and a target of at least 24 px (WCAG 2.5.8); row-action links used to be about 16 px tall. Labels and confirms did not change. Every table cell's `data-label` (the label on the card shown below 1024 px) now reuses its column header's translation, so a German operator no longer sees English labels at phone width. Empty states go through `shared/_empty_row`. `spec/pgbus/web/view_conventions_spec.rb` pins all of this, and the contrast and compiled-CSS guards now scan every helper file. Closes #496.
+
+- **Dashboard quality gates: an axe WCAG 2.1 AA gate in light and dark mode, monthly Lighthouse budgets, one stub data source.** `spec/system/accessibility_spec.rb` runs axe-core 4.13 (from `node_modules`, no gem) on every dashboard page — dashboard, queues + detail, jobs with every state tab and an expanded row, job detail, recurring + detail, processes, events + detail, batches + detail, DLQ + detail, outbox, locks, insights — in both themes, against rich sample data and against the empty states (tags `wcag2a`, `wcag2aa`, `wcag21aa`, and `wcag22aa` for target size, which Lighthouse scores too); any violation fails the system job, and there is no way to accept one. The page list is `lighthouserc.dashboard.json`, shared with Lighthouse, and a route-coverage example fails when an HTML GET route of the engine has no audited URL. Dark mode is entered the way a user enters it (stored preference, fresh page), so the body's colour transition never runs under the audit. Lighthouse runs monthly and on demand (`.github/workflows/lighthouse.yml`, `lhci collect → upload → assert`, score table in the job summary, 30-day artifact) and never gates a merge; `bin/lighthouse` runs it locally against `rake dummy:server`. The dummy server now uses `Pgbus::Test::StubDataSource#fill_sample_data!` — the same rows the gate audits — instead of its own stub, which lacked 12 methods and answered 500 on `/pgbus/events`, `/pgbus/dlq` and `/pgbus/locks`; Events, Outbox and the Insights tables now render with data there. The static contrast guard learned the shades the browser caught. `AGENTS.md`, `.claude/rules/dashboard.md`, `.claude/rules/testing.md` and `/lfg` say how a dashboard change is tested. Closes #498.
+
+- **feat(web): the Jobs page is one list that says why each job is there and when it will run.** The old page had two unrelated tables — "Failed Jobs" and "Enqueued Jobs" — that listed a retrying job twice (its failed-event row and its queue message) and never said what state a job was in. `/jobs` is now a single table with the same columns for every row (ID · Job · Queue · State · Attempts · Next run / Reason · Actions), filter tabs with counts (All / Ready / Scheduled / Running / Retrying / Blocked) and a pager. Each row says in words why it is waiting: scheduled with its run time; ready with the number of jobs ahead of it, or the reason nothing will pick it up (queue paused, no worker drains the queue, no healthy worker); running with when it was claimed and when its lease expires; retrying with the failed attempt, the error and the next attempt (or that the next read dead-letters it); blocked by `limits_concurrency` with its key and slot usage, linked to the Locks page. Failed-event rows whose message has left its queue show as retrying ("message no longer in queue"). The state is derived in SQL (new `Web::DataSource#job_rows` / `#job_state_counts` / `#jobs_ahead`, in `DataSource::JobList`) so tabs, counts and paging agree; tab counts are capped at 10k per queue so a huge queue never forces a full scan on auto-refresh. `DataSource#jobs` and `#failed_events` are unchanged. The dashboard's "failed jobs" links (`/jobs?status=failed`) land on the Retrying tab. **Dark mode:** every dashboard page now pairs each light colour with a `dark:` variant (hostnames, PIDs, headers and badges were near-invisible on dark cards), rows no longer paint a darker stripe inside their card, and the empty-state colspans match their tables. A static spec (`spec/pgbus/web/dark_mode_contrast_spec.rb`) keeps it that way. Closes #489.
+
+- **`rake bench:worker_profile` measures the event consumer too.** A consumer cell drives a real `Process::Consumer` over an event-bus topic queue (plain handler, so the transport is measured, not the dedup table) with the same harness as the worker cell. `WP_BENCH_ROLE` (`worker`, `consumer`, `both`) and `WP_BENCH_READ_AHEAD` (passed as `read_ahead:`) are new knobs. Refs #486.
+
 - **`coalesce:` now works on the `Turbo::StreamsChannel` broadcast path.** `Pgbus::Streams::Stream#broadcast` has accepted `coalesce:` since #171, but the Turbo patch forwarded only four options (`durable:`, `exclude:`, `visible_to:`, `event:`), so nothing that broadcasts through `Turbo::StreamsChannel.broadcast_*_to` — every `Turbo::Broadcastable` model, and phlex-reactive's `Streamable.broadcast_to` — could debounce. A background job settling a 177-row collection therefore pushed 177 idempotent replaces of the same count badge at every peer. Any targeted helper now takes `coalesce:` (`@order.broadcast_replace_to :account, coalesce: 100`), as does a direct channel call (`Turbo::StreamsChannel.broadcast_replace_to(@account, target: "order-count", coalesce: true)`), with the same semantics as the direct API: opt-in, per `(stream, target)`, last-write-wins, `true` = the 50ms default window, an integer = the window in ms. Two things the option's plumbing needed beyond a fifth thread-local: the coalescing key, which dedupes on `(stream, target)` while `broadcast_stream_to` only ever sees the rendered `content:` — it is now resolved from the frame's own `target:`/`targets:` through the same `ActionView::RecordIdentifier.dom_id` conversion Turbo applies, so a record target keys on `order_7` rather than on a per-instance `#to_s`; and extraction of the pgbus options at the channel itself, since a direct `Turbo::StreamsChannel` call never passes through `Turbo::Broadcastable` and so previously leaked `durable: true` (and now `coalesce:`) into Turbo's renderer. `broadcast_refresh_to` and `broadcast_render_to` carry no target and so cannot coalesce — they raise the existing actionable "requires target:" error rather than guessing a key — and the `_later_to` variants remain out of scope, as the thread-local cannot follow into a job. The wire is byte-identical when `coalesce:` is absent. Closes #465.
 
 - **Concurrency is now visible: a section on the Locks page, three gauges, a home stat card and an MCP tool.** `limits_concurrency … on_conflict: :block` became durable in #460 — a parked job is never dropped and a key never runs past `to:` — but nothing showed it, so an operator whose per-record pipeline looked "stuck" had no way to see that the key's slot was held, how many jobs were parked behind it, or how long the oldest had waited, short of SQL. `pgbus_semaphores` and `pgbus_blocked_executions` appeared on no page, in no gauge, in no tool. One new read, `Web::DataSource#concurrency_stats`, now feeds four surfaces: the Locks page's **Concurrency** section (parked jobs, oldest wait, slots held, keys at limit, then a row per key with value / limit, lease state, parked count and oldest wait, auto-refreshing like the other dashboard panels); a **Parked jobs** card on the dashboard home; three unlabelled gauges on both `/pgbus/api/metrics` and the AppSignal probe (`pgbus_concurrency_blocked_executions`, `pgbus_concurrency_blocked_oldest_age_seconds`, `pgbus_concurrency_slots_held` — keys are per-record, so a per-key label would be an unbounded series); and a read-only `pgbus_concurrency` MCP tool returning the same data. The key rows come from a FULL OUTER JOIN, not a LEFT one: a key can have parked jobs and no semaphore (its holder died and the sweep removed the row) or a semaphore and nothing parked, and both shapes matter to an operator. Each row carries two guarded escape hatches. **Release** drops the semaphore and promotes through `Concurrency::BlockedExecution.promote_next`, so it takes each slot through the same guarded upsert an enqueue uses and can never reintroduce the over-admission #460 closed; when the lease is still fresh the confirm says so, because a job is then probably still running and releasing lets another start beside it. **Discard parked** drops a single key's parked jobs, naming the exact count in the confirm, and resolves the bookkeeping those jobs will never resolve themselves — a parked batch child is marked failed so its batch stops waiting and `on_failure` fires (#413), and an `:until_executed` uniqueness lock is released rather than orphaned (#423); each discard logs a warning naming the job and emits `pgbus.blocked_execution_discarded`. There is deliberately no cross-key "discard all parked": not losing work is the whole point of `:block`. Nothing is added to the enqueue or execute path. Refs #461.
@@ -10,8 +30,38 @@
 
 - **The batches list and a batch's progress panel now refresh themselves.** Batches was the only dashboard list whose `turbo-frame` was missing `data-auto-refresh`, and the batch detail page had no frame at all, so watching a batch drain meant reloading by hand. Both now poll on the existing `web_refresh_interval` (5s, pauses while you interact) like every other list — the batch progress panel is served by `GET /batches/:id?frame=progress`. This reuses the dashboard's turbo-frame polling rather than the SSE stack in `lib/pgbus/web/stream_app.rb`: that is the host-app-facing stream feature, and pointing it at the dashboard would open a PG `LISTEN` connection per viewer to move four integers.
 
+- **`pgbus doctor` reports the Ruby JIT, and the boot banner names it.** A new "Ruby JIT" check is `:ok` when YJIT (or ZJIT) is on, when the Ruby has no JIT, or in a local (development/test) environment, and warns otherwise — never fails, never enables anything; turning YJIT on stays with Rails' `config.yjit`. The supervisor's first boot line now ends `jit=yjit|zjit|none`. Measured reason: with Postgres close by, a worker is CPU-bound under the GVL and YJIT is worth ~14–20 % jobs/s (`docs/performance.md`). Refs #484.
+
+- **`rake bench:worker_profile`: jobs/s and where a job's time goes, for a real worker.** Drives a real `Pgbus::Process::Worker` against real PGMQ and splits each thread's time (Postgres wait, pool-checkout wait, waiting for work, GVL wait, and whose code was on-CPU) with vernier, locally and optionally behind toxiproxy-injected latency. `vernier` joins the test group; it is not a runtime dependency. The same change repairs `benchmarks/integration_bench.rb`, which no longer ran against 1.0's `ensures_uniqueness`. Refs #484.
+
+### Changed
+
+- **The Jobs page leaves event-handler queues to the Events page.** `/pgbus/jobs` without a queue filter no longer lists EventBus handler-queue messages (they appeared on both pages), and its Retry All, Discard All and Discard All Enqueued skip handler queues: Retry All would have re-sent a copy of a failed event whose message still sits in its handler queue, running the handler twice. A handler queue's own queue page (`/pgbus/queues/:name`, `/pgbus/jobs?queue=`) still lists its messages. `DataSource#jobs` and the MCP `jobs` tool are unchanged. Refs #494.
+
+- **feat(web): the queue page shows the unified Jobs list and explains the queue in words.** `/pgbus/queues/:name` used to render its own message table: raw `vt`, timestamps and `read_ct` columns, no state, no reason, no pager, and Retry/Discard hidden inside the expansion. It now renders the same list as `/pgbus/jobs` (state tabs with counts, state badge, attempts, reason, row expansion, bulk discard, pager, auto-refresh), scoped to that physical queue. On a priority sub-table (`_pN`) the scope follows `DataSource#job_rows` as it is: blocked jobs of the logical queue appear on every level's page (they are parked before a level is chosen), and a failed event whose message sits in a sibling level can show as "no longer in queue". Tabs, the pager and the auto-refresh source stay on the queue's own URL. Above the list, a summary says why the queue is in its state: paused by an operator (with the reason) or by the circuit breaker (failures, when it resumes, trip number); which worker capsules drain it, or that none does; whether a healthy worker is alive; how many jobs are claimable now, how long the oldest has waited and how many are parked; and, for a `_pN` table, its priority level, with links to the other levels. The header's `|`-separated raw seconds become labelled metrics (`pgbus_duration`). A dead-letter queue page shows its summary and a link to the Dead Letter page instead of a list. "Discard All Enqueued" acts on every queue, so it is hidden on a queue page and on `/pgbus/jobs?queue=…`; Purge Queue covers that case. Retry, Discard, Discard Selected and Discard All Enqueued on the Jobs list now redirect back to the page they were clicked on. New `Web::DataSource#queue_pause_state` / `#queue_drainers` (in `DataSource::QueueSummary`) and the pure presenter `Web::QueueSummary`. `DataSource#jobs`, `#queue_detail`, `#queue_paused?` and `#queue_health_detail` and the MCP tools are unchanged. Closes #491.
+
+- **Dashboard: one timestamp and duration presentation, translated, always in `Time.zone`.** Each page used to render time its own way: raw `Time#to_s` or ISO strings in expanded rows and on show pages, bare integer seconds for queue ages ("3600" and "3600s"), and hard-coded English "5m ago" / "in 2h". A new pure presenter, `Pgbus::Web::TimeFormat`, accepts every shape the data sources return (`Time`, `TimeWithZone`, `DateTime`, ISO or `Time#to_s` strings, epoch seconds, nil). It formats them in `Time.zone`, with every word from `pgbus.helpers.time` in all 12 locales. Lists now show `<time datetime="…UTC…" title="…absolute…">5m ago</time>`, so the exact value is in the tooltip. Expanded rows and show pages show the absolute followed by the relative ("2026-10-10 02:08:54 CEST (5m ago)"), and every age is a duration with units. Job reasons render their times as `<time>` through `_html` locale keys, so `%{error}` and the other arguments are still escaped. Relative times refresh with the existing auto-refresh frames; no new JavaScript. Two fixes come with it: a future `vt` on the queue page rendered as "-1d ago", and the scheduled-job ETA raised `NoMethodError` on a String time. Removed (engine-internal): `pgbus_time_ago`, `pgbus_time_ago_future`, `pgbus_job_eta`, and the `pgbus.outbox.index.stats.oldest_age` / `no_age` keys. The `pgbus.jobs.list.reasons.*` keys are now `*_html`. No `Web::DataSource` return shape changed. Two guards keep it that way: the RuboCop cop `Pgbus/DashboardTimeFormatting` flags `strftime`, `time_ago_in_words`, `distance_of_time_in_words`, `to_fs` and `l`/`localize` in `app/` and `lib/pgbus/web/` (outside `TimeFormat`). `spec/pgbus/web/time_conventions_spec.rb` does the same for the ERB views, which RuboCop cannot parse, and also fails on a raw `*_at` / `vt` / `*_age_sec` field printed without a helper. The rules are written down in `.claude/rules/dashboard.md` and `AGENTS.md`. The queue-age column headers lose their "(s)" suffix, since the cells now carry units. Closes #497.
+
+- **CI now runs every spec under `spec/`.** The Ruby × Rails matrix job ran only `spec/pgbus/` and `spec/generators/`, so the 17 dashboard/API request specs, the i18n-tasks checks across all 12 locales, the custom RuboCop cop specs, `spec/pgbus_spec.rb` and the worker-profile harness spec ran only on a developer's machine — a PR could break a controller or ship a missing translation and still go green. All of them are DB- and browser-free and now run on every matrix leg, so the request specs exercise the controllers on Rails 7.1 and 8.x. A new static spec, `spec/pgbus/ci_coverage_spec.rb`, parses the `rspec` paths out of `.github/workflows/main.yml` and fails when any spec file outside `spec/dummy` and `spec/support` is not run by a CI job (option values such as `-I spec` are not counted as paths), or when the workflow names a spec path that does not exist. Closes #499.
+
+- **perf(worker, consumer): opt-in read-ahead, so one read feeds many jobs when Postgres is on another host.** A worker read `qty = free threads`; once its pool was busy, threads freed one at a time, so it made one read round trip per job. Under +1 ms of latency that capped a 12-thread worker at ~1.3–1.7 k no-op jobs/s however many threads it had (#484), and the event consumer's loop had the same shape. New `config.read_ahead` (default **0**, off; overridable per capsule and per `event_consumers` entry with `read_ahead:`) lets either loop claim up to N messages beyond its free threads. Claimed messages wait in a new `Process::ClaimBuffer`, kept invisible by a new `VisibilityHeartbeat.hold`, until a thread frees. They are handed back to the queue (`set_vt` 0) the moment the worker drains, recycles, pauses or stops, or the consumer shuts down or recycles, with `read_ct` already counted. Buffered claims count as in flight, so `prefetch_limit` still caps everything claimed. Measured with `rake bench:worker_profile` (12 threads, no-op jobs, +1 ms per round trip, rounds 2–3 of three interleaved rounds on a loaded machine): **worker 1 286–1 731 → 2 300–2 462 jobs/s, consumer 1 326–1 779 → 2 288–2 599** at `read_ahead` 12. Locally both are within noise, because there the worker is CPU-bound under the GVL; `read_ahead` 0 overlaps the baseline's run-to-run range in 7 of 8 cells; consumer / proxied / no JIT sits 2–18 % below, plausibly load noise but unproven. A returned message's next claimant logs it as a zombie redelivery when `zombie_detection` is on, and `validate!` warns when `read_ahead > 0` meets `visibility_heartbeat = false`, since nothing then keeps a buffered message invisible. With read-ahead on, the next ceiling under latency is the connection pool (checkout wait 2–3 % → 14–15 %), so raise `pool_size` with it. `VisibilityHeartbeat`'s unregister now drops only the entry it registered, so a stale hold released after a job's own tracking took over cannot drop that job's heartbeat. Worker heartbeat metadata gains `"buffered"`. Refs #486.
+
+- **perf(executor): a successful first delivery no longer deletes from `pgbus_failed_events`.** Every successful job ran a `DELETE … WHERE queue_name = $1 AND msg_id = $2` through ActiveRecord, but only a failed attempt ever writes that table, so a job on its first delivery cannot have a row. The DELETE now runs only on redelivery (`read_ct > 1`), still clearing the row a failed attempt left. Measured with `rake bench:worker_profile` (12 threads, no-op jobs, local Postgres): **4 569 → 5 823 jobs/s with YJIT (+27 %), 3 797 → 5 119 without (+35 %)**; under +1 ms injected latency the change is within noise, because there the worker's single reader is the ceiling. This is a per-job query removed, not faster job code. The executor's debug tag is also built only when debug logging is on (9 → 8 pgbus-owned objects per job, now a hard budget). `Client#drop_queue` now also deletes the dropped queue's `pgbus_failed_events` rows: PGMQ restarts `msg_id`s when a queue is recreated, and a surviving row would point the dashboard's retry/discard at an unrelated new message — previously a first-delivery success happened to sweep such a row, and that sweep is gone. Refs #484.
+
 ### Fixed
 
+- **Insights charts are readable in dark mode.** The charts were themed only through `foreColor`: in dark mode the donut kept white slice separators, axis borders and the crosshair label stayed light, and the "No data" fallback was `text-gray-400 dark:text-gray-500` (below WCAG AA, and invisible to the contrast guard because it lived in JS). `charts.js` now themes every chart from one `palette(isDark)` (theme mode, axis and grid colours, donut separators in the card colour, legend and tooltip, animations off under `prefers-reduced-motion`) and redraws on the dark-mode toggle without a reload. The latency headers and chart titles no longer carry "(ms)" / "(jobs/min)"; the P95 latency card's yellow is now AA. The static contrast guard scans the dashboard's JS modules too. Refs #492.
+
+- **The dashboard home page reads the process rows and the stream registry once per render.** `DataSource` memoizes the raw process rows, `stream_queue_names` and `queue_health_stats` per instance (cleared by `reset_cache!`, as `queues_with_metrics` already was), so the stats, the verdict and the panels share one query each. `HealthApp` now calls `reset_cache!` on an injected DataSource before every `/readyz` check, so a long-lived one never answers with the first check's facts (it already did that for queues). Refs #492.
+
+- **A failed event handler now leaves a trace, like a failed job.** `Process::Consumer` only logged a handler error, tripped the circuit breaker and recorded a stat, so `pgbus_failed_events` never heard of it: the Jobs page showed a failed-and-waiting event as Ready ("lease expired, worker likely died") or Running, no page could say which attempt failed with what error, and a dead-lettered event's DLQ block had no error. The consumer now records the failure (logical queue name, `retry_count` = attempt − 1), clears it when a redelivery succeeds (only when `read_ct > 1`, so a first-delivery success adds no query, which a unit spec pins; `WP_BENCH_ROLE=consumer rake bench:worker_profile`, main vs branch interleaved three times on the same machine under load average ~17, gave 1,120–4,932 vs 2,120–3,197 jobs/s: no difference measurable through that noise), and clears it after the dead-letter move (read the error, move, then clear). The dashboard's Discard, Mark Handled, Edit & Retry and Reroute clear an event's failed row with its message. Refs #494.
+
+- **Replay of a processed event works.** `DataSource#replay_event` published `event["payload"] || "{}"` to a topic named after the handler class (`pgbus_processed_events` has neither a payload nor a routing key), and no page offered it. It now finds the event's archived envelope in its handler queue's archive and re-delivers it to that handler only (not to every subscriber of the routing key), under a new `event_id` with `replayed_from` set to the original, so the idempotency claim is new and the handler runs again. The dashboard offers Replay only when the handler is still registered and the archive still holds the event (`archive_retention`), and otherwise says which of the two is missing. Refs #494.
+
+- **fix(web): the recurring task toggle buttons had no dark-mode colours, and the pager's disabled Previous/Next failed contrast.** The Enable/Disable buttons built their class in a Ruby string concatenation that the static contrast guard could not read, so `bg-yellow-100`/`bg-green-100` glowed on dark cards. They now use the button helper, and the guard reads class strings built from `"…" \ "…"` continuations and `#{…}` ternaries. The shared pager's disabled span was gray-400 on gray-100 (axe colour-contrast); it is now gray-600 / dark gray-400 and marked `aria-disabled`. Refs #496.
+
+- **Dashboard contrast and markup fixes (WCAG 2.1 AA, both themes).** The Queues list's Pause/Resume/Purge/Delete buttons are now 24px tall (WCAG 2.2 target size), and the Insights time-range switcher is a `div role="group"` rather than a `nav` with an invalid role. The queue detail Pause button and the Events Reroute button were white on yellow (1.9:1) and now carry dark text on `yellow-400`; process metadata chips were grey on grey in both themes; the batches "failed" count (`red-500`), the "tables need vacuum" note (`amber-500`), the "view tasks" link (`blue-500`), the queue list's Resume/Pause links (`green-600`/`yellow-600`), amber/green table cells and link hover colours (`indigo-500`) were below 4.5:1 on white and move to `-700`/`-800`; the indigo job-state badge and the Jobs list's row actions were below 4.5:1 in dark mode and move to `-300`. The Outbox and Insights stat cards now wrap their `<dt>`/`<dd>` in a `<dl>`, and the Insights section headings are `<h2>` under the page `<h1>` instead of skipping a level. No copy changes. Refs #498.
+
 - **A still-running event handler is no longer re-run as if it had crashed, and a slow one is no longer redelivered mid-run.** Two halves of one hole, follow-up to #469. First: `Process::Consumer` had no equivalent of `ActiveJob::Executor#with_visibility_heartbeat`, so an event message's visibility timeout was never re-armed — a handler slower than `visibility_timeout` (30s by default) was redelivered *while it was still running*, `read_ct` climbed on every redelivery, and after `max_retries` the event was dead-lettered without the handler ever raising. Workers have been protected from this since the heartbeat landed; event consumers never were. The consumer now tracks each message through `VisibilityHeartbeat` for exactly as long as its handlers run, releasing the entry before the archive so a beat can never re-arm a message that is already gone. Second: `Handler#claim_idempotency?` read a `pgbus_processed_events` row with `completed_at IS NULL` as proof that the previous holder had been killed mid-handler, and re-ran. That state equally describes a handler that is simply still running — on another thread, another fork or another host — so a second delivery (a genuine redelivery, or a duplicate envelope, whose independent visibility timeouts no heartbeat can serialize) executed the handler *concurrently with* the live one, which is precisely the double-execution `idempotent!` exists to prevent. The claim now carries liveness rather than only a claim instant: `EventBus::ClaimBeat` refreshes every in-flight claim's `processed_at` from the same beat that re-arms the message's visibility timeout, so message visibility and claim liveness go quiet together when a process dies. A pending claim silent for longer than twice the heartbeat interval is abandoned and re-runs, exactly as before; a fresher one is owned and the delivery skips, deferring to the holder — which either completes (nothing is lost) or fails, leaving its own message for visibility-timeout redelivery to recover. A skip is also no longer silent: `pgbus.event_skipped` carries the reason (`:completed`, `:cached` or `:owned`), the claim's age in seconds and the delivery's `read_ct`, and the metrics subscriber counts it as `pgbus_event_count` with `status: "skipped"` and the reason as a tag. No migration, no new column, and no change on a table that has not run `pgbus:add_processed_event_completion` — a single-phase claim has no pending state and therefore no ownership question. Closes #470.
 
 - **An event is no longer handed to every handler whose pattern matches it, so a wildcard handler ran once per subscriber on the topic.** Each subscriber gets its own queue (`Subscriber#setup!` creates and binds one), so a topic with N matching subscribers puts N copies of every event on the bus — but `Consumer#handle_message` resolved handlers with `Registry#handlers_for(routing_key)`, a pattern match that ignored which queue the message had just been read from, and ran every match. Each of the N copies therefore fanned out to all N handlers: **every handler ran N times per event**, with four `"#"` subscribers meaning four invocations of each handler for every event on the bus. `idempotent!` was the only thing hiding it, and it hides it imperfectly by design: the two-phase claim (#385) deliberately *re-runs* when the existing claim is still pending, on the theory that the prior holder crashed — but a pending claim is also exactly what a handler still running on another host looks like. So one event could have host A win the claim and start a 700ms handler while host B, reading a different subscriber's copy of the same event a few hundred milliseconds later, dispatched to that same handler, lost the claim insert, read `completed_at IS NULL` as "crashed, re-run", and executed it concurrently. In one host app that meant a "task completed" record written twice and the user emailed twice, with a single row in `pgbus_processed_events`, `read_ct = 1` on every archived copy, and the second execution's side effects timestamped inside the first's window — dispatch, not redelivery. A non-idempotent handler simply ran N times with no guard at all. `handlers_for` now takes a required `queue_name:` and selects on ownership *and* pattern, so a message read from queue Q goes to Q's owner(s) alone and each handler runs exactly once per event; duplicate execution again requires a real redelivery, which is the at-least-once contract the docs describe. The keyword is required rather than defaulted precisely so a keyword-less call cannot silently restore the fan-out — the two callers that genuinely want the pattern view (the `Testing.inline!` publish path and `Testing::EventStore#drain!`, neither of which has a queue, since the event never reaches PGMQ) moved to an explicitly named `Registry#subscribers_matching`, where per-subscriber delivery counts already match what owner-only dispatch now produces in production. Two handlers registered against the same explicit `queue_name:` still both run on that queue's delivery. The pattern check is kept alongside the ownership check because a routing key the owner's pattern does not match means a stale `pgmq.topic_bindings` row, and a stale binding must not run the handler. A message on a queue no subscriber in this process owns is still archived rather than looped through visibility-timeout redelivery into the DLQ — that was already the behavior when `handlers_for` returned `[]` — but it is no longer silent: it logs a warning naming the queue and routing key, rate-limited to once per queue per process so a permanently stale binding cannot flood the log, and emits `pgbus.event_unrouted` on every occurrence so the real rate stays visible in metrics. The pending-claim re-run in `Handler#claim_idempotency?` is deliberately untouched here; it is a separate question now that dispatch no longer manufactures the concurrency it was misreading. Closes #469.

data/README.md

@@ -26,6 +26,7 @@ PostgreSQL-native job processing and event bus for Rails, built on [PGMQ](https:
   - [Client-level circuit breaker (database-down)](#client-level-circuit-breaker-database-down)
   - [Read timeouts (libpq-native)](#read-timeouts-libpq-native)
   - [Prefetch flow control](#prefetch-flow-control)
+  - [Read-ahead](#read-ahead)
   - [Worker recycling](#worker-recycling)
   - [Retry backoff](#retry-backoff)
 - [Routing and ordering](#routing-and-ordering)
@@ -541,6 +542,24 @@ end
 
 The worker tracks in-flight messages with an atomic counter and only fetches `min(idle_threads, prefetch_available)` messages per cycle. The counter is decremented in an `ensure` block so it never gets stuck.
 
+### Read-ahead
+
+By default a worker reads as many messages as it has free threads. Once the pool is busy, threads free one at a time, so the worker ends up making one read round trip per job. That costs nothing when Postgres is on the same host. When Postgres is on another host, that round trip becomes the throughput ceiling. `read_ahead` lets a worker or event consumer claim up to N messages beyond its free threads and hold them until a thread frees:
+
+```ruby
+Pgbus.configure do |config|
+  config.read_ahead = 12                                   # global; 0 = off (default)
+  config.capsule :api, queues: %w[api], threads: 12, read_ahead: 12   # per capsule
+  config.event_consumers = [{ topics: ["orders.#"], threads: 8, read_ahead: 8 }]
+end
+```
+
+- **When to turn it on:** when Postgres is not on the same host as the worker. Start with `read_ahead` equal to `threads`. With a local database it buys little, because the worker is CPU-bound there. Under +1 ms of latency, `read_ahead` 12 took a 12-thread worker from ~1.3–1.7k to ~2.3–2.5k no-op jobs/s (`docs/performance.md`, "Read-ahead"). The next ceiling is the connection pool, so raise `pool_size` with it.
+- **Visibility:** buffered messages are kept invisible by the visibility heartbeat until their job starts, so a long wait never lets another worker claim them. With `visibility_heartbeat = false` nothing holds them, and a message buffered past `visibility_timeout` can run twice; `validate!` warns about that combination.
+- **Shutdown and recycle:** when a worker drains, recycles, pauses or shuts down, buffered messages go straight back to the queue (`set_vt` 0), not after `visibility_timeout`. Their `read_ct` has already been incremented, the same as after a crash, so with `zombie_detection` on, the worker that picks one up logs it as a zombie redelivery.
+- **With `prefetch_limit`:** buffered messages count as in flight, so `prefetch_limit` still caps everything a worker has claimed.
+- **Multi-queue workers:** the plain multi-queue read (no priority, fair share or group mode) can claim and discard rows past its limit. A larger read makes that worse; see `docs/performance.md`.
+
 ### Worker recycling
 
 Pgbus workers recycle themselves to prevent memory bloat. This is the main reliability difference vs. solid_queue, which leaves workers alive forever.
@@ -555,6 +574,41 @@ end
 
 When a limit is hit, the worker drains its thread pool, exits, and the supervisor forks a fresh process. RSS memory is sampled from `/proc/self/statm` (Linux) or `ps -o rss` (macOS).
 
+A capsule can set its own limits. They apply to that capsule's workers only, and any limit it leaves unset (or sets to `nil`) falls back to the global value, so a capsule can raise or lower a limit but not switch it off while the global one is set. One heavyweight capsule no longer forces a high global limit on the lightweight ones:
+
+```ruby
+Pgbus.configure do |config|
+  config.max_memory_mb = 512                      # every other capsule
+  config.capsule :render, queues: %w[render], threads: 1, processes: 4,
+                 max_memory_mb: 1_536, max_worker_lifetime: 6.hours
+end
+```
+
+### Threads vs processes
+
+A capsule is one forked process with `threads:` threads by default. Ruby threads in one process share the GVL, so threads help when jobs wait on I/O (HTTP, mail, the database) and do not help when jobs burn CPU (template rendering, report or PDF generation, image work, large serialization): a capsule with `threads: 8` running CPU-bound jobs still uses about one core.
+
+`processes: N` forks N identical workers for one capsule. Each is its own process with its own GVL, so N processes use up to N cores:
+
+```ruby
+config.capsule :render, queues: %w[render], threads: 1, processes: 4
+```
+
+| Jobs are mostly… | Use |
+|---|---|
+| Waiting on I/O | `threads:` (or `execution_mode: :async`), `processes: 1` |
+| Burning CPU | `processes:` up to the cores you can give the capsule, `threads: 1`–`2` |
+
+- **One queue, N processes.** All N workers read the same queues with `FOR UPDATE SKIP LOCKED`; there is no sharding, so enqueue-time uniqueness and concurrency controls behave as with one process. The queue-overlap rule for named capsules is unchanged: the forks belong to one capsule.
+- **Each fork is supervised on its own.** It has its own restart backoff, liveness pipe, wake pipe and `pgbus_processes` row. A crashed fork restarts alone; its siblings keep running. The boot banner prints `processes=N`, readiness counts every fork, and the dashboard's Processes page labels each row `capsule render · process 2/4`.
+- **Memory.** Budget roughly `boot RSS + peak job RSS` per process. The forks share the booted app's memory copy-on-write until they write to it, so N forks cost less than N separate supervisors, which each boot the app.
+- **Connections.** Every process has its own pool (`pool=` in the boot banner), so a host holds about `processes × pool_size` connections for the capsule. Under the default `worker_notify_scope: :supervisor` the host still holds one LISTEN connection; under `:fork` each fork holds its own, and `pgbus doctor`'s connection budget counts them.
+- **Shutdown.** The stop timeout applies to every fork at once, so N forks drain in parallel and need no longer `shutdown_timeout`.
+- **With `single_active_consumer: true`** each queue is locked by one fork at a time; a fork can still process the capsule's other queues. For a single-queue capsule that makes the extra forks hot standbys: use `processes:` there for failover, not throughput.
+- `processes:` must be a positive Integer (default `1`, so existing configs are unchanged). It is set on `c.capsule` or on an Array-form `workers` entry; the string DSL has no syntax for it.
+
+The three most common reasons to reach for `processes:` or per-capsule limits: CPU-bound rendering (PDFs, reports, templates) that one GVL serializes; one memory-hungry capsule among light ones, which gets its own `max_memory_mb:` instead of a separate container with a raised limit; and the same capsule string listed twice to get two processes, which becomes one named capsule with `processes: 2`. Use cases, a decision table and the memory and connection math for a job host: [Sizing capsules](https://pgbus.zoolutions.llc/docs/sizing-capsules).
+
 ### Retry backoff
 
 When a job fails, Pgbus extends the PGMQ visibility timeout with exponential backoff so retries are spread out instead of bunched at fixed intervals:
@@ -1157,9 +1211,9 @@ All tools are read-only — no tool mutates state, and there is no raw-SQL passt
 | `pgbus_processes` | Every process with kind, pid, heartbeat age, and `healthy`/`stale`/`stalled` status. |
 | `pgbus_jobs` / `pgbus_job_detail` | Inspect enqueued messages (`read_ct`, `vt`, `enqueued_at`). Paginated. |
 | `pgbus_dlq` / `pgbus_dlq_detail` | Dead-letter inspection. Paginated. |
-| `pgbus_locks` | Active uniqueness locks (the leaked-lock diagnostic). |
+| `pgbus_locks` | Active uniqueness locks (the leaked-lock diagnostic). Paginated (`page`, `per_page` max 100; returns `total`, capped at 10,000, and `has_more`). |
 | `pgbus_throughput` / `pgbus_stats` | Recent throughput time series and status counts. |
-| `pgbus_recurring` | Recurring task schedule + last/next run times. |
+| `pgbus_recurring` | Recurring task schedule + last/next run times. All tasks unless `page` / `per_page` is given (then `total` is capped at 10,000); returns `total`, `has_more`. |
 
 #### Security
 
@@ -1167,7 +1221,7 @@ The server is built to be safe against a production datastore:
 
 - **Read-only by default.** No tool mutates state and no arbitrary-query tool exists.
 - **Payloads redacted.** Message bodies, headers, and job arguments are replaced with `[redacted]` unless payloads are explicitly allowed **and** `include_payloads: true` is passed on the call. Both gates must be open. Allow payloads with `PGBUS_MCP_ALLOW_PAYLOADS=1` (stdio) or `Pgbus::MCP.rack_app(allow_payloads: true)` (HTTP).
-- **Bounded queries.** Every list tool paginates with a row cap (`pgbus_jobs` / `pgbus_dlq` cap at 100 rows/page; time windows cap at 1440 minutes).
+- **Bounded queries.** Every list tool paginates with a row cap (`pgbus_jobs` / `pgbus_dlq` / `pgbus_locks` / `pgbus_concurrency` cap at 100 rows/page; time windows cap at 1440 minutes). The one exception is `pgbus_recurring` called without `page`/`per_page`, which returns every task as it always has.
 - **Reuses your DB credentials.** No new privileged path — it reads through the app's existing connection config.
 - **Authentication.**
   - *HTTP:* set `token:` (clients send `Authorization: Bearer <token>`) or a custom `auth:` callable; unauthenticated requests get `401`.
@@ -1321,11 +1375,11 @@ If the orchestrator's stop grace period is *shorter* than `shutdown_timeout`, do
 [Pgbus] boot: pgmq_schema_mode=auto pgmq_version=1.4.0
 [Pgbus] boot: listen_notify=true worker_notify_wakeup=true
 [Pgbus] boot: roles=workers,dispatcher,scheduler
-[Pgbus] boot: capsule=critical queues=critical threads=5 mode=threads
-[Pgbus] boot: capsule=default queues=default,mailers threads=10 mode=threads
+[Pgbus] boot: capsule=critical queues=critical threads=5 processes=1 mode=threads
+[Pgbus] boot: capsule=default queues=default,mailers threads=10 processes=1 mode=threads
 ```
 
-It states: the pgbus version; the connection target reduced to `host/dbname` (never the password) across all three `connection_options` forms (`database_url` string, `connection_params` hash, AR-derived hash); the resolved pool size; `pgmq_schema_mode` and the installed PGMQ version (best-effort — `unknown` on any error); `listen_notify` and `worker_notify_wakeup?`; the roles that will actually boot (honoring `config.roles`); and one line per worker capsule (name, queues, threads, execution mode) and per event consumer (topics, threads).
+It states: the pgbus version; the connection target reduced to `host/dbname` (never the password) across all three `connection_options` forms (`database_url` string, `connection_params` hash, AR-derived hash); the resolved pool size; `pgmq_schema_mode` and the installed PGMQ version (best-effort — `unknown` on any error); `listen_notify` and `worker_notify_wakeup?`; the roles that will actually boot (honoring `config.roles`); and one line per worker capsule (name, queues, threads, processes, execution mode) and per event consumer (topics, threads).
 
 Every DB-dependent field is wrapped so a transient failure degrades that field to `unknown` — the banner can never abort boot.
 
@@ -2031,11 +2085,12 @@ pgbus help      # Show help
 
 #### pgbus doctor
 
-A single preflight command that answers "is this environment healthy enough to run?" — useful as a deploy or CI gate. It runs 11 checks and never raises; a broken environment turns every probe into a failed/warned check instead of a crash:
+A single preflight command that answers "is this environment healthy enough to run?" — useful as a deploy or CI gate. It runs 12 checks and never raises; a broken environment turns every probe into a failed/warned check instead of a crash:
 
 | Check | Fails (`:fail`) when | Warns (`:warn`) when |
 |---|---|---|
 | Configuration | `Configuration#validate!` raises | — |
+| Ruby JIT | — | YJIT is available but off outside a local environment (a CPU-bound worker with Postgres nearby runs ~15% fewer jobs/s without it; no measurable effect when network-bound); reports only, never enables |
 | Database | Unreachable (`SELECT 1` via `Client#ping`) | — |
 | PGMQ schema | Schema not installed | Installed but untracked, or behind the vendored version |
 | Queues | A configured queue has no PGMQ table | — |
@@ -2117,17 +2172,17 @@ The dashboard is a mountable Rails engine at `/pgbus` with:
 
 - **Overview** — queue depths, enqueued count, active processes, failure count, throughput rate
 - **Queues** — per-queue metrics, purge/pause/resume/delete actions
-- **Jobs** — enqueued and failed jobs, retry/discard actions
+- **Jobs** — one list of every job with its state (Ready / Scheduled / Running / Retrying / Blocked), why it is waiting and when it will run, filter tabs with counts, paging, retry/discard actions
 - **Dead letter** — DLQ messages with retry/discard, bulk actions
-- **Processes** — active workers/dispatcher/consumers with heartbeat status **and per-worker throughput** (e.g. `12.4/s processed · 0.2/s failed`)
-- **Events** — registered subscribers and processed events
+- **Processes** — every process with its status in words: why it is Healthy, Stalled or Stale (against which limit) and what happens next (the supervisor's watchdog restarts it, the dispatcher removes it within 5 minutes, or nothing will); what it is doing (queues or topics, busy threads, buffered claims, jobs/s); recycle progress (`1,240 / 10,000 jobs · 310 / 512 MB · up 42m 0s / 1h 0m`); uptime; and the supervisor that owns it
+- **Events** — one list of pending events with their state (Ready / Scheduled / Handling / Retrying), the handler that will run each one and why it is waiting (jobs ahead, queue paused, no running consumer subscribes to its pattern), the failed attempt and its error, tabs, paging and the existing actions; the processed-events audit says whether each claim completed, is being handled or went silent, with Replay for completed claims still in the archive; subscribers say whether a running consumer drains them
 - **Outbox** — transactional outbox entries pending publication
 - **Locks** — active job uniqueness locks with state (queued/executing), owner PID@hostname, age
 - **Insights** — throughput chart (jobs/min), status distribution donut, slowest job classes table
 
 All tables use Turbo Frames for periodic auto-refresh without page reloads. Destructive actions use styled confirmation dialogs (not browser `confirm()`), and flash messages appear as auto-dismissing toast notifications.
 
-Each worker snapshots its live in-process rate counter (dequeued/processed/failed) into its heartbeat metadata on every beat, so the Processes panel shows a cluster-wide, near-real-time throughput view with no extra query. Zero rates are omitted from the rendered string. Non-worker processes (dispatcher, scheduler, consumers) carry no rate data and render only their static boot metadata, unchanged.
+Every process registers `started_at`, and processes the supervisor forks register `supervisor_pid` (its pid). Workers and consumers refresh the pool's `capacity`/`busy`, `buffered`, `memory_mb`, their processed/handled counts and their recycle limits (`max_jobs`, `max_memory_mb`, `max_lifetime`; a worker uses its capsule's own limits when set) once per 60 s beat, never per job. Workers also report live rates; consumers do not. A process from before these keys existed shows "—" where a fact is missing. A row is grouped under a supervisor only when that supervisor's row is on the same host. The stall limits the page quotes are derived from the dashboard host's `stall_threshold`: workers get it as is, dispatchers and schedulers add their sleep interval, consumers add their polling allowance. `pgbus_worker_pool_capacity` and `pgbus_worker_pool_busy` (`/pgbus/api/metrics`) read the same `capacity`/`busy` keys.
 
 #### Queue management
 
@@ -2256,9 +2311,9 @@ Curated headline options for the README. The full operator reference (with types
 | `retry_backoff` | `5` | Base delay in seconds for VT-based retry backoff (exponential: `base * 2^(attempt-1)`) |
 | `retry_backoff_max` | `300` | Maximum retry delay in seconds (caps the exponential curve) |
 | `retry_backoff_jitter` | `0.15` | Jitter factor (0-1) added to retry delays to spread retries |
-| `max_jobs_per_worker` | `nil` | Recycle worker after N jobs (nil = unlimited) |
-| `max_memory_mb` | `nil` | Recycle worker when memory exceeds N MB |
-| `max_worker_lifetime` | `nil` | Recycle worker after N seconds. Accepts seconds or Duration. |
+| `max_jobs_per_worker` | `nil` | Recycle worker after N jobs (nil = unlimited). Overridable per capsule. |
+| `max_memory_mb` | `nil` | Recycle worker when memory exceeds N MB. Overridable per capsule. |
+| `max_worker_lifetime` | `nil` | Recycle worker after N seconds. Accepts seconds or Duration. Overridable per capsule. |
 | `listen_notify` | `true` | Use PGMQ's LISTEN/NOTIFY for instant wake-up |
 | `worker_notify_wakeup` | `nil` (follows `listen_notify`) | Dedicated LISTEN connection for worker wake-up. `nil` follows `listen_notify`; set `false` to force polling. |
 | `worker_notify_host` / `worker_notify_port` / `worker_notify_database_url` | `nil` | Override host/port/URL for the worker notify LISTEN connection (e.g. direct primary port when jobs go through PgBouncer) |

data/Rakefile

@@ -25,7 +25,8 @@ namespace :bench do
   # no-DB unit suite that bench:all runs in CI.
   db_benches = %w[connection_pool_bench integration_bench streams_bench streams_read_pool_bench
                   execution_modes_bench pool_swap_bench pool_autoscale_bench job_burst_bench
-                  notify_wake_bench notify_chaos_bench streams_hub_bench fair_read_bench].freeze
+                  notify_wake_bench notify_chaos_bench streams_hub_bench fair_read_bench
+                  worker_profile_bench].freeze
   # The unit suite is every *_bench.rb that doesn't need a database, derived
   # from the directory so a new unit bench is picked up automatically (kept in
   # sync with bench:one, which globs the same files).
@@ -60,6 +61,11 @@ namespace :bench do
     ruby "benchmarks/integration_bench.rb"
   end
 
+  desc "Run worker profiling bench: jobs/s and where a job's time goes (requires PGBUS_DATABASE_URL)"
+  task :worker_profile do
+    ruby "benchmarks/worker_profile_bench.rb"
+  end
+
   desc "Run fair share read benchmark (requires PGBUS_DATABASE_URL)"
   task :fair_read do
     ruby "benchmarks/fair_read_bench.rb"

data/app/controllers/concerns/pgbus/job_listing.rb

@@ -0,0 +1,37 @@
+# frozen_string_literal: true
+
+module Pgbus
+  # Loads the unified Jobs list (issue #489) for any page that shows it: the
+  # Jobs page across every queue, a queue page for one physical queue (#491).
+  module JobListing
+    # Tabs of the unified list; "all" is no filter.
+    STATES = ["all", *Web::JobState::STATES].freeze
+
+    private
+
+    # list_path: ->(extra) { url } for the tabs, the pager and the frame's
+    # auto-refresh source, so each page keeps them on its own URL. extra is
+    # symbol-keyed, never request.query_parameters (string keys would repeat
+    # a param in the URL).
+    #
+    # exclude: physical queues to leave out (the Jobs page leaves the EventBus
+    # handler queues to the Events page, issue #494).
+    def load_job_list(queue_name:, list_path:, exclude: nil)
+      @state = job_state_param
+      @page = page_param
+      @per_page = per_page
+      @rows = data_source.job_rows(state: (@state unless @state == "all"), queue_name: queue_name, exclude: exclude,
+                                   page: @page, per_page: @per_page)
+      @counts = data_source.job_state_counts(queue_name: queue_name, exclude: exclude)
+      @ahead = data_source.jobs_ahead(@rows)
+      @state_context = data_source.job_list_context
+      @list_scoped = !queue_name.nil?
+      @list_path = ->(extra) { list_path.call({ state: (@state unless @state == "all") }.merge(extra)) }
+    end
+
+    def job_state_param
+      state = params[:state].presence || ("retrying" if params[:status] == "failed")
+      STATES.include?(state) ? state : "all"
+    end
+  end
+end

data/app/controllers/concerns/pgbus/process_listing.rb

@@ -0,0 +1,17 @@
+# frozen_string_literal: true
+
+module Pgbus
+  # Loads the process rows and their words (issue #493) for the Processes
+  # page and the home page's panel, so both order and describe them alike.
+  module ProcessListing
+    private
+
+    def load_processes
+      thresholds = data_source.process_thresholds
+      rows = data_source.processes(thresholds: thresholds)
+      context = data_source.process_context(rows: rows, thresholds: thresholds)
+      @processes = Web::ProcessState.order(rows, context)
+      @process_states = @processes.to_h { |p| [p[:id], Web::ProcessState.present(p, context)] }
+    end
+  end
+end

data/app/controllers/pgbus/application_controller.rb

@@ -17,6 +17,7 @@ module Pgbus
     layout "pgbus/application"
 
     helper Pgbus::ApplicationHelper unless self < Pgbus::ApplicationHelper
+    helper Pgbus::ButtonHelper unless self < Pgbus::ButtonHelper
 
     # Make `pgbus` route proxy available in views (e.g. pgbus.root_path).
     # With isolate_namespace, the non-prefixed helpers (root_path) work inside
@@ -75,8 +76,10 @@ module Pgbus
       @data_source ||= Pgbus.configuration.web_data_source || Web::DataSource.new
     end
 
-    def page_param
-      [params[:page].to_i, 1].max
+    # key – the query param; a second list on one page uses <list>_page.
+    def page_param(key = :page)
+      value = params[key]
+      [value.is_a?(String) ? value.to_i : 1, 1].max
     end
 
     def per_page

data/app/controllers/pgbus/batches_controller.rb

@@ -3,7 +3,10 @@
 module Pgbus
   class BatchesController < ApplicationController
     def index
-      @batches = data_source.batches
+      @page = page_param
+      @per_page = per_page
+      @batches = data_source.batches(page: @page, per_page: @per_page)
+      @batches_count = data_source.list_count(:batches)
       render_frame("pgbus/batches/batches_table") if params[:frame] == "list"
     end
 

data/app/controllers/pgbus/dashboard_controller.rb

@@ -2,16 +2,18 @@
 
 module Pgbus
   class DashboardController < ApplicationController
+    include ProcessListing
+
     def show
       case params[:frame]
       when "stats"
-        @stats = data_source.summary_stats
+        load_stats
         render_frame("pgbus/dashboard/stats_cards")
       when "queues"
-        @queues = data_source.queues_with_metrics
+        load_queues
         render_frame("pgbus/dashboard/queues_table")
       when "processes"
-        @processes = data_source.processes
+        load_processes
         render_frame("pgbus/dashboard/processes_table")
       when "failures"
         @recent_failures = data_source.failed_events(page: 1, per_page: 5)
@@ -20,12 +22,24 @@ module Pgbus
         @health = data_source.queue_health_stats
         render_frame("pgbus/dashboard/queue_health")
       else
-        @stats = data_source.summary_stats
-        @queues = data_source.queues_with_metrics
-        @processes = data_source.processes
+        load_stats
+        load_queues
+        load_processes
         @recent_failures = data_source.failed_events(page: 1, per_page: 5)
         @health = data_source.queue_health_stats
       end
     end
+
+    private
+
+    def load_stats
+      @stats = data_source.summary_stats
+      @signals = data_source.dashboard_signals
+    end
+
+    def load_queues
+      @queues = data_source.queues_with_metrics
+      @queue_lines = data_source.queue_status_lines
+    end
   end
 end

data/app/controllers/pgbus/dead_letter_controller.rb

@@ -2,17 +2,28 @@
 
 module Pgbus
   class DeadLetterController < ApplicationController
+    ERROR_CLASS_MAX = 200
+
     def index
       @page = page_param
       @per_page = per_page
-      @messages = data_source.dlq_messages(page: @page, per_page: @per_page)
-      @total_count = data_source.dlq_total_count
+      @dlq = dlq_param
+      @error_class = error_class_param
+      filters = { dlq: @dlq, error_class: @error_class }
+      @messages = data_source.dlq_messages(page: @page, per_page: @per_page, **filters)
+      @total_count = data_source.dlq_total_count(**filters)
       @total_pages = (@total_count.to_f / @per_page).ceil
+      @dlq_counts = data_source.dlq_counts_by_queue
+      @dlq_rows = @messages.map { |m| [m, Pgbus::Web::DeadLetterReason.present(m)] }
+      # One place builds every list URL (frame source, pager, chips), so the
+      # filters survive refresh and paging.
+      @list_path = ->(extra = {}) { pgbus.dead_letter_index_path(filters.merge(extra).compact) }
       render_frame("pgbus/dead_letter/messages_table") if params[:frame] == "list"
     end
 
     def show
-      @message = data_source.dlq_message_detail(params[:id].to_i)
+      @message = data_source.dlq_message_detail(params[:id].to_i, queue_name: dlq_param(:queue_name))
+      @reason = Pgbus::Web::DeadLetterReason.present(@message) if @message
     end
 
     def retry
@@ -63,5 +74,18 @@ module Pgbus
       end
       redirect_to dead_letter_index_path, notice: t("pgbus.dead_letter.index.discarded_selected", count: count)
     end
+
+    private
+
+    # A full DLQ name or nil; the data source also checks it is a known DLQ.
+    def dlq_param(key = :dlq)
+      name = params[key].to_s
+      name if name.end_with?(Pgbus::DEAD_LETTER_SUFFIX) && name.match?(/\A\w+\z/)
+    end
+
+    def error_class_param
+      value = params[:error_class].to_s
+      value if value.present? && value.length <= ERROR_CLASS_MAX
+    end
   end
 end

data/app/controllers/pgbus/events_controller.rb

@@ -2,14 +2,35 @@
 
 module Pgbus
   class EventsController < ApplicationController
+    # Tabs of the pending list; "all" is no filter.
+    STATES = ["all", *Web::EventState::STATES].freeze
+
+    # One pending-events list (the job list scoped to the handler queues) and
+    # the processed-events audit, each in its own auto-refreshing frame with
+    # its own pager (page / processed_page), issue #494.
     def index
-      @events = data_source.processed_events(page: page_param, per_page: per_page)
-      @subscribers = data_source.registered_subscribers
-      @pending = data_source.pending_events(page: page_param, per_page: per_page)
+      @state = STATES.include?(params[:state]) ? params[:state] : "all"
+      @queue = params[:queue].presence
+      @page = page_param
+      @processed_page = page_param(:processed_page)
+      @per_page = per_page
+      if params[:frame] == "processed"
+        load_processed_events
+        return render_frame("pgbus/events/processed_table")
+      end
+
+      load_pending_events
+      return render_frame("pgbus/events/list") if params[:frame] == "list"
+
+      load_processed_events
     end
 
     def show
       @event = data_source.processed_event(params[:id])
+      return unless @event
+
+      @claim = Web::EventState.processed(@event, now: Time.now, claim_window: EventBus::Handler.claim_ownership_window)
+      @replay_state = data_source.event_replay_states([@event])[@event["id"]]
     end
 
     def replay
@@ -53,6 +74,10 @@ module Pgbus
       return reject_unknown_queue(:payload_update_failed) unless registered_queue?(queue_name)
 
       new_payload = params[:payload].to_s
+      # The editor shows the payload through PayloadFilter: saving a marker
+      # back would overwrite the redacted secret for good.
+      return redirect_to(events_path, alert: t("pgbus.events.flash.payload_redacted")) if new_payload.include?(Web::PayloadFilter::FILTERED)
+
       if data_source.edit_event_payload(queue_name, params[:id], new_payload)
         redirect_to events_path, notice: t("pgbus.events.flash.payload_updated")
       else
@@ -104,6 +129,30 @@ module Pgbus
 
     private
 
+    def load_pending_events
+      @rows = data_source.event_rows(state: (@state unless @state == "all"), queue_name: @queue,
+                                     page: @page, per_page: @per_page)
+      @counts = data_source.event_state_counts(queue_name: @queue)
+      @ahead = data_source.events_ahead(@rows)
+      @state_context = data_source.event_list_context
+      @subscribers = data_source.registered_subscribers
+    end
+
+    def load_processed_events
+      @events = data_source.processed_events(page: @processed_page, per_page: @per_page)
+      @processed_count = data_source.list_count(:processed_events)
+      @replay_states = data_source.event_replay_states(@events)
+      @claim_window = EventBus::Handler.claim_ownership_window
+    end
+
+    # Every list link keeps the other list's page, so paging one frame never
+    # resets the other.
+    def events_list_path(extra = {})
+      events_path({ state: (@state unless @state == "all"), queue: @queue, page: (@page if @page > 1),
+                    processed_page: (@processed_page if @processed_page > 1) }.merge(extra).compact)
+    end
+    helper_method :events_list_path
+
     def registered_queues
       @registered_queues ||= data_source.handler_queue_physical_names
     end