pgbus 0.15.3 → 0.16.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 9ccee33eff147c5f62d51634db10d7d2529806a9b84d3373c8e66c4b93dfe68d
-  data.tar.gz: b3ebfda680231e62bc65a92822af22bcbb33a6666bfa9997be15387dcf2100cf
+  metadata.gz: 2043a2be7395c42d40398ebbb862de8c3e142dcb17967980b069ac8e23313fb2
+  data.tar.gz: 850d0b62d0fe06528f166e15b136efb11981ef5bbbeeae95e9f5a37ef282f7e1
 SHA512:
-  metadata.gz: 76e267110b0064e3b257fa10d0dab5a05f9705b6dabc5b86884cf3655df5125498cc5957c589662d04603653d40b516c5f8d073db9d1937829bc012979c91f00
-  data.tar.gz: ddb7653b85459ff790f48386932f9096950386cb52643eb146c3f3f6c3dd53a8383935df80b69efa124874f7018884c9211264c35aa656494c0e8ff760bf7bd2
+  metadata.gz: dab60674d617fd909bd54a1f4904eb0262611ac4e146390ac828decbb0e4145503a362a95c0f5b0c5e5ef565e17891a12aa1556ae7c28f3d6b336e7ddcbff5aa
+  data.tar.gz: 10787c60e18aa5211d29872f0da662d9a6b98a12f1f4a35c8f37b6115046cad0d16b53e04254c827833dd412024b3173cc38eeb50daa1e7e84b30c1edf935276

data/CHANGELOG.md

@@ -2,6 +2,8 @@
 
 ### Fixed
 
+- **The unbound-lock reaper no longer counts a dead-lettered copy as a live message, so a `:until_executed` job that once dead-lettered can be enqueued again.** `Client#uniqueness_keys_present` scanned every queue in `pgmq.meta` for the payload's `pgbus_uniqueness_key` — including `*_dlq` queues. A dead-lettered copy keeps the original payload (key included) but is not in flight: the executor already released the lock when it moved the message. For a legacy `pending`/`msg_id=0` row (pre-#418 enqueue, or a bind that failed) the DLQ copy made the key look "still here" forever, the reaper kept the row, and an `on_conflict: :discard` job — a heartbeat, typically — was discarded on every enqueue until someone purged the DLQ by hand. Seen in production as three heartbeats silently dead for nine days behind a 468-message DLQ. Dead-letter queues are now skipped in the scan; bound-lock probes (`message_exists?`) were never affected because they resolve the logical queue's physical tables only.
+- **`Pgbus::Testing.disabled!` can no longer turn a Capybara teardown race into a hung test process (issue #443).** The `streams_test_mode` stub closed immediately, so a page on a stream-bearing layout had its `EventSource` reconnect every ~3s for the whole example; a reconnect landing after `disabled!` had switched test mode off — a config-level `after` hook runs *before* `Capybara.reset_sessions!` — took the real path and started a live `Streamer` (listener/dispatcher/heartbeat threads + a LISTEN connection) inside the RSpec process. Its orphaned threads then shared the test's pinned AR connection and CI died at the job timeout with only `message type 0x5a arrived from server while idle`. Three changes: the stub now emits `retry: 86400000` so the browser does not reconnect at all; `Streamer::Instance#shutdown!` snapshots every component's threads (`#threads` on Listener/Dispatcher/Heartbeat/OutboundPump/HubClient/FailoverListener), logs one error naming those still alive after their bounded joins, and returns that list (`Streamer.reset!` forwards it, `nil` when nothing was live); and `Testing.disabled!` raises `Pgbus::Testing::StreamerLeakError` — pointing at `config.append_after` — when threads leaked, while a live streamer that shut down cleanly is only logged. README and docs now recommend `config.append_after` for the teardown hook.
 - **`Pgbus::MCP.rack_app` works on a real hostname again with `mcp` 0.23+ / 1.x.** Since mcp 0.23 the `StreamableHTTPTransport` validates the `Host` header (DNS-rebinding protection, on by default, loopback hosts only) and pgbus had no way to pass the transport's options through — so a gated mount at `https://app.example.com/pgbus/mcp` answered every request `403 "Invalid Host header"`, and consumers pinned `mcp < 1.0` to dodge it (which only helps while the lock stays on 0.22). The rack app now exposes `allowed_hosts:`, `allowed_origins:` and `dns_rebinding_protection:`; the check **follows the gate by default** — off when `token:`/`auth:` is configured (a rebound browser page can never carry the bearer secret, so the check is redundant there), on for the warned-about unauthenticated mount — and `true`/`false` forces it. `mcp >= 0.23` is the floor for `rack_app` (older gems raise `Pgbus::Error` naming the fix); the gem's own bundle now tracks `mcp` 1.x, so lift that `< 1.0` pin. Stdio (`pgbus mcp`) is unaffected.
 
 - **Worker forks no longer kill the supervisor's shared LISTEN connection (issue #437).** Under `worker_notify_scope = :supervisor` every fork produced one `[Pgbus::NotifyListener] connection error (PG::ConnectionBad: PQconsumeInput() server closed the connection unexpectedly …) — reconnecting` in the supervisor ~2 s after `Worker started`, and a LISTEN gap (polling fallback) until `reconnect!` completed. The child's fork hygiene closed the Ruby `IO` wrapper of the inherited LISTEN socket, but pg builds `socket_io` with `autoclose=false`, so the fd stayed open — and when the child's GC freed the inherited `PG::Connection`, its `PQfinish` sent a libpq Terminate down that fd, i.e. down the **parent's** connection. `NotifyListener#close_inherited_socket!` now repoints the fd at `/dev/null` (`socket_io.reopen(IO::NULL)`, the ActiveRecord `PostgreSQLAdapter#discard!` idiom) so the eventual `PQfinish` is harmless. Regression-covered by a real-fork integration spec that asserts the parent's `pgbus-listen` backend pid is unchanged and no reconnect is logged. Refs #437.
@@ -18,6 +20,8 @@
 - **Fair share scheduling for event-bus consumers (issue #427).** Follow-up to #426: a subscriber queue is FIFO, so a bulk import emitting `orders.created` 100 000 times for one tenant put every other tenant's events behind it in every handler subscribed to that topic. New `config.event_fair_share = ->(event) { key | [key, weight] | nil }` — the event twin of `fair_share` — receives the `Pgbus::Event` (routing key, the payload object as passed to publish, headers) at publish time on every path (`Pgbus.publish`, `publish_later`, `Pgbus::Outbox.publish_event`) and merges the same `pgbus_fair_key` / `pgbus_fair_weight` into the **event envelope** (a sibling of `event_id` / `published_at`, never inside the user payload, so `event.payload` is unchanged in handlers). Because the key lives at `message->>'pgbus_fair_key'` for events exactly as for jobs, #426's `read_batch_fair` SQL and `q_<queue>_fair_idx` index are reused verbatim; PGMQ's topic fan-out copies the tag to every bound subscriber queue, and the outbox (which stores the envelope) carries it across the relay with no extra work — a system writing `pgbus_outbox_entries` directly can set the key in the envelope JSON. `Consumer#fetch_messages` now fair-reads each active subscriber queue in list order with the remaining capacity (strict across queues, fair within; circuit-breaker-paused queues still skipped), ensures the fair index on its queues at boot, and `Subscriber#setup!` ensures it at queue creation. Independent of `fair_share` (enable either side or both); no interaction with `group_mode` (worker-only). Default-off is byte-identical. Refs #427.
 - **Fair share scheduling across tenants — weighted, work-conserving (issue #426).** A queue is FIFO, so one tenant enqueuing 100 000 jobs put every other tenant's work behind them; nothing in pgbus prevented it (multi-queue reads are strict list-order, `group_mode` serializes PGMQ FIFO groups and pgbus never set its header for jobs, `limits_concurrency` is a non-work-conserving cap). New `config.fair_share = ->(job) { key | [key, weight] | nil }` is evaluated at enqueue on every adapter path (`enqueue`, `enqueue_at`, `perform_all_later`); the key and weight ride inside the job payload as `pgbus_fair_key` / `pgbus_fair_weight` (same pattern as `pgbus_concurrency_key`, so they survive blocked-execution promotion, DLQ retry and dashboard retry). Workers then read with the new `Client#read_batch_fair` — one statement that enumerates the keys with visible messages via a loose index scan, ranks each key's oldest visible messages, and takes the `qty` lowest `rank / weight`: weight 3 vs 1 is a 3:1 split while both have work, a lone tenant still fills the whole batch, and the cost scales with the number of keys that have visible work, not backlog depth (`rake bench:fair_read`: 1.05 ms with 1 key, 4.7 ms with 200 keys, 100k-row backlog; numbers in `docs/performance.md`). Strict list-order priority across queues is preserved (fair within each queue); with `priority_levels` it is strict between levels and fair within a level; mutually exclusive with `group_mode` (`validate!` and `Worker.new` both raise). The supporting expression index `q_<queue>_fair_idx ((COALESCE(message->>'pgbus_fair_key','')), vt, msg_id)` is created at queue creation when the option is on, and built `CONCURRENTLY` by each worker for the queues it already serves (`Client#ensure_fair_index`, memoized, logs the `DROP INDEX` remediation if a concurrent build is interrupted). Within a key messages are taken oldest-visible first (`vt, msg_id`) — a deliberate deviation from `pgmq.read`'s pure `msg_id` order so a tenant's whole visible backlog is never sorted per read. Event-bus consumers get the same in #427 (below). Refs #426.
 
+- **Unique batches: a run-scoped lock held from `enqueue` until the batch finishes.** `ensures_uniqueness` guards one job, but a multi-job run — an import fanned out into chunk jobs, a pipeline whose jobs enqueue the next stage — has no single job that spans the run, so a lock on the entry job releases the moment that job finishes and the next scheduler tick starts a second run on top of the first. `Pgbus::Batch.new(uniqueness_key: "perfecta-daily", on_conflict: :reject | :discard | :log)` takes a row in `pgbus_uniqueness_keys` at `#enqueue` (lock_key `batch:<key>`, queue_name `batch:<batch_id>`), keeps it through every open-batch add, and releases it in the single-winner finish (`finish_if_needed`) that also fires the callbacks — so completion, the stalled-batch sweep and cleanup all release it. A second batch with the same key raises `Pgbus::Batch::AlreadyRunning`, or skips its block and answers `discarded?`. The dispatcher's reaper judges these rows by the owning batch's status (`Batch.lock_orphaned?`: finished or missing → release) instead of scanning queues, so a nightly run may hold its lock for hours without being reaped, and a batch whose release failed is healed on the next cleanup cycle. `Batch#uniqueness_key` / `#on_conflict` / `#lock_key` readers are new; `Batch.find` handles stay non-unique (the lock belongs to the run that created it).
+- **Visibility heartbeat: a running job's message no longer expires underneath it.** PGMQ hands a message to one reader for `visibility_timeout`; a job that ran longer was redelivered while still running — a second copy started, `read_ct` climbed on every redelivery, and after `max_retries` the message was dead-lettered without the job ever raising. Every long import, nightly cleanup and report job was exposed, and the usual workaround (a huge global `visibility_timeout`) made crash recovery equally slow for every queue. The executor now tracks each in-flight message with `Pgbus::VisibilityHeartbeat`: one thread per worker process re-arms the VT (`set_vt` back to `visibility_timeout`) every `visibility_heartbeat_interval` seconds — default a third of `visibility_timeout` — for as long as `perform` runs, and drops the entry before archive or the retry backoff touch the same message. The timeout therefore only fires for a process that is gone (crash, SIGKILL), which is what it exists for; `drain_timeout` still bounds what a deploy waits for. Works in both execution modes (the extension never runs inside the job's fiber), forgets the parent's entries after a fork, and is stopped by `Worker#shutdown`. New config: `visibility_heartbeat` (default `true`; `false` restores plain PGMQ semantics) and `visibility_heartbeat_interval` (seconds or Duration, must stay below `visibility_timeout`; validated at boot). Per-class opt-out: `pgbus_visibility_heartbeat false` (mixin included on `ActiveJob::Base` by the engine). New event `pgbus.job_visibility_extended` (`queue`, `job_class`, `msg_id`, `vt`, `extensions`) and counter `pgbus_visibility_extended` tagged `queue`/`job_class` — a class that shows up there is one that outlives the timeout. Event-bus handlers are not covered yet.
 - **Batches v2: open batches, a `batch` accessor on jobs and callbacks, and configured callback instances (issue #415).** ⚠️ **Breaking (pre-1.0)**: `Pgbus::Batch.find` now returns a rehydrated `Pgbus::Batch` handle instead of the raw attributes Hash — read the values off the handle (`status`, `total_jobs`, `completed_jobs`, `failed_jobs`, `pending_jobs`, `progress_percentage`, `finished?`, `description`, `properties`), or query `Pgbus::BatchEntry` for a row. Three capabilities: **(1) Open batches.** `batch.enqueue` is re-callable while the batch is unfinished — the second and later calls add to the existing group (`BatchEntry.increment_total_jobs!`, guarded on an unfinished row) instead of hitting the unique index on `batch_id`, so multi-stage workflows are possible. Adding to a finished batch raises `Pgbus::Batch::AlreadyFinished`. Execution rows are inserted as the block enqueues, before `total_jobs` is bumped, so the single-winner finish always sees outstanding work; a `check_finished!` after the bump covers a block whose jobs all completed while it was still open. Membership stays explicit: only jobs enqueued inside an `enqueue` block join the batch, so a fan-out from a batched job never silently extends it. **(2) `batch` accessor.** New `Pgbus::ActiveJob::BatchId` mixin (included on `ActiveJob::Base` from the engine alongside `Concurrency` and `Uniqueness`) adds `batch_id` / `callback_batch_id` accessors, round-trips both through `serialize`/`deserialize` — omitted from the payload when unset, so an unbatched job's serialized hash is unchanged — and exposes a memoized `batch` reader. The executor assigns `batch_id` from the payload's `pgbus_batch_id` before `perform`, so a running job can call `batch.enqueue` to add siblings. **(3) Configured callback instances.** `on_finish:` / `on_success:` / `on_failure:` accept an ActiveJob instance as well as a class: `on_finish: ReportJob.new.set(queue: :critical, wait: 5.minutes)`. `.set` options resolve at batch-creation time (matching solid_queue) into new `on_finish_job` / `on_success_job` / `on_failure_job` jsonb columns; at fire time the job is deserialized, given `callback_batch_id`, has its own `batch_id` cleared (a callback is never a member of the batch it reports on) and is enqueued on its configured queue and `scheduled_at`. Legacy class-name callbacks keep the `perform_later(properties)` signature — deprecated at 1.0 in favour of `batch.properties`. The dashboard shows a configured callback's `job_class`. Existing installs: `rails generate pgbus:add_batch_callback_jobs` (or `pgbus:update`); apps that have not migrated keep storing bare classes only. Refs #415.
 
 - **Batch completion is now self-healing via execution-row tracking (issue #414).** ⚠️ **Restart pgbus workers after `rails generate pgbus:add_batch_executions` / `db:migrate`** — a process that still writes `discarded_jobs` will error once that column is gone. `pgbus_batches` no longer finishes on `completed + discarded == total` — that arithmetic stalled a batch *forever* if a worker crashed between `archive_from` and `signal_batch_completed`, and a crash mid-`Batch#enqueue` left an immortal `pending` row (`cleanup_batches` only deletes `finished`). Each batched job now gets a `pgbus_batch_executions` row (identity = ActiveJob `job_id`) inserted *before* `send_message`, with `msg_id` + `queue_name` backfilled after send. The batch finishes when no rows remain (single-winner `UPDATE … AND NOT EXISTS`, plus a fresh `exists?` re-check for Postgres READ COMMITTED). A dispatcher sweep (`config.batch_sweep_interval`, default 5 minutes) repairs four crash windows: stale rows whose PGMQ message is gone (archived → completed, DLQ → failed, missing → completed with a warning); orphan rows with `msg_id` NULL older than 5 minutes (enqueue crashed between insert and send — blocked concurrency jobs are excluded); stalled `pending` batches; stalled `processing` batches with zero rows. `on_failure:` / `failed_jobs` are canonical (`on_discard:` / `discarded_jobs` remain deprecated aliases until 1.0). `config.batch_retention` (default 7 days, `nil` disables) replaces the hardcoded cleanup window. Existing installs: `rails generate pgbus:add_batch_executions` (or `pgbus:update`). Unmigrated apps keep the counter path via `Batch.executions_migrated?`. Refs #414.

data/README.md

@@ -575,6 +575,30 @@ class FragileApiJob < ApplicationJob
 end
 ```
 
+### Long-running jobs: visibility heartbeat
+
+PGMQ hands a message to one reader for `visibility_timeout` seconds. Without help, a job that runs longer is redelivered *while it is still running*: a second copy starts, `read_ct` climbs on every redelivery, and after `max_retries` the message is dead-lettered — all without the job ever raising.
+
+Pgbus keeps that from happening. While `perform` runs, a per-process heartbeat thread re-arms the message's visibility timeout every `visibility_heartbeat_interval` seconds (default: a third of `visibility_timeout`), so the timeout only fires for a process that is actually gone — a crash or a `SIGKILL` — which is what it exists for. The heartbeat is dropped before the message is archived or the retry backoff adjusts its VT, so neither path changes.
+
+```ruby
+Pgbus.configure do |config|
+  config.visibility_timeout = 10.minutes
+  config.visibility_heartbeat = true            # default
+  config.visibility_heartbeat_interval = 2.minutes  # default: visibility_timeout / 3
+end
+```
+
+Opt a job class out (a job that must be redelivered if it stalls, for example):
+
+```ruby
+class WatchdogJob < ApplicationJob
+  pgbus_visibility_heartbeat false
+end
+```
+
+Each extension emits `pgbus.job_visibility_extended` and increments the `pgbus_visibility_extended` counter (tags: `queue`, `job_class`) — a job class that shows up there is one that outlives `visibility_timeout`, so you can size the timeout from data instead of guessing. `drain_timeout` still bounds what a graceful shutdown waits for; a job still running past it is redelivered after the timeout, as before. Event-bus handlers are not covered yet.
+
 ### Async execution mode (fibers)
 
 Workers can optionally execute jobs as fibers instead of threads. This is ideal for I/O-bound workloads (HTTP calls, email delivery, LLM API calls) where jobs spend most of their time waiting on network I/O.
@@ -743,6 +767,24 @@ class BatchFinishedJob < ApplicationJob
 end
 ```
 
+#### Unique batches (run-scoped locks)
+
+`ensures_uniqueness` protects one job. A multi-job run — a nightly import that fans out into hundreds of chunk jobs, a pipeline whose jobs enqueue the next stage — has no single job that lives for the whole run, so a lock on the entry job releases the moment that job finishes and the next scheduler tick starts a second run on top of the first. Give the batch the lock instead:
+
+```ruby
+batch = Pgbus::Batch.new(
+  uniqueness_key: "perfecta-daily",
+  on_conflict: :discard,           # :reject (raise AlreadyRunning), :discard, :log
+  on_success: ExportPricesJob
+)
+
+batch.enqueue do
+  ImportChunkJob.perform_later("products", 0)
+end
+```
+
+The lock is taken at `enqueue` and held until the batch finishes — through every open-batch `batch.enqueue` add from inside a running job — then released in the same single-winner finish that fires the callbacks, so completion, the stalled-batch sweep and cleanup all release it. While it is held, another `Pgbus::Batch.new(uniqueness_key: "perfecta-daily").enqueue { ... }` raises `Pgbus::Batch::AlreadyRunning` (`:reject`), or skips its block and returns a batch that answers `discarded?` (`:discard`, `:log`). The row lives in `pgbus_uniqueness_keys` as `batch:<key>`, so it never collides with a job's key and shows up in the dashboard like any other lock; the reaper judges it by the batch's status, never by queue contents.
+
 #### How batches work
 
 1. `Batch.new(...)` creates a tracking row in `pgbus_batches` with `status: "pending"`
@@ -907,6 +949,7 @@ c.metrics_backend = MyOpenTelemetryBackend.new
 | Metric | Type | Tags |
 |--------|------|------|
 | `pgbus_queue_job_count` | counter | `queue`, `job_class`, `status` (`processed`/`failed`/`dead_lettered`) |
+| `pgbus_visibility_extended` | counter | `queue`, `job_class` — one per re-armed visibility timeout; a class listed here outlives `visibility_timeout` |
 | `pgbus_job_duration_ms` | histogram | `queue`, `job_class` |
 | `pgbus_event_count` | counter | `handler`, `routing_key`, `status` |
 | `pgbus_event_duration_ms` | histogram | `handler`, `routing_key` |
@@ -1751,7 +1794,12 @@ require "pgbus/testing/rspec"
 
 RSpec.configure do |config|
   config.before { Pgbus::Testing.fake! }
-  config.after do
+  # append_after, not after: config-level `after` hooks run in reverse
+  # registration order, so one registered after `capybara/rspec` runs BEFORE
+  # Capybara.reset_sessions! — while the browser page is still open.
+  # append_after runs once the page is closed and its pending SSE requests
+  # are drained (see "SSE streams in tests" below).
+  config.append_after do
     Pgbus::Testing.disabled!
     Pgbus::Testing.store.clear!
   end
@@ -1763,7 +1811,7 @@ Or scope it to specific groups:
 ```ruby
 RSpec.configure do |config|
   config.before(:each, :pgbus) { Pgbus::Testing.fake! }
-  config.after(:each, :pgbus) do
+  config.append_after(:each, :pgbus) do
     Pgbus::Testing.disabled!
     Pgbus::Testing.store.clear!
   end
@@ -1934,10 +1982,14 @@ HTTP/1.1 200 OK
 Content-Type: text/event-stream
 Cache-Control: no-cache, no-transform
 
+retry: 86400000
+
 : pgbus test mode — connection accepted, no polling
 ```
 
-This is a valid SSE response that the browser's EventSource will accept. No `Streamer` singleton is created, no PG LISTEN connection is opened, and no dispatcher/heartbeat/listener threads are spawned.
+This is a valid SSE response that the browser's EventSource will accept. No `Streamer` singleton is created, no PG LISTEN connection is opened, and no dispatcher/heartbeat/listener threads are spawned. The `retry:` directive tells `EventSource` to wait 24 hours before reconnecting: without it a page left open re-requests the closed stub every ~3 seconds for the whole example, and that reconnect storm is what turns a teardown race into a real streamer running inside the test process.
+
+**Hook ordering with Capybara:** `Pgbus::Testing.disabled!` turns `streams_test_mode` back off. If it runs while the browser page is still open, a reconnect landing in that window takes the real stream path and starts a live `Streamer` — with listener/dispatcher/heartbeat threads and a LISTEN connection — inside your RSpec process. RSpec runs config-level `after` hooks in reverse registration order, so a plain `config.after { Pgbus::Testing.disabled! }` registered after `capybara/rspec` fires *before* `Capybara.reset_sessions!`. Register it with `config.append_after` (as in the snippet above) so the page is closed first. If a live streamer does get started and its threads outlive the bounded shutdown, `disabled!` raises `Pgbus::Testing::StreamerLeakError` with this diagnosis rather than letting the suite hang on a corrupted shared connection; a streamer that shut down cleanly is only logged as a warning, because that is expected inside `Pgbus::Testing.disabled! do ... end` real-stream tests.
 
 **Testing actual stream delivery:** If you need to verify end-to-end SSE message delivery in integration tests, disable `streams_test_mode` and use the `PumaTestHarness` from the pgbus test support:
 
@@ -2190,6 +2242,8 @@ Curated headline options for the README. The full operator reference (with types
 | `roles` | `nil` (all) | Supervisor role filter — usually set via CLI flags (`--workers-only` etc.) |
 | `polling_interval` | `0.1` | Seconds between polls (LISTEN/NOTIFY is primary) |
 | `visibility_timeout` | `30` | Time before unacked message becomes visible again. Accepts seconds or `ActiveSupport::Duration` (e.g. `10.minutes`) |
+| `visibility_heartbeat` | `true` | Re-arm a running job's visibility timeout while `perform` runs, so long jobs are not redelivered or dead-lettered mid-run. `false` restores plain PGMQ semantics |
+| `visibility_heartbeat_interval` | `nil` (`visibility_timeout / 3`) | Seconds (or Duration) between two extensions; must stay below `visibility_timeout` |
 | `max_retries` | `5` | Failed reads before routing to dead letter queue |
 | `retry_backoff` | `5` | Base delay in seconds for VT-based retry backoff (exponential: `base * 2^(attempt-1)`) |
 | `retry_backoff_max` | `300` | Maximum retry delay in seconds (caps the exponential curve) |

data/config/i18n-tasks.yml

@@ -22,6 +22,7 @@ search:
   exclude:
     - "*.md"
     - app/assets/images
+    - app/frontend/pgbus/vendor
   strict: false
 
 translation:

data/config/locales/da.yml

@@ -168,8 +168,6 @@ da:
         reroute_failed: Kunne ikke omdirigere begivenhed.
         rerouted: Begivenhed omdirigeret til målhandler.
       index:
-        discard_all: Forkast alle
-        discard_all_confirm: Forkast alle ventende begivenheder? Dette kan ikke fortrydes.
         discard_selected: Forkast valgte
         discard_selected_confirm: Forkast valgte begivenheder?
         pending_empty: Ingen ventende begivenheder

data/config/locales/de.yml

@@ -168,8 +168,6 @@ de:
         reroute_failed: Ereignis konnte nicht umgeleitet werden.
         rerouted: Ereignis an Ziel-Handler umgeleitet.
       index:
-        discard_all: Alle verwerfen
-        discard_all_confirm: Alle ausstehenden Ereignisse verwerfen? Dies kann nicht rückgängig gemacht werden.
         discard_selected: Ausgewählte verwerfen
         discard_selected_confirm: Ausgewählte Ereignisse verwerfen?
         pending_empty: Keine ausstehenden Ereignisse

data/config/locales/en.yml

@@ -168,8 +168,6 @@ en:
         reroute_failed: Could not reroute event.
         rerouted: Event rerouted to target handler.
       index:
-        discard_all: Discard All
-        discard_all_confirm: Discard all pending events? This cannot be undone.
         discard_selected: Discard Selected
         discard_selected_confirm: Discard selected events?
         pending_empty: No pending events

data/config/locales/es.yml

@@ -168,8 +168,6 @@ es:
         reroute_failed: No se pudo redirigir el evento.
         rerouted: Evento redirigido al manejador objetivo.
       index:
-        discard_all: Descartar todo
-        discard_all_confirm: "¿Descartar todos los eventos pendientes? Esto no se puede deshacer."
         discard_selected: Descartar seleccionados
         discard_selected_confirm: "¿Descartar eventos seleccionados?"
         pending_empty: No hay eventos pendientes

data/config/locales/fi.yml

@@ -168,8 +168,6 @@ fi:
         reroute_failed: Tapahtumaa ei voitu uudelleenreitittää.
         rerouted: Tapahtuma uudelleenreititetty kohdekäsittelijälle.
       index:
-        discard_all: Hylkää kaikki
-        discard_all_confirm: Hylätäänkö kaikki odottavat tapahtumat? Tätä ei voi peruuttaa.
         discard_selected: Hylkää valitut
         discard_selected_confirm: Hylätäänkö valitut tapahtumat?
         pending_empty: Ei odottavia tapahtumia

data/config/locales/fr.yml

@@ -168,8 +168,6 @@ fr:
         reroute_failed: Impossible de rediriger l'événement.
         rerouted: Événement redirigé vers le gestionnaire cible.
       index:
-        discard_all: Tout jeter
-        discard_all_confirm: Jeter tous les événements en attente ? Cette action est irréversible.
         discard_selected: Jeter la sélection
         discard_selected_confirm: Jeter les événements sélectionnés ?
         pending_empty: Aucun événement en attente

data/config/locales/it.yml

@@ -168,8 +168,6 @@ it:
         reroute_failed: Impossibile reindirizzare l'evento.
         rerouted: Evento reindirizzato al gestore di destinazione.
       index:
-        discard_all: Scarta tutto
-        discard_all_confirm: Scartare tutti gli eventi in sospeso? Questa operazione non può essere annullata.
         discard_selected: Scarta selezionati
         discard_selected_confirm: Scartare gli eventi selezionati?
         pending_empty: Nessun evento in sospeso

data/config/locales/ja.yml

@@ -168,8 +168,6 @@ ja:
         reroute_failed: イベントをルーティングできませんでした。
         rerouted: イベントがターゲットハンドラーにルーティングされました。
       index:
-        discard_all: すべて破棄
-        discard_all_confirm: 保留中のすべてのイベントを破棄しますか？これは元に戻せません。
         discard_selected: 選択を破棄
         discard_selected_confirm: 選択したイベントを破棄しますか？
         pending_empty: 保留中のイベントはありません

data/config/locales/nb.yml

@@ -168,8 +168,6 @@ nb:
         reroute_failed: Kunne ikke omdirigere hendelse.
         rerouted: Hendelse omdirigert til målbehandler.
       index:
-        discard_all: Forkast alle
-        discard_all_confirm: Forkast alle ventende hendelser? Dette kan ikke angres.
         discard_selected: Forkast valgte
         discard_selected_confirm: Forkast valgte hendelser?
         pending_empty: Ingen ventende hendelser

data/config/locales/nl.yml

@@ -168,8 +168,6 @@ nl:
         reroute_failed: Kon gebeurtenis niet omleiden.
         rerouted: Gebeurtenis omgeleid naar doelhandler.
       index:
-        discard_all: Alles Verwijderen
-        discard_all_confirm: Alle openstaande gebeurtenissen verwijderen? Dit kan niet ongedaan worden gemaakt.
         discard_selected: Geselecteerde Verwijderen
         discard_selected_confirm: Geselecteerde gebeurtenissen verwijderen?
         pending_empty: Geen openstaande gebeurtenissen

data/config/locales/pt.yml

@@ -168,8 +168,6 @@ pt:
         reroute_failed: Não foi possível redirecionar o evento.
         rerouted: Evento redirecionado para o manipulador alvo.
       index:
-        discard_all: Descartar Todos
-        discard_all_confirm: Descartar todos os eventos pendentes? Esta ação não pode ser desfeita.
         discard_selected: Descartar Selecionados
         discard_selected_confirm: Descartar eventos selecionados?
         pending_empty: Nenhum evento pendente

data/config/locales/sv.yml

@@ -168,8 +168,6 @@ sv:
         reroute_failed: Kunde inte omdirigera händelsen.
         rerouted: Händelsen omdirigerades till målhanteraren.
       index:
-        discard_all: Kassera alla
-        discard_all_confirm: Kassera alla väntande händelser? Detta kan inte ångras.
         discard_selected: Kassera valda
         discard_selected_confirm: Kassera valda händelser?
         pending_empty: Inga väntande händelser

data/lib/pgbus/active_job/executor.rb

@@ -96,7 +96,7 @@ module Pgbus
           # `batch` (and `batch.enqueue` for open batches) work inside a job.
           assign_batch_id(job, payload)
           Pgbus.logger.debug { "[Pgbus::Executor] running #{tag} job_class=#{job_class}" }
-          execute_job(job)
+          with_visibility_heartbeat(job, queue_name, msg_id, source_queue) { execute_job(job) }
           # retry_on re-enqueues from inside perform_now and returns normally:
           # this attempt is done (archive it) but the job is not — the retry
           # message carries the batch tag and signals on its own outcome.
@@ -174,6 +174,20 @@ module Pgbus
         Pgbus.logger.warn { "[Pgbus] Uniqueness release failed: #{e.message}" }
       end
 
+      # Keep the message invisible while perform runs (see VisibilityHeartbeat).
+      # Wraps only the perform: the heartbeat must be gone before archive or
+      # the retry backoff touches the same message's VT.
+      def with_visibility_heartbeat(job, queue_name, msg_id, source_queue, &)
+        klass = job.class
+        per_job = klass.respond_to?(:pgbus_visibility_heartbeat_enabled) ? klass.pgbus_visibility_heartbeat_enabled : nil
+        return yield if per_job == false
+
+        VisibilityHeartbeat.track(
+          client: client, queue_name: source_queue || queue_name, prefixed: source_queue.nil?,
+          msg_id: msg_id, job_class: klass.name, config: config, &
+        )
+      end
+
       def execute_job(job)
         if defined?(Rails) && Rails.respond_to?(:application) && Rails.application
           wrapper = reloading? ? Rails.application.reloader : Rails.application.executor

data/lib/pgbus/batch.rb

@@ -6,18 +6,39 @@ require "json"
 module Pgbus
   class Batch
     class AlreadyFinished < Error; end
+    # Raised by #enqueue when a batch with the same uniqueness_key is still
+    # running and on_conflict: is :reject.
+    class AlreadyRunning < Error; end
 
     METADATA_KEY = "pgbus_batch_id"
 
+    # A unique batch holds one row in pgbus_uniqueness_keys from #enqueue until
+    # the batch finishes. The row's lock_key carries the caller's key under
+    # this prefix (so a job's ensures_uniqueness key never collides with it)
+    # and its queue_name names the owning batch, which is how the reaper
+    # tells a live run from an orphan (see .lock_orphaned?).
+    LOCK_KEY_PREFIX = "batch:"
+    LOCK_QUEUE_PREFIX = "batch:"
+    VALID_CONFLICTS = %i[reject discard log].freeze
+
     attr_reader :batch_id, :properties, :description,
-                :on_finish, :on_success, :on_failure
+                :on_finish, :on_success, :on_failure,
+                :uniqueness_key, :on_conflict
 
     def on_discard
       on_failure
     end
 
-    def initialize(on_finish: nil, on_success: nil, on_discard: nil, on_failure: nil, description: nil, properties: {})
+    # @param uniqueness_key [String, nil] at most one unfinished batch with
+    #   this key may exist; see #enqueue for what happens to the next one
+    # @param on_conflict [Symbol] :reject (raise AlreadyRunning), :discard
+    #   (skip the block silently) or :log (skip the block, warn)
+    def initialize(on_finish: nil, on_success: nil, on_discard: nil, on_failure: nil, description: nil, properties: {},
+                   uniqueness_key: nil, on_conflict: :reject)
       raise ArgumentError, "pass on_failure: only — on_discard: is a deprecated alias" if on_discard && on_failure
+      unless VALID_CONFLICTS.include?(on_conflict)
+        raise ArgumentError, "on_conflict must be one of #{VALID_CONFLICTS.join(", ")}, got #{on_conflict.inspect}"
+      end
 
       if on_discard
         Pgbus.logger.warn do
@@ -31,9 +52,20 @@ module Pgbus
       @on_failure = on_failure || on_discard
       @description = description
       @properties = properties
+      @uniqueness_key = uniqueness_key&.to_s
+      @on_conflict = on_conflict
+      @discarded = false
       @started = false
     end
 
+    # True when #enqueue found another batch with the same uniqueness_key
+    # still running and skipped this one (on_conflict: :discard or :log).
+    def discarded? = @discarded
+
+    def lock_key
+      "#{LOCK_KEY_PREFIX}#{uniqueness_key}" if uniqueness_key
+    end
+
     # Enqueue a group of jobs as a batch. Jobs enqueued inside the block join
     # this batch.
     #
@@ -45,8 +77,14 @@ module Pgbus
     # Raises Pgbus::Batch::AlreadyFinished once the batch has finished.
     def enqueue(&)
       return reopen(&) if @started
+      return self unless acquire_lock!
 
-      create_record
+      begin
+        create_record
+      rescue StandardError
+        self.class.release_lock(batch_id)
+        raise
+      end
       @started = true
       count_jobs(&)
       start_processing
@@ -134,6 +172,43 @@ module Pgbus
       BatchEntry.stale(before: older_than).delete_all
     end
 
+    # --- run-scoped uniqueness lock ------------------------------------
+
+    # queue_name stored on the uniqueness row of a unique batch.
+    def self.lock_queue_name(batch_id)
+      "#{LOCK_QUEUE_PREFIX}#{batch_id}"
+    end
+
+    # True for a pgbus_uniqueness_keys row that belongs to a batch rather
+    # than to a message.
+    def self.lock_row?(queue_name)
+      queue_name.to_s.start_with?(LOCK_QUEUE_PREFIX)
+    end
+
+    # For the dispatcher's reaper: a batch lock is an orphan once its batch
+    # has finished (the release in finish_if_needed failed) or its row is
+    # gone (cleanup). A batch that is still pending/processing keeps the lock
+    # no matter how old it is — a nightly run legitimately holds it for
+    # hours. Any lookup error keeps the lock; the reaper never deletes in
+    # doubt.
+    def self.lock_orphaned?(queue_name)
+      batch_id = queue_name.to_s.delete_prefix(LOCK_QUEUE_PREFIX)
+      record = BatchEntry.find_by(batch_id: batch_id)
+      record.nil? || record.status == "finished"
+    rescue StandardError => e
+      Pgbus.logger.debug { "[Pgbus] Batch lock lookup failed for #{queue_name}: #{e.message}" }
+      false
+    end
+
+    # Drop the uniqueness row a batch holds, if any. Fail-soft: the reaper
+    # releases the row once the batch is finished.
+    def self.release_lock(batch_id)
+      UniquenessKey.where(queue_name: lock_queue_name(batch_id)).delete_all
+    rescue StandardError => e
+      Pgbus.logger.debug { "[Pgbus] Batch lock release failed for #{batch_id}: #{e.message}" }
+      0
+    end
+
     def self.executions_migrated?
       return true if @executions_migrated
 
@@ -329,6 +404,9 @@ module Pgbus
 
         fire_callbacks(result[:record])
         instrument_finished(result[:record])
+        # Every finish path (completion, sweep) funnels through here, so this
+        # is the one place a unique batch gives its run lock back.
+        release_lock(result[:record].batch_id) if result[:record].respond_to?(:batch_id)
         result
       end
 
@@ -438,9 +516,34 @@ module Pgbus
       @on_finish = nil
       @on_success = nil
       @on_failure = nil
+      @uniqueness_key = nil
+      @on_conflict = :reject
+      @discarded = false
       @started = true
     end
 
+    # Take the run lock for a unique batch. Returns true when the batch may
+    # proceed (no key, or the lock was won); false when another batch holds
+    # the key and on_conflict is :discard or :log. :reject raises.
+    def acquire_lock! # rubocop:disable Naming/PredicateMethod
+      return true unless uniqueness_key
+
+      acquired = UniquenessKey.acquire!(lock_key, queue_name: self.class.lock_queue_name(batch_id), msg_id: 0)
+      UniquenessKey.clear_bind_stamp!(lock_key)
+      return true if acquired
+
+      case on_conflict
+      when :reject
+        raise AlreadyRunning, "Batch #{uniqueness_key.inspect} is already running"
+      when :discard
+        Pgbus.logger.info { "[Pgbus] Discarding batch #{uniqueness_key.inspect}: a batch with that key is still running" }
+      else
+        Pgbus.logger.warn { "[Pgbus] Batch #{uniqueness_key.inspect} skipped: a batch with that key is still running" }
+      end
+      @discarded = true
+      false
+    end
+
     # Add to an already-created batch. Each job counts itself in (guarded
     # increment + execution row, see .track_enqueue) as it is enqueued, so an
     # add into a finished batch raises at perform_later before anything is

data/lib/pgbus/client.rb

@@ -821,6 +821,13 @@ module Pgbus
                       .map { |row| row["queue_name"] }
           names.each do |name|
             break if found.size == keys.size
+            # A dead-lettered copy still carries the payload's uniqueness key,
+            # but it is not in flight: the executor released the lock when it
+            # moved the message. Counting it as "present" would pin an unbound
+            # lock for as long as the DLQ row exists, and a `:until_executed` +
+            # `on_conflict: :discard` job would then be discarded on every
+            # enqueue until someone purged the DLQ by hand.
+            next if name.end_with?(Pgbus::DEAD_LETTER_SUFFIX)
 
             sanitized = begin
               QueueNameValidator.sanitize!(name)

data/lib/pgbus/configuration.rb

@@ -12,7 +12,14 @@ module Pgbus
 
     # Worker settings
     attr_accessor :polling_interval, :prefetch_limit, :execution_mode
-    attr_reader :workers, :visibility_timeout # rubocop:disable Style/AccessorGrouping
+    # visibility_heartbeat / visibility_heartbeat_interval: while a job runs,
+    # its message's visibility timeout is re-armed every interval seconds
+    # (default: a third of visibility_timeout), so a job that outlives the
+    # timeout is not redelivered — and eventually dead-lettered — while it is
+    # still running. The timeout then only fires for a process that is gone.
+    # Set visibility_heartbeat = false to restore plain PGMQ semantics.
+    attr_accessor :visibility_heartbeat
+    attr_reader :workers, :visibility_timeout, :visibility_heartbeat_interval # rubocop:disable Style/AccessorGrouping
 
     # Supervisor role selection.
     # nil = boot all roles (default behavior).
@@ -266,6 +273,8 @@ module Pgbus
       @roles = nil
       @polling_interval = 0.1
       @visibility_timeout = 30
+      @visibility_heartbeat = true
+      @visibility_heartbeat_interval = nil
 
       @prefetch_limit = nil
       @execution_mode = :threads
@@ -762,6 +771,7 @@ module Pgbus
         raise Pgbus::ConfigurationError,
               "visibility_timeout must be > 0"
       end
+      validate_visibility_heartbeat!
       raise Pgbus::ConfigurationError, "max_retries must be >= 0" unless max_retries.is_a?(Integer) && max_retries >= 0
       raise Pgbus::ConfigurationError, "retry_backoff must be > 0" unless retry_backoff.is_a?(Numeric) && retry_backoff.positive?
       unless retry_backoff_max.is_a?(Numeric) && retry_backoff_max.positive?
@@ -1230,6 +1240,27 @@ module Pgbus
     # Numeric values are stored unchanged (preserving Float for sub-second
     # values). Duration values are coerced to Integer seconds via .to_i.
 
+    def visibility_heartbeat_interval=(value)
+      @visibility_heartbeat_interval = coerce_duration!(value, :visibility_heartbeat_interval)
+    end
+
+    # Seconds between two extensions of a running job's visibility timeout:
+    # the configured interval, or a third of visibility_timeout.
+    def effective_visibility_heartbeat_interval
+      visibility_heartbeat_interval || (visibility_timeout / 3.0)
+    end
+
+    def validate_visibility_heartbeat!
+      raise Pgbus::ConfigurationError, "visibility_heartbeat must be true or false" unless [true, false].include?(visibility_heartbeat)
+      return if visibility_heartbeat_interval.nil?
+
+      interval = visibility_heartbeat_interval
+      return if interval.is_a?(Numeric) && interval.positive? && interval < visibility_timeout
+
+      raise Pgbus::ConfigurationError,
+            "visibility_heartbeat_interval must be > 0 and below visibility_timeout (#{visibility_timeout}s)"
+    end
+
     def visibility_timeout=(value)
       @visibility_timeout = coerce_duration!(value, :visibility_timeout)
     end

data/lib/pgbus/engine.rb

@@ -76,6 +76,7 @@ module Pgbus
         include Pgbus::Uniqueness
         include Pgbus::ActiveJob::BatchId
         include Pgbus::ActiveJob::CurrentAttributes
+        include Pgbus::VisibilityHeartbeat::JobMixin
       end
     end
 

data/lib/pgbus/instrumentation.rb

@@ -16,6 +16,8 @@ module Pgbus
   #   pgbus.job_completed          — job archived successfully
   #   pgbus.job_failed             — job raised; carries :exception_object
   #   pgbus.job_dead_lettered      — job exceeded max_retries and was DLQ-routed
+  #   pgbus.job_visibility_extended — heartbeat re-armed a running job's visibility timeout
+  #                                  payload: queue, job_class, msg_id, vt, extensions
   #   pgbus.event_processed        — event handler succeeded
   #   pgbus.event_failed           — event handler raised; carries :exception_object
   #   pgbus.stream.broadcast       — stream broadcast (sync or deferred)

data/lib/pgbus/metrics/subscriber.rb

@@ -30,6 +30,7 @@ module Pgbus
             subscribe("pgbus.job_completed") { |event| on_job_completed(event) },
             subscribe("pgbus.job_failed") { |event| on_job_failed(event) },
             subscribe("pgbus.job_dead_lettered") { |event| on_job_dead_lettered(event) },
+            subscribe("pgbus.job_visibility_extended") { |event| on_job_visibility_extended(event) },
             subscribe("pgbus.event_processed") { |event| on_event_processed(event) },
             subscribe("pgbus.event_failed") { |event| on_event_failed(event) },
             subscribe("pgbus.client.send_message") { |event| on_send_message(event) },
@@ -111,6 +112,16 @@ module Pgbus
           )
         end
 
+        # One increment per re-armed visibility timeout: a job class that
+        # shows up here is one that outlives visibility_timeout.
+        def on_job_visibility_extended(event)
+          payload = event.payload
+          backend.increment(
+            "#{METRIC_PREFIX}visibility_extended", 1,
+            compact(queue: payload[:queue], job_class: payload[:job_class])
+          )
+        end
+
         # ── Event handler ─────────────────────────────────────────────────
 
         def on_event_processed(event)

data/lib/pgbus/process/dispatcher.rb

@@ -461,8 +461,14 @@ module Pgbus
         candidates = keys.select { |key| key.created_at && key.created_at < threshold && key.queue_name }
         return 0 if candidates.empty?
 
+        # A unique batch's lock is judged by its batch, not by any queue: it
+        # is held for the whole run (hours, legitimately) and is an orphan
+        # only once the batch has finished or its row is gone.
+        batch_locks, candidates = candidates.partition { |key| Batch.lock_row?(key.queue_name) }
+        orphaned = batch_locks.select { |key| Batch.lock_orphaned?(key.queue_name) }
+
         bound, unbound = candidates.partition { |key| bound_lock?(key) }
-        orphaned = bound.select { |key| message_gone?(key) }
+        orphaned.concat(bound.select { |key| message_gone?(key) })
         orphaned.concat(gone_unbound_locks(unbound))
 
         return 0 if orphaned.empty?

data/lib/pgbus/process/worker.rb

@@ -844,6 +844,7 @@ module Pgbus
         # proven it won't finish; waiting another full window would push the
         # worker past the supervisor's shutdown_timeout deadline (issue #386).
         @pool.wait_for_termination(POOL_TERMINATION_WAIT)
+        VisibilityHeartbeat.stop
         @stat_buffer&.stop
         @queue_lock&.unlock_all
         @heartbeat&.stop

data/lib/pgbus/testing.rb

@@ -18,6 +18,11 @@ module Pgbus
     MODES = %i[fake inline disabled].freeze
     MODE_KEY = :__pgbus_test_mode
 
+    # Raised by `disabled!` when a live Pgbus::Web::Streamer had to be torn
+    # down and some of its threads did not stop within their join budget —
+    # a red spec with a diagnosis instead of a hung test process (issue #443).
+    class StreamerLeakError < Pgbus::Error; end
+
     # Thread-safe in-memory store for events captured in fake/inline mode.
     class EventStore
       def initialize
@@ -70,15 +75,17 @@ module Pgbus
       def mode!(mode, &block)
         raise ArgumentError, "Unknown mode: #{mode}. Valid modes: #{MODES.join(", ")}" unless MODES.include?(mode)
 
-        sync_streams_test_mode!(mode)
-
+        # Record the mode BEFORE syncing streams: a StreamerLeakError raised
+        # by the teardown must still leave the process in the requested mode.
         unless block
           Thread.main[MODE_KEY] = mode
+          sync_streams_test_mode!(mode)
           return
         end
 
         old = Thread.current[MODE_KEY]
         Thread.current[MODE_KEY] = mode
+        sync_streams_test_mode!(mode)
         yield
       ensure
         if block
@@ -112,11 +119,34 @@ module Pgbus
 
         if mode == :disabled
           Pgbus.configuration.streams_test_mode = false
-          Pgbus::Web::Streamer.reset! if defined?(Pgbus::Web::Streamer)
+          reset_streamer!
         else
           Pgbus.configuration.streams_test_mode = true
         end
       end
+
+      # A live streamer at this point is legitimate inside a
+      # `disabled! do ... end` real-stream test, so a clean teardown only
+      # warns. Threads that outlived shutdown! are a different matter: left
+      # alone they share the test's pinned AR connection with the test thread
+      # and the suite hangs with no diagnosis (issue #443) — so raise.
+      def reset_streamer!
+        return unless defined?(Pgbus::Web::Streamer)
+
+        leaked = Pgbus::Web::Streamer.reset!
+        return if leaked.nil?
+
+        if leaked.empty?
+          Pgbus.logger.warn { "[Pgbus::Testing] disabled! tore down a live Pgbus::Web::Streamer that was started during the test" }
+          return
+        end
+
+        raise StreamerLeakError, <<~MSG
+          Pgbus::Testing.disabled! shut down a live Pgbus::Web::Streamer but its #{leaked.join(", ")} thread(s) did not stop within their join budget.
+          A live streamer inside the test process means an SSE request reached the real stream path while streams_test_mode was off — usually because Pgbus::Testing.disabled! ran in an RSpec `after` hook before Capybara reset the browser session (page still open, EventSource still reconnecting).
+          Register the hook with `config.append_after { Pgbus::Testing.disabled! }` so it runs after Capybara.reset_sessions!. See the README section "SSE streams in tests".
+        MSG
+      end
     end
   end
 end

data/lib/pgbus/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Pgbus
-  VERSION = "0.15.3"
+  VERSION = "0.16.0"
 end

data/lib/pgbus/visibility_heartbeat.rb

@@ -0,0 +1,194 @@
+# frozen_string_literal: true
+
+module Pgbus
+  # Keeps a running job's PGMQ message invisible for as long as the job is
+  # actually running.
+  #
+  # PGMQ hands a message to one reader for `visibility_timeout` seconds. A job
+  # that runs longer is redelivered while it is still running: a second copy
+  # starts, `read_ct` climbs on every redelivery, and after `max_retries` the
+  # message is dead-lettered — all without the job ever raising. The heartbeat
+  # extends the VT of every in-flight message on a fixed cadence, so the
+  # timeout only ever fires for a job whose process is gone (crash, SIGKILL),
+  # which is the case it exists for.
+  #
+  # One background thread per process, started lazily by the first tracked
+  # job and stopped by Worker#shutdown. Entries are keyed by physical queue +
+  # msg_id and hold the client that read the message, so async and threaded
+  # execution modes both work: the extension runs on this thread, never
+  # inside the job's fiber. A fork forgets the parent's entries.
+  #
+  #   Pgbus::VisibilityHeartbeat.track(client:, queue_name:, msg_id:) { job.perform_now }
+  #
+  # Disable globally with `config.visibility_heartbeat = false`, tune the
+  # cadence with `config.visibility_heartbeat_interval`, or opt a job class
+  # out with `pgbus_visibility_heartbeat false`.
+  module VisibilityHeartbeat
+    Entry = Struct.new(:client, :queue_name, :prefixed, :msg_id, :job_class, :extended_at, :extensions,
+                       keyword_init: true)
+
+    # Per-job opt-out, included on ActiveJob::Base by the engine:
+    #
+    #   class ShortJob < ApplicationJob
+    #     pgbus_visibility_heartbeat false
+    #   end
+    module JobMixin
+      extend ActiveSupport::Concern
+
+      included do
+        class_attribute :pgbus_visibility_heartbeat_enabled, instance_writer: false, default: nil
+      end
+
+      class_methods do
+        def pgbus_visibility_heartbeat(enabled = true) # rubocop:disable Style/OptionalBooleanParameter
+          self.pgbus_visibility_heartbeat_enabled = enabled ? true : false
+        end
+      end
+    end
+
+    class << self
+      # Track the message for the duration of the block.
+      #
+      # @param client [Pgbus::Client] the client that read the message
+      # @param queue_name [String] logical name, or physical when prefixed: false
+      # @param msg_id [Integer]
+      # @param prefixed [Boolean] whether queue_name still needs the prefix
+      # @param job_class [String, nil] for logging and instrumentation
+      # @param config [Pgbus::Configuration]
+      def track(client:, queue_name:, msg_id:, prefixed: true, job_class: nil, config: Pgbus.configuration)
+        return yield unless config.visibility_heartbeat
+
+        entry = Entry.new(client: client, queue_name: queue_name, prefixed: prefixed, msg_id: msg_id.to_i,
+                          job_class: job_class, extended_at: monotonic_now, extensions: 0)
+        register(entry, config)
+        begin
+          yield
+        ensure
+          unregister(entry)
+        end
+      end
+
+      # Extend every tracked message whose last extension is older than the
+      # heartbeat interval. Public so tests and callers without the thread
+      # can drive it.
+      def tick!(now: monotonic_now, config: Pgbus.configuration)
+        interval = config.effective_visibility_heartbeat_interval
+        due = synchronize { entries.values.select { |entry| now - entry.extended_at >= interval } }
+        due.each { |entry| extend!(entry, now: now, config: config) }
+        due.size
+      end
+
+      def tracked_count
+        synchronize { entries.size }
+      end
+
+      # Stop the background thread. Tracked entries are kept: a job still
+      # running during shutdown can drive tick! itself, and Worker#shutdown
+      # only calls this once the pool has drained.
+      def stop
+        thread = synchronize do
+          @running = false
+          current = @thread
+          @thread = nil
+          current
+        end
+        return unless thread
+
+        thread.wakeup if thread.alive?
+        thread.join(1)
+      end
+
+      # Forget every entry and stop the thread. Test helper.
+      def reset!
+        stop
+        synchronize { @entries = {} }
+      end
+
+      private
+
+      def register(entry, config)
+        synchronize do
+          forget_parent_entries!
+          entries[key_for(entry)] = entry
+          ensure_thread(config)
+        end
+      end
+
+      def unregister(entry)
+        synchronize { entries.delete(key_for(entry)) }
+      end
+
+      def extend!(entry, now:, config:)
+        vt = config.visibility_timeout
+        entry.client.set_visibility_timeout(entry.queue_name, entry.msg_id, vt: vt, prefixed: entry.prefixed)
+        entry.extended_at = now
+        entry.extensions += 1
+        Instrumentation.instrument(
+          "pgbus.job_visibility_extended",
+          queue: entry.queue_name, job_class: entry.job_class, msg_id: entry.msg_id, vt: vt,
+          extensions: entry.extensions
+        )
+        Pgbus.logger.debug do
+          "[Pgbus::VisibilityHeartbeat] extended msg_id=#{entry.msg_id} queue=#{entry.queue_name} " \
+            "job_class=#{entry.job_class} vt=#{vt} extensions=#{entry.extensions}"
+        end
+      rescue StandardError => e
+        # The next tick retries; the message simply keeps its current VT.
+        Pgbus.logger.warn do
+          "[Pgbus::VisibilityHeartbeat] could not extend msg_id=#{entry.msg_id} queue=#{entry.queue_name}: " \
+            "#{e.class}: #{e.message}"
+        end
+      end
+
+      # Caller holds the mutex. Start the ticker once per process.
+      def ensure_thread(config)
+        return if @running && @thread&.alive?
+
+        @running = true
+        @thread = Thread.new { run_loop(config) }
+        @thread.name = "pgbus-visibility-heartbeat"
+      end
+
+      def run_loop(config)
+        while @running
+          # Half the interval keeps every extension inside [interval, 1.5 * interval]
+          # of the previous one — at most half the visibility timeout.
+          sleep([config.effective_visibility_heartbeat_interval / 2.0, 0.05].max)
+          break unless @running
+
+          tick!(config: config)
+        end
+      rescue StandardError => e
+        Pgbus.logger.error { "[Pgbus::VisibilityHeartbeat] ticker died: #{e.class}: #{e.message}" }
+        synchronize { @running = false }
+      end
+
+      # Entries registered before a fork belong to the parent's jobs; the
+      # thread did not survive the fork either.
+      def forget_parent_entries!
+        return if @pid == ::Process.pid
+
+        @pid = ::Process.pid
+        @entries = {}
+        @running = false
+        @thread = nil
+      end
+
+      def entries
+        @entries ||= {}
+      end
+
+      def key_for(entry)
+        [entry.queue_name, entry.msg_id]
+      end
+
+      def synchronize(&)
+        (@mutex ||= Mutex.new).synchronize(&)
+      end
+
+      def monotonic_now
+        ::Process.clock_gettime(::Process::CLOCK_MONOTONIC)
+      end
+    end
+  end
+end

data/lib/pgbus/web/stream_app.rb

@@ -31,6 +31,8 @@ module Pgbus
     # full request lifecycle.
     class StreamApp
       PATH_PREFIX = "/pgbus/streams"
+      # EventSource reconnect delay handed to the streams_test_mode stub (24h).
+      TEST_MODE_RETRY_MS = 86_400_000
       private_constant :PATH_PREFIX
 
       def initialize(streamer: nil, config: nil, logger: nil, authorize: nil)
@@ -221,8 +223,15 @@ module Pgbus
         [500, { "content-type" => "text/plain" }, ["pgbus: internal error"]]
       end
 
+      # A closed SSE response makes EventSource reconnect every ~3s by default.
+      # In a browser-driven test that storm keeps hitting the endpoint for the
+      # whole example, so a reconnect landing after the harness has turned
+      # streams_test_mode back off (teardown ordering) would start a LIVE
+      # streamer inside the test process (issue #443). Telling the browser to
+      # wait a day before retrying closes that window.
       def test_mode_stub
-        body = ": pgbus test mode — connection accepted, no polling\n\n"
+        retry_line = Pgbus::Streams::Envelope.retry_directive(TEST_MODE_RETRY_MS)
+        body = "#{retry_line}: pgbus test mode — connection accepted, no polling\n\n"
         [200, sse_headers, [body]]
       end
     end

data/lib/pgbus/web/streamer/failover_listener.rb

@@ -115,6 +115,13 @@ module Pgbus
           end
         end
 
+        # Snapshot of this component's live thread(s). Instance#shutdown! captures
+        # it BEFORE calling #stop so a join that timed out is still observable
+        # after #stop has cleared the reference (issue #443).
+        def threads
+          current_impl.threads
+        end
+
         def stop
           current_impl.stop
         end

data/lib/pgbus/web/streamer/heartbeat.rb

@@ -47,6 +47,13 @@ module Pgbus
           self
         end
 
+        # Snapshot of this component's live thread(s). Instance#shutdown! captures
+        # it BEFORE calling #stop so a join that timed out is still observable
+        # after #stop has cleared the reference (issue #443).
+        def threads
+          [@thread].compact
+        end
+
         def stop
           return unless @running
 

data/lib/pgbus/web/streamer/hub_client.rb

@@ -98,6 +98,13 @@ module Pgbus
           nil
         end
 
+        # Snapshot of this component's live thread(s). Instance#shutdown! captures
+        # it BEFORE calling #stop so a join that timed out is still observable
+        # after #stop has cleared the reference (issue #443).
+        def threads
+          [@reader].compact
+        end
+
         def stop
           @stopping = true
           close_quietly(@sock)

data/lib/pgbus/web/streamer/instance.rb

@@ -135,16 +135,28 @@ module Pgbus
         #
         # Bounded by the configured write deadline per connection; a dead
         # client drops instantly, a slow one stalls for at most write_deadline_ms.
+        #
+        # Every component join is bounded, so a thread blocked in a slow
+        # client write or a libpq call can outlive its #stop. That is not
+        # silent any more (issue #443): the names of the components whose
+        # threads are still alive afterwards are logged once and RETURNED
+        # (empty array = clean shutdown, also on a repeat call). The sentinel
+        # + socket close still run — Connection#close takes the same mutex as
+        # the writer, so it cannot fire mid-write even under a stuck thread.
         def shutdown!
           @shutdown_mutex.synchronize do
-            return unless @started
+            return [] unless @started
 
             @started = false
+            threads = component_threads
             safely { @heartbeat.stop }
             safely { @listener.stop }
             safely { @dispatcher.stop }
             safely { @pump&.stop }
+            leaked = threads.select { |_, list| list.any?(&:alive?) }.keys
+            report_leaked_threads(leaked) if leaked.any?
             close_all_connections
+            leaked
           end
         end
 
@@ -224,6 +236,25 @@ module Pgbus
           @logger.warn { "[Pgbus::Streamer::Instance] component stop raised: #{e.class}: #{e.message}" }
         end
 
+        # Captured BEFORE the stops: every component nils its thread
+        # reference in #stop whether or not the join succeeded.
+        def component_threads
+          {
+            "heartbeat" => @heartbeat.threads,
+            "listener" => @listener.threads,
+            "dispatcher" => @dispatcher.threads,
+            "pump" => @pump ? @pump.threads : []
+          }
+        end
+
+        def report_leaked_threads(leaked)
+          @logger.error do
+            "[Pgbus::Streamer::Instance] shutdown! finished but the #{leaked.join(", ")} thread(s) are " \
+              "still running past their join budget; they exit on their own once their current blocking " \
+              "call returns. In a test process this means a live streamer was started mid-suite (issue #443)."
+          end
+        end
+
         # Off-thread durable fanout writer (issue #321). nil (the default) keeps
         # fanout writes inline on the dispatcher thread. When on_dead fires
         # (a write failed), the pump posts a DisconnectMessage onto the shared

data/lib/pgbus/web/streamer/listener.rb

@@ -117,6 +117,13 @@ module Pgbus
           !@conn.nil?
         end
 
+        # Snapshot of this component's live thread(s). Instance#shutdown! captures
+        # it BEFORE calling #stop so a join that timed out is still observable
+        # after #stop has cleared the reference (issue #443).
+        def threads
+          [@thread].compact
+        end
+
         def stop
           return unless @running
 

data/lib/pgbus/web/streamer/outbound_pump.rb

@@ -99,6 +99,13 @@ module Pgbus
         # join each worker bounded by the write deadline. Never Thread#kill — a
         # kill mid write_nonblock corrupts IO state (mirrors
         # StreamEventDispatcher#stop). Idempotent.
+        # Snapshot of this component's live thread(s). Instance#shutdown! captures
+        # it BEFORE calling #stop so a join that timed out is still observable
+        # after #stop has cleared the reference (issue #443).
+        def threads
+          @threads.dup
+        end
+
         def stop
           return self unless @started
 

data/lib/pgbus/web/streamer/stream_event_dispatcher.rb

@@ -150,6 +150,13 @@ module Pgbus
           self
         end
 
+        # Snapshot of this component's live thread(s). Instance#shutdown! captures
+        # it BEFORE calling #stop so a join that timed out is still observable
+        # after #stop has cleared the reference (issue #443).
+        def threads
+          [@thread].compact
+        end
+
         def stop
           return unless @running
 

data/lib/pgbus/web/streamer.rb

@@ -41,6 +41,9 @@ module Pgbus
           @current_mutex.synchronize { @current&.stream_counter }
         end
 
+        # Tears down the current instance, if any. Returns nil when there was
+        # nothing to reset, otherwise Instance#shutdown!'s list of component
+        # names whose threads survived their bounded joins ([] = clean).
         def reset!
           instance = nil
           @current_mutex.synchronize do

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: pgbus
 version: !ruby/object:Gem::Version
-  version: 0.15.3
+  version: 0.16.0
 platform: ruby
 authors:
 - Mikael Henriksson
@@ -394,6 +394,7 @@ files:
 - lib/pgbus/testing/rspec.rb
 - lib/pgbus/uniqueness.rb
 - lib/pgbus/version.rb
+- lib/pgbus/visibility_heartbeat.rb
 - lib/pgbus/web/authentication.rb
 - lib/pgbus/web/data_source.rb
 - lib/pgbus/web/health_app.rb