pgbus 0.15.4 → 0.16.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 385f64fc61d6cb0c6c7c2cb1ae9e7286a73bdf1da97545ff7d409884243085e8
4
- data.tar.gz: ea244f460ce992e31d234ab30d1ad9d083a20ae3493f745c04abee2e1a389a39
3
+ metadata.gz: 2043a2be7395c42d40398ebbb862de8c3e142dcb17967980b069ac8e23313fb2
4
+ data.tar.gz: 850d0b62d0fe06528f166e15b136efb11981ef5bbbeeae95e9f5a37ef282f7e1
5
5
  SHA512:
6
- metadata.gz: a9a3926554aec518d1c59c14749ebb345eb8cce02467dfed0df4c9bb0778eee2e3899517bd1d2533446269bee9bf2e1ed2a935a0658c1b727973aa68d1c7f4c7
7
- data.tar.gz: 939083f3a554c6539da4f15167517c21ccee8314dcc732b9b29abf4e2271c97f1d1ef48e24d3b64bbe91e7956c177210e57a692b327d720730f340f205330b59
6
+ metadata.gz: dab60674d617fd909bd54a1f4904eb0262611ac4e146390ac828decbb0e4145503a362a95c0f5b0c5e5ef565e17891a12aa1556ae7c28f3d6b336e7ddcbff5aa
7
+ data.tar.gz: 10787c60e18aa5211d29872f0da662d9a6b98a12f1f4a35c8f37b6115046cad0d16b53e04254c827833dd412024b3173cc38eeb50daa1e7e84b30c1edf935276
data/CHANGELOG.md CHANGED
@@ -2,6 +2,7 @@
2
2
 
3
3
  ### Fixed
4
4
 
5
+ - **The unbound-lock reaper no longer counts a dead-lettered copy as a live message, so a `:until_executed` job that once dead-lettered can be enqueued again.** `Client#uniqueness_keys_present` scanned every queue in `pgmq.meta` for the payload's `pgbus_uniqueness_key` — including `*_dlq` queues. A dead-lettered copy keeps the original payload (key included) but is not in flight: the executor already released the lock when it moved the message. For a legacy `pending`/`msg_id=0` row (pre-#418 enqueue, or a bind that failed) the DLQ copy made the key look "still here" forever, the reaper kept the row, and an `on_conflict: :discard` job — a heartbeat, typically — was discarded on every enqueue until someone purged the DLQ by hand. Seen in production as three heartbeats silently dead for nine days behind a 468-message DLQ. Dead-letter queues are now skipped in the scan; bound-lock probes (`message_exists?`) were never affected because they resolve the logical queue's physical tables only.
5
6
  - **`Pgbus::Testing.disabled!` can no longer turn a Capybara teardown race into a hung test process (issue #443).** The `streams_test_mode` stub closed immediately, so a page on a stream-bearing layout had its `EventSource` reconnect every ~3s for the whole example; a reconnect landing after `disabled!` had switched test mode off — a config-level `after` hook runs *before* `Capybara.reset_sessions!` — took the real path and started a live `Streamer` (listener/dispatcher/heartbeat threads + a LISTEN connection) inside the RSpec process. Its orphaned threads then shared the test's pinned AR connection and CI died at the job timeout with only `message type 0x5a arrived from server while idle`. Three changes: the stub now emits `retry: 86400000` so the browser does not reconnect at all; `Streamer::Instance#shutdown!` snapshots every component's threads (`#threads` on Listener/Dispatcher/Heartbeat/OutboundPump/HubClient/FailoverListener), logs one error naming those still alive after their bounded joins, and returns that list (`Streamer.reset!` forwards it, `nil` when nothing was live); and `Testing.disabled!` raises `Pgbus::Testing::StreamerLeakError` — pointing at `config.append_after` — when threads leaked, while a live streamer that shut down cleanly is only logged. README and docs now recommend `config.append_after` for the teardown hook.
6
7
  - **`Pgbus::MCP.rack_app` works on a real hostname again with `mcp` 0.23+ / 1.x.** Since mcp 0.23 the `StreamableHTTPTransport` validates the `Host` header (DNS-rebinding protection, on by default, loopback hosts only) and pgbus had no way to pass the transport's options through — so a gated mount at `https://app.example.com/pgbus/mcp` answered every request `403 "Invalid Host header"`, and consumers pinned `mcp < 1.0` to dodge it (which only helps while the lock stays on 0.22). The rack app now exposes `allowed_hosts:`, `allowed_origins:` and `dns_rebinding_protection:`; the check **follows the gate by default** — off when `token:`/`auth:` is configured (a rebound browser page can never carry the bearer secret, so the check is redundant there), on for the warned-about unauthenticated mount — and `true`/`false` forces it. `mcp >= 0.23` is the floor for `rack_app` (older gems raise `Pgbus::Error` naming the fix); the gem's own bundle now tracks `mcp` 1.x, so lift that `< 1.0` pin. Stdio (`pgbus mcp`) is unaffected.
7
8
 
@@ -19,6 +20,8 @@
19
20
  - **Fair share scheduling for event-bus consumers (issue #427).** Follow-up to #426: a subscriber queue is FIFO, so a bulk import emitting `orders.created` 100 000 times for one tenant put every other tenant's events behind it in every handler subscribed to that topic. New `config.event_fair_share = ->(event) { key | [key, weight] | nil }` — the event twin of `fair_share` — receives the `Pgbus::Event` (routing key, the payload object as passed to publish, headers) at publish time on every path (`Pgbus.publish`, `publish_later`, `Pgbus::Outbox.publish_event`) and merges the same `pgbus_fair_key` / `pgbus_fair_weight` into the **event envelope** (a sibling of `event_id` / `published_at`, never inside the user payload, so `event.payload` is unchanged in handlers). Because the key lives at `message->>'pgbus_fair_key'` for events exactly as for jobs, #426's `read_batch_fair` SQL and `q_<queue>_fair_idx` index are reused verbatim; PGMQ's topic fan-out copies the tag to every bound subscriber queue, and the outbox (which stores the envelope) carries it across the relay with no extra work — a system writing `pgbus_outbox_entries` directly can set the key in the envelope JSON. `Consumer#fetch_messages` now fair-reads each active subscriber queue in list order with the remaining capacity (strict across queues, fair within; circuit-breaker-paused queues still skipped), ensures the fair index on its queues at boot, and `Subscriber#setup!` ensures it at queue creation. Independent of `fair_share` (enable either side or both); no interaction with `group_mode` (worker-only). Default-off is byte-identical. Refs #427.
20
21
  - **Fair share scheduling across tenants — weighted, work-conserving (issue #426).** A queue is FIFO, so one tenant enqueuing 100 000 jobs put every other tenant's work behind them; nothing in pgbus prevented it (multi-queue reads are strict list-order, `group_mode` serializes PGMQ FIFO groups and pgbus never set its header for jobs, `limits_concurrency` is a non-work-conserving cap). New `config.fair_share = ->(job) { key | [key, weight] | nil }` is evaluated at enqueue on every adapter path (`enqueue`, `enqueue_at`, `perform_all_later`); the key and weight ride inside the job payload as `pgbus_fair_key` / `pgbus_fair_weight` (same pattern as `pgbus_concurrency_key`, so they survive blocked-execution promotion, DLQ retry and dashboard retry). Workers then read with the new `Client#read_batch_fair` — one statement that enumerates the keys with visible messages via a loose index scan, ranks each key's oldest visible messages, and takes the `qty` lowest `rank / weight`: weight 3 vs 1 is a 3:1 split while both have work, a lone tenant still fills the whole batch, and the cost scales with the number of keys that have visible work, not backlog depth (`rake bench:fair_read`: 1.05 ms with 1 key, 4.7 ms with 200 keys, 100k-row backlog; numbers in `docs/performance.md`). Strict list-order priority across queues is preserved (fair within each queue); with `priority_levels` it is strict between levels and fair within a level; mutually exclusive with `group_mode` (`validate!` and `Worker.new` both raise). The supporting expression index `q_<queue>_fair_idx ((COALESCE(message->>'pgbus_fair_key','')), vt, msg_id)` is created at queue creation when the option is on, and built `CONCURRENTLY` by each worker for the queues it already serves (`Client#ensure_fair_index`, memoized, logs the `DROP INDEX` remediation if a concurrent build is interrupted). Within a key messages are taken oldest-visible first (`vt, msg_id`) — a deliberate deviation from `pgmq.read`'s pure `msg_id` order so a tenant's whole visible backlog is never sorted per read. Event-bus consumers get the same in #427 (below). Refs #426.
21
22
 
23
+ - **Unique batches: a run-scoped lock held from `enqueue` until the batch finishes.** `ensures_uniqueness` guards one job, but a multi-job run — an import fanned out into chunk jobs, a pipeline whose jobs enqueue the next stage — has no single job that spans the run, so a lock on the entry job releases the moment that job finishes and the next scheduler tick starts a second run on top of the first. `Pgbus::Batch.new(uniqueness_key: "perfecta-daily", on_conflict: :reject | :discard | :log)` takes a row in `pgbus_uniqueness_keys` at `#enqueue` (lock_key `batch:<key>`, queue_name `batch:<batch_id>`), keeps it through every open-batch add, and releases it in the single-winner finish (`finish_if_needed`) that also fires the callbacks — so completion, the stalled-batch sweep and cleanup all release it. A second batch with the same key raises `Pgbus::Batch::AlreadyRunning`, or skips its block and answers `discarded?`. The dispatcher's reaper judges these rows by the owning batch's status (`Batch.lock_orphaned?`: finished or missing → release) instead of scanning queues, so a nightly run may hold its lock for hours without being reaped, and a batch whose release failed is healed on the next cleanup cycle. `Batch#uniqueness_key` / `#on_conflict` / `#lock_key` readers are new; `Batch.find` handles stay non-unique (the lock belongs to the run that created it).
24
+ - **Visibility heartbeat: a running job's message no longer expires underneath it.** PGMQ hands a message to one reader for `visibility_timeout`; a job that ran longer was redelivered while still running — a second copy started, `read_ct` climbed on every redelivery, and after `max_retries` the message was dead-lettered without the job ever raising. Every long import, nightly cleanup and report job was exposed, and the usual workaround (a huge global `visibility_timeout`) made crash recovery equally slow for every queue. The executor now tracks each in-flight message with `Pgbus::VisibilityHeartbeat`: one thread per worker process re-arms the VT (`set_vt` back to `visibility_timeout`) every `visibility_heartbeat_interval` seconds — default a third of `visibility_timeout` — for as long as `perform` runs, and drops the entry before archive or the retry backoff touch the same message. The timeout therefore only fires for a process that is gone (crash, SIGKILL), which is what it exists for; `drain_timeout` still bounds what a deploy waits for. Works in both execution modes (the extension never runs inside the job's fiber), forgets the parent's entries after a fork, and is stopped by `Worker#shutdown`. New config: `visibility_heartbeat` (default `true`; `false` restores plain PGMQ semantics) and `visibility_heartbeat_interval` (seconds or Duration, must stay below `visibility_timeout`; validated at boot). Per-class opt-out: `pgbus_visibility_heartbeat false` (mixin included on `ActiveJob::Base` by the engine). New event `pgbus.job_visibility_extended` (`queue`, `job_class`, `msg_id`, `vt`, `extensions`) and counter `pgbus_visibility_extended` tagged `queue`/`job_class` — a class that shows up there is one that outlives the timeout. Event-bus handlers are not covered yet.
22
25
  - **Batches v2: open batches, a `batch` accessor on jobs and callbacks, and configured callback instances (issue #415).** ⚠️ **Breaking (pre-1.0)**: `Pgbus::Batch.find` now returns a rehydrated `Pgbus::Batch` handle instead of the raw attributes Hash — read the values off the handle (`status`, `total_jobs`, `completed_jobs`, `failed_jobs`, `pending_jobs`, `progress_percentage`, `finished?`, `description`, `properties`), or query `Pgbus::BatchEntry` for a row. Three capabilities: **(1) Open batches.** `batch.enqueue` is re-callable while the batch is unfinished — the second and later calls add to the existing group (`BatchEntry.increment_total_jobs!`, guarded on an unfinished row) instead of hitting the unique index on `batch_id`, so multi-stage workflows are possible. Adding to a finished batch raises `Pgbus::Batch::AlreadyFinished`. Execution rows are inserted as the block enqueues, before `total_jobs` is bumped, so the single-winner finish always sees outstanding work; a `check_finished!` after the bump covers a block whose jobs all completed while it was still open. Membership stays explicit: only jobs enqueued inside an `enqueue` block join the batch, so a fan-out from a batched job never silently extends it. **(2) `batch` accessor.** New `Pgbus::ActiveJob::BatchId` mixin (included on `ActiveJob::Base` from the engine alongside `Concurrency` and `Uniqueness`) adds `batch_id` / `callback_batch_id` accessors, round-trips both through `serialize`/`deserialize` — omitted from the payload when unset, so an unbatched job's serialized hash is unchanged — and exposes a memoized `batch` reader. The executor assigns `batch_id` from the payload's `pgbus_batch_id` before `perform`, so a running job can call `batch.enqueue` to add siblings. **(3) Configured callback instances.** `on_finish:` / `on_success:` / `on_failure:` accept an ActiveJob instance as well as a class: `on_finish: ReportJob.new.set(queue: :critical, wait: 5.minutes)`. `.set` options resolve at batch-creation time (matching solid_queue) into new `on_finish_job` / `on_success_job` / `on_failure_job` jsonb columns; at fire time the job is deserialized, given `callback_batch_id`, has its own `batch_id` cleared (a callback is never a member of the batch it reports on) and is enqueued on its configured queue and `scheduled_at`. Legacy class-name callbacks keep the `perform_later(properties)` signature — deprecated at 1.0 in favour of `batch.properties`. The dashboard shows a configured callback's `job_class`. Existing installs: `rails generate pgbus:add_batch_callback_jobs` (or `pgbus:update`); apps that have not migrated keep storing bare classes only. Refs #415.
23
26
 
24
27
  - **Batch completion is now self-healing via execution-row tracking (issue #414).** ⚠️ **Restart pgbus workers after `rails generate pgbus:add_batch_executions` / `db:migrate`** — a process that still writes `discarded_jobs` will error once that column is gone. `pgbus_batches` no longer finishes on `completed + discarded == total` — that arithmetic stalled a batch *forever* if a worker crashed between `archive_from` and `signal_batch_completed`, and a crash mid-`Batch#enqueue` left an immortal `pending` row (`cleanup_batches` only deletes `finished`). Each batched job now gets a `pgbus_batch_executions` row (identity = ActiveJob `job_id`) inserted *before* `send_message`, with `msg_id` + `queue_name` backfilled after send. The batch finishes when no rows remain (single-winner `UPDATE … AND NOT EXISTS`, plus a fresh `exists?` re-check for Postgres READ COMMITTED). A dispatcher sweep (`config.batch_sweep_interval`, default 5 minutes) repairs four crash windows: stale rows whose PGMQ message is gone (archived → completed, DLQ → failed, missing → completed with a warning); orphan rows with `msg_id` NULL older than 5 minutes (enqueue crashed between insert and send — blocked concurrency jobs are excluded); stalled `pending` batches; stalled `processing` batches with zero rows. `on_failure:` / `failed_jobs` are canonical (`on_discard:` / `discarded_jobs` remain deprecated aliases until 1.0). `config.batch_retention` (default 7 days, `nil` disables) replaces the hardcoded cleanup window. Existing installs: `rails generate pgbus:add_batch_executions` (or `pgbus:update`). Unmigrated apps keep the counter path via `Batch.executions_migrated?`. Refs #414.
data/README.md CHANGED
@@ -575,6 +575,30 @@ class FragileApiJob < ApplicationJob
575
575
  end
576
576
  ```
577
577
 
578
+ ### Long-running jobs: visibility heartbeat
579
+
580
+ PGMQ hands a message to one reader for `visibility_timeout` seconds. Without help, a job that runs longer is redelivered *while it is still running*: a second copy starts, `read_ct` climbs on every redelivery, and after `max_retries` the message is dead-lettered — all without the job ever raising.
581
+
582
+ Pgbus keeps that from happening. While `perform` runs, a per-process heartbeat thread re-arms the message's visibility timeout every `visibility_heartbeat_interval` seconds (default: a third of `visibility_timeout`), so the timeout only fires for a process that is actually gone — a crash or a `SIGKILL` — which is what it exists for. The heartbeat is dropped before the message is archived or the retry backoff adjusts its VT, so neither path changes.
583
+
584
+ ```ruby
585
+ Pgbus.configure do |config|
586
+ config.visibility_timeout = 10.minutes
587
+ config.visibility_heartbeat = true # default
588
+ config.visibility_heartbeat_interval = 2.minutes # default: visibility_timeout / 3
589
+ end
590
+ ```
591
+
592
+ Opt a job class out (a job that must be redelivered if it stalls, for example):
593
+
594
+ ```ruby
595
+ class WatchdogJob < ApplicationJob
596
+ pgbus_visibility_heartbeat false
597
+ end
598
+ ```
599
+
600
+ Each extension emits `pgbus.job_visibility_extended` and increments the `pgbus_visibility_extended` counter (tags: `queue`, `job_class`) — a job class that shows up there is one that outlives `visibility_timeout`, so you can size the timeout from data instead of guessing. `drain_timeout` still bounds what a graceful shutdown waits for; a job still running past it is redelivered after the timeout, as before. Event-bus handlers are not covered yet.
601
+
578
602
  ### Async execution mode (fibers)
579
603
 
580
604
  Workers can optionally execute jobs as fibers instead of threads. This is ideal for I/O-bound workloads (HTTP calls, email delivery, LLM API calls) where jobs spend most of their time waiting on network I/O.
@@ -743,6 +767,24 @@ class BatchFinishedJob < ApplicationJob
743
767
  end
744
768
  ```
745
769
 
770
+ #### Unique batches (run-scoped locks)
771
+
772
+ `ensures_uniqueness` protects one job. A multi-job run — a nightly import that fans out into hundreds of chunk jobs, a pipeline whose jobs enqueue the next stage — has no single job that lives for the whole run, so a lock on the entry job releases the moment that job finishes and the next scheduler tick starts a second run on top of the first. Give the batch the lock instead:
773
+
774
+ ```ruby
775
+ batch = Pgbus::Batch.new(
776
+ uniqueness_key: "perfecta-daily",
777
+ on_conflict: :discard, # :reject (raise AlreadyRunning), :discard, :log
778
+ on_success: ExportPricesJob
779
+ )
780
+
781
+ batch.enqueue do
782
+ ImportChunkJob.perform_later("products", 0)
783
+ end
784
+ ```
785
+
786
+ The lock is taken at `enqueue` and held until the batch finishes — through every open-batch `batch.enqueue` add from inside a running job — then released in the same single-winner finish that fires the callbacks, so completion, the stalled-batch sweep and cleanup all release it. While it is held, another `Pgbus::Batch.new(uniqueness_key: "perfecta-daily").enqueue { ... }` raises `Pgbus::Batch::AlreadyRunning` (`:reject`), or skips its block and returns a batch that answers `discarded?` (`:discard`, `:log`). The row lives in `pgbus_uniqueness_keys` as `batch:<key>`, so it never collides with a job's key and shows up in the dashboard like any other lock; the reaper judges it by the batch's status, never by queue contents.
787
+
746
788
  #### How batches work
747
789
 
748
790
  1. `Batch.new(...)` creates a tracking row in `pgbus_batches` with `status: "pending"`
@@ -907,6 +949,7 @@ c.metrics_backend = MyOpenTelemetryBackend.new
907
949
  | Metric | Type | Tags |
908
950
  |--------|------|------|
909
951
  | `pgbus_queue_job_count` | counter | `queue`, `job_class`, `status` (`processed`/`failed`/`dead_lettered`) |
952
+ | `pgbus_visibility_extended` | counter | `queue`, `job_class` — one per re-armed visibility timeout; a class listed here outlives `visibility_timeout` |
910
953
  | `pgbus_job_duration_ms` | histogram | `queue`, `job_class` |
911
954
  | `pgbus_event_count` | counter | `handler`, `routing_key`, `status` |
912
955
  | `pgbus_event_duration_ms` | histogram | `handler`, `routing_key` |
@@ -2199,6 +2242,8 @@ Curated headline options for the README. The full operator reference (with types
2199
2242
  | `roles` | `nil` (all) | Supervisor role filter — usually set via CLI flags (`--workers-only` etc.) |
2200
2243
  | `polling_interval` | `0.1` | Seconds between polls (LISTEN/NOTIFY is primary) |
2201
2244
  | `visibility_timeout` | `30` | Time before unacked message becomes visible again. Accepts seconds or `ActiveSupport::Duration` (e.g. `10.minutes`) |
2245
+ | `visibility_heartbeat` | `true` | Re-arm a running job's visibility timeout while `perform` runs, so long jobs are not redelivered or dead-lettered mid-run. `false` restores plain PGMQ semantics |
2246
+ | `visibility_heartbeat_interval` | `nil` (`visibility_timeout / 3`) | Seconds (or Duration) between two extensions; must stay below `visibility_timeout` |
2202
2247
  | `max_retries` | `5` | Failed reads before routing to dead letter queue |
2203
2248
  | `retry_backoff` | `5` | Base delay in seconds for VT-based retry backoff (exponential: `base * 2^(attempt-1)`) |
2204
2249
  | `retry_backoff_max` | `300` | Maximum retry delay in seconds (caps the exponential curve) |
@@ -22,6 +22,7 @@ search:
22
22
  exclude:
23
23
  - "*.md"
24
24
  - app/assets/images
25
+ - app/frontend/pgbus/vendor
25
26
  strict: false
26
27
 
27
28
  translation:
@@ -168,8 +168,6 @@ da:
168
168
  reroute_failed: Kunne ikke omdirigere begivenhed.
169
169
  rerouted: Begivenhed omdirigeret til målhandler.
170
170
  index:
171
- discard_all: Forkast alle
172
- discard_all_confirm: Forkast alle ventende begivenheder? Dette kan ikke fortrydes.
173
171
  discard_selected: Forkast valgte
174
172
  discard_selected_confirm: Forkast valgte begivenheder?
175
173
  pending_empty: Ingen ventende begivenheder
@@ -168,8 +168,6 @@ de:
168
168
  reroute_failed: Ereignis konnte nicht umgeleitet werden.
169
169
  rerouted: Ereignis an Ziel-Handler umgeleitet.
170
170
  index:
171
- discard_all: Alle verwerfen
172
- discard_all_confirm: Alle ausstehenden Ereignisse verwerfen? Dies kann nicht rückgängig gemacht werden.
173
171
  discard_selected: Ausgewählte verwerfen
174
172
  discard_selected_confirm: Ausgewählte Ereignisse verwerfen?
175
173
  pending_empty: Keine ausstehenden Ereignisse
@@ -168,8 +168,6 @@ en:
168
168
  reroute_failed: Could not reroute event.
169
169
  rerouted: Event rerouted to target handler.
170
170
  index:
171
- discard_all: Discard All
172
- discard_all_confirm: Discard all pending events? This cannot be undone.
173
171
  discard_selected: Discard Selected
174
172
  discard_selected_confirm: Discard selected events?
175
173
  pending_empty: No pending events
@@ -168,8 +168,6 @@ es:
168
168
  reroute_failed: No se pudo redirigir el evento.
169
169
  rerouted: Evento redirigido al manejador objetivo.
170
170
  index:
171
- discard_all: Descartar todo
172
- discard_all_confirm: "¿Descartar todos los eventos pendientes? Esto no se puede deshacer."
173
171
  discard_selected: Descartar seleccionados
174
172
  discard_selected_confirm: "¿Descartar eventos seleccionados?"
175
173
  pending_empty: No hay eventos pendientes
@@ -168,8 +168,6 @@ fi:
168
168
  reroute_failed: Tapahtumaa ei voitu uudelleenreitittää.
169
169
  rerouted: Tapahtuma uudelleenreititetty kohdekäsittelijälle.
170
170
  index:
171
- discard_all: Hylkää kaikki
172
- discard_all_confirm: Hylätäänkö kaikki odottavat tapahtumat? Tätä ei voi peruuttaa.
173
171
  discard_selected: Hylkää valitut
174
172
  discard_selected_confirm: Hylätäänkö valitut tapahtumat?
175
173
  pending_empty: Ei odottavia tapahtumia
@@ -168,8 +168,6 @@ fr:
168
168
  reroute_failed: Impossible de rediriger l'événement.
169
169
  rerouted: Événement redirigé vers le gestionnaire cible.
170
170
  index:
171
- discard_all: Tout jeter
172
- discard_all_confirm: Jeter tous les événements en attente ? Cette action est irréversible.
173
171
  discard_selected: Jeter la sélection
174
172
  discard_selected_confirm: Jeter les événements sélectionnés ?
175
173
  pending_empty: Aucun événement en attente
@@ -168,8 +168,6 @@ it:
168
168
  reroute_failed: Impossibile reindirizzare l'evento.
169
169
  rerouted: Evento reindirizzato al gestore di destinazione.
170
170
  index:
171
- discard_all: Scarta tutto
172
- discard_all_confirm: Scartare tutti gli eventi in sospeso? Questa operazione non può essere annullata.
173
171
  discard_selected: Scarta selezionati
174
172
  discard_selected_confirm: Scartare gli eventi selezionati?
175
173
  pending_empty: Nessun evento in sospeso
@@ -168,8 +168,6 @@ ja:
168
168
  reroute_failed: イベントをルーティングできませんでした。
169
169
  rerouted: イベントがターゲットハンドラーにルーティングされました。
170
170
  index:
171
- discard_all: すべて破棄
172
- discard_all_confirm: 保留中のすべてのイベントを破棄しますか?これは元に戻せません。
173
171
  discard_selected: 選択を破棄
174
172
  discard_selected_confirm: 選択したイベントを破棄しますか?
175
173
  pending_empty: 保留中のイベントはありません
@@ -168,8 +168,6 @@ nb:
168
168
  reroute_failed: Kunne ikke omdirigere hendelse.
169
169
  rerouted: Hendelse omdirigert til målbehandler.
170
170
  index:
171
- discard_all: Forkast alle
172
- discard_all_confirm: Forkast alle ventende hendelser? Dette kan ikke angres.
173
171
  discard_selected: Forkast valgte
174
172
  discard_selected_confirm: Forkast valgte hendelser?
175
173
  pending_empty: Ingen ventende hendelser
@@ -168,8 +168,6 @@ nl:
168
168
  reroute_failed: Kon gebeurtenis niet omleiden.
169
169
  rerouted: Gebeurtenis omgeleid naar doelhandler.
170
170
  index:
171
- discard_all: Alles Verwijderen
172
- discard_all_confirm: Alle openstaande gebeurtenissen verwijderen? Dit kan niet ongedaan worden gemaakt.
173
171
  discard_selected: Geselecteerde Verwijderen
174
172
  discard_selected_confirm: Geselecteerde gebeurtenissen verwijderen?
175
173
  pending_empty: Geen openstaande gebeurtenissen
@@ -168,8 +168,6 @@ pt:
168
168
  reroute_failed: Não foi possível redirecionar o evento.
169
169
  rerouted: Evento redirecionado para o manipulador alvo.
170
170
  index:
171
- discard_all: Descartar Todos
172
- discard_all_confirm: Descartar todos os eventos pendentes? Esta ação não pode ser desfeita.
173
171
  discard_selected: Descartar Selecionados
174
172
  discard_selected_confirm: Descartar eventos selecionados?
175
173
  pending_empty: Nenhum evento pendente
@@ -168,8 +168,6 @@ sv:
168
168
  reroute_failed: Kunde inte omdirigera händelsen.
169
169
  rerouted: Händelsen omdirigerades till målhanteraren.
170
170
  index:
171
- discard_all: Kassera alla
172
- discard_all_confirm: Kassera alla väntande händelser? Detta kan inte ångras.
173
171
  discard_selected: Kassera valda
174
172
  discard_selected_confirm: Kassera valda händelser?
175
173
  pending_empty: Inga väntande händelser
@@ -96,7 +96,7 @@ module Pgbus
96
96
  # `batch` (and `batch.enqueue` for open batches) work inside a job.
97
97
  assign_batch_id(job, payload)
98
98
  Pgbus.logger.debug { "[Pgbus::Executor] running #{tag} job_class=#{job_class}" }
99
- execute_job(job)
99
+ with_visibility_heartbeat(job, queue_name, msg_id, source_queue) { execute_job(job) }
100
100
  # retry_on re-enqueues from inside perform_now and returns normally:
101
101
  # this attempt is done (archive it) but the job is not — the retry
102
102
  # message carries the batch tag and signals on its own outcome.
@@ -174,6 +174,20 @@ module Pgbus
174
174
  Pgbus.logger.warn { "[Pgbus] Uniqueness release failed: #{e.message}" }
175
175
  end
176
176
 
177
+ # Keep the message invisible while perform runs (see VisibilityHeartbeat).
178
+ # Wraps only the perform: the heartbeat must be gone before archive or
179
+ # the retry backoff touches the same message's VT.
180
+ def with_visibility_heartbeat(job, queue_name, msg_id, source_queue, &)
181
+ klass = job.class
182
+ per_job = klass.respond_to?(:pgbus_visibility_heartbeat_enabled) ? klass.pgbus_visibility_heartbeat_enabled : nil
183
+ return yield if per_job == false
184
+
185
+ VisibilityHeartbeat.track(
186
+ client: client, queue_name: source_queue || queue_name, prefixed: source_queue.nil?,
187
+ msg_id: msg_id, job_class: klass.name, config: config, &
188
+ )
189
+ end
190
+
177
191
  def execute_job(job)
178
192
  if defined?(Rails) && Rails.respond_to?(:application) && Rails.application
179
193
  wrapper = reloading? ? Rails.application.reloader : Rails.application.executor
data/lib/pgbus/batch.rb CHANGED
@@ -6,18 +6,39 @@ require "json"
6
6
  module Pgbus
7
7
  class Batch
8
8
  class AlreadyFinished < Error; end
9
+ # Raised by #enqueue when a batch with the same uniqueness_key is still
10
+ # running and on_conflict: is :reject.
11
+ class AlreadyRunning < Error; end
9
12
 
10
13
  METADATA_KEY = "pgbus_batch_id"
11
14
 
15
+ # A unique batch holds one row in pgbus_uniqueness_keys from #enqueue until
16
+ # the batch finishes. The row's lock_key carries the caller's key under
17
+ # this prefix (so a job's ensures_uniqueness key never collides with it)
18
+ # and its queue_name names the owning batch, which is how the reaper
19
+ # tells a live run from an orphan (see .lock_orphaned?).
20
+ LOCK_KEY_PREFIX = "batch:"
21
+ LOCK_QUEUE_PREFIX = "batch:"
22
+ VALID_CONFLICTS = %i[reject discard log].freeze
23
+
12
24
  attr_reader :batch_id, :properties, :description,
13
- :on_finish, :on_success, :on_failure
25
+ :on_finish, :on_success, :on_failure,
26
+ :uniqueness_key, :on_conflict
14
27
 
15
28
  def on_discard
16
29
  on_failure
17
30
  end
18
31
 
19
- def initialize(on_finish: nil, on_success: nil, on_discard: nil, on_failure: nil, description: nil, properties: {})
32
+ # @param uniqueness_key [String, nil] at most one unfinished batch with
33
+ # this key may exist; see #enqueue for what happens to the next one
34
+ # @param on_conflict [Symbol] :reject (raise AlreadyRunning), :discard
35
+ # (skip the block silently) or :log (skip the block, warn)
36
+ def initialize(on_finish: nil, on_success: nil, on_discard: nil, on_failure: nil, description: nil, properties: {},
37
+ uniqueness_key: nil, on_conflict: :reject)
20
38
  raise ArgumentError, "pass on_failure: only — on_discard: is a deprecated alias" if on_discard && on_failure
39
+ unless VALID_CONFLICTS.include?(on_conflict)
40
+ raise ArgumentError, "on_conflict must be one of #{VALID_CONFLICTS.join(", ")}, got #{on_conflict.inspect}"
41
+ end
21
42
 
22
43
  if on_discard
23
44
  Pgbus.logger.warn do
@@ -31,9 +52,20 @@ module Pgbus
31
52
  @on_failure = on_failure || on_discard
32
53
  @description = description
33
54
  @properties = properties
55
+ @uniqueness_key = uniqueness_key&.to_s
56
+ @on_conflict = on_conflict
57
+ @discarded = false
34
58
  @started = false
35
59
  end
36
60
 
61
+ # True when #enqueue found another batch with the same uniqueness_key
62
+ # still running and skipped this one (on_conflict: :discard or :log).
63
+ def discarded? = @discarded
64
+
65
+ def lock_key
66
+ "#{LOCK_KEY_PREFIX}#{uniqueness_key}" if uniqueness_key
67
+ end
68
+
37
69
  # Enqueue a group of jobs as a batch. Jobs enqueued inside the block join
38
70
  # this batch.
39
71
  #
@@ -45,8 +77,14 @@ module Pgbus
45
77
  # Raises Pgbus::Batch::AlreadyFinished once the batch has finished.
46
78
  def enqueue(&)
47
79
  return reopen(&) if @started
80
+ return self unless acquire_lock!
48
81
 
49
- create_record
82
+ begin
83
+ create_record
84
+ rescue StandardError
85
+ self.class.release_lock(batch_id)
86
+ raise
87
+ end
50
88
  @started = true
51
89
  count_jobs(&)
52
90
  start_processing
@@ -134,6 +172,43 @@ module Pgbus
134
172
  BatchEntry.stale(before: older_than).delete_all
135
173
  end
136
174
 
175
+ # --- run-scoped uniqueness lock ------------------------------------
176
+
177
+ # queue_name stored on the uniqueness row of a unique batch.
178
+ def self.lock_queue_name(batch_id)
179
+ "#{LOCK_QUEUE_PREFIX}#{batch_id}"
180
+ end
181
+
182
+ # True for a pgbus_uniqueness_keys row that belongs to a batch rather
183
+ # than to a message.
184
+ def self.lock_row?(queue_name)
185
+ queue_name.to_s.start_with?(LOCK_QUEUE_PREFIX)
186
+ end
187
+
188
+ # For the dispatcher's reaper: a batch lock is an orphan once its batch
189
+ # has finished (the release in finish_if_needed failed) or its row is
190
+ # gone (cleanup). A batch that is still pending/processing keeps the lock
191
+ # no matter how old it is — a nightly run legitimately holds it for
192
+ # hours. Any lookup error keeps the lock; the reaper never deletes in
193
+ # doubt.
194
+ def self.lock_orphaned?(queue_name)
195
+ batch_id = queue_name.to_s.delete_prefix(LOCK_QUEUE_PREFIX)
196
+ record = BatchEntry.find_by(batch_id: batch_id)
197
+ record.nil? || record.status == "finished"
198
+ rescue StandardError => e
199
+ Pgbus.logger.debug { "[Pgbus] Batch lock lookup failed for #{queue_name}: #{e.message}" }
200
+ false
201
+ end
202
+
203
+ # Drop the uniqueness row a batch holds, if any. Fail-soft: the reaper
204
+ # releases the row once the batch is finished.
205
+ def self.release_lock(batch_id)
206
+ UniquenessKey.where(queue_name: lock_queue_name(batch_id)).delete_all
207
+ rescue StandardError => e
208
+ Pgbus.logger.debug { "[Pgbus] Batch lock release failed for #{batch_id}: #{e.message}" }
209
+ 0
210
+ end
211
+
137
212
  def self.executions_migrated?
138
213
  return true if @executions_migrated
139
214
 
@@ -329,6 +404,9 @@ module Pgbus
329
404
 
330
405
  fire_callbacks(result[:record])
331
406
  instrument_finished(result[:record])
407
+ # Every finish path (completion, sweep) funnels through here, so this
408
+ # is the one place a unique batch gives its run lock back.
409
+ release_lock(result[:record].batch_id) if result[:record].respond_to?(:batch_id)
332
410
  result
333
411
  end
334
412
 
@@ -438,9 +516,34 @@ module Pgbus
438
516
  @on_finish = nil
439
517
  @on_success = nil
440
518
  @on_failure = nil
519
+ @uniqueness_key = nil
520
+ @on_conflict = :reject
521
+ @discarded = false
441
522
  @started = true
442
523
  end
443
524
 
525
+ # Take the run lock for a unique batch. Returns true when the batch may
526
+ # proceed (no key, or the lock was won); false when another batch holds
527
+ # the key and on_conflict is :discard or :log. :reject raises.
528
+ def acquire_lock! # rubocop:disable Naming/PredicateMethod
529
+ return true unless uniqueness_key
530
+
531
+ acquired = UniquenessKey.acquire!(lock_key, queue_name: self.class.lock_queue_name(batch_id), msg_id: 0)
532
+ UniquenessKey.clear_bind_stamp!(lock_key)
533
+ return true if acquired
534
+
535
+ case on_conflict
536
+ when :reject
537
+ raise AlreadyRunning, "Batch #{uniqueness_key.inspect} is already running"
538
+ when :discard
539
+ Pgbus.logger.info { "[Pgbus] Discarding batch #{uniqueness_key.inspect}: a batch with that key is still running" }
540
+ else
541
+ Pgbus.logger.warn { "[Pgbus] Batch #{uniqueness_key.inspect} skipped: a batch with that key is still running" }
542
+ end
543
+ @discarded = true
544
+ false
545
+ end
546
+
444
547
  # Add to an already-created batch. Each job counts itself in (guarded
445
548
  # increment + execution row, see .track_enqueue) as it is enqueued, so an
446
549
  # add into a finished batch raises at perform_later before anything is
data/lib/pgbus/client.rb CHANGED
@@ -821,6 +821,13 @@ module Pgbus
821
821
  .map { |row| row["queue_name"] }
822
822
  names.each do |name|
823
823
  break if found.size == keys.size
824
+ # A dead-lettered copy still carries the payload's uniqueness key,
825
+ # but it is not in flight: the executor released the lock when it
826
+ # moved the message. Counting it as "present" would pin an unbound
827
+ # lock for as long as the DLQ row exists, and a `:until_executed` +
828
+ # `on_conflict: :discard` job would then be discarded on every
829
+ # enqueue until someone purged the DLQ by hand.
830
+ next if name.end_with?(Pgbus::DEAD_LETTER_SUFFIX)
824
831
 
825
832
  sanitized = begin
826
833
  QueueNameValidator.sanitize!(name)
@@ -12,7 +12,14 @@ module Pgbus
12
12
 
13
13
  # Worker settings
14
14
  attr_accessor :polling_interval, :prefetch_limit, :execution_mode
15
- attr_reader :workers, :visibility_timeout # rubocop:disable Style/AccessorGrouping
15
+ # visibility_heartbeat / visibility_heartbeat_interval: while a job runs,
16
+ # its message's visibility timeout is re-armed every interval seconds
17
+ # (default: a third of visibility_timeout), so a job that outlives the
18
+ # timeout is not redelivered — and eventually dead-lettered — while it is
19
+ # still running. The timeout then only fires for a process that is gone.
20
+ # Set visibility_heartbeat = false to restore plain PGMQ semantics.
21
+ attr_accessor :visibility_heartbeat
22
+ attr_reader :workers, :visibility_timeout, :visibility_heartbeat_interval # rubocop:disable Style/AccessorGrouping
16
23
 
17
24
  # Supervisor role selection.
18
25
  # nil = boot all roles (default behavior).
@@ -266,6 +273,8 @@ module Pgbus
266
273
  @roles = nil
267
274
  @polling_interval = 0.1
268
275
  @visibility_timeout = 30
276
+ @visibility_heartbeat = true
277
+ @visibility_heartbeat_interval = nil
269
278
 
270
279
  @prefetch_limit = nil
271
280
  @execution_mode = :threads
@@ -762,6 +771,7 @@ module Pgbus
762
771
  raise Pgbus::ConfigurationError,
763
772
  "visibility_timeout must be > 0"
764
773
  end
774
+ validate_visibility_heartbeat!
765
775
  raise Pgbus::ConfigurationError, "max_retries must be >= 0" unless max_retries.is_a?(Integer) && max_retries >= 0
766
776
  raise Pgbus::ConfigurationError, "retry_backoff must be > 0" unless retry_backoff.is_a?(Numeric) && retry_backoff.positive?
767
777
  unless retry_backoff_max.is_a?(Numeric) && retry_backoff_max.positive?
@@ -1230,6 +1240,27 @@ module Pgbus
1230
1240
  # Numeric values are stored unchanged (preserving Float for sub-second
1231
1241
  # values). Duration values are coerced to Integer seconds via .to_i.
1232
1242
 
1243
+ def visibility_heartbeat_interval=(value)
1244
+ @visibility_heartbeat_interval = coerce_duration!(value, :visibility_heartbeat_interval)
1245
+ end
1246
+
1247
+ # Seconds between two extensions of a running job's visibility timeout:
1248
+ # the configured interval, or a third of visibility_timeout.
1249
+ def effective_visibility_heartbeat_interval
1250
+ visibility_heartbeat_interval || (visibility_timeout / 3.0)
1251
+ end
1252
+
1253
+ def validate_visibility_heartbeat!
1254
+ raise Pgbus::ConfigurationError, "visibility_heartbeat must be true or false" unless [true, false].include?(visibility_heartbeat)
1255
+ return if visibility_heartbeat_interval.nil?
1256
+
1257
+ interval = visibility_heartbeat_interval
1258
+ return if interval.is_a?(Numeric) && interval.positive? && interval < visibility_timeout
1259
+
1260
+ raise Pgbus::ConfigurationError,
1261
+ "visibility_heartbeat_interval must be > 0 and below visibility_timeout (#{visibility_timeout}s)"
1262
+ end
1263
+
1233
1264
  def visibility_timeout=(value)
1234
1265
  @visibility_timeout = coerce_duration!(value, :visibility_timeout)
1235
1266
  end
data/lib/pgbus/engine.rb CHANGED
@@ -76,6 +76,7 @@ module Pgbus
76
76
  include Pgbus::Uniqueness
77
77
  include Pgbus::ActiveJob::BatchId
78
78
  include Pgbus::ActiveJob::CurrentAttributes
79
+ include Pgbus::VisibilityHeartbeat::JobMixin
79
80
  end
80
81
  end
81
82
 
@@ -16,6 +16,8 @@ module Pgbus
16
16
  # pgbus.job_completed — job archived successfully
17
17
  # pgbus.job_failed — job raised; carries :exception_object
18
18
  # pgbus.job_dead_lettered — job exceeded max_retries and was DLQ-routed
19
+ # pgbus.job_visibility_extended — heartbeat re-armed a running job's visibility timeout
20
+ # payload: queue, job_class, msg_id, vt, extensions
19
21
  # pgbus.event_processed — event handler succeeded
20
22
  # pgbus.event_failed — event handler raised; carries :exception_object
21
23
  # pgbus.stream.broadcast — stream broadcast (sync or deferred)
@@ -30,6 +30,7 @@ module Pgbus
30
30
  subscribe("pgbus.job_completed") { |event| on_job_completed(event) },
31
31
  subscribe("pgbus.job_failed") { |event| on_job_failed(event) },
32
32
  subscribe("pgbus.job_dead_lettered") { |event| on_job_dead_lettered(event) },
33
+ subscribe("pgbus.job_visibility_extended") { |event| on_job_visibility_extended(event) },
33
34
  subscribe("pgbus.event_processed") { |event| on_event_processed(event) },
34
35
  subscribe("pgbus.event_failed") { |event| on_event_failed(event) },
35
36
  subscribe("pgbus.client.send_message") { |event| on_send_message(event) },
@@ -111,6 +112,16 @@ module Pgbus
111
112
  )
112
113
  end
113
114
 
115
+ # One increment per re-armed visibility timeout: a job class that
116
+ # shows up here is one that outlives visibility_timeout.
117
+ def on_job_visibility_extended(event)
118
+ payload = event.payload
119
+ backend.increment(
120
+ "#{METRIC_PREFIX}visibility_extended", 1,
121
+ compact(queue: payload[:queue], job_class: payload[:job_class])
122
+ )
123
+ end
124
+
114
125
  # ── Event handler ─────────────────────────────────────────────────
115
126
 
116
127
  def on_event_processed(event)
@@ -461,8 +461,14 @@ module Pgbus
461
461
  candidates = keys.select { |key| key.created_at && key.created_at < threshold && key.queue_name }
462
462
  return 0 if candidates.empty?
463
463
 
464
+ # A unique batch's lock is judged by its batch, not by any queue: it
465
+ # is held for the whole run (hours, legitimately) and is an orphan
466
+ # only once the batch has finished or its row is gone.
467
+ batch_locks, candidates = candidates.partition { |key| Batch.lock_row?(key.queue_name) }
468
+ orphaned = batch_locks.select { |key| Batch.lock_orphaned?(key.queue_name) }
469
+
464
470
  bound, unbound = candidates.partition { |key| bound_lock?(key) }
465
- orphaned = bound.select { |key| message_gone?(key) }
471
+ orphaned.concat(bound.select { |key| message_gone?(key) })
466
472
  orphaned.concat(gone_unbound_locks(unbound))
467
473
 
468
474
  return 0 if orphaned.empty?
@@ -844,6 +844,7 @@ module Pgbus
844
844
  # proven it won't finish; waiting another full window would push the
845
845
  # worker past the supervisor's shutdown_timeout deadline (issue #386).
846
846
  @pool.wait_for_termination(POOL_TERMINATION_WAIT)
847
+ VisibilityHeartbeat.stop
847
848
  @stat_buffer&.stop
848
849
  @queue_lock&.unlock_all
849
850
  @heartbeat&.stop
data/lib/pgbus/version.rb CHANGED
@@ -1,5 +1,5 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  module Pgbus
4
- VERSION = "0.15.4"
4
+ VERSION = "0.16.0"
5
5
  end
@@ -0,0 +1,194 @@
1
+ # frozen_string_literal: true
2
+
3
+ module Pgbus
4
+ # Keeps a running job's PGMQ message invisible for as long as the job is
5
+ # actually running.
6
+ #
7
+ # PGMQ hands a message to one reader for `visibility_timeout` seconds. A job
8
+ # that runs longer is redelivered while it is still running: a second copy
9
+ # starts, `read_ct` climbs on every redelivery, and after `max_retries` the
10
+ # message is dead-lettered — all without the job ever raising. The heartbeat
11
+ # extends the VT of every in-flight message on a fixed cadence, so the
12
+ # timeout only ever fires for a job whose process is gone (crash, SIGKILL),
13
+ # which is the case it exists for.
14
+ #
15
+ # One background thread per process, started lazily by the first tracked
16
+ # job and stopped by Worker#shutdown. Entries are keyed by physical queue +
17
+ # msg_id and hold the client that read the message, so async and threaded
18
+ # execution modes both work: the extension runs on this thread, never
19
+ # inside the job's fiber. A fork forgets the parent's entries.
20
+ #
21
+ # Pgbus::VisibilityHeartbeat.track(client:, queue_name:, msg_id:) { job.perform_now }
22
+ #
23
+ # Disable globally with `config.visibility_heartbeat = false`, tune the
24
+ # cadence with `config.visibility_heartbeat_interval`, or opt a job class
25
+ # out with `pgbus_visibility_heartbeat false`.
26
+ module VisibilityHeartbeat
27
+ Entry = Struct.new(:client, :queue_name, :prefixed, :msg_id, :job_class, :extended_at, :extensions,
28
+ keyword_init: true)
29
+
30
+ # Per-job opt-out, included on ActiveJob::Base by the engine:
31
+ #
32
+ # class ShortJob < ApplicationJob
33
+ # pgbus_visibility_heartbeat false
34
+ # end
35
+ module JobMixin
36
+ extend ActiveSupport::Concern
37
+
38
+ included do
39
+ class_attribute :pgbus_visibility_heartbeat_enabled, instance_writer: false, default: nil
40
+ end
41
+
42
+ class_methods do
43
+ def pgbus_visibility_heartbeat(enabled = true) # rubocop:disable Style/OptionalBooleanParameter
44
+ self.pgbus_visibility_heartbeat_enabled = enabled ? true : false
45
+ end
46
+ end
47
+ end
48
+
49
+ class << self
50
+ # Track the message for the duration of the block.
51
+ #
52
+ # @param client [Pgbus::Client] the client that read the message
53
+ # @param queue_name [String] logical name, or physical when prefixed: false
54
+ # @param msg_id [Integer]
55
+ # @param prefixed [Boolean] whether queue_name still needs the prefix
56
+ # @param job_class [String, nil] for logging and instrumentation
57
+ # @param config [Pgbus::Configuration]
58
+ def track(client:, queue_name:, msg_id:, prefixed: true, job_class: nil, config: Pgbus.configuration)
59
+ return yield unless config.visibility_heartbeat
60
+
61
+ entry = Entry.new(client: client, queue_name: queue_name, prefixed: prefixed, msg_id: msg_id.to_i,
62
+ job_class: job_class, extended_at: monotonic_now, extensions: 0)
63
+ register(entry, config)
64
+ begin
65
+ yield
66
+ ensure
67
+ unregister(entry)
68
+ end
69
+ end
70
+
71
+ # Extend every tracked message whose last extension is older than the
72
+ # heartbeat interval. Public so tests and callers without the thread
73
+ # can drive it.
74
+ def tick!(now: monotonic_now, config: Pgbus.configuration)
75
+ interval = config.effective_visibility_heartbeat_interval
76
+ due = synchronize { entries.values.select { |entry| now - entry.extended_at >= interval } }
77
+ due.each { |entry| extend!(entry, now: now, config: config) }
78
+ due.size
79
+ end
80
+
81
+ def tracked_count
82
+ synchronize { entries.size }
83
+ end
84
+
85
+ # Stop the background thread. Tracked entries are kept: a job still
86
+ # running during shutdown can drive tick! itself, and Worker#shutdown
87
+ # only calls this once the pool has drained.
88
+ def stop
89
+ thread = synchronize do
90
+ @running = false
91
+ current = @thread
92
+ @thread = nil
93
+ current
94
+ end
95
+ return unless thread
96
+
97
+ thread.wakeup if thread.alive?
98
+ thread.join(1)
99
+ end
100
+
101
+ # Forget every entry and stop the thread. Test helper.
102
+ def reset!
103
+ stop
104
+ synchronize { @entries = {} }
105
+ end
106
+
107
+ private
108
+
109
+ def register(entry, config)
110
+ synchronize do
111
+ forget_parent_entries!
112
+ entries[key_for(entry)] = entry
113
+ ensure_thread(config)
114
+ end
115
+ end
116
+
117
+ def unregister(entry)
118
+ synchronize { entries.delete(key_for(entry)) }
119
+ end
120
+
121
+ def extend!(entry, now:, config:)
122
+ vt = config.visibility_timeout
123
+ entry.client.set_visibility_timeout(entry.queue_name, entry.msg_id, vt: vt, prefixed: entry.prefixed)
124
+ entry.extended_at = now
125
+ entry.extensions += 1
126
+ Instrumentation.instrument(
127
+ "pgbus.job_visibility_extended",
128
+ queue: entry.queue_name, job_class: entry.job_class, msg_id: entry.msg_id, vt: vt,
129
+ extensions: entry.extensions
130
+ )
131
+ Pgbus.logger.debug do
132
+ "[Pgbus::VisibilityHeartbeat] extended msg_id=#{entry.msg_id} queue=#{entry.queue_name} " \
133
+ "job_class=#{entry.job_class} vt=#{vt} extensions=#{entry.extensions}"
134
+ end
135
+ rescue StandardError => e
136
+ # The next tick retries; the message simply keeps its current VT.
137
+ Pgbus.logger.warn do
138
+ "[Pgbus::VisibilityHeartbeat] could not extend msg_id=#{entry.msg_id} queue=#{entry.queue_name}: " \
139
+ "#{e.class}: #{e.message}"
140
+ end
141
+ end
142
+
143
+ # Caller holds the mutex. Start the ticker once per process.
144
+ def ensure_thread(config)
145
+ return if @running && @thread&.alive?
146
+
147
+ @running = true
148
+ @thread = Thread.new { run_loop(config) }
149
+ @thread.name = "pgbus-visibility-heartbeat"
150
+ end
151
+
152
+ def run_loop(config)
153
+ while @running
154
+ # Half the interval keeps every extension inside [interval, 1.5 * interval]
155
+ # of the previous one — at most half the visibility timeout.
156
+ sleep([config.effective_visibility_heartbeat_interval / 2.0, 0.05].max)
157
+ break unless @running
158
+
159
+ tick!(config: config)
160
+ end
161
+ rescue StandardError => e
162
+ Pgbus.logger.error { "[Pgbus::VisibilityHeartbeat] ticker died: #{e.class}: #{e.message}" }
163
+ synchronize { @running = false }
164
+ end
165
+
166
+ # Entries registered before a fork belong to the parent's jobs; the
167
+ # thread did not survive the fork either.
168
+ def forget_parent_entries!
169
+ return if @pid == ::Process.pid
170
+
171
+ @pid = ::Process.pid
172
+ @entries = {}
173
+ @running = false
174
+ @thread = nil
175
+ end
176
+
177
+ def entries
178
+ @entries ||= {}
179
+ end
180
+
181
+ def key_for(entry)
182
+ [entry.queue_name, entry.msg_id]
183
+ end
184
+
185
+ def synchronize(&)
186
+ (@mutex ||= Mutex.new).synchronize(&)
187
+ end
188
+
189
+ def monotonic_now
190
+ ::Process.clock_gettime(::Process::CLOCK_MONOTONIC)
191
+ end
192
+ end
193
+ end
194
+ end
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: pgbus
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.15.4
4
+ version: 0.16.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - Mikael Henriksson
@@ -394,6 +394,7 @@ files:
394
394
  - lib/pgbus/testing/rspec.rb
395
395
  - lib/pgbus/uniqueness.rb
396
396
  - lib/pgbus/version.rb
397
+ - lib/pgbus/visibility_heartbeat.rb
397
398
  - lib/pgbus/web/authentication.rb
398
399
  - lib/pgbus/web/data_source.rb
399
400
  - lib/pgbus/web/health_app.rb