pgbus 0.16.5 → 0.16.7
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +9 -0
- data/README.md +8 -0
- data/lib/pgbus/event_bus/claim_beat.rb +80 -0
- data/lib/pgbus/event_bus/handler.rb +137 -34
- data/lib/pgbus/event_bus/publisher.rb +1 -1
- data/lib/pgbus/event_bus/registry.rb +24 -1
- data/lib/pgbus/event_bus/stale_connection_retry.rb +85 -0
- data/lib/pgbus/instrumentation.rb +4 -0
- data/lib/pgbus/metrics/subscriber.rb +14 -0
- data/lib/pgbus/process/consumer.rb +71 -4
- data/lib/pgbus/streams/broadcast_opts.rb +70 -0
- data/lib/pgbus/streams/broadcastable_override.rb +7 -20
- data/lib/pgbus/streams/turbo_broadcastable.rb +65 -4
- data/lib/pgbus/testing.rb +1 -1
- data/lib/pgbus/version.rb +1 -1
- data/lib/pgbus/visibility_heartbeat.rb +23 -4
- metadata +4 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 82a6db71dd099a9b4fa24cb0a278184a3aed8bbee7acfb3f3cc6ef6b55064038
|
|
4
|
+
data.tar.gz: ad7034ee934aee91afdb4a6278200f8178caf0da609d49e57bc6bbcf27d3b7b5
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 86977d3f705b8e3732211956582334f6e49a60e426af1082fdf585a175c9a189c9dc9ec83d5ceeaac2f0be7dcee40c1414e55a99225fcc90ac955537d8b4f01c
|
|
7
|
+
data.tar.gz: 3af2ae55db88774c89f3082b70e5456201c9a85ffd994d43beb197fe0f52d282209fc5de57db66c44de10afe34b37d37fa5404522eabb97d4eb2e91cbfc80da7
|
data/CHANGELOG.md
CHANGED
|
@@ -2,12 +2,21 @@
|
|
|
2
2
|
|
|
3
3
|
### Added
|
|
4
4
|
|
|
5
|
+
- **`coalesce:` now works on the `Turbo::StreamsChannel` broadcast path.** `Pgbus::Streams::Stream#broadcast` has accepted `coalesce:` since #171, but the Turbo patch forwarded only four options (`durable:`, `exclude:`, `visible_to:`, `event:`), so nothing that broadcasts through `Turbo::StreamsChannel.broadcast_*_to` — every `Turbo::Broadcastable` model, and phlex-reactive's `Streamable.broadcast_to` — could debounce. A background job settling a 177-row collection therefore pushed 177 idempotent replaces of the same count badge at every peer. Any targeted helper now takes `coalesce:` (`@order.broadcast_replace_to :account, coalesce: 100`), as does a direct channel call (`Turbo::StreamsChannel.broadcast_replace_to(@account, target: "order-count", coalesce: true)`), with the same semantics as the direct API: opt-in, per `(stream, target)`, last-write-wins, `true` = the 50ms default window, an integer = the window in ms. Two things the option's plumbing needed beyond a fifth thread-local: the coalescing key, which dedupes on `(stream, target)` while `broadcast_stream_to` only ever sees the rendered `content:` — it is now resolved from the frame's own `target:`/`targets:` through the same `ActionView::RecordIdentifier.dom_id` conversion Turbo applies, so a record target keys on `order_7` rather than on a per-instance `#to_s`; and extraction of the pgbus options at the channel itself, since a direct `Turbo::StreamsChannel` call never passes through `Turbo::Broadcastable` and so previously leaked `durable: true` (and now `coalesce:`) into Turbo's renderer. `broadcast_refresh_to` and `broadcast_render_to` carry no target and so cannot coalesce — they raise the existing actionable "requires target:" error rather than guessing a key — and the `_later_to` variants remain out of scope, as the thread-local cannot follow into a job. The wire is byte-identical when `coalesce:` is absent. Closes #465.
|
|
6
|
+
|
|
5
7
|
- **Concurrency is now visible: a section on the Locks page, three gauges, a home stat card and an MCP tool.** `limits_concurrency … on_conflict: :block` became durable in #460 — a parked job is never dropped and a key never runs past `to:` — but nothing showed it, so an operator whose per-record pipeline looked "stuck" had no way to see that the key's slot was held, how many jobs were parked behind it, or how long the oldest had waited, short of SQL. `pgbus_semaphores` and `pgbus_blocked_executions` appeared on no page, in no gauge, in no tool. One new read, `Web::DataSource#concurrency_stats`, now feeds four surfaces: the Locks page's **Concurrency** section (parked jobs, oldest wait, slots held, keys at limit, then a row per key with value / limit, lease state, parked count and oldest wait, auto-refreshing like the other dashboard panels); a **Parked jobs** card on the dashboard home; three unlabelled gauges on both `/pgbus/api/metrics` and the AppSignal probe (`pgbus_concurrency_blocked_executions`, `pgbus_concurrency_blocked_oldest_age_seconds`, `pgbus_concurrency_slots_held` — keys are per-record, so a per-key label would be an unbounded series); and a read-only `pgbus_concurrency` MCP tool returning the same data. The key rows come from a FULL OUTER JOIN, not a LEFT one: a key can have parked jobs and no semaphore (its holder died and the sweep removed the row) or a semaphore and nothing parked, and both shapes matter to an operator. Each row carries two guarded escape hatches. **Release** drops the semaphore and promotes through `Concurrency::BlockedExecution.promote_next`, so it takes each slot through the same guarded upsert an enqueue uses and can never reintroduce the over-admission #460 closed; when the lease is still fresh the confirm says so, because a job is then probably still running and releasing lets another start beside it. **Discard parked** drops a single key's parked jobs, naming the exact count in the confirm, and resolves the bookkeeping those jobs will never resolve themselves — a parked batch child is marked failed so its batch stops waiting and `on_failure` fires (#413), and an `:until_executed` uniqueness lock is released rather than orphaned (#423); each discard logs a warning naming the job and emits `pgbus.blocked_execution_discarded`. There is deliberately no cross-key "discard all parked": not losing work is the whole point of `:block`. Nothing is added to the enqueue or execute path. Refs #461.
|
|
6
8
|
|
|
7
9
|
- **The batches list and a batch's progress panel now refresh themselves.** Batches was the only dashboard list whose `turbo-frame` was missing `data-auto-refresh`, and the batch detail page had no frame at all, so watching a batch drain meant reloading by hand. Both now poll on the existing `web_refresh_interval` (5s, pauses while you interact) like every other list — the batch progress panel is served by `GET /batches/:id?frame=progress`. This reuses the dashboard's turbo-frame polling rather than the SSE stack in `lib/pgbus/web/stream_app.rb`: that is the host-app-facing stream feature, and pointing it at the dashboard would open a PG `LISTEN` connection per viewer to move four integers.
|
|
8
10
|
|
|
9
11
|
### Fixed
|
|
10
12
|
|
|
13
|
+
- **A still-running event handler is no longer re-run as if it had crashed, and a slow one is no longer redelivered mid-run.** Two halves of one hole, follow-up to #469. First: `Process::Consumer` had no equivalent of `ActiveJob::Executor#with_visibility_heartbeat`, so an event message's visibility timeout was never re-armed — a handler slower than `visibility_timeout` (30s by default) was redelivered *while it was still running*, `read_ct` climbed on every redelivery, and after `max_retries` the event was dead-lettered without the handler ever raising. Workers have been protected from this since the heartbeat landed; event consumers never were. The consumer now tracks each message through `VisibilityHeartbeat` for exactly as long as its handlers run, releasing the entry before the archive so a beat can never re-arm a message that is already gone. Second: `Handler#claim_idempotency?` read a `pgbus_processed_events` row with `completed_at IS NULL` as proof that the previous holder had been killed mid-handler, and re-ran. That state equally describes a handler that is simply still running — on another thread, another fork or another host — so a second delivery (a genuine redelivery, or a duplicate envelope, whose independent visibility timeouts no heartbeat can serialize) executed the handler *concurrently with* the live one, which is precisely the double-execution `idempotent!` exists to prevent. The claim now carries liveness rather than only a claim instant: `EventBus::ClaimBeat` refreshes every in-flight claim's `processed_at` from the same beat that re-arms the message's visibility timeout, so message visibility and claim liveness go quiet together when a process dies. A pending claim silent for longer than twice the heartbeat interval is abandoned and re-runs, exactly as before; a fresher one is owned and the delivery skips, deferring to the holder — which either completes (nothing is lost) or fails, leaving its own message for visibility-timeout redelivery to recover. A skip is also no longer silent: `pgbus.event_skipped` carries the reason (`:completed`, `:cached` or `:owned`), the claim's age in seconds and the delivery's `read_ct`, and the metrics subscriber counts it as `pgbus_event_count` with `status: "skipped"` and the reason as a tag. No migration, no new column, and no change on a table that has not run `pgbus:add_processed_event_completion` — a single-phase claim has no pending state and therefore no ownership question. Closes #470.
|
|
14
|
+
|
|
15
|
+
- **An event is no longer handed to every handler whose pattern matches it, so a wildcard handler ran once per subscriber on the topic.** Each subscriber gets its own queue (`Subscriber#setup!` creates and binds one), so a topic with N matching subscribers puts N copies of every event on the bus — but `Consumer#handle_message` resolved handlers with `Registry#handlers_for(routing_key)`, a pattern match that ignored which queue the message had just been read from, and ran every match. Each of the N copies therefore fanned out to all N handlers: **every handler ran N times per event**, with four `"#"` subscribers meaning four invocations of each handler for every event on the bus. `idempotent!` was the only thing hiding it, and it hides it imperfectly by design: the two-phase claim (#385) deliberately *re-runs* when the existing claim is still pending, on the theory that the prior holder crashed — but a pending claim is also exactly what a handler still running on another host looks like. So one event could have host A win the claim and start a 700ms handler while host B, reading a different subscriber's copy of the same event a few hundred milliseconds later, dispatched to that same handler, lost the claim insert, read `completed_at IS NULL` as "crashed, re-run", and executed it concurrently. In one host app that meant a "task completed" record written twice and the user emailed twice, with a single row in `pgbus_processed_events`, `read_ct = 1` on every archived copy, and the second execution's side effects timestamped inside the first's window — dispatch, not redelivery. A non-idempotent handler simply ran N times with no guard at all. `handlers_for` now takes a required `queue_name:` and selects on ownership *and* pattern, so a message read from queue Q goes to Q's owner(s) alone and each handler runs exactly once per event; duplicate execution again requires a real redelivery, which is the at-least-once contract the docs describe. The keyword is required rather than defaulted precisely so a keyword-less call cannot silently restore the fan-out — the two callers that genuinely want the pattern view (the `Testing.inline!` publish path and `Testing::EventStore#drain!`, neither of which has a queue, since the event never reaches PGMQ) moved to an explicitly named `Registry#subscribers_matching`, where per-subscriber delivery counts already match what owner-only dispatch now produces in production. Two handlers registered against the same explicit `queue_name:` still both run on that queue's delivery. The pattern check is kept alongside the ownership check because a routing key the owner's pattern does not match means a stale `pgmq.topic_bindings` row, and a stale binding must not run the handler. A message on a queue no subscriber in this process owns is still archived rather than looped through visibility-timeout redelivery into the DLQ — that was already the behavior when `handlers_for` returned `[]` — but it is no longer silent: it logs a warning naming the queue and routing key, rate-limited to once per queue per process so a permanently stale binding cannot flood the log, and emits `pgbus.event_unrouted` on every occurrence so the real rate stays visible in metrics. The pending-claim re-run in `Handler#claim_idempotency?` is deliberately untouched here; it is a separate question now that dispatch no longer manufactures the concurrency it was misreading. Closes #469.
|
|
16
|
+
|
|
17
|
+
- **A dropped ActiveRecord socket no longer pages the host app for an event that was handled successfully.** EventBus consumers are long-lived threads holding a leased AR connection, and a pooler restart, an admin disconnect or a brief failover can kill that socket at any point. Rails reconnects most statements transparently, but `Relation#update_all` is marked `allow_retry: false` — and that is exactly the phase-2 claim stamp in `Handler#complete_claim!`, the one AR write that happens *after* `handle` has already returned. A drop there surfaced as `ActiveRecord::ConnectionFailed` ("PQconsumeInput() SSL error: unexpected eof while reading") on a message whose work was done, so the host app's exception tracker paged for a false failure and PGMQ redelivered the event for a re-run. `EventBus::StaleConnectionRetry` now reconnects this thread's lease (via `ConnectionPool#active_connection?`, the accessor that exists across the supported Rails range) and repeats the stamp once; a second drop still raises, leaving the message to VT redelivery. Only the leased connections are reconnected — `clear_all_connections!` would yank sockets out from under sibling consumers in the same process, turning one recoverable drop into many. The retryable patterns are deliberately *broader* than `Client::STALE_CONNECTION_PATTERNS` and for the opposite reason: that list excludes mid-flight drops because a half-committed enqueue would duplicate a message, whereas this only ever repeats an idempotent `SET completed_at = <now>`. Phase 1 (`claim_idempotency?`) is deliberately not wrapped — its INSERT may have committed before the socket died, and on a legacy schema the retry's empty `result.rows` would read as "another consumer owns this claim", turning a recoverable drop into a silently skipped event.
|
|
18
|
+
|
|
19
|
+
|
|
11
20
|
- **The AppSignal probe's gauges and every MCP tool response were frozen at their first read.** `Web::DataSource` memoizes `queues_with_metrics` for the life of the instance — correct for the dashboard, which builds one instance per request, but the AppSignal `Probe::Runner` holds a single DataSource for the life of the process and `MCP::Server.build` injects one into the server context, so both replayed the first read forever: queue depth in AppSignal never moved after the first minute, and `pgbus_queues` returned the same snapshot for every call in a session. `DataSource#reset_cache!` drops the memos, and both long-lived consumers call it — the probe once per iteration, `MCP::BaseTool.data_source_from` once per tool call. Found while reviewing the concurrency aggregates added in the same release, which memoize the same way.
|
|
12
21
|
|
|
13
22
|
- **`limits_concurrency … on_conflict: :block` now constrains a job without ever losing it, and never runs more than `to:` at once.** The same bug class solid_queue closed in rails/solid_queue#712 and #761, checked against pgbus's implementation. Four defects, each with a regression spec:
|
data/README.md
CHANGED
|
@@ -221,6 +221,14 @@ Pgbus::EventBus::Registry.instance.subscribe(
|
|
|
221
221
|
)
|
|
222
222
|
```
|
|
223
223
|
|
|
224
|
+
Each subscriber gets its own queue, and a queue is consumed only by the
|
|
225
|
+
handler(s) registered against it: one published event matching N subscribers
|
|
226
|
+
becomes N deliveries, one per handler, so every handler runs exactly once per
|
|
227
|
+
event. (Two handlers sharing an explicit `queue_name:` both run on that queue's
|
|
228
|
+
delivery.) A message on a queue no subscriber in this process owns — a stale
|
|
229
|
+
topic binding left by a renamed or removed handler — is archived, logged once
|
|
230
|
+
per queue, and reported as `pgbus.event_unrouted`.
|
|
231
|
+
|
|
224
232
|
`idempotent!` uses a **two-phase claim**: a *pending* row in
|
|
225
233
|
`pgbus_processed_events` is inserted before `handle` runs, and only stamped
|
|
226
234
|
`completed_at` after `handle` returns. Deduplication applies to **completed**
|
|
@@ -0,0 +1,80 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module Pgbus
|
|
4
|
+
module EventBus
|
|
5
|
+
# Liveness for the idempotency claims in flight for one PGMQ message.
|
|
6
|
+
#
|
|
7
|
+
# A two-phase claim (issue #385) is a `pgbus_processed_events` row with
|
|
8
|
+
# `completed_at IS NULL`. That state says "claimed, not finished" — it does
|
|
9
|
+
# NOT say whether the claimer is dead or simply still running (issue #470).
|
|
10
|
+
# Handler resolves the ambiguity by age: a claim whose `processed_at` has
|
|
11
|
+
# gone quiet for longer than the ownership window is abandoned, anything
|
|
12
|
+
# fresher is owned. For that age to mean "silence" rather than
|
|
13
|
+
# "time since the claim was taken", something has to keep stamping it while
|
|
14
|
+
# the handler runs. This is that something.
|
|
15
|
+
#
|
|
16
|
+
# One beat per message, created by Process::Consumer and handed to every
|
|
17
|
+
# handler dispatched for it. A handler registers its claim for exactly the
|
|
18
|
+
# duration of `handle` and the consumer's VisibilityHeartbeat `on_beat` hook
|
|
19
|
+
# drives #touch! on the same cadence that re-arms the message's visibility
|
|
20
|
+
# timeout — so the claim and the message go quiet together when the process
|
|
21
|
+
# dies, and both stay fresh while it lives.
|
|
22
|
+
#
|
|
23
|
+
# #touch! runs on the heartbeat ticker thread while #register / #release run
|
|
24
|
+
# on a pool thread, hence the mutex.
|
|
25
|
+
class ClaimBeat
|
|
26
|
+
def initialize
|
|
27
|
+
@mutex = Mutex.new
|
|
28
|
+
@claims = []
|
|
29
|
+
end
|
|
30
|
+
|
|
31
|
+
def register(event_id, handler_class)
|
|
32
|
+
claim = [event_id, handler_class]
|
|
33
|
+
@mutex.synchronize { @claims << claim unless @claims.include?(claim) }
|
|
34
|
+
self
|
|
35
|
+
end
|
|
36
|
+
|
|
37
|
+
def release(event_id, handler_class)
|
|
38
|
+
@mutex.synchronize { @claims.delete([event_id, handler_class]) }
|
|
39
|
+
self
|
|
40
|
+
end
|
|
41
|
+
|
|
42
|
+
def size
|
|
43
|
+
@mutex.synchronize { @claims.size }
|
|
44
|
+
end
|
|
45
|
+
|
|
46
|
+
def empty?
|
|
47
|
+
size.zero?
|
|
48
|
+
end
|
|
49
|
+
|
|
50
|
+
# Refresh every registered claim's liveness stamp. Returns the number of
|
|
51
|
+
# claims touched. No-op on a legacy schema: without `completed_at` there
|
|
52
|
+
# are no pending claims to keep alive, and Handler's single-phase
|
|
53
|
+
# fallback never consults the age.
|
|
54
|
+
#
|
|
55
|
+
# A claim that fails to update is logged and skipped rather than raised:
|
|
56
|
+
# this runs inside the visibility heartbeat's beat, and one unwritable
|
|
57
|
+
# row must not cost every other in-flight message its VT extension.
|
|
58
|
+
def touch!
|
|
59
|
+
return 0 unless ProcessedEvent.completion_column?
|
|
60
|
+
|
|
61
|
+
now = Time.now.utc
|
|
62
|
+
@mutex.synchronize { @claims.dup }.count { |event_id, handler_class| touch(event_id, handler_class, now) }
|
|
63
|
+
end
|
|
64
|
+
|
|
65
|
+
private
|
|
66
|
+
|
|
67
|
+
def touch(event_id, handler_class, now)
|
|
68
|
+
ProcessedEvent
|
|
69
|
+
.where(event_id: event_id, handler_class: handler_class, completed_at: nil)
|
|
70
|
+
.update_all(processed_at: now)
|
|
71
|
+
true
|
|
72
|
+
rescue StandardError => e
|
|
73
|
+
Pgbus.logger.warn do
|
|
74
|
+
"[Pgbus] Could not refresh idempotency claim #{handler_class}/#{event_id}: #{e.class}: #{e.message}"
|
|
75
|
+
end
|
|
76
|
+
false
|
|
77
|
+
end
|
|
78
|
+
end
|
|
79
|
+
end
|
|
80
|
+
end
|
|
@@ -17,8 +17,20 @@ module Pgbus
|
|
|
17
17
|
end
|
|
18
18
|
end
|
|
19
19
|
|
|
20
|
-
|
|
21
|
-
|
|
20
|
+
# Outcome of the two-phase claim. `age` is how long the losing delivery
|
|
21
|
+
# found the existing claim to have been silent, in seconds (nil unless
|
|
22
|
+
# the claim was pending).
|
|
23
|
+
ClaimResult = Data.define(:status, :age) do
|
|
24
|
+
def granted?
|
|
25
|
+
status == :claimed
|
|
26
|
+
end
|
|
27
|
+
end
|
|
28
|
+
|
|
29
|
+
# @param claim_beat [ClaimBeat, nil] the message's claim-liveness beat,
|
|
30
|
+
# supplied by Process::Consumer. Absent for a hand-rolled caller: the
|
|
31
|
+
# handler still runs, its claim simply ages from the claim instant.
|
|
32
|
+
def process(message, claim_beat: nil)
|
|
33
|
+
with_rails_executor { process!(message, claim_beat) }
|
|
22
34
|
end
|
|
23
35
|
|
|
24
36
|
def handle(event)
|
|
@@ -27,12 +39,18 @@ module Pgbus
|
|
|
27
39
|
|
|
28
40
|
private
|
|
29
41
|
|
|
30
|
-
def process!(message)
|
|
42
|
+
def process!(message, claim_beat = nil)
|
|
31
43
|
raw = JSON.parse(message.message)
|
|
32
44
|
event = build_event(raw)
|
|
33
45
|
routing_key = raw.dig("headers", "routing_key") || raw["routing_key"]
|
|
34
46
|
|
|
35
|
-
|
|
47
|
+
if self.class.idempotent?
|
|
48
|
+
claim = claim_idempotency(event.event_id)
|
|
49
|
+
unless claim.granted?
|
|
50
|
+
instrument_skip(claim, event, message, routing_key)
|
|
51
|
+
return :skipped
|
|
52
|
+
end
|
|
53
|
+
end
|
|
36
54
|
|
|
37
55
|
instrument_payload = {
|
|
38
56
|
event_id: event.event_id,
|
|
@@ -42,11 +60,13 @@ module Pgbus
|
|
|
42
60
|
read_ct: message.read_ct.to_i,
|
|
43
61
|
msg_id: message.msg_id.to_i
|
|
44
62
|
}
|
|
45
|
-
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
|
|
49
|
-
|
|
63
|
+
with_claim_beat(claim_beat, event.event_id) do
|
|
64
|
+
Instrumentation.instrument("pgbus.event_processed", instrument_payload) do
|
|
65
|
+
# Publisher's Current attributes (issue #431) are set for the handler
|
|
66
|
+
# and reverted after (CurrentAttributes#set semantics); the Rails
|
|
67
|
+
# executor wrap above additionally resets at completion.
|
|
68
|
+
Pgbus::CurrentAttributes.restore(event.context) { handle(event) }
|
|
69
|
+
end
|
|
50
70
|
end
|
|
51
71
|
complete_claim!(event.event_id) if self.class.idempotent?
|
|
52
72
|
:handled
|
|
@@ -64,7 +84,7 @@ module Pgbus
|
|
|
64
84
|
|
|
65
85
|
# Mirrors Pgbus::ActiveJob::Executor#execute_job: wrap the handler
|
|
66
86
|
# invocation in Rails.application.executor (or the reloader in dev)
|
|
67
|
-
# so AR connections leased by `claim_idempotency
|
|
87
|
+
# so AR connections leased by `claim_idempotency` and `handle` are
|
|
68
88
|
# released back to the pool when this method returns. Without the
|
|
69
89
|
# wrap, every consumed event leaks one AR connection on the consumer
|
|
70
90
|
# thread — in dev that wedges `clear_reloadable_connections!`,
|
|
@@ -113,24 +133,32 @@ module Pgbus
|
|
|
113
133
|
|
|
114
134
|
# Two-phase idempotency claim (issue #385). Phase 1: atomically claim
|
|
115
135
|
# via INSERT ... ON CONFLICT DO NOTHING with completed_at NULL — a
|
|
116
|
-
# *pending* claim. Returns
|
|
136
|
+
# *pending* claim. Returns a ClaimResult whose status is one of:
|
|
117
137
|
#
|
|
118
|
-
#
|
|
119
|
-
#
|
|
120
|
-
#
|
|
121
|
-
#
|
|
122
|
-
#
|
|
123
|
-
#
|
|
138
|
+
# :claimed — insert won (fresh claim), or the row was purged between
|
|
139
|
+
# the losing insert and the read, or an existing pending
|
|
140
|
+
# claim has gone silent for longer than the ownership
|
|
141
|
+
# window: the holder is dead by the heartbeat's own
|
|
142
|
+
# definition, so re-run rather than silently drop the
|
|
143
|
+
# execution a SIGKILL interrupted.
|
|
144
|
+
# :completed — the execution already finished. Skip.
|
|
145
|
+
# :owned — pending, and its liveness stamp is fresh: the holder is
|
|
146
|
+
# still running (issue #470). Skip — running `handle`
|
|
147
|
+
# concurrently with the holder is exactly the
|
|
148
|
+
# double-execution `idempotent!` promises not to do. The
|
|
149
|
+
# holder either completes (nothing lost) or fails, leaving
|
|
150
|
+
# its own message for VT redelivery to recover.
|
|
151
|
+
# :cached — a completed execution already in this process's memory.
|
|
124
152
|
#
|
|
125
|
-
#
|
|
126
|
-
#
|
|
127
|
-
# the in-memory dedup cache.
|
|
153
|
+
# Phase 2 is complete_claim! after handle returns; only completed
|
|
154
|
+
# executions enter the in-memory dedup cache.
|
|
128
155
|
#
|
|
129
156
|
# Legacy fallback: without the completed_at column (upgraded gem,
|
|
130
|
-
# not-yet-migrated table) this degrades to the old single-phase claim
|
|
131
|
-
|
|
157
|
+
# not-yet-migrated table) this degrades to the old single-phase claim,
|
|
158
|
+
# which has no pending state and therefore no ownership question.
|
|
159
|
+
def claim_idempotency(event_id)
|
|
132
160
|
cache_key = dedup_key(event_id)
|
|
133
|
-
return
|
|
161
|
+
return ClaimResult.new(status: :cached, age: nil) if self.class.dedup_cache.seen?(cache_key)
|
|
134
162
|
|
|
135
163
|
result = ProcessedEvent.insert(
|
|
136
164
|
{ event_id: event_id, handler_class: self.class.name, processed_at: Time.now.utc },
|
|
@@ -139,18 +167,79 @@ module Pgbus
|
|
|
139
167
|
|
|
140
168
|
unless ProcessedEvent.completion_column?
|
|
141
169
|
self.class.dedup_cache.mark!(cache_key)
|
|
142
|
-
return result.rows.any?
|
|
170
|
+
return ClaimResult.new(status: result.rows.any? ? :claimed : :completed, age: nil)
|
|
143
171
|
end
|
|
144
172
|
|
|
145
|
-
return
|
|
173
|
+
return ClaimResult.new(status: :claimed, age: nil) if result.rows.any?
|
|
146
174
|
|
|
147
|
-
|
|
148
|
-
|
|
149
|
-
|
|
150
|
-
|
|
175
|
+
inspect_existing_claim(event_id, cache_key)
|
|
176
|
+
end
|
|
177
|
+
|
|
178
|
+
# The insert lost, so a row exists (or existed). `pick` returns nil for
|
|
179
|
+
# the whole row when it has since been purged — not a pending claim,
|
|
180
|
+
# nothing is running, so claim it.
|
|
181
|
+
def inspect_existing_claim(event_id, cache_key)
|
|
182
|
+
completed_at, processed_at = ProcessedEvent
|
|
183
|
+
.where(event_id: event_id, handler_class: self.class.name)
|
|
184
|
+
.pick(:completed_at, :processed_at)
|
|
151
185
|
|
|
152
|
-
|
|
153
|
-
|
|
186
|
+
if completed_at
|
|
187
|
+
self.class.dedup_cache.mark!(cache_key)
|
|
188
|
+
return ClaimResult.new(status: :completed, age: nil)
|
|
189
|
+
end
|
|
190
|
+
|
|
191
|
+
return ClaimResult.new(status: :claimed, age: nil) if processed_at.nil?
|
|
192
|
+
|
|
193
|
+
age = Time.now.utc - processed_at.to_time.utc
|
|
194
|
+
return ClaimResult.new(status: :owned, age: age) if age < claim_ownership_window
|
|
195
|
+
|
|
196
|
+
ClaimResult.new(status: :claimed, age: age)
|
|
197
|
+
end
|
|
198
|
+
|
|
199
|
+
# How long a pending claim may stay silent before its holder counts as
|
|
200
|
+
# dead. ClaimBeat refreshes a live claim from the visibility heartbeat,
|
|
201
|
+
# which lands every extension inside [interval, 1.5 * interval] of the
|
|
202
|
+
# previous one — two intervals leaves margin for a late beat without
|
|
203
|
+
# stretching the window past the visibility timeout it rides on.
|
|
204
|
+
#
|
|
205
|
+
# With the heartbeat disabled a claim is never refreshed, so the window
|
|
206
|
+
# degrades to "roughly the first two thirds of one visibility timeout
|
|
207
|
+
# after the claim" — a redelivery, which cannot arrive before the VT has
|
|
208
|
+
# lapsed, still re-runs exactly as it did before issue #470.
|
|
209
|
+
def claim_ownership_window
|
|
210
|
+
Pgbus.configuration.effective_visibility_heartbeat_interval * 2
|
|
211
|
+
end
|
|
212
|
+
|
|
213
|
+
# Register this claim with the message's beat for exactly the duration of
|
|
214
|
+
# handle: before it, there is nothing to keep alive; after it,
|
|
215
|
+
# complete_claim! owns the row and a beat touching processed_at would
|
|
216
|
+
# race the completion stamp.
|
|
217
|
+
def with_claim_beat(claim_beat, event_id)
|
|
218
|
+
return yield unless claim_beat && self.class.idempotent? && ProcessedEvent.completion_column?
|
|
219
|
+
|
|
220
|
+
claim_beat.register(event_id, self.class.name)
|
|
221
|
+
begin
|
|
222
|
+
yield
|
|
223
|
+
ensure
|
|
224
|
+
claim_beat.release(event_id, self.class.name)
|
|
225
|
+
end
|
|
226
|
+
end
|
|
227
|
+
|
|
228
|
+
# A skip used to be silent, which made an over-eager re-run (issue #470)
|
|
229
|
+
# invisible in production: nothing distinguished "deduplicated" from
|
|
230
|
+
# "deferred to a live holder". The claim age and read_ct are what tell
|
|
231
|
+
# an operator which one happened.
|
|
232
|
+
def instrument_skip(claim, event, message, routing_key)
|
|
233
|
+
Instrumentation.instrument(
|
|
234
|
+
"pgbus.event_skipped",
|
|
235
|
+
event_id: event.event_id,
|
|
236
|
+
handler: self.class.name,
|
|
237
|
+
routing_key: routing_key,
|
|
238
|
+
reason: claim.status,
|
|
239
|
+
claim_age: claim.age,
|
|
240
|
+
read_ct: message.read_ct.to_i,
|
|
241
|
+
msg_id: message.msg_id.to_i
|
|
242
|
+
)
|
|
154
243
|
end
|
|
155
244
|
|
|
156
245
|
# Phase 2: stamp the claim completed and only then admit it to the
|
|
@@ -158,12 +247,26 @@ module Pgbus
|
|
|
158
247
|
# already cached at claim time). If this write fails, process!'s rescue
|
|
159
248
|
# re-raises, the consumer leaves the message for VT redelivery, and the
|
|
160
249
|
# still-pending claim re-runs — at-least-once, never a silent drop.
|
|
250
|
+
#
|
|
251
|
+
# Wrapped in StaleConnectionRetry because this is the one AR write that
|
|
252
|
+
# happens AFTER handle() has already succeeded: a socket dropped here
|
|
253
|
+
# costs the host app a paging exception for work that was in fact done.
|
|
254
|
+
# The stamp is an idempotent `SET completed_at = <now>`, so repeating a
|
|
255
|
+
# statement that may already have committed is safe.
|
|
256
|
+
#
|
|
257
|
+
# Phase 1 (claim_idempotency) is deliberately NOT wrapped. Its INSERT
|
|
258
|
+
# may have committed before the socket died, and on a legacy schema the
|
|
259
|
+
# retry's empty `result.rows` would read as "someone else owns this
|
|
260
|
+
# claim" and report :completed — turning a recoverable drop into a
|
|
261
|
+
# silently skipped event. VT redelivery is the correct recovery there.
|
|
161
262
|
def complete_claim!(event_id)
|
|
162
263
|
return unless ProcessedEvent.completion_column?
|
|
163
264
|
|
|
164
|
-
|
|
165
|
-
|
|
166
|
-
|
|
265
|
+
StaleConnectionRetry.call(context: self.class.name) do
|
|
266
|
+
ProcessedEvent
|
|
267
|
+
.where(event_id: event_id, handler_class: self.class.name)
|
|
268
|
+
.update_all(completed_at: Time.now.utc)
|
|
269
|
+
end
|
|
167
270
|
self.class.dedup_cache.mark!(dedup_key(event_id))
|
|
168
271
|
end
|
|
169
272
|
|
|
@@ -25,7 +25,7 @@ module Pgbus
|
|
|
25
25
|
Pgbus::Testing.store.push_event(event)
|
|
26
26
|
|
|
27
27
|
if Pgbus::Testing.inline? && delay.to_i <= 0
|
|
28
|
-
Pgbus::EventBus::Registry.instance.
|
|
28
|
+
Pgbus::EventBus::Registry.instance.subscribers_matching(routing_key).each do |subscriber|
|
|
29
29
|
Pgbus::CurrentAttributes.restore(event.context) { subscriber.handler_class.new.handle(event) }
|
|
30
30
|
end
|
|
31
31
|
end
|
|
@@ -62,7 +62,30 @@ module Pgbus
|
|
|
62
62
|
end
|
|
63
63
|
end
|
|
64
64
|
|
|
65
|
-
|
|
65
|
+
# Subscribers a message read from +queue_name+ must be dispatched to
|
|
66
|
+
# (issue #469). Every subscriber gets its own queue, so a topic with N
|
|
67
|
+
# matching subscribers produces N queue copies of each event; selecting by
|
|
68
|
+
# pattern alone fanned every copy out to every match, running each handler
|
|
69
|
+
# N times per event — and, across hosts, concurrently. Ownership is the
|
|
70
|
+
# primary filter; the pattern check still applies because a routing key
|
|
71
|
+
# the owner's pattern does not match means a stale pgmq.topic_bindings
|
|
72
|
+
# row, and a stale binding must not run the handler.
|
|
73
|
+
#
|
|
74
|
+
# +queue_name+ is required on purpose: a keyword-less call would silently
|
|
75
|
+
# restore the fan-out this closed. Callers that genuinely want the
|
|
76
|
+
# pattern view (the Testing inline/drain paths, which never touch a
|
|
77
|
+
# queue) use #subscribers_matching.
|
|
78
|
+
def handlers_for(routing_key, queue_name:)
|
|
79
|
+
@subscribers.select do |s|
|
|
80
|
+
s.queue_name == queue_name && matches?(s.pattern, routing_key)
|
|
81
|
+
end
|
|
82
|
+
end
|
|
83
|
+
|
|
84
|
+
# Pattern-only selection, with no queue in play. Used by the Testing
|
|
85
|
+
# inline/drain paths, where the event never reaches PGMQ and each matching
|
|
86
|
+
# subscriber is invoked exactly once — the same per-subscriber delivery
|
|
87
|
+
# count owner-only dispatch produces in production.
|
|
88
|
+
def subscribers_matching(routing_key)
|
|
66
89
|
@subscribers.select { |s| matches?(s.pattern, routing_key) }
|
|
67
90
|
end
|
|
68
91
|
|
|
@@ -0,0 +1,85 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module Pgbus
|
|
4
|
+
module EventBus
|
|
5
|
+
# Reconnect + one retry for an ActiveRecord socket that died underneath a
|
|
6
|
+
# long-lived consumer thread.
|
|
7
|
+
#
|
|
8
|
+
# A consumer holds its leased connection for the life of the message, and a
|
|
9
|
+
# pooler restart (PgBouncer `server_idle_timeout`), an admin disconnect or a
|
|
10
|
+
# brief failover can kill that socket at any point. Rails reconnects most
|
|
11
|
+
# statements transparently, but `Relation#update_all` is marked
|
|
12
|
+
# `allow_retry: false`, so the drop propagates — and the only call that
|
|
13
|
+
# matters here, the phase-2 claim stamp, is exactly an `update_all`. The
|
|
14
|
+
# host app saw a paging exception on a message whose handler had in fact
|
|
15
|
+
# succeeded.
|
|
16
|
+
#
|
|
17
|
+
# Deliberately narrower than `Client::STALE_CONNECTION_PATTERNS`, and for
|
|
18
|
+
# the opposite reason. That list excludes mid-flight drops because a
|
|
19
|
+
# half-committed *enqueue* would duplicate a message on retry. This one
|
|
20
|
+
# only ever wraps an idempotent `UPDATE ... SET completed_at = <now>`, so a
|
|
21
|
+
# statement that may already have committed is safe to repeat — which is
|
|
22
|
+
# what makes the mid-flight shapes retryable here and not there.
|
|
23
|
+
module StaleConnectionRetry
|
|
24
|
+
# The socket dying under a statement, not a server refusing us. A refused
|
|
25
|
+
# or timed-out connection is an outage: retrying it in-process buys
|
|
26
|
+
# nothing and hides the outage behind a doubled statement timeout.
|
|
27
|
+
#
|
|
28
|
+
# "ssl syscall error" is libpq's wording when the peer vanished without a
|
|
29
|
+
# TLS close_notify — the same drop as "unexpected eof while reading",
|
|
30
|
+
# reported from the syscall layer instead. It is in
|
|
31
|
+
# Client::STALE_CONNECTION_PATTERNS for the same reason.
|
|
32
|
+
TRANSIENT_DROP = /
|
|
33
|
+
PQconsumeInput|
|
|
34
|
+
server\ closed\ the\ connection\ unexpectedly|
|
|
35
|
+
unexpected\ eof\ while\ reading|
|
|
36
|
+
ssl\ syscall\ error
|
|
37
|
+
/ix
|
|
38
|
+
|
|
39
|
+
module_function
|
|
40
|
+
|
|
41
|
+
# Runs the block, and on a transient drop reconnects this thread's lease
|
|
42
|
+
# and runs it exactly once more. A second drop raises: the socket is not
|
|
43
|
+
# coming back inside this attempt, and PGMQ's visibility timeout is the
|
|
44
|
+
# right recovery — it redelivers to a healthy consumer.
|
|
45
|
+
#
|
|
46
|
+
# context — an identifier for the log line (the handler class name).
|
|
47
|
+
def call(context: nil, &block)
|
|
48
|
+
block.call
|
|
49
|
+
rescue ActiveRecord::ConnectionFailed => e
|
|
50
|
+
raise unless transient_drop?(e)
|
|
51
|
+
|
|
52
|
+
reconnect_leased!
|
|
53
|
+
Pgbus.logger.warn do
|
|
54
|
+
"[Pgbus::EventBus] Retrying after stale ActiveRecord connection drop" \
|
|
55
|
+
"#{" (#{context})" if context}: #{e.message}"
|
|
56
|
+
end
|
|
57
|
+
block.call
|
|
58
|
+
end
|
|
59
|
+
|
|
60
|
+
def transient_drop?(error)
|
|
61
|
+
return false unless error.is_a?(ActiveRecord::ConnectionFailed)
|
|
62
|
+
|
|
63
|
+
[error.message, error.cause&.message].compact.any? { |message| TRANSIENT_DROP.match?(message) }
|
|
64
|
+
end
|
|
65
|
+
|
|
66
|
+
# Only the connections this thread has actually leased.
|
|
67
|
+
# `clear_all_connections!` would yank sockets out from under sibling
|
|
68
|
+
# consumers sharing the process, turning one recoverable drop into many.
|
|
69
|
+
# `active_connection?` returns the lease or nil, and is the accessor this
|
|
70
|
+
# gem uses everywhere (Streams#current_open_transaction documents why we
|
|
71
|
+
# never reach for `ActiveRecord::Base.connection`). Its `active_connection`
|
|
72
|
+
# alias is `:nodoc:` and does not exist at all before Rails 7.2 — below
|
|
73
|
+
# our floor, so calling it would raise NoMethodError on the one path that
|
|
74
|
+
# exists to recover from an error.
|
|
75
|
+
def reconnect_leased!
|
|
76
|
+
ActiveRecord::Base.connection_handler.each_connection_pool(:all) do |pool|
|
|
77
|
+
connection = pool.active_connection?
|
|
78
|
+
next unless connection
|
|
79
|
+
|
|
80
|
+
connection.reconnect!
|
|
81
|
+
end
|
|
82
|
+
end
|
|
83
|
+
end
|
|
84
|
+
end
|
|
85
|
+
end
|
|
@@ -20,6 +20,10 @@ module Pgbus
|
|
|
20
20
|
# payload: queue, job_class, msg_id, vt, extensions
|
|
21
21
|
# pgbus.event_processed — event handler succeeded
|
|
22
22
|
# pgbus.event_failed — event handler raised; carries :exception_object
|
|
23
|
+
# pgbus.event_unrouted — a consumer read an event from a queue no
|
|
24
|
+
# subscriber in this process owns (stale topic
|
|
25
|
+
# binding); the message is archived
|
|
26
|
+
# payload: queue_name, routing_key
|
|
23
27
|
# pgbus.stream.broadcast — stream broadcast (sync or deferred)
|
|
24
28
|
# pgbus.outbox.publish — outbox row created
|
|
25
29
|
# pgbus.recurring.enqueue — scheduler enqueued a due recurring task
|
|
@@ -33,6 +33,7 @@ module Pgbus
|
|
|
33
33
|
subscribe("pgbus.job_visibility_extended") { |event| on_job_visibility_extended(event) },
|
|
34
34
|
subscribe("pgbus.event_processed") { |event| on_event_processed(event) },
|
|
35
35
|
subscribe("pgbus.event_failed") { |event| on_event_failed(event) },
|
|
36
|
+
subscribe("pgbus.event_skipped") { |event| on_event_skipped(event) },
|
|
36
37
|
subscribe("pgbus.client.send_message") { |event| on_send_message(event) },
|
|
37
38
|
subscribe("pgbus.client.send_batch") { |event| on_send_batch(event) },
|
|
38
39
|
subscribe("pgbus.client.read_batch") { |event| on_read_batch(event) },
|
|
@@ -139,6 +140,19 @@ module Pgbus
|
|
|
139
140
|
)
|
|
140
141
|
end
|
|
141
142
|
|
|
143
|
+
# An idempotent handler that did not run this delivery. `reason` is what
|
|
144
|
+
# makes the skip readable: :completed/:cached is deduplication working,
|
|
145
|
+
# :owned means a second delivery arrived while the holder was still
|
|
146
|
+
# running and was deferred to it (issue #470).
|
|
147
|
+
def on_event_skipped(event)
|
|
148
|
+
payload = event.payload
|
|
149
|
+
backend.increment(
|
|
150
|
+
"#{METRIC_PREFIX}event_count", 1,
|
|
151
|
+
compact(handler: payload[:handler], routing_key: payload[:routing_key],
|
|
152
|
+
status: "skipped", reason: payload[:reason])
|
|
153
|
+
)
|
|
154
|
+
end
|
|
155
|
+
|
|
142
156
|
# ── Client (PGMQ wrapper) ─────────────────────────────────────────
|
|
143
157
|
|
|
144
158
|
def on_send_message(event)
|
|
@@ -78,6 +78,11 @@ module Pgbus
|
|
|
78
78
|
)
|
|
79
79
|
@registry = EventBus::Registry.instance
|
|
80
80
|
@circuit_breaker = Pgbus::CircuitBreaker.new(config: config)
|
|
81
|
+
# Queues already warned about for an unroutable message (issue #469).
|
|
82
|
+
# handle_message runs on the execution pool, so this is touched from
|
|
83
|
+
# several threads — Concurrent::Set makes add? the atomic
|
|
84
|
+
# test-and-set the once-per-queue guarantee needs.
|
|
85
|
+
@unrouted_queues = Concurrent::Set.new
|
|
81
86
|
# stat_buffer: :default means "build one iff config.stats_enabled";
|
|
82
87
|
# passing an explicit value (including nil) overrides that for tests.
|
|
83
88
|
@stat_buffer =
|
|
@@ -222,10 +227,12 @@ module Pgbus
|
|
|
222
227
|
raw = JSON.parse(message.message)
|
|
223
228
|
routing_key = raw.dig("headers", "routing_key") || raw["routing_key"]
|
|
224
229
|
|
|
225
|
-
handlers = @registry.handlers_for(routing_key || "")
|
|
226
|
-
|
|
227
|
-
|
|
228
|
-
|
|
230
|
+
handlers = @registry.handlers_for(routing_key || "", queue_name: queue_name)
|
|
231
|
+
|
|
232
|
+
if handlers.empty?
|
|
233
|
+
report_unrouted(queue_name, routing_key)
|
|
234
|
+
else
|
|
235
|
+
dispatch(handlers, message, queue_name)
|
|
229
236
|
end
|
|
230
237
|
|
|
231
238
|
Pgbus.client.archive_message(queue_name, message.msg_id.to_i)
|
|
@@ -247,6 +254,65 @@ module Pgbus
|
|
|
247
254
|
@jobs_processed.increment
|
|
248
255
|
end
|
|
249
256
|
|
|
257
|
+
# Run every handler that owns this message, with the message's visibility
|
|
258
|
+
# timeout held open for as long as they take (issue #470).
|
|
259
|
+
#
|
|
260
|
+
# Without this the consumer had no equivalent of
|
|
261
|
+
# ActiveJob::Executor#with_visibility_heartbeat: an event handler slower
|
|
262
|
+
# than config.visibility_timeout (30s by default) was redelivered *while
|
|
263
|
+
# still running*, a second consumer read the same envelope, and the
|
|
264
|
+
# holder's pending idempotency claim was indistinguishable from a claim
|
|
265
|
+
# left by a crash — so the handler ran twice, concurrently.
|
|
266
|
+
#
|
|
267
|
+
# The beat also refreshes the claims the handlers register, which is what
|
|
268
|
+
# lets Handler tell "holder still running" from "holder died": message
|
|
269
|
+
# visibility and claim liveness go quiet together when this process does.
|
|
270
|
+
#
|
|
271
|
+
# Tracking ends before the caller archives — a beat must never re-arm the
|
|
272
|
+
# VT of a message that is already gone (same rule as the executor's).
|
|
273
|
+
def dispatch(handlers, message, queue_name)
|
|
274
|
+
claim_beat = EventBus::ClaimBeat.new
|
|
275
|
+
|
|
276
|
+
VisibilityHeartbeat.track(
|
|
277
|
+
client: Pgbus.client,
|
|
278
|
+
queue_name: queue_name,
|
|
279
|
+
msg_id: message.msg_id.to_i,
|
|
280
|
+
job_class: "EventConsumer",
|
|
281
|
+
config: config,
|
|
282
|
+
on_beat: -> { claim_beat.touch! }
|
|
283
|
+
) do
|
|
284
|
+
handlers.each { |subscriber| subscriber.handler_class.new.process(message, claim_beat: claim_beat) }
|
|
285
|
+
end
|
|
286
|
+
end
|
|
287
|
+
|
|
288
|
+
# No subscriber in this process owns +queue_name+ (a stale
|
|
289
|
+
# pgmq.topic_bindings row left by a renamed or removed handler, another
|
|
290
|
+
# app bound to the same bus), or the owner's pattern no longer matches the
|
|
291
|
+
# routing key. The message is archived by the caller either way — looping
|
|
292
|
+
# it through VT redelivery would only walk it into the DLQ — but that used
|
|
293
|
+
# to happen in total silence. The warning is rate-limited to once per
|
|
294
|
+
# queue per process so a permanently stale binding cannot flood the log;
|
|
295
|
+
# the instrumentation fires on every message, so the real rate stays
|
|
296
|
+
# visible in metrics (issue #469).
|
|
297
|
+
def report_unrouted(queue_name, routing_key)
|
|
298
|
+
first_for_queue = @unrouted_queues.add?(queue_name)
|
|
299
|
+
|
|
300
|
+
if first_for_queue
|
|
301
|
+
Pgbus.logger.warn do
|
|
302
|
+
"[Pgbus] Consumer read an unroutable event from queue #{queue_name} " \
|
|
303
|
+
"(routing_key=#{routing_key.inspect}): no subscriber in this process owns that queue. " \
|
|
304
|
+
"Archiving. This usually means a stale topic binding — a handler was renamed or removed " \
|
|
305
|
+
"without unbinding its queue. Further unrouted messages on this queue are not logged."
|
|
306
|
+
end
|
|
307
|
+
end
|
|
308
|
+
|
|
309
|
+
Pgbus::Instrumentation.instrument(
|
|
310
|
+
"pgbus.event_unrouted",
|
|
311
|
+
queue_name: queue_name,
|
|
312
|
+
routing_key: routing_key
|
|
313
|
+
)
|
|
314
|
+
end
|
|
315
|
+
|
|
250
316
|
# Record a job stat for the handled message, mirroring the shape the
|
|
251
317
|
# executor pushes (Executor#record_stat) so consumer and worker throughput
|
|
252
318
|
# land in the same pgbus_job_stats table. No-op unless stats are enabled.
|
|
@@ -521,6 +587,7 @@ module Pgbus
|
|
|
521
587
|
# wait IS its drain window — bound it by the same knob workers use
|
|
522
588
|
# instead of a hardcoded 30s (issue #386).
|
|
523
589
|
@pool.wait_for_termination(config.drain_timeout)
|
|
590
|
+
VisibilityHeartbeat.stop
|
|
524
591
|
@stat_buffer&.stop
|
|
525
592
|
@heartbeat&.stop
|
|
526
593
|
restore_signals
|
|
@@ -0,0 +1,70 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module Pgbus
|
|
4
|
+
module Streams
|
|
5
|
+
# The thread-local channel that carries pgbus-specific broadcast options
|
|
6
|
+
# from a `Turbo::Broadcastable` / `Turbo::StreamsChannel` call site down to
|
|
7
|
+
# `TurboBroadcastable#broadcast_stream_to`, which is the only place with a
|
|
8
|
+
# `Pgbus::Streams::Stream` to hand them to.
|
|
9
|
+
#
|
|
10
|
+
# turbo-rails' broadcast helpers funnel everything through
|
|
11
|
+
# `broadcast_stream_to(*streamables, content:)` — a signature with nowhere
|
|
12
|
+
# to put `durable:`, `exclude:`, `visible_to:`, `event:` or `coalesce:`.
|
|
13
|
+
# Rather than fork turbo's whole helper surface, the patches pull those
|
|
14
|
+
# kwargs out before calling `super` and stash them here for the duration of
|
|
15
|
+
# the broadcast.
|
|
16
|
+
#
|
|
17
|
+
# Both patches (`BroadcastableOverride` on the model concern and
|
|
18
|
+
# `TurboBroadcastable` on the channel) share this module so there is one
|
|
19
|
+
# definition of which keys exist and how they are saved/restored.
|
|
20
|
+
module BroadcastOpts
|
|
21
|
+
# Options a caller may pass to any Turbo broadcast helper. They are
|
|
22
|
+
# deleted from the kwargs so they never reach turbo-rails' renderer.
|
|
23
|
+
KEYS = %i[durable exclude visible_to event coalesce].freeze
|
|
24
|
+
|
|
25
|
+
# `coalesce_target` is not a caller-facing kwarg — it is derived from the
|
|
26
|
+
# broadcast's own `target:`/`targets:` by the channel patch, because
|
|
27
|
+
# coalescing keys on `(stream, target)` and `broadcast_stream_to` never
|
|
28
|
+
# sees the target otherwise.
|
|
29
|
+
THREAD_LOCALS = {
|
|
30
|
+
durable: :pgbus_broadcast_durable,
|
|
31
|
+
exclude: :pgbus_broadcast_exclude,
|
|
32
|
+
visible_to: :pgbus_broadcast_visible_to,
|
|
33
|
+
event: :pgbus_broadcast_event,
|
|
34
|
+
coalesce: :pgbus_broadcast_coalesce,
|
|
35
|
+
coalesce_target: :pgbus_broadcast_coalesce_target
|
|
36
|
+
}.freeze
|
|
37
|
+
|
|
38
|
+
# Removes the pgbus options from `kwargs` (mutating it) and returns them.
|
|
39
|
+
# Keys the caller did not pass are absent from the result, so an outer
|
|
40
|
+
# `BroadcastOpts.with` block's values survive an inner broadcast that
|
|
41
|
+
# doesn't set them.
|
|
42
|
+
def self.extract!(kwargs)
|
|
43
|
+
KEYS.each_with_object({}) do |key, opts|
|
|
44
|
+
opts[key] = kwargs.delete(key) if kwargs.key?(key)
|
|
45
|
+
end
|
|
46
|
+
end
|
|
47
|
+
|
|
48
|
+
# Sets the thread-locals for the given options for the duration of the
|
|
49
|
+
# block, restoring the previous values afterwards (nesting-safe), even on
|
|
50
|
+
# error. Only the keys passed are touched.
|
|
51
|
+
def self.with(**opts)
|
|
52
|
+
previous = {}
|
|
53
|
+
|
|
54
|
+
opts.each do |key, value|
|
|
55
|
+
tl_key = THREAD_LOCALS.fetch(key)
|
|
56
|
+
previous[tl_key] = Thread.current[tl_key]
|
|
57
|
+
Thread.current[tl_key] = value
|
|
58
|
+
end
|
|
59
|
+
|
|
60
|
+
yield
|
|
61
|
+
ensure
|
|
62
|
+
previous.each { |tl_key, value| Thread.current[tl_key] = value }
|
|
63
|
+
end
|
|
64
|
+
|
|
65
|
+
def self.[](key)
|
|
66
|
+
Thread.current[THREAD_LOCALS.fetch(key)]
|
|
67
|
+
end
|
|
68
|
+
end
|
|
69
|
+
end
|
|
70
|
+
end
|
|
@@ -40,7 +40,7 @@ module Pgbus
|
|
|
40
40
|
# don't understand. We pull them out of kwargs (so they never reach
|
|
41
41
|
# turbo-rails' renderer) and thread them to broadcast_stream_to via
|
|
42
42
|
# thread-locals, mirroring the original durable: shim.
|
|
43
|
-
PGBUS_BROADCAST_OPTS =
|
|
43
|
+
PGBUS_BROADCAST_OPTS = BroadcastOpts::KEYS
|
|
44
44
|
|
|
45
45
|
BROADCAST_METHODS.each do |method_name|
|
|
46
46
|
define_method(method_name) do |*streamables, **kwargs|
|
|
@@ -128,32 +128,19 @@ module Pgbus
|
|
|
128
128
|
private
|
|
129
129
|
|
|
130
130
|
def extract_pgbus_broadcast_opts(kwargs)
|
|
131
|
-
|
|
132
|
-
opts[key] = kwargs.delete(key) if kwargs.key?(key)
|
|
133
|
-
end
|
|
131
|
+
BroadcastOpts.extract!(kwargs)
|
|
134
132
|
end
|
|
135
133
|
|
|
136
134
|
# Set the pgbus broadcast thread-locals for the duration of the block,
|
|
137
135
|
# restoring previous values afterwards (nested/concurrent-safe). Only
|
|
138
136
|
# keys actually passed are touched, so unrelated outer broadcasts keep
|
|
139
137
|
# their values.
|
|
140
|
-
def with_pgbus_broadcast_opts(durable: :__unset__, exclude: :__unset__, visible_to: :__unset__,
|
|
141
|
-
|
|
142
|
-
|
|
143
|
-
|
|
144
|
-
|
|
145
|
-
previous[tl_key] = Thread.current[tl_key]
|
|
146
|
-
Thread.current[tl_key] = value
|
|
147
|
-
end
|
|
148
|
-
|
|
149
|
-
set.call(:pgbus_broadcast_durable, durable)
|
|
150
|
-
set.call(:pgbus_broadcast_exclude, exclude)
|
|
151
|
-
set.call(:pgbus_broadcast_visible_to, visible_to)
|
|
152
|
-
set.call(:pgbus_broadcast_event, event)
|
|
138
|
+
def with_pgbus_broadcast_opts(durable: :__unset__, exclude: :__unset__, visible_to: :__unset__,
|
|
139
|
+
event: :__unset__, coalesce: :__unset__, &)
|
|
140
|
+
opts = { durable: durable, exclude: exclude, visible_to: visible_to, event: event, coalesce: coalesce }
|
|
141
|
+
opts.reject! { |_key, value| value == :__unset__ }
|
|
153
142
|
|
|
154
|
-
|
|
155
|
-
ensure
|
|
156
|
-
previous.each { |tl_key, value| Thread.current[tl_key] = value }
|
|
143
|
+
BroadcastOpts.with(**opts, &)
|
|
157
144
|
end
|
|
158
145
|
end
|
|
159
146
|
end
|
|
@@ -34,9 +34,57 @@ module Pgbus
|
|
|
34
34
|
# long as `Turbo.signed_stream_verifier_key` is set, which the Rails
|
|
35
35
|
# app is already responsible for).
|
|
36
36
|
module TurboBroadcastable
|
|
37
|
+
# Every targeted broadcast helper (`broadcast_replace_to`,
|
|
38
|
+
# `broadcast_append_to`, `broadcast_remove_to`, …) funnels through
|
|
39
|
+
# `broadcast_action_to`, so this is the one place that sees both the
|
|
40
|
+
# caller's kwargs and the `target:`/`targets:` the frame will carry.
|
|
41
|
+
#
|
|
42
|
+
# It does two things the model-level `BroadcastableOverride` can't:
|
|
43
|
+
#
|
|
44
|
+
# 1. Extracts the pgbus options from a *direct* channel call —
|
|
45
|
+
# `Turbo::StreamsChannel.broadcast_replace_to(..., coalesce: true)` —
|
|
46
|
+
# which never passes through `Turbo::Broadcastable` at all (this is
|
|
47
|
+
# the path phlex-reactive's `Streamable.broadcast_to` takes).
|
|
48
|
+
# 2. Records the coalescing key. `coalesce:` dedupes on
|
|
49
|
+
# `(stream, target)`, but `broadcast_stream_to` only receives the
|
|
50
|
+
# rendered `content:` — the target is gone by then. We resolve it the
|
|
51
|
+
# same way turbo will (`convert_to_turbo_stream_dom_id`), so a record
|
|
52
|
+
# target keys on its stable dom_id rather than its object id.
|
|
53
|
+
#
|
|
54
|
+
# Resolving the key is skipped entirely unless coalescing was actually
|
|
55
|
+
# requested, so the uncoalesced path stays byte-identical and free.
|
|
56
|
+
def broadcast_action_to(*streamables, action:, target: nil, targets: nil, **rendering)
|
|
57
|
+
opts = BroadcastOpts.extract!(rendering)
|
|
58
|
+
|
|
59
|
+
coalesce = opts.key?(:coalesce) ? opts[:coalesce] : BroadcastOpts[:coalesce]
|
|
60
|
+
opts[:coalesce_target] = pgbus_coalesce_target(target, targets) if coalesce
|
|
61
|
+
|
|
62
|
+
BroadcastOpts.with(**opts) do
|
|
63
|
+
super(*streamables, action: action, target: target, targets: targets, **rendering)
|
|
64
|
+
end
|
|
65
|
+
end
|
|
66
|
+
|
|
67
|
+
# The two helpers that don't pass through `broadcast_action_to`. They
|
|
68
|
+
# carry no target, so they can't coalesce (`Stream#broadcast` raises an
|
|
69
|
+
# actionable error, same as `Pgbus.stream(x).broadcast(coalesce:)`
|
|
70
|
+
# without one) — but they still need their pgbus options pulled out of
|
|
71
|
+
# the kwargs, or a direct channel call leaks `durable: true` into
|
|
72
|
+
# turbo's renderer and it ends up as an HTML attribute.
|
|
73
|
+
def broadcast_refresh_to(*streamables, **attributes)
|
|
74
|
+
BroadcastOpts.with(**BroadcastOpts.extract!(attributes)) do
|
|
75
|
+
super(*streamables, **attributes)
|
|
76
|
+
end
|
|
77
|
+
end
|
|
78
|
+
|
|
79
|
+
def broadcast_render_to(*streamables, **rendering)
|
|
80
|
+
BroadcastOpts.with(**BroadcastOpts.extract!(rendering)) do
|
|
81
|
+
super(*streamables, **rendering)
|
|
82
|
+
end
|
|
83
|
+
end
|
|
84
|
+
|
|
37
85
|
def broadcast_stream_to(*streamables, content:)
|
|
38
86
|
name = stream_name_from(streamables)
|
|
39
|
-
override =
|
|
87
|
+
override = BroadcastOpts[:durable]
|
|
40
88
|
# When no explicit thread-local override is present, let the config
|
|
41
89
|
# resolver decide: it checks `streams_durable_patterns` first (exact
|
|
42
90
|
# string or regex match), then falls back to
|
|
@@ -46,11 +94,24 @@ module Pgbus
|
|
|
46
94
|
durable = override.nil? ? Pgbus.configuration.stream_durable?(name) : override
|
|
47
95
|
Pgbus.stream(name, durable: durable).broadcast(
|
|
48
96
|
content,
|
|
49
|
-
exclude:
|
|
50
|
-
visible_to:
|
|
51
|
-
event:
|
|
97
|
+
exclude: BroadcastOpts[:exclude],
|
|
98
|
+
visible_to: BroadcastOpts[:visible_to],
|
|
99
|
+
event: BroadcastOpts[:event],
|
|
100
|
+
coalesce: BroadcastOpts[:coalesce],
|
|
101
|
+
target: BroadcastOpts[:coalesce_target]
|
|
52
102
|
)
|
|
53
103
|
end
|
|
104
|
+
|
|
105
|
+
private
|
|
106
|
+
|
|
107
|
+
# The coalescing key, resolved exactly as turbo resolves the rendered
|
|
108
|
+
# `target=`/`targets=` attribute. `targets:` (a CSS selector) is the
|
|
109
|
+
# fallback turbo itself uses when `target:` is absent; we drop the `#`
|
|
110
|
+
# selector prefix because the key is never rendered — it only has to be
|
|
111
|
+
# stable and distinct.
|
|
112
|
+
def pgbus_coalesce_target(target, targets)
|
|
113
|
+
convert_to_turbo_stream_dom_id(target) || convert_to_turbo_stream_dom_id(targets)
|
|
114
|
+
end
|
|
54
115
|
end
|
|
55
116
|
|
|
56
117
|
# Apply the patch to Turbo::StreamsChannel's singleton class. Idempotent:
|
data/lib/pgbus/testing.rb
CHANGED
|
@@ -58,7 +58,7 @@ module Pgbus
|
|
|
58
58
|
event = @mutex.synchronize { @events.first }
|
|
59
59
|
break unless event
|
|
60
60
|
|
|
61
|
-
Pgbus::EventBus::Registry.instance.
|
|
61
|
+
Pgbus::EventBus::Registry.instance.subscribers_matching(event.routing_key).each do |subscriber|
|
|
62
62
|
# Restore the publisher's Current (issue #431) like the consumer does.
|
|
63
63
|
Pgbus::CurrentAttributes.restore(event.context) { subscriber.handler_class.new.handle(event) }
|
|
64
64
|
end
|
data/lib/pgbus/version.rb
CHANGED
|
@@ -25,9 +25,15 @@ module Pgbus
|
|
|
25
25
|
# out with `pgbus_visibility_heartbeat false`.
|
|
26
26
|
module VisibilityHeartbeat
|
|
27
27
|
# `concurrency` is `[key, duration]` for a concurrency-limited job, else
|
|
28
|
-
# nil
|
|
28
|
+
# nil. `on_beat` is an optional callable run on every extension — the event
|
|
29
|
+
# consumer uses it to refresh its handlers' idempotency claims (issue #470).
|
|
30
|
+
#
|
|
31
|
+
# The ninth member takes the struct out of the 80-byte slot it used to fit
|
|
32
|
+
# (measured: 80 → 160). That is paid at most once per in-flight message, so
|
|
33
|
+
# the whole table is bounded by the execution pool's capacity — a handful of
|
|
34
|
+
# entries per process, not one per enqueued job.
|
|
29
35
|
Entry = Struct.new(:client, :queue_name, :prefixed, :msg_id, :job_class, :extended_at, :extensions,
|
|
30
|
-
:concurrency, keyword_init: true)
|
|
36
|
+
:concurrency, :on_beat, keyword_init: true)
|
|
31
37
|
|
|
32
38
|
# Per-job opt-out, included on ActiveJob::Base by the engine:
|
|
33
39
|
#
|
|
@@ -59,13 +65,14 @@ module Pgbus
|
|
|
59
65
|
# @param config [Pgbus::Configuration]
|
|
60
66
|
# @param concurrency [Array(String, Numeric), nil] semaphore key to keep alive alongside
|
|
61
67
|
# the message and how far to push its expiry on each beat
|
|
68
|
+
# @param on_beat [#call, nil] run after each extension; its failures are contained
|
|
62
69
|
def track(client:, queue_name:, msg_id:, prefixed: true, job_class: nil, config: Pgbus.configuration,
|
|
63
|
-
concurrency: nil)
|
|
70
|
+
concurrency: nil, on_beat: nil)
|
|
64
71
|
return yield unless config.visibility_heartbeat
|
|
65
72
|
|
|
66
73
|
entry = Entry.new(client: client, queue_name: queue_name, prefixed: prefixed, msg_id: msg_id.to_i,
|
|
67
74
|
job_class: job_class, extended_at: monotonic_now, extensions: 0,
|
|
68
|
-
concurrency: concurrency)
|
|
75
|
+
concurrency: concurrency, on_beat: on_beat)
|
|
69
76
|
register(entry, config)
|
|
70
77
|
begin
|
|
71
78
|
yield
|
|
@@ -130,6 +137,7 @@ module Pgbus
|
|
|
130
137
|
entry.extended_at = now
|
|
131
138
|
entry.extensions += 1
|
|
132
139
|
touch_semaphore(entry)
|
|
140
|
+
run_on_beat(entry)
|
|
133
141
|
Instrumentation.instrument(
|
|
134
142
|
"pgbus.job_visibility_extended",
|
|
135
143
|
queue: entry.queue_name, job_class: entry.job_class, msg_id: entry.msg_id, vt: vt,
|
|
@@ -186,6 +194,17 @@ module Pgbus
|
|
|
186
194
|
end
|
|
187
195
|
end
|
|
188
196
|
|
|
197
|
+
# Same containment as touch_semaphore: a lease the beat keeps alive
|
|
198
|
+
# alongside the message must never cost the message its extension.
|
|
199
|
+
def run_on_beat(entry)
|
|
200
|
+
entry.on_beat&.call
|
|
201
|
+
rescue StandardError => e
|
|
202
|
+
Pgbus.logger.warn do
|
|
203
|
+
"[Pgbus::VisibilityHeartbeat] on_beat hook failed for msg_id=#{entry.msg_id} " \
|
|
204
|
+
"queue=#{entry.queue_name}: #{e.class}: #{e.message}"
|
|
205
|
+
end
|
|
206
|
+
end
|
|
207
|
+
|
|
189
208
|
def forget_parent_entries!
|
|
190
209
|
return if @pid == ::Process.pid
|
|
191
210
|
|
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: pgbus
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.16.
|
|
4
|
+
version: 0.16.7
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Mikael Henriksson
|
|
@@ -291,9 +291,11 @@ files:
|
|
|
291
291
|
- lib/pgbus/engine.rb
|
|
292
292
|
- lib/pgbus/error_reporter.rb
|
|
293
293
|
- lib/pgbus/event.rb
|
|
294
|
+
- lib/pgbus/event_bus/claim_beat.rb
|
|
294
295
|
- lib/pgbus/event_bus/handler.rb
|
|
295
296
|
- lib/pgbus/event_bus/publisher.rb
|
|
296
297
|
- lib/pgbus/event_bus/registry.rb
|
|
298
|
+
- lib/pgbus/event_bus/stale_connection_retry.rb
|
|
297
299
|
- lib/pgbus/event_bus/subscriber.rb
|
|
298
300
|
- lib/pgbus/execution_pools.rb
|
|
299
301
|
- lib/pgbus/execution_pools/async_pool.rb
|
|
@@ -375,6 +377,7 @@ files:
|
|
|
375
377
|
- lib/pgbus/serializer.rb
|
|
376
378
|
- lib/pgbus/stat_buffer.rb
|
|
377
379
|
- lib/pgbus/streams.rb
|
|
380
|
+
- lib/pgbus/streams/broadcast_opts.rb
|
|
378
381
|
- lib/pgbus/streams/broadcastable_override.rb
|
|
379
382
|
- lib/pgbus/streams/coalescer.rb
|
|
380
383
|
- lib/pgbus/streams/cursor.rb
|