pgbus 0.9.0 → 0.9.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +16 -0
- data/app/assets/javascripts/pgbus/stream_source_element.js +150 -5
- data/app/views/pgbus/batches/_batches_table.html.erb +2 -1
- data/lib/pgbus/configuration.rb +107 -0
- data/lib/pgbus/execution_pools/async_pool.rb +44 -6
- data/lib/pgbus/process/notify_listener.rb +268 -0
- data/lib/pgbus/process/worker.rb +76 -3
- data/lib/pgbus/streams/coalescer.rb +88 -0
- data/lib/pgbus/streams/envelope.rb +20 -1
- data/lib/pgbus/streams/key.rb +35 -2
- data/lib/pgbus/streams/renderer.rb +67 -0
- data/lib/pgbus/streams.rb +150 -1
- data/lib/pgbus/version.rb +1 -1
- data/lib/pgbus/web/stream_app.rb +8 -4
- data/lib/pgbus/web/streamer/connection.rb +15 -1
- data/lib/pgbus/web/streamer/falcon_connection.rb +9 -1
- data/lib/pgbus/web/streamer/heartbeat.rb +23 -1
- data/lib/pgbus/web/streamer/stream_event_dispatcher.rb +129 -14
- data/lib/pgbus.rb +11 -0
- metadata +4 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 562d32e977685559437bdb1d3817fd02feb4fd9485b9101241a64d25d37162b1
|
|
4
|
+
data.tar.gz: eeacf0de199bfdc49c5b68523c5bbae7b1eff7970637742f3ec3e4391e4a3b06
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: a06cdb8eeecef32d6fa024f1fadac4f8c22758a11c91b1e9f5b00266869ea92b068c58a239a543c9409375d7155a59a5ec05f17cad59d7a23a49c90ecf74d7cd
|
|
7
|
+
data.tar.gz: e2a941feb09d3462bb9a83cda9b48821cf93649b53e9e7305af4a3b43b95bccca7690f40ebcb42857c4b9118ebeb592bcfacc317302e3a71b39be1c4f78493aa
|
data/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,21 @@
|
|
|
1
1
|
## [Unreleased]
|
|
2
2
|
|
|
3
|
+
### Added
|
|
4
|
+
|
|
5
|
+
- **Streams: optional publish-side coalescing for high-frequency broadcasts.** A chatty reactive component (live cursor, typing indicator, progress bar) can fan out many small broadcasts per second. Pass `coalesce:` (a window in milliseconds, or `true` for the 50ms default) with `target:` to `broadcast`/`broadcast_render` to batch per `(stream, target)` and publish only the *latest* frame within the window — superseded frames never hit the bus (no PGMQ insert, no NOTIFY, no fan-out). Last-write-wins, so it is opt-in and only safe for idempotent `replace`/`update` of a stable target (exactly the high-frequency case). The new `Pgbus::Streams::Coalescer` is a process-wide, thread-safe, trailing-edge-with-max-wait debounce: the first submit per window schedules the flush (bounding latency to one window) and later submits only overwrite the buffered payload; the flush re-enters the normal broadcast path, so a coalesced frame is just a deferred ordinary broadcast that still composes with `visible_to`/`exclude`/`event`/`durable`. Refs #171.
|
|
6
|
+
|
|
7
|
+
- **Streams: typed SSE event names on broadcasts.** A broadcast can set the SSE `event:` field — `Pgbus.stream(name).broadcast(html, event: "presence")` (and `broadcast_render(..., event:)`) — while keeping the payload a Turbo Stream, so clients route on the typed name instead of sniffing the HTML. The default (`nil` or `"turbo-stream"`) is omitted from the JSONB payload to avoid redundancy, but is still set on the SSE frame's `event:` line (the connection adapter falls back to `turbo-stream`), so default consumers still get the standard `message`/turbo-stream path. The event flows through the JSONB payload → `StreamEnvelope.event` → the SSE frame's `event:` line (both the Puma and Falcon connection adapters). On the client, `<pgbus-stream-source>` dispatches a typed broadcast as a generic `pgbus:event` (`{ event, data, msgId }`) and a named `pgbus:<event>` (`{ data, msgId }`) for `addEventListener` ergonomics; declare typed event names via the element's `listen-events` attribute so they survive the EventSource reconnect path. Refs #170.
|
|
8
|
+
|
|
9
|
+
- **Streams: optional connection-driven presence.** Streams matching `config.streams_presence_patterns` (exact string or Regexp, mirroring `streams_durable_patterns`) now auto-join a member on SSE connect, auto-leave on disconnect, and refresh `last_seen_at` on the keepalive heartbeat — no explicit `join`/`leave`/sweeper wiring. Identity comes from the connection's authorize-hook context: the built-in extractor handles a Hash with `:member_id`/`:id` (plus optional `:metadata`) or any object responding to `#id`, and `config.streams_presence_member` accepts a custom `->(context) { { id:, metadata: } }`. Membership work runs on the dispatcher thread (which already releases AR connections each pass); the heartbeat posts a batched `PresenceTouchMessage` per tick. Presence failures are logged and swallowed so a presence-DB hiccup can't knock a live SSE connection out of the registry. Anonymous connections (no derivable member) are simply skipped. Refs #169.
|
|
10
|
+
|
|
11
|
+
- **Streams: msg_id (revision) exposed to the client for optimistic-UI reconciliation.** Each delivered turbo-stream frame already carries its monotonic `msg_id` as the SSE `id:`; `<pgbus-stream-source>` now surfaces it two ways: the standard `message` MessageEvent sets `lastEventId` to the msg_id (Turbo ignores it; a reactive runtime listening for `message` reads the revision with no pgbus-specific API), and a new `pgbus:message` CustomEvent carries `{ msgId, data }` (msgId is a Number when numeric; a negative value marks an ephemeral frame). Documents the reconciliation pattern: track the highest applied msgId per target and skip the morph if a newer revision was already applied, so a late echo can't clobber a newer optimistic edit. Complements #165 (exclude handles the actor; the revision handles out-of-order delivery for everyone else). Refs #168.
|
|
12
|
+
|
|
13
|
+
- **Streams: `Stream#broadcast_render` — render a renderable and broadcast it in one call.** `Pgbus.stream(name).broadcast_render(renderable:, target:, action: :replace, exclude:, visible_to:, durable:)` renders a Phlex component, a ViewComponent, or a pre-rendered HTML string into a complete `<turbo-stream action target><template>…</template></turbo-stream>` tag and broadcasts it atomically — removing the off-request render + tag-building boilerplate (and the easy-to-get-wrong view context) from every call site. `action` defaults to `:replace`; content-less actions (`remove`) emit no `<template>`. `exclude:`/`visible_to:`/`durable:` forward to `#broadcast`, so actor-echo suppression and audience filtering compose. Self-contained (no turbo-rails/ActionView/Phlex dependency); the renderable is resolved via `String` → `#call` (Phlex) → `#render_in` (ViewComponent) → `#to_s`. Refs #166.
|
|
14
|
+
|
|
15
|
+
- **Streams: actor-echo suppression via `exclude:` on broadcasts.** A broadcast can now name a connection id to skip: `Pgbus.stream(name).broadcast(html, exclude: connection_id)` (and `visible_to:` composes with it). The dispatcher skips delivery to that one SSE connection, so an actor who triggered a change does not receive the echo of its own broadcast — it already applied the change via its action's HTTP response, and re-applying the SSE echo would double-apply (re-run animations, clobber optimistic edits). Every server-minted SSE connection now exposes its id to the page: the server sends a `pgbus:connected` frame right after the open handshake, and `<pgbus-stream-source>` captures it onto the element's `connection-id` attribute and re-dispatches it as a `pgbus:connected` event (also surfaced in `pgbus:open`'s detail). A page reads its connection id and sends it back as the `X-Pgbus-Connection` header on action requests; the broadcaster passes `exclude: request.headers["X-Pgbus-Connection"]`. Reuses the existing per-connection delivery (Filters) path. Refs #165.
|
|
16
|
+
|
|
17
|
+
- **Streams: `Pgbus.stream_key` is idempotent for an already-built key, plus `Pgbus.stream_key!`.** A single `String` argument is now treated as a pre-built pgbus stream key and returned unchanged (after the queue-name budget check) instead of tripping the colon-separator guard. This lets a consumer hold one `stream_key` value and pass it to both `turbo_stream_from` and the broadcaster without `stream_key("chat:lobby")` raising `ArgumentError`. The guard still fires for the genuinely ambiguous multi-fragment join (`stream_key("a:b", :c)`), and `Symbol`/record fragments with colons are still rejected (a colon there never came from `stream_key`). `Pgbus.stream_key!(key)` accepts a pre-built key explicitly (String required, budget enforced). Refs #167.
|
|
18
|
+
|
|
3
19
|
### Breaking Changes
|
|
4
20
|
|
|
5
21
|
- **Queue names must be alphanumeric and underscores only.** Queue names containing dashes (e.g., `my-app-queue`) will now raise `ArgumentError`. Rename to underscored form (e.g., `my_app_queue`) before upgrading. This restriction prevents SQL injection via PGMQ queue identifiers, which are interpolated into table names and cannot be parameterized.
|
|
@@ -17,12 +17,32 @@
|
|
|
17
17
|
// channel — compatibility shim; ignored
|
|
18
18
|
//
|
|
19
19
|
// Events (dispatched on the element):
|
|
20
|
-
//
|
|
20
|
+
// message (MessageEvent) data = turbo-stream HTML;
|
|
21
|
+
// lastEventId = the frame's msg_id (revision)
|
|
22
|
+
// pgbus:message { msgId, data } — same frame, for optimistic-UI
|
|
23
|
+
// reconciliation (skip morph if a newer rev for the
|
|
24
|
+
// target was already applied). See #168.
|
|
25
|
+
// pgbus:event { event, data, msgId } — a typed broadcast (any SSE
|
|
26
|
+
// event name other than turbo-stream). See #170.
|
|
27
|
+
// pgbus:<event> { data, msgId } — the same typed broadcast, named
|
|
28
|
+
// for ergonomic addEventListener("pgbus:presence").
|
|
29
|
+
// pgbus:open { lastEventId, connectionId }
|
|
30
|
+
// pgbus:connected { connectionId }
|
|
21
31
|
// pgbus:replay-start { fromId, toId }
|
|
22
32
|
// pgbus:replay-end {}
|
|
23
33
|
// pgbus:gap-detected { lastSeenId, archiveOldestId }
|
|
24
34
|
// pgbus:close { code, reason }
|
|
25
35
|
//
|
|
36
|
+
// Connection id (issue #165 — actor-echo suppression): the server sends a
|
|
37
|
+
// `pgbus:connected` frame right after the open handshake carrying the
|
|
38
|
+
// server-minted connection id. This element captures it, reflects it onto
|
|
39
|
+
// the `connection-id` attribute, and re-dispatches it as `pgbus:connected`.
|
|
40
|
+
// A page reads it (from the element or a `<meta name="pgbus-connection-id">`
|
|
41
|
+
// the app mirrors it to) and sends it back as the `X-Pgbus-Connection`
|
|
42
|
+
// header on action requests. The broadcaster then passes
|
|
43
|
+
// `exclude: request.headers["X-Pgbus-Connection"]` so the actor does not
|
|
44
|
+
// receive the echo of its own broadcast.
|
|
45
|
+
//
|
|
26
46
|
// The element integrates with Turbo via connectStreamSource /
|
|
27
47
|
// disconnectStreamSource + dispatching MessageEvent("message") so Turbo
|
|
28
48
|
// Stream HTML is automatically consumed by the existing StreamObserver.
|
|
@@ -48,9 +68,17 @@ class PgbusStreamSourceElement extends HTMLElement {
|
|
|
48
68
|
this.abortController = null
|
|
49
69
|
this.eventSource = null
|
|
50
70
|
this.lastEventId = null
|
|
71
|
+
this.connectionId = null
|
|
51
72
|
this.closed = false
|
|
52
73
|
}
|
|
53
74
|
|
|
75
|
+
// The server-minted connection id for this SSE connection, or null
|
|
76
|
+
// until the `pgbus:connected` frame arrives. Public read accessor so
|
|
77
|
+
// a reactive runtime can grab it without poking at attributes.
|
|
78
|
+
get pgbusConnectionId() {
|
|
79
|
+
return this.connectionId
|
|
80
|
+
}
|
|
81
|
+
|
|
54
82
|
connectedCallback() {
|
|
55
83
|
this.closed = false
|
|
56
84
|
connectStreamSource(this)
|
|
@@ -80,6 +108,12 @@ class PgbusStreamSourceElement extends HTMLElement {
|
|
|
80
108
|
// the URL. Parses the SSE event stream by hand because EventSource
|
|
81
109
|
// doesn't expose custom query strings uniformly across browsers.
|
|
82
110
|
async openFetchStream() {
|
|
111
|
+
// Each transport open is a fresh connection: the server mints a new id
|
|
112
|
+
// and sends a new pgbus:connected frame. Clear the previous id so
|
|
113
|
+
// pgbus:open (which fires before that frame arrives) can't surface a
|
|
114
|
+
// stale connection id — a stale id would produce a wrong X-Pgbus-Connection
|
|
115
|
+
// exclude header during reconnect windows.
|
|
116
|
+
this.resetConnectionId()
|
|
83
117
|
const url = this.buildUrl({ includeSince: true })
|
|
84
118
|
this.abortController = new AbortController()
|
|
85
119
|
|
|
@@ -99,7 +133,7 @@ class PgbusStreamSourceElement extends HTMLElement {
|
|
|
99
133
|
|
|
100
134
|
this.setAttribute("connected", "")
|
|
101
135
|
this.dispatchEvent(new CustomEvent("pgbus:open", {
|
|
102
|
-
detail: { lastEventId: this.lastEventId }
|
|
136
|
+
detail: { lastEventId: this.lastEventId, connectionId: this.connectionId }
|
|
103
137
|
}))
|
|
104
138
|
|
|
105
139
|
const reader = response.body.getReader()
|
|
@@ -137,13 +171,17 @@ class PgbusStreamSourceElement extends HTMLElement {
|
|
|
137
171
|
switchToEventSource() {
|
|
138
172
|
if (this.closed) return
|
|
139
173
|
|
|
174
|
+
// Fresh connection on reconnect — drop the previous connection id so
|
|
175
|
+
// pgbus:open can't emit a stale one before the new pgbus:connected
|
|
176
|
+
// frame lands. See openFetchStream.
|
|
177
|
+
this.resetConnectionId()
|
|
140
178
|
const url = this.buildUrl({ includeSince: true })
|
|
141
179
|
this.eventSource = new EventSource(url, { withCredentials: true })
|
|
142
180
|
|
|
143
181
|
this.eventSource.addEventListener("open", () => {
|
|
144
182
|
this.setAttribute("connected", "")
|
|
145
183
|
this.dispatchEvent(new CustomEvent("pgbus:open", {
|
|
146
|
-
detail: { lastEventId: this.lastEventId }
|
|
184
|
+
detail: { lastEventId: this.lastEventId, connectionId: this.connectionId }
|
|
147
185
|
}))
|
|
148
186
|
})
|
|
149
187
|
|
|
@@ -153,7 +191,11 @@ class PgbusStreamSourceElement extends HTMLElement {
|
|
|
153
191
|
|
|
154
192
|
this.eventSource.addEventListener("turbo-stream", (event) => {
|
|
155
193
|
this.lastEventId = event.lastEventId
|
|
156
|
-
this.
|
|
194
|
+
this.emitTurboStream(event.data, event.lastEventId)
|
|
195
|
+
})
|
|
196
|
+
|
|
197
|
+
this.eventSource.addEventListener("pgbus:connected", (event) => {
|
|
198
|
+
this.handleConnected(event.data)
|
|
157
199
|
})
|
|
158
200
|
|
|
159
201
|
this.eventSource.addEventListener("pgbus:gap-detected", (event) => {
|
|
@@ -166,6 +208,15 @@ class PgbusStreamSourceElement extends HTMLElement {
|
|
|
166
208
|
detail: { code: "shutdown", reason: "worker restart" }
|
|
167
209
|
}))
|
|
168
210
|
})
|
|
211
|
+
|
|
212
|
+
// Typed broadcasts (issue #170): EventSource only fires listeners
|
|
213
|
+
// registered by name, so we register one per declared typed event.
|
|
214
|
+
for (const name of this.declaredTypedEvents()) {
|
|
215
|
+
this.eventSource.addEventListener(name, (event) => {
|
|
216
|
+
if (event.lastEventId) this.lastEventId = event.lastEventId
|
|
217
|
+
this.emitTypedEvent(name, event.data, event.lastEventId)
|
|
218
|
+
})
|
|
219
|
+
}
|
|
169
220
|
}
|
|
170
221
|
|
|
171
222
|
// Parses a single SSE event block (: comment | id: ... | event: ... | data: ...)
|
|
@@ -185,7 +236,9 @@ class PgbusStreamSourceElement extends HTMLElement {
|
|
|
185
236
|
if (id !== null) this.lastEventId = id
|
|
186
237
|
|
|
187
238
|
if (event === "turbo-stream") {
|
|
188
|
-
this.
|
|
239
|
+
this.emitTurboStream(data, id)
|
|
240
|
+
} else if (event === "pgbus:connected") {
|
|
241
|
+
this.handleConnected(data)
|
|
189
242
|
} else if (event === "pgbus:gap-detected") {
|
|
190
243
|
this.dispatchEvent(new CustomEvent("pgbus:gap-detected", {
|
|
191
244
|
detail: this.safeJsonParse(data)
|
|
@@ -194,9 +247,101 @@ class PgbusStreamSourceElement extends HTMLElement {
|
|
|
194
247
|
this.dispatchEvent(new CustomEvent("pgbus:close", {
|
|
195
248
|
detail: { code: "shutdown", reason: "worker restart" }
|
|
196
249
|
}))
|
|
250
|
+
} else {
|
|
251
|
+
// A typed broadcast (issue #170): event: presence | reactive | ...
|
|
252
|
+
this.emitTypedEvent(event, data, id)
|
|
197
253
|
}
|
|
198
254
|
}
|
|
199
255
|
|
|
256
|
+
// Captures the server-minted connection id from a `pgbus:connected`
|
|
257
|
+
// frame: stores it, reflects it onto the `connection-id` attribute (so
|
|
258
|
+
// it's visible in the DOM / to MutationObservers), and re-dispatches it
|
|
259
|
+
// as a `pgbus:connected` CustomEvent. Idempotent across reconnects — the
|
|
260
|
+
// server mints a fresh id per connection, so a reconnect updates it.
|
|
261
|
+
handleConnected(data) {
|
|
262
|
+
const detail = this.safeJsonParse(data)
|
|
263
|
+
const connectionId = detail && detail.connectionId
|
|
264
|
+
if (!connectionId) return
|
|
265
|
+
|
|
266
|
+
this.connectionId = connectionId
|
|
267
|
+
this.setAttribute("connection-id", connectionId)
|
|
268
|
+
this.dispatchEvent(new CustomEvent("pgbus:connected", {
|
|
269
|
+
detail: { connectionId }
|
|
270
|
+
}))
|
|
271
|
+
}
|
|
272
|
+
|
|
273
|
+
// Clears the cached connection id and its reflected attribute. Called at
|
|
274
|
+
// the start of every transport open so a reconnect doesn't carry the
|
|
275
|
+
// previous connection's id into pgbus:open before the new pgbus:connected
|
|
276
|
+
// frame arrives.
|
|
277
|
+
resetConnectionId() {
|
|
278
|
+
this.connectionId = null
|
|
279
|
+
this.removeAttribute("connection-id")
|
|
280
|
+
}
|
|
281
|
+
|
|
282
|
+
// Delivers a turbo-stream frame to two audiences (issue #168):
|
|
283
|
+
//
|
|
284
|
+
// 1. Turbo's StreamObserver, via a `message` MessageEvent whose `data`
|
|
285
|
+
// is the turbo-stream HTML. The MessageEvent's standard
|
|
286
|
+
// `lastEventId` field carries the frame's msg_id, so a reactive
|
|
287
|
+
// runtime that listens for `message` can read the revision without
|
|
288
|
+
// any pgbus-specific API. (Turbo ignores lastEventId.)
|
|
289
|
+
//
|
|
290
|
+
// 2. A reactive runtime doing optimistic-UI reconciliation, via a
|
|
291
|
+
// `pgbus:message` CustomEvent carrying { msgId, data }. msgId is the
|
|
292
|
+
// monotonic per-stream revision (a negative value marks an ephemeral
|
|
293
|
+
// frame that was never persisted). Pattern: track the highest msgId
|
|
294
|
+
// you have applied per target; when a frame arrives, skip the morph
|
|
295
|
+
// if you have already applied a newer revision for that target —
|
|
296
|
+
// this stops a late echo from clobbering a newer optimistic edit.
|
|
297
|
+
// Complements #165: exclude handles the actor; this handles
|
|
298
|
+
// out-of-order delivery for everyone else.
|
|
299
|
+
//
|
|
300
|
+
// msgId is parsed to a Number when numeric so consumers can compare
|
|
301
|
+
// revisions with `>` directly; left as-is otherwise.
|
|
302
|
+
emitTurboStream(data, id) {
|
|
303
|
+
const msgId = id === null || id === undefined || id === "" ? null
|
|
304
|
+
: (Number.isNaN(Number(id)) ? id : Number(id))
|
|
305
|
+
|
|
306
|
+
const message = new MessageEvent("message", { data, lastEventId: id == null ? "" : String(id) })
|
|
307
|
+
this.dispatchEvent(message)
|
|
308
|
+
|
|
309
|
+
this.dispatchEvent(new CustomEvent("pgbus:message", {
|
|
310
|
+
detail: { msgId, data }
|
|
311
|
+
}))
|
|
312
|
+
}
|
|
313
|
+
|
|
314
|
+
// Delivers a typed broadcast (issue #170) — a frame whose SSE event name
|
|
315
|
+
// is something other than turbo-stream (e.g. "presence", "reactive").
|
|
316
|
+
// The payload is still whatever the broadcaster sent (usually a Turbo
|
|
317
|
+
// Stream); the typed name lets a client route without sniffing the HTML.
|
|
318
|
+
// Dispatched two ways for ergonomics:
|
|
319
|
+
// - a generic `pgbus:event` { event, data, msgId } (one listener for all)
|
|
320
|
+
// - a named `pgbus:<event>` { data, msgId } (addEventListener by name)
|
|
321
|
+
emitTypedEvent(event, data, id) {
|
|
322
|
+
const msgId = id === null || id === undefined || id === "" ? null
|
|
323
|
+
: (Number.isNaN(Number(id)) ? id : Number(id))
|
|
324
|
+
|
|
325
|
+
this.dispatchEvent(new CustomEvent("pgbus:event", {
|
|
326
|
+
detail: { event, data, msgId }
|
|
327
|
+
}))
|
|
328
|
+
this.dispatchEvent(new CustomEvent(`pgbus:${event}`, {
|
|
329
|
+
detail: { data, msgId }
|
|
330
|
+
}))
|
|
331
|
+
}
|
|
332
|
+
|
|
333
|
+
// Typed event names the EventSource (reconnect) path should listen for,
|
|
334
|
+
// declared by the app via the `listen-events` attribute (comma- or
|
|
335
|
+
// space-separated). EventSource only invokes listeners registered by
|
|
336
|
+
// name, so unlike the fetch path it cannot route unknown typed events
|
|
337
|
+
// generically; declaring them here keeps typed delivery working across
|
|
338
|
+
// reconnects. The fetch (initial) path always routes typed events.
|
|
339
|
+
declaredTypedEvents() {
|
|
340
|
+
const raw = this.getAttribute("listen-events")
|
|
341
|
+
if (!raw) return []
|
|
342
|
+
return raw.split(/[\s,]+/).filter((name) => name && name !== "turbo-stream")
|
|
343
|
+
}
|
|
344
|
+
|
|
200
345
|
buildUrl({ includeSince }) {
|
|
201
346
|
const src = this.getAttribute("src")
|
|
202
347
|
if (!includeSince || !this.lastEventId) return src
|
|
@@ -16,7 +16,8 @@
|
|
|
16
16
|
<tr class="hover:bg-gray-50 dark:hover:bg-gray-700/50">
|
|
17
17
|
<td data-label="Batch ID" class="px-4 py-3 text-sm">
|
|
18
18
|
<%= link_to batch[:batch_id][0..7], pgbus.batch_path(batch[:batch_id]),
|
|
19
|
-
class: "font-mono text-indigo-600 dark:text-indigo-400 hover:underline"
|
|
19
|
+
class: "font-mono text-indigo-600 dark:text-indigo-400 hover:underline",
|
|
20
|
+
data: { turbo_frame: "_top" } %>
|
|
20
21
|
</td>
|
|
21
22
|
<td data-label="Description" class="px-4 py-3 text-sm text-gray-700 dark:text-gray-300 max-w-xs truncate">
|
|
22
23
|
<%= batch[:description] || "—" %>
|
data/lib/pgbus/configuration.rb
CHANGED
|
@@ -113,9 +113,18 @@ module Pgbus
|
|
|
113
113
|
:streams_stats_enabled, :streams_test_mode,
|
|
114
114
|
:streams_orphan_sweep_interval, :streams_orphan_threshold,
|
|
115
115
|
:streams_durable_patterns,
|
|
116
|
+
:streams_presence_patterns, :streams_presence_member,
|
|
116
117
|
:streams_host, :streams_port, :streams_database_url
|
|
117
118
|
attr_reader :streams_default_broadcast_mode # rubocop:disable Style/AccessorGrouping
|
|
118
119
|
|
|
120
|
+
# NOTIFY-gated worker wakeups. When true, each Worker fork owns a
|
|
121
|
+
# dedicated NotifyListener PG connection that LISTENs on its queues'
|
|
122
|
+
# INSERT channels and wakes the loop on a real insert. Defaults to the
|
|
123
|
+
# value of listen_notify. The worker_notify_* overrides mirror
|
|
124
|
+
# streams_* so the LISTEN connection can bypass PgBouncer.
|
|
125
|
+
attr_accessor :worker_notify_wakeup,
|
|
126
|
+
:worker_notify_host, :worker_notify_port, :worker_notify_database_url
|
|
127
|
+
|
|
119
128
|
# AppSignal integration (auto-loaded when ::Appsignal is defined and this is true).
|
|
120
129
|
# Set to false to opt out without uninstalling the appsignal gem.
|
|
121
130
|
attr_accessor :appsignal_enabled, :appsignal_probe_enabled
|
|
@@ -170,6 +179,11 @@ module Pgbus
|
|
|
170
179
|
|
|
171
180
|
@listen_notify = true
|
|
172
181
|
|
|
182
|
+
@worker_notify_wakeup = nil
|
|
183
|
+
@worker_notify_host = nil
|
|
184
|
+
@worker_notify_port = nil
|
|
185
|
+
@worker_notify_database_url = nil
|
|
186
|
+
|
|
173
187
|
@pgmq_schema_mode = :auto
|
|
174
188
|
|
|
175
189
|
@event_consumers = nil
|
|
@@ -247,6 +261,15 @@ module Pgbus
|
|
|
247
261
|
@streams_orphan_sweep_interval = 3600 # 1 hour
|
|
248
262
|
@streams_orphan_threshold = 86_400 # 24 hours
|
|
249
263
|
@streams_durable_patterns = []
|
|
264
|
+
# Streams matching these patterns get connection-driven presence:
|
|
265
|
+
# auto-join on SSE connect, auto-leave on disconnect, touch on the
|
|
266
|
+
# keepalive heartbeat (issue #169). Empty by default (opt-in).
|
|
267
|
+
@streams_presence_patterns = []
|
|
268
|
+
# Extracts a presence member { id:, metadata: } from a connection's
|
|
269
|
+
# authorize-hook context. Defaults to nil, which uses the built-in
|
|
270
|
+
# extractor (see #presence_member_for): a Hash with :member_id/:id,
|
|
271
|
+
# or an object responding to #id.
|
|
272
|
+
@streams_presence_member = nil
|
|
250
273
|
|
|
251
274
|
# AppSignal: auto-on when the appsignal gem is loaded; probe runs in
|
|
252
275
|
# the same process, so the operator can disable it independently.
|
|
@@ -411,6 +434,12 @@ module Pgbus
|
|
|
411
434
|
|
|
412
435
|
raise ArgumentError, "streams_durable_patterns must be an Array of strings/regex" unless streams_durable_patterns.is_a?(Array)
|
|
413
436
|
|
|
437
|
+
raise ArgumentError, "streams_presence_patterns must be an Array of strings/regex" unless streams_presence_patterns.is_a?(Array)
|
|
438
|
+
|
|
439
|
+
if !streams_presence_member.nil? && !streams_presence_member.respond_to?(:call)
|
|
440
|
+
raise ArgumentError, "streams_presence_member must respond to #call (a Proc/lambda) or be nil"
|
|
441
|
+
end
|
|
442
|
+
|
|
414
443
|
return if streams_orphan_threshold.nil?
|
|
415
444
|
return if streams_orphan_threshold.is_a?(Numeric) && streams_orphan_threshold.positive?
|
|
416
445
|
|
|
@@ -427,6 +456,35 @@ module Pgbus
|
|
|
427
456
|
streams_default_broadcast_mode == :durable
|
|
428
457
|
end
|
|
429
458
|
|
|
459
|
+
# Returns true if the given stream name should have connection-driven
|
|
460
|
+
# presence based on `streams_presence_patterns` (exact string or
|
|
461
|
+
# Regexp match). Presence is opt-in, so the default (no patterns) is
|
|
462
|
+
# false. See issue #169.
|
|
463
|
+
def stream_presence?(name)
|
|
464
|
+
patterns = streams_presence_patterns || []
|
|
465
|
+
patterns.any? { |p| p.is_a?(Regexp) ? p.match?(name) : p == name }
|
|
466
|
+
end
|
|
467
|
+
|
|
468
|
+
# Derives a presence member { id:, metadata: } from a connection's
|
|
469
|
+
# authorize-hook context, or nil when no member can be derived (an
|
|
470
|
+
# anonymous connection — presence is simply skipped). Uses
|
|
471
|
+
# `streams_presence_member` when configured; otherwise the built-in
|
|
472
|
+
# extractor handles the common shapes:
|
|
473
|
+
# - Hash with :member_id (or :id) and optional :metadata
|
|
474
|
+
# - an object responding to #id (e.g. a User model)
|
|
475
|
+
# The id is always coerced to a String and metadata defaults to {}.
|
|
476
|
+
def presence_member_for(context)
|
|
477
|
+
return nil if context.nil?
|
|
478
|
+
|
|
479
|
+
raw = streams_presence_member ? streams_presence_member.call(context) : default_presence_member(context)
|
|
480
|
+
return nil unless raw.is_a?(Hash)
|
|
481
|
+
|
|
482
|
+
id = raw[:id]
|
|
483
|
+
return nil if id.nil? || id.to_s.empty?
|
|
484
|
+
|
|
485
|
+
{ id: id.to_s, metadata: raw[:metadata] || {} }
|
|
486
|
+
end
|
|
487
|
+
|
|
430
488
|
# Set the worker capsule list. Accepts:
|
|
431
489
|
#
|
|
432
490
|
# String — parsed via Pgbus::Configuration::CapsuleDSL into capsules
|
|
@@ -687,8 +745,57 @@ module Pgbus
|
|
|
687
745
|
end
|
|
688
746
|
end
|
|
689
747
|
|
|
748
|
+
# Connection options for the Worker's dedicated NotifyListener connection.
|
|
749
|
+
# Mirrors streams_connection_options: defaults to the base connection_options,
|
|
750
|
+
# overridable via worker_notify_database_url / worker_notify_host /
|
|
751
|
+
# worker_notify_port so the LISTEN connection can bypass PgBouncer.
|
|
752
|
+
def worker_notify_connection_options
|
|
753
|
+
return worker_notify_database_url if worker_notify_database_url
|
|
754
|
+
|
|
755
|
+
base = connection_options
|
|
756
|
+
return base unless worker_notify_host || worker_notify_port
|
|
757
|
+
|
|
758
|
+
case base
|
|
759
|
+
when Hash
|
|
760
|
+
result = base.dup
|
|
761
|
+
result[:host] = worker_notify_host if worker_notify_host
|
|
762
|
+
result[:port] = worker_notify_port if worker_notify_port
|
|
763
|
+
result
|
|
764
|
+
when String
|
|
765
|
+
parts = [base]
|
|
766
|
+
parts << "host=#{worker_notify_host}" if worker_notify_host
|
|
767
|
+
parts << "port=#{worker_notify_port}" if worker_notify_port
|
|
768
|
+
parts.join(" ")
|
|
769
|
+
else
|
|
770
|
+
base
|
|
771
|
+
end
|
|
772
|
+
end
|
|
773
|
+
|
|
774
|
+
# Resolved notify wakeup flag: defaults to listen_notify when nil.
|
|
775
|
+
def worker_notify_wakeup?
|
|
776
|
+
if @worker_notify_wakeup.nil?
|
|
777
|
+
listen_notify
|
|
778
|
+
else
|
|
779
|
+
@worker_notify_wakeup
|
|
780
|
+
end
|
|
781
|
+
end
|
|
782
|
+
|
|
690
783
|
private
|
|
691
784
|
|
|
785
|
+
# Built-in presence-member extractor used when no custom
|
|
786
|
+
# `streams_presence_member` is configured. Returns a { id:, metadata: }
|
|
787
|
+
# Hash or nil; #presence_member_for normalizes the id/metadata.
|
|
788
|
+
def default_presence_member(context)
|
|
789
|
+
if context.is_a?(Hash)
|
|
790
|
+
id = context[:member_id] || context[:id]
|
|
791
|
+
return nil if id.nil?
|
|
792
|
+
|
|
793
|
+
{ id: id, metadata: context[:metadata] || {} }
|
|
794
|
+
elsif context.respond_to?(:id)
|
|
795
|
+
{ id: context.id, metadata: {} }
|
|
796
|
+
end
|
|
797
|
+
end
|
|
798
|
+
|
|
692
799
|
# Coerce a duration setting value to a positive Numeric.
|
|
693
800
|
#
|
|
694
801
|
# Accepts an ActiveSupport::Duration (coerced to Integer seconds via .to_i)
|
|
@@ -5,8 +5,6 @@ module Pgbus
|
|
|
5
5
|
class AsyncPool
|
|
6
6
|
attr_reader :capacity
|
|
7
7
|
|
|
8
|
-
IDLE_WAIT_INTERVAL = 0.01
|
|
9
|
-
|
|
10
8
|
def initialize(capacity:, on_state_change: nil)
|
|
11
9
|
@capacity = capacity
|
|
12
10
|
@on_state_change = on_state_change
|
|
@@ -17,6 +15,7 @@ module Pgbus
|
|
|
17
15
|
@fatal_error = nil
|
|
18
16
|
@boot_queue = Thread::Queue.new
|
|
19
17
|
@pending = Thread::Queue.new
|
|
18
|
+
@wake_rd, @wake_wr = IO.pipe
|
|
20
19
|
|
|
21
20
|
validate_dependencies!
|
|
22
21
|
@reactor_thread = start_reactor
|
|
@@ -32,6 +31,7 @@ module Pgbus
|
|
|
32
31
|
reserve_capacity!
|
|
33
32
|
reserved = true
|
|
34
33
|
@pending << block
|
|
34
|
+
wake_reactor
|
|
35
35
|
rescue StandardError
|
|
36
36
|
restore_capacity if reserved
|
|
37
37
|
raise
|
|
@@ -52,6 +52,7 @@ module Pgbus
|
|
|
52
52
|
|
|
53
53
|
@shutdown_flag = true
|
|
54
54
|
end
|
|
55
|
+
wake_reactor
|
|
55
56
|
end
|
|
56
57
|
|
|
57
58
|
def shutdown?
|
|
@@ -96,7 +97,18 @@ module Pgbus
|
|
|
96
97
|
@boot_queue << :ready
|
|
97
98
|
|
|
98
99
|
wait_for_executions(semaphore)
|
|
99
|
-
wait_for_inflight
|
|
100
|
+
wait_for_inflight(task)
|
|
101
|
+
ensure
|
|
102
|
+
begin
|
|
103
|
+
@wake_rd.close
|
|
104
|
+
rescue IOError
|
|
105
|
+
nil
|
|
106
|
+
end
|
|
107
|
+
begin
|
|
108
|
+
@wake_wr.close
|
|
109
|
+
rescue IOError
|
|
110
|
+
nil
|
|
111
|
+
end
|
|
100
112
|
end
|
|
101
113
|
rescue Exception => e
|
|
102
114
|
register_fatal_error(e)
|
|
@@ -110,10 +122,35 @@ module Pgbus
|
|
|
110
122
|
schedule_pending(semaphore)
|
|
111
123
|
break if shutdown? && @pending.empty?
|
|
112
124
|
|
|
113
|
-
|
|
125
|
+
wait_for_wake if @pending.empty?
|
|
114
126
|
end
|
|
115
127
|
end
|
|
116
128
|
|
|
129
|
+
# Fiber-aware wait: yields the reactor fiber until the main thread
|
|
130
|
+
# writes a wake byte via wake_reactor. IO#wait_readable integrates
|
|
131
|
+
# with the Async scheduler so other fibers continue running.
|
|
132
|
+
def wait_for_wake
|
|
133
|
+
return if @wake_rd.closed?
|
|
134
|
+
|
|
135
|
+
@wake_rd.wait_readable
|
|
136
|
+
drain_wake_pipe
|
|
137
|
+
rescue IOError, Errno::EBADF
|
|
138
|
+
nil
|
|
139
|
+
end
|
|
140
|
+
|
|
141
|
+
def drain_wake_pipe
|
|
142
|
+
@wake_rd.read_nonblock(256)
|
|
143
|
+
rescue IOError, SystemCallError
|
|
144
|
+
nil
|
|
145
|
+
end
|
|
146
|
+
|
|
147
|
+
# Thread-safe: called from the main thread to wake the reactor fiber.
|
|
148
|
+
def wake_reactor
|
|
149
|
+
@wake_wr.write_nonblock(".")
|
|
150
|
+
rescue IO::WaitWritable, IOError, Errno::EBADF, Errno::EPIPE
|
|
151
|
+
nil
|
|
152
|
+
end
|
|
153
|
+
|
|
117
154
|
def schedule_pending(semaphore)
|
|
118
155
|
while (block = next_pending)
|
|
119
156
|
semaphore.async do
|
|
@@ -160,6 +197,7 @@ module Pgbus
|
|
|
160
197
|
@available_capacity += 1
|
|
161
198
|
@available_capacity.positive?
|
|
162
199
|
end
|
|
200
|
+
wake_reactor
|
|
163
201
|
@on_state_change&.call if should_notify
|
|
164
202
|
end
|
|
165
203
|
|
|
@@ -174,8 +212,8 @@ module Pgbus
|
|
|
174
212
|
raise error if error
|
|
175
213
|
end
|
|
176
214
|
|
|
177
|
-
def wait_for_inflight
|
|
178
|
-
sleep(
|
|
215
|
+
def wait_for_inflight(task)
|
|
216
|
+
task.sleep(0.01) while inflight?
|
|
179
217
|
end
|
|
180
218
|
|
|
181
219
|
def inflight?
|