pgbus 0.13.2 → 0.13.4
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +8 -0
- data/README.md +1 -0
- data/app/assets/javascripts/pgbus/stream_source_element.js +6 -2
- data/app/views/pgbus/dashboard/_queues_table.html.erb +5 -3
- data/app/views/pgbus/queues/_queues_list.html.erb +5 -3
- data/app/views/pgbus/queues/show.html.erb +3 -0
- data/config/locales/da.yml +7 -2
- data/config/locales/de.yml +7 -2
- data/config/locales/en.yml +7 -2
- data/config/locales/es.yml +7 -2
- data/config/locales/fi.yml +7 -2
- data/config/locales/fr.yml +7 -2
- data/config/locales/it.yml +7 -2
- data/config/locales/ja.yml +7 -2
- data/config/locales/nb.yml +7 -2
- data/config/locales/nl.yml +7 -2
- data/config/locales/pt.yml +7 -2
- data/config/locales/sv.yml +7 -2
- data/lib/pgbus/cli.rb +6 -4
- data/lib/pgbus/client/notify_stream.rb +25 -3
- data/lib/pgbus/client.rb +44 -0
- data/lib/pgbus/integrations/appsignal/probe.rb +8 -4
- data/lib/pgbus/mcp/tools/queues_tool.rb +6 -0
- data/lib/pgbus/streams/coalescer.rb +6 -0
- data/lib/pgbus/streams/envelope.rb +26 -11
- data/lib/pgbus/streams.rb +49 -1
- data/lib/pgbus/version.rb +1 -1
- data/lib/pgbus/web/data_source.rb +8 -1
- data/lib/pgbus/web/metrics_serializer.rb +9 -0
- metadata +1 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 9f59e5211aa8a7dedbbca32416bf5f82943848229ea5cf0a967f3681f14c4613
|
|
4
|
+
data.tar.gz: 8db3311f9117fbb1980982bb4f0de9a25ac610ff59898e8877631fdbc5c92f5f
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 478a033b2857060eec4240e8556e789af8055fdc34fc3c17639646159e168289e1f7e0ea80443137b9980294dde3d114488abbb76317e7a2f079d2df22e79fb5
|
|
7
|
+
data.tar.gz: 9e9ae02d538d96d30f141bed4a5da3e0420841da0364b452d0aa347b6ce242f9eae605514c148805542bdd24b44f949592e0d155f58b7a20f765e70231fd04b3
|
data/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,13 @@
|
|
|
1
1
|
## [Unreleased]
|
|
2
2
|
|
|
3
|
+
### Fixed
|
|
4
|
+
|
|
5
|
+
- **SSE delivery no longer strips newlines from broadcast payloads — multiline payloads are framed as consecutive `data:` lines per the SSE spec (issue #392).** `Streams::Envelope.message` collapsed `\r`/`\n` in the payload to nothing before writing the single `data:` line, silently corrupting any whitespace-significant broadcast (pre-formatted `<pre>` content, textarea seeds, JSON-in-data frames) on **both** the ephemeral and durable delivery paths — HTML's whitespace tolerance is why it went unnoticed. A multiline payload is now split on `\r\n`/`\r`/`\n` into consecutive `data:` lines, which EventSource clients rejoin with `\n`, making delivery lossless (a trailing newline survives via an empty final `data:` line; `\r` variants normalize to `\n` — SSE line terminators cannot be carried raw). The original injection defense is preserved: every payload line carries the `data:` prefix followed by one space, so a crafted payload still cannot forge `id:`/`event:` fields, and single-line fields (event names, comments) still strip newlines. The `<pgbus-stream-source>` element's fetch-path parser had the matching client-side bug — it joined `data:` lines without `\n` *and* `trim()`ed payload whitespace — and now follows EventSource semantics (join with `\n`, strip only the single leading space). Refs #392.
|
|
6
|
+
|
|
7
|
+
- **Ephemeral broadcasts over the PG NOTIFY payload cap no longer fail — loudly on the sync path, silently in the coalescer — they auto-degrade to a durable publish (issue #391).** Ephemeral frames ride the NOTIFY payload itself, which PostgreSQL caps below 8000 bytes. Any rendered-component broadcast (a progress card with Tailwind classes easily exceeds it) previously raised `PGMQ::Errors::ConnectionError: … payload string too long` — an error class that sent diagnosis toward the connection, not the payload — and on the `coalesce:` path that raise happened inside the coalescer's flush thread, reaching no caller, no ErrorReporter, no log: small frames delivered, big frames vanished, and the operator saw "SSE works but updates don't arrive". Three changes: **(1)** `Stream#broadcast` now measures the wrapped JSON before the NOTIFY and publishes an over-budget frame durably instead (payload stored in PGMQ, the queue's insert trigger fires the NOTIFY as a bare wake on the same channel the subscriber already LISTENs on) — delivery semantics preserved on both the sync and coalesced paths, warn-logged and instrumented (`pgbus.stream.broadcast` with `ephemeral_fallback: true`). **(2)** Direct `Client#notify_stream` callers get publish-time validation: a typed `Pgbus::Streams::PayloadTooLarge` raised at the call site for payloads exceeding `Pgbus::Client::NotifyStream::NOTIFY_PAYLOAD_LIMIT_BYTES` (7999 bytes, the largest accepted payload), naming the stream, the byte count, and the durable-mode escape hatch. **(3)** The coalescer's flush thread routes every flush error through `ErrorReporter` (same report-don't-log reasoning as #352) — a background thread swallowing delivery failures is invisible to APM by construction. ⚠️ **Upgrade note for 0.13 installs:** `streams_default_broadcast_mode` defaults to `:ephemeral`, and that default is a **behavior change** for apps broadcasting rendered components (what `broadcast_render`-style usage produces) — before this fix, any frame over ~8KB was silently lost or misdiagnosed. Durable is the right mode for turbo-stream UI regardless (since-id replay needs the archive): pin `config.streams_default_broadcast_mode = :durable`, or use `streams_durable_patterns` for the streams that need it; the auto-fallback now covers whatever stays ephemeral. Refs #391.
|
|
8
|
+
|
|
9
|
+
- **Queue-age metrics no longer count vt-parked (scheduled/retrying) messages — one delayed job stops reading as a degraded queue (issue #389).** ⚠️ **Behavior change on the AppSignal `pgbus_queue_latency` gauge.** pgmq's `oldest_msg_age_sec` is computed from `enqueued_at` and ignores `vt`, but a job enqueued with `wait:` or parked on a long retry backoff lives in the queue table with a future `vt` — that *is* the delayed-delivery mechanism. So a single parked message made the age metric grow at wall-clock rate for hours on an otherwise drained queue, and any latency alert thresholding on it fired continuously ("oldest message is 17045s old" on a healthy queue with depth 1, `read_ct` 0). Every metrics surface now also exposes **`oldest_claimable_age_sec`** — `now() - min(vt)` over rows with `vt <= now()`, i.e. the age of the oldest message actually *eligible for pickup*: an immediately-enqueued message contributes from enqueue time (matching the old number on a plain backlog), a scheduled/backoff-parked message contributes nothing until due, an in-flight message (vt pushed forward) is excluded, and nil means "no claimable backlog" even when the table is non-empty. Surfaces: `Web::DataSource` (dashboard, JSON API, MCP `pgbus_queues` tool), a new Prometheus gauge `pgbus_queue_oldest_claimable_age_seconds`, a new AppSignal gauge `pgbus_queue_oldest_claimable_age_seconds`, `Pgbus::Client#oldest_claimable_ages` (raw-SQL reader, since pgmq's `metrics_result` type is frozen upstream), and a CLAIMABLE column in `pgbus queues`. The AppSignal **`pgbus_queue_latency` gauge now derives from the claimable age** and always emits — `(claimable_age || 0) * 1000`, 0 = no claimable backlog — so existing latency alerts stop false-firing with no dashboard changes; the raw `pgbus_queue_oldest_message_age_seconds` gauge keeps its enqueue-time semantics everywhere. The dashboard queue tables additionally split depth into **Parked** (`depth − visible`) and show the claimable age in place of the raw age, so a queue holding only backoff retries reads visibly healthy. Refs #389.
|
|
10
|
+
|
|
3
11
|
### Added
|
|
4
12
|
|
|
5
13
|
- **Health-checked rolling restarts for the job container (issue #386).** ⚠️ **Behavior change on the standalone `/readyz`.** The supervisor's `health_port` server previously answered `/readyz` with the cluster-wide HealthAnalyzer verdict — so during a rolling deploy a freshly-booted container could pass an orchestrator's health gate on the strength of the *old* container's still-heartbeating workers, and the old container (with all its capacity) was stopped before the new one had forked a single child. The standalone `/readyz` is now **container-local**: 200 only when *this* supervisor verified its connection, bootstrapped queues, forked every configured child, and all of them are currently alive — with 503 bodies `BOOTING` (pre-boot), `DEGRADED` (a child died and is waiting out crash-restart backoff — precisely the state a deploy gate must fail on, keeping the old container running), and `DRAINING` (stop signal received). No database access on the probe path; the supervisor publishes an immutable snapshot per monitor pass and the accept thread reads it. The Rails-mounted `Pgbus::Web::HealthApp` keeps the cluster-wide verdict unchanged. Alongside it: **`pgbus-health`**, a shipped executable probe for docker `HEALTHCHECK` blocks (plain Ruby + stdlib sockets, loads neither Bundler nor the gem — cheap at 1–5s intervals, works in curl-less images; exit 0/1/2 = healthy/unhealthy/usage), and a README "Rolling restarts (Kamal, docker)" guide covering the healthcheck block, stop-timeout alignment, overlap-window duplicate-supervisor safety, and the `read_ct`-vs-deploy-kill DLQ caveat. Refs #386.
|
data/README.md
CHANGED
|
@@ -991,6 +991,7 @@ When `config.metrics_enabled = true` (default), the dashboard exposes Prometheus
|
|
|
991
991
|
|
|
992
992
|
| Metric | Description |
|
|
993
993
|
|--------|-------------|
|
|
994
|
+
| `pgbus_queue_oldest_claimable_age_seconds` | Age of the oldest message eligible for pickup (visibility timeout elapsed) — safe to alert on: scheduled/backoff-parked messages don't count until due; the series is omitted entirely when no claimable backlog exists (while the raw `pgbus_queue_oldest_message_age_seconds` gauge may still report a parked message's age) |
|
|
994
995
|
| `pgbus_table_dead_tuples` | Dead tuple count per PGMQ table |
|
|
995
996
|
| `pgbus_table_live_tuples` | Live tuple count per PGMQ table |
|
|
996
997
|
| `pgbus_table_bloat_ratio` | Dead / (dead + live) per table |
|
|
@@ -225,13 +225,17 @@ class PgbusStreamSourceElement extends HTMLElement {
|
|
|
225
225
|
|
|
226
226
|
let id = null
|
|
227
227
|
let event = "message"
|
|
228
|
-
|
|
228
|
+
const dataLines = []
|
|
229
229
|
|
|
230
230
|
for (const line of block.split("\n")) {
|
|
231
231
|
if (line.startsWith("id:")) id = line.slice(3).trim()
|
|
232
232
|
else if (line.startsWith("event:")) event = line.slice(6).trim()
|
|
233
|
-
|
|
233
|
+
// Per the SSE spec: strip only a single leading space after the colon
|
|
234
|
+
// (never trim — payload whitespace is significant, issue #392) and
|
|
235
|
+
// rejoin consecutive data: lines with \n, matching native EventSource.
|
|
236
|
+
else if (line.startsWith("data:")) dataLines.push(line.slice(5).replace(/^ /, ""))
|
|
234
237
|
}
|
|
238
|
+
const data = dataLines.join("\n")
|
|
235
239
|
|
|
236
240
|
if (id !== null) this.lastEventId = id
|
|
237
241
|
|
|
@@ -12,7 +12,8 @@
|
|
|
12
12
|
<th class="px-4 py-3 text-left text-xs font-medium uppercase text-gray-500"><%= t("pgbus.dashboard.queues_table.headers.queue") %></th>
|
|
13
13
|
<th class="px-4 py-3 text-right text-xs font-medium uppercase text-gray-500"><%= t("pgbus.dashboard.queues_table.headers.depth") %></th>
|
|
14
14
|
<th class="px-4 py-3 text-right text-xs font-medium uppercase text-gray-500"><%= t("pgbus.dashboard.queues_table.headers.visible") %></th>
|
|
15
|
-
<th class="px-4 py-3 text-right text-xs font-medium uppercase text-gray-500"><%= t("pgbus.dashboard.queues_table.headers.
|
|
15
|
+
<th class="px-4 py-3 text-right text-xs font-medium uppercase text-gray-500"><%= t("pgbus.dashboard.queues_table.headers.parked") %></th>
|
|
16
|
+
<th class="px-4 py-3 text-right text-xs font-medium uppercase text-gray-500"><%= t("pgbus.dashboard.queues_table.headers.oldest_claimable") %></th>
|
|
16
17
|
<th class="px-4 py-3 text-right text-xs font-medium uppercase text-gray-500"><%= t("pgbus.dashboard.queues_table.headers.total") %></th>
|
|
17
18
|
</tr>
|
|
18
19
|
</thead>
|
|
@@ -25,12 +26,13 @@
|
|
|
25
26
|
</td>
|
|
26
27
|
<td data-label="Depth" class="px-4 py-3 text-sm text-right text-gray-700"><%= pgbus_number(q[:queue_length]) %></td>
|
|
27
28
|
<td data-label="Visible" class="px-4 py-3 text-sm text-right text-gray-700"><%= pgbus_number(q[:queue_visible_length]) %></td>
|
|
28
|
-
<td data-label="
|
|
29
|
+
<td data-label="Parked" class="px-4 py-3 text-sm text-right text-gray-500"><%= pgbus_number(q[:parked_length]) %></td>
|
|
30
|
+
<td data-label="Oldest claimable" class="px-4 py-3 text-sm text-right text-gray-500"><%= q[:oldest_claimable_age_sec] || "—" %></td>
|
|
29
31
|
<td data-label="Total" class="px-4 py-3 text-sm text-right text-gray-500"><%= pgbus_number(q[:total_messages]) %></td>
|
|
30
32
|
</tr>
|
|
31
33
|
<% end %>
|
|
32
34
|
<% if @queues.empty? %>
|
|
33
|
-
<tr><td colspan="
|
|
35
|
+
<tr><td colspan="6" class="px-4 py-8 text-center text-sm text-gray-400"><%= t("pgbus.dashboard.queues_table.empty") %></td></tr>
|
|
34
36
|
<% end %>
|
|
35
37
|
</tbody>
|
|
36
38
|
</table>
|
|
@@ -6,7 +6,8 @@
|
|
|
6
6
|
<th class="px-4 py-3 text-left text-xs font-medium uppercase text-gray-500"><%= t("pgbus.queues.queues_list.headers.queue") %></th>
|
|
7
7
|
<th class="px-4 py-3 text-right text-xs font-medium uppercase text-gray-500"><%= t("pgbus.queues.queues_list.headers.depth") %></th>
|
|
8
8
|
<th class="px-4 py-3 text-right text-xs font-medium uppercase text-gray-500"><%= t("pgbus.queues.queues_list.headers.visible") %></th>
|
|
9
|
-
<th class="px-4 py-3 text-right text-xs font-medium uppercase text-gray-500"><%= t("pgbus.queues.queues_list.headers.
|
|
9
|
+
<th class="px-4 py-3 text-right text-xs font-medium uppercase text-gray-500"><%= t("pgbus.queues.queues_list.headers.parked") %></th>
|
|
10
|
+
<th class="px-4 py-3 text-right text-xs font-medium uppercase text-gray-500"><%= t("pgbus.queues.queues_list.headers.oldest_claimable") %></th>
|
|
10
11
|
<th class="px-4 py-3 text-right text-xs font-medium uppercase text-gray-500"><%= t("pgbus.queues.queues_list.headers.newest") %></th>
|
|
11
12
|
<th class="px-4 py-3 text-right text-xs font-medium uppercase text-gray-500"><%= t("pgbus.queues.queues_list.headers.total_ever") %></th>
|
|
12
13
|
<th class="px-4 py-3 text-right text-xs font-medium uppercase text-gray-500"><%= t("pgbus.queues.queues_list.headers.actions") %></th>
|
|
@@ -24,7 +25,8 @@
|
|
|
24
25
|
</td>
|
|
25
26
|
<td data-label="Depth" class="px-4 py-3 text-sm text-right font-mono text-gray-700"><%= pgbus_number(q[:queue_length]) %></td>
|
|
26
27
|
<td data-label="Visible" class="px-4 py-3 text-sm text-right font-mono text-gray-700"><%= pgbus_number(q[:queue_visible_length]) %></td>
|
|
27
|
-
<td data-label="
|
|
28
|
+
<td data-label="Parked" class="px-4 py-3 text-sm text-right font-mono text-gray-500"><%= pgbus_number(q[:parked_length]) %></td>
|
|
29
|
+
<td data-label="Oldest claimable" class="px-4 py-3 text-sm text-right text-gray-500"><%= q[:oldest_claimable_age_sec] || "—" %></td>
|
|
28
30
|
<td data-label="Newest" class="px-4 py-3 text-sm text-right text-gray-500"><%= q[:newest_msg_age_sec] || "—" %></td>
|
|
29
31
|
<td data-label="Total" class="px-4 py-3 text-sm text-right text-gray-500"><%= pgbus_number(q[:total_messages]) %></td>
|
|
30
32
|
<td data-label="Actions" class="px-4 py-3 text-sm text-right space-x-2">
|
|
@@ -51,7 +53,7 @@
|
|
|
51
53
|
</tr>
|
|
52
54
|
<% end %>
|
|
53
55
|
<% if @queues.empty? %>
|
|
54
|
-
<tr><td colspan="
|
|
56
|
+
<tr><td colspan="8" class="px-4 py-8 text-center text-sm text-gray-400"><%= t("pgbus.queues.queues_list.empty") %></td></tr>
|
|
55
57
|
<% end %>
|
|
56
58
|
</tbody>
|
|
57
59
|
</table>
|
|
@@ -13,6 +13,9 @@
|
|
|
13
13
|
<p class="text-sm text-gray-500 mt-1">
|
|
14
14
|
<%= t("pgbus.queues.show.depth") %> <span class="font-mono"><%= @queue[:queue_length] %></span> |
|
|
15
15
|
<%= t("pgbus.queues.show.visible") %> <span class="font-mono"><%= @queue[:queue_visible_length] %></span> |
|
|
16
|
+
<%= t("pgbus.queues.show.parked") %> <span class="font-mono"><%= @queue[:parked_length] %></span> |
|
|
17
|
+
<%= t("pgbus.queues.show.oldest") %> <span class="font-mono"><%= @queue[:oldest_msg_age_sec] || "—" %></span> |
|
|
18
|
+
<%= t("pgbus.queues.show.oldest_claimable") %> <span class="font-mono"><%= @queue[:oldest_claimable_age_sec] || "—" %></span> |
|
|
16
19
|
<%= t("pgbus.queues.show.total") %> <span class="font-mono"><%= pgbus_number(@queue[:total_messages]) %></span>
|
|
17
20
|
</p>
|
|
18
21
|
<% end %>
|
data/config/locales/da.yml
CHANGED
|
@@ -65,7 +65,8 @@ da:
|
|
|
65
65
|
empty: Ingen køer fundet
|
|
66
66
|
headers:
|
|
67
67
|
depth: Dybde
|
|
68
|
-
|
|
68
|
+
oldest_claimable: Ældste tilgængelig (s)
|
|
69
|
+
parked: Parkeret
|
|
69
70
|
queue: Kø
|
|
70
71
|
total: Total
|
|
71
72
|
visible: Synlig
|
|
@@ -475,7 +476,8 @@ da:
|
|
|
475
476
|
actions: Handlinger
|
|
476
477
|
depth: Dybde
|
|
477
478
|
newest: Nyeste (s)
|
|
478
|
-
|
|
479
|
+
oldest_claimable: Ældste tilgængelig (s)
|
|
480
|
+
parked: Parkeret
|
|
479
481
|
queue: Kø
|
|
480
482
|
total_ever: Total nogensinde
|
|
481
483
|
visible: Synlig
|
|
@@ -516,6 +518,9 @@ da:
|
|
|
516
518
|
scheduled: 'Planlagt:'
|
|
517
519
|
timezone: 'Tidszone:'
|
|
518
520
|
visible_at: 'Synlig fra:'
|
|
521
|
+
oldest: 'Ældste:'
|
|
522
|
+
oldest_claimable: 'Ældste tilgængelig:'
|
|
523
|
+
parked: 'Parkeret:'
|
|
519
524
|
pause: Pause
|
|
520
525
|
pause_confirm: Pause behandling?
|
|
521
526
|
purge_confirm: Rens alle beskeder?
|
data/config/locales/de.yml
CHANGED
|
@@ -65,7 +65,8 @@ de:
|
|
|
65
65
|
empty: Keine Warteschlangen gefunden
|
|
66
66
|
headers:
|
|
67
67
|
depth: Tiefe
|
|
68
|
-
|
|
68
|
+
oldest_claimable: Älteste verfügbar (s)
|
|
69
|
+
parked: Geparkt
|
|
69
70
|
queue: Warteschlange
|
|
70
71
|
total: Gesamt
|
|
71
72
|
visible: Sichtbar
|
|
@@ -475,7 +476,8 @@ de:
|
|
|
475
476
|
actions: Aktionen
|
|
476
477
|
depth: Tiefe
|
|
477
478
|
newest: Neueste (s)
|
|
478
|
-
|
|
479
|
+
oldest_claimable: Älteste verfügbar (s)
|
|
480
|
+
parked: Geparkt
|
|
479
481
|
queue: Warteschlange
|
|
480
482
|
total_ever: Insgesamt jemals
|
|
481
483
|
visible: Sichtbar
|
|
@@ -516,6 +518,9 @@ de:
|
|
|
516
518
|
scheduled: 'Geplant:'
|
|
517
519
|
timezone: 'Zeitzone:'
|
|
518
520
|
visible_at: 'Sichtbar ab:'
|
|
521
|
+
oldest: 'Älteste:'
|
|
522
|
+
oldest_claimable: 'Älteste verfügbar:'
|
|
523
|
+
parked: 'Geparkt:'
|
|
519
524
|
pause: Pause
|
|
520
525
|
pause_confirm: Verarbeitung pausieren?
|
|
521
526
|
purge_confirm: Alle Nachrichten löschen?
|
data/config/locales/en.yml
CHANGED
|
@@ -65,7 +65,8 @@ en:
|
|
|
65
65
|
empty: No queues found
|
|
66
66
|
headers:
|
|
67
67
|
depth: Depth
|
|
68
|
-
|
|
68
|
+
oldest_claimable: Oldest claimable (s)
|
|
69
|
+
parked: Parked
|
|
69
70
|
queue: Queue
|
|
70
71
|
total: Total
|
|
71
72
|
visible: Visible
|
|
@@ -475,7 +476,8 @@ en:
|
|
|
475
476
|
actions: Actions
|
|
476
477
|
depth: Depth
|
|
477
478
|
newest: Newest (s)
|
|
478
|
-
|
|
479
|
+
oldest_claimable: Oldest claimable (s)
|
|
480
|
+
parked: Parked
|
|
479
481
|
queue: Queue
|
|
480
482
|
total_ever: Total Ever
|
|
481
483
|
visible: Visible
|
|
@@ -516,6 +518,9 @@ en:
|
|
|
516
518
|
scheduled: 'Scheduled at:'
|
|
517
519
|
timezone: 'Timezone:'
|
|
518
520
|
visible_at: 'Visible at:'
|
|
521
|
+
oldest: 'Oldest:'
|
|
522
|
+
oldest_claimable: 'Oldest claimable:'
|
|
523
|
+
parked: 'Parked:'
|
|
519
524
|
pause: Pause
|
|
520
525
|
pause_confirm: Pause processing?
|
|
521
526
|
purge_confirm: Purge all messages?
|
data/config/locales/es.yml
CHANGED
|
@@ -65,7 +65,8 @@ es:
|
|
|
65
65
|
empty: No se encontraron colas
|
|
66
66
|
headers:
|
|
67
67
|
depth: Profundidad
|
|
68
|
-
|
|
68
|
+
oldest_claimable: Más antiguo disponible (s)
|
|
69
|
+
parked: Aparcados
|
|
69
70
|
queue: Cola
|
|
70
71
|
total: Total
|
|
71
72
|
visible: Visible
|
|
@@ -475,7 +476,8 @@ es:
|
|
|
475
476
|
actions: Acciones
|
|
476
477
|
depth: Profundidad
|
|
477
478
|
newest: Más nuevo (s)
|
|
478
|
-
|
|
479
|
+
oldest_claimable: Más antiguo disponible (s)
|
|
480
|
+
parked: Aparcados
|
|
479
481
|
queue: Cola
|
|
480
482
|
total_ever: Total acumulado
|
|
481
483
|
visible: Visible
|
|
@@ -516,6 +518,9 @@ es:
|
|
|
516
518
|
scheduled: 'Programado:'
|
|
517
519
|
timezone: 'Zona horaria:'
|
|
518
520
|
visible_at: 'Visible en:'
|
|
521
|
+
oldest: 'Más antiguo:'
|
|
522
|
+
oldest_claimable: 'Más antiguo disponible:'
|
|
523
|
+
parked: 'Aparcados:'
|
|
519
524
|
pause: Pausar
|
|
520
525
|
pause_confirm: "¿Pausar el procesamiento?"
|
|
521
526
|
purge_confirm: "¿Purgar todos los mensajes?"
|
data/config/locales/fi.yml
CHANGED
|
@@ -65,7 +65,8 @@ fi:
|
|
|
65
65
|
empty: Jonot eivät löytyneet
|
|
66
66
|
headers:
|
|
67
67
|
depth: Syvyys
|
|
68
|
-
|
|
68
|
+
oldest_claimable: Vanhin saatavilla (s)
|
|
69
|
+
parked: Pysäköidyt
|
|
69
70
|
queue: Jono
|
|
70
71
|
total: Yhteensä
|
|
71
72
|
visible: Näkyvissä
|
|
@@ -475,7 +476,8 @@ fi:
|
|
|
475
476
|
actions: Toiminnot
|
|
476
477
|
depth: Syvyys
|
|
477
478
|
newest: Uusimmat (s)
|
|
478
|
-
|
|
479
|
+
oldest_claimable: Vanhin saatavilla (s)
|
|
480
|
+
parked: Pysäköidyt
|
|
479
481
|
queue: Jono
|
|
480
482
|
total_ever: Yhteensä koskaan
|
|
481
483
|
visible: Näkyvissä
|
|
@@ -516,6 +518,9 @@ fi:
|
|
|
516
518
|
scheduled: 'Aikataulutettu:'
|
|
517
519
|
timezone: 'Aikavyöhyke:'
|
|
518
520
|
visible_at: 'Näkyvissä:'
|
|
521
|
+
oldest: 'Vanhin:'
|
|
522
|
+
oldest_claimable: 'Vanhin saatavilla:'
|
|
523
|
+
parked: 'Pysäköidyt:'
|
|
519
524
|
pause: Tauko
|
|
520
525
|
pause_confirm: Keskeytetäänkö käsittely?
|
|
521
526
|
purge_confirm: Tyhjennetäänkö kaikki viestit?
|
data/config/locales/fr.yml
CHANGED
|
@@ -65,7 +65,8 @@ fr:
|
|
|
65
65
|
empty: Aucune file d'attente trouvée
|
|
66
66
|
headers:
|
|
67
67
|
depth: Profondeur
|
|
68
|
-
|
|
68
|
+
oldest_claimable: Plus ancien disponible (s)
|
|
69
|
+
parked: Différés
|
|
69
70
|
queue: File d'attente
|
|
70
71
|
total: Total
|
|
71
72
|
visible: Visible
|
|
@@ -475,7 +476,8 @@ fr:
|
|
|
475
476
|
actions: Actions
|
|
476
477
|
depth: Profondeur
|
|
477
478
|
newest: Le plus récent (s)
|
|
478
|
-
|
|
479
|
+
oldest_claimable: Plus ancien disponible (s)
|
|
480
|
+
parked: Différés
|
|
479
481
|
queue: File d'attente
|
|
480
482
|
total_ever: Total jamais
|
|
481
483
|
visible: Visible
|
|
@@ -516,6 +518,9 @@ fr:
|
|
|
516
518
|
scheduled: 'Planifié :'
|
|
517
519
|
timezone: 'Fuseau horaire :'
|
|
518
520
|
visible_at: 'Visible à :'
|
|
521
|
+
oldest: 'Plus ancien :'
|
|
522
|
+
oldest_claimable: 'Plus ancien disponible :'
|
|
523
|
+
parked: 'Différés :'
|
|
519
524
|
pause: Pause
|
|
520
525
|
pause_confirm: Mettre en pause le traitement ?
|
|
521
526
|
purge_confirm: Purger tous les messages ?
|
data/config/locales/it.yml
CHANGED
|
@@ -65,7 +65,8 @@ it:
|
|
|
65
65
|
empty: Nessuna coda trovata
|
|
66
66
|
headers:
|
|
67
67
|
depth: Profondità
|
|
68
|
-
|
|
68
|
+
oldest_claimable: Più vecchio disponibile (s)
|
|
69
|
+
parked: Posticipati
|
|
69
70
|
queue: Coda
|
|
70
71
|
total: Totale
|
|
71
72
|
visible: Visibile
|
|
@@ -475,7 +476,8 @@ it:
|
|
|
475
476
|
actions: Azioni
|
|
476
477
|
depth: Profondità
|
|
477
478
|
newest: Più recente (s)
|
|
478
|
-
|
|
479
|
+
oldest_claimable: Più vecchio disponibile (s)
|
|
480
|
+
parked: Posticipati
|
|
479
481
|
queue: Coda
|
|
480
482
|
total_ever: Totale mai
|
|
481
483
|
visible: Visibile
|
|
@@ -516,6 +518,9 @@ it:
|
|
|
516
518
|
scheduled: 'Programmato:'
|
|
517
519
|
timezone: 'Fuso orario:'
|
|
518
520
|
visible_at: 'Visibile alle:'
|
|
521
|
+
oldest: 'Più vecchio:'
|
|
522
|
+
oldest_claimable: 'Più vecchio disponibile:'
|
|
523
|
+
parked: 'Posticipati:'
|
|
519
524
|
pause: Pausa
|
|
520
525
|
pause_confirm: Mettere in pausa l'elaborazione?
|
|
521
526
|
purge_confirm: Eliminare tutti i messaggi?
|
data/config/locales/ja.yml
CHANGED
|
@@ -65,7 +65,8 @@ ja:
|
|
|
65
65
|
empty: キューが見つかりません
|
|
66
66
|
headers:
|
|
67
67
|
depth: 深さ
|
|
68
|
-
|
|
68
|
+
oldest_claimable: 取得可能な最古 (秒)
|
|
69
|
+
parked: 待機中
|
|
69
70
|
queue: キュー
|
|
70
71
|
total: 合計
|
|
71
72
|
visible: 表示中
|
|
@@ -475,7 +476,8 @@ ja:
|
|
|
475
476
|
actions: アクション
|
|
476
477
|
depth: 深さ
|
|
477
478
|
newest: 最新 (秒)
|
|
478
|
-
|
|
479
|
+
oldest_claimable: 取得可能な最古 (秒)
|
|
480
|
+
parked: 待機中
|
|
479
481
|
queue: キュー
|
|
480
482
|
total_ever: 合計数
|
|
481
483
|
visible: 表示中
|
|
@@ -516,6 +518,9 @@ ja:
|
|
|
516
518
|
scheduled: スケジュール済み:
|
|
517
519
|
timezone: タイムゾーン:
|
|
518
520
|
visible_at: 表示可能日時:
|
|
521
|
+
oldest: '最古:'
|
|
522
|
+
oldest_claimable: '取得可能な最古:'
|
|
523
|
+
parked: '待機中:'
|
|
519
524
|
pause: 一時停止
|
|
520
525
|
pause_confirm: 処理を一時停止しますか?
|
|
521
526
|
purge_confirm: すべてのメッセージを削除しますか?
|
data/config/locales/nb.yml
CHANGED
|
@@ -65,7 +65,8 @@ nb:
|
|
|
65
65
|
empty: Ingen køer funnet
|
|
66
66
|
headers:
|
|
67
67
|
depth: Dybde
|
|
68
|
-
|
|
68
|
+
oldest_claimable: Eldste tilgjengelig (s)
|
|
69
|
+
parked: Parkert
|
|
69
70
|
queue: Kø
|
|
70
71
|
total: Totalt
|
|
71
72
|
visible: Synlig
|
|
@@ -475,7 +476,8 @@ nb:
|
|
|
475
476
|
actions: Handlinger
|
|
476
477
|
depth: Dybde
|
|
477
478
|
newest: Nyeste (s)
|
|
478
|
-
|
|
479
|
+
oldest_claimable: Eldste tilgjengelig (s)
|
|
480
|
+
parked: Parkert
|
|
479
481
|
queue: Kø
|
|
480
482
|
total_ever: Totalt noensinne
|
|
481
483
|
visible: Synlig
|
|
@@ -516,6 +518,9 @@ nb:
|
|
|
516
518
|
scheduled: 'Planlagt:'
|
|
517
519
|
timezone: 'Tidssone:'
|
|
518
520
|
visible_at: 'Synlig fra:'
|
|
521
|
+
oldest: 'Eldste:'
|
|
522
|
+
oldest_claimable: 'Eldste tilgjengelig:'
|
|
523
|
+
parked: 'Parkert:'
|
|
519
524
|
pause: Pause
|
|
520
525
|
pause_confirm: Pause behandling?
|
|
521
526
|
purge_confirm: Rens alle meldinger?
|
data/config/locales/nl.yml
CHANGED
|
@@ -65,7 +65,8 @@ nl:
|
|
|
65
65
|
empty: Geen wachtrijen gevonden
|
|
66
66
|
headers:
|
|
67
67
|
depth: Diepte
|
|
68
|
-
|
|
68
|
+
oldest_claimable: Oudste beschikbaar (s)
|
|
69
|
+
parked: Geparkeerd
|
|
69
70
|
queue: Wachtrij
|
|
70
71
|
total: Totaal
|
|
71
72
|
visible: Zichtbaar
|
|
@@ -475,7 +476,8 @@ nl:
|
|
|
475
476
|
actions: Acties
|
|
476
477
|
depth: Diepte
|
|
477
478
|
newest: Nieuwste (s)
|
|
478
|
-
|
|
479
|
+
oldest_claimable: Oudste beschikbaar (s)
|
|
480
|
+
parked: Geparkeerd
|
|
479
481
|
queue: Wachtrij
|
|
480
482
|
total_ever: Totaal ooit
|
|
481
483
|
visible: Zichtbaar
|
|
@@ -516,6 +518,9 @@ nl:
|
|
|
516
518
|
scheduled: 'Gepland:'
|
|
517
519
|
timezone: 'Tijdzone:'
|
|
518
520
|
visible_at: 'Zichtbaar op:'
|
|
521
|
+
oldest: 'Oudste:'
|
|
522
|
+
oldest_claimable: 'Oudste beschikbaar:'
|
|
523
|
+
parked: 'Geparkeerd:'
|
|
519
524
|
pause: Pauzeren
|
|
520
525
|
pause_confirm: Verwerking pauzeren?
|
|
521
526
|
purge_confirm: Alle berichten verwijderen?
|
data/config/locales/pt.yml
CHANGED
|
@@ -65,7 +65,8 @@ pt:
|
|
|
65
65
|
empty: Nenhuma fila encontrada
|
|
66
66
|
headers:
|
|
67
67
|
depth: Profundidade
|
|
68
|
-
|
|
68
|
+
oldest_claimable: Mais antigo disponível (s)
|
|
69
|
+
parked: Adiados
|
|
69
70
|
queue: Fila
|
|
70
71
|
total: Total
|
|
71
72
|
visible: Visível
|
|
@@ -475,7 +476,8 @@ pt:
|
|
|
475
476
|
actions: Ações
|
|
476
477
|
depth: Profundidade
|
|
477
478
|
newest: Mais novo (s)
|
|
478
|
-
|
|
479
|
+
oldest_claimable: Mais antigo disponível (s)
|
|
480
|
+
parked: Adiados
|
|
479
481
|
queue: Fila
|
|
480
482
|
total_ever: Total de todos os tempos
|
|
481
483
|
visible: Visível
|
|
@@ -516,6 +518,9 @@ pt:
|
|
|
516
518
|
scheduled: 'Agendado:'
|
|
517
519
|
timezone: 'Fuso horário:'
|
|
518
520
|
visible_at: 'Visível em:'
|
|
521
|
+
oldest: 'Mais antigo:'
|
|
522
|
+
oldest_claimable: 'Mais antigo disponível:'
|
|
523
|
+
parked: 'Adiados:'
|
|
519
524
|
pause: Pausar
|
|
520
525
|
pause_confirm: Pausar processamento?
|
|
521
526
|
purge_confirm: Limpar todas as mensagens?
|
data/config/locales/sv.yml
CHANGED
|
@@ -65,7 +65,8 @@ sv:
|
|
|
65
65
|
empty: Inga köer hittades
|
|
66
66
|
headers:
|
|
67
67
|
depth: Djup
|
|
68
|
-
|
|
68
|
+
oldest_claimable: Äldsta tillgänglig (s)
|
|
69
|
+
parked: Parkerade
|
|
69
70
|
queue: Kö
|
|
70
71
|
total: Totalt
|
|
71
72
|
visible: Synliga
|
|
@@ -475,7 +476,8 @@ sv:
|
|
|
475
476
|
actions: Åtgärder
|
|
476
477
|
depth: Djup
|
|
477
478
|
newest: Nyaste (s)
|
|
478
|
-
|
|
479
|
+
oldest_claimable: Äldsta tillgänglig (s)
|
|
480
|
+
parked: Parkerade
|
|
479
481
|
queue: Kö
|
|
480
482
|
total_ever: Totalt någonsin
|
|
481
483
|
visible: Synliga
|
|
@@ -516,6 +518,9 @@ sv:
|
|
|
516
518
|
scheduled: 'Schemalagt:'
|
|
517
519
|
timezone: 'Tidszon:'
|
|
518
520
|
visible_at: 'Synlig vid:'
|
|
521
|
+
oldest: 'Äldsta:'
|
|
522
|
+
oldest_claimable: 'Äldsta tillgänglig:'
|
|
523
|
+
parked: 'Parkerade:'
|
|
519
524
|
pause: Pausa
|
|
520
525
|
pause_confirm: Pausa bearbetning?
|
|
521
526
|
purge_confirm: Rensa alla meddelanden?
|
data/lib/pgbus/cli.rb
CHANGED
|
@@ -177,14 +177,16 @@ module Pgbus
|
|
|
177
177
|
def list_queues
|
|
178
178
|
Pgbus.client.list_queues
|
|
179
179
|
metrics = Pgbus.client.metrics
|
|
180
|
+
claimable_ages = Pgbus.client.oldest_claimable_ages
|
|
180
181
|
|
|
181
|
-
puts "QUEUE DEPTH VISIBLE OLDEST (s) TOTAL "
|
|
182
|
-
puts "-" *
|
|
182
|
+
puts "QUEUE DEPTH VISIBLE OLDEST (s) CLAIMABLE (s) TOTAL "
|
|
183
|
+
puts "-" * 111
|
|
183
184
|
|
|
184
185
|
Array(metrics).each do |m|
|
|
185
|
-
puts format("%-40s %-10s %-10s %-15s %-15s",
|
|
186
|
+
puts format("%-40s %-10s %-10s %-15s %-15s %-15s",
|
|
186
187
|
m.queue_name, m.queue_length, m.queue_visible_length,
|
|
187
|
-
m.oldest_msg_age_sec || "-", m.
|
|
188
|
+
m.oldest_msg_age_sec || "-", claimable_ages[m.queue_name] || "-",
|
|
189
|
+
m.total_messages)
|
|
188
190
|
end
|
|
189
191
|
end
|
|
190
192
|
|
|
@@ -14,15 +14,24 @@ module Pgbus
|
|
|
14
14
|
# no orphan tables.
|
|
15
15
|
#
|
|
16
16
|
# The payload is JSON-serialized into the NOTIFY's optional payload
|
|
17
|
-
# parameter
|
|
18
|
-
#
|
|
19
|
-
#
|
|
17
|
+
# parameter. Postgres caps NOTIFY payloads at < 8000 bytes; oversized
|
|
18
|
+
# payloads raise a typed Pgbus::Streams::PayloadTooLarge here, at the
|
|
19
|
+
# call site, instead of surfacing as a misleading
|
|
20
|
+
# PGMQ::Errors::ConnectionError ("payload string too long") from deep
|
|
21
|
+
# inside the driver (issue #391). Callers needing large payloads should
|
|
22
|
+
# use durable mode (which inserts into PGMQ).
|
|
20
23
|
module NotifyStream
|
|
24
|
+
# PostgreSQL rejects NOTIFY payloads of 8000 bytes or more
|
|
25
|
+
# ("payload string too long"), so 7999 is the largest deliverable
|
|
26
|
+
# payload.
|
|
27
|
+
NOTIFY_PAYLOAD_LIMIT_BYTES = 7999
|
|
28
|
+
|
|
21
29
|
def notify_stream(stream_name, payload)
|
|
22
30
|
full_name = config.queue_name(stream_name)
|
|
23
31
|
sanitized = QueueNameValidator.sanitize!(full_name)
|
|
24
32
|
channel = "pgmq.q_#{sanitized}.INSERT"
|
|
25
33
|
json = payload.is_a?(String) ? payload : JSON.generate(payload)
|
|
34
|
+
validate_notify_payload_size!(stream_name, json)
|
|
26
35
|
|
|
27
36
|
Instrumentation.instrument("pgbus.stream.notify", stream: stream_name, bytes: json.bytesize) do
|
|
28
37
|
with_stale_connection_retry do
|
|
@@ -34,6 +43,19 @@ module Pgbus
|
|
|
34
43
|
end
|
|
35
44
|
end
|
|
36
45
|
end
|
|
46
|
+
|
|
47
|
+
private
|
|
48
|
+
|
|
49
|
+
def validate_notify_payload_size!(stream_name, json)
|
|
50
|
+
return if json.bytesize <= NOTIFY_PAYLOAD_LIMIT_BYTES
|
|
51
|
+
|
|
52
|
+
raise Pgbus::Streams::PayloadTooLarge,
|
|
53
|
+
"Ephemeral broadcast on stream #{stream_name.inspect} is #{json.bytesize} bytes; " \
|
|
54
|
+
"PostgreSQL caps NOTIFY payloads at #{NOTIFY_PAYLOAD_LIMIT_BYTES} bytes. " \
|
|
55
|
+
"Use durable mode for large payloads (payload stored in PGMQ, NOTIFY as wake) — " \
|
|
56
|
+
"e.g. broadcast(..., durable: true), a streams_durable_patterns match, or " \
|
|
57
|
+
"streams_default_broadcast_mode = :durable."
|
|
58
|
+
end
|
|
37
59
|
end
|
|
38
60
|
end
|
|
39
61
|
end
|
data/lib/pgbus/client.rb
CHANGED
|
@@ -543,6 +543,38 @@ module Pgbus
|
|
|
543
543
|
end
|
|
544
544
|
end
|
|
545
545
|
|
|
546
|
+
# Age (seconds) of the oldest message actually eligible for pickup, i.e.
|
|
547
|
+
# whose visibility timeout has elapsed. Unlike pgmq's oldest_msg_age_sec
|
|
548
|
+
# (computed from enqueued_at), a scheduled or backoff-parked message —
|
|
549
|
+
# future vt — contributes nothing until it comes due, so a queue holding
|
|
550
|
+
# only parked messages reads nil ("no claimable backlog") instead of an
|
|
551
|
+
# age growing at wall-clock rate (issue #389). pgmq's metrics_result type
|
|
552
|
+
# is frozen upstream, so this lives here rather than in the SQL function.
|
|
553
|
+
#
|
|
554
|
+
# With a queue name: the age for that (prefixed) queue, or nil.
|
|
555
|
+
# Without: a hash of every physical queue in pgmq.meta to its age.
|
|
556
|
+
#
|
|
557
|
+
# Routes through the pooled @pgmq.with_connection (health-checked, bounded
|
|
558
|
+
# by the statement/socket timeouts applied at Client#initialize) rather
|
|
559
|
+
# than a fresh unbounded PG.connect per call — same rationale as
|
|
560
|
+
# notify_trigger_current?. synchronized: on the shared-Proc path @pgmq
|
|
561
|
+
# rides the AR raw connection, so the query must serialize against
|
|
562
|
+
# concurrent PGMQ operations. One checkout spans all per-queue queries;
|
|
563
|
+
# nothing nests inside it, so the shared pool_size=1 path is safe.
|
|
564
|
+
def oldest_claimable_ages(queue_name = nil)
|
|
565
|
+
synchronized do
|
|
566
|
+
@pgmq.with_connection do |conn|
|
|
567
|
+
if queue_name
|
|
568
|
+
claimable_age_for(conn, config.queue_name(queue_name))
|
|
569
|
+
else
|
|
570
|
+
names = conn.exec("SELECT queue_name FROM pgmq.meta ORDER BY queue_name")
|
|
571
|
+
.map { |row| row["queue_name"] }
|
|
572
|
+
names.to_h { |name| [name, claimable_age_for(conn, name)] }
|
|
573
|
+
end
|
|
574
|
+
end
|
|
575
|
+
end
|
|
576
|
+
end
|
|
577
|
+
|
|
546
578
|
# Snapshot of the PGMQ connection pool: {size:, available:, pool_timeout:}.
|
|
547
579
|
#
|
|
548
580
|
# Reads pgmq-ruby's own pool counters (@pgmq.stats -> {size:, available:})
|
|
@@ -948,6 +980,18 @@ module Pgbus
|
|
|
948
980
|
end
|
|
949
981
|
end
|
|
950
982
|
|
|
983
|
+
# queue_name is a physical (already prefixed) queue name; sanitized to a
|
|
984
|
+
# bare identifier before interpolation, same as the dashboard's DataSource.
|
|
985
|
+
def claimable_age_for(conn, queue_name)
|
|
986
|
+
qtable = "q_#{QueueNameValidator.sanitize!(queue_name)}"
|
|
987
|
+
row = conn.exec(<<~SQL).first
|
|
988
|
+
SELECT EXTRACT(epoch FROM (NOW() - min(vt)))::int AS age_sec
|
|
989
|
+
FROM pgmq.#{qtable}
|
|
990
|
+
WHERE vt <= NOW()
|
|
991
|
+
SQL
|
|
992
|
+
row && row["age_sec"]&.to_i
|
|
993
|
+
end
|
|
994
|
+
|
|
951
995
|
def with_raw_connection
|
|
952
996
|
opts = config.connection_options
|
|
953
997
|
owned = false
|
|
@@ -85,10 +85,14 @@ module Pgbus
|
|
|
85
85
|
gauge "queue_visible_depth", q[:queue_visible_length], tags
|
|
86
86
|
gauge "queue_paused", q[:paused] ? 1 : 0, tags
|
|
87
87
|
age = q[:oldest_msg_age_sec]
|
|
88
|
-
if age
|
|
89
|
-
|
|
90
|
-
|
|
91
|
-
|
|
88
|
+
gauge "queue_oldest_message_age_seconds", age, tags if age
|
|
89
|
+
claimable_age = q[:oldest_claimable_age_sec]
|
|
90
|
+
gauge "queue_oldest_claimable_age_seconds", claimable_age, tags if claimable_age
|
|
91
|
+
# Latency = time the oldest *claimable* message has waited for
|
|
92
|
+
# pickup; a queue holding only vt-parked (scheduled/backoff)
|
|
93
|
+
# messages is healthy, so 0 — not the raw enqueued_at age, which
|
|
94
|
+
# grows at wall-clock rate on a parked message (issue #389).
|
|
95
|
+
gauge "queue_latency", (claimable_age || 0) * 1_000, tags
|
|
92
96
|
end
|
|
93
97
|
rescue StandardError => e
|
|
94
98
|
log_failure("queue metrics", e)
|
|
@@ -14,6 +14,12 @@ module Pgbus
|
|
|
14
14
|
(messages whose visibility timeout has expired and are ready to be
|
|
15
15
|
claimed), oldest/newest message age in seconds, lifetime total, and
|
|
16
16
|
paused state. Use this to answer "are any queues backed up?".
|
|
17
|
+
oldest_claimable_age_sec is the age of the oldest message actually
|
|
18
|
+
eligible for pickup — nil means no message is currently claimable:
|
|
19
|
+
every remaining message is scheduled, backoff-parked, or in flight
|
|
20
|
+
with a future visibility timeout. A queue whose oldest_msg_age_sec
|
|
21
|
+
keeps growing while oldest_claimable_age_sec stays nil has no
|
|
22
|
+
starving backlog — nothing is waiting for a worker.
|
|
17
23
|
DESC
|
|
18
24
|
|
|
19
25
|
input_schema(properties: {}, required: [])
|
|
@@ -73,6 +73,12 @@ module Pgbus
|
|
|
73
73
|
return unless entry
|
|
74
74
|
|
|
75
75
|
@flush.call(stream_name: stream_name, target: target, payload: entry.payload, opts: entry.opts)
|
|
76
|
+
rescue StandardError => e
|
|
77
|
+
# The flush runs on the scheduler's thread — a raise here reaches no
|
|
78
|
+
# caller, so a swallowed error is invisible to APM by construction
|
|
79
|
+
# (issue #391: an oversized ephemeral frame died here without a
|
|
80
|
+
# trace). Route through ErrorReporter so configured reporters see it.
|
|
81
|
+
ErrorReporter.report(e, { component: "streams.coalescer", stream: stream_name, target: target })
|
|
76
82
|
end
|
|
77
83
|
|
|
78
84
|
# Default scheduler backed by Concurrent::ScheduledTask. Kept as a
|
|
@@ -12,13 +12,17 @@ module Pgbus
|
|
|
12
12
|
# - `comment(text)` — a heartbeat or sentinel that the SSE parser ignores
|
|
13
13
|
# - `retry_directive(ms)` — tells `EventSource` how long to wait before reconnecting
|
|
14
14
|
#
|
|
15
|
-
# All frames end with `\n\n` (the SSE event terminator).
|
|
16
|
-
#
|
|
17
|
-
#
|
|
18
|
-
#
|
|
19
|
-
#
|
|
15
|
+
# All frames end with `\n\n` (the SSE event terminator). A multiline payload is
|
|
16
|
+
# framed as consecutive `data:` lines (issue #392) — the client rejoins them with
|
|
17
|
+
# `\n`, so delivery is lossless. `\r\n` and lone `\r` are also SSE line terminators,
|
|
18
|
+
# so they become `data:` line breaks too (rejoined as `\n`; SSE cannot represent a
|
|
19
|
+
# raw `\r`). Every payload line carries the `data: ` prefix, so a crafted payload
|
|
20
|
+
# cannot inject forged id:/event: fields. Single-line fields (`event:`, comments)
|
|
21
|
+
# still strip newlines — there a `\r`/`\n` would terminate the field early and
|
|
22
|
+
# permit SSE field injection.
|
|
20
23
|
module Envelope
|
|
21
24
|
NEWLINES = /[\r\n]+/
|
|
25
|
+
DATA_LINE_BREAK = /\r\n|\r|\n/
|
|
22
26
|
|
|
23
27
|
RESPONSE_HEADERS = "HTTP/1.1 200 OK\r\n" \
|
|
24
28
|
"content-type: text/event-stream\r\n" \
|
|
@@ -31,11 +35,13 @@ module Pgbus
|
|
|
31
35
|
raise ArgumentError, "id is required" if id.nil?
|
|
32
36
|
raise ArgumentError, "event is required" if event.nil? || event.to_s.empty?
|
|
33
37
|
|
|
34
|
-
#
|
|
35
|
-
#
|
|
36
|
-
#
|
|
37
|
-
#
|
|
38
|
-
|
|
38
|
+
# The event name is a single SSE field line, so newlines are stripped —
|
|
39
|
+
# an unescaped \r/\n would terminate the field early and let a crafted
|
|
40
|
+
# value inject extra SSE fields (a forged id:/data:) into the frame.
|
|
41
|
+
# The payload is framed as one `data:` line per payload line instead:
|
|
42
|
+
# every line carries the `data: ` prefix, which is both spec-correct
|
|
43
|
+
# (the client rejoins with \n) and injection-safe.
|
|
44
|
+
"id: #{id}\nevent: #{strip_newlines(event.to_s)}\n#{data_lines(data.to_s)}\n"
|
|
39
45
|
end
|
|
40
46
|
|
|
41
47
|
def self.comment(text)
|
|
@@ -71,7 +77,16 @@ module Pgbus
|
|
|
71
77
|
str.gsub(NEWLINES, "")
|
|
72
78
|
end
|
|
73
79
|
|
|
74
|
-
|
|
80
|
+
# One `data: <line>\n` per payload line. The -1 limit keeps trailing
|
|
81
|
+
# empty strings, so a payload ending in \n round-trips as an empty
|
|
82
|
+
# final `data:` line (the client's rejoin restores the newline).
|
|
83
|
+
def self.data_lines(str)
|
|
84
|
+
lines = str.split(DATA_LINE_BREAK, -1)
|
|
85
|
+
lines = [""] if lines.empty? # "".split → [] — an empty payload still gets its data: line
|
|
86
|
+
lines.map { |line| "data: #{line}\n" }.join
|
|
87
|
+
end
|
|
88
|
+
|
|
89
|
+
private_class_method :strip_newlines, :data_lines
|
|
75
90
|
end
|
|
76
91
|
end
|
|
77
92
|
end
|
data/lib/pgbus/streams.rb
CHANGED
|
@@ -15,6 +15,18 @@ module Pgbus
|
|
|
15
15
|
# this specifically can rescue Pgbus::Streams::StreamNameTooLong.
|
|
16
16
|
class StreamNameTooLong < ArgumentError; end
|
|
17
17
|
|
|
18
|
+
# Raised when an ephemeral broadcast's JSON payload exceeds PostgreSQL's
|
|
19
|
+
# NOTIFY payload budget (< 8000 bytes). Ephemeral frames ride the NOTIFY
|
|
20
|
+
# itself, so the cap is a hard PostgreSQL limit — durable mode (payload
|
|
21
|
+
# stored in PGMQ, NOTIFY as a bare wake) has no such cap.
|
|
22
|
+
#
|
|
23
|
+
# Stream#broadcast never raises this: an oversized ephemeral frame
|
|
24
|
+
# auto-degrades to a durable publish (issue #391). The error exists for
|
|
25
|
+
# direct Client#notify_stream callers, where the previous failure mode
|
|
26
|
+
# was a misleading PGMQ::Errors::ConnectionError ("payload string too
|
|
27
|
+
# long") that pointed diagnosis at the connection instead of the payload.
|
|
28
|
+
class PayloadTooLarge < Pgbus::Error; end
|
|
29
|
+
|
|
18
30
|
# The default SSE `event:` name for a broadcast frame. Turbo's
|
|
19
31
|
# StreamObserver consumes frames the client re-dispatches as the
|
|
20
32
|
# `message` DOM event; the client maps this SSE event name to
|
|
@@ -261,11 +273,47 @@ module Pgbus
|
|
|
261
273
|
|
|
262
274
|
private
|
|
263
275
|
|
|
276
|
+
# Ephemeral frames ride the NOTIFY payload itself, which PostgreSQL
|
|
277
|
+
# caps below 8000 bytes. A frame over the cap auto-degrades to a
|
|
278
|
+
# durable publish (issue #391): payload stored in PGMQ, the queue's
|
|
279
|
+
# insert trigger fires the NOTIFY as a bare wake on the same channel
|
|
280
|
+
# the subscriber already LISTENs on — delivery semantics preserved,
|
|
281
|
+
# cap irrelevant. The JSON is generated once here and passed
|
|
282
|
+
# pre-serialized to notify_stream so the size check costs no extra
|
|
283
|
+
# allocation on the hot path.
|
|
264
284
|
def broadcast_ephemeral(wrapped)
|
|
265
|
-
|
|
285
|
+
json = JSON.generate(wrapped)
|
|
286
|
+
return durable_fallback(wrapped, json.bytesize) if json.bytesize > Client::NotifyStream::NOTIFY_PAYLOAD_LIMIT_BYTES
|
|
287
|
+
|
|
288
|
+
@client.notify_stream(@name, json)
|
|
266
289
|
nil
|
|
267
290
|
end
|
|
268
291
|
|
|
292
|
+
# The durable degrade path for an oversized ephemeral frame. Stays
|
|
293
|
+
# fire-and-forget like the ephemeral path it replaces: no after_commit
|
|
294
|
+
# deferral (pg_notify runs on the PGMQ pool connection, outside the
|
|
295
|
+
# request's AR transaction, so the ephemeral path never deferred
|
|
296
|
+
# either). Returns the msg_id like any durable publish.
|
|
297
|
+
def durable_fallback(wrapped, bytes)
|
|
298
|
+
Pgbus.logger.warn do
|
|
299
|
+
"[Pgbus::Streams] ephemeral broadcast on #{@name.inspect} is #{bytes} bytes " \
|
|
300
|
+
"(NOTIFY cap is #{Client::NotifyStream::NOTIFY_PAYLOAD_LIMIT_BYTES}); " \
|
|
301
|
+
"publishing durably instead. Consider durable mode for this stream " \
|
|
302
|
+
"(streams_durable_patterns or durable: true) to skip this check."
|
|
303
|
+
end
|
|
304
|
+
ensure_queue!
|
|
305
|
+
instrument_payload = {
|
|
306
|
+
stream: @name,
|
|
307
|
+
visible_to: wrapped["visible_to"],
|
|
308
|
+
deferred: false,
|
|
309
|
+
bytes: wrapped["html"].bytesize,
|
|
310
|
+
ephemeral_fallback: true
|
|
311
|
+
}
|
|
312
|
+
Instrumentation.instrument("pgbus.stream.broadcast", instrument_payload) do
|
|
313
|
+
@client.send_stream_message(@name, wrapped)
|
|
314
|
+
end
|
|
315
|
+
end
|
|
316
|
+
|
|
269
317
|
# Submits a frame to the process-wide coalescer instead of
|
|
270
318
|
# broadcasting now. Requires a target (the dedupe key — there's no
|
|
271
319
|
# way to last-write-win without one). The window is `coalesce` in ms
|
data/lib/pgbus/version.rb
CHANGED
|
@@ -1177,7 +1177,8 @@ module Pgbus
|
|
|
1177
1177
|
(SELECT EXTRACT(epoch FROM (NOW() - min(enqueued_at)))::int FROM pgmq.#{qtable}) AS oldest_msg_age_sec,
|
|
1178
1178
|
(SELECT CASE WHEN is_called THEN last_value ELSE 0 END FROM pgmq.#{seq_name}) AS total_messages,
|
|
1179
1179
|
(SELECT max(read_ct) FROM pgmq.#{qtable}) AS max_read_ct,
|
|
1180
|
-
(SELECT count(*) FROM pgmq.#{qtable} WHERE vt <= NOW() AND read_ct = 0) AS visible_unread_length
|
|
1180
|
+
(SELECT count(*) FROM pgmq.#{qtable} WHERE vt <= NOW() AND read_ct = 0) AS visible_unread_length,
|
|
1181
|
+
(SELECT EXTRACT(epoch FROM (NOW() - min(vt)))::int FROM pgmq.#{qtable} WHERE vt <= NOW()) AS oldest_claimable_age_sec
|
|
1181
1182
|
SQL
|
|
1182
1183
|
rescue StandardError => e
|
|
1183
1184
|
Pgbus.logger.debug { "[Pgbus::Web] Skipping queue metrics for #{name}: #{e.message}" }
|
|
@@ -1193,7 +1194,9 @@ module Pgbus
|
|
|
1193
1194
|
name: row["queue_name"],
|
|
1194
1195
|
queue_length: row["queue_length"].to_i,
|
|
1195
1196
|
queue_visible_length: row["queue_visible_length"].to_i,
|
|
1197
|
+
parked_length: row["queue_length"].to_i - row["queue_visible_length"].to_i,
|
|
1196
1198
|
oldest_msg_age_sec: row["oldest_msg_age_sec"]&.to_i,
|
|
1199
|
+
oldest_claimable_age_sec: row["oldest_claimable_age_sec"]&.to_i,
|
|
1197
1200
|
newest_msg_age_sec: row["newest_msg_age_sec"]&.to_i,
|
|
1198
1201
|
total_messages: row["total_messages"].to_i,
|
|
1199
1202
|
max_read_ct: row["max_read_ct"]&.to_i,
|
|
@@ -1216,6 +1219,7 @@ module Pgbus
|
|
|
1216
1219
|
count(CASE WHEN vt <= NOW() THEN 1 END) AS queue_visible_length,
|
|
1217
1220
|
EXTRACT(epoch FROM (NOW() - max(enqueued_at)))::int AS newest_msg_age_sec,
|
|
1218
1221
|
EXTRACT(epoch FROM (NOW() - min(enqueued_at)))::int AS oldest_msg_age_sec,
|
|
1222
|
+
EXTRACT(epoch FROM (NOW() - min(vt) FILTER (WHERE vt <= NOW())))::int AS oldest_claimable_age_sec,
|
|
1219
1223
|
max(read_ct) AS max_read_ct,
|
|
1220
1224
|
count(CASE WHEN vt <= NOW() AND read_ct = 0 THEN 1 END) AS visible_unread_length
|
|
1221
1225
|
FROM pgmq.#{qtable}
|
|
@@ -1229,6 +1233,7 @@ module Pgbus
|
|
|
1229
1233
|
q_summary.queue_visible_length,
|
|
1230
1234
|
q_summary.newest_msg_age_sec,
|
|
1231
1235
|
q_summary.oldest_msg_age_sec,
|
|
1236
|
+
q_summary.oldest_claimable_age_sec,
|
|
1232
1237
|
q_summary.max_read_ct,
|
|
1233
1238
|
q_summary.visible_unread_length,
|
|
1234
1239
|
all_metrics.total_messages
|
|
@@ -1241,7 +1246,9 @@ module Pgbus
|
|
|
1241
1246
|
name: queue_name,
|
|
1242
1247
|
queue_length: row["queue_length"].to_i,
|
|
1243
1248
|
queue_visible_length: row["queue_visible_length"].to_i,
|
|
1249
|
+
parked_length: row["queue_length"].to_i - row["queue_visible_length"].to_i,
|
|
1244
1250
|
oldest_msg_age_sec: row["oldest_msg_age_sec"]&.to_i,
|
|
1251
|
+
oldest_claimable_age_sec: row["oldest_claimable_age_sec"]&.to_i,
|
|
1245
1252
|
newest_msg_age_sec: row["newest_msg_age_sec"]&.to_i,
|
|
1246
1253
|
total_messages: row["total_messages"].to_i,
|
|
1247
1254
|
max_read_ct: row["max_read_ct"]&.to_i,
|
|
@@ -54,6 +54,15 @@ module Pgbus
|
|
|
54
54
|
end
|
|
55
55
|
end
|
|
56
56
|
|
|
57
|
+
gauge(lines, "pgbus_queue_oldest_claimable_age_seconds",
|
|
58
|
+
"Age of the oldest message eligible for pickup (visibility timeout elapsed)") do
|
|
59
|
+
queues.filter_map do |q|
|
|
60
|
+
next unless q[:oldest_claimable_age_sec]
|
|
61
|
+
|
|
62
|
+
[q[:oldest_claimable_age_sec], { queue: q[:name] }]
|
|
63
|
+
end
|
|
64
|
+
end
|
|
65
|
+
|
|
57
66
|
gauge(lines, "pgbus_queue_paused", "Whether the queue is paused (1) or active (0)") do
|
|
58
67
|
queues.map { |q| [q[:paused] ? 1 : 0, { queue: q[:name] }] }
|
|
59
68
|
end
|