pgbus 0.27.0 → 0.27.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +9 -0
- data/README.md +2 -1
- data/Rakefile +7 -1
- data/app/views/pgbus/dashboard/_queues_table.html.erb +9 -5
- data/app/views/pgbus/queues/_queues_list.html.erb +10 -6
- data/config/locales/da.yml +1 -0
- data/config/locales/de.yml +1 -0
- data/config/locales/en.yml +1 -0
- data/config/locales/es.yml +1 -0
- data/config/locales/fi.yml +1 -0
- data/config/locales/fr.yml +1 -0
- data/config/locales/it.yml +1 -0
- data/config/locales/ja.yml +1 -0
- data/config/locales/nb.yml +1 -0
- data/config/locales/nl.yml +1 -0
- data/config/locales/pt.yml +1 -0
- data/config/locales/sv.yml +1 -0
- data/lib/pgbus/configuration.rb +11 -1
- data/lib/pgbus/integrations/appsignal/probe.rb +20 -1
- data/lib/pgbus/mcp/health_analyzer.rb +32 -2
- data/lib/pgbus/mcp/tools/dlq_detail_tool.rb +11 -2
- data/lib/pgbus/mcp/tools/queues_tool.rb +3 -0
- data/lib/pgbus/version.rb +1 -1
- data/lib/pgbus/web/data_source/dead_letter.rb +35 -16
- data/lib/pgbus/web/data_source/job_list.rb +1 -1
- data/lib/pgbus/web/data_source/queue_summary.rb +1 -1
- data/lib/pgbus/web/data_source.rb +176 -56
- data/lib/pgbus/web/metrics_serializer.rb +34 -5
- data/lib/pgbus/web/queue_summary.rb +6 -1
- data/sig/pgbus/configuration.rbs +2 -0
- metadata +1 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 6b6f949cf25759a3498971f5c00a1ba4c3beee0bdcadac41c6bd978af96ba392
|
|
4
|
+
data.tar.gz: 0cd2f5abd43c6f5b380f2439b40553dd707ec0283dc42fbe696d6dd2f0a0172e
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 2d9fa8de5bcae929a3546fba35c7734e83ca9794aa582adc69cb9fee0aadbd397061324cea4e00fa5fcb77bc79e11e39909ff96fd4c27e83d8f94c7acca46a42
|
|
7
|
+
data.tar.gz: 7fba6d23df3a93fa218b2f23cb59b6a942a45ebdd8aadd57cf00f01d3f110adb7eb406086aed7922b0b54931bb937cfe366fef5230bb01fc517dd50d3bb64c0a
|
data/CHANGELOG.md
CHANGED
|
@@ -24,6 +24,15 @@
|
|
|
24
24
|
|
|
25
25
|
### Fixed
|
|
26
26
|
|
|
27
|
+
- **`Web::DataSource#queues_with_metrics` reads 1 000 queues in a tenth of the time, and a metrics query that fails or times out no longer reads as "no queues" (issue #677).** The read was one statement with nine scalar subqueries per queue (eight scans of each queue table plus one sequence read), so its cost grew with the queue count until `statement_timeout` cancelled it, and the rescue returned `[]`: the dashboard, the dead-letter views, `pgbus_health`, the Prometheus scrape and the AppSignal probe all read a timeout as an empty, healthy system. Inside a caller's transaction the cancelled statement also aborted it.
|
|
28
|
+
|
|
29
|
+
- **One scan per queue, in chunks.** Each queue's metrics are one aggregate over its table, and the queues are read `web_queue_metrics_chunk_size` (new, default 100) per statement, each statement in its own savepoint. Same machine, one message per queue: 100 queues 57.7 ms → 37.4 ms, 1 000 queues 3 711 ms → 373 ms, 5 000 queues 126 s → 1.9 s (`rake bench:queue_metrics`, new). A healthy queue's hash is unchanged.
|
|
30
|
+
- **A failed chunk is visible.** Its queues stay in the list with `metrics_unavailable: true` and `nil` counts; the other chunks still report, and an enclosing transaction stays usable. After 3 chunks that timed out (a cancelled statement, a lock wait, a deadlock, a connection that cannot be established) the rest are not read and come back unavailable at once, so a database where every chunk times out costs a few `statement_timeout`s, not one per chunk; a chunk that fails at once (a queue whose table is gone) never stops the others. Totals (`summary_stats` `total_depth` / `dlq_depth`, the `pgbus_dlq_depth` and AppSignal summary gauges) leave an unmeasured queue out rather than guess, so they read low while the verdict is DEGRADED. New `DataSource#queue_metrics_errors` returns `[]` or `[{ queues: [...], error: "Class: message" }]` (`queues: []` when `pgmq.meta` itself could not be read). `pgbus_health` / `/readyz` report `DEGRADED` with a reason naming the queues; the dashboard queue tables say "Metrics unavailable" in place of the counts; the AppSignal probe skips the queue's gauges and increments `pgbus_queue_metrics_unavailable`; the Prometheus scrape writes no count for it and adds the `pgbus_queue_metrics_unavailable` gauge; the MCP `pgbus_queues` tool carries the flag per row; the dead-letter total counts the rows of a DLQ whose metrics are unavailable (in a savepoint) rather than reading it as 0. An unreadable queue list increments the AppSignal counter untagged, writes one unlabelled Prometheus sample and leaves `pgbus_dlq_depth` out of the scrape; a failure after the queue list was read keeps every queue, flagged.
|
|
31
|
+
- **Names without metrics.** New `DataSource#queue_names` reads `pgmq.meta` only (memoized, cleared by `reset_cache!`). The job list, `dlq_messages`, `dlq_message_detail`, discard-all, priority siblings and the MCP `pgbus_dlq_detail` scan use it and run no metrics statement (the dead-letter page's counts, `dlq_total_count` and `dlq_counts_by_queue`, still read the metrics).
|
|
32
|
+
- **Throughput.** `summary_stats[:throughput_rate]` was always 0.0: `compute_throughput` called `Process.clock_gettime` inside `Pgbus::Web`, where `Process` is `Pgbus::Process`, and rescued the `NoMethodError`. It now reads the clock, never reports a negative rate (a dropped queue lowers the total), and keeps its last complete snapshot while any queue's metrics are unavailable.
|
|
33
|
+
|
|
34
|
+
Refs #677.
|
|
35
|
+
|
|
27
36
|
- **A pooled connection in `connection_guc_mode = :session` keeps its database.yml `variables` when it is reset (issue #671).** In `:session` mode the `variables` (`statement_timeout`, `lock_timeout` and the like) are applied with `SET` after connect, and `PG::Connection#reset` opens a new backend without them. pgmq-ruby's checkout health check resets an unhealthy pooled connection (`finished?` or `CONNECTION_BAD` since #332, and an intermediate state such as `CONNECTION_STARTED` since pgmq-ruby 0.9.0), so that connection went on serving the job or streams pool with the server's defaults. Each connection the `:session` connect lambda opens now re-applies its `SET`s when it is reset, whoever resets it. A connection whose `SET`s do not all complete, after a connect or a reset, is closed before the error reaches the caller, so it is never pooled without them; with pg 1.5.9 or later the next checkout resets it and applies the `SET`s again (an older pg cannot reset a closed connection, so it is replaced when the pool is reloaded before the last stale-connection retry; the call itself is retried only for a stale-connection error, so a `SET` that always fails, such as an invalid value, fails each time). The completion pipeline's own reset (#619) no longer applies the `SET`s itself (they ran twice), and a `SET` that fails there is now logged as `Resetting the connection or re-applying its session variables failed` and leaves the connection closed, where before the session carried on without its `variables`. `:options` mode, a String URL and the shared ActiveRecord connection are unaffected. Refs #671.
|
|
28
37
|
|
|
29
38
|
## [0.25.1] - 2026-11-15
|
data/README.md
CHANGED
|
@@ -1025,7 +1025,7 @@ When the `appsignal` gem is loaded in your app, Pgbus auto-installs a subscriber
|
|
|
1025
1025
|
|
|
1026
1026
|
- **Background-job transactions** for every ActiveJob run and every event-bus handler invocation. Action names follow the AppSignal convention: `MyJob#perform`, `MyHandler#handle`. Tags include `queue`, `job_class`/`handler`, `routing_key`, `attempts`, and the `active_job_id` / `provider_job_id`. `enqueued_at` becomes the AppSignal `queue_start` timestamp so "time on queue" shows up correctly in the timeline. The transaction is current for the whole execution (deserialize, `perform`/`handle`, archive), so `Appsignal.add_tags`, `Appsignal.set_error` and `Appsignal.instrument` called from job or handler code attach to it, and the error an execution raised is reported on it, also when it was raised before `perform` (an unknown job class, arguments that cannot be deserialized).
|
|
1027
1027
|
- **Custom counters and distributions** for sends, reads, broadcasts, outbox publishes, recurring scheduling, and worker recycles. All metric names are prefixed `pgbus_`.
|
|
1028
|
-
- **A minutely probe** that gauges queue depth (visible vs total), oldest message age per queue, DLQ depth, failed events count, dead-tuple totals, MVCC horizon age, active processes, and stream connection estimates.
|
|
1028
|
+
- **A minutely probe** that gauges queue depth (visible vs total), oldest message age per queue, DLQ depth, failed events count, dead-tuple totals, MVCC horizon age, active processes, and stream connection estimates. A queue whose metrics could not be read (its metrics query failed or hit `statement_timeout`) gets no gauges that minute; it increments the `pgbus_queue_metrics_unavailable` counter, tagged by queue, instead (untagged when the queue list itself could not be read).
|
|
1029
1029
|
|
|
1030
1030
|
There is nothing to wire up — load the appsignal gem and the integration installs itself in a Rails initializer. To opt out:
|
|
1031
1031
|
|
|
@@ -1205,6 +1205,7 @@ When `config.metrics_enabled = true` (default), the dashboard exposes Prometheus
|
|
|
1205
1205
|
| Metric | Description |
|
|
1206
1206
|
|--------|-------------|
|
|
1207
1207
|
| `pgbus_queue_oldest_claimable_age_seconds` | Age of the oldest message eligible for pickup (visibility timeout elapsed) — safe to alert on: scheduled/backoff-parked messages don't count until due; the series is omitted entirely when no claimable backlog exists (while the raw `pgbus_queue_oldest_message_age_seconds` gauge may still report a parked message's age) |
|
|
1208
|
+
| `pgbus_queue_metrics_unavailable` | 1 when the queue's metrics could not be read this scrape (its metrics query failed or hit `statement_timeout`), else 0; such a queue has no depth or age samples, rather than a false 0. One unlabelled `1` sample when the queue list itself could not be read |
|
|
1208
1209
|
| `pgbus_table_dead_tuples` | Dead tuple count per PGMQ table |
|
|
1209
1210
|
| `pgbus_table_live_tuples` | Live tuple count per PGMQ table |
|
|
1210
1211
|
| `pgbus_table_bloat_ratio` | Dead / (dead + live) per table |
|
data/Rakefile
CHANGED
|
@@ -26,7 +26,8 @@ namespace :bench do
|
|
|
26
26
|
db_benches = %w[connection_pool_bench integration_bench streams_bench streams_read_pool_bench
|
|
27
27
|
execution_modes_bench pool_swap_bench pool_autoscale_bench job_burst_bench
|
|
28
28
|
notify_wake_bench notify_chaos_bench streams_hub_bench fair_read_bench
|
|
29
|
-
worker_profile_bench workflow_insights_bench streams_scale_bench stat_flush_bench
|
|
29
|
+
worker_profile_bench workflow_insights_bench streams_scale_bench stat_flush_bench
|
|
30
|
+
queue_metrics_bench].freeze
|
|
30
31
|
# The unit suite is every *_bench.rb that doesn't need a database, derived
|
|
31
32
|
# from the directory so a new unit bench is picked up automatically (kept in
|
|
32
33
|
# sync with bench:one, which globs the same files).
|
|
@@ -78,6 +79,11 @@ namespace :bench do
|
|
|
78
79
|
ruby "benchmarks/workflow_insights_bench.rb"
|
|
79
80
|
end
|
|
80
81
|
|
|
82
|
+
desc "Time queues_with_metrics at 100 / 1 000 / 5 000 queues (requires PGBUS_DATABASE_URL)"
|
|
83
|
+
task :queue_metrics do
|
|
84
|
+
ruby "benchmarks/queue_metrics_bench.rb"
|
|
85
|
+
end
|
|
86
|
+
|
|
81
87
|
desc "Run fair share read benchmark (requires PGBUS_DATABASE_URL)"
|
|
82
88
|
task :fair_read do
|
|
83
89
|
ruby "benchmarks/fair_read_bench.rb"
|
|
@@ -26,11 +26,15 @@
|
|
|
26
26
|
<%= pgbus_queue_badge(q[:name]) %>
|
|
27
27
|
</td>
|
|
28
28
|
<td data-label="<%= t("pgbus.dashboard.queues_table.headers.status") %>" class="px-4 py-3 text-sm"><%= pgbus_queue_status(@queue_lines[q[:name]]) %></td>
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
|
|
33
|
-
|
|
29
|
+
<% if q[:metrics_unavailable] %>
|
|
30
|
+
<td colspan="5" data-label="<%= t("pgbus.dashboard.queues_table.headers.depth") %>" class="px-4 py-3 text-sm text-right text-yellow-800 dark:text-yellow-200"><%= t("pgbus.queues.show.summary.metrics_unavailable_html") %></td>
|
|
31
|
+
<% else %>
|
|
32
|
+
<td data-label="<%= t("pgbus.dashboard.queues_table.headers.depth") %>" class="px-4 py-3 text-sm text-right text-gray-700 dark:text-gray-200"><%= pgbus_number(q[:queue_length]) %></td>
|
|
33
|
+
<td data-label="<%= t("pgbus.dashboard.queues_table.headers.visible") %>" class="px-4 py-3 text-sm text-right text-gray-700 dark:text-gray-200"><%= pgbus_number(q[:queue_visible_length]) %></td>
|
|
34
|
+
<td data-label="<%= t("pgbus.dashboard.queues_table.headers.parked") %>" class="px-4 py-3 text-sm text-right text-gray-500 dark:text-gray-400"><%= pgbus_number(q[:parked_length]) %></td>
|
|
35
|
+
<td data-label="<%= t("pgbus.dashboard.queues_table.headers.oldest_claimable") %>" class="px-4 py-3 text-sm text-right text-gray-500 dark:text-gray-400"><%= pgbus_duration(q[:oldest_claimable_age_sec]) %></td>
|
|
36
|
+
<td data-label="<%= t("pgbus.dashboard.queues_table.headers.total") %>" class="px-4 py-3 text-sm text-right text-gray-500 dark:text-gray-400"><%= pgbus_number(q[:total_messages]) %></td>
|
|
37
|
+
<% end %>
|
|
34
38
|
</tr>
|
|
35
39
|
<% end %>
|
|
36
40
|
<% if @queues.empty? %>
|
|
@@ -23,12 +23,16 @@
|
|
|
23
23
|
<%= pgbus_paused_badge(q[:paused]) %>
|
|
24
24
|
<% end %>
|
|
25
25
|
</td>
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
|
|
26
|
+
<% if q[:metrics_unavailable] %>
|
|
27
|
+
<td colspan="6" data-label="<%= t("pgbus.queues.queues_list.headers.depth") %>" class="px-4 py-3 text-sm text-right text-yellow-800 dark:text-yellow-200"><%= t("pgbus.queues.show.summary.metrics_unavailable_html") %></td>
|
|
28
|
+
<% else %>
|
|
29
|
+
<td data-label="<%= t("pgbus.queues.queues_list.headers.depth") %>" class="px-4 py-3 text-sm text-right font-mono text-gray-700 dark:text-gray-200"><%= pgbus_number(q[:queue_length]) %></td>
|
|
30
|
+
<td data-label="<%= t("pgbus.queues.queues_list.headers.visible") %>" class="px-4 py-3 text-sm text-right font-mono text-gray-700 dark:text-gray-200"><%= pgbus_number(q[:queue_visible_length]) %></td>
|
|
31
|
+
<td data-label="<%= t("pgbus.queues.queues_list.headers.parked") %>" class="px-4 py-3 text-sm text-right font-mono text-gray-500 dark:text-gray-400"><%= pgbus_number(q[:parked_length]) %></td>
|
|
32
|
+
<td data-label="<%= t("pgbus.queues.queues_list.headers.oldest_claimable") %>" class="px-4 py-3 text-sm text-right text-gray-500 dark:text-gray-400"><%= pgbus_duration(q[:oldest_claimable_age_sec]) %></td>
|
|
33
|
+
<td data-label="<%= t("pgbus.queues.queues_list.headers.newest") %>" class="px-4 py-3 text-sm text-right text-gray-500 dark:text-gray-400"><%= pgbus_duration(q[:newest_msg_age_sec]) %></td>
|
|
34
|
+
<td data-label="<%= t("pgbus.queues.queues_list.headers.total_ever") %>" class="px-4 py-3 text-sm text-right text-gray-500 dark:text-gray-400"><%= pgbus_number(q[:total_messages]) %></td>
|
|
35
|
+
<% end %>
|
|
32
36
|
<td data-label="<%= t("pgbus.queues.queues_list.headers.actions") %>" class="px-4 py-3 text-sm text-right space-x-2">
|
|
33
37
|
<% if q[:paused] %>
|
|
34
38
|
<%= pgbus_button_to t("pgbus.queues.queues_list.resume"), pgbus.resume_queue_path(name: q[:name]),
|
data/config/locales/da.yml
CHANGED
|
@@ -973,6 +973,7 @@ da:
|
|
|
973
973
|
other: Tømmes af en wildcard-worker-kapsel (%{count} sunde workers)
|
|
974
974
|
empty_html: Intet venter
|
|
975
975
|
handler_html: Event handler-kø — tømmes af consumers (%{count} sunde)
|
|
976
|
+
metrics_unavailable_html: 'Metrikker utilgængelige: metrikforespørgslen fejlede eller fik timeout'
|
|
976
977
|
no_consumers_html:
|
|
977
978
|
one: Event handler-kø uden sund consumer — %{count} event venter
|
|
978
979
|
other: Event handler-kø uden sund consumer — %{count} events venter
|
data/config/locales/de.yml
CHANGED
|
@@ -973,6 +973,7 @@ de:
|
|
|
973
973
|
other: Abgearbeitet von einer Wildcard-Worker-Kapsel (%{count} gesunde Worker)
|
|
974
974
|
empty_html: Nichts wartet
|
|
975
975
|
handler_html: Event-Handler-Queue — von Consumern abgearbeitet (%{count} gesund)
|
|
976
|
+
metrics_unavailable_html: 'Metriken nicht verfügbar: Die Metrikabfrage ist fehlgeschlagen oder hat das Zeitlimit überschritten'
|
|
976
977
|
no_consumers_html:
|
|
977
978
|
one: Event-Handler-Queue ohne gesunden Consumer — %{count} Event wartet
|
|
978
979
|
other: Event-Handler-Queue ohne gesunden Consumer — %{count} Events warten
|
data/config/locales/en.yml
CHANGED
|
@@ -973,6 +973,7 @@ en:
|
|
|
973
973
|
other: Drained by a wildcard worker capsule (%{count} healthy workers)
|
|
974
974
|
empty_html: Nothing waiting
|
|
975
975
|
handler_html: Event handler queue — drained by consumers (%{count} healthy)
|
|
976
|
+
metrics_unavailable_html: 'Metrics unavailable: the metrics query failed or timed out'
|
|
976
977
|
no_consumers_html:
|
|
977
978
|
one: Event handler queue with no healthy consumer — %{count} event waits
|
|
978
979
|
other: Event handler queue with no healthy consumer — %{count} events wait
|
data/config/locales/es.yml
CHANGED
|
@@ -973,6 +973,7 @@ es:
|
|
|
973
973
|
other: La vacía una cápsula de workers comodín (%{count} workers sanos)
|
|
974
974
|
empty_html: Nada en espera
|
|
975
975
|
handler_html: Cola de manejador de eventos — la vacían los consumidores (%{count} sanos)
|
|
976
|
+
metrics_unavailable_html: 'Métricas no disponibles: la consulta de métricas falló o agotó el tiempo de espera'
|
|
976
977
|
no_consumers_html:
|
|
977
978
|
one: Cola de manejador de eventos sin consumidor sano — %{count} evento espera
|
|
978
979
|
other: Cola de manejador de eventos sin consumidor sano — %{count} eventos esperan
|
data/config/locales/fi.yml
CHANGED
|
@@ -973,6 +973,7 @@ fi:
|
|
|
973
973
|
other: Tyhjentää jokerimerkki-worker-kapseli (%{count} tervettä workeria)
|
|
974
974
|
empty_html: Mitään ei odota
|
|
975
975
|
handler_html: Tapahtumankäsittelijän jono — consumerit tyhjentävät (%{count} tervettä)
|
|
976
|
+
metrics_unavailable_html: 'Mittarit eivät ole saatavilla: mittarikysely epäonnistui tai aikakatkaistiin'
|
|
976
977
|
no_consumers_html:
|
|
977
978
|
one: Tapahtumankäsittelijän jono ilman tervettä consumeria — %{count} tapahtuma odottaa
|
|
978
979
|
other: Tapahtumankäsittelijän jono ilman tervettä consumeria — %{count} tapahtumaa odottaa
|
data/config/locales/fr.yml
CHANGED
|
@@ -973,6 +973,7 @@ fr:
|
|
|
973
973
|
other: Vidée par une capsule de workers joker (%{count} workers sains)
|
|
974
974
|
empty_html: Rien en attente
|
|
975
975
|
handler_html: File de gestionnaire d'événements — vidée par les consommateurs (%{count} sains)
|
|
976
|
+
metrics_unavailable_html: 'Métriques indisponibles : la requête des métriques a échoué ou a expiré'
|
|
976
977
|
no_consumers_html:
|
|
977
978
|
one: File de gestionnaire d'événements sans consommateur sain — %{count} événement attend
|
|
978
979
|
other: File de gestionnaire d'événements sans consommateur sain — %{count} événements attendent
|
data/config/locales/it.yml
CHANGED
|
@@ -973,6 +973,7 @@ it:
|
|
|
973
973
|
other: Svuotata da una capsula di worker jolly (%{count} worker sani)
|
|
974
974
|
empty_html: Niente in attesa
|
|
975
975
|
handler_html: Coda del gestore di eventi — svuotata dai consumer (%{count} sani)
|
|
976
|
+
metrics_unavailable_html: 'Metriche non disponibili: la query delle metriche non è riuscita o è scaduta'
|
|
976
977
|
no_consumers_html:
|
|
977
978
|
one: Coda del gestore di eventi senza consumer sano — %{count} evento in attesa
|
|
978
979
|
other: Coda del gestore di eventi senza consumer sano — %{count} eventi in attesa
|
data/config/locales/ja.yml
CHANGED
|
@@ -973,6 +973,7 @@ ja:
|
|
|
973
973
|
other: ワイルドカードのワーカーカプセルが処理(正常なワーカー %{count})
|
|
974
974
|
empty_html: 待機中のものはありません
|
|
975
975
|
handler_html: イベントハンドラーキュー — コンシューマーが処理(正常 %{count})
|
|
976
|
+
metrics_unavailable_html: 'メトリクスを取得できません: メトリクスのクエリが失敗したかタイムアウトしました'
|
|
976
977
|
no_consumers_html:
|
|
977
978
|
one: 正常なコンシューマーがいないイベントハンドラーキュー — %{count} 件のイベントが待機中
|
|
978
979
|
other: 正常なコンシューマーがいないイベントハンドラーキュー — %{count} 件のイベントが待機中
|
data/config/locales/nb.yml
CHANGED
|
@@ -973,6 +973,7 @@ nb:
|
|
|
973
973
|
other: Tømmes av en jokertegn-worker-kapsel (%{count} friske workers)
|
|
974
974
|
empty_html: Ingenting venter
|
|
975
975
|
handler_html: Hendelsesbehandler-kø — tømmes av consumers (%{count} friske)
|
|
976
|
+
metrics_unavailable_html: 'Metrikker utilgjengelige: metrikkspørringen feilet eller fikk tidsavbrudd'
|
|
976
977
|
no_consumers_html:
|
|
977
978
|
one: Hendelsesbehandler-kø uten frisk consumer — %{count} hendelse venter
|
|
978
979
|
other: Hendelsesbehandler-kø uten frisk consumer — %{count} hendelser venter
|
data/config/locales/nl.yml
CHANGED
|
@@ -973,6 +973,7 @@ nl:
|
|
|
973
973
|
other: Leeggemaakt door een wildcard-workercapsule (%{count} gezonde workers)
|
|
974
974
|
empty_html: Niets in de wacht
|
|
975
975
|
handler_html: Event-handlerwachtrij — leeggemaakt door consumers (%{count} gezond)
|
|
976
|
+
metrics_unavailable_html: 'Metrieken niet beschikbaar: de metriekenquery is mislukt of verlopen'
|
|
976
977
|
no_consumers_html:
|
|
977
978
|
one: Event-handlerwachtrij zonder gezonde consumer — %{count} event wacht
|
|
978
979
|
other: Event-handlerwachtrij zonder gezonde consumer — %{count} events wachten
|
data/config/locales/pt.yml
CHANGED
|
@@ -973,6 +973,7 @@ pt:
|
|
|
973
973
|
other: Esvaziada por uma cápsula de workers curinga (%{count} workers saudáveis)
|
|
974
974
|
empty_html: Nada à espera
|
|
975
975
|
handler_html: Fila de handler de eventos — esvaziada pelos consumidores (%{count} saudáveis)
|
|
976
|
+
metrics_unavailable_html: 'Métricas indisponíveis: a consulta de métricas falhou ou expirou'
|
|
976
977
|
no_consumers_html:
|
|
977
978
|
one: Fila de handler de eventos sem consumidor saudável — %{count} evento espera
|
|
978
979
|
other: Fila de handler de eventos sem consumidor saudável — %{count} eventos esperam
|
data/config/locales/sv.yml
CHANGED
|
@@ -973,6 +973,7 @@ sv:
|
|
|
973
973
|
other: Töms av en jokertecken-worker-kapsel (%{count} friska workers)
|
|
974
974
|
empty_html: Inget väntar
|
|
975
975
|
handler_html: Händelsehanterarkö — töms av consumers (%{count} friska)
|
|
976
|
+
metrics_unavailable_html: 'Mätvärden otillgängliga: mätvärdesfrågan misslyckades eller överskred tidsgränsen'
|
|
976
977
|
no_consumers_html:
|
|
977
978
|
one: Händelsehanterarkö utan frisk consumer — %{count} händelse väntar
|
|
978
979
|
other: Händelsehanterarkö utan frisk consumer — %{count} händelser väntar
|
data/lib/pgbus/configuration.rb
CHANGED
|
@@ -301,7 +301,12 @@ module Pgbus
|
|
|
301
301
|
attr_accessor :web_auth, :web_refresh_interval, :web_per_page, :web_live_updates, :web_data_source,
|
|
302
302
|
:insights_default_minutes, :base_controller_class, :return_to_app_url, :web_actor,
|
|
303
303
|
:metrics_enabled,
|
|
304
|
-
:web_filter_parameters, :web_filter_sensitive
|
|
304
|
+
:web_filter_parameters, :web_filter_sensitive,
|
|
305
|
+
# Queues per statement when Web::DataSource#queues_with_metrics
|
|
306
|
+
# reads every queue's metrics (issue #677): each chunk is its
|
|
307
|
+
# own statement in its own savepoint, so a cancelled chunk
|
|
308
|
+
# marks only its queues unavailable.
|
|
309
|
+
:web_queue_metrics_chunk_size
|
|
305
310
|
|
|
306
311
|
# The MCP server's workflow write tools (issue #550): pgbus_workflow_resume,
|
|
307
312
|
# _cancel, _skip_step, _resume_at, _run_now and _signal. Off by default, and
|
|
@@ -504,6 +509,7 @@ module Pgbus
|
|
|
504
509
|
@metrics_enabled = true
|
|
505
510
|
@web_filter_parameters = nil # nil = auto-detect from Rails, then fall back to defaults
|
|
506
511
|
@web_filter_sensitive = true
|
|
512
|
+
@web_queue_metrics_chunk_size = 100
|
|
507
513
|
@deprecation_warned = Set.new
|
|
508
514
|
|
|
509
515
|
# HTTP health endpoints — nil port disables the standalone server.
|
|
@@ -1123,6 +1129,10 @@ module Pgbus
|
|
|
1123
1129
|
raise Pgbus::ConfigurationError, "insights_default_minutes must be a positive integer"
|
|
1124
1130
|
end
|
|
1125
1131
|
|
|
1132
|
+
unless web_queue_metrics_chunk_size.is_a?(Integer) && web_queue_metrics_chunk_size.positive?
|
|
1133
|
+
raise Pgbus::ConfigurationError, "web_queue_metrics_chunk_size must be a positive integer"
|
|
1134
|
+
end
|
|
1135
|
+
|
|
1126
1136
|
raise Pgbus::ConfigurationError, "require_primary must be true or false" unless [true, false].include?(require_primary)
|
|
1127
1137
|
raise Pgbus::ConfigurationError, "mcp_write_tools must be true or false" unless [true, false].include?(mcp_write_tools)
|
|
1128
1138
|
|
|
@@ -92,8 +92,17 @@ module Pgbus
|
|
|
92
92
|
end
|
|
93
93
|
|
|
94
94
|
def track_queues
|
|
95
|
-
data_source.queues_with_metrics
|
|
95
|
+
queues = data_source.queues_with_metrics
|
|
96
|
+
# The queue list itself could not be read: count it once, untagged,
|
|
97
|
+
# rather than emit nothing (an empty list reads as a quiet fleet).
|
|
98
|
+
counter("queue_metrics_unavailable", 1) if queues.empty? && queue_list_unavailable?
|
|
99
|
+
|
|
100
|
+
queues.each do |q|
|
|
96
101
|
tags = { queue: q[:name] }
|
|
102
|
+
# Issue #677: an unmeasured queue reports no gauge (a missing
|
|
103
|
+
# point, not a false 0) and counts as unavailable instead.
|
|
104
|
+
next counter("queue_metrics_unavailable", 1, tags) if q[:metrics_unavailable]
|
|
105
|
+
|
|
97
106
|
gauge "queue_depth", q[:queue_length], tags
|
|
98
107
|
gauge "queue_visible_depth", q[:queue_visible_length], tags
|
|
99
108
|
gauge "queue_paused", q[:paused] ? 1 : 0, tags
|
|
@@ -175,6 +184,16 @@ module Pgbus
|
|
|
175
184
|
::Appsignal.set_gauge("#{METRIC_PREFIX}#{key}", value, tags)
|
|
176
185
|
end
|
|
177
186
|
|
|
187
|
+
def queue_list_unavailable?
|
|
188
|
+
return false unless data_source.respond_to?(:queue_metrics_errors)
|
|
189
|
+
|
|
190
|
+
data_source.queue_metrics_errors.any? { |e| Array(e[:queues]).empty? }
|
|
191
|
+
end
|
|
192
|
+
|
|
193
|
+
def counter(key, value, tags = {})
|
|
194
|
+
::Appsignal.increment_counter("#{METRIC_PREFIX}#{key}", value, tags)
|
|
195
|
+
end
|
|
196
|
+
|
|
178
197
|
def log_failure(label, error)
|
|
179
198
|
Pgbus.logger.debug do
|
|
180
199
|
"[Pgbus::AppSignal::Probe] #{label} failed: #{error.class}: #{error.message}"
|
|
@@ -13,8 +13,9 @@ module Pgbus
|
|
|
13
13
|
# but its claim loop has stopped advancing (status :stalled),
|
|
14
14
|
# OR backlog with live-but-idle workers and zero claim progress.
|
|
15
15
|
# DEGRADED — something is wrong but not the wedge: stale processes, a
|
|
16
|
-
# paused queue holding a backlog, growing DLQ,
|
|
17
|
-
# pinned by a long-running transaction
|
|
16
|
+
# paused queue holding a backlog, growing DLQ, MVCC horizon
|
|
17
|
+
# pinned by a long-running transaction, or queue metrics that
|
|
18
|
+
# could not be read (issue #677).
|
|
18
19
|
# OK — draining normally / nothing actionable.
|
|
19
20
|
class HealthAnalyzer
|
|
20
21
|
# A worker is considered to have idle capacity unless its metadata
|
|
@@ -161,6 +162,9 @@ module Pgbus
|
|
|
161
162
|
|
|
162
163
|
reasons << "queue health stats unavailable: #{health_error.class}: #{health_error.message}" if health_error
|
|
163
164
|
|
|
165
|
+
metrics_reason = metrics_unavailable_reason(queues)
|
|
166
|
+
reasons << metrics_reason if metrics_reason
|
|
167
|
+
|
|
164
168
|
stale = processes.select { |p| p[:status].to_s == "stale" }
|
|
165
169
|
reasons << "#{stale.size} process(es) stale (no recent heartbeat)" if stale.any?
|
|
166
170
|
|
|
@@ -189,6 +193,32 @@ module Pgbus
|
|
|
189
193
|
reasons
|
|
190
194
|
end
|
|
191
195
|
|
|
196
|
+
# Issue #677: a queue whose metrics could not be read (its chunk failed or
|
|
197
|
+
# hit statement_timeout) has nil counts, so it can never look backed up
|
|
198
|
+
# or wedged. Without this reason a fleet too large to measure would read
|
|
199
|
+
# OK. queue_metrics_errors also covers a pgmq.meta read that failed, when
|
|
200
|
+
# there are no rows to flag; a custom data source without it still
|
|
201
|
+
# reports its flagged rows.
|
|
202
|
+
def metrics_unavailable_reason(queues)
|
|
203
|
+
unavailable = queues.select { |q| q[:metrics_unavailable] }
|
|
204
|
+
errors = queue_metrics_errors
|
|
205
|
+
return if unavailable.empty? && errors.empty?
|
|
206
|
+
|
|
207
|
+
# First line only: a database error carries a multi-line SQL excerpt.
|
|
208
|
+
cause = errors.first && " (#{errors.first[:error].to_s.lines.first.to_s.strip})"
|
|
209
|
+
if unavailable.any?
|
|
210
|
+
"queue metrics unavailable for #{unavailable.size} queue(s)#{cause}: #{backlog_names(unavailable)}"
|
|
211
|
+
else
|
|
212
|
+
"queue metrics unavailable#{cause}: the queue list could not be read"
|
|
213
|
+
end
|
|
214
|
+
end
|
|
215
|
+
|
|
216
|
+
def queue_metrics_errors
|
|
217
|
+
return [] unless @data_source.respond_to?(:queue_metrics_errors)
|
|
218
|
+
|
|
219
|
+
Array(@data_source.queue_metrics_errors)
|
|
220
|
+
end
|
|
221
|
+
|
|
192
222
|
# Issue #366: fingerprint-matched streams excluded from the wedge signal
|
|
193
223
|
# still need the registry for orphan sweep bookkeeping and durable
|
|
194
224
|
# wildcard safety. Surface a DEGRADED hint so operators run the backfill.
|
|
@@ -57,10 +57,13 @@ module Pgbus
|
|
|
57
57
|
# refuse to guess if more than one matches. Returns an error response
|
|
58
58
|
# on ambiguity, nil otherwise (so the caller proceeds with the normal
|
|
59
59
|
# first-match lookup).
|
|
60
|
+
# Names only (issue #677): finding a message runs no metrics statement.
|
|
61
|
+
# A custom web_data_source without queue_names falls back to the names
|
|
62
|
+
# of its queues_with_metrics.
|
|
60
63
|
def self.ambiguity_check(data_source, msg_id)
|
|
61
64
|
dlq_suffix = Pgbus::DEAD_LETTER_SUFFIX
|
|
62
|
-
dlqs = data_source.
|
|
63
|
-
matches = dlqs.
|
|
65
|
+
dlqs = queue_names_of(data_source).select { |name| name.to_s.end_with?(dlq_suffix) }
|
|
66
|
+
matches = dlqs.select { |name| data_source.job_detail(name, msg_id) }
|
|
64
67
|
return nil if matches.size <= 1
|
|
65
68
|
|
|
66
69
|
error_response(
|
|
@@ -68,6 +71,12 @@ module Pgbus
|
|
|
68
71
|
"(#{matches.join(", ")}). Pass `queue:` to disambiguate."
|
|
69
72
|
)
|
|
70
73
|
end
|
|
74
|
+
|
|
75
|
+
def self.queue_names_of(data_source)
|
|
76
|
+
return data_source.queue_names if data_source.respond_to?(:queue_names)
|
|
77
|
+
|
|
78
|
+
data_source.queues_with_metrics.map { |q| q[:name] }
|
|
79
|
+
end
|
|
71
80
|
end
|
|
72
81
|
end
|
|
73
82
|
end
|
|
@@ -20,6 +20,9 @@ module Pgbus
|
|
|
20
20
|
with a future visibility timeout. A queue whose oldest_msg_age_sec
|
|
21
21
|
keeps growing while oldest_claimable_age_sec stays nil has no
|
|
22
22
|
starving backlog — nothing is waiting for a worker.
|
|
23
|
+
A queue with metrics_unavailable: true could not be measured (its
|
|
24
|
+
metrics query failed or timed out): its counts and ages are null,
|
|
25
|
+
meaning unknown, not zero; pgbus_health reports it as DEGRADED.
|
|
23
26
|
DESC
|
|
24
27
|
|
|
25
28
|
input_schema(properties: {}, required: [])
|
data/lib/pgbus/version.rb
CHANGED
|
@@ -9,7 +9,7 @@ module Pgbus
|
|
|
9
9
|
# The header path the error-class filter reads (DeadLetterHeader::KEY).
|
|
10
10
|
ERROR_CLASS_SQL = "headers #>> '{#{Pgbus::DeadLetterHeader::KEY},error_class}'".freeze
|
|
11
11
|
|
|
12
|
-
# DLQ queue names from
|
|
12
|
+
# DLQ queue names from queue_names are already fully qualified
|
|
13
13
|
# (e.g., "pgbus_default_dlq"), so we use them directly without re-prefixing.
|
|
14
14
|
# dlq: one DLQ by its full name (unknown names list nothing);
|
|
15
15
|
# error_class: only messages whose dead-letter block names that class.
|
|
@@ -25,27 +25,37 @@ module Pgbus
|
|
|
25
25
|
[]
|
|
26
26
|
end
|
|
27
27
|
|
|
28
|
+
# From the cached metrics, unless a filter needs the rows or a DLQ's
|
|
29
|
+
# metrics are unavailable (issue #677): then a COUNT over the tables,
|
|
30
|
+
# so the pager does not read a DLQ that lists rows as empty (unless
|
|
31
|
+
# that COUNT fails too: then 0, logged as an error).
|
|
28
32
|
def dlq_total_count(dlq: nil, error_class: nil)
|
|
29
33
|
names = dlq_queue_names(dlq)
|
|
30
34
|
return 0 if names.empty?
|
|
31
|
-
return dlq_counts_by_queue.values_at(*names).sum unless error_class
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
|
|
35
|
+
return dlq_counts_by_queue.values_at(*names).sum unless error_class || dlq_unmeasured?(names)
|
|
36
|
+
|
|
37
|
+
# In a savepoint: the DLQ whose metrics timed out may time out here
|
|
38
|
+
# too, and that must not abort a caller's transaction.
|
|
39
|
+
where = "#{ERROR_CLASS_SQL} = $1" if error_class
|
|
40
|
+
connection.transaction(requires_new: true) do
|
|
41
|
+
connection.select_value(
|
|
42
|
+
"SELECT COUNT(*) FROM (#{queue_message_fragments(names, where)}) AS combined",
|
|
43
|
+
"Pgbus DLQ Count",
|
|
44
|
+
error_class ? [error_class] : []
|
|
45
|
+
).to_i
|
|
46
|
+
end
|
|
38
47
|
rescue StandardError => e
|
|
39
|
-
Pgbus.logger.
|
|
48
|
+
Pgbus.logger.error { "[Pgbus::Web] Error fetching DLQ count: #{e.class}: #{e.message}" }
|
|
40
49
|
0
|
|
41
50
|
end
|
|
42
51
|
|
|
43
|
-
# { "pgbus_default_dlq" => 12, ... } for the filter chips
|
|
44
|
-
# cached metrics (no query of its
|
|
52
|
+
# { "pgbus_default_dlq" => 12, ... } for the filter chips: every DLQ
|
|
53
|
+
# from queue_names, its count from the cached metrics (no query of its
|
|
54
|
+
# own). A DLQ whose metrics are unavailable (issue #677) stays listed
|
|
55
|
+
# and counts 0.
|
|
45
56
|
def dlq_counts_by_queue
|
|
46
|
-
queues_with_metrics
|
|
47
|
-
|
|
48
|
-
.to_h { |q| [q[:name], q[:queue_length].to_i] }
|
|
57
|
+
lengths = queues_with_metrics.to_h { |q| [q[:name], q[:queue_length].to_i] }
|
|
58
|
+
dlq_names.to_h { |name| [name, lengths.fetch(name, 0)] }
|
|
49
59
|
rescue StandardError => e
|
|
50
60
|
Pgbus.logger.debug { "[Pgbus::Web] Error fetching DLQ counts: #{e.message}" }
|
|
51
61
|
{}
|
|
@@ -140,11 +150,20 @@ module Pgbus
|
|
|
140
150
|
|
|
141
151
|
private
|
|
142
152
|
|
|
143
|
-
# Only names that are real DLQs ever reach sanitize_name and SQL.
|
|
153
|
+
# Only names that are real DLQs ever reach sanitize_name and SQL. Names
|
|
154
|
+
# only: listing or opening a DLQ message runs no metrics statement.
|
|
144
155
|
def dlq_queue_names(dlq)
|
|
145
|
-
names =
|
|
156
|
+
names = dlq_names
|
|
146
157
|
dlq.nil? ? names : names & [dlq]
|
|
147
158
|
end
|
|
159
|
+
|
|
160
|
+
def dlq_names
|
|
161
|
+
queue_names.select { |name| name.end_with?(Pgbus::DEAD_LETTER_SUFFIX) }
|
|
162
|
+
end
|
|
163
|
+
|
|
164
|
+
def dlq_unmeasured?(names)
|
|
165
|
+
queues_with_metrics.any? { |q| q[:metrics_unavailable] && names.include?(q[:name]) }
|
|
166
|
+
end
|
|
148
167
|
end
|
|
149
168
|
end
|
|
150
169
|
end
|
|
@@ -136,7 +136,7 @@ module Pgbus
|
|
|
136
136
|
# exist. Excluded queues stay known, so their failed rows never show up
|
|
137
137
|
# as orphans of an unknown queue.
|
|
138
138
|
def job_list_scope(queue_name, queues: nil, exclude: nil)
|
|
139
|
-
names =
|
|
139
|
+
names = queue_names.reject { |n| n.end_with?(Pgbus::DEAD_LETTER_SUFFIX) }
|
|
140
140
|
names &= queues if queues
|
|
141
141
|
return nil if queues && names.empty?
|
|
142
142
|
|
|
@@ -112,7 +112,7 @@ module Pgbus
|
|
|
112
112
|
end
|
|
113
113
|
|
|
114
114
|
def priority_siblings(name, logical)
|
|
115
|
-
|
|
115
|
+
queue_names.select do |n|
|
|
116
116
|
n != name && n.match?(Pgbus::Web::QueueSummary::PRIORITY_LEVEL) && logical_queue_name(n) == logical
|
|
117
117
|
end
|
|
118
118
|
end
|
|
@@ -28,6 +28,26 @@ module Pgbus
|
|
|
28
28
|
# sweep picks up whatever is left on its next pass.
|
|
29
29
|
PROMOTE_CAP = 100
|
|
30
30
|
|
|
31
|
+
# The metrics of a queue whose chunk could not be read (issue #677):
|
|
32
|
+
# unknown, so nil rather than zero, next to metrics_unavailable: true.
|
|
33
|
+
UNAVAILABLE_METRICS = %i[queue_length queue_visible_length parked_length oldest_msg_age_sec
|
|
34
|
+
oldest_claimable_age_sec newest_msg_age_sec total_messages max_read_ct
|
|
35
|
+
visible_unread_length oldest_unread_claimable_age_sec].to_h { |k| [k, nil] }.freeze
|
|
36
|
+
|
|
37
|
+
# Timed-out metrics chunks after which queues_with_metrics stops reading
|
|
38
|
+
# (issue #677): the rest of the queues come back unavailable, so a
|
|
39
|
+
# database where every chunk times out costs a few statement_timeouts,
|
|
40
|
+
# not one per chunk.
|
|
41
|
+
MAX_FAILED_METRICS_CHUNKS = 3
|
|
42
|
+
|
|
43
|
+
# The chunk failures that cost time (a cancelled statement, a lock wait,
|
|
44
|
+
# a deadlock, which PostgreSQL detects only after deadlock_timeout, a
|
|
45
|
+
# connection that cannot be established) and so count toward MAX_FAILED_METRICS_CHUNKS. A
|
|
46
|
+
# failure that returns at once (a queue whose table is gone) does not:
|
|
47
|
+
# it must never stop the healthy queues after it from being read.
|
|
48
|
+
SLOW_METRICS_FAILURES = [ActiveRecord::QueryCanceled, ActiveRecord::LockWaitTimeout, ActiveRecord::Deadlocked,
|
|
49
|
+
ActiveRecord::ConnectionNotEstablished].freeze
|
|
50
|
+
|
|
31
51
|
def initialize(client: Pgbus.client)
|
|
32
52
|
@client = client
|
|
33
53
|
@last_throughput_snapshot = nil
|
|
@@ -37,10 +57,12 @@ module Pgbus
|
|
|
37
57
|
# Dashboard summary
|
|
38
58
|
def summary_stats
|
|
39
59
|
queues = queues_with_metrics
|
|
40
|
-
|
|
41
|
-
|
|
60
|
+
# A queue with metrics_unavailable (issue #677) has nil counts: it adds
|
|
61
|
+
# nothing it cannot vouch for.
|
|
62
|
+
total_depth = queues.sum { |q| q[:queue_length].to_i }
|
|
63
|
+
total_visible = queues.sum { |q| q[:queue_visible_length].to_i }
|
|
42
64
|
dlq_suffix = Pgbus::DEAD_LETTER_SUFFIX
|
|
43
|
-
dlq_depth = queues.select { |q| q[:name].end_with?(dlq_suffix) }.sum { |q| q[:queue_length] }
|
|
65
|
+
dlq_depth = queues.select { |q| q[:name].end_with?(dlq_suffix) }.sum { |q| q[:queue_length].to_i }
|
|
44
66
|
|
|
45
67
|
throughput = compute_throughput(queues)
|
|
46
68
|
|
|
@@ -67,7 +89,9 @@ module Pgbus
|
|
|
67
89
|
# responses would freeze at the first read. Both call this per iteration
|
|
68
90
|
# / per tool call.
|
|
69
91
|
def reset_cache!
|
|
92
|
+
@queue_names = nil
|
|
70
93
|
@queues_with_metrics = nil
|
|
94
|
+
@queue_metrics_errors = nil
|
|
71
95
|
@concurrency_summary = nil
|
|
72
96
|
@queue_health_stats = nil
|
|
73
97
|
@process_rows = nil
|
|
@@ -75,6 +99,19 @@ module Pgbus
|
|
|
75
99
|
self
|
|
76
100
|
end
|
|
77
101
|
|
|
102
|
+
# Every queue's physical name, from pgmq.meta alone: no queue table is
|
|
103
|
+
# read (issue #677). Callers that need only names (the job list scope,
|
|
104
|
+
# the DLQ selection, priority siblings) use this instead of paying for
|
|
105
|
+
# queues_with_metrics. Memoized like it, cleared by reset_cache!, and
|
|
106
|
+
# shared with it, so a page that wants both reads pgmq.meta once. A
|
|
107
|
+
# failed read is logged and answers [] without memoizing.
|
|
108
|
+
def queue_names
|
|
109
|
+
@queue_names ||= read_queue_names
|
|
110
|
+
rescue StandardError => e
|
|
111
|
+
Pgbus.logger.error { "[Pgbus::Web] Error reading queue names: #{e.class}: #{e.message}" }
|
|
112
|
+
[]
|
|
113
|
+
end
|
|
114
|
+
|
|
78
115
|
# Queues — query via ActiveRecord for reliability in web processes
|
|
79
116
|
# (avoids PGMQ client connection issues when the web server uses a
|
|
80
117
|
# different connection lifecycle than the worker processes).
|
|
@@ -84,24 +121,57 @@ module Pgbus
|
|
|
84
121
|
# message rows and the total count); without the memo each call re-runs
|
|
85
122
|
# the meta query + batched metrics. Mutations redirect to a fresh request
|
|
86
123
|
# with a new instance, so a per-request memo never serves stale data.
|
|
124
|
+
#
|
|
125
|
+
# A queue whose metrics could not be read (its chunk failed or hit
|
|
126
|
+
# statement_timeout, issue #677) stays in the list with
|
|
127
|
+
# metrics_unavailable: true and nil counts; queue_metrics_errors says why.
|
|
87
128
|
def queues_with_metrics
|
|
88
129
|
@queues_with_metrics ||= fetch_queues_with_metrics
|
|
89
130
|
end
|
|
90
131
|
|
|
132
|
+
# Why the last queues_with_metrics read is incomplete: [] when every
|
|
133
|
+
# queue reported, else one { queues: [names], error: "Class: message" }
|
|
134
|
+
# per failed statement (queues: [] when pgmq.meta itself could not be
|
|
135
|
+
# read). Runs that read when none has run yet. An empty
|
|
136
|
+
# queues_with_metrics with no errors is an empty system; with errors it
|
|
137
|
+
# is an unknown one.
|
|
138
|
+
def queue_metrics_errors
|
|
139
|
+
queues_with_metrics
|
|
140
|
+
@queue_metrics_errors || []
|
|
141
|
+
end
|
|
142
|
+
|
|
143
|
+
# queues: [] in queue_metrics_errors means pgmq.meta itself could not be
|
|
144
|
+
# read; a failure after it answered keeps the names, flagged.
|
|
91
145
|
def fetch_queues_with_metrics
|
|
92
|
-
|
|
93
|
-
|
|
146
|
+
@queue_metrics_errors = []
|
|
147
|
+
names = @queue_names ||= read_queue_names
|
|
148
|
+
return [] if names.empty?
|
|
94
149
|
|
|
95
150
|
paused_queues = paused_queue_names.to_set
|
|
96
|
-
batched_queue_metrics(
|
|
151
|
+
batched_queue_metrics(names).map do |q|
|
|
97
152
|
q.merge(paused: paused_queues.include?(logical_queue_name(q[:name])))
|
|
98
153
|
end
|
|
99
154
|
rescue StandardError => e
|
|
100
155
|
Pgbus.logger.error { "[Pgbus::Web] Error fetching queue metrics: #{e.class}: #{e.message}" }
|
|
156
|
+
return unavailable_queue_metrics(names, "#{e.class}: #{e.message}") if names
|
|
157
|
+
|
|
158
|
+
@queue_metrics_errors << { queues: [], error: "#{e.class}: #{e.message}" }
|
|
101
159
|
[]
|
|
102
160
|
end
|
|
103
161
|
private :fetch_queues_with_metrics
|
|
104
162
|
|
|
163
|
+
# A name sanitize_name rejects never reaches a caller: before #677 the
|
|
164
|
+
# metrics read dropped it, and every caller took its names from there.
|
|
165
|
+
def read_queue_names
|
|
166
|
+
connection.select_values("SELECT queue_name FROM pgmq.meta ORDER BY queue_name").select do |name|
|
|
167
|
+
sanitize_name(name)
|
|
168
|
+
rescue ArgumentError => e
|
|
169
|
+
Pgbus.logger.debug { "[Pgbus::Web] Skipping queue #{name.inspect}: #{e.message}" }
|
|
170
|
+
false
|
|
171
|
+
end
|
|
172
|
+
end
|
|
173
|
+
private :read_queue_names
|
|
174
|
+
|
|
105
175
|
# Physical queue names known to back streams: the pgbus_stream_queues
|
|
106
176
|
# registry plus any dormant pre-registry queues discovered by the
|
|
107
177
|
# archive msg_id index fingerprint (issue #366). Stream delivery is a
|
|
@@ -248,20 +318,20 @@ module Pgbus
|
|
|
248
318
|
def discard_all_enqueued
|
|
249
319
|
dlq_suffix = Pgbus::DEAD_LETTER_SUFFIX
|
|
250
320
|
handlers = handler_queue_physical_names
|
|
251
|
-
|
|
321
|
+
names = queue_names.reject { |n| n.end_with?(dlq_suffix) || handlers.include?(n) }
|
|
252
322
|
total = 0
|
|
253
323
|
|
|
254
|
-
|
|
255
|
-
messages = query_queue_messages_raw(
|
|
324
|
+
names.each do |name|
|
|
325
|
+
messages = query_queue_messages_raw(name, 10_000, 0)
|
|
256
326
|
next if messages.empty?
|
|
257
327
|
|
|
258
328
|
release_locks_for_messages(messages)
|
|
259
329
|
|
|
260
330
|
ids = messages.map { |m| m[:msg_id].to_i }
|
|
261
|
-
@client.archive_batch(
|
|
331
|
+
@client.archive_batch(name, ids, prefixed: false)
|
|
262
332
|
total += ids.size
|
|
263
333
|
rescue StandardError => e
|
|
264
|
-
Pgbus.logger.debug { "[Pgbus::Web] Error discarding enqueued messages from #{
|
|
334
|
+
Pgbus.logger.debug { "[Pgbus::Web] Error discarding enqueued messages from #{name}: #{e.message}" }
|
|
265
335
|
end
|
|
266
336
|
|
|
267
337
|
total
|
|
@@ -1280,8 +1350,7 @@ module Pgbus
|
|
|
1280
1350
|
|
|
1281
1351
|
def all_queue_messages(limit, offset)
|
|
1282
1352
|
dlq_suffix = Pgbus::DEAD_LETTER_SUFFIX
|
|
1283
|
-
|
|
1284
|
-
paginated_queue_messages(queues.map { |q| q[:name] }, limit, offset)
|
|
1353
|
+
paginated_queue_messages(queue_names.reject { |n| n.end_with?(dlq_suffix) }, limit, offset)
|
|
1285
1354
|
end
|
|
1286
1355
|
|
|
1287
1356
|
# Returns messages from multiple PGMQ queues in a single paginated
|
|
@@ -1324,53 +1393,98 @@ module Pgbus
|
|
|
1324
1393
|
end.join("\nUNION ALL\n")
|
|
1325
1394
|
end
|
|
1326
1395
|
|
|
1396
|
+
# Every queue's metrics, web_queue_metrics_chunk_size queues per
|
|
1397
|
+
# statement (issue #677). Each queue's table is scanned once, by one
|
|
1398
|
+
# aggregate, so a statement's cost grows with its chunk, not with the
|
|
1399
|
+
# whole fleet. Each chunk runs in its own savepoint: a chunk that fails
|
|
1400
|
+
# (statement_timeout included) rolls back to it, so a caller's open
|
|
1401
|
+
# transaction stays usable, and only that chunk's queues come back as
|
|
1402
|
+
# unavailable. Rows follow +queue_names+ order.
|
|
1403
|
+
#
|
|
1404
|
+
# Chunks run one after another, so a database where every chunk times
|
|
1405
|
+
# out would cost chunks × statement_timeout. After
|
|
1406
|
+
# MAX_FAILED_METRICS_CHUNKS failed chunks the rest are not run: their
|
|
1407
|
+
# queues come back unavailable at once.
|
|
1327
1408
|
def batched_queue_metrics(queue_names)
|
|
1328
|
-
|
|
1409
|
+
chunk_size = Pgbus.configuration.web_queue_metrics_chunk_size
|
|
1410
|
+
rows = []
|
|
1411
|
+
failed = 0
|
|
1412
|
+
queue_names.each_slice(chunk_size).with_index do |chunk, index|
|
|
1413
|
+
if failed >= MAX_FAILED_METRICS_CHUNKS
|
|
1414
|
+
skipped = queue_names.drop(index * chunk_size)
|
|
1415
|
+
rows.concat(unavailable_queue_metrics(skipped, "not read: #{failed} metrics chunks timed out before them"))
|
|
1416
|
+
break
|
|
1417
|
+
end
|
|
1329
1418
|
|
|
1330
|
-
|
|
1331
|
-
|
|
1332
|
-
|
|
1333
|
-
|
|
1334
|
-
|
|
1335
|
-
|
|
1336
|
-
|
|
1337
|
-
|
|
1338
|
-
|
|
1339
|
-
|
|
1340
|
-
|
|
1341
|
-
|
|
1342
|
-
|
|
1343
|
-
(SELECT count(*) FROM pgmq.#{qtable} WHERE vt <= NOW() AND read_ct = 0) AS visible_unread_length,
|
|
1344
|
-
(SELECT EXTRACT(epoch FROM (NOW() - min(vt)))::int FROM pgmq.#{qtable} WHERE vt <= NOW()) AS oldest_claimable_age_sec,
|
|
1345
|
-
(SELECT EXTRACT(epoch FROM (NOW() - min(vt)))::int FROM pgmq.#{qtable} WHERE vt <= NOW() AND read_ct = 0) AS oldest_unread_claimable_age_sec
|
|
1346
|
-
SQL
|
|
1347
|
-
rescue StandardError => e
|
|
1348
|
-
Pgbus.logger.debug { "[Pgbus::Web] Skipping queue metrics for #{name}: #{e.message}" }
|
|
1349
|
-
nil
|
|
1419
|
+
chunk_rows, error = chunk_queue_metrics(chunk)
|
|
1420
|
+
failed += 1 if SLOW_METRICS_FAILURES.any? { |klass| error.is_a?(klass) }
|
|
1421
|
+
rows.concat(chunk_rows)
|
|
1422
|
+
end
|
|
1423
|
+
rows
|
|
1424
|
+
end
|
|
1425
|
+
|
|
1426
|
+
# [rows, nil], or [unavailable rows, the error] for a failed chunk.
|
|
1427
|
+
# +chunk+ holds names read_queue_names already validated.
|
|
1428
|
+
def chunk_queue_metrics(chunk)
|
|
1429
|
+
sql = chunk.map { |name| "(#{queue_metrics_select(name)})" }.join("\nUNION ALL\n")
|
|
1430
|
+
rows = connection.transaction(requires_new: true) do
|
|
1431
|
+
connection.select_all(sql, "Pgbus Batched Queue Metrics").to_a
|
|
1350
1432
|
end
|
|
1433
|
+
by_name = rows.to_h { |row| [row["queue_name"], row] }
|
|
1434
|
+
[chunk.filter_map { |name| by_name[name] && queue_metrics_hash(name, by_name[name]) }, nil]
|
|
1435
|
+
rescue StandardError => e
|
|
1436
|
+
[unavailable_queue_metrics(chunk, "#{e.class}: #{e.message}"), e]
|
|
1437
|
+
end
|
|
1438
|
+
|
|
1439
|
+
# One row for one queue: a single aggregate over its table, plus the
|
|
1440
|
+
# msg_id sequence for the lifetime total. The same expressions as
|
|
1441
|
+
# queue_metrics_via_sql, so a queue reads the same on both paths.
|
|
1442
|
+
def queue_metrics_select(name)
|
|
1443
|
+
qtable = "q_#{sanitize_name(name)}"
|
|
1444
|
+
<<~SQL.strip
|
|
1445
|
+
SELECT #{connection.quote(name)} AS queue_name,
|
|
1446
|
+
m.queue_length, m.queue_visible_length, m.newest_msg_age_sec, m.oldest_msg_age_sec,
|
|
1447
|
+
(SELECT CASE WHEN is_called THEN last_value ELSE 0 END FROM pgmq.#{qtable}_msg_id_seq) AS total_messages,
|
|
1448
|
+
m.max_read_ct, m.visible_unread_length, m.oldest_claimable_age_sec, m.oldest_unread_claimable_age_sec
|
|
1449
|
+
FROM (
|
|
1450
|
+
SELECT count(*) AS queue_length,
|
|
1451
|
+
count(*) FILTER (WHERE vt <= NOW()) AS queue_visible_length,
|
|
1452
|
+
EXTRACT(epoch FROM (NOW() - max(enqueued_at)))::int AS newest_msg_age_sec,
|
|
1453
|
+
EXTRACT(epoch FROM (NOW() - min(enqueued_at)))::int AS oldest_msg_age_sec,
|
|
1454
|
+
max(read_ct) AS max_read_ct,
|
|
1455
|
+
count(*) FILTER (WHERE vt <= NOW() AND read_ct = 0) AS visible_unread_length,
|
|
1456
|
+
EXTRACT(epoch FROM (NOW() - min(vt) FILTER (WHERE vt <= NOW())))::int AS oldest_claimable_age_sec,
|
|
1457
|
+
EXTRACT(epoch FROM (NOW() - min(vt) FILTER (WHERE vt <= NOW() AND read_ct = 0)))::int AS oldest_unread_claimable_age_sec
|
|
1458
|
+
FROM pgmq.#{qtable}
|
|
1459
|
+
) AS m
|
|
1460
|
+
SQL
|
|
1461
|
+
end
|
|
1351
1462
|
|
|
1352
|
-
|
|
1463
|
+
def queue_metrics_hash(name, row)
|
|
1464
|
+
{
|
|
1465
|
+
name: name,
|
|
1466
|
+
queue_length: row["queue_length"].to_i,
|
|
1467
|
+
queue_visible_length: row["queue_visible_length"].to_i,
|
|
1468
|
+
parked_length: row["queue_length"].to_i - row["queue_visible_length"].to_i,
|
|
1469
|
+
oldest_msg_age_sec: row["oldest_msg_age_sec"]&.to_i,
|
|
1470
|
+
oldest_claimable_age_sec: row["oldest_claimable_age_sec"]&.to_i,
|
|
1471
|
+
newest_msg_age_sec: row["newest_msg_age_sec"]&.to_i,
|
|
1472
|
+
total_messages: row["total_messages"].to_i,
|
|
1473
|
+
max_read_ct: row["max_read_ct"]&.to_i,
|
|
1474
|
+
visible_unread_length: row["visible_unread_length"].to_i,
|
|
1475
|
+
oldest_unread_claimable_age_sec: row["oldest_unread_claimable_age_sec"]&.to_i
|
|
1476
|
+
}
|
|
1477
|
+
end
|
|
1353
1478
|
|
|
1354
|
-
|
|
1355
|
-
|
|
1356
|
-
|
|
1357
|
-
|
|
1358
|
-
|
|
1359
|
-
|
|
1360
|
-
queue_visible_length: row["queue_visible_length"].to_i,
|
|
1361
|
-
parked_length: row["queue_length"].to_i - row["queue_visible_length"].to_i,
|
|
1362
|
-
oldest_msg_age_sec: row["oldest_msg_age_sec"]&.to_i,
|
|
1363
|
-
oldest_claimable_age_sec: row["oldest_claimable_age_sec"]&.to_i,
|
|
1364
|
-
newest_msg_age_sec: row["newest_msg_age_sec"]&.to_i,
|
|
1365
|
-
total_messages: row["total_messages"].to_i,
|
|
1366
|
-
max_read_ct: row["max_read_ct"]&.to_i,
|
|
1367
|
-
visible_unread_length: row["visible_unread_length"].to_i,
|
|
1368
|
-
oldest_unread_claimable_age_sec: row["oldest_unread_claimable_age_sec"]&.to_i
|
|
1369
|
-
}
|
|
1479
|
+
# A failed chunk keeps its queues, flagged, with every metric nil: a
|
|
1480
|
+
# count we could not read is unknown, not zero.
|
|
1481
|
+
def unavailable_queue_metrics(names, error)
|
|
1482
|
+
Pgbus.logger.error do
|
|
1483
|
+
"[Pgbus::Web] Queue metrics unavailable for #{names.size} queue(s) (#{names.first(3).join(", ")}" \
|
|
1484
|
+
"#{", …" if names.size > 3}): #{error}"
|
|
1370
1485
|
end
|
|
1371
|
-
|
|
1372
|
-
|
|
1373
|
-
[]
|
|
1486
|
+
(@queue_metrics_errors ||= []) << { queues: names, error: error }
|
|
1487
|
+
names.map { |name| { name: name, **UNAVAILABLE_METRICS, metrics_unavailable: true } }
|
|
1374
1488
|
end
|
|
1375
1489
|
|
|
1376
1490
|
def queue_metrics_via_sql(queue_name)
|
|
@@ -1509,14 +1623,20 @@ module Pgbus
|
|
|
1509
1623
|
QueueNameValidator.sanitize!(name)
|
|
1510
1624
|
end
|
|
1511
1625
|
|
|
1626
|
+
# A read with a queue's metrics unavailable (issue #677) leaves the
|
|
1627
|
+
# snapshot alone: its lifetime total is unknown, and counting it as zero
|
|
1628
|
+
# would read as a drop now and a spike once it reports again.
|
|
1512
1629
|
def compute_throughput(queues)
|
|
1630
|
+
return 0.0 if queues.any? { |q| q[:metrics_unavailable] }
|
|
1631
|
+
|
|
1513
1632
|
current_totals = queues.sum { |q| q[:total_messages] }
|
|
1514
|
-
now = Process.clock_gettime(Process::CLOCK_MONOTONIC)
|
|
1633
|
+
now = ::Process.clock_gettime(::Process::CLOCK_MONOTONIC)
|
|
1515
1634
|
|
|
1516
1635
|
if @last_throughput_snapshot && @last_throughput_at
|
|
1517
1636
|
elapsed = now - @last_throughput_at
|
|
1518
1637
|
delta = current_totals - @last_throughput_snapshot
|
|
1519
|
-
|
|
1638
|
+
# A dropped queue lowers the lifetime total: never a negative rate.
|
|
1639
|
+
rate = elapsed.positive? ? [(delta / elapsed).round(1), 0.0].max : 0.0
|
|
1520
1640
|
end
|
|
1521
1641
|
|
|
1522
1642
|
@last_throughput_snapshot = current_totals
|
|
@@ -36,8 +36,13 @@ module Pgbus
|
|
|
36
36
|
private
|
|
37
37
|
|
|
38
38
|
def append_queue_metrics(lines)
|
|
39
|
-
|
|
40
|
-
return if
|
|
39
|
+
all = @data_source.queues_with_metrics
|
|
40
|
+
return append_queue_list_unavailable(lines) if all.empty?
|
|
41
|
+
|
|
42
|
+
# A queue whose metrics could not be read (issue #677) has nil counts:
|
|
43
|
+
# it gets no count sample (format_value would write a false 0), only
|
|
44
|
+
# its paused state and pgbus_queue_metrics_unavailable.
|
|
45
|
+
queues = all.reject { |q| q[:metrics_unavailable] }
|
|
41
46
|
|
|
42
47
|
gauge(lines, "pgbus_queue_depth", "Number of messages in the queue (including invisible)") do
|
|
43
48
|
queues.map { |q| [q[:queue_length], { queue: q[:name] }] }
|
|
@@ -69,12 +74,32 @@ module Pgbus
|
|
|
69
74
|
end
|
|
70
75
|
|
|
71
76
|
gauge(lines, "pgbus_queue_paused", "Whether the queue is paused (1) or active (0)") do
|
|
72
|
-
|
|
77
|
+
all.map { |q| [q[:paused] ? 1 : 0, { queue: q[:name] }] }
|
|
78
|
+
end
|
|
79
|
+
|
|
80
|
+
gauge(lines, "pgbus_queue_metrics_unavailable",
|
|
81
|
+
"Whether the queue's metrics could not be read this scrape (1) or were read (0)") do
|
|
82
|
+
all.map { |q| [q[:metrics_unavailable] ? 1 : 0, { queue: q[:name] }] }
|
|
73
83
|
end
|
|
74
84
|
rescue StandardError => e
|
|
75
85
|
Pgbus.logger.debug { "[Pgbus::Metrics] Error serializing queue metrics: #{e.message}" }
|
|
76
86
|
end
|
|
77
87
|
|
|
88
|
+
# No queue rows because pgmq.meta could not be read (issue #677): one
|
|
89
|
+
# unlabelled sample, so the scrape says so instead of going quiet.
|
|
90
|
+
def append_queue_list_unavailable(lines)
|
|
91
|
+
return unless queue_list_unavailable?
|
|
92
|
+
|
|
93
|
+
gauge(lines, "pgbus_queue_metrics_unavailable",
|
|
94
|
+
"Whether the queue's metrics could not be read this scrape (1) or were read (0)") { [[1]] }
|
|
95
|
+
end
|
|
96
|
+
|
|
97
|
+
def queue_list_unavailable?
|
|
98
|
+
return false unless @data_source.respond_to?(:queue_metrics_errors)
|
|
99
|
+
|
|
100
|
+
@data_source.queue_metrics_errors.any? { |e| Array(e[:queues]).empty? }
|
|
101
|
+
end
|
|
102
|
+
|
|
78
103
|
def append_job_metrics(lines)
|
|
79
104
|
counts = @data_source.job_status_counts
|
|
80
105
|
unless counts.empty?
|
|
@@ -168,8 +193,12 @@ module Pgbus
|
|
|
168
193
|
[[stats[:failed_count]]]
|
|
169
194
|
end
|
|
170
195
|
|
|
171
|
-
|
|
172
|
-
|
|
196
|
+
# An unreadable queue list (issue #677) makes the DLQ total unknown,
|
|
197
|
+
# not 0: no sample, so a dlq_depth alert does not clear on it.
|
|
198
|
+
unless queue_list_unavailable?
|
|
199
|
+
gauge(lines, "pgbus_dlq_depth", "Total messages across all dead letter queues") do
|
|
200
|
+
[[stats[:dlq_depth]]]
|
|
201
|
+
end
|
|
173
202
|
end
|
|
174
203
|
rescue StandardError => e
|
|
175
204
|
Pgbus.logger.debug { "[Pgbus::Metrics] Error serializing summary metrics: #{e.message}" }
|
|
@@ -19,13 +19,18 @@ module Pgbus
|
|
|
19
19
|
|
|
20
20
|
module_function
|
|
21
21
|
|
|
22
|
+
# A queue whose metrics could not be read (issue #677) says so right
|
|
23
|
+
# after any pause, ahead of the drain line: "no worker — N claimable"
|
|
24
|
+
# would quote an unknown count as 0.
|
|
22
25
|
def present(detail, pause_state, drainers, max_retries:)
|
|
23
26
|
name = detail[:name].to_s
|
|
24
27
|
dlq = name.end_with?(Pgbus::DEAD_LETTER_SUFFIX)
|
|
28
|
+
unavailable = line("metrics_unavailable", :yellow) if detail[:metrics_unavailable]
|
|
25
29
|
[
|
|
26
30
|
pause_line(pause_state),
|
|
31
|
+
unavailable,
|
|
27
32
|
drain_line(dlq, detail, drainers, max_retries),
|
|
28
|
-
(backlog_line(detail) unless dlq || drainers[:stream]),
|
|
33
|
+
(backlog_line(detail) unless unavailable || dlq || drainers[:stream]),
|
|
29
34
|
priority_line(drainers)
|
|
30
35
|
].compact
|
|
31
36
|
end
|
data/sig/pgbus/configuration.rbs
CHANGED
|
@@ -327,6 +327,8 @@ module Pgbus
|
|
|
327
327
|
def web_live_updates=: (untyped) -> untyped
|
|
328
328
|
def web_per_page: () -> Integer
|
|
329
329
|
def web_per_page=: (untyped) -> untyped
|
|
330
|
+
def web_queue_metrics_chunk_size: () -> Integer
|
|
331
|
+
def web_queue_metrics_chunk_size=: (untyped) -> untyped
|
|
330
332
|
def web_refresh_interval: () -> Integer
|
|
331
333
|
def web_refresh_interval=: (untyped) -> untyped
|
|
332
334
|
def worker_notify_connection_options: () -> untyped
|