pgbus 0.27.0 → 0.27.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 1b883f03f4b08531d1e4847486beb29842afde42a872453606cae421952a86e6
4
- data.tar.gz: '09f28e7f82f7e843ce8bd8aeda05c53481ade115e6f66fb2bc2197ae45d081df'
3
+ metadata.gz: 6b6f949cf25759a3498971f5c00a1ba4c3beee0bdcadac41c6bd978af96ba392
4
+ data.tar.gz: 0cd2f5abd43c6f5b380f2439b40553dd707ec0283dc42fbe696d6dd2f0a0172e
5
5
  SHA512:
6
- metadata.gz: ca5ce2420a6df278b3aae31ebcca0c0dc1980373a8954254de83310a75c56574cda6b3549787a662b38ade7a7fa6a8d24a2471b2ba97981f22e04a6884f4c4fb
7
- data.tar.gz: fecd5c8e775ef8367f9adceee008b6b02afea7b842ff9827bf000993e3fabb82534a49464e81b68e2fd35d5cb62ecd6cacff6efdd6c04d5eebb08c6e9859a7a7
6
+ metadata.gz: 2d9fa8de5bcae929a3546fba35c7734e83ca9794aa582adc69cb9fee0aadbd397061324cea4e00fa5fcb77bc79e11e39909ff96fd4c27e83d8f94c7acca46a42
7
+ data.tar.gz: 7fba6d23df3a93fa218b2f23cb59b6a942a45ebdd8aadd57cf00f01d3f110adb7eb406086aed7922b0b54931bb937cfe366fef5230bb01fc517dd50d3bb64c0a
data/CHANGELOG.md CHANGED
@@ -24,6 +24,15 @@
24
24
 
25
25
  ### Fixed
26
26
 
27
+ - **`Web::DataSource#queues_with_metrics` reads 1 000 queues in a tenth of the time, and a metrics query that fails or times out no longer reads as "no queues" (issue #677).** The read was one statement with nine scalar subqueries per queue (eight scans of each queue table plus one sequence read), so its cost grew with the queue count until `statement_timeout` cancelled it, and the rescue returned `[]`: the dashboard, the dead-letter views, `pgbus_health`, the Prometheus scrape and the AppSignal probe all read a timeout as an empty, healthy system. Inside a caller's transaction the cancelled statement also aborted it.
28
+
29
+ - **One scan per queue, in chunks.** Each queue's metrics are one aggregate over its table, and the queues are read `web_queue_metrics_chunk_size` (new, default 100) per statement, each statement in its own savepoint. Same machine, one message per queue: 100 queues 57.7 ms → 37.4 ms, 1 000 queues 3 711 ms → 373 ms, 5 000 queues 126 s → 1.9 s (`rake bench:queue_metrics`, new). A healthy queue's hash is unchanged.
30
+ - **A failed chunk is visible.** Its queues stay in the list with `metrics_unavailable: true` and `nil` counts; the other chunks still report, and an enclosing transaction stays usable. After 3 chunks that timed out (a cancelled statement, a lock wait, a deadlock, a connection that cannot be established) the rest are not read and come back unavailable at once, so a database where every chunk times out costs a few `statement_timeout`s, not one per chunk; a chunk that fails at once (a queue whose table is gone) never stops the others. Totals (`summary_stats` `total_depth` / `dlq_depth`, the `pgbus_dlq_depth` and AppSignal summary gauges) leave an unmeasured queue out rather than guess, so they read low while the verdict is DEGRADED. New `DataSource#queue_metrics_errors` returns `[]` or `[{ queues: [...], error: "Class: message" }]` (`queues: []` when `pgmq.meta` itself could not be read). `pgbus_health` / `/readyz` report `DEGRADED` with a reason naming the queues; the dashboard queue tables say "Metrics unavailable" in place of the counts; the AppSignal probe skips the queue's gauges and increments `pgbus_queue_metrics_unavailable`; the Prometheus scrape writes no count for it and adds the `pgbus_queue_metrics_unavailable` gauge; the MCP `pgbus_queues` tool carries the flag per row; the dead-letter total counts the rows of a DLQ whose metrics are unavailable (in a savepoint) rather than reading it as 0. An unreadable queue list increments the AppSignal counter untagged, writes one unlabelled Prometheus sample and leaves `pgbus_dlq_depth` out of the scrape; a failure after the queue list was read keeps every queue, flagged.
31
+ - **Names without metrics.** New `DataSource#queue_names` reads `pgmq.meta` only (memoized, cleared by `reset_cache!`). The job list, `dlq_messages`, `dlq_message_detail`, discard-all, priority siblings and the MCP `pgbus_dlq_detail` scan use it and run no metrics statement (the dead-letter page's counts, `dlq_total_count` and `dlq_counts_by_queue`, still read the metrics).
32
+ - **Throughput.** `summary_stats[:throughput_rate]` was always 0.0: `compute_throughput` called `Process.clock_gettime` inside `Pgbus::Web`, where `Process` is `Pgbus::Process`, and rescued the `NoMethodError`. It now reads the clock, never reports a negative rate (a dropped queue lowers the total), and keeps its last complete snapshot while any queue's metrics are unavailable.
33
+
34
+ Refs #677.
35
+
27
36
  - **A pooled connection in `connection_guc_mode = :session` keeps its database.yml `variables` when it is reset (issue #671).** In `:session` mode the `variables` (`statement_timeout`, `lock_timeout` and the like) are applied with `SET` after connect, and `PG::Connection#reset` opens a new backend without them. pgmq-ruby's checkout health check resets an unhealthy pooled connection (`finished?` or `CONNECTION_BAD` since #332, and an intermediate state such as `CONNECTION_STARTED` since pgmq-ruby 0.9.0), so that connection went on serving the job or streams pool with the server's defaults. Each connection the `:session` connect lambda opens now re-applies its `SET`s when it is reset, whoever resets it. A connection whose `SET`s do not all complete, after a connect or a reset, is closed before the error reaches the caller, so it is never pooled without them; with pg 1.5.9 or later the next checkout resets it and applies the `SET`s again (an older pg cannot reset a closed connection, so it is replaced when the pool is reloaded before the last stale-connection retry; the call itself is retried only for a stale-connection error, so a `SET` that always fails, such as an invalid value, fails each time). The completion pipeline's own reset (#619) no longer applies the `SET`s itself (they ran twice), and a `SET` that fails there is now logged as `Resetting the connection or re-applying its session variables failed` and leaves the connection closed, where before the session carried on without its `variables`. `:options` mode, a String URL and the shared ActiveRecord connection are unaffected. Refs #671.
28
37
 
29
38
  ## [0.25.1] - 2026-11-15
data/README.md CHANGED
@@ -1025,7 +1025,7 @@ When the `appsignal` gem is loaded in your app, Pgbus auto-installs a subscriber
1025
1025
 
1026
1026
  - **Background-job transactions** for every ActiveJob run and every event-bus handler invocation. Action names follow the AppSignal convention: `MyJob#perform`, `MyHandler#handle`. Tags include `queue`, `job_class`/`handler`, `routing_key`, `attempts`, and the `active_job_id` / `provider_job_id`. `enqueued_at` becomes the AppSignal `queue_start` timestamp so "time on queue" shows up correctly in the timeline. The transaction is current for the whole execution (deserialize, `perform`/`handle`, archive), so `Appsignal.add_tags`, `Appsignal.set_error` and `Appsignal.instrument` called from job or handler code attach to it, and the error an execution raised is reported on it, also when it was raised before `perform` (an unknown job class, arguments that cannot be deserialized).
1027
1027
  - **Custom counters and distributions** for sends, reads, broadcasts, outbox publishes, recurring scheduling, and worker recycles. All metric names are prefixed `pgbus_`.
1028
- - **A minutely probe** that gauges queue depth (visible vs total), oldest message age per queue, DLQ depth, failed events count, dead-tuple totals, MVCC horizon age, active processes, and stream connection estimates.
1028
+ - **A minutely probe** that gauges queue depth (visible vs total), oldest message age per queue, DLQ depth, failed events count, dead-tuple totals, MVCC horizon age, active processes, and stream connection estimates. A queue whose metrics could not be read (its metrics query failed or hit `statement_timeout`) gets no gauges that minute; it increments the `pgbus_queue_metrics_unavailable` counter, tagged by queue, instead (untagged when the queue list itself could not be read).
1029
1029
 
1030
1030
  There is nothing to wire up — load the appsignal gem and the integration installs itself in a Rails initializer. To opt out:
1031
1031
 
@@ -1205,6 +1205,7 @@ When `config.metrics_enabled = true` (default), the dashboard exposes Prometheus
1205
1205
  | Metric | Description |
1206
1206
  |--------|-------------|
1207
1207
  | `pgbus_queue_oldest_claimable_age_seconds` | Age of the oldest message eligible for pickup (visibility timeout elapsed) — safe to alert on: scheduled/backoff-parked messages don't count until due; the series is omitted entirely when no claimable backlog exists (while the raw `pgbus_queue_oldest_message_age_seconds` gauge may still report a parked message's age) |
1208
+ | `pgbus_queue_metrics_unavailable` | 1 when the queue's metrics could not be read this scrape (its metrics query failed or hit `statement_timeout`), else 0; such a queue has no depth or age samples, rather than a false 0. One unlabelled `1` sample when the queue list itself could not be read |
1208
1209
  | `pgbus_table_dead_tuples` | Dead tuple count per PGMQ table |
1209
1210
  | `pgbus_table_live_tuples` | Live tuple count per PGMQ table |
1210
1211
  | `pgbus_table_bloat_ratio` | Dead / (dead + live) per table |
data/Rakefile CHANGED
@@ -26,7 +26,8 @@ namespace :bench do
26
26
  db_benches = %w[connection_pool_bench integration_bench streams_bench streams_read_pool_bench
27
27
  execution_modes_bench pool_swap_bench pool_autoscale_bench job_burst_bench
28
28
  notify_wake_bench notify_chaos_bench streams_hub_bench fair_read_bench
29
- worker_profile_bench workflow_insights_bench streams_scale_bench stat_flush_bench].freeze
29
+ worker_profile_bench workflow_insights_bench streams_scale_bench stat_flush_bench
30
+ queue_metrics_bench].freeze
30
31
  # The unit suite is every *_bench.rb that doesn't need a database, derived
31
32
  # from the directory so a new unit bench is picked up automatically (kept in
32
33
  # sync with bench:one, which globs the same files).
@@ -78,6 +79,11 @@ namespace :bench do
78
79
  ruby "benchmarks/workflow_insights_bench.rb"
79
80
  end
80
81
 
82
+ desc "Time queues_with_metrics at 100 / 1 000 / 5 000 queues (requires PGBUS_DATABASE_URL)"
83
+ task :queue_metrics do
84
+ ruby "benchmarks/queue_metrics_bench.rb"
85
+ end
86
+
81
87
  desc "Run fair share read benchmark (requires PGBUS_DATABASE_URL)"
82
88
  task :fair_read do
83
89
  ruby "benchmarks/fair_read_bench.rb"
@@ -26,11 +26,15 @@
26
26
  <%= pgbus_queue_badge(q[:name]) %>
27
27
  </td>
28
28
  <td data-label="<%= t("pgbus.dashboard.queues_table.headers.status") %>" class="px-4 py-3 text-sm"><%= pgbus_queue_status(@queue_lines[q[:name]]) %></td>
29
- <td data-label="<%= t("pgbus.dashboard.queues_table.headers.depth") %>" class="px-4 py-3 text-sm text-right text-gray-700 dark:text-gray-200"><%= pgbus_number(q[:queue_length]) %></td>
30
- <td data-label="<%= t("pgbus.dashboard.queues_table.headers.visible") %>" class="px-4 py-3 text-sm text-right text-gray-700 dark:text-gray-200"><%= pgbus_number(q[:queue_visible_length]) %></td>
31
- <td data-label="<%= t("pgbus.dashboard.queues_table.headers.parked") %>" class="px-4 py-3 text-sm text-right text-gray-500 dark:text-gray-400"><%= pgbus_number(q[:parked_length]) %></td>
32
- <td data-label="<%= t("pgbus.dashboard.queues_table.headers.oldest_claimable") %>" class="px-4 py-3 text-sm text-right text-gray-500 dark:text-gray-400"><%= pgbus_duration(q[:oldest_claimable_age_sec]) %></td>
33
- <td data-label="<%= t("pgbus.dashboard.queues_table.headers.total") %>" class="px-4 py-3 text-sm text-right text-gray-500 dark:text-gray-400"><%= pgbus_number(q[:total_messages]) %></td>
29
+ <% if q[:metrics_unavailable] %>
30
+ <td colspan="5" data-label="<%= t("pgbus.dashboard.queues_table.headers.depth") %>" class="px-4 py-3 text-sm text-right text-yellow-800 dark:text-yellow-200"><%= t("pgbus.queues.show.summary.metrics_unavailable_html") %></td>
31
+ <% else %>
32
+ <td data-label="<%= t("pgbus.dashboard.queues_table.headers.depth") %>" class="px-4 py-3 text-sm text-right text-gray-700 dark:text-gray-200"><%= pgbus_number(q[:queue_length]) %></td>
33
+ <td data-label="<%= t("pgbus.dashboard.queues_table.headers.visible") %>" class="px-4 py-3 text-sm text-right text-gray-700 dark:text-gray-200"><%= pgbus_number(q[:queue_visible_length]) %></td>
34
+ <td data-label="<%= t("pgbus.dashboard.queues_table.headers.parked") %>" class="px-4 py-3 text-sm text-right text-gray-500 dark:text-gray-400"><%= pgbus_number(q[:parked_length]) %></td>
35
+ <td data-label="<%= t("pgbus.dashboard.queues_table.headers.oldest_claimable") %>" class="px-4 py-3 text-sm text-right text-gray-500 dark:text-gray-400"><%= pgbus_duration(q[:oldest_claimable_age_sec]) %></td>
36
+ <td data-label="<%= t("pgbus.dashboard.queues_table.headers.total") %>" class="px-4 py-3 text-sm text-right text-gray-500 dark:text-gray-400"><%= pgbus_number(q[:total_messages]) %></td>
37
+ <% end %>
34
38
  </tr>
35
39
  <% end %>
36
40
  <% if @queues.empty? %>
@@ -23,12 +23,16 @@
23
23
  <%= pgbus_paused_badge(q[:paused]) %>
24
24
  <% end %>
25
25
  </td>
26
- <td data-label="<%= t("pgbus.queues.queues_list.headers.depth") %>" class="px-4 py-3 text-sm text-right font-mono text-gray-700 dark:text-gray-200"><%= pgbus_number(q[:queue_length]) %></td>
27
- <td data-label="<%= t("pgbus.queues.queues_list.headers.visible") %>" class="px-4 py-3 text-sm text-right font-mono text-gray-700 dark:text-gray-200"><%= pgbus_number(q[:queue_visible_length]) %></td>
28
- <td data-label="<%= t("pgbus.queues.queues_list.headers.parked") %>" class="px-4 py-3 text-sm text-right font-mono text-gray-500 dark:text-gray-400"><%= pgbus_number(q[:parked_length]) %></td>
29
- <td data-label="<%= t("pgbus.queues.queues_list.headers.oldest_claimable") %>" class="px-4 py-3 text-sm text-right text-gray-500 dark:text-gray-400"><%= pgbus_duration(q[:oldest_claimable_age_sec]) %></td>
30
- <td data-label="<%= t("pgbus.queues.queues_list.headers.newest") %>" class="px-4 py-3 text-sm text-right text-gray-500 dark:text-gray-400"><%= pgbus_duration(q[:newest_msg_age_sec]) %></td>
31
- <td data-label="<%= t("pgbus.queues.queues_list.headers.total_ever") %>" class="px-4 py-3 text-sm text-right text-gray-500 dark:text-gray-400"><%= pgbus_number(q[:total_messages]) %></td>
26
+ <% if q[:metrics_unavailable] %>
27
+ <td colspan="6" data-label="<%= t("pgbus.queues.queues_list.headers.depth") %>" class="px-4 py-3 text-sm text-right text-yellow-800 dark:text-yellow-200"><%= t("pgbus.queues.show.summary.metrics_unavailable_html") %></td>
28
+ <% else %>
29
+ <td data-label="<%= t("pgbus.queues.queues_list.headers.depth") %>" class="px-4 py-3 text-sm text-right font-mono text-gray-700 dark:text-gray-200"><%= pgbus_number(q[:queue_length]) %></td>
30
+ <td data-label="<%= t("pgbus.queues.queues_list.headers.visible") %>" class="px-4 py-3 text-sm text-right font-mono text-gray-700 dark:text-gray-200"><%= pgbus_number(q[:queue_visible_length]) %></td>
31
+ <td data-label="<%= t("pgbus.queues.queues_list.headers.parked") %>" class="px-4 py-3 text-sm text-right font-mono text-gray-500 dark:text-gray-400"><%= pgbus_number(q[:parked_length]) %></td>
32
+ <td data-label="<%= t("pgbus.queues.queues_list.headers.oldest_claimable") %>" class="px-4 py-3 text-sm text-right text-gray-500 dark:text-gray-400"><%= pgbus_duration(q[:oldest_claimable_age_sec]) %></td>
33
+ <td data-label="<%= t("pgbus.queues.queues_list.headers.newest") %>" class="px-4 py-3 text-sm text-right text-gray-500 dark:text-gray-400"><%= pgbus_duration(q[:newest_msg_age_sec]) %></td>
34
+ <td data-label="<%= t("pgbus.queues.queues_list.headers.total_ever") %>" class="px-4 py-3 text-sm text-right text-gray-500 dark:text-gray-400"><%= pgbus_number(q[:total_messages]) %></td>
35
+ <% end %>
32
36
  <td data-label="<%= t("pgbus.queues.queues_list.headers.actions") %>" class="px-4 py-3 text-sm text-right space-x-2">
33
37
  <% if q[:paused] %>
34
38
  <%= pgbus_button_to t("pgbus.queues.queues_list.resume"), pgbus.resume_queue_path(name: q[:name]),
@@ -973,6 +973,7 @@ da:
973
973
  other: Tømmes af en wildcard-worker-kapsel (%{count} sunde workers)
974
974
  empty_html: Intet venter
975
975
  handler_html: Event handler-kø — tømmes af consumers (%{count} sunde)
976
+ metrics_unavailable_html: 'Metrikker utilgængelige: metrikforespørgslen fejlede eller fik timeout'
976
977
  no_consumers_html:
977
978
  one: Event handler-kø uden sund consumer — %{count} event venter
978
979
  other: Event handler-kø uden sund consumer — %{count} events venter
@@ -973,6 +973,7 @@ de:
973
973
  other: Abgearbeitet von einer Wildcard-Worker-Kapsel (%{count} gesunde Worker)
974
974
  empty_html: Nichts wartet
975
975
  handler_html: Event-Handler-Queue — von Consumern abgearbeitet (%{count} gesund)
976
+ metrics_unavailable_html: 'Metriken nicht verfügbar: Die Metrikabfrage ist fehlgeschlagen oder hat das Zeitlimit überschritten'
976
977
  no_consumers_html:
977
978
  one: Event-Handler-Queue ohne gesunden Consumer — %{count} Event wartet
978
979
  other: Event-Handler-Queue ohne gesunden Consumer — %{count} Events warten
@@ -973,6 +973,7 @@ en:
973
973
  other: Drained by a wildcard worker capsule (%{count} healthy workers)
974
974
  empty_html: Nothing waiting
975
975
  handler_html: Event handler queue — drained by consumers (%{count} healthy)
976
+ metrics_unavailable_html: 'Metrics unavailable: the metrics query failed or timed out'
976
977
  no_consumers_html:
977
978
  one: Event handler queue with no healthy consumer — %{count} event waits
978
979
  other: Event handler queue with no healthy consumer — %{count} events wait
@@ -973,6 +973,7 @@ es:
973
973
  other: La vacía una cápsula de workers comodín (%{count} workers sanos)
974
974
  empty_html: Nada en espera
975
975
  handler_html: Cola de manejador de eventos — la vacían los consumidores (%{count} sanos)
976
+ metrics_unavailable_html: 'Métricas no disponibles: la consulta de métricas falló o agotó el tiempo de espera'
976
977
  no_consumers_html:
977
978
  one: Cola de manejador de eventos sin consumidor sano — %{count} evento espera
978
979
  other: Cola de manejador de eventos sin consumidor sano — %{count} eventos esperan
@@ -973,6 +973,7 @@ fi:
973
973
  other: Tyhjentää jokerimerkki-worker-kapseli (%{count} tervettä workeria)
974
974
  empty_html: Mitään ei odota
975
975
  handler_html: Tapahtumankäsittelijän jono — consumerit tyhjentävät (%{count} tervettä)
976
+ metrics_unavailable_html: 'Mittarit eivät ole saatavilla: mittarikysely epäonnistui tai aikakatkaistiin'
976
977
  no_consumers_html:
977
978
  one: Tapahtumankäsittelijän jono ilman tervettä consumeria — %{count} tapahtuma odottaa
978
979
  other: Tapahtumankäsittelijän jono ilman tervettä consumeria — %{count} tapahtumaa odottaa
@@ -973,6 +973,7 @@ fr:
973
973
  other: Vidée par une capsule de workers joker (%{count} workers sains)
974
974
  empty_html: Rien en attente
975
975
  handler_html: File de gestionnaire d'événements — vidée par les consommateurs (%{count} sains)
976
+ metrics_unavailable_html: 'Métriques indisponibles : la requête des métriques a échoué ou a expiré'
976
977
  no_consumers_html:
977
978
  one: File de gestionnaire d'événements sans consommateur sain — %{count} événement attend
978
979
  other: File de gestionnaire d'événements sans consommateur sain — %{count} événements attendent
@@ -973,6 +973,7 @@ it:
973
973
  other: Svuotata da una capsula di worker jolly (%{count} worker sani)
974
974
  empty_html: Niente in attesa
975
975
  handler_html: Coda del gestore di eventi — svuotata dai consumer (%{count} sani)
976
+ metrics_unavailable_html: 'Metriche non disponibili: la query delle metriche non è riuscita o è scaduta'
976
977
  no_consumers_html:
977
978
  one: Coda del gestore di eventi senza consumer sano — %{count} evento in attesa
978
979
  other: Coda del gestore di eventi senza consumer sano — %{count} eventi in attesa
@@ -973,6 +973,7 @@ ja:
973
973
  other: ワイルドカードのワーカーカプセルが処理(正常なワーカー %{count})
974
974
  empty_html: 待機中のものはありません
975
975
  handler_html: イベントハンドラーキュー — コンシューマーが処理(正常 %{count})
976
+ metrics_unavailable_html: 'メトリクスを取得できません: メトリクスのクエリが失敗したかタイムアウトしました'
976
977
  no_consumers_html:
977
978
  one: 正常なコンシューマーがいないイベントハンドラーキュー — %{count} 件のイベントが待機中
978
979
  other: 正常なコンシューマーがいないイベントハンドラーキュー — %{count} 件のイベントが待機中
@@ -973,6 +973,7 @@ nb:
973
973
  other: Tømmes av en jokertegn-worker-kapsel (%{count} friske workers)
974
974
  empty_html: Ingenting venter
975
975
  handler_html: Hendelsesbehandler-kø — tømmes av consumers (%{count} friske)
976
+ metrics_unavailable_html: 'Metrikker utilgjengelige: metrikkspørringen feilet eller fikk tidsavbrudd'
976
977
  no_consumers_html:
977
978
  one: Hendelsesbehandler-kø uten frisk consumer — %{count} hendelse venter
978
979
  other: Hendelsesbehandler-kø uten frisk consumer — %{count} hendelser venter
@@ -973,6 +973,7 @@ nl:
973
973
  other: Leeggemaakt door een wildcard-workercapsule (%{count} gezonde workers)
974
974
  empty_html: Niets in de wacht
975
975
  handler_html: Event-handlerwachtrij — leeggemaakt door consumers (%{count} gezond)
976
+ metrics_unavailable_html: 'Metrieken niet beschikbaar: de metriekenquery is mislukt of verlopen'
976
977
  no_consumers_html:
977
978
  one: Event-handlerwachtrij zonder gezonde consumer — %{count} event wacht
978
979
  other: Event-handlerwachtrij zonder gezonde consumer — %{count} events wachten
@@ -973,6 +973,7 @@ pt:
973
973
  other: Esvaziada por uma cápsula de workers curinga (%{count} workers saudáveis)
974
974
  empty_html: Nada à espera
975
975
  handler_html: Fila de handler de eventos — esvaziada pelos consumidores (%{count} saudáveis)
976
+ metrics_unavailable_html: 'Métricas indisponíveis: a consulta de métricas falhou ou expirou'
976
977
  no_consumers_html:
977
978
  one: Fila de handler de eventos sem consumidor saudável — %{count} evento espera
978
979
  other: Fila de handler de eventos sem consumidor saudável — %{count} eventos esperam
@@ -973,6 +973,7 @@ sv:
973
973
  other: Töms av en jokertecken-worker-kapsel (%{count} friska workers)
974
974
  empty_html: Inget väntar
975
975
  handler_html: Händelsehanterarkö — töms av consumers (%{count} friska)
976
+ metrics_unavailable_html: 'Mätvärden otillgängliga: mätvärdesfrågan misslyckades eller överskred tidsgränsen'
976
977
  no_consumers_html:
977
978
  one: Händelsehanterarkö utan frisk consumer — %{count} händelse väntar
978
979
  other: Händelsehanterarkö utan frisk consumer — %{count} händelser väntar
@@ -301,7 +301,12 @@ module Pgbus
301
301
  attr_accessor :web_auth, :web_refresh_interval, :web_per_page, :web_live_updates, :web_data_source,
302
302
  :insights_default_minutes, :base_controller_class, :return_to_app_url, :web_actor,
303
303
  :metrics_enabled,
304
- :web_filter_parameters, :web_filter_sensitive
304
+ :web_filter_parameters, :web_filter_sensitive,
305
+ # Queues per statement when Web::DataSource#queues_with_metrics
306
+ # reads every queue's metrics (issue #677): each chunk is its
307
+ # own statement in its own savepoint, so a cancelled chunk
308
+ # marks only its queues unavailable.
309
+ :web_queue_metrics_chunk_size
305
310
 
306
311
  # The MCP server's workflow write tools (issue #550): pgbus_workflow_resume,
307
312
  # _cancel, _skip_step, _resume_at, _run_now and _signal. Off by default, and
@@ -504,6 +509,7 @@ module Pgbus
504
509
  @metrics_enabled = true
505
510
  @web_filter_parameters = nil # nil = auto-detect from Rails, then fall back to defaults
506
511
  @web_filter_sensitive = true
512
+ @web_queue_metrics_chunk_size = 100
507
513
  @deprecation_warned = Set.new
508
514
 
509
515
  # HTTP health endpoints — nil port disables the standalone server.
@@ -1123,6 +1129,10 @@ module Pgbus
1123
1129
  raise Pgbus::ConfigurationError, "insights_default_minutes must be a positive integer"
1124
1130
  end
1125
1131
 
1132
+ unless web_queue_metrics_chunk_size.is_a?(Integer) && web_queue_metrics_chunk_size.positive?
1133
+ raise Pgbus::ConfigurationError, "web_queue_metrics_chunk_size must be a positive integer"
1134
+ end
1135
+
1126
1136
  raise Pgbus::ConfigurationError, "require_primary must be true or false" unless [true, false].include?(require_primary)
1127
1137
  raise Pgbus::ConfigurationError, "mcp_write_tools must be true or false" unless [true, false].include?(mcp_write_tools)
1128
1138
 
@@ -92,8 +92,17 @@ module Pgbus
92
92
  end
93
93
 
94
94
  def track_queues
95
- data_source.queues_with_metrics.each do |q|
95
+ queues = data_source.queues_with_metrics
96
+ # The queue list itself could not be read: count it once, untagged,
97
+ # rather than emit nothing (an empty list reads as a quiet fleet).
98
+ counter("queue_metrics_unavailable", 1) if queues.empty? && queue_list_unavailable?
99
+
100
+ queues.each do |q|
96
101
  tags = { queue: q[:name] }
102
+ # Issue #677: an unmeasured queue reports no gauge (a missing
103
+ # point, not a false 0) and counts as unavailable instead.
104
+ next counter("queue_metrics_unavailable", 1, tags) if q[:metrics_unavailable]
105
+
97
106
  gauge "queue_depth", q[:queue_length], tags
98
107
  gauge "queue_visible_depth", q[:queue_visible_length], tags
99
108
  gauge "queue_paused", q[:paused] ? 1 : 0, tags
@@ -175,6 +184,16 @@ module Pgbus
175
184
  ::Appsignal.set_gauge("#{METRIC_PREFIX}#{key}", value, tags)
176
185
  end
177
186
 
187
+ def queue_list_unavailable?
188
+ return false unless data_source.respond_to?(:queue_metrics_errors)
189
+
190
+ data_source.queue_metrics_errors.any? { |e| Array(e[:queues]).empty? }
191
+ end
192
+
193
+ def counter(key, value, tags = {})
194
+ ::Appsignal.increment_counter("#{METRIC_PREFIX}#{key}", value, tags)
195
+ end
196
+
178
197
  def log_failure(label, error)
179
198
  Pgbus.logger.debug do
180
199
  "[Pgbus::AppSignal::Probe] #{label} failed: #{error.class}: #{error.message}"
@@ -13,8 +13,9 @@ module Pgbus
13
13
  # but its claim loop has stopped advancing (status :stalled),
14
14
  # OR backlog with live-but-idle workers and zero claim progress.
15
15
  # DEGRADED — something is wrong but not the wedge: stale processes, a
16
- # paused queue holding a backlog, growing DLQ, or MVCC horizon
17
- # pinned by a long-running transaction.
16
+ # paused queue holding a backlog, growing DLQ, MVCC horizon
17
+ # pinned by a long-running transaction, or queue metrics that
18
+ # could not be read (issue #677).
18
19
  # OK — draining normally / nothing actionable.
19
20
  class HealthAnalyzer
20
21
  # A worker is considered to have idle capacity unless its metadata
@@ -161,6 +162,9 @@ module Pgbus
161
162
 
162
163
  reasons << "queue health stats unavailable: #{health_error.class}: #{health_error.message}" if health_error
163
164
 
165
+ metrics_reason = metrics_unavailable_reason(queues)
166
+ reasons << metrics_reason if metrics_reason
167
+
164
168
  stale = processes.select { |p| p[:status].to_s == "stale" }
165
169
  reasons << "#{stale.size} process(es) stale (no recent heartbeat)" if stale.any?
166
170
 
@@ -189,6 +193,32 @@ module Pgbus
189
193
  reasons
190
194
  end
191
195
 
196
+ # Issue #677: a queue whose metrics could not be read (its chunk failed or
197
+ # hit statement_timeout) has nil counts, so it can never look backed up
198
+ # or wedged. Without this reason a fleet too large to measure would read
199
+ # OK. queue_metrics_errors also covers a pgmq.meta read that failed, when
200
+ # there are no rows to flag; a custom data source without it still
201
+ # reports its flagged rows.
202
+ def metrics_unavailable_reason(queues)
203
+ unavailable = queues.select { |q| q[:metrics_unavailable] }
204
+ errors = queue_metrics_errors
205
+ return if unavailable.empty? && errors.empty?
206
+
207
+ # First line only: a database error carries a multi-line SQL excerpt.
208
+ cause = errors.first && " (#{errors.first[:error].to_s.lines.first.to_s.strip})"
209
+ if unavailable.any?
210
+ "queue metrics unavailable for #{unavailable.size} queue(s)#{cause}: #{backlog_names(unavailable)}"
211
+ else
212
+ "queue metrics unavailable#{cause}: the queue list could not be read"
213
+ end
214
+ end
215
+
216
+ def queue_metrics_errors
217
+ return [] unless @data_source.respond_to?(:queue_metrics_errors)
218
+
219
+ Array(@data_source.queue_metrics_errors)
220
+ end
221
+
192
222
  # Issue #366: fingerprint-matched streams excluded from the wedge signal
193
223
  # still need the registry for orphan sweep bookkeeping and durable
194
224
  # wildcard safety. Surface a DEGRADED hint so operators run the backfill.
@@ -57,10 +57,13 @@ module Pgbus
57
57
  # refuse to guess if more than one matches. Returns an error response
58
58
  # on ambiguity, nil otherwise (so the caller proceeds with the normal
59
59
  # first-match lookup).
60
+ # Names only (issue #677): finding a message runs no metrics statement.
61
+ # A custom web_data_source without queue_names falls back to the names
62
+ # of its queues_with_metrics.
60
63
  def self.ambiguity_check(data_source, msg_id)
61
64
  dlq_suffix = Pgbus::DEAD_LETTER_SUFFIX
62
- dlqs = data_source.queues_with_metrics.select { |q| q[:name].to_s.end_with?(dlq_suffix) }
63
- matches = dlqs.map { |q| q[:name] }.select { |name| data_source.job_detail(name, msg_id) }
65
+ dlqs = queue_names_of(data_source).select { |name| name.to_s.end_with?(dlq_suffix) }
66
+ matches = dlqs.select { |name| data_source.job_detail(name, msg_id) }
64
67
  return nil if matches.size <= 1
65
68
 
66
69
  error_response(
@@ -68,6 +71,12 @@ module Pgbus
68
71
  "(#{matches.join(", ")}). Pass `queue:` to disambiguate."
69
72
  )
70
73
  end
74
+
75
+ def self.queue_names_of(data_source)
76
+ return data_source.queue_names if data_source.respond_to?(:queue_names)
77
+
78
+ data_source.queues_with_metrics.map { |q| q[:name] }
79
+ end
71
80
  end
72
81
  end
73
82
  end
@@ -20,6 +20,9 @@ module Pgbus
20
20
  with a future visibility timeout. A queue whose oldest_msg_age_sec
21
21
  keeps growing while oldest_claimable_age_sec stays nil has no
22
22
  starving backlog — nothing is waiting for a worker.
23
+ A queue with metrics_unavailable: true could not be measured (its
24
+ metrics query failed or timed out): its counts and ages are null,
25
+ meaning unknown, not zero; pgbus_health reports it as DEGRADED.
23
26
  DESC
24
27
 
25
28
  input_schema(properties: {}, required: [])
data/lib/pgbus/version.rb CHANGED
@@ -1,5 +1,5 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  module Pgbus
4
- VERSION = "0.27.0"
4
+ VERSION = "0.27.1"
5
5
  end
@@ -9,7 +9,7 @@ module Pgbus
9
9
  # The header path the error-class filter reads (DeadLetterHeader::KEY).
10
10
  ERROR_CLASS_SQL = "headers #>> '{#{Pgbus::DeadLetterHeader::KEY},error_class}'".freeze
11
11
 
12
- # DLQ queue names from queues_with_metrics are already fully qualified
12
+ # DLQ queue names from queue_names are already fully qualified
13
13
  # (e.g., "pgbus_default_dlq"), so we use them directly without re-prefixing.
14
14
  # dlq: one DLQ by its full name (unknown names list nothing);
15
15
  # error_class: only messages whose dead-letter block names that class.
@@ -25,27 +25,37 @@ module Pgbus
25
25
  []
26
26
  end
27
27
 
28
+ # From the cached metrics, unless a filter needs the rows or a DLQ's
29
+ # metrics are unavailable (issue #677): then a COUNT over the tables,
30
+ # so the pager does not read a DLQ that lists rows as empty (unless
31
+ # that COUNT fails too: then 0, logged as an error).
28
32
  def dlq_total_count(dlq: nil, error_class: nil)
29
33
  names = dlq_queue_names(dlq)
30
34
  return 0 if names.empty?
31
- return dlq_counts_by_queue.values_at(*names).sum unless error_class
32
-
33
- connection.select_value(
34
- "SELECT COUNT(*) FROM (#{queue_message_fragments(names, "#{ERROR_CLASS_SQL} = $1")}) AS combined",
35
- "Pgbus DLQ Count",
36
- [error_class]
37
- ).to_i
35
+ return dlq_counts_by_queue.values_at(*names).sum unless error_class || dlq_unmeasured?(names)
36
+
37
+ # In a savepoint: the DLQ whose metrics timed out may time out here
38
+ # too, and that must not abort a caller's transaction.
39
+ where = "#{ERROR_CLASS_SQL} = $1" if error_class
40
+ connection.transaction(requires_new: true) do
41
+ connection.select_value(
42
+ "SELECT COUNT(*) FROM (#{queue_message_fragments(names, where)}) AS combined",
43
+ "Pgbus DLQ Count",
44
+ error_class ? [error_class] : []
45
+ ).to_i
46
+ end
38
47
  rescue StandardError => e
39
- Pgbus.logger.debug { "[Pgbus::Web] Error fetching DLQ count: #{e.message}" }
48
+ Pgbus.logger.error { "[Pgbus::Web] Error fetching DLQ count: #{e.class}: #{e.message}" }
40
49
  0
41
50
  end
42
51
 
43
- # { "pgbus_default_dlq" => 12, ... } for the filter chips, from the
44
- # cached metrics (no query of its own).
52
+ # { "pgbus_default_dlq" => 12, ... } for the filter chips: every DLQ
53
+ # from queue_names, its count from the cached metrics (no query of its
54
+ # own). A DLQ whose metrics are unavailable (issue #677) stays listed
55
+ # and counts 0.
45
56
  def dlq_counts_by_queue
46
- queues_with_metrics
47
- .select { |q| q[:name].end_with?(Pgbus::DEAD_LETTER_SUFFIX) }
48
- .to_h { |q| [q[:name], q[:queue_length].to_i] }
57
+ lengths = queues_with_metrics.to_h { |q| [q[:name], q[:queue_length].to_i] }
58
+ dlq_names.to_h { |name| [name, lengths.fetch(name, 0)] }
49
59
  rescue StandardError => e
50
60
  Pgbus.logger.debug { "[Pgbus::Web] Error fetching DLQ counts: #{e.message}" }
51
61
  {}
@@ -140,11 +150,20 @@ module Pgbus
140
150
 
141
151
  private
142
152
 
143
- # Only names that are real DLQs ever reach sanitize_name and SQL.
153
+ # Only names that are real DLQs ever reach sanitize_name and SQL. Names
154
+ # only: listing or opening a DLQ message runs no metrics statement.
144
155
  def dlq_queue_names(dlq)
145
- names = dlq_counts_by_queue.keys
156
+ names = dlq_names
146
157
  dlq.nil? ? names : names & [dlq]
147
158
  end
159
+
160
+ def dlq_names
161
+ queue_names.select { |name| name.end_with?(Pgbus::DEAD_LETTER_SUFFIX) }
162
+ end
163
+
164
+ def dlq_unmeasured?(names)
165
+ queues_with_metrics.any? { |q| q[:metrics_unavailable] && names.include?(q[:name]) }
166
+ end
148
167
  end
149
168
  end
150
169
  end
@@ -136,7 +136,7 @@ module Pgbus
136
136
  # exist. Excluded queues stay known, so their failed rows never show up
137
137
  # as orphans of an unknown queue.
138
138
  def job_list_scope(queue_name, queues: nil, exclude: nil)
139
- names = queues_with_metrics.map { |q| q[:name] }.reject { |n| n.end_with?(Pgbus::DEAD_LETTER_SUFFIX) }
139
+ names = queue_names.reject { |n| n.end_with?(Pgbus::DEAD_LETTER_SUFFIX) }
140
140
  names &= queues if queues
141
141
  return nil if queues && names.empty?
142
142
 
@@ -112,7 +112,7 @@ module Pgbus
112
112
  end
113
113
 
114
114
  def priority_siblings(name, logical)
115
- queues_with_metrics.map { |q| q[:name] }.select do |n|
115
+ queue_names.select do |n|
116
116
  n != name && n.match?(Pgbus::Web::QueueSummary::PRIORITY_LEVEL) && logical_queue_name(n) == logical
117
117
  end
118
118
  end
@@ -28,6 +28,26 @@ module Pgbus
28
28
  # sweep picks up whatever is left on its next pass.
29
29
  PROMOTE_CAP = 100
30
30
 
31
+ # The metrics of a queue whose chunk could not be read (issue #677):
32
+ # unknown, so nil rather than zero, next to metrics_unavailable: true.
33
+ UNAVAILABLE_METRICS = %i[queue_length queue_visible_length parked_length oldest_msg_age_sec
34
+ oldest_claimable_age_sec newest_msg_age_sec total_messages max_read_ct
35
+ visible_unread_length oldest_unread_claimable_age_sec].to_h { |k| [k, nil] }.freeze
36
+
37
+ # Timed-out metrics chunks after which queues_with_metrics stops reading
38
+ # (issue #677): the rest of the queues come back unavailable, so a
39
+ # database where every chunk times out costs a few statement_timeouts,
40
+ # not one per chunk.
41
+ MAX_FAILED_METRICS_CHUNKS = 3
42
+
43
+ # The chunk failures that cost time (a cancelled statement, a lock wait,
44
+ # a deadlock, which PostgreSQL detects only after deadlock_timeout, a
45
+ # connection that cannot be established) and so count toward MAX_FAILED_METRICS_CHUNKS. A
46
+ # failure that returns at once (a queue whose table is gone) does not:
47
+ # it must never stop the healthy queues after it from being read.
48
+ SLOW_METRICS_FAILURES = [ActiveRecord::QueryCanceled, ActiveRecord::LockWaitTimeout, ActiveRecord::Deadlocked,
49
+ ActiveRecord::ConnectionNotEstablished].freeze
50
+
31
51
  def initialize(client: Pgbus.client)
32
52
  @client = client
33
53
  @last_throughput_snapshot = nil
@@ -37,10 +57,12 @@ module Pgbus
37
57
  # Dashboard summary
38
58
  def summary_stats
39
59
  queues = queues_with_metrics
40
- total_depth = queues.sum { |q| q[:queue_length] }
41
- total_visible = queues.sum { |q| q[:queue_visible_length] }
60
+ # A queue with metrics_unavailable (issue #677) has nil counts: it adds
61
+ # nothing it cannot vouch for.
62
+ total_depth = queues.sum { |q| q[:queue_length].to_i }
63
+ total_visible = queues.sum { |q| q[:queue_visible_length].to_i }
42
64
  dlq_suffix = Pgbus::DEAD_LETTER_SUFFIX
43
- dlq_depth = queues.select { |q| q[:name].end_with?(dlq_suffix) }.sum { |q| q[:queue_length] }
65
+ dlq_depth = queues.select { |q| q[:name].end_with?(dlq_suffix) }.sum { |q| q[:queue_length].to_i }
44
66
 
45
67
  throughput = compute_throughput(queues)
46
68
 
@@ -67,7 +89,9 @@ module Pgbus
67
89
  # responses would freeze at the first read. Both call this per iteration
68
90
  # / per tool call.
69
91
  def reset_cache!
92
+ @queue_names = nil
70
93
  @queues_with_metrics = nil
94
+ @queue_metrics_errors = nil
71
95
  @concurrency_summary = nil
72
96
  @queue_health_stats = nil
73
97
  @process_rows = nil
@@ -75,6 +99,19 @@ module Pgbus
75
99
  self
76
100
  end
77
101
 
102
+ # Every queue's physical name, from pgmq.meta alone: no queue table is
103
+ # read (issue #677). Callers that need only names (the job list scope,
104
+ # the DLQ selection, priority siblings) use this instead of paying for
105
+ # queues_with_metrics. Memoized like it, cleared by reset_cache!, and
106
+ # shared with it, so a page that wants both reads pgmq.meta once. A
107
+ # failed read is logged and answers [] without memoizing.
108
+ def queue_names
109
+ @queue_names ||= read_queue_names
110
+ rescue StandardError => e
111
+ Pgbus.logger.error { "[Pgbus::Web] Error reading queue names: #{e.class}: #{e.message}" }
112
+ []
113
+ end
114
+
78
115
  # Queues — query via ActiveRecord for reliability in web processes
79
116
  # (avoids PGMQ client connection issues when the web server uses a
80
117
  # different connection lifecycle than the worker processes).
@@ -84,24 +121,57 @@ module Pgbus
84
121
  # message rows and the total count); without the memo each call re-runs
85
122
  # the meta query + batched metrics. Mutations redirect to a fresh request
86
123
  # with a new instance, so a per-request memo never serves stale data.
124
+ #
125
+ # A queue whose metrics could not be read (its chunk failed or hit
126
+ # statement_timeout, issue #677) stays in the list with
127
+ # metrics_unavailable: true and nil counts; queue_metrics_errors says why.
87
128
  def queues_with_metrics
88
129
  @queues_with_metrics ||= fetch_queues_with_metrics
89
130
  end
90
131
 
132
+ # Why the last queues_with_metrics read is incomplete: [] when every
133
+ # queue reported, else one { queues: [names], error: "Class: message" }
134
+ # per failed statement (queues: [] when pgmq.meta itself could not be
135
+ # read). Runs that read when none has run yet. An empty
136
+ # queues_with_metrics with no errors is an empty system; with errors it
137
+ # is an unknown one.
138
+ def queue_metrics_errors
139
+ queues_with_metrics
140
+ @queue_metrics_errors || []
141
+ end
142
+
143
+ # queues: [] in queue_metrics_errors means pgmq.meta itself could not be
144
+ # read; a failure after it answered keeps the names, flagged.
91
145
  def fetch_queues_with_metrics
92
- queue_names = connection.select_values("SELECT queue_name FROM pgmq.meta ORDER BY queue_name")
93
- return [] if queue_names.empty?
146
+ @queue_metrics_errors = []
147
+ names = @queue_names ||= read_queue_names
148
+ return [] if names.empty?
94
149
 
95
150
  paused_queues = paused_queue_names.to_set
96
- batched_queue_metrics(queue_names).map do |q|
151
+ batched_queue_metrics(names).map do |q|
97
152
  q.merge(paused: paused_queues.include?(logical_queue_name(q[:name])))
98
153
  end
99
154
  rescue StandardError => e
100
155
  Pgbus.logger.error { "[Pgbus::Web] Error fetching queue metrics: #{e.class}: #{e.message}" }
156
+ return unavailable_queue_metrics(names, "#{e.class}: #{e.message}") if names
157
+
158
+ @queue_metrics_errors << { queues: [], error: "#{e.class}: #{e.message}" }
101
159
  []
102
160
  end
103
161
  private :fetch_queues_with_metrics
104
162
 
163
+ # A name sanitize_name rejects never reaches a caller: before #677 the
164
+ # metrics read dropped it, and every caller took its names from there.
165
+ def read_queue_names
166
+ connection.select_values("SELECT queue_name FROM pgmq.meta ORDER BY queue_name").select do |name|
167
+ sanitize_name(name)
168
+ rescue ArgumentError => e
169
+ Pgbus.logger.debug { "[Pgbus::Web] Skipping queue #{name.inspect}: #{e.message}" }
170
+ false
171
+ end
172
+ end
173
+ private :read_queue_names
174
+
105
175
  # Physical queue names known to back streams: the pgbus_stream_queues
106
176
  # registry plus any dormant pre-registry queues discovered by the
107
177
  # archive msg_id index fingerprint (issue #366). Stream delivery is a
@@ -248,20 +318,20 @@ module Pgbus
248
318
  def discard_all_enqueued
249
319
  dlq_suffix = Pgbus::DEAD_LETTER_SUFFIX
250
320
  handlers = handler_queue_physical_names
251
- queues = queues_with_metrics.reject { |q| q[:name].end_with?(dlq_suffix) || handlers.include?(q[:name]) }
321
+ names = queue_names.reject { |n| n.end_with?(dlq_suffix) || handlers.include?(n) }
252
322
  total = 0
253
323
 
254
- queues.each do |q|
255
- messages = query_queue_messages_raw(q[:name], 10_000, 0)
324
+ names.each do |name|
325
+ messages = query_queue_messages_raw(name, 10_000, 0)
256
326
  next if messages.empty?
257
327
 
258
328
  release_locks_for_messages(messages)
259
329
 
260
330
  ids = messages.map { |m| m[:msg_id].to_i }
261
- @client.archive_batch(q[:name], ids, prefixed: false)
331
+ @client.archive_batch(name, ids, prefixed: false)
262
332
  total += ids.size
263
333
  rescue StandardError => e
264
- Pgbus.logger.debug { "[Pgbus::Web] Error discarding enqueued messages from #{q[:name]}: #{e.message}" }
334
+ Pgbus.logger.debug { "[Pgbus::Web] Error discarding enqueued messages from #{name}: #{e.message}" }
265
335
  end
266
336
 
267
337
  total
@@ -1280,8 +1350,7 @@ module Pgbus
1280
1350
 
1281
1351
  def all_queue_messages(limit, offset)
1282
1352
  dlq_suffix = Pgbus::DEAD_LETTER_SUFFIX
1283
- queues = queues_with_metrics.reject { |q| q[:name].end_with?(dlq_suffix) }
1284
- paginated_queue_messages(queues.map { |q| q[:name] }, limit, offset)
1353
+ paginated_queue_messages(queue_names.reject { |n| n.end_with?(dlq_suffix) }, limit, offset)
1285
1354
  end
1286
1355
 
1287
1356
  # Returns messages from multiple PGMQ queues in a single paginated
@@ -1324,53 +1393,98 @@ module Pgbus
1324
1393
  end.join("\nUNION ALL\n")
1325
1394
  end
1326
1395
 
1396
+ # Every queue's metrics, web_queue_metrics_chunk_size queues per
1397
+ # statement (issue #677). Each queue's table is scanned once, by one
1398
+ # aggregate, so a statement's cost grows with its chunk, not with the
1399
+ # whole fleet. Each chunk runs in its own savepoint: a chunk that fails
1400
+ # (statement_timeout included) rolls back to it, so a caller's open
1401
+ # transaction stays usable, and only that chunk's queues come back as
1402
+ # unavailable. Rows follow +queue_names+ order.
1403
+ #
1404
+ # Chunks run one after another, so a database where every chunk times
1405
+ # out would cost chunks × statement_timeout. After
1406
+ # MAX_FAILED_METRICS_CHUNKS failed chunks the rest are not run: their
1407
+ # queues come back unavailable at once.
1327
1408
  def batched_queue_metrics(queue_names)
1328
- return [] if queue_names.empty?
1409
+ chunk_size = Pgbus.configuration.web_queue_metrics_chunk_size
1410
+ rows = []
1411
+ failed = 0
1412
+ queue_names.each_slice(chunk_size).with_index do |chunk, index|
1413
+ if failed >= MAX_FAILED_METRICS_CHUNKS
1414
+ skipped = queue_names.drop(index * chunk_size)
1415
+ rows.concat(unavailable_queue_metrics(skipped, "not read: #{failed} metrics chunks timed out before them"))
1416
+ break
1417
+ end
1329
1418
 
1330
- unions = queue_names.filter_map do |name|
1331
- sanitized = sanitize_name(name)
1332
- qtable = "q_#{sanitized}"
1333
- seq_name = "#{qtable}_msg_id_seq"
1334
- <<~SQL
1335
- SELECT
1336
- #{connection.quote(name)} AS queue_name,
1337
- (SELECT count(*) FROM pgmq.#{qtable}) AS queue_length,
1338
- (SELECT count(*) FROM pgmq.#{qtable} WHERE vt <= NOW()) AS queue_visible_length,
1339
- (SELECT EXTRACT(epoch FROM (NOW() - max(enqueued_at)))::int FROM pgmq.#{qtable}) AS newest_msg_age_sec,
1340
- (SELECT EXTRACT(epoch FROM (NOW() - min(enqueued_at)))::int FROM pgmq.#{qtable}) AS oldest_msg_age_sec,
1341
- (SELECT CASE WHEN is_called THEN last_value ELSE 0 END FROM pgmq.#{seq_name}) AS total_messages,
1342
- (SELECT max(read_ct) FROM pgmq.#{qtable}) AS max_read_ct,
1343
- (SELECT count(*) FROM pgmq.#{qtable} WHERE vt <= NOW() AND read_ct = 0) AS visible_unread_length,
1344
- (SELECT EXTRACT(epoch FROM (NOW() - min(vt)))::int FROM pgmq.#{qtable} WHERE vt <= NOW()) AS oldest_claimable_age_sec,
1345
- (SELECT EXTRACT(epoch FROM (NOW() - min(vt)))::int FROM pgmq.#{qtable} WHERE vt <= NOW() AND read_ct = 0) AS oldest_unread_claimable_age_sec
1346
- SQL
1347
- rescue StandardError => e
1348
- Pgbus.logger.debug { "[Pgbus::Web] Skipping queue metrics for #{name}: #{e.message}" }
1349
- nil
1419
+ chunk_rows, error = chunk_queue_metrics(chunk)
1420
+ failed += 1 if SLOW_METRICS_FAILURES.any? { |klass| error.is_a?(klass) }
1421
+ rows.concat(chunk_rows)
1422
+ end
1423
+ rows
1424
+ end
1425
+
1426
+ # [rows, nil], or [unavailable rows, the error] for a failed chunk.
1427
+ # +chunk+ holds names read_queue_names already validated.
1428
+ def chunk_queue_metrics(chunk)
1429
+ sql = chunk.map { |name| "(#{queue_metrics_select(name)})" }.join("\nUNION ALL\n")
1430
+ rows = connection.transaction(requires_new: true) do
1431
+ connection.select_all(sql, "Pgbus Batched Queue Metrics").to_a
1350
1432
  end
1433
+ by_name = rows.to_h { |row| [row["queue_name"], row] }
1434
+ [chunk.filter_map { |name| by_name[name] && queue_metrics_hash(name, by_name[name]) }, nil]
1435
+ rescue StandardError => e
1436
+ [unavailable_queue_metrics(chunk, "#{e.class}: #{e.message}"), e]
1437
+ end
1438
+
1439
+ # One row for one queue: a single aggregate over its table, plus the
1440
+ # msg_id sequence for the lifetime total. The same expressions as
1441
+ # queue_metrics_via_sql, so a queue reads the same on both paths.
1442
+ def queue_metrics_select(name)
1443
+ qtable = "q_#{sanitize_name(name)}"
1444
+ <<~SQL.strip
1445
+ SELECT #{connection.quote(name)} AS queue_name,
1446
+ m.queue_length, m.queue_visible_length, m.newest_msg_age_sec, m.oldest_msg_age_sec,
1447
+ (SELECT CASE WHEN is_called THEN last_value ELSE 0 END FROM pgmq.#{qtable}_msg_id_seq) AS total_messages,
1448
+ m.max_read_ct, m.visible_unread_length, m.oldest_claimable_age_sec, m.oldest_unread_claimable_age_sec
1449
+ FROM (
1450
+ SELECT count(*) AS queue_length,
1451
+ count(*) FILTER (WHERE vt <= NOW()) AS queue_visible_length,
1452
+ EXTRACT(epoch FROM (NOW() - max(enqueued_at)))::int AS newest_msg_age_sec,
1453
+ EXTRACT(epoch FROM (NOW() - min(enqueued_at)))::int AS oldest_msg_age_sec,
1454
+ max(read_ct) AS max_read_ct,
1455
+ count(*) FILTER (WHERE vt <= NOW() AND read_ct = 0) AS visible_unread_length,
1456
+ EXTRACT(epoch FROM (NOW() - min(vt) FILTER (WHERE vt <= NOW())))::int AS oldest_claimable_age_sec,
1457
+ EXTRACT(epoch FROM (NOW() - min(vt) FILTER (WHERE vt <= NOW() AND read_ct = 0)))::int AS oldest_unread_claimable_age_sec
1458
+ FROM pgmq.#{qtable}
1459
+ ) AS m
1460
+ SQL
1461
+ end
1351
1462
 
1352
- return [] if unions.empty?
1463
+ def queue_metrics_hash(name, row)
1464
+ {
1465
+ name: name,
1466
+ queue_length: row["queue_length"].to_i,
1467
+ queue_visible_length: row["queue_visible_length"].to_i,
1468
+ parked_length: row["queue_length"].to_i - row["queue_visible_length"].to_i,
1469
+ oldest_msg_age_sec: row["oldest_msg_age_sec"]&.to_i,
1470
+ oldest_claimable_age_sec: row["oldest_claimable_age_sec"]&.to_i,
1471
+ newest_msg_age_sec: row["newest_msg_age_sec"]&.to_i,
1472
+ total_messages: row["total_messages"].to_i,
1473
+ max_read_ct: row["max_read_ct"]&.to_i,
1474
+ visible_unread_length: row["visible_unread_length"].to_i,
1475
+ oldest_unread_claimable_age_sec: row["oldest_unread_claimable_age_sec"]&.to_i
1476
+ }
1477
+ end
1353
1478
 
1354
- sql = unions.join(" UNION ALL ")
1355
- rows = connection.select_all(sql, "Pgbus Batched Queue Metrics")
1356
- rows.to_a.map do |row|
1357
- {
1358
- name: row["queue_name"],
1359
- queue_length: row["queue_length"].to_i,
1360
- queue_visible_length: row["queue_visible_length"].to_i,
1361
- parked_length: row["queue_length"].to_i - row["queue_visible_length"].to_i,
1362
- oldest_msg_age_sec: row["oldest_msg_age_sec"]&.to_i,
1363
- oldest_claimable_age_sec: row["oldest_claimable_age_sec"]&.to_i,
1364
- newest_msg_age_sec: row["newest_msg_age_sec"]&.to_i,
1365
- total_messages: row["total_messages"].to_i,
1366
- max_read_ct: row["max_read_ct"]&.to_i,
1367
- visible_unread_length: row["visible_unread_length"].to_i,
1368
- oldest_unread_claimable_age_sec: row["oldest_unread_claimable_age_sec"]&.to_i
1369
- }
1479
+ # A failed chunk keeps its queues, flagged, with every metric nil: a
1480
+ # count we could not read is unknown, not zero.
1481
+ def unavailable_queue_metrics(names, error)
1482
+ Pgbus.logger.error do
1483
+ "[Pgbus::Web] Queue metrics unavailable for #{names.size} queue(s) (#{names.first(3).join(", ")}" \
1484
+ "#{", …" if names.size > 3}): #{error}"
1370
1485
  end
1371
- rescue StandardError => e
1372
- Pgbus.logger.error { "[Pgbus::Web] Error fetching batched queue metrics: #{e.class}: #{e.message}" }
1373
- []
1486
+ (@queue_metrics_errors ||= []) << { queues: names, error: error }
1487
+ names.map { |name| { name: name, **UNAVAILABLE_METRICS, metrics_unavailable: true } }
1374
1488
  end
1375
1489
 
1376
1490
  def queue_metrics_via_sql(queue_name)
@@ -1509,14 +1623,20 @@ module Pgbus
1509
1623
  QueueNameValidator.sanitize!(name)
1510
1624
  end
1511
1625
 
1626
+ # A read with a queue's metrics unavailable (issue #677) leaves the
1627
+ # snapshot alone: its lifetime total is unknown, and counting it as zero
1628
+ # would read as a drop now and a spike once it reports again.
1512
1629
  def compute_throughput(queues)
1630
+ return 0.0 if queues.any? { |q| q[:metrics_unavailable] }
1631
+
1513
1632
  current_totals = queues.sum { |q| q[:total_messages] }
1514
- now = Process.clock_gettime(Process::CLOCK_MONOTONIC)
1633
+ now = ::Process.clock_gettime(::Process::CLOCK_MONOTONIC)
1515
1634
 
1516
1635
  if @last_throughput_snapshot && @last_throughput_at
1517
1636
  elapsed = now - @last_throughput_at
1518
1637
  delta = current_totals - @last_throughput_snapshot
1519
- rate = elapsed.positive? ? (delta / elapsed).round(1) : 0.0
1638
+ # A dropped queue lowers the lifetime total: never a negative rate.
1639
+ rate = elapsed.positive? ? [(delta / elapsed).round(1), 0.0].max : 0.0
1520
1640
  end
1521
1641
 
1522
1642
  @last_throughput_snapshot = current_totals
@@ -36,8 +36,13 @@ module Pgbus
36
36
  private
37
37
 
38
38
  def append_queue_metrics(lines)
39
- queues = @data_source.queues_with_metrics
40
- return if queues.empty?
39
+ all = @data_source.queues_with_metrics
40
+ return append_queue_list_unavailable(lines) if all.empty?
41
+
42
+ # A queue whose metrics could not be read (issue #677) has nil counts:
43
+ # it gets no count sample (format_value would write a false 0), only
44
+ # its paused state and pgbus_queue_metrics_unavailable.
45
+ queues = all.reject { |q| q[:metrics_unavailable] }
41
46
 
42
47
  gauge(lines, "pgbus_queue_depth", "Number of messages in the queue (including invisible)") do
43
48
  queues.map { |q| [q[:queue_length], { queue: q[:name] }] }
@@ -69,12 +74,32 @@ module Pgbus
69
74
  end
70
75
 
71
76
  gauge(lines, "pgbus_queue_paused", "Whether the queue is paused (1) or active (0)") do
72
- queues.map { |q| [q[:paused] ? 1 : 0, { queue: q[:name] }] }
77
+ all.map { |q| [q[:paused] ? 1 : 0, { queue: q[:name] }] }
78
+ end
79
+
80
+ gauge(lines, "pgbus_queue_metrics_unavailable",
81
+ "Whether the queue's metrics could not be read this scrape (1) or were read (0)") do
82
+ all.map { |q| [q[:metrics_unavailable] ? 1 : 0, { queue: q[:name] }] }
73
83
  end
74
84
  rescue StandardError => e
75
85
  Pgbus.logger.debug { "[Pgbus::Metrics] Error serializing queue metrics: #{e.message}" }
76
86
  end
77
87
 
88
+ # No queue rows because pgmq.meta could not be read (issue #677): one
89
+ # unlabelled sample, so the scrape says so instead of going quiet.
90
+ def append_queue_list_unavailable(lines)
91
+ return unless queue_list_unavailable?
92
+
93
+ gauge(lines, "pgbus_queue_metrics_unavailable",
94
+ "Whether the queue's metrics could not be read this scrape (1) or were read (0)") { [[1]] }
95
+ end
96
+
97
+ def queue_list_unavailable?
98
+ return false unless @data_source.respond_to?(:queue_metrics_errors)
99
+
100
+ @data_source.queue_metrics_errors.any? { |e| Array(e[:queues]).empty? }
101
+ end
102
+
78
103
  def append_job_metrics(lines)
79
104
  counts = @data_source.job_status_counts
80
105
  unless counts.empty?
@@ -168,8 +193,12 @@ module Pgbus
168
193
  [[stats[:failed_count]]]
169
194
  end
170
195
 
171
- gauge(lines, "pgbus_dlq_depth", "Total messages across all dead letter queues") do
172
- [[stats[:dlq_depth]]]
196
+ # An unreadable queue list (issue #677) makes the DLQ total unknown,
197
+ # not 0: no sample, so a dlq_depth alert does not clear on it.
198
+ unless queue_list_unavailable?
199
+ gauge(lines, "pgbus_dlq_depth", "Total messages across all dead letter queues") do
200
+ [[stats[:dlq_depth]]]
201
+ end
173
202
  end
174
203
  rescue StandardError => e
175
204
  Pgbus.logger.debug { "[Pgbus::Metrics] Error serializing summary metrics: #{e.message}" }
@@ -19,13 +19,18 @@ module Pgbus
19
19
 
20
20
  module_function
21
21
 
22
+ # A queue whose metrics could not be read (issue #677) says so right
23
+ # after any pause, ahead of the drain line: "no worker — N claimable"
24
+ # would quote an unknown count as 0.
22
25
  def present(detail, pause_state, drainers, max_retries:)
23
26
  name = detail[:name].to_s
24
27
  dlq = name.end_with?(Pgbus::DEAD_LETTER_SUFFIX)
28
+ unavailable = line("metrics_unavailable", :yellow) if detail[:metrics_unavailable]
25
29
  [
26
30
  pause_line(pause_state),
31
+ unavailable,
27
32
  drain_line(dlq, detail, drainers, max_retries),
28
- (backlog_line(detail) unless dlq || drainers[:stream]),
33
+ (backlog_line(detail) unless unavailable || dlq || drainers[:stream]),
29
34
  priority_line(drainers)
30
35
  ].compact
31
36
  end
@@ -327,6 +327,8 @@ module Pgbus
327
327
  def web_live_updates=: (untyped) -> untyped
328
328
  def web_per_page: () -> Integer
329
329
  def web_per_page=: (untyped) -> untyped
330
+ def web_queue_metrics_chunk_size: () -> Integer
331
+ def web_queue_metrics_chunk_size=: (untyped) -> untyped
330
332
  def web_refresh_interval: () -> Integer
331
333
  def web_refresh_interval=: (untyped) -> untyped
332
334
  def worker_notify_connection_options: () -> untyped
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: pgbus
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.27.0
4
+ version: 0.27.1
5
5
  platform: ruby
6
6
  authors:
7
7
  - Mikael Henriksson