pgbus 0.12.0 → 0.12.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +7 -0
- data/lib/generators/pgbus/templates/upgrade_pgmq.rb.erb +7 -0
- data/lib/pgbus/client.rb +9 -7
- data/lib/pgbus/concurrency.rb +14 -11
- data/lib/pgbus/configuration.rb +44 -0
- data/lib/pgbus/mcp/health_analyzer.rb +11 -5
- data/lib/pgbus/pgmq_schema.rb +40 -0
- data/lib/pgbus/recurring/schedule.rb +32 -11
- data/lib/pgbus/support.rb +22 -0
- data/lib/pgbus/uniqueness.rb +18 -7
- data/lib/pgbus/version.rb +1 -1
- data/lib/pgbus/web/data_source.rb +13 -0
- metadata +2 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: a966ed340b2cd097f82b926f90647a0d481df35da76bb4ef06f4e3c13e3d5692
|
|
4
|
+
data.tar.gz: 779124d08216918b02a162f2a1263c4115cd7a7fb9404dc0055dc940da28793f
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: b25519a1d9ea9e1e2700b959a5ebcf03adf8efbcb130001f1e962067e0def2546b1644e199d24d2b849b29ba67f15bc83eef5f41506ffde3d6ea5d79da752af7
|
|
7
|
+
data.tar.gz: f0ec03ccc2b075b90f6b87cc5eb1315d5d3834b2cdd7c9c19fc851f350196f6860b941ec0566aac0315f531bf29356f9a1f94b68e852aa497eb642833ddc3800
|
data/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,11 @@
|
|
|
1
1
|
## [Unreleased]
|
|
2
2
|
|
|
3
|
+
### Fixed
|
|
4
|
+
|
|
5
|
+
- **`streams_pool_database_url` / `streams_pool_host` / `streams_pool_port` — route the streams PGMQ pool independently of the streamer's LISTEN connection (issue #358).** 0.12.0 started building the dedicated streams pool from `streams_connection_options` (correct for a separate streams database), but that method is also how pooler-bypass installs pin the LISTEN connection to the **direct** Postgres port (`streams_port = 5432`, the documented "workers go through PgBouncer, streamer goes direct" pattern) — so upgrading silently moved up to `streams_pool_size` connections **per process** onto the direct port, whose `max_connections` ceiling on managed Postgres is typically low. A modest fleet exhausts it (`FATAL: remaining connection slots are reserved for roles with the SUPERUSER attribute`): `StreamApp` then 500s every SSE connect and durable broadcasts fail at publish, intermittently, because the pools are lazy. Only LISTEN actually needs the direct port (it dies at transaction-pool COMMIT boundaries); the pool's broadcast INSERTs and replay reads are plain pooler-safe SQL. The new `streams_pool_*` triple routes the POOL independently — applied to the base options exactly like the `streams_*` and `worker_notify_*` groups: set `streams_pool_port` back to the pooled port and only the LISTEN pins remain direct. Default `nil` = follows `streams_connection_options`, byte-identical to 0.12.0, so separate-streams-DB installs are unchanged. Refs #358.
|
|
6
|
+
- **The health verdict no longer reads durable stream queues as a wedged fleet (issue #359).** The Process-liveness signal (`pgbus doctor`, `pgbus_health`, the MCP health tool) treats *visible messages with `read_ct=0` while workers are alive* as the silent-worker-wedge signature — but durable stream delivery is a non-consuming peek, so **every** stream queue matches it permanently by design. On a streams-heavy install the verdict screamed STALLED listing hundreds of healthy stream queues, burying real wedges in noise. `HealthAnalyzer` now excludes queues registered in the `pgbus_stream_queues` registry from the operational set (exactly like DLQs) — out of the STALLED/DEGRADED reasons and the backlog totals — via a new `Web::DataSource#stream_queue_names` (loaded fresh per verdict; degrades to an empty set on pre-registry installs). Same bug class as #308/#309, same registry cure. Refs #359.
|
|
7
|
+
- **The `upgrade_pgmq` migration no longer silently kills NOTIFY-gated wakeups fleet-wide (issue #360).** The generated migration drops every pgmq function **with CASCADE** — which also drops the `trigger_notify_queue_insert_listeners` trigger from every existing queue table (it depends on the dropped `pgmq.notify_queue_listeners()`), and `install_sql` re-created the functions but never the per-queue triggers. Result: after every PGMQ schema upgrade, NOTIFY wakeups (#174) silently died and workers fell back to `polling_interval` polling — nothing errored; latency degraded and DB poll load rose. Trigger re-install only happened in `ensure_single_queue` (per-process memoized), so on the common deploy ordering where the job supervisor boots *before* migrations run, the supervisor installed triggers at boot and the migration wiped them minutes later — with nothing left to re-install them until a full process restart. New `PgmqSchema.reinstall_notify_triggers_sql` replays `pgmq.enable_notify_insert` for every row in `pgmq.notify_insert_throttle` (a table, preserved by the upgrade, recording exactly which queues had notify and at what throttle; the function is idempotent), and the migration template runs it after `install_sql` — restoring every trigger at its original interval. No-op on vendored versions without the notify feature. **Already ran an upgrade migration?** Run `Pgbus::PgmqSchema.reinstall_notify_triggers_sql` once (e.g. from a console: `ActiveRecord::Base.connection.execute(...)`) or restart the fleet; `pgbus doctor`'s LISTEN/NOTIFY check confirms the repair. Refs #360.
|
|
8
|
+
|
|
3
9
|
### Breaking Changes
|
|
4
10
|
|
|
5
11
|
- **`config.validate!` now rejects malformed values for 12 core config keys at boot instead of failing deep in a worker or silently misbehaving (issue #335).** Before the 1.0 surface freeze, these headline job-path keys had **zero** validation while streams keys had 14 checks — so a bad value surfaced as a crash inside a worker/dispatcher/poller/scheduler thread, per-enqueue, or by silently corrupting every derived queue name / leaving the dashboard open. `Pgbus.configure` (eager-validating by default) now raises `Pgbus::ConfigurationError` naming the key for: the worker-recycling trio `max_jobs_per_worker`/`max_memory_mb`/`max_worker_lifetime` (positive number or `nil` to disable); the intervals `dispatch_interval`/`outbox_poll_interval`/`recurring_schedule_interval` (positive number); `outbox_batch_size` (positive integer); `default_priority` (non-negative integer — `0` is a valid level); `queue_prefix`/`default_queue` (non-empty String — an empty prefix silently produced malformed queue names); `web_auth` (a callable or `nil` — a non-callable silently left the dashboard open); `error_reporters` (an Array); and `connects_to` (a Hash or `nil` — previously a malformed value raised a raw `TypeError` deep in engine boot). This only rejects configuration that was already broken; if a `Pgbus.configure` block set one of these to an invalid value it will now fail loudly at boot (the intended pre-freeze tightening). Set `config.eager_validation = false` to defer. Refs #335, #282.
|
|
@@ -38,6 +44,7 @@
|
|
|
38
44
|
|
|
39
45
|
### Fixed
|
|
40
46
|
|
|
47
|
+
- **`ensures_uniqueness` / `limits_concurrency` declared on a base class now reach every subclass — previously the declaration was silently inert for them (issue #357).** Both macros stored their config in a class-level ivar (`@pgbus_uniqueness` / `@pgbus_concurrency`) and every lookup read `active_job.class.pgbus_uniqueness` without walking ancestors — class-level ivars are not inherited in Ruby, so `class RecurringBase < ApplicationJob; ensures_uniqueness strategy: :until_executed; end` gave subclasses NO uniqueness key, no scheduler overlap protection, and no #333 guard, with no warning that the natural reading of the code was false. The readers now fall back to the nearest ancestor's declaration (a class's own declaration still beats an inherited one). Coupled fix, required to make inheritance safe rather than harmful: no default key proc is stored anymore — the old `key || ->(*) { name }` captured the DECLARING class, so an inheritance fix alone would have collapsed every subclass into ONE shared key (the base class's name), and with `on_conflict: :discard` sibling jobs would silently discard each other's enqueues — strictly worse than inert. The class-name default is now resolved from the ENQUEUED job's class at all three call sites (`Uniqueness.resolve_key`, `Concurrency.resolve_key`, and the recurring scheduler's `resolve_uniqueness_key`), and `config[:key]` is `nil` when no explicit `key:` was given. Consumer note: concrete-class declarations resolve to exactly the same keys as before, but a base-class declaration that was silently inert now ACTIVATES per-subclass uniqueness/concurrency on upgrade — audit any base-class declarations (the macro finally does what it says). In the scheduler path the no-key default is additionally qualified with the recurring task's `args:` (`SyncJob:["site_a"]`) so two tasks pointing at the same job class with different arguments don't share one lock — the scheduler's analogue of the #333 raise-at-enqueue guard, which a scheduler loop can't use — and a failure to resolve ANY uniqueness key — default or user-supplied key proc — now fails closed (skips the tick, logs, and reaches `error_reporters`) instead of degrading to nil and enqueuing WITHOUT a lock, which silently disabled the very protection the proc configured. Both macros also now reject a non-nil, non-callable `key:` (e.g. `key: false`) at definition time with the documented ArgumentError — previously `false` slipped the truthiness guard and surfaced as a `NoMethodError` at enqueue (uniqueness) or silently acted as the class-name default (concurrency). Measured (benchmark-ips, enqueue-path budget per docs/performance.md): the shared `Support.call_key_proc` dispatch is ~17% faster than the old inline dispatch (136ns vs 160ns/op), the class-name default is 3.7× faster than the old stored-proc call (23ns vs 86ns/op), and the inherited-config ancestor walk adds ~15ns/op. Refs #357, #333.
|
|
41
48
|
- **The Ruby-Timeout read fallback no longer leaves a wedged `CONNECTION_OK` connection in the pgmq pool to re-hang the next read (issue #354).** On the one path where libpq can't bound a hung socket (dedicated connection on non-Linux hosts or libpq < 12), the last-resort `Timeout.timeout` interrupts the read via `Thread#raise` — and libpq may leave the pooled `PG::Connection` reporting `CONNECTION_OK` while it will in fact re-hang on reuse, invisible to pgmq-ruby's checkout health check (it isn't `CONNECTION_BAD`). This was a documented KNOWN LIMITATION waiting on a public pool-reload upstream (cf. mensfeld/pgmq-ruby#94); pgmq-ruby 0.7.1 shipped it, so the fallback now raises an internal `ReadTimeoutError` subclass and reloads the job pool before re-raising — the poisoned connection (already checked back in by `connection_pool`'s ensure) is dropped and the pool rebuilds lazily. A clean server-side `statement_timeout` cancel still never triggers a reload (the connection is healthy), and a reload failure is logged without masking the timeout. Requires pgmq-ruby >= 0.7.1 (dependency bumped to `~> 0.7.1`). Refs #354.
|
|
42
49
|
- **`connection_guc_mode = :session` no longer kills every SSE stream and NOTIFY wakeup with `PG::Error: invalid connection option "variables"` (issue #352).** In `:session` mode, `forward_connection_variables` deliberately leaves the database.yml `variables:` hash on the connection options for the *caller* to strip and apply via post-connect `SET` (a transaction-mode pooler rejects the libpq `options` startup param — the reason `:session` mode exists). `Pgbus::Client#wrap_session_gucs` honors that contract on both pgmq pools, but the two dedicated raw-connect paths — the streamer's `build_raw_pg_connection` and the worker `NotifyListener` — passed the hash straight to `PG.connect`, which rejects the non-libpq `:variables` key. Result: the first stream request per process raised, `StreamApp` rescued it into a 500 (so **every** `/pgbus/streams/…` request failed from then on), and NOTIFY-gated wakeups died with workers silently falling back to polling. Both paths now connect through the new `Pgbus::DedicatedConnection.connect`, which strips `:variables` and applies each GUC via post-connect `SET` — dedicated connections *keep* the operator's GUCs (`statement_timeout`, `timezone`, …), matching the pooled paths, and the mechanism works on both a direct port and a session-mode pooler. A guard spec confines raw `PG.connect` call sites to `Client` and `DedicatedConnection`, so the next dedicated-connection path can't silently reintroduce the bypass. Refs #352.
|
|
43
50
|
- **AppSignal main dashboard: job and event lines now split by `job_class` and `routing_key`.** Applied the upstream review suggestions committed on appsignal/public_config#80 to the vendored `dashboard.json`: the "Job status per queue" panel gains a `job_class` wildcard tag filter and label (`%job_class% - %queue% - %status%`), and "Event handler status" gains `routing_key` (`%handler% - %status% - %routing_key%`). Without the wildcard tag entry a `%placeholder%` in the line label never resolves, so lines from different job classes / routing keys were collapsed together. The subscriber already emitted both tags; only the dashboard definition lagged.
|
|
@@ -8,6 +8,13 @@ class UpgradePgmqToV<%= target_version_slug.camelize %> < ActiveRecord::Migratio
|
|
|
8
8
|
# This uses the vendored SQL which doesn't require the pgmq extension.
|
|
9
9
|
execute Pgbus::PgmqSchema.install_sql("<%= target_version %>")
|
|
10
10
|
|
|
11
|
+
# Re-install the per-queue NOTIFY insert triggers the function drop
|
|
12
|
+
# cascaded away (pgmq.notify_insert_throttle records which queues had
|
|
13
|
+
# them and at what throttle; enable_notify_insert is idempotent).
|
|
14
|
+
# Without this, NOTIFY-gated wakeups silently die and workers fall back
|
|
15
|
+
# to polling until each queue is re-ensured by a process restart.
|
|
16
|
+
execute Pgbus::PgmqSchema.reinstall_notify_triggers_sql
|
|
17
|
+
|
|
11
18
|
# Record the upgrade
|
|
12
19
|
execute <<~SQL
|
|
13
20
|
CREATE TABLE IF NOT EXISTS pgbus_pgmq_schema_versions (
|
data/lib/pgbus/client.rb
CHANGED
|
@@ -89,15 +89,17 @@ module Pgbus
|
|
|
89
89
|
# persistent connection instead of a fresh PG.connect per call. Its own
|
|
90
90
|
# PGMQ::Client → its own connection_pool, sized independently of worker
|
|
91
91
|
# thread counts.
|
|
92
|
-
# Build the streams pool from
|
|
93
|
-
# to
|
|
94
|
-
#
|
|
95
|
-
#
|
|
96
|
-
#
|
|
97
|
-
#
|
|
92
|
+
# Build the streams pool from streams_pool_connection_options (defaults
|
|
93
|
+
# to streams_connection_options so a separate streams DB carries the
|
|
94
|
+
# pool with it — issue #315 — but overridable via streams_pool_* so a
|
|
95
|
+
# pooler-bypass install keeps the pool off the direct port — issue
|
|
96
|
+
# #358), bounds-applied, and tagged with a per-process application_name
|
|
97
|
+
# so the autoscaler can count peer processes from pg_stat_activity
|
|
98
|
+
# (issue #323 P1/P2). Snapshot it so a hot-swap rebuilds a
|
|
99
|
+
# byte-identical pool at a new size.
|
|
98
100
|
@streams_conn_opts = wrap_session_gucs(
|
|
99
101
|
tag_application_name(
|
|
100
|
-
apply_connection_bounds(config.
|
|
102
|
+
apply_connection_bounds(config.streams_pool_connection_options)
|
|
101
103
|
)
|
|
102
104
|
)
|
|
103
105
|
@streams_pgmq = PGMQ::Client.new(@streams_conn_opts, pool_size: config.streams_pool_size,
|
data/lib/pgbus/concurrency.rb
CHANGED
|
@@ -16,25 +16,30 @@ module Pgbus
|
|
|
16
16
|
#
|
|
17
17
|
# Options:
|
|
18
18
|
# to: Maximum concurrent jobs for the same key (required)
|
|
19
|
-
# key: Proc receiving job arguments, returns a string key.
|
|
19
|
+
# key: Proc receiving job arguments, returns a string key.
|
|
20
|
+
# Default: the ENQUEUED job's class name (resolved at
|
|
21
|
+
# resolve time, so an inherited declaration keys each
|
|
22
|
+
# subclass separately — issue #357).
|
|
20
23
|
# duration: Safety expiry for semaphore (default: 15 minutes)
|
|
21
24
|
# on_conflict: What to do when limit is reached — :block, :discard, or :raise (default: :block)
|
|
22
25
|
def limits_concurrency(to:, key: nil, duration: 15 * 60, on_conflict: :block) # rubocop:disable Naming/MethodParameterName
|
|
23
26
|
raise ArgumentError, "to: must be a positive integer" unless to.is_a?(Integer) && to.positive?
|
|
24
27
|
raise ArgumentError, "on_conflict must be :block, :discard, or :raise" unless %i[block discard raise].include?(on_conflict)
|
|
25
28
|
raise ArgumentError, "duration must be a positive number" unless duration.is_a?(Numeric) && duration.positive?
|
|
26
|
-
raise ArgumentError, "key must be callable (Proc or lambda)" if key && !key.respond_to?(:call)
|
|
29
|
+
raise ArgumentError, "key must be callable (Proc or lambda)" if !key.nil? && !key.respond_to?(:call)
|
|
27
30
|
|
|
28
31
|
@pgbus_concurrency = {
|
|
29
32
|
limit: to,
|
|
30
|
-
key: key
|
|
33
|
+
key: key,
|
|
31
34
|
duration: duration,
|
|
32
35
|
on_conflict: on_conflict
|
|
33
36
|
}.freeze
|
|
34
37
|
end
|
|
35
38
|
|
|
39
|
+
# The nearest declaration in the ancestor chain wins — same inheritance
|
|
40
|
+
# contract as Uniqueness#pgbus_uniqueness (issue #357).
|
|
36
41
|
def pgbus_concurrency
|
|
37
|
-
@pgbus_concurrency
|
|
42
|
+
@pgbus_concurrency || (superclass.pgbus_concurrency if superclass.respond_to?(:pgbus_concurrency))
|
|
38
43
|
end
|
|
39
44
|
end
|
|
40
45
|
|
|
@@ -47,13 +52,11 @@ module Pgbus
|
|
|
47
52
|
config = active_job.class.pgbus_concurrency
|
|
48
53
|
return nil unless config
|
|
49
54
|
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
|
|
55
|
-
config[:key].call(*args)
|
|
56
|
-
end
|
|
55
|
+
# Class-name default, resolved from the ENQUEUED job's class so an
|
|
56
|
+
# inherited declaration keys each subclass separately (#357).
|
|
57
|
+
return active_job.class.name unless config[:key]
|
|
58
|
+
|
|
59
|
+
Support.call_key_proc(config[:key], active_job.arguments)
|
|
57
60
|
end
|
|
58
61
|
|
|
59
62
|
# Inject the resolved concurrency key into the job's serialized payload.
|
data/lib/pgbus/configuration.rb
CHANGED
|
@@ -167,6 +167,7 @@ module Pgbus
|
|
|
167
167
|
:streams_durable_patterns, :streams_broadcast_queue,
|
|
168
168
|
:streams_presence_patterns, :streams_presence_member,
|
|
169
169
|
:streams_host, :streams_port, :streams_database_url,
|
|
170
|
+
:streams_pool_host, :streams_pool_port, :streams_pool_database_url,
|
|
170
171
|
:streams_pool_size, :streams_pool_timeout,
|
|
171
172
|
:streams_fanout_write_deadline_ms, :streams_dispatch_queue_limit,
|
|
172
173
|
:streams_writer_threads, :streams_writer_buffer_limit,
|
|
@@ -338,6 +339,22 @@ module Pgbus
|
|
|
338
339
|
# thread-safe and streams keep using the single serialized connection.
|
|
339
340
|
@streams_pool_size = 5
|
|
340
341
|
@streams_pool_timeout = 5
|
|
342
|
+
# Streams-POOL-only connection overrides (issue #358). The pool above
|
|
343
|
+
# defaults to wherever the streamer connects (streams_host/port/
|
|
344
|
+
# database_url), which is right for a separate streams database — but
|
|
345
|
+
# wrong for the pooler-bypass pattern, where streams_port pins the
|
|
346
|
+
# LISTEN connection to the direct Postgres port purely because LISTEN
|
|
347
|
+
# dies in transaction pooling. The pool's traffic (broadcast INSERTs,
|
|
348
|
+
# replay reads) is plain pooler-safe SQL, and on a direct port with a
|
|
349
|
+
# low max_connections ceiling every process's streams_pool_size
|
|
350
|
+
# connections eat scarce direct slots. Set any of these to route the
|
|
351
|
+
# POOL independently of the LISTEN connection:
|
|
352
|
+
#
|
|
353
|
+
# c.streams_port = 5432 # LISTEN bypasses the pooler
|
|
354
|
+
# c.streams_pool_port = 6432 # the pool stays on the pooler
|
|
355
|
+
@streams_pool_host = nil
|
|
356
|
+
@streams_pool_port = nil
|
|
357
|
+
@streams_pool_database_url = nil
|
|
341
358
|
# Self-tuning streams-pool autoscaling (issue #323). Opt-in, default off.
|
|
342
359
|
# When true, a per-web-process control loop grows the dedicated streams
|
|
343
360
|
# pool into a FAIR SHARE of live Postgres connection headroom under
|
|
@@ -1214,6 +1231,33 @@ module Pgbus
|
|
|
1214
1231
|
override_connection_options(url: streams_database_url, host: streams_host, port: streams_port)
|
|
1215
1232
|
end
|
|
1216
1233
|
|
|
1234
|
+
# Connection options for the dedicated streams PGMQ pool — durable-broadcast
|
|
1235
|
+
# publish INSERTs and the dispatcher's replay reads (issue #315). Defaults
|
|
1236
|
+
# to `streams_connection_options`, so a genuinely separate streams database
|
|
1237
|
+
# carries the pool with it. When any of `streams_pool_database_url` /
|
|
1238
|
+
# `streams_pool_host` / `streams_pool_port` is set, the pool is routed
|
|
1239
|
+
# independently — applied to the BASE options, exactly like the other two
|
|
1240
|
+
# override groups.
|
|
1241
|
+
#
|
|
1242
|
+
# Why this exists (issue #358): only the streamer's LISTEN connection needs
|
|
1243
|
+
# to bypass a transaction-mode pooler (LISTEN dies at COMMIT boundaries);
|
|
1244
|
+
# the pool's INSERT/SELECT traffic is pooler-safe. Without this split, a
|
|
1245
|
+
# `streams_port` direct-port pin drags streams_pool_size connections per
|
|
1246
|
+
# process onto the direct port's low max_connections ceiling:
|
|
1247
|
+
#
|
|
1248
|
+
# c.streams_port = 5432 # LISTEN bypasses the pooler
|
|
1249
|
+
# c.streams_pool_port = 6432 # the pool stays on the pooler
|
|
1250
|
+
#
|
|
1251
|
+
# Precedence: streams_pool_database_url > streams_pool_host/port over the
|
|
1252
|
+
# base > streams_connection_options.
|
|
1253
|
+
def streams_pool_connection_options
|
|
1254
|
+
return streams_connection_options unless streams_pool_database_url || streams_pool_host || streams_pool_port
|
|
1255
|
+
|
|
1256
|
+
override_connection_options(
|
|
1257
|
+
url: streams_pool_database_url, host: streams_pool_host, port: streams_pool_port
|
|
1258
|
+
)
|
|
1259
|
+
end
|
|
1260
|
+
|
|
1217
1261
|
# Connection options for the Worker's dedicated NotifyListener connection.
|
|
1218
1262
|
# Mirrors streams_connection_options: defaults to the base connection_options,
|
|
1219
1263
|
# overridable via worker_notify_database_url / worker_notify_host /
|
|
@@ -32,13 +32,19 @@ module Pgbus
|
|
|
32
32
|
def verdict
|
|
33
33
|
queues = @data_source.queues_with_metrics
|
|
34
34
|
processes = @data_source.processes
|
|
35
|
+
stream_names = @data_source.stream_queue_names
|
|
35
36
|
health, health_error = safe_queue_health
|
|
36
37
|
|
|
37
|
-
# Partition queues once:
|
|
38
|
-
#
|
|
39
|
-
#
|
|
40
|
-
#
|
|
41
|
-
|
|
38
|
+
# Partition queues once: the operational set excludes DLQs and
|
|
39
|
+
# registered stream queues, and the backlog is the subset with
|
|
40
|
+
# visible, claimable messages. Stream queues are excluded like DLQs
|
|
41
|
+
# (issue #359): stream delivery is a non-consuming peek, so their
|
|
42
|
+
# messages are permanently visible with read_ct=0 — exactly the wedge
|
|
43
|
+
# signature — and counting them makes the verdict scream STALLED on
|
|
44
|
+
# every streams-heavy install. Paused queues are removed from the
|
|
45
|
+
# STALLED backlog (an intentional pause is not the wedge — it's
|
|
46
|
+
# reported under DEGRADED), but kept in `non_dlq` for the summary.
|
|
47
|
+
non_dlq = queues.reject { |q| dlq?(q) || stream_names.include?(q[:name].to_s) }
|
|
42
48
|
backlog = non_dlq.select { |q| q[:queue_visible_length].to_i.positive? }
|
|
43
49
|
active_backlog = backlog.reject { |q| q[:paused] }
|
|
44
50
|
|
data/lib/pgbus/pgmq_schema.rb
CHANGED
|
@@ -135,6 +135,46 @@ module Pgbus
|
|
|
135
135
|
SQL
|
|
136
136
|
end
|
|
137
137
|
|
|
138
|
+
# SQL to re-install the per-queue NOTIFY insert triggers that
|
|
139
|
+
# drop_pgmq_functions_sql cascades away (issue #360). Dropping
|
|
140
|
+
# pgmq.notify_queue_listeners() with CASCADE also drops the
|
|
141
|
+
# trigger_notify_queue_insert_listeners trigger from every queue table;
|
|
142
|
+
# install_sql re-creates the function but nothing re-creates the
|
|
143
|
+
# triggers, so NOTIFY-gated wakeups silently die fleet-wide until each
|
|
144
|
+
# queue happens to be re-ensured by a process restart.
|
|
145
|
+
#
|
|
146
|
+
# The dropped state is fully recoverable: pgmq.notify_insert_throttle is
|
|
147
|
+
# a TABLE (preserved by the upgrade — its rows record exactly which
|
|
148
|
+
# queues had notify enabled and at what throttle, FK-bound to pgmq.meta
|
|
149
|
+
# so it can't reference a dropped queue), and pgmq.enable_notify_insert
|
|
150
|
+
# is idempotent. Replaying it per recorded row restores every trigger at
|
|
151
|
+
# its original interval. No-ops when the throttle table or the enable
|
|
152
|
+
# function is absent (a vendored version without the notify feature).
|
|
153
|
+
def reinstall_notify_triggers_sql
|
|
154
|
+
<<~SQL
|
|
155
|
+
DO $$
|
|
156
|
+
DECLARE
|
|
157
|
+
r RECORD;
|
|
158
|
+
BEGIN
|
|
159
|
+
IF to_regclass('pgmq.notify_insert_throttle') IS NULL THEN
|
|
160
|
+
RETURN;
|
|
161
|
+
END IF;
|
|
162
|
+
IF to_regprocedure('pgmq.enable_notify_insert(text, integer)') IS NULL THEN
|
|
163
|
+
RETURN;
|
|
164
|
+
END IF;
|
|
165
|
+
|
|
166
|
+
-- The FOR loop iterates a snapshot, so enable_notify_insert's
|
|
167
|
+
-- internal DELETE + re-INSERT of the same throttle row is safe.
|
|
168
|
+
FOR r IN
|
|
169
|
+
SELECT queue_name, throttle_interval_ms
|
|
170
|
+
FROM pgmq.notify_insert_throttle
|
|
171
|
+
LOOP
|
|
172
|
+
PERFORM pgmq.enable_notify_insert(r.queue_name, r.throttle_interval_ms);
|
|
173
|
+
END LOOP;
|
|
174
|
+
END $$;
|
|
175
|
+
SQL
|
|
176
|
+
end
|
|
177
|
+
|
|
138
178
|
private
|
|
139
179
|
|
|
140
180
|
# Strips extension-specific blocks (pg_extension_config_dump, pg_depend checks)
|
|
@@ -139,7 +139,7 @@ module Pgbus
|
|
|
139
139
|
return nil unless config
|
|
140
140
|
return nil unless config[:strategy] == :until_executed
|
|
141
141
|
|
|
142
|
-
key = resolve_uniqueness_key(config, task)
|
|
142
|
+
key = resolve_uniqueness_key(job_class, config, task)
|
|
143
143
|
return nil unless key
|
|
144
144
|
|
|
145
145
|
acquired = UniquenessKey.acquire!(key, queue_name: resolve_queue(task), msg_id: 0)
|
|
@@ -155,6 +155,7 @@ module Pgbus
|
|
|
155
155
|
"[Pgbus] Uniqueness lock check failed for #{task.key}: #{e.class}: #{e.message} — " \
|
|
156
156
|
"skipping enqueue to prevent duplicates"
|
|
157
157
|
end
|
|
158
|
+
ErrorReporter.report(e, { action: "recurring_uniqueness_lock", task: task.key })
|
|
158
159
|
:already_locked # Fail closed — skip enqueue when lock check errors
|
|
159
160
|
end
|
|
160
161
|
|
|
@@ -168,19 +169,39 @@ module Pgbus
|
|
|
168
169
|
end
|
|
169
170
|
|
|
170
171
|
# Resolve the uniqueness key for a recurring task.
|
|
171
|
-
#
|
|
172
|
-
|
|
172
|
+
# With no explicit key: the key is the SCHEDULED job class's name —
|
|
173
|
+
# resolved here, not via a stored proc, so a declaration inherited from
|
|
174
|
+
# a base class keys each subclass separately (issue #357) — qualified
|
|
175
|
+
# with the task's arguments so two recurring tasks pointing at the same
|
|
176
|
+
# job class with different args: don't share one lock (the scheduler
|
|
177
|
+
# can't use #333's raise-at-enqueue guard, so it disambiguates instead).
|
|
178
|
+
#
|
|
179
|
+
# Every failure here — default path or user-supplied key proc — must
|
|
180
|
+
# propagate to acquire_uniqueness_lock's fail-closed rescue (skip this
|
|
181
|
+
# tick, log, report). Returning nil instead would mean "no key
|
|
182
|
+
# configured", and the task would be enqueued WITHOUT a lock: a broken
|
|
183
|
+
# key proc silently disabling the very protection it configures. The
|
|
184
|
+
# warn adds the failing-proc context before re-raising.
|
|
185
|
+
def resolve_uniqueness_key(job_class, config, task)
|
|
173
186
|
key_proc = config[:key]
|
|
174
187
|
args = task.arguments || []
|
|
188
|
+
return default_uniqueness_key(job_class, args) unless key_proc
|
|
175
189
|
|
|
176
|
-
|
|
177
|
-
key_proc.call
|
|
178
|
-
|
|
179
|
-
|
|
190
|
+
begin
|
|
191
|
+
args.empty? ? key_proc.call : key_proc.call(*args)
|
|
192
|
+
rescue StandardError => e
|
|
193
|
+
Pgbus.logger.warn { "[Pgbus] Could not resolve uniqueness key for #{task.key}: #{e.message}" }
|
|
194
|
+
raise
|
|
180
195
|
end
|
|
181
|
-
|
|
182
|
-
|
|
183
|
-
|
|
196
|
+
end
|
|
197
|
+
|
|
198
|
+
# Class-name default for the scheduler path, args-qualified so distinct
|
|
199
|
+
# argument sets get distinct locks. JSON keeps the key deterministic
|
|
200
|
+
# and readable in the dashboard/locks table.
|
|
201
|
+
def default_uniqueness_key(job_class, args)
|
|
202
|
+
return job_class.name if args.empty?
|
|
203
|
+
|
|
204
|
+
"#{job_class.name}:#{JSON.generate(args)}"
|
|
184
205
|
end
|
|
185
206
|
|
|
186
207
|
# Inject uniqueness metadata into the payload so the executor releases
|
|
@@ -197,7 +218,7 @@ module Pgbus
|
|
|
197
218
|
return payload unless config
|
|
198
219
|
return payload unless config[:strategy] == :until_executed
|
|
199
220
|
|
|
200
|
-
key = resolve_uniqueness_key(config, task)
|
|
221
|
+
key = resolve_uniqueness_key(job_class, config, task)
|
|
201
222
|
return payload unless key
|
|
202
223
|
|
|
203
224
|
payload.merge(
|
|
@@ -0,0 +1,22 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module Pgbus
|
|
4
|
+
# Shared internal helpers.
|
|
5
|
+
module Support
|
|
6
|
+
module_function
|
|
7
|
+
|
|
8
|
+
# Call a user-supplied key proc with a job's arguments, preserving
|
|
9
|
+
# ActiveJob's keyword-argument convention: a trailing Hash whose keys are
|
|
10
|
+
# all Symbols is splatted as keywords; everything else is positional.
|
|
11
|
+
# Single implementation for Uniqueness.resolve_key and
|
|
12
|
+
# Concurrency.resolve_key so the dispatch semantics cannot drift apart.
|
|
13
|
+
def call_key_proc(key_proc, args)
|
|
14
|
+
last = args.last
|
|
15
|
+
if last.is_a?(Hash) && last.each_key.all?(Symbol)
|
|
16
|
+
key_proc.call(*args[...-1], **last)
|
|
17
|
+
else
|
|
18
|
+
key_proc.call(*args)
|
|
19
|
+
end
|
|
20
|
+
end
|
|
21
|
+
end
|
|
22
|
+
end
|
data/lib/pgbus/uniqueness.rb
CHANGED
|
@@ -60,7 +60,7 @@ module Pgbus
|
|
|
60
60
|
raise ArgumentError, "unknown keyword: #{opts.keys.first.inspect}" if opts.any?
|
|
61
61
|
raise ArgumentError, "strategy must be one of: #{VALID_STRATEGIES.join(", ")}" unless VALID_STRATEGIES.include?(strategy)
|
|
62
62
|
raise ArgumentError, "on_conflict must be one of: #{VALID_CONFLICTS.join(", ")}" unless VALID_CONFLICTS.include?(on_conflict)
|
|
63
|
-
raise ArgumentError, "key must be callable (Proc or lambda)" if key && !key.respond_to?(:call)
|
|
63
|
+
raise ArgumentError, "key must be callable (Proc or lambda)" if !key.nil? && !key.respond_to?(:call)
|
|
64
64
|
|
|
65
65
|
# Record whether an explicit key was given. With NO explicit key the key
|
|
66
66
|
# defaults to the class name; that is safe for a no-argument job (one
|
|
@@ -70,16 +70,26 @@ module Pgbus
|
|
|
70
70
|
# order into ONE per-class singleton. resolve_key raises at resolve time
|
|
71
71
|
# when the class-name default meets non-empty arguments (see #333); the
|
|
72
72
|
# no-arg case keeps working. explicit_key marks which is which.
|
|
73
|
+
#
|
|
74
|
+
# No proc is stored for the default — resolve_key derives it from the
|
|
75
|
+
# ENQUEUED job's class. A stored `->(*) { name }` would capture the
|
|
76
|
+
# DECLARING class, so a base-class declaration would collapse every
|
|
77
|
+
# subclass into one shared key once configs became inheritable (#357).
|
|
73
78
|
@pgbus_uniqueness = {
|
|
74
79
|
strategy: strategy,
|
|
75
|
-
key: key
|
|
80
|
+
key: key,
|
|
76
81
|
explicit_key: !key.nil?,
|
|
77
82
|
on_conflict: on_conflict
|
|
78
83
|
}.freeze
|
|
79
84
|
end
|
|
80
85
|
|
|
86
|
+
# The nearest declaration in the ancestor chain wins: a class's own
|
|
87
|
+
# `ensures_uniqueness` beats an inherited one, and a base-class
|
|
88
|
+
# declaration reaches every subclass (issue #357 — class-level ivars
|
|
89
|
+
# don't inherit, so without the superclass walk a base declaration was
|
|
90
|
+
# silently inert for subclasses).
|
|
81
91
|
def pgbus_uniqueness
|
|
82
|
-
@pgbus_uniqueness
|
|
92
|
+
@pgbus_uniqueness || (superclass.pgbus_uniqueness if superclass.respond_to?(:pgbus_uniqueness))
|
|
83
93
|
end
|
|
84
94
|
end
|
|
85
95
|
|
|
@@ -90,11 +100,12 @@ module Pgbus
|
|
|
90
100
|
|
|
91
101
|
args = active_job.arguments
|
|
92
102
|
guard_class_name_default!(active_job, config, args)
|
|
93
|
-
|
|
94
|
-
|
|
95
|
-
config[:key].call(*args[...-1], **last)
|
|
103
|
+
key = if config[:explicit_key]
|
|
104
|
+
Support.call_key_proc(config[:key], args)
|
|
96
105
|
else
|
|
97
|
-
|
|
106
|
+
# Class-name default, resolved from the ENQUEUED job's class so
|
|
107
|
+
# an inherited declaration keys each subclass separately (#357).
|
|
108
|
+
active_job.class.name
|
|
98
109
|
end
|
|
99
110
|
|
|
100
111
|
# Automatically serialize GlobalID-compatible objects (e.g. ActiveRecord models)
|
data/lib/pgbus/version.rb
CHANGED
|
@@ -65,6 +65,19 @@ module Pgbus
|
|
|
65
65
|
end
|
|
66
66
|
private :fetch_queues_with_metrics
|
|
67
67
|
|
|
68
|
+
# Physical queue names registered as stream queues (the
|
|
69
|
+
# pgbus_stream_queues registry). Stream delivery is a non-consuming
|
|
70
|
+
# peek, so stream messages sit visible with read_ct=0 forever — health
|
|
71
|
+
# verdicts must not read that as a wedge (issue #359). Loaded fresh per
|
|
72
|
+
# call (reset + one query): verdicts run on coarse intervals and a
|
|
73
|
+
# long-lived process must see streams registered since the last check.
|
|
74
|
+
# Degrades to an empty Set when the registry table is absent
|
|
75
|
+
# (pre-migration installs) or unreadable — StreamQueue swallows both.
|
|
76
|
+
def stream_queue_names
|
|
77
|
+
StreamQueue.reset_cache!
|
|
78
|
+
StreamQueue.all_names
|
|
79
|
+
end
|
|
80
|
+
|
|
68
81
|
# name is the full PGMQ queue name (e.g. "pgbus_default") as returned
|
|
69
82
|
# by queues_with_metrics. No prefix is added.
|
|
70
83
|
def queue_detail(name)
|
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: pgbus
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.12.
|
|
4
|
+
version: 0.12.1
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Mikael Henriksson
|
|
@@ -365,6 +365,7 @@ files:
|
|
|
365
365
|
- lib/pgbus/streams/turbo_broadcastable.rb
|
|
366
366
|
- lib/pgbus/streams/turbo_stream_override.rb
|
|
367
367
|
- lib/pgbus/streams/watermark_cache_middleware.rb
|
|
368
|
+
- lib/pgbus/support.rb
|
|
368
369
|
- lib/pgbus/table_maintenance.rb
|
|
369
370
|
- lib/pgbus/testing.rb
|
|
370
371
|
- lib/pgbus/testing/assertions.rb
|