pgbus 0.12.4 → 0.13.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +10 -0
- data/README.md +2 -1
- data/Rakefile +17 -1
- data/lib/pgbus/client.rb +15 -0
- data/lib/pgbus/configuration/capsule_dsl.rb +8 -0
- data/lib/pgbus/configuration.rb +65 -5
- data/lib/pgbus/dedicated_connection.rb +29 -2
- data/lib/pgbus/doctor.rb +47 -2
- data/lib/pgbus/event_bus/registry.rb +22 -0
- data/lib/pgbus/process/consumer.rb +57 -22
- data/lib/pgbus/process/notify_hub.rb +273 -0
- data/lib/pgbus/process/notify_listener.rb +47 -2
- data/lib/pgbus/process/supervisor.rb +146 -34
- data/lib/pgbus/process/wake_pipe.rb +129 -0
- data/lib/pgbus/process/wildcard_queue_resolver.rb +35 -0
- data/lib/pgbus/process/worker.rb +58 -35
- data/lib/pgbus/version.rb +1 -1
- data/lib/pgbus/web/streamer/failover_listener.rb +130 -0
- data/lib/pgbus/web/streamer/hub_client.rb +199 -0
- data/lib/pgbus/web/streamer/hub_protocol.rb +85 -0
- data/lib/pgbus/web/streamer/instance.rb +69 -20
- data/lib/pgbus/web/streamer/listener.rb +17 -2
- data/lib/pgbus/web/streamer/master_hub.rb +414 -0
- data/lib/pgbus/web/streamer/master_hub_boot.rb +149 -0
- data/lib/puma/plugin/pgbus_streams.rb +38 -2
- metadata +9 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 8c955f8fb2ac4e66fb2db388505ee83341aa6dec7c00f606162f7fba7c29cb75
|
|
4
|
+
data.tar.gz: a31e43c6f48cb10609d73dc507e9d177d25a299ae25c6f5d5a39a045018c32e8
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 7cd3fba720a5ab724fe4f0b5b89168cb2df4ba68e0a5a35cca725093724f8f2b2109c379a2263442111367001a52d2eda440eef8e1c2469653d2022d2a92d148
|
|
7
|
+
data.tar.gz: 4a4e5e1c2c229c599541e61ac6ad5602dfbfcf057b346b22b0271c10a48752c41075c7166b43dd55ece7ef4a30d79ec3fd6f5cac5b898b2578ecb90c73bcc2a8
|
data/CHANGELOG.md
CHANGED
|
@@ -1,7 +1,17 @@
|
|
|
1
1
|
## [Unreleased]
|
|
2
2
|
|
|
3
|
+
### Added
|
|
4
|
+
|
|
5
|
+
- **Streams: one LISTEN connection per web host — `streams_listen_scope` (issue #382).** ⚠️ **Default behavior change.** Previously every Puma worker lazily opened its own dedicated streams LISTEN connection on first SSE use, so a web host pinned one direct connection per worker. Under the new default (`streams_listen_scope = :master`) the `pgbus_streams` Puma plugin runs a **MasterHub** in the preforking master: ONE `Web::Streamer::Listener` on the refcounted union of every worker's stream channels, fanning wakes — **including ephemeral payloads** — out to workers over a Unix domain socket with length-prefixed frames (`Streamer::HubProtocol`). Workers connect lazily (nothing is inherited across fork) and the synchronous `ensure_listening` ack contract is preserved cross-process: a sub is registered before LISTEN executes and acked only after, so the no-lost-broadcast guarantee holds. Backpressure follows the streams rules: durable wakes are droppable at a per-worker cap (they self-heal via `read_after`), **ephemeral wakes are never dropped** — a worker that stops draining is evicted, which triggers its own fallback. **Fallback is per-worker listeners, not loss**: whenever the hub is absent or dies (no `preload_app!`, single-mode Puma, crash, eviction) each worker's `FailoverListener` swaps in a real per-worker `Listener` and re-LISTENs its recorded subscriptions — connection footprint balloons back to pre-#382 levels (census-visible) but no broadcast semantics change; the worker stays local until it recycles. Measured (local PG, n=50): the master→worker hop is noise-level free — single-broadcast SSE roundtrip p50 16.00ms via the hub vs 16.93ms per-worker. **`:master` effectively requires `preload_app!`** (the hub waits for the app's pgbus initializer; without it the deadline expires quietly and workers stay per-worker). **Rollback:** `config.streams_listen_scope = :process`. Refs #382, builds on the #381 patterns.
|
|
6
|
+
|
|
7
|
+
- **Host-level shared LISTEN: `worker_notify_scope` — the supervisor now owns ONE direct LISTEN connection for the whole host (issue #381).** ⚠️ **Default behavior change.** Previously every worker fork and every consumer fork opened its own dedicated LISTEN connection (`NotifyListener`), so a host's direct-connection footprint scaled with fork count — on transaction-pool PgBouncer platforms those connections come out of the scarcest slice of `max_connections`, and a 5-capsule + 2-consumer host pinned 7. Under the new default (`config.worker_notify_scope = :supervisor`) the supervisor runs a single `NotifyHub`: one `NotifyListener` on the union of every capsule's and consumer's queue channels (wildcards via the shared resolver, consumer sets via the registry), fanning wakes out to forks over per-fork pipes (`W` wake / `H` healthy / `P` degraded bytes; a fork whose pipe reports degraded or reaches EOF falls back to fast polling exactly like a failed local listener). Footprint drops to **1 direct LISTEN connection per job host**, verified by integration test: routing is per-fork (an insert wakes only the forks reading that queue, wildcard capsules unconditionally), and `pg_terminate_backend` on the shared connection is survived — reconnect, re-LISTEN, wakes flow again. **Rollback:** `config.worker_notify_scope = :fork` restores the previous per-fork listeners byte-for-byte. Dedicated LISTEN connections are now census-tagged `application_name=pgbus-listen` so `pg_stat_activity` can count them. Refs #381.
|
|
8
|
+
- **`pgbus doctor`: new "Connection budget" check (issue #381).** Prints how many direct LISTEN connections the current config pins — 1 per host under `:supervisor` scope, capsules + consumers under `:fork` (honoring `config.roles`), plus a "+1 per web-server process (streams)" clause — so operators can do pooler capacity math from the doctor output alone. Informational, always `:ok`. Refs #381.
|
|
9
|
+
- **Benchmarks: `rake bench:notify_wake` and `rake bench:notify_chaos` (issue #381).** Wake-path latency (send → wake, p50/p95/p99, direct vs hub-mediated), empty-read cost, LISTEN connection census, and failure-mode measurements (killed LISTEN backend, wedged fork, FD churn, fan-out cost). Refs #381.
|
|
10
|
+
|
|
3
11
|
### Changed
|
|
4
12
|
|
|
13
|
+
- **Multi-queue strict-priority contract documented and pinned (issue #381).** The capsule DSL's "list order = strict priority" promise rides on Postgres's Append node filling `read_multi`'s `UNION ALL … LIMIT` from earlier-listed subqueries first — incidental behavior nothing upstream promises. The contract (and the vt-claim caveat: subquery rows discarded by the outer LIMIT are still claimed and go invisible for one visibility timeout) is now documented on `Client#read_multi` and the capsule DSL, and pinned by an integration canary (`spec/integration/multi_queue_priority_spec.rb`) so a planner or pgmq-ruby change breaks loudly instead of silently. `Consumer#wake_timeout` also gained the live-but-deaf listener consult (`delivering?`) for parity with Worker (issue #332). Refs #381.
|
|
14
|
+
|
|
5
15
|
- **README: doctor table and configuration reference catch up to the shipped surface (issue #369).** The doctor section still said "six checks" and listed only the original six; `Pgbus::Doctor::CHECKS` has 10 (GlobalID allowlist, Broadcast queue, Primary affinity, Dedicated connections were missing — the first two are the ones most likely to warn after upgrade on nil defaults). The configuration reference table was missing ~20 shipped options operators actually hit: `allowed_global_id_models` (incl. `[]` = deny-all), streams GC (`streams_orphan_threshold` / `streams_orphan_sweep_interval` — 24h / hourly defaults that auto-drop dormant durable stream queues), retention/durable/presence/broadcast-queue knobs, `streams_pool_*` / LISTEN connection overrides, `doctor_on_boot`, `require_primary`, `connects_to`, `zombie_detection`, `connection_guc_mode`, `worker_notify_*`, `group_mode`, and neighbors. A unit drift guard pins the README doctor table to every `Doctor::CHECKS` name. Full typed reference remains on the docs site. Refs #369.
|
|
6
16
|
|
|
7
17
|
### Fixed
|
data/README.md
CHANGED
|
@@ -1862,7 +1862,7 @@ pgbus help # Show help
|
|
|
1862
1862
|
|
|
1863
1863
|
#### pgbus doctor
|
|
1864
1864
|
|
|
1865
|
-
A single preflight command that answers "is this environment healthy enough to run?" — useful as a deploy or CI gate. It runs
|
|
1865
|
+
A single preflight command that answers "is this environment healthy enough to run?" — useful as a deploy or CI gate. It runs 11 checks and never raises; a broken environment turns every probe into a failed/warned check instead of a crash:
|
|
1866
1866
|
|
|
1867
1867
|
| Check | Fails (`:fail`) when | Warns (`:warn`) when |
|
|
1868
1868
|
|---|---|---|
|
|
@@ -1876,6 +1876,7 @@ A single preflight command that answers "is this environment healthy enough to r
|
|
|
1876
1876
|
| Broadcast queue | — | Turbo broadcasts share the default queue in production, or `streams_broadcast_queue` is set but no worker capsule drains it |
|
|
1877
1877
|
| Primary affinity | — | Job connection is on a read-only replica (`pg_is_in_recovery`) — a read/write-splitting pooler may be stalling jobs |
|
|
1878
1878
|
| Dedicated connections | Streamer LISTEN and/or worker notify dedicated path cannot connect | — |
|
|
1879
|
+
| Connection budget | — (informational: prints how many direct LISTEN connections the current config pins — 1 per host under `worker_notify_scope: :supervisor`, one per fork under `:fork`; streams add 1 per web host under `streams_listen_scope: :master` or 1 per web process under `:process`) | — |
|
|
1879
1880
|
|
|
1880
1881
|
```bash
|
|
1881
1882
|
pgbus doctor # prints the report; exit 1 unless every check passed
|
data/Rakefile
CHANGED
|
@@ -24,7 +24,8 @@ namespace :bench do
|
|
|
24
24
|
# Benches that need a real PostgreSQL/PGMQ (or boot Puma) — excluded from the
|
|
25
25
|
# no-DB unit suite that bench:all runs in CI.
|
|
26
26
|
db_benches = %w[connection_pool_bench integration_bench streams_bench streams_read_pool_bench
|
|
27
|
-
execution_modes_bench pool_swap_bench pool_autoscale_bench job_burst_bench
|
|
27
|
+
execution_modes_bench pool_swap_bench pool_autoscale_bench job_burst_bench
|
|
28
|
+
notify_wake_bench notify_chaos_bench streams_hub_bench].freeze
|
|
28
29
|
# The unit suite is every *_bench.rb that doesn't need a database, derived
|
|
29
30
|
# from the directory so a new unit bench is picked up automatically (kept in
|
|
30
31
|
# sync with bench:one, which globs the same files).
|
|
@@ -84,6 +85,21 @@ namespace :bench do
|
|
|
84
85
|
ruby "benchmarks/job_burst_bench.rb"
|
|
85
86
|
end
|
|
86
87
|
|
|
88
|
+
desc "Run NOTIFY wake-path benchmark (#381 wake latency + connection census; requires PGBUS_DATABASE_URL)"
|
|
89
|
+
task :notify_wake do
|
|
90
|
+
ruby "benchmarks/notify_wake_bench.rb"
|
|
91
|
+
end
|
|
92
|
+
|
|
93
|
+
desc "Run NotifyHub failure-mode measurements (#381 chaos scenarios; requires PGBUS_DATABASE_URL)"
|
|
94
|
+
task :notify_chaos do
|
|
95
|
+
ruby "benchmarks/notify_chaos_bench.rb"
|
|
96
|
+
end
|
|
97
|
+
|
|
98
|
+
desc "Run streams master-hub latency benchmark (#382 hop cost + census; requires PGBUS_DATABASE_URL)"
|
|
99
|
+
task :streams_hub do
|
|
100
|
+
ruby "benchmarks/streams_hub_bench.rb"
|
|
101
|
+
end
|
|
102
|
+
|
|
87
103
|
desc "Run a single benchmark: rake bench:one[client_bench]"
|
|
88
104
|
task :one, [:name] do |_t, args|
|
|
89
105
|
name = args[:name] or abort "Usage: rake bench:one[serialization_bench|client_bench|...]"
|
data/lib/pgbus/client.rb
CHANGED
|
@@ -434,6 +434,21 @@ module Pgbus
|
|
|
434
434
|
# caller receives up to `queue_count * qty` messages. Pass `limit:` to cap
|
|
435
435
|
# the total across all queues — required when feeding a fixed-size pool,
|
|
436
436
|
# otherwise the pool can overflow on multi-queue reads (issue #123).
|
|
437
|
+
#
|
|
438
|
+
# STRICT-PRIORITY CONTRACT (issue #381): when `limit:` is smaller than the
|
|
439
|
+
# total available, earlier-listed queues win — the capsule DSL's "list
|
|
440
|
+
# order = strict priority" promise rides on this. The mechanism is
|
|
441
|
+
# incidental: pgmq-ruby builds `pgmq.read(q1) UNION ALL pgmq.read(q2) …
|
|
442
|
+
# LIMIT n`, and Postgres's Append node fills the LIMIT from the subqueries
|
|
443
|
+
# in written order. Nothing upstream promises that, so the contract is
|
|
444
|
+
# pinned by spec/integration/multi_queue_priority_spec.rb — if that canary
|
|
445
|
+
# ever breaks, switch callers to ordered per-queue reads (the
|
|
446
|
+
# Worker#fetch_prioritized pattern) instead of relying on this method.
|
|
447
|
+
#
|
|
448
|
+
# vt-claim caveat: each subquery may claim (set vt on) up to `qty` rows
|
|
449
|
+
# even when the outer LIMIT discards them — a discarded row goes invisible
|
|
450
|
+
# for one visibility timeout without being processed. Size `qty`/`limit`
|
|
451
|
+
# accordingly on latency-sensitive queues.
|
|
437
452
|
def read_multi(queue_names, qty:, vt: nil, limit: nil)
|
|
438
453
|
full_names = queue_names.map { |q| config.queue_name(q) }
|
|
439
454
|
guarded_read do
|
|
@@ -19,6 +19,14 @@ module Pgbus
|
|
|
19
19
|
# * wildcard, matches all queues
|
|
20
20
|
# *_ trailing wildcard, prefix match (e.g. "staging_*")
|
|
21
21
|
#
|
|
22
|
+
# "List order = strict priority" is enforced two ways at runtime: with
|
|
23
|
+
# priority_levels > 1 the worker reads queues one at a time in list order
|
|
24
|
+
# (Worker#fetch_prioritized); otherwise it relies on Client#read_multi's
|
|
25
|
+
# UNION ALL filling its LIMIT from earlier-listed queues first — an
|
|
26
|
+
# Append-node behavior Postgres does not formally promise, pinned by
|
|
27
|
+
# spec/integration/multi_queue_priority_spec.rb (see Client#read_multi
|
|
28
|
+
# for the contract and its vt-claim caveat).
|
|
29
|
+
#
|
|
22
30
|
# Returns +Array<Hash>+ in the same shape as the legacy +workers:+ array,
|
|
23
31
|
# so the rest of the codebase can consume it without changes:
|
|
24
32
|
#
|
data/lib/pgbus/configuration.rb
CHANGED
|
@@ -179,14 +179,22 @@ module Pgbus
|
|
|
179
179
|
:streams_pool_autoscale_interval, :streams_application_name
|
|
180
180
|
attr_reader :streams_default_broadcast_mode # rubocop:disable Style/AccessorGrouping
|
|
181
181
|
|
|
182
|
-
# NOTIFY-gated worker wakeups. When true,
|
|
183
|
-
#
|
|
184
|
-
#
|
|
185
|
-
#
|
|
186
|
-
#
|
|
182
|
+
# NOTIFY-gated worker wakeups. When true, workers are woken by a
|
|
183
|
+
# NotifyListener PG connection that LISTENs on their queues' INSERT
|
|
184
|
+
# channels instead of blind-polling. Defaults to the value of
|
|
185
|
+
# listen_notify. The worker_notify_* overrides mirror streams_* so the
|
|
186
|
+
# LISTEN connection can bypass PgBouncer.
|
|
187
187
|
attr_accessor :worker_notify_wakeup,
|
|
188
188
|
:worker_notify_host, :worker_notify_port, :worker_notify_database_url
|
|
189
189
|
|
|
190
|
+
# Where the LISTEN connection lives (issue #381):
|
|
191
|
+
# :supervisor (default) — ONE shared NotifyListener in the supervisor
|
|
192
|
+
# process for the whole host; forks are woken over per-fork pipes.
|
|
193
|
+
# Direct-connection footprint: 1, regardless of capsule/consumer count.
|
|
194
|
+
# :fork — one NotifyListener (and one dedicated PG connection) per
|
|
195
|
+
# worker/consumer fork: the pre-0.13 behavior, kept as an escape hatch.
|
|
196
|
+
attr_reader :worker_notify_scope
|
|
197
|
+
|
|
190
198
|
# AppSignal integration (auto-loaded when ::Appsignal is defined and this is true).
|
|
191
199
|
# Set to false to opt out without uninstalling the appsignal gem.
|
|
192
200
|
attr_accessor :appsignal_enabled, :appsignal_probe_enabled
|
|
@@ -261,6 +269,8 @@ module Pgbus
|
|
|
261
269
|
@listen_notify = true
|
|
262
270
|
|
|
263
271
|
@worker_notify_wakeup = nil
|
|
272
|
+
@worker_notify_scope = :supervisor
|
|
273
|
+
@streams_listen_scope = :master
|
|
264
274
|
@worker_notify_host = nil
|
|
265
275
|
@worker_notify_port = nil
|
|
266
276
|
@worker_notify_database_url = nil
|
|
@@ -623,6 +633,56 @@ module Pgbus
|
|
|
623
633
|
@doctor_on_boot = coerced
|
|
624
634
|
end
|
|
625
635
|
|
|
636
|
+
# Where the streams LISTEN connection lives (issue #382):
|
|
637
|
+
# :master (default) — ONE shared listener in the preforking web master
|
|
638
|
+
# (MasterHub); workers connect lazily over a Unix socket and fall back
|
|
639
|
+
# to a per-worker listener whenever the hub is absent or dies.
|
|
640
|
+
# :process — one listener per web process: the pre-0.13 behavior, and
|
|
641
|
+
# the automatic behavior on single-mode / non-preforking servers.
|
|
642
|
+
attr_reader :streams_listen_scope
|
|
643
|
+
|
|
644
|
+
VALID_STREAMS_LISTEN_SCOPES = %i[master process].freeze
|
|
645
|
+
|
|
646
|
+
def streams_listen_scope=(scope)
|
|
647
|
+
coerced = case scope
|
|
648
|
+
when Symbol then scope
|
|
649
|
+
when String then scope.to_sym
|
|
650
|
+
else
|
|
651
|
+
raise Pgbus::ConfigurationError,
|
|
652
|
+
"Invalid streams_listen_scope type: #{scope.class}. " \
|
|
653
|
+
"Must be :master (one shared LISTEN connection per web host) or :process (one per worker)"
|
|
654
|
+
end
|
|
655
|
+
unless VALID_STREAMS_LISTEN_SCOPES.include?(coerced)
|
|
656
|
+
raise Pgbus::ConfigurationError,
|
|
657
|
+
"Invalid streams_listen_scope: #{coerced.inspect}. " \
|
|
658
|
+
"Must be :master (one shared LISTEN connection per web host) or :process (one per worker)"
|
|
659
|
+
end
|
|
660
|
+
|
|
661
|
+
@streams_listen_scope = coerced
|
|
662
|
+
end
|
|
663
|
+
|
|
664
|
+
VALID_WORKER_NOTIFY_SCOPES = %i[supervisor fork].freeze
|
|
665
|
+
|
|
666
|
+
# Validated at assignment time like the other enum options. A String is
|
|
667
|
+
# coerced so YAML-ish configs work.
|
|
668
|
+
def worker_notify_scope=(scope)
|
|
669
|
+
coerced = case scope
|
|
670
|
+
when Symbol then scope
|
|
671
|
+
when String then scope.to_sym
|
|
672
|
+
else
|
|
673
|
+
raise Pgbus::ConfigurationError,
|
|
674
|
+
"Invalid worker_notify_scope type: #{scope.class}. " \
|
|
675
|
+
"Must be :supervisor (one shared LISTEN connection per host) or :fork (one per fork)"
|
|
676
|
+
end
|
|
677
|
+
unless VALID_WORKER_NOTIFY_SCOPES.include?(coerced)
|
|
678
|
+
raise Pgbus::ConfigurationError,
|
|
679
|
+
"Invalid worker_notify_scope: #{coerced.inspect}. " \
|
|
680
|
+
"Must be :supervisor (one shared LISTEN connection per host) or :fork (one per fork)"
|
|
681
|
+
end
|
|
682
|
+
|
|
683
|
+
@worker_notify_scope = coerced
|
|
684
|
+
end
|
|
685
|
+
|
|
626
686
|
def validate!
|
|
627
687
|
if pool_size && !(pool_size.is_a?(Numeric) && pool_size.positive?)
|
|
628
688
|
raise Pgbus::ConfigurationError, "pool_size must be a positive number or nil (auto-tune)"
|
|
@@ -16,10 +16,17 @@ module Pgbus
|
|
|
16
16
|
module DedicatedConnection
|
|
17
17
|
module_function
|
|
18
18
|
|
|
19
|
+
# Stable census tag for every dedicated LISTEN connection, so operators can
|
|
20
|
+
# count them: SELECT count(*) FROM pg_stat_activity WHERE application_name
|
|
21
|
+
# = 'pgbus-listen' (issue #381 connection budget). Applied as
|
|
22
|
+
# fallback_application_name so an explicit application_name in the URL /
|
|
23
|
+
# hash — or PGAPPNAME in the environment — always wins.
|
|
24
|
+
APP_NAME = "pgbus-listen"
|
|
25
|
+
|
|
19
26
|
def connect(opts)
|
|
20
27
|
require "pg" unless defined?(::PG::Connection)
|
|
21
28
|
case opts
|
|
22
|
-
when String then ::PG.connect(opts)
|
|
29
|
+
when String then ::PG.connect(with_app_name(opts))
|
|
23
30
|
when Hash then connect_from_hash(opts)
|
|
24
31
|
else
|
|
25
32
|
raise Pgbus::ConfigurationError,
|
|
@@ -28,9 +35,23 @@ module Pgbus
|
|
|
28
35
|
end
|
|
29
36
|
end
|
|
30
37
|
|
|
38
|
+
# Append the census tag to a conninfo String — URL query-param style for
|
|
39
|
+
# postgres:// URLs, space-separated keyword style otherwise. A string that
|
|
40
|
+
# already mentions application_name (either keyword) is left untouched.
|
|
41
|
+
def with_app_name(conninfo)
|
|
42
|
+
return conninfo if conninfo.include?("application_name")
|
|
43
|
+
|
|
44
|
+
if conninfo.include?("://")
|
|
45
|
+
separator = conninfo.include?("?") ? "&" : "?"
|
|
46
|
+
"#{conninfo}#{separator}fallback_application_name=#{APP_NAME}"
|
|
47
|
+
else
|
|
48
|
+
"#{conninfo} fallback_application_name=#{APP_NAME}".strip
|
|
49
|
+
end
|
|
50
|
+
end
|
|
51
|
+
|
|
31
52
|
def connect_from_hash(opts)
|
|
32
53
|
variables = opts[:variables]
|
|
33
|
-
conn = ::PG.connect(**opts.except(:variables))
|
|
54
|
+
conn = ::PG.connect(**with_app_name_hash(opts.except(:variables)))
|
|
34
55
|
begin
|
|
35
56
|
variables&.each { |name, value| conn.exec("SET #{name} = '#{value}'") }
|
|
36
57
|
rescue StandardError
|
|
@@ -44,6 +65,12 @@ module Pgbus
|
|
|
44
65
|
conn
|
|
45
66
|
end
|
|
46
67
|
|
|
68
|
+
def with_app_name_hash(opts)
|
|
69
|
+
return opts if opts.key?(:application_name) || opts.key?(:fallback_application_name)
|
|
70
|
+
|
|
71
|
+
opts.merge(fallback_application_name: APP_NAME)
|
|
72
|
+
end
|
|
73
|
+
|
|
47
74
|
def close_quietly(conn)
|
|
48
75
|
conn.close
|
|
49
76
|
rescue StandardError
|
data/lib/pgbus/doctor.rb
CHANGED
|
@@ -5,7 +5,7 @@ require "pgbus/mcp/health_analyzer"
|
|
|
5
5
|
|
|
6
6
|
module Pgbus
|
|
7
7
|
# Preflight diagnostics for a pgbus deployment — the single command that
|
|
8
|
-
# answers "is this environment healthy enough to run?". Runs
|
|
8
|
+
# answers "is this environment healthy enough to run?". Runs eleven checks and
|
|
9
9
|
# returns a machine-readable result plus a human report, so `pgbus doctor`
|
|
10
10
|
# and `rake pgbus:doctor` can gate a deploy or CI run (exit 0 on success,
|
|
11
11
|
# 1 on any failure).
|
|
@@ -40,7 +40,8 @@ module Pgbus
|
|
|
40
40
|
"GlobalID allowlist" => :check_allowed_global_id_models,
|
|
41
41
|
"Broadcast queue" => :check_broadcast_queue,
|
|
42
42
|
"Primary affinity" => :check_primary,
|
|
43
|
-
"Dedicated connections" => :check_dedicated_connections
|
|
43
|
+
"Dedicated connections" => :check_dedicated_connections,
|
|
44
|
+
"Connection budget" => :check_connection_budget
|
|
44
45
|
}.freeze
|
|
45
46
|
|
|
46
47
|
# Process liveness reads the pgbus_processes table (via HealthAnalyzer), so
|
|
@@ -366,6 +367,50 @@ module Pgbus
|
|
|
366
367
|
Check.new(name: "Dedicated connections", status: :fail, detail: "#{e.class}: #{e.message}")
|
|
367
368
|
end
|
|
368
369
|
|
|
370
|
+
# How many direct LISTEN connections this config pins at steady state
|
|
371
|
+
# (issue #381) — informational (always :ok) so operators can do capacity
|
|
372
|
+
# math on the pooler's direct-connection budget from the doctor output.
|
|
373
|
+
# Under :supervisor scope the whole host shares 1; under :fork it is one
|
|
374
|
+
# per worker/consumer fork. Streams add one per web-server process, which
|
|
375
|
+
# the doctor cannot count from here, so it is reported as a clause.
|
|
376
|
+
def check_connection_budget
|
|
377
|
+
capsules = @config.role_enabled?(:workers) ? Array(@config.workers).size : 0
|
|
378
|
+
consumers = @config.role_enabled?(:consumers) ? Array(@config.event_consumers).size : 0
|
|
379
|
+
|
|
380
|
+
count =
|
|
381
|
+
if !@config.worker_notify_wakeup?
|
|
382
|
+
0
|
|
383
|
+
elsif @config.worker_notify_scope == :supervisor
|
|
384
|
+
(capsules + consumers).positive? ? 1 : 0
|
|
385
|
+
else
|
|
386
|
+
capsules + consumers
|
|
387
|
+
end
|
|
388
|
+
|
|
389
|
+
detail = format(
|
|
390
|
+
"%<count>d direct LISTEN connection%<plural>s pinned (scope=%<scope>s; " \
|
|
391
|
+
"%<capsules>d capsule%<cap_plural>s + %<consumers>d consumer%<con_plural>s%<share>s)",
|
|
392
|
+
count: count, plural: count == 1 ? "" : "s", scope: @config.worker_notify_scope,
|
|
393
|
+
capsules: capsules, cap_plural: capsules == 1 ? "" : "s",
|
|
394
|
+
consumers: consumers, con_plural: consumers == 1 ? "" : "s",
|
|
395
|
+
share: count == 1 && @config.worker_notify_scope == :supervisor ? " share it" : ""
|
|
396
|
+
)
|
|
397
|
+
detail += streams_budget_clause if @config.streams_enabled
|
|
398
|
+
Check.new(name: "Connection budget", status: :ok, detail: detail)
|
|
399
|
+
rescue StandardError => e
|
|
400
|
+
Check.new(name: "Connection budget", status: :warn, detail: "#{e.class}: #{e.message}")
|
|
401
|
+
end
|
|
402
|
+
|
|
403
|
+
# Streams add their own LISTEN footprint on web hosts: one per host with
|
|
404
|
+
# the master hub (#382, the default — workers fall back per-worker only
|
|
405
|
+
# during a hub outage), one per web process under :process scope.
|
|
406
|
+
def streams_budget_clause
|
|
407
|
+
if @config.streams_listen_scope == :master
|
|
408
|
+
" + 1 per web host (streams master hub; per-worker fallback during a hub outage costs 1 per web process)"
|
|
409
|
+
else
|
|
410
|
+
" + 1 per web-server process (streams)"
|
|
411
|
+
end
|
|
412
|
+
end
|
|
413
|
+
|
|
369
414
|
# Open one dedicated connection the way the runtime does, verify it
|
|
370
415
|
# answers, close it. Returns nil on success, "label: error" on failure.
|
|
371
416
|
def probe_dedicated_connection(label, opts)
|
|
@@ -80,6 +80,28 @@ module Pgbus
|
|
|
80
80
|
@mutex.synchronize { @subscribers.clear }
|
|
81
81
|
end
|
|
82
82
|
|
|
83
|
+
# Logical queue names a consumer subscribed to +topics+ reads from — the
|
|
84
|
+
# derivation Consumer#setup_subscriptions uses, exposed here so the
|
|
85
|
+
# supervisor-owned NotifyHub (issue #381) computes the same LISTEN set
|
|
86
|
+
# the consumer forks actually read. The overlap check is deliberately
|
|
87
|
+
# coarse: any topic filter ending in "#" claims every subscriber (read
|
|
88
|
+
# more queues rather than risk an uncovered subscriber).
|
|
89
|
+
def queue_names_for_topics(topics)
|
|
90
|
+
subscribers = @mutex.synchronize { @subscribers.dup }
|
|
91
|
+
subscribers
|
|
92
|
+
.select { |s| topics.any? { |t| pattern_overlaps?(t, s.pattern) } }
|
|
93
|
+
.map(&:queue_name)
|
|
94
|
+
.uniq
|
|
95
|
+
end
|
|
96
|
+
|
|
97
|
+
# Preserved verbatim from Consumer#pattern_overlaps?: true when either
|
|
98
|
+
# side is a superset of the other by the cheap prefix/suffix rules.
|
|
99
|
+
def pattern_overlaps?(topic_filter, subscription_pattern)
|
|
100
|
+
topic_filter == subscription_pattern ||
|
|
101
|
+
topic_filter.end_with?("#") ||
|
|
102
|
+
subscription_pattern.start_with?(topic_filter.delete_suffix(".#"))
|
|
103
|
+
end
|
|
104
|
+
|
|
83
105
|
private
|
|
84
106
|
|
|
85
107
|
# Rake task-name prefixes during which pgbus must NOT open a PGMQ
|
|
@@ -14,6 +14,9 @@ module Pgbus
|
|
|
14
14
|
# notify_retry_at is writable so a test can re-arm the backoff window
|
|
15
15
|
# between successive ensure_notify_listener calls.
|
|
16
16
|
attr_accessor :notify_listener, :notify_retry_at
|
|
17
|
+
# Supervisor-mediated wake source (issue #381): non-nil iff the
|
|
18
|
+
# supervisor forked us with a wake pipe. Readable as a test seam.
|
|
19
|
+
attr_reader :wake_pipe
|
|
17
20
|
# stat_buffer is writable so a test can swap in a buffer double after
|
|
18
21
|
# construction and assert graceful_shutdown / check_recycle / shutdown flush
|
|
19
22
|
# it (mirrors Worker#stat_buffer).
|
|
@@ -57,7 +60,7 @@ module Pgbus
|
|
|
57
60
|
queue_names: nil, liveness_pipe: nil, stat_buffer: :default,
|
|
58
61
|
notify_listener: nil, notify_retry_at: 0.0,
|
|
59
62
|
notify_retry_backoff: NOTIFY_RETRY_BASE_SECONDS,
|
|
60
|
-
started_at_monotonic: nil)
|
|
63
|
+
started_at_monotonic: nil, wake_pipe: nil)
|
|
61
64
|
@topics = Array(topics)
|
|
62
65
|
@threads = threads
|
|
63
66
|
@config = config
|
|
@@ -96,6 +99,9 @@ module Pgbus
|
|
|
96
99
|
# supervisor forked us with one. Written from stamp_loop_tick so the
|
|
97
100
|
# watchdog can detect a wedged consumer even when the database is down.
|
|
98
101
|
@liveness_pipe = liveness_pipe
|
|
102
|
+
# Supervisor wake pipe (read end): when present the fork owns NO
|
|
103
|
+
# LISTEN connection — wakes and hub health arrive as bytes (issue #381).
|
|
104
|
+
@wake_pipe = wake_pipe ? WakePipe.new(wake_pipe, wake_signal: @wake_signal) : nil
|
|
99
105
|
end
|
|
100
106
|
|
|
101
107
|
# The last wall-clock loop-tick stamp (Time.now.to_f) fed to the
|
|
@@ -110,7 +116,7 @@ module Pgbus
|
|
|
110
116
|
setup_signals
|
|
111
117
|
start_heartbeat
|
|
112
118
|
setup_subscriptions
|
|
113
|
-
|
|
119
|
+
start_wake_source
|
|
114
120
|
Pgbus.logger.info do
|
|
115
121
|
"[Pgbus] Consumer started: topics=#{topics.join(",")} threads=#{threads} " \
|
|
116
122
|
"notify_wakeup=#{notify_wakeup?} pid=#{::Process.pid}"
|
|
@@ -150,10 +156,13 @@ module Pgbus
|
|
|
150
156
|
private
|
|
151
157
|
|
|
152
158
|
def setup_subscriptions
|
|
153
|
-
|
|
154
|
-
|
|
155
|
-
|
|
156
|
-
|
|
159
|
+
# An injected queue_names: seed (test seam — the ctor documents nil as
|
|
160
|
+
# "derive from the registry") survives #run. Derivation is shared with
|
|
161
|
+
# the supervisor-owned NotifyHub (issue #381) so the LISTEN union
|
|
162
|
+
# covers exactly the queues this fork reads.
|
|
163
|
+
return unless @queue_names.nil?
|
|
164
|
+
|
|
165
|
+
@queue_names = @registry.queue_names_for_topics(topics)
|
|
157
166
|
end
|
|
158
167
|
|
|
159
168
|
def consume
|
|
@@ -273,13 +282,6 @@ module Pgbus
|
|
|
273
282
|
end
|
|
274
283
|
end
|
|
275
284
|
|
|
276
|
-
def pattern_overlaps?(topic_filter, subscription_pattern)
|
|
277
|
-
# Simple check: if either is a subset of the other
|
|
278
|
-
topic_filter == subscription_pattern ||
|
|
279
|
-
topic_filter.end_with?("#") ||
|
|
280
|
-
subscription_pattern.start_with?(topic_filter.delete_suffix(".#"))
|
|
281
|
-
end
|
|
282
|
-
|
|
283
285
|
# Signal the loop to exit cleanly once a recycle limit is hit. The clean
|
|
284
286
|
# exit gets an immediate supervisor restart (supervisor.rb:305-307), so a
|
|
285
287
|
# fresh fork replaces this one before its memory grows unbounded — the
|
|
@@ -351,21 +353,49 @@ module Pgbus
|
|
|
351
353
|
end
|
|
352
354
|
|
|
353
355
|
def wake_timeout
|
|
354
|
-
# A dead listener (running? false) will never wake the loop, so treat
|
|
355
|
-
# as absent and keep polling at the short interval until
|
|
356
|
-
# ensure_notify_listener restarts it.
|
|
357
|
-
#
|
|
358
|
-
|
|
356
|
+
# A dead listener (running? false) will never wake the loop, so treat
|
|
357
|
+
# it as absent and keep polling at the short interval until
|
|
358
|
+
# ensure_notify_listener restarts it. A live-but-deaf listener
|
|
359
|
+
# (delivering? false) is the same story (issue #332 — parity with
|
|
360
|
+
# Worker#wake_timeout). Under :supervisor scope the hub's H/P
|
|
361
|
+
# broadcasts (via WakePipe) carry the same signal. Only a live,
|
|
362
|
+
# delivering wake source earns the long NOTIFY-mode ceiling.
|
|
363
|
+
return config.polling_interval unless notify_wakeup? && wake_source_delivering?
|
|
359
364
|
|
|
360
365
|
[config.polling_interval, NOTIFY_FALLBACK_POLL_SECONDS].max
|
|
361
366
|
end
|
|
362
367
|
|
|
368
|
+
def wake_source_delivering?
|
|
369
|
+
return @wake_pipe.delivering? if @wake_pipe
|
|
370
|
+
|
|
371
|
+
@notify_listener&.running? && @notify_listener.delivering?
|
|
372
|
+
end
|
|
373
|
+
|
|
374
|
+
# :supervisor scope: the fork opens NO LISTEN connection; the WakePipe
|
|
375
|
+
# watcher is the wake source, and a missing pipe (hub failed to start)
|
|
376
|
+
# means plain polling — never a local listener (see Worker's twin).
|
|
377
|
+
# :fork scope: the fork-local NotifyListener.
|
|
378
|
+
def start_wake_source
|
|
379
|
+
return @wake_pipe.start if @wake_pipe
|
|
380
|
+
|
|
381
|
+
start_notify_listener if local_listener_scope?
|
|
382
|
+
end
|
|
383
|
+
|
|
384
|
+
def local_listener_scope?
|
|
385
|
+
config.worker_notify_scope == :fork
|
|
386
|
+
end
|
|
387
|
+
|
|
388
|
+
def stop_wake_source
|
|
389
|
+
@wake_pipe&.stop
|
|
390
|
+
@notify_listener&.stop
|
|
391
|
+
end
|
|
392
|
+
|
|
363
393
|
def start_notify_listener
|
|
364
394
|
return unless notify_wakeup?
|
|
365
395
|
|
|
366
396
|
@notify_listener = NotifyListener.new(
|
|
367
397
|
physical_queues: physical_queue_names,
|
|
368
|
-
on_wake: -> { @wake_signal.notify! },
|
|
398
|
+
on_wake: ->(_channel) { @wake_signal.notify! },
|
|
369
399
|
connection_options: config.worker_notify_connection_options,
|
|
370
400
|
health_check_ms: (config.polling_interval * 1000).to_i.clamp(250, 5_000),
|
|
371
401
|
logger: Pgbus.logger
|
|
@@ -382,6 +412,10 @@ module Pgbus
|
|
|
382
412
|
# a persistent outage retries on 5s→…→300s intervals, not every tick
|
|
383
413
|
# (mirrors Worker#ensure_notify_listener).
|
|
384
414
|
def ensure_notify_listener
|
|
415
|
+
# Supervisor scope: self-healing is the hub's job (once per host);
|
|
416
|
+
# a pipe-less fork under that scope stays on plain polling.
|
|
417
|
+
return if @wake_pipe
|
|
418
|
+
return unless local_listener_scope?
|
|
385
419
|
return unless notify_wakeup?
|
|
386
420
|
return if @notify_listener&.running?
|
|
387
421
|
return if monotonic_now < @notify_retry_at
|
|
@@ -409,9 +443,10 @@ module Pgbus
|
|
|
409
443
|
@notify_listener = nil
|
|
410
444
|
end
|
|
411
445
|
|
|
446
|
+
# Through config.queue_name so normalized subscriber queue names LISTEN
|
|
447
|
+
# on the channel their table actually notifies (see Worker's twin).
|
|
412
448
|
def physical_queue_names
|
|
413
|
-
|
|
414
|
-
@queue_names.map { |q| "#{prefix}#{q}" }
|
|
449
|
+
@queue_names.map { |q| config.queue_name(q) }
|
|
415
450
|
end
|
|
416
451
|
|
|
417
452
|
def start_heartbeat
|
|
@@ -443,7 +478,7 @@ module Pgbus
|
|
|
443
478
|
end
|
|
444
479
|
|
|
445
480
|
def shutdown
|
|
446
|
-
|
|
481
|
+
stop_wake_source
|
|
447
482
|
@pool.shutdown
|
|
448
483
|
@pool.wait_for_termination(30)
|
|
449
484
|
@stat_buffer&.stop
|