pgbus 0.12.3 → 0.13.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +9 -0
- data/README.md +2 -1
- data/Rakefile +12 -1
- data/lib/pgbus/client.rb +15 -0
- data/lib/pgbus/configuration/capsule_dsl.rb +8 -0
- data/lib/pgbus/configuration.rb +36 -5
- data/lib/pgbus/dedicated_connection.rb +29 -2
- data/lib/pgbus/doctor.rb +36 -2
- data/lib/pgbus/event_bus/registry.rb +22 -0
- data/lib/pgbus/process/consumer.rb +57 -22
- data/lib/pgbus/process/notify_hub.rb +273 -0
- data/lib/pgbus/process/notify_listener.rb +103 -26
- data/lib/pgbus/process/supervisor.rb +146 -34
- data/lib/pgbus/process/wake_pipe.rb +129 -0
- data/lib/pgbus/process/wildcard_queue_resolver.rb +35 -0
- data/lib/pgbus/process/worker.rb +58 -35
- data/lib/pgbus/version.rb +1 -1
- data/lib/pgbus/web/streamer/listener.rb +49 -30
- metadata +4 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: '0589d2d6e69eb8b2ba5863cb8c8e2779bca29528030410508e1930fcfbd0d774'
|
|
4
|
+
data.tar.gz: c491442cd7b1d2bdccc5ff95153dc31abb424111906e17872b6b112830a1cde7
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 63e8480dcd7efb70bd7933cba734c23d7baa6f9e50ff658b575720da5ccc53e43fde1e4aca0fc7c30b6715448d040c1bb2fff472f5ac1686ae3e62853beb0bdd
|
|
7
|
+
data.tar.gz: 186da9576934f14264e9d9989c1a035d1c725e8283d1a735692da90f1610f4c538776b558a526d0c536466f92e405ffcbf99fd5bf229f9bf9b92dd1e0d7ec876
|
data/CHANGELOG.md
CHANGED
|
@@ -1,11 +1,20 @@
|
|
|
1
1
|
## [Unreleased]
|
|
2
2
|
|
|
3
|
+
### Added
|
|
4
|
+
|
|
5
|
+
- **Host-level shared LISTEN: `worker_notify_scope` — the supervisor now owns ONE direct LISTEN connection for the whole host (issue #381).** ⚠️ **Default behavior change.** Previously every worker fork and every consumer fork opened its own dedicated LISTEN connection (`NotifyListener`), so a host's direct-connection footprint scaled with fork count — on transaction-pool PgBouncer platforms those connections come out of the scarcest slice of `max_connections`, and a 5-capsule + 2-consumer host pinned 7. Under the new default (`config.worker_notify_scope = :supervisor`) the supervisor runs a single `NotifyHub`: one `NotifyListener` on the union of every capsule's and consumer's queue channels (wildcards via the shared resolver, consumer sets via the registry), fanning wakes out to forks over per-fork pipes (`W` wake / `H` healthy / `P` degraded bytes; a fork whose pipe reports degraded or reaches EOF falls back to fast polling exactly like a failed local listener). Footprint drops to **1 direct LISTEN connection per job host**, verified by integration test: routing is per-fork (an insert wakes only the forks reading that queue, wildcard capsules unconditionally), and `pg_terminate_backend` on the shared connection is survived — reconnect, re-LISTEN, wakes flow again. **Rollback:** `config.worker_notify_scope = :fork` restores the previous per-fork listeners byte-for-byte. Dedicated LISTEN connections are now census-tagged `application_name=pgbus-listen` so `pg_stat_activity` can count them. Refs #381.
|
|
6
|
+
- **`pgbus doctor`: new "Connection budget" check (issue #381).** Prints how many direct LISTEN connections the current config pins — 1 per host under `:supervisor` scope, capsules + consumers under `:fork` (honoring `config.roles`), plus a "+1 per web-server process (streams)" clause — so operators can do pooler capacity math from the doctor output alone. Informational, always `:ok`. Refs #381.
|
|
7
|
+
- **Benchmarks: `rake bench:notify_wake` and `rake bench:notify_chaos` (issue #381).** Wake-path latency (send → wake, p50/p95/p99, direct vs hub-mediated), empty-read cost, LISTEN connection census, and failure-mode measurements (killed LISTEN backend, wedged fork, FD churn, fan-out cost). Refs #381.
|
|
8
|
+
|
|
3
9
|
### Changed
|
|
4
10
|
|
|
11
|
+
- **Multi-queue strict-priority contract documented and pinned (issue #381).** The capsule DSL's "list order = strict priority" promise rides on Postgres's Append node filling `read_multi`'s `UNION ALL … LIMIT` from earlier-listed subqueries first — incidental behavior nothing upstream promises. The contract (and the vt-claim caveat: subquery rows discarded by the outer LIMIT are still claimed and go invisible for one visibility timeout) is now documented on `Client#read_multi` and the capsule DSL, and pinned by an integration canary (`spec/integration/multi_queue_priority_spec.rb`) so a planner or pgmq-ruby change breaks loudly instead of silently. `Consumer#wake_timeout` also gained the live-but-deaf listener consult (`delivering?`) for parity with Worker (issue #332). Refs #381.
|
|
12
|
+
|
|
5
13
|
- **README: doctor table and configuration reference catch up to the shipped surface (issue #369).** The doctor section still said "six checks" and listed only the original six; `Pgbus::Doctor::CHECKS` has 10 (GlobalID allowlist, Broadcast queue, Primary affinity, Dedicated connections were missing — the first two are the ones most likely to warn after upgrade on nil defaults). The configuration reference table was missing ~20 shipped options operators actually hit: `allowed_global_id_models` (incl. `[]` = deny-all), streams GC (`streams_orphan_threshold` / `streams_orphan_sweep_interval` — 24h / hourly defaults that auto-drop dormant durable stream queues), retention/durable/presence/broadcast-queue knobs, `streams_pool_*` / LISTEN connection overrides, `doctor_on_boot`, `require_primary`, `connects_to`, `zombie_detection`, `connection_guc_mode`, `worker_notify_*`, `group_mode`, and neighbors. A unit drift guard pins the README doctor table to every `Doctor::CHECKS` name. Full typed reference remains on the docs site. Refs #369.
|
|
6
14
|
|
|
7
15
|
### Fixed
|
|
8
16
|
|
|
17
|
+
- **SIGTERM no longer segfaults the process: both LISTEN listeners stopped closing their PG connection from the stopping thread (issue #375).** `Process::NotifyListener#stop` (supervisor thread, on SIGTERM) closed the listener's dedicated `PG::Connection` to interrupt the blocking `wait_for_notify`, but left `@conn` pointing at it — so the listener thread unwound into `run_loop`'s `ensure` and ran `safe_unlisten_all`, exec'ing `UNLISTEN` on a connection the other thread was concurrently freeing. `PG::Connection#close` is `PQfinish`: it frees the PGconn **and its OpenSSL objects**, so `PQsendQuery` walked into freed TLS state and the whole worker died with `[BUG] Segmentation fault` — reproducibly, on essentially every container-replacement deploy against a TLS Postgres, sometimes in several forked workers at once. `rescue PG::Error` cannot catch a C-level SEGV, so the "safe" in `safe_unlisten_all` never held. **The connection is now single-owner**: the listener thread is the only thread that may exec, wait, or close on it, from build through teardown. `#stop` signals by clearing `@running` and joining — nothing more — and the listener thread closes its own connection in `run_loop`'s `ensure`. The stop is observed within one `wait_for_notify` timeout (`health_check_ms`, **250 ms** in the default configuration since workers/consumers derive it from `polling_interval`), so `#stop`'s join budget is now `health_check_ms + 5s` instead of a flat 5 s — a flat timeout could expire before a listener with a large `health_check_ms` had even one chance to observe the stop. The teardown `UNLISTEN` round-trip is **dropped entirely** rather than merely made safe: it ran immediately before the close, and closing a session deregisters every `LISTEN` server-side, so it bought nothing at any time. The identical defect in `Web::Streamer::Listener` — same close-from-`#stop`, same `UNLISTEN`-in-`ensure`, and there without even a mutex around `@conn` — is fixed the same way, so the dashboard's Puma worker stops crashing on the same deploy; its listener thread now also closes its own connection (previously only `#stop` did). Cost: shutdown can take up to one health-check cycle longer per listener — measured against a real PostgreSQL at the default `health_check_ms` of 250: **~207 ms on a fully idle queue, ~5 ms when the queue has any NOTIFY traffic** (a notification returns the wait immediately, and the loop then sees the cleared flag). 40 start/stop cycles under concurrent NOTIFY load leaked zero `LISTEN` backends. Thanks to the reporter for the crash dumps and the root-cause analysis. Refs #375.
|
|
9
18
|
- **The health verdict no longer emits false STALLED reports — `max_read_ct` was never populated, and the wedge signal counted queues no capsule drains (issue #367).** Two correctness defects in `Pgbus::MCP::HealthAnalyzer` (surfaced through `pgbus doctor` / `pgbus_health` / the MCP health tool). **(1)** The `all_unread?` wedge check read `:max_read_ct`, but the metrics query never selected it — so the guard always degenerated to "never claimed" and a busy-but-healthy queue caught mid-burst produced a STALLED verdict with a factually wrong "read_ct=0 (never claimed)" reason. The metrics query now exposes a per-queue **`visible_unread_length`** (`count(*) WHERE vt <= NOW() AND read_ct = 0`) and the analyzer keys the wedge off *visible, never-claimed* messages. Counting per visible message (not `max(read_ct)` over the whole table) means one retried message left in-queue after its backoff, or an in-flight message claimed by a peer, can no longer veto the signal for a pile of genuinely-unclaimed jobs. **(2)** The verdict reasoned about **every** non-DLQ/non-stream queue against the global worker fleet — but a worker can only claim from queues its capsule subscribes to, so "M workers alive but never claimed" was vacuous for a queue nobody drains (ad-hoc queues, unregistered stream queues — see #366). The analyzer now intersects the STALLED backlog with `Web::DataSource#drained_queue_names` (each configured capsule's queues, priority `_pN` sub-tables expanded via the client's queue strategy, unioned with EventBus handler queues; `nil` for a `*` wildcard = drains everything, fail-open on error). Queues nobody drains get their own DEGRADED signal — *"N queue(s) hold messages but no capsule is configured to drain them"* — instead of being folded into the worker-wedge verdict. A heart-beating-but-`:stalled` worker is still reported STALLED regardless of which queue holds the backlog. Reason strings now truncate to the first 10 queue names with `(+N more)` so a flagged fleet of hundreds of queues doesn't produce a multi-KB log line. Refs #367, #366.
|
|
10
19
|
- **`allowed_global_id_models` now actually guards ActiveJob arguments, not only EventBus payloads (issue #368).** The doctor warned in production that `nil` means "allow-all GlobalID arguments", but the allowlist was only enforced in `Serializer.locate_global_id` — reached from EventBus `_global_id` payloads — while the ordinary job path (`Executor` → `ActiveJob::Base.deserialize` → Rails' unrestricted `GlobalID::Locator`) never checked it. Operators who set an allowlist after following the doctor had a false sense of security; the common `SomeJob.perform_later(record)` pattern was unguarded. Job deserialization now goes through `Serializer.deserialize_job_data`, which walks `_aj_globalid` keys (including nested arrays/hashes) and reuses the same gate as EventBus when the allowlist is set; `nil` remains zero-cost allow-all. Rejected models raise `Pgbus::SerializationError` and are treated as a normal job failure. Apps with ActiveStorage attachments should include `ActiveStorage::Blob` (and related models) on the allowlist. Docs + doctor copy updated. Refs #368.
|
|
11
20
|
- **Dormant pre-registry stream queues no longer false-STALL the doctor or escape the orphan sweep (issue #366).** Stream queues created before the `pgbus_stream_queues` migration only register on their next broadcast — completed checkout-style flows, ended chats, and one-shot progress streams never broadcast again, so they stay unregistered forever. Health/doctor then treated their permanent `read_ct=0` visible backlog as a worker wedge (STALLED on ~hundreds of healthy stream queues), and the orphan sweep skipped them because it only iterated the registry. Detection is fingerprint-based (the archive `a_<queue>_msg_id_idx` index that only `ensure_stream_queue` creates — no name heuristics). `StreamQueue.known_names` = registry ∪ fingerprints, used by health exclusion, orphan sweep / stream-archive prune, and wildcard workers; `rake pgbus:streams:backfill_registry` persists the missing rows; doctor surfaces a DEGRADED hint pointing at the backfill when unregistered fingerprints remain; `add_stream_queues` generator post-install mentions the task. Refs #366.
|
data/README.md
CHANGED
|
@@ -1862,7 +1862,7 @@ pgbus help # Show help
|
|
|
1862
1862
|
|
|
1863
1863
|
#### pgbus doctor
|
|
1864
1864
|
|
|
1865
|
-
A single preflight command that answers "is this environment healthy enough to run?" — useful as a deploy or CI gate. It runs
|
|
1865
|
+
A single preflight command that answers "is this environment healthy enough to run?" — useful as a deploy or CI gate. It runs 11 checks and never raises; a broken environment turns every probe into a failed/warned check instead of a crash:
|
|
1866
1866
|
|
|
1867
1867
|
| Check | Fails (`:fail`) when | Warns (`:warn`) when |
|
|
1868
1868
|
|---|---|---|
|
|
@@ -1876,6 +1876,7 @@ A single preflight command that answers "is this environment healthy enough to r
|
|
|
1876
1876
|
| Broadcast queue | — | Turbo broadcasts share the default queue in production, or `streams_broadcast_queue` is set but no worker capsule drains it |
|
|
1877
1877
|
| Primary affinity | — | Job connection is on a read-only replica (`pg_is_in_recovery`) — a read/write-splitting pooler may be stalling jobs |
|
|
1878
1878
|
| Dedicated connections | Streamer LISTEN and/or worker notify dedicated path cannot connect | — |
|
|
1879
|
+
| Connection budget | — (informational: prints how many direct LISTEN connections the current config pins — 1 per host under `worker_notify_scope: :supervisor`, one per fork under `:fork`, plus 1 per web process when streams are enabled) | — |
|
|
1879
1880
|
|
|
1880
1881
|
```bash
|
|
1881
1882
|
pgbus doctor # prints the report; exit 1 unless every check passed
|
data/Rakefile
CHANGED
|
@@ -24,7 +24,8 @@ namespace :bench do
|
|
|
24
24
|
# Benches that need a real PostgreSQL/PGMQ (or boot Puma) — excluded from the
|
|
25
25
|
# no-DB unit suite that bench:all runs in CI.
|
|
26
26
|
db_benches = %w[connection_pool_bench integration_bench streams_bench streams_read_pool_bench
|
|
27
|
-
execution_modes_bench pool_swap_bench pool_autoscale_bench job_burst_bench
|
|
27
|
+
execution_modes_bench pool_swap_bench pool_autoscale_bench job_burst_bench
|
|
28
|
+
notify_wake_bench notify_chaos_bench].freeze
|
|
28
29
|
# The unit suite is every *_bench.rb that doesn't need a database, derived
|
|
29
30
|
# from the directory so a new unit bench is picked up automatically (kept in
|
|
30
31
|
# sync with bench:one, which globs the same files).
|
|
@@ -84,6 +85,16 @@ namespace :bench do
|
|
|
84
85
|
ruby "benchmarks/job_burst_bench.rb"
|
|
85
86
|
end
|
|
86
87
|
|
|
88
|
+
desc "Run NOTIFY wake-path benchmark (#381 wake latency + connection census; requires PGBUS_DATABASE_URL)"
|
|
89
|
+
task :notify_wake do
|
|
90
|
+
ruby "benchmarks/notify_wake_bench.rb"
|
|
91
|
+
end
|
|
92
|
+
|
|
93
|
+
desc "Run NotifyHub failure-mode measurements (#381 chaos scenarios; requires PGBUS_DATABASE_URL)"
|
|
94
|
+
task :notify_chaos do
|
|
95
|
+
ruby "benchmarks/notify_chaos_bench.rb"
|
|
96
|
+
end
|
|
97
|
+
|
|
87
98
|
desc "Run a single benchmark: rake bench:one[client_bench]"
|
|
88
99
|
task :one, [:name] do |_t, args|
|
|
89
100
|
name = args[:name] or abort "Usage: rake bench:one[serialization_bench|client_bench|...]"
|
data/lib/pgbus/client.rb
CHANGED
|
@@ -434,6 +434,21 @@ module Pgbus
|
|
|
434
434
|
# caller receives up to `queue_count * qty` messages. Pass `limit:` to cap
|
|
435
435
|
# the total across all queues — required when feeding a fixed-size pool,
|
|
436
436
|
# otherwise the pool can overflow on multi-queue reads (issue #123).
|
|
437
|
+
#
|
|
438
|
+
# STRICT-PRIORITY CONTRACT (issue #381): when `limit:` is smaller than the
|
|
439
|
+
# total available, earlier-listed queues win — the capsule DSL's "list
|
|
440
|
+
# order = strict priority" promise rides on this. The mechanism is
|
|
441
|
+
# incidental: pgmq-ruby builds `pgmq.read(q1) UNION ALL pgmq.read(q2) …
|
|
442
|
+
# LIMIT n`, and Postgres's Append node fills the LIMIT from the subqueries
|
|
443
|
+
# in written order. Nothing upstream promises that, so the contract is
|
|
444
|
+
# pinned by spec/integration/multi_queue_priority_spec.rb — if that canary
|
|
445
|
+
# ever breaks, switch callers to ordered per-queue reads (the
|
|
446
|
+
# Worker#fetch_prioritized pattern) instead of relying on this method.
|
|
447
|
+
#
|
|
448
|
+
# vt-claim caveat: each subquery may claim (set vt on) up to `qty` rows
|
|
449
|
+
# even when the outer LIMIT discards them — a discarded row goes invisible
|
|
450
|
+
# for one visibility timeout without being processed. Size `qty`/`limit`
|
|
451
|
+
# accordingly on latency-sensitive queues.
|
|
437
452
|
def read_multi(queue_names, qty:, vt: nil, limit: nil)
|
|
438
453
|
full_names = queue_names.map { |q| config.queue_name(q) }
|
|
439
454
|
guarded_read do
|
|
@@ -19,6 +19,14 @@ module Pgbus
|
|
|
19
19
|
# * wildcard, matches all queues
|
|
20
20
|
# *_ trailing wildcard, prefix match (e.g. "staging_*")
|
|
21
21
|
#
|
|
22
|
+
# "List order = strict priority" is enforced two ways at runtime: with
|
|
23
|
+
# priority_levels > 1 the worker reads queues one at a time in list order
|
|
24
|
+
# (Worker#fetch_prioritized); otherwise it relies on Client#read_multi's
|
|
25
|
+
# UNION ALL filling its LIMIT from earlier-listed queues first — an
|
|
26
|
+
# Append-node behavior Postgres does not formally promise, pinned by
|
|
27
|
+
# spec/integration/multi_queue_priority_spec.rb (see Client#read_multi
|
|
28
|
+
# for the contract and its vt-claim caveat).
|
|
29
|
+
#
|
|
22
30
|
# Returns +Array<Hash>+ in the same shape as the legacy +workers:+ array,
|
|
23
31
|
# so the rest of the codebase can consume it without changes:
|
|
24
32
|
#
|
data/lib/pgbus/configuration.rb
CHANGED
|
@@ -179,14 +179,22 @@ module Pgbus
|
|
|
179
179
|
:streams_pool_autoscale_interval, :streams_application_name
|
|
180
180
|
attr_reader :streams_default_broadcast_mode # rubocop:disable Style/AccessorGrouping
|
|
181
181
|
|
|
182
|
-
# NOTIFY-gated worker wakeups. When true,
|
|
183
|
-
#
|
|
184
|
-
#
|
|
185
|
-
#
|
|
186
|
-
#
|
|
182
|
+
# NOTIFY-gated worker wakeups. When true, workers are woken by a
|
|
183
|
+
# NotifyListener PG connection that LISTENs on their queues' INSERT
|
|
184
|
+
# channels instead of blind-polling. Defaults to the value of
|
|
185
|
+
# listen_notify. The worker_notify_* overrides mirror streams_* so the
|
|
186
|
+
# LISTEN connection can bypass PgBouncer.
|
|
187
187
|
attr_accessor :worker_notify_wakeup,
|
|
188
188
|
:worker_notify_host, :worker_notify_port, :worker_notify_database_url
|
|
189
189
|
|
|
190
|
+
# Where the LISTEN connection lives (issue #381):
|
|
191
|
+
# :supervisor (default) — ONE shared NotifyListener in the supervisor
|
|
192
|
+
# process for the whole host; forks are woken over per-fork pipes.
|
|
193
|
+
# Direct-connection footprint: 1, regardless of capsule/consumer count.
|
|
194
|
+
# :fork — one NotifyListener (and one dedicated PG connection) per
|
|
195
|
+
# worker/consumer fork: the pre-0.13 behavior, kept as an escape hatch.
|
|
196
|
+
attr_reader :worker_notify_scope
|
|
197
|
+
|
|
190
198
|
# AppSignal integration (auto-loaded when ::Appsignal is defined and this is true).
|
|
191
199
|
# Set to false to opt out without uninstalling the appsignal gem.
|
|
192
200
|
attr_accessor :appsignal_enabled, :appsignal_probe_enabled
|
|
@@ -261,6 +269,7 @@ module Pgbus
|
|
|
261
269
|
@listen_notify = true
|
|
262
270
|
|
|
263
271
|
@worker_notify_wakeup = nil
|
|
272
|
+
@worker_notify_scope = :supervisor
|
|
264
273
|
@worker_notify_host = nil
|
|
265
274
|
@worker_notify_port = nil
|
|
266
275
|
@worker_notify_database_url = nil
|
|
@@ -623,6 +632,28 @@ module Pgbus
|
|
|
623
632
|
@doctor_on_boot = coerced
|
|
624
633
|
end
|
|
625
634
|
|
|
635
|
+
VALID_WORKER_NOTIFY_SCOPES = %i[supervisor fork].freeze
|
|
636
|
+
|
|
637
|
+
# Validated at assignment time like the other enum options. A String is
|
|
638
|
+
# coerced so YAML-ish configs work.
|
|
639
|
+
def worker_notify_scope=(scope)
|
|
640
|
+
coerced = case scope
|
|
641
|
+
when Symbol then scope
|
|
642
|
+
when String then scope.to_sym
|
|
643
|
+
else
|
|
644
|
+
raise Pgbus::ConfigurationError,
|
|
645
|
+
"Invalid worker_notify_scope type: #{scope.class}. " \
|
|
646
|
+
"Must be :supervisor (one shared LISTEN connection per host) or :fork (one per fork)"
|
|
647
|
+
end
|
|
648
|
+
unless VALID_WORKER_NOTIFY_SCOPES.include?(coerced)
|
|
649
|
+
raise Pgbus::ConfigurationError,
|
|
650
|
+
"Invalid worker_notify_scope: #{coerced.inspect}. " \
|
|
651
|
+
"Must be :supervisor (one shared LISTEN connection per host) or :fork (one per fork)"
|
|
652
|
+
end
|
|
653
|
+
|
|
654
|
+
@worker_notify_scope = coerced
|
|
655
|
+
end
|
|
656
|
+
|
|
626
657
|
def validate!
|
|
627
658
|
if pool_size && !(pool_size.is_a?(Numeric) && pool_size.positive?)
|
|
628
659
|
raise Pgbus::ConfigurationError, "pool_size must be a positive number or nil (auto-tune)"
|
|
@@ -16,10 +16,17 @@ module Pgbus
|
|
|
16
16
|
module DedicatedConnection
|
|
17
17
|
module_function
|
|
18
18
|
|
|
19
|
+
# Stable census tag for every dedicated LISTEN connection, so operators can
|
|
20
|
+
# count them: SELECT count(*) FROM pg_stat_activity WHERE application_name
|
|
21
|
+
# = 'pgbus-listen' (issue #381 connection budget). Applied as
|
|
22
|
+
# fallback_application_name so an explicit application_name in the URL /
|
|
23
|
+
# hash — or PGAPPNAME in the environment — always wins.
|
|
24
|
+
APP_NAME = "pgbus-listen"
|
|
25
|
+
|
|
19
26
|
def connect(opts)
|
|
20
27
|
require "pg" unless defined?(::PG::Connection)
|
|
21
28
|
case opts
|
|
22
|
-
when String then ::PG.connect(opts)
|
|
29
|
+
when String then ::PG.connect(with_app_name(opts))
|
|
23
30
|
when Hash then connect_from_hash(opts)
|
|
24
31
|
else
|
|
25
32
|
raise Pgbus::ConfigurationError,
|
|
@@ -28,9 +35,23 @@ module Pgbus
|
|
|
28
35
|
end
|
|
29
36
|
end
|
|
30
37
|
|
|
38
|
+
# Append the census tag to a conninfo String — URL query-param style for
|
|
39
|
+
# postgres:// URLs, space-separated keyword style otherwise. A string that
|
|
40
|
+
# already mentions application_name (either keyword) is left untouched.
|
|
41
|
+
def with_app_name(conninfo)
|
|
42
|
+
return conninfo if conninfo.include?("application_name")
|
|
43
|
+
|
|
44
|
+
if conninfo.include?("://")
|
|
45
|
+
separator = conninfo.include?("?") ? "&" : "?"
|
|
46
|
+
"#{conninfo}#{separator}fallback_application_name=#{APP_NAME}"
|
|
47
|
+
else
|
|
48
|
+
"#{conninfo} fallback_application_name=#{APP_NAME}".strip
|
|
49
|
+
end
|
|
50
|
+
end
|
|
51
|
+
|
|
31
52
|
def connect_from_hash(opts)
|
|
32
53
|
variables = opts[:variables]
|
|
33
|
-
conn = ::PG.connect(**opts.except(:variables))
|
|
54
|
+
conn = ::PG.connect(**with_app_name_hash(opts.except(:variables)))
|
|
34
55
|
begin
|
|
35
56
|
variables&.each { |name, value| conn.exec("SET #{name} = '#{value}'") }
|
|
36
57
|
rescue StandardError
|
|
@@ -44,6 +65,12 @@ module Pgbus
|
|
|
44
65
|
conn
|
|
45
66
|
end
|
|
46
67
|
|
|
68
|
+
def with_app_name_hash(opts)
|
|
69
|
+
return opts if opts.key?(:application_name) || opts.key?(:fallback_application_name)
|
|
70
|
+
|
|
71
|
+
opts.merge(fallback_application_name: APP_NAME)
|
|
72
|
+
end
|
|
73
|
+
|
|
47
74
|
def close_quietly(conn)
|
|
48
75
|
conn.close
|
|
49
76
|
rescue StandardError
|
data/lib/pgbus/doctor.rb
CHANGED
|
@@ -5,7 +5,7 @@ require "pgbus/mcp/health_analyzer"
|
|
|
5
5
|
|
|
6
6
|
module Pgbus
|
|
7
7
|
# Preflight diagnostics for a pgbus deployment — the single command that
|
|
8
|
-
# answers "is this environment healthy enough to run?". Runs
|
|
8
|
+
# answers "is this environment healthy enough to run?". Runs eleven checks and
|
|
9
9
|
# returns a machine-readable result plus a human report, so `pgbus doctor`
|
|
10
10
|
# and `rake pgbus:doctor` can gate a deploy or CI run (exit 0 on success,
|
|
11
11
|
# 1 on any failure).
|
|
@@ -40,7 +40,8 @@ module Pgbus
|
|
|
40
40
|
"GlobalID allowlist" => :check_allowed_global_id_models,
|
|
41
41
|
"Broadcast queue" => :check_broadcast_queue,
|
|
42
42
|
"Primary affinity" => :check_primary,
|
|
43
|
-
"Dedicated connections" => :check_dedicated_connections
|
|
43
|
+
"Dedicated connections" => :check_dedicated_connections,
|
|
44
|
+
"Connection budget" => :check_connection_budget
|
|
44
45
|
}.freeze
|
|
45
46
|
|
|
46
47
|
# Process liveness reads the pgbus_processes table (via HealthAnalyzer), so
|
|
@@ -366,6 +367,39 @@ module Pgbus
|
|
|
366
367
|
Check.new(name: "Dedicated connections", status: :fail, detail: "#{e.class}: #{e.message}")
|
|
367
368
|
end
|
|
368
369
|
|
|
370
|
+
# How many direct LISTEN connections this config pins at steady state
|
|
371
|
+
# (issue #381) — informational (always :ok) so operators can do capacity
|
|
372
|
+
# math on the pooler's direct-connection budget from the doctor output.
|
|
373
|
+
# Under :supervisor scope the whole host shares 1; under :fork it is one
|
|
374
|
+
# per worker/consumer fork. Streams add one per web-server process, which
|
|
375
|
+
# the doctor cannot count from here, so it is reported as a clause.
|
|
376
|
+
def check_connection_budget
|
|
377
|
+
capsules = @config.role_enabled?(:workers) ? Array(@config.workers).size : 0
|
|
378
|
+
consumers = @config.role_enabled?(:consumers) ? Array(@config.event_consumers).size : 0
|
|
379
|
+
|
|
380
|
+
count =
|
|
381
|
+
if !@config.worker_notify_wakeup?
|
|
382
|
+
0
|
|
383
|
+
elsif @config.worker_notify_scope == :supervisor
|
|
384
|
+
(capsules + consumers).positive? ? 1 : 0
|
|
385
|
+
else
|
|
386
|
+
capsules + consumers
|
|
387
|
+
end
|
|
388
|
+
|
|
389
|
+
detail = format(
|
|
390
|
+
"%<count>d direct LISTEN connection%<plural>s pinned (scope=%<scope>s; " \
|
|
391
|
+
"%<capsules>d capsule%<cap_plural>s + %<consumers>d consumer%<con_plural>s%<share>s)",
|
|
392
|
+
count: count, plural: count == 1 ? "" : "s", scope: @config.worker_notify_scope,
|
|
393
|
+
capsules: capsules, cap_plural: capsules == 1 ? "" : "s",
|
|
394
|
+
consumers: consumers, con_plural: consumers == 1 ? "" : "s",
|
|
395
|
+
share: count == 1 && @config.worker_notify_scope == :supervisor ? " share it" : ""
|
|
396
|
+
)
|
|
397
|
+
detail += " + 1 per web-server process (streams)" if @config.streams_enabled
|
|
398
|
+
Check.new(name: "Connection budget", status: :ok, detail: detail)
|
|
399
|
+
rescue StandardError => e
|
|
400
|
+
Check.new(name: "Connection budget", status: :warn, detail: "#{e.class}: #{e.message}")
|
|
401
|
+
end
|
|
402
|
+
|
|
369
403
|
# Open one dedicated connection the way the runtime does, verify it
|
|
370
404
|
# answers, close it. Returns nil on success, "label: error" on failure.
|
|
371
405
|
def probe_dedicated_connection(label, opts)
|
|
@@ -80,6 +80,28 @@ module Pgbus
|
|
|
80
80
|
@mutex.synchronize { @subscribers.clear }
|
|
81
81
|
end
|
|
82
82
|
|
|
83
|
+
# Logical queue names a consumer subscribed to +topics+ reads from — the
|
|
84
|
+
# derivation Consumer#setup_subscriptions uses, exposed here so the
|
|
85
|
+
# supervisor-owned NotifyHub (issue #381) computes the same LISTEN set
|
|
86
|
+
# the consumer forks actually read. The overlap check is deliberately
|
|
87
|
+
# coarse: any topic filter ending in "#" claims every subscriber (read
|
|
88
|
+
# more queues rather than risk an uncovered subscriber).
|
|
89
|
+
def queue_names_for_topics(topics)
|
|
90
|
+
subscribers = @mutex.synchronize { @subscribers.dup }
|
|
91
|
+
subscribers
|
|
92
|
+
.select { |s| topics.any? { |t| pattern_overlaps?(t, s.pattern) } }
|
|
93
|
+
.map(&:queue_name)
|
|
94
|
+
.uniq
|
|
95
|
+
end
|
|
96
|
+
|
|
97
|
+
# Preserved verbatim from Consumer#pattern_overlaps?: true when either
|
|
98
|
+
# side is a superset of the other by the cheap prefix/suffix rules.
|
|
99
|
+
def pattern_overlaps?(topic_filter, subscription_pattern)
|
|
100
|
+
topic_filter == subscription_pattern ||
|
|
101
|
+
topic_filter.end_with?("#") ||
|
|
102
|
+
subscription_pattern.start_with?(topic_filter.delete_suffix(".#"))
|
|
103
|
+
end
|
|
104
|
+
|
|
83
105
|
private
|
|
84
106
|
|
|
85
107
|
# Rake task-name prefixes during which pgbus must NOT open a PGMQ
|
|
@@ -14,6 +14,9 @@ module Pgbus
|
|
|
14
14
|
# notify_retry_at is writable so a test can re-arm the backoff window
|
|
15
15
|
# between successive ensure_notify_listener calls.
|
|
16
16
|
attr_accessor :notify_listener, :notify_retry_at
|
|
17
|
+
# Supervisor-mediated wake source (issue #381): non-nil iff the
|
|
18
|
+
# supervisor forked us with a wake pipe. Readable as a test seam.
|
|
19
|
+
attr_reader :wake_pipe
|
|
17
20
|
# stat_buffer is writable so a test can swap in a buffer double after
|
|
18
21
|
# construction and assert graceful_shutdown / check_recycle / shutdown flush
|
|
19
22
|
# it (mirrors Worker#stat_buffer).
|
|
@@ -57,7 +60,7 @@ module Pgbus
|
|
|
57
60
|
queue_names: nil, liveness_pipe: nil, stat_buffer: :default,
|
|
58
61
|
notify_listener: nil, notify_retry_at: 0.0,
|
|
59
62
|
notify_retry_backoff: NOTIFY_RETRY_BASE_SECONDS,
|
|
60
|
-
started_at_monotonic: nil)
|
|
63
|
+
started_at_monotonic: nil, wake_pipe: nil)
|
|
61
64
|
@topics = Array(topics)
|
|
62
65
|
@threads = threads
|
|
63
66
|
@config = config
|
|
@@ -96,6 +99,9 @@ module Pgbus
|
|
|
96
99
|
# supervisor forked us with one. Written from stamp_loop_tick so the
|
|
97
100
|
# watchdog can detect a wedged consumer even when the database is down.
|
|
98
101
|
@liveness_pipe = liveness_pipe
|
|
102
|
+
# Supervisor wake pipe (read end): when present the fork owns NO
|
|
103
|
+
# LISTEN connection — wakes and hub health arrive as bytes (issue #381).
|
|
104
|
+
@wake_pipe = wake_pipe ? WakePipe.new(wake_pipe, wake_signal: @wake_signal) : nil
|
|
99
105
|
end
|
|
100
106
|
|
|
101
107
|
# The last wall-clock loop-tick stamp (Time.now.to_f) fed to the
|
|
@@ -110,7 +116,7 @@ module Pgbus
|
|
|
110
116
|
setup_signals
|
|
111
117
|
start_heartbeat
|
|
112
118
|
setup_subscriptions
|
|
113
|
-
|
|
119
|
+
start_wake_source
|
|
114
120
|
Pgbus.logger.info do
|
|
115
121
|
"[Pgbus] Consumer started: topics=#{topics.join(",")} threads=#{threads} " \
|
|
116
122
|
"notify_wakeup=#{notify_wakeup?} pid=#{::Process.pid}"
|
|
@@ -150,10 +156,13 @@ module Pgbus
|
|
|
150
156
|
private
|
|
151
157
|
|
|
152
158
|
def setup_subscriptions
|
|
153
|
-
|
|
154
|
-
|
|
155
|
-
|
|
156
|
-
|
|
159
|
+
# An injected queue_names: seed (test seam — the ctor documents nil as
|
|
160
|
+
# "derive from the registry") survives #run. Derivation is shared with
|
|
161
|
+
# the supervisor-owned NotifyHub (issue #381) so the LISTEN union
|
|
162
|
+
# covers exactly the queues this fork reads.
|
|
163
|
+
return unless @queue_names.nil?
|
|
164
|
+
|
|
165
|
+
@queue_names = @registry.queue_names_for_topics(topics)
|
|
157
166
|
end
|
|
158
167
|
|
|
159
168
|
def consume
|
|
@@ -273,13 +282,6 @@ module Pgbus
|
|
|
273
282
|
end
|
|
274
283
|
end
|
|
275
284
|
|
|
276
|
-
def pattern_overlaps?(topic_filter, subscription_pattern)
|
|
277
|
-
# Simple check: if either is a subset of the other
|
|
278
|
-
topic_filter == subscription_pattern ||
|
|
279
|
-
topic_filter.end_with?("#") ||
|
|
280
|
-
subscription_pattern.start_with?(topic_filter.delete_suffix(".#"))
|
|
281
|
-
end
|
|
282
|
-
|
|
283
285
|
# Signal the loop to exit cleanly once a recycle limit is hit. The clean
|
|
284
286
|
# exit gets an immediate supervisor restart (supervisor.rb:305-307), so a
|
|
285
287
|
# fresh fork replaces this one before its memory grows unbounded — the
|
|
@@ -351,21 +353,49 @@ module Pgbus
|
|
|
351
353
|
end
|
|
352
354
|
|
|
353
355
|
def wake_timeout
|
|
354
|
-
# A dead listener (running? false) will never wake the loop, so treat
|
|
355
|
-
# as absent and keep polling at the short interval until
|
|
356
|
-
# ensure_notify_listener restarts it.
|
|
357
|
-
#
|
|
358
|
-
|
|
356
|
+
# A dead listener (running? false) will never wake the loop, so treat
|
|
357
|
+
# it as absent and keep polling at the short interval until
|
|
358
|
+
# ensure_notify_listener restarts it. A live-but-deaf listener
|
|
359
|
+
# (delivering? false) is the same story (issue #332 — parity with
|
|
360
|
+
# Worker#wake_timeout). Under :supervisor scope the hub's H/P
|
|
361
|
+
# broadcasts (via WakePipe) carry the same signal. Only a live,
|
|
362
|
+
# delivering wake source earns the long NOTIFY-mode ceiling.
|
|
363
|
+
return config.polling_interval unless notify_wakeup? && wake_source_delivering?
|
|
359
364
|
|
|
360
365
|
[config.polling_interval, NOTIFY_FALLBACK_POLL_SECONDS].max
|
|
361
366
|
end
|
|
362
367
|
|
|
368
|
+
def wake_source_delivering?
|
|
369
|
+
return @wake_pipe.delivering? if @wake_pipe
|
|
370
|
+
|
|
371
|
+
@notify_listener&.running? && @notify_listener.delivering?
|
|
372
|
+
end
|
|
373
|
+
|
|
374
|
+
# :supervisor scope: the fork opens NO LISTEN connection; the WakePipe
|
|
375
|
+
# watcher is the wake source, and a missing pipe (hub failed to start)
|
|
376
|
+
# means plain polling — never a local listener (see Worker's twin).
|
|
377
|
+
# :fork scope: the fork-local NotifyListener.
|
|
378
|
+
def start_wake_source
|
|
379
|
+
return @wake_pipe.start if @wake_pipe
|
|
380
|
+
|
|
381
|
+
start_notify_listener if local_listener_scope?
|
|
382
|
+
end
|
|
383
|
+
|
|
384
|
+
def local_listener_scope?
|
|
385
|
+
config.worker_notify_scope == :fork
|
|
386
|
+
end
|
|
387
|
+
|
|
388
|
+
def stop_wake_source
|
|
389
|
+
@wake_pipe&.stop
|
|
390
|
+
@notify_listener&.stop
|
|
391
|
+
end
|
|
392
|
+
|
|
363
393
|
def start_notify_listener
|
|
364
394
|
return unless notify_wakeup?
|
|
365
395
|
|
|
366
396
|
@notify_listener = NotifyListener.new(
|
|
367
397
|
physical_queues: physical_queue_names,
|
|
368
|
-
on_wake: -> { @wake_signal.notify! },
|
|
398
|
+
on_wake: ->(_channel) { @wake_signal.notify! },
|
|
369
399
|
connection_options: config.worker_notify_connection_options,
|
|
370
400
|
health_check_ms: (config.polling_interval * 1000).to_i.clamp(250, 5_000),
|
|
371
401
|
logger: Pgbus.logger
|
|
@@ -382,6 +412,10 @@ module Pgbus
|
|
|
382
412
|
# a persistent outage retries on 5s→…→300s intervals, not every tick
|
|
383
413
|
# (mirrors Worker#ensure_notify_listener).
|
|
384
414
|
def ensure_notify_listener
|
|
415
|
+
# Supervisor scope: self-healing is the hub's job (once per host);
|
|
416
|
+
# a pipe-less fork under that scope stays on plain polling.
|
|
417
|
+
return if @wake_pipe
|
|
418
|
+
return unless local_listener_scope?
|
|
385
419
|
return unless notify_wakeup?
|
|
386
420
|
return if @notify_listener&.running?
|
|
387
421
|
return if monotonic_now < @notify_retry_at
|
|
@@ -409,9 +443,10 @@ module Pgbus
|
|
|
409
443
|
@notify_listener = nil
|
|
410
444
|
end
|
|
411
445
|
|
|
446
|
+
# Through config.queue_name so normalized subscriber queue names LISTEN
|
|
447
|
+
# on the channel their table actually notifies (see Worker's twin).
|
|
412
448
|
def physical_queue_names
|
|
413
|
-
|
|
414
|
-
@queue_names.map { |q| "#{prefix}#{q}" }
|
|
449
|
+
@queue_names.map { |q| config.queue_name(q) }
|
|
415
450
|
end
|
|
416
451
|
|
|
417
452
|
def start_heartbeat
|
|
@@ -443,7 +478,7 @@ module Pgbus
|
|
|
443
478
|
end
|
|
444
479
|
|
|
445
480
|
def shutdown
|
|
446
|
-
|
|
481
|
+
stop_wake_source
|
|
447
482
|
@pool.shutdown
|
|
448
483
|
@pool.wait_for_termination(30)
|
|
449
484
|
@stat_buffer&.stop
|