pgbus 0.25.0 → 0.25.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +8 -0
- data/README.md +5 -3
- data/lib/pgbus/client.rb +2 -3
- data/lib/pgbus/configuration.rb +25 -11
- data/lib/pgbus/dedicated_connection.rb +91 -3
- data/lib/pgbus/doctor.rb +24 -2
- data/lib/pgbus/streams/presence.rb +104 -13
- data/lib/pgbus/version.rb +1 -1
- data/lib/pgbus/web/streamer/instance.rb +7 -4
- data/lib/pgbus/web/streamer/listener.rb +32 -17
- data/lib/pgbus/web/streamer/master_hub.rb +6 -1
- data/lib/pgbus/web/streamer/stream_event_dispatcher.rb +1 -1
- data/sig/pgbus/configuration.rbs +2 -0
- metadata +1 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: d2e222e8f30d9e69de08103ecf574c4166a4804d90aff9db01df780451255b0f
|
|
4
|
+
data.tar.gz: 8744453ae4652d2dc6e163b81c5104b5e25e62bf6c7dff57c5b4e5a5c5807208
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: e0f35bcb78c6f707f6adb837f6845c21143543050891bc5dd677d935d571b2dfb93fe6e3d048ceecd2a4d75596c5f811cee2fa6076dba0c51c5b030a1ad527bb
|
|
7
|
+
data.tar.gz: c4d933bc1d90cae70c82888f31a6bbc73d83cc1a8e1ca9beccd754c310d03289e4edce4170a11a85f05c6310ecb95ec6a005c50cf9791fc5ab0b594252f0c5a3
|
data/CHANGELOG.md
CHANGED
|
@@ -4,6 +4,14 @@
|
|
|
4
4
|
|
|
5
5
|
- **Deploy tasks: `after:` makes a task wait until the tasks it names have completed, and every run records its outcome (issue #660).** `after:` takes a task key or a list of keys declared in the loaded configuration (any of `deploy_tasks_files`). The scheduler keeps such a task armed, and does not enqueue it, until each named task has a run with `completed_at` at its current version (any revision; an `every_deploy` dependency needs one on this revision). A rollover run and a Run now run both count, so a failed dependency that an operator fixes and runs again with Run now unblocks its dependents without a new deploy. A waiting task records no failure and no backoff, and fires on the first tick after its last dependency completes; the check is one query per tick, made only while an armed task has dependencies. At scheduler boot a task whose `after:` names an unknown key (`after: unknown task 'x'`), sits on a cycle (`after: cycle a → b → a`, a self-reference included), or names a task that is itself skipped (`after: 'x' is skipped`) is skipped with a warning. Run outcomes: `Deploy::Runner#enqueue` now creates the `pgbus_deploy_task_runs` row before the enqueue and passes its id in the job payload under `pgbus_deploy_task_run_id` (`Pgbus::ActiveJob::DeployTaskRunId`, included on `ActiveJob::Base` next to `BatchId`; an unmarked job's payload is unchanged), fills in `queue_name` and `msg_id` after it, and deletes the row when the enqueue fails. The executor stamps `completed_at` when `perform` returned having done its work without re-enqueuing the job (a `retry_on` attempt whose retry is queued, or a `discard_on` block that queues one, leaves it to the retry; a job that rescues inside `perform` and returns has completed) and `failed_at` when ActiveJob discarded the job without queuing a retry (`discard_on`, with or without a block, or a `retry_on` that gave up through its block, where one without a block re-raises and ends dead-lettered as before; `Pgbus::ActiveJob::DeployTaskRunId` marks the instance in ActiveJob's `run_after_discard_procs`, after the job's own `after_discard` procs, so a discarded dependency reads `failed` and its dependents keep waiting, issue #662), when the job is dead-lettered, discarded as a `:while_executing` duplicate, or its retry never left (dropped by its own concurrency limit, `on_conflict: :discard`, or aborted by a `before_enqueue` callback); a failed write is reported and never fails the job, and a job without the key pays one Hash lookup. An enqueue the job's own `on_conflict: :discard` / `:log` drops now counts as a failed enqueue (the run is deleted and the task re-armed with backoff). A run row without `queue_name` (its enqueue never finished) does not count as a run, so the version stays owed. The dispatcher's pruning keeps the newest completed run per task and version. Dashboard: a task's state says "Waiting for N dependencies to complete" and lists each one as `not run`, `running` or `failed`; the task page shows `after:` and an Outcome column in the run history; Run now on a waiting task is refused with a flash naming the dependencies (`DataSource#enqueue_deploy_task_now(id, ignore_dependencies: false)` returns `:blocked`) unless "Ignore dependencies" is ticked on the task page. `pgbus_deploy_tasks` (MCP) lists `dependencies` and `waiting_on`. New generator `pgbus:add_deploy_task_run_outcomes` adds `pgbus_deploy_task_runs.completed_at` / `failed_at`, `pgbus_deploy_tasks.dependencies` and an index on `(task_key, completed_at)`; `pgbus:update` queues it, and fresh `pgbus:install` / `pgbus:add_deploy_tasks` include the columns. Until it is migrated, a task with `after:` is skipped with a warning naming the generator and no outcome is written. **Upgrade note:** runs recorded before the migration have no outcome and satisfy nothing, so when an `after:` names a task whose only runs predate the upgrade, Run now that task once (or bump its version; an `every_deploy` dependency needs a run on the current revision, so only Run now helps). Delivery stays at-least-once: a scheduler that dies between claiming a task and enqueuing it leaves the task unarmed until the next scheduler boot re-arms it, and a run whose enqueue could not be recorded runs again after one, so deploy tasks must be idempotent. Refs #660.
|
|
6
6
|
|
|
7
|
+
### Changed
|
|
8
|
+
|
|
9
|
+
- **Changed default: the streamer's LISTEN keepalive `streams_listen_health_check_ms` is 5 000 ms instead of 250, and the listener's ack and stop budgets no longer follow it (issue #664).** Every web process running the streamer sent `SELECT 1` on its LISTEN connection four times a second, idle or not: 345 600 queries a day per process, the most frequent statement on a small fleet's database. Since #630 a connect no longer waits on that cycle (the listener wakes through a pipe), so the interval only bounds how long a LISTEN the server dropped on a still-open socket goes unnoticed; now one `SELECT 1` per 5 s idle cycle, 20× fewer. The interval was also wired into three wait budgets, so raising it slowed the failure paths: they now come from a new setting, **`streams_listen_ack_timeout_ms`** (default 1 250, a positive Integer, today's 250 + 1 000). `Listener#ensure_listening` waits at most that for a stuck listener's ack, `Listener#stop` joins for that plus its 5 s grace, and with `streams_listen_scope :master` the worker's `HubClient` waits that plus 1 s before failing over, as before relative to the hub's own budget; the master hub's listener gets the same ack budget. The streamer's dedicated LISTEN connection (per worker, and the master hub's) now opens with libpq TCP keepalives, `keepalives=1 keepalives_idle=5 keepalives_interval=2 keepalives_count=3 tcp_user_timeout=11000`, so a dead socket errors within ~11 s by these settings (5 s idle + 3 probes 2 s apart, and the matching `tcp_user_timeout`; on Linux, where `tcp_user_timeout` applies; the bound follows the settings, it was not measured) even between keepalives, and the listener reconnects and re-LISTENs; a keyword the connection options already set wins, a connection that names a service (`service=` or `PGSERVICE`) gets none so its service file's values stay in force, and on libpq < 12 none is added (the same gate `Client` uses for its read bounds, now `Pgbus::DedicatedConnection.libpq_supports_socket_bounds?`). The worker `NotifyListener` and the doctor's connection are unchanged. The pool autoscaler's check runs on the keepalive tick, so a `streams_pool_autoscale_interval` under 5 s (the default is 300 s) now runs every 5 s. A dedicated connection URL ending in a bare `?` no longer gets an empty `?&` parameter that failed the connect. An explicit `streams_listen_health_check_ms` keeps exactly its value: **pin `config.streams_listen_health_check_ms = 250` for the old keepalive.** If you set it to 5 000 as the workaround, the ack budget drops from 6 s back to 1.25 s. Measured on the same machine (`rake bench:streams`, which pins 100 ms; `rake bench:streams_scale` at M = 100 with the default, 250 ms before and 5 000 ms after, two rounds each): section 4 concurrent connect 0.91–3.07 ms per connect before, 0.86–3.06 ms after; scale bench, all 100 clients registered in 0.11–0.12 s before, 0.09–0.11 s after (one 0.24 s round at load 32, 0.09 s on a re-run), 0 misses either way. Refs #664.
|
|
10
|
+
|
|
11
|
+
### Fixed
|
|
12
|
+
|
|
13
|
+
- **A presence table without the seat columns is reported, and its schema probe backs off (issue #665).** `Pgbus::Streams::Presence.seats_supported?` kept a "no seats" answer for 30 s, so an app that upgraded without running `rails g pgbus:add_presence_seats` queried `information_schema.columns` twice a minute per process, forever, and nothing said why. A "no" is now kept for 30 s, then twice as long after each further "no", up to 10 minutes (`SCHEMA_RECHECK_MAX_SECONDS = 600`); a true answer is still kept for the life of the process, `reset_schema_cache!` still re-probes at once and starts the backoff again from 30 s, and two threads that probe at the same moment count as one step. The backoff holds in a process that makes no member-keyed `leave`, `touch` or `sweep!`: each of those still re-probes a cached "no" before it runs, because it would delete or refresh seat rows on a table migrated under the process, and restarts the backoff at 30 s. Connection-driven presence on such a table still touches once per connection per heartbeat, unchanged here (#666). A member-keyed `join` that hits the migrated index still re-probes at once. The first "no" logs one warning per process naming the fix (`rails generate pgbus:add_presence_seats && rails db:migrate`). `pgbus doctor` has a fifteenth check, `Presence seats`: a warning with the same fix when `pgbus_presence_members` exists without `connection_id`, ok when the column exists or presence is not installed; it runs in the boot preflight and never blocks a strict boot. New `Pgbus::Streams::Presence.seats_state` (`:absent`, `:missing`, `:present`), read on the connection presence uses. Refs #665.
|
|
14
|
+
|
|
7
15
|
## [0.24.0] - 2026-11-11
|
|
8
16
|
|
|
9
17
|
### Added
|
data/README.md
CHANGED
|
@@ -1556,7 +1556,8 @@ Pgbus.configure do |c|
|
|
|
1556
1556
|
c.streams_peer_close_interval = 1.0 # notice a closed client at once; nil = heartbeat only
|
|
1557
1557
|
c.streams_max_connections = 2_000 # per web-server process (Puma worker or Falcon process)
|
|
1558
1558
|
c.streams_idle_timeout = 3_600 # close idle connections after 1h
|
|
1559
|
-
c.streams_listen_health_check_ms =
|
|
1559
|
+
c.streams_listen_health_check_ms = 5_000 # PG LISTEN keepalive: one SELECT 1 per idle cycle, the bound on noticing a dropped LISTEN (a dead socket: ~11 s by the TCP keepalive settings it adds when the connection options set none of their own; libpq >= 12, none added on a `service=` / PGSERVICE connection)
|
|
1560
|
+
c.streams_listen_ack_timeout_ms = 1_250 # how long a stream's first connect waits for a stuck listener (a live one acks in a thread hop); :master scope adds 1 s
|
|
1560
1561
|
c.streams_write_deadline_ms = 5_000 # hijacked-socket write_nonblock deadline (connect replay, sentinels)
|
|
1561
1562
|
c.streams_fanout_write_deadline_ms = 250 # hijacked-socket fan-out and heartbeat writes; a slower client reconnects (Falcon's streaming body ignores both)
|
|
1562
1563
|
c.streams_falcon_streaming_body = false # opt-in: Falcon-native streaming body
|
|
@@ -1961,7 +1962,7 @@ Pgbus.configure do |c|
|
|
|
1961
1962
|
end
|
|
1962
1963
|
```
|
|
1963
1964
|
|
|
1964
|
-
Keep each ttl above `streams_heartbeat_interval` (pgbus warns at boot otherwise): a shorter one expires live seats between touches. On a seat table `sweep!` only expires member-keyed rows; seats expire through the heartbeat. A process that probed the table before the migration re-checks
|
|
1965
|
+
Keep each ttl above `streams_heartbeat_interval` (pgbus warns at boot otherwise): a shorter one expires live seats between touches. On a seat table `sweep!` only expires member-keyed rows; seats expire through the heartbeat. A process that probed the table before the migration re-checks after 30 seconds, then after twice as long each time it still finds no seats, up to every 10 minutes. A member-keyed `leave`, `touch` or `sweep!` re-checks before it runs and starts that backoff again at 30 seconds, and a member-keyed `join` outside a transaction re-checks at once if it hits the new index. The first time it finds no seats it logs one warning naming the migration, and `pgbus doctor` warns under `Presence seats`; restart the processes after migrating to pick the columns up at once. A raising hook is logged, not propagated. Library authors can probe for seat support with `Pgbus::Streams::Presence.instance_method(:join).parameters.include?([:key, :connection_id])` (the gem) and `Pgbus::Streams::Presence.seats_supported?` (the table).
|
|
1965
1966
|
|
|
1966
1967
|
### Stream stats (opt-in)
|
|
1967
1968
|
|
|
@@ -2242,7 +2243,7 @@ pgbus help # Show help
|
|
|
2242
2243
|
|
|
2243
2244
|
#### pgbus doctor
|
|
2244
2245
|
|
|
2245
|
-
A single preflight command that answers "is this environment healthy enough to run?" — useful as a deploy or CI gate. It runs
|
|
2246
|
+
A single preflight command that answers "is this environment healthy enough to run?" — useful as a deploy or CI gate. It runs 15 checks and never raises; a broken environment turns every probe into a failed/warned check instead of a crash:
|
|
2246
2247
|
|
|
2247
2248
|
| Check | Fails (`:fail`) when | Warns (`:warn`) when |
|
|
2248
2249
|
|---|---|---|
|
|
@@ -2260,6 +2261,7 @@ A single preflight command that answers "is this environment healthy enough to r
|
|
|
2260
2261
|
| Dedicated connections | Streamer LISTEN and/or worker notify dedicated path cannot connect | — |
|
|
2261
2262
|
| Connection budget | — (informational: prints how many direct LISTEN connections the current config pins — 1 per host under `worker_notify_scope: :supervisor`, one per fork under `:fork`; streams add 1 per web host under `streams_listen_scope: :master` or 1 per web process under `:process`) | — |
|
|
2262
2263
|
| Completion commit | — (informational: prints whether completions commit synchronously, the default, or with `synchronous_commit` off under `completion_synchronous_commit = false`) | `completion_synchronous_commit = false` with `native_statements = false`, or with pgbus on ActiveRecord's shared connection, where it has no effect |
|
|
2264
|
+
| Presence seats | — | `pgbus_presence_members` exists without the `connection_id` column: presence runs member-keyed (one row per member, no seats) until `rails generate pgbus:add_presence_seats && rails db:migrate` runs; ok when the column exists or presence is not installed |
|
|
2263
2265
|
|
|
2264
2266
|
```bash
|
|
2265
2267
|
pgbus doctor # prints the report; exit 1 unless every check passed
|
data/lib/pgbus/client.rb
CHANGED
|
@@ -2253,10 +2253,9 @@ module Pgbus
|
|
|
2253
2253
|
end
|
|
2254
2254
|
|
|
2255
2255
|
# Whether the linked libpq accepts the tcp_user_timeout / keepalives conninfo
|
|
2256
|
-
# keywords
|
|
2257
|
-
# option" and fails the connection, so we must not emit them there.
|
|
2256
|
+
# keywords (libpq 12+). One gate for every connection pgbus opens.
|
|
2258
2257
|
def libpq_supports_socket_bounds?
|
|
2259
|
-
|
|
2258
|
+
DedicatedConnection.libpq_supports_socket_bounds?
|
|
2260
2259
|
end
|
|
2261
2260
|
|
|
2262
2261
|
# Hash form maps 1:1 to libpq keywords, so no escaping/encoding is needed.
|
data/lib/pgbus/configuration.rb
CHANGED
|
@@ -328,6 +328,7 @@ module Pgbus
|
|
|
328
328
|
attr_accessor :streams_enabled, :streams_path, :streams_signed_name_secret,
|
|
329
329
|
:streams_default_retention, :streams_retention, :streams_heartbeat_interval, :streams_peer_close_interval,
|
|
330
330
|
:streams_max_connections, :streams_idle_timeout, :streams_listen_health_check_ms,
|
|
331
|
+
:streams_listen_ack_timeout_ms,
|
|
331
332
|
:streams_write_deadline_ms, :streams_falcon_streaming_body,
|
|
332
333
|
:streams_stats_enabled, :streams_test_mode,
|
|
333
334
|
:streams_orphan_sweep_interval, :streams_orphan_threshold,
|
|
@@ -574,7 +575,9 @@ module Pgbus
|
|
|
574
575
|
# Slow maintenance cadence (seconds) — the autoscaler runs as a periodic
|
|
575
576
|
# check on the streamer's idle LISTEN connection (pghero-style), not a busy
|
|
576
577
|
# loop. 5 minutes: the interval itself debounces, so there is no per-sample
|
|
577
|
-
# hysteresis; a sustained burst converges over a few checks.
|
|
578
|
+
# hysteresis; a sustained burst converges over a few checks. The check
|
|
579
|
+
# runs on the listener's keepalive tick, so an interval shorter than
|
|
580
|
+
# streams_listen_health_check_ms runs once per keepalive instead.
|
|
578
581
|
@streams_pool_autoscale_interval = 300.0
|
|
579
582
|
# Distinctive application_name prefix stamped on streams-pool connections
|
|
580
583
|
# so the autoscaler can count peer processes (DISTINCT application_name
|
|
@@ -595,16 +598,23 @@ module Pgbus
|
|
|
595
598
|
@streams_peer_close_interval = 1.0
|
|
596
599
|
@streams_max_connections = 2_000
|
|
597
600
|
@streams_idle_timeout = 3_600 # 1 hour
|
|
598
|
-
#
|
|
599
|
-
#
|
|
600
|
-
#
|
|
601
|
-
#
|
|
602
|
-
#
|
|
603
|
-
#
|
|
604
|
-
#
|
|
605
|
-
#
|
|
606
|
-
#
|
|
607
|
-
@streams_listen_health_check_ms =
|
|
601
|
+
# 5 s — the keepalive interval of the streamer's PG LISTEN
|
|
602
|
+
# connection: one `SELECT 1` per idle cycle, and the bound on how
|
|
603
|
+
# long a LISTEN the server dropped on a still-open socket goes
|
|
604
|
+
# unnoticed. A socket whose peer is gone is also detected, between
|
|
605
|
+
# keepalives, by the TCP keepalives on that connection: ~11 s by their
|
|
606
|
+
# settings (not measured; libpq >= 12; not added to a connection that
|
|
607
|
+
# names a service, whose service file keeps them). Connects do not wait on
|
|
608
|
+
# this cycle: the listener wakes through a pipe (issue #630). Was
|
|
609
|
+
# 250 ms, which ran 4 queries a second per web process (issue #664).
|
|
610
|
+
@streams_listen_health_check_ms = 5_000
|
|
611
|
+
# 1.25 s — how long a stream's first connect waits for a stuck local
|
|
612
|
+
# Listener to acknowledge its ensure_listening; #stop's join budget
|
|
613
|
+
# is this plus a grace. With streams_listen_scope :master, HubClient
|
|
614
|
+
# waits this plus 1 s before failing over. Independent of the
|
|
615
|
+
# keepalive above (issue #664). A live listener acks within a thread
|
|
616
|
+
# hop.
|
|
617
|
+
@streams_listen_ack_timeout_ms = 1_250
|
|
608
618
|
@streams_write_deadline_ms = 5_000
|
|
609
619
|
# Deadline (ms) for a socket write made INSIDE the dispatcher's
|
|
610
620
|
# serial fanout loop (handle_durable_wake / handle_ephemeral_wake).
|
|
@@ -1308,6 +1318,10 @@ module Pgbus
|
|
|
1308
1318
|
raise Pgbus::ConfigurationError, "streams_listen_health_check_ms must be a positive integer"
|
|
1309
1319
|
end
|
|
1310
1320
|
|
|
1321
|
+
unless streams_listen_ack_timeout_ms.is_a?(Integer) && streams_listen_ack_timeout_ms.positive?
|
|
1322
|
+
raise Pgbus::ConfigurationError, "streams_listen_ack_timeout_ms must be a positive integer"
|
|
1323
|
+
end
|
|
1324
|
+
|
|
1311
1325
|
unless streams_write_deadline_ms.is_a?(Integer) && streams_write_deadline_ms.positive?
|
|
1312
1326
|
raise Pgbus::ConfigurationError, "streams_write_deadline_ms must be a positive integer"
|
|
1313
1327
|
end
|
|
@@ -23,6 +23,20 @@ module Pgbus
|
|
|
23
23
|
# hash — or PGAPPNAME in the environment — always wins.
|
|
24
24
|
APP_NAME = "pgbus-listen"
|
|
25
25
|
|
|
26
|
+
# libpq TCP keepalives for the streamer's LISTEN connection (issue #664):
|
|
27
|
+
# probing starts after 5 idle seconds, every 2 s, 3 times, so a dead
|
|
28
|
+
# socket errors within ~11 s even between the listener's own keepalive
|
|
29
|
+
# queries; tcp_user_timeout bounds data the peer never acknowledges by
|
|
30
|
+
# the same 11 s. Client-side keywords: never in the startup packet, so a
|
|
31
|
+
# pooler cannot reject them.
|
|
32
|
+
LISTEN_KEEPALIVES = {
|
|
33
|
+
keepalives: 1,
|
|
34
|
+
keepalives_idle: 5,
|
|
35
|
+
keepalives_interval: 2,
|
|
36
|
+
keepalives_count: 3,
|
|
37
|
+
tcp_user_timeout: 11_000
|
|
38
|
+
}.freeze
|
|
39
|
+
|
|
26
40
|
def connect(opts)
|
|
27
41
|
require "pg" unless defined?(::PG::Connection)
|
|
28
42
|
case opts
|
|
@@ -41,14 +55,88 @@ module Pgbus
|
|
|
41
55
|
def with_app_name(conninfo)
|
|
42
56
|
return conninfo if conninfo.include?("application_name")
|
|
43
57
|
|
|
44
|
-
if
|
|
45
|
-
|
|
46
|
-
"#{conninfo}#{separator}fallback_application_name=#{APP_NAME}"
|
|
58
|
+
if url?(conninfo)
|
|
59
|
+
"#{conninfo}#{query_separator(conninfo)}fallback_application_name=#{APP_NAME}"
|
|
47
60
|
else
|
|
48
61
|
"#{conninfo} fallback_application_name=#{APP_NAME}".strip
|
|
49
62
|
end
|
|
50
63
|
end
|
|
51
64
|
|
|
65
|
+
# connect, plus LISTEN_KEEPALIVES: the streamer's LISTEN connection. A
|
|
66
|
+
# keyword the options already set wins.
|
|
67
|
+
def connect_with_keepalives(opts)
|
|
68
|
+
require "pg" unless defined?(::PG::Connection)
|
|
69
|
+
connect(with_keepalives(opts))
|
|
70
|
+
end
|
|
71
|
+
|
|
72
|
+
# Whether the linked libpq accepts the tcp_user_timeout / keepalives conninfo
|
|
73
|
+
# keywords. Added in libpq 12; an older libpq raises "invalid connection
|
|
74
|
+
# option" and fails the connection, so they must not be emitted there.
|
|
75
|
+
def libpq_supports_socket_bounds?
|
|
76
|
+
defined?(::PG) && ::PG.respond_to?(:library_version) && ::PG.library_version >= 120_000
|
|
77
|
+
end
|
|
78
|
+
|
|
79
|
+
# Fill in the LISTEN_KEEPALIVES keywords the options do not already set.
|
|
80
|
+
# Unchanged on libpq < 12 (see libpq_supports_socket_bounds?), and for a
|
|
81
|
+
# connection that names a service (service=, or PGSERVICE): libpq applies
|
|
82
|
+
# the service file only to keywords the conninfo leaves unset, so an added
|
|
83
|
+
# keyword would override the service's own.
|
|
84
|
+
def with_keepalives(opts)
|
|
85
|
+
return opts unless libpq_supports_socket_bounds?
|
|
86
|
+
return opts if ENV.fetch("PGSERVICE", nil).to_s != ""
|
|
87
|
+
|
|
88
|
+
case opts
|
|
89
|
+
when Hash
|
|
90
|
+
return opts if opts.key?(:service) || opts.key?("service")
|
|
91
|
+
|
|
92
|
+
missing = LISTEN_KEEPALIVES.reject { |key, _| opts.key?(key) || opts.key?(key.to_s) }
|
|
93
|
+
opts.merge(missing)
|
|
94
|
+
when String then append_keepalives(opts)
|
|
95
|
+
else opts
|
|
96
|
+
end
|
|
97
|
+
end
|
|
98
|
+
|
|
99
|
+
# URL query-param style for postgres:// URLs, space-separated keyword
|
|
100
|
+
# style otherwise (as with_app_name does). The keywords already set are
|
|
101
|
+
# read with libpq's own parser, so a keyword quoted inside another value
|
|
102
|
+
# does not count; a conninfo libpq cannot parse is left for PG.connect to
|
|
103
|
+
# report.
|
|
104
|
+
def append_keepalives(conninfo)
|
|
105
|
+
set = conninfo_keywords(conninfo)
|
|
106
|
+
return conninfo if set.nil? || set.include?("service")
|
|
107
|
+
|
|
108
|
+
missing = LISTEN_KEEPALIVES.reject { |key, _| set.include?(key.to_s) }
|
|
109
|
+
return conninfo if missing.empty?
|
|
110
|
+
|
|
111
|
+
if url?(conninfo)
|
|
112
|
+
"#{conninfo}#{query_separator(conninfo)}#{missing.map { |key, value| "#{key}=#{value}" }.join("&")}"
|
|
113
|
+
else
|
|
114
|
+
"#{conninfo} #{missing.map { |key, value| "#{key}=#{value}" }.join(" ")}".strip
|
|
115
|
+
end
|
|
116
|
+
end
|
|
117
|
+
|
|
118
|
+
# A postgres:// or postgresql:// URL, as libpq tells them apart: by the
|
|
119
|
+
# prefix, so a key=value conninfo with :// inside a value (a password)
|
|
120
|
+
# stays key=value.
|
|
121
|
+
def url?(conninfo)
|
|
122
|
+
conninfo.start_with?("postgres://", "postgresql://")
|
|
123
|
+
end
|
|
124
|
+
|
|
125
|
+
# The keywords a conninfo String sets, or nil when libpq cannot parse it.
|
|
126
|
+
def conninfo_keywords(conninfo)
|
|
127
|
+
::PG::Connection.conninfo_parse(conninfo).filter_map { |option| option[:keyword] if option[:val] }
|
|
128
|
+
rescue ::PG::Error
|
|
129
|
+
nil
|
|
130
|
+
end
|
|
131
|
+
|
|
132
|
+
# "?" to start a URL's query string, "&" to extend one, nothing after a
|
|
133
|
+
# trailing "?" or "&" (an empty parameter would fail the connect).
|
|
134
|
+
def query_separator(url)
|
|
135
|
+
return "" if url.end_with?("?", "&")
|
|
136
|
+
|
|
137
|
+
url.include?("?") ? "&" : "?"
|
|
138
|
+
end
|
|
139
|
+
|
|
52
140
|
def connect_from_hash(opts)
|
|
53
141
|
variables = opts[:variables]
|
|
54
142
|
conn = ::PG.connect(**with_app_name_hash(opts.except(:variables)))
|
data/lib/pgbus/doctor.rb
CHANGED
|
@@ -5,7 +5,7 @@ require "pgbus/mcp/health_analyzer"
|
|
|
5
5
|
|
|
6
6
|
module Pgbus
|
|
7
7
|
# Preflight diagnostics for a pgbus deployment — the single command that
|
|
8
|
-
# answers "is this environment healthy enough to run?". Runs
|
|
8
|
+
# answers "is this environment healthy enough to run?". Runs fifteen checks and
|
|
9
9
|
# returns a machine-readable result plus a human report, so `pgbus doctor`
|
|
10
10
|
# and `rake pgbus:doctor` can gate a deploy or CI run (exit 0 on success,
|
|
11
11
|
# 1 on any failure).
|
|
@@ -44,7 +44,8 @@ module Pgbus
|
|
|
44
44
|
"Primary affinity" => :check_primary,
|
|
45
45
|
"Dedicated connections" => :check_dedicated_connections,
|
|
46
46
|
"Connection budget" => :check_connection_budget,
|
|
47
|
-
"Completion commit" => :check_completion_commit
|
|
47
|
+
"Completion commit" => :check_completion_commit,
|
|
48
|
+
"Presence seats" => :check_presence_seats
|
|
48
49
|
}.freeze
|
|
49
50
|
|
|
50
51
|
# Process liveness reads the pgbus_processes table (via HealthAnalyzer), so
|
|
@@ -448,6 +449,27 @@ module Pgbus
|
|
|
448
449
|
Check.new(name: "Completion commit", status: :warn, detail: "#{e.class}: #{e.message}")
|
|
449
450
|
end
|
|
450
451
|
|
|
452
|
+
# Presence seats (issue #665). Without the add_presence_seats migration
|
|
453
|
+
# presence runs member-keyed (one row per member, no seats), and every
|
|
454
|
+
# process re-probes information_schema with backoff. A supported degraded
|
|
455
|
+
# mode, so a warning; ok when the columns exist or presence is not installed.
|
|
456
|
+
# Read through Presence, not the Client: presence reads ActiveRecord's
|
|
457
|
+
# connection (Pgbus::BusRecord under connects_to), which a database_url or
|
|
458
|
+
# connection_params Client connection need not share.
|
|
459
|
+
def check_presence_seats
|
|
460
|
+
case Pgbus::Streams::Presence.seats_state
|
|
461
|
+
when :absent then Check.new(name: "Presence seats", status: :ok, detail: "presence not installed")
|
|
462
|
+
when :present then Check.new(name: "Presence seats", status: :ok, detail: "pgbus_presence_members migrated with seats")
|
|
463
|
+
else
|
|
464
|
+
Check.new(name: "Presence seats", status: :warn,
|
|
465
|
+
detail: "pgbus_presence_members has no connection_id column — presence runs member-keyed " \
|
|
466
|
+
"(one row per member, no seats) and each process re-probes the schema; run " \
|
|
467
|
+
"`#{Pgbus::Streams::Presence::SEATS_MIGRATION_FIX}`")
|
|
468
|
+
end
|
|
469
|
+
rescue StandardError => e
|
|
470
|
+
Check.new(name: "Presence seats", status: :warn, detail: "could not read the schema (#{e.class}: #{e.message})")
|
|
471
|
+
end
|
|
472
|
+
|
|
451
473
|
# How many direct LISTEN connections this config pins at steady state
|
|
452
474
|
# (issue #381) — informational (always :ok) so operators can do capacity
|
|
453
475
|
# math on the pooler's direct-connection budget from the doctor output.
|
|
@@ -65,19 +65,39 @@ module Pgbus
|
|
|
65
65
|
RECORD_COLUMNS = "member_id, metadata, joined_at, last_seen_at"
|
|
66
66
|
SEAT_RECORD_COLUMNS = "member_id, metadata, joined_at, last_seen_at, connection_id, seat, state"
|
|
67
67
|
|
|
68
|
-
# How long
|
|
69
|
-
# again, so a table migrated under a running process is picked
|
|
68
|
+
# How long the first "no seats" answer is trusted before the schema is
|
|
69
|
+
# probed again, so a table migrated under a running process is picked
|
|
70
|
+
# up. Each further "no" doubles the wait, up to SCHEMA_RECHECK_MAX_SECONDS
|
|
71
|
+
# (issue #665).
|
|
70
72
|
SCHEMA_RECHECK_SECONDS = 30
|
|
73
|
+
SCHEMA_RECHECK_MAX_SECONDS = 600
|
|
74
|
+
|
|
75
|
+
# What the doctor and the one-time warning tell an operator to run.
|
|
76
|
+
SEATS_MIGRATION_FIX = "rails generate pgbus:add_presence_seats && rails db:migrate"
|
|
77
|
+
|
|
78
|
+
# NULL: no pgbus_presence_members table; false: the table without the
|
|
79
|
+
# seat columns; true: seats. The same schema predicate as the
|
|
80
|
+
# seats_supported? probe.
|
|
81
|
+
SEATS_STATE_SQL = <<~SQL
|
|
82
|
+
SELECT bool_or(column_name = 'connection_id') FROM information_schema.columns
|
|
83
|
+
WHERE table_name = 'pgbus_presence_members' AND table_schema = ANY (current_schemas(false))
|
|
84
|
+
SQL
|
|
71
85
|
|
|
72
86
|
@schema_mutex = Mutex.new
|
|
73
87
|
@seats_supported = nil
|
|
74
88
|
@probed_at = nil
|
|
89
|
+
@recheck_seconds = SCHEMA_RECHECK_SECONDS
|
|
75
90
|
@schema_generation = 0
|
|
91
|
+
# Survives reset_schema_cache!: the warning is once per process.
|
|
92
|
+
@seats_warning_logged = false
|
|
76
93
|
|
|
77
94
|
class << self
|
|
78
95
|
# True when pgbus_presence_members has the seat columns (the
|
|
79
96
|
# add_presence_seats migration ran). A true answer is kept for the
|
|
80
|
-
# life of the process
|
|
97
|
+
# life of the process. A false one is kept for SCHEMA_RECHECK_SECONDS,
|
|
98
|
+
# then twice as long after each further false answer, up to
|
|
99
|
+
# SCHEMA_RECHECK_MAX_SECONDS (issue #665); the first one logs a
|
|
100
|
+
# warning naming the migration, once per process.
|
|
81
101
|
#
|
|
82
102
|
# The answer is read and written under @schema_mutex only (issue
|
|
83
103
|
# #527). A successful probe is kept as above; a probe that raises is
|
|
@@ -90,7 +110,7 @@ module Pgbus
|
|
|
90
110
|
# probe that a reset_schema_cache! overtook stores nothing.
|
|
91
111
|
def seats_supported?
|
|
92
112
|
cached, generation = @schema_mutex.synchronize do
|
|
93
|
-
[(@seats_supported if schema_answer_fresh?(@seats_supported, @probed_at)), @schema_generation]
|
|
113
|
+
[(@seats_supported if schema_answer_fresh?(@seats_supported, @probed_at, @recheck_seconds)), @schema_generation]
|
|
94
114
|
end
|
|
95
115
|
return cached unless cached.nil?
|
|
96
116
|
|
|
@@ -100,12 +120,9 @@ module Pgbus
|
|
|
100
120
|
AND table_schema = ANY (current_schemas(false))
|
|
101
121
|
SQL
|
|
102
122
|
supported = !rows.to_a.empty?
|
|
103
|
-
|
|
104
|
-
|
|
105
|
-
|
|
106
|
-
@probed_at = monotonic_now
|
|
107
|
-
@seats_supported ||= supported
|
|
108
|
-
end
|
|
123
|
+
answer, warn = store_schema_answer(supported, generation)
|
|
124
|
+
warn_seats_missing if warn
|
|
125
|
+
answer
|
|
109
126
|
end
|
|
110
127
|
|
|
111
128
|
# seats_supported?, but a cached "no seats" is probed again first.
|
|
@@ -127,6 +144,22 @@ module Pgbus
|
|
|
127
144
|
end
|
|
128
145
|
end
|
|
129
146
|
|
|
147
|
+
# What Pgbus::Doctor reports (issue #665): :absent (no
|
|
148
|
+
# pgbus_presence_members table, or no ActiveRecord database to reach
|
|
149
|
+
# it), :missing (add_presence_seats has not run) or :present. Read on
|
|
150
|
+
# the connection presence uses, leased for this statement only, and
|
|
151
|
+
# never cached. A database that is configured but down raises.
|
|
152
|
+
def seats_state
|
|
153
|
+
return :absent unless defined?(::ActiveRecord::Base)
|
|
154
|
+
|
|
155
|
+
value = presence_pool&.with_connection { |conn| conn.select_value(SEATS_STATE_SQL) }
|
|
156
|
+
case value
|
|
157
|
+
when nil then :absent
|
|
158
|
+
when true, "t" then :present
|
|
159
|
+
else :missing
|
|
160
|
+
end
|
|
161
|
+
end
|
|
162
|
+
|
|
130
163
|
# Deletes every seat row (connection_id IS NOT NULL) that has not
|
|
131
164
|
# been touched within its stream's ttl and returns the deleted
|
|
132
165
|
# records, each with its "stream_name". Member-keyed rows are left
|
|
@@ -197,13 +230,71 @@ module Pgbus
|
|
|
197
230
|
"Pgbus::Streams::Presence requires ActiveRecord (no AR connection available)"
|
|
198
231
|
end
|
|
199
232
|
|
|
200
|
-
|
|
233
|
+
record_base.connection
|
|
234
|
+
end
|
|
235
|
+
|
|
236
|
+
def record_base
|
|
237
|
+
Pgbus.configuration.connects_to ? Pgbus::BusRecord : ::ActiveRecord::Base
|
|
201
238
|
end
|
|
202
239
|
|
|
203
|
-
|
|
240
|
+
# The pool presence reads from, or nil when ActiveRecord has none
|
|
241
|
+
# defined for it (a process on database_url alone): connection_pool
|
|
242
|
+
# raises ConnectionNotEstablished only for a missing pool, never for
|
|
243
|
+
# a database that is down.
|
|
244
|
+
def presence_pool
|
|
245
|
+
record_base.connection_pool
|
|
246
|
+
rescue ::ActiveRecord::ConnectionNotEstablished
|
|
247
|
+
nil
|
|
248
|
+
end
|
|
249
|
+
|
|
250
|
+
def schema_answer_fresh?(supported, probed_at, recheck_seconds)
|
|
204
251
|
return false if supported.nil? || probed_at.nil?
|
|
205
252
|
|
|
206
|
-
supported || monotonic_now - probed_at <
|
|
253
|
+
supported || monotonic_now - probed_at < recheck_seconds
|
|
254
|
+
end
|
|
255
|
+
|
|
256
|
+
# Stores a probe's answer unless a reset_schema_cache! overtook it.
|
|
257
|
+
# Returns the answer to give and whether to log the missing-seats
|
|
258
|
+
# warning (the first "no" of the process).
|
|
259
|
+
#
|
|
260
|
+
# A "no" over a fresh "no" (two threads probed at once) keeps the
|
|
261
|
+
# first one's time and step, so concurrent probes are one step. A
|
|
262
|
+
# true answer is never taken back.
|
|
263
|
+
def store_schema_answer(supported, generation)
|
|
264
|
+
@schema_mutex.synchronize do
|
|
265
|
+
next [supported, false] unless generation == @schema_generation
|
|
266
|
+
|
|
267
|
+
if supported || @seats_supported
|
|
268
|
+
@probed_at = monotonic_now
|
|
269
|
+
next [@seats_supported = true, false]
|
|
270
|
+
end
|
|
271
|
+
next [false, false] if schema_answer_fresh?(@seats_supported, @probed_at, @recheck_seconds)
|
|
272
|
+
|
|
273
|
+
@recheck_seconds = next_recheck_seconds(@seats_supported, @recheck_seconds)
|
|
274
|
+
@probed_at = monotonic_now
|
|
275
|
+
@seats_supported = false
|
|
276
|
+
warn = !@seats_warning_logged
|
|
277
|
+
@seats_warning_logged = true
|
|
278
|
+
[false, warn]
|
|
279
|
+
end
|
|
280
|
+
end
|
|
281
|
+
|
|
282
|
+
# A "no" after a stale "no" doubles the wait, up to the cap; a "no"
|
|
283
|
+
# after no answer (the first probe, or after reset_schema_cache!)
|
|
284
|
+
# starts at SCHEMA_RECHECK_SECONDS.
|
|
285
|
+
def next_recheck_seconds(previous_answer, previous_seconds)
|
|
286
|
+
return SCHEMA_RECHECK_SECONDS if previous_answer.nil?
|
|
287
|
+
|
|
288
|
+
[previous_seconds * 2, SCHEMA_RECHECK_MAX_SECONDS].min
|
|
289
|
+
end
|
|
290
|
+
|
|
291
|
+
def warn_seats_missing
|
|
292
|
+
Pgbus.logger&.warn(
|
|
293
|
+
"[Pgbus::Streams::Presence] pgbus_presence_members has no seat columns (connection_id): presence " \
|
|
294
|
+
"runs member-keyed, one row per member, and the schema keeps being re-checked. Run `#{SEATS_MIGRATION_FIX}` to get seats " \
|
|
295
|
+
"(if the table does not exist, run `rails generate pgbus:add_presence && rails db:migrate` instead: " \
|
|
296
|
+
"it already creates the seat columns)."
|
|
297
|
+
)
|
|
207
298
|
end
|
|
208
299
|
|
|
209
300
|
def expire_stream_seats(stream_name, connection_ids, ttl)
|
data/lib/pgbus/version.rb
CHANGED
|
@@ -190,12 +190,13 @@ module Pgbus
|
|
|
190
190
|
return nil if socket_path.nil? || socket_path.empty?
|
|
191
191
|
|
|
192
192
|
# The worker's ack deadline must exceed the master's own internal
|
|
193
|
-
# ensure_listening budget (its listener's
|
|
193
|
+
# ensure_listening budget (its listener's ack budget), so it gets
|
|
194
|
+
# that plus 1 s. Independent of the keepalive (issue #664).
|
|
194
195
|
failover = nil
|
|
195
196
|
client = HubClient.new(
|
|
196
197
|
socket_path: socket_path,
|
|
197
198
|
dispatch_queue: @dispatch_queue,
|
|
198
|
-
ack_timeout: (@config.
|
|
199
|
+
ack_timeout: (@config.streams_listen_ack_timeout_ms / 1000.0) + 1.0,
|
|
199
200
|
# failover is assigned right below; a transport death in the gap
|
|
200
201
|
# is caught by the FailoverListener's synchronous ensure path.
|
|
201
202
|
on_failure: -> { failover&.fail_over! },
|
|
@@ -221,6 +222,7 @@ module Pgbus
|
|
|
221
222
|
pg_connection: pg_connection,
|
|
222
223
|
dispatch_queue: @dispatch_queue,
|
|
223
224
|
health_check_ms: @config.streams_listen_health_check_ms,
|
|
225
|
+
ack_timeout_ms: @config.streams_listen_ack_timeout_ms,
|
|
224
226
|
# Opt-in dispatch-queue backpressure (issue #315 item 3). 0 =
|
|
225
227
|
# unbounded (default). The queue itself stays an unbounded
|
|
226
228
|
# Queue.new so the request-thread Connect push and the dispatcher's
|
|
@@ -335,11 +337,12 @@ module Pgbus
|
|
|
335
337
|
# runs its own PrimaryValidator (retrying on a replica) and skips the
|
|
336
338
|
# probe to stay cheap, mirroring Pgbus::Process::NotifyListener.
|
|
337
339
|
# String/Hash go through DedicatedConnection so a :session-mode
|
|
338
|
-
# `:variables` key never reaches PG.connect (issue #352)
|
|
340
|
+
# `:variables` key never reaches PG.connect (issue #352), with TCP
|
|
341
|
+
# keepalives so a dead socket errors between keepalives (issue #664).
|
|
339
342
|
def build_raw_pg_connection
|
|
340
343
|
opts = @config.streams_connection_options
|
|
341
344
|
case opts
|
|
342
|
-
when String, Hash then Pgbus::DedicatedConnection.
|
|
345
|
+
when String, Hash then Pgbus::DedicatedConnection.connect_with_keepalives(opts)
|
|
343
346
|
when Proc
|
|
344
347
|
# The Proc branch in connection_options typically returns
|
|
345
348
|
# ActiveRecord::Base.connection.raw_connection — a pooled
|
|
@@ -28,8 +28,10 @@ module Pgbus
|
|
|
28
28
|
# the pipe is touched from other threads
|
|
29
29
|
#
|
|
30
30
|
# Health check: once per health_check_ms the listener runs `SELECT 1` as
|
|
31
|
-
# a
|
|
32
|
-
# cycle, and a stream of wakes or notifications cannot postpone it.
|
|
31
|
+
# a keepalive, on a deadline: an idle connection sees exactly one per
|
|
32
|
+
# cycle, and a stream of wakes or notifications cannot postpone it. The
|
|
33
|
+
# interval only bounds how long a silently dropped LISTEN goes unnoticed:
|
|
34
|
+
# no wait budget is derived from it (ack_timeout_ms is, issue #664). If
|
|
33
35
|
# it raises, the listener rebuilds a FRESH connection via
|
|
34
36
|
# connection_factory and re-LISTENs every channel in `@listening_to`.
|
|
35
37
|
# This is the fix for design doc §11 #1 (silently dropped LISTEN
|
|
@@ -62,10 +64,14 @@ module Pgbus
|
|
|
62
64
|
|
|
63
65
|
RECONNECT_BACKOFF_SECONDS = 0.5
|
|
64
66
|
|
|
65
|
-
# Grace added to
|
|
66
|
-
#
|
|
67
|
+
# Grace added to the ack budget when #stop joins the listener thread.
|
|
68
|
+
# See #stop_join_timeout.
|
|
67
69
|
STOP_JOIN_GRACE_SECONDS = 5
|
|
68
70
|
|
|
71
|
+
# Default for ack_timeout_ms: the old 250 ms cycle plus its 1 s margin,
|
|
72
|
+
# now fixed instead of following the keepalive (issue #664).
|
|
73
|
+
DEFAULT_ACK_TIMEOUT_MS = 1_250
|
|
74
|
+
|
|
69
75
|
attr_reader :listening_to
|
|
70
76
|
|
|
71
77
|
# @param connection_factory [#call] builds a FRESH PG connection on each
|
|
@@ -84,14 +90,18 @@ module Pgbus
|
|
|
84
90
|
# most once per `maintenance.interval` seconds with THIS connection, so a
|
|
85
91
|
# maintenance query (e.g. pg_stat_activity headroom) reuses the existing
|
|
86
92
|
# idle connection instead of opening its own. nil disables it.
|
|
93
|
+
# `ack_timeout_ms:` bounds how long ensure_listening waits for this
|
|
94
|
+
# thread's ack, and (plus a grace) how long #stop joins it
|
|
95
|
+
# (streams_listen_ack_timeout_ms; issue #664).
|
|
87
96
|
def initialize(pg_connection:, dispatch_queue:, health_check_ms:,
|
|
88
|
-
connection_factory:,
|
|
89
|
-
logger: Pgbus.logger, clock: nil)
|
|
97
|
+
connection_factory:, ack_timeout_ms: DEFAULT_ACK_TIMEOUT_MS, dispatch_queue_limit: 0,
|
|
98
|
+
maintenance: nil, logger: Pgbus.logger, clock: nil)
|
|
90
99
|
raise ArgumentError, "connection_factory is required" unless connection_factory.respond_to?(:call)
|
|
91
100
|
|
|
92
101
|
@conn = pg_connection
|
|
93
102
|
@dispatch_queue = dispatch_queue
|
|
94
103
|
@health_check_ms = health_check_ms
|
|
104
|
+
@ack_timeout_ms = ack_timeout_ms
|
|
95
105
|
@logger = logger
|
|
96
106
|
@connection_factory = connection_factory
|
|
97
107
|
@dispatch_queue_limit = dispatch_queue_limit
|
|
@@ -182,9 +192,12 @@ module Pgbus
|
|
|
182
192
|
# a process-killing SEGV, not a rescuable PG::Error (issue #375).
|
|
183
193
|
# The listener thread is the sole owner of @conn and closes it in
|
|
184
194
|
# run_loop's ensure; clearing @running above is the whole stop signal,
|
|
185
|
-
# and the wake makes the thread see it at once. The join budget
|
|
186
|
-
#
|
|
187
|
-
# keepalive or
|
|
195
|
+
# and the wake makes the thread see it at once. The join budget
|
|
196
|
+
# (the ack budget plus a grace, 6.25 s by default) covers a thread
|
|
197
|
+
# inside a normal keepalive or reconnect attempt. A keepalive stuck
|
|
198
|
+
# on a dead socket can outlast it (libpq's TCP bound is ~11 s): #stop
|
|
199
|
+
# then logs "did not stop" and the thread exits on its own once
|
|
200
|
+
# libpq raises.
|
|
188
201
|
wake
|
|
189
202
|
joined = @thread.nil? || @thread.join(stop_join_timeout)
|
|
190
203
|
# A thread past its budget stays referenced, so #start refuses to
|
|
@@ -208,9 +221,9 @@ module Pgbus
|
|
|
208
221
|
# in the read_after result nor delivered as a WakeMessage.
|
|
209
222
|
#
|
|
210
223
|
# The wake pipe makes the typical wait a thread hop (issue #630). The
|
|
211
|
-
# wait is bounded by
|
|
212
|
-
#
|
|
213
|
-
#
|
|
224
|
+
# wait is bounded by the ack budget (streams_listen_ack_timeout_ms,
|
|
225
|
+
# default 1.25 s; issue #664), so a stuck listener can't hang the
|
|
226
|
+
# dispatcher forever.
|
|
214
227
|
def ensure_listening(queue_name)
|
|
215
228
|
# A connect the dispatcher drains during shutdown (Instance#shutdown!
|
|
216
229
|
# stops the listener first) must not wait out the ack budget.
|
|
@@ -411,18 +424,20 @@ module Pgbus
|
|
|
411
424
|
|
|
412
425
|
# Caller wait budget for ensure_listening's ack. The wake pipe
|
|
413
426
|
# makes the usual wait a thread hop; a thread busy in a keepalive
|
|
414
|
-
# or a reconnect gets
|
|
415
|
-
#
|
|
427
|
+
# or a reconnect gets ack_timeout_ms, so the dispatcher fails loud
|
|
428
|
+
# rather than hanging if the thread is dead. Not derived from the
|
|
429
|
+
# keepalive interval: raising that must not slow this failure path
|
|
430
|
+
# (issue #664).
|
|
416
431
|
def ack_timeout
|
|
417
|
-
|
|
432
|
+
@ack_timeout_ms / 1000.0
|
|
418
433
|
end
|
|
419
434
|
|
|
420
435
|
# How long #stop waits for the listener thread to notice the cleared
|
|
421
436
|
# @running and finish teardown. The wake pipe interrupts its select at
|
|
422
437
|
# once, but it may be inside a keepalive or a reconnect attempt, so the
|
|
423
|
-
# budget
|
|
438
|
+
# budget is the ack budget plus grace, independent of the keepalive.
|
|
424
439
|
def stop_join_timeout
|
|
425
|
-
|
|
440
|
+
ack_timeout + STOP_JOIN_GRACE_SECONDS
|
|
426
441
|
end
|
|
427
442
|
|
|
428
443
|
# The channel joins @listening_to BEFORE the exec: if the LISTEN
|
|
@@ -158,13 +158,18 @@ module Pgbus
|
|
|
158
158
|
|
|
159
159
|
def default_listener_factory
|
|
160
160
|
lambda do |dispatch_queue:|
|
|
161
|
-
|
|
161
|
+
# TCP keepalives and the ack budget as on a per-worker listener:
|
|
162
|
+
# in :master scope this is the connection that LISTENs (issue #664).
|
|
163
|
+
build_connection = lambda do
|
|
164
|
+
Pgbus::DedicatedConnection.connect_with_keepalives(@config.streams_connection_options)
|
|
165
|
+
end
|
|
162
166
|
conn = build_connection.call
|
|
163
167
|
Pgbus::Process::PrimaryValidator.validate_primary!(conn)
|
|
164
168
|
Listener.new(
|
|
165
169
|
pg_connection: conn,
|
|
166
170
|
dispatch_queue: dispatch_queue,
|
|
167
171
|
health_check_ms: @config.streams_listen_health_check_ms,
|
|
172
|
+
ack_timeout_ms: @config.streams_listen_ack_timeout_ms,
|
|
168
173
|
connection_factory: build_connection,
|
|
169
174
|
dispatch_queue_limit: @config.streams_dispatch_queue_limit,
|
|
170
175
|
logger: @logger
|
|
@@ -398,7 +398,7 @@ module Pgbus
|
|
|
398
398
|
#
|
|
399
399
|
# The fast path holds while the listener reconnects, on purpose.
|
|
400
400
|
# Waiting on a reconnecting listener would cost every connect the
|
|
401
|
-
# full ack budget (
|
|
401
|
+
# full ack budget (streams_listen_ack_timeout_ms) for as long as the
|
|
402
402
|
# database stays unreachable: the storm this rule removes. A
|
|
403
403
|
# connect in that window has the same exposure as every connection
|
|
404
404
|
# already registered: a durable stream's next wake after the
|
data/sig/pgbus/configuration.rbs
CHANGED
|
@@ -239,6 +239,8 @@ module Pgbus
|
|
|
239
239
|
def streams_idle_timeout=: (untyped) -> untyped
|
|
240
240
|
def streams_listen_health_check_ms: () -> Integer
|
|
241
241
|
def streams_listen_health_check_ms=: (untyped) -> untyped
|
|
242
|
+
def streams_listen_ack_timeout_ms: () -> Integer
|
|
243
|
+
def streams_listen_ack_timeout_ms=: (untyped) -> untyped
|
|
242
244
|
def streams_listen_scope: () -> Symbol
|
|
243
245
|
def streams_listen_scope=: (untyped scope) -> untyped
|
|
244
246
|
def streams_max_connections: () -> Integer
|