pgbus 0.25.0 → 0.25.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 1b3feabb297f44a982729733bf7c45e3dfa7aa0ce389e43bb6451b699636bf8f
4
- data.tar.gz: 4729a4f127dee012b73d4d5c714bbab574ef134d83147be9d66741bb0ef556d3
3
+ metadata.gz: d2e222e8f30d9e69de08103ecf574c4166a4804d90aff9db01df780451255b0f
4
+ data.tar.gz: 8744453ae4652d2dc6e163b81c5104b5e25e62bf6c7dff57c5b4e5a5c5807208
5
5
  SHA512:
6
- metadata.gz: 0ba843ed880d64b2250d9c7bc122a3d78d4c4e5726d6a87408d91b703419585c93ff3c37e46bc65ad5c788bd7ea5c2542a22323cada6e9b9fefe81fc2c9ed223
7
- data.tar.gz: 8bd1142c5aaa183cb3c1f6b27b66bbe6e69d8f988b3d08068dac6934d57eb48bff444da6b3fc04bb71a1e1224b377e15f60aad1e2206cd2a4026bb39adbd5c79
6
+ metadata.gz: e0f35bcb78c6f707f6adb837f6845c21143543050891bc5dd677d935d571b2dfb93fe6e3d048ceecd2a4d75596c5f811cee2fa6076dba0c51c5b030a1ad527bb
7
+ data.tar.gz: c4d933bc1d90cae70c82888f31a6bbc73d83cc1a8e1ca9beccd754c310d03289e4edce4170a11a85f05c6310ecb95ec6a005c50cf9791fc5ab0b594252f0c5a3
data/CHANGELOG.md CHANGED
@@ -4,6 +4,14 @@
4
4
 
5
5
  - **Deploy tasks: `after:` makes a task wait until the tasks it names have completed, and every run records its outcome (issue #660).** `after:` takes a task key or a list of keys declared in the loaded configuration (any of `deploy_tasks_files`). The scheduler keeps such a task armed, and does not enqueue it, until each named task has a run with `completed_at` at its current version (any revision; an `every_deploy` dependency needs one on this revision). A rollover run and a Run now run both count, so a failed dependency that an operator fixes and runs again with Run now unblocks its dependents without a new deploy. A waiting task records no failure and no backoff, and fires on the first tick after its last dependency completes; the check is one query per tick, made only while an armed task has dependencies. At scheduler boot a task whose `after:` names an unknown key (`after: unknown task 'x'`), sits on a cycle (`after: cycle a → b → a`, a self-reference included), or names a task that is itself skipped (`after: 'x' is skipped`) is skipped with a warning. Run outcomes: `Deploy::Runner#enqueue` now creates the `pgbus_deploy_task_runs` row before the enqueue and passes its id in the job payload under `pgbus_deploy_task_run_id` (`Pgbus::ActiveJob::DeployTaskRunId`, included on `ActiveJob::Base` next to `BatchId`; an unmarked job's payload is unchanged), fills in `queue_name` and `msg_id` after it, and deletes the row when the enqueue fails. The executor stamps `completed_at` when `perform` returned having done its work without re-enqueuing the job (a `retry_on` attempt whose retry is queued, or a `discard_on` block that queues one, leaves it to the retry; a job that rescues inside `perform` and returns has completed) and `failed_at` when ActiveJob discarded the job without queuing a retry (`discard_on`, with or without a block, or a `retry_on` that gave up through its block, where one without a block re-raises and ends dead-lettered as before; `Pgbus::ActiveJob::DeployTaskRunId` marks the instance in ActiveJob's `run_after_discard_procs`, after the job's own `after_discard` procs, so a discarded dependency reads `failed` and its dependents keep waiting, issue #662), when the job is dead-lettered, discarded as a `:while_executing` duplicate, or its retry never left (dropped by its own concurrency limit, `on_conflict: :discard`, or aborted by a `before_enqueue` callback); a failed write is reported and never fails the job, and a job without the key pays one Hash lookup. An enqueue the job's own `on_conflict: :discard` / `:log` drops now counts as a failed enqueue (the run is deleted and the task re-armed with backoff). A run row without `queue_name` (its enqueue never finished) does not count as a run, so the version stays owed. The dispatcher's pruning keeps the newest completed run per task and version. Dashboard: a task's state says "Waiting for N dependencies to complete" and lists each one as `not run`, `running` or `failed`; the task page shows `after:` and an Outcome column in the run history; Run now on a waiting task is refused with a flash naming the dependencies (`DataSource#enqueue_deploy_task_now(id, ignore_dependencies: false)` returns `:blocked`) unless "Ignore dependencies" is ticked on the task page. `pgbus_deploy_tasks` (MCP) lists `dependencies` and `waiting_on`. New generator `pgbus:add_deploy_task_run_outcomes` adds `pgbus_deploy_task_runs.completed_at` / `failed_at`, `pgbus_deploy_tasks.dependencies` and an index on `(task_key, completed_at)`; `pgbus:update` queues it, and fresh `pgbus:install` / `pgbus:add_deploy_tasks` include the columns. Until it is migrated, a task with `after:` is skipped with a warning naming the generator and no outcome is written. **Upgrade note:** runs recorded before the migration have no outcome and satisfy nothing, so when an `after:` names a task whose only runs predate the upgrade, Run now that task once (or bump its version; an `every_deploy` dependency needs a run on the current revision, so only Run now helps). Delivery stays at-least-once: a scheduler that dies between claiming a task and enqueuing it leaves the task unarmed until the next scheduler boot re-arms it, and a run whose enqueue could not be recorded runs again after one, so deploy tasks must be idempotent. Refs #660.
6
6
 
7
+ ### Changed
8
+
9
+ - **Changed default: the streamer's LISTEN keepalive `streams_listen_health_check_ms` is 5 000 ms instead of 250, and the listener's ack and stop budgets no longer follow it (issue #664).** Every web process running the streamer sent `SELECT 1` on its LISTEN connection four times a second, idle or not: 345 600 queries a day per process, the most frequent statement on a small fleet's database. Since #630 a connect no longer waits on that cycle (the listener wakes through a pipe), so the interval only bounds how long a LISTEN the server dropped on a still-open socket goes unnoticed; now one `SELECT 1` per 5 s idle cycle, 20× fewer. The interval was also wired into three wait budgets, so raising it slowed the failure paths: they now come from a new setting, **`streams_listen_ack_timeout_ms`** (default 1 250, a positive Integer, today's 250 + 1 000). `Listener#ensure_listening` waits at most that for a stuck listener's ack, `Listener#stop` joins for that plus its 5 s grace, and with `streams_listen_scope :master` the worker's `HubClient` waits that plus 1 s before failing over, as before relative to the hub's own budget; the master hub's listener gets the same ack budget. The streamer's dedicated LISTEN connection (per worker, and the master hub's) now opens with libpq TCP keepalives, `keepalives=1 keepalives_idle=5 keepalives_interval=2 keepalives_count=3 tcp_user_timeout=11000`, so a dead socket errors within ~11 s by these settings (5 s idle + 3 probes 2 s apart, and the matching `tcp_user_timeout`; on Linux, where `tcp_user_timeout` applies; the bound follows the settings, it was not measured) even between keepalives, and the listener reconnects and re-LISTENs; a keyword the connection options already set wins, a connection that names a service (`service=` or `PGSERVICE`) gets none so its service file's values stay in force, and on libpq < 12 none is added (the same gate `Client` uses for its read bounds, now `Pgbus::DedicatedConnection.libpq_supports_socket_bounds?`). The worker `NotifyListener` and the doctor's connection are unchanged. The pool autoscaler's check runs on the keepalive tick, so a `streams_pool_autoscale_interval` under 5 s (the default is 300 s) now runs every 5 s. A dedicated connection URL ending in a bare `?` no longer gets an empty `?&` parameter that failed the connect. An explicit `streams_listen_health_check_ms` keeps exactly its value: **pin `config.streams_listen_health_check_ms = 250` for the old keepalive.** If you set it to 5 000 as the workaround, the ack budget drops from 6 s back to 1.25 s. Measured on the same machine (`rake bench:streams`, which pins 100 ms; `rake bench:streams_scale` at M = 100 with the default, 250 ms before and 5 000 ms after, two rounds each): section 4 concurrent connect 0.91–3.07 ms per connect before, 0.86–3.06 ms after; scale bench, all 100 clients registered in 0.11–0.12 s before, 0.09–0.11 s after (one 0.24 s round at load 32, 0.09 s on a re-run), 0 misses either way. Refs #664.
10
+
11
+ ### Fixed
12
+
13
+ - **A presence table without the seat columns is reported, and its schema probe backs off (issue #665).** `Pgbus::Streams::Presence.seats_supported?` kept a "no seats" answer for 30 s, so an app that upgraded without running `rails g pgbus:add_presence_seats` queried `information_schema.columns` twice a minute per process, forever, and nothing said why. A "no" is now kept for 30 s, then twice as long after each further "no", up to 10 minutes (`SCHEMA_RECHECK_MAX_SECONDS = 600`); a true answer is still kept for the life of the process, `reset_schema_cache!` still re-probes at once and starts the backoff again from 30 s, and two threads that probe at the same moment count as one step. The backoff holds in a process that makes no member-keyed `leave`, `touch` or `sweep!`: each of those still re-probes a cached "no" before it runs, because it would delete or refresh seat rows on a table migrated under the process, and restarts the backoff at 30 s. Connection-driven presence on such a table still touches once per connection per heartbeat, unchanged here (#666). A member-keyed `join` that hits the migrated index still re-probes at once. The first "no" logs one warning per process naming the fix (`rails generate pgbus:add_presence_seats && rails db:migrate`). `pgbus doctor` has a fifteenth check, `Presence seats`: a warning with the same fix when `pgbus_presence_members` exists without `connection_id`, ok when the column exists or presence is not installed; it runs in the boot preflight and never blocks a strict boot. New `Pgbus::Streams::Presence.seats_state` (`:absent`, `:missing`, `:present`), read on the connection presence uses. Refs #665.
14
+
7
15
  ## [0.24.0] - 2026-11-11
8
16
 
9
17
  ### Added
data/README.md CHANGED
@@ -1556,7 +1556,8 @@ Pgbus.configure do |c|
1556
1556
  c.streams_peer_close_interval = 1.0 # notice a closed client at once; nil = heartbeat only
1557
1557
  c.streams_max_connections = 2_000 # per web-server process (Puma worker or Falcon process)
1558
1558
  c.streams_idle_timeout = 3_600 # close idle connections after 1h
1559
- c.streams_listen_health_check_ms = 250 # PG LISTEN keepalive cadence + the ack budget for a stuck listener; a stream's first connect waits a thread hop on a live one, later connects not at all
1559
+ c.streams_listen_health_check_ms = 5_000 # PG LISTEN keepalive: one SELECT 1 per idle cycle, the bound on noticing a dropped LISTEN (a dead socket: ~11 s by the TCP keepalive settings it adds when the connection options set none of their own; libpq >= 12, none added on a `service=` / PGSERVICE connection)
1560
+ c.streams_listen_ack_timeout_ms = 1_250 # how long a stream's first connect waits for a stuck listener (a live one acks in a thread hop); :master scope adds 1 s
1560
1561
  c.streams_write_deadline_ms = 5_000 # hijacked-socket write_nonblock deadline (connect replay, sentinels)
1561
1562
  c.streams_fanout_write_deadline_ms = 250 # hijacked-socket fan-out and heartbeat writes; a slower client reconnects (Falcon's streaming body ignores both)
1562
1563
  c.streams_falcon_streaming_body = false # opt-in: Falcon-native streaming body
@@ -1961,7 +1962,7 @@ Pgbus.configure do |c|
1961
1962
  end
1962
1963
  ```
1963
1964
 
1964
- Keep each ttl above `streams_heartbeat_interval` (pgbus warns at boot otherwise): a shorter one expires live seats between touches. On a seat table `sweep!` only expires member-keyed rows; seats expire through the heartbeat. A process that probed the table before the migration re-checks within 30 seconds (and at once if a member-keyed `join` outside a transaction hits the new index). A raising hook is logged, not propagated. Library authors can probe for seat support with `Pgbus::Streams::Presence.instance_method(:join).parameters.include?([:key, :connection_id])` (the gem) and `Pgbus::Streams::Presence.seats_supported?` (the table).
1965
+ Keep each ttl above `streams_heartbeat_interval` (pgbus warns at boot otherwise): a shorter one expires live seats between touches. On a seat table `sweep!` only expires member-keyed rows; seats expire through the heartbeat. A process that probed the table before the migration re-checks after 30 seconds, then after twice as long each time it still finds no seats, up to every 10 minutes. A member-keyed `leave`, `touch` or `sweep!` re-checks before it runs and starts that backoff again at 30 seconds, and a member-keyed `join` outside a transaction re-checks at once if it hits the new index. The first time it finds no seats it logs one warning naming the migration, and `pgbus doctor` warns under `Presence seats`; restart the processes after migrating to pick the columns up at once. A raising hook is logged, not propagated. Library authors can probe for seat support with `Pgbus::Streams::Presence.instance_method(:join).parameters.include?([:key, :connection_id])` (the gem) and `Pgbus::Streams::Presence.seats_supported?` (the table).
1965
1966
 
1966
1967
  ### Stream stats (opt-in)
1967
1968
 
@@ -2242,7 +2243,7 @@ pgbus help # Show help
2242
2243
 
2243
2244
  #### pgbus doctor
2244
2245
 
2245
- A single preflight command that answers "is this environment healthy enough to run?" — useful as a deploy or CI gate. It runs 14 checks and never raises; a broken environment turns every probe into a failed/warned check instead of a crash:
2246
+ A single preflight command that answers "is this environment healthy enough to run?" — useful as a deploy or CI gate. It runs 15 checks and never raises; a broken environment turns every probe into a failed/warned check instead of a crash:
2246
2247
 
2247
2248
  | Check | Fails (`:fail`) when | Warns (`:warn`) when |
2248
2249
  |---|---|---|
@@ -2260,6 +2261,7 @@ A single preflight command that answers "is this environment healthy enough to r
2260
2261
  | Dedicated connections | Streamer LISTEN and/or worker notify dedicated path cannot connect | — |
2261
2262
  | Connection budget | — (informational: prints how many direct LISTEN connections the current config pins — 1 per host under `worker_notify_scope: :supervisor`, one per fork under `:fork`; streams add 1 per web host under `streams_listen_scope: :master` or 1 per web process under `:process`) | — |
2262
2263
  | Completion commit | — (informational: prints whether completions commit synchronously, the default, or with `synchronous_commit` off under `completion_synchronous_commit = false`) | `completion_synchronous_commit = false` with `native_statements = false`, or with pgbus on ActiveRecord's shared connection, where it has no effect |
2264
+ | Presence seats | — | `pgbus_presence_members` exists without the `connection_id` column: presence runs member-keyed (one row per member, no seats) until `rails generate pgbus:add_presence_seats && rails db:migrate` runs; ok when the column exists or presence is not installed |
2263
2265
 
2264
2266
  ```bash
2265
2267
  pgbus doctor # prints the report; exit 1 unless every check passed
data/lib/pgbus/client.rb CHANGED
@@ -2253,10 +2253,9 @@ module Pgbus
2253
2253
  end
2254
2254
 
2255
2255
  # Whether the linked libpq accepts the tcp_user_timeout / keepalives conninfo
2256
- # keywords. Added in libpq 12; an older libpq raises "invalid connection
2257
- # option" and fails the connection, so we must not emit them there.
2256
+ # keywords (libpq 12+). One gate for every connection pgbus opens.
2258
2257
  def libpq_supports_socket_bounds?
2259
- defined?(PG) && PG.respond_to?(:library_version) && PG.library_version >= 120_000
2258
+ DedicatedConnection.libpq_supports_socket_bounds?
2260
2259
  end
2261
2260
 
2262
2261
  # Hash form maps 1:1 to libpq keywords, so no escaping/encoding is needed.
@@ -328,6 +328,7 @@ module Pgbus
328
328
  attr_accessor :streams_enabled, :streams_path, :streams_signed_name_secret,
329
329
  :streams_default_retention, :streams_retention, :streams_heartbeat_interval, :streams_peer_close_interval,
330
330
  :streams_max_connections, :streams_idle_timeout, :streams_listen_health_check_ms,
331
+ :streams_listen_ack_timeout_ms,
331
332
  :streams_write_deadline_ms, :streams_falcon_streaming_body,
332
333
  :streams_stats_enabled, :streams_test_mode,
333
334
  :streams_orphan_sweep_interval, :streams_orphan_threshold,
@@ -574,7 +575,9 @@ module Pgbus
574
575
  # Slow maintenance cadence (seconds) — the autoscaler runs as a periodic
575
576
  # check on the streamer's idle LISTEN connection (pghero-style), not a busy
576
577
  # loop. 5 minutes: the interval itself debounces, so there is no per-sample
577
- # hysteresis; a sustained burst converges over a few checks.
578
+ # hysteresis; a sustained burst converges over a few checks. The check
579
+ # runs on the listener's keepalive tick, so an interval shorter than
580
+ # streams_listen_health_check_ms runs once per keepalive instead.
578
581
  @streams_pool_autoscale_interval = 300.0
579
582
  # Distinctive application_name prefix stamped on streams-pool connections
580
583
  # so the autoscaler can count peer processes (DISTINCT application_name
@@ -595,16 +598,23 @@ module Pgbus
595
598
  @streams_peer_close_interval = 1.0
596
599
  @streams_max_connections = 2_000
597
600
  @streams_idle_timeout = 3_600 # 1 hour
598
- # 250ms — this value plays two roles: (1) the TCP keepalive
599
- # interval for the streamer's PG LISTEN connection, and (2) the
600
- # budget (plus 1s) a stream's first connect waits for a stuck local
601
- # Listener to acknowledge its ensure_listening (with
602
- # streams_listen_scope :master, HubClient waits this cycle plus 2s
603
- # before failing over). A live listener
604
- # is woken through a pipe and acks within a thread hop, and a
605
- # connect on a stream that already has a subscriber does not
606
- # wait at all (issue #630), so connects no longer pay this cycle.
607
- @streams_listen_health_check_ms = 250
601
+ # 5 s — the keepalive interval of the streamer's PG LISTEN
602
+ # connection: one `SELECT 1` per idle cycle, and the bound on how
603
+ # long a LISTEN the server dropped on a still-open socket goes
604
+ # unnoticed. A socket whose peer is gone is also detected, between
605
+ # keepalives, by the TCP keepalives on that connection: ~11 s by their
606
+ # settings (not measured; libpq >= 12; not added to a connection that
607
+ # names a service, whose service file keeps them). Connects do not wait on
608
+ # this cycle: the listener wakes through a pipe (issue #630). Was
609
+ # 250 ms, which ran 4 queries a second per web process (issue #664).
610
+ @streams_listen_health_check_ms = 5_000
611
+ # 1.25 s — how long a stream's first connect waits for a stuck local
612
+ # Listener to acknowledge its ensure_listening; #stop's join budget
613
+ # is this plus a grace. With streams_listen_scope :master, HubClient
614
+ # waits this plus 1 s before failing over. Independent of the
615
+ # keepalive above (issue #664). A live listener acks within a thread
616
+ # hop.
617
+ @streams_listen_ack_timeout_ms = 1_250
608
618
  @streams_write_deadline_ms = 5_000
609
619
  # Deadline (ms) for a socket write made INSIDE the dispatcher's
610
620
  # serial fanout loop (handle_durable_wake / handle_ephemeral_wake).
@@ -1308,6 +1318,10 @@ module Pgbus
1308
1318
  raise Pgbus::ConfigurationError, "streams_listen_health_check_ms must be a positive integer"
1309
1319
  end
1310
1320
 
1321
+ unless streams_listen_ack_timeout_ms.is_a?(Integer) && streams_listen_ack_timeout_ms.positive?
1322
+ raise Pgbus::ConfigurationError, "streams_listen_ack_timeout_ms must be a positive integer"
1323
+ end
1324
+
1311
1325
  unless streams_write_deadline_ms.is_a?(Integer) && streams_write_deadline_ms.positive?
1312
1326
  raise Pgbus::ConfigurationError, "streams_write_deadline_ms must be a positive integer"
1313
1327
  end
@@ -23,6 +23,20 @@ module Pgbus
23
23
  # hash — or PGAPPNAME in the environment — always wins.
24
24
  APP_NAME = "pgbus-listen"
25
25
 
26
+ # libpq TCP keepalives for the streamer's LISTEN connection (issue #664):
27
+ # probing starts after 5 idle seconds, every 2 s, 3 times, so a dead
28
+ # socket errors within ~11 s even between the listener's own keepalive
29
+ # queries; tcp_user_timeout bounds data the peer never acknowledges by
30
+ # the same 11 s. Client-side keywords: never in the startup packet, so a
31
+ # pooler cannot reject them.
32
+ LISTEN_KEEPALIVES = {
33
+ keepalives: 1,
34
+ keepalives_idle: 5,
35
+ keepalives_interval: 2,
36
+ keepalives_count: 3,
37
+ tcp_user_timeout: 11_000
38
+ }.freeze
39
+
26
40
  def connect(opts)
27
41
  require "pg" unless defined?(::PG::Connection)
28
42
  case opts
@@ -41,14 +55,88 @@ module Pgbus
41
55
  def with_app_name(conninfo)
42
56
  return conninfo if conninfo.include?("application_name")
43
57
 
44
- if conninfo.include?("://")
45
- separator = conninfo.include?("?") ? "&" : "?"
46
- "#{conninfo}#{separator}fallback_application_name=#{APP_NAME}"
58
+ if url?(conninfo)
59
+ "#{conninfo}#{query_separator(conninfo)}fallback_application_name=#{APP_NAME}"
47
60
  else
48
61
  "#{conninfo} fallback_application_name=#{APP_NAME}".strip
49
62
  end
50
63
  end
51
64
 
65
+ # connect, plus LISTEN_KEEPALIVES: the streamer's LISTEN connection. A
66
+ # keyword the options already set wins.
67
+ def connect_with_keepalives(opts)
68
+ require "pg" unless defined?(::PG::Connection)
69
+ connect(with_keepalives(opts))
70
+ end
71
+
72
+ # Whether the linked libpq accepts the tcp_user_timeout / keepalives conninfo
73
+ # keywords. Added in libpq 12; an older libpq raises "invalid connection
74
+ # option" and fails the connection, so they must not be emitted there.
75
+ def libpq_supports_socket_bounds?
76
+ defined?(::PG) && ::PG.respond_to?(:library_version) && ::PG.library_version >= 120_000
77
+ end
78
+
79
+ # Fill in the LISTEN_KEEPALIVES keywords the options do not already set.
80
+ # Unchanged on libpq < 12 (see libpq_supports_socket_bounds?), and for a
81
+ # connection that names a service (service=, or PGSERVICE): libpq applies
82
+ # the service file only to keywords the conninfo leaves unset, so an added
83
+ # keyword would override the service's own.
84
+ def with_keepalives(opts)
85
+ return opts unless libpq_supports_socket_bounds?
86
+ return opts if ENV.fetch("PGSERVICE", nil).to_s != ""
87
+
88
+ case opts
89
+ when Hash
90
+ return opts if opts.key?(:service) || opts.key?("service")
91
+
92
+ missing = LISTEN_KEEPALIVES.reject { |key, _| opts.key?(key) || opts.key?(key.to_s) }
93
+ opts.merge(missing)
94
+ when String then append_keepalives(opts)
95
+ else opts
96
+ end
97
+ end
98
+
99
+ # URL query-param style for postgres:// URLs, space-separated keyword
100
+ # style otherwise (as with_app_name does). The keywords already set are
101
+ # read with libpq's own parser, so a keyword quoted inside another value
102
+ # does not count; a conninfo libpq cannot parse is left for PG.connect to
103
+ # report.
104
+ def append_keepalives(conninfo)
105
+ set = conninfo_keywords(conninfo)
106
+ return conninfo if set.nil? || set.include?("service")
107
+
108
+ missing = LISTEN_KEEPALIVES.reject { |key, _| set.include?(key.to_s) }
109
+ return conninfo if missing.empty?
110
+
111
+ if url?(conninfo)
112
+ "#{conninfo}#{query_separator(conninfo)}#{missing.map { |key, value| "#{key}=#{value}" }.join("&")}"
113
+ else
114
+ "#{conninfo} #{missing.map { |key, value| "#{key}=#{value}" }.join(" ")}".strip
115
+ end
116
+ end
117
+
118
+ # A postgres:// or postgresql:// URL, as libpq tells them apart: by the
119
+ # prefix, so a key=value conninfo with :// inside a value (a password)
120
+ # stays key=value.
121
+ def url?(conninfo)
122
+ conninfo.start_with?("postgres://", "postgresql://")
123
+ end
124
+
125
+ # The keywords a conninfo String sets, or nil when libpq cannot parse it.
126
+ def conninfo_keywords(conninfo)
127
+ ::PG::Connection.conninfo_parse(conninfo).filter_map { |option| option[:keyword] if option[:val] }
128
+ rescue ::PG::Error
129
+ nil
130
+ end
131
+
132
+ # "?" to start a URL's query string, "&" to extend one, nothing after a
133
+ # trailing "?" or "&" (an empty parameter would fail the connect).
134
+ def query_separator(url)
135
+ return "" if url.end_with?("?", "&")
136
+
137
+ url.include?("?") ? "&" : "?"
138
+ end
139
+
52
140
  def connect_from_hash(opts)
53
141
  variables = opts[:variables]
54
142
  conn = ::PG.connect(**with_app_name_hash(opts.except(:variables)))
data/lib/pgbus/doctor.rb CHANGED
@@ -5,7 +5,7 @@ require "pgbus/mcp/health_analyzer"
5
5
 
6
6
  module Pgbus
7
7
  # Preflight diagnostics for a pgbus deployment — the single command that
8
- # answers "is this environment healthy enough to run?". Runs fourteen checks and
8
+ # answers "is this environment healthy enough to run?". Runs fifteen checks and
9
9
  # returns a machine-readable result plus a human report, so `pgbus doctor`
10
10
  # and `rake pgbus:doctor` can gate a deploy or CI run (exit 0 on success,
11
11
  # 1 on any failure).
@@ -44,7 +44,8 @@ module Pgbus
44
44
  "Primary affinity" => :check_primary,
45
45
  "Dedicated connections" => :check_dedicated_connections,
46
46
  "Connection budget" => :check_connection_budget,
47
- "Completion commit" => :check_completion_commit
47
+ "Completion commit" => :check_completion_commit,
48
+ "Presence seats" => :check_presence_seats
48
49
  }.freeze
49
50
 
50
51
  # Process liveness reads the pgbus_processes table (via HealthAnalyzer), so
@@ -448,6 +449,27 @@ module Pgbus
448
449
  Check.new(name: "Completion commit", status: :warn, detail: "#{e.class}: #{e.message}")
449
450
  end
450
451
 
452
+ # Presence seats (issue #665). Without the add_presence_seats migration
453
+ # presence runs member-keyed (one row per member, no seats), and every
454
+ # process re-probes information_schema with backoff. A supported degraded
455
+ # mode, so a warning; ok when the columns exist or presence is not installed.
456
+ # Read through Presence, not the Client: presence reads ActiveRecord's
457
+ # connection (Pgbus::BusRecord under connects_to), which a database_url or
458
+ # connection_params Client connection need not share.
459
+ def check_presence_seats
460
+ case Pgbus::Streams::Presence.seats_state
461
+ when :absent then Check.new(name: "Presence seats", status: :ok, detail: "presence not installed")
462
+ when :present then Check.new(name: "Presence seats", status: :ok, detail: "pgbus_presence_members migrated with seats")
463
+ else
464
+ Check.new(name: "Presence seats", status: :warn,
465
+ detail: "pgbus_presence_members has no connection_id column — presence runs member-keyed " \
466
+ "(one row per member, no seats) and each process re-probes the schema; run " \
467
+ "`#{Pgbus::Streams::Presence::SEATS_MIGRATION_FIX}`")
468
+ end
469
+ rescue StandardError => e
470
+ Check.new(name: "Presence seats", status: :warn, detail: "could not read the schema (#{e.class}: #{e.message})")
471
+ end
472
+
451
473
  # How many direct LISTEN connections this config pins at steady state
452
474
  # (issue #381) — informational (always :ok) so operators can do capacity
453
475
  # math on the pooler's direct-connection budget from the doctor output.
@@ -65,19 +65,39 @@ module Pgbus
65
65
  RECORD_COLUMNS = "member_id, metadata, joined_at, last_seen_at"
66
66
  SEAT_RECORD_COLUMNS = "member_id, metadata, joined_at, last_seen_at, connection_id, seat, state"
67
67
 
68
- # How long a "no seats" answer is trusted before the schema is probed
69
- # again, so a table migrated under a running process is picked up.
68
+ # How long the first "no seats" answer is trusted before the schema is
69
+ # probed again, so a table migrated under a running process is picked
70
+ # up. Each further "no" doubles the wait, up to SCHEMA_RECHECK_MAX_SECONDS
71
+ # (issue #665).
70
72
  SCHEMA_RECHECK_SECONDS = 30
73
+ SCHEMA_RECHECK_MAX_SECONDS = 600
74
+
75
+ # What the doctor and the one-time warning tell an operator to run.
76
+ SEATS_MIGRATION_FIX = "rails generate pgbus:add_presence_seats && rails db:migrate"
77
+
78
+ # NULL: no pgbus_presence_members table; false: the table without the
79
+ # seat columns; true: seats. The same schema predicate as the
80
+ # seats_supported? probe.
81
+ SEATS_STATE_SQL = <<~SQL
82
+ SELECT bool_or(column_name = 'connection_id') FROM information_schema.columns
83
+ WHERE table_name = 'pgbus_presence_members' AND table_schema = ANY (current_schemas(false))
84
+ SQL
71
85
 
72
86
  @schema_mutex = Mutex.new
73
87
  @seats_supported = nil
74
88
  @probed_at = nil
89
+ @recheck_seconds = SCHEMA_RECHECK_SECONDS
75
90
  @schema_generation = 0
91
+ # Survives reset_schema_cache!: the warning is once per process.
92
+ @seats_warning_logged = false
76
93
 
77
94
  class << self
78
95
  # True when pgbus_presence_members has the seat columns (the
79
96
  # add_presence_seats migration ran). A true answer is kept for the
80
- # life of the process; a false one for SCHEMA_RECHECK_SECONDS.
97
+ # life of the process. A false one is kept for SCHEMA_RECHECK_SECONDS,
98
+ # then twice as long after each further false answer, up to
99
+ # SCHEMA_RECHECK_MAX_SECONDS (issue #665); the first one logs a
100
+ # warning naming the migration, once per process.
81
101
  #
82
102
  # The answer is read and written under @schema_mutex only (issue
83
103
  # #527). A successful probe is kept as above; a probe that raises is
@@ -90,7 +110,7 @@ module Pgbus
90
110
  # probe that a reset_schema_cache! overtook stores nothing.
91
111
  def seats_supported?
92
112
  cached, generation = @schema_mutex.synchronize do
93
- [(@seats_supported if schema_answer_fresh?(@seats_supported, @probed_at)), @schema_generation]
113
+ [(@seats_supported if schema_answer_fresh?(@seats_supported, @probed_at, @recheck_seconds)), @schema_generation]
94
114
  end
95
115
  return cached unless cached.nil?
96
116
 
@@ -100,12 +120,9 @@ module Pgbus
100
120
  AND table_schema = ANY (current_schemas(false))
101
121
  SQL
102
122
  supported = !rows.to_a.empty?
103
- @schema_mutex.synchronize do
104
- next supported unless generation == @schema_generation
105
-
106
- @probed_at = monotonic_now
107
- @seats_supported ||= supported
108
- end
123
+ answer, warn = store_schema_answer(supported, generation)
124
+ warn_seats_missing if warn
125
+ answer
109
126
  end
110
127
 
111
128
  # seats_supported?, but a cached "no seats" is probed again first.
@@ -127,6 +144,22 @@ module Pgbus
127
144
  end
128
145
  end
129
146
 
147
+ # What Pgbus::Doctor reports (issue #665): :absent (no
148
+ # pgbus_presence_members table, or no ActiveRecord database to reach
149
+ # it), :missing (add_presence_seats has not run) or :present. Read on
150
+ # the connection presence uses, leased for this statement only, and
151
+ # never cached. A database that is configured but down raises.
152
+ def seats_state
153
+ return :absent unless defined?(::ActiveRecord::Base)
154
+
155
+ value = presence_pool&.with_connection { |conn| conn.select_value(SEATS_STATE_SQL) }
156
+ case value
157
+ when nil then :absent
158
+ when true, "t" then :present
159
+ else :missing
160
+ end
161
+ end
162
+
130
163
  # Deletes every seat row (connection_id IS NOT NULL) that has not
131
164
  # been touched within its stream's ttl and returns the deleted
132
165
  # records, each with its "stream_name". Member-keyed rows are left
@@ -197,13 +230,71 @@ module Pgbus
197
230
  "Pgbus::Streams::Presence requires ActiveRecord (no AR connection available)"
198
231
  end
199
232
 
200
- Pgbus.configuration.connects_to ? Pgbus::BusRecord.connection : ::ActiveRecord::Base.connection
233
+ record_base.connection
234
+ end
235
+
236
+ def record_base
237
+ Pgbus.configuration.connects_to ? Pgbus::BusRecord : ::ActiveRecord::Base
201
238
  end
202
239
 
203
- def schema_answer_fresh?(supported, probed_at)
240
+ # The pool presence reads from, or nil when ActiveRecord has none
241
+ # defined for it (a process on database_url alone): connection_pool
242
+ # raises ConnectionNotEstablished only for a missing pool, never for
243
+ # a database that is down.
244
+ def presence_pool
245
+ record_base.connection_pool
246
+ rescue ::ActiveRecord::ConnectionNotEstablished
247
+ nil
248
+ end
249
+
250
+ def schema_answer_fresh?(supported, probed_at, recheck_seconds)
204
251
  return false if supported.nil? || probed_at.nil?
205
252
 
206
- supported || monotonic_now - probed_at < SCHEMA_RECHECK_SECONDS
253
+ supported || monotonic_now - probed_at < recheck_seconds
254
+ end
255
+
256
+ # Stores a probe's answer unless a reset_schema_cache! overtook it.
257
+ # Returns the answer to give and whether to log the missing-seats
258
+ # warning (the first "no" of the process).
259
+ #
260
+ # A "no" over a fresh "no" (two threads probed at once) keeps the
261
+ # first one's time and step, so concurrent probes are one step. A
262
+ # true answer is never taken back.
263
+ def store_schema_answer(supported, generation)
264
+ @schema_mutex.synchronize do
265
+ next [supported, false] unless generation == @schema_generation
266
+
267
+ if supported || @seats_supported
268
+ @probed_at = monotonic_now
269
+ next [@seats_supported = true, false]
270
+ end
271
+ next [false, false] if schema_answer_fresh?(@seats_supported, @probed_at, @recheck_seconds)
272
+
273
+ @recheck_seconds = next_recheck_seconds(@seats_supported, @recheck_seconds)
274
+ @probed_at = monotonic_now
275
+ @seats_supported = false
276
+ warn = !@seats_warning_logged
277
+ @seats_warning_logged = true
278
+ [false, warn]
279
+ end
280
+ end
281
+
282
+ # A "no" after a stale "no" doubles the wait, up to the cap; a "no"
283
+ # after no answer (the first probe, or after reset_schema_cache!)
284
+ # starts at SCHEMA_RECHECK_SECONDS.
285
+ def next_recheck_seconds(previous_answer, previous_seconds)
286
+ return SCHEMA_RECHECK_SECONDS if previous_answer.nil?
287
+
288
+ [previous_seconds * 2, SCHEMA_RECHECK_MAX_SECONDS].min
289
+ end
290
+
291
+ def warn_seats_missing
292
+ Pgbus.logger&.warn(
293
+ "[Pgbus::Streams::Presence] pgbus_presence_members has no seat columns (connection_id): presence " \
294
+ "runs member-keyed, one row per member, and the schema keeps being re-checked. Run `#{SEATS_MIGRATION_FIX}` to get seats " \
295
+ "(if the table does not exist, run `rails generate pgbus:add_presence && rails db:migrate` instead: " \
296
+ "it already creates the seat columns)."
297
+ )
207
298
  end
208
299
 
209
300
  def expire_stream_seats(stream_name, connection_ids, ttl)
data/lib/pgbus/version.rb CHANGED
@@ -1,5 +1,5 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  module Pgbus
4
- VERSION = "0.25.0"
4
+ VERSION = "0.25.1"
5
5
  end
@@ -190,12 +190,13 @@ module Pgbus
190
190
  return nil if socket_path.nil? || socket_path.empty?
191
191
 
192
192
  # The worker's ack deadline must exceed the master's own internal
193
- # ensure_listening budget (its listener's health-check cycle + 1s).
193
+ # ensure_listening budget (its listener's ack budget), so it gets
194
+ # that plus 1 s. Independent of the keepalive (issue #664).
194
195
  failover = nil
195
196
  client = HubClient.new(
196
197
  socket_path: socket_path,
197
198
  dispatch_queue: @dispatch_queue,
198
- ack_timeout: (@config.streams_listen_health_check_ms / 1000.0) + 2.0,
199
+ ack_timeout: (@config.streams_listen_ack_timeout_ms / 1000.0) + 1.0,
199
200
  # failover is assigned right below; a transport death in the gap
200
201
  # is caught by the FailoverListener's synchronous ensure path.
201
202
  on_failure: -> { failover&.fail_over! },
@@ -221,6 +222,7 @@ module Pgbus
221
222
  pg_connection: pg_connection,
222
223
  dispatch_queue: @dispatch_queue,
223
224
  health_check_ms: @config.streams_listen_health_check_ms,
225
+ ack_timeout_ms: @config.streams_listen_ack_timeout_ms,
224
226
  # Opt-in dispatch-queue backpressure (issue #315 item 3). 0 =
225
227
  # unbounded (default). The queue itself stays an unbounded
226
228
  # Queue.new so the request-thread Connect push and the dispatcher's
@@ -335,11 +337,12 @@ module Pgbus
335
337
  # runs its own PrimaryValidator (retrying on a replica) and skips the
336
338
  # probe to stay cheap, mirroring Pgbus::Process::NotifyListener.
337
339
  # String/Hash go through DedicatedConnection so a :session-mode
338
- # `:variables` key never reaches PG.connect (issue #352).
340
+ # `:variables` key never reaches PG.connect (issue #352), with TCP
341
+ # keepalives so a dead socket errors between keepalives (issue #664).
339
342
  def build_raw_pg_connection
340
343
  opts = @config.streams_connection_options
341
344
  case opts
342
- when String, Hash then Pgbus::DedicatedConnection.connect(opts)
345
+ when String, Hash then Pgbus::DedicatedConnection.connect_with_keepalives(opts)
343
346
  when Proc
344
347
  # The Proc branch in connection_options typically returns
345
348
  # ActiveRecord::Base.connection.raw_connection — a pooled
@@ -28,8 +28,10 @@ module Pgbus
28
28
  # the pipe is touched from other threads
29
29
  #
30
30
  # Health check: once per health_check_ms the listener runs `SELECT 1` as
31
- # a TCP keepalive, on a deadline: an idle connection sees exactly one per
32
- # cycle, and a stream of wakes or notifications cannot postpone it. If
31
+ # a keepalive, on a deadline: an idle connection sees exactly one per
32
+ # cycle, and a stream of wakes or notifications cannot postpone it. The
33
+ # interval only bounds how long a silently dropped LISTEN goes unnoticed:
34
+ # no wait budget is derived from it (ack_timeout_ms is, issue #664). If
33
35
  # it raises, the listener rebuilds a FRESH connection via
34
36
  # connection_factory and re-LISTENs every channel in `@listening_to`.
35
37
  # This is the fix for design doc §11 #1 (silently dropped LISTEN
@@ -62,10 +64,14 @@ module Pgbus
62
64
 
63
65
  RECONNECT_BACKOFF_SECONDS = 0.5
64
66
 
65
- # Grace added to one health-check cycle when #stop joins the listener
66
- # thread. See #stop_join_timeout.
67
+ # Grace added to the ack budget when #stop joins the listener thread.
68
+ # See #stop_join_timeout.
67
69
  STOP_JOIN_GRACE_SECONDS = 5
68
70
 
71
+ # Default for ack_timeout_ms: the old 250 ms cycle plus its 1 s margin,
72
+ # now fixed instead of following the keepalive (issue #664).
73
+ DEFAULT_ACK_TIMEOUT_MS = 1_250
74
+
69
75
  attr_reader :listening_to
70
76
 
71
77
  # @param connection_factory [#call] builds a FRESH PG connection on each
@@ -84,14 +90,18 @@ module Pgbus
84
90
  # most once per `maintenance.interval` seconds with THIS connection, so a
85
91
  # maintenance query (e.g. pg_stat_activity headroom) reuses the existing
86
92
  # idle connection instead of opening its own. nil disables it.
93
+ # `ack_timeout_ms:` bounds how long ensure_listening waits for this
94
+ # thread's ack, and (plus a grace) how long #stop joins it
95
+ # (streams_listen_ack_timeout_ms; issue #664).
87
96
  def initialize(pg_connection:, dispatch_queue:, health_check_ms:,
88
- connection_factory:, dispatch_queue_limit: 0, maintenance: nil,
89
- logger: Pgbus.logger, clock: nil)
97
+ connection_factory:, ack_timeout_ms: DEFAULT_ACK_TIMEOUT_MS, dispatch_queue_limit: 0,
98
+ maintenance: nil, logger: Pgbus.logger, clock: nil)
90
99
  raise ArgumentError, "connection_factory is required" unless connection_factory.respond_to?(:call)
91
100
 
92
101
  @conn = pg_connection
93
102
  @dispatch_queue = dispatch_queue
94
103
  @health_check_ms = health_check_ms
104
+ @ack_timeout_ms = ack_timeout_ms
95
105
  @logger = logger
96
106
  @connection_factory = connection_factory
97
107
  @dispatch_queue_limit = dispatch_queue_limit
@@ -182,9 +192,12 @@ module Pgbus
182
192
  # a process-killing SEGV, not a rescuable PG::Error (issue #375).
183
193
  # The listener thread is the sole owner of @conn and closes it in
184
194
  # run_loop's ensure; clearing @running above is the whole stop signal,
185
- # and the wake makes the thread see it at once. The join budget still
186
- # covers a full health-check cycle: the thread may be inside a
187
- # keepalive or a reconnect rather than the select.
195
+ # and the wake makes the thread see it at once. The join budget
196
+ # (the ack budget plus a grace, 6.25 s by default) covers a thread
197
+ # inside a normal keepalive or reconnect attempt. A keepalive stuck
198
+ # on a dead socket can outlast it (libpq's TCP bound is ~11 s): #stop
199
+ # then logs "did not stop" and the thread exits on its own once
200
+ # libpq raises.
188
201
  wake
189
202
  joined = @thread.nil? || @thread.join(stop_join_timeout)
190
203
  # A thread past its budget stays referenced, so #start refuses to
@@ -208,9 +221,9 @@ module Pgbus
208
221
  # in the read_after result nor delivered as a WakeMessage.
209
222
  #
210
223
  # The wake pipe makes the typical wait a thread hop (issue #630). The
211
- # wait is bounded by one health-check cycle
212
- # (streams_listen_health_check_ms, default 250ms) plus a small margin,
213
- # so a stuck listener can't hang the dispatcher forever.
224
+ # wait is bounded by the ack budget (streams_listen_ack_timeout_ms,
225
+ # default 1.25 s; issue #664), so a stuck listener can't hang the
226
+ # dispatcher forever.
214
227
  def ensure_listening(queue_name)
215
228
  # A connect the dispatcher drains during shutdown (Instance#shutdown!
216
229
  # stops the listener first) must not wait out the ack budget.
@@ -411,18 +424,20 @@ module Pgbus
411
424
 
412
425
  # Caller wait budget for ensure_listening's ack. The wake pipe
413
426
  # makes the usual wait a thread hop; a thread busy in a keepalive
414
- # or a reconnect gets one health-check cycle plus a margin, so the
415
- # dispatcher fails loud rather than hanging if the thread is dead.
427
+ # or a reconnect gets ack_timeout_ms, so the dispatcher fails loud
428
+ # rather than hanging if the thread is dead. Not derived from the
429
+ # keepalive interval: raising that must not slow this failure path
430
+ # (issue #664).
416
431
  def ack_timeout
417
- (@health_check_ms / 1000.0) + 1.0
432
+ @ack_timeout_ms / 1000.0
418
433
  end
419
434
 
420
435
  # How long #stop waits for the listener thread to notice the cleared
421
436
  # @running and finish teardown. The wake pipe interrupts its select at
422
437
  # once, but it may be inside a keepalive or a reconnect attempt, so the
423
- # budget stays one health-check cycle plus grace.
438
+ # budget is the ack budget plus grace, independent of the keepalive.
424
439
  def stop_join_timeout
425
- (@health_check_ms / 1000.0) + STOP_JOIN_GRACE_SECONDS
440
+ ack_timeout + STOP_JOIN_GRACE_SECONDS
426
441
  end
427
442
 
428
443
  # The channel joins @listening_to BEFORE the exec: if the LISTEN
@@ -158,13 +158,18 @@ module Pgbus
158
158
 
159
159
  def default_listener_factory
160
160
  lambda do |dispatch_queue:|
161
- build_connection = -> { Pgbus::DedicatedConnection.connect(@config.streams_connection_options) }
161
+ # TCP keepalives and the ack budget as on a per-worker listener:
162
+ # in :master scope this is the connection that LISTENs (issue #664).
163
+ build_connection = lambda do
164
+ Pgbus::DedicatedConnection.connect_with_keepalives(@config.streams_connection_options)
165
+ end
162
166
  conn = build_connection.call
163
167
  Pgbus::Process::PrimaryValidator.validate_primary!(conn)
164
168
  Listener.new(
165
169
  pg_connection: conn,
166
170
  dispatch_queue: dispatch_queue,
167
171
  health_check_ms: @config.streams_listen_health_check_ms,
172
+ ack_timeout_ms: @config.streams_listen_ack_timeout_ms,
168
173
  connection_factory: build_connection,
169
174
  dispatch_queue_limit: @config.streams_dispatch_queue_limit,
170
175
  logger: @logger
@@ -398,7 +398,7 @@ module Pgbus
398
398
  #
399
399
  # The fast path holds while the listener reconnects, on purpose.
400
400
  # Waiting on a reconnecting listener would cost every connect the
401
- # full ack budget (health_check_ms + 1 s) for as long as the
401
+ # full ack budget (streams_listen_ack_timeout_ms) for as long as the
402
402
  # database stays unreachable: the storm this rule removes. A
403
403
  # connect in that window has the same exposure as every connection
404
404
  # already registered: a durable stream's next wake after the
@@ -239,6 +239,8 @@ module Pgbus
239
239
  def streams_idle_timeout=: (untyped) -> untyped
240
240
  def streams_listen_health_check_ms: () -> Integer
241
241
  def streams_listen_health_check_ms=: (untyped) -> untyped
242
+ def streams_listen_ack_timeout_ms: () -> Integer
243
+ def streams_listen_ack_timeout_ms=: (untyped) -> untyped
242
244
  def streams_listen_scope: () -> Symbol
243
245
  def streams_listen_scope=: (untyped scope) -> untyped
244
246
  def streams_max_connections: () -> Integer
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: pgbus
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.25.0
4
+ version: 0.25.1
5
5
  platform: ruby
6
6
  authors:
7
7
  - Mikael Henriksson