pgbus 0.26.0 → 0.26.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +6 -2
- data/lib/pgbus/client/async_commit.rb +12 -17
- data/lib/pgbus/client/session_gucs.rb +58 -0
- data/lib/pgbus/client.rb +4 -13
- data/lib/pgbus/version.rb +1 -1
- metadata +2 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 8b69998354e78e47fc483f2c03b2fd2d4856031ad0a1e2db00d4ba7da7f40547
|
|
4
|
+
data.tar.gz: 3bfad5779917d58b0afd88da6b6cb680c3086f879a76de66aa139c318476bbcc
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 48fffb0b444afea7ba76f4d53e65426cbfe9cea96af16f1aa674586dc1141106e246f51b117da7d533ad153185a3189d3d6b730521eed7c21ce5acec3d3c3df9
|
|
7
|
+
data.tar.gz: 7e580515d3b53981e973556c1fd440dfe91700a30f554b650cc82c4843d7286161a1f7ff07c73799a7988303e04bf6d433e88bd37951ef47d1e1ae5d7b42c417
|
data/CHANGELOG.md
CHANGED
|
@@ -4,12 +4,16 @@
|
|
|
4
4
|
|
|
5
5
|
- **A member-keyed presence `touch` no longer probes `information_schema` on a table without the seat columns, and a forced re-check no longer restarts the backoff (issue #666).** On a `pgbus_presence_members` table where `rails g pgbus:add_presence_seats` has not run, every member-keyed `touch`, `leave` and `sweep!` re-probed a cached "no seats" answer before it ran and restarted the #665 backoff at 30 s. The streamer touches once per connection per heartbeat, so a process with N member-keyed connections sent N catalog queries every `streams_heartbeat_interval` (15 s by default), and its backoff never took effect. A member-keyed `touch` now uses the cached answer and re-checks only on the backoff's schedule: three touches after a reset send one probe instead of four, and none while a "no" is cached. Assumed: on a table migrated under a running process, such a `touch` can also refresh that member's seat rows until the next re-check (at most 10 minutes, `SCHEMA_RECHECK_MAX_SECONDS`); it is an `UPDATE` of `last_seen_at`, so nothing is deleted. A member-keyed `leave` and `sweep!` still re-check a cached "no" before they delete, so they never delete seat rows on a migrated table (#515), now with one probe instead of up to two, answered from that probe. That re-check leaves the backoff alone: over a fresh "no" it changes neither the step nor the schedule however many disconnects force one, and over a "no" already due it is the timed re-check, one step. `reset_schema_cache!` still re-probes at once and starts again from 30 s. Refs #666.
|
|
6
6
|
|
|
7
|
+
- **pgmq-ruby 0.9.0 (was 0.8.2); the gemspec now requires `~> 0.9.0` (issue #656).** What reaches pgbus users is the connection pool: pgmq-ruby's checkout health check now resets a pooled connection in any state other than `CONNECTION_OK`, so a connection a `Thread#raise` left half-way through a reset (`CONNECTION_STARTED`) is recovered on the next checkout. Before, the checkout handed it out again, libpq refused every send (`no connection to the server`), and pgbus's stale-connection retry only recovered it on its last attempt by dropping both pools (#645): now the call succeeds on the first attempt, with no retry, no pool reload and no warning. pgmq-ruby also resets a connection whose call raised `PG::ConnectionBad` or `PG::UnableToSend` before it goes back to the pool (not a connection pgbus hands it through a callable, which it neither resets nor closes: the shared ActiveRecord connection, and dedicated connections in `connection_guc_mode = :session` with database.yml `variables`); native statements re-prepare on the new backend as on any reconnect. pgbus keeps its reload before the last stale retry for a connection that still fails after the health check passed it. A checkout reset of a connection in `connection_guc_mode = :session` with `variables` applies its `SET`s again (#671). `produce_topic` and `produce_batch_topic` now raise `PGMQ::Errors::MessageInvalidError` for a message that is not a String or headers that are neither a String nor nil, as `produce` has since 0.8.0. `Client#publish_to_topic` (what `Pgbus.publish` and the outbox poller call; `Pgbus::Testing`'s fake and inline modes never reach PGMQ) serializes the payload and headers to JSON Strings first, so it is unaffected; `headers: false`, which pgmq-ruby dropped before with an integer delay (and stored as JSON `false` with a fractional one) and now rejects, is passed as nil, so it is dropped at any delay. `spec/integration/connection_failure_spec.rb` now asserts the mid-reset connection is recovered at checkout (it failed on 0.8.2), in place of the two examples that pinned the old last-retry recovery. Refs #656.
|
|
8
|
+
|
|
9
|
+
### Fixed
|
|
10
|
+
|
|
11
|
+
- **A pooled connection in `connection_guc_mode = :session` keeps its database.yml `variables` when it is reset (issue #671).** In `:session` mode the `variables` (`statement_timeout`, `lock_timeout` and the like) are applied with `SET` after connect, and `PG::Connection#reset` opens a new backend without them. pgmq-ruby's checkout health check resets an unhealthy pooled connection (`finished?` or `CONNECTION_BAD` since #332, and an intermediate state such as `CONNECTION_STARTED` since pgmq-ruby 0.9.0), so that connection went on serving the job or streams pool with the server's defaults. Each connection the `:session` connect lambda opens now re-applies its `SET`s when it is reset, whoever resets it. A connection whose `SET`s do not all complete, after a connect or a reset, is closed before the error reaches the caller, so it is never pooled without them; with pg 1.5.9 or later the next checkout resets it and applies the `SET`s again (an older pg cannot reset a closed connection, so it is replaced when the pool is reloaded before the last stale-connection retry; the call itself is retried only for a stale-connection error, so a `SET` that always fails, such as an invalid value, fails each time). The completion pipeline's own reset (#619) no longer applies the `SET`s itself (they ran twice), and a `SET` that fails there is now logged as `Resetting the connection or re-applying its session variables failed` and leaves the connection closed, where before the session carried on without its `variables`. `:options` mode, a String URL and the shared ActiveRecord connection are unaffected. Refs #671.
|
|
12
|
+
|
|
7
13
|
## [0.25.1] - 2026-11-15
|
|
8
14
|
|
|
9
15
|
### Changed
|
|
10
16
|
|
|
11
|
-
- **pgmq-ruby 0.9.0 (was 0.8.2); the gemspec now requires `~> 0.9.0` (issue #656).** What reaches pgbus users is the connection pool: pgmq-ruby's checkout health check now resets a pooled connection in any state other than `CONNECTION_OK`, so a connection a `Thread#raise` left half-way through a reset (`CONNECTION_STARTED`) is recovered on the next checkout. Before, the checkout handed it out again, libpq refused every send (`no connection to the server`), and pgbus's stale-connection retry only recovered it on its last attempt by dropping both pools (#645): now the call succeeds on the first attempt, with no retry, no pool reload and no warning. pgmq-ruby also resets a connection whose call raised `PG::ConnectionBad` or `PG::UnableToSend` before it goes back to the pool (not a connection pgbus hands it through a callable, which it neither resets nor closes: the shared ActiveRecord connection, and dedicated connections in `connection_guc_mode = :session` with database.yml `variables`); native statements re-prepare on the new backend as on any reconnect. pgbus keeps its reload before the last stale retry for a connection that still fails after the health check passed it. A checkout reset of a connection in `connection_guc_mode = :session` with `variables` does not re-run its `SET`s, as for a `CONNECTION_BAD` one before (#671). `produce_topic` and `produce_batch_topic` now raise `PGMQ::Errors::MessageInvalidError` for a message that is not a String or headers that are neither a String nor nil, as `produce` has since 0.8.0. `Client#publish_to_topic` (what `Pgbus.publish` and the outbox poller call; `Pgbus::Testing`'s fake and inline modes never reach PGMQ) serializes the payload and headers to JSON Strings first, so it is unaffected; `headers: false`, which pgmq-ruby dropped before with an integer delay (and stored as JSON `false` with a fractional one) and now rejects, is passed as nil, so it is dropped at any delay. `spec/integration/connection_failure_spec.rb` now asserts the mid-reset connection is recovered at checkout (it failed on 0.8.2), in place of the two examples that pinned the old last-retry recovery. Refs #656.
|
|
12
|
-
|
|
13
17
|
- **Changed default: the streamer's LISTEN keepalive `streams_listen_health_check_ms` is 5 000 ms instead of 250, and the listener's ack and stop budgets no longer follow it (issue #664).** Every web process running the streamer sent `SELECT 1` on its LISTEN connection four times a second, idle or not: 345 600 queries a day per process, the most frequent statement on a small fleet's database. Since #630 a connect no longer waits on that cycle (the listener wakes through a pipe), so the interval only bounds how long a LISTEN the server dropped on a still-open socket goes unnoticed; now one `SELECT 1` per 5 s idle cycle, 20× fewer. The interval was also wired into three wait budgets, so raising it slowed the failure paths: they now come from a new setting, **`streams_listen_ack_timeout_ms`** (default 1 250, a positive Integer, today's 250 + 1 000). `Listener#ensure_listening` waits at most that for a stuck listener's ack, `Listener#stop` joins for that plus its 5 s grace, and with `streams_listen_scope :master` the worker's `HubClient` waits that plus 1 s before failing over, as before relative to the hub's own budget; the master hub's listener gets the same ack budget. The streamer's dedicated LISTEN connection (per worker, and the master hub's) now opens with libpq TCP keepalives, `keepalives=1 keepalives_idle=5 keepalives_interval=2 keepalives_count=3 tcp_user_timeout=11000`, so a dead socket errors within ~11 s by these settings (5 s idle + 3 probes 2 s apart, and the matching `tcp_user_timeout`; on Linux, where `tcp_user_timeout` applies; the bound follows the settings, it was not measured) even between keepalives, and the listener reconnects and re-LISTENs; a keyword the connection options already set wins, a connection that names a service (`service=` or `PGSERVICE`) gets none so its service file's values stay in force, and on libpq < 12 none is added (the same gate `Client` uses for its read bounds, now `Pgbus::DedicatedConnection.libpq_supports_socket_bounds?`). The worker `NotifyListener` and the doctor's connection are unchanged. The pool autoscaler's check runs on the keepalive tick, so a `streams_pool_autoscale_interval` under 5 s (the default is 300 s) now runs every 5 s. A dedicated connection URL ending in a bare `?` no longer gets an empty `?&` parameter that failed the connect. An explicit `streams_listen_health_check_ms` keeps exactly its value: **pin `config.streams_listen_health_check_ms = 250` for the old keepalive.** If you set it to 5 000 as the workaround, the ack budget drops from 6 s back to 1.25 s. Measured on the same machine (`rake bench:streams`, which pins 100 ms; `rake bench:streams_scale` at M = 100 with the default, 250 ms before and 5 000 ms after, two rounds each): section 4 concurrent connect 0.91–3.07 ms per connect before, 0.86–3.06 ms after; scale bench, all 100 clients registered in 0.11–0.12 s before, 0.09–0.11 s after (one 0.24 s round at load 32, 0.09 s on a re-run), 0 misses either way. Refs #664.
|
|
14
18
|
|
|
15
19
|
### Fixed
|
|
@@ -103,8 +103,11 @@ module Pgbus
|
|
|
103
103
|
# A connection back in the pool in pipeline mode would break the next
|
|
104
104
|
# caller. A lost one is left as it is: pgmq-ruby resets a bad connection
|
|
105
105
|
# at its next checkout, and a reset leaves pipeline mode. One that cannot
|
|
106
|
-
# leave (results left unread by an interrupted read) is reset here.
|
|
106
|
+
# leave (results left unread by an interrupted read) is reset here. One
|
|
107
|
+
# a failed reset closed (its :session SETs failed, SessionGucs) is left
|
|
108
|
+
# closed too: pgmq-ruby resets it at the next checkout.
|
|
107
109
|
def leave_pipeline(conn)
|
|
110
|
+
return if conn.finished?
|
|
108
111
|
return if conn.pipeline_status == PG::PQ_PIPELINE_OFF || conn.status == PG::CONNECTION_BAD
|
|
109
112
|
|
|
110
113
|
conn.exit_pipeline_mode
|
|
@@ -116,32 +119,24 @@ module Pgbus
|
|
|
116
119
|
# fails (the server is gone) is logged and left there, so the error the
|
|
117
120
|
# caller is already raising is the one that surfaces; pgmq-ruby resets a
|
|
118
121
|
# bad connection at its next checkout.
|
|
119
|
-
# A reset reconnects with the same conninfo (the read bounds ride in it)
|
|
120
|
-
#
|
|
121
|
-
#
|
|
122
|
+
# A reset reconnects with the same conninfo (the read bounds ride in it);
|
|
123
|
+
# in :session GUC mode the connection's own reset applies the SETs again
|
|
124
|
+
# and closes it when they fail (SessionGucs, issue #671).
|
|
122
125
|
def discard_session(conn, reason)
|
|
123
126
|
Pgbus.logger.warn { "[Pgbus::Client] #{reason}, resetting the connection" }
|
|
124
|
-
|
|
125
|
-
conn.reset
|
|
126
|
-
rescue PG::Error => e
|
|
127
|
-
Pgbus.logger.warn { "[Pgbus::Client] Resetting the connection failed: #{e.message}" }
|
|
128
|
-
return
|
|
129
|
-
end
|
|
130
|
-
reapply_session_gucs(conn)
|
|
131
|
-
end
|
|
132
|
-
|
|
133
|
-
def reapply_session_gucs(conn)
|
|
134
|
-
apply_session_gucs(conn, @session_gucs) if @session_gucs
|
|
127
|
+
conn.reset
|
|
135
128
|
rescue PG::Error => e
|
|
136
129
|
Pgbus.logger.warn do
|
|
137
|
-
"[Pgbus::Client]
|
|
130
|
+
"[Pgbus::Client] Resetting the connection or re-applying its session variables failed: #{e.message}"
|
|
138
131
|
end
|
|
139
132
|
end
|
|
140
133
|
|
|
141
134
|
# The failed BEGIN block (INERROR) is still open after the sync. A failed
|
|
142
135
|
# COMMIT has already ended it (IDLE); a lost connection reports UNKNOWN
|
|
143
|
-
# and has nothing to roll back.
|
|
136
|
+
# and has nothing to roll back, as does one a reset closed.
|
|
144
137
|
def rollback_async_commit(conn)
|
|
138
|
+
return if conn.finished?
|
|
139
|
+
|
|
145
140
|
conn.exec("ROLLBACK") if [PG::PQTRANS_INERROR, PG::PQTRANS_INTRANS].include?(conn.transaction_status)
|
|
146
141
|
end
|
|
147
142
|
end
|
|
@@ -0,0 +1,58 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module Pgbus
|
|
4
|
+
class Client
|
|
5
|
+
# The database.yml variables that connection_guc_mode :session applies with
|
|
6
|
+
# SET after connect (issue #332), on every session a pooled connection has.
|
|
7
|
+
#
|
|
8
|
+
# PG::Connection#reset opens a new backend from the same conninfo and does
|
|
9
|
+
# not replay the SETs. pgmq-ruby resets an unhealthy pooled connection at
|
|
10
|
+
# checkout (finished?, CONNECTION_BAD, and since 0.9.0 an intermediate
|
|
11
|
+
# state such as CONNECTION_STARTED), and AsyncCommit#discard_session resets
|
|
12
|
+
# one whose pipeline state cannot be trusted. So each connection #connect
|
|
13
|
+
# opens is extended with this module, whose #reset (and #async_reset,
|
|
14
|
+
# #sync_reset) applies the SETs again (issue #671).
|
|
15
|
+
#
|
|
16
|
+
# A connection whose SETs did not all complete is closed before the error
|
|
17
|
+
# propagates: it is never pooled CONNECTION_OK without them. The next
|
|
18
|
+
# checkout sees it finished? and resets it, which tries again (pg 1.5.9 and
|
|
19
|
+
# later reset a closed connection; an older pg raises, and the pool reload
|
|
20
|
+
# before the last stale-connection retry replaces it).
|
|
21
|
+
#
|
|
22
|
+
# One instance per pool, built at Client construction; it holds only the
|
|
23
|
+
# SET statements, in a frozen array nothing mutates, so concurrent
|
|
24
|
+
# connections share it safely.
|
|
25
|
+
class SessionGucs < Module
|
|
26
|
+
def initialize(variables)
|
|
27
|
+
super()
|
|
28
|
+
statements = variables.map { |name, value| "SET #{name} = '#{value}'" }.freeze
|
|
29
|
+
|
|
30
|
+
# Runs the block (a reset, or nothing after a connect), then the SETs.
|
|
31
|
+
# One ensure covers both, so an async raise between the reset and the
|
|
32
|
+
# first SET also closes the connection.
|
|
33
|
+
define_method(:pgbus_with_session_gucs) do |&before|
|
|
34
|
+
applied = false
|
|
35
|
+
before&.call
|
|
36
|
+
statements.each { |sql| exec(sql) }
|
|
37
|
+
applied = true
|
|
38
|
+
self
|
|
39
|
+
ensure
|
|
40
|
+
close unless applied || finished?
|
|
41
|
+
end
|
|
42
|
+
|
|
43
|
+
# pg's async_reset is an alias bound to PG::Connection#reset and its
|
|
44
|
+
# sync_reset is C, so a hook on reset alone would not see either. None
|
|
45
|
+
# of the three calls another, so each reset applies the SETs once.
|
|
46
|
+
%i[reset async_reset sync_reset].each do |name|
|
|
47
|
+
define_method(name) do
|
|
48
|
+
pgbus_with_session_gucs { super() }
|
|
49
|
+
end
|
|
50
|
+
end
|
|
51
|
+
end
|
|
52
|
+
|
|
53
|
+
def connect(pg_opts)
|
|
54
|
+
PG.connect(pg_opts).extend(self).pgbus_with_session_gucs
|
|
55
|
+
end
|
|
56
|
+
end
|
|
57
|
+
end
|
|
58
|
+
end
|
data/lib/pgbus/client.rb
CHANGED
|
@@ -138,9 +138,6 @@ module Pgbus
|
|
|
138
138
|
# safe on this dedicated-connection branch — never on the shared-AR Proc
|
|
139
139
|
# path, where statement_timeout would leak into application queries.
|
|
140
140
|
conn_opts = apply_connection_bounds(conn_opts)
|
|
141
|
-
# What the connect lambda SETs per session (:session GUC mode), kept so
|
|
142
|
-
# a session reset after a failed completion pipeline gets it again.
|
|
143
|
-
@session_gucs = session_gucs(conn_opts)
|
|
144
141
|
conn_opts = wrap_session_gucs(conn_opts)
|
|
145
142
|
@pgmq = PGMQ::Client.new(conn_opts, pool_size: config.resolved_pool_size, pool_timeout: config.pool_timeout)
|
|
146
143
|
@pgmq_mutex = nil
|
|
@@ -2204,17 +2201,15 @@ module Pgbus
|
|
|
2204
2201
|
# PG::Connection each call (pgmq guards against a shared object). Applies to
|
|
2205
2202
|
# a Hash with `:variables` only — a String URL or no variables passes through
|
|
2206
2203
|
# unchanged, as does :options mode (where forward_connection_variables
|
|
2207
|
-
# already baked the GUCs into `options`). See issue #332.
|
|
2204
|
+
# already baked the GUCs into `options`). See issue #332. Each connection
|
|
2205
|
+
# re-applies the SETs when it is reset (issue #671, SessionGucs).
|
|
2208
2206
|
def wrap_session_gucs(conn_opts)
|
|
2209
2207
|
variables = session_gucs(conn_opts)
|
|
2210
2208
|
return conn_opts unless variables
|
|
2211
2209
|
|
|
2212
2210
|
pg_opts = conn_opts.except(:variables)
|
|
2213
|
-
|
|
2214
|
-
|
|
2215
|
-
apply_session_gucs(conn, variables)
|
|
2216
|
-
conn
|
|
2217
|
-
end
|
|
2211
|
+
gucs = SessionGucs.new(variables)
|
|
2212
|
+
-> { gucs.connect(pg_opts) }
|
|
2218
2213
|
end
|
|
2219
2214
|
|
|
2220
2215
|
# The database.yml variables :session GUC mode applies with SET after
|
|
@@ -2226,10 +2221,6 @@ module Pgbus
|
|
|
2226
2221
|
variables unless variables.nil? || variables.empty?
|
|
2227
2222
|
end
|
|
2228
2223
|
|
|
2229
|
-
def apply_session_gucs(conn, variables)
|
|
2230
|
-
variables.each { |name, value| conn.exec("SET #{name} = '#{value}'") }
|
|
2231
|
-
end
|
|
2232
|
-
|
|
2233
2224
|
def apply_connection_bounds(conn_opts)
|
|
2234
2225
|
timeout = config.read_timeout
|
|
2235
2226
|
return conn_opts unless timeout&.positive?
|
data/lib/pgbus/version.rb
CHANGED
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: pgbus
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.26.
|
|
4
|
+
version: 0.26.1
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Mikael Henriksson
|
|
@@ -368,6 +368,7 @@ files:
|
|
|
368
368
|
- lib/pgbus/client/notify_stream.rb
|
|
369
369
|
- lib/pgbus/client/read_after.rb
|
|
370
370
|
- lib/pgbus/client/resizable_pool.rb
|
|
371
|
+
- lib/pgbus/client/session_gucs.rb
|
|
371
372
|
- lib/pgbus/concurrency.rb
|
|
372
373
|
- lib/pgbus/concurrency/blocked_execution.rb
|
|
373
374
|
- lib/pgbus/concurrency/semaphore.rb
|