pgbus 0.26.1 → 0.27.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +16 -0
- data/README.md +28 -0
- data/lib/pgbus/client/notify_stream.rb +16 -0
- data/lib/pgbus/dedicated_connection.rb +20 -7
- data/lib/pgbus/engine.rb +8 -0
- data/lib/pgbus/open_transaction.rb +96 -0
- data/lib/pgbus/signals/middleware.rb +23 -0
- data/lib/pgbus/signals.rb +282 -0
- data/lib/pgbus/streams.rb +3 -39
- data/lib/pgbus/testing/minitest.rb +1 -0
- data/lib/pgbus/testing.rb +36 -0
- data/lib/pgbus/version.rb +1 -1
- data/lib/pgbus/web/streamer/failover_listener.rb +132 -27
- data/lib/pgbus/web/streamer/hub_client.rb +46 -15
- data/lib/pgbus/web/streamer/hub_protocol.rb +6 -0
- data/lib/pgbus/web/streamer/instance.rb +37 -2
- data/lib/pgbus/web/streamer/listener.rb +113 -24
- data/lib/pgbus/web/streamer/master_hub.rb +87 -2
- data/lib/pgbus/web/streamer/signal_delivery.rb +136 -0
- data/lib/pgbus/web/streamer.rb +10 -0
- data/lib/pgbus.rb +21 -0
- data/sig/pgbus/client.rbs +1 -0
- data/sig/pgbus.rbs +13 -0
- metadata +5 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 1b883f03f4b08531d1e4847486beb29842afde42a872453606cae421952a86e6
|
|
4
|
+
data.tar.gz: '09f28e7f82f7e843ce8bd8aeda05c53481ade115e6f66fb2bc2197ae45d081df'
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: ca5ce2420a6df278b3aae31ebcca0c0dc1980373a8954254de83310a75c56574cda6b3549787a662b38ade7a7fa6a8d24a2471b2ba97981f22e04a6884f4c4fb
|
|
7
|
+
data.tar.gz: fecd5c8e775ef8367f9adceee008b6b02afea7b842ff9827bf000993e3fabb82534a49464e81b68e2fd35d5cb62ecd6cacff6efdd6c04d5eebb08c6e9859a7a7
|
data/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,21 @@
|
|
|
1
1
|
## [Unreleased]
|
|
2
2
|
|
|
3
|
+
### Added
|
|
4
|
+
|
|
5
|
+
- **Process signals: `Pgbus.on_signal(name) { |payload| ... }` runs a block in every web process when any process calls `Pgbus.signal(name, payload = nil)` (issue #674).** This covers a per-process in-memory cache that every Puma worker must drop when its data changes, which had no supported route: the event bus delivers an event to one competing consumer in a worker process, and the streamer only woke SSE connections.
|
|
6
|
+
|
|
7
|
+
- **Sending.** A signal is a `NOTIFY` on `<queue_prefix>_signal_<name>`, sent through `Pgbus.client` (new `Client#notify_signal`). Inside an open ActiveRecord transaction it is sent at `COMMIT` and never on rollback. A `transaction(joinable: false)` block sends at once: the fixture transaction of an app's transactional tests is one, so an `after_commit { Pgbus.signal(...) }` still sends there. The payload is an optional String under NOTIFY's 7999-byte cap; an empty String arrives as `nil`. A name may use lowercase letters, digits and underscores, and its channel must fit the queue-name budget; anything else raises `ArgumentError`.
|
|
8
|
+
- **Transport.** The streamer's LISTEN connection carries the signal: each worker's `Listener` under `streams_listen_scope :process`, the `MasterHub`'s under `:master`. The hub fans it out over the hub socket with new frames: `ssub`/`sack`, refcounted per channel like a stream, and `signal`/`gap`, never dropped by the backpressure policy; a worker evicted for not draining fails over and reports a gap. No worker opens a LISTEN connection of its own. A worker whose master predates signals gets no `sack` and fails over to its own listener. A signal NOTIFY never enters the stream dispatch queue and is never dropped at `streams_dispatch_queue_limit`.
|
|
9
|
+
- **Running the blocks.** Subscribers run on a new per-process `SignalDelivery` thread, started by the first signal for a registered name, each inside `Rails.application.executor.wrap`. A block that raises is reported through `error_reporters` and the others still run. With 1 000 signals waiting behind a slow block, a further one is dropped and reported as a gap.
|
|
10
|
+
- **Eager start.** With at least one subscriber, a process's first request starts its streamer on a background thread (new `Pgbus::Signals::Middleware`, which the engine installs when `streams_enabled`). A process that never served an SSE connection still receives signals, and no request waits on the LISTEN connect. A failed start is reported and retried by a request 30 s later. The middleware adds about 100 ns to a request (65 ns once started; 1M calls, this machine). Without a subscriber nothing changes: no connection, no thread.
|
|
11
|
+
- **Gaps.** Delivery is at most once. Every `on_gap:` callable runs once, once the LISTEN is in place (gaps that arrive before it runs coalesce), after each gap: the process's first LISTEN (nothing sent before it reached the process), a listener reconnect, a worker failover from the master hub, or the hub's own listener reconnecting.
|
|
12
|
+
- **Failover retry.** A worker whose hub died and whose own listener cannot be built now retries the build every 5 s in the background, for streams too (it used to stay degraded until it recycled), then replays every subscription and runs `on_gap`; a retry in flight never holds up the worker's shutdown. The streamer's dedicated LISTEN connection (per worker, and the master hub's) now also opens with `connect_timeout=10`, beside the #664 keepalives (a keyword the connection options already set wins, and so does `PGCONNECT_TIMEOUT`; on libpq < 12, where #664 adds no keepalive keyword, `connect_timeout` is still added), so a connect to an unreachable streams host fails in 10 s instead of waiting out the OS's TCP retries.
|
|
13
|
+
- **Testing.** `Pgbus::Testing.fake!` records signals in `Pgbus::Testing.signals` (a `SignalRecord` with `name` and `payload`; `clear_signals!` empties it) without a NOTIFY. `inline!` also runs the blocks synchronously on the caller's thread, without the Rails executor, so `Current` attributes survive. Both record at the commit of a `transaction do … end` block, never on its rollback.
|
|
14
|
+
- **Scope.** Web processes only: pgbus worker processes are not subscribed.
|
|
15
|
+
- **Internals.** The open-transaction probe `Streams::Stream` used is now `Pgbus::OpenTransaction.current`, unchanged. On Rails 7.1, whose transactions have no `after_commit`, deferral goes through a transaction record.
|
|
16
|
+
|
|
17
|
+
Refs #674.
|
|
18
|
+
|
|
3
19
|
### Changed
|
|
4
20
|
|
|
5
21
|
- **A member-keyed presence `touch` no longer probes `information_schema` on a table without the seat columns, and a forced re-check no longer restarts the backoff (issue #666).** On a `pgbus_presence_members` table where `rails g pgbus:add_presence_seats` has not run, every member-keyed `touch`, `leave` and `sweep!` re-probed a cached "no seats" answer before it ran and restarted the #665 backoff at 30 s. The streamer touches once per connection per heartbeat, so a process with N member-keyed connections sent N catalog queries every `streams_heartbeat_interval` (15 s by default), and its backoff never took effect. A member-keyed `touch` now uses the cached answer and re-checks only on the backoff's schedule: three touches after a reset send one probe instead of four, and none while a "no" is cached. Assumed: on a table migrated under a running process, such a `touch` can also refresh that member's seat rows until the next re-check (at most 10 minutes, `SCHEMA_RECHECK_MAX_SECONDS`); it is an `UPDATE` of `last_seen_at`, so nothing is deleted. A member-keyed `leave` and `sweep!` still re-check a cached "no" before they delete, so they never delete seat rows on a migrated table (#515), now with one probe instead of up to two, answered from that probe. That re-check leaves the backoff alone: over a fresh "no" it changes neither the step nor the schedule however many disconnects force one, and over a "no" already due it is the timed re-check, one step. `reset_schema_cache!` still re-probes at once and starts again from 30 s. Refs #666.
|
data/README.md
CHANGED
|
@@ -47,6 +47,7 @@ PostgreSQL-native job processing and event bus for Rails, built on [PGMQ](https:
|
|
|
47
47
|
- [Health endpoints (liveness / readiness)](#health-endpoints-liveness--readiness)
|
|
48
48
|
- [Boot diagnostics banner](#boot-diagnostics-banner)
|
|
49
49
|
- [Real-time broadcasts](#real-time-broadcasts-turbo-streams-replacement)
|
|
50
|
+
- [Process signals](#process-signals)
|
|
50
51
|
- [Testing](#testing)
|
|
51
52
|
- [RSpec setup](#rspec-setup)
|
|
52
53
|
- [Minitest / TestUnit setup](#minitest--testunit-setup)
|
|
@@ -1987,6 +1988,29 @@ The Insights tab gains a "Real-time Streams" section with counts of broadcasts /
|
|
|
1987
1988
|
|
|
1988
1989
|
Overhead on a real Puma + PGMQ setup (`bundle exec rake bench:streams`): the most visible cost is an INSERT per connect/disconnect pair, which shows up under thundering-herd connect scenarios (K=50 concurrent connects: ~+20% per-connect latency). Steady-state broadcast and fanout numbers stay in the run-to-run noise band. Enable it if Insights is useful; leave it off if the write traffic worries you.
|
|
1989
1990
|
|
|
1991
|
+
### Process signals
|
|
1992
|
+
|
|
1993
|
+
Run a block in every web process when any process sends a signal: the way to tell every Puma worker "drop your in-memory cache now" when the data behind it changes.
|
|
1994
|
+
|
|
1995
|
+
```ruby
|
|
1996
|
+
# config/initializers/widgets.rb
|
|
1997
|
+
WIDGETS = Concurrent::Map.new
|
|
1998
|
+
|
|
1999
|
+
Pgbus.on_signal(:widgets_changed, on_gap: -> { WIDGETS.clear }) { |_payload| WIDGETS.clear }
|
|
2000
|
+
|
|
2001
|
+
class Widget < ApplicationRecord
|
|
2002
|
+
after_commit { Pgbus.signal(:widgets_changed) }
|
|
2003
|
+
end
|
|
2004
|
+
```
|
|
2005
|
+
|
|
2006
|
+
- `Pgbus.signal(name, payload = nil)` sends a `NOTIFY` on `<queue_prefix>_signal_<name>`. Inside an open ActiveRecord transaction it is sent at `COMMIT`, and never on rollback. The exception is a `transaction(joinable: false)` block (the kind transactional tests open around each example): there it is sent at once. The payload is an optional String under NOTIFY's 7999-byte cap; an empty String arrives as `nil`.
|
|
2007
|
+
- `Pgbus.on_signal(name) { |payload| ... }` registers a block in this process (register in an initializer, like `Pgbus::Streams.filters`). Names are lowercase letters, digits and underscores, and the channel must fit the queue-name budget.
|
|
2008
|
+
- The signal rides the streamer's LISTEN connection: one per web process under `streams_listen_scope :process`, the master hub's under `:master` (fanned out to every worker, so no worker opens a LISTEN connection of its own). With at least one subscriber, a process's first request starts that listener on a background thread (`Pgbus::Signals::Middleware`, installed by the engine; no request waits on the connect), so a process that never served an SSE connection still receives signals. A failed start is reported and retried by a request 30 s later. Without a subscriber nothing changes: no connection, no thread. Under `:master`, a worker that starts before the master hub has bound its socket uses a LISTEN connection of its own until it recycles, as an SSE-started worker does.
|
|
2009
|
+
- Blocks run on a dedicated thread (not the LISTEN thread, not the stream dispatcher), each inside the Rails executor. A block that raises is reported through `error_reporters` and the other blocks still run. When 1 000 signals are waiting behind a slow block, a further one is dropped and reported as a gap.
|
|
2010
|
+
- Delivery is at most once, within about a second. A signal sent while a process was not listening is lost: before its first LISTEN (whatever it cached at boot, or before a fork under `preload_app!`), while its LISTEN connection reconnected, or while a worker failed over from the master hub. After each such gap, the first LISTEN included, every `on_gap` runs once the LISTEN is in place (gaps that arrive while one is still waiting to run share that run), so it can assume it missed signals. A subscriber whose state can go stale needs an `on_gap`: no request waits for the first LISTEN, so a signal sent between a process's first request and its LISTEN reaches it only as that first gap. A worker whose hub died and whose own listener cannot connect retries every 5 s and runs `on_gap` when it succeeds.
|
|
2011
|
+
- Web processes only: pgbus worker processes do not receive signals. Signals need `streams_enabled` (the default) and a dedicated streams connection, as SSE does: pgbus builds it from `database_url`, a `connection_params` Hash, or the ActiveRecord database configuration, never from a Proc.
|
|
2012
|
+
- In tests, `Pgbus::Testing.fake!` records signals in `Pgbus::Testing.signals` without sending them, and `Pgbus::Testing.inline!` also runs the blocks synchronously on the calling thread, without the Rails executor, so `Current` attributes survive. Both record when the NOTIFY would be sent: at the commit of a `transaction do ... end` block, never on its rollback (the fixture transaction of transactional tests is not waited for). Set a mode in your suite (`config.before { Pgbus::Testing.fake! }`, as for SSE): with `Pgbus::Testing.disabled!` and an `on_signal` registered, the first request spec starts a real streamer, with a LISTEN connection and threads outside the test transaction.
|
|
2013
|
+
|
|
1990
2014
|
## Testing
|
|
1991
2015
|
|
|
1992
2016
|
Pgbus ships opt-in test helpers for both RSpec and Minitest. The testing module is **never autoloaded** by Zeitwerk -- you must `require` it explicitly, so it cannot leak into production.
|
|
@@ -2022,6 +2046,7 @@ RSpec.configure do |config|
|
|
|
2022
2046
|
config.append_after do
|
|
2023
2047
|
Pgbus::Testing.disabled!
|
|
2024
2048
|
Pgbus::Testing.store.clear!
|
|
2049
|
+
Pgbus::Testing.clear_signals!
|
|
2025
2050
|
end
|
|
2026
2051
|
end
|
|
2027
2052
|
```
|
|
@@ -2034,6 +2059,7 @@ RSpec.configure do |config|
|
|
|
2034
2059
|
config.append_after(:each, :pgbus) do
|
|
2035
2060
|
Pgbus::Testing.disabled!
|
|
2036
2061
|
Pgbus::Testing.store.clear!
|
|
2062
|
+
Pgbus::Testing.clear_signals!
|
|
2037
2063
|
end
|
|
2038
2064
|
end
|
|
2039
2065
|
|
|
@@ -2162,6 +2188,8 @@ Pgbus::Testing.disabled? # => true/false
|
|
|
2162
2188
|
|
|
2163
2189
|
The `:inline` mode skips delayed publishes (`delay: > 0`) -- those are captured in the store but not dispatched. Use `Pgbus::Testing.store.drain!` to manually dispatch all captured events including delayed ones.
|
|
2164
2190
|
|
|
2191
|
+
`Pgbus.signal` follows the same modes: `:fake` records `Pgbus::Testing::SignalRecord`s (`name`, `payload`) in `Pgbus::Testing.signals` (`Pgbus::Testing.clear_signals!` empties it), `:inline` records them and runs the process's `on_signal` blocks synchronously, and `:disabled` sends the NOTIFY. See [Process signals](#process-signals).
|
|
2192
|
+
|
|
2165
2193
|
### SSE streams in tests
|
|
2166
2194
|
|
|
2167
2195
|
When using `use_transactional_fixtures = true` (the default in Rails), pgbus SSE streams are incompatible with transactional test isolation. The `rack.hijack` mechanism spawns background threads that acquire their own database connections outside the test transaction, which causes:
|
|
@@ -44,6 +44,22 @@ module Pgbus
|
|
|
44
44
|
end
|
|
45
45
|
end
|
|
46
46
|
|
|
47
|
+
# Fire-and-forget NOTIFY for a process signal (issue #674) on its channel,
|
|
48
|
+
# `<queue_prefix>_signal_<name>` (Pgbus::Signals validates the name and
|
|
49
|
+
# the payload: a String within the cap, or nil for none).
|
|
50
|
+
def notify_signal(name, payload = nil)
|
|
51
|
+
channel = config.queue_name("signal_#{name}")
|
|
52
|
+
Instrumentation.instrument("pgbus.signal.notify", signal: name.to_s, bytes: payload.to_s.bytesize) do
|
|
53
|
+
with_stale_connection_retry do
|
|
54
|
+
synchronized do
|
|
55
|
+
@pgmq.__send__(:with_connection) do |conn|
|
|
56
|
+
conn.exec_params("SELECT pg_notify($1, $2)", [channel, payload])
|
|
57
|
+
end
|
|
58
|
+
end
|
|
59
|
+
end
|
|
60
|
+
end
|
|
61
|
+
end
|
|
62
|
+
|
|
47
63
|
private
|
|
48
64
|
|
|
49
65
|
def validate_notify_payload_size!(stream_name, json)
|
|
@@ -27,14 +27,18 @@ module Pgbus
|
|
|
27
27
|
# probing starts after 5 idle seconds, every 2 s, 3 times, so a dead
|
|
28
28
|
# socket errors within ~11 s even between the listener's own keepalive
|
|
29
29
|
# queries; tcp_user_timeout bounds data the peer never acknowledges by
|
|
30
|
-
# the same 11 s.
|
|
31
|
-
#
|
|
30
|
+
# the same 11 s. connect_timeout bounds the connect itself (issue #674:
|
|
31
|
+
# a streamer starting for process signals, and a hub worker rebuilding
|
|
32
|
+
# its own listener, otherwise wait on an unreachable host as long as the
|
|
33
|
+
# OS lets the TCP handshake retry). Client-side keywords: never in the
|
|
34
|
+
# startup packet, so a pooler cannot reject them.
|
|
32
35
|
LISTEN_KEEPALIVES = {
|
|
33
36
|
keepalives: 1,
|
|
34
37
|
keepalives_idle: 5,
|
|
35
38
|
keepalives_interval: 2,
|
|
36
39
|
keepalives_count: 3,
|
|
37
|
-
tcp_user_timeout: 11_000
|
|
40
|
+
tcp_user_timeout: 11_000,
|
|
41
|
+
connect_timeout: 10
|
|
38
42
|
}.freeze
|
|
39
43
|
|
|
40
44
|
def connect(opts)
|
|
@@ -76,20 +80,29 @@ module Pgbus
|
|
|
76
80
|
defined?(::PG) && ::PG.respond_to?(:library_version) && ::PG.library_version >= 120_000
|
|
77
81
|
end
|
|
78
82
|
|
|
83
|
+
# LISTEN_KEEPALIVES, without connect_timeout when PGCONNECT_TIMEOUT is set:
|
|
84
|
+
# a conninfo keyword beats the environment, and the operator's value wins.
|
|
85
|
+
def listen_keywords
|
|
86
|
+
keywords = libpq_supports_socket_bounds? ? LISTEN_KEEPALIVES : LISTEN_KEEPALIVES.slice(:connect_timeout)
|
|
87
|
+
return keywords if ENV.fetch("PGCONNECT_TIMEOUT", nil).to_s.empty?
|
|
88
|
+
|
|
89
|
+
keywords.except(:connect_timeout)
|
|
90
|
+
end
|
|
91
|
+
|
|
79
92
|
# Fill in the LISTEN_KEEPALIVES keywords the options do not already set.
|
|
80
|
-
#
|
|
93
|
+
# Only connect_timeout on libpq < 12 (see libpq_supports_socket_bounds?;
|
|
94
|
+
# connect_timeout is older than that), and nothing for a
|
|
81
95
|
# connection that names a service (service=, or PGSERVICE): libpq applies
|
|
82
96
|
# the service file only to keywords the conninfo leaves unset, so an added
|
|
83
97
|
# keyword would override the service's own.
|
|
84
98
|
def with_keepalives(opts)
|
|
85
|
-
return opts unless libpq_supports_socket_bounds?
|
|
86
99
|
return opts if ENV.fetch("PGSERVICE", nil).to_s != ""
|
|
87
100
|
|
|
88
101
|
case opts
|
|
89
102
|
when Hash
|
|
90
103
|
return opts if opts.key?(:service) || opts.key?("service")
|
|
91
104
|
|
|
92
|
-
missing =
|
|
105
|
+
missing = listen_keywords.reject { |key, _| opts.key?(key) || opts.key?(key.to_s) }
|
|
93
106
|
opts.merge(missing)
|
|
94
107
|
when String then append_keepalives(opts)
|
|
95
108
|
else opts
|
|
@@ -105,7 +118,7 @@ module Pgbus
|
|
|
105
118
|
set = conninfo_keywords(conninfo)
|
|
106
119
|
return conninfo if set.nil? || set.include?("service")
|
|
107
120
|
|
|
108
|
-
missing =
|
|
121
|
+
missing = listen_keywords.reject { |key, _| set.include?(key.to_s) }
|
|
109
122
|
return conninfo if missing.empty?
|
|
110
123
|
|
|
111
124
|
if url?(conninfo)
|
data/lib/pgbus/engine.rb
CHANGED
|
@@ -160,6 +160,14 @@ module Pgbus
|
|
|
160
160
|
app.middleware.use Pgbus::Streams::WatermarkCacheMiddleware if Pgbus.configuration.streams_enabled
|
|
161
161
|
end
|
|
162
162
|
|
|
163
|
+
# Process signals (issue #674): start the streamer's LISTEN before a
|
|
164
|
+
# process's first request when a signal is registered. Installed whenever
|
|
165
|
+
# streams are enabled, since on_signal usually runs in an initializer after
|
|
166
|
+
# this one; with no subscriber it does nothing.
|
|
167
|
+
initializer "pgbus.signals.middleware" do |app|
|
|
168
|
+
app.middleware.use Pgbus::Signals::Middleware if Pgbus.configuration.streams_enabled
|
|
169
|
+
end
|
|
170
|
+
|
|
163
171
|
# Make the <pgbus-stream-source> element available to the host app's
|
|
164
172
|
# asset pipeline (Propshaft or Sprockets). The importmap pin points at
|
|
165
173
|
# the minified build (rake frontend:js). Propshaft rewrites its
|
|
@@ -0,0 +1,96 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module Pgbus
|
|
4
|
+
# The caller's open ActiveRecord transaction, for work that must wait for
|
|
5
|
+
# its COMMIT: a durable stream broadcast (Streams::Stream) and a process
|
|
6
|
+
# signal (Signals). Guarded by defined? so both stay usable without Rails.
|
|
7
|
+
module OpenTransaction
|
|
8
|
+
# Runs nothing itself: hands the commit hook a block that runs after
|
|
9
|
+
# the commit (and never on rollback).
|
|
10
|
+
#
|
|
11
|
+
# Duck-typed as a record for Rails 7.1, whose transactions have no public
|
|
12
|
+
# after_commit: ActiveRecord calls committed! / rolledback! on every record
|
|
13
|
+
# added to a transaction. 7.2+ use Transaction#after_commit instead.
|
|
14
|
+
class CommitHook
|
|
15
|
+
def initialize(block)
|
|
16
|
+
@block = block
|
|
17
|
+
end
|
|
18
|
+
|
|
19
|
+
def before_committed!; end
|
|
20
|
+
|
|
21
|
+
# should_run_callbacks is false for a duplicate of a record already in
|
|
22
|
+
# the transaction; a hook is only ever equal to itself.
|
|
23
|
+
def committed!(should_run_callbacks: true, **)
|
|
24
|
+
@block.call if should_run_callbacks
|
|
25
|
+
end
|
|
26
|
+
|
|
27
|
+
def rolledback!(*, **); end
|
|
28
|
+
|
|
29
|
+
def trigger_transactional_callbacks?
|
|
30
|
+
true
|
|
31
|
+
end
|
|
32
|
+
end
|
|
33
|
+
|
|
34
|
+
module_function
|
|
35
|
+
|
|
36
|
+
# Returns the current AR transaction if one is open, nil otherwise.
|
|
37
|
+
# AR's NullTransaction yields immediately from after_commit, so we have to
|
|
38
|
+
# check #open? explicitly — otherwise every call path would hit the
|
|
39
|
+
# "deferred" branch and the caller would lose its return value.
|
|
40
|
+
#
|
|
41
|
+
# The probe must only inspect a connection the calling thread/fiber
|
|
42
|
+
# ALREADY leases — `connection_pool.active_connection?`, never
|
|
43
|
+
# `ActiveRecord::Base.connection`. On Rails 7.2+ `.connection` takes a
|
|
44
|
+
# sticky executor-scoped lease: on a non-executor thread (the
|
|
45
|
+
# Coalescer's flush thread, app worker threads) it is never released —
|
|
46
|
+
# one pool connection pinned per thread — and inside
|
|
47
|
+
# `with_connection` the sticky flag defeats the block-exit release, so
|
|
48
|
+
# the CALLER's connection leaks when its thread dies. Both variants
|
|
49
|
+
# exhausted an exactly-sized pool deterministically (Zazu fan-out
|
|
50
|
+
# incident, 2026-08-05). Semantics are unchanged: a transaction is
|
|
51
|
+
# per-lease, so a thread holding no connection can have no open
|
|
52
|
+
# transaction to defer on — the old code's fresh checkout always
|
|
53
|
+
# answered nil anyway, at the price of the leak.
|
|
54
|
+
def current
|
|
55
|
+
return nil unless defined?(::ActiveRecord::Base)
|
|
56
|
+
|
|
57
|
+
connection = ::ActiveRecord::Base.connection_pool.active_connection?
|
|
58
|
+
return nil unless connection
|
|
59
|
+
|
|
60
|
+
transaction = connection.current_transaction
|
|
61
|
+
transaction if transaction.open?
|
|
62
|
+
rescue StandardError => e
|
|
63
|
+
# Defensive: if AR is loaded but not yet connected (e.g. a
|
|
64
|
+
# Rake task invoked before Rails boot), don't let the transaction
|
|
65
|
+
# probe break the caller. Debug-logged so a misconfigured
|
|
66
|
+
# app can diagnose "why isn't my work deferring?" without impacting
|
|
67
|
+
# production performance (debug is off by default).
|
|
68
|
+
Pgbus.logger.debug do
|
|
69
|
+
"[Pgbus::OpenTransaction] transaction probe failed (#{e.class}: #{e.message}); " \
|
|
70
|
+
"falling back to running now"
|
|
71
|
+
end
|
|
72
|
+
nil
|
|
73
|
+
end
|
|
74
|
+
|
|
75
|
+
# Registers +block+ to run after the COMMIT of the caller's open,
|
|
76
|
+
# joinable transaction and returns that transaction; returns nil (and
|
|
77
|
+
# registers nothing) when there is none, so the caller runs the work now.
|
|
78
|
+
#
|
|
79
|
+
# A transaction that is not joinable is not deferred on: in an app's
|
|
80
|
+
# transactional tests that is the fixture transaction, which never
|
|
81
|
+
# commits, so a model's `after_commit { Pgbus.signal(...) }` would never
|
|
82
|
+
# send. Inside it, a `transaction do … end` block is joinable and its
|
|
83
|
+
# commit runs the callbacks, as a production commit does.
|
|
84
|
+
def defer(&block)
|
|
85
|
+
transaction = current
|
|
86
|
+
return nil unless transaction&.joinable?
|
|
87
|
+
|
|
88
|
+
if transaction.respond_to?(:after_commit)
|
|
89
|
+
transaction.after_commit(&block)
|
|
90
|
+
else
|
|
91
|
+
::ActiveRecord::Base.connection_pool.active_connection?.add_transaction_record(CommitHook.new(block))
|
|
92
|
+
end
|
|
93
|
+
transaction
|
|
94
|
+
end
|
|
95
|
+
end
|
|
96
|
+
end
|
|
@@ -0,0 +1,23 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module Pgbus
|
|
4
|
+
module Signals
|
|
5
|
+
# Starts this process's streamer, on a background thread, at its first
|
|
6
|
+
# request when a process signal is registered (issue #674), so a web
|
|
7
|
+
# process that never served an SSE connection still receives signals. The
|
|
8
|
+
# request does not wait for the LISTEN: what it (or anything before it)
|
|
9
|
+
# read before the LISTEN is covered by the on_gap the streamer runs once
|
|
10
|
+
# its first LISTEN is in place. After that it costs one atomic read.
|
|
11
|
+
# Installed by Pgbus::Engine when streams are enabled.
|
|
12
|
+
class Middleware
|
|
13
|
+
def initialize(app)
|
|
14
|
+
@app = app
|
|
15
|
+
end
|
|
16
|
+
|
|
17
|
+
def call(env)
|
|
18
|
+
Pgbus::Signals.start_listening!
|
|
19
|
+
@app.call(env)
|
|
20
|
+
end
|
|
21
|
+
end
|
|
22
|
+
end
|
|
23
|
+
end
|
|
@@ -0,0 +1,282 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require "concurrent/atomic/atomic_reference"
|
|
4
|
+
|
|
5
|
+
module Pgbus
|
|
6
|
+
# Process signals (issue #674): run a block in every web process when any
|
|
7
|
+
# process sends a signal — the "drop your in-memory cache now" message.
|
|
8
|
+
#
|
|
9
|
+
# # config/initializers/widgets.rb
|
|
10
|
+
# Pgbus.on_signal(:widgets_changed, on_gap: -> { WIDGETS.clear }) { |_payload| WIDGETS.clear }
|
|
11
|
+
#
|
|
12
|
+
# # anywhere, after the change (inside a transaction it is sent at COMMIT)
|
|
13
|
+
# Pgbus.signal(:widgets_changed)
|
|
14
|
+
# Pgbus.signal(:widgets_changed, "id=42") # a String payload, under NOTIFY's 7999-byte cap
|
|
15
|
+
#
|
|
16
|
+
# Transport: a NOTIFY on one channel per signal, `<queue_prefix>_signal_<name>`,
|
|
17
|
+
# LISTENed by the connection the streamer already owns: the per-worker
|
|
18
|
+
# Listener under `streams_listen_scope :process`, the MasterHub's under
|
|
19
|
+
# `:master` (fanned out to the workers over the hub socket). Subscribers run
|
|
20
|
+
# on the streamer's signal delivery thread, never on the LISTEN thread, each
|
|
21
|
+
# inside the Rails executor; one that raises is reported and the others run.
|
|
22
|
+
#
|
|
23
|
+
# The registry is process-local, like Pgbus::Streams.filters: register in an
|
|
24
|
+
# initializer. With at least one subscriber, the first request a process
|
|
25
|
+
# serves starts the streamer on a background thread (Signals::Middleware),
|
|
26
|
+
# so a process that never served an SSE connection still listens and no
|
|
27
|
+
# request waits on the connect; with none, nothing changes.
|
|
28
|
+
#
|
|
29
|
+
# Delivery is at-most-once: a NOTIFY sent while a process was not LISTENing
|
|
30
|
+
# (before its streamer started, a reconnect, a hub failover) is lost. Every
|
|
31
|
+
# subscriber's `on_gap` runs once after each such gap, once the LISTEN is in
|
|
32
|
+
# place (the first LISTEN of a process included), so it can assume it
|
|
33
|
+
# missed signals.
|
|
34
|
+
module Signals
|
|
35
|
+
NAME_PATTERN = /\A[a-z0-9_]+\z/
|
|
36
|
+
|
|
37
|
+
# How long a failed eager start waits before the next request tries again.
|
|
38
|
+
START_RETRY_SECONDS = 30
|
|
39
|
+
|
|
40
|
+
# One registered block, and the optional callable run after a LISTEN gap.
|
|
41
|
+
Subscriber = Data.define(:name, :callback, :on_gap)
|
|
42
|
+
|
|
43
|
+
# Built at load, never `||=` (AGENTS.md, Never Do 5). @lock guards
|
|
44
|
+
# @subscribers (a frozen Hash of name → frozen Array of Subscriber,
|
|
45
|
+
# replaced whole on write, so a reader iterates a snapshot). @started_pid
|
|
46
|
+
# is the pid whose streamer started (an atomic, so the per-request check
|
|
47
|
+
# takes no lock once started). @start_lock guards the rest of the
|
|
48
|
+
# eager-start bookkeeping and is only held for constant-time work: the
|
|
49
|
+
# connect runs on a starter thread, outside it.
|
|
50
|
+
@lock = Mutex.new
|
|
51
|
+
@start_lock = Mutex.new
|
|
52
|
+
@subscribers = {}.freeze
|
|
53
|
+
@started_pid = Concurrent::AtomicReference.new(nil)
|
|
54
|
+
@starting_pid = nil
|
|
55
|
+
@failed_at = nil
|
|
56
|
+
|
|
57
|
+
class << self
|
|
58
|
+
# Pgbus.on_signal. Returns the Subscriber.
|
|
59
|
+
def subscribe(name, on_gap: nil, &block)
|
|
60
|
+
name = validate_name!(name)
|
|
61
|
+
raise ArgumentError, "Pgbus.on_signal(#{name.inspect}) needs a block" unless block
|
|
62
|
+
raise ArgumentError, "on_gap: must respond to #call (got #{on_gap.class})" if on_gap && !on_gap.respond_to?(:call)
|
|
63
|
+
|
|
64
|
+
subscriber = Subscriber.new(name: name, callback: block, on_gap: on_gap)
|
|
65
|
+
@lock.synchronize do
|
|
66
|
+
@subscribers = @subscribers.merge(name => [*@subscribers[name], subscriber].freeze).freeze
|
|
67
|
+
end
|
|
68
|
+
# A streamer already running in this process LISTENs on the new
|
|
69
|
+
# channel now; one started later subscribes every channel at start.
|
|
70
|
+
Pgbus::Web::Streamer.listen_signal(channel_for(name))
|
|
71
|
+
subscriber
|
|
72
|
+
end
|
|
73
|
+
|
|
74
|
+
# Pgbus.signal. Sends the NOTIFY now, or at the COMMIT of the caller's
|
|
75
|
+
# open, joinable ActiveRecord transaction (nothing is sent on its
|
|
76
|
+
# rollback; OpenTransaction.defer explains joinable). Returns nil.
|
|
77
|
+
def publish(name, payload = nil)
|
|
78
|
+
name = validate_name!(name)
|
|
79
|
+
validate_payload!(payload)
|
|
80
|
+
# A NOTIFY delivers an empty payload as none: the same in every mode.
|
|
81
|
+
# A frozen copy, so a caller mutating its String afterwards changes
|
|
82
|
+
# neither a deferred NOTIFY nor a test recording.
|
|
83
|
+
payload = payload.empty? ? nil : payload.dup.freeze if payload
|
|
84
|
+
return testing_signal(name, payload) if testing?
|
|
85
|
+
|
|
86
|
+
send_now = -> { Pgbus.client.notify_signal(name, payload) }
|
|
87
|
+
send_now.call unless OpenTransaction.defer(&send_now)
|
|
88
|
+
nil
|
|
89
|
+
end
|
|
90
|
+
|
|
91
|
+
def names
|
|
92
|
+
@lock.synchronize { @subscribers.keys }
|
|
93
|
+
end
|
|
94
|
+
|
|
95
|
+
def subscribed?
|
|
96
|
+
@lock.synchronize { @subscribers.any? }
|
|
97
|
+
end
|
|
98
|
+
|
|
99
|
+
def gap_handlers?
|
|
100
|
+
@lock.synchronize { @subscribers.values.flatten.any?(&:on_gap) }
|
|
101
|
+
end
|
|
102
|
+
|
|
103
|
+
# The NOTIFY channel of +name+: `<queue_prefix>_signal_<name>`, through
|
|
104
|
+
# config.queue_name, so it is bounded by the queue-name budget (47) and
|
|
105
|
+
# always a valid, unquoted-safe identifier under PostgreSQL's 63.
|
|
106
|
+
def channel_for(name)
|
|
107
|
+
Pgbus.configuration.queue_name("signal_#{name}")
|
|
108
|
+
end
|
|
109
|
+
|
|
110
|
+
def channels
|
|
111
|
+
names.map { |name| channel_for(name) }
|
|
112
|
+
end
|
|
113
|
+
|
|
114
|
+
# The registered signal whose channel is +channel+, or nil.
|
|
115
|
+
def name_for(channel)
|
|
116
|
+
names.find { |name| channel_for(name) == channel }
|
|
117
|
+
end
|
|
118
|
+
|
|
119
|
+
# Runs every subscriber of +name+ with +payload+, in registration order.
|
|
120
|
+
# On the streamer's signal delivery thread each runs in the Rails
|
|
121
|
+
# executor; Pgbus::Testing's inline mode passes executor: false, since it
|
|
122
|
+
# runs on the caller's thread, whose CurrentAttributes the executor's
|
|
123
|
+
# completion would reset. Never raises.
|
|
124
|
+
def deliver(name, payload, executor: true)
|
|
125
|
+
subscribers_of(name).each do |subscriber|
|
|
126
|
+
run_safely({ action: "signal_subscriber", signal: name }, executor: executor) do
|
|
127
|
+
subscriber.callback.call(payload)
|
|
128
|
+
end
|
|
129
|
+
end
|
|
130
|
+
nil
|
|
131
|
+
end
|
|
132
|
+
|
|
133
|
+
# Runs every subscriber's on_gap once. Never raises.
|
|
134
|
+
def deliver_gap
|
|
135
|
+
all = @lock.synchronize { @subscribers.values.flatten }
|
|
136
|
+
all.each do |subscriber|
|
|
137
|
+
next unless subscriber.on_gap
|
|
138
|
+
|
|
139
|
+
run_safely({ action: "signal_on_gap", signal: subscriber.name }, executor: true) { subscriber.on_gap.call }
|
|
140
|
+
end
|
|
141
|
+
nil
|
|
142
|
+
end
|
|
143
|
+
|
|
144
|
+
# The eager start Signals::Middleware runs on every request: with a
|
|
145
|
+
# subscriber, start this process's streamer once, on a background
|
|
146
|
+
# thread, so no request waits on the LISTEN connect (a connect to an
|
|
147
|
+
# unreachable streams host can take as long as the OS lets it). The
|
|
148
|
+
# streamer runs every on_gap once its first LISTEN is in place, which
|
|
149
|
+
# covers whatever the process cached before. A no-op under
|
|
150
|
+
# streams_test_mode (Pgbus::Testing.fake!/inline!), with streams
|
|
151
|
+
# disabled, and in a schema rake task. Never raises into the request: a
|
|
152
|
+
# failure is reported, and a request START_RETRY_SECONDS later tries
|
|
153
|
+
# again. Once started, it costs one atomic read.
|
|
154
|
+
def start_listening!
|
|
155
|
+
pid = ::Process.pid
|
|
156
|
+
return if started_pid == pid
|
|
157
|
+
return unless subscribed? && startable?
|
|
158
|
+
|
|
159
|
+
@start_lock.synchronize do
|
|
160
|
+
return if started_pid == pid || @starting_pid == pid
|
|
161
|
+
return if @failed_at && (monotonic_now - @failed_at) < START_RETRY_SECONDS
|
|
162
|
+
|
|
163
|
+
@starting_pid = pid
|
|
164
|
+
begin
|
|
165
|
+
# One short-lived starter thread per attempt, owned by this
|
|
166
|
+
# module: it ends when Streamer.current returns or raises.
|
|
167
|
+
Thread.new { start_streamer(pid) } # rubocop:disable ThreadSafety/NewThread
|
|
168
|
+
rescue ThreadError => e
|
|
169
|
+
# At the process's thread limit: not this request's problem.
|
|
170
|
+
@starting_pid = nil
|
|
171
|
+
@failed_at = monotonic_now
|
|
172
|
+
Pgbus::ErrorReporter.report(e, { action: "signals_start" })
|
|
173
|
+
end
|
|
174
|
+
end
|
|
175
|
+
nil
|
|
176
|
+
end
|
|
177
|
+
|
|
178
|
+
# Streamer.reset! tore this process's streamer down: the next request
|
|
179
|
+
# starts a new one.
|
|
180
|
+
def streamer_reset!
|
|
181
|
+
@start_lock.synchronize { @started_pid.set(nil) }
|
|
182
|
+
end
|
|
183
|
+
|
|
184
|
+
# A fresh registry. Tests only.
|
|
185
|
+
def reset!
|
|
186
|
+
@lock.synchronize { @subscribers = {}.freeze }
|
|
187
|
+
@start_lock.synchronize do
|
|
188
|
+
@started_pid.set(nil)
|
|
189
|
+
@starting_pid = nil
|
|
190
|
+
@failed_at = nil
|
|
191
|
+
end
|
|
192
|
+
end
|
|
193
|
+
|
|
194
|
+
private
|
|
195
|
+
|
|
196
|
+
def started_pid
|
|
197
|
+
# An AtomicReference built at load and never reassigned: reading it
|
|
198
|
+
# takes no lock (the per-request fast path).
|
|
199
|
+
@started_pid.get # rubocop:disable ThreadSafety/ClassInstanceVariable
|
|
200
|
+
end
|
|
201
|
+
|
|
202
|
+
def start_streamer(pid)
|
|
203
|
+
Pgbus::Web::Streamer.current
|
|
204
|
+
@start_lock.synchronize do
|
|
205
|
+
@started_pid.set(pid)
|
|
206
|
+
@failed_at = nil
|
|
207
|
+
end
|
|
208
|
+
rescue StandardError => e
|
|
209
|
+
@start_lock.synchronize { @failed_at = monotonic_now }
|
|
210
|
+
Pgbus::ErrorReporter.report(e, { action: "signals_start" })
|
|
211
|
+
ensure
|
|
212
|
+
# Also for a starter killed by a non-StandardError, so a later
|
|
213
|
+
# request can try again.
|
|
214
|
+
@start_lock.synchronize { @starting_pid = nil }
|
|
215
|
+
end
|
|
216
|
+
|
|
217
|
+
def startable?
|
|
218
|
+
config = Pgbus.configuration
|
|
219
|
+
config.streams_enabled && !config.streams_test_mode && !Pgbus.database_task?
|
|
220
|
+
end
|
|
221
|
+
|
|
222
|
+
def subscribers_of(name)
|
|
223
|
+
@lock.synchronize { @subscribers[name.to_s] } || []
|
|
224
|
+
end
|
|
225
|
+
|
|
226
|
+
def validate_name!(name)
|
|
227
|
+
string = name.to_s
|
|
228
|
+
unless NAME_PATTERN.match?(string)
|
|
229
|
+
raise ArgumentError,
|
|
230
|
+
"Invalid signal name #{name.inspect}: use lowercase letters, digits and underscores only"
|
|
231
|
+
end
|
|
232
|
+
|
|
233
|
+
budget = QueueNameValidator::MAX_QUEUE_NAME_LENGTH
|
|
234
|
+
channel = "#{Pgbus.configuration.queue_prefix}_signal_#{string}"
|
|
235
|
+
if channel.length > budget
|
|
236
|
+
raise ArgumentError,
|
|
237
|
+
"Signal name #{string.inspect} is too long: its channel #{channel.inspect} is " \
|
|
238
|
+
"#{channel.length} chars, the budget is #{budget}"
|
|
239
|
+
end
|
|
240
|
+
|
|
241
|
+
string
|
|
242
|
+
end
|
|
243
|
+
|
|
244
|
+
def validate_payload!(payload)
|
|
245
|
+
return if payload.nil?
|
|
246
|
+
raise ArgumentError, "A signal payload must be a String or nil (got #{payload.class})" unless payload.is_a?(String)
|
|
247
|
+
|
|
248
|
+
limit = Client::NotifyStream::NOTIFY_PAYLOAD_LIMIT_BYTES
|
|
249
|
+
return if payload.bytesize <= limit
|
|
250
|
+
|
|
251
|
+
raise ArgumentError, "A signal payload is #{payload.bytesize} bytes; NOTIFY caps it at #{limit}"
|
|
252
|
+
end
|
|
253
|
+
|
|
254
|
+
def run_safely(context, executor:, &)
|
|
255
|
+
executor ? with_executor(&) : yield
|
|
256
|
+
rescue StandardError => e
|
|
257
|
+
Pgbus::ErrorReporter.report(e, context)
|
|
258
|
+
end
|
|
259
|
+
|
|
260
|
+
def with_executor(&)
|
|
261
|
+
if defined?(::Rails) && ::Rails.respond_to?(:application) && ::Rails.application
|
|
262
|
+
::Rails.application.executor.wrap(&)
|
|
263
|
+
else
|
|
264
|
+
yield
|
|
265
|
+
end
|
|
266
|
+
end
|
|
267
|
+
|
|
268
|
+
def testing?
|
|
269
|
+
defined?(Pgbus::Testing) && !Pgbus::Testing.disabled?
|
|
270
|
+
end
|
|
271
|
+
|
|
272
|
+
def testing_signal(name, payload)
|
|
273
|
+
Pgbus::Testing.record_signal(name, payload)
|
|
274
|
+
nil
|
|
275
|
+
end
|
|
276
|
+
|
|
277
|
+
def monotonic_now
|
|
278
|
+
::Process.clock_gettime(::Process::CLOCK_MONOTONIC)
|
|
279
|
+
end
|
|
280
|
+
end
|
|
281
|
+
end
|
|
282
|
+
end
|
data/lib/pgbus/streams.rb
CHANGED
|
@@ -391,46 +391,10 @@ module Pgbus
|
|
|
391
391
|
end
|
|
392
392
|
end
|
|
393
393
|
|
|
394
|
-
#
|
|
395
|
-
#
|
|
396
|
-
# non-Rails contexts. AR's NullTransaction yields immediately from
|
|
397
|
-
# after_commit, so we have to check #open? explicitly — otherwise
|
|
398
|
-
# every call path would hit the "deferred" branch and we'd lose the
|
|
399
|
-
# msg_id return value.
|
|
400
|
-
#
|
|
401
|
-
# The probe must only inspect a connection the calling thread/fiber
|
|
402
|
-
# ALREADY leases — `connection_pool.active_connection?`, never
|
|
403
|
-
# `ActiveRecord::Base.connection`. On Rails 7.2+ `.connection` takes a
|
|
404
|
-
# sticky executor-scoped lease: on a non-executor thread (the
|
|
405
|
-
# Coalescer's flush thread, app worker threads) it is never released —
|
|
406
|
-
# one pool connection pinned per thread — and inside
|
|
407
|
-
# `with_connection` the sticky flag defeats the block-exit release, so
|
|
408
|
-
# the CALLER's connection leaks when its thread dies. Both variants
|
|
409
|
-
# exhausted an exactly-sized pool deterministically (Zazu fan-out
|
|
410
|
-
# incident, 2026-08-05). Semantics are unchanged: a transaction is
|
|
411
|
-
# per-lease, so a thread holding no connection can have no open
|
|
412
|
-
# transaction to defer on — the old code's fresh checkout always
|
|
413
|
-
# answered nil anyway, at the price of the leak.
|
|
394
|
+
# The caller's open AR transaction, or nil (Pgbus::OpenTransaction.current
|
|
395
|
+
# explains the lease-safe probe).
|
|
414
396
|
def current_open_transaction
|
|
415
|
-
|
|
416
|
-
|
|
417
|
-
connection = ::ActiveRecord::Base.connection_pool.active_connection?
|
|
418
|
-
return nil unless connection
|
|
419
|
-
|
|
420
|
-
transaction = connection.current_transaction
|
|
421
|
-
transaction if transaction.open?
|
|
422
|
-
rescue StandardError => e
|
|
423
|
-
# Defensive: if AR is loaded but not yet connected (e.g. a
|
|
424
|
-
# Rake task invoked before Rails boot), don't let the transaction
|
|
425
|
-
# probe break the broadcast. Debug-logged so a misconfigured
|
|
426
|
-
# app can diagnose "why aren't my transactional broadcasts
|
|
427
|
-
# deferring?" without impacting production performance
|
|
428
|
-
# (debug is off by default).
|
|
429
|
-
Pgbus.logger.debug do
|
|
430
|
-
"[Pgbus::Streams::Stream] transaction probe failed (#{e.class}: #{e.message}); " \
|
|
431
|
-
"falling back to synchronous broadcast"
|
|
432
|
-
end
|
|
433
|
-
nil
|
|
397
|
+
OpenTransaction.current
|
|
434
398
|
end
|
|
435
399
|
end
|
|
436
400
|
end
|