pgbus 0.11.3 → 0.11.4

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: df6f1cbda597599aaad006242fbb0b964edaa8fa599c413dfde949fc3b7e19a2
-  data.tar.gz: 4a57b939f12238bf4aac8fc601ee5cd529fd1311e6c6994dacfb55f925545e4b
+  metadata.gz: c43ef6f3c3efe57de0f647a0811f83bcf98f08e49c63e2da3d6b9cae1c4d3f16
+  data.tar.gz: edb510a56a9410f8cbf127504c3e31ae459561620a7ef62c7011c97f665908ae
 SHA512:
-  metadata.gz: 336b503f39a11252b461b98c124617d16f252a20e8ef343ea48d637d8552a6048ff8c0d9465d95fcb7c737616010da63669a8db3eacb9971d79b6ae9286ed432
-  data.tar.gz: 33d5b40c21ef5213f4014f27577e19c617ee0e71420dc3c3432c4e3d35f941cafbe0e6a41d33bac42cc9ec2eb65a159d3685f1cce28ff44e97a14ca9749134ab
+  metadata.gz: 21f9209e70c028b51c2794ac2a73d71f55b06e0a1e23fede2c7a074c27c732bf88959a859c2f9f79de8484cd4cb51cc4c876fc801ab6981fc05541926312dd34
+  data.tar.gz: f15cf91037ccee58333e0e74648db8336acf40511ef495a608428ff839e340e5c6ddd06a89cde323d2a6f08ea5b2f5abfbd9717acfbd5ed3d2e3023a1b47f84b

data/CHANGELOG.md

@@ -10,6 +10,7 @@
 
 ### Added
 
+- **Doctor check 10: "Dedicated connections" — the preflight now opens the streamer/notify-listener connections exactly the way the runtime does (issue #352).** The streamer's LISTEN connection and the worker `NotifyListener` bypass the Client's pgmq pools entirely (they own a single raw `PG` connection each), so every existing doctor check — all of which probe through `Pgbus::Client` — passed while both dedicated paths were broken. The new check builds each configured dedicated connection via `Pgbus::DedicatedConnection` (streams when `streams_enabled`, notify when `worker_notify_wakeup?`), runs `SELECT 1`, and closes it, failing with the underlying PG error and the affected path label. Not strict-fatal (same reasoning as the Database check: a transient DB blip at boot must not lockstep-abort a fleet), but with `doctor_on_boot` it turns "every SSE request 500s after deploy" into a named failing check in the boot report. Refs #352.
 - **`pgbus dashboard` — print the vendored AppSignal dashboards as import-ready JSON.** The gem ships four AppSignal dashboard definitions (the main one submitted to appsignal/public_config#80 plus the health/streams/throughput extras), but they're stored in the automated-dashboard format — a `metric_keys` trigger wrapping the `dashboard` object — which AppSignal's "Import dashboard" dialog doesn't accept, so trying one on a real app meant hand-unwrapping JSON. `pgbus dashboard [main|health|streams|throughput]` prints the inner dashboard object ready to paste into the import dialog (`pgbus dashboard health | pbcopy`); `pgbus dashboard --list` enumerates the available names and titles.
 - **`config.doctor_on_boot` (+ `pgbus start --doctor` / `--doctor-strict`) — run the doctor preflight inside the booting supervisor, one Rails boot instead of two (issue #347).** The recommended deploy preflight was `bin/pgbus doctor || true` in the container entrypoint before `bin/pgbus start` — but both commands `require config/environment`, so the job container boots the full Rails app **twice** on every deploy (cost scaling with app boot time), and running doctor pre-supervisor makes the `Process liveness` check false-fail (no workers exist yet), which is exactly why the entrypoint has to swallow the exit code with `|| true` — losing any chance to gate on a genuinely-fatal finding. Now `config.doctor_on_boot = :report` (or `pgbus start --doctor`) runs the checks in the already-booted supervisor, after the DB is verified reachable and queues are bootstrapped but **before any worker is forked**: one boot, and the worker-dependent `Process liveness` check is skipped (it has no workers to observe yet and would false-fail on stale prior-generation rows during a redeploy). `config.doctor_on_boot = :strict` (or `--doctor-strict`) additionally **refuses to boot** — raising before forking anything, so the `ensure shutdown` path tears down the heartbeat/health server and no child starts — but *only* on a genuinely-fatal, non-transient check: a `Configuration` failure (a real config bug) or an **absent** PGMQ schema. It deliberately does **not** abort on `Queues`/`Database` failures — those are the transients the lenient queue bootstrap is built to ride out (children crash-and-backoff until the DB recovers), and `verify_connection!` already gated a hard-down DB moments earlier; making them strict-fatal would take down a whole fleet's cold boot in lockstep on a momentarily-saturated primary. Default `nil`/`false` = off (byte-identical to today; the supervisor constructs no `Doctor` and does zero extra work). The standalone `pgbus doctor` command and its exit-code semantics are unchanged. Refs #347.
 - **`Pgbus::Streams::PhlexHelpers` — a Phlex-includable `pgbus_stream_from`, and `Pgbus.stream_name_budget` (issue #334).** Two small helpers that stop consuming apps reimplementing pgbus glue. (1) `pgbus_stream_from` is a Rails view helper, so a Phlex component couldn't call it without hand-registering an output-helper macro (phlex-rails ships no `Phlex::Rails::Helpers::PgbusStreamFrom`). `include Pgbus::Streams::PhlexHelpers` now bridges it exactly like phlex-rails' own `TurboStreamFrom`. phlex-rails stays an **optional** dependency — the module references its `HelperMacros` and is loaded on demand (`require "pgbus/streams/phlex_helpers"`), never eagerly. (2) `Pgbus.stream_name_budget` exposes the maximum stream-name length (the pgmq queue-name cap minus the queue prefix) so apps size/truncate stream identifiers up front instead of hand-computing `MAX_QUEUE_NAME_LENGTH - queue_prefix.length - 1` at every call site. Refs #334.
@@ -28,12 +29,14 @@
 
 ### Changed
 
+- **Streaming/notify failures now reach `config.error_reporters`, not just the log (issue #352).** Four error-level rescues were logger-only, so an error tracker (AppSignal/Sentry/…) never saw them: a `StreamApp` request rescued into a 500 (which, for a broken streamer config, is *every* SSE request on the worker), a `NotifyListener` fatal thread exit, a `NotifyListener` reconnect failure, and a streamer `Listener` reconnect failure. All four now go through `Pgbus::ErrorReporter.report` with an `action:` context tag (`stream_connect`, `notify_listener_fatal`, `notify_listener_reconnect`, `streamer_reconnect`); ErrorReporter logs as before, so operators without reporters lose nothing. Refs #352.
 - **Fanout socket writes now use a 250ms deadline (was the full 5s `streams_write_deadline_ms`), so a transiently-slow SSE client is marked dead ~20× sooner during broadcast fanout.** This is the behavior change behind the head-of-line-blocking fix (see Added, #315): a client that can't absorb a broadcast frame within `streams_fanout_write_deadline_ms` (default 250ms) is dropped and reconnects, replaying the missed frames from the durable archive (durable streams) or getting a fresh re-render (ephemeral). Connect-time replay is unaffected (keeps the full deadline). To restore the previous timing, set `config.streams_fanout_write_deadline_ms = config.streams_write_deadline_ms`. Refs #315.
 - **The generated `config/initializers/pgbus.rb` documents every config group and bakes in realtime broadcast isolation.** The initializer that `rails generate pgbus:install` now writes (see Breaking Changes for the YAML removal) documents each config group inline — queues, pool, retries, recycling, dispatcher, event consumers — with the gem defaults shown as commented reference, and it sets the #311 realtime broadcast isolation by default: `c.streams_broadcast_queue = "realtime"` plus a dedicated `c.capsule :realtime` that drains it (so `pgbus doctor` stays quiet — no undrained-queue footgun). The **code default in `configuration.rb` stays `nil`**, so a programmatic `Pgbus.configure` and existing installs are unchanged; the recommended setup lives in the generated Ruby. Refs #317.
 - **1.0 API freeze: config renames as deprecated aliases, and removal of provably-dead surface.** Renaming now is cheap; renaming after 1.0 is a major-version break, so the semver-audit renames land as **deprecated aliases** (old name works, warns once, removed in 2.0): `skip_recurring` → `recurring_enabled` (positive polarity, the only negative toggle among the `*_enabled` switches — the alias inverts the boolean); `dashboard_filter_parameters`/`dashboard_filter_sensitive` → `web_filter_parameters`/`web_filter_sensitive` (unify on the incumbent `web_` prefix); and `recurring_tasks_file` (singular), which now warns once when set alongside the plural `recurring_tasks_files` instead of being silently ignored. Both spellings of each alias resolve to the same setter. **Removed** (each verified zero-caller in the audit): the `lock_ttl:` keyword on `ensures_uniqueness` (validated and stored in metadata but read by nothing — passing it now raises `ArgumentError` naming the removal and the upgrade guide); the `pgbus:add_job_locks` generator and the `Pgbus::JobLock` model (new installs use `pgbus:add_uniqueness_keys`; `pgbus:migrate_job_locks` still retires the legacy `pgbus_job_locks` table); the internal `with_pgbus_durable` streams helper (use `with_pgbus_broadcast_opts(durable:)`); and the streamer's `reconnect_via_reset` fallback (`connection_factory` is now required and always injected, so reconnect always rebuilds a fresh connection). `log_format=` no longer clobbers a custom logger's formatter, and `streams_presence_patterns`/`streams_presence_member`, `group_mode`, and `streams_falcon_streaming_body` are annotated **experimental** (exempt from the 1.0 stability promise). The `workers=` Array form is documented as permanent (not "legacy") — it is the only way to express N identical anonymous capsules. Refs #283.
 
 ### Fixed
 
+- **`connection_guc_mode = :session` no longer kills every SSE stream and NOTIFY wakeup with `PG::Error: invalid connection option "variables"` (issue #352).** In `:session` mode, `forward_connection_variables` deliberately leaves the database.yml `variables:` hash on the connection options for the *caller* to strip and apply via post-connect `SET` (a transaction-mode pooler rejects the libpq `options` startup param — the reason `:session` mode exists). `Pgbus::Client#wrap_session_gucs` honors that contract on both pgmq pools, but the two dedicated raw-connect paths — the streamer's `build_raw_pg_connection` and the worker `NotifyListener` — passed the hash straight to `PG.connect`, which rejects the non-libpq `:variables` key. Result: the first stream request per process raised, `StreamApp` rescued it into a 500 (so **every** `/pgbus/streams/…` request failed from then on), and NOTIFY-gated wakeups died with workers silently falling back to polling. Both paths now connect through the new `Pgbus::DedicatedConnection.connect`, which strips `:variables` and applies each GUC via post-connect `SET` — dedicated connections *keep* the operator's GUCs (`statement_timeout`, `timezone`, …), matching the pooled paths, and the mechanism works on both a direct port and a session-mode pooler. A guard spec confines raw `PG.connect` call sites to `Client` and `DedicatedConnection`, so the next dedicated-connection path can't silently reintroduce the bypass. Refs #352.
 - **AppSignal main dashboard: job and event lines now split by `job_class` and `routing_key`.** Applied the upstream review suggestions committed on appsignal/public_config#80 to the vendored `dashboard.json`: the "Job status per queue" panel gains a `job_class` wildcard tag filter and label (`%job_class% - %queue% - %status%`), and "Event handler status" gains `routing_key` (`%handler% - %status% - %routing_key%`). Without the wildcard tag entry a `%placeholder%` in the line label never resolves, so lines from different job classes / routing keys were collapsed together. The subscriber already emitted both tags; only the dashboard definition lagged.
 - **Migration generators now route to the pgbus database when `connects_to` is configured, even without an explicit `--database` (issue #344).** `rails g pgbus:add_stream_queues` (and every sibling generator via `Generators::MigrationPath`) keyed the separate-database decision purely on the `--database` flag. An app that had already set `config.connects_to = { database: { writing: :pgbus } }` (so its pgbus migrations live in `db/pgbus_migrate/` per database.yml's `migrations_paths:`) still got the migration written to the **primary** DB's `db/migrate/` on a bare invocation — where it then ran against the wrong database — and the "Next steps" output told the operator to run `rails db:migrate` instead of `rails db:migrate:pgbus`. `MigrationPath` now auto-detects the target database from `connects_to` (via the existing `DatabaseTargetDetector`: runtime config → initializer scan → application.rb scan) when `--database` is absent; an explicit `--database` still wins. The detected name drives both the migration path (resolving `migrations_paths` for that database, falling back to the `db/pgbus_migrate` convention) and the `db:migrate:<name>` suffix in every generator's post-install output. An app with no separate database is unaffected. Refs #344.
 - **`extract_ar_connection_hash` no longer forces `host: "localhost"` / `port: 5432` on a socket-based (host-less) database.yml — pgmq's connections now match ActiveRecord's on Unix-socket dev setups (issue #343).** A local database.yml with no `host:` is a Unix-socket connection: ActiveRecord connects via libpq's default socket (`PGHOST` / the default socket dir), but pgbus's AR-config extraction defaulted the absent `host`/`port` to TCP `localhost:5432`, silently diverging from AR and pointing pgmq's dedicated raw connections at a *different* server (or nothing) on any machine where the socket dir isn't `localhost`. Apps on socket-based dev DBs therefore couldn't use the AR-extraction path at all and had to pin an explicit `connection_params` Hash whose only real job was *not* defaulting `host` (cosmos carried an `after_initialize` block for exactly this). The two fallbacks are dropped: `host`/`port` now pass through as-is and `.compact` removes them when absent, so libpq applies its own socket defaults and matches AR. A config that *does* set `host`/`port` is byte-identical to before. Refs #343.

data/lib/pgbus/dedicated_connection.rb

@@ -0,0 +1,53 @@
+# frozen_string_literal: true
+
+module Pgbus
+  # Single choke point for opening a DEDICATED PG connection outside the
+  # pgmq pools — the streamer's LISTEN connection and the worker
+  # NotifyListener. Every such path MUST route through here (enforced by
+  # spec/pgbus/pg_connect_guard_spec.rb): in :session GUC mode,
+  # Configuration#forward_connection_variables leaves the database.yml
+  # `variables:` hash on the connection options for the caller to apply
+  # post-connect, and :variables is not a libpq keyword — a raw
+  # PG.connect(**opts) fails with `invalid connection option "variables"`
+  # (issue #352). This mirrors Client#wrap_session_gucs, which does the
+  # same for pool connections: the GUCs are applied via post-connect SET
+  # because a transaction-mode pooler rejects the libpq `options` startup
+  # param (the reason :session mode exists).
+  module DedicatedConnection
+    module_function
+
+    def connect(opts)
+      require "pg" unless defined?(::PG::Connection)
+      case opts
+      when String then ::PG.connect(opts)
+      when Hash then connect_from_hash(opts)
+      else
+        raise Pgbus::ConfigurationError,
+              "Cannot build a dedicated PG connection from #{opts.class}. " \
+              "Set database_url or connection_params so pgbus can open its own connection."
+      end
+    end
+
+    def connect_from_hash(opts)
+      variables = opts[:variables]
+      conn = ::PG.connect(**opts.except(:variables))
+      begin
+        variables&.each { |name, value| conn.exec("SET #{name} = '#{value}'") }
+      rescue StandardError
+        # A failing SET (e.g. a bogus GUC name in database.yml variables:)
+        # must not orphan the freshly opened socket — the reconnect loops
+        # retry on a tight backoff and would leak one server connection per
+        # attempt until PostgreSQL exhausts max_connections.
+        close_quietly(conn)
+        raise
+      end
+      conn
+    end
+
+    def close_quietly(conn)
+      conn.close
+    rescue StandardError
+      nil
+    end
+  end
+end

data/lib/pgbus/doctor.rb

@@ -5,16 +5,19 @@ require "pgbus/mcp/health_analyzer"
 
 module Pgbus
   # Preflight diagnostics for a pgbus deployment — the single command that
-  # answers "is this environment healthy enough to run?". Runs nine checks and
+  # answers "is this environment healthy enough to run?". Runs ten checks and
   # returns a machine-readable result plus a human report, so `pgbus doctor`
   # and `rake pgbus:doctor` can gate a deploy or CI run (exit 0 on success,
   # 1 on any failure).
   #
-  # Doctor never touches PGMQ or PostgreSQL directly: every probe goes through
-  # Pgbus::Client (DB, PGMQ schema, queues, NOTIFY) or Pgbus::Web::DataSource
-  # (process liveness, via Pgbus::MCP::HealthAnalyzer). It also never raises —
-  # a broken environment turns into :fail results, never a crash — so it is
-  # safe to run against a database that is down or a half-installed schema.
+  # Doctor probes through Pgbus::Client (DB, PGMQ schema, queues, NOTIFY),
+  # Pgbus::Web::DataSource (process liveness, via Pgbus::MCP::HealthAnalyzer),
+  # or Pgbus::DedicatedConnection (the streamer/notify-listener check — a
+  # deliberate exception to "everything via Client": those runtime paths
+  # bypass the Client's pools, so probing via the Client cannot catch a
+  # broken dedicated path; see issue #352). It never raises — a broken
+  # environment turns into :fail results, never a crash — so it is safe to
+  # run against a database that is down or a half-installed schema.
   class Doctor
     # A single check result. status is one of :ok, :warn, :fail.
     Check = Struct.new(:name, :status, :detail) do
@@ -36,7 +39,8 @@ module Pgbus
       "Process liveness" => :check_processes,
       "GlobalID allowlist" => :check_allowed_global_id_models,
       "Broadcast queue" => :check_broadcast_queue,
-      "Primary affinity" => :check_primary
+      "Primary affinity" => :check_primary,
+      "Dedicated connections" => :check_dedicated_connections
     }.freeze
 
     # Process liveness reads the pgbus_processes table (via HealthAnalyzer), so
@@ -331,6 +335,51 @@ module Pgbus
       Check.new(name: "Primary affinity", status: :warn, detail: "could not determine (#{e.class}: #{e.message})")
     end
 
+    # 10. Dedicated connections — the streamer's LISTEN connection and the
+    # worker NotifyListener bypass the Client's pgmq pools and connect via
+    # DedicatedConnection, so a broken dedicated path is invisible to every
+    # other check (issue #352: :session-mode `:variables` broke ONLY these
+    # connections while every Client path worked). Open each configured
+    # dedicated connection exactly the way the runtime does, probe it, close
+    # it. NOT strict-fatal, same reasoning as the Database check: a transient
+    # DB blip at boot must not lockstep-abort a fleet.
+    def check_dedicated_connections
+      targets = []
+      targets << ["streams", @config.streams_connection_options] if @config.streams_enabled
+      targets << ["worker notify", @config.worker_notify_connection_options] if @config.worker_notify_wakeup?
+
+      if targets.empty?
+        return Check.new(name: "Dedicated connections", status: :ok,
+                         detail: "disabled in config (streams + notify wakeup off)")
+      end
+
+      failures = targets.filter_map { |label, opts| probe_dedicated_connection(label, opts) }
+      if failures.empty?
+        Check.new(name: "Dedicated connections", status: :ok,
+                  detail: "#{targets.map(&:first).join(" + ")} connect OK")
+      else
+        Check.new(name: "Dedicated connections", status: :fail, detail: failures.join("; "))
+      end
+    rescue StandardError => e
+      Check.new(name: "Dedicated connections", status: :fail, detail: "#{e.class}: #{e.message}")
+    end
+
+    # Open one dedicated connection the way the runtime does, verify it
+    # answers, close it. Returns nil on success, "label: error" on failure.
+    def probe_dedicated_connection(label, opts)
+      conn = Pgbus::DedicatedConnection.connect(opts)
+      conn.exec("SELECT 1")
+      nil
+    rescue StandardError => e
+      "#{label}: #{e.class}: #{e.message}"
+    ensure
+      begin
+        conn&.close if conn.respond_to?(:close)
+      rescue StandardError
+        nil
+      end
+    end
+
     # True when some configured worker capsule drains the given queue — either
     # by naming it explicitly or via a "*" wildcard (which drains every queue).
     def worker_drains?(queue)

data/lib/pgbus/process/notify_listener.rb

@@ -146,7 +146,9 @@ module Pgbus
           wait_once
         end
       rescue StandardError => e
-        @logger.error { "[Pgbus::NotifyListener] fatal: #{e.class}: #{e.message}" } if running?
+        # Report, don't just log: a listener that dies at boot silently
+        # degrades every worker to polling (issue #352).
+        ErrorReporter.report(e, { action: "notify_listener_fatal" }) if running?
       ensure
         # Clear @running so #start can spawn a fresh thread after a fatal exit
         # (e.g. build_connection raising at boot). Without this, the dead
@@ -242,7 +244,7 @@ module Pgbus
             # partially-built conn is orphaned and the next retry just allocates
             # another one — leaking PG connections on repeated failures.
             close_quietly(new_conn)
-            @logger.error { "[Pgbus::NotifyListener] reconnect failed: #{e.class}: #{e.message}" }
+            ErrorReporter.report(e, { action: "notify_listener_reconnect" })
             sleep RECONNECT_BACKOFF_SECONDS
             next
           end
@@ -255,11 +257,11 @@ module Pgbus
         end
       end
 
+      # String/Hash go through DedicatedConnection so a :session-mode
+      # `:variables` key never reaches PG.connect (issue #352).
       def build_connection
-        require "pg" unless defined?(::PG::Connection)
         case @connection_options
-        when String then ::PG.connect(@connection_options)
-        when Hash   then ::PG.connect(**@connection_options)
+        when String, Hash then Pgbus::DedicatedConnection.connect(@connection_options)
         else
           raise Pgbus::ConfigurationError,
                 "NotifyListener cannot build a PG connection from #{@connection_options.class}. " \

data/lib/pgbus/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Pgbus
-  VERSION = "0.11.3"
+  VERSION = "0.11.4"
 end

data/lib/pgbus/web/stream_app.rb

@@ -74,7 +74,11 @@ module Pgbus
           unsupported_server
         end
       rescue StandardError => e
-        logger.error { "[Pgbus::StreamApp] #{e.class}: #{e.message}" }
+        # Report, don't just log: a broken streamer config (e.g. bad
+        # streams_connection_options) 500s EVERY stream request on this
+        # worker — logger-only made that invisible to APM (issue #352).
+        ErrorReporter.report(e, { action: "stream_connect" }, config: config)
+        logger.debug { e.backtrace&.join("\n") }
         server_error
       end
 

data/lib/pgbus/web/streamer/instance.rb

@@ -230,16 +230,16 @@ module Pgbus
           probe(validate_primary(build_raw_pg_connection))
         end
 
-        # Raw PG.connect with no validation or probe. This is what the Listener's
+        # Raw connect with no validation or probe. This is what the Listener's
         # connection_factory calls on every reconnect attempt: the reconnect loop
         # runs its own PrimaryValidator (retrying on a replica) and skips the
         # probe to stay cheap, mirroring Pgbus::Process::NotifyListener.
+        # String/Hash go through DedicatedConnection so a :session-mode
+        # `:variables` key never reaches PG.connect (issue #352).
         def build_raw_pg_connection
-          require "pg" unless defined?(::PG::Connection)
           opts = @config.streams_connection_options
           case opts
-          when String then ::PG.connect(opts)
-          when Hash   then ::PG.connect(**opts)
+          when String, Hash then Pgbus::DedicatedConnection.connect(opts)
           when Proc
             # The Proc branch in connection_options typically returns
             # ActiveRecord::Base.connection.raw_connection — a pooled

data/lib/pgbus/web/streamer/listener.rb

@@ -303,7 +303,7 @@ module Pgbus
               # don't leak PG connections, then back off and retry rather than
               # letting the exception kill the listener thread silently.
               close_quietly(new_conn)
-              @logger.error { "[Pgbus::Streamer::Listener] reconnect failed: #{e.class}: #{e.message}" }
+              Pgbus::ErrorReporter.report(e, { action: "streamer_reconnect" })
               sleep RECONNECT_BACKOFF_SECONDS
               next
             end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: pgbus
 version: !ruby/object:Gem::Version
-  version: 0.11.3
+  version: 0.11.4
 platform: ruby
 authors:
 - Mikael Henriksson
@@ -266,6 +266,7 @@ files:
 - lib/pgbus/concurrency/semaphore.rb
 - lib/pgbus/configuration.rb
 - lib/pgbus/configuration/capsule_dsl.rb
+- lib/pgbus/dedicated_connection.rb
 - lib/pgbus/dedup_cache.rb
 - lib/pgbus/doctor.rb
 - lib/pgbus/engine.rb