pgbus 0.9.11 → 0.11.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +11 -0
- data/Rakefile +49 -4
- data/lib/pgbus/active_job/adapter.rb +11 -0
- data/lib/pgbus/client.rb +63 -5
- data/lib/pgbus/configuration.rb +190 -52
- data/lib/pgbus/doctor.rb +26 -2
- data/lib/pgbus/event_bus/registry.rb +61 -2
- data/lib/pgbus/process/notify_listener.rb +18 -2
- data/lib/pgbus/process/worker.rb +19 -3
- data/lib/pgbus/streams/phlex_helpers.rb +32 -0
- data/lib/pgbus/uniqueness.rb +31 -0
- data/lib/pgbus/version.rb +1 -1
- data/lib/pgbus/web/authentication.rb +23 -0
- data/lib/pgbus.rb +18 -0
- metadata +2 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 44b8e6016eb94d129ba3f90831ee6e3f8eb213c928e61766c35fa4641f0a518b
|
|
4
|
+
data.tar.gz: d35475deb1eaaf59e893c7096b10bc85498482f5940fc099e5add48af54aecbd
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: db2122fe97f2b77dd6812e5e9a5264850b4eb84ff3c6094cc2c0e11a148989fa3ae502664e040da26b1ea1bf11b764c2b1f9659a58f9d4a6b663be54c1027cba
|
|
7
|
+
data.tar.gz: 597210a0238131f0a7912b0cf135dc4d89ce66ab830354dca3b3805dbc70a9ccbd2106830dea85b17c68be73fb79868ff9c43d27813e84e6f457dfbe996107d3
|
data/CHANGELOG.md
CHANGED
|
@@ -2,11 +2,17 @@
|
|
|
2
2
|
|
|
3
3
|
### Breaking Changes
|
|
4
4
|
|
|
5
|
+
- **`config.validate!` now rejects malformed values for 12 core config keys at boot instead of failing deep in a worker or silently misbehaving (issue #335).** Before the 1.0 surface freeze, these headline job-path keys had **zero** validation while streams keys had 14 checks — so a bad value surfaced as a crash inside a worker/dispatcher/poller/scheduler thread, per-enqueue, or by silently corrupting every derived queue name / leaving the dashboard open. `Pgbus.configure` (eager-validating by default) now raises `Pgbus::ConfigurationError` naming the key for: the worker-recycling trio `max_jobs_per_worker`/`max_memory_mb`/`max_worker_lifetime` (positive number or `nil` to disable); the intervals `dispatch_interval`/`outbox_poll_interval`/`recurring_schedule_interval` (positive number); `outbox_batch_size` (positive integer); `default_priority` (non-negative integer — `0` is a valid level); `queue_prefix`/`default_queue` (non-empty String — an empty prefix silently produced malformed queue names); `web_auth` (a callable or `nil` — a non-callable silently left the dashboard open); `error_reporters` (an Array); and `connects_to` (a Hash or `nil` — previously a malformed value raised a raw `TypeError` deep in engine boot). This only rejects configuration that was already broken; if a `Pgbus.configure` block set one of these to an invalid value it will now fail loudly at boot (the intended pre-freeze tightening). Set `config.eager_validation = false` to defer. Refs #335, #282.
|
|
6
|
+
- **`config.streams_queue_prefix` is removed (issue #335).** It had been an inert no-op since #308 — stream queues are named like job queues (`#{queue_prefix}_<name>`) and identified via the `pgbus_stream_queues` registry, so the prefix affected nothing — yet it was still a writable accessor documented as "Prefix for stream queues" (false). Rather than carry a dead, misleadingly-documented key into the 1.0 semver promise, the accessor is **removed**: `c.streams_queue_prefix = ...` now raises `NoMethodError`. **Action:** delete the line from your initializer — it never did anything. Refs #335, #308.
|
|
7
|
+
- **`ensures_uniqueness strategy: :until_executed` with no `key:` now raises when the job is enqueued *with arguments* — the class-name default silently collapsed per-record jobs (issue #333).** Without an explicit `key:`, the uniqueness key defaulted to the job class name. For a job that takes per-record arguments (`ImportOrderJob.perform_later(order_id)`), every distinct `order_id` resolved to the same class-name key and collapsed into ONE per-class singleton — order 1 and order 2 shared a lock, so the second was silently discarded/rejected — with no warning. The guard is at **resolve time, argument-aware**: a no-key `:until_executed` job enqueued **with arguments** now raises `ArgumentError` naming the collapse and pointing at the upgrade guide; **a no-argument job keeps the class-name default** (a legitimate "one logical instance at a time" — e.g. a recurring `CleanupJob` that must not overlap itself — is unaffected). `:while_executing` is unaffected (it acquires per-invocation at execution start, not by class-name identity at enqueue). Fix: pass `key: ->(*args) { "job-#{args.first}" }` on any argument-taking `:until_executed` job. Refs #333.
|
|
5
8
|
- **YAML config (`config/pgbus.yml`) is removed for 1.0 — the Ruby initializer is the only config surface.** Ruby was always the real config surface (every production setup is a `Pgbus.configure` initializer); `config/pgbus.yml` was only ever a legacy convenience, and pre-1.0 is the last low-risk moment to drop it before the config shape becomes a compatibility promise. Concretely: the engine no longer loads `config/pgbus.yml` at boot (the `pgbus.configure` initializer and `Pgbus::ConfigLoader` are deleted); `rails generate pgbus:install` generates `config/initializers/pgbus.rb` (done for 1.0, see below); and `rails generate pgbus:update` no longer converts YAML → initializer (`Pgbus::Generators::ConfigConverter`, its `--source`/`--destination`/`--skip-config` options, and the `pgbus.yml.erb` template are deleted) — `pgbus:update` now only detects and installs missing migrations. **If you still have a `config/pgbus.yml`, it is no longer loaded** — the engine warns once at boot (pointing at the docs) so the inert file doesn't quietly diverge from your real config. Port its settings into `config/initializers/pgbus.rb` as a `Pgbus.configure` block and delete the YAML; the last release that auto-converts it via `pgbus:update` is 0.9.x, so run that conversion (or port by hand) **before** upgrading to 1.0. The recurring-tasks YAML (`config/recurring.yml`, `Pgbus::Recurring::ConfigLoader`) is a separate surface and is unchanged. Refs #317.
|
|
6
9
|
- **The error hierarchy is unified under `Pgbus::Error` for the 1.0 API freeze.** Several call sites raised bare stdlib errors that no `rescue Pgbus::Error` could catch, freezing an inconsistent contract into 1.0. Now: **`Configuration#validate!` and every config setter raise `Pgbus::ConfigurationError`** (was `ArgumentError`) — this is the one that can break existing code: a `rescue ArgumentError` around `Pgbus.configure` or a boot-time config read will no longer catch config failures; switch to `rescue Pgbus::Error` (or `rescue Pgbus::ConfigurationError`). The ActiveJob adapter's `perform_all_later` msg_id-count mismatch now raises **`Pgbus::EnqueueError`** (new, was `RuntimeError`); `ExecutionPools::AsyncPool` shutdown/at-capacity raises now use **`Pgbus::ExecutionPoolError`** (new, was `RuntimeError`); `Serializer#locate_global_id` GlobalID rejections raise **`Pgbus::SerializationError`** (was `ArgumentError`). Three error classes that bypassed the base — `PgmqSchema::VersionNotFoundError` and `Streams::SignedName::InvalidSignedName`/`MissingSecret` — are re-parented under `Pgbus::Error`. Three classes that reject a malformed *argument shape* — `Configuration::CapsuleDSL::ParseError`, `Streams::Cursor::InvalidCursor`, `Streams::StreamNameTooLong` — **deliberately stay `ArgumentError` subclasses**; the policy ("argument-shape errors are `ArgumentError`, operational errors are `Pgbus::Error`") is documented in `lib/pgbus.rb` and pinned by `spec/pgbus/error_hierarchy_spec.rb`. Messages are unchanged. Refs #282.
|
|
7
10
|
|
|
8
11
|
### Added
|
|
9
12
|
|
|
13
|
+
- **`Pgbus::Streams::PhlexHelpers` — a Phlex-includable `pgbus_stream_from`, and `Pgbus.stream_name_budget` (issue #334).** Two small helpers that stop consuming apps reimplementing pgbus glue. (1) `pgbus_stream_from` is a Rails view helper, so a Phlex component couldn't call it without hand-registering an output-helper macro (phlex-rails ships no `Phlex::Rails::Helpers::PgbusStreamFrom`). `include Pgbus::Streams::PhlexHelpers` now bridges it exactly like phlex-rails' own `TurboStreamFrom`. phlex-rails stays an **optional** dependency — the module references its `HelperMacros` and is loaded on demand (`require "pgbus/streams/phlex_helpers"`), never eagerly. (2) `Pgbus.stream_name_budget` exposes the maximum stream-name length (the pgmq queue-name cap minus the queue prefix) so apps size/truncate stream identifiers up front instead of hand-computing `MAX_QUEUE_NAME_LENGTH - queue_prefix.length - 1` at every call site. Refs #334.
|
|
14
|
+
- **`EventBus::Registry#setup_all!(safe: true)` — a boot/rake-safe way to eagerly set up event subscribers (issue #334).** Calling `setup_all!` opens a PGMQ connection per subscriber (to create its queue + bind its topic), which is wrong during a `db:`/schema rake task (a live connection blocks `DROP DATABASE`) and crashes boot if the database isn't up yet — so apps wrapped it in a hand-rolled multi-class rescue plus a rake-task skip. `setup_all!(safe: true)` bakes that in: it skips entirely in a schema/asset rake context and swallows a connection error with a warning instead of raising. The default `setup_all!` is unchanged (raises). Refs #334.
|
|
15
|
+
- **Connection-pooler safety (`config.require_primary`, `config.connection_guc_mode`) — pgbus is now correct-and-loud behind a transaction/read-write-splitting pooler instead of silently stalling (issue #332).** Three linked fixes. (1) **Primary affinity:** a read/write-splitting pooler (pgdog/pgcat) can route pgmq's VOLATILE `read`/`archive`/`delete` to a read replica, where workers read nothing (`read_ct=0`) and jobs stop with a **healthy heartbeat** — the supervisor watchdog keys on loop advancement, not read progress, so nothing surfaced the stall. The job read path had zero primary-affinity defense (only the LISTEN/streamer paths validated the primary via `PrimaryValidator`). Setting `config.require_primary = true` (default **false** — a single-primary deployment is byte-identical) makes `verify_connection!` reject a connection that lands on a replica (`pg_is_in_recovery() => t`) at supervisor boot, raising a clear `ConfigurationError` instead of forking children that read nothing. (2) **GUC forwarding:** pgbus's AR-config extraction dropped database.yml's `variables:` block, so `client_min_messages` (and any GUC) never reached pgmq's dedicated raw connections (NOTICE flooding that three consuming apps hand-patched via `connection_params`). GUCs are now forwarded, on both the AR-extracted and explicit-`connection_params` paths, by `config.connection_guc_mode`: `:options` (default) bakes them into the libpq `options` STARTUP param (byte-compatible with today for the read-timeout `statement_timeout`, now also carrying `variables:`), while `:session` applies them via post-connect `SET` on a fresh connection instead — for a transaction-mode PgBouncer that **rejects** the `options` startup param with a FATAL. `pgbus doctor` gains a ninth check ("Primary affinity") that warns (never fails) when the job connection is currently on a replica, naming the direct-port remediation. DB-gated integration tests prove both GUC modes round-trip a forwarded GUC and that `require_primary` verifies on a real primary. Refs #332.
|
|
10
16
|
- **Job-burst tuning guidance + gate benchmark (`rake bench:job_burst`), and a decision NOT to autoscale the job pool (issue #323 phase 3).** Following the issue's measure-first mandate, `benchmarks/job_burst_bench.rb` sweeps worker thread count under a job burst in two connection regimes (pool matched to threads vs. pool fixed small). Measured finding: a job holds a pgmq connection only for the `read_batch` + `archive` round-trip (not the job body), so **the DB connection pool is the hard burst ceiling, not the thread count** — in the fixed-pool regime `peak_busy` never exceeds `pool_size` no matter how many threads run (extra threads just queue on checkout, latency climbs, throughput plateaus), while matching the pool to the thread count scales throughput ~linearly (≈8× from 2→16). The conclusion — issue #323's explicitly sanctioned outcome — is that **elastic job threads on a fixed connection pool cannot help**, so pgbus does *not* autoscale the job execution pool; the fix for job bursts is **static headroom**: raise `threads` (and let `pool_size` follow, or keep it ≥ `threads`). Documented in the Performance & tuning docs. The streams-pool autoscaling below stays the elastic story where it *does* pay off (the streams pool's connection cost genuinely scales with its size). Refs #323.
|
|
11
17
|
- **Fan-out writer throughput gate benchmark (`writer_burst_bench.rb`), and a decision NOT to make the writer pool elastic (issue #323 phase 1).** Following the issue's measure-first mandate, `benchmarks/writer_burst_bench.rb` drives a sustained fan-out burst (a fleet of connections × many wakes) through the `OutboundPump` while sweeping `streams_writer_threads`, across fast/typical socket-write profiles. Measured finding: throughput scales ~linearly with writer count (≈6× from 2→16) — but that is generic parallelism (N workers finish N independent blocking writes ~N× faster), which argues for the **existing static `streams_writer_threads` knob**, not for elastic writers: a fixed-N sweep can't demonstrate the *spikiness* that would make a static count wasteful enough to justify dynamic resize, and the genuinely harmful case (a slow/congested client) is already **shed** by the #320/#322 fan-out deadline + mark-dead path rather than absorbed by scaling writers. Given the elastic-writer rewrite is the issue's highest-risk phase (it would touch the ordering/cursor invariant #321's whole review exists to protect — mis-sizing risks out-of-order SSE frames), the conclusion — issue #323's sanctioned outcome — is that pgbus does **not** autoscale the writer pool; the fan-out-throughput knob is **static `streams_writer_threads`, sized for your peak fan-out fleet**. Documented in the Performance & tuning docs. Refs #323.
|
|
12
18
|
- **Self-tuning elastic streams-pool autoscaling (`config.streams_pool_autoscale`, opt-in, default off).** The dedicated streams pool (durable-broadcast publish + dispatcher replay reads) is a fixed `streams_pool_size` (default 5); under a genuine burst of SSE clients it saturates, and a saturated pool *serialises* replay reads (it doesn't error — the checkout waits; see #324), slowing fan-out. Setting `config.streams_pool_autoscale = true` runs a **periodic maintenance check** (default every 5 minutes, `config.streams_pool_autoscale_interval`) that **grows the streams pool into a fair share of live Postgres connection headroom** while it's saturated and **shrinks it back to `streams_pool_size`** when the burst passes — with no connection-count target to tune: every threshold derives from live `max_connections`. The check adds **no extra connection and no extra thread** — it's a lightweight `pg_stat_activity` query that runs on the streamer's *existing idle LISTEN connection* (a pghero-`capture_query_stats`-style periodic task, run on the listener thread in its health-check window). It counts peer stream processes by a per-process `application_name` tag (`config.streams_application_name`, default `"pgbus_streams"`), and each process grows only into its own fair share of free connections (`free / peers`, claiming ¼, bounded by a per-grow step and half the remaining headroom) so N forked processes provably can't collectively exhaust the database — even in a synchronized cold-boot herd. If the database runs *critically* low on free connections, every process **emergency-shrinks to baseline immediately**, overriding the saturation signal (protecting the DB wins). `config.streams_pool_max` is an **optional** hard per-process ceiling (`nil` = the dynamic fair share is the cap). No per-sample hysteresis or cooldown is needed — the slow cadence is itself the debounce: one grow (or shrink) step per check, and a sustained burst converges over a few checks. Built on the #325 hot-swap primitive; the decision object is a pure function (no thread, no connection), DB-free unit-tested (scripted headroom, incl. convergence-across-checks and emergency-shrink), with a DB-gated integration test proving real grow/shrink/emergency and a `rake bench:pool_autoscale` benchmark (check cost ~0.3 ms; grow-under-saturation `3→7→10`). **Two prerequisite fixes ship with it:** streams-pool connections are now tagged with a per-process `application_name`, and the streams pool is now built from `streams_connection_options` (honoring `streams_database_url`/host/port) rather than the job-database options — the latter a latent wrong-DB bug in the existing streams pool that only surfaced with a separate streams database. Default-off = byte-identical (the Listener runs no maintenance). A pure-*publisher* worker process (no streamer, only `send_stream_message`) autoscales too: each publish opportunistically triggers the same check, throttled to at most once per `streams_pool_autoscale_interval` across all publisher threads (a lock-free compare-and-set claims the window). The publish thread does only that CAS and then hands the work to a background single-worker executor — it never runs the `pg_stat_activity` query or the resize inline (off the hot path), and a quiet process spawns no thread. The query runs through the *job* pool (so it never competes with the streams pool it grows), and the publisher decision is **grow-only**: it grows under its own publish pressure and emergency-shrinks on DB exhaustion, but leaves normal idle shrink to the streamer/consumer that sees the sustained-idle picture. Fail-soft throughout, so a broadcast never fails on a telemetry query. For steady load, a larger static `streams_pool_size` remains the simpler choice — autoscaling earns its keep under genuinely bursty streams traffic. `PoolAutoscaler` lives in `Pgbus::Streams::` (it is a streams-pool controller, not web-specific). Refs #323.
|
|
@@ -26,6 +32,11 @@
|
|
|
26
32
|
|
|
27
33
|
### Fixed
|
|
28
34
|
|
|
35
|
+
- **`config.log_format` no longer breaks `ActiveSupport::TaggedLogging` — tags are preserved instead of dropped, and `logger.tagged` no longer raises (issue #334).** TaggedLogging works by extending the logger's formatter instance with a module that prepends the current tags. Installing a fresh pgbus formatter (`LogFormatter::JSON`/`Text`) stripped that extension, so `logger.tagged("req-42") { ... }` dropped the tag — and, worse, raised `NoMethodError: undefined method 'tagged'` on the pgbus formatter (a deploy-breaker several apps hit, then worked around with a dedicated `$stdout` logger). `log_format=` now re-extends the replacement formatter with `ActiveSupport::TaggedLogging::Formatter` when the previous one carried it, so tagging keeps working. Refs #334.
|
|
36
|
+
- **The "Dashboard is accessible without authentication" warning is no longer a false positive when a gating `base_controller_class` is set (issue #334).** The warning fired whenever `web_auth` was nil, ignoring that a non-default `base_controller_class` (e.g. an `AdminController` with its own `before_action`) already gates the dashboard — so apps set a redundant `web_auth` lambda purely to silence the log line. The warning now stays quiet when `base_controller_class` is anything other than the default `::ActionController::Base`. Refs #334.
|
|
37
|
+
- **`retry_on` on an `:until_executed` job no longer dead-letters the job instead of retrying (issue #333).** The uniqueness key is released only on success or DLQ, but ActiveJob's `retry_on` re-enqueues from *inside* `perform_now` (after incrementing `executions`), while the executor still holds the key. That retry re-enqueue hit the job's own still-held key, was rejected as a duplicate (`JobNotUnique` under `on_conflict: :reject`), and the original message dead-lettered — so a job with both `ensures_uniqueness :until_executed` and `retry_on` lost its retry and DLQ'd on the first transient failure. A retry re-enqueue (`executions > 0`) is now recognized as the same logical job re-acquiring its own key and is allowed through; the existing key row correctly stays held until the job finally succeeds or dead-letters. Cross-job uniqueness (a genuine fresh duplicate, `executions == 0`) is still rejected. Proven end-to-end with a DB-gated integration test (fail-once-then-succeed runs to success exactly once, no `JobNotUnique`, no DLQ, key cleaned up). Refs #333.
|
|
38
|
+
- **A wildcard (`queues: ['*']`) worker no longer adopts event-bus subscriber queues and dead-letters the events (issue #333).** Event-subscriber queues share the job namespace (`#{queue_prefix}_<handler>`) but carry event payloads, not ActiveJob jobs. `resolve_wildcard_queues` excluded only `_dlq` and stream queues, so a wildcard worker claimed event queues too, handed each event to the ActiveJob executor, failed to deserialize it, and DLQ-moved it out of the consumer's reach — forcing apps that run the event bus to hand-maintain an explicit queue list. Wildcard resolution now also excludes every registered event-subscriber queue (new `EventBus::Registry#event_queue_names`), mirroring the existing stream-queue exclusion. Refs #333.
|
|
39
|
+
- **A pooler-deaf NOTIFY listener no longer raises the worker poll floor to 15s — worse latency than a plain no-NOTIFY deployment (issue #332).** pgbus already detects a transaction-mode pooler / replica LISTEN connection with a self-NOTIFY probe (`NotifyProbe`) and logs a loud, actionable error, but the probe's boolean result was **discarded**: a live-but-deaf listener still reported `running? == true`, so `Worker#wake_timeout` treated it as a healthy NOTIFY source and raised the empty-fetch poll floor to `NOTIFY_FALLBACK_POLL_SECONDS` (15s) — behind a pooler the loop would then sleep up to 15s between polls while jobs sat ready. `NotifyListener` now records the probe result and exposes `delivering?`; `wake_timeout` treats a non-delivering listener as absent and keeps polling at the fast `polling_interval` until a direct-port connection is configured. The streamer path needed no change — its listener already waits on the short `streams_listen_health_check_ms` and has no equivalent 15s ceiling. Refs #332.
|
|
29
40
|
- **The dispatcher no longer logs a scary `Concurrency cleanup failed: ... terminating connection due to administrator command` WARN when an idle DB connection is killed between maintenance cycles.** The dispatcher is a long-lived loop that queries the DB on coarse intervals (concurrency cleanup every 300s, most tasks hourly) and, until now, kept its pooled ActiveRecord connection leased across the idle sleep. When the server terminated that idle backend — routine in **local development** (a Postgres restart, `rails db:migrate` calling `pg_terminate_backend` on idle backends, foreman/overmind restarting the DB) — the next cycle reused the dead socket and *every* maintenance task raised `PG::ConnectionBad` with PostgreSQL's `terminating connection due to administrator command` text, surfaced as an alarming WARN even though the process was healthy and self-heals on the next tick. Two changes: (1) `run_maintenance` now returns every leased AR connection to the pool after each cycle (`clear_active_connections!`, mirroring the Streamer's existing `release_ar_connections`), so an idle backend is never reused — the next cycle checks out afresh and AR's `verify!`-on-checkout transparently discards a dead socket and reconnects; (2) every maintenance task's failure log now routes through `log_maintenance_failure`, which recognizes the terminated-idle-connection signal and logs a calm INFO ("reconnecting on the next cycle") instead of a WARN. Non-connection failures still WARN unchanged, and maintenance backoff is unaffected (a self-healing cycle resets the failure streak on the next successful reconnect).
|
|
30
41
|
- **`priority_levels > 1` no longer silently breaks durable streams.** With job priority enabled, `QueueFactory.for` selects `PriorityStrategy`, and `Client#send_message` routes *every* send — including durable stream broadcasts (which pass no `priority:`) — to a `_p<default_priority>` sub-queue. But the streamer LISTENs and replays exclusively on the **bare** `pgmq.q_<stream>` queue (`read_after`, `notify_stream`), so the broadcast never reached the browser; on a fresh DB, `ensure_stream_queue` also raised `PG::UndefinedTable` because `ensure_queue` created only `_p0.._pN`, never the bare table the NOTIFY trigger and archive index attach to. Durable broadcasts now go through a dedicated `Client#send_stream_message` that always targets the bare queue, and `ensure_stream_queue` creates the bare queue directly via `ensure_single_queue` — both independent of the priority strategy, mirroring the bare-name read path. Streams are delivered by a non-consuming peek; priority sub-queues are meaningless for them. Ephemeral broadcasts (the default mode) were never affected. Refs #310.
|
|
31
42
|
- **Stream maintenance and wildcard workers now correctly identify stream queues (finishes the #306/#307 orphan-sweep fix and closes a durable-stream data-loss bug).** `streams_queue_prefix` (default `pgbus_stream`) was consulted only by the dispatcher's `prune_stream_archives` and `sweep_orphan_streams`, but no code ever *named* a queue with it — real stream queues are `#{queue_prefix}_<name>` (`pgbus_<name>`). So both filters matched **zero** real stream queues: per-stream retention never ran, and the age-based orphan drop shipped in #307 was inert. Worse, a *job* queue whose logical name started with `stream_` (physical `pgbus_stream_x`) *did* match and could be dropped. All stream detection now goes through the new `pgbus_stream_queues` registry (see Added): `prune_stream_archives` and `sweep_orphan_streams` iterate registered streams (reviving #306/#307), `compact_archives` skips registered streams so each stream archive is compacted by exactly one path at its per-stream retention (not the 7-day job default), and — closing #309 — a `queues: ['*']` worker's `resolve_wildcard_queues` now excludes registered stream queues, so wildcard workers no longer claim durable broadcasts, fail to deserialize them, and DLQ-move them out of the stream's replay history. `streams_queue_prefix` is retained for backward compatibility but is now inert (documented). Refs #308, #309.
|
data/Rakefile
CHANGED
|
@@ -209,17 +209,62 @@ task :release, %i[version force] do |_t, args|
|
|
|
209
209
|
success "Updated #{version_file}"
|
|
210
210
|
end
|
|
211
211
|
|
|
212
|
+
# Step 1b: Regenerate the frozen lockfiles that pin the pgbus path gem, so the
|
|
213
|
+
# bump ships with them in sync. These are installed with `--frozen`/deployment
|
|
214
|
+
# in CI, so if they still name the OLD version they instant-fail (the Rails 7.1
|
|
215
|
+
# leg with exit 16, and docs-CI on any docs change). Regenerating here keeps the
|
|
216
|
+
# version-pin drift out of the release commit instead of surfacing on the next PR.
|
|
217
|
+
header "Frozen lockfiles"
|
|
218
|
+
# The ONLY thing a version bump changes in these frozen lockfiles is the pgbus
|
|
219
|
+
# path-gem pin — so bump exactly that line, in place, with a string edit.
|
|
220
|
+
#
|
|
221
|
+
# We deliberately do NOT run `bundle lock` here: a full re-resolve trips over
|
|
222
|
+
# constraints that have nothing to do with pgbus. Concretely, docs/Gemfile.lock
|
|
223
|
+
# carries a broad PLATFORMS list (…-gnu / …-musl / arm-linux) for which a
|
|
224
|
+
# platform gem like `thruster` ships no variant, so `bundle lock` fails with
|
|
225
|
+
# "Could not find gems matching 'thruster' valid for all resolution platforms"
|
|
226
|
+
# on any machine whose cache doesn't already hold those exact gems — aborting
|
|
227
|
+
# the release. `bundle lock --local` was even worse (wrong-file write + no
|
|
228
|
+
# fetch). A targeted pin edit sidesteps all of it, is deterministic on any
|
|
229
|
+
# machine, and produces the minimal 2-line diff (the PATH spec + the
|
|
230
|
+
# DEPENDENCIES pin). See #338/#341 and the surgical-bump fix.
|
|
231
|
+
frozen_lockfiles = %w[gemfiles/rails_7_1.gemfile.lock docs/Gemfile.lock]
|
|
232
|
+
regenerated_lockfiles = []
|
|
233
|
+
frozen_lockfiles.each do |lockfile|
|
|
234
|
+
unless File.exist?(lockfile)
|
|
235
|
+
skip "#{lockfile} not present"
|
|
236
|
+
next
|
|
237
|
+
end
|
|
238
|
+
|
|
239
|
+
content = File.read(lockfile)
|
|
240
|
+
# Matches both the PATH-source spec (" pgbus (X.Y.Z)") and the
|
|
241
|
+
# DEPENDENCIES pin (" pgbus (X.Y.Z)"), leaving everything else untouched.
|
|
242
|
+
bumped = content.gsub(/^(\s+pgbus) \([^)]*\)$/, "\\1 (#{new_version})")
|
|
243
|
+
|
|
244
|
+
if bumped == content
|
|
245
|
+
skip "#{lockfile} — no pgbus pin to bump"
|
|
246
|
+
next
|
|
247
|
+
end
|
|
248
|
+
|
|
249
|
+
File.write(lockfile, bumped)
|
|
250
|
+
regenerated_lockfiles << lockfile
|
|
251
|
+
success "Bumped pgbus pin in #{lockfile}"
|
|
252
|
+
end
|
|
253
|
+
|
|
212
254
|
# Step 2: Verify gem builds cleanly
|
|
213
255
|
header "Build verification"
|
|
214
256
|
sh("gem build pgbus.gemspec --strict")
|
|
215
257
|
sh("rm -f pgbus-*.gem")
|
|
216
258
|
success "Gem builds cleanly"
|
|
217
259
|
|
|
218
|
-
# Step 3: Commit version bump
|
|
260
|
+
# Step 3: Commit version bump (+ any re-synced lockfiles)
|
|
219
261
|
header "Git commit"
|
|
220
|
-
|
|
221
|
-
|
|
222
|
-
|
|
262
|
+
paths_to_stage = [version_file, *regenerated_lockfiles]
|
|
263
|
+
staged_changes = paths_to_stage.any? do |path|
|
|
264
|
+
!`git diff #{path}`.strip.empty? || !`git diff --cached #{path}`.strip.empty?
|
|
265
|
+
end
|
|
266
|
+
if staged_changes
|
|
267
|
+
paths_to_stage.each { |path| sh("git add #{path}") }
|
|
223
268
|
sh("git commit -m 'chore: bump version to #{new_version}'")
|
|
224
269
|
success "Committed version bump"
|
|
225
270
|
else
|
|
@@ -112,6 +112,17 @@ module Pgbus
|
|
|
112
112
|
uniqueness_key = Uniqueness.extract_key(payload_hash)
|
|
113
113
|
return false unless uniqueness_key
|
|
114
114
|
|
|
115
|
+
# A retry re-enqueue is the SAME logical job re-acquiring its OWN key.
|
|
116
|
+
# ActiveJob increments `executions` at the start of perform_now (before
|
|
117
|
+
# the body), then retry_on re-enqueues from inside perform_now — while
|
|
118
|
+
# the executor still holds the key (it releases only on success/DLQ). So
|
|
119
|
+
# a re-enqueue with executions > 0 would otherwise hit its own held key,
|
|
120
|
+
# be rejected as a duplicate (JobNotUnique), and dead-letter the original
|
|
121
|
+
# while losing the retry. Let the retry through; the existing key row
|
|
122
|
+
# correctly stays held until the job finally succeeds or dead-letters.
|
|
123
|
+
# See issue #333.
|
|
124
|
+
return false if active_job.executions.to_i.positive?
|
|
125
|
+
|
|
115
126
|
result = Uniqueness.acquire_enqueue_lock(uniqueness_key, active_job)
|
|
116
127
|
|
|
117
128
|
# :no_lock means no enqueue-time lock needed (e.g. :while_executing strategy)
|
data/lib/pgbus/client.rb
CHANGED
|
@@ -79,7 +79,7 @@ module Pgbus
|
|
|
79
79
|
# Both raise clean PG errors — no Ruby Timeout, no Thread#raise. Only
|
|
80
80
|
# safe on this dedicated-connection branch — never on the shared-AR Proc
|
|
81
81
|
# path, where statement_timeout would leak into application queries.
|
|
82
|
-
conn_opts = apply_connection_bounds(conn_opts)
|
|
82
|
+
conn_opts = wrap_session_gucs(apply_connection_bounds(conn_opts))
|
|
83
83
|
@pgmq = PGMQ::Client.new(conn_opts, pool_size: config.resolved_pool_size, pool_timeout: config.pool_timeout)
|
|
84
84
|
@pgmq_mutex = nil
|
|
85
85
|
# Dedicated streams pool (issue #315): isolates the durable-stream
|
|
@@ -95,8 +95,10 @@ module Pgbus
|
|
|
95
95
|
# with a per-process application_name so the autoscaler can count peer
|
|
96
96
|
# processes from pg_stat_activity (issue #323 P1/P2). Snapshot it so a
|
|
97
97
|
# hot-swap rebuilds a byte-identical pool at a new size.
|
|
98
|
-
@streams_conn_opts =
|
|
99
|
-
|
|
98
|
+
@streams_conn_opts = wrap_session_gucs(
|
|
99
|
+
tag_application_name(
|
|
100
|
+
apply_connection_bounds(config.streams_connection_options)
|
|
101
|
+
)
|
|
100
102
|
)
|
|
101
103
|
@streams_pgmq = PGMQ::Client.new(@streams_conn_opts, pool_size: config.streams_pool_size,
|
|
102
104
|
pool_timeout: config.streams_pool_timeout)
|
|
@@ -163,9 +165,22 @@ module Pgbus
|
|
|
163
165
|
# carries the underlying error plus which config source was in use.
|
|
164
166
|
def verify_connection!
|
|
165
167
|
synchronized do
|
|
166
|
-
@pgmq.with_connection
|
|
168
|
+
@pgmq.with_connection do |conn|
|
|
169
|
+
conn.exec("SELECT 1")
|
|
170
|
+
# When require_primary is set, reject a connection that landed on a
|
|
171
|
+
# read-only replica at boot rather than letting a read/write-splitting
|
|
172
|
+
# pooler silently route pgmq's VOLATILE read/archive to a standby,
|
|
173
|
+
# where workers read nothing and jobs stop with a healthy heartbeat
|
|
174
|
+
# (issue #332). Off by default, so a single-primary deployment is
|
|
175
|
+
# unaffected.
|
|
176
|
+
Process::PrimaryValidator.validate_primary!(conn) if config.require_primary
|
|
177
|
+
end
|
|
167
178
|
end
|
|
168
179
|
true
|
|
180
|
+
rescue Process::ReplicaConnectionError => e
|
|
181
|
+
raise ConfigurationError,
|
|
182
|
+
"Database connection via #{connection_source} landed on a read-only replica " \
|
|
183
|
+
"(require_primary is set): #{e.message}"
|
|
169
184
|
rescue PGMQ::Errors::ConnectionError, PG::Error => e
|
|
170
185
|
raise ConfigurationError, "Database connection failed via #{connection_source}: #{e.message}"
|
|
171
186
|
end
|
|
@@ -182,6 +197,17 @@ module Pgbus
|
|
|
182
197
|
true
|
|
183
198
|
end
|
|
184
199
|
|
|
200
|
+
# Whether the job connection currently lands on a read-only replica
|
|
201
|
+
# (pg_is_in_recovery() => t). Used by the doctor to warn about a
|
|
202
|
+
# read/write-splitting pooler that could route pgmq's VOLATILE read/archive
|
|
203
|
+
# to a standby, silently stalling job processing (issue #332). Raw PG error
|
|
204
|
+
# propagates so the caller can render the reason.
|
|
205
|
+
def in_recovery?
|
|
206
|
+
with_raw_connection do |conn|
|
|
207
|
+
conn.exec(Process::PrimaryValidator::RECOVERY_QUERY).getvalue(0, 0) == "t"
|
|
208
|
+
end
|
|
209
|
+
end
|
|
210
|
+
|
|
185
211
|
# The logical queue names pgbus expects to exist based on the configuration
|
|
186
212
|
# (default queue + worker capsules + recurring tasks). Public wrapper around
|
|
187
213
|
# collect_configured_queues so the doctor can diff configured-vs-existing
|
|
@@ -890,7 +916,14 @@ module Pgbus
|
|
|
890
916
|
PG.connect(opts)
|
|
891
917
|
when Hash
|
|
892
918
|
owned = true
|
|
893
|
-
|
|
919
|
+
# :variables is a database.yml convention, not a libpq keyword —
|
|
920
|
+
# strip it before PG.connect and apply the GUCs via SET so this
|
|
921
|
+
# raw bootstrap/DDL connection matches the pooled connections
|
|
922
|
+
# (issue #332). Empty/absent variables is a plain connect.
|
|
923
|
+
variables = opts[:variables]
|
|
924
|
+
conn = PG.connect(**opts.except(:variables))
|
|
925
|
+
variables&.each { |name, value| conn.exec("SET #{name} = '#{value}'") }
|
|
926
|
+
conn
|
|
894
927
|
else
|
|
895
928
|
raise ConfigurationError, "Cannot resolve raw PG connection from #{opts.class}"
|
|
896
929
|
end
|
|
@@ -1299,6 +1332,31 @@ module Pgbus
|
|
|
1299
1332
|
end
|
|
1300
1333
|
end
|
|
1301
1334
|
|
|
1335
|
+
# In :session GUC mode, a Hash conn_opts carrying database.yml `:variables`
|
|
1336
|
+
# must apply those GUCs via post-connect `SET` rather than the libpq
|
|
1337
|
+
# `options` STARTUP param (which a transaction-mode PgBouncer rejects). We
|
|
1338
|
+
# can't pass `:variables` to PG.connect (not a libpq keyword), so wrap the
|
|
1339
|
+
# opts in a fresh-connect factory Proc: pgmq-ruby natively accepts a callable
|
|
1340
|
+
# per pool slot (pgmq connection.rb), and it must return a UNIQUE
|
|
1341
|
+
# PG::Connection each call (pgmq guards against a shared object). Applies to
|
|
1342
|
+
# a Hash with `:variables` only — a String URL or no variables passes through
|
|
1343
|
+
# unchanged, as does :options mode (where forward_connection_variables
|
|
1344
|
+
# already baked the GUCs into `options`). See issue #332.
|
|
1345
|
+
def wrap_session_gucs(conn_opts)
|
|
1346
|
+
return conn_opts unless config.connection_guc_mode == :session
|
|
1347
|
+
return conn_opts unless conn_opts.is_a?(Hash)
|
|
1348
|
+
|
|
1349
|
+
variables = conn_opts[:variables]
|
|
1350
|
+
return conn_opts if variables.nil? || variables.empty?
|
|
1351
|
+
|
|
1352
|
+
pg_opts = conn_opts.except(:variables)
|
|
1353
|
+
lambda do
|
|
1354
|
+
conn = PG.connect(pg_opts)
|
|
1355
|
+
variables.each { |name, value| conn.exec("SET #{name} = '#{value}'") }
|
|
1356
|
+
conn
|
|
1357
|
+
end
|
|
1358
|
+
end
|
|
1359
|
+
|
|
1302
1360
|
def apply_connection_bounds(conn_opts)
|
|
1303
1361
|
timeout = config.read_timeout
|
|
1304
1362
|
return conn_opts unless timeout&.positive?
|
data/lib/pgbus/configuration.rb
CHANGED
|
@@ -102,6 +102,27 @@ module Pgbus
|
|
|
102
102
|
# Requires a matching entry in config/database.yml under the "pgbus" key.
|
|
103
103
|
attr_accessor :connects_to
|
|
104
104
|
|
|
105
|
+
# Reject a job-pool connection that landed on a read-only replica
|
|
106
|
+
# (pg_is_in_recovery() => t). Default false — a correctly-configured
|
|
107
|
+
# single-primary deployment sees no change. Set true when a read/write
|
|
108
|
+
# splitting pooler (pgdog/pgcat) could route pgmq's VOLATILE read/archive
|
|
109
|
+
# to a replica, which otherwise makes workers silently read nothing.
|
|
110
|
+
# See issue #332.
|
|
111
|
+
attr_accessor :require_primary
|
|
112
|
+
|
|
113
|
+
# How pgbus forwards database.yml GUCs (`variables:` such as
|
|
114
|
+
# client_min_messages, and the read-timeout statement_timeout) onto its
|
|
115
|
+
# dedicated pgmq connections:
|
|
116
|
+
# :options (default) — bake them into the libpq `options` STARTUP param
|
|
117
|
+
# (-c key=value). Works everywhere EXCEPT a
|
|
118
|
+
# transaction-mode PgBouncer, which rejects the
|
|
119
|
+
# `options` startup param with a FATAL.
|
|
120
|
+
# :session — apply them via post-connect `SET` statements on a
|
|
121
|
+
# fresh connection instead, so a transaction-mode
|
|
122
|
+
# pooler accepts them. Dedicated-connection path only.
|
|
123
|
+
# See issue #332.
|
|
124
|
+
attr_reader :connection_guc_mode
|
|
125
|
+
|
|
105
126
|
# Zombie message detection — logs a warning when a message is redelivered
|
|
106
127
|
# (read_ct > 1) without any prior failure recorded in pgbus_failed_events.
|
|
107
128
|
attr_accessor :zombie_detection
|
|
@@ -124,7 +145,7 @@ module Pgbus
|
|
|
124
145
|
attr_accessor :health_port, :health_bind
|
|
125
146
|
|
|
126
147
|
# Streams (turbo-rails replacement, SSE-based)
|
|
127
|
-
attr_accessor :streams_enabled, :streams_path, :
|
|
148
|
+
attr_accessor :streams_enabled, :streams_path, :streams_signed_name_secret,
|
|
128
149
|
:streams_default_retention, :streams_retention, :streams_heartbeat_interval,
|
|
129
150
|
:streams_max_connections, :streams_idle_timeout, :streams_listen_health_check_ms,
|
|
130
151
|
:streams_write_deadline_ms, :streams_falcon_streaming_body,
|
|
@@ -248,6 +269,8 @@ module Pgbus
|
|
|
248
269
|
@stats_flush_interval = StatBuffer::DEFAULT_FLUSH_INTERVAL
|
|
249
270
|
|
|
250
271
|
@connects_to = nil
|
|
272
|
+
@require_primary = false
|
|
273
|
+
@connection_guc_mode = :options
|
|
251
274
|
|
|
252
275
|
@web_auth = nil
|
|
253
276
|
@web_refresh_interval = 5000
|
|
@@ -268,12 +291,11 @@ module Pgbus
|
|
|
268
291
|
|
|
269
292
|
@streams_enabled = true
|
|
270
293
|
@streams_path = nil
|
|
271
|
-
#
|
|
272
|
-
#
|
|
273
|
-
#
|
|
274
|
-
#
|
|
275
|
-
#
|
|
276
|
-
@streams_queue_prefix = "pgbus_stream"
|
|
294
|
+
# streams_queue_prefix was removed in 1.0 (issue #335): it had been an
|
|
295
|
+
# inert no-op since #308 (stream queues are named like job queues,
|
|
296
|
+
# `#{queue_prefix}_<name>`, and identified via the pgbus_stream_queues
|
|
297
|
+
# registry). Setting it now raises NoMethodError — remove it from your
|
|
298
|
+
# initializer.
|
|
277
299
|
# Streamer-only connection overrides. The Streamer's Listener owns a
|
|
278
300
|
# dedicated long-lived `wait_for_notify` PG connection that can't go
|
|
279
301
|
# through a PgBouncer in transaction mode (LISTEN/NOTIFY don't survive
|
|
@@ -472,12 +494,23 @@ module Pgbus
|
|
|
472
494
|
# pgbus-installed formatter. A user who set a custom logger with a custom
|
|
473
495
|
# formatter keeps it — log_format only records the intended format for
|
|
474
496
|
# pgbus's own defaults, it must not silently clobber their formatting.
|
|
475
|
-
|
|
476
|
-
|
|
477
|
-
|
|
478
|
-
|
|
479
|
-
|
|
480
|
-
|
|
497
|
+
old_formatter = @logger&.formatter
|
|
498
|
+
return unless pgbus_installable_formatter?(old_formatter)
|
|
499
|
+
|
|
500
|
+
new_formatter = case format
|
|
501
|
+
when :json then LogFormatter::JSON.new
|
|
502
|
+
when :text then LogFormatter::Text.new
|
|
503
|
+
end
|
|
504
|
+
|
|
505
|
+
# ActiveSupport::TaggedLogging works by extending the logger's formatter
|
|
506
|
+
# instance with a module that prepends the current tags (and adds a
|
|
507
|
+
# #tagged method the logger delegates to). Installing a fresh pgbus
|
|
508
|
+
# formatter would strip that, so `logger.tagged { ... }` would drop tags —
|
|
509
|
+
# and, worse, raise NoMethodError: undefined method `tagged` on our
|
|
510
|
+
# formatter. Re-extend the new formatter with the same module when the old
|
|
511
|
+
# one carried it, so tagging keeps working (issue #334).
|
|
512
|
+
preserve_tagged_logging(old_formatter, new_formatter)
|
|
513
|
+
@logger.formatter = new_formatter
|
|
481
514
|
end
|
|
482
515
|
|
|
483
516
|
VALID_BROADCAST_MODES = %i[ephemeral durable].freeze
|
|
@@ -510,6 +543,18 @@ module Pgbus
|
|
|
510
543
|
@group_mode = coerced
|
|
511
544
|
end
|
|
512
545
|
|
|
546
|
+
VALID_CONNECTION_GUC_MODES = %i[options session].freeze
|
|
547
|
+
|
|
548
|
+
def connection_guc_mode=(mode)
|
|
549
|
+
mode = mode.to_sym
|
|
550
|
+
unless VALID_CONNECTION_GUC_MODES.include?(mode)
|
|
551
|
+
raise Pgbus::ConfigurationError,
|
|
552
|
+
"Invalid connection_guc_mode: #{mode}. Must be one of: #{VALID_CONNECTION_GUC_MODES.join(", ")}"
|
|
553
|
+
end
|
|
554
|
+
|
|
555
|
+
@connection_guc_mode = mode
|
|
556
|
+
end
|
|
557
|
+
|
|
513
558
|
VALID_PGMQ_SCHEMA_MODES = %i[auto extension embedded].freeze
|
|
514
559
|
|
|
515
560
|
def pgmq_schema_mode=(mode)
|
|
@@ -594,12 +639,65 @@ module Pgbus
|
|
|
594
639
|
raise Pgbus::ConfigurationError, "insights_default_minutes must be a positive integer"
|
|
595
640
|
end
|
|
596
641
|
|
|
642
|
+
raise Pgbus::ConfigurationError, "require_primary must be true or false" unless [true, false].include?(require_primary)
|
|
643
|
+
|
|
644
|
+
validate_job_path_gaps!
|
|
597
645
|
validate_streams!
|
|
598
646
|
validate_metrics_backend!
|
|
599
647
|
|
|
600
648
|
self
|
|
601
649
|
end
|
|
602
650
|
|
|
651
|
+
# Pre-1.0 surface-freeze: reject malformed values for core job-path keys at
|
|
652
|
+
# boot rather than failing deep in a worker/dispatcher/poller/scheduler
|
|
653
|
+
# thread, per-enqueue, or by silently corrupting queue names / leaving the
|
|
654
|
+
# dashboard open. Mirrors the idioms already used above (issue #335).
|
|
655
|
+
def validate_job_path_gaps!
|
|
656
|
+
# Worker recycling trio: nil disables the limit; a set value must be a
|
|
657
|
+
# positive Numeric (mirror stall_threshold).
|
|
658
|
+
{ max_jobs_per_worker: max_jobs_per_worker, max_memory_mb: max_memory_mb,
|
|
659
|
+
max_worker_lifetime: max_worker_lifetime }.each do |name, value|
|
|
660
|
+
next if value.nil? || (value.is_a?(Numeric) && value.positive?)
|
|
661
|
+
|
|
662
|
+
raise Pgbus::ConfigurationError, "#{name} must be a positive number or nil to disable"
|
|
663
|
+
end
|
|
664
|
+
|
|
665
|
+
# Interval knobs: positive Numeric, never nil (mirror polling_interval).
|
|
666
|
+
{ dispatch_interval: dispatch_interval, outbox_poll_interval: outbox_poll_interval,
|
|
667
|
+
recurring_schedule_interval: recurring_schedule_interval }.each do |name, value|
|
|
668
|
+
raise Pgbus::ConfigurationError, "#{name} must be > 0" unless value.is_a?(Numeric) && value.positive?
|
|
669
|
+
end
|
|
670
|
+
|
|
671
|
+
unless outbox_batch_size.is_a?(Integer) && outbox_batch_size.positive?
|
|
672
|
+
raise Pgbus::ConfigurationError, "outbox_batch_size must be a positive integer"
|
|
673
|
+
end
|
|
674
|
+
|
|
675
|
+
# 0 is a valid priority level (queue_factory clamps to 0..priority_levels-1).
|
|
676
|
+
unless default_priority.is_a?(Integer) && default_priority >= 0
|
|
677
|
+
raise Pgbus::ConfigurationError, "default_priority must be a non-negative integer"
|
|
678
|
+
end
|
|
679
|
+
|
|
680
|
+
# Queue-name components: a nil/empty/non-String prefix silently corrupts
|
|
681
|
+
# every derived queue name (mirror health_bind).
|
|
682
|
+
{ queue_prefix: queue_prefix, default_queue: default_queue }.each do |name, value|
|
|
683
|
+
raise Pgbus::ConfigurationError, "#{name} must be a non-empty String" unless value.is_a?(String) && !value.empty?
|
|
684
|
+
end
|
|
685
|
+
|
|
686
|
+
# web_auth gates the dashboard; a non-callable value silently leaves it
|
|
687
|
+
# open (mirror streams_presence_member).
|
|
688
|
+
unless web_auth.nil? || web_auth.respond_to?(:call)
|
|
689
|
+
raise Pgbus::ConfigurationError, "web_auth must respond to #call (a Proc/lambda) or be nil"
|
|
690
|
+
end
|
|
691
|
+
|
|
692
|
+
raise Pgbus::ConfigurationError, "error_reporters must be an Array" unless error_reporters.is_a?(Array)
|
|
693
|
+
|
|
694
|
+
# connects_to is splatted into ActiveRecord's connects_to at engine boot;
|
|
695
|
+
# a non-Hash raises a raw TypeError there — surface a clean message here.
|
|
696
|
+
return if connects_to.nil? || connects_to.is_a?(Hash)
|
|
697
|
+
|
|
698
|
+
raise Pgbus::ConfigurationError, "connects_to must be a Hash or nil"
|
|
699
|
+
end
|
|
700
|
+
|
|
603
701
|
def validate_streams!
|
|
604
702
|
unless streams_default_retention.is_a?(Numeric) && streams_default_retention >= 0
|
|
605
703
|
raise Pgbus::ConfigurationError, "streams_default_retention must be a non-negative number"
|
|
@@ -1039,7 +1137,17 @@ module Pgbus
|
|
|
1039
1137
|
if database_url
|
|
1040
1138
|
database_url
|
|
1041
1139
|
elsif connection_params
|
|
1042
|
-
connection_params
|
|
1140
|
+
# An explicit connection_params Hash may carry a database.yml-style
|
|
1141
|
+
# `:variables` block; forward it the same way the AR-extracted path does
|
|
1142
|
+
# so client_min_messages etc. actually reach pgmq's connections and a
|
|
1143
|
+
# non-libpq :variables key never reaches PG.connect (issue #332). A
|
|
1144
|
+
# non-Hash connection_params (e.g. a Proc returning a raw connection)
|
|
1145
|
+
# passes through untouched.
|
|
1146
|
+
if connection_params.is_a?(Hash)
|
|
1147
|
+
forward_connection_variables(connection_params.except(:variables), connection_params[:variables])
|
|
1148
|
+
else
|
|
1149
|
+
connection_params
|
|
1150
|
+
end
|
|
1043
1151
|
elsif defined?(ActiveRecord::Base)
|
|
1044
1152
|
# Extract connection config from ActiveRecord so pgmq-ruby creates its
|
|
1045
1153
|
# own dedicated PG connections. Sharing AR's raw_connection via a Proc
|
|
@@ -1067,28 +1175,7 @@ module Pgbus
|
|
|
1067
1175
|
#
|
|
1068
1176
|
# Precedence: streams_database_url > streams_host/port override > base.
|
|
1069
1177
|
def streams_connection_options
|
|
1070
|
-
|
|
1071
|
-
|
|
1072
|
-
base = connection_options
|
|
1073
|
-
return base unless streams_host || streams_port
|
|
1074
|
-
|
|
1075
|
-
case base
|
|
1076
|
-
when Hash
|
|
1077
|
-
result = base.dup
|
|
1078
|
-
result[:host] = streams_host if streams_host
|
|
1079
|
-
result[:port] = streams_port if streams_port
|
|
1080
|
-
result
|
|
1081
|
-
when String
|
|
1082
|
-
# libpq's conninfo parser takes later key=value pairs as overrides
|
|
1083
|
-
# for earlier ones, so we just append. Handles both URI form
|
|
1084
|
-
# (postgres://...) and key=value form.
|
|
1085
|
-
parts = [base]
|
|
1086
|
-
parts << "host=#{streams_host}" if streams_host
|
|
1087
|
-
parts << "port=#{streams_port}" if streams_port
|
|
1088
|
-
parts.join(" ")
|
|
1089
|
-
else
|
|
1090
|
-
base
|
|
1091
|
-
end
|
|
1178
|
+
override_connection_options(url: streams_database_url, host: streams_host, port: streams_port)
|
|
1092
1179
|
end
|
|
1093
1180
|
|
|
1094
1181
|
# Connection options for the Worker's dedicated NotifyListener connection.
|
|
@@ -1096,38 +1183,64 @@ module Pgbus
|
|
|
1096
1183
|
# overridable via worker_notify_database_url / worker_notify_host /
|
|
1097
1184
|
# worker_notify_port so the LISTEN connection can bypass PgBouncer.
|
|
1098
1185
|
def worker_notify_connection_options
|
|
1099
|
-
|
|
1186
|
+
override_connection_options(url: worker_notify_database_url, host: worker_notify_host, port: worker_notify_port)
|
|
1187
|
+
end
|
|
1188
|
+
|
|
1189
|
+
# Resolved notify wakeup flag: defaults to listen_notify when nil.
|
|
1190
|
+
def worker_notify_wakeup?
|
|
1191
|
+
if @worker_notify_wakeup.nil?
|
|
1192
|
+
listen_notify
|
|
1193
|
+
else
|
|
1194
|
+
@worker_notify_wakeup
|
|
1195
|
+
end
|
|
1196
|
+
end
|
|
1197
|
+
|
|
1198
|
+
private
|
|
1199
|
+
|
|
1200
|
+
# Shared resolver for a dedicated connection (streamer LISTEN / worker
|
|
1201
|
+
# NotifyListener) that can override the base `connection_options` to bypass a
|
|
1202
|
+
# pooler. Precedence: a full `url` wins outright; otherwise `host`/`port`
|
|
1203
|
+
# surgically override the base (a Hash gets dup+merge, a String URL/conninfo
|
|
1204
|
+
# gets appended key=value pairs libpq treats as later-wins overrides); with
|
|
1205
|
+
# no override, or a base that is neither Hash nor String, the base passes
|
|
1206
|
+
# through unchanged. Extracted from the byte-identical
|
|
1207
|
+
# streams_connection_options / worker_notify_connection_options (issue #335).
|
|
1208
|
+
def override_connection_options(url:, host:, port:)
|
|
1209
|
+
return url if url
|
|
1100
1210
|
|
|
1101
1211
|
base = connection_options
|
|
1102
|
-
return base unless
|
|
1212
|
+
return base unless host || port
|
|
1103
1213
|
|
|
1104
1214
|
case base
|
|
1105
1215
|
when Hash
|
|
1106
1216
|
result = base.dup
|
|
1107
|
-
result[:host] =
|
|
1108
|
-
result[:port] =
|
|
1217
|
+
result[:host] = host if host
|
|
1218
|
+
result[:port] = port if port
|
|
1109
1219
|
result
|
|
1110
1220
|
when String
|
|
1111
1221
|
parts = [base]
|
|
1112
|
-
parts << "host=#{
|
|
1113
|
-
parts << "port=#{
|
|
1222
|
+
parts << "host=#{host}" if host
|
|
1223
|
+
parts << "port=#{port}" if port
|
|
1114
1224
|
parts.join(" ")
|
|
1115
1225
|
else
|
|
1116
1226
|
base
|
|
1117
1227
|
end
|
|
1118
1228
|
end
|
|
1119
1229
|
|
|
1120
|
-
#
|
|
1121
|
-
|
|
1122
|
-
|
|
1123
|
-
|
|
1124
|
-
|
|
1125
|
-
|
|
1126
|
-
|
|
1230
|
+
# When the logger's previous formatter was extended with
|
|
1231
|
+
# ActiveSupport::TaggedLogging::Formatter (the case for a Rails
|
|
1232
|
+
# Rails.logger), extend the replacement pgbus formatter with the same module
|
|
1233
|
+
# so `logger.tagged`/`push_tags` keep prepending tags instead of dropping
|
|
1234
|
+
# them (and calling #tagged on our formatter stops raising NoMethodError).
|
|
1235
|
+
# No-op when TaggedLogging isn't loaded or the old formatter wasn't tagged.
|
|
1236
|
+
def preserve_tagged_logging(old_formatter, new_formatter)
|
|
1237
|
+
return unless new_formatter
|
|
1238
|
+
return unless defined?(::ActiveSupport::TaggedLogging::Formatter)
|
|
1239
|
+
return unless old_formatter.is_a?(::ActiveSupport::TaggedLogging::Formatter)
|
|
1240
|
+
|
|
1241
|
+
new_formatter.extend(::ActiveSupport::TaggedLogging::Formatter)
|
|
1127
1242
|
end
|
|
1128
1243
|
|
|
1129
|
-
private
|
|
1130
|
-
|
|
1131
1244
|
# True when log_format= may install its own formatter: no formatter set yet
|
|
1132
1245
|
# (nil), a pgbus formatter we installed, or a framework DEFAULT formatter
|
|
1133
1246
|
# (Ruby's Logger::Formatter, Rails' SimpleFormatter) that the app didn't
|
|
@@ -1326,13 +1439,15 @@ module Pgbus
|
|
|
1326
1439
|
# Rails 7.1+ db_config.configuration_hash returns the full config
|
|
1327
1440
|
config_hash = db_config.configuration_hash
|
|
1328
1441
|
|
|
1329
|
-
{
|
|
1442
|
+
base = {
|
|
1330
1443
|
host: config_hash[:host] || "localhost",
|
|
1331
1444
|
port: (config_hash[:port] || 5432).to_i,
|
|
1332
1445
|
dbname: config_hash[:database],
|
|
1333
1446
|
user: config_hash[:username],
|
|
1334
1447
|
password: config_hash[:password]
|
|
1335
1448
|
}.compact
|
|
1449
|
+
|
|
1450
|
+
forward_connection_variables(base, config_hash[:variables])
|
|
1336
1451
|
rescue StandardError => e
|
|
1337
1452
|
# Fallback to Proc path if AR config extraction fails (e.g., adapter
|
|
1338
1453
|
# doesn't expose standard config keys). Log a warning since this path
|
|
@@ -1348,5 +1463,28 @@ module Pgbus
|
|
|
1348
1463
|
-> { ActiveRecord::Base.connection.raw_connection }
|
|
1349
1464
|
end
|
|
1350
1465
|
end
|
|
1466
|
+
|
|
1467
|
+
# database.yml's `variables:` block (e.g. client_min_messages) is a Rails
|
|
1468
|
+
# convention, not a libpq keyword — pgmq-ruby passes the hash straight to
|
|
1469
|
+
# PG.connect, which would ignore/reject a `:variables` key. So carry the GUCs
|
|
1470
|
+
# forward by the mechanism the operator's pooler tolerates:
|
|
1471
|
+
# :options — bake them into the libpq `options` STARTUP param
|
|
1472
|
+
# (-c key=value), appended to any existing options. A
|
|
1473
|
+
# transaction-mode PgBouncer rejects this param.
|
|
1474
|
+
# :session — leave the raw `:variables` hash on the returned options so the
|
|
1475
|
+
# Client applies them via post-connect `SET` (pooler-safe).
|
|
1476
|
+
# Empty/absent variables is a no-op on both paths. See issue #332.
|
|
1477
|
+
def forward_connection_variables(base, variables)
|
|
1478
|
+
return base if variables.nil? || variables.empty?
|
|
1479
|
+
|
|
1480
|
+
case connection_guc_mode
|
|
1481
|
+
when :session
|
|
1482
|
+
base.merge(variables: variables)
|
|
1483
|
+
else
|
|
1484
|
+
gucs = variables.map { |k, v| "-c #{k}=#{v}" }.join(" ")
|
|
1485
|
+
options = [base[:options], gucs].compact.reject(&:empty?).join(" ")
|
|
1486
|
+
base.merge(options: options)
|
|
1487
|
+
end
|
|
1488
|
+
end
|
|
1351
1489
|
end
|
|
1352
1490
|
end
|
data/lib/pgbus/doctor.rb
CHANGED
|
@@ -5,7 +5,7 @@ require "pgbus/mcp/health_analyzer"
|
|
|
5
5
|
|
|
6
6
|
module Pgbus
|
|
7
7
|
# Preflight diagnostics for a pgbus deployment — the single command that
|
|
8
|
-
# answers "is this environment healthy enough to run?". Runs
|
|
8
|
+
# answers "is this environment healthy enough to run?". Runs nine checks and
|
|
9
9
|
# returns a machine-readable result plus a human report, so `pgbus doctor`
|
|
10
10
|
# and `rake pgbus:doctor` can gate a deploy or CI run (exit 0 on success,
|
|
11
11
|
# 1 on any failure).
|
|
@@ -42,7 +42,8 @@ module Pgbus
|
|
|
42
42
|
check_notify,
|
|
43
43
|
check_processes,
|
|
44
44
|
check_allowed_global_id_models,
|
|
45
|
-
check_broadcast_queue
|
|
45
|
+
check_broadcast_queue,
|
|
46
|
+
check_primary
|
|
46
47
|
].map(&:to_h)
|
|
47
48
|
end
|
|
48
49
|
|
|
@@ -248,6 +249,29 @@ module Pgbus
|
|
|
248
249
|
Check.new(name: "Broadcast queue", status: :fail, detail: "#{e.class}: #{e.message}")
|
|
249
250
|
end
|
|
250
251
|
|
|
252
|
+
# 9. Primary affinity — pooler safety (issue #332). A read/write-splitting
|
|
253
|
+
# pooler (pgdog/pgcat) can route pgmq's VOLATILE read/archive to a read
|
|
254
|
+
# replica, where workers read nothing and jobs stop with a healthy
|
|
255
|
+
# heartbeat. If the job connection currently lands on a replica
|
|
256
|
+
# (pg_is_in_recovery() => t), warn with the direct-port remediation. A
|
|
257
|
+
# warning, never a failure: a deliberate replica-read setup is rare but
|
|
258
|
+
# possible, and require_primary is the enforcement knob for those who want a
|
|
259
|
+
# hard stop.
|
|
260
|
+
def check_primary
|
|
261
|
+
if @client.in_recovery?
|
|
262
|
+
return Check.new(name: "Primary affinity", status: :warn,
|
|
263
|
+
detail: "job connection is on a read-only replica (pg_is_in_recovery() => t) — " \
|
|
264
|
+
"a read/write-splitting pooler may be routing pgmq reads to a standby, " \
|
|
265
|
+
"silently stalling jobs. Point the connection at the DIRECT primary port " \
|
|
266
|
+
"(worker_notify_* / streams_* overrides) and set require_primary to reject " \
|
|
267
|
+
"a replica at boot")
|
|
268
|
+
end
|
|
269
|
+
|
|
270
|
+
Check.new(name: "Primary affinity", status: :ok, detail: "on primary")
|
|
271
|
+
rescue StandardError => e
|
|
272
|
+
Check.new(name: "Primary affinity", status: :warn, detail: "could not determine (#{e.class}: #{e.message})")
|
|
273
|
+
end
|
|
274
|
+
|
|
251
275
|
# True when some configured worker capsule drains the given queue — either
|
|
252
276
|
# by naming it explicitly or via a "*" wildcard (which drains every queue).
|
|
253
277
|
def worker_drains?(queue)
|
|
@@ -28,20 +28,79 @@ module Pgbus
|
|
|
28
28
|
subscriber
|
|
29
29
|
end
|
|
30
30
|
|
|
31
|
-
|
|
32
|
-
|
|
31
|
+
# Set up every registered subscriber (creates its queue + binds its topic
|
|
32
|
+
# via Pgbus.client, which opens a PGMQ connection).
|
|
33
|
+
#
|
|
34
|
+
# safe: false (default) — set up unconditionally; a connection error
|
|
35
|
+
# propagates. Use when you know the database is up.
|
|
36
|
+
# safe: true — boot/rake-safe: skip entirely in a schema/db: rake context
|
|
37
|
+
# (opening a PGMQ connection there would block DROP DATABASE and isn't
|
|
38
|
+
# wanted during schema load / asset precompile), and swallow a connection
|
|
39
|
+
# error with a warning instead of crashing boot when the DB isn't ready.
|
|
40
|
+
# This is the path host apps should call from an initializer so they stop
|
|
41
|
+
# hand-wrapping setup_all! in a multi-class rescue (issue #334).
|
|
42
|
+
def setup_all!(safe: false)
|
|
43
|
+
return if safe && schema_task_context?
|
|
44
|
+
|
|
45
|
+
# Snapshot under the mutex — subscribe/clear! mutate @subscribers under
|
|
46
|
+
# @mutex, and setup! does DB I/O we must NOT hold the lock across, so
|
|
47
|
+
# iterate a copy taken atomically.
|
|
48
|
+
subscribers = @mutex.synchronize { @subscribers.dup }
|
|
49
|
+
|
|
50
|
+
subscribers.each do |subscriber|
|
|
51
|
+
subscriber.setup!
|
|
52
|
+
rescue PGMQ::Errors::ConnectionError, PG::ConnectionBad => e
|
|
53
|
+
# Only a genuine CONNECTION failure ("database isn't up yet") is
|
|
54
|
+
# tolerable under safe:; a PG::Error subclass like a syntax/permission/
|
|
55
|
+
# missing-table error is a real setup bug and must still surface.
|
|
56
|
+
raise unless safe
|
|
57
|
+
|
|
58
|
+
Pgbus.logger.warn do
|
|
59
|
+
"[Pgbus] EventBus subscriber setup skipped (#{e.class}: #{e.message}) — " \
|
|
60
|
+
"the database isn't reachable yet; subscribers set up on the next attempt."
|
|
61
|
+
end
|
|
62
|
+
end
|
|
33
63
|
end
|
|
34
64
|
|
|
35
65
|
def handlers_for(routing_key)
|
|
36
66
|
@subscribers.select { |s| matches?(s.pattern, routing_key) }
|
|
37
67
|
end
|
|
38
68
|
|
|
69
|
+
# Physical PGMQ queue names for every registered event subscriber, so a
|
|
70
|
+
# wildcard (`queues: ['*']`) worker can exclude them — an event queue
|
|
71
|
+
# carries event payloads, not ActiveJob jobs, and a job worker that adopts
|
|
72
|
+
# one fails to deserialize and DLQ-moves the event (issue #333). Returns a
|
|
73
|
+
# Set of prefixed names (`#{queue_prefix}_<subscriber>`), matching the
|
|
74
|
+
# pgmq.meta rows the wildcard resolver diffs against.
|
|
75
|
+
def event_queue_names
|
|
76
|
+
@subscribers.to_set { |s| Pgbus.configuration.queue_name(s.queue_name) }
|
|
77
|
+
end
|
|
78
|
+
|
|
39
79
|
def clear!
|
|
40
80
|
@mutex.synchronize { @subscribers.clear }
|
|
41
81
|
end
|
|
42
82
|
|
|
43
83
|
private
|
|
44
84
|
|
|
85
|
+
# Rake task-name prefixes during which pgbus must NOT open a PGMQ
|
|
86
|
+
# connection: creating/dropping/migrating/loading the schema (a live
|
|
87
|
+
# connection blocks DROP DATABASE) and asset precompile (no DB expected).
|
|
88
|
+
SCHEMA_TASK_PREFIXES = %w[db: db_test: assets: webpacker: yarn:].freeze
|
|
89
|
+
private_constant :SCHEMA_TASK_PREFIXES
|
|
90
|
+
|
|
91
|
+
# True when running inside a rake schema/asset task, where setup_all!(safe:)
|
|
92
|
+
# should skip rather than open a connection. Detected from the invoked rake
|
|
93
|
+
# task names; false outside a Rake run.
|
|
94
|
+
def schema_task_context?
|
|
95
|
+
return false unless defined?(::Rake) && ::Rake.respond_to?(:application)
|
|
96
|
+
|
|
97
|
+
tasks = ::Rake.application.top_level_tasks
|
|
98
|
+
tasks.any? { |t| SCHEMA_TASK_PREFIXES.any? { |prefix| t.to_s.start_with?(prefix) } }
|
|
99
|
+
rescue StandardError => e
|
|
100
|
+
Pgbus.logger.debug { "[Pgbus] schema_task_context? detection failed, assuming non-schema: #{e.class}: #{e.message}" }
|
|
101
|
+
false
|
|
102
|
+
end
|
|
103
|
+
|
|
45
104
|
def matches?(pattern, routing_key)
|
|
46
105
|
regex = pattern
|
|
47
106
|
.gsub(".", "\\.")
|
|
@@ -49,12 +49,25 @@ module Pgbus
|
|
|
49
49
|
@running = false
|
|
50
50
|
@thread = nil
|
|
51
51
|
@conn = nil
|
|
52
|
+
# Optimistic until the start-time self-probe runs: assume NOTIFY delivery
|
|
53
|
+
# works so a not-yet-probed listener isn't mistaken for a pooler-deaf one.
|
|
54
|
+
@delivering = true
|
|
52
55
|
end
|
|
53
56
|
|
|
54
57
|
def listening_to
|
|
55
58
|
@state_mutex.synchronize { @listening_to.dup }
|
|
56
59
|
end
|
|
57
60
|
|
|
61
|
+
# Whether the start-time self-probe confirmed this connection can actually
|
|
62
|
+
# receive a NOTIFY. False when a transaction-mode pooler or replica
|
|
63
|
+
# silently drops LISTEN: the thread is still alive (running? == true) but
|
|
64
|
+
# will never wake the loop. The Worker/Consumer consult this so a
|
|
65
|
+
# live-but-deaf listener is treated as absent for wake-timeout purposes —
|
|
66
|
+
# fast polling, not the 15s NOTIFY ceiling (issue #332).
|
|
67
|
+
def delivering?
|
|
68
|
+
@state_mutex.synchronize { @delivering }
|
|
69
|
+
end
|
|
70
|
+
|
|
58
71
|
def start
|
|
59
72
|
@state_mutex.synchronize do
|
|
60
73
|
return self if @running
|
|
@@ -116,8 +129,11 @@ module Pgbus
|
|
|
116
129
|
# One-shot delivery self-probe on the initial connection only. A pooler
|
|
117
130
|
# or replica that silently breaks LISTEN/NOTIFY is surfaced here with an
|
|
118
131
|
# actionable error; the listener still runs and degrades to polling.
|
|
119
|
-
# Reconnects skip the probe to stay cheap.
|
|
120
|
-
|
|
132
|
+
# Reconnects skip the probe to stay cheap. Record the result so the
|
|
133
|
+
# owning worker can drop back to fast polling instead of the 15s NOTIFY
|
|
134
|
+
# ceiling when this connection can't actually deliver (issue #332).
|
|
135
|
+
delivering = NotifyProbe.probe_notify_delivery!(conn, logger: @logger)
|
|
136
|
+
@state_mutex.synchronize { @delivering = delivering }
|
|
121
137
|
@state_mutex.synchronize { @conn = conn }
|
|
122
138
|
drain_commands
|
|
123
139
|
|
data/lib/pgbus/process/worker.rb
CHANGED
|
@@ -415,11 +415,20 @@ module Pgbus
|
|
|
415
415
|
Pgbus::StreamQueue.reset_cache!
|
|
416
416
|
stream_names = Pgbus::StreamQueue.all_names
|
|
417
417
|
|
|
418
|
+
# Event-bus subscriber queues also share the job namespace (pgbus_<handler>)
|
|
419
|
+
# but carry event payloads, not ActiveJob jobs. A wildcard worker that
|
|
420
|
+
# adopts one hands the event to the executor, which fails to deserialize
|
|
421
|
+
# it and DLQ-moves it out of the consumer's reach — so an app running the
|
|
422
|
+
# event bus had to hand-maintain an explicit queue list. Exclude them,
|
|
423
|
+
# like stream queues (issue #333).
|
|
424
|
+
event_names = Pgbus::EventBus::Registry.instance.event_queue_names
|
|
425
|
+
|
|
418
426
|
conn = Pgbus.configuration.connects_to ? Pgbus::BusRecord.connection : ActiveRecord::Base.connection
|
|
419
427
|
all_queues = conn.select_values("SELECT queue_name FROM pgmq.meta ORDER BY queue_name")
|
|
420
428
|
resolved = all_queues
|
|
421
429
|
.reject { |q| q.end_with?(dlq_suffix) }
|
|
422
430
|
.reject { |q| stream_names.include?(q) }
|
|
431
|
+
.reject { |q| event_names.include?(q) }
|
|
423
432
|
.map { |q| q.delete_prefix(prefix) }
|
|
424
433
|
|
|
425
434
|
if resolved.empty?
|
|
@@ -605,13 +614,20 @@ module Pgbus
|
|
|
605
614
|
def wake_timeout
|
|
606
615
|
# A dead listener (running? false) will never wake the loop, so treat
|
|
607
616
|
# it as absent and keep polling at the short interval until
|
|
608
|
-
# ensure_notify_listener restarts it.
|
|
609
|
-
#
|
|
610
|
-
|
|
617
|
+
# ensure_notify_listener restarts it. A live-but-deaf listener
|
|
618
|
+
# (delivering? false — a transaction-mode pooler or replica that drops
|
|
619
|
+
# LISTEN) is the same story: it will never wake us, so don't raise the
|
|
620
|
+
# poll floor to the 15s ceiling behind it (issue #332). Only a live,
|
|
621
|
+
# delivering listener earns the long NOTIFY-mode ceiling.
|
|
622
|
+
return effective_polling_interval unless notify_wakeup? && listener_delivering?
|
|
611
623
|
|
|
612
624
|
[effective_polling_interval, config.polling_interval, NOTIFY_FALLBACK_POLL_SECONDS].max
|
|
613
625
|
end
|
|
614
626
|
|
|
627
|
+
def listener_delivering?
|
|
628
|
+
@notify_listener&.running? && @notify_listener.delivering?
|
|
629
|
+
end
|
|
630
|
+
|
|
615
631
|
def effective_polling_interval
|
|
616
632
|
return config.polling_interval if @consumer_priority.zero?
|
|
617
633
|
|
|
@@ -0,0 +1,32 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module Pgbus
|
|
4
|
+
module Streams
|
|
5
|
+
# Phlex-includable version of the +pgbus_stream_from+ Rails view helper, so
|
|
6
|
+
# a Phlex component can subscribe a page to a pgbus stream the same way it
|
|
7
|
+
# would with turbo-rails' +turbo_stream_from+:
|
|
8
|
+
#
|
|
9
|
+
# class ChatRoom < Phlex::HTML
|
|
10
|
+
# include Pgbus::Streams::PhlexHelpers
|
|
11
|
+
#
|
|
12
|
+
# def view_template
|
|
13
|
+
# pgbus_stream_from(@room)
|
|
14
|
+
# end
|
|
15
|
+
# end
|
|
16
|
+
#
|
|
17
|
+
# phlex-rails doesn't ship a +Phlex::Rails::Helpers::PgbusStreamFrom+, so
|
|
18
|
+
# apps used to hand-register this module locally. It is defined exactly like
|
|
19
|
+
# phlex-rails' own +Phlex::Rails::Helpers::TurboStreamFrom+ (register the
|
|
20
|
+
# method as an output helper resolved against the Rails view context).
|
|
21
|
+
#
|
|
22
|
+
# This file is NOT autoloaded — phlex-rails is an OPTIONAL dependency, so the
|
|
23
|
+
# module references +Phlex::Rails::HelperMacros+, which only exists when the
|
|
24
|
+
# app has phlex-rails. Require it explicitly from a Phlex component or an
|
|
25
|
+
# initializer: require "pgbus/streams/phlex_helpers".
|
|
26
|
+
module PhlexHelpers
|
|
27
|
+
extend Phlex::Rails::HelperMacros
|
|
28
|
+
|
|
29
|
+
register_output_helper def pgbus_stream_from(...) = nil
|
|
30
|
+
end
|
|
31
|
+
end
|
|
32
|
+
end
|
data/lib/pgbus/uniqueness.rb
CHANGED
|
@@ -62,9 +62,18 @@ module Pgbus
|
|
|
62
62
|
raise ArgumentError, "on_conflict must be one of: #{VALID_CONFLICTS.join(", ")}" unless VALID_CONFLICTS.include?(on_conflict)
|
|
63
63
|
raise ArgumentError, "key must be callable (Proc or lambda)" if key && !key.respond_to?(:call)
|
|
64
64
|
|
|
65
|
+
# Record whether an explicit key was given. With NO explicit key the key
|
|
66
|
+
# defaults to the class name; that is safe for a no-argument job (one
|
|
67
|
+
# logical instance — e.g. a recurring CleanupJob that must not overlap
|
|
68
|
+
# itself) but a silent-correctness footgun for a job that takes per-record
|
|
69
|
+
# arguments: `ImportOrderJob.perform_later(order_id)` would collapse every
|
|
70
|
+
# order into ONE per-class singleton. resolve_key raises at resolve time
|
|
71
|
+
# when the class-name default meets non-empty arguments (see #333); the
|
|
72
|
+
# no-arg case keeps working. explicit_key marks which is which.
|
|
65
73
|
@pgbus_uniqueness = {
|
|
66
74
|
strategy: strategy,
|
|
67
75
|
key: key || ->(*) { name },
|
|
76
|
+
explicit_key: !key.nil?,
|
|
68
77
|
on_conflict: on_conflict
|
|
69
78
|
}.freeze
|
|
70
79
|
end
|
|
@@ -80,6 +89,7 @@ module Pgbus
|
|
|
80
89
|
return nil unless config
|
|
81
90
|
|
|
82
91
|
args = active_job.arguments
|
|
92
|
+
guard_class_name_default!(active_job, config, args)
|
|
83
93
|
last = args.last
|
|
84
94
|
key = if last.is_a?(Hash) && last.each_key.all?(Symbol)
|
|
85
95
|
config[:key].call(*args[...-1], **last)
|
|
@@ -93,6 +103,27 @@ module Pgbus
|
|
|
93
103
|
key
|
|
94
104
|
end
|
|
95
105
|
|
|
106
|
+
# Guards the class-name default key against the per-record collapse
|
|
107
|
+
# footgun (#333). When an :until_executed job was declared with NO explicit
|
|
108
|
+
# key (so the key is the class name) AND is enqueued WITH arguments, every
|
|
109
|
+
# distinct argument set would resolve to the same class-name key and
|
|
110
|
+
# collapse into one per-class singleton — almost never what the caller
|
|
111
|
+
# wants. Raise with an actionable message. A no-argument job keeps the
|
|
112
|
+
# class-name default (one logical instance, e.g. a recurring task that must
|
|
113
|
+
# not overlap itself), and :while_executing is unaffected (it acquires
|
|
114
|
+
# per-invocation at execution start, not by class-name identity at enqueue).
|
|
115
|
+
def guard_class_name_default!(active_job, config, args)
|
|
116
|
+
return if config[:explicit_key]
|
|
117
|
+
return unless config[:strategy] == :until_executed
|
|
118
|
+
return if args.nil? || args.empty?
|
|
119
|
+
|
|
120
|
+
raise ArgumentError,
|
|
121
|
+
"#{active_job.class.name} uses ensures_uniqueness strategy: :until_executed with no key: " \
|
|
122
|
+
"but is enqueued with arguments — the default key is the class name, which would collapse " \
|
|
123
|
+
"every distinct argument set into one per-class singleton. Pass an explicit " \
|
|
124
|
+
"key: ->(*args) { ... } that includes the arguments. See https://pgbus.dev/docs/upgrading-pgbus"
|
|
125
|
+
end
|
|
126
|
+
|
|
96
127
|
def inject_metadata(active_job, payload_hash)
|
|
97
128
|
config = uniqueness_config(active_job)
|
|
98
129
|
return payload_hash unless config
|
data/lib/pgbus/version.rb
CHANGED
|
@@ -13,6 +13,12 @@ module Pgbus
|
|
|
13
13
|
attr_accessor :auth_warned
|
|
14
14
|
end
|
|
15
15
|
|
|
16
|
+
# Default base controller — anything else means the app deliberately
|
|
17
|
+
# chose a controller (e.g. an AdminController) that already gates access,
|
|
18
|
+
# so the dashboard isn't actually open and the warning is a false positive.
|
|
19
|
+
DEFAULT_BASE_CONTROLLER = "::ActionController::Base"
|
|
20
|
+
private_constant :DEFAULT_BASE_CONTROLLER
|
|
21
|
+
|
|
16
22
|
private
|
|
17
23
|
|
|
18
24
|
def authenticate_pgbus!
|
|
@@ -30,6 +36,10 @@ module Pgbus
|
|
|
30
36
|
|
|
31
37
|
def warn_unauthenticated_dashboard
|
|
32
38
|
return if Pgbus::Web::Authentication.auth_warned
|
|
39
|
+
# A non-default base_controller_class signals host-level auth is in
|
|
40
|
+
# place; don't nag apps that gate the dashboard via a custom controller
|
|
41
|
+
# (they had to set a redundant web_auth lambda just to silence us — #334).
|
|
42
|
+
return unless default_base_controller?
|
|
33
43
|
|
|
34
44
|
Pgbus.logger.warn do
|
|
35
45
|
"[Pgbus] Dashboard is accessible without authentication. " \
|
|
@@ -38,6 +48,19 @@ module Pgbus
|
|
|
38
48
|
end
|
|
39
49
|
Pgbus::Web::Authentication.auth_warned = true
|
|
40
50
|
end
|
|
51
|
+
|
|
52
|
+
# True when base_controller_class is (still) the default ActionController::Base.
|
|
53
|
+
# Normalizes both sides so a Class value, the "::"-prefixed string, and the
|
|
54
|
+
# bare "ActionController::Base" string all compare equal — otherwise the
|
|
55
|
+
# warning would wrongly fire for an app that set the default in another form.
|
|
56
|
+
def default_base_controller?
|
|
57
|
+
normalize_controller_name(Pgbus.configuration.base_controller_class) ==
|
|
58
|
+
normalize_controller_name(DEFAULT_BASE_CONTROLLER)
|
|
59
|
+
end
|
|
60
|
+
|
|
61
|
+
def normalize_controller_name(value)
|
|
62
|
+
value.to_s.delete_prefix("::")
|
|
63
|
+
end
|
|
41
64
|
end
|
|
42
65
|
end
|
|
43
66
|
end
|
data/lib/pgbus.rb
CHANGED
|
@@ -103,6 +103,11 @@ module Pgbus
|
|
|
103
103
|
# the gem and loads the subsystem explicitly.
|
|
104
104
|
loader.ignore("#{__dir__}/pgbus/mcp")
|
|
105
105
|
loader.ignore("#{__dir__}/pgbus/mcp.rb")
|
|
106
|
+
# The Phlex stream helper references Phlex::Rails::HelperMacros from the
|
|
107
|
+
# optional `phlex-rails` gem. Keep it out of Zeitwerk so eager_load never
|
|
108
|
+
# touches those constants; a Phlex app requires it explicitly via
|
|
109
|
+
# `require "pgbus/streams/phlex_helpers"`.
|
|
110
|
+
loader.ignore("#{__dir__}/pgbus/streams/phlex_helpers.rb")
|
|
106
111
|
# Vendor integrations are loaded conditionally (when the vendor gem
|
|
107
112
|
# is present) by lib/pgbus/engine.rb. Keeping them out of Zeitwerk
|
|
108
113
|
# means we don't reference vendor constants at autoload time.
|
|
@@ -209,6 +214,19 @@ module Pgbus
|
|
|
209
214
|
Streams::Key.stream_key!(key)
|
|
210
215
|
end
|
|
211
216
|
|
|
217
|
+
# The maximum length (in characters) a stream name may be before it
|
|
218
|
+
# overflows the pgbus queue-name budget. A stream name becomes the physical
|
|
219
|
+
# queue `#{queue_prefix}_<name>`, and PGMQ caps queue names, so the usable
|
|
220
|
+
# budget is `QueueNameValidator::MAX_QUEUE_NAME_LENGTH - queue_prefix.length
|
|
221
|
+
# - 1`. Depends on `config.queue_prefix`. Use it to size or truncate stream
|
|
222
|
+
# identifiers up front instead of hand-computing the budget at every call
|
|
223
|
+
# site.
|
|
224
|
+
#
|
|
225
|
+
# name = candidate.length > Pgbus.stream_name_budget ? Pgbus.stream_key(record) : candidate
|
|
226
|
+
def stream_name_budget
|
|
227
|
+
Streams::Key.queue_name_budget
|
|
228
|
+
end
|
|
229
|
+
|
|
212
230
|
# Publish an event to the bus — the top-level shortcut for
|
|
213
231
|
# `Pgbus::EventBus::Publisher.publish`, symmetric with `Pgbus.stream`.
|
|
214
232
|
#
|
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: pgbus
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.
|
|
4
|
+
version: 0.11.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Mikael Henriksson
|
|
@@ -352,6 +352,7 @@ files:
|
|
|
352
352
|
- lib/pgbus/streams/envelope.rb
|
|
353
353
|
- lib/pgbus/streams/filters.rb
|
|
354
354
|
- lib/pgbus/streams/key.rb
|
|
355
|
+
- lib/pgbus/streams/phlex_helpers.rb
|
|
355
356
|
- lib/pgbus/streams/pool_autoscaler.rb
|
|
356
357
|
- lib/pgbus/streams/pool_trigger.rb
|
|
357
358
|
- lib/pgbus/streams/presence.rb
|