pgbus 0.13.8 → 0.13.9

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 403234bf91b0ae06bd9c1331b6356450c98cc7a062a7ebcfb45f159950d07c2c
-  data.tar.gz: a314687f0071c9588433dcd452c146178cc1afc52a2e46a675e998004bdbef2f
+  metadata.gz: 03f5f0c1044bc23fb429b3e24e13e3df5bfe6bffe5452ea3c746b24de081f822
+  data.tar.gz: bf1c7f3f71811469fef78fb8d93a66a77668229e0293d2e3442f62abedf6badf
 SHA512:
-  metadata.gz: 80ce61932ee3f9c34562c353219ae5c19a2a1b312c8418e6ed645e32c66ac2e7f590edd7b9cc861e4f13b4bd784a7f2712d4b45b11f0a0c49b30fa117a664501
-  data.tar.gz: 91856c566af986f8de528c40e036a00548adf0d7bb0d6e1fc96abb887120613df0b9f8fb22a04292bf4bb794baf06e6a7668b823b05fed730d52b420ff5cfc9b
+  metadata.gz: 0f82fba75de8a30b79887b4535cf47865d624126dc7f81cdbba190490be2b57ad1ea69c5d6a1d8ea902b4d48bb85c7b19920326d18cbc9c411201f30ac5ee571
+  data.tar.gz: '039f1c2bcc8e4c65188f03794f42ca5e31aebc67ed4a9de869ed885306e7052ea370af331e57f9f104ca9ae65ae51ef4caf690516993966c92a4d117c91f7bd0'

data/CHANGELOG.md

@@ -2,6 +2,10 @@
 
 ### Fixed
 
+- **Concurrent queue creation no longer kills the loser with `PG::UniqueViolation` on `pg_class` (issue #404).** The sibling of the #403 trigger race, one DDL step earlier: `CREATE TABLE IF NOT EXISTS` is not race-safe — two backends creating a not-yet-existing queue both pass the existence check (READ COMMITTED; neither sees the other's uncommitted catalog rows), both insert into `pg_class`, and the loser raises `unique_violation` on `pg_class_relname_nsp_index` instead of the friendly `duplicate_table`. All the dedup guards (`@queues_created`, `Stream#@ensured`, `synchronized`) are process-local, so with lazily-created per-record durable stream queues two processes first-touching the same brand-new stream hit `pgmq.create` simultaneously — and via `send_message`'s `ensure_queue` the same window applies to plain job queues on cold post-deploy herds (production: AppSignal incident #963, an unrescued `PG::UniqueViolation` out of an unrelated `after_commit`). Three DDL steps now rescue the duplicate as success, mirroring the #397/#403 shape: **(1)** `create_queue_table` — `pgmq.create` is a single statement and therefore atomic, so when the loser unblocks the winner has committed the whole queue; the loser re-checks `pgmq.meta` and returns, retrying `pgmq.create` once when the re-check can't confirm (e.g. a leftover physical table without a meta row) with a second failure propagating (the retry's own error — the original duplicate is preserved deeper in its cause chain via Ruby's implicit `$!` chaining); **(2)** `create_fifo_index_if_needed` under group_mode — the duplicate proves the index exists; **(3)** the stream archive `msg_id` index in `ensure_stream_queue` — a raw `CREATE INDEX IF NOT EXISTS` with the same catalog race, where the loser now proceeds to register and memoize the stream instead of dropping the first broadcast. Duplicate detection reuses `DUPLICATE_INSTALL_ERROR_CLASSES`, matched directly (raw `conn.exec` paths) or as the `cause` of pgmq-ruby's `ConnectionError` wrapper. No locking added, no happy-path cost. Refs #404.
+
+- **Concurrent lazy stream-queue ensures no longer drop the loser's broadcast with `PG::DuplicateObject` on the NOTIFY trigger (issue #403).** `Client#enable_notify_if_needed` is check-then-act — `notify_trigger_current?` is a plain SELECT and the surrounding `synchronized` is a process-local mutex — so after a deploy, when every process's `@queues_created` memo is cold, two processes handling the first broadcast for the same stream both see "trigger not current" and both run PGMQ's `DROP TRIGGER IF EXISTS` + `CREATE CONSTRAINT TRIGGER` cycle. The loser's CREATE blocks on the winner's table lock and fails with `duplicate_object` once the winner commits — surfaced as `PGMQ::Errors::ConnectionError`, killing that broadcast (in production: a deploy-time thundering herd turning unrelated `after_commit` writes into 500s, AppSignal incident #546). The existing mitigations all miss this window: the 0.12.x `notify_trigger_current?` check made repeat ensures cheap but left the TOCTOU race, the #397 advisory lock serializes schema install only, and the supervisor's pre-fork bootstrap covers job-container children but not web processes. The loser now treats the duplicate as success — the trigger provably exists because a concurrent caller just created it (same shape as the #397 duplicate-object rescue on schema install; matched on the trigger identifier, which survives message localization, plus the `PG::DuplicateObject` cause or the "already exists" text). Before returning, it re-checks `notify_trigger_current?`: racing ensures pass the same throttle so the check normally confirms and returns, but a job-queue ensure (250ms) racing the stream override (0ms) can leave the wrong interval installed — that mismatch retries `enable_notify_insert` once to converge; a second loss propagates. No locking added, no happy-path cost. Refs #403.
+
 - **`StreamQueue.record!` no longer depends on Rails' pool schema-cache index resolution — one bad probe stopped poisoning stream registration for the process lifetime (issue #401).** `record!` used `upsert(unique_by: :queue_name)`, which resolves the unique index through the connection pool's schema cache. That cache stores a negative `data_source_exists?` answer permanently, and `SchemaCache#indexes` returns `[]` (uncached) whenever the cached probe says false — while the `table_exists?` guard at the top of `record!` is a live query. So a single wrong first probe on the pool cache (observed under PgBouncer transaction pooling in production, and from a coalescer flush thread racing foreground test DB work in CI) made the guard pass and the upsert raise `ArgumentError: No unique index found for queue_name` — swallowed at DEBUG — on **every** subsequent `record!` in that process until restart, leaving streams unregistered from that process's perspective (maintenance, orphan sweep, and wildcard classification degrade). The registry write is now a raw `INSERT … ON CONFLICT (queue_name) DO NOTHING` on the model's connection: the unique index is owned by the gem's own migration, so there is nothing for Rails to resolve, no schema-cache traffic leaves the hot first-broadcast path, and a poisoned cache can no longer break registration (pinned by an integration regression spec that deliberately poisons the pool cache, plus a `sql.active_record` assertion that no `SCHEMA` query is issued once the `table_exists?` memo is warm). Failure logging is now class-aware: a transient database error (`ActiveRecord::ActiveRecordError`) still logs at DEBUG per attempt, but a non-database failure — the bug-signal class the old `ArgumentError` belonged to — logs at WARN once per process (DEBUG thereafter) instead of drowning a process-lifetime malfunction in per-broadcast DEBUG spam. Return values and `backfill!`/`all_names` cache semantics are unchanged. Refs #401.
 
 - **The schema-install transaction framing no longer commits or destroys a caller's open transaction (#398 review follow-up).** The #397 fix wrapped check+install in `BEGIN`…`COMMIT`/`ROLLBACK` unconditionally. On the Proc-supplied shared-connection path (the Rails lambda), the connection can arrive **mid-transaction** — e.g. `perform_later` inside an application `transaction do` block — where `BEGIN` is a warning-level no-op and the matching `COMMIT`/`ROLLBACK` then commits half of, or destroys, the *caller's* transaction. The framing is now `transaction_status`-aware: an idle connection gets the owned `BEGIN`…`COMMIT` as before; a connection already inside a transaction rides it via `SAVEPOINT pgbus_pgmq_install` / `RELEASE` (`ROLLBACK TO SAVEPOINT` on failure), so the caller's transaction is never touched. On the savepoint path the advisory lock joins the caller's transaction and is held until it ends — over-holding only delays a concurrent installer, never corrupts it — and `@schema_ensured` is NOT cached there: the install is only durable once the caller commits, so a cached true after an outer rollback would skip every future check against a missing schema. The same durability rule now governs `@queues_created`: queue DDL on the shared Proc-supplied connection joins the caller's open transaction, so queue creation there runs uncached (idempotent `CREATE IF NOT EXISTS`) and the next ensure re-checks — a cache write outliving a caller rollback would make later message operations fail against a missing queue. All shared-connection access in these paths — including the transaction-status probe and the schema install itself — holds the per-instance connection mutex, restoring the single-owner invariant the #397 fix had narrowed. Refs #398.

data/lib/pgbus/client/ensure_stream_queue.rb

@@ -46,10 +46,19 @@ module Pgbus
           ON pgmq.a_#{sanitized} (msg_id)
         SQL
 
-        synchronized do
-          with_raw_connection do |conn|
-            conn.exec(sql)
+        begin
+          synchronized do
+            with_raw_connection do |conn|
+              conn.exec(sql)
+            end
           end
+        rescue StandardError => e
+          # CREATE INDEX IF NOT EXISTS shares pgmq.create's catalog race
+          # (issue #404): two first-broadcasts both pass the existence
+          # check and the loser gets a raw unique_violation on pg_class.
+          # The duplicate proves the index exists — carry on to register
+          # and memoize.
+          raise unless duplicate_relation_error?(e)
         end
 
         # Record the physical queue name so maintenance (stream-archive prune,

data/lib/pgbus/client.rb

@@ -64,6 +64,12 @@ module Pgbus
     # re-running the trigger DDL on every queue.
     NOTIFY_THROTTLE_MS = 250
 
+    # PGMQ's per-queue NOTIFY trigger name, as created by
+    # pgmq.enable_notify_insert — used to recognize the duplicate-trigger
+    # race loser (issue #403).
+    NOTIFY_TRIGGER_NAME = "trigger_notify_queue_insert_listeners"
+    private_constant :NOTIFY_TRIGGER_NAME
+
     # Load the pgmq-ruby gem, defining the PGMQ module before requiring it so
     # Zeitwerk's eager_load (called inside pgmq.rb) can resolve the constant.
     # Without the pre-definition, Ruby 4.0 + Zeitwerk 2.7.5 raises NameError
@@ -1177,7 +1183,29 @@ module Pgbus
     end
 
     # Runs inside synchronized — callers own the connection mutex.
+    #
+    # CREATE TABLE IF NOT EXISTS is not race-safe: two backends creating a
+    # not-yet-existing queue both pass the existence check (READ COMMITTED
+    # — neither sees the other's uncommitted catalog rows), both insert
+    # into pg_class, and the loser raises unique_violation on
+    # pg_class_relname_nsp_index instead of the friendly duplicate_table
+    # (issue #404 — the sibling of the #403 trigger race, one DDL step
+    # earlier; `synchronized` is process-local, so nothing serializes
+    # this across processes). pgmq.create is a single statement and
+    # therefore atomic: by the time the loser unblocks, the winner has
+    # committed the WHOLE queue — tables, indexes, and the pgmq.meta row
+    # — so re-check meta and return (the winner also ran autovacuum
+    # tuning). The retry covers the can't-confirm case (e.g. a leftover
+    # physical table without a meta row); its failure propagates as the
+    # retry's own error, with the original duplicate preserved deeper in
+    # the cause chain (raised while $! held it, so Ruby chains it).
     def create_queue_table(name)
+      @pgmq.create(name)
+      tune_autovacuum(name)
+    rescue StandardError => e
+      raise unless duplicate_relation_error?(e)
+      return if queue_registered?(name)
+
       @pgmq.create(name)
       tune_autovacuum(name)
     end
@@ -1203,17 +1231,77 @@ module Pgbus
       end
     end
 
+    # notify_trigger_current? is a plain SELECT and `synchronized` is a
+    # process-local mutex, so this check-then-act races across processes:
+    # after a deploy every process's @queues_created memo is cold, and two
+    # web processes handling the first broadcast for the same lazy stream
+    # queue both see "trigger not current" and both run PGMQ's
+    # DROP + CREATE CONSTRAINT TRIGGER cycle. The loser's CREATE blocks on
+    # the winner's table lock and fails with PG::DuplicateObject once the
+    # winner commits (issue #403). The duplicate proves the trigger exists,
+    # so treat it as success — same shape as the #397 duplicate-object
+    # rescue on schema install. Re-check the throttle first: racing ensures
+    # pass the same value, but a job-queue ensure (250ms) can race the
+    # stream override (0ms), so a mismatch means the winner installed a
+    # different interval — retry once to converge; a second loss propagates.
     def enable_notify_if_needed(full_name, throttle_ms)
       return unless config.listen_notify
       return if notify_trigger_current?(full_name, throttle_ms)
 
+      @pgmq.enable_notify_insert(full_name, throttle_interval_ms: throttle_ms)
+    rescue PGMQ::Errors::ConnectionError => e
+      raise unless duplicate_notify_trigger_error?(e)
+      return if notify_trigger_current?(full_name, throttle_ms)
+
       @pgmq.enable_notify_insert(full_name, throttle_interval_ms: throttle_ms)
     end
 
+    # Matched on the trigger name (an identifier — survives server-side
+    # message localization) plus either the PG::DuplicateObject cause set by
+    # pgmq-ruby's `raise … ConnectionError` inside `rescue PG::Error` (the
+    # defined? guard mirrors the other PG::… checks in this file: a cause
+    # can only be a PG::DuplicateObject when the class is loaded) or the
+    # English "already exists" text when a wrapper dropped the cause.
+    def duplicate_notify_trigger_error?(error)
+      message = error.message.to_s
+      return false unless message.include?(NOTIFY_TRIGGER_NAME)
+
+      (defined?(PG::DuplicateObject) && error.cause.is_a?(PG::DuplicateObject)) ||
+        message.include?("already exists")
+    end
+
     def create_fifo_index_if_needed(full_name)
       return unless config.group_mode
 
       @pgmq.create_fifo_index(full_name)
+    rescue StandardError => e
+      # CREATE INDEX IF NOT EXISTS has the same catalog race as
+      # pgmq.create (issue #404): the loser's duplicate proves a
+      # concurrent ensure created the index.
+      raise unless duplicate_relation_error?(e)
+    end
+
+    # A relation-creation race loser's error: one of the duplicate DDL
+    # classes directly (raw conn.exec paths), or wrapped — pgmq-ruby
+    # raises ConnectionError inside `rescue PG::Error`, so Ruby sets the
+    # duplicate as its cause automatically.
+    def duplicate_relation_error?(error)
+      duplicate_install_error?(error) || duplicate_install_error?(error.cause)
+    end
+
+    # Whether pgmq.meta records the queue — the authoritative "create
+    # committed" signal (pgmq.create writes it atomically with the
+    # tables). The pooled checkout is a sequential sibling of the failed
+    # create's (already returned when the exception unwound), so there is
+    # no nested checkout — same reasoning as notify_trigger_current?.
+    def queue_registered?(full_name)
+      @pgmq.with_connection do |conn|
+        conn.exec_params("SELECT 1 FROM pgmq.meta WHERE queue_name = $1 LIMIT 1", [full_name]).ntuples.positive?
+      end
+    rescue StandardError
+      # Can't confirm (aborted caller transaction, schema not ready) —
+      # fall through to the retry, which surfaces the state honestly.
+      false
     end
 
     # Check whether the NOTIFY trigger already exists on this queue with the
@@ -1238,7 +1326,7 @@ module Pgbus
           JOIN pg_namespace n ON c.relnamespace = n.oid
           WHERE n.nspname = 'pgmq'
             AND c.relname = pgmq.format_table_name($1, 'q')
-            AND t.tgname = 'trigger_notify_queue_insert_listeners'
+            AND t.tgname = '#{NOTIFY_TRIGGER_NAME}'
             AND EXISTS (
               SELECT 1 FROM pgmq.notify_insert_throttle
               WHERE queue_name = $1

data/lib/pgbus/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Pgbus
-  VERSION = "0.13.8"
+  VERSION = "0.13.9"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: pgbus
 version: !ruby/object:Gem::Version
-  version: 0.13.8
+  version: 0.13.9
 platform: ruby
 authors:
 - Mikael Henriksson