pgbus 0.16.7 → 0.17.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 82a6db71dd099a9b4fa24cb0a278184a3aed8bbee7acfb3f3cc6ef6b55064038
-  data.tar.gz: ad7034ee934aee91afdb4a6278200f8178caf0da609d49e57bc6bbcf27d3b7b5
+  metadata.gz: 11b7ad58d91fe1eb9c0388008907523c77324f286f27a0e97a94353e4a8577b0
+  data.tar.gz: '0707593bb7c826f48f9a84db9a666057aaed3407b0d1cb559007f57caf080be2'
 SHA512:
-  metadata.gz: 86977d3f705b8e3732211956582334f6e49a60e426af1082fdf585a175c9a189c9dc9ec83d5ceeaac2f0be7dcee40c1414e55a99225fcc90ac955537d8b4f01c
-  data.tar.gz: 3af2ae55db88774c89f3082b70e5456201c9a85ffd994d43beb197fe0f52d282209fc5de57db66c44de10afe34b37d37fa5404522eabb97d4eb2e91cbfc80da7
+  metadata.gz: 0f1422dea206dbeaac10b1fd7dfa74da2338c007cffb48d7a2b5429bf3a94e6f3e78b5795832d0a2f45346fe4350d4c7355d16b773247420894ea5234c50db85
+  data.tar.gz: 6d30c3583b7e6b5fa9dc89cf42115f57b182dea18b2d363f1ec413d6096ba32b76d75f008dcf7a2f25de3baf9ceba7e17f1edbb382b96e2aeb23bf52a8824af2

data/CHANGELOG.md

@@ -6,6 +6,8 @@
 
 - **Concurrency is now visible: a section on the Locks page, three gauges, a home stat card and an MCP tool.** `limits_concurrency … on_conflict: :block` became durable in #460 — a parked job is never dropped and a key never runs past `to:` — but nothing showed it, so an operator whose per-record pipeline looked "stuck" had no way to see that the key's slot was held, how many jobs were parked behind it, or how long the oldest had waited, short of SQL. `pgbus_semaphores` and `pgbus_blocked_executions` appeared on no page, in no gauge, in no tool. One new read, `Web::DataSource#concurrency_stats`, now feeds four surfaces: the Locks page's **Concurrency** section (parked jobs, oldest wait, slots held, keys at limit, then a row per key with value / limit, lease state, parked count and oldest wait, auto-refreshing like the other dashboard panels); a **Parked jobs** card on the dashboard home; three unlabelled gauges on both `/pgbus/api/metrics` and the AppSignal probe (`pgbus_concurrency_blocked_executions`, `pgbus_concurrency_blocked_oldest_age_seconds`, `pgbus_concurrency_slots_held` — keys are per-record, so a per-key label would be an unbounded series); and a read-only `pgbus_concurrency` MCP tool returning the same data. The key rows come from a FULL OUTER JOIN, not a LEFT one: a key can have parked jobs and no semaphore (its holder died and the sweep removed the row) or a semaphore and nothing parked, and both shapes matter to an operator. Each row carries two guarded escape hatches. **Release** drops the semaphore and promotes through `Concurrency::BlockedExecution.promote_next`, so it takes each slot through the same guarded upsert an enqueue uses and can never reintroduce the over-admission #460 closed; when the lease is still fresh the confirm says so, because a job is then probably still running and releasing lets another start beside it. **Discard parked** drops a single key's parked jobs, naming the exact count in the confirm, and resolves the bookkeeping those jobs will never resolve themselves — a parked batch child is marked failed so its batch stops waiting and `on_failure` fires (#413), and an `:until_executed` uniqueness lock is released rather than orphaned (#423); each discard logs a warning naming the job and emits `pgbus.blocked_execution_discarded`. There is deliberately no cross-key "discard all parked": not losing work is the whole point of `:block`. Nothing is added to the enqueue or execute path. Refs #461.
 
+- **PGMQ: vendored schema v1.13.0, and the upgrade migration now carries upstream's table fixups.** The 1.12.0 → 1.13.0 delta is entirely partitioned-queue work plus one metrics attribute: `pgmq.create_partitioned` gains a `premake` argument (its three-argument form dropped, so pgmq-ruby's three-argument call binds to the new signature with the default), new partitioned queues use `msg_id … GENERATED BY DEFAULT AS IDENTITY` rather than `GENERATED ALWAYS`, and `pgmq.metrics_result` gains `default_partition_length` (NULL for non-partitioned queues, ignored by pgmq-ruby, which reads rows by name). Nothing pgbus calls through `Pgbus::Client` changes, and pgbus creates only partitioned *archives*, whose `msg_id` is a plain column — so no client or pgmq-ruby change. Fresh embedded installs get 1.13.0; existing *embedded* installs: `rake pgbus:pgmq:status`, then `rails generate pgbus:upgrade_pgmq` + `rails db:migrate` (`rails db:migrate:pgbus` on a separate-database install). An `:extension` install is upgraded by PostgreSQL itself (`ALTER EXTENSION pgmq UPDATE`) once the server has the new version packaged — not by this migration, whose function drop refuses to remove extension-owned objects. **New:** because the upgrade works by dropping every pgmq function and composite type and re-running the target schema, it could never carry an upstream change to an existing *table*. The generated migration now reads the version in `pgbus_pgmq_schema_versions` and applies the fixup vendored for each version between it and the target (`lib/pgbus/pgmq_schema/fixups/pgmq_v<VERSION>.sql`, discovered by the same glob as the schema files, via `PgmqSchema.fixups_sql`). This hop's fixup is upstream's own guarded block moving existing partitioned queues to `GENERATED BY DEFAULT`; it selects on `pgmq.meta.is_partitioned`, so it is a no-op unless partitioned queues were created through PGMQ directly. Fixups must be idempotent — an install with no recorded version gets all of them. Closes #459.
+
 - **The batches list and a batch's progress panel now refresh themselves.** Batches was the only dashboard list whose `turbo-frame` was missing `data-auto-refresh`, and the batch detail page had no frame at all, so watching a batch drain meant reloading by hand. Both now poll on the existing `web_refresh_interval` (5s, pauses while you interact) like every other list — the batch progress panel is served by `GET /batches/:id?frame=progress`. This reuses the dashboard's turbo-frame polling rather than the SSE stack in `lib/pgbus/web/stream_app.rb`: that is the host-app-facing stream feature, and pointing it at the dashboard would open a PG `LISTEN` connection per viewer to move four integers.
 
 ### Fixed

data/Rakefile

@@ -16,7 +16,7 @@ require "rubocop/rake_task"
 # on the unresolvable gem inheritance. Passing explicit paths stops the
 # discovery. docs/ lints itself in the Docs site workflow. (Same fix as docs-kit.)
 RuboCop::RakeTask.new do |task|
-  task.patterns = %w[app benchmarks config lib spec Gemfile Rakefile pgbus.gemspec]
+  task.patterns = %w[app benchmarks config lib rakelib spec Gemfile Rakefile pgbus.gemspec]
 end
 
 namespace :bench do
@@ -164,173 +164,8 @@ task :build do
   sh("rm -rf /tmp/gem-verify #{gem_file}")
 end
 
-desc "Release a new version (rake release[1.2.3] or rake release[pre] or rake release[1.2.3,force])"
-task :release, %i[version force] do |_t, args|
-  require_relative "lib/pgbus/version"
-
-  def info(msg)    = puts "\e[34m→\e[0m #{msg}"
-  def success(msg) = puts "\e[32m✓\e[0m #{msg}"
-  def skip(msg)    = puts "\e[33m⊘\e[0m #{msg} \e[33m(skipped)\e[0m"
-  def warn(msg)    = puts "\e[33m⚠\e[0m #{msg}"
-  def error(msg)   = puts "\e[31m✗\e[0m #{msg}"
-  def header(msg)  = puts "\n\e[1;36m#{msg}\e[0m\n#{"─" * msg.length}"
-
-  new_version = args[:version]
-  abort "\e[31mUsage: rake release[X.Y.Z] or rake release[X.Y.Z,force]\e[0m" unless new_version
-
-  force = args[:force]&.to_s&.downcase == "force"
-
-  dirty = `git status --porcelain`.strip
-  abort "\e[31mAborting: working directory is not clean.\e[0m\n#{dirty}" unless dirty.empty?
-
-  current = Pgbus::VERSION
-  prerelease = new_version.match?(/alpha|beta|rc|pre/) || new_version == "pre"
-
-  if new_version == "pre"
-    new_version = current
-    prerelease = true
-  end
-
-  tag = "v#{new_version}"
-  version_file = "lib/pgbus/version.rb"
-
-  title = "Release #{tag}"
-  title += " (force)" if force
-  header title
-  info "Current version: #{current}"
-  info "New version:     #{new_version}"
-  info "Pre-release:     #{prerelease}"
-
-  # Step 0: Force cleanup — delete existing release and tag
-  if force
-    header "Force cleanup"
-    if system("gh release view #{tag} >/dev/null 2>&1")
-      sh("gh release delete #{tag} --yes --cleanup-tag")
-      success "Deleted release and remote tag #{tag}"
-    else
-      skip "No release #{tag} to delete"
-    end
-
-    if system("git rev-parse #{tag} >/dev/null 2>&1")
-      sh("git tag -d #{tag}")
-      success "Deleted local tag #{tag}"
-    else
-      skip "No local tag #{tag} to delete"
-    end
-  end
-
-  # Step 1: Update version file
-  header "Version"
-  if new_version == current
-    skip "Version already #{new_version}"
-  else
-    content = File.read(version_file)
-    content.sub!(/VERSION = ".*"/, "VERSION = \"#{new_version}\"")
-    File.write(version_file, content)
-    success "Updated #{version_file}"
-  end
-
-  # Step 1b: Regenerate the frozen lockfiles that pin the pgbus path gem, so the
-  # bump ships with them in sync. These are installed with `--frozen`/deployment
-  # in CI, so if they still name the OLD version they instant-fail (the root
-  # Gemfile.lock on every main-Gemfile leg AND release.yml's own `bundle install`,
-  # the Rails 7.1 leg with exit 16, and docs-CI on any docs change). Regenerating
-  # here keeps the version-pin drift out of the release commit instead of
-  # surfacing on the next PR — or, worse, in the Release workflow itself.
-  header "Frozen lockfiles"
-  # The ONLY thing a version bump changes in these frozen lockfiles is the pgbus
-  # path-gem pin — so bump exactly that line, in place, with a string edit.
-  #
-  # We deliberately do NOT run `bundle lock` here: a full re-resolve trips over
-  # constraints that have nothing to do with pgbus. Concretely, docs/Gemfile.lock
-  # carries a broad PLATFORMS list (…-gnu / …-musl / arm-linux) for which a
-  # platform gem like `thruster` ships no variant, so `bundle lock` fails with
-  # "Could not find gems matching 'thruster' valid for all resolution platforms"
-  # on any machine whose cache doesn't already hold those exact gems — aborting
-  # the release. `bundle lock --local` was even worse (wrong-file write + no
-  # fetch). A targeted pin edit sidesteps all of it, is deterministic on any
-  # machine, and produces the minimal 2-line diff (the PATH spec + the
-  # DEPENDENCIES pin). See #338/#341 and the surgical-bump fix.
-  frozen_lockfiles = %w[Gemfile.lock gemfiles/rails_7_1.gemfile.lock docs/Gemfile.lock]
-  regenerated_lockfiles = []
-  frozen_lockfiles.each do |lockfile|
-    unless File.exist?(lockfile)
-      skip "#{lockfile} not present"
-      next
-    end
-
-    content = File.read(lockfile)
-    # Matches both the PATH-source spec ("    pgbus (X.Y.Z)") and the
-    # DEPENDENCIES pin ("  pgbus (X.Y.Z)"), leaving everything else untouched.
-    bumped = content.gsub(/^(\s+pgbus) \([^)]*\)$/, "\\1 (#{new_version})")
-
-    if bumped == content
-      skip "#{lockfile} — no pgbus pin to bump"
-      next
-    end
-
-    File.write(lockfile, bumped)
-    regenerated_lockfiles << lockfile
-    success "Bumped pgbus pin in #{lockfile}"
-  end
-
-  # Step 2: Verify gem builds cleanly
-  header "Build verification"
-  sh("gem build pgbus.gemspec --strict")
-  sh("rm -f pgbus-*.gem")
-  success "Gem builds cleanly"
-
-  # Step 3: Commit version bump (+ any re-synced lockfiles)
-  header "Git commit"
-  paths_to_stage = [version_file, *regenerated_lockfiles]
-  staged_changes = paths_to_stage.any? do |path|
-    !`git diff #{path}`.strip.empty? || !`git diff --cached #{path}`.strip.empty?
-  end
-  if staged_changes
-    paths_to_stage.each { |path| sh("git add #{path}") }
-    sh("git commit -m 'chore: bump version to #{new_version}'")
-    success "Committed version bump"
-  else
-    skip "No version change to commit"
-  end
-
-  # Step 4: Push to origin
-  header "Git push"
-  local_sha = `git rev-parse HEAD`.strip
-  remote_sha = `git rev-parse origin/main 2>/dev/null`.strip
-  if local_sha == remote_sha
-    skip "origin/main already at #{local_sha[0..6]}"
-  else
-    sh("git push origin main")
-    success "Pushed to origin/main"
-  end
-
-  # Step 5: Create release
-  header "Release"
-  tag_exists = system("git rev-parse #{tag} >/dev/null 2>&1")
-  release_exists = system("gh release view #{tag} >/dev/null 2>&1")
-
-  if release_exists
-    skip "Release #{tag} already exists (use force to re-create)"
-  elsif tag_exists
-    info "Tag #{tag} exists, creating release from it"
-    pre_flag = prerelease ? "--prerelease" : ""
-    sh("gh release create #{tag} --generate-notes #{pre_flag}".strip)
-    success "Release #{tag} created from existing tag"
-  else
-    pre_flag = prerelease ? "--prerelease" : ""
-    sh("gh release create #{tag} --generate-notes --target main #{pre_flag}".strip)
-    success "Release #{tag} created"
-  end
-
-  puts ""
-  success "\e[1mRelease #{tag} complete!\e[0m CI will handle the rest:"
-  puts "    • Run tests"
-  puts "    • Build + verify gem"
-  puts "    • Sign with Sigstore"
-  puts "    • Publish to RubyGems"
-  puts "    • Upload assets to the release"
-end
+# `rake release[X.Y.Z]` lives in rakelib/release.rake (shared across the
+# zoolutions gems); `bin/release` is its interactive front door.
 
 namespace :frontend do
   # app/frontend/pgbus/style.css is a committed build artifact — the engine ships

data/lib/generators/pgbus/templates/upgrade_pgmq.rb.erb

@@ -8,6 +8,16 @@ class UpgradePgmqToV<%= target_version_slug.camelize %> < ActiveRecord::Migratio
     # This uses the vendored SQL which doesn't require the pgmq extension.
     execute Pgbus::PgmqSchema.install_sql("<%= target_version %>")
 
+    # Apply the table fixups upstream ships for every hop between the
+    # version this database records and the target. Dropping and re-creating
+    # the functions cannot carry a change to an existing TABLE (1.13.0 moves
+    # partitioned queues' msg_id to GENERATED BY DEFAULT), so those ride
+    # here. Runs after install_sql because the fixups call
+    # pgmq.format_table_name, which the drop step removed. Each fixup is
+    # idempotent, so an install with no recorded version gets them all.
+    fixups = Pgbus::PgmqSchema.fixups_sql(after: installed_pgmq_version, upto: "<%= target_version %>")
+    execute fixups unless fixups.empty?
+
     # Re-install the per-queue NOTIFY insert triggers the function drop
     # cascaded away (pgmq.notify_insert_throttle records which queues had
     # them and at what throttle; enable_notify_insert is idempotent).
@@ -34,4 +44,16 @@ class UpgradePgmqToV<%= target_version_slug.camelize %> < ActiveRecord::Migratio
       "PGMQ schema downgrade is not supported. " \
       "Restore from a database backup if needed."
   end
+
+  private
+
+  # The version this database last recorded, or nil when it has never been
+  # tracked (a pre-tracking install, or one bootstrapped by hand).
+  def installed_pgmq_version
+    return nil unless connection.table_exists?("pgbus_pgmq_schema_versions")
+
+    connection.select_value(
+      "SELECT version FROM pgbus_pgmq_schema_versions ORDER BY installed_at DESC LIMIT 1"
+    )
+  end
 end

data/lib/pgbus/pgmq_schema/fixups/pgmq_v1.13.0.sql

@@ -0,0 +1,49 @@
+-- PGMQ 1.13.0 table fixup.
+--
+-- Adapted from the upstream migration
+-- pgmq-extension/sql/pgmq--1.12.0--1.13.0.sql at tag v1.13.0
+-- (blob a380ee56e856c5799dcd1486342ce910b96da64e), which is the only part of
+-- that file pgbus's upgrade cannot already reproduce: dropping every function
+-- and composite type and re-running the target schema recreates functions and
+-- types, but never touches an existing table.
+--
+-- One deliberate deviation from upstream: the guard resolves the queue table by
+-- OID via to_regclass instead of matching information_schema.columns.table_name
+-- against the formatted name. See the comment on the guard for why.
+--
+-- Every file in this directory MUST be idempotent and self-guarded. The
+-- upgrade migration applies each fixup between the recorded installed version
+-- and the target, and an install whose version was never recorded gets all of
+-- them, so a fixup can be replayed against a database that already has it.
+
+-- Partitioned queues: msg_id becomes GENERATED BY DEFAULT AS IDENTITY.
+-- pg_partman's partition_data_* tooling moves rows out of the default
+-- partition by inserting them with their existing msg_id, which
+-- GENERATED ALWAYS refuses. pgmq's own writers never supply msg_id, so this
+-- changes nothing for them.
+DO $$
+DECLARE
+    queue_record RECORD;
+    qtable TEXT;
+BEGIN
+    FOR queue_record IN SELECT queue_name FROM pgmq.meta WHERE is_partitioned LOOP
+        qtable := pgmq.format_table_name(queue_record.queue_name, 'q');
+
+        -- Resolved by OID rather than by comparing qtable to a catalog name:
+        -- validate_queue_name caps the queue at 47 *characters*, but PostgreSQL
+        -- truncates identifiers at 63 *bytes*, so a multibyte name can produce a
+        -- qtable string that never equals the stored relation name. A name
+        -- comparison would silently skip such a queue and leave msg_id at
+        -- ALWAYS, which is the exact failure this fixup exists to prevent.
+        IF EXISTS (
+            SELECT 1
+            FROM pg_catalog.pg_attribute
+            WHERE attrelid = pg_catalog.to_regclass(FORMAT('pgmq.%I', qtable))
+            AND attname = 'msg_id'
+            AND attidentity = 'a'
+        ) THEN
+            EXECUTE FORMAT('ALTER TABLE pgmq.%I ALTER COLUMN msg_id SET GENERATED BY DEFAULT', qtable);
+        END IF;
+    END LOOP;
+END;
+$$;