pgbus 0.16.7 → 0.17.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 82a6db71dd099a9b4fa24cb0a278184a3aed8bbee7acfb3f3cc6ef6b55064038
-  data.tar.gz: ad7034ee934aee91afdb4a6278200f8178caf0da609d49e57bc6bbcf27d3b7b5
+  metadata.gz: 7b7955f6b4ccdd4d3e4f8d8e372e9283ca0b7f801d7a6909d29f03fde3425ce7
+  data.tar.gz: f26d692d0e9b979e9c4b77b3e28cafdab4d3c50e32292e793e1885d94f0d5943
 SHA512:
-  metadata.gz: 86977d3f705b8e3732211956582334f6e49a60e426af1082fdf585a175c9a189c9dc9ec83d5ceeaac2f0be7dcee40c1414e55a99225fcc90ac955537d8b4f01c
-  data.tar.gz: 3af2ae55db88774c89f3082b70e5456201c9a85ffd994d43beb197fe0f52d282209fc5de57db66c44de10afe34b37d37fa5404522eabb97d4eb2e91cbfc80da7
+  metadata.gz: 30c539f2494acaeffdd10301bba047d6dca86ab2bf3ae776ac6d8d3239f4b5e5a05d530135932cf15122b37dd02acd50de86026315365ea10c085e0100201ebf
+  data.tar.gz: 596d817366be305b4abd3c9f1975e6e004ef2448976f58bfb36c9c9b1cd15d0abbc830a98bfd9900d5fc2cfc07b2f6257c1f158b7f8b09a653e97e55c590cc92

data/CHANGELOG.md

@@ -2,12 +2,26 @@
 
 ### Added
 
+- **`rake bench:worker_profile` measures the event consumer too.** A consumer cell drives a real `Process::Consumer` over an event-bus topic queue (plain handler, so the transport is measured, not the dedup table) with the same harness as the worker cell. `WP_BENCH_ROLE` (`worker`, `consumer`, `both`) and `WP_BENCH_READ_AHEAD` (passed as `read_ahead:`) are new knobs. Refs #486.
+
 - **`coalesce:` now works on the `Turbo::StreamsChannel` broadcast path.** `Pgbus::Streams::Stream#broadcast` has accepted `coalesce:` since #171, but the Turbo patch forwarded only four options (`durable:`, `exclude:`, `visible_to:`, `event:`), so nothing that broadcasts through `Turbo::StreamsChannel.broadcast_*_to` — every `Turbo::Broadcastable` model, and phlex-reactive's `Streamable.broadcast_to` — could debounce. A background job settling a 177-row collection therefore pushed 177 idempotent replaces of the same count badge at every peer. Any targeted helper now takes `coalesce:` (`@order.broadcast_replace_to :account, coalesce: 100`), as does a direct channel call (`Turbo::StreamsChannel.broadcast_replace_to(@account, target: "order-count", coalesce: true)`), with the same semantics as the direct API: opt-in, per `(stream, target)`, last-write-wins, `true` = the 50ms default window, an integer = the window in ms. Two things the option's plumbing needed beyond a fifth thread-local: the coalescing key, which dedupes on `(stream, target)` while `broadcast_stream_to` only ever sees the rendered `content:` — it is now resolved from the frame's own `target:`/`targets:` through the same `ActionView::RecordIdentifier.dom_id` conversion Turbo applies, so a record target keys on `order_7` rather than on a per-instance `#to_s`; and extraction of the pgbus options at the channel itself, since a direct `Turbo::StreamsChannel` call never passes through `Turbo::Broadcastable` and so previously leaked `durable: true` (and now `coalesce:`) into Turbo's renderer. `broadcast_refresh_to` and `broadcast_render_to` carry no target and so cannot coalesce — they raise the existing actionable "requires target:" error rather than guessing a key — and the `_later_to` variants remain out of scope, as the thread-local cannot follow into a job. The wire is byte-identical when `coalesce:` is absent. Closes #465.
 
 - **Concurrency is now visible: a section on the Locks page, three gauges, a home stat card and an MCP tool.** `limits_concurrency … on_conflict: :block` became durable in #460 — a parked job is never dropped and a key never runs past `to:` — but nothing showed it, so an operator whose per-record pipeline looked "stuck" had no way to see that the key's slot was held, how many jobs were parked behind it, or how long the oldest had waited, short of SQL. `pgbus_semaphores` and `pgbus_blocked_executions` appeared on no page, in no gauge, in no tool. One new read, `Web::DataSource#concurrency_stats`, now feeds four surfaces: the Locks page's **Concurrency** section (parked jobs, oldest wait, slots held, keys at limit, then a row per key with value / limit, lease state, parked count and oldest wait, auto-refreshing like the other dashboard panels); a **Parked jobs** card on the dashboard home; three unlabelled gauges on both `/pgbus/api/metrics` and the AppSignal probe (`pgbus_concurrency_blocked_executions`, `pgbus_concurrency_blocked_oldest_age_seconds`, `pgbus_concurrency_slots_held` — keys are per-record, so a per-key label would be an unbounded series); and a read-only `pgbus_concurrency` MCP tool returning the same data. The key rows come from a FULL OUTER JOIN, not a LEFT one: a key can have parked jobs and no semaphore (its holder died and the sweep removed the row) or a semaphore and nothing parked, and both shapes matter to an operator. Each row carries two guarded escape hatches. **Release** drops the semaphore and promotes through `Concurrency::BlockedExecution.promote_next`, so it takes each slot through the same guarded upsert an enqueue uses and can never reintroduce the over-admission #460 closed; when the lease is still fresh the confirm says so, because a job is then probably still running and releasing lets another start beside it. **Discard parked** drops a single key's parked jobs, naming the exact count in the confirm, and resolves the bookkeeping those jobs will never resolve themselves — a parked batch child is marked failed so its batch stops waiting and `on_failure` fires (#413), and an `:until_executed` uniqueness lock is released rather than orphaned (#423); each discard logs a warning naming the job and emits `pgbus.blocked_execution_discarded`. There is deliberately no cross-key "discard all parked": not losing work is the whole point of `:block`. Nothing is added to the enqueue or execute path. Refs #461.
 
+- **PGMQ: vendored schema v1.13.0, and the upgrade migration now carries upstream's table fixups.** The 1.12.0 → 1.13.0 delta is entirely partitioned-queue work plus one metrics attribute: `pgmq.create_partitioned` gains a `premake` argument (its three-argument form dropped, so pgmq-ruby's three-argument call binds to the new signature with the default), new partitioned queues use `msg_id … GENERATED BY DEFAULT AS IDENTITY` rather than `GENERATED ALWAYS`, and `pgmq.metrics_result` gains `default_partition_length` (NULL for non-partitioned queues, ignored by pgmq-ruby, which reads rows by name). Nothing pgbus calls through `Pgbus::Client` changes, and pgbus creates only partitioned *archives*, whose `msg_id` is a plain column — so no client or pgmq-ruby change. Fresh embedded installs get 1.13.0; existing *embedded* installs: `rake pgbus:pgmq:status`, then `rails generate pgbus:upgrade_pgmq` + `rails db:migrate` (`rails db:migrate:pgbus` on a separate-database install). An `:extension` install is upgraded by PostgreSQL itself (`ALTER EXTENSION pgmq UPDATE`) once the server has the new version packaged — not by this migration, whose function drop refuses to remove extension-owned objects. **New:** because the upgrade works by dropping every pgmq function and composite type and re-running the target schema, it could never carry an upstream change to an existing *table*. The generated migration now reads the version in `pgbus_pgmq_schema_versions` and applies the fixup vendored for each version between it and the target (`lib/pgbus/pgmq_schema/fixups/pgmq_v<VERSION>.sql`, discovered by the same glob as the schema files, via `PgmqSchema.fixups_sql`). This hop's fixup is upstream's own guarded block moving existing partitioned queues to `GENERATED BY DEFAULT`; it selects on `pgmq.meta.is_partitioned`, so it is a no-op unless partitioned queues were created through PGMQ directly. Fixups must be idempotent — an install with no recorded version gets all of them. Closes #459.
+
 - **The batches list and a batch's progress panel now refresh themselves.** Batches was the only dashboard list whose `turbo-frame` was missing `data-auto-refresh`, and the batch detail page had no frame at all, so watching a batch drain meant reloading by hand. Both now poll on the existing `web_refresh_interval` (5s, pauses while you interact) like every other list — the batch progress panel is served by `GET /batches/:id?frame=progress`. This reuses the dashboard's turbo-frame polling rather than the SSE stack in `lib/pgbus/web/stream_app.rb`: that is the host-app-facing stream feature, and pointing it at the dashboard would open a PG `LISTEN` connection per viewer to move four integers.
 
+- **`pgbus doctor` reports the Ruby JIT, and the boot banner names it.** A new "Ruby JIT" check is `:ok` when YJIT (or ZJIT) is on, when the Ruby has no JIT, or in a local (development/test) environment, and warns otherwise — never fails, never enables anything; turning YJIT on stays with Rails' `config.yjit`. The supervisor's first boot line now ends `jit=yjit|zjit|none`. Measured reason: with Postgres close by, a worker is CPU-bound under the GVL and YJIT is worth ~14–20 % jobs/s (`docs/performance.md`). Refs #484.
+
+- **`rake bench:worker_profile`: jobs/s and where a job's time goes, for a real worker.** Drives a real `Pgbus::Process::Worker` against real PGMQ and splits each thread's time (Postgres wait, pool-checkout wait, waiting for work, GVL wait, and whose code was on-CPU) with vernier, locally and optionally behind toxiproxy-injected latency. `vernier` joins the test group; it is not a runtime dependency. The same change repairs `benchmarks/integration_bench.rb`, which no longer ran against 1.0's `ensures_uniqueness`. Refs #484.
+
+### Changed
+
+- **perf(worker, consumer): opt-in read-ahead, so one read feeds many jobs when Postgres is on another host.** A worker read `qty = free threads`; once its pool was busy, threads freed one at a time, so it made one read round trip per job. Under +1 ms of latency that capped a 12-thread worker at ~1.3–1.7 k no-op jobs/s however many threads it had (#484), and the event consumer's loop had the same shape. New `config.read_ahead` (default **0**, off; overridable per capsule and per `event_consumers` entry with `read_ahead:`) lets either loop claim up to N messages beyond its free threads. Claimed messages wait in a new `Process::ClaimBuffer`, kept invisible by a new `VisibilityHeartbeat.hold`, until a thread frees. They are handed back to the queue (`set_vt` 0) the moment the worker drains, recycles, pauses or stops, or the consumer shuts down or recycles, with `read_ct` already counted. Buffered claims count as in flight, so `prefetch_limit` still caps everything claimed. Measured with `rake bench:worker_profile` (12 threads, no-op jobs, +1 ms per round trip, rounds 2–3 of three interleaved rounds on a loaded machine): **worker 1 286–1 731 → 2 300–2 462 jobs/s, consumer 1 326–1 779 → 2 288–2 599** at `read_ahead` 12. Locally both are within noise, because there the worker is CPU-bound under the GVL; `read_ahead` 0 overlaps the baseline's run-to-run range in 7 of 8 cells; consumer / proxied / no JIT sits 2–18 % below, plausibly load noise but unproven. A returned message's next claimant logs it as a zombie redelivery when `zombie_detection` is on, and `validate!` warns when `read_ahead > 0` meets `visibility_heartbeat = false`, since nothing then keeps a buffered message invisible. With read-ahead on, the next ceiling under latency is the connection pool (checkout wait 2–3 % → 14–15 %), so raise `pool_size` with it. `VisibilityHeartbeat`'s unregister now drops only the entry it registered, so a stale hold released after a job's own tracking took over cannot drop that job's heartbeat. Worker heartbeat metadata gains `"buffered"`. Refs #486.
+
+- **perf(executor): a successful first delivery no longer deletes from `pgbus_failed_events`.** Every successful job ran a `DELETE … WHERE queue_name = $1 AND msg_id = $2` through ActiveRecord, but only a failed attempt ever writes that table, so a job on its first delivery cannot have a row. The DELETE now runs only on redelivery (`read_ct > 1`), still clearing the row a failed attempt left. Measured with `rake bench:worker_profile` (12 threads, no-op jobs, local Postgres): **4 569 → 5 823 jobs/s with YJIT (+27 %), 3 797 → 5 119 without (+35 %)**; under +1 ms injected latency the change is within noise, because there the worker's single reader is the ceiling. This is a per-job query removed, not faster job code. The executor's debug tag is also built only when debug logging is on (9 → 8 pgbus-owned objects per job, now a hard budget). `Client#drop_queue` now also deletes the dropped queue's `pgbus_failed_events` rows: PGMQ restarts `msg_id`s when a queue is recreated, and a surviving row would point the dashboard's retry/discard at an unrelated new message — previously a first-delivery success happened to sweep such a row, and that sweep is gone. Refs #484.
+
 ### Fixed
 
 - **A still-running event handler is no longer re-run as if it had crashed, and a slow one is no longer redelivered mid-run.** Two halves of one hole, follow-up to #469. First: `Process::Consumer` had no equivalent of `ActiveJob::Executor#with_visibility_heartbeat`, so an event message's visibility timeout was never re-armed — a handler slower than `visibility_timeout` (30s by default) was redelivered *while it was still running*, `read_ct` climbed on every redelivery, and after `max_retries` the event was dead-lettered without the handler ever raising. Workers have been protected from this since the heartbeat landed; event consumers never were. The consumer now tracks each message through `VisibilityHeartbeat` for exactly as long as its handlers run, releasing the entry before the archive so a beat can never re-arm a message that is already gone. Second: `Handler#claim_idempotency?` read a `pgbus_processed_events` row with `completed_at IS NULL` as proof that the previous holder had been killed mid-handler, and re-ran. That state equally describes a handler that is simply still running — on another thread, another fork or another host — so a second delivery (a genuine redelivery, or a duplicate envelope, whose independent visibility timeouts no heartbeat can serialize) executed the handler *concurrently with* the live one, which is precisely the double-execution `idempotent!` exists to prevent. The claim now carries liveness rather than only a claim instant: `EventBus::ClaimBeat` refreshes every in-flight claim's `processed_at` from the same beat that re-arms the message's visibility timeout, so message visibility and claim liveness go quiet together when a process dies. A pending claim silent for longer than twice the heartbeat interval is abandoned and re-runs, exactly as before; a fresher one is owned and the delivery skips, deferring to the holder — which either completes (nothing is lost) or fails, leaving its own message for visibility-timeout redelivery to recover. A skip is also no longer silent: `pgbus.event_skipped` carries the reason (`:completed`, `:cached` or `:owned`), the claim's age in seconds and the delivery's `read_ct`, and the metrics subscriber counts it as `pgbus_event_count` with `status: "skipped"` and the reason as a tag. No migration, no new column, and no change on a table that has not run `pgbus:add_processed_event_completion` — a single-phase claim has no pending state and therefore no ownership question. Closes #470.

data/README.md

@@ -26,6 +26,7 @@ PostgreSQL-native job processing and event bus for Rails, built on [PGMQ](https:
   - [Client-level circuit breaker (database-down)](#client-level-circuit-breaker-database-down)
   - [Read timeouts (libpq-native)](#read-timeouts-libpq-native)
   - [Prefetch flow control](#prefetch-flow-control)
+  - [Read-ahead](#read-ahead)
   - [Worker recycling](#worker-recycling)
   - [Retry backoff](#retry-backoff)
 - [Routing and ordering](#routing-and-ordering)
@@ -541,6 +542,24 @@ end
 
 The worker tracks in-flight messages with an atomic counter and only fetches `min(idle_threads, prefetch_available)` messages per cycle. The counter is decremented in an `ensure` block so it never gets stuck.
 
+### Read-ahead
+
+By default a worker reads as many messages as it has free threads. Once the pool is busy, threads free one at a time, so the worker ends up making one read round trip per job. That costs nothing when Postgres is on the same host. When Postgres is on another host, that round trip becomes the throughput ceiling. `read_ahead` lets a worker or event consumer claim up to N messages beyond its free threads and hold them until a thread frees:
+
+```ruby
+Pgbus.configure do |config|
+  config.read_ahead = 12                                   # global; 0 = off (default)
+  config.capsule :api, queues: %w[api], threads: 12, read_ahead: 12   # per capsule
+  config.event_consumers = [{ topics: ["orders.#"], threads: 8, read_ahead: 8 }]
+end
+```
+
+- **When to turn it on:** when Postgres is not on the same host as the worker. Start with `read_ahead` equal to `threads`. With a local database it buys little, because the worker is CPU-bound there. Under +1 ms of latency, `read_ahead` 12 took a 12-thread worker from ~1.3–1.7k to ~2.3–2.5k no-op jobs/s (`docs/performance.md`, "Read-ahead"). The next ceiling is the connection pool, so raise `pool_size` with it.
+- **Visibility:** buffered messages are kept invisible by the visibility heartbeat until their job starts, so a long wait never lets another worker claim them. With `visibility_heartbeat = false` nothing holds them, and a message buffered past `visibility_timeout` can run twice; `validate!` warns about that combination.
+- **Shutdown and recycle:** when a worker drains, recycles, pauses or shuts down, buffered messages go straight back to the queue (`set_vt` 0), not after `visibility_timeout`. Their `read_ct` has already been incremented, the same as after a crash, so with `zombie_detection` on, the worker that picks one up logs it as a zombie redelivery.
+- **With `prefetch_limit`:** buffered messages count as in flight, so `prefetch_limit` still caps everything a worker has claimed.
+- **Multi-queue workers:** the plain multi-queue read (no priority, fair share or group mode) can claim and discard rows past its limit. A larger read makes that worse; see `docs/performance.md`.
+
 ### Worker recycling
 
 Pgbus workers recycle themselves to prevent memory bloat. This is the main reliability difference vs. solid_queue, which leaves workers alive forever.
@@ -2031,11 +2050,12 @@ pgbus help      # Show help
 
 #### pgbus doctor
 
-A single preflight command that answers "is this environment healthy enough to run?" — useful as a deploy or CI gate. It runs 11 checks and never raises; a broken environment turns every probe into a failed/warned check instead of a crash:
+A single preflight command that answers "is this environment healthy enough to run?" — useful as a deploy or CI gate. It runs 12 checks and never raises; a broken environment turns every probe into a failed/warned check instead of a crash:
 
 | Check | Fails (`:fail`) when | Warns (`:warn`) when |
 |---|---|---|
 | Configuration | `Configuration#validate!` raises | — |
+| Ruby JIT | — | YJIT is available but off outside a local environment (a CPU-bound worker with Postgres nearby runs ~15% fewer jobs/s without it; no measurable effect when network-bound); reports only, never enables |
 | Database | Unreachable (`SELECT 1` via `Client#ping`) | — |
 | PGMQ schema | Schema not installed | Installed but untracked, or behind the vendored version |
 | Queues | A configured queue has no PGMQ table | — |

data/Rakefile

@@ -16,7 +16,7 @@ require "rubocop/rake_task"
 # on the unresolvable gem inheritance. Passing explicit paths stops the
 # discovery. docs/ lints itself in the Docs site workflow. (Same fix as docs-kit.)
 RuboCop::RakeTask.new do |task|
-  task.patterns = %w[app benchmarks config lib spec Gemfile Rakefile pgbus.gemspec]
+  task.patterns = %w[app benchmarks config lib rakelib spec Gemfile Rakefile pgbus.gemspec]
 end
 
 namespace :bench do
@@ -25,7 +25,8 @@ namespace :bench do
   # no-DB unit suite that bench:all runs in CI.
   db_benches = %w[connection_pool_bench integration_bench streams_bench streams_read_pool_bench
                   execution_modes_bench pool_swap_bench pool_autoscale_bench job_burst_bench
-                  notify_wake_bench notify_chaos_bench streams_hub_bench fair_read_bench].freeze
+                  notify_wake_bench notify_chaos_bench streams_hub_bench fair_read_bench
+                  worker_profile_bench].freeze
   # The unit suite is every *_bench.rb that doesn't need a database, derived
   # from the directory so a new unit bench is picked up automatically (kept in
   # sync with bench:one, which globs the same files).
@@ -60,6 +61,11 @@ namespace :bench do
     ruby "benchmarks/integration_bench.rb"
   end
 
+  desc "Run worker profiling bench: jobs/s and where a job's time goes (requires PGBUS_DATABASE_URL)"
+  task :worker_profile do
+    ruby "benchmarks/worker_profile_bench.rb"
+  end
+
   desc "Run fair share read benchmark (requires PGBUS_DATABASE_URL)"
   task :fair_read do
     ruby "benchmarks/fair_read_bench.rb"
@@ -164,173 +170,8 @@ task :build do
   sh("rm -rf /tmp/gem-verify #{gem_file}")
 end
 
-desc "Release a new version (rake release[1.2.3] or rake release[pre] or rake release[1.2.3,force])"
-task :release, %i[version force] do |_t, args|
-  require_relative "lib/pgbus/version"
-
-  def info(msg)    = puts "\e[34m→\e[0m #{msg}"
-  def success(msg) = puts "\e[32m✓\e[0m #{msg}"
-  def skip(msg)    = puts "\e[33m⊘\e[0m #{msg} \e[33m(skipped)\e[0m"
-  def warn(msg)    = puts "\e[33m⚠\e[0m #{msg}"
-  def error(msg)   = puts "\e[31m✗\e[0m #{msg}"
-  def header(msg)  = puts "\n\e[1;36m#{msg}\e[0m\n#{"─" * msg.length}"
-
-  new_version = args[:version]
-  abort "\e[31mUsage: rake release[X.Y.Z] or rake release[X.Y.Z,force]\e[0m" unless new_version
-
-  force = args[:force]&.to_s&.downcase == "force"
-
-  dirty = `git status --porcelain`.strip
-  abort "\e[31mAborting: working directory is not clean.\e[0m\n#{dirty}" unless dirty.empty?
-
-  current = Pgbus::VERSION
-  prerelease = new_version.match?(/alpha|beta|rc|pre/) || new_version == "pre"
-
-  if new_version == "pre"
-    new_version = current
-    prerelease = true
-  end
-
-  tag = "v#{new_version}"
-  version_file = "lib/pgbus/version.rb"
-
-  title = "Release #{tag}"
-  title += " (force)" if force
-  header title
-  info "Current version: #{current}"
-  info "New version:     #{new_version}"
-  info "Pre-release:     #{prerelease}"
-
-  # Step 0: Force cleanup — delete existing release and tag
-  if force
-    header "Force cleanup"
-    if system("gh release view #{tag} >/dev/null 2>&1")
-      sh("gh release delete #{tag} --yes --cleanup-tag")
-      success "Deleted release and remote tag #{tag}"
-    else
-      skip "No release #{tag} to delete"
-    end
-
-    if system("git rev-parse #{tag} >/dev/null 2>&1")
-      sh("git tag -d #{tag}")
-      success "Deleted local tag #{tag}"
-    else
-      skip "No local tag #{tag} to delete"
-    end
-  end
-
-  # Step 1: Update version file
-  header "Version"
-  if new_version == current
-    skip "Version already #{new_version}"
-  else
-    content = File.read(version_file)
-    content.sub!(/VERSION = ".*"/, "VERSION = \"#{new_version}\"")
-    File.write(version_file, content)
-    success "Updated #{version_file}"
-  end
-
-  # Step 1b: Regenerate the frozen lockfiles that pin the pgbus path gem, so the
-  # bump ships with them in sync. These are installed with `--frozen`/deployment
-  # in CI, so if they still name the OLD version they instant-fail (the root
-  # Gemfile.lock on every main-Gemfile leg AND release.yml's own `bundle install`,
-  # the Rails 7.1 leg with exit 16, and docs-CI on any docs change). Regenerating
-  # here keeps the version-pin drift out of the release commit instead of
-  # surfacing on the next PR — or, worse, in the Release workflow itself.
-  header "Frozen lockfiles"
-  # The ONLY thing a version bump changes in these frozen lockfiles is the pgbus
-  # path-gem pin — so bump exactly that line, in place, with a string edit.
-  #
-  # We deliberately do NOT run `bundle lock` here: a full re-resolve trips over
-  # constraints that have nothing to do with pgbus. Concretely, docs/Gemfile.lock
-  # carries a broad PLATFORMS list (…-gnu / …-musl / arm-linux) for which a
-  # platform gem like `thruster` ships no variant, so `bundle lock` fails with
-  # "Could not find gems matching 'thruster' valid for all resolution platforms"
-  # on any machine whose cache doesn't already hold those exact gems — aborting
-  # the release. `bundle lock --local` was even worse (wrong-file write + no
-  # fetch). A targeted pin edit sidesteps all of it, is deterministic on any
-  # machine, and produces the minimal 2-line diff (the PATH spec + the
-  # DEPENDENCIES pin). See #338/#341 and the surgical-bump fix.
-  frozen_lockfiles = %w[Gemfile.lock gemfiles/rails_7_1.gemfile.lock docs/Gemfile.lock]
-  regenerated_lockfiles = []
-  frozen_lockfiles.each do |lockfile|
-    unless File.exist?(lockfile)
-      skip "#{lockfile} not present"
-      next
-    end
-
-    content = File.read(lockfile)
-    # Matches both the PATH-source spec ("    pgbus (X.Y.Z)") and the
-    # DEPENDENCIES pin ("  pgbus (X.Y.Z)"), leaving everything else untouched.
-    bumped = content.gsub(/^(\s+pgbus) \([^)]*\)$/, "\\1 (#{new_version})")
-
-    if bumped == content
-      skip "#{lockfile} — no pgbus pin to bump"
-      next
-    end
-
-    File.write(lockfile, bumped)
-    regenerated_lockfiles << lockfile
-    success "Bumped pgbus pin in #{lockfile}"
-  end
-
-  # Step 2: Verify gem builds cleanly
-  header "Build verification"
-  sh("gem build pgbus.gemspec --strict")
-  sh("rm -f pgbus-*.gem")
-  success "Gem builds cleanly"
-
-  # Step 3: Commit version bump (+ any re-synced lockfiles)
-  header "Git commit"
-  paths_to_stage = [version_file, *regenerated_lockfiles]
-  staged_changes = paths_to_stage.any? do |path|
-    !`git diff #{path}`.strip.empty? || !`git diff --cached #{path}`.strip.empty?
-  end
-  if staged_changes
-    paths_to_stage.each { |path| sh("git add #{path}") }
-    sh("git commit -m 'chore: bump version to #{new_version}'")
-    success "Committed version bump"
-  else
-    skip "No version change to commit"
-  end
-
-  # Step 4: Push to origin
-  header "Git push"
-  local_sha = `git rev-parse HEAD`.strip
-  remote_sha = `git rev-parse origin/main 2>/dev/null`.strip
-  if local_sha == remote_sha
-    skip "origin/main already at #{local_sha[0..6]}"
-  else
-    sh("git push origin main")
-    success "Pushed to origin/main"
-  end
-
-  # Step 5: Create release
-  header "Release"
-  tag_exists = system("git rev-parse #{tag} >/dev/null 2>&1")
-  release_exists = system("gh release view #{tag} >/dev/null 2>&1")
-
-  if release_exists
-    skip "Release #{tag} already exists (use force to re-create)"
-  elsif tag_exists
-    info "Tag #{tag} exists, creating release from it"
-    pre_flag = prerelease ? "--prerelease" : ""
-    sh("gh release create #{tag} --generate-notes #{pre_flag}".strip)
-    success "Release #{tag} created from existing tag"
-  else
-    pre_flag = prerelease ? "--prerelease" : ""
-    sh("gh release create #{tag} --generate-notes --target main #{pre_flag}".strip)
-    success "Release #{tag} created"
-  end
-
-  puts ""
-  success "\e[1mRelease #{tag} complete!\e[0m CI will handle the rest:"
-  puts "    • Run tests"
-  puts "    • Build + verify gem"
-  puts "    • Sign with Sigstore"
-  puts "    • Publish to RubyGems"
-  puts "    • Upload assets to the release"
-end
+# `rake release[X.Y.Z]` lives in rakelib/release.rake (shared across the
+# zoolutions gems); `bin/release` is its interactive front door.
 
 namespace :frontend do
   # app/frontend/pgbus/style.css is a committed build artifact — the engine ships

data/lib/generators/pgbus/templates/upgrade_pgmq.rb.erb

@@ -8,6 +8,16 @@ class UpgradePgmqToV<%= target_version_slug.camelize %> < ActiveRecord::Migratio
     # This uses the vendored SQL which doesn't require the pgmq extension.
     execute Pgbus::PgmqSchema.install_sql("<%= target_version %>")
 
+    # Apply the table fixups upstream ships for every hop between the
+    # version this database records and the target. Dropping and re-creating
+    # the functions cannot carry a change to an existing TABLE (1.13.0 moves
+    # partitioned queues' msg_id to GENERATED BY DEFAULT), so those ride
+    # here. Runs after install_sql because the fixups call
+    # pgmq.format_table_name, which the drop step removed. Each fixup is
+    # idempotent, so an install with no recorded version gets them all.
+    fixups = Pgbus::PgmqSchema.fixups_sql(after: installed_pgmq_version, upto: "<%= target_version %>")
+    execute fixups unless fixups.empty?
+
     # Re-install the per-queue NOTIFY insert triggers the function drop
     # cascaded away (pgmq.notify_insert_throttle records which queues had
     # them and at what throttle; enable_notify_insert is idempotent).
@@ -34,4 +44,16 @@ class UpgradePgmqToV<%= target_version_slug.camelize %> < ActiveRecord::Migratio
       "PGMQ schema downgrade is not supported. " \
       "Restore from a database backup if needed."
   end
+
+  private
+
+  # The version this database last recorded, or nil when it has never been
+  # tracked (a pre-tracking install, or one bootstrapped by hand).
+  def installed_pgmq_version
+    return nil unless connection.table_exists?("pgbus_pgmq_schema_versions")
+
+    connection.select_value(
+      "SELECT version FROM pgbus_pgmq_schema_versions ORDER BY installed_at DESC LIMIT 1"
+    )
+  end
 end

data/lib/pgbus/active_job/executor.rb

@@ -19,8 +19,7 @@ module Pgbus
 
       def execute(message, queue_name, source_queue: nil)
         execution_start = monotonic_now
-        tag = "msg_id=#{message.msg_id} queue=#{queue_name} read_ct=#{message.read_ct}"
-        Pgbus.logger.debug { "[Pgbus::Executor] start #{tag}" }
+        Pgbus.logger.debug { "[Pgbus::Executor] start #{log_tag(message, queue_name)}" }
 
         payload = JSON.parse(message.message)
         job_class = payload["job_class"]
@@ -42,7 +41,7 @@ module Pgbus
             read_ct: read_count,
             msg_id: message.msg_id.to_i
           )
-          Pgbus.logger.debug { "[Pgbus::Executor] dead_lettered #{tag} job_class=#{job_class}" }
+          Pgbus.logger.debug { "[Pgbus::Executor] dead_lettered #{log_tag(message, queue_name)} job_class=#{job_class}" }
           return :dead_lettered
         end
         uniqueness_key = Uniqueness.extract_key(payload)
@@ -70,7 +69,7 @@ module Pgbus
           end
         end
 
-        Pgbus.logger.debug { "[Pgbus::Executor] deserialized #{tag} job_class=#{job_class}" }
+        Pgbus.logger.debug { "[Pgbus::Executor] deserialized #{log_tag(message, queue_name)} job_class=#{job_class}" }
         job_succeeded = false
         retried = false
 
@@ -95,13 +94,13 @@ module Pgbus
           # job data, so hand it to the job before perform — that is what makes
           # `batch` (and `batch.enqueue` for open batches) work inside a job.
           assign_batch_id(job, payload)
-          Pgbus.logger.debug { "[Pgbus::Executor] running #{tag} job_class=#{job_class}" }
+          Pgbus.logger.debug { "[Pgbus::Executor] running #{log_tag(message, queue_name)} job_class=#{job_class}" }
           with_visibility_heartbeat(job, queue_name, msg_id, source_queue, payload) { execute_job(job) }
           # retry_on re-enqueues from inside perform_now and returns normally:
           # this attempt is done (archive it) but the job is not — the retry
           # message carries the batch tag and signals on its own outcome.
           retried = Batch.retry_reenqueued?(payload["job_id"])
-          Pgbus.logger.debug { "[Pgbus::Executor] perform_returned #{tag} job_class=#{job_class}" }
+          Pgbus.logger.debug { "[Pgbus::Executor] perform_returned #{log_tag(message, queue_name)} job_class=#{job_class}" }
           # Archiving is the exact-once claim on this execution.
           # :already_archived means another worker archived the message (our
           # heartbeat lapsed and it was redelivered): that worker owns the
@@ -109,20 +108,24 @@ module Pgbus
           # concurrency slot a second time (rails/solid_queue#761).
           if archive_from(queue_name, msg_id, source_queue: source_queue) == :already_archived
             Pgbus.logger.warn do
-              "[Pgbus::Executor] already archived elsewhere, skipping signals #{tag} job_class=#{job_class}"
+              "[Pgbus::Executor] already archived elsewhere, skipping signals #{log_tag(message, queue_name)} job_class=#{job_class}"
             end
             release_duplicate_execution_lock(uniqueness_key, uniqueness_strategy, queue_name, msg_id)
             return :duplicate
           end
-          Pgbus.logger.debug { "[Pgbus::Executor] archived #{tag} job_class=#{job_class}" }
+          Pgbus.logger.debug { "[Pgbus::Executor] archived #{log_tag(message, queue_name)} job_class=#{job_class}" }
           job_succeeded = true
           release_uniqueness_lock(uniqueness_key)
-          FailedEventRecorder.clear!(queue_name: queue_name, msg_id: msg_id)
+          # Only FailedEventRecorder.record! writes this table, and only after a
+          # failed attempt, so a first delivery has no row: skip the per-job
+          # DELETE round trip (issue #484). A stale row from a dropped-and-
+          # recreated queue reusing this msg_id is the dispatcher's to sweep.
+          FailedEventRecorder.clear!(queue_name: queue_name, msg_id: msg_id) if read_count > 1
         end
 
         instrument("pgbus.job_completed", queue: queue_name, job_class: job_class)
         record_stat(payload, queue_name, "success", execution_start, message: message)
-        Pgbus.logger.debug { "[Pgbus::Executor] done #{tag} job_class=#{job_class}" }
+        Pgbus.logger.debug { "[Pgbus::Executor] done #{log_tag(message, queue_name)} job_class=#{job_class}" }
         :success
       rescue *FATAL_EXCEPTIONS
         # Process-fatal: propagate so the supervisor/OS can react.
@@ -150,7 +153,7 @@ module Pgbus
           exception_object: e
         )
         record_stat(payload, queue_name, "failed", execution_start, message: message)
-        Pgbus.logger.debug { "[Pgbus::Executor] failed #{tag} job_class=#{payload&.dig("job_class")} error=#{e.class}" }
+        Pgbus.logger.debug { "[Pgbus::Executor] failed #{log_tag(message, queue_name)} job_class=#{payload&.dig("job_class")} error=#{e.class}" }
         # Don't signal concurrency on transient failure — the job will be retried.
         # Semaphore is released only on success or dead-lettering.
         :failed
@@ -170,6 +173,12 @@ module Pgbus
 
       private
 
+      # Built inside the logger blocks so a job pays for the string only when
+      # debug logging is on (issue #484).
+      def log_tag(message, queue_name)
+        "msg_id=#{message.msg_id} queue=#{queue_name} read_ct=#{message.read_ct}"
+      end
+
       def assign_batch_id(job, payload)
         batch_id = payload[Batch::METADATA_KEY]
         return unless batch_id && job.respond_to?(:batch_id=)

data/lib/pgbus/client.rb

@@ -699,6 +699,9 @@ module Pgbus
         synchronized { @pgmq.drop_queue(name) }
       end
       @queues_created.delete(name)
+      # Failed-event rows are keyed by the name the executor saw (physical or
+      # logical); a recreated queue reuses msg_ids, so neither may survive.
+      FailedEventRecorder.clear_queue!([name, name.delete_prefix("#{config.queue_prefix}_")].uniq)
       result
     end
 

data/lib/pgbus/configuration.rb

@@ -12,6 +12,12 @@ module Pgbus
 
     # Worker settings
     attr_accessor :polling_interval, :prefetch_limit, :execution_mode
+    # read_ahead (issue #486): how many messages a worker or event consumer
+    # claims beyond its free threads and holds, heartbeated, until a thread
+    # frees. Under database latency one read then feeds many jobs instead of
+    # one. 0 (default) is off. Overridable per capsule / event_consumers entry
+    # (`read_ahead:`); prefetch_limit still caps everything claimed.
+    attr_reader :read_ahead
     # visibility_heartbeat / visibility_heartbeat_interval: while a job runs,
     # its message's visibility timeout is re-armed every interval seconds
     # (default: a third of visibility_timeout), so a job that outlives the
@@ -277,6 +283,7 @@ module Pgbus
       @visibility_heartbeat_interval = nil
 
       @prefetch_limit = nil
+      @read_ahead = 0
       @execution_mode = :threads
 
       @max_jobs_per_worker = nil
@@ -573,6 +580,16 @@ module Pgbus
       (0...priority_levels).map { |p| priority_queue_name(name, p) }
     end
 
+    def read_ahead=(value)
+      @read_ahead = value.nil? ? 0 : value
+    end
+
+    # Read-ahead for one capsule or event_consumers entry, falling back to
+    # the global read_ahead (mirrors execution_mode_for).
+    def read_ahead_for(entry)
+      entry.fetch(:read_ahead, nil) || read_ahead
+    end
+
     # Returns the execution mode for a specific worker config hash,
     # falling back to the global execution_mode setting.
     def execution_mode_for(worker_config)
@@ -828,6 +845,8 @@ module Pgbus
               "prefetch_limit must be > 0"
       end
 
+      validate_read_ahead!
+
       if priority_levels && !(priority_levels.is_a?(Integer) && priority_levels >= 1 && priority_levels <= 10)
         raise Pgbus::ConfigurationError, "priority_levels must be an integer between 1 and 10"
       end
@@ -846,6 +865,33 @@ module Pgbus
       self
     end
 
+    # Global, per-capsule and per-consumer read_ahead: a non-negative Integer.
+    def validate_read_ahead!
+      settings = [["read_ahead", read_ahead]]
+      Array(workers).each { |w| settings << ["worker read_ahead", w[:read_ahead]] unless w[:read_ahead].nil? }
+      Array(event_consumers).each { |c| settings << ["event consumer read_ahead", c[:read_ahead]] unless c[:read_ahead].nil? }
+
+      settings.each do |name, value|
+        next if value.is_a?(Integer) && !value.negative?
+
+        raise Pgbus::ConfigurationError, "#{name} must be a non-negative Integer (got #{value.inspect})"
+      end
+      warn_read_ahead_without_heartbeat(settings)
+    end
+
+    # Buffered claims are kept invisible only by the visibility heartbeat.
+    # Without it a message buffered longer than visibility_timeout can be
+    # claimed by another worker while it still waits here. Legal, but warned.
+    def warn_read_ahead_without_heartbeat(settings)
+      return if visibility_heartbeat
+      return unless settings.any? { |_name, value| value.positive? }
+
+      Pgbus.logger.warn do
+        "[Pgbus] read_ahead is on but visibility_heartbeat is false — a message buffered longer than " \
+          "visibility_timeout (#{visibility_timeout}s) can be claimed and run by another worker"
+      end
+    end
+
     # An explicit shutdown_timeout must be a positive number; nil keeps the
     # derived drain_timeout + margin default. A value below drain_timeout is
     # legal but self-defeating (the supervisor SIGKILLs workers mid-drain), so

data/lib/pgbus/doctor.rb

@@ -5,7 +5,7 @@ require "pgbus/mcp/health_analyzer"
 
 module Pgbus
   # Preflight diagnostics for a pgbus deployment — the single command that
-  # answers "is this environment healthy enough to run?". Runs eleven checks and
+  # answers "is this environment healthy enough to run?". Runs twelve checks and
   # returns a machine-readable result plus a human report, so `pgbus doctor`
   # and `rake pgbus:doctor` can gate a deploy or CI run (exit 0 on success,
   # 1 on any failure).
@@ -32,6 +32,7 @@ module Pgbus
     # stable identity of each check (used in the report and to select subsets).
     CHECKS = {
       "Configuration" => :check_configuration,
+      "Ruby JIT" => :check_ruby_jit,
       "Database" => :check_database,
       "PGMQ schema" => :check_pgmq_schema,
       "Queues" => :check_queues,
@@ -166,6 +167,33 @@ module Pgbus
       Check.new(name: "Configuration", status: :fail, detail: "#{e.class}: #{e.message}")
     end
 
+    # 1b. Ruby JIT (issue #484). With Postgres close by, a worker is GVL-bound
+    # and YJIT is worth ~15-20% jobs/s (docs/performance.md). Rails enables it via
+    # config.yjit (load_defaults 7.2+, or 8.0+ outside local envs). Reports
+    # only — enabling a JIT is the application's decision, never pgbus's.
+    def check_ruby_jit
+      label = RubyJit.label
+      return Check.new(name: "Ruby JIT", status: :ok, detail: "#{label.upcase} enabled") unless label == "none"
+      return Check.new(name: "Ruby JIT", status: :ok, detail: "no YJIT available on this Ruby") unless RubyJit.yjit_available?
+      return Check.new(name: "Ruby JIT", status: :ok, detail: "YJIT off in a local environment (expected)") if local_env?
+
+      Check.new(name: "Ruby JIT", status: :warn,
+                detail: "YJIT is available but off — a CPU-bound worker (Postgres nearby) runs ~15% fewer jobs/s " \
+                        "without it; no measurable effect when network-bound. " \
+                        "Enable it with config.yjit = true (or load_defaults 7.2+) or RUBY_YJIT_ENABLE=1")
+    rescue StandardError => e
+      Check.new(name: "Ruby JIT", status: :warn, detail: "could not determine (#{e.class}: #{e.message})")
+    end
+
+    # A Rails app outside development/test. Without Rails the environment is
+    # unknown, so assume local and stay quiet rather than warn on a guess.
+    def local_env?
+      return true unless defined?(Rails) && Rails.respond_to?(:env) && Rails.env
+
+      env = Rails.env
+      env.respond_to?(:local?) ? env.local? : %w[development test].include?(env.to_s)
+    end
+
     # 2. Database connectivity — SELECT 1 via Client#ping.
     def check_database
       @client.ping

data/lib/pgbus/failed_event_recorder.rb

@@ -47,6 +47,23 @@ module Pgbus
         false
       end
 
+      # Drops every row recorded under any of `queue_names`. Called when a
+      # queue is dropped: PGMQ restarts msg_ids on recreate, so a surviving
+      # row would aim the dashboard's retry/discard at an unrelated message.
+      # Queue names are validated word characters, so the array literal is safe.
+      def clear_queue!(queue_names)
+        connection.exec_delete(
+          "DELETE FROM pgbus_failed_events WHERE queue_name = ANY($1::text[])",
+          "FailedEvent Clear Queue",
+          ["{#{queue_names.join(",")}}"]
+        )
+      rescue StandardError => e
+        Pgbus.logger.error do
+          "[Pgbus] Failed to clear failed events for dropped queue(s) #{queue_names.join(", ")}: " \
+            "#{e.class}: #{e.message}"
+        end
+      end
+
       def clear!(queue_name:, msg_id:)
         connection.exec_delete(
           "DELETE FROM pgbus_failed_events WHERE queue_name = $1 AND msg_id = $2",