pgbus 0.19.1 → 0.19.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: fc9755d4591c2de1b1a8ba1d5f6e4c20073bdac018049f31572338e354a8f6bb
-  data.tar.gz: ee79c33310552873c05ceb8c2c133079aafb1c07a8b8fb1074ac79d76cafe68f
+  metadata.gz: cb2ec8b622a127040f7ec0d98d514397b66fe2bda51d910bdf7109aee1cb9201
+  data.tar.gz: be53e0ecc3dc3700cd8daa769cf067190244dc4b957d1c8e022adb2515f695df
 SHA512:
-  metadata.gz: a5b4f622fb2503d67dd38cdf7b6175fd8437a82c07d536d4274a8a57fec877125e08e97583763817a33b3727f148262fdc3c0eeb9ee91d3230ce7a04057ee3a6
-  data.tar.gz: 819e82ba9a43259e8c1be15f835e6c23922fd530d7cc793dcda401a9f93bc1c592eb0f910449b244218bf6c0a0c20275090381fe0f1f8b052b4e6bfbc1321f38
+  metadata.gz: 65ad9a9f4bf9a3bdb9263a0603178370595b2b552336c4a74dacfb5d5575cf14f75ec1e59c2dbcec3c148bd0d5e6a8782fbf47bca89e5d0eb68ba5504b74743c
+  data.tar.gz: a3389974026f5626913e1038c39cfb099f8f982dcd5df968ad7ba09e0b6da747a3e6a41b4823ea843415558d3bd576db7922836e3556c43c94ae56c48f2ff529

data/CHANGELOG.md

@@ -3,6 +3,12 @@
 ### Added
 
 - **Deploy tasks: one-time work owed by a release, run once the fleet has rolled over.** Declare a backfill, data fix or cache warm in `config/deploy_tasks.yml` (the shape of `recurring.yml` minus `schedule:`, plus `version:` and `every_deploy:`) or `config.deploy_tasks`; the scheduler enqueues it once per `(key, version)` (or once per revision for an `every_deploy:` task) after every live pgbus process reports the new app revision and a worker on it is alive, so it never runs on old code during the overlap window and two schedulers never both fire it. It is not exactly-once: a failure between the enqueue and recording the run can repeat a task after a scheduler restart, and a scheduler killed between claiming and enqueueing loses that arm, so deploy tasks must be idempotent. Every pgbus process stamps the revision (`config.app_revision`, `PGBUS_REVISION`, `DASH_VERSION`, `KAMAL_VERSION` or the `REVISION` file) into its heartbeat. The dashboard gets a "Deploy tasks" page with the rollover status, Run now and Enable/Disable; run history lives in `pgbus_deploy_task_runs` (`deploy_task_run_retention`, default 90 days); `pgbus.deploy_task.enqueue` is instrumented and the MCP server gains `pgbus_deploy_tasks`. Set up with `rails generate pgbus:add_deploy_tasks` (`pgbus:update` detects the missing tables). Refs #536.
+- **`pgbus.event_dead_lettered`: the event consumer says when it dead-letters an event.** A job that exhausted `max_retries` emitted `pgbus.job_dead_lettered`; an event moved to `<queue>_dlq` the same way emitted nothing, so a pager, an error tracker or a metrics counter had no push signal for dead events. `Process::Consumer` now emits `pgbus.event_dead_lettered` once per dead-lettered event, after `move_to_dead_letter` returns and the failed-event row is cleared, with `queue_name` (logical), `routing_key`, `handler` (the class that owns the queue in this process, or nil), `event_id`, `read_ct`, `msg_id` and `error` (the class of the last recorded failure, or nil). `handler` is nil unless exactly one subscriber owns the queue. A body that is not a JSON object (unparseable text, or a jsonb scalar or array written straight to the queue) is still dead-lettered, with `routing_key`, `handler` and `event_id` nil, and its `error` is the consumer's own failure to read it (`JSON::ParserError`, `NoMethodError` or `TypeError`): no handler ran. A move that raises emits nothing. A subscriber that raises is logged at debug level and does not change the outcome: the message stays dead-lettered, with one `dead_lettered` stat, no `failed` stat and no circuit-breaker failure. The AppSignal and metrics subscribers count it as `pgbus_event_count` with `status: "dead_lettered"`, tagged `handler` and `routing_key`. `pgbus.job_dead_lettered` and the consumer's success path are unchanged: the envelope is parsed a second time only on the dead-letter branch. Refs #543.
+- **feat(batches): `Pgbus.batch do |b| … end`, a builder for starting a batch.** `b.on_complete`, `b.on_success` and `b.on_failure` take an ActiveJob class or a configured instance; `b.add(*jobs)` takes job instances (each keeps its own `.set(wait:, queue:, priority:)`) and sends them in one bulk enqueue. The batch row exists, and a `uniqueness_key` is held, before the block runs, and the batch stays pending until the block returns, so the calls can come in any order and a job that finishes while the block is still adding cannot finish the batch early. Every `perform_later` inside the block joins the batch, as inside `Batch#enqueue`. With `on_conflict: :discard` or `:log` and the key held, the block does not run. Inside a job, `batch(description: …) { |b| … }` opens a new, independent batch; `batch` without a block still reads the job's own batch. A member job adds siblings with `batch.add(NextJob.new(id))`, the open-batch form of `batch.enqueue { … }`. `Pgbus::Batch.start` is the same as `Pgbus.batch`, and `Pgbus::Batch.new` + `#enqueue` stay as the lower-level API. Refs #538.
+
+### Changed
+
+- **The batch completion callback is `on_complete`.** `Pgbus::Batch.new(on_complete:)`, the builder's `b.on_complete` and the `#on_complete` reader replace `on_finish`, which keeps working as a deprecated alias until 1.0 and logs one warning per process. The database columns stay `on_finish_class` / `on_finish_job` until the 1.0 schema cleanup, so batches created before upgrading still fire, and the `pgbus.batch_finished` instrumentation event keeps its name. The dashboard's batch page labels the callback "On Complete". Refs #538.
 
 ## [0.19.0] - 2026-10-16
 

data/README.md

@@ -801,28 +801,27 @@ How Pgbus integrates with your application's transactions and tracks groups of r
 Coordinate groups of jobs with callbacks when all complete:
 
 ```ruby
-batch = Pgbus::Batch.new(
-  on_finish: BatchFinishedJob,
-  on_success: BatchSucceededJob,
-  on_discard: BatchFailedJob,
-  description: "Import users",
-  properties: { initiated_by: current_user.id }
-)
-
-batch.enqueue do
-  users.each { |user| ImportUserJob.perform_later(user.id) }
+batch = Pgbus.batch(description: "Import users", properties: { initiated_by: current_user.id }) do |b|
+  b.on_complete BatchFinishedJob
+  b.on_success  BatchSucceededJob
+  b.on_failure  BatchFailedJob
+  b.add(*users.map { |user| ImportUserJob.new(user.id) })
 end
 ```
 
+`b.add` takes job instances (each keeps its own `.set(wait:, queue:)`) and sends them in one bulk enqueue. The batch row exists before the block runs and stays pending until it returns, so a job that finishes while the block is still adding cannot finish the batch early. Set callbacks first: if the block raises after an `add`, the added jobs still run and the batch finishes later with only the callbacks set by then. A block that goes `config.batch_stall_threshold` (default 5 minutes) without adding a job is started by the stalled-batch sweep, so keep slow work outside it. Callbacks are job classes or configured instances, never blocks: they are stored on the batch row.
+
+Inside a job, `batch { |b| ... }` opens a new batch the same way, and a job that is already a member adds siblings with `batch.add(NextJob.new(id))`. `Pgbus::Batch.new(on_complete: ...)` plus `batch.enqueue { ... }` remains the lower-level form.
+
 #### Callbacks
 
 | Callback | Fired when |
 |----------|------------|
-| `on_finish` | All jobs completed (success or discard) |
-| `on_success` | All jobs completed successfully (zero discarded) |
-| `on_discard` | At least one job was dead-lettered |
+| `on_complete` | All jobs completed (success or failure). `on_finish` is a deprecated alias until 1.0 |
+| `on_success` | All jobs completed successfully (zero failed) |
+| `on_failure` | At least one job was dead-lettered. `on_discard` is a deprecated alias until 1.0 |
 
-Callback jobs receive the batch `properties` hash as their argument:
+Callback jobs given as a bare class receive the batch `properties` hash as their argument:
 
 ```ruby
 class BatchFinishedJob < ApplicationJob
@@ -838,23 +837,18 @@ end
 `ensures_uniqueness` protects one job. A multi-job run — a nightly import that fans out into hundreds of chunk jobs, a pipeline whose jobs enqueue the next stage — has no single job that lives for the whole run, so a lock on the entry job releases the moment that job finishes and the next scheduler tick starts a second run on top of the first. Give the batch the lock instead:
 
 ```ruby
-batch = Pgbus::Batch.new(
-  uniqueness_key: "perfecta-daily",
-  on_conflict: :discard,           # :reject (raise AlreadyRunning), :discard, :log
-  on_success: ExportPricesJob
-)
-
-batch.enqueue do
-  ImportChunkJob.perform_later("products", 0)
+Pgbus.batch(uniqueness_key: "perfecta-daily", on_conflict: :discard) do |b|   # :reject (raise AlreadyRunning), :discard, :log
+  b.on_success ExportPricesJob
+  b.add(ImportChunkJob.new("products", 0))
 end
 ```
 
-The lock is taken at `enqueue` and held until the batch finishes — through every open-batch `batch.enqueue` add from inside a running job — then released in the same single-winner finish that fires the callbacks, so completion, the stalled-batch sweep and cleanup all release it. While it is held, another `Pgbus::Batch.new(uniqueness_key: "perfecta-daily").enqueue { ... }` raises `Pgbus::Batch::AlreadyRunning` (`:reject`), or skips its block and returns a batch that answers `discarded?` (`:discard`, `:log`). The row lives in `pgbus_uniqueness_keys` as `batch:<key>`, so it never collides with a job's key and shows up in the dashboard like any other lock; the reaper judges it by the batch's status, never by queue contents.
+The lock is taken at the top of the block (or at `enqueue`) and held until the batch finishes — through every open-batch `batch.add` from inside a running job — then released in the same single-winner finish that fires the callbacks, so completion, the stalled-batch sweep and cleanup all release it. While it is held, another `Pgbus.batch(uniqueness_key: "perfecta-daily") { ... }` raises `Pgbus::Batch::AlreadyRunning` (`:reject`), or skips its block and returns a batch that answers `discarded?` (`:discard`, `:log`). The row lives in `pgbus_uniqueness_keys` as `batch:<key>`, so it never collides with a job's key and shows up in the dashboard like any other lock; the reaper judges it by the batch's status, never by queue contents.
 
 #### How batches work
 
-1. `Batch.new(...)` creates a tracking row in `pgbus_batches` with `status: "pending"`
-2. `batch.enqueue { ... }` tags each enqueued job with the `pgbus_batch_id` in its payload
+1. The top of the `Pgbus.batch` block (or the first `batch.enqueue`) creates a tracking row in `pgbus_batches` with `status: "pending"`
+2. Each job enqueued in the block is tagged with the `pgbus_batch_id` in its payload
 3. After each job completes or is dead-lettered, the executor atomically updates the batch counters
 4. When `completed_jobs + discarded_jobs == total_jobs`, the batch status flips to `"finished"` and callback jobs are enqueued
 5. The dispatcher cleans up finished batches older than 7 days
@@ -1017,7 +1011,7 @@ c.metrics_backend = MyOpenTelemetryBackend.new
 | `pgbus_queue_job_count` | counter | `queue`, `job_class`, `status` (`processed`/`failed`/`dead_lettered`) |
 | `pgbus_visibility_extended` | counter | `queue`, `job_class` — one per re-armed visibility timeout; a class listed here outlives `visibility_timeout` |
 | `pgbus_job_duration_ms` | histogram | `queue`, `job_class` |
-| `pgbus_event_count` | counter | `handler`, `routing_key`, `status` |
+| `pgbus_event_count` | counter | `handler`, `routing_key`, `status` (`processed`/`failed`/`skipped`/`dead_lettered`), plus `reason` on `skipped` only (`completed`/`cached`/`owned`) |
 | `pgbus_event_duration_ms` | histogram | `handler`, `routing_key` |
 | `pgbus_messages_sent` / `pgbus_messages_read` | counter | `queue` |
 | `pgbus_stream_broadcast_count` | counter | `stream`, `deferred` |
@@ -1052,7 +1046,7 @@ ActiveSupport::Notifications.subscribe(/^pgbus\./) do |name, start, finish, _id,
 end
 ```
 
-Metrics-relevant events emitted (the subset the metrics subscriber maps; see `lib/pgbus/instrumentation.rb` for the full `pgbus.*` catalog): `pgbus.executor.execute`, `pgbus.job_completed`, `pgbus.job_failed`, `pgbus.job_dead_lettered`, `pgbus.event_processed`, `pgbus.event_failed`, `pgbus.client.send_message`, `pgbus.client.send_batch`, `pgbus.client.read_batch`, `pgbus.client.pool`, `pgbus.stream.broadcast`, `pgbus.outbox.publish`, `pgbus.recurring.enqueue`, `pgbus.worker.recycle`, `pgbus.consumer.recycle`. Payload keys are documented in `lib/pgbus/instrumentation.rb`.
+Metrics-relevant events emitted (the subset the metrics subscriber maps; see `lib/pgbus/instrumentation.rb` for the full `pgbus.*` catalog): `pgbus.executor.execute`, `pgbus.job_completed`, `pgbus.job_failed`, `pgbus.job_dead_lettered`, `pgbus.event_processed`, `pgbus.event_failed`, `pgbus.event_dead_lettered`, `pgbus.event_skipped`, `pgbus.job_visibility_extended`, `pgbus.client.send_message`, `pgbus.client.send_batch`, `pgbus.client.read_batch`, `pgbus.client.pool`, `pgbus.stream.broadcast`, `pgbus.outbox.publish`, `pgbus.recurring.enqueue`, `pgbus.worker.recycle`, `pgbus.consumer.recycle`. Payload keys are documented in `lib/pgbus/instrumentation.rb`.
 
 ### Connection pool metrics
 

data/config/locales/en.yml

@@ -23,7 +23,7 @@ en:
         finished_at: Finished At
         not_found: Batch not found
         on_failure: On Failure
-        on_finish: On Finish
+        on_finish: On Complete
         on_success: On Success
         progress: Progress
         properties: Properties

data/lib/generators/pgbus/add_batch_callback_jobs_generator.rb

@@ -33,7 +33,10 @@ module Pgbus
         say "  2. Restart pgbus: bin/pgbus start"
         say ""
         say "You can now pass a configured ActiveJob instance to a batch:"
-        say "  Pgbus::Batch.new(on_finish: ReportJob.new.set(queue: :critical, wait: 5.minutes))"
+        say "  Pgbus.batch do |b|"
+        say "    b.on_complete ReportJob.new.set(queue: :critical, wait: 5.minutes)"
+        say "    b.add(*rows.map { |row| ImportRowJob.new(row.id) })"
+        say "  end"
         say ""
       end
 

data/lib/pgbus/active_job/batch_id.rb

@@ -8,9 +8,9 @@ module Pgbus
     #
     # * +batch_id+ — the batch this job is a *member* of. Assigned by the
     #   executor from the payload's +pgbus_batch_id+ before +perform+, so a
-    #   running job can call +batch.enqueue+ to add siblings.
+    #   running job can call +batch.add+ to add siblings.
     # * +callback_batch_id+ — the batch this job *reports on*. Set on
-    #   +on_finish+/+on_success+/+on_failure+ jobs at fire time. A callback is
+    #   +on_complete+/+on_success+/+on_failure+ jobs at fire time. A callback is
     #   never a member of the batch it reports on, so its +batch_id+ is nil.
     #
     # Both round-trip through +serialize+/+deserialize+, and both are omitted
@@ -36,8 +36,14 @@ module Pgbus
         self.callback_batch_id = job_data["callback_batch_id"]
       end
 
-      # The batch this job reports on, or is a member of. nil when neither.
-      def batch
+      # Without a block: the batch this job reports on, or is a member of; nil
+      # when neither. With a block: open a new, independent batch through
+      # Pgbus.batch (this job's own membership is untouched).
+      #
+      #   batch(description: "Import") { |b| b.add(*rows.map { |r| RowJob.new(r.id) }) }
+      def batch(**options, &block)
+        return Pgbus.batch(**options, &block) if block
+        raise ArgumentError, "batch(...) with options needs a block: batch(description: ...) { |b| ... }" if options.any?
         return @batch if defined?(@batch)
 
         id = callback_batch_id || batch_id

data/lib/pgbus/batch/builder.rb

@@ -0,0 +1,92 @@
+# frozen_string_literal: true
+
+module Pgbus
+  class Batch
+    # What Pgbus.batch (and +batch { |b| ... }+ inside a job) yields: callback
+    # setters and +add+ on a batch that is open for the length of the block.
+    #
+    #   Pgbus.batch(description: "Import #{source.id}") do |b|
+    #     b.on_complete ImportDoneJob.new.set(queue: :critical)
+    #     b.on_failure  ImportFailedJob
+    #     b.add(*source.rows.map { |row| ImportRowJob.new(row.id) })
+    #   end
+    #
+    # The batch row exists (and a uniqueness key is held) before the block
+    # runs, and it stays pending until the block returns, so the order of the
+    # calls does not matter and a job that completes during the block cannot
+    # finish the batch early. Once the block has returned the builder is
+    # closed: add more jobs through +batch.add+ on the returned batch.
+    class Builder
+      CLOSED_MESSAGE = "the batch builder only works inside its Pgbus.batch block; " \
+                       "add more jobs later with batch.add on the batch Pgbus.batch returned"
+      UNTAGGED_MESSAGE = "b.add ran on a thread or fiber outside its Pgbus.batch block (or inside another " \
+                         "batch's block), where its jobs would not join the batch; call b.add from the block itself"
+
+      # Flatten +jobs+ and check that each one is an ActiveJob instance.
+      def self.job_instances(jobs)
+        jobs = jobs.flatten
+        invalid = jobs.grep_v(::ActiveJob::Base)
+        return jobs if invalid.empty?
+
+        raise ArgumentError,
+              "a batch takes job instances (MyJob.new(args), optionally .set(wait:, queue:)), got #{invalid.first.inspect}"
+      end
+
+      def initialize(batch)
+        @batch = batch
+        @closed = false
+      end
+
+      def on_complete(job = nil, &) = set_callback(:on_complete, job, &)
+
+      def on_success(job = nil, &) = set_callback(:on_success, job, &)
+
+      def on_failure(job = nil, &) = set_callback(:on_failure, job, &)
+
+      # Deprecated alias of #on_complete until 1.0.
+      def on_finish(job = nil, &)
+        Batch.warn_on_finish_deprecated
+        set_callback(:on_complete, job, &)
+      end
+
+      # Enqueue job instances into the batch with one bulk send (and anything
+      # the block enqueues). Returns the jobs. Membership is the batch tag the
+      # block holds (Thread.current, so fiber-local), so an add from another
+      # thread or fiber, or from inside a nested batch's block, raises rather
+      # than send jobs that would not be counted.
+      def add(*jobs)
+        raise ArgumentError, CLOSED_MESSAGE if @closed
+        raise ArgumentError, UNTAGGED_MESSAGE unless Thread.current[:pgbus_batch_id] == @batch.batch_id
+
+        jobs = self.class.job_instances(jobs)
+        ::ActiveJob.perform_all_later(jobs) if jobs.any?
+        yield if block_given?
+        jobs
+      end
+
+      # Called by Batch.start when the block returns or raises.
+      def close
+        @closed = true
+      end
+
+      private
+
+      def set_callback(slot, job, &block)
+        raise ArgumentError, CLOSED_MESSAGE if @closed
+
+        unless block.nil? && callback_job?(job)
+          raise ArgumentError,
+                "#{slot} takes an ActiveJob class or a configured instance (MyJob.new.set(queue: :critical)), " \
+                "not #{block ? "a block" : job.inspect}: callbacks are stored on the batch row"
+        end
+
+        @batch.send(:write_callback, slot, job)
+        self
+      end
+
+      def callback_job?(job)
+        (job.is_a?(Class) && job < ::ActiveJob::Base) || job.is_a?(::ActiveJob::Base)
+      end
+    end
+  end
+end

data/lib/pgbus/batch.rb

@@ -21,21 +21,57 @@ module Pgbus
     LOCK_QUEUE_PREFIX = "batch:"
     VALID_CONFLICTS = %i[reject discard log].freeze
 
+    # The callback slots, each mapped to the prefix of its pgbus_batches
+    # columns. on_complete keeps the on_finish_* columns until the 1.0 schema
+    # cleanup, so rows written before the rename still fire.
+    CALLBACK_COLUMNS = { on_complete: "on_finish", on_success: "on_success", on_failure: "on_failure" }.freeze
+
     attr_reader :batch_id, :properties, :description,
-                :on_finish, :on_success, :on_failure,
+                :on_complete, :on_success, :on_failure,
                 :uniqueness_key, :on_conflict
 
+    # Deprecated alias of #on_complete until 1.0.
+    def on_finish = on_complete
+
     def on_discard
       on_failure
     end
 
+    # Open a batch, yield a Pgbus::Batch::Builder, and start the batch when
+    # the block returns. Every job the block enqueues (through +b.add+ or a
+    # plain +perform_later+) joins the batch, which stays pending, and so
+    # cannot finish, until the block is done. Keywords go to Batch.new.
+    # Membership is the block's own thread or fiber (the tag lives in
+    # Thread.current): b.add raises anywhere else, and a plain perform_later
+    # from a thread or fiber the block spawns does not join the batch.
+    #
+    #   Pgbus::Batch.start(description: "Import") do |b|
+    #     b.on_complete ImportDoneJob
+    #     b.add(*rows.map { |row| ImportRowJob.new(row.id) })
+    #   end
+    def self.start(**, &block)
+      raise ArgumentError, "Pgbus.batch needs a block: Pgbus.batch { |b| b.add(...) }" unless block
+
+      batch = new(**)
+      builder = Builder.new(batch)
+      begin
+        batch.enqueue { yield builder }
+      ensure
+        builder.close
+      end
+      batch
+    end
+
+    # @param on_complete [Class, ActiveJob::Base, nil] fired when every job
+    #   reached a terminal state (+on_finish:+ is a deprecated alias)
     # @param uniqueness_key [String, nil] at most one unfinished batch with
     #   this key may exist; see #enqueue for what happens to the next one
     # @param on_conflict [Symbol] :reject (raise AlreadyRunning), :discard
     #   (skip the block silently) or :log (skip the block, warn)
-    def initialize(on_finish: nil, on_success: nil, on_discard: nil, on_failure: nil, description: nil, properties: {},
-                   uniqueness_key: nil, on_conflict: :reject)
+    def initialize(on_complete: nil, on_finish: nil, on_success: nil, on_discard: nil, on_failure: nil,
+                   description: nil, properties: {}, uniqueness_key: nil, on_conflict: :reject)
       raise ArgumentError, "pass on_failure: only — on_discard: is a deprecated alias" if on_discard && on_failure
+      raise ArgumentError, "pass on_complete: only — on_finish: is a deprecated alias" if on_finish && on_complete
       unless VALID_CONFLICTS.include?(on_conflict)
         raise ArgumentError, "on_conflict must be one of #{VALID_CONFLICTS.join(", ")}, got #{on_conflict.inspect}"
       end
@@ -45,9 +81,10 @@ module Pgbus
           "[Pgbus] Batch on_discard: is deprecated and will be removed in 1.0 — use on_failure: instead"
         end
       end
+      self.class.warn_on_finish_deprecated if on_finish
 
       @batch_id = SecureRandom.uuid
-      @on_finish = on_finish
+      @on_complete = on_complete || on_finish
       @on_success = on_success
       @on_failure = on_failure || on_discard
       @description = description
@@ -91,6 +128,23 @@ module Pgbus
       self
     end
 
+    # Enqueue ActiveJob instances (and anything the block enqueues) into this
+    # batch: +enqueue { ActiveJob.perform_all_later(jobs) }+. Instances keep
+    # their own +.set(queue:, wait:, priority:)+ options. On a started batch
+    # this is an open-batch add, so a job running inside the batch can call
+    # +batch.add(NextJob.new(id))+. Returns the jobs; on a unique batch that
+    # another run discarded (#discarded?) nothing was sent.
+    def add(*jobs, &block)
+      jobs = Builder.job_instances(jobs)
+      return jobs if jobs.empty? && block.nil?
+
+      enqueue do
+        ::ActiveJob.perform_all_later(jobs) if jobs.any?
+        yield if block
+      end
+      jobs
+    end
+
     # --- readers on a live batch ---------------------------------------
 
     def status = record&.status
@@ -210,11 +264,22 @@ module Pgbus
     end
 
     # Schema-probe memos (a true answer is kept for the life of the process)
-    # and the warn-once flag, read and written under @schema_lock (issue
+    # and the warn-once flags, read and written under @schema_lock (issue
     # #527). The probes run outside it: a duplicate probe on a race is
     # harmless, a lock held across a query is not.
     @schema_lock = Mutex.new
 
+    # on_finish: is in most batch calls written before the rename (issue
+    # #538), so it warns once per process rather than on every batch.
+    def self.warn_on_finish_deprecated
+      first = @schema_lock.synchronize { @warned_on_finish ? false : (@warned_on_finish = true) }
+      return unless first
+
+      Pgbus.logger.warn do
+        "[Pgbus] Batch on_finish is deprecated and will be removed in 1.0 — use on_complete instead"
+      end
+    end
+
     def self.executions_migrated?
       return true if @schema_lock.synchronize { @executions_migrated }
 
@@ -243,6 +308,7 @@ module Pgbus
         @executions_migrated = nil
         @callback_jobs_migrated = nil
         @warned_callback_jobs_unmigrated = nil
+        @warned_on_finish = nil
       end
     end
 
@@ -521,7 +587,7 @@ module Pgbus
       @batch_id = record.batch_id
       @description = record.description
       @properties = parse_properties(record.properties)
-      @on_finish = nil
+      @on_complete = nil
       @on_success = nil
       @on_failure = nil
       @uniqueness_key = nil
@@ -572,20 +638,37 @@ module Pgbus
       attrs = {
         batch_id: batch_id,
         description: description,
-        on_finish_class: callback_class_name(on_finish),
-        on_success_class: callback_class_name(on_success),
         properties: JSON.generate(properties),
         status: "pending"
       }
-      if self.class.executions_migrated?
-        attrs[:on_failure_class] = callback_class_name(on_failure)
-      else
-        attrs[:on_discard_class] = callback_class_name(on_failure)
-      end
-      attrs.merge!(callback_job_attributes) if self.class.callback_jobs_migrated?
+      CALLBACK_COLUMNS.each_key { |slot| attrs.merge!(callback_attributes(slot, public_send(slot))) }
       @record = BatchEntry.create!(attrs)
     end
 
+    # Replace one callback on a batch that has not finished yet (the
+    # builder's setters, inside the Pgbus.batch block, while the row is
+    # still pending and so cannot fire its callbacks).
+    def write_callback(slot, callback)
+      updated = BatchEntry.where(batch_id: batch_id, status: %w[pending processing])
+                          .update_all(callback_attributes(slot, callback))
+      raise AlreadyFinished, "Can't change the callbacks of an already finished batch" if updated.zero?
+
+      instance_variable_set(:"@#{slot}", callback)
+      reload
+    end
+
+    # The pgbus_batches columns for one callback slot. Before the
+    # add_batch_executions migration the failure class lives in
+    # on_discard_class; before add_batch_callback_jobs there is no *_job
+    # column, and a configured instance degrades to its class.
+    def callback_attributes(slot, callback)
+      prefix = CALLBACK_COLUMNS.fetch(slot)
+      class_column = slot == :on_failure && !self.class.executions_migrated? ? "on_discard" : prefix
+      attrs = { "#{class_column}_class": callback_class_name(callback) }
+      attrs[:"#{prefix}_job"] = serialize_callback(callback) if self.class.callback_jobs_migrated?
+      attrs
+    end
+
     # A callback given as a bare class keeps the legacy *_class column; a
     # configured ActiveJob instance is serialized now (so .set options resolve
     # at creation, matching solid_queue) into the *_job jsonb column. Before
@@ -600,14 +683,6 @@ module Pgbus
       callback.class.name
     end
 
-    def callback_job_attributes
-      {
-        on_finish_job: serialize_callback(on_finish),
-        on_success_job: serialize_callback(on_success),
-        on_failure_job: serialize_callback(on_failure)
-      }
-    end
-
     def serialize_callback(callback)
       return nil if callback.nil? || callback.is_a?(Class)
 

data/lib/pgbus/generators/migration_detector.rb

@@ -263,7 +263,7 @@ module Pgbus
         [:add_batch_executions]
       end
 
-      # Configured callback instances (on_finish: MyJob.new.set(...)) need
+      # Configured callback instances (on_complete: MyJob.new.set(...)) need
       # somewhere to store the serialized job — issue #415.
       def batch_callback_jobs_migrations
         return [] unless table_exists?("pgbus_batches")

data/lib/pgbus/instrumentation.rb

@@ -21,6 +21,17 @@ module Pgbus
   #                                  payload: queue, job_class, msg_id, vt, extensions
   #   pgbus.event_processed        — event handler succeeded
   #   pgbus.event_failed           — event handler raised; carries :exception_object
+  #   pgbus.event_dead_lettered    — a consumer moved an event that exceeded max_retries
+  #                                  to its DLQ; fires once, after the move
+  #                                  payload: queue_name, routing_key, handler, event_id,
+  #                                  read_ct, msg_id, error (the class of the last
+  #                                  recorded failure: a handler's error, or the
+  #                                  consumer's own parse error for a body it could
+  #                                  not route; nil when none was recorded). Any of
+  #                                  routing_key/handler/event_id can be nil: all
+  #                                  three when the body is not a JSON object, a
+  #                                  field the envelope lacks, and handler when no
+  #                                  single subscriber in this process owns the queue
   #   pgbus.event_unrouted         — a consumer read an event from a queue no
   #                                  subscriber in this process owns (stale topic
   #                                  binding); the message is archived

data/lib/pgbus/integrations/appsignal/subscriber.rb

@@ -60,6 +60,7 @@ module Pgbus
               subscribe("pgbus.job_dead_lettered") { |event| on_job_dead_lettered(event) },
               subscribe("pgbus.event_processed") { |event| on_event_processed(event) },
               subscribe("pgbus.event_failed") { |event| on_event_failed(event) },
+              subscribe("pgbus.event_dead_lettered") { |event| on_event_dead_lettered(event) },
               subscribe("pgbus.client.send_message") { |event| on_send_message(event) },
               subscribe("pgbus.client.send_batch") { |event| on_send_batch(event) },
               subscribe("pgbus.client.read_batch") { |event| on_read_batch(event) },
@@ -175,6 +176,20 @@ module Pgbus
             ::Appsignal.set_error(err) if err && ::Appsignal.respond_to?(:set_error)
           end
 
+          # A counter only, like on_job_dead_lettered: nothing is raised at
+          # move time, and the handler's error already went out through
+          # pgbus.event_failed on the attempt that raised (issue #543). A body
+          # that is not JSON leaves handler and routing_key nil; compact drops
+          # those tags, as Metrics::Subscriber does.
+          def on_event_dead_lettered(event)
+            payload = event.payload
+            ::Appsignal.increment_counter(
+              "#{METRIC_PREFIX}event_count",
+              1,
+              { handler: payload[:handler], routing_key: payload[:routing_key], status: "dead_lettered" }.compact
+            )
+          end
+
           # ── Client (PGMQ wrapper) ───────────────────────────────────────
 
           def on_send_message(event)

data/lib/pgbus/metrics/subscriber.rb

@@ -63,6 +63,7 @@ module Pgbus
             subscribe("pgbus.job_visibility_extended") { |event| on_job_visibility_extended(event) },
             subscribe("pgbus.event_processed") { |event| on_event_processed(event) },
             subscribe("pgbus.event_failed") { |event| on_event_failed(event) },
+            subscribe("pgbus.event_dead_lettered") { |event| on_event_dead_lettered(event) },
             subscribe("pgbus.event_skipped") { |event| on_event_skipped(event) },
             subscribe("pgbus.client.send_message") { |event| on_send_message(event) },
             subscribe("pgbus.client.send_batch") { |event| on_send_batch(event) },
@@ -154,6 +155,17 @@ module Pgbus
           )
         end
 
+        # The consumer moved an event that exhausted max_retries to its DLQ
+        # (issue #543). A body that is not JSON leaves handler and routing_key
+        # nil; compact drops those tags.
+        def on_event_dead_lettered(event)
+          payload = event.payload
+          backend.increment(
+            "#{METRIC_PREFIX}event_count", 1,
+            compact(handler: payload[:handler], routing_key: payload[:routing_key], status: "dead_lettered")
+          )
+        end
+
         # An idempotent handler that did not run this delivery. `reason` is what
         # makes the skip readable: :completed/:cached is deduplication working,
         # :owned means a second delivery arrived while the holder was still

data/lib/pgbus/process/consumer.rb

@@ -289,9 +289,9 @@ module Pgbus
         @jobs_processed.increment
       end
 
-      # Order: read the last error, move with the dead-letter block, then
-      # clear the failed-event row. A move that raises keeps the row, so the
-      # next attempt still carries the reason.
+      # Order: read the last error, move with the dead-letter block, clear the
+      # failed-event row, then announce it. A move that raises keeps the row,
+      # so the next attempt still carries the reason, and announces nothing.
       def dead_letter(message, queue_name)
         last_error = FailedEventRecorder.last_error(queue_name: queue_name, msg_id: message.msg_id.to_i)
         headers = DeadLetterHeader.build(
@@ -301,6 +301,59 @@ module Pgbus
         )
         Pgbus.client.move_to_dead_letter(queue_name, message, headers: headers)
         FailedEventRecorder.clear!(queue_name: queue_name, msg_id: message.msg_id.to_i)
+        instrument_dead_lettered(message, queue_name, last_error)
+      end
+
+      # pgbus.event_dead_lettered, the event twin of the executor's
+      # pgbus.job_dead_lettered (issue #543). The dead-letter branch runs
+      # before handle_message parses the envelope, so it is parsed here: once
+      # per dead-lettered message, never on the success path. The move has
+      # already happened, so nothing here may turn the dead-letter into a
+      # failure (a failed stat, a circuit-breaker trip): the guard is the
+      # executor's instrument rescue, widened on purpose to cover building the
+      # payload as well as the subscribers, and logs at the same debug level.
+      def instrument_dead_lettered(message, queue_name, last_error)
+        envelope = dead_letter_envelope(message)
+        routing_key = envelope_routing_key(envelope)
+        Pgbus::Instrumentation.instrument(
+          "pgbus.event_dead_lettered",
+          queue_name: queue_name, routing_key: routing_key,
+          handler: owning_handler_name(routing_key, queue_name), event_id: envelope["event_id"],
+          read_ct: message.read_ct.to_i, msg_id: message.msg_id.to_i, error: last_error&.dig(:error_class)
+        )
+      rescue StandardError => e
+        Pgbus.logger.debug { "[Pgbus] Notification failure pgbus.event_dead_lettered: #{e.class}: #{e.message}" }
+      end
+
+      # The body that killed a message may not be a JSON object at all: PGMQ
+      # stores jsonb, so a producer writing straight to the queue can leave a
+      # scalar or an array, and the success path's dig raises on those every
+      # read. Any of them reads as an empty envelope, so the notification
+      # still fires.
+      def dead_letter_envelope(message)
+        raw = JSON.parse(message.message)
+        raw.is_a?(Hash) ? raw : {}
+      rescue JSON::ParserError, TypeError
+        {}
+      end
+
+      # The success path's lookup (headers first, then the top level),
+      # tolerating a headers value that is not an object.
+      def envelope_routing_key(envelope)
+        headers = envelope["headers"]
+        (headers["routing_key"] if headers.is_a?(Hash)) || envelope["routing_key"]
+      end
+
+      # The handler of the one subscriber that owns +queue_name+ in this
+      # process, or nil when there is none, no usable routing key, or more than
+      # one: #469 makes one owner per queue the convention, but Registry does
+      # not enforce it, and dispatch runs every owner, so naming one of several
+      # would misattribute the dead letter.
+      def owning_handler_name(routing_key, queue_name)
+        return unless routing_key.is_a?(String)
+
+        owners = @registry.handlers_for(routing_key, queue_name: queue_name)
+        owners.first.handler_class.name if owners.one?
       end
 
       # The executor's handle_failure, for events (issue #494): one upsert per

data/lib/pgbus/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Pgbus
-  VERSION = "0.19.1"
+  VERSION = "0.19.2"
 end

data/lib/pgbus.rb

@@ -298,6 +298,17 @@ module Pgbus
       Streams::Key.queue_name_budget
     end
 
+    # Open a batch, yield a Pgbus::Batch::Builder, and start the batch when
+    # the block returns — the top-level shortcut for Pgbus::Batch.start.
+    #
+    #   Pgbus.batch(description: "Import") do |b|
+    #     b.on_complete ImportDoneJob
+    #     b.add(*rows.map { |row| ImportRowJob.new(row.id) })
+    #   end
+    def batch(**, &)
+      Batch.start(**, &)
+    end
+
     # Publish an event to the bus — the top-level shortcut for
     # `Pgbus::EventBus::Publisher.publish`, symmetric with `Pgbus.stream`.
     #

data/sig/pgbus.rbs

@@ -49,6 +49,13 @@ module Pgbus
     end
   end
 
+  # Partial: only the types Pgbus.batch names. Batch's own methods are not
+  # covered by spec/pgbus/rbs_spec.rb yet.
+  class Batch
+    class Builder
+    end
+  end
+
   def self.configuration: () -> Configuration
   def self.configure: () { (Configuration) -> void } -> Configuration
   def self.finalize!: () -> Configuration
@@ -57,6 +64,8 @@ module Pgbus
   def self.stream_key: (*untyped parts, ?digest_bits: Integer) -> String
   def self.stream_key!: (String key) -> String
   def self.stream_name_budget: () -> Integer
+  # Keywords go to Pgbus::Batch.new (description:, properties:, on_complete:, uniqueness_key:, …).
+  def self.batch: (**untyped options) { (Batch::Builder) -> void } -> Batch
   # delay: seconds, as a Numeric or an ActiveSupport::Duration (1.hour).
   def self.publish: (String routing_key, untyped payload, ?headers: Hash[String, untyped]?, ?delay: Numeric | _ToI) -> untyped
   def self.publish_later: (String routing_key, untyped payload, delay: Numeric | _ToI, ?headers: Hash[String, untyped]?) -> untyped

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: pgbus
 version: !ruby/object:Gem::Version
-  version: 0.19.1
+  version: 0.19.2
 platform: ruby
 authors:
 - Mikael Henriksson
@@ -307,6 +307,7 @@ files:
 - lib/pgbus/active_job/executor.rb
 - lib/pgbus/autovacuum_tuning.rb
 - lib/pgbus/batch.rb
+- lib/pgbus/batch/builder.rb
 - lib/pgbus/batch/sweep.rb
 - lib/pgbus/bus_record.rb
 - lib/pgbus/circuit_breaker.rb