pgbus 0.14.0 → 0.14.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 66fe070813c414410d4b55f2b9c9f1dfcacd70236f97d22f8e037ec7a2d906c4
-  data.tar.gz: 285e8851e43a02384734c0dfda2ae240ea4450554b5c8dca478e351b04a57528
+  metadata.gz: 6ba7f4aa13c44c41c30fb021cbb3e087d643b147895a80a84ddbc022f40088a7
+  data.tar.gz: 3015145e2f795d83d543d786c4f5d462854123f3a9e8eb946c2b74de059ee31a
 SHA512:
-  metadata.gz: 160efab6bf57a74ab7d190b127b7e0f64426fc17877407a87c0f70a3ba328bea8e0dfa6c44e2477263e746b16a23544843a044e1eb88c285147a6e31e129f404
-  data.tar.gz: 7dc944582757565e3d0c8339661e533745f750731da645f2aaf6693673083b31b470f1b92265e880278b7b5720a593cc266eb9d3a11aaf554a527780f7cd481b
+  metadata.gz: dc4d0ccfac9ab0f4f57c271fe98e2a7716cfbac10daba7ca5b0ad81b59f43a51a775a1780e36faa395f23ed8084376a7e21645390e8a5f6463e1100a4db007d6
+  data.tar.gz: 684ce73b5e1e43be5d9616319da6d29ed22acf98e5384d662b63d7020ed234d8b94e2783f97a45062ae532633507b0cf44afe0007a68ea4894b55b23e631b018

data/CHANGELOG.md

@@ -1,7 +1,37 @@
 ## [Unreleased]
 
+### Added
+
+- **Batches v2: open batches, a `batch` accessor on jobs and callbacks, and configured callback instances (issue #415).** ⚠️ **Breaking (pre-1.0)**: `Pgbus::Batch.find` now returns a rehydrated `Pgbus::Batch` handle instead of the raw attributes Hash — read the values off the handle (`status`, `total_jobs`, `completed_jobs`, `failed_jobs`, `pending_jobs`, `progress_percentage`, `finished?`, `description`, `properties`), or query `Pgbus::BatchEntry` for a row. Three capabilities: **(1) Open batches.** `batch.enqueue` is re-callable while the batch is unfinished — the second and later calls add to the existing group (`BatchEntry.increment_total_jobs!`, guarded on an unfinished row) instead of hitting the unique index on `batch_id`, so multi-stage workflows are possible. Adding to a finished batch raises `Pgbus::Batch::AlreadyFinished`. Execution rows are inserted as the block enqueues, before `total_jobs` is bumped, so the single-winner finish always sees outstanding work; a `check_finished!` after the bump covers a block whose jobs all completed while it was still open. Membership stays explicit: only jobs enqueued inside an `enqueue` block join the batch, so a fan-out from a batched job never silently extends it. **(2) `batch` accessor.** New `Pgbus::ActiveJob::BatchId` mixin (included on `ActiveJob::Base` from the engine alongside `Concurrency` and `Uniqueness`) adds `batch_id` / `callback_batch_id` accessors, round-trips both through `serialize`/`deserialize` — omitted from the payload when unset, so an unbatched job's serialized hash is unchanged — and exposes a memoized `batch` reader. The executor assigns `batch_id` from the payload's `pgbus_batch_id` before `perform`, so a running job can call `batch.enqueue` to add siblings. **(3) Configured callback instances.** `on_finish:` / `on_success:` / `on_failure:` accept an ActiveJob instance as well as a class: `on_finish: ReportJob.new.set(queue: :critical, wait: 5.minutes)`. `.set` options resolve at batch-creation time (matching solid_queue) into new `on_finish_job` / `on_success_job` / `on_failure_job` jsonb columns; at fire time the job is deserialized, given `callback_batch_id`, has its own `batch_id` cleared (a callback is never a member of the batch it reports on) and is enqueued on its configured queue and `scheduled_at`. Legacy class-name callbacks keep the `perform_later(properties)` signature — deprecated at 1.0 in favour of `batch.properties`. The dashboard shows a configured callback's `job_class`. Existing installs: `rails generate pgbus:add_batch_callback_jobs` (or `pgbus:update`); apps that have not migrated keep storing bare classes only. Refs #415.
+
+- **Batch completion is now self-healing via execution-row tracking (issue #414).** ⚠️ **Restart pgbus workers after `rails generate pgbus:add_batch_executions` / `db:migrate`** — a process that still writes `discarded_jobs` will error once that column is gone. `pgbus_batches` no longer finishes on `completed + discarded == total` — that arithmetic stalled a batch *forever* if a worker crashed between `archive_from` and `signal_batch_completed`, and a crash mid-`Batch#enqueue` left an immortal `pending` row (`cleanup_batches` only deletes `finished`). Each batched job now gets a `pgbus_batch_executions` row (identity = ActiveJob `job_id`) inserted *before* `send_message`, with `msg_id` + `queue_name` backfilled after send. The batch finishes when no rows remain (single-winner `UPDATE … AND NOT EXISTS`, plus a fresh `exists?` re-check for Postgres READ COMMITTED). A dispatcher sweep (`config.batch_sweep_interval`, default 5 minutes) repairs four crash windows: stale rows whose PGMQ message is gone (archived → completed, DLQ → failed, missing → completed with a warning); orphan rows with `msg_id` NULL older than 5 minutes (enqueue crashed between insert and send — blocked concurrency jobs are excluded); stalled `pending` batches; stalled `processing` batches with zero rows. `on_failure:` / `failed_jobs` are canonical (`on_discard:` / `discarded_jobs` remain deprecated aliases until 1.0). `config.batch_retention` (default 7 days, `nil` disables) replaces the hardcoded cleanup window. Existing installs: `rails generate pgbus:add_batch_executions` (or `pgbus:update`). Unmigrated apps keep the counter path via `Batch.executions_migrated?`. Refs #414.
+
 ### Fixed
 
+- **A `retry_on` re-enqueue now stays in its batch — batch completion means terminal completion (issue #424).** `retry_on` rescues inside `perform_now`, re-enqueues the same job (new PGMQ message, same ActiveJob `job_id`, `executions > 0`) and returns normally; the executor then archived the original message and signalled `job_completed` — deleting the execution row and bumping `completed_jobs` — while the retry message carried no batch tag at all. `on_success` could fire with a retry still pending, and a retry that later dead-lettered never fired `on_failure`. The adapter now recognises a retry re-enqueue (a job with a `batch_id` from the `BatchId` mixin and `executions > 0`, outside any `Batch#enqueue` block), re-tags it into its batch **without** counting it again (`Batch.track_retry`: the existing row is kept via `ON CONFLICT DO NOTHING`, the backfill re-points it at the retry message) and records the `job_id` in a per-thread set once the retry is live (sent, or parked as a blocked execution); the executor consults that set after `perform` and skips the completion signal for an attempt that re-enqueued itself, clearing the set per execute. A first-attempt job that merely has a `batch_id` outside a block is still not tagged (membership stays explicit), and `callback_batch_id` never re-tags. Batches with `retry_on` jobs finish later — correctly. Refs #424.
+
+- **Batched jobs count themselves in per job, in one transaction, before the send — `AlreadyFinished` now raises at `perform_later`, and no batch path can leave `total_jobs` out of step with its rows (issue #423).** Open batches (#422) inserted execution rows during the block and bumped `total_jobs` once at the end. Three ways that went wrong: a re-opened block that raised mid-way left rows counted but the total un-bumped, so the batch could never satisfy `completed + failed = total` and lived forever; `AlreadyFinished` was checked against a stale memoized row and only enforced after the block, so adding to a finished batch enqueued every job *and then* raised; and the stalled-pending sweep recomputed `total_jobs` from the row count, discarding `completed_jobs`/`failed_jobs` already incremented while the block ran — a crashed `pending` batch with any early completions was immortal too. Every batched enqueue (individual, bulk, blocked-promote) now runs `BatchEntry.increment_total_jobs!` — guarded on an unfinished row — and the execution-row insert inside one `BatchEntry.transaction`, *before* `send_message`; a `perform_all_later` increments once by N. The invariant `total_jobs == outstanding rows + completed_jobs + failed_jobs` holds at every commit point, so the first block only flips `pending → processing` (guarded) and re-checks, a re-opened block never bumps, the empty-batch path goes through the same single-winner finish (and now emits `pgbus.batch_finished`), and the sweep never rewrites totals. Discards at enqueue time (`uncount`) decrement and delete in one transaction. The legacy counter path only auto-finishes a `processing` batch (totals now grow mid-block). **Internal:** `Thread.current[:pgbus_batch_job_count]` is gone — nothing reads it; a spec that stubbed it should assert on `Pgbus::Batch.track_enqueue` instead. Cost: one extra `UPDATE pgbus_batches` per batched job (one per bulk send); plain enqueue is untouched. Refs #423, #415.
+
+- **The stalled-batch sweep no longer un-counts a live job, and no longer flips a pending batch that is still actively enqueuing (issue #423).** Phase 2 deleted every `msg_id IS NULL` execution row older than the threshold and decremented `total_jobs`, assuming the enqueue died between row insert and send — but a send that landed whose backfill then failed leaves the same shape with a **live** message, and un-counting it let the batch finish (and fire `on_success`) before that job ran. The row now stores the logical queue at insert, and the sweep probes it by `job_id` first: message present → keep; probe inconclusive → keep; present in the DLQ → resolve as failed; nowhere → orphan as before. `Client#message_with_job_id?` expands a logical name to every `_pN` sub-queue under priority routing (it probed a single bare table that does not exist there and returned "unknown"). Phase 3 treats a `pending` batch as stalled only after `config.batch_stall_threshold` (new, default 5 minutes, replaces the hardcoded constant) with **no execution row inserted in that window**, so a long-running enqueue block is not closed under the caller. Refs #423.
+
+- **A `:while_executing` job that fails once can retry — the lock is released on failure and re-acquired by the same message after a crash (issue #423).** The lock was acquired at execution start with `msg_id = 0` and released only on success or dead-letter. After an exception the row stayed, and the retry of the *same message* hit `INSERT … ON CONFLICT DO NOTHING` against its own row → `:skipped`, on every read, until `read_ct` exceeded `max_retries` and the job dead-lettered without a second real attempt; a process kill mid-job did the same. The executor now binds the lock to the message it is executing (`msg_id` + queue) and releases it in the failure path (`:until_executed` is untouched — that lock is held until success/DLQ by design); `UniquenessKey.acquire!(reacquire_same_message: true)` treats a conflict with a row pointing at the **same** `msg_id` as this message's own previous attempt (PGMQ's visibility timeout guarantees nobody else holds it) and re-acquires, while a different `msg_id` is still a genuine duplicate. Refs #423.
+
+- **A batch callback configured as an ActiveJob instance on a database that has not run `add_batch_callback_jobs` is no longer silently dropped (issue #423).** It degrades to its class in the legacy `on_*_class` column — the callback still fires, on its default queue — and logs a WARN once per process that `.set` options are ignored until the migration runs. Refs #423.
+
+- **A concurrency-blocked job keeps its priority when promoted (issue #423).** `BlockedExecution.release_next!` returned only `queue_name` and `payload`, so `promote_next` re-sent without `priority:` and, under `priority_levels > 1`, every promoted job landed on the default sub-queue regardless of what the enqueuer asked for. The row's `priority` is now returned and passed to `send_message` and to the execution-row backfill's target queue. Refs #423.
+
+- **A batch whose `enqueue` block crashed before enqueuing anything is no longer immortal (#414 follow-up).** The stalled-batch sweep moved such a batch from `pending` to `processing` with `total_jobs = 0`, but `BatchEntry.finish_if_empty!` required `total_jobs > 0` and `Sweep#counters_terminal?` required `total_jobs.positive?` — guards added so a pre-migration in-flight batch (zero execution rows, counters short of `total_jobs`) is not closed empty. Nothing could then close the crashed batch: it sat in `processing` forever, `cleanup_batches` only deletes `finished` rows so it accumulated, and `on_finish` never fired — the same leak #414 set out to remove, relocated one status along. Both guards now test `completed_jobs + failed_jobs = total_jobs` alone: `total_jobs = 0` with zero counters *is* terminal, while a legacy in-flight batch still has counters short of a positive `total_jobs` and is still left on the counter path. Refs #414.
+
+- **`ActiveJob.perform_all_later` no longer raises `PG::UndefinedTable` when priority routing is enabled.** `Client#send_batch` targeted `config.queue_name(queue)` — the bare `pgbus_<queue>` table — while the priority strategy only ever creates `_p0.._pN`, so every bulk enqueue under `priority_levels > 1` failed outright (and, had the table existed, no worker would have read it: consumers poll the sub-queues). `send_batch` now routes through the same `QueueFactory` strategy as `send_message` and takes an optional `priority:`, `Adapter#enqueue_all` groups bulk jobs by queue **and** priority so a mixed-priority `perform_all_later` produces one batch per level, and `Outbox::Poller#publish_queue_batch` — which already grouped entries by priority and then discarded it — passes it through (its single-send fallback now does too, so both paths place messages identically). Under the default single-queue strategy the target is unchanged. This also stopped bulk-enqueued batch jobs from leaking `pgbus_batch_executions` rows on the failed send. Refs #413.
+
+- **`:until_executed` uniqueness keys no longer stick at `pending`/`msg_id=0` so the orphan reaper can now release true orphans (issue #418).** The adapter inserted the lock row as `queue_name: "pending", msg_id: 0` before `send_message` and never UPDATEd it with the real logical queue and PGMQ `msg_id`. The dispatcher's reaper then probed `pgmq.q_<prefix>_pending` — a table that does not exist — `Client#message_exists?` returned `nil` (UndefinedTable), and `nil` is treated as "still here" so leaked keys lived forever. Heartbeat jobs using `on_conflict: :discard` stayed discarded for hours or days with no matching queue message. Three cooperating fixes: **(1)** acquire with the logical queue name, then `UniquenessKey.bind!` after a successful send (adapter and recurring scheduler; retry re-enqueues do not re-bind — issue #333). A bind error is fail-soft. **(2)** Unbound rows (`pending` and/or `msg_id=0`) are reaped by scanning live `pgmq.q_*` tables via `Client#uniqueness_keys_present` — never by querying the synthetic pending queue. In-flight jobs that still look like `pending`/`0` are kept if any queue payload still carries the key. Age floor remains `2 * visibility_timeout`. **(3)** The executor releases the lock once, immediately after archive (fail-soft), so a kill during `pgbus.job_completed` cannot leak a lock whose message is already gone. There is deliberately **no** second `ensure` DELETE: a key-only DELETE after a committed release can drop a successor that re-acquired the same key; a failed release is instead healed by the reaper once the archived message is provably gone. Bound `message_exists?` lookups on a logical name also check priority `_pN` sub-queues. Do **not** bulk-delete `pending`/`msg_id=0` rows on upgrade — the reaper heals true orphans on the next cleanup cycle. Refs #418.
+
+- **A job that combines `ensures_uniqueness` (`:until_executed`) with `limits_concurrency on_conflict: :discard` no longer leaves the uniqueness lock held forever.** Enqueue acquires the uniqueness lock first, then the semaphore; a `:discard` conflict dropped the job without sending a message, so no executor could ever release the lock and later equivalent jobs were blocked until someone deleted the `pgbus_uniqueness_keys` row. The discard path now rolls back the lock this enqueue acquired. A uniqueness *duplicate* discard still does not release — that lock belongs to the in-flight job. `:block` conflicts still keep the lock, because `BlockedExecution` stores the tagged payload and the job runs when the semaphore frees. Refs #413.
+
+- **`ActiveJob.perform_all_later` no longer lets jobs escape batch tracking or bypass concurrency limits (issue #413).** `Adapter#enqueue_all`'s bulk path (`enqueue_immediate`) serialized payloads directly to `send_batch`, skipping the metadata-injection steps the individual `enqueue`/`enqueue_at` paths run. Two consequences: **(1)** jobs bulk-enqueued inside a `Pgbus::Batch#enqueue` block were neither tagged with `pgbus_batch_id` nor counted into `total_jobs`, so the batch could finish — and fire `on_finish`/`on_success` — before those jobs ever ran; **(2)** jobs whose class declares `pgbus_concurrency` were sent without semaphore acquisition, silently ignoring their concurrency limit. Bulk payloads now pass through `inject_batch_metadata` (tagging + counting), and concurrency-configured jobs are partitioned out of the bulk path into the individual enqueue path — exactly how uniqueness-configured jobs were already routed, and for the same reason: per-job locks cannot be acquired in a bulk send. Plain jobs keep the fast `send_batch` path unchanged. Refs #413.
+
+- **The 0.14.0 purge/drop guard no longer crashes on Rails 8.0/8.1 — pool ownership is resolved version-tolerantly (issue #411).** `Pgbus::BusRecord.disconnect_all_pools!` (added in #410 for the #409 wedge) identified its own pools via `pool.connection_class` — an API that exists on Rails 7.1/7.2 but was replaced in Rails 8.0 by `ConnectionPool#connection_descriptor` (a `ConnectionDescriptor` whose `#name` is the owning class name; 8.x `PoolConfig` has no `connection_class` either, so the obvious fallback route doesn't exist). Since the guard is prepended onto every purge/drop path, the `NoMethodError` aborted **every** `db:test:purge` / `db:prepare` on current stable Rails, forcing consumers to cap at `< 0.14`. Ownership is now resolved against whichever API the running Rails exposes — `connection_descriptor&.name == name` on 8.0+ (a nil descriptor, e.g. a null pool, never matches), `connection_class == self` on 7.x — with identical semantics: only `connects_to`-created BusRecord pools are disconnected, `ActiveRecord::Base`'s pool is never touched. The reason CI missed this: the spec stubbed `connection_class` on plain RSpec doubles, which happily fake methods the running Rails doesn't define. The spec now also exercises verifying doubles (`instance_double`) against the real `ConnectionPool`, so both existing matrix legs (7.1 and latest 8.x) fail on any future drift in the ownership API. Refs #411.
+
 - **Boot-time BusRecord connections no longer wedge `db:test:purge` — pgbus disconnects its own pools before every database purge/drop (issue #409).** With pgbus on a dedicated database (`config.connects_to`), any boot-time touch of a pgbus model left an idle session on that database for the life of the process. Rails' purge/drop only disconnects the connection it establishes for the target db_config — it knows nothing about gem-owned pools — so the rake process's own idle session blocked its own `DROP DATABASE`: a hang without a `statement_timeout`, a fast `ActiveRecord::QueryCanceled` with one, and either way `db:test:prepare` / `maintain_test_schema!` permanently broken on that machine (every retry re-boots, re-opens the connection, and re-blocks; terminating sessions externally can't help). Three changes: **(1)** `Pgbus::DatabaseTasksGuard` is prepended onto `ActiveRecord::Tasks::DatabaseTasks` at boot, so every route to a purge/drop — `db:test:purge` / `db:purge` / `db:drop` and their per-database variants, `maintain_test_schema!`'s in-process purge, parallel-testing's `TestDatabases` — first runs `Pgbus::BusRecord.disconnect_all_pools!` (all roles; a no-op on primary-database installs, `ActiveRecord::Base`'s pool is never touched). **(2)** The task-aware guard the event-bus registry already used internally is now public as **`Pgbus.database_task?`** — true while the process runs a `db:*` / `assets:*`-family rake task — so apps can skip their own boot-time warm-ups in the contexts where a database may legitimately not exist. **(3)** `pgbus:tune_autovacuum` (enhanced onto `db:schema:load`) no longer holds a permanent connection lease from `BusRecord.connection`: it checks out via `connection_pool.with_connection` and disconnects the pool afterward, leaving no idle session behind inside longer rake chains. Refs #409.
 
 - **Concurrent queue creation no longer kills the loser with `PG::UniqueViolation` on `pg_class` (issue #404).** The sibling of the #403 trigger race, one DDL step earlier: `CREATE TABLE IF NOT EXISTS` is not race-safe — two backends creating a not-yet-existing queue both pass the existence check (READ COMMITTED; neither sees the other's uncommitted catalog rows), both insert into `pg_class`, and the loser raises `unique_violation` on `pg_class_relname_nsp_index` instead of the friendly `duplicate_table`. All the dedup guards (`@queues_created`, `Stream#@ensured`, `synchronized`) are process-local, so with lazily-created per-record durable stream queues two processes first-touching the same brand-new stream hit `pgmq.create` simultaneously — and via `send_message`'s `ensure_queue` the same window applies to plain job queues on cold post-deploy herds (production: AppSignal incident #963, an unrescued `PG::UniqueViolation` out of an unrelated `after_commit`). Three DDL steps now rescue the duplicate as success, mirroring the #397/#403 shape: **(1)** `create_queue_table` — `pgmq.create` is a single statement and therefore atomic, so when the loser unblocks the winner has committed the whole queue; the loser re-checks `pgmq.meta` and returns, retrying `pgmq.create` once when the re-check can't confirm (e.g. a leftover physical table without a meta row) with a second failure propagating (the retry's own error — the original duplicate is preserved deeper in its cause chain via Ruby's implicit `$!` chaining); **(2)** `create_fifo_index_if_needed` under group_mode — the duplicate proves the index exists; **(3)** the stream archive `msg_id` index in `ensure_stream_queue` — a raw `CREATE INDEX IF NOT EXISTS` with the same catalog race, where the loser now proceeds to register and memoize the stream instead of dropping the first broadcast. Duplicate detection reuses `DUPLICATE_INSTALL_ERROR_CLASSES`, matched directly (raw `conn.exec` paths) or as the `cause` of pgmq-ruby's `ConnectionError` wrapper. No locking added, no happy-path cost. Refs #404.

data/app/models/pgbus/batch_entry.rb

@@ -4,14 +4,79 @@ module Pgbus
   class BatchEntry < BusRecord
     self.table_name = "pgbus_batches"
 
-    COUNTER_COLUMNS = %w[completed_jobs discarded_jobs].freeze
+    # discarded_jobs remains valid until the add_batch_executions migration
+    # folds it into failed_jobs. Both names are accepted so gem-before-migrate
+    # and gem-after-migrate stay incrementable.
+    COUNTER_COLUMNS = %w[completed_jobs discarded_jobs failed_jobs].freeze
 
     scope :finished, -> { where(status: "finished") }
     scope :stale, ->(before:) { finished.where("finished_at < ?", before) }
+    scope :pending, -> { where(status: "pending") }
+    scope :processing, -> { where(status: "processing") }
+    scope :without_executions, -> { where.not(batch_id: BatchExecution.select(:batch_id)) }
 
-    # Atomically increment the counter and detect if this update caused the
-    # batch to finish. Uses row-level locking to prevent duplicate callbacks.
+    # Deprecated alias until 1.0: after the column is dropped this reads failed_jobs.
+    def discarded_jobs
+      has_attribute?(:discarded_jobs) ? self[:discarded_jobs] : failed_jobs
+    end
+
+    # Atomically add n to total_jobs on an unfinished batch. Returns true.
+    # Raises Batch::AlreadyFinished when the row is already finished (0 rows
+    # updated) — the adder-before-insert contract open batches (#415) rely on.
+    def self.increment_total_jobs!(batch_id, count) # rubocop:disable Naming/PredicateMethod
+      updated = where(batch_id: batch_id, status: %w[pending processing])
+                .update_all(["total_jobs = total_jobs + ?", count])
+      raise Batch::AlreadyFinished, "Can't add jobs into an already finished batch" if updated.zero?
+
+      true
+    end
+
+    # Reverse of increment_total_jobs! for a job that will never run. Floored
+    # at zero; a finished row is left alone.
+    def self.decrement_total_jobs!(batch_id)
+      where(batch_id: batch_id, status: %w[pending processing])
+        .update_all(["total_jobs = GREATEST(total_jobs - 1, 0)"])
+    end
+
+    # Single-winner finish: status is processing AND no execution rows remain.
+    # Join-free NOT EXISTS so the subquery stays in this UPDATE's WHERE.
+    # Returns the number of rows updated (0 or 1).
+    def self.finish_if_empty!(batch_id)
+      # Counters must already be terminal so a pre-migration in-flight batch
+      # (zero execution rows, total_jobs = N, counters short of N) is not
+      # closed empty. total_jobs = 0 with zero counters IS terminal: that is a
+      # batch whose enqueue block crashed before it enqueued anything, and
+      # nothing else will ever close it.
+      where(batch_id: batch_id, status: "processing")
+        .without_executions
+        .where("completed_jobs + failed_jobs = total_jobs")
+        .update_all(status: "finished", finished_at: Time.current)
+    end
+
+    # Finish the batch if every job already reached a terminal state. Used
+    # after total_jobs is published: completion signals that arrived while the
+    # enqueue block was still open saw total_jobs == 0 and could not finish
+    # the batch themselves (PR #417). Row lock + status guard keep it
+    # idempotent against concurrent completion signals.
     # Returns { just_finished:, record: } or nil if batch not found.
+    def self.check_finished!(batch_id)
+      return Batch.try_finish!(batch_id) if Batch.executions_migrated?
+
+      transaction do
+        record = lock.find_by(batch_id: batch_id)
+        return nil unless record
+        return { record: record, just_finished: false } if record.status == "finished"
+        return { record: record, just_finished: false } unless record.completed_jobs + record.discarded_jobs == record.total_jobs
+
+        record.update!(status: "finished", finished_at: Time.current)
+        { record: record, just_finished: true }
+      end
+    end
+
+    # Atomically increment the counter and, on the pre-migration path, detect
+    # if this update caused the batch to finish. Uses row-level locking to
+    # prevent duplicate callbacks. Returns { just_finished:, record: } or nil
+    # if batch not found.
     def self.increment_counter!(batch_id, column)
       raise ArgumentError, "Invalid column: #{column}" unless COUNTER_COLUMNS.include?(column)
 
@@ -21,10 +86,17 @@ module Pgbus
 
         record.increment!(column)
 
-        just_finished = record.completed_jobs + record.discarded_jobs == record.total_jobs
-        record.update!(status: "finished", finished_at: Time.current) if just_finished && record.status != "finished"
+        return { record: record, just_finished: false } if Batch.executions_migrated?
+
+        # total_jobs grows per job while the block is still open (issue #423),
+        # so completed == total can be momentarily true on a pending batch.
+        # Only a processing batch may auto-finish; check_finished! at the end
+        # of the block covers the pending case.
+        counters_match = record.completed_jobs + record.discarded_jobs == record.total_jobs
+        just_finished = counters_match && record.status == "processing"
+        record.update!(status: "finished", finished_at: Time.current) if just_finished
 
-        { record: record, just_finished: just_finished && record.status == "finished" }
+        { record: record, just_finished: just_finished }
       end
     end
   end

data/app/models/pgbus/batch_execution.rb

@@ -0,0 +1,28 @@
+# frozen_string_literal: true
+
+module Pgbus
+  class BatchExecution < BusRecord
+    self.table_name = "pgbus_batch_executions"
+
+    # One row per outstanding batched job. Inserted before send_message so a
+    # crash cannot produce an untracked in-flight job. ON CONFLICT DO NOTHING
+    # makes a retry re-enqueue of the same ActiveJob id a no-op.
+    # Raw SQL rather than insert_all(unique_by:) — Rails resolves unique_by
+    # through the schema cache (issue #401).
+    # queue_name is the logical queue at insert (so the sweep can probe for a
+    # live message before the msg_id backfill lands); backfill! overwrites it
+    # with the physical target after send.
+    def self.insert_for!(batch_id:, job_id:, queue_name: nil)
+      connection.exec_query(
+        "INSERT INTO #{table_name} (batch_id, job_id, queue_name, created_at) " \
+        "VALUES ($1, $2, $3, $4) ON CONFLICT (job_id) DO NOTHING",
+        "BatchExecution Insert",
+        [batch_id, job_id, queue_name, Time.current]
+      )
+    end
+
+    def self.backfill!(job_id, msg_id:, queue_name:)
+      where(job_id: job_id).update_all(msg_id: msg_id, queue_name: queue_name)
+    end
+  end
+end

data/app/models/pgbus/blocked_execution.rb

@@ -22,7 +22,7 @@ module Pgbus
             LIMIT 1
             FOR UPDATE SKIP LOCKED
           )
-          RETURNING queue_name, payload
+          RETURNING queue_name, payload, priority
         SQL
         "Pgbus Blocked Release",
         [concurrency_key, now]
@@ -34,7 +34,7 @@ module Pgbus
       payload = row["payload"]
       payload = JSON.parse(payload) if payload.is_a?(String)
 
-      { queue_name: row["queue_name"], payload: payload }
+      { queue_name: row["queue_name"], payload: payload, priority: row["priority"] }
     end
   end
 end

data/app/models/pgbus/uniqueness_key.rb

@@ -8,16 +8,62 @@ module Pgbus
     # Atomically try to acquire a uniqueness lock via INSERT ... ON CONFLICT.
     # PostgreSQL's unique index on lock_key guarantees at most one caller wins.
     # Returns true if acquired (row inserted), false if already locked.
-    def self.acquire!(lock_key, queue_name:, msg_id:) # rubocop:disable Naming/PredicateMethod
-      connection.exec_query(
+    #
+    # reacquire_same_message: true (the :while_executing executor path) treats
+    # a conflict with a row that already points at THIS msg_id as acquired —
+    # it is this message's own previous attempt (PGMQ's visibility timeout
+    # guarantees nobody else holds the message), left behind by a crash. A
+    # conflict with a different msg_id is a genuine concurrent execution.
+    def self.acquire!(lock_key, queue_name:, msg_id:, reacquire_same_message: false) # rubocop:disable Naming/PredicateMethod
+      on_conflict = if reacquire_same_message && msg_id.to_i.positive?
+                      "DO UPDATE SET queue_name = EXCLUDED.queue_name " \
+                        "WHERE #{table_name}.msg_id = EXCLUDED.msg_id"
+                    else
+                      "DO NOTHING"
+                    end
+      result = connection.exec_query(
         "INSERT INTO #{table_name} (lock_key, queue_name, msg_id) " \
-        "VALUES ($1, $2, $3) ON CONFLICT (lock_key) DO NOTHING RETURNING lock_key",
+        "VALUES ($1, $2, $3) ON CONFLICT (lock_key) #{on_conflict} RETURNING lock_key, created_at",
         "UniquenessKey Acquire", [lock_key, queue_name, msg_id]
-      ).rows.any?
+      )
+      row = result.rows.first
+      return false unless row
+
+      # Ownership stamp for bind!: a successor acquire of the same key after
+      # this row is released must not inherit this enqueue's msg_id.
+      stamps = Thread.current[:pgbus_uniqueness_created_at] ||= {}
+      stamps[lock_key] = row[1]
+      true
+    end
+
+    # Bind a pre-produce lock to the real queue and PGMQ msg_id after send.
+    # Does not touch created_at — the reaper's age floor is from acquire time.
+    # Restricted to this enqueue's unbound row (msg_id=0, matching created_at
+    # when acquire! stamped one) so a completed job's bind cannot retarget a
+    # successor that re-acquired the key.
+    def self.bind!(lock_key, queue_name:, msg_id:)
+      stamps = Thread.current[:pgbus_uniqueness_created_at]
+      created_at = stamps&.delete(lock_key)
+      sql = "UPDATE #{table_name} SET queue_name = $2, msg_id = $3 " \
+            "WHERE lock_key = $1 AND msg_id = 0"
+      binds = [lock_key, queue_name, msg_id]
+      if created_at
+        sql += " AND created_at = $4"
+        binds << created_at
+      end
+      connection.exec_update(sql, "UniquenessKey Bind", binds)
+    end
+
+    # Drop the bind ownership stamp without touching the lock row. Used when
+    # this thread acquired the key but will not bind (concurrency :block, or
+    # enqueue returning after a failed send already rolled the lock back).
+    def self.clear_bind_stamp!(lock_key)
+      Thread.current[:pgbus_uniqueness_created_at]&.delete(lock_key)
     end
 
     # Release a uniqueness lock after job completion or DLQ.
     def self.release!(lock_key)
+      Thread.current[:pgbus_uniqueness_created_at]&.delete(lock_key)
       connection.exec_delete(
         "DELETE FROM #{table_name} WHERE lock_key = $1",
         "UniquenessKey Release", [lock_key]

data/app/views/pgbus/batches/_batches_table.html.erb

@@ -28,7 +28,7 @@
             <td data-label="Progress" class="px-4 py-3 text-sm">
               <div class="flex items-center space-x-2">
                 <div class="w-24 bg-gray-200 dark:bg-gray-700 rounded-full h-2">
-                  <div class="h-2 rounded-full <%= batch[:discarded_jobs].to_i > 0 ? 'bg-amber-500' : 'bg-green-500' %>"
+                  <div class="h-2 rounded-full <%= batch[:failed_jobs].to_i > 0 ? 'bg-amber-500' : 'bg-green-500' %>"
                        style="width: <%= batch[:progress_pct] %>%"></div>
                 </div>
                 <span class="text-xs text-gray-500 dark:text-gray-400 font-mono"><%= batch[:progress_pct] %>%</span>
@@ -36,8 +36,8 @@
             </td>
             <td data-label="Jobs" class="px-4 py-3 text-sm text-right font-mono text-gray-500 dark:text-gray-400">
               <%= batch[:completed_jobs] %>/<%= batch[:total_jobs] %>
-              <% if batch[:discarded_jobs].to_i > 0 %>
-                <span class="text-red-500">(<%= batch[:discarded_jobs] %> dlq)</span>
+              <% if batch[:failed_jobs].to_i > 0 %>
+                <span class="text-red-500">(<%= batch[:failed_jobs] %> failed)</span>
               <% end %>
             </td>
             <td data-label="Created" class="px-4 py-3 text-sm text-right text-gray-500 dark:text-gray-400">

data/app/views/pgbus/batches/show.html.erb

@@ -18,7 +18,7 @@
   <h2 class="text-sm font-semibold text-gray-900 dark:text-white mb-4"><%= t("pgbus.batches.show.progress") %></h2>
 
   <div class="w-full bg-gray-200 dark:bg-gray-700 rounded-full h-4 mb-3">
-    <div class="h-4 rounded-full transition-all duration-300 <%= @batch[:discarded_jobs].to_i > 0 ? 'bg-amber-500' : 'bg-green-500' %>"
+    <div class="h-4 rounded-full transition-all duration-300 <%= @batch[:failed_jobs].to_i > 0 ? 'bg-amber-500' : 'bg-green-500' %>"
          style="width: <%= @batch[:progress_pct] %>%"></div>
   </div>
 
@@ -32,12 +32,12 @@
       <dd class="mt-1 text-2xl font-semibold text-green-600 dark:text-green-400 font-mono"><%= @batch[:completed_jobs] %></dd>
     </div>
     <div>
-      <dt class="text-xs font-medium text-gray-500 dark:text-gray-400 uppercase"><%= t("pgbus.batches.show.discarded") %></dt>
-      <dd class="mt-1 text-2xl font-semibold <%= @batch[:discarded_jobs].to_i > 0 ? 'text-red-600 dark:text-red-400' : 'text-gray-900 dark:text-white' %> font-mono"><%= @batch[:discarded_jobs] %></dd>
+      <dt class="text-xs font-medium text-gray-500 dark:text-gray-400 uppercase"><%= t("pgbus.batches.show.failed") %></dt>
+      <dd class="mt-1 text-2xl font-semibold <%= @batch[:failed_jobs].to_i > 0 ? 'text-red-600 dark:text-red-400' : 'text-gray-900 dark:text-white' %> font-mono"><%= @batch[:failed_jobs] %></dd>
     </div>
     <div>
       <dt class="text-xs font-medium text-gray-500 dark:text-gray-400 uppercase"><%= t("pgbus.batches.show.remaining") %></dt>
-      <dd class="mt-1 text-2xl font-semibold text-gray-900 dark:text-white font-mono"><%= @batch[:total_jobs] - @batch[:completed_jobs] - @batch[:discarded_jobs] %></dd>
+      <dd class="mt-1 text-2xl font-semibold text-gray-900 dark:text-white font-mono"><%= @batch[:pending_jobs] || (@batch[:total_jobs] - @batch[:completed_jobs] - @batch[:failed_jobs]) %></dd>
     </div>
   </div>
 </div>
@@ -74,10 +74,10 @@
         <dd class="mt-1 text-sm font-mono text-gray-900 dark:text-white sm:col-span-2 sm:mt-0"><%= @batch[:on_success_class] %></dd>
       </div>
     <% end %>
-    <% if @batch[:on_discard_class] %>
+    <% if @batch[:on_failure_class] || @batch[:on_discard_class] %>
       <div class="px-4 py-3 sm:grid sm:grid-cols-3 sm:gap-4">
-        <dt class="text-sm font-medium text-gray-500 dark:text-gray-400"><%= t("pgbus.batches.show.on_discard") %></dt>
-        <dd class="mt-1 text-sm font-mono text-gray-900 dark:text-white sm:col-span-2 sm:mt-0"><%= @batch[:on_discard_class] %></dd>
+        <dt class="text-sm font-medium text-gray-500 dark:text-gray-400"><%= t("pgbus.batches.show.on_failure") %></dt>
+        <dd class="mt-1 text-sm font-mono text-gray-900 dark:text-white sm:col-span-2 sm:mt-0"><%= @batch[:on_failure_class] || @batch[:on_discard_class] %></dd>
       </div>
     <% end %>
     <% if @batch[:properties].present? %>

data/config/locales/da.yml

@@ -19,10 +19,10 @@ da:
         completed: Fuldført
         created_at: Oprettet den
         details: Detaljer
-        discarded: Kasseret
+        failed: Mislykkedes
         finished_at: Afsluttet den
         not_found: Batch ikke fundet
-        on_discard: Ved kassering
+        on_failure: Ved fejl
         on_finish: Ved afslutning
         on_success: Ved succes
         progress: Fremgang

data/config/locales/de.yml

@@ -19,10 +19,10 @@ de:
         completed: Abgeschlossen
         created_at: Erstellt am
         details: Details
-        discarded: Verworfen
+        failed: Fehlgeschlagen
         finished_at: Beendet am
         not_found: Stapel nicht gefunden
-        on_discard: Bei Verwerfung
+        on_failure: Im Fehlerfall
         on_finish: Bei Beendigung
         on_success: Bei Erfolg
         progress: Fortschritt

data/config/locales/en.yml

@@ -19,10 +19,10 @@ en:
         completed: Completed
         created_at: Created At
         details: Details
-        discarded: Discarded
+        failed: Failed
         finished_at: Finished At
         not_found: Batch not found
-        on_discard: On Discard
+        on_failure: On Failure
         on_finish: On Finish
         on_success: On Success
         progress: Progress

data/config/locales/es.yml

@@ -19,10 +19,10 @@ es:
         completed: Completados
         created_at: Creado el
         details: Detalles
-        discarded: Descartados
+        failed: Fallados
         finished_at: Finalizado el
         not_found: Lote no encontrado
-        on_discard: Al descartar
+        on_failure: Al fallar
         on_finish: Al finalizar
         on_success: Al tener éxito
         progress: Progreso

data/config/locales/fi.yml

@@ -19,10 +19,10 @@ fi:
         completed: Valmiit
         created_at: Luotu
         details: Tiedot
-        discarded: Hylätyt
+        failed: Epäonnistuneet
         finished_at: Valmistunut
         not_found: Erää ei löytynyt
-        on_discard: Hylkäyksessä
+        on_failure: Epäonnistumisessa
         on_finish: Valmistumisessa
         on_success: Onnistumisessa
         progress: Edistyminen

data/config/locales/fr.yml

@@ -19,10 +19,10 @@ fr:
         completed: Terminées
         created_at: Créé le
         details: Détails
-        discarded: Rejetées
+        failed: Échouées
         finished_at: Terminé le
         not_found: Lot non trouvé
-        on_discard: Au rejet
+        on_failure: En cas d'échec
         on_finish: À la fin
         on_success: Au succès
         progress: Progression

data/config/locales/it.yml

@@ -19,10 +19,10 @@ it:
         completed: Completati
         created_at: Creato il
         details: Dettagli
-        discarded: Scartati
+        failed: Falliti
         finished_at: Terminato il
         not_found: Lotto non trovato
-        on_discard: Allo scarto
+        on_failure: Al fallimento
         on_finish: Al termine
         on_success: Al successo
         progress: Progresso

data/config/locales/ja.yml

@@ -19,10 +19,10 @@ ja:
         completed: 完了
         created_at: 作成日時
         details: 詳細
-        discarded: 破棄済み
+        failed: 失敗
         finished_at: 完了日時
         not_found: バッチが見つかりません
-        on_discard: 破棄時
+        on_failure: 失敗時
         on_finish: 完了時
         on_success: 成功時
         progress: 進捗

data/config/locales/nb.yml

@@ -19,10 +19,10 @@ nb:
         completed: Fullført
         created_at: Opprettet den
         details: Detaljer
-        discarded: Forkastet
+        failed: Mislyktes
         finished_at: Avsluttet den
         not_found: Gruppe ikke funnet
-        on_discard: Ved forkastning
+        on_failure: Ved feil
         on_finish: Ved avslutning
         on_success: Ved suksess
         progress: Fremdrift

data/config/locales/nl.yml

@@ -19,10 +19,10 @@ nl:
         completed: Voltooid
         created_at: Aangemaakt op
         details: Details
-        discarded: Verworpen
+        failed: Mislukt
         finished_at: Voltooid op
         not_found: Groep niet gevonden
-        on_discard: Bij verwerping
+        on_failure: Bij mislukking
         on_finish: Bij voltooiing
         on_success: Bij succes
         progress: Voortgang

data/config/locales/pt.yml

@@ -19,10 +19,10 @@ pt:
         completed: Concluídos
         created_at: Criado em
         details: Detalhes
-        discarded: Descartados
+        failed: Falhados
         finished_at: Finalizado em
         not_found: Lote não encontrado
-        on_discard: Ao descartar
+        on_failure: Ao falhar
         on_finish: Ao finalizar
         on_success: Ao ter sucesso
         progress: Progresso

data/config/locales/sv.yml

@@ -19,10 +19,10 @@ sv:
         completed: Slutförda
         created_at: Skapad den
         details: Detaljer
-        discarded: Kasserade
+        failed: Misslyckades
         finished_at: Avslutad den
         not_found: Grupp hittades inte
-        on_discard: Vid kassering
+        on_failure: Vid misslyckande
         on_finish: Vid avslut
         on_success: Vid framgång
         progress: Framsteg

data/lib/generators/pgbus/add_batch_callback_jobs_generator.rb

@@ -0,0 +1,47 @@
+# frozen_string_literal: true
+
+require "rails/generators"
+require "rails/generators/active_record"
+require_relative "migration_path"
+
+module Pgbus
+  module Generators
+    class AddBatchCallbackJobsGenerator < Rails::Generators::Base
+      include ActiveRecord::Generators::Migration
+      include MigrationPath
+
+      source_root File.expand_path("templates", __dir__)
+
+      desc "Add jsonb callback-job columns to pgbus_batches (issue #415)"
+
+      class_option :database,
+                   type: :string,
+                   default: nil,
+                   desc: "Use a separate database for pgbus tables (e.g. --database=pgbus)"
+
+      def create_migration_file
+        migration_template "add_batch_callback_jobs.rb.erb",
+                           File.join(pgbus_migrate_path, "add_pgbus_batch_callback_jobs.rb")
+      end
+
+      def display_post_install
+        say ""
+        say "Pgbus configured batch callbacks installed!", :green
+        say ""
+        say "Next steps:"
+        say "  1. Run: rails db:migrate#{migrate_command_suffix}"
+        say "  2. Restart pgbus: bin/pgbus start"
+        say ""
+        say "You can now pass a configured ActiveJob instance to a batch:"
+        say "  Pgbus::Batch.new(on_finish: ReportJob.new.set(queue: :critical, wait: 5.minutes))"
+        say ""
+      end
+
+      private
+
+      def migration_version
+        "[#{ActiveRecord::Migration.current_version}]"
+      end
+    end
+  end
+end

data/lib/generators/pgbus/add_batch_executions_generator.rb

@@ -0,0 +1,44 @@
+# frozen_string_literal: true
+
+require "rails/generators"
+require "rails/generators/active_record"
+require_relative "migration_path"
+
+module Pgbus
+  module Generators
+    class AddBatchExecutionsGenerator < Rails::Generators::Base
+      include ActiveRecord::Generators::Migration
+      include MigrationPath
+
+      source_root File.expand_path("templates", __dir__)
+
+      desc "Add pgbus_batch_executions and rename batch failure columns (issue #414)"
+
+      class_option :database,
+                   type: :string,
+                   default: nil,
+                   desc: "Use a separate database for pgbus tables (e.g. --database=pgbus)"
+
+      def create_migration_file
+        migration_template "add_batch_executions.rb.erb",
+                           File.join(pgbus_migrate_path, "add_pgbus_batch_executions.rb")
+      end
+
+      def display_post_install
+        say ""
+        say "Pgbus batch execution-row tracking installed!", :green
+        say ""
+        say "Next steps:"
+        say "  1. Run: rails db:migrate#{migrate_command_suffix}"
+        say "  2. Restart pgbus: bin/pgbus start"
+        say ""
+      end
+
+      private
+
+      def migration_version
+        "[#{ActiveRecord::Migration.current_version}]"
+      end
+    end
+  end
+end