solid_queue-coalesce 0.1.0 → 0.2.0

data/UPGRADING.md

@@ -0,0 +1,76 @@
+# Upgrading to version 0.2.x
+This version replaces the gem's tables and the options of `coalesce`. There's no in-place migration for waiting calls, so drain them first. The generator skips an app that already has its migration, so this upgrade needs a migration of your own.
+
+Stop enqueuing jobs that use `coalesce`, and on 0.1.0 wait until none of those jobs are left. This query includes scheduled, blocked, running and failed jobs, so retry or discard the failed ones:
+
+```ruby
+SolidQueue::Job.where(finished_at: nil, class_name: %w[ ReindexDocumentsJob SendDigestJob ]).count
+```
+
+Stop your Solid Queue processes, add this migration to your queue database's migrations path, usually `db/queue_migrate`, and run it. It deletes the rows in both tables, including the calls of any finished jobs you've preserved:
+
+```ruby
+class ReplaceSolidQueueCoalesceTables < ActiveRecord::Migration[7.1]
+  def up
+    drop_table :solid_queue_coalesced_arguments
+    drop_table :solid_queue_coalesce_keys
+
+    create_table :solid_queue_coalesce_keys do |t|
+      t.string :key, null: false
+      t.bigint :job_id
+      t.integer :calls_count, null: false, default: 0
+      t.datetime :created_at, null: false
+
+      t.index :key, unique: true
+      t.index :job_id, unique: true
+      t.foreign_key :solid_queue_jobs, column: :job_id, on_delete: :cascade
+    end
+
+    create_table :solid_queue_coalesced_calls do |t|
+      t.bigint :job_id, null: false
+      t.text :arguments, null: false
+      t.datetime :created_at, null: false
+
+      t.index :job_id
+      t.foreign_key :solid_queue_jobs, column: :job_id, on_delete: :cascade
+    end
+  end
+
+  def down
+    raise ActiveRecord::IrreversibleMigration
+  end
+end
+```
+
+```bash
+bin/rails db:migrate:queue
+```
+
+If you use a single database, put the migration in `db/migrate` and run `bin/rails db:migrate`.
+
+Deploy these changes together with the migration:
+
+| 0.1.0 | 0.2.0 |
+| --- | --- |
+| `collect: true` | `on_conflict: :collect` |
+| `limit: 100` | `max_calls: 100` |
+| `flush_when_full: true`, which was the default | `on_full: :dispatch` |
+| `flush_when_full: false` | Remove it. A full job waits for its time by default |
+| `perform(array)` on a collecting job | `perform_calls(calls)`, and read `call.arguments` |
+| `coalesced?` | `coalesce_status`, which is `:discarded` or `:collected` for a call that joined a job |
+| `:dropped` in `coalesce.solid_queue` | `:discarded` |
+| `action` in `coalesce.solid_queue` | `status` |
+| `drop_coalesced_argument.solid_queue` | `drop_coalesced_call.solid_queue` |
+| `argument_id` in that event's payload | `call_id` |
+| `SolidQueue::CoalescedArgument` | `SolidQueue::CoalescedCall` |
+
+These behaviours change too:
+
+- Calls that refer to deleted records are skipped for that attempt, and their rows stay until the job is destroyed. Retrying or re-running the job reports each still-missing call again; a restored record's call can run. Allow for these payloads in finished-job retention. To decide which records to process in your own code, enqueue IDs and look them up in `perform_calls` instead of passing records as GlobalIDs.
+- `drop_coalesced_call.solid_queue` reports a skip on an attempt, with no deletion outcome. Count these events as skipped attempts, or deduplicate by `call_id` if you need unique calls.
+- If you relied on reaching `limit` to dispatch a scheduled job early, jobs now wait for their time unless you set `on_full: :dispatch`.
+- A retry runs all of the job's calls again, so make `perform_calls` idempotent. Cursors over `perform`'s array don't carry over.
+- `SolidQueue::Job::CoalescedInBatchError` is gone. Remove any `rescue` of it, because Ruby would raise `NameError` when an exception reaches that `rescue`. Coalesced jobs that a `rescue` kept out of a batch now join it.
+- The key column uses the database's default collation. If that's case-insensitive, keys that differ only by case now coalesce together. Use a case-sensitive collation in the migration above if they have to stay apart.
+
+If you run pre-release code from `main`, remove `call_deleted` from notification subscribers, log parsers and metric labels. This field was added after 0.1.0 and removed before 0.2.0.

data/app/models/solid_queue/coalesce_key.rb

@@ -1,10 +1,13 @@
 # frozen_string_literal: true
 
 module SolidQueue
-  # A row means "this job is the open job for this key": it exists only while
-  # its job exists and hasn't started. job_id is null only between the insert and
-  # the end of the admission that inserted it, so it's never visible to others.
-  class CoalesceKey < Record
+  # A row means "this job is the open job for this key": it exists only while its
+  # job exists, hasn't started and has room. job_id is null only between the insert
+  # and the end of the admission that inserted it, so it's never visible to others.
+  # calls_count is how many calls the job owns: at least one if it's collecting, and
+  # zero if it's discarding. A sealed or started job has no row, and so no count.
+  class CoalesceKey < Record # :nodoc:
+    # Enqueuing a job that uses +coalesce+ before the gem's migration has run.
     class PendingMigrations < StandardError
       def initialize(message = "The coalescing schema hasn't been installed yet. Run `bin/rails generate solid_queue:coalesce:install` to copy the migration to your application, and then `bin/rails db:migrate` to run it")
         super
@@ -12,28 +15,33 @@ module SolidQueue
     end
 
     belongs_to :job, optional: true
-    has_many :coalesced_arguments
-
-    scope :stranded, -> { left_joins(job: :failed_execution).merge(Job.finished.or(Job.where.not(failed_execution: { id: nil }))) }
 
     class << self
       # Until the app has run the gem's migration everything else works as before.
       def migrated?
-        @migrated ||= table_exists? && CoalescedArgument.table_exists?
+        @migrated ||= table_exists? && CoalescedCall.table_exists?
       end
 
       # Returns the row for this key, inserted if nobody holds it, and locked so
-      # a job start can't claim or delete it until this transaction commits. A row
+      # a job start can't delete it until this transaction commits. A row
       # with a null job_id is ours: the caller has to create the job for it.
-      def admit(key)
+      def admit(key, lock_clause = share_lock)
         loop do
           insert_unless_held(key)
 
-          admitted = lock(share_lock).find_by(key: key)
+          admitted = lock(lock_clause).find_by(key: key)
           return admitted if admitted
         end
       end
 
+      # Admission for a collecting job, which goes on to change the row. Nearly every
+      # call finds a row, so it tries the lock first. On MySQL, an insert that hits a
+      # duplicate locks the row and the gap next to it, and the calls queued on a row
+      # that gets sealed deadlock on those locks.
+      def admit_exclusively(key)
+        lock_if_present(key) || admit(key, true)
+      end
+
       # Locks through the key's index, like admission. Going through the job_id
       # index, even by deleting by it, takes InnoDB's locks on this row's indexes in
       # the opposite order to admission, and that deadlocks.
@@ -43,16 +51,30 @@ module SolidQueue
         end
       end
 
-      def release_stranded(batch_size)
-        stranded.limit(batch_size).pluck(:key).each do |key|
-          transaction do
-            coalesce_key = lock.find_by(key: key)
-            coalesce_key.release(claiming: !coalesce_key.job.finished?) if coalesce_key&.stranded?
-          end
+      # Lets go of a job from when the key's class collected. Checks again under the
+      # lock, because another call may have got here first. Doesn't check for the row
+      # first like lock_if_present: inside an older MySQL snapshot that plain read can
+      # miss a row that admission's locking read keeps finding, and we'd never seal it.
+      def seal_collecting(key)
+        coalesce_key = lock.find_by(key: key)
+        coalesce_key.destroy! if coalesce_key&.collecting?
+      end
+
+      # For jobs deleted in bulk, which skips their destroy callback. In the order of
+      # the keys, so that two of these can't deadlock with each other.
+      def release_all(job_ids)
+        where(job_id: job_ids).order(:key).pluck(:key, :job_id).each do |key, job_id|
+          lock.find_by(key: key, job_id: job_id)&.destroy!
         end
       end
 
       private
+        # MySQL's check is because locking a row that isn't there takes a gap lock,
+        # and that blocks the insert of every other new key.
+        def lock_if_present(key)
+          lock.find_by(key: key) if supports_insert_conflict_target? || exists?(key: key)
+        end
+
         # MariaDB and MySQL before 8 don't have FOR SHARE, and this means the same on MySQL 8
         def share_lock
           connection_pool.with_connection { |connection| connection.adapter_name == "PostgreSQL" } ? "FOR SHARE" : "LOCK IN SHARE MODE"
@@ -62,71 +84,39 @@ module SolidQueue
         # a duplicate would abort an enclosing transaction on PostgreSQL. On MySQL it
         # also avoids a deadlock: a plain INSERT that hits a duplicate takes a shared
         # lock, and two of those on a row being deleted can't both go on to insert.
+        # created_at starts the max_wait clock, so it's set here and not by the database:
+        # MySQL would write it in the server's time zone, whatever the app's is.
         def insert_unless_held(key)
           if supports_insert_conflict_target?
-            insert({ key: key }, unique_by: :key)
+            insert({ key: key, created_at: Time.current }, unique_by: :key)
           else
-            insert({ key: key })
+            insert({ key: key, created_at: Time.current })
           end
         end
     end
 
-    def stranded?
-      job.finished? || job.failed?
+    # How many more calls the open job can take. A job created while its class was
+    # discarding runs from its own arguments, so a call that joined it would replace them.
+    def capacity(max_calls)
+      discarding? ? 0 : max_calls - calls_count
     end
 
-    def collect(active_jobs)
-      CoalescedArgument.insert_all(active_jobs.map { |active_job| argument_attributes_from(active_job) })
+    def collecting?
+      calls_count.positive?
     end
 
-    # Early flush. Bounded by the limit, however big the backlog is. A concurrent
-    # dispatcher holding the scheduled execution is dispatching it already, and if
-    # it's been dispatched before we got here there's nothing to lock.
-    def flush_if_full(limit)
-      if at_least?(limit)
-        scheduled = ScheduledExecution.where(job_id: job_id).non_blocking_lock.pluck(:job_id)
-        ScheduledExecution.dispatch_jobs(scheduled) if scheduled.any?
-      end
+    def discarding?
+      job_id.present? && calls_count.zero?
     end
 
-    # Hands the waiting rows over as the job leaves the queue. Requires this row
-    # to be locked: that's what waits for in-flight appends, and it serializes this
-    # with other starts. Reads of the rows lock too, because on MySQL a plain read
-    # could use a snapshot older than the appends we waited for, and deleting this
-    # row with rows we didn't see would cascade to them.
-    #
-    # A finished job won't run again, so it claims nothing when there's a next job
-    # that can process the rows. Without a limit there can't be one: the class that
-    # would process them is gone, or stopped collecting.
-    def release(claiming: true)
-      limit = job.coalesce_limit
-      claim(claiming || limit.nil? ? limit : 0)
-
-      if oldest = coalesced_arguments.order(:id).lock.first
-        next_job = job.enqueue_next_coalesced(oldest, due_now: next_job_due_now?(limit))
-        update_columns(job_id: next_job.id)
-      else
+    # Records calls added to the job. A full job is sealed by deleting this row, so
+    # the next call opens a new one, and nothing about the sealed job changes.
+    def fill(job_id, count, max_calls)
+      if count >= capacity(max_calls)
         destroy!
+      else
+        update_columns(job_id: job_id, calls_count: calls_count + count)
       end
     end
-
-    private
-      def next_job_due_now?(limit)
-        limit.present? && job.coalesce_flush_when_full? && at_least?(limit)
-      end
-
-      def argument_attributes_from(active_job)
-        { coalesce_key_id: id, arguments: active_job.serialize["arguments"], scheduled_at: active_job.scheduled_at }
-      end
-
-      def at_least?(count)
-        coalesced_arguments.order(:id).offset(count - 1).limit(1).pick(:id).present?
-      end
-
-      # A nil limit means we can't know it, so nothing is left behind
-      def claim(limit)
-        claimed = limit ? coalesced_arguments.where(id: coalesced_arguments.order(:id).limit(limit).lock.pluck(:id)) : coalesced_arguments
-        claimed.update_all(job_id: job_id, coalesce_key_id: nil)
-      end
   end
 end

data/app/models/solid_queue/coalesced_call.rb

@@ -0,0 +1,22 @@
+# frozen_string_literal: true
+
+module SolidQueue
+  # The Active Job payload of one collected perform_later call. It belongs to one
+  # job from the moment it's admitted, across that job's retries, until the job is
+  # destroyed.
+  class CoalescedCall < Record
+    belongs_to :job
+
+    serialize :arguments, coder: JSON
+
+    ##
+    # :singleton-method: waiting
+    #
+    # Calls whose job hasn't finished, isn't running and hasn't failed, for measuring
+    # the backlog.
+    #
+    #   SolidQueue::CoalescedCall.waiting.count
+    #   SolidQueue::CoalescedCall.waiting.minimum(:created_at)
+    scope :waiting, -> { where(job_id: Job.where(finished_at: nil).where.missing(:claimed_execution, :failed_execution).select(:id)) }
+  end
+end

data/app/models/solid_queue/job/coalescible.rb

@@ -2,41 +2,53 @@
 
 module SolidQueue
   class Job
-    class CoalescedInBatchError < EnqueueError; end
-
-    module Coalescible
+    module Coalescible # :nodoc: all
       extend ActiveSupport::Concern
 
       included do
-        has_many :coalesced_arguments
+        has_many :coalesced_calls
+
+        # Before anything else, so the key is locked before the job, as in admission.
+        # Left to the cascade, they'd lock in the opposite order and deadlock.
+        before_destroy :release_coalesce_key, prepend: true
       end
 
       class_methods do
-        # Every call in active_jobs has the same key. Returns the job if the first
-        # call created one, nil if the calls joined the open job.
-        def enqueue_coalesced(key, active_jobs)
+        # Every call in active_jobs has the same key. Returns the job if a call
+        # created one, nil if the calls joined the open job.
+        def enqueue_coalesced(key, active_jobs, payloads = nil)
+          return collect_coalesced(key, active_jobs, payloads || active_jobs.map(&:serialize)) if active_jobs.first.class.coalesce_collect?
+
           enqueued_job, open_job_id = raising_enqueue_errors do
-            retrying_deadlocks { transaction { admit_coalesced(key, active_jobs) } }
+            retrying_deadlocks { admit_discarding(key, active_jobs) }
           end
 
-          joined = enqueued_job ? active_jobs.drop(1) : active_jobs
-          record_coalesced(key, joined, open_job_id)
+          record_coalesced(key, active_jobs, enqueued_job&.persisted?, open_job_id)
 
           enqueued_job
         end
 
-        def enqueue_all_coalesced(active_jobs)
-          active_jobs.group_by(&:coalesce_key).sort_by(&:first).each do |key, calls|
-            enqueue_coalesced(key, calls)
+        # So a key or a payload that can't be built raises before anything is written.
+        # A repeated :discard call is never stored, so its arguments don't have to
+        # serialize, as with perform_later. The first one's only get checked here,
+        # because its job is built later, if there's no open job to find.
+        def prepare_all_coalesced(active_jobs)
+          active_jobs.group_by(&:coalesce_key).sort_by(&:first).map do |key, calls|
+            if calls.first.class.coalesce_collect?
+              [ key, calls, calls.map(&:serialize) ]
+            else
+              calls.first.serialize
+              [ key, calls ]
+            end
           end
         end
 
-        def ensure_coalescible(active_jobs)
-          raise CoalesceKey::PendingMigrations unless CoalesceKey.migrated?
+        def enqueue_all_coalesced(prepared_calls)
+          prepared_calls.each { |key, calls, payloads| enqueue_coalesced(key, calls, payloads) }
+        end
 
-          if Batch.current_batch_id || active_jobs.any?(&:batch_id)
-            raise CoalescedInBatchError, "#{active_jobs.first.class.name} coalesces its calls, so it can't be enqueued as part of a batch: a call that joins an open job adds no job to the batch"
-          end
+        def ensure_coalescible
+          raise CoalesceKey::PendingMigrations unless CoalesceKey.migrated?
         end
 
         MAX_COALESCE_DEADLOCK_RETRIES = 5
@@ -65,12 +77,12 @@ module SolidQueue
           # block creates the new job, and sets provider_job_id to it.
           def continuing_coalesced_run(active_job)
             previous_job_id = active_job.provider_job_id
-            moving_calls = previous_job_id && active_job.class.coalesce_collect && CoalesceKey.migrated?
+            moving_calls = previous_job_id && active_job.class.coalesce_collect? && CoalesceKey.migrated?
 
             enqueued_job = raising_enqueue_errors do
               transaction do
                 yield.tap do |job|
-                  CoalescedArgument.where(job_id: previous_job_id).update_all(job_id: job.id) if moving_calls && job.persisted?
+                  CoalescedCall.where(job_id: previous_job_id).update_all(job_id: job.id) if moving_calls && job.persisted?
                 end
               end
             end
@@ -93,112 +105,212 @@ module SolidQueue
             raise enqueue_error
           end
 
-          def admit_coalesced(key, active_jobs)
-            job_class = active_jobs.first.class
+          # A key with calls belongs to a job from when the class collected. A call
+          # can't count on that job, which fails without running it, so it seals the
+          # job and tries again. Sealing is outside the transaction that found it: every
+          # call waiting on the key holds the same share lock, and upgrading it deadlocks
+          # with all of them. Inside the caller's own transaction it can't be outside,
+          # so that deadlock can reach the caller, as in retrying_deadlocks.
+          def admit_discarding(key, active_jobs)
+            loop do
+              admitted = transaction { admit_unless_collecting(key, active_jobs) }
+              return admitted if admitted
+
+              transaction { CoalesceKey.seal_collecting(key) }
+            end
+          end
+
+          def admit_unless_collecting(key, active_jobs)
             coalesce_key = CoalesceKey.admit(key)
-            enqueued_job = create_for_coalesce_key(coalesce_key, active_jobs.first) if coalesce_key.job_id.nil?
 
-            if job_class.coalesce_collect && coalesce_key.job_id
-              coalesce_key.collect(active_jobs)
-              coalesce_key.flush_if_full(job_class.coalesce_limit) if job_class.coalesce_flush_when_full
-            end
+            unless coalesce_key.collecting?
+              enqueued_job = create_for_coalesce_key(coalesce_key, active_jobs.first) if coalesce_key.job_id.nil?
+              debounce_coalesced(coalesce_key, coalesce_key.job_id, enqueued_job ? active_jobs.drop(1) : active_jobs)
 
-            [ enqueued_job, coalesce_key.job_id ]
+              [ enqueued_job, coalesce_key.job_id ]
+            end
           end
 
           def create_for_coalesce_key(coalesce_key, active_job)
-            create!(**attributes_from_active_job(active_job)).tap do |job|
+            attributes = attributes_from_active_job(active_job)
+
+            create!(**attributes, arguments: attributes[:arguments].merge("coalesce_on_conflict" => "discard")).tap do |job|
               # limits_concurrency with on_conflict: :discard destroys the job right away
               if job.persisted?
                 coalesce_key.update_columns(job_id: job.id)
               else
                 coalesce_key.destroy!
               end
-
-              active_job.provider_job_id = job.id if job.persisted?
-              active_job.successfully_enqueued = job.persisted?
             end
           end
 
-          def record_coalesced(key, active_jobs, open_job_id)
-            active_jobs.each do |active_job|
-              collected = active_job.class.coalesce_collect
+          # Takes one job's worth of calls at a time, each in its own transaction.
+          # A call's outcome is set once its transaction is over, so a deadlock retry
+          # can't leave one behind. Serialized once and before any job exists, so the
+          # first call's payload is like every other call's, and a retry doesn't take
+          # a new snapshot.
+          def collect_coalesced(key, active_jobs, payloads)
+            max_calls = active_jobs.first.class.coalesce_max_calls
+            pending = active_jobs.zip(payloads)
+            enqueued_job = nil
+
+            until pending.empty?
+              admitted, created_job, job_id = raising_enqueue_errors do
+                retrying_deadlocks { transaction { admit_collected(key, pending, max_calls) } }
+              end
 
-              active_job.provider_job_id = open_job_id
-              active_job.successfully_enqueued = collected
-              active_job.coalesced = open_job_id.present?
+              record_coalesced(key, pending.shift(admitted).map(&:first), created_job.present?, job_id)
+              enqueued_job ||= created_job
+            end
 
-              if open_job_id
-                SolidQueue.instrument(:coalesce, key: key, job_id: open_job_id, active_job_id: active_job.job_id, action: collected ? :collected : :dropped)
+            enqueued_job
+          end
+
+          # Returns how many of the pending calls went in, and where. It can be none:
+          # a smaller max_calls can leave the open job over its capacity, and a job
+          # from when the class discarded has none, and that only seals it.
+          def admit_collected(key, pending, max_calls)
+            coalesce_key = CoalesceKey.admit_exclusively(key)
+            calls = pending.first([ coalesce_key.capacity(max_calls), 0 ].max)
+
+            if calls.empty?
+              coalesce_key.destroy!
+              [ 0 ]
+            else
+              created_job = create_collecting(*calls.first) if coalesce_key.job_id.nil?
+              job_id = coalesce_key.job_id || created_job.id
+
+              CoalescedCall.insert_all(calls.map { |_, payload| { job_id: job_id, arguments: payload } })
+              coalesce_key.fill(job_id, calls.size, max_calls)
+
+              if coalesce_key.destroyed? && calls.first.first.class.coalesce_on_full == :dispatch
+                dispatch_full(job_id)
+              else
+                debounce_coalesced(coalesce_key, job_id, calls.drop(created_job ? 1 : 0).map(&:first))
               end
+
+              [ calls.size, created_job, job_id ]
             end
           end
-      end
 
-      def coalesced?
-        job_class.try(:coalesce_key).present?
-      end
+          # The time moves too, or retrying the job after it fails would wait for the old one.
+          def dispatch_full(job_id)
+            scheduled = lock_scheduled(ScheduledExecution.where(job_id: job_id))
 
-      def coalesce_limit
-        job_class.try(:coalesce_limit)
-      end
+            if scheduled.any?
+              where(id: scheduled).update_all(scheduled_at: Time.current)
+              ScheduledExecution.dispatch_jobs(scheduled)
+            end
+          end
 
-      def coalesce_flush_when_full?
-        job_class.try(:coalesce_flush_when_full)
+          # The calls that joined the job move it once, to the latest of their times,
+          # capped at max_wait from when its key opened. Each open job has its own key,
+          # so each has its own clock. A sealed job has no key, so nothing moves it again.
+          def debounce_coalesced(coalesce_key, job_id, joined_jobs)
+            job_class = joined_jobs.first&.class
+
+            if job_id && job_class&.coalesce_debounce
+              scheduled_at = joined_jobs.map(&:scheduled_at).max
+              scheduled_at = [ scheduled_at, coalesce_key.created_at + job_class.coalesce_max_wait ].min if job_class.coalesce_max_wait
+
+              move_later(job_id, scheduled_at)
+            end
+          end
+
+          # Only an execution more than a second before the new time matches. That keeps
+          # moves later-only, and saves a hot key from writing on every call. The time
+          # moves on the job too, because a failed job's retry goes by the job's. If this
+          # skips a row that's held, the job runs early, and the key still covers the call.
+          def move_later(job_id, scheduled_at)
+            scheduled = lock_scheduled(ScheduledExecution.where(job_id: job_id, scheduled_at: ...(scheduled_at - 1.second)))
+
+            if scheduled.any?
+              where(id: scheduled).update_all(scheduled_at: scheduled_at)
+              ScheduledExecution.where(job_id: scheduled).update_all(scheduled_at: scheduled_at)
+            end
+          end
+
+          # A job that's ready, blocked or claimed has no scheduled execution, and has
+          # nothing to do. SKIP LOCKED because we hold the key, and a discard holds the
+          # execution while it waits for the key. Whoever holds the row is dispatching
+          # or discarding the job already. On MySQL, locking a row that isn't there takes
+          # a gap lock that blocks other jobs being scheduled, so it checks first.
+          def lock_scheduled(executions)
+            executions.exists? ? executions.non_blocking_lock.pluck(:job_id) : []
+          end
+
+          # Not attributes_from_active_job, which serializes the call again, and that
+          # could raise after earlier jobs committed.
+          def create_collecting(active_job, payload)
+            attributes = {
+              queue_name: active_job.queue_name || singleton_class::DEFAULT_QUEUE_NAME,
+              active_job_id: active_job.job_id,
+              priority: active_job.priority || singleton_class::DEFAULT_PRIORITY,
+              scheduled_at: active_job.scheduled_at,
+              class_name: active_job.class.name,
+              arguments: payload.merge("coalesce_on_conflict" => "collect"),
+              concurrency_key: active_job.concurrency_key
+            }
+            attributes[:batch_id] = active_job.batch_id if Batch.migrated?
+
+            create!(**attributes)
+          end
+
+          # A call has an outcome only if it was admitted: when on_conflict: :discard
+          # destroys the job that was just created, nothing holds the key.
+          def record_coalesced(key, active_jobs, created, job_id)
+            active_jobs.each_with_index do |active_job, index|
+              collect = active_job.class.coalesce_collect?
+              created_by_this_call = created && index.zero?
+
+              active_job.provider_job_id = job_id
+              active_job.successfully_enqueued = created_by_this_call || collect
+              active_job.send(:coalesce_status=, job_id && (created_by_this_call ? :created : (collect ? :collected : :discarded)))
+            end
+
+            # After every outcome, because a subscriber can raise. These calls are stored,
+            # so one without an outcome would look like it needs enqueuing again.
+            if job_id
+              active_jobs.each do |active_job|
+                SolidQueue.instrument(:coalesce, status: active_job.coalesce_status, key: key, job_id: job_id, active_job_id: active_job.job_id, class_name: active_job.class.name)
+              end
+            end
+          end
       end
 
       # The start step. Returns what the Active Job needs to know about this run,
       # which stays in memory instead of going into its serialized data.
+      #
+      # Closing commits before the calls are read, in a statement of their own. The
+      # read that finds the key in lock_for_release sets the snapshot of a MySQL
+      # REPEATABLE READ transaction, so a read in the same transaction could miss
+      # calls committed while it waited for the lock.
       def start_coalesced_run
         return unless coalesced? && CoalesceKey.migrated?
 
-        self.class.retrying_deadlocks do
-          transaction do
-            release_coalesce_key
-            { "calls" => coalesced_arguments.order(:id).pluck(:id, :arguments), "concurrency_key" => concurrency_key }
-          end
-        end
-      end
+        self.class.retrying_deadlocks { transaction { release_coalesce_key } }
 
-      # The next job for the same key, picking up from the rows the job that's
-      # starting left waiting. Its arguments are the raw ones in the oldest row:
-      # deserializing them here would let a deleted record break the handover.
-      def enqueue_next_coalesced(oldest, due_now:)
-        due_at = (oldest.scheduled_at unless due_now) || Time.current
-        active_job_id = SecureRandom.uuid
-
-        self.class.create!(
-          queue_name: queue_name,
-          class_name: class_name,
-          priority: priority,
-          active_job_id: active_job_id,
-          scheduled_at: due_at,
-          concurrency_key: concurrency_key,
-          arguments: next_coalesced_arguments(active_job_id, oldest, due_at)
-        )
+        { "calls" => coalesced_calls.order(:id).pluck(:id, :arguments), "concurrency_key" => concurrency_key }
       end
 
       private
-        def release_coalesce_key
-          return unless CoalesceKey.migrated?
+        # Whether this job's class coalesces
+        def coalesced?
+          job_class.try(:coalesce_key).present?
+        end
 
-          if job_class&.coalesce_key && !job_class.coalesce_collect
-            CoalesceKey.lock_for_release(id)&.destroy!
-          elsif job_class.nil? || coalesced?
-            CoalesceKey.lock_for_release(id)&.release
-          end
+        # Admission marks a job before giving it a key, and the class covers jobs from
+        # before the marker. Anything that can't answer gets the read, so that failing
+        # or discarding a job still works when its payload or its class can't be loaded.
+        def may_hold_coalesce_key?
+          !arguments.is_a?(Hash) || arguments.key?("coalesce_on_conflict") || coalesced?
+        rescue StandardError, ScriptError
+          true
         end
 
-        def next_coalesced_arguments(active_job_id, oldest, due_at)
-          arguments.except("continuation", "resumptions").merge(
-            "job_id" => active_job_id,
-            "provider_job_id" => nil,
-            "arguments" => oldest.arguments,
-            "executions" => 0,
-            "exception_executions" => {},
-            "enqueued_at" => Time.now.utc.iso8601(9),
-            "scheduled_at" => due_at.utc.iso8601(9)
-          )
+        # A sealed job has no row, and lock_for_release won't touch a newer job's
+        def release_coalesce_key
+          CoalesceKey.lock_for_release(id)&.destroy! if may_hold_coalesce_key? && CoalesceKey.migrated?
         end
     end
   end