deimos-ruby 2.5.4 → 2.6.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: cd09242a06bc94c004fb1fbeff2f6462e3b94d9aefce48894a1378eafdad1aa6
-  data.tar.gz: 1d9c45e6048b08216fc9ef6c424c204c02d14d661e33a516da7c248cc8667e6a
+  metadata.gz: 3399390b0e3dc85c67c95c4d633d3ce65efb80258848e73c6b0fa5d9244db13e
+  data.tar.gz: d236d19cd353f581121dbf561cba34ff78a192bd636ac992fcb1ad87b5913375
 SHA512:
-  metadata.gz: 34bed6ee88463b0345bb3d361bc489d57fdc1acbe472937b49e8ef6ae552e4cf9b9ca390606f180aeeebfb5864646dd45dd4fe3b1a6e35fa752d9d3fa65ee6c3
-  data.tar.gz: cd7df7022342e21b40c1d68b6696f0026bbaff946e34b04de7cc4ca46fc82930c0fa19316150196ddf5bf19bf1f908116ffedc380e332a5c9352d954e063bf69
+  metadata.gz: 502f900a5d5220c55d13bbc50ab329c826d3da953b14adfbaaaf1668e229d2ede46268729ed3c7bc631045feb6927fe85126a6b990f28b4f00630efdb0bb5501
+  data.tar.gz: 44d082485bfb68be5b2c49f228ab29626eb5c7b80b98873284a7844fba55c881ac805d0ce9db2c9d2f44f8b68eb781f541a5d1d0099b94f2f7ee236a29197093

data/CHANGELOG.md

@@ -5,7 +5,11 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
-## Unreleased
+## 2.6.0 - 2026-08-13
+
+- Breaking: a failed batch database write is now retried one record at a time, so one unpersistable record no longer loses the whole batch. `Deimos::BatchFallbackError` is raised naming the keys that still failed.
+- Feature: new `batch_message_fallback` topic setting, default true. Set it to false to keep the previous all-or-nothing behaviour.
+- Feature: new `deimos.batch_consumption.initial_failure` instrumentation event, fired when a batch write fails and the records are about to be retried individually.
 
 ## 2.5.4 - 2026-07-15
 

data/README.md

@@ -323,6 +323,12 @@ produced by Phobos and RubyKafka):
 * `deimos.batch_consumption.invalid_records` - sent when the consumer has rejected records returned from `filtered_records`. Limited by `max_db_batch_size`.
   * consumer: class of the consumer that rejected these records
   * records: Rejected records (of type `Deimos::ActiveRecordConsume::BatchRecord`)
+* `deimos.batch_consumption.initial_failure` - sent when a bulk database write failed and the consumer is about to retry it one record at a time. Fired once per failing write, and fired even if nothing ends up being salvaged.
+  * consumer: class of the consumer that is retrying
+  * topic: name of the topic being consumed
+  * operation: `:upsert_records` or `:remove_records`
+  * count: number of records or messages in the write that failed
+  * error: the exception the bulk write raised
   
 # Consumers
 
@@ -565,6 +571,36 @@ By default, batches will be compacted before processing, i.e. only the last
 message for each unique key in a batch will actually be processed. To change
 this behaviour, call `compacted false` inside of your consumer definition.
 
+#### Failure handling
+
+Because a batch is written in a single SQL statement inside a single transaction, one record
+which cannot be persisted would otherwise roll back every other record written alongside it.
+To avoid losing an entire batch to one bad record, Deimos retries the database write one record
+at a time when the bulk write fails. Everything that can be saved on its own is saved, and once
+the whole batch has been attempted a single `Deimos::BatchFallbackError` is raised. Its
+`failures` attribute holds the `[message, error]` pairs that could not be saved, and its message
+names their keys. From there it is handled by `reraise_errors`/`fatal_error` like any other
+consumer error.
+
+Only the database write is retried. Message-level work — `pre_process`, building and filtering
+records, and the `valid_records`/`invalid_records` events — happens exactly once for the group
+either way, so nothing is applied twice. The
+`deimos.batch_consumption.initial_failure` event fires when a bulk write fails and the records
+are about to be retried individually.
+
+The original error is reraised unchanged, rather than wrapped, in three cases:
+
+* Writes of a single record, where there is nothing to isolate.
+* Deadlocks and lock wait timeouts, which are transient contention on the whole write (already
+  retried by `DeadlockRetry`) rather than a problem with a particular record.
+* When nothing could be saved on its own. Isolating salvaged nothing, so the failure was
+  never about one bad record - it is something systemic, such as an unreachable database - and
+  the original error describes that better.
+
+Retrying row by row trades throughput for durability, which is not always the right trade for a
+very large batch on a hot topic. Set `batch_message_fallback false` on the topic to keep
+the previous all-or-nothing behaviour and let the whole batch fail.
+
 A sample batch consumer would look as follows:
 
 ```ruby

data/docs/CONFIGURATION.md

@@ -133,6 +133,7 @@ The following are additional settings that can be added to the `topic` block in
 | replace_associations     | true              | If false, append to associations in multi-table imports rather than replacing them.                                                                                                                                                                                                                                                                                                                                        |
 | bulk_import_id_generator | nil               | Block to determine the bulk_import_id generated during bulk consumption. If no block is specified the provided/default block from the consumers configuration will be used.                                                                                                                                                                                                                                                |
 | save_associations_first  | false             | Whether to save associated records of primary class prior to upserting primary records. Foreign key of associated records are assigned to the record class prior to saving the record class                                                                                                                                                                                                                                |
+| batch_message_fallback   | true              | When a bulk database write fails, retry it one record at a time so that a single failed record doesn't lose the whole batch. Set to false to skip this behavior, e.g. where retrying row by row is too slow.                                                                                                                                                                                                                |
 
 ### Defining Consumers
 

data/lib/deimos/active_record_consume/batch_consumption.rb

@@ -24,6 +24,7 @@ module Deimos
       # in the same operation as they would interfere with each other. Thus
       # they are split
       # @return [void]
+      # @raise [BatchFallbackError] if some messages could not be saved even on their own.
       def consume_batch
         filtered = messages.select { |message| process_message?(message) }
         skipped_count = messages.size - filtered.size
@@ -39,11 +40,15 @@ module Deimos
         Deimos.config.tracer.active_span.set_tag('topic', tag)
 
         Karafka.monitor.instrument('deimos.ar_consumer.consume_batch', { topic: tag }) do
-          if @compacted && deimos_messages.map(&:key).compact.any?
-            update_database(compact_messages(deimos_messages))
-          else
-            uncompacted_update(deimos_messages)
-          end
+          failures = if @compacted && deimos_messages.map(&:key).compact.any?
+                       update_database(compact_messages(deimos_messages))
+                     else
+                       uncompacted_update(deimos_messages)
+                     end
+
+          # Raised only once every slice and group has been attempted, so that a message which
+          # can't be persisted never stops the rest of the batch from being saved.
+          raise BatchFallbackError, failures if failures.any?
         end
 
         post_process_batch(deimos_messages)
@@ -128,45 +133,40 @@ module Deimos
       # All messages are split into slices containing only unique keys, and
       # each slice is handles as its own batch.
       # @param messages [Array<Message>] List of messages.
-      # @return [void]
+      # @return [Array<Array(Message, StandardError)>] messages that could not be saved even on
+      #   their own, paired with their error.
       def uncompacted_update(messages)
         BatchSlicer.
           slice(messages).
-          each(&method(:update_database))
+          flat_map(&method(:update_database))
       end
 
       # Perform database operations for a group of messages.
       # All messages with payloads are passed to upsert_records.
       # All tombstones messages are passed to remove_records.
       # @param messages [Array<Message>] List of messages.
-      # @return [void]
+      # @return [Array<Array(Message, StandardError)>] messages that could not be saved even on
+      #   their own, paired with their error.
       def update_database(messages)
         # Find all upserted records (i.e. that have a payload) and all
         # deleted record (no payload)
         removed, upserted = messages.partition { |m| delete_record?(m) }
 
         max_db_batch_size = self.class.config[:max_db_batch_size]
-        if upserted.any?
-          if max_db_batch_size
-            upserted.each_slice(max_db_batch_size) { |group| upsert_records(group) }
-          else
-            upsert_records(upserted)
-          end
-        end
+        upsert_groups = max_db_batch_size ? upserted.each_slice(max_db_batch_size).to_a : [upserted]
+        remove_groups = max_db_batch_size ? removed.each_slice(max_db_batch_size).to_a : [removed]
 
-        return if removed.empty?
-
-        if max_db_batch_size
-          removed.each_slice(max_db_batch_size) { |group| remove_records(group) }
-        else
-          remove_records(removed)
-        end
+        upsert_groups.reject(&:empty?).flat_map { |group| upsert_records(group) } +
+          remove_groups.reject(&:empty?).flat_map { |group| remove_group(group) }
       end
 
-      # Upsert any non-deleted records
+      # Upsert any non-deleted records. Everything that operates on the messages - pre-processing,
+      # building and filtering records, instrumentation - happens exactly once here; only the
+      # database write is retried if it fails, so nothing gets applied twice.
       # @param messages [Array<Message>] List of messages for a group of
       # records to either be updated or inserted.
-      # @return [void]
+      # @return [Array<Array(Message, StandardError)>] messages whose records could not be saved
+      #   even on their own, paired with their error.
       def upsert_records(messages)
         record_list = build_records(messages)
         invalid = filter_records(record_list)
@@ -176,7 +176,7 @@ module Deimos
                                        consumer: self.class
                                      })
         end
-        return if record_list.empty?
+        return [] if record_list.empty?
 
         key_col_proc = self.method(:key_columns).to_proc
         col_proc = self.method(:columns).to_proc
@@ -188,10 +188,104 @@ module Deimos
                                   bulk_import_id_generator: self.bulk_import_id_generator,
                                   save_associations_first: self.save_associations_first,
                                   bulk_import_id_column: self.bulk_import_id_column)
+        saved, failures = save_record_list(record_list, updater)
         Karafka.monitor.instrument('deimos.batch_consumption.valid_records', {
-                                     records: updater.mass_update(record_list),
+                                     records: saved,
                                      consumer: self.class
                                    })
+        failures
+      end
+
+      # Write a list of records to the database. The list is written in a single statement inside
+      # a single transaction, so one record which can't be persisted would otherwise take down
+      # every other record written alongside it. Unless the topic turns
+      # `batch_message_fallback` off, retry the write one record at a time so the healthy
+      # ones still land - and only the write, so that message-level work isn't repeated.
+      # @param record_list [BatchRecordList]
+      # @param updater [MassUpdater]
+      # @return [Array(Array<ActiveRecord::Base>, Array<Array(Message, StandardError)>)] the
+      #   records that were saved, and the messages that could not be saved with their error.
+      def save_record_list(record_list, updater)
+        [updater.mass_update(record_list), []]
+      rescue StandardError => e
+        raise unless self.batch_message_fallback
+        # Nothing to isolate from a single record, and deadlocks/lock wait timeouts are transient
+        # contention on the whole write which DeadlockRetry has already retried - they don't point
+        # at a bad record, so retrying row by row only multiplies the work.
+        raise if record_list.batch_records.size <= 1 || Deimos::Utils::DeadlockRetry.deadlock?(e)
+
+        save_records_individually(record_list, updater, e)
+      end
+
+      # @param record_list [BatchRecordList]
+      # @param updater [MassUpdater]
+      # @param batch_error [StandardError] the error the bulk write raised.
+      # @return [Array(Array<ActiveRecord::Base>, Array<Array(Message, StandardError)>)]
+      def save_records_individually(record_list, updater, batch_error)
+        report_initial_failure(:upsert_records, record_list.batch_records.size, batch_error)
+
+        saved = []
+        failures = []
+        record_list.batch_records.each do |batch_record|
+          saved.concat(updater.mass_update(BatchRecordList.new([batch_record])))
+        rescue StandardError => e
+          failures << [batch_record.message, e]
+        end
+
+        # Nothing could be saved on its own, so this was never about one bad record - it's
+        # something systemic (the database is unreachable, ...). The original error describes that
+        # better than a BatchFallbackError listing every key.
+        raise batch_error if saved.empty?
+
+        [saved, failures]
+      end
+
+      # Delete the records for a group of tombstones, falling back to one message at a time if the
+      # bulk delete fails. Unlike upserts there is no record building, pre-processing or
+      # instrumentation on this path, so the whole operation can safely be retried per message.
+      # @param messages [Array<Message>]
+      # @return [Array<Array(Message, StandardError)>]
+      def remove_group(messages)
+        remove_records(messages)
+        []
+      rescue StandardError => e
+        raise unless self.batch_message_fallback
+        raise if messages.size <= 1 || Deimos::Utils::DeadlockRetry.deadlock?(e)
+
+        report_initial_failure(:remove_records, messages.size, e)
+
+        failures = []
+        messages.each do |message|
+          remove_records([message])
+        rescue StandardError => individual_error
+          failures << [message, individual_error]
+        end
+        raise e if failures.size == messages.size
+
+        failures
+      end
+
+      # Log and announce that a bulk write failed and is about to be retried one at a time.
+      # @param operation [Symbol] `:upsert_records` or `:remove_records`.
+      # @param count [Integer] how many records or messages were in the failed write.
+      # @param error [StandardError]
+      # @return [void]
+      def report_initial_failure(operation, count, error)
+        Deimos::Logging.log_warn(
+          message: 'Batch database write failed, retrying one at a time',
+          handler: self.class.name,
+          topic: self.topic.name,
+          operation: operation,
+          count: count,
+          error_message: error.message
+        )
+        Karafka.monitor.instrument('deimos.batch_consumption.initial_failure', {
+                                     consumer: self.class,
+                                     topic: self.topic.name,
+                                     operation: operation,
+                                     count: count,
+                                     error: error
+                                   })
       end
 
       # @param record_list [BatchRecordList]
@@ -226,10 +320,13 @@ module Deimos
                   self.bulk_import_id_column
                 end
 
-          BatchRecord.new(klass: @klass,
-                          attributes: attrs,
-                          bulk_import_column: col,
-                          bulk_import_id_generator: self.bulk_import_id_generator)
+          record = BatchRecord.new(klass: @klass,
+                                   attributes: attrs,
+                                   bulk_import_column: col,
+                                   bulk_import_id_generator: self.bulk_import_id_generator)
+          # Keep the message so a record which can't be saved can be reported by its Kafka key.
+          record.message = m
+          record
         end
         BatchRecordList.new(records.compact)
       end

data/lib/deimos/active_record_consume/batch_record.rb

@@ -18,6 +18,9 @@ module Deimos
       attr_accessor :bulk_import_column
       # @return [Boolean] true if the primary key was supplied in the input attributes,
       attr_accessor :primary_key_preset
+      # @return [Deimos::Message,nil] the message this record was built from, if any. Used to
+      # report the Kafka key of a record which could not be saved. Not set on sub-records.
+      attr_accessor :message
 
       delegate :valid?, :errors, :send, :attributes, to: :record
 

data/lib/deimos/active_record_consumer.rb

@@ -65,6 +65,11 @@ module Deimos
       topic.save_associations_first
     end
 
+    # @return [Boolean]
+    def batch_message_fallback
+      topic.batch_message_fallback
+    end
+
     def key_decoder
       self.topic.serializers[:key]&.backend
     end

data/lib/deimos/exceptions.rb

@@ -2,4 +2,21 @@
 
 module Deimos
   class MissingImplementationError < StandardError; end
+
+  # Raised when a batch database operation failed and the messages were retried one at a time.
+  # Every message that could be saved on its own has been saved; this carries the ones that could
+  # not, so that the offending keys show up in logging and error reporting.
+  class BatchFallbackError < StandardError
+    # @return [Array<Array(Deimos::Message, StandardError)>] each message that failed on its own,
+    #   paired with the error it raised.
+    attr_reader :failures
+
+    # @param failures [Array<Array(Deimos::Message, StandardError)>]
+    def initialize(failures)
+      @failures = failures
+      details = failures.map { |message, error| "#{message.key.inspect} (#{error.message})" }
+      super("#{failures.size} message(s) could not be saved individually after the batch " \
+            "failed. Failed keys: #{details.join(', ')}")
+    end
+  end
 end

data/lib/deimos/ext/consumer_route.rb

@@ -10,7 +10,8 @@ module Deimos
                   each_message
                   reraise_errors
                   fatal_error
-                  save_associations_first).freeze
+                  save_associations_first
+                  batch_message_fallback).freeze
       Config = Struct.new(*FIELDS, keyword_init: true)
 
       FIELDS.each do |field|
@@ -19,6 +20,7 @@ module Deimos
             bulk_import_id_column: :bulk_import_id,
             replace_associations: true,
             each_message: false,
+            batch_message_fallback: true,
             reraise_errors: Rails.env.test?,
             bulk_import_id_generator: proc { SecureRandom.uuid },
             fatal_error: proc { false }

data/lib/deimos/utils/deadlock_retry.rb

@@ -21,6 +21,15 @@ module Deimos
           'deadlock detected'
         ].freeze
 
+        # Whether the given exception is a deadlock or lock wait timeout, i.e. transient
+        # contention on the database rather than a problem with the data being written.
+        # @param error [Exception]
+        # @return [Boolean]
+        def deadlock?(error)
+          error.is_a?(ActiveRecord::StatementInvalid) &&
+            DEADLOCK_MESSAGES.any? { |m| error.message.include?(m) }
+        end
+
         # Retry the given block when encountering a deadlock. For any other
         # exceptions, they are reraised. This is used to handle cases where
         # the database may be busy but the transaction would succeed if
@@ -45,7 +54,7 @@ module Deimos
             end
           rescue ActiveRecord::StatementInvalid => e
             # Reraise if not a known deadlock
-            raise if DEADLOCK_MESSAGES.none? { |m| e.message.include?(m) }
+            raise unless deadlock?(e)
 
             # Reraise if all retries exhausted
             raise if count <= 0

data/lib/deimos/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Deimos
-  VERSION = '2.5.4'
+  VERSION = '2.6.0'
 end

data/lib/deimos.rb

@@ -49,6 +49,7 @@ module Deimos
     deimos.encode_message
     deimos.batch_consumption.invalid_records
     deimos.batch_consumption.valid_records
+    deimos.batch_consumption.initial_failure
     deimos.outbox.produce
   ).freeze
 

data/spec/active_record_batch_consumer_spec.rb

@@ -460,6 +460,291 @@ module ActiveRecordBatchConsumerTest
       end
     end
 
+    describe 'individual fallback when a batch operation fails' do
+      # `test_id: ''` passes Avro validation (it's a string) but fails the model's
+      # `validates :test_id, presence: true`, so `MassUpdater` raises before it issues any SQL.
+      # That's the poison-message case: one record that can never be persisted.
+      let(:poison_payload) { { test_id: '', some_int: 3 } }
+
+      let(:consumer_class) do
+        Class.new(described_class) do
+          record_class Widget
+          compacted false
+        end
+      end
+
+      before(:each) do
+        register_consumer(consumer_class, 'MySchema',
+                          key_config: { plain: true },
+                          configs: { reraise_errors: true })
+      end
+
+      it 'should not raise when the batch succeeds' do
+        publish_batch(
+          [
+            { key: 1, payload: { test_id: 'abc', some_int: 1 } },
+            { key: 2, payload: { test_id: 'def', some_int: 2 } }
+          ]
+        )
+
+        expect(all_widgets).
+          to contain_exactly(have_attributes(id: 1, test_id: 'abc'),
+                             have_attributes(id: 2, test_id: 'def'))
+      end
+
+      context 'when one record in the batch cannot be persisted' do
+
+        it 'should save every record except the bad one and report the failed key' do
+          expect {
+            publish_batch(
+              [
+                { key: 1, payload: { test_id: 'abc', some_int: 1 } },
+                { key: 2, payload: poison_payload },
+                { key: 3, payload: { test_id: 'ghi', some_int: 3 } }
+              ]
+            )
+          }.to raise_error(Deimos::BatchFallbackError, /Failed keys: "2"/)
+
+          expect(all_widgets).
+            to contain_exactly(have_attributes(id: 1, test_id: 'abc', some_int: 1),
+                               have_attributes(id: 3, test_id: 'ghi', some_int: 3))
+        end
+
+        it 'should collect the failures of every bad record in the batch' do
+          expect {
+            publish_batch(
+              [
+                { key: 1, payload: poison_payload },
+                { key: 2, payload: { test_id: 'def', some_int: 2 } },
+                { key: 3, payload: poison_payload }
+              ]
+            )
+          }.to raise_error(Deimos::BatchFallbackError) { |error|
+            expect(error.failures.map { |message, _| message.key }).to contain_exactly('1', '3')
+            expect(error.failures.map(&:last)).to all(be_a(ActiveRecord::RecordInvalid))
+          }
+
+          expect(all_widgets).to contain_exactly(have_attributes(id: 2, test_id: 'def'))
+        end
+
+        it 'should raise the original error when no record could be saved on its own' do
+          # Isolating salvaged nothing, so the failure was never about one bad message. The
+          # original error is more useful than a BatchFallbackError listing every key.
+          expect {
+            publish_batch(
+              [
+                { key: 1, payload: poison_payload },
+                { key: 2, payload: poison_payload }
+              ]
+            )
+          }.to raise_error(ActiveRecord::RecordInvalid)
+
+          expect(all_widgets).to be_empty
+        end
+
+        it 'should emit an instrumentation event naming the topic and operation' do
+          events = []
+          Karafka.monitor.subscribe('deimos.batch_consumption.initial_failure') do |event|
+            events << event.payload
+          end
+
+          expect {
+            publish_batch(
+              [
+                { key: 1, payload: { test_id: 'abc', some_int: 1 } },
+                { key: 2, payload: poison_payload }
+              ]
+            )
+          }.to raise_error(Deimos::BatchFallbackError)
+
+          expect(events.size).to eq(1)
+          expect(events.first).to include(consumer: consumer_class,
+                                          topic: 'my-topic',
+                                          operation: :upsert_records,
+                                          count: 2)
+        end
+
+        it 'should only re-run the database write, not the message-level processing' do
+          # Retrying by breaking the batch into writes of one must not re-run anything that
+          # already ran for the batch as a whole. pre_process is called once for the group, and
+          # valid_records is announced once with every record that made it, rather than once per
+          # record.
+          pre_processed = []
+          allow_any_instance_of(consumer_class).to receive(:pre_process) do |_, messages|
+            pre_processed << messages.map(&:key)
+          end
+
+          valid_records = []
+          Karafka.monitor.subscribe('deimos.batch_consumption.valid_records') do |event|
+            valid_records << event.payload[:records]
+          end
+
+          expect {
+            publish_batch(
+              [
+                { key: 1, payload: { test_id: 'abc', some_int: 1 } },
+                { key: 2, payload: poison_payload },
+                { key: 3, payload: { test_id: 'ghi', some_int: 3 } }
+              ]
+            )
+          }.to raise_error(Deimos::BatchFallbackError)
+
+          expect(pre_processed).to eq([%w(1 2 3)])
+          expect(valid_records.size).to eq(1)
+          expect(valid_records.first.map(&:test_id)).to contain_exactly('abc', 'ghi')
+        end
+
+        it 'should not retry individually when the batch failed on a deadlock' do
+          allow(Deimos::Utils::DeadlockRetry).to receive(:sleep)
+          allow(Widget).to receive(:import!).
+            and_raise(ActiveRecord::Deadlocked.new('Lock wait timeout exceeded'))
+
+          expect {
+            publish_batch(
+              [
+                { key: 1, payload: { test_id: 'abc', some_int: 1 } },
+                { key: 2, payload: { test_id: 'def', some_int: 2 } }
+              ]
+            )
+          }.to raise_error(ActiveRecord::Deadlocked)
+
+          # Only DeadlockRetry's own 3 attempts - no per-message retries on top.
+          expect(Widget).to have_received(:import!).exactly(3).times
+        end
+      end
+
+      context 'with a single-message batch' do
+        it 'should raise the original error rather than wrapping it' do
+          expect {
+            publish_batch([{ key: 1, payload: poison_payload }])
+          }.to raise_error(ActiveRecord::RecordInvalid)
+
+          expect(all_widgets).to be_empty
+        end
+      end
+
+      context 'when batch_message_fallback is turned off' do
+        before(:each) do
+          register_consumer(consumer_class, 'MySchema',
+                            key_config: { plain: true },
+                            configs: { reraise_errors: true,
+                                       batch_message_fallback: false })
+        end
+
+        it 'should lose the whole batch to a single bad record' do
+          expect {
+            publish_batch(
+              [
+                { key: 1, payload: { test_id: 'abc', some_int: 1 } },
+                { key: 2, payload: poison_payload },
+                { key: 3, payload: { test_id: 'ghi', some_int: 3 } }
+              ]
+            )
+          }.to raise_error(ActiveRecord::RecordInvalid)
+
+          expect(all_widgets).to be_empty
+        end
+
+        it 'should not attempt any individual writes' do
+          allow(Widget).to receive(:import!).and_call_original
+
+          expect {
+            publish_batch(
+              [
+                { key: 1, payload: { test_id: 'abc', some_int: 1 } },
+                { key: 2, payload: poison_payload }
+              ]
+            )
+          }.to raise_error(ActiveRecord::RecordInvalid)
+
+          # The batch write raises during validation, before any import is issued.
+          expect(Widget).not_to have_received(:import!)
+        end
+      end
+
+      context 'with keys repeated across BatchSlicer slices' do
+        it 'should still process the later slices after an earlier one fails' do
+          # Key 1 appears twice, so BatchSlicer splits the batch into two slices:
+          # [key 1 (poison), key 2] and [key 1 (valid)]. The valid update for key 1 sits in the
+          # second slice, so it only lands if a failure in the first slice doesn't abort the rest.
+          expect {
+            publish_batch(
+              [
+                { key: 1, payload: poison_payload },
+                { key: 1, payload: { test_id: 'later', some_int: 2 } },
+                { key: 2, payload: { test_id: 'ok', some_int: 3 } }
+              ]
+            )
+          }.to raise_error(Deimos::BatchFallbackError, /Failed keys: "1"/)
+
+          expect(all_widgets).
+            to contain_exactly(have_attributes(id: 1, test_id: 'later', some_int: 2),
+                               have_attributes(id: 2, test_id: 'ok', some_int: 3))
+        end
+      end
+
+      context 'with max_db_batch_size' do
+        let(:consumer_class) do
+          Class.new(described_class) do
+            record_class Widget
+            compacted false
+            max_db_batch_size 2
+          end
+        end
+
+        it 'should still process the later groups after an earlier one fails' do
+          expect {
+            publish_batch(
+              [
+                { key: 1, payload: { test_id: 'abc', some_int: 1 } },
+                { key: 2, payload: poison_payload },
+                { key: 3, payload: { test_id: 'ghi', some_int: 3 } },
+                { key: 4, payload: { test_id: 'jkl', some_int: 4 } }
+              ]
+            )
+          }.to raise_error(Deimos::BatchFallbackError, /Failed keys: "2"/)
+
+          expect(all_widgets).
+            to contain_exactly(have_attributes(id: 1, test_id: 'abc'),
+                               have_attributes(id: 3, test_id: 'ghi'),
+                               have_attributes(id: 4, test_id: 'jkl'))
+        end
+      end
+
+      context 'when removing records fails' do
+        let(:consumer_class) do
+          Class.new(described_class) do
+            record_class Widget
+            compacted false
+
+            def remove_records(messages)
+              raise 'cannot delete widget 2' if messages.any? { |m| m.key.to_s == '2' }
+
+              super
+            end
+          end
+        end
+
+        it 'should delete the records it can and report the one it cannot' do
+          Widget.create!(id: 1, test_id: 'abc', some_int: 1)
+          Widget.create!(id: 2, test_id: 'def', some_int: 2)
+          Widget.create!(id: 3, test_id: 'ghi', some_int: 3)
+
+          expect {
+            publish_batch(
+              [
+                { key: 1, payload: nil },
+                { key: 2, payload: nil },
+                { key: 3, payload: nil }
+              ]
+            )
+          }.to raise_error(Deimos::BatchFallbackError, /Failed keys: "2"/)
+
+          expect(all_widgets).to contain_exactly(have_attributes(id: 2, test_id: 'def'))
+        end
+      end
+    end
+
     describe 'skipping records' do
       before(:each) do
         register_consumer(consumer_class,

data/spec/utils/deadlock_retry_spec.rb

@@ -7,6 +7,43 @@ RSpec.describe Deimos::Utils::DeadlockRetry do
     allow(described_class).to receive(:sleep)
   end
 
+  # `deadlock?` matches against the exception message, so the examples below are literal
+  # fragments as each engine emits them, padded with surrounding text to pin substring rather
+  # than equality matching. No database is exercised here - these are hand-built exceptions, so
+  # they verify the predicate agrees with DEADLOCK_MESSAGES, not that any engine really produces
+  # those strings. See the `each_db_config` specs for tests that run against real engines.
+  describe '.deadlock?' do
+    it 'should match MySQL deadlocks and lock wait timeouts' do
+      expect(described_class).to be_deadlock(
+        ActiveRecord::Deadlocked.new('Mysql2::Error: Deadlock found when trying to get lock')
+      )
+      expect(described_class).to be_deadlock(
+        ActiveRecord::StatementInvalid.new('Lock wait timeout exceeded; try restarting')
+      )
+    end
+
+    it 'should match Postgres deadlock detection' do
+      expect(described_class).to be_deadlock(
+        ActiveRecord::Deadlocked.new('PG::TRDeadlockDetected: ERROR: deadlock detected')
+      )
+    end
+
+    it 'should not match a database error whose message is not a known deadlock' do
+      expect(described_class).not_to be_deadlock(ActiveRecord::StatementInvalid.new('Oops!!'))
+    end
+
+    it 'should not match a non-database error that happens to mention a deadlock' do
+      expect(described_class).not_to be_deadlock(StandardError.new('deadlock detected'))
+    end
+
+    it 'should not match validation failures' do
+      # Batch consumption depends on this: if a RecordInvalid looked like a deadlock, the guard
+      # in `save_record_list` would reraise it and the individual fallback would never fire for
+      # the poison-record case it exists to handle.
+      expect(described_class).not_to be_deadlock(ActiveRecord::RecordInvalid.new)
+    end
+  end
+
   describe 'deadlock handling' do
     let(:batch) { [{ key: 1, payload: { test_id: 'abc', some_int: 3 } }] }
 

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: deimos-ruby
 version: !ruby/object:Gem::Version
-  version: 2.5.4
+  version: 2.6.0
 platform: ruby
 authors:
 - Daniel Orner