deimos-ruby 2.6.0.pre.beta1 → 2.6.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: ba96f4ae17cd71ddcda0499103bfaedc88d2757e5e6eb6034b3cffb9e9898034
-  data.tar.gz: f707d6ae52803d9e2cdca87b03c409ba5f47bc1ceaad243fb80b8bd91a2b4dff
+  metadata.gz: 3399390b0e3dc85c67c95c4d633d3ce65efb80258848e73c6b0fa5d9244db13e
+  data.tar.gz: d236d19cd353f581121dbf561cba34ff78a192bd636ac992fcb1ad87b5913375
 SHA512:
-  metadata.gz: 6151699039457db7eecc81f33bdae4ab921f68220ba75db0b6ce935555a112903b280f942c35cd459bb298128cc4e3f4240f26c6538bbe94b0c0924ad0bd8958
-  data.tar.gz: d1882baeffc9e69998fd704559b7c21f83e966e365c4e96b60f073412b40c2b242a8749e0e9890379cdfc88d96e2687dcf5d9293015f3e8ee15608c78c5bc3fb
+  metadata.gz: 502f900a5d5220c55d13bbc50ab329c826d3da953b14adfbaaaf1668e229d2ede46268729ed3c7bc631045feb6927fe85126a6b990f28b4f00630efdb0bb5501
+  data.tar.gz: 44d082485bfb68be5b2c49f228ab29626eb5c7b80b98873284a7844fba55c881ac805d0ce9db2c9d2f44f8b68eb781f541a5d1d0099b94f2f7ee236a29197093

data/CHANGELOG.md

@@ -5,9 +5,15 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
-## Unreleased
+## 2.6.0 - 2026-08-13
 
-- Major change: Extract Avro schema class generation out of Deimos and into `avro-gen-ruby`.
+- Breaking: a failed batch database write is now retried one record at a time, so one unpersistable record no longer loses the whole batch. `Deimos::BatchFallbackError` is raised naming the keys that still failed.
+- Feature: new `batch_message_fallback` topic setting, default true. Set it to false to keep the previous all-or-nothing behaviour.
+- Feature: new `deimos.batch_consumption.initial_failure` instrumentation event, fired when a batch write fails and the records are about to be retried individually.
+
+## 2.5.4 - 2026-07-15
+
+- Fix: `DeadlockRetry.wrap` coerces its `tags` argument with `Array(...)`, so a scalar topic String (passed by `MassUpdater`/batch consumption) no longer raises `NoMethodError` on `String#to_a` inside the rescue, which had defeated the deadlock retry on the first deadlock.
 
 ## 2.5.3 - 2026-04-28
 

data/README.md

@@ -323,6 +323,12 @@ produced by Phobos and RubyKafka):
 * `deimos.batch_consumption.invalid_records` - sent when the consumer has rejected records returned from `filtered_records`. Limited by `max_db_batch_size`.
   * consumer: class of the consumer that rejected these records
   * records: Rejected records (of type `Deimos::ActiveRecordConsume::BatchRecord`)
+* `deimos.batch_consumption.initial_failure` - sent when a bulk database write failed and the consumer is about to retry it one record at a time. Fired once per failing write, and fired even if nothing ends up being salvaged.
+  * consumer: class of the consumer that is retrying
+  * topic: name of the topic being consumed
+  * operation: `:upsert_records` or `:remove_records`
+  * count: number of records or messages in the write that failed
+  * error: the exception the bulk write raised
   
 # Consumers
 
@@ -565,6 +571,36 @@ By default, batches will be compacted before processing, i.e. only the last
 message for each unique key in a batch will actually be processed. To change
 this behaviour, call `compacted false` inside of your consumer definition.
 
+#### Failure handling
+
+Because a batch is written in a single SQL statement inside a single transaction, one record
+which cannot be persisted would otherwise roll back every other record written alongside it.
+To avoid losing an entire batch to one bad record, Deimos retries the database write one record
+at a time when the bulk write fails. Everything that can be saved on its own is saved, and once
+the whole batch has been attempted a single `Deimos::BatchFallbackError` is raised. Its
+`failures` attribute holds the `[message, error]` pairs that could not be saved, and its message
+names their keys. From there it is handled by `reraise_errors`/`fatal_error` like any other
+consumer error.
+
+Only the database write is retried. Message-level work — `pre_process`, building and filtering
+records, and the `valid_records`/`invalid_records` events — happens exactly once for the group
+either way, so nothing is applied twice. The
+`deimos.batch_consumption.initial_failure` event fires when a bulk write fails and the records
+are about to be retried individually.
+
+The original error is reraised unchanged, rather than wrapped, in three cases:
+
+* Writes of a single record, where there is nothing to isolate.
+* Deadlocks and lock wait timeouts, which are transient contention on the whole write (already
+  retried by `DeadlockRetry`) rather than a problem with a particular record.
+* When nothing could be saved on its own. Isolating salvaged nothing, so the failure was
+  never about one bad record - it is something systemic, such as an unreachable database - and
+  the original error describes that better.
+
+Retrying row by row trades throughput for durability, which is not always the right trade for a
+very large batch on a hot topic. Set `batch_message_fallback false` on the topic to keep
+the previous all-or-nothing behaviour and let the whole batch fail.
+
 A sample batch consumer would look as follows:
 
 ```ruby
@@ -840,23 +876,19 @@ DB backend only when your rake task is running.
 
 # Generated Schema Classes
 
-Deimos generates classes from Avro schemas using the
-[avro-gen-ruby](https://github.com/flipp-oss/avro-gen-ruby) gem (namespace `AvroGen`).
-These classes are documented with YARD to aid in IDE auto-complete, and will help to move
-errors closer to the code.
+Deimos offers a way to generate classes from Avro schemas. These classes are documented
+with YARD to aid in IDE auto-complete, and will help to move errors closer to the code.
 
-Add the following configurations for schema class generation:
+Add the following configurations for schema class generation: 
 
 ```ruby
-config.avrogen.generated_class_path 'path/to/generated/classes' # Defaults to 'app/lib/schema_classes'
+config.schema.generated_class_path 'path/to/generated/classes' # Defaults to 'app/lib/schema_classes'
 ```
 
 Run the following command to generate schema classes in your application. It will generate classes for every configured consumer or producer by `Deimos.configure`:
 
     bundle exec rake deimos:generate_schema_classes
 
-(The standalone `bundle exec rake avro:generate` task generates classes for every schema on disk, without the Kafka-aware key/tombstone handling.)
-
 Add the following configurations to start using generated schema classes in your application's Consumers and Producers:
 
     config.schema.use_schema_classes true
@@ -868,14 +900,7 @@ Note that if you have a schema in your repo but have not configured a producer o
 
 One additional configuration option indicates whether nested records should be generated as top-level classes or would remain nested inside the generated class for its parent schema. The default is to nest them, as a flattened structure can have one sub-schema clobber another sub-schema defined in a different top-level schema.
 
-    config.avrogen.nest_child_schemas = false # Flatten all classes into one directory
-
-> **Note:** The schema-class generation settings moved from `config.schema.*` to
-> `config.avrogen.*`. The old `config.schema.generated_class_path` /
-> `nest_child_schemas` / `use_full_namespace` / `schema_namespace_map` settings still
-> work but are deprecated. Generated classes now inherit from
-> `AvroGen::SchemaClass::Record`/`Enum`; previously-generated files referencing
-> `Deimos::SchemaClass::*` still load, and `bundle exec rake avro:upgrade` rewrites them.
+    config.schema.nest_child_schemas = false # Flatten all classes into one directory
 
 You can generate a tombstone message (with only a key and no value) by calling the `YourSchemaClass.tombstone(key)` method. If you're using a `:field` key config, you can pass in just the key scalar value. If using a key schema, you can pass it in as a hash or as another schema class.
 
@@ -888,7 +913,7 @@ Examples of consumers would look like this:
 ```ruby
 class MyConsumer < Deimos::Consumer
   def consume_message(message)
-    # Same method as before but message.payload is now an instance of AvroGen::SchemaClass::Record
+    # Same method as before but message.payload is now an instance of Deimos::SchemaClass::Record
     # rather than a hash. 
     # You can interact with the schema class instance in the following way: 
     do_something(message.payload.test_id, message.payload.some_int)
@@ -902,7 +927,7 @@ end
 class MyActiveRecordConsumer < Deimos::ActiveRecordConsumer
   record_class Widget
   # Any method that expects a message payload as a hash will instead
-  # receive an instance of AvroGen::SchemaClass::Record.
+  # receive an instance of Deimos::SchemaClass::Record.
   def record_attributes(payload, key)
     # You can interact with the schema class instance in the following way:
     super.merge(:some_field => "some_value-#{payload.test_id}")
@@ -942,9 +967,9 @@ class MyActiveRecordProducer < Deimos::ActiveRecordProducer
   record_class Widget
   # @param attributes [Hash]
   # @param _record [Widget]
-  # @return [AvroGen::SchemaClass::Record]
+  # @return [Deimos::SchemaClass::Record]
   def self.generate_payload(attributes, _record)
-    # This method converts your ActiveRecord into a AvroGen::SchemaClass::Record. You will be able to use super
+    # This method converts your ActiveRecord into a Deimos::SchemaClass::Record. You will be able to use super
     # as an instance of Schemas::MySchema and set values that are not on your ActiveRecord schema.
     res = super
     res.some_value = "some_value-#{res.test_id}"

data/deimos-ruby.gemspec

@@ -17,7 +17,6 @@ Gem::Specification.new do |spec|
   spec.executables   = spec.files.grep(%r{^bin/}) { |f| File.basename(f) }
   spec.require_paths = ['lib']
 
-  spec.add_dependency('avro-gen-ruby')
   spec.add_dependency('benchmark', '~> 0.5')
   spec.add_dependency('fig_tree', '~> 0.2.0')
   spec.add_dependency('karafka', '~> 2.0')

data/docs/CONFIGURATION.md

@@ -60,17 +60,10 @@ things you need to reference into local variables before calling `configure`.
 | schema.password             | nil                      | Basic auth password.                                                                                                                                       |
 | schema.path                 | nil                      | Local path to find your schemas.                                                                                                                           |
 | schema.use_schema_classes   | false                    | Set this to true to use generated schema classes in your application.                                                                                      |
-
-#### Schema Class Generation (`avrogen`)
-
-Schema class generation is provided by the [avro-gen-ruby](https://github.com/flipp-oss/avro-gen-ruby) gem (namespace `AvroGen`). These settings are forwarded to `AvroGen.config`. The equivalent `schema.*` settings still work but are deprecated — using one prints a warning pointing to the `avrogen.*` setting, and you can migrate generated files with `rake avro:upgrade`.
-
-| Config name                   | Default                  | Description                                                                                                                                              |
-|-------------------------------|--------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------|
-| avrogen.generated_class_path  | `app/lib/schema_classes` | Local path to generated schema classes. (was `schema.generated_class_path`)                                                                              |
-| avrogen.nest_child_schemas    | true                     | Set to true to nest subschemas within the generated class for the parent schema. (was `schema.nest_child_schemas`)                                       |
-| avrogen.use_full_namespace    | false                    | Set to true to generate folders for schemas matching the full namespace. (was `schema.use_full_namespace`)                                               |
-| avrogen.schema_namespace_map  | {}                       | A map of namespace prefixes to base module name(s). Example: { 'com.mycompany.suborg' => ['SchemaClasses'] }. Requires `use_full_namespace` to be true. (was `schema.schema_namespace_map`) |
+| schema.generated_class_path | `app/lib/schema_classes` | Local path to generated schema classes.                                                                                                                    |
+| schema.nest_child_schemas   | false                    | Set to true to nest subschemas within the generated class for the parent schema.                                                                           |
+| schema.use_full_namespace   | false                    | Set to true to generate folders for schemas matching the full namespace.                                                                                   |
+| schema.schema_namespace_map | {}                       | A map of namespace prefixes to base module name(s). Example: { 'com.mycompany.suborg' => ['SchemaClasses'] }. Requires `use_full_namespace` to be true.    |
 
 ### Outbox Configuration
 
@@ -140,6 +133,7 @@ The following are additional settings that can be added to the `topic` block in
 | replace_associations     | true              | If false, append to associations in multi-table imports rather than replacing them.                                                                                                                                                                                                                                                                                                                                        |
 | bulk_import_id_generator | nil               | Block to determine the bulk_import_id generated during bulk consumption. If no block is specified the provided/default block from the consumers configuration will be used.                                                                                                                                                                                                                                                |
 | save_associations_first  | false             | Whether to save associated records of primary class prior to upserting primary records. Foreign key of associated records are assigned to the record class prior to saving the record class                                                                                                                                                                                                                                |
+| batch_message_fallback   | true              | When a bulk database write fails, retry it one record at a time so that a single failed record doesn't lose the whole batch. Set to false to skip this behavior, e.g. where retrying row by row is too slow.                                                                                                                                                                                                                |
 
 ### Defining Consumers
 

data/lib/deimos/active_record_consume/batch_consumption.rb

@@ -24,6 +24,7 @@ module Deimos
       # in the same operation as they would interfere with each other. Thus
       # they are split
       # @return [void]
+      # @raise [BatchFallbackError] if some messages could not be saved even on their own.
       def consume_batch
         filtered = messages.select { |message| process_message?(message) }
         skipped_count = messages.size - filtered.size
@@ -39,11 +40,15 @@ module Deimos
         Deimos.config.tracer.active_span.set_tag('topic', tag)
 
         Karafka.monitor.instrument('deimos.ar_consumer.consume_batch', { topic: tag }) do
-          if @compacted && deimos_messages.map(&:key).compact.any?
-            update_database(compact_messages(deimos_messages))
-          else
-            uncompacted_update(deimos_messages)
-          end
+          failures = if @compacted && deimos_messages.map(&:key).compact.any?
+                       update_database(compact_messages(deimos_messages))
+                     else
+                       uncompacted_update(deimos_messages)
+                     end
+
+          # Raised only once every slice and group has been attempted, so that a message which
+          # can't be persisted never stops the rest of the batch from being saved.
+          raise BatchFallbackError, failures if failures.any?
         end
 
         post_process_batch(deimos_messages)
@@ -76,7 +81,7 @@ module Deimos
       def record_key(key)
         if key.nil?
           {}
-        elsif key.is_a?(Hash) || key.is_a?(AvroGen::SchemaClass::Record)
+        elsif key.is_a?(Hash) || key.is_a?(SchemaClass::Record)
           self.key_converter.convert(key)
         elsif self.topic.key_config[:field].nil?
           { @klass.primary_key => key }
@@ -128,45 +133,40 @@ module Deimos
       # All messages are split into slices containing only unique keys, and
       # each slice is handles as its own batch.
       # @param messages [Array<Message>] List of messages.
-      # @return [void]
+      # @return [Array<Array(Message, StandardError)>] messages that could not be saved even on
+      #   their own, paired with their error.
       def uncompacted_update(messages)
         BatchSlicer.
           slice(messages).
-          each(&method(:update_database))
+          flat_map(&method(:update_database))
       end
 
       # Perform database operations for a group of messages.
       # All messages with payloads are passed to upsert_records.
       # All tombstones messages are passed to remove_records.
       # @param messages [Array<Message>] List of messages.
-      # @return [void]
+      # @return [Array<Array(Message, StandardError)>] messages that could not be saved even on
+      #   their own, paired with their error.
       def update_database(messages)
         # Find all upserted records (i.e. that have a payload) and all
         # deleted record (no payload)
         removed, upserted = messages.partition { |m| delete_record?(m) }
 
         max_db_batch_size = self.class.config[:max_db_batch_size]
-        if upserted.any?
-          if max_db_batch_size
-            upserted.each_slice(max_db_batch_size) { |group| upsert_records(group) }
-          else
-            upsert_records(upserted)
-          end
-        end
+        upsert_groups = max_db_batch_size ? upserted.each_slice(max_db_batch_size).to_a : [upserted]
+        remove_groups = max_db_batch_size ? removed.each_slice(max_db_batch_size).to_a : [removed]
 
-        return if removed.empty?
-
-        if max_db_batch_size
-          removed.each_slice(max_db_batch_size) { |group| remove_records(group) }
-        else
-          remove_records(removed)
-        end
+        upsert_groups.reject(&:empty?).flat_map { |group| upsert_records(group) } +
+          remove_groups.reject(&:empty?).flat_map { |group| remove_group(group) }
       end
 
-      # Upsert any non-deleted records
+      # Upsert any non-deleted records. Everything that operates on the messages - pre-processing,
+      # building and filtering records, instrumentation - happens exactly once here; only the
+      # database write is retried if it fails, so nothing gets applied twice.
       # @param messages [Array<Message>] List of messages for a group of
       # records to either be updated or inserted.
-      # @return [void]
+      # @return [Array<Array(Message, StandardError)>] messages whose records could not be saved
+      #   even on their own, paired with their error.
       def upsert_records(messages)
         record_list = build_records(messages)
         invalid = filter_records(record_list)
@@ -176,7 +176,7 @@ module Deimos
                                        consumer: self.class
                                      })
         end
-        return if record_list.empty?
+        return [] if record_list.empty?
 
         key_col_proc = self.method(:key_columns).to_proc
         col_proc = self.method(:columns).to_proc
@@ -188,10 +188,104 @@ module Deimos
                                   bulk_import_id_generator: self.bulk_import_id_generator,
                                   save_associations_first: self.save_associations_first,
                                   bulk_import_id_column: self.bulk_import_id_column)
+        saved, failures = save_record_list(record_list, updater)
         Karafka.monitor.instrument('deimos.batch_consumption.valid_records', {
-                                     records: updater.mass_update(record_list),
+                                     records: saved,
                                      consumer: self.class
                                    })
+        failures
+      end
+
+      # Write a list of records to the database. The list is written in a single statement inside
+      # a single transaction, so one record which can't be persisted would otherwise take down
+      # every other record written alongside it. Unless the topic turns
+      # `batch_message_fallback` off, retry the write one record at a time so the healthy
+      # ones still land - and only the write, so that message-level work isn't repeated.
+      # @param record_list [BatchRecordList]
+      # @param updater [MassUpdater]
+      # @return [Array(Array<ActiveRecord::Base>, Array<Array(Message, StandardError)>)] the
+      #   records that were saved, and the messages that could not be saved with their error.
+      def save_record_list(record_list, updater)
+        [updater.mass_update(record_list), []]
+      rescue StandardError => e
+        raise unless self.batch_message_fallback
+        # Nothing to isolate from a single record, and deadlocks/lock wait timeouts are transient
+        # contention on the whole write which DeadlockRetry has already retried - they don't point
+        # at a bad record, so retrying row by row only multiplies the work.
+        raise if record_list.batch_records.size <= 1 || Deimos::Utils::DeadlockRetry.deadlock?(e)
+
+        save_records_individually(record_list, updater, e)
+      end
+
+      # @param record_list [BatchRecordList]
+      # @param updater [MassUpdater]
+      # @param batch_error [StandardError] the error the bulk write raised.
+      # @return [Array(Array<ActiveRecord::Base>, Array<Array(Message, StandardError)>)]
+      def save_records_individually(record_list, updater, batch_error)
+        report_initial_failure(:upsert_records, record_list.batch_records.size, batch_error)
+
+        saved = []
+        failures = []
+        record_list.batch_records.each do |batch_record|
+          saved.concat(updater.mass_update(BatchRecordList.new([batch_record])))
+        rescue StandardError => e
+          failures << [batch_record.message, e]
+        end
+
+        # Nothing could be saved on its own, so this was never about one bad record - it's
+        # something systemic (the database is unreachable, ...). The original error describes that
+        # better than a BatchFallbackError listing every key.
+        raise batch_error if saved.empty?
+
+        [saved, failures]
+      end
+
+      # Delete the records for a group of tombstones, falling back to one message at a time if the
+      # bulk delete fails. Unlike upserts there is no record building, pre-processing or
+      # instrumentation on this path, so the whole operation can safely be retried per message.
+      # @param messages [Array<Message>]
+      # @return [Array<Array(Message, StandardError)>]
+      def remove_group(messages)
+        remove_records(messages)
+        []
+      rescue StandardError => e
+        raise unless self.batch_message_fallback
+        raise if messages.size <= 1 || Deimos::Utils::DeadlockRetry.deadlock?(e)
+
+        report_initial_failure(:remove_records, messages.size, e)
+
+        failures = []
+        messages.each do |message|
+          remove_records([message])
+        rescue StandardError => individual_error
+          failures << [message, individual_error]
+        end
+        raise e if failures.size == messages.size
+
+        failures
+      end
+
+      # Log and announce that a bulk write failed and is about to be retried one at a time.
+      # @param operation [Symbol] `:upsert_records` or `:remove_records`.
+      # @param count [Integer] how many records or messages were in the failed write.
+      # @param error [StandardError]
+      # @return [void]
+      def report_initial_failure(operation, count, error)
+        Deimos::Logging.log_warn(
+          message: 'Batch database write failed, retrying one at a time',
+          handler: self.class.name,
+          topic: self.topic.name,
+          operation: operation,
+          count: count,
+          error_message: error.message
+        )
+        Karafka.monitor.instrument('deimos.batch_consumption.initial_failure', {
+                                     consumer: self.class,
+                                     topic: self.topic.name,
+                                     operation: operation,
+                                     count: count,
+                                     error: error
+                                   })
       end
 
       # @param record_list [BatchRecordList]
@@ -226,10 +320,13 @@ module Deimos
                   self.bulk_import_id_column
                 end
 
-          BatchRecord.new(klass: @klass,
-                          attributes: attrs,
-                          bulk_import_column: col,
-                          bulk_import_id_generator: self.bulk_import_id_generator)
+          record = BatchRecord.new(klass: @klass,
+                                   attributes: attrs,
+                                   bulk_import_column: col,
+                                   bulk_import_id_generator: self.bulk_import_id_generator)
+          # Keep the message so a record which can't be saved can be reported by its Kafka key.
+          record.message = m
+          record
         end
         BatchRecordList.new(records.compact)
       end

data/lib/deimos/active_record_consume/batch_record.rb

@@ -18,6 +18,9 @@ module Deimos
       attr_accessor :bulk_import_column
       # @return [Boolean] true if the primary key was supplied in the input attributes,
       attr_accessor :primary_key_preset
+      # @return [Deimos::Message,nil] the message this record was built from, if any. Used to
+      # report the Kafka key of a record which could not be saved. Not set on sub-records.
+      attr_accessor :message
 
       delegate :valid?, :errors, :send, :attributes, to: :record
 

data/lib/deimos/active_record_consume/message_consumption.rb

@@ -13,7 +13,7 @@ module Deimos
       # Default is to use the primary key column and the value of the first
       # field in the key.
       # @param klass [Class<ActiveRecord::Base>]
-      # @param _payload [Hash,AvroGen::SchemaClass::Record]
+      # @param _payload [Hash,Deimos::SchemaClass::Record]
       # @param key [Object]
       # @return [ActiveRecord::Base]
       def fetch_record(klass, _payload, key)
@@ -23,7 +23,7 @@ module Deimos
 
       # Assign a key to a new record.
       # @param record [ActiveRecord::Base]
-      # @param _payload [Hash,AvroGen::SchemaClass::Record]
+      # @param _payload [Hash,Deimos::SchemaClass::Record]
       # @param key [Object]
       # @return [void]
       def assign_key(record, _payload, key)

data/lib/deimos/active_record_consumer.rb

@@ -65,6 +65,11 @@ module Deimos
       topic.save_associations_first
     end
 
+    # @return [Boolean]
+    def batch_message_fallback
+      topic.batch_message_fallback
+    end
+
     def key_decoder
       self.topic.serializers[:key]&.backend
     end
@@ -90,7 +95,7 @@ module Deimos
 
     # Override this method (with `super`) if you want to add/change the default
     # attributes set to the new/existing record.
-    # @param payload [Hash,AvroGen::SchemaClass::Record]
+    # @param payload [Hash,Deimos::SchemaClass::Record]
     # @param _key [String]
     # @return [Hash]
     def record_attributes(payload, _key=nil)

data/lib/deimos/active_record_producer.rb

@@ -81,7 +81,7 @@ module Deimos
         return payload if self.karafka_config.use_schema_classes.nil? &&
                           !Deimos.config.schema.use_schema_classes
 
-        AvroGen::SchemaClass.instance(payload, encoder.schema, encoder.namespace)
+        Utils::SchemaClass.instance(payload, encoder.schema, encoder.namespace)
       end
 
       # Deletion payload for a record by default, delegate to the

data/lib/deimos/config/configuration.rb

@@ -15,7 +15,6 @@ module Deimos
       config.deserializers[:payload].try(:reset_backend)
       config.deserializers[:key].try(:reset_backend)
     end
-    Deimos::SchemaClass.sync_config!
     if self.config.schema.use_schema_classes
       load_generated_schema_classes
     end
@@ -43,16 +42,16 @@ module Deimos
     # Loads generated classes
     # @return [void]
     def load_generated_schema_classes
-      path = AvroGen.config.generated_class_path
-      if path.nil?
-        raise 'Cannot use schema classes without a generated class path. ' \
-              'Please set AvroGen.config.generated_class_path.'
+      if Deimos.config.schema.generated_class_path.nil?
+        raise 'Cannot use schema classes without schema.generated_class_path. ' \
+              'Please provide a directory.'
       end
 
-      Dir["./#{path}/**/*.rb"].
+      Dir["./#{Deimos.config.schema.generated_class_path}/**/*.rb"].
         each { |f| require f }
     rescue LoadError
-      raise 'Cannot load schema classes. Please regenerate classes with rake avro:generate.'
+      raise 'Cannot load schema classes. Please regenerate classes with' \
+            'rake deimos:generate_schema_models.'
     end
 
     # Ensure everything is set up correctly for the DB backend.
@@ -174,7 +173,7 @@ module Deimos
       # @return [String]
       setting :path
 
-      # @deprecated Use config.avrogen.generated_class_path instead.
+      # Local path for schema classes to be generated in.
       # @return [String]
       setting :generated_class_path, 'app/lib/schema_classes'
 
@@ -182,31 +181,6 @@ module Deimos
       # @return [Boolean]
       setting :use_schema_classes
 
-      # @deprecated Use config.avrogen.nest_child_schemas instead.
-      # @return [Boolean]
-      setting :nest_child_schemas, true
-
-      # @deprecated Use config.avrogen.use_full_namespace instead.
-      # @return [Boolean]
-      setting :use_full_namespace, false
-
-      # @deprecated Use config.avrogen.schema_namespace_map instead.
-      # @return [Hash]
-      setting :schema_namespace_map, {}
-
-      # The base directory for generated protobuf key schemas.
-      setting :proto_schema_key_path, 'protos'
-    end
-
-    # Schema class generation settings, forwarded to the avro-gen-ruby gem
-    # (AvroGen.config). These previously lived under `schema` (still supported,
-    # but deprecated).
-    setting :avrogen do
-
-      # Local path for schema classes to be generated in.
-      # @return [String]
-      setting :generated_class_path, 'app/lib/schema_classes'
-
       # Set to false to generate child schemas as their own files.
       # @return [Boolean]
       setting :nest_child_schemas, true
@@ -217,9 +191,12 @@ module Deimos
 
       # Use this option to reduce nesting when using use_full_namespace.
       # For example: { 'com.mycompany.suborg' => 'SchemaClasses' }
-      # would replace a prefix matching the given key with the module name SchemaClasses.
+      # would replace a prefixed with the given key with the module name SchemaClasses.
       # @return [Hash]
       setting :schema_namespace_map, {}
+
+      # The base directory for generated protobuf key schemas.
+      setting :proto_schema_key_path, 'protos'
     end
 
     # The configured metrics provider.

data/lib/deimos/exceptions.rb

@@ -2,4 +2,21 @@
 
 module Deimos
   class MissingImplementationError < StandardError; end
+
+  # Raised when a batch database operation failed and the messages were retried one at a time.
+  # Every message that could be saved on its own has been saved; this carries the ones that could
+  # not, so that the offending keys show up in logging and error reporting.
+  class BatchFallbackError < StandardError
+    # @return [Array<Array(Deimos::Message, StandardError)>] each message that failed on its own,
+    #   paired with the error it raised.
+    attr_reader :failures
+
+    # @param failures [Array<Array(Deimos::Message, StandardError)>]
+    def initialize(failures)
+      @failures = failures
+      details = failures.map { |message, error| "#{message.key.inspect} (#{error.message})" }
+      super("#{failures.size} message(s) could not be saved individually after the batch " \
+            "failed. Failed keys: #{details.join(', ')}")
+    end
+  end
 end

data/lib/deimos/ext/consumer_route.rb

@@ -10,7 +10,8 @@ module Deimos
                   each_message
                   reraise_errors
                   fatal_error
-                  save_associations_first).freeze
+                  save_associations_first
+                  batch_message_fallback).freeze
       Config = Struct.new(*FIELDS, keyword_init: true)
 
       FIELDS.each do |field|
@@ -19,6 +20,7 @@ module Deimos
             bulk_import_id_column: :bulk_import_id,
             replace_associations: true,
             each_message: false,
+            batch_message_fallback: true,
             reraise_errors: Rails.env.test?,
             bulk_import_id_generator: proc { SecureRandom.uuid },
             fatal_error: proc { false }

data/lib/deimos/ext/producer_middleware.rb

@@ -5,7 +5,7 @@ module Deimos
     class << self
 
       def allowed_classes
-        arr = [Hash, AvroGen::SchemaClass::Record]
+        arr = [Hash, SchemaClass::Record]
         if defined?(Google::Protobuf)
           arr.push(Google::Protobuf.const_get(:AbstractMessage))
         end
@@ -26,7 +26,7 @@ module Deimos
                     self.allowed_classes.none? { |k| message[:payload].is_a?(k) }
 
           payload = message[:payload]
-          payload = payload.to_h if payload.nil? || payload.is_a?(AvroGen::SchemaClass::Record)
+          payload = payload.to_h if payload.nil? || payload.is_a?(SchemaClass::Record)
           m = Deimos::Message.new(payload,
                                   headers: message[:headers],
                                   partition_key: message[:partition_key])

data/lib/deimos/logging.rb

@@ -30,15 +30,6 @@ module Deimos
         log_add(:warn, *args)
       end
 
-      # Emit a deprecation warning at most once per unique message.
-      # @param msg [String]
-      def deprecate(msg)
-        @deprecations ||= Set.new
-        return unless @deprecations.add?(msg)
-
-        warn("DEPRECATION WARNING: #{msg}")
-      end
-
       def metadata_log_text(metadata)
         metadata.to_h.slice(:timestamp, :offset, :first_offset, :last_offset, :partition, :topic, :size)
       end