deimos-ruby 1.24.2 → 2.0.0.pre.alpha1

data/docs/UPGRADING.md

@@ -1,5 +1,242 @@
 # Upgrading Deimos
 
+## Upgrading to 2.x
+
+2.x is a major rewrite from 1.0. The underlying library has been changed from [Phobos](https://github.com/phobos/phobos) to [Karafka](https://karafka.io/). This change has given us an opportunity to fix some issues and deprecated code paths from version 1.0 as well as provide much more functionality by integrating more fully with the Karafka ecosystem.
+
+For a deeper dive into the internal changes, please see [...]().
+
+There are a number of breaking changes. We provide a `v2` generator to attempt to auto-fix many of these breaking changes automatically. To run the generator:
+
+    KARAFKA_BOOT_FILE=false rails g deimos:v2
+
+### Running Deimos
+
+Instead of running `rake deimos:start`, you now run your Kafka consumers the same way any Karafka consumers are run: `karafka server`.
+
+### Configuration
+
+In V1, Deimos configuration was all done in a single `Deimos.configure` block, including Kafka configs, consumers and producers:
+
+```ruby
+Deimos.configure do
+  producers.schema_namespace 'com.my-namespace'
+  kafka.seed_brokers ['my-broker:9092']
+
+  consumer do
+    class_name 'MyConsumer'
+    topic 'MyTopic'
+    session_timeout 30
+    schema 'MySchema'
+    key_config field: :id
+    namespace 'com.my-namespace'
+  end
+
+  producer do
+    class_name 'MyProducer'
+    topic 'MyTopic2'
+    schema 'MySchema2'
+    key_config none: true
+  end
+end  
+```
+
+In V2, the `Deimos.configure` block now only takes Deimos-specific settings, and is **not** used to configure producers and consumers. Kafka settings now go in the Karafka `kafka` setting method, and producers and consumers use Karafka [routing](https://karafka.io/docs/Routing/). There are Deimos-specific extensions to routing to apply to consumers and producers, either via a `defaults` block (applying to all consumers and producers) or in individual `topic` blocks:
+
+```ruby
+Deimos.configure do
+  producers.schema_namespace 'com.my-namespace'
+end
+
+class KarafkaApp < Karafka::App
+  setup do |config|
+    config.kafka = {
+      "bootstrap.servers": "my-broker:9092"
+    }
+  end
+  
+  routes.draw do
+    defaults do
+      namespace "com.my-namespace"
+    end
+
+    topic "MyTopic" do
+      # Karafka settings
+      consumer MyConsumer
+      kafka({"session.timeout.ms": 30_000})
+      # Deimos settings
+      schema "MySchema" # the res
+      key_config({field: id})
+    end
+    
+    topic "MyTopic2" do
+      # these are all Deimos settings since Karafka doesn't actually do per-topic producer configs
+      producer_class MyProducer
+      schema 'MySchema2'
+      key_config none: true
+    end
+  end
+end
+```
+
+This configuration must be in a file called `karafka.rb` at the root of your application. The V2 generator will generate this file for you. Without the generator, if you have this file and start up your app with the old `Deimos.configure` code, you will get notifications of the correct places to put these settings.
+
+
+### Removed deprecations
+
+The following were deprecated in version 1.x and are removed in 2.0.
+
+* The `kafka_producer` method for KafkaSource is no longer supported. Please use `kafka_producers`. (This is not addressed by the V2 generator.)
+
+```ruby
+# before:
+class MyRecord < ApplicationRecord
+  def kafka_producer
+    MyProducer
+  end
+end
+
+# after:
+class MyRecord < ApplicationRecord
+  def kafka_producers
+    [MyProducer]
+  end
+end
+```
+
+* The `record_attributes` method for ActiveRecordConsumer now must take two parameters, not one. (The V2 generator can fix this.)
+
+```ruby
+# before:
+class MyConsumer < Deimos::ActiveRecordConsumer
+  def record_attributes(payload)
+    # ...
+  end
+end
+
+# after:
+class MyConsumer < Deimos::ActiveRecordConsumer
+  def record_attributes(payload, key)
+    # ...
+  end
+end
+```
+
+* The `BatchConsumer` class has been removed. Please use the `Consumer` class.
+* You can no longer configure your application using a `phobos.yml` file. The V2 generator will not be able to work on apps using this approach.
+* Removed `test_consume_invalid_message` and `test_consume_batch_invalid_message` test helpers. These did not serve a useful purpose.
+* The following deprecated testing functions have been removed: `stub_producers_and_consumers!`, `stub_producer`, `stub_consumer`, `stub_batch_consumer`. These have not done anything in a long time.
+
+### Major breaking changes
+* Since Karafka only supports Ruby >= 3.0, that means Deimos also only supports those versions.
+* Deimos no longer supports a separate logger from Karafka. When you configure a Karafka logger, Deimos will use that logger for all its logging. (Deimos logs will be prefixed with a `[Deimos]` tag.)
+* The `:db` backend has been renamed to `:outbox`. All associated classes (like `DbProducer`) have likewise been renamed. The Rake task has also been renamed to `rake deimos:outbox`.
+* The `SchemaControllerMixin` has been removed as there was no serious usage for it.
+* `InlineConsumer` has been removed - Karafka Pro has an [Iterator API](https://karafka.io/docs/Pro-Iterator-API/) that does the same thing. There also has been no evidence that it was used (and was probably pretty buggy).
+* The `:test` backend has been removed and the `Deimos::TestHelpers` module is now largely powered by [karafka-testing](https://github.com/karafka/karafka-testing/). This means that you can no longer use `Deimos::Backends::Test.sent_messages` - you need to use `Deimos::TestHelpers.sent_messages`. (The V2 generator should fix this.)
+* Individual consumer and producer settings now live within Karafka route configuration. This means you can no longer call e.g. `consumer.schema` to retrieve this information, as settings are no longer stored directly on the consumer and producer objects (it is still available, but via different methods).
+* Consumers should no longer define a `consume` method, as the semantics have changed with Karafka. Instead, you can define a `consume_message` or `consume_batch` method. Both of these methods now take Karafka `Message` objects instead of hashes. The V2 generator can handle translating this for you, but if you create new consumers, you should take advantage of the Karafka functionality and use it first-class.
+* Phobos `delivery_method` is no longer relevant. Instead, specify an `each_message` setting for your consumer. If set to true, you should define a `consume_message` method. Otherwise, you should define a `consume_batch` method. (Note that this is the reverse from the previous default, which assumed `delivery_method: message`.) The V2 generator will create the correct setting for each consumer.
+
+```ruby
+# before:
+class MyConsumer < Deimos::Consumer
+  def consume(payload, metadata)
+    # payload and metadata are both hashes
+  end
+
+  # OR with delivery_method: inline_batch
+  def batch_consume(payloads, metadata)
+    # payloads is an array of hashes, metadata is a hash
+  end
+end
+
+# now:
+class MyConsumer < Deimos::Consumer
+  def consume_batch
+    payloads = messages.payloads # messages is an instance method and `payloads` will return the decoded hashes
+  end
+
+  # OR with batch(false)
+  def consume_message(message)
+    # message is a Karafka Message object
+    payload = message.payload
+    key = message.key # etc.
+  end
+end
+```
+
+### Metrics
+
+The following metrics have been **removed** in favor of Karafka's more robust [DataDog metrics](https://karafka.io/docs/Monitoring-and-Logging/#datadog-and-statsd-integration) and WaterDrop's [DataDog metrics](https://karafka.io/docs/WaterDrop-Monitoring-and-Logging/#datadog-and-statsd-integration):
+* `consumer_lag` (use `consumer.lags`)
+* `handler` (use `consumer.consumed.time_taken`)
+* `publish` (use `produced_sync` and `produced_async`)
+* `publish_error` (use `deliver.errors`)
+
+You will need to manually add the DataDog MetricsListener as shown in the above pages.
+
+The following metrics have been **renamed**:
+
+* `db_producer.insert` -> `outbox.insert`
+* `db_producer.process` -> `outbox.process`
+
+### Instrumentation
+
+Deimos's own instrumentation layer has been removed in favor of Karafka's. You can still subscribe to Deimos notifications - you simply do it via Karafka's monitor instead of Deimos's.
+
+```ruby
+# before:
+Deimos.subscribe('encode_messages') do |event|	
+    # ... 
+end
+
+# after:
+Karafka.monitor.subscribe('deimos.encode_messages') do |event|
+  # ...
+end
+```
+
+Note that Karafka's monitors do not support the legacy "splatted" subscribe:
+```ruby
+Deimos.subscribe("event") do |*args|
+  payload = ActiveSupport::Notifications::Event.new(*args).payload
+end
+```
+
+The following instrumentation events have been **removed** in favor of Karafka's [events](https://karafka.io/docs/Monitoring-and-Logging/#subscribing-to-the-instrumentation-events):
+
+* `produce_error` (use `error.occurred`)
+
+The following events have been **renamed**:
+* `encode_messages` -> `deimos.encode_message` (**note that only one message is instrumented at a time now**)
+* `db_producer.produce` -> `deimos.outbox.produce`
+* `batch_consumption.valid_records` -> `deimos.batch_consumption.valid_records`
+* `batch_consumption.invalid_records` -> `deimos.batch_consumption.invalid_records`
+
+### Additional breaking changes
+* `key_config` now defaults to `{none: true}` instead of erroring out if not set.
+* `fatal_error?` now receives a Karafka `messages` object instead of a payload hash or array of hashes.
+* `watched_attributes` has been moved from the corresponding ActiveRecord class to the ActiveRecordProducer class. The object being watched is passed into the method.
+* Removed `TestHelpers.full_integration_test!` and `kafka_test!` as Karafka does not currently support these use cases. If we need them back, we will need to put in changes to the testing library to support them.
+* `test_consume_message` and `test_consume_batch` used to not fully validate schemas when using the `:avro_validation` backend. Now these are fully validated, which may cause test errors when upgrading.
+
+### New functionality
+
+* When setting up a Datadog metrics client, you can pass `:karafka_namespace`, `:karafka_distribution_mode`, or `:rd_kafka_metrics` tags to specify the Karafka settings for Datadog metrics.
+- The `payload_log` setting now works for consumers as well as producers, as it is now a topic setting.
+- You can publish messages **without a Deimos Producer class**. Karafka producers take a hash with `:message`, `:topic`, `:key`, `:headers` and `:partition_key` keys. As long as the topic is configured in `karafka.rb`, you don't need a special class to send the message. You can simply call `Karafka.producer.produce()`.
+- The only features that are now available on the bare Producer (as opposed to ActiveRecordProducer) class are:
+    - Outbox backend
+    - Instance method to determine partition key (rather than passing it in)
+    - Using `Deimos.disable_producers`
+- If you need these features, you must continue using a `Deimos::Producer`.
+- You can now call `.produce(messages)` directly on a `Deimos::Producer` which allows for use of these features while still passing a Karafka message hash. This removes the need to add a `payload_key` key into your payload. This is now the recommended method to use in a Deimos Producer.
+
+### New deprecations
+* For testing, you no longer have to call `unit_test!` to get the right settings. It is handled automatically by Karafka. The only thing this method now does is set the schema backend to `:avro_validation`, and you can do that in a single line.
+* The `skip_expectation` and `call_original` arguments to `test_consume_message` and `test_consume_batch` have been deprecated and no longer need to be provided. The assumption is that `call_original` is always true.
+
 ## Upgrading from < 1.5.0 to >= 1.5.0
 
 If you are using Confluent's schema registry to Avro-encode your

data/lib/deimos/active_record_consume/batch_consumption.rb

@@ -4,6 +4,7 @@ require 'deimos/active_record_consume/batch_slicer'
 require 'deimos/active_record_consume/batch_record'
 require 'deimos/active_record_consume/batch_record_list'
 require 'deimos/active_record_consume/mass_updater'
+require 'deimos/consume/batch_consumption'
 
 require 'deimos/utils/deadlock_retry'
 require 'deimos/message'
@@ -14,28 +15,26 @@ module Deimos
     # Methods for consuming batches of messages and saving them to the database
     # in bulk ActiveRecord operations.
     module BatchConsumption
+      include Deimos::Consume::BatchConsumption
+
       # Handle a batch of Kafka messages. Batches are split into "slices",
       # which are groups of independent messages that can be processed together
       # in a single database operation.
       # If two messages in a batch have the same key, we cannot process them
       # in the same operation as they would interfere with each other. Thus
       # they are split
-      # @param payloads [Array<Hash,Deimos::SchemaClass::Record>] Decoded payloads
-      # @param metadata [Hash] Information about batch, including keys.
       # @return [void]
-      def consume_batch(payloads, metadata)
-        messages = payloads.
-          zip(metadata[:keys]).
-          map { |p, k| Deimos::Message.new(p, nil, key: k) }
+      def consume_batch
+        deimos_messages = messages.map { |p| Deimos::Message.new(p.payload, key: p.key) }
 
-        tag = metadata[:topic]
+        tag = topic.name
         Deimos.config.tracer.active_span.set_tag('topic', tag)
 
-        Deimos.instrument('ar_consumer.consume_batch', tag) do
-          if @compacted || self.class.config[:no_keys]
-            update_database(compact_messages(messages))
+        Karafka.monitor.instrument('deimos.ar_consumer.consume_batch', {topic: tag}) do
+          if @compacted && deimos_messages.map(&:key).compact.any?
+            update_database(compact_messages(deimos_messages))
           else
-            uncompacted_update(messages)
+            uncompacted_update(deimos_messages)
           end
         end
       end
@@ -68,11 +67,11 @@ module Deimos
         if key.nil?
           {}
         elsif key.is_a?(Hash)
-          @key_converter.convert(key)
-        elsif self.class.config[:key_field].nil?
+          self.key_converter.convert(key)
+        elsif self.topic.key_config[:field].nil?
           { @klass.primary_key => key }
         else
-            { self.class.config[:key_field] => key }
+          { self.topic.key_config[:field].to_s => key }
         end
       end
 
@@ -83,7 +82,7 @@ module Deimos
       def deleted_query(records)
         keys = records.
           map { |m| record_key(m.key)[@klass.primary_key] }.
-          reject(&:nil?)
+          compact
 
         @klass.unscoped.where(@klass.primary_key => keys)
       end
@@ -154,10 +153,10 @@ module Deimos
         record_list = build_records(messages)
         invalid = filter_records(record_list)
         if invalid.any?
-          ActiveSupport::Notifications.instrument('batch_consumption.invalid_records', {
-                                                    records: invalid,
-                                                    consumer: self.class
-                                                  })
+          Karafka.monitor.instrument('deimos.batch_consumption.invalid_records', {
+            records: invalid,
+            consumer: self.class
+          })
         end
         return if record_list.empty?
 
@@ -167,12 +166,14 @@ module Deimos
         updater = MassUpdater.new(@klass,
                                   key_col_proc: key_col_proc,
                                   col_proc: col_proc,
-                                  replace_associations: self.class.replace_associations,
-                                  bulk_import_id_generator: self.class.bulk_import_id_generator)
-        ActiveSupport::Notifications.instrument('batch_consumption.valid_records', {
-                                                  records: updater.mass_update(record_list),
-                                                  consumer: self.class
-                                                })
+                                  replace_associations: self.replace_associations,
+                                  bulk_import_id_generator: self.bulk_import_id_generator,
+                                  save_associations_first: self.save_associations_first,
+                                  bulk_import_id_column: self.bulk_import_id_column)
+        Karafka.monitor.instrument('deimos.batch_consumption.valid_records', {
+          records: updater.mass_update(record_list),
+          consumer: self.class
+        })
       end
 
       # @param record_list [BatchRecordList]
@@ -203,14 +204,14 @@ module Deimos
           attrs = attrs.merge(record_key(m.key))
           next unless attrs
 
-          col = if @klass.column_names.include?(self.class.bulk_import_id_column.to_s)
-                  self.class.bulk_import_id_column
+          col = if @klass.column_names.include?(self.bulk_import_id_column.to_s)
+                  self.bulk_import_id_column
                 end
 
           BatchRecord.new(klass: @klass,
                           attributes: attrs,
                           bulk_import_column: col,
-                          bulk_import_id_generator: self.class.bulk_import_id_generator)
+                          bulk_import_id_generator: self.bulk_import_id_generator)
         end
         BatchRecordList.new(records.compact)
       end

data/lib/deimos/active_record_consume/mass_updater.rb

@@ -20,10 +20,13 @@ module Deimos
       # @param col_proc [Proc<Class < ActiveRecord::Base>]
       # @param replace_associations [Boolean]
       def initialize(klass, key_col_proc: nil, col_proc: nil,
-                     replace_associations: true, bulk_import_id_generator: nil)
+                     replace_associations: true, bulk_import_id_generator: nil, save_associations_first: false,
+                     bulk_import_id_column: nil)
         @klass = klass
         @replace_associations = replace_associations
         @bulk_import_id_generator = bulk_import_id_generator
+        @save_associations_first = save_associations_first
+        @bulk_import_id_column = bulk_import_id_column&.to_s
 
         @key_cols = {}
         @key_col_proc = key_col_proc
@@ -84,15 +87,67 @@ module Deimos
         end
       end
 
+      # Assign associated records to corresponding primary records
+      # @param record_list [BatchRecordList] RecordList of primary records for this consumer
+      # @return [Hash]
+      def assign_associations(record_list)
+        associations_info = {}
+        record_list.associations.each do |assoc|
+          col = @bulk_import_id_column if assoc.klass.column_names.include?(@bulk_import_id_column)
+          associations_info[[assoc, col]] = []
+        end
+        record_list.batch_records.each do |primary_batch_record|
+          associations_info.each_key do |assoc, col|
+            batch_record = BatchRecord.new(klass: assoc.klass,
+                                           attributes: primary_batch_record.associations[assoc.name],
+                                           bulk_import_column: col,
+                                           bulk_import_id_generator: @bulk_import_id_generator)
+            # Associate this associated batch record's record with the primary record to
+            # retrieve foreign_key after associated records have been saved and primary
+            # keys have been filled
+            primary_batch_record.record.assign_attributes({ assoc.name => batch_record.record })
+            associations_info[[assoc, col]] << batch_record
+          end
+        end
+        associations_info
+      end
+
+      # Save associated records and fill foreign keys on RecordList records
+      # @param record_list [BatchRecordList] RecordList of primary records for this consumer
+      # @param associations_info [Hash] Contains association info
+      def save_associations_first(record_list, associations_info)
+        associations_info.each_value do |records|
+          assoc_record_list = BatchRecordList.new(records)
+          Deimos::Utils::DeadlockRetry.wrap(Deimos.config.tracer.active_span.get_tag('topic')) do
+            save_records_to_database(assoc_record_list)
+          end
+          import_associations(assoc_record_list)
+        end
+        record_list.records.each do |record|
+          associations_info.each_key do |assoc, _|
+            record.assign_attributes({ assoc.foreign_key => record.send(assoc.name).id })
+          end
+        end
+      end
+
       # @param record_list [BatchRecordList]
       # @return [Array<ActiveRecord::Base>]
       def mass_update(record_list)
         # The entire batch should be treated as one transaction so that if
         # any message fails, the whole thing is rolled back or retried
         # if there is deadlock
-        Deimos::Utils::DeadlockRetry.wrap(Deimos.config.tracer.active_span.get_tag('topic')) do
-          save_records_to_database(record_list)
-          import_associations(record_list) if record_list.associations.any?
+
+        if @save_associations_first
+          associations_info = assign_associations(record_list)
+          save_associations_first(record_list, associations_info)
+          Deimos::Utils::DeadlockRetry.wrap(Deimos.config.tracer.active_span.get_tag('topic')) do
+            save_records_to_database(record_list)
+          end
+        else
+          Deimos::Utils::DeadlockRetry.wrap(Deimos.config.tracer.active_span.get_tag('topic')) do
+            save_records_to_database(record_list)
+            import_associations(record_list) if record_list.associations.any?
+          end
         end
         record_list.records
       end

data/lib/deimos/active_record_consume/message_consumption.rb

@@ -1,10 +1,13 @@
 # frozen_string_literal: true
 
+require 'deimos/consume/message_consumption'
+
 module Deimos
   module ActiveRecordConsume
     # Methods for consuming individual messages and saving them to the database
     # as ActiveRecord instances.
     module MessageConsumption
+      include Deimos::Consume::MessageConsumption
       # Find the record specified by the given payload and key.
       # Default is to use the primary key column and the value of the first
       # field in the key.
@@ -26,38 +29,29 @@ module Deimos
         record[record.class.primary_key] = key
       end
 
-      # @param payload [Hash,Deimos::SchemaClass::Record] Decoded payloads
-      # @param metadata [Hash] Information about batch, including keys.
-      # @return [void]
-      def consume(payload, metadata)
-        unless self.process_message?(payload)
-          Deimos.config.logger.debug(
-            message: 'Skipping processing of message',
-            payload: payload,
-            metadata: metadata
-          )
+      # @param message [Karafka::Messages::Message]
+      def consume_message(message)
+        unless self.process_message?(message)
+          Deimos::Logging.log_debug(
+              message: 'Skipping processing of message',
+              payload: message.payload.to_h,
+              metadata: Deimos::Logging.metadata_log_text(message.metadata)
+            )
           return
         end
 
-        key = metadata.with_indifferent_access[:key]
         klass = self.class.config[:record_class]
-        record = fetch_record(klass, (payload || {}).with_indifferent_access, key)
-        if payload.nil?
+        record = fetch_record(klass, message.payload.to_h.with_indifferent_access, message.key)
+        if message.payload.nil?
           destroy_record(record)
           return
         end
         if record.blank?
           record = klass.new
-          assign_key(record, payload, key)
+          assign_key(record, message.payload, message.key)
         end
 
-        # for backwards compatibility
-        # TODO next major release we should deprecate this
-        attrs = if self.method(:record_attributes).parameters.size == 2
-                  record_attributes(payload.with_indifferent_access, key)
-                else
-                  record_attributes(payload.with_indifferent_access)
-                end
+        attrs = record_attributes((message.payload || {}).with_indifferent_access, message.key)
         # don't use attributes= - bypass Rails < 5 attr_protected
         attrs.each do |k, v|
           record.send("#{k}=", v)

data/lib/deimos/active_record_consumer.rb

@@ -30,21 +30,6 @@ module Deimos
         config[:record_class] = klass
       end
 
-      # @return [String,nil]
-      def bulk_import_id_column
-        config[:bulk_import_id_column]
-      end
-
-      # @return [Proc]
-      def bulk_import_id_generator
-        config[:bulk_import_id_generator]
-      end
-
-      # @return [Boolean]
-      def replace_associations
-        config[:replace_associations]
-      end
-
       # @param val [Boolean] Turn pre-compaction of the batch on or off. If true,
       # only the last message for each unique key in a batch is processed.
       # @return [void]
@@ -57,18 +42,48 @@ module Deimos
       def max_db_batch_size(limit)
         config[:max_db_batch_size] = limit
       end
+
+    end
+
+    # @return [Boolean]
+    def replace_associations
+      self.topic.replace_associations
+    end
+
+    # @return [String,nil]
+    def bulk_import_id_column
+      self.topic.bulk_import_id_column
+    end
+
+    # @return [Proc]
+    def bulk_import_id_generator
+      topic.bulk_import_id_generator
+    end
+
+    # @return [Boolean]
+    def save_associations_first
+      topic.save_associations_first
+    end
+
+    def key_decoder
+      self.topic.serializers[:key]&.backend
     end
 
     # Setup
     def initialize
       @klass = self.class.config[:record_class]
-      @converter = ActiveRecordConsume::SchemaModelConverter.new(self.class.decoder, @klass)
+      @compacted = self.class.config[:compacted] != false
+    end
 
-      if self.class.config[:key_schema]
-        @key_converter = ActiveRecordConsume::SchemaModelConverter.new(self.class.key_decoder, @klass)
-      end
+    def converter
+      decoder = self.topic.deserializers[:payload].backend
+      @converter ||= ActiveRecordConsume::SchemaModelConverter.new(decoder, @klass)
+    end
 
-      @compacted = self.class.config[:compacted] != false
+    def key_converter
+      decoder = self.topic.deserializers[:key]&.backend
+      return nil if decoder.nil?
+      @key_converter ||= ActiveRecordConsume::SchemaModelConverter.new(decoder, @klass)
     end
 
     # Override this method (with `super`) if you want to add/change the default
@@ -77,7 +92,7 @@ module Deimos
     # @param _key [String]
     # @return [Hash]
     def record_attributes(payload, _key=nil)
-      @converter.convert(payload)
+      self.converter.convert(payload)
     end
 
     # Override this message to conditionally save records