deimos-ruby 2.1.13 → 2.2.0.pre.beta1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: f53dee42dc005e4b2d17f8e88f0bcd358d3985bddd595af90c05727b98ad47b6
-  data.tar.gz: 56e3494c8c255d223b79214a78766a4140cda5fc0355fe3cbc38cf414fd99027
+  metadata.gz: 5e3d172790c33077ab5f0dc188a8a28b9bb7050cb612bbac833111e7e1caafbe
+  data.tar.gz: 89040d16c763b278cb0dee57c609f3d9386e00d0d8d8a9f331b2b56b22600752
 SHA512:
-  metadata.gz: d0672ef183346354f9df0a4ea3104ea330c52bb25ebc5b3daa373fc97a64be50a242f0e686e1dca7417a083401c1fc0dac9270e651e127a447a46660b48c58e3
-  data.tar.gz: efbf308dd6ec940af43091747c5420bfa95873d20b6f7c1b107af13f225448a191d7016850b4f0baac3b20552d6a8c54c73a22b49bb8b4f48e0a753c2dd4b025
+  metadata.gz: 20e247295e0724264d1db8dbd32a9267663acbbbf6478c32a78333a742b2538195a353bf0892a3ac5f0ea1e27f6dead3836af551e40e05c1f54071b76ba3cf3e
+  data.tar.gz: cdec52113639f6cf7c5c79671386ffd103bbfb16ebf2500913e2015789af8416117ec8b69923fcdc7804cd29a3efe887415e19a4d452443afa2d81649458b028

data/.github/workflows/ci.yml

@@ -36,7 +36,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        ruby: [ '3.0', '3.1', '3.2', '3.3', '3.4' ]
+        ruby: [ '3.2', '3.3', '3.4' ]
 
     steps:
       - uses: actions/checkout@v3

data/CHANGELOG.md

@@ -7,6 +7,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## UNRELEASED
 
+- Feature: Support Protobuf as a schema backend.
+- Feature: Add the ability to specify the schema backend per topic.
+
 # 2.1.13 - 2025-10-03
 - Fix: Union types where the first type is string and has a default value of empty string was incorrectly turning the field into a required argument, which crashed when trying to instantiate it.
 

data/README.md

@@ -84,21 +84,19 @@ For a full configuration reference, please see [the configuration docs ](docs/CO
 
 # Schemas
 
-Deimos was originally written only supporting Avro encoding via a schema registry.
-This has since been expanded to a plugin architecture allowing messages to be
-encoded and decoded via any schema specification you wish. 
+Deimos has a plugin architecture allowing messages to be encoded and decoded via any schema specification you wish. 
 
 Currently we have the following possible schema backends:
 * Avro Local (use pure Avro)
 * Avro Schema Registry (use the Confluent Schema Registry)
 * Avro Validation (validate using an Avro schema but leave decoded - this is useful
   for unit testing and development)
+* Protobuf Schema Registry (use Protobuf with the Confluent Schema Registry)
 * Mock (no actual encoding/decoding).
 
-Note that to use Avro-encoding, you must include the [avro_turf](https://github.com/dasch/avro_turf) gem in your
-Gemfile.
+Note that to use Protobuf, you must include the [proto_turf](https://github.com/flipp-oss/proto_turf) gem in your Gemfile.
 
-Other possible schemas could include [Protobuf](https://developers.google.com/protocol-buffers), [JSONSchema](https://json-schema.org/), etc. Feel free to
+Other possible schemas could [JSONSchema](https://json-schema.org/), etc. Feel free to
 contribute!
 
 To create a new schema backend, please see the existing examples [here](lib/deimos/schema_backends).
@@ -141,6 +139,8 @@ class MyProducer < Deimos::Producer
 end
 ```
 
+Note that if you are using Protobuf, you need to pass a Protobuf message object as the payload - you can't use a bare hash.
+
 ## Auto-added Fields
 
 If your schema has a field called `message_id`, and the payload you give
@@ -276,6 +276,9 @@ MyProducer.publish({
                    })
 ```
 
+> [!IMPORTANT]
+> Protobuf should *not* be used as a key schema, since the binary encoding is [unstable](https://protobuf.dev/programming-guides/encoding/#implications) and may break partitioning. Deimos will automatically convert key fields to plain values and key hashes to JSON.
+
 ## Instrumentation
 
 Deimos will send events through the [Karafka instrumentation monitor](https://karafka.io/docs/Monitoring-and-Logging/#subscribing-to-the-instrumentation-events). 
@@ -1091,6 +1094,10 @@ Bug reports and pull requests are welcome on GitHub at https://github.com/flipp-
 
 If making changes to the generator, you should regenerate the test schema classes by running `bundle exec ./regenerate_test_schema_classes.rb` .
 
+You can regenerate test Protobuf classes by running:
+
+    protoc -I spec/protos --ruby_out=spec/gen --ruby_opt=paths=source_relative spec/protos/**/*.proto
+
 You can/should re-generate RBS types when methods or classes change by running the following:
 
     rbs collection install # if you haven't done it 

data/deimos-ruby.gemspec

@@ -29,6 +29,7 @@ Gem::Specification.new do |spec|
   spec.add_development_dependency('database_cleaner', '~> 2.1')
   spec.add_development_dependency('ddtrace', '>= 0.11')
   spec.add_development_dependency('dogstatsd-ruby', '>= 4.2')
+  spec.add_development_dependency('proto_turf')
   spec.add_development_dependency('guard', '~> 2')
   spec.add_development_dependency('guard-rspec', '~> 4')
   spec.add_development_dependency('guard-rubocop', '~> 1')

data/docs/CONFIGURATION.md

@@ -52,18 +52,18 @@ things you need to reference into local variables before calling `configure`.
 
 ### Schema Configuration
 
-| Config name                 | Default                  | Description                                                                                                                                             |
-|-----------------------------|--------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------|
-| schema.backend              | `:mock`                  | Backend representing the schema encoder/decoder. You can see a full list [here](../lib/deimos/schema_backends).                                         |
-| schema.registry_url         | `http://localhost:8081`  | URL of the Confluent schema registry.                                                                                                                   |
-| schema.user                 | nil                      | Basic auth user.                                                                                                                                        |
-| schema.password             | nil                      | Basic auth password.                                                                                                                                    |
-| schema.path                 | nil                      | Local path to find your schemas.                                                                                                                        |
-| schema.use_schema_classes   | false                    | Set this to true to use generated schema classes in your application.                                                                                   |
-| schema.generated_class_path | `app/lib/schema_classes` | Local path to generated schema classes.                                                                                                                 |
-| schema.nest_child_schemas   | false                    | Set to true to nest subschemas within the generated class for the parent schema.                                                                        |
-| schema.use_full_namespace   | false                    | Set to true to generate folders for schemas matching the full namespace.                                                                                |
-| schema.schema_namespace_map | {}                       | A map of namespace prefixes to base module name(s). Example: { 'com.mycompany.suborg' => ['SchemaClasses'] }. Requires `use_full_namespace` to be true. |
+| Config name                 | Default                  | Description                                                                                                                                                |
+|-----------------------------|--------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| schema.backend              | `:mock`                  | Backend representing the schema encoder/decoder. You can see a full list [here](../lib/deimos/schema_backends).                                            |
+| schema.registry_url         | `http://localhost:8081`  | URL of the Confluent schema registry.                                                                                                                      |
+| schema.user                 | nil                      | Basic auth user.                                                                                                                                           |
+| schema.password             | nil                      | Basic auth password.                                                                                                                                       |
+| schema.path                 | nil                      | Local path to find your schemas.                                                                                                                           |
+| schema.use_schema_classes   | false                    | Set this to true to use generated schema classes in your application.                                                                                      |
+| schema.generated_class_path | `app/lib/schema_classes` | Local path to generated schema classes.                                                                                                                    |
+| schema.nest_child_schemas   | false                    | Set to true to nest subschemas within the generated class for the parent schema.                                                                           |
+| schema.use_full_namespace   | false                    | Set to true to generate folders for schemas matching the full namespace.                                                                                   |
+| schema.schema_namespace_map | {}                       | A map of namespace prefixes to base module name(s). Example: { 'com.mycompany.suborg' => ['SchemaClasses'] }. Requires `use_full_namespace` to be true.    |
 
 ### Outbox Configuration
 
@@ -113,8 +113,8 @@ The following are additional settings that can be added to the `topic` block in
 | Config name        | Default | Description                                                                                                                                                                                     |
 |--------------------|---------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
 | payload_log        | :full   | Determines how much data is logged per payload.</br>`:full` - all keys and payloads are logged.</br>`:keys` - only keys are logged.</br>`:count` - only the total count of messages are logged. |
-| schema             | nil     | Name of the schema to use to encode data before producing.                                                                                                                                      |
-| namespace          | nil     | Namespace of the schema to use when finding it locally.                                                                                                                                         |
+| schema             | nil     | Name of the schema to use to encode data before producing. For Avro, namespace and schema are separated, but protobuf uses only the fully resolved name including package.                      |
+| namespace          | nil     | Namespace of the schema to use when finding it locally. Leave blank for protobuf.                                                                                                               |
 | key_config         | nil     | Configuration hash for message keys. See [Kafka Message Keys](../README.md#kafka-message-keys).                                                                                                 |
 | use_schema_classes | nil     | Set to true or false to enable or disable using the producers schema classes. See [Generated Schema Classes](../README.md#generated-schema-classes).                                            |
 

data/lib/deimos/active_record_consume/message_consumption.rb

@@ -41,7 +41,11 @@ module Deimos
         end
 
         klass = self.class.config[:record_class]
-        record = fetch_record(klass, message.payload.to_h.with_indifferent_access, message.key)
+        payload = message.payload
+        if payload.is_a?(Hash) || payload.nil? || payload.is_a?(SchemaClass::Record)
+          payload = payload.to_h.with_indifferent_access
+        end
+        record = fetch_record(klass, payload, message.key)
         if delete_record?(message)
           destroy_record(record)
           return

data/lib/deimos/config/configuration.rb

@@ -27,7 +27,9 @@ module Deimos # rubocop:disable Metrics/ModuleLength
         if transcoder.respond_to?(:key_field) && transcoder.key_field
           transcoder.backend = Deimos.schema_backend(schema: config.schema,
                                                      namespace: config.namespace)
-          transcoder.backend.generate_key_schema(transcoder.key_field)
+          if transcoder.backend.supports_key_schemas?
+            transcoder.backend.generate_key_schema(transcoder.key_field)
+          end
         end
       end
     end

data/lib/deimos/consume/message_consumption.rb

@@ -43,15 +43,27 @@ module Deimos
       # @param exception [Throwable]
       # @param message [Karafka::Messages::Message]
       def _handle_message_error(exception, message)
-        Deimos::Logging.log_warn(
-          message: 'Error consuming message',
-          handler: self.class.name,
-          metadata: Deimos::Logging.metadata_log_text(message.metadata),
-          key: message.key,
-          data: message.payload,
-          error_message: exception.message,
-          error: exception.backtrace
-        )
+        begin
+          Deimos::Logging.log_warn(
+            message: 'Error consuming message',
+            handler: self.class.name,
+            metadata: Deimos::Logging.metadata_log_text(message.metadata),
+            key: message.key,
+            data: message.payload,
+            error_message: exception.message,
+            error: exception.backtrace
+          )
+        rescue # serialization issues
+          Deimos::Logging.log_warn(
+            message: 'Error consuming message',
+            handler: self.class.name,
+            metadata: Deimos::Logging.metadata_log_text(message.metadata),
+            key: message.raw_key,
+            data: message.raw_payload,
+            error_message: exception.message,
+            error: exception.backtrace
+          )
+        end
 
         _error(exception, Karafka::Messages::Messages.new([message], messages.metadata))
       end

data/lib/deimos/ext/producer_middleware.rb

@@ -1,8 +1,17 @@
 module Deimos
 
   module ProducerMiddleware
+
     class << self
 
+      def allowed_classes
+        arr = [Hash, SchemaClass::Record]
+        if defined?(Google::Protobuf)
+          arr.push(Google::Protobuf.const_get(:AbstractMessage))
+        end
+        @allowed_classes ||= arr.freeze
+      end
+
       def call(message)
         Karafka.monitor.instrument(
           'deimos.encode_message',
@@ -13,9 +22,12 @@ module Deimos
 
           config = Deimos.karafka_config_for(topic: message[:topic])
           return message if config.nil? || config.schema.nil?
-          return if message[:payload] && !message[:payload].is_a?(Hash) && !message[:payload].is_a?(SchemaClass::Record)
+          return if message[:payload] &&
+                    self.allowed_classes.none? { |k| message[:payload].is_a?(k) }
 
-          m = Deimos::Message.new(message[:payload].to_h,
+          payload = message[:payload]
+          payload = payload.to_h if payload.nil? || payload.is_a?(SchemaClass::Record)
+          m = Deimos::Message.new(payload,
                                   headers: message[:headers],
                                   partition_key: message[:partition_key])
           _process_message(m, message, config)
@@ -90,10 +102,19 @@ module Deimos
       # @param key_transcoder [Deimos::Transcoder]
       # @return [String]
       def _retrieve_key(payload, key_transcoder)
-        key = payload.delete(:payload_key)
+        key = payload.try(:delete, :payload_key)
         return key if key || !key_transcoder.respond_to?(:key_field)
 
-        key_transcoder.key_field ? payload[key_transcoder.key_field] : nil
+        if key_transcoder.key_field
+          key = key_transcoder.key_field.to_s.split('.')
+          current = payload
+          key.each do |k|
+            current = current[k] if current
+          end
+          current
+        else
+          nil
+        end
       end
     end
   end

data/lib/deimos/ext/schema_route.rb

@@ -10,6 +10,7 @@ module Deimos
         schema: nil,
         namespace: nil,
         key_config: {none: true},
+        schema_backend: nil,
         use_schema_classes: Deimos.config.schema.use_schema_classes
       }.each do |field, default|
         define_method(field) do |*args|
@@ -26,6 +27,7 @@ module Deimos
         payload = Transcoder.new(
             schema: schema,
             namespace: namespace,
+            backend: schema_backend,
             use_schema_classes: use_schema_classes,
             topic: name
           )
@@ -35,6 +37,7 @@ module Deimos
         if key_config[:plain]
           key = Transcoder.new(
             schema: schema,
+            backend: schema_backend,
             namespace: namespace,
             use_schema_classes: use_schema_classes,
             topic: name
@@ -44,6 +47,7 @@ module Deimos
           if key_config[:field]
             key = Transcoder.new(
               schema: schema,
+              backend: schema_backend,
               namespace: namespace,
               use_schema_classes: use_schema_classes,
               key_field: key_config[:field].to_s,
@@ -52,6 +56,7 @@ module Deimos
           elsif key_config[:schema]
             key = Transcoder.new(
               schema: key_config[:schema] || schema,
+              backend: schema_backend,
               namespace: namespace,
               use_schema_classes: use_schema_classes,
               topic: self.name

data/lib/deimos/message.rb

@@ -23,7 +23,8 @@ module Deimos
     # @param key [String, Integer, Hash]
     # @param partition_key [Integer]
     def initialize(payload, topic: nil, key: nil, headers: nil, partition_key: nil)
-      @payload = payload&.with_indifferent_access
+      @payload = payload
+      @payload = @payload.with_indifferent_access if @payload.is_a?(Hash)
       @topic = topic
       @key = key
       @headers = headers&.with_indifferent_access
@@ -35,7 +36,7 @@ module Deimos
     # @param fields [Array<String>] existing name fields in the schema.
     # @return [void]
     def add_fields(fields)
-      return if @payload.except(:payload_key, :partition_key).blank?
+      return if @payload.to_h.except(:payload_key, :partition_key).blank?
 
       if fields.include?('message_id')
         @payload['message_id'] ||= SecureRandom.uuid

data/lib/deimos/producer.rb

@@ -111,15 +111,20 @@ module Deimos
         backend = determine_backend_class(sync, force_send)
 
         messages = Array(payloads).map do |p|
+          payload = p
+          payload = payload.to_h if p.is_a?(SchemaClass::Record)
           m = {
-            payload: p&.to_h,
+            payload: payload,
             headers: headers,
             topic: topic,
             partition_key: self.partition_key(p)
           }
           if m.dig(:payload, :key).present? && m.dig(:payload, :message).present?
-            m[:key] = m[:payload][:key].to_h
-            m[:payload] = m[:payload][:message].to_h
+            m[:key] = m[:payload][:key]
+            m[:key] = m[:key].to_h if m[:key].nil? || m[:key].is_a?(SchemaClass::Record)
+            m[:payload] = m[:payload][:message]
+            m[:payload] = m[:payload].to_h if m[:payload].nil? ||
+                                              m[:payload].is_a?(SchemaClass::Record)
           end
           m
         end

data/lib/deimos/schema_backends/avro_base.rb

@@ -18,6 +18,15 @@ module Deimos
         @schema_store = AvroTurf::MutableSchemaStore.new(path: Deimos.config.schema.path)
       end
 
+      def supports_key_schemas?
+        true
+      end
+
+      # @return [Boolean]
+      def supports_class_generation?
+        true
+      end
+
       # @override
       def encode_key(key_id, key, topic: nil)
         begin

data/lib/deimos/schema_backends/avro_schema_registry.rb

@@ -21,7 +21,7 @@ module Deimos
 
       # @return [AvroTurf::Messaging]
       def avro_turf_messaging
-        @avro_turf_messaging ||= AvroTurf::Messaging.new(
+         @avro_turf_messaging ||= AvroTurf::Messaging.new(
           schema_store: @schema_store,
           registry_url: Deimos.config.schema.registry_url,
           schemas_path: Deimos.config.schema.path,

data/lib/deimos/schema_backends/base.rb

@@ -41,6 +41,16 @@ module Deimos
         @namespace = namespace
       end
 
+      # @return [Boolean]
+      def supports_key_schemas?
+        false
+      end
+
+      # @return [Boolean]
+      def supports_class_generation?
+        false
+      end
+
       # Encode a payload with a schema. Public method.
       # @param payload [Hash]
       # @param schema [String,Symbol]

data/lib/deimos/schema_backends/proto_base.rb

@@ -0,0 +1,92 @@
+# frozen_string_literal: true
+
+require_relative 'base'
+require 'proto_turf'
+
+module Deimos
+  module SchemaBackends
+    # Encode / decode using Avro, either locally or via schema registry.
+    class ProtoBase < Base
+      SQL_MAP = {
+        string: :string,
+        int32: :integer,
+        uint32: :integer,
+        sint32: :integer,
+        fixed32: :integer,
+        sfixed32: :integer,
+        int64: :bigint,
+        uint64: :bigint,
+        sint64: :bigint,
+        fixed64: :bigint,
+        sfixed64: :bigint,
+        bool: :boolean,
+        bytes: :string,
+        float: :float,
+        message: :record
+      }
+      def proto_schema(schema=@schema)
+        Google::Protobuf::DescriptorPool.generated_pool.lookup(schema)
+      end
+
+      # @override
+      def encode_key(key_id, key, topic: nil)
+        if key.is_a?(Hash)
+          key_id ? key[key_id].to_s : key.sort.to_h.to_json
+        else
+          key.to_s
+        end
+      end
+
+      # @override
+      def decode_key(payload, key_id)
+        val = JSON.parse(payload) rescue payload
+        key_id ? val[key_id.to_s] : val
+      end
+
+      # :nodoc:
+      def sql_type(field)
+        type = field.type
+        return SQL_MAP[type] if SQL_MAP[type]
+        return :array if type.repeated?
+
+        if type == :double
+          warn('Protobuf `double` type turns into SQL `float` type. Please ensure you have the correct `limit` set.')
+          return :float
+        end
+
+        :string
+      end
+
+      def coerce(payload)
+        payload
+      end
+
+      # @override
+      def coerce_field(field, value)
+      end
+
+      # @override
+      def schema_fields
+        proto_schema.to_a.map do |f|
+          SchemaField.new(f.name, f.subtype&.name || 'record', [], nil)
+        end
+      end
+
+      # @override
+      def validate(payload, schema:)
+      end
+
+      # @override
+      def self.mock_backend
+        :mock
+      end
+
+      def generate_key_schema(field_name)
+        raise 'Protobuf cannot generate key schemas! Please use field_config :plain'
+      end
+
+    private
+
+    end
+  end
+end

data/lib/deimos/schema_backends/proto_schema_registry.rb

@@ -0,0 +1,33 @@
+# frozen_string_literal: true
+
+require_relative 'proto_base'
+require 'proto_turf'
+
+module Deimos
+  module SchemaBackends
+    # Encode / decode using the Protobuf schema registry.
+    class ProtoSchemaRegistry < ProtoBase
+
+      # @override
+      def decode_payload(payload, schema:)
+        self.class.proto_turf.decode(payload)
+      end
+
+      # @override
+      def encode_payload(payload, schema: nil, topic: nil)
+        msg = payload.is_a?(Hash) ? proto_schema.msgclass.new(**payload) : payload
+        self.class.proto_turf.encode(msg, subject: topic)
+      end
+
+    private
+
+      # @return [ProtoTurf]
+      def self.proto_turf
+        @proto_turf ||= ProtoTurf.new(
+          registry_url: Deimos.config.schema.registry_url,
+          logger: Karafka.logger
+        )
+      end
+    end
+  end
+end

data/lib/deimos/transcoder.rb

@@ -7,18 +7,22 @@ module Deimos
     # @param namespace [String]
     # @param key_field [Symbol]
     # @param use_schema_classes [Boolean]
+    # @param backend [Symbol]
     # @param topic [String]
-    def initialize(schema:, namespace:, key_field: nil, use_schema_classes: nil, topic: nil)
+    def initialize(schema:, namespace:, key_field: nil, use_schema_classes: nil, topic: nil, backend: nil)
       @schema = schema
       @namespace = namespace
       self.key_field = key_field
       @use_schema_classes = use_schema_classes
+      @backend_type = backend
       @topic = topic
     end
 
     # @return [Class < Deimos::SchemaBackends::Base]
     def backend
-      @backend ||= Deimos.schema_backend(schema: @schema, namespace: @namespace)
+      @backend ||= Deimos.schema_backend(schema: @schema,
+                                         namespace: @namespace,
+                                         backend: @backend_type)
     end
 
     # for use in test helpers

data/lib/deimos/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Deimos
-  VERSION = '2.1.13'
+  VERSION = '2.2.0-beta1'
 end

data/lib/deimos.rb

@@ -54,32 +54,34 @@ module Deimos
 
   class << self
 
+    # @param backend [Symbol, nil]
     # @return [Class<Deimos::SchemaBackends::Base>]
-    def schema_backend_class
-      backend = Deimos.config.schema.backend.to_s
+    def schema_backend_class(backend: nil)
+      backend ||= Deimos.config.schema.backend
 
       require "deimos/schema_backends/#{backend}"
 
-      "Deimos::SchemaBackends::#{backend.classify}".constantize
+      "Deimos::SchemaBackends::#{backend.to_s.classify}".constantize
     end
 
     # @param schema [String, Symbol]
     # @param namespace [String]
     # @return [Deimos::SchemaBackends::Base]
-    def schema_backend(schema:, namespace:)
+    def schema_backend(schema:, namespace:, backend: Deimos.config.schema.backend)
       if config.schema.use_schema_classes
         # Initialize an instance of the provided schema
         # in the event the schema class is an override, the inherited
         # schema and namespace will be applied
         schema_class = Utils::SchemaClass.klass(schema, namespace)
         if schema_class.nil?
-          schema_backend_class.new(schema: schema, namespace: namespace)
+          schema_backend_class(backend: backend).new(schema: schema, namespace: namespace)
         else
           schema_instance = schema_class.allocate
-          schema_backend_class.new(schema: schema_instance.schema, namespace: schema_instance.namespace)
+          schema_backend_class(backend: backend).
+            new(schema: schema_instance.schema, namespace: schema_instance.namespace)
         end
       else
-        schema_backend_class.new(schema: schema, namespace: namespace)
+        schema_backend_class(backend: backend).new(schema: schema, namespace: namespace)
       end
     end
 

data/lib/generators/deimos/schema_class_generator.rb

@@ -48,11 +48,17 @@ module Deimos
         # @param namespace [String]
         # @param key_config [Hash,nil]
         # @return [void]
-        def generate_classes(schema_name, namespace, key_config)
-          schema_base = Deimos.schema_backend(schema: schema_name, namespace: namespace)
+        def generate_classes(schema_name, namespace, key_config, backend: nil)
+          schema_base = Deimos.schema_backend(schema: schema_name,
+                                              namespace: namespace,
+                                              backend: backend)
+          return unless schema_base.supports_class_generation?
+
           schema_base.load_schema
           if key_config&.dig(:schema)
-            key_schema_base = Deimos.schema_backend(schema: key_config[:schema], namespace: namespace)
+            key_schema_base = Deimos.schema_backend(schema: key_config[:schema],
+                                                    namespace: namespace,
+                                                    backend: backend)
             key_schema_base.load_schema
             generate_class_from_schema_base(key_schema_base, key_config: nil)
           end
@@ -174,7 +180,7 @@ module Deimos
 
           found_schemas["#{namespace}.#{schema_name}"] = key_schema_name
           found_schemas["#{namespace}.#{key_schema_name}"] = nil
-          generate_classes(schema_name, namespace, config.key_config)
+          generate_classes(schema_name, namespace, config.key_config, backend: config.schema_backend)
         end
 
         generate_from_schema_files(found_schemas.keys)
@@ -184,7 +190,8 @@ module Deimos
     private
 
       def generate_from_schema_files(found_schemas)
-        schema_store = AvroTurf::MutableSchemaStore.new(path: Deimos.config.schema.path)
+        path = Deimos.config.schema.path || Deimos.config.schema.paths[:avro].first
+        schema_store = AvroTurf::MutableSchemaStore.new(path: path)
         schema_store.load_schemas!
         schema_store.schemas.values.sort_by { |s| "#{s.namespace}#{s.name}" }.each do |schema|
           name = "#{schema.namespace}.#{schema.name}"

data/spec/gen/sample/v1/sample_pb.rb

@@ -0,0 +1,20 @@
+# frozen_string_literal: true
+# Generated by the protocol buffer compiler.  DO NOT EDIT!
+# source: sample/v1/sample.proto
+
+require 'google/protobuf'
+
+require 'google/protobuf/timestamp_pb'
+
+
+descriptor_data = "\n\x16sample/v1/sample.proto\x12\tsample.v1\x1a\x1fgoogle/protobuf/timestamp.proto\"7\n\rNestedMessage\x12\x12\n\nnested_str\x18\x01 \x01(\t\x12\x12\n\nnested_num\x18\x02 \x01(\x05\"\xdb\x02\n\rSampleMessage\x12\x0b\n\x03str\x18\x01 \x01(\t\x12\x0b\n\x03num\x18\x02 \x01(\x05\x12\x0f\n\x07str_arr\x18\x03 \x03(\t\x12\x0c\n\x04\x66lag\x18\x04 \x01(\x08\x12-\n\ttimestamp\x18\x05 \x01(\x0b\x32\x1a.google.protobuf.Timestamp\x12*\n\x06nested\x18\x06 \x01(\x0b\x32\x18.sample.v1.NestedMessageH\x00\x12\x13\n\tunion_str\x18\x07 \x01(\tH\x00\x12\x32\n\x10non_union_nested\x18\x08 \x01(\x0b\x32\x18.sample.v1.NestedMessage\x12\x35\n\x07str_map\x18\t \x03(\x0b\x32$.sample.v1.SampleMessage.StrMapEntry\x1a-\n\x0bStrMapEntry\x12\x0b\n\x03key\x18\x01 \x01(\t\x12\r\n\x05value\x18\x02 \x01(\t:\x02\x38\x01\x42\x07\n\x05unionb\x06proto3"
+
+pool = Google::Protobuf::DescriptorPool.generated_pool
+pool.add_serialized_file(descriptor_data)
+
+module Sample
+  module V1
+    NestedMessage = ::Google::Protobuf::DescriptorPool.generated_pool.lookup("sample.v1.NestedMessage").msgclass
+    SampleMessage = ::Google::Protobuf::DescriptorPool.generated_pool.lookup("sample.v1.SampleMessage").msgclass
+  end
+end

data/spec/protos/sample/v1/sample.proto

@@ -0,0 +1,24 @@
+syntax = "proto3";
+
+package sample.v1;
+
+import "google/protobuf/timestamp.proto";
+
+message NestedMessage {
+  string nested_str = 1;
+  int32 nested_num = 2;
+}
+
+message SampleMessage {
+  string str = 1;
+  int32 num = 2;
+  repeated string str_arr = 3;
+  bool flag = 4;
+  google.protobuf.Timestamp timestamp = 5;
+  oneof union {
+    NestedMessage nested = 6;
+    string union_str = 7;
+  }
+  NestedMessage non_union_nested = 8;
+  map<string, string> str_map = 9;
+}

data/spec/schema_backends/proto_schema_registry_spec.rb

@@ -0,0 +1,50 @@
+# frozen_string_literal: true
+
+require 'deimos/schema_backends/proto_schema_registry'
+require_relative "#{__dir__}/../gen/sample/v1/sample_pb"
+
+RSpec.describe Deimos::SchemaBackends::ProtoSchemaRegistry do
+  let(:payload) do
+    Sample::V1::SampleMessage.new(
+      str: "string",
+      num: 123,
+      str_arr: %w[one two],
+      flag: true,
+      timestamp: Time.utc(2017, 1, 1),
+      nested: Sample::V1::NestedMessage.new(nested_str: "string"),
+      non_union_nested: Sample::V1::NestedMessage.new(nested_num: 456),
+      str_map: {'foo' => 'bar'}
+    )
+  end
+
+  let(:backend) { described_class.new(schema: 'sample.v1.SampleMessage') }
+
+  specify('#encode_key') do
+    expect(backend.encode_key(nil, 789)).to eq('789')
+    expect(backend.encode_key(nil, 'string')).to eq('string')
+    expect(backend.encode_key(nil, {foo: 'bar'})).to eq('{"foo":"bar"}')
+    expect(backend.encode_key(:foo, 'bar')).to eq('bar')
+  end
+
+  specify('#decode_key') do
+    expect(backend.decode_key('789', nil)).to eq(789)
+    expect(backend.decode_key('{"foo":"bar"}', :foo)).to eq('bar')
+    expect(backend.decode_key('{"foo":"bar"}', nil)).to eq({"foo" => 'bar'})
+  end
+
+  it 'should encode and decode correctly' do
+    proto_turf = instance_double(ProtoTurf)
+    expect(proto_turf).to receive(:encode).
+      with(payload, subject: 'topic').
+      and_return('encoded-payload')
+    expect(proto_turf).to receive(:decode).
+      with('encoded-payload').
+      and_return(payload)
+    allow(described_class).to receive(:proto_turf).and_return(proto_turf)
+    results = backend.encode(payload, topic: 'topic')
+    expect(results).to eq('encoded-payload')
+    results = backend.decode(results)
+    expect(results).to eq(payload)
+  end
+
+end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: deimos-ruby
 version: !ruby/object:Gem::Version
-  version: 2.1.13
+  version: 2.2.0.pre.beta1
 platform: ruby
 authors:
 - Daniel Orner
@@ -161,6 +161,20 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '4.2'
+- !ruby/object:Gem::Dependency
+  name: proto_turf
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
 - !ruby/object:Gem::Dependency
   name: guard
   requirement: !ruby/object:Gem::Requirement
@@ -483,6 +497,8 @@ files:
 - lib/deimos/schema_backends/base.rb
 - lib/deimos/schema_backends/mock.rb
 - lib/deimos/schema_backends/plain.rb
+- lib/deimos/schema_backends/proto_base.rb
+- lib/deimos/schema_backends/proto_schema_registry.rb
 - lib/deimos/schema_class/base.rb
 - lib/deimos/schema_class/enum.rb
 - lib/deimos/schema_class/record.rb
@@ -539,6 +555,7 @@ files:
 - spec/batch_consumer_spec.rb
 - spec/consumer_spec.rb
 - spec/deimos_spec.rb
+- spec/gen/sample/v1/sample_pb.rb
 - spec/generators/active_record_generator_spec.rb
 - spec/generators/schema_class/my_schema_spec.rb
 - spec/generators/schema_class/my_schema_with_circular_reference_spec.rb
@@ -553,12 +570,14 @@ files:
 - spec/logging_spec.rb
 - spec/message_spec.rb
 - spec/producer_spec.rb
+- spec/protos/sample/v1/sample.proto
 - spec/rake_spec.rb
 - spec/schema_backends/avro_base_shared.rb
 - spec/schema_backends/avro_local_spec.rb
 - spec/schema_backends/avro_schema_registry_spec.rb
 - spec/schema_backends/avro_validation_spec.rb
 - spec/schema_backends/base_spec.rb
+- spec/schema_backends/proto_schema_registry_spec.rb
 - spec/schemas/com/my-namespace/Generated.avsc
 - spec/schemas/com/my-namespace/MyNestedSchema.avsc
 - spec/schemas/com/my-namespace/MySchema.avsc
@@ -669,6 +688,7 @@ test_files:
 - spec/batch_consumer_spec.rb
 - spec/consumer_spec.rb
 - spec/deimos_spec.rb
+- spec/gen/sample/v1/sample_pb.rb
 - spec/generators/active_record_generator_spec.rb
 - spec/generators/schema_class/my_schema_spec.rb
 - spec/generators/schema_class/my_schema_with_circular_reference_spec.rb
@@ -683,12 +703,14 @@ test_files:
 - spec/logging_spec.rb
 - spec/message_spec.rb
 - spec/producer_spec.rb
+- spec/protos/sample/v1/sample.proto
 - spec/rake_spec.rb
 - spec/schema_backends/avro_base_shared.rb
 - spec/schema_backends/avro_local_spec.rb
 - spec/schema_backends/avro_schema_registry_spec.rb
 - spec/schema_backends/avro_validation_spec.rb
 - spec/schema_backends/base_spec.rb
+- spec/schema_backends/proto_schema_registry_spec.rb
 - spec/schemas/com/my-namespace/Generated.avsc
 - spec/schemas/com/my-namespace/MyNestedSchema.avsc
 - spec/schemas/com/my-namespace/MySchema.avsc