karafka-sidekiq-backend 1.2.0.beta4 → 1.4.0

data/README.md

@@ -1,6 +1,6 @@
 # Karafka Sidekiq Backend
 
-[![Build Status](https://travis-ci.org/karafka/sidekiq-backend.png)](https://travis-ci.org/karafka/karafka-sidekiq-backend)
+[![Build Status](https://github.com/karafka/sidekiq-backend/workflows/ci/badge.svg)](https://github.com/karafka/sidekiq-backend/actions?query=workflow%3Aci)
 [![Join the chat at https://gitter.im/karafka/karafka](https://badges.gitter.im/karafka/karafka.svg)](https://gitter.im/karafka/karafka?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
 
 [Karafka Sidekiq Backend](https://github.com/karafka/sidekiq-backend) provides support for consuming (processing) received Kafka messages inside of Sidekiq workers.
@@ -45,7 +45,8 @@ or on a per topic level:
 App.routes.draw do
   consumer_group :videos_consumer do
     topic :binary_video_details do
-      controller Videos::DetailsController
+      backend :sidekiq
+      consumer Videos::DetailsConsumer
       worker Workers::DetailsWorker
       interchanger Interchangers::MyCustomInterchanger
     end
@@ -53,7 +54,7 @@ App.routes.draw do
 end
 ```
 
-You don't need to do anything beyond that. Karafka will know, that you want to run your controllers ```#perform``` method in a background job.
+You don't need to do anything beyond that. Karafka will know, that you want to run your consumer's ```#consume``` method in a background job.
 
 ## Configuration
 
@@ -67,7 +68,7 @@ There are two options you can set inside of the ```topic``` block:
 
 ### Workers
 
-Karafka by default will build a worker that will correspond to each of your controllers (so you will have a pair - controller and a worker). All of them will inherit from ```ApplicationWorker``` and will share all its settings.
+Karafka by default will build a worker that will correspond to each of your consumers (so you will have a pair - consumer and a worker). All of them will inherit from ```ApplicationWorker``` and will share all its settings.
 
 To run Sidekiq you should have sidekiq.yml file in *config* folder. The example of ```sidekiq.yml``` file will be generated to config/sidekiq.yml.example once you run ```bundle exec karafka install```.
 
@@ -75,17 +76,17 @@ However, if you want to use a raw Sidekiq worker (without any Karafka additional
 
 ```ruby
 topic :incoming_messages do
-  controller MessagesController
+  consumer MessagesConsumer
   worker MyCustomWorker
 end
 ```
 
-Note that even then, you need to specify a controller that will schedule a background task.
+Note that even then, you need to specify a consumer that will schedule a background task.
 
 Custom workers need to provide a ```#perform_async``` method. It needs to accept two arguments:
 
  - ```topic_id``` - first argument is a current topic id from which a given message comes
- - ```params_batch``` - all the params that came from Kafka + additional metadata. This data format might be changed if you use custom interchangers. Otherwise it will be an instance of Karafka::Params::ParamsBatch.
+ - ```params_batch``` - all the params that came from Kafka + additional metadata. This data format might be changed if you use custom interchangers. Otherwise, it will be an instance of Karafka::Params::ParamsBatch.
 
 **Note**: If you use custom interchangers, keep in mind, that params inside params batch might be in two states: parsed or unparsed when passed to #perform_async. This means, that if you use custom interchangers and/or custom workers, you might want to look into Karafka's sources to see exactly how it works.
 
@@ -93,19 +94,50 @@ Custom workers need to provide a ```#perform_async``` method. It needs to accept
 
 Custom interchangers target issues with non-standard (binary, etc.) data that we want to store when we do ```#perform_async```. This data might be corrupted when fetched in a worker (see [this](https://github.com/karafka/karafka/issues/30) issue). With custom interchangers, you can encode/compress data before it is being passed to scheduling and decode/decompress it when it gets into the worker.
 
+To specify the interchanger for a topic, specify the interchanger inside routes like this:
+
+```ruby
+App.routes.draw do
+  consumer_group :videos_consumer do
+    topic :binary_video_details do
+      consumer Videos::DetailsConsumer
+      interchanger Interchangers::MyCustomInterchanger
+    end
+  end
+end
+```
+Each custom interchanger should define `encode` to encode params before they get stored in Redis, and `decode` to convert the params to hash format, as shown below:
+
+```ruby
+class Base64Interchanger
+  class << self
+    def encode(params_batch)
+      # Note, that you need to cast the params_batch to an array in order to get it work
+      # in sidekiq later
+      Base64.encode64(Marshal.dump(params_batch.to_a))
+    end
+
+    def decode(params_string)
+      Marshal.load(Base64.decode64(params_string))
+    end
+  end
+end
+
+```
+
 **Warning**: if you decide to use slow interchangers, they might significantly slow down Karafka.
 
 ## References
 
 * [Karafka framework](https://github.com/karafka/karafka)
-* [Karafka Sidekiq Backend Travis CI](https://travis-ci.org/karafka/karafka-sidekiq-backend)
+* [Karafka Sidekiq Backend Actions CI](https://github.com/karafka/sidekiq-backend/actions?query=workflow%3Aci)
 * [Karafka Sidekiq Backend Coditsu](https://app.coditsu.io/karafka/repositories/karafka-sidekiq-backend)
 
 ## Note on contributions
 
 First, thank you for considering contributing to Karafka! It's people like you that make the open source community such a great community!
 
-Each pull request must pass all the rspec specs and meet our quality requirements.
+Each pull request must pass all the RSpec specs and meet our quality requirements.
 
 To check if everything is as it should be, we use [Coditsu](https://coditsu.io) that combines multiple linters and code analyzers for both code and documentation. Once you're done with your changes, submit a pull request.
 

data/certs/mensfeld.pem

@@ -0,0 +1,25 @@
+-----BEGIN CERTIFICATE-----
+MIIEODCCAqCgAwIBAgIBATANBgkqhkiG9w0BAQsFADAjMSEwHwYDVQQDDBhtYWNp
+ZWovREM9bWVuc2ZlbGQvREM9cGwwHhcNMjAwODExMDkxNTM3WhcNMjEwODExMDkx
+NTM3WjAjMSEwHwYDVQQDDBhtYWNpZWovREM9bWVuc2ZlbGQvREM9cGwwggGiMA0G
+CSqGSIb3DQEBAQUAA4IBjwAwggGKAoIBgQDCpXsCgmINb6lHBXXBdyrgsBPSxC4/
+2H+weJ6L9CruTiv2+2/ZkQGtnLcDgrD14rdLIHK7t0o3EKYlDT5GhD/XUVhI15JE
+N7IqnPUgexe1fbZArwQ51afxz2AmPQN2BkB2oeQHXxnSWUGMhvcEZpfbxCCJH26w
+hS0Ccsma8yxA6hSlGVhFVDuCr7c2L1di6cK2CtIDpfDaWqnVNJEwBYHIxrCoWK5g
+sIGekVt/admS9gRhIMaIBg+Mshth5/DEyWO2QjteTodItlxfTctrfmiAl8X8T5JP
+VXeLp5SSOJ5JXE80nShMJp3RFnGw5fqjX/ffjtISYh78/By4xF3a25HdWH9+qO2Z
+tx0wSGc9/4gqNM0APQnjN/4YXrGZ4IeSjtE+OrrX07l0TiyikzSLFOkZCAp8oBJi
+Fhlosz8xQDJf7mhNxOaZziqASzp/hJTU/tuDKl5+ql2icnMv5iV/i6SlmvU29QNg
+LCV71pUv0pWzN+OZbHZKWepGhEQ3cG9MwvkCAwEAAaN3MHUwCQYDVR0TBAIwADAL
+BgNVHQ8EBAMCBLAwHQYDVR0OBBYEFImGed2AXS070ohfRidiCEhXEUN+MB0GA1Ud
+EQQWMBSBEm1hY2llakBtZW5zZmVsZC5wbDAdBgNVHRIEFjAUgRJtYWNpZWpAbWVu
+c2ZlbGQucGwwDQYJKoZIhvcNAQELBQADggGBAKiHpwoENVrMi94V1zD4o8/6G3AU
+gWz4udkPYHTZLUy3dLznc/sNjdkJFWT3E6NKYq7c60EpJ0m0vAEg5+F5pmNOsvD3
+2pXLj9kisEeYhR516HwXAvtngboUcb75skqvBCU++4Pu7BRAPjO1/ihLSBexbwSS
+fF+J5OWNuyHHCQp+kGPLtXJe2yUYyvSWDj3I2//Vk0VhNOIlaCS1+5/P3ZJThOtm
+zJUBI7h3HgovwRpcnmk2mXTmU4Zx/bCzX8EA6VY0khEvnmiq7S6eBF0H9qH8KyQ6
+EkVLpvmUDFcf/uNaBQdazEMB5jYtwoA8gQlANETNGPi51KlkukhKgaIEDMkBDJOx
+65N7DzmkcyY0/GwjIVIxmRhcrCt1YeCUElmfFx0iida1/YRm6sB2AXqScc1+ECRi
+2DND//YJUikn1zwbz1kT70XmHd97B4Eytpln7K+M1u2g1pHVEPW4owD/ammXNpUy
+nt70FcDD4yxJQ+0YNiHd0N8IcVBM1TMIVctMNQ==
+-----END CERTIFICATE-----

data/karafka-sidekiq-backend.gemspec

@@ -9,16 +9,21 @@ Gem::Specification.new do |spec|
   spec.version     = Karafka::Backends::Sidekiq::VERSION
   spec.platform    = Gem::Platform::RUBY
   spec.authors     = ['Maciej Mensfeld']
-  spec.email       = %w[maciej@coditsu.io]
+  spec.email       = %w[maciej@mensfeld.pl]
   spec.homepage    = 'https://github.com/karafka/karafka-sidekiq-backend'
   spec.summary     = 'Karafka Sidekiq backend for background messages processing'
   spec.description = 'Karafka Sidekiq backend for background messages processing'
-  spec.license     = 'MIT'
+  spec.license     = 'LGPL-3.0'
 
-  spec.add_dependency 'karafka', '>= 1.2.0.beta4'
+  spec.add_dependency 'karafka', '~> 1.4.0.rc2'
   spec.add_dependency 'sidekiq', '>= 4.2'
-  spec.required_ruby_version = '>= 2.3.0'
+  spec.required_ruby_version = '>= 2.5.0'
 
+  if $PROGRAM_NAME.end_with?('gem')
+    spec.signing_key = File.expand_path('~/.ssh/gem-private_key.pem')
+  end
+
+  spec.cert_chain    = %w[certs/mensfeld.pem]
   spec.files         = `git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(spec)/}) }
   spec.executables   = spec.files.grep(%r{^bin/}) { |f| File.basename(f) }
   spec.require_paths = %w[lib]

data/lib/karafka-sidekiq-backend.rb

@@ -1,34 +1,3 @@
 # frozen_string_literal: true
 
-# @note Active Support is already included since Karafka is using it directly so no need
-#   to require it again in the gemspec, etc
-%w[
-  karafka
-  sidekiq
-  active_support/core_ext/class/subclasses
-].each(&method(:require))
-
-# Karafka framework namespace
-module Karafka
-  # Namespace for all the backends that process data
-  module Backends
-    # Sidekiq Karafka backend
-    module Sidekiq
-      class << self
-        # @return [String] path to Karafka gem root core
-        def core_root
-          Pathname.new(File.expand_path('karafka', __dir__))
-        end
-      end
-    end
-  end
-end
-
-# Uses Karafka loader to load all the sources that this backend needs
-Karafka::Loader.load!(Karafka::Backends::Sidekiq.core_root)
-Karafka::AttributesMap.prepend(Karafka::Extensions::SidekiqAttributesMap)
-# Register internal events for instrumentation
-%w[
-  backends.sidekiq.process
-  backends.sidekiq.base_worker.perform
-].each(&Karafka.monitor.method(:register_event))
+require 'karafka_sidekiq_backend'

data/lib/karafka/backends/sidekiq.rb

@@ -1,11 +1,12 @@
 # frozen_string_literal: true
 
 module Karafka
+  # Namespace for alternative processing backends for Karafka framework
   module Backends
     # Sidekiq backend that schedules stuff to Sidekiq worker for delayed execution
     module Sidekiq
       # Karafka Sidekiq backend version
-      VERSION = '1.2.0.beta4'
+      VERSION = '1.4.0'
 
       # Enqueues the execution of perform method into a worker.
       # @note Each worker needs to have a class #perform_async method that will allow us to pass
@@ -16,7 +17,8 @@ module Karafka
         Karafka.monitor.instrument('backends.sidekiq.process', caller: self) do
           topic.worker.perform_async(
             topic.id,
-            topic.interchanger.encode(params_batch)
+            topic.interchanger.encode(params_batch),
+            respond_to?(:batch_metadata) ? batch_metadata.to_h : nil
           )
         end
       end

data/lib/karafka/base_worker.rb

@@ -5,11 +5,30 @@ module Karafka
   class BaseWorker
     include Sidekiq::Worker
 
+    class << self
+      # Returns the base worker class for application.
+      #
+      # @return [Class] first worker that inherited from Karafka::BaseWorker. Karafka
+      #   assumes that it is the base worker for an application.
+      # @raise [Karafka::Errors::BaseWorkerDescentantMissing] raised when application
+      #   base worker was not defined.
+      def base_worker
+        @inherited || raise(Errors::BaseWorkerDescentantMissing)
+      end
+
+      # @param subclass [Class] subclass of the worker
+      # @return [Class] subclass of the worker that was selected
+      def inherited(subclass)
+        @inherited ||= subclass
+      end
+    end
+
     # Executes the logic that lies in #perform Karafka consumer method
     # @param topic_id [String] Unique topic id that we will use to find a proper topic
-    # @param params_batch [Array] Array with messages batch
-    def perform(topic_id, params_batch)
-      consumer = consumer(topic_id, params_batch)
+    # @param params_batch [Array<Hash>] Array with messages batch
+    # @param metadata [Hash, nil] hash with all the metadata or nil if not present
+    def perform(topic_id, params_batch, metadata)
+      consumer = consumer(topic_id, params_batch, metadata)
 
       Karafka.monitor.instrument(
         'backends.sidekiq.base_worker.perform',
@@ -20,12 +39,27 @@ module Karafka
 
     private
 
+    # @see `#perform` for exact params descriptions
+    # @param topic_id [String]
+    # @param params_batch [Array<Hash>]
+    # @param metadata [Hash, nil]
     # @return [Karafka::Consumer] descendant of Karafka::BaseConsumer that matches the topic
     #   with params_batch assigned already (consumer is ready to use)
-    def consumer(topic_id, params_batch)
+    def consumer(topic_id, params_batch, metadata)
       topic = Karafka::Routing::Router.find(topic_id)
-      consumer = topic.consumer.new
-      consumer.params_batch = topic.interchanger.decode(params_batch)
+      consumer = topic.consumer.new(topic)
+      consumer.params_batch = Params::Builders::ParamsBatch.from_array(
+        topic.interchanger.decode(params_batch),
+        topic
+      )
+
+      if topic.batch_fetching
+        consumer.batch_metadata = Params::Builders::BatchMetadata.from_hash(
+          metadata,
+          topic
+        )
+      end
+
       consumer
     end
   end

data/lib/karafka/extensions/batch_metadata_builder.rb

@@ -0,0 +1,23 @@
+# frozen_string_literal: true
+
+module Karafka
+  module Extensions
+    # Extension for metadata builder to allow building metadata from a hash
+    module BatchMetadataBuilder
+      # Builds metadata from hash
+      # @param hash [Hash] hash with metadata
+      # @param topic [Karafka::Routing::Topic] topic instance
+      # @return [Karafka::Params::Metadata] metadata instance
+      def from_hash(hash, topic)
+        # Parser needs to be merged as this is the only non-serializable object
+        # so it gets reconstructed from the topic
+        Karafka::Params::BatchMetadata
+          .new(
+            **hash
+              .merge('deserializer' => topic.deserializer)
+              .transform_keys(&:to_sym)
+          )
+      end
+    end
+  end
+end

data/lib/karafka/extensions/params_batch_builder.rb

@@ -0,0 +1,21 @@
+# frozen_string_literal: true
+
+module Karafka
+  module Extensions
+    # Extension for params batch builder for reconstruction of the batch from an array
+    module ParamsBatchBuilder
+      # Builds params batch from array of hashes
+      # @param array [Array<Hash>] array with hash messages
+      # @param topic [Karafka::Routing::Topic] topic for which we build the batch
+      # @return [Karafka::Params::ParamsBatch] built batch
+      # @note We rebuild the params batch from array after the serialization
+      def from_array(array, topic)
+        params_array = array.map do |hash|
+          Karafka::Params::Builders::Params.from_hash(hash, topic)
+        end
+
+        Karafka::Params::ParamsBatch.new(params_array).freeze
+      end
+    end
+  end
+end

data/lib/karafka/extensions/params_builder.rb

@@ -0,0 +1,24 @@
+# frozen_string_literal: true
+
+module Karafka
+  module Extensions
+    # Extension for rebuilding params from a hash
+    module ParamsBuilder
+      # Builds params from a hash
+      # @param hash [Hash] hash with params details
+      # @param topic [Karafka::Routing::Topic] topic for which we build the params
+      # @return [Karafka::Params::Params] built params
+      def from_hash(hash, topic)
+        metadata = Karafka::Params::Metadata.new(
+          **hash
+            .fetch('metadata')
+            .merge('deserializer' => topic.deserializer)
+            .transform_keys(&:to_sym)
+        ).freeze
+
+        Karafka::Params::Params
+          .new(hash.fetch('raw_payload'), metadata)
+      end
+    end
+  end
+end

data/lib/karafka/extensions/sidekiq_attributes_map.rb

@@ -1,6 +1,7 @@
 # frozen_string_literal: true
 
 module Karafka
+  # Extensions for Karafka components
   module Extensions
     # Additional Karafka::Attributes map topic attributes that can be used when worker
     # is active and we use sidekiq backend

data/lib/karafka/extensions/sidekiq_topic_attributes.rb

@@ -1,8 +1,6 @@
 # frozen_string_literal: true
 
 module Karafka
-  # Namespace for additional extensions that we include into some Karafka components, to gain
-  # extra features that we require
   module Extensions
     # Additional Karafka::Routing::Topic methods that are required to work with Sidekiq backend
     module SidekiqTopicAttributes
@@ -16,7 +14,7 @@ module Karafka
       # @return [Class] Interchanger class (not an instance) that we want to use to interchange
       #   params between Karafka server and Karafka background job
       def interchanger
-        @interchanger ||= Karafka::Interchanger
+        @interchanger ||= Karafka::Interchanger.new
       end
 
       # Creates attributes writers for worker and interchanger, so they can be overwritten
@@ -28,5 +26,3 @@ module Karafka
     end
   end
 end
-
-Karafka::Routing::Topic.include Karafka::Extensions::SidekiqTopicAttributes

data/lib/karafka/extensions/stdout_listener.rb

@@ -0,0 +1,27 @@
+# frozen_string_literal: true
+
+module Karafka
+  module Extensions
+    # Additional methods for listener that listen on instrumentation related to the Sidekiq
+    # backend of Karafka
+    module StdoutListener
+      # Logs info about scheduling of a certain dataset with a Sidekiq backend
+      # @param event [Dry::Events::Event] event details including payload
+      def on_backends_sidekiq_process(event)
+        count = event[:caller].send(:params_batch).size
+        topic = event[:caller].topic.name
+        time = event[:time]
+        info "Scheduling of #{count} messages to Sidekiq on topic #{topic} took #{time} ms"
+      end
+
+      # Logs ino about processing certain events with a given Sidekiq worker
+      # @param event [Dry::Events::Event] event details including payload
+      def on_backends_sidekiq_base_worker_perform(event)
+        count = event[:consumer].send(:params_batch).size
+        topic = event[:consumer].topic.name
+        time = event[:time]
+        info "Sidekiq processing of topic #{topic} with #{count} messages took #{time} ms"
+      end
+    end
+  end
+end

data/lib/karafka/interchanger.rb

@@ -1,7 +1,7 @@
 # frozen_string_literal: true
 
 module Karafka
-  # Interchangers allow us to format/encode/pack data that is being send to perform_async
+  # Interchanger allows us to format/encode/pack data that is being send to perform_async
   # This is meant to target mostly issues with data encoding like this one:
   # https://github.com/mperham/sidekiq/issues/197
   # Each custom interchanger should implement following methods:
@@ -9,26 +9,22 @@ module Karafka
   #   - decode - decoded params back to a hash format that we can use
   #
   # This interchanger uses default Sidekiq options to exchange data
-  # @note Since we use symbols for Karafka params (performance reasons), they will be
-  #   deserialized into string versions. Keep that in mind.
   class Interchanger
-    class << self
-      # @param params_batch [Karafka::Params::ParamsBatch] Karafka params batch object
-      # @return [Karafka::Params::ParamsBatch] parsed params batch. There are too many problems
-      #   with passing unparsed data from Karafka to Sidekiq, to make it a default. In case you
-      #   need this, please implement your own interchanger.
-      def encode(params_batch)
-        params_batch.parsed
+    # @param params_batch [Karafka::Params::ParamsBatch] Karafka params batch object
+    # @return [Array<Hash>] Array with hash built out of params data
+    def encode(params_batch)
+      params_batch.map do |param|
+        {
+          raw_payload: param.raw_payload,
+          metadata: param.metadata.to_h
+        }
       end
+    end
 
-      # @param params_batch [Array<Hash>] Sidekiq params that are now an array
-      # @note Since Sidekiq does not like symbols, we restore symbolized keys for system keys, so
-      #   everything can work as expected. Keep in mind, that custom data will always be assigned
-      #   with string keys per design. To change it, please change this interchanger and create
-      #   your own custom parser
-      def decode(params_batch)
-        params_batch
-      end
+    # @param params_batch [Array<Hash>] Sidekiq params that are now an array
+    # @return [Array<Hash>] exactly what we've fetched from Sidekiq
+    def decode(params_batch)
+      params_batch
     end
   end
 end