karafka 1.0.1 → 1.1.0.alpha1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: c7d3b30943ab27b3ae2d8609e645590ca918ff2b
-  data.tar.gz: 0b6b31b10d070c07d45ab6bb8bc177e94d4d9c3c
+  metadata.gz: 3a4a0b649f34461fbeef206fdc2c92bd3e6d6ff2
+  data.tar.gz: 87c937be4ee2bdfdc9b1957e458754e88cb15231
 SHA512:
-  metadata.gz: b8a3f16977e6af10f6cda783c326be7bb8540b483cacfe225fc7c422cad091d9014950c6a6de4a9f5cd033b974a41de73bc445e7813063d12013684f056d35df
-  data.tar.gz: 2e425da4e9e420ad092b3b563e5fa796f3f73159bc9c5d6cdf77bbf8463842f0a9379fe156b54328a3f65144f08ec657fc6f65aa01295a3efc40181e4bd0d858
+  metadata.gz: bee787850078c780417e90a87ea5945b12ef84b70e49a888696fd37454de8301fa25130a0f2b736ac8ea9df2a72a58837fe37aced3a3d2f7261bf11546cf23d6
+  data.tar.gz: 3c42797b1d29ca0d107b570c42d9f7b28e45c1f6c3664d8acd055379594569eb2fd801c8c9e44bc9120ee001fe2c4ade401577cb82024689549c9ff67b6177cb

data/CHANGELOG.md

@@ -1,5 +1,29 @@
 # Karafka framework changelog
 
+## 1.1.0 Unreleased
+- Gem bump
+- Switch from Celluloid to native Thread management
+- Improved shutdown process
+- Introduced optional fetch callbacks and moved current the ```after_received``` there as well
+- Karafka will raise Errors::InvalidPauseTimeout exception when trying to pause but timeout set to 0
+- Allow float for timeouts and other time based second settings
+- Renamed MessagesProcessor to Processor and MessagesConsumer to Consumer - we don't process and don't consumer anything else so it was pointless to keep this "namespace"
+- #232 - Remove unused ActiveSupport require
+- #214 - Expose consumer on a controller layer
+- #193 - Process shutdown callbacks
+- Fixed accessibility of ```#params_batch``` from the outside of the controller
+- connection_pool config options are no longer required
+- celluloid config options are no longer required
+- ```#perform``` is not renamed to ```#consume``` with warning level on using the old one (deprecated)
+- #235 - Rename perform to consume
+- Upgrade to ruby-kafka 0.5
+- Due to redesign of Waterdrop concurrency setting is no longer needed
+- #236 - Manual offset management
+- WaterDrop 1.0.0 support with async
+- Renamed ```batch_consuming``` option to ```batch_fetching``` as it is not a consumption (with processing) but a process of fetching messages from Kafka. The messages is considered consumed, when it is processed.
+- Renamed ```batch_processing``` to ```batch_consuming``` to resemble Kafka concept of consuming messages.
+- Renamed ```after_received``` to ```after_fetched``` to normalize the naming conventions.
+
 ## 1.0.1
 - #210 - LoadError: cannot load such file -- [...]/karafka.rb
 - Ruby 2.4.2 as a default (+travis integration)
@@ -60,11 +84,11 @@
 
 ### New features and improvements
 
-- batch processing thanks to ```#batch_processing``` flag and ```#params_batch``` on controllers
+- batch processing thanks to ```#batch_consuming``` flag and ```#params_batch``` on controllers
 - ```#topic``` method on an controller instance to make a clear distinction in between params and route details
 - Changed routing model (still compatible with 0.5) to allow better resources management
 - Lower memory requirements due to object creation limitation (2-3 times less objects on each new message)
-- Introduced the ```#batch_processing``` config flag (config for #126) that can be set per each consumer_group
+- Introduced the ```#batch_consuming``` config flag (config for #126) that can be set per each consumer_group
 - Added support for partition, offset and partition key in the params hash
 - ```name``` option in config renamed to ```client_id```
 - Long running controllers with ```persistent``` flag on a topic config level, to make controller instances persistent between messages batches (single controller instance per topic per partition no per messages batch) - turned on by default
@@ -77,7 +101,7 @@
 - ```start_from_beginning``` moved into kafka scope (```kafka.start_from_beginning```)
 - Router no longer checks for route uniqueness - now you can define same routes for multiple kafkas and do a lot of crazy stuff, so it's your responsibility to check uniqueness
 - Change in the way we identify topics in between Karafka and Sidekiq workers. If you upgrade, please make sure, all the jobs scheduled in Sidekiq are finished before the upgrade.
-- ```batch_mode``` renamed to ```batch_consuming```
+- ```batch_mode``` renamed to ```batch_fetching```
 - Renamed content to value to better resemble ruby-kafka internal messages naming convention
 - When having a responder with ```required``` topics and not using ```#respond_with``` at all, it will raise an exception
 - Renamed ```inline_mode``` to ```inline_processing``` to resemble other settings conventions

data/Gemfile

@@ -5,6 +5,7 @@ source 'https://rubygems.org'
 gemspec
 
 group :development, :test do
+  gem 'waterdrop'
   gem 'timecop'
   gem 'rspec'
   gem 'simplecov'

data/Gemfile.lock

@@ -1,18 +1,17 @@
 PATH
   remote: .
   specs:
-    karafka (1.0.1)
+    karafka (1.1.0.alpha1)
       activesupport (>= 5.0)
-      celluloid
       dry-configurable (~> 0.7)
       dry-validation (~> 0.11)
       envlogic (~> 1.0)
       multi_json (>= 1.12)
       rake (>= 11.3)
       require_all (>= 1.4)
-      ruby-kafka (>= 0.4)
+      ruby-kafka (>= 0.5)
       thor (~> 0.19)
-      waterdrop (~> 0.4)
+      waterdrop (>= 1.0.alpha2)
 
 GEM
   remote: https://rubygems.org/
@@ -22,25 +21,10 @@ GEM
       i18n (~> 0.7)
       minitest (~> 5.1)
       tzinfo (~> 1.1)
-    celluloid (0.17.3)
-      celluloid-essentials
-      celluloid-extras
-      celluloid-fsm
-      celluloid-pool
-      celluloid-supervision
-      timers (>= 4.1.1)
-    celluloid-essentials (0.20.5)
-      timers (>= 4.1.1)
-    celluloid-extras (0.20.5)
-      timers (>= 4.1.1)
-    celluloid-fsm (0.20.5)
-      timers (>= 4.1.1)
-    celluloid-pool (0.20.5)
-      timers (>= 4.1.1)
-    celluloid-supervision (0.20.6)
-      timers (>= 4.1.1)
     concurrent-ruby (1.0.5)
-    connection_pool (2.2.1)
+    delivery_boy (0.2.2)
+      king_konf (~> 0.1.8)
+      ruby-kafka (~> 0.4)
     diff-lcs (1.3)
     docile (1.1.5)
     dry-configurable (0.7.0)
@@ -72,11 +56,11 @@ GEM
       dry-types (~> 0.12.0)
     envlogic (1.0.4)
       activesupport
-    hitimes (1.2.6)
     i18n (0.9.0)
       concurrent-ruby (~> 1.0)
     inflecto (0.0.2)
     json (2.1.0)
+    king_konf (0.1.8)
     minitest (5.10.3)
     multi_json (1.12.2)
     null-logger (0.1.4)
@@ -95,7 +79,7 @@ GEM
       diff-lcs (>= 1.2.0, < 2.0)
       rspec-support (~> 3.7.0)
     rspec-support (3.7.0)
-    ruby-kafka (0.4.3)
+    ruby-kafka (0.5.0)
     simplecov (0.15.1)
       docile (~> 1.1.0)
       json (>= 1.8, < 3)
@@ -104,17 +88,14 @@ GEM
     thor (0.20.0)
     thread_safe (0.3.6)
     timecop (0.9.1)
-    timers (4.1.2)
-      hitimes
     tzinfo (1.2.4)
       thread_safe (~> 0.1)
-    waterdrop (0.4.0)
-      bundler
-      connection_pool
-      dry-configurable (~> 0.6)
+    waterdrop (1.0.0.alpha2)
+      delivery_boy (>= 0.2.2)
+      dry-configurable (~> 0.7)
+      dry-validation (~> 0.11)
       null-logger
-      rake
-      ruby-kafka (~> 0.4)
+      ruby-kafka (>= 0.5)
 
 PLATFORMS
   ruby
@@ -124,6 +105,7 @@ DEPENDENCIES
   rspec
   simplecov
   timecop
+  waterdrop
 
 BUNDLED WITH
    1.15.4

data/README.md

@@ -80,7 +80,7 @@ Unfortunately, it does not yet support independent forks, however you should be
 Please run:
 
 ```bash
-bundle exec rake
+bundle exec rspec spec
 ```
 
 to check if everything is in order. After that you can submit a pull request.

data/karafka.gemspec

@@ -17,16 +17,15 @@ Gem::Specification.new do |spec|
   spec.license     = 'MIT'
 
   spec.add_dependency 'activesupport', '>= 5.0'
-  spec.add_dependency 'celluloid'
   spec.add_dependency 'dry-configurable', '~> 0.7'
   spec.add_dependency 'dry-validation', '~> 0.11'
   spec.add_dependency 'envlogic', '~> 1.0'
   spec.add_dependency 'multi_json', '>= 1.12'
   spec.add_dependency 'rake', '>= 11.3'
   spec.add_dependency 'require_all', '>= 1.4'
-  spec.add_dependency 'ruby-kafka', '>= 0.4'
+  spec.add_dependency 'ruby-kafka', '>= 0.5'
   spec.add_dependency 'thor', '~> 0.19'
-  spec.add_dependency 'waterdrop', '~> 0.4'
+  spec.add_dependency 'waterdrop', '>= 1.0.alpha2'
 
   spec.required_ruby_version = '>= 2.3.0'
 

data/lib/karafka.rb

@@ -2,8 +2,6 @@
 
 %w[
   English
-  bundler
-  celluloid/current
   waterdrop
   kafka
   envlogic
@@ -14,7 +12,6 @@
   dry-configurable
   dry-validation
   active_support/callbacks
-  active_support/core_ext/class/subclasses
   active_support/core_ext/hash/indifferent_access
   active_support/descendants_tracker
   active_support/inflector
@@ -67,3 +64,5 @@ module Karafka
 end
 
 Karafka::Loader.load!(Karafka.core_root)
+Kafka::Consumer.prepend(Karafka::Patches::RubyKafka)
+Dry::Configurable::Config.prepend(Karafka::Patches::DryConfigurable)

data/lib/karafka/attributes_map.rb

@@ -21,7 +21,7 @@ module Karafka
             offset_retention_time heartbeat_interval
           ],
           subscription: %i[start_from_beginning max_bytes_per_partition],
-          consuming: %i[min_bytes max_wait_time],
+          consuming: %i[min_bytes max_wait_time automatically_mark_as_processed],
           pausing: %i[pause_timeout],
           # All the options that are under kafka config namespace, but are not used
           # directly with kafka api, but from the Karafka user perspective, they are
@@ -37,7 +37,7 @@ module Karafka
           name
           parser
           responder
-          batch_processing
+          batch_consuming
           persistent
         ]).uniq
       end
@@ -52,7 +52,7 @@ module Karafka
         #   only when proxying details go ruby-kafka. We use ignored fields internally in karafka
         ignored_settings = config_adapter[:subscription]
         defined_settings = config_adapter.values.flatten
-        karafka_settings = %i[batch_consuming]
+        karafka_settings = %i[batch_fetching]
         # This is a drity and bad hack of dry-configurable to get keys before setting values
         dynamically_proxied = Karafka::Setup::Config
                               ._settings

data/lib/karafka/backends/inline.rb

@@ -7,10 +7,10 @@ module Karafka
     module Inline
       private
 
-      # Executes perform code immediately (without enqueuing)
+      # Executes consume code immediately (without enqueuing)
       def process
         Karafka.monitor.notice(self.class, params_batch)
-        perform
+        consume
       end
     end
   end

data/lib/karafka/base_controller.rb

@@ -3,57 +3,8 @@
 # Karafka module namespace
 module Karafka
   # Base controller from which all Karafka controllers should inherit
-  # Similar to Rails controllers we can define after_received callbacks
-  # that will be executed
-  #
-  # Note that if after_received return false, the chain will be stopped and
-  #   the perform method won't be executed
-  #
-  # @example Create simple controller
-  #   class ExamplesController < Karafka::BaseController
-  #     def perform
-  #       # some logic here
-  #     end
-  #   end
-  #
-  # @example Create a controller with a block after_received
-  #   class ExampleController < Karafka::BaseController
-  #     after_received do
-  #       # Here we should have some checking logic
-  #       # If false is returned, won't schedule a perform action
-  #     end
-  #
-  #     def perform
-  #       # some logic here
-  #     end
-  #   end
-  #
-  # @example Create a controller with a method after_received
-  #   class ExampleController < Karafka::BaseController
-  #     after_received :after_received_method
-  #
-  #     def perform
-  #       # some logic here
-  #     end
-  #
-  #     private
-  #
-  #     def after_received_method
-  #       # Here we should have some checking logic
-  #       # If false is returned, won't schedule a perform action
-  #     end
-  #   end
   class BaseController
     extend ActiveSupport::DescendantsTracker
-    include ActiveSupport::Callbacks
-
-    # The call method is wrapped with a set of callbacks
-    # We won't run perform at the backend if any of the callbacks
-    # returns false
-    # @see http://api.rubyonrails.org/classes/ActiveSupport/Callbacks/ClassMethods.html#method-i-get_callbacks
-    define_callbacks :after_received
-
-    attr_reader :params_batch
 
     class << self
       attr_reader :topic
@@ -66,21 +17,6 @@ module Karafka
         @topic = topic
         Controllers::Includer.call(self)
       end
-
-      # Creates a callback that will be executed after receiving message but before executing the
-      #   backend for processing
-      # @param method_name [Symbol, String] method name or nil if we plan to provide a block
-      # @yield A block with a code that should be executed before scheduling
-      # @example Define a block after_received callback
-      #   after_received do
-      #     # logic here
-      #   end
-      #
-      # @example Define a class name after_received callback
-      #   after_received :method_name
-      def after_received(method_name = nil, &block)
-        set_callback :after_received, :before, method_name ? method_name : block
-      end
     end
 
     # @return [Karafka::Routing::Topic] topic to which a given controller is subscribed
@@ -100,18 +36,32 @@ module Karafka
     # Executes the default controller flow, runs callbacks and if not halted
     # will call process method of a proper backend
     def call
-      run_callbacks :after_received do
-        process
-      end
+      process
     end
 
     private
 
-    # Method that will perform business logic on data received from Kafka
+    # We make it private as it should be accesible only from the inside of a controller
+    attr_reader :params_batch
+
+    # @return [Karafka::Connection::Consumer] messages consumer that can be used to
+    #    commit manually offset or pause / stop consumer based on the business logic
+    def consumer
+      Persistence::Consumer.read
+    end
+
+    # Method that will perform business logic and on data received from Kafka (it will consume
+    #   the data)
     # @note This method needs bo be implemented in a subclass. We stub it here as a failover if
     #   someone forgets about it or makes on with typo
-    def perform
+    def consume
       raise NotImplementedError, 'Implement this in a subclass'
     end
+
+    # Alias for #consume method and deprecation warning
+    def perform
+      Karafka.logger.warn('[DEPRECATION WARNING]: please use #consume instead of #perform')
+      consume
+    end
   end
 end

data/lib/karafka/base_responder.rb

@@ -147,12 +147,11 @@ module Karafka
         # @note By default will not change topic (if default mapper used)
         mapped_topic = Karafka::App.config.topic_mapper.outgoing(topic)
 
-        data_elements.each do |(data, options)|
-          ::WaterDrop::Message.new(
-            mapped_topic,
+        data_elements.each do |data, options|
+          producer(options).call(
             data,
-            options
-          ).send!
+            options.merge(topic: mapped_topic)
+          )
         end
       end
     end
@@ -176,5 +175,11 @@ module Karafka
       messages_buffer[topic.to_s] ||= []
       messages_buffer[topic.to_s] << [@parser_class.generate(data), options]
     end
+
+    # @param options [Hash] options for waterdrop
+    # @return [Class] WaterDrop producer (sync or async based on the settings)
+    def producer(options)
+      options[:async] ? WaterDrop::AsyncProducer : WaterDrop::SyncProducer
+    end
   end
 end

data/lib/karafka/cli/info.rb

@@ -15,9 +15,8 @@ module Karafka
           "Karafka framework version: #{Karafka::VERSION}",
           "Application client id: #{config.client_id}",
           "Backend: #{config.backend}",
+          "Batch fetching: #{config.batch_fetching}",
           "Batch consuming: #{config.batch_consuming}",
-          "Batch processing: #{config.batch_processing}",
-          "Number of threads: #{config.concurrency}",
           "Boot file: #{Karafka.boot_file}",
           "Environment: #{Karafka.env}",
           "Kafka seed brokers: #{config.kafka.seed_brokers}"

data/lib/karafka/cli/server.rb

@@ -20,21 +20,19 @@ module Karafka
 
         if cli.options[:daemon]
           FileUtils.mkdir_p File.dirname(cli.options[:pid])
-          # For some reason Celluloid spins threads that break forking
-          # Threads are not shutdown immediately so deamonization will stale until
-          # those threads are killed by Celluloid manager (via timeout)
-          # There's nothing initialized here yet, so instead we shutdown celluloid
-          # and run it again when we need (after fork)
-          Celluloid.shutdown
           daemonize
-          Celluloid.boot
         end
 
         # We assign active topics on a server level, as only server is expected to listen on
         # part of the topics
         Karafka::Server.consumer_groups = cli.options[:consumer_groups]
 
-        # Remove pidfile on shutdown, just before the server instance is going to be GCed
+        # Remove pidfile on stop, just before the server instance is going to be GCed
+        # We want to delay the moment in which the pidfile is removed as much as we can,
+        # so instead of removing it after the server stops running, we rely on the gc moment
+        # when this object gets removed (it is a bit later), so it is closer to the actual
+        # system process end. We do that, so monitoring and deployment tools that rely on pids
+        # won't alarm or start new system process up until the current one is finished
         ObjectSpace.define_finalizer(self, proc { send(:clean) })
 
         # After we fork, we can boot celluloid again