karafka 2.0.0.alpha6 → 2.0.0.beta3

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 862df94b26c24809f82e07f71c39433b90ef08f68f053a004a87aa466b872dac
-  data.tar.gz: 2c533cbd6c271fe282f59c2030d9cc885555242bb8bc9316d0264a5ccfd694a0
+  metadata.gz: 86b352cc7737bde8484567662d546165793496adb6a923e33aa7e4823e4a456f
+  data.tar.gz: 2a180ba7b177db9f4c67de387262d47ac3eb2580860a548992912cd89a049da7
 SHA512:
-  metadata.gz: 8ce3720e535d65f121bcbfd957286cbc41404f2aff2751622183367862db55f8231c286d5bbede4a18649eafcd085952a7fc8e7569a74dbd3633c9db906114e5
-  data.tar.gz: e77819ccd2be263b02958fcee71ce9228c93c7dee62fabbea61711fbbdbffa997bb9a85a7f668cf8bb596b7a98a773ace560ad5075c46e70f20cd302b354b0fe
+  metadata.gz: 207700c3e1fab4d3370de7eddae72453fc180c65d817f4d7c021929327e5464fcd8a9bb1610c641080f82a60bc038e4517dc7bc86c0aa4c83852dd9467441e75
+  data.tar.gz: ab94239255ff841e0728c6e6585d4145a9c4f42dfd02d8df062975bd28ad3caf2cef50f5488d30a1356ad49e2529f772379bc947ba5804a15f12e56989cf9a7a

data/.ruby-version

@@ -1 +1 @@
-3.1.0
+3.1.2

data/CHANGELOG.md

@@ -1,5 +1,45 @@
 # Karafka framework changelog
 
+## 2.0.0-beta3 (2022-06-14)
+- Jobs building responsibility extracted out of the listener code base.
+- Fix a case where specs supervisor would try to kill no longer running process (#868)
+- Fix an instable integration spec that could misbehave under load
+- Commit offsets prior to pausing partitions to ensure that the latest offset is always committed
+- Fix a case where consecutive CTRL+C (non-stop) would case an exception during forced shutdown
+- Add missing `consumer.prepared.error` into `LoggerListener`
+- Delegate partition resuming from the consumers to listeners threads.
+- Add support for Long Running Jobs (LRJ) for ActiveJob [PRO]
+- Add support for Long Running Jobs for consumers [PRO]
+- Allow `active_job_topic` to accept a block for extra topic related settings
+- Remove no longer needed logger threads
+- Auto-adapt number of processes for integration specs based on the number of CPUs
+- Introduce an integration spec runner that prints everything to stdout (better for development)
+- Introduce extra integration specs for various ActiveJob usage scenarios
+- Rename consumer method `#prepared` to `#prepare` to reflect better its use-case
+- For test and dev raise an error when expired license key is used (never for non dev)
+- Add worker related monitor events (`worker.process` and `worker.processed`)
+- Update `LoggerListener` to include more useful information about processing and polling messages
+
+## 2.0.0-beta2 (2022-06-07)
+- Abstract away notion of topics groups (until now it was just an array)
+- Optimize how jobs queue is closed. Since we enqueue jobs only from the listeners, we can safely close jobs queue once listeners are done. By extracting this responsibility from listeners, we remove corner cases and race conditions. Note here: for non-blocking jobs we do wait for them to finish while running the `poll`. This ensures, that for async jobs that are long-living, we do not reach `max.poll.interval`.
+- `Shutdown` jobs are executed in workers to align all the jobs behaviours.
+- `Shutdown` jobs are always blocking.
+- Notion of `ListenersBatch` was introduced similar to `WorkersBatch` to abstract this concept.
+- Change default `shutdown_timeout` to be more than `max_wait_time` not to cause forced shutdown when no messages are being received from Kafka.
+- Abstract away scheduling of revocation and shutdown jobs for both default and pro schedulers
+- Introduce a second (internal) messages buffer to distinguish between raw messages buffer and karafka messages buffer
+- Move messages and their metadata remap process to the listener thread to allow for their inline usage
+- Change how we wait in the shutdown phase, so shutdown jobs can still use Kafka connection even if they run for a longer period of time. This will prevent us from being kicked out from the group early.
+- Introduce validation that ensures, that `shutdown_timeout` is more than `max_wait_time`. This will prevent users from ending up with a config that could lead to frequent forceful shutdowns.
+
+## 2.0.0-beta1 (2022-05-22)
+- Update the jobs queue blocking engine and allow for non-blocking jobs execution
+- Provide `#prepared` hook that always runs before the fetching loop is unblocked
+- [Pro] Introduce performance tracker for scheduling optimizer
+- Provide ability to pause (`#pause`) and resume (`#resume`) given partitions from the consumers
+- Small integration specs refactoring + specs for pausing scenarios
+
 ## 2.0.0-alpha6 (2022-04-17)
 - Fix a bug, where upon missing boot file and Rails, railtie would fail with a generic exception (#818) 
 - Fix an issue with parallel pristine specs colliding with each other during `bundle install` (#820)
@@ -26,12 +66,12 @@
 
 ## 2.0.0-alpha2 (2022-02-19)
 - Require `kafka` keys to be symbols
-- Added ActiveJob Pro adapter
+- [Pro] Added ActiveJob Pro adapter
 - Small updates to the license and docs
 
 ## 2.0.0-alpha1 (2022-01-30)
 - Change license to `LGPL-3.0`
-- Introduce a Pro subscription
+- [Pro] Introduce a Pro subscription
 - Switch from `ruby-kafka` to `librdkafka` as an underlying driver
 - Introduce fully automatic integration tests that go through the whole server lifecycle
 - Integrate WaterDrop tightly with autoconfiguration inheritance and an option to redefine it

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    karafka (2.0.0.alpha6)
+    karafka (2.0.0.beta3)
       dry-configurable (~> 0.13)
       dry-monitor (~> 0.5)
       dry-validation (~> 1.7)
@@ -13,10 +13,10 @@ PATH
 GEM
   remote: https://rubygems.org/
   specs:
-    activejob (7.0.2.3)
-      activesupport (= 7.0.2.3)
+    activejob (7.0.3)
+      activesupport (= 7.0.3)
       globalid (>= 0.3.6)
-    activesupport (7.0.2.3)
+    activesupport (7.0.3)
       concurrent-ruby (~> 1.0, >= 1.0.2)
       i18n (>= 1.6, < 2)
       minitest (>= 5.1)
@@ -25,7 +25,7 @@ GEM
     concurrent-ruby (1.1.10)
     diff-lcs (1.5.0)
     docile (1.4.0)
-    dry-configurable (0.14.0)
+    dry-configurable (0.15.0)
       concurrent-ruby (~> 1.0)
       dry-core (~> 0.6)
     dry-container (0.9.0)
@@ -45,7 +45,7 @@ GEM
       dry-configurable (~> 0.13, >= 0.13.0)
       dry-core (~> 0.5, >= 0.5)
       dry-events (~> 0.2)
-    dry-schema (1.9.1)
+    dry-schema (1.9.2)
       concurrent-ruby (~> 1.0)
       dry-configurable (~> 0.13, >= 0.13.0)
       dry-core (~> 0.5, >= 0.5)
@@ -58,12 +58,12 @@ GEM
       dry-core (~> 0.5, >= 0.5)
       dry-inflector (~> 0.1, >= 0.1.2)
       dry-logic (~> 1.0, >= 1.0.2)
-    dry-validation (1.8.0)
+    dry-validation (1.8.1)
       concurrent-ruby (~> 1.0)
       dry-container (~> 0.7, >= 0.7.1)
       dry-core (~> 0.5, >= 0.5)
       dry-initializer (~> 3.0)
-      dry-schema (~> 1.9, >= 1.9.1)
+      dry-schema (~> 1.8, >= 1.8.0)
     factory_bot (6.2.1)
       activesupport (>= 5.0.0)
     ffi (1.15.5)
@@ -121,4 +121,4 @@ DEPENDENCIES
   simplecov
 
 BUNDLED WITH
-   2.3.10
+   2.3.11

data/bin/integrations

@@ -11,21 +11,21 @@ require 'open3'
 require 'fileutils'
 require 'pathname'
 require 'tmpdir'
+require 'etc'
 
 ROOT_PATH = Pathname.new(File.expand_path(File.join(File.dirname(__FILE__), '../')))
 
-# Raised from the parent process if any of the integration tests fails
-IntegrationTestError = Class.new(StandardError)
-
 # How many child processes with integration specs do we want to run in parallel
-# When the value is high, there's a problem with thread allocation on Github
-CONCURRENCY = 4
+# When the value is high, there's a problem with thread allocation on Github CI, tht is why
+# we limit it. Locally we can run a lot of those, as many of them have sleeps and do not use a lot
+# of CPU
+CONCURRENCY = ENV.key?('CI') ? 5 : Etc.nprocessors * 2
 
 # Abstraction around a single test scenario execution process
 class Scenario
   # How long a scenario can run before we kill it
   # This is a fail-safe just in case something would hang
-  MAX_RUN_TIME = 60 * 2
+  MAX_RUN_TIME = 3 * 60 # 3 minutes tops
 
   # There are rare cases where Karafka may force shutdown for some of the integration cases
   # This includes exactly those
@@ -73,7 +73,12 @@ class Scenario
     # If the thread is running too long, kill it
     if current_time - @started_at > MAX_RUN_TIME
       @wait_thr.kill
-      Process.kill('TERM', pid)
+
+      begin
+        Process.kill('TERM', pid)
+      # It may finish right after we want to kill it, that's why we ignore this
+      rescue Errno::ESRCH
+      end
     end
 
     # We read it so it won't grow as we use our default logger that prints to both test.log and
@@ -106,14 +111,15 @@ class Scenario
 
   # Prints a status report when scenario is finished and stdout if it failed
   def report
-    result = success? ? "\e[#{32}m#{'OK'}\e[0m" : "\e[#{31}m#{'FAILED'}\e[0m"
-
-    puts "#{result} #{name}"
-
-    unless success?
+    if success?
+      print "\e[#{32}m#{'.'}\e[0m"
+    else
+      puts
+      puts "\e[#{31}m#{'[FAILED]'}\e[0m #{name}"
       puts "Exit code: #{exit_code}"
       puts @stdout_tail
       puts @stderr.read
+      puts
     end
   end
 
@@ -204,5 +210,21 @@ while finished_scenarios.size < scenarios.size
   sleep(0.1)
 end
 
-# Fail all if any of the tests does not have expected exit code
-raise IntegrationTestError unless finished_scenarios.all?(&:success?)
+failed_scenarios = finished_scenarios.reject(&:success?)
+
+# Report once more on the failed jobs
+# This will only list scenarios that failed without printing their stdout here.
+if failed_scenarios.empty?
+  puts
+else
+  puts "\nFailed scenarios:\n\n"
+
+  failed_scenarios.each do |scenario|
+    puts "\e[#{31}m#{'[FAILED]'}\e[0m #{scenario.name}"
+  end
+
+  puts
+
+  # Exit with 1 if not all scenarios were successful
+  exit 1
+end

data/bin/scenario

@@ -0,0 +1,29 @@
+#!/usr/bin/env ruby
+
+# Runner for non-parallel execution of a single scenario.
+# It prints all the info stdout, etc and basically replaces itself with the scenario execution.
+# It is useful when we work with a single spec and we need all the debug info
+
+raise 'This code needs to be executed WITHOUT bundle exec' if Kernel.const_defined?(:Bundler)
+
+require 'open3'
+require 'fileutils'
+require 'pathname'
+require 'tmpdir'
+require 'etc'
+
+ROOT_PATH = Pathname.new(File.expand_path(File.join(File.dirname(__FILE__), '../')))
+
+# Load all the specs
+specs = Dir[ROOT_PATH.join('spec/integrations/**/*.rb')]
+
+# If filters is provided, apply
+# Allows to provide several filters one after another and applies all of them
+ARGV.each do |filter|
+  specs.delete_if { |name| !name.include?(filter) }
+end
+
+raise ArgumentError, "No integration specs with filters: #{ARGV.join(', ')}" if specs.empty?
+raise ArgumentError, "Many specs found with filters: #{ARGV.join(', ')}" if specs.size != 1
+
+exec("bundle exec ruby -r #{ROOT_PATH}/spec/integrations_helper.rb #{specs[0]}")

data/config/errors.yml

@@ -2,6 +2,7 @@ en:
   dry_validation:
     errors:
       max_timeout_vs_pause_max_timeout: pause_timeout must be less or equal to pause_max_timeout
+      shutdown_timeout_vs_max_wait_time: shutdown_timeout must be more than max_wait_time
       topics_names_not_unique: all topic names within a single consumer group must be unique
       required_usage_count: Given topic must be used at least once
       consumer_groups_inclusion: Unknown consumer group

data/docker-compose.yml

@@ -16,6 +16,9 @@ services:
       KAFKA_CREATE_TOPICS:
         "integrations_0_02:2:1,\
          integrations_1_02:2:1,\
+         integrations_2_02:2:1,\
+         integrations_3_02:2:1,\
+         integrations_4_02:2:1,\
          integrations_0_03:3:1,\
          integrations_1_03:3:1,\
          integrations_2_03:3:1,\

data/karafka.gemspec

@@ -14,7 +14,7 @@ Gem::Specification.new do |spec|
   spec.homepage    = 'https://karafka.io'
   spec.summary     = 'Ruby based framework for working with Apache Kafka'
   spec.description = 'Framework used to simplify Apache Kafka based Ruby applications development'
-  spec.license     = 'LGPL-3.0'
+  spec.licenses    = ['LGPL-3.0', 'Commercial']
 
   spec.add_dependency 'dry-configurable', '~> 0.13'
   spec.add_dependency 'dry-monitor', '~> 0.5'

data/lib/active_job/karafka.rb

@@ -14,8 +14,8 @@ begin
   # We extend routing builder by adding a simple wrapper for easier jobs topics defining
   # This needs to be extended here as it is going to be used in karafka routes, hence doing that in
   # the railtie initializer would be too late
-  ::Karafka::Routing::Builder.include ::Karafka::ActiveJob::RoutingExtensions
-  ::Karafka::Routing::Proxy.include ::Karafka::ActiveJob::RoutingExtensions
+  ::Karafka::Routing::Builder.include ::Karafka::ActiveJob::Routing::Extensions
+  ::Karafka::Routing::Proxy.include ::Karafka::ActiveJob::Routing::Extensions
 rescue LoadError
   # We extend ActiveJob stuff in the railtie
 end

data/lib/karafka/active_job/routing/extensions.rb

@@ -0,0 +1,31 @@
+# frozen_string_literal: true
+
+module Karafka
+  # ActiveJob related Karafka stuff
+  module ActiveJob
+    # Karafka routing ActiveJob related components
+    module Routing
+      # Routing extensions for ActiveJob
+      module Extensions
+        # This method simplifies routes definition for ActiveJob topics / queues by auto-injecting
+        # the consumer class
+        # @param name [String, Symbol] name of the topic where ActiveJobs jobs should go
+        # @param block [Proc] block that we can use for some extra configuration
+        def active_job_topic(name, &block)
+          topic(name) do
+            consumer App.config.internal.active_job.consumer
+
+            next unless block
+
+            instance_eval(&block)
+
+            # This is handled by our custom ActiveJob consumer
+            # Without this, default behaviour would cause messages to skip upon shutdown as the
+            # offset would be committed for the last message
+            manual_offset_management true
+          end
+        end
+      end
+    end
+  end
+end

data/lib/karafka/base_consumer.rb

@@ -10,13 +10,38 @@ module Karafka
     attr_accessor :messages
     # @return [Karafka::Connection::Client] kafka connection client
     attr_accessor :client
-    # @return [Karafka::TimeTrackers::Pause] current topic partition pause
-    attr_accessor :pause
+    # @return [Karafka::TimeTrackers::Pause] current topic partition pause tracker
+    attr_accessor :pause_tracker
     # @return [Waterdrop::Producer] producer instance
     attr_accessor :producer
 
+    # Can be used to run preparation code
+    #
+    # @private
+    # @note This should not be used by the end users as it is part of the lifecycle of things but
+    #   not as part of the public api. This can act as a hook when creating non-blocking
+    #   consumers and doing other advanced stuff
+    def on_prepare
+      Karafka.monitor.instrument('consumer.prepared', caller: self) do
+        prepare
+      end
+
+      true
+    rescue StandardError => e
+      Karafka.monitor.instrument(
+        'error.occurred',
+        error: e,
+        caller: self,
+        type: 'consumer.prepare.error'
+      )
+
+      false
+    end
+
     # Executes the default consumer flow.
     #
+    # @return [Boolean] true if there was no exception, otherwise false.
+    #
     # @note We keep the seek offset tracking, and use it to compensate for async offset flushing
     #   that may not yet kick in when error occurs. That way we pause always on the last processed
     #   message.
@@ -24,15 +49,17 @@ module Karafka
       Karafka.monitor.instrument('consumer.consumed', caller: self) do
         consume
 
-        pause.reset
+        pause_tracker.reset
 
         # Mark as consumed only if manual offset management is not on
-        return if topic.manual_offset_management
+        next if topic.manual_offset_management
 
         # We use the non-blocking one here. If someone needs the blocking one, can implement it
         # with manual offset management
         mark_as_consumed(messages.last)
       end
+
+      true
     rescue StandardError => e
       Karafka.monitor.instrument(
         'error.occurred',
@@ -40,8 +67,10 @@ module Karafka
         caller: self,
         type: 'consumer.consume.error'
       )
-      client.pause(topic.name, messages.first.partition, @seek_offset || messages.first.offset)
-      pause.pause
+
+      pause(@seek_offset || messages.first.offset)
+
+      false
     end
 
     # Trigger method for running on shutdown.
@@ -51,6 +80,8 @@ module Karafka
       Karafka.monitor.instrument('consumer.revoked', caller: self) do
         revoked
       end
+
+      true
     rescue StandardError => e
       Karafka.monitor.instrument(
         'error.occurred',
@@ -58,6 +89,8 @@ module Karafka
         caller: self,
         type: 'consumer.revoked.error'
       )
+
+      false
     end
 
     # Trigger method for running on shutdown.
@@ -67,6 +100,8 @@ module Karafka
       Karafka.monitor.instrument('consumer.shutdown', caller: self) do
         shutdown
       end
+
+      true
     rescue StandardError => e
       Karafka.monitor.instrument(
         'error.occurred',
@@ -74,10 +109,16 @@ module Karafka
         caller: self,
         type: 'consumer.shutdown.error'
       )
+
+      false
     end
 
     private
 
+    # Method that gets called in the blocking flow allowing to setup any type of resources or to
+    # send additional commands to Kafka before the proper execution starts.
+    def prepare; end
+
     # Method that will perform business logic and on data received from Kafka (it will consume
     #   the data)
     # @note This method needs bo be implemented in a subclass. We stub it here as a failover if
@@ -97,6 +138,10 @@ module Karafka
     # Marks message as consumed in an async way.
     #
     # @param message [Messages::Message] last successfully processed message.
+    # @note We keep track of this offset in case we would mark as consumed and got error when
+    #   processing another message. In case like this we do not pause on the message we've already
+    #   processed but rather at the next one. This applies to both sync and async versions of this
+    #   method.
     def mark_as_consumed(message)
       client.mark_as_consumed(message)
       @seek_offset = message.offset + 1
@@ -110,6 +155,29 @@ module Karafka
       @seek_offset = message.offset + 1
     end
 
+    # Pauses processing on a given offset for the current topic partition
+    #
+    # After given partition is resumed, it will continue processing from the given offset
+    # @param offset [Integer] offset from which we want to restart the processing
+    # @param timeout [Integer, nil] how long in milliseconds do we want to pause or nil to use the
+    #   default exponential pausing strategy defined for retries
+    def pause(offset, timeout = nil)
+      timeout ? pause_tracker.pause(timeout) : pause_tracker.pause
+
+      client.pause(
+        messages.metadata.topic,
+        messages.metadata.partition,
+        offset
+      )
+    end
+
+    # Resumes processing of the current topic partition
+    def resume
+      # This is sufficient to expire a partition pause, as with it will be resumed by the listener
+      # thread before the next poll.
+      pause_tracker.expire
+    end
+
     # Seeks in the context of current topic and partition
     #
     # @param offset [Integer] offset where we want to seek

data/lib/karafka/connection/client.rb

@@ -30,7 +30,7 @@ module Karafka
         @mutex = Mutex.new
         @closed = false
         @subscription_group = subscription_group
-        @buffer = MessagesBuffer.new
+        @buffer = RawMessagesBuffer.new
         @rebalance_manager = RebalanceManager.new
         @kafka = build_consumer
         # Marks if we need to offset. If we did not store offsets, we should not commit the offset
@@ -86,8 +86,7 @@ module Karafka
       # @param message [Karafka::Messages::Message]
       def store_offset(message)
         @mutex.synchronize do
-          @offsetting = true
-          @kafka.store_offset(message)
+          internal_store_offset(message)
         end
       end
 
@@ -104,14 +103,7 @@ module Karafka
       def commit_offsets(async: true)
         @mutex.lock
 
-        return unless @offsetting
-
-        @kafka.commit(nil, async)
-        @offsetting = false
-      rescue Rdkafka::RdkafkaError => e
-        return if e.code == :no_offset
-
-        raise e
+        internal_commit_offsets(async: async)
       ensure
         @mutex.unlock
       end
@@ -128,7 +120,11 @@ module Karafka
       #
       # @param message [Messages::Message, Messages::Seek] message to which we want to seek to
       def seek(message)
+        @mutex.lock
+
         @kafka.seek(message)
+      ensure
+        @mutex.unlock
       end
 
       # Pauses given partition and moves back to last successful offset processed.
@@ -144,15 +140,17 @@ module Karafka
         # Do not pause if the client got closed, would not change anything
         return if @closed
 
+        pause_msg = Messages::Seek.new(topic, partition, offset)
+
+        internal_commit_offsets(async: false)
+
         tpl = topic_partition_list(topic, partition)
 
         return unless tpl
 
         @kafka.pause(tpl)
 
-        pause_msg = Messages::Seek.new(topic, partition, offset)
-
-        seek(pause_msg)
+        @kafka.seek(pause_msg)
       ensure
         @mutex.unlock
       end
@@ -166,6 +164,11 @@ module Karafka
 
         return if @closed
 
+        # Always commit synchronously offsets if any when we resume
+        # This prevents resuming without offset in case it would not be committed prior
+        # We can skip performance penalty since resuming should not happen too often
+        internal_commit_offsets(async: false)
+
         tpl = topic_partition_list(topic, partition)
 
         return unless tpl
@@ -214,11 +217,31 @@ module Karafka
 
       private
 
+      # Non thread-safe offset storing method
+      # @param message [Karafka::Messages::Message]
+      def internal_store_offset(message)
+        @offsetting = true
+        @kafka.store_offset(message)
+      end
+
+      # Non thread-safe message committing method
+      # @param async [Boolean] should the commit happen async or sync (async by default)
+      def internal_commit_offsets(async: true)
+        return unless @offsetting
+
+        @kafka.commit(nil, async)
+        @offsetting = false
+      rescue Rdkafka::RdkafkaError => e
+        return if e.code == :no_offset
+
+        raise e
+      end
+
       # Commits the stored offsets in a sync way and closes the consumer.
       def close
-        commit_offsets!
-
         @mutex.synchronize do
+          internal_commit_offsets(async: false)
+
           @closed = true
 
           # Remove callbacks runners that were registered