karafka 2.0.16 → 2.0.18

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 8cc0ac8b318d7c40e8974e2e26753c68cc5f611bbcd0e4fbea94e3eb275002dd
-  data.tar.gz: db638edaecc662ac8a0f7d627f4bcae6e1e1e83500a49c4dfa4a4f32ee8f78c9
+  metadata.gz: 17800cf4d60529269b31340d76245ed2035b463f03daceeb482241528eb5d681
+  data.tar.gz: 9caed11b4f7221d9c5e9c33607ba965a0fd9c75c7ce037d40e38df119a8255ea
 SHA512:
-  metadata.gz: 6875e7e152b4150f397f3830c828412e0477409483c81e714525161e5263cff686c0cb9b46a6c11cb5b914dc51087e4bac95d5dcdc0e25c7233825b363ed7e6c
-  data.tar.gz: 1f35d410350dc3fbe97462e979f3df832484df1f1f5119e16fe2c28210d01a05e1d3e086c73e1a3b70ba3c6cae9a25dd50e0907ca454b68c9166efcabd22f972
+  metadata.gz: 5de87acf0d5dbef037c2e980d1354c398fe47bf9a14312f772b346f59bd6f638aa0d4444b9d32984f494cd66dc3cb752f843dc72534ad9ec7df367ecde100dfc
+  data.tar.gz: 8577247eeb4eae1d5abc9e74de9c2c95787535662ed8a2a664824871fe1907abb7aa6e14619926cde218cfb06db5496ec0c38dd6fc3fc3ad0efe0b5702fe067e

data/CHANGELOG.md

@@ -1,5 +1,35 @@
 # Karafka framework changelog
 
+## 2.0.18 (2022-11-18)
+- **[Feature]** Support quiet mode via `TSTP` signal. When used, Karafka will finish processing current messages, run `shutdown` jobs, and switch to a quiet mode where no new work is being accepted. At the same time, it will keep the consumer group quiet, and thus no rebalance will be triggered. This can be particularly useful during deployments.
+- [Improvement] Trigger `#revoked` for jobs in case revocation would happen during shutdown when jobs are still running. This should ensure, we get a notion of revocation for Pro LRJ jobs even when revocation happening upon shutdown (#1150).
+- [Improvement] Stabilize the shutdown procedure for consumer groups with many subscription groups that have non-aligned processing cost per batch.
+- [Improvement] Remove double loading of Karafka via Rails railtie.
+- [Fix] Fix invalid class references in YARD docs.
+- [Fix] prevent parallel closing of many clients.
+- [Fix] fix a case where information about revocation for a combination of LRJ + VP would not be dispatched until all VP work is done.
+
+## 2.0.17 (2022-11-10)
+- [Fix] Few typos around DLQ and Pro DLQ Dispatch original metadata naming.
+- [Fix] Narrow the components lookup to the appropriate scope (#1114)
+
+### Upgrade notes
+
+1. Replace `original-*` references from DLQ dispatched metadata with `original_*`
+
+```ruby
+# DLQ topic consumption
+def consume
+  messages.each do |broken_message|
+    topic = broken_message.metadata['original_topic'] # was original-topic
+    partition = broken_message.metadata['original_partition'] # was original-partition
+    offset = broken_message.metadata['original_offset'] # was original-offset
+
+    Rails.logger.error "This message is broken: #{topic}/#{partition}/#{offset}"
+  end
+end
+```
+
 ## 2.0.16 (2022-11-09)
 - **[Breaking]** Disable the root `manual_offset_management` setting and require it to be configured per topic. This is part of "topic features" configuration extraction for better code organization.
 - **[Feature]** Introduce **Dead Letter Queue** feature and Pro **Enhanced Dead Letter Queue** feature
@@ -26,7 +56,6 @@
 - [Specs] Split specs into regular and pro to simplify how resources are loaded
 - [Specs] Add specs to ensure, that all the Pro components have a proper per-file license (#1099)
 
-
 ### Upgrade notes
 
 1. Remove the `manual_offset_management` setting from the main config if you use it:

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    karafka (2.0.16)
+    karafka (2.0.18)
       karafka-core (>= 2.0.2, < 3.0.0)
       rdkafka (>= 0.12)
       thor (>= 0.20)

data/README.md

@@ -4,6 +4,8 @@
 [![Gem Version](https://badge.fury.io/rb/karafka.svg)](http://badge.fury.io/rb/karafka)
 [![Join the chat at https://slack.karafka.io](https://raw.githubusercontent.com/karafka/misc/master/slack.svg)](https://slack.karafka.io)
 
+**Note**: Upgrade notes for migration from Karafka `1.4` to Karafka `2.0` can be found [here](https://karafka.io/docs/Upgrades-2.0/).
+
 ## About Karafka
 
 Karafka is a Ruby and Rails multi-threaded efficient Kafka processing framework that:

data/lib/karafka/app.rb

@@ -14,11 +14,12 @@ module Karafka
           .builder
       end
 
-      # @return [Array<Karafka::Routing::SubscriptionGroup>] active subscription groups
+      # @return [Hash] active subscription groups grouped based on consumer group in a hash
       def subscription_groups
         consumer_groups
           .active
-          .flat_map(&:subscription_groups)
+          .map { |consumer_group| [consumer_group, consumer_group.subscription_groups] }
+          .to_h
       end
 
       # Just a nicer name for the consumer groups

data/lib/karafka/connection/client.rb

@@ -17,7 +17,11 @@ module Karafka
       # How many times should we retry polling in case of a failure
       MAX_POLL_RETRIES = 20
 
-      private_constant :MAX_POLL_RETRIES
+      # We want to make sure we never close several clients in the same moment to prevent
+      # potential race conditions and other issues
+      SHUTDOWN_MUTEX = Mutex.new
+
+      private_constant :MAX_POLL_RETRIES, :SHUTDOWN_MUTEX
 
       # Creates a new consumer instance.
       #
@@ -281,24 +285,26 @@ module Karafka
 
       # Commits the stored offsets in a sync way and closes the consumer.
       def close
-        @mutex.synchronize do
-          # Once client is closed, we should not close it again
-          # This could only happen in case of a race-condition when forceful shutdown happens
-          # and triggers this from a different thread
-          return if @closed
-
-          @closed = true
-
-          internal_commit_offsets(async: false)
-
-          # Remove callbacks runners that were registered
-          ::Karafka::Instrumentation.statistics_callbacks.delete(@subscription_group.id)
-          ::Karafka::Instrumentation.error_callbacks.delete(@subscription_group.id)
-
-          @kafka.close
-          @buffer.clear
-          # @note We do not clear rebalance manager here as we may still have revocation info here
-          # that we want to consider valid prior to running another reconnection
+        # Allow only one client to be closed at the same time
+        SHUTDOWN_MUTEX.synchronize do
+          # Make sure that no other operations are happening on this client when we close it
+          @mutex.synchronize do
+            # Once client is closed, we should not close it again
+            # This could only happen in case of a race-condition when forceful shutdown happens
+            # and triggers this from a different thread
+            return if @closed
+
+            @closed = true
+
+            # Remove callbacks runners that were registered
+            ::Karafka::Instrumentation.statistics_callbacks.delete(@subscription_group.id)
+            ::Karafka::Instrumentation.error_callbacks.delete(@subscription_group.id)
+
+            @kafka.close
+            @buffer.clear
+            # @note We do not clear rebalance manager here as we may still have revocation info here
+            # that we want to consider valid prior to running another reconnection
+          end
         end
       end
 

data/lib/karafka/connection/consumer_group_status.rb

@@ -0,0 +1,38 @@
+# frozen_string_literal: true
+
+module Karafka
+  module Connection
+    # This object represents a collective status of execution of group of listeners running inside
+    # of one consumer group but potentially in separate subscription groups.
+    #
+    # There are cases when we do not want to close a given client when others from the same
+    # consumer group are running because it can cause instabilities due to early shutdown of some
+    # of the clients out of same consumer group.
+    #
+    # Here we can track it and only shutdown listeners when all work in a group is done.
+    class ConsumerGroupStatus
+      # @param group_size [Integer] number of separate subscription groups in a consumer group
+      def initialize(group_size)
+        @mutex = Mutex.new
+        @active_size = group_size
+      end
+
+      # @return [Boolean] Are there any listeners that are still doing any type of work. If not,
+      #   it means a consumer group is safe to be shutdown fully.
+      def working?
+        @active_size.positive?
+      end
+
+      # Decrements number of working listeners in the group by one until there's none
+      def finish
+        @mutex.synchronize do
+          @active_size -= 1
+
+          return if @active_size >= 0
+
+          raise Errors::InvalidConsumerGroupStatusError, @active_size
+        end
+      end
+    end
+  end
+end

data/lib/karafka/connection/listener.rb

@@ -14,13 +14,20 @@ module Karafka
       # @return [String] id of this listener
       attr_reader :id
 
+      # Mutex for things we do not want to do in parallel even in different consumer groups
+      MUTEX = Mutex.new
+
+      private_constant :MUTEX
+
+      # @param consumer_group_status [Karafka::Connection::ConsumerGroupStatus]
       # @param subscription_group [Karafka::Routing::SubscriptionGroup]
       # @param jobs_queue [Karafka::Processing::JobsQueue] queue where we should push work
       # @return [Karafka::Connection::Listener] listener instance
-      def initialize(subscription_group, jobs_queue)
+      def initialize(consumer_group_status, subscription_group, jobs_queue)
         proc_config = ::Karafka::App.config.internal.processing
 
         @id = SecureRandom.uuid
+        @consumer_group_status = consumer_group_status
         @subscription_group = subscription_group
         @jobs_queue = jobs_queue
         @coordinators = Processing::CoordinatorsBuffer.new
@@ -61,13 +68,17 @@ module Karafka
       #
       # @note We wrap it with a mutex exactly because of the above case of forceful shutdown
       def shutdown
-        return if @stopped
-
-        @mutex.synchronize do
-          @stopped = true
-          @executors.clear
-          @coordinators.reset
-          @client.stop
+        # We want to make sure that we never close two librdkafka clients at the same time. I'm not
+        # particularly fond of it's shutdown API being fully thread-safe
+        MUTEX.synchronize do
+          return if @stopped
+
+          @mutex.synchronize do
+            @stopped = true
+            @executors.clear
+            @coordinators.reset
+            @client.stop
+          end
         end
       end
 
@@ -82,7 +93,8 @@ module Karafka
       #   Kafka connections / Internet connection issues / Etc. Business logic problems should not
       #   propagate this far.
       def fetch_loop
-        until Karafka::App.stopping?
+        # Run the main loop as long as we are not stopping or moving into quiet mode
+        until Karafka::App.stopping? || Karafka::App.quieting?
           Karafka.monitor.instrument(
             'connection.listener.fetch_loop',
             caller: self,
@@ -122,7 +134,7 @@ module Karafka
           wait
         end
 
-        # If we are stopping we will no longer schedule any jobs despite polling.
+        # If we are stopping we will no longer schedule any regular jobs despite polling.
         # We need to keep polling not to exceed the `max.poll.interval` for long-running
         # non-blocking jobs and we need to allow them to finish. We however do not want to
         # enqueue any new jobs. It's worth keeping in mind that it is the end user responsibility
@@ -131,7 +143,14 @@ module Karafka
         #
         # We do not care about resuming any partitions or lost jobs as we do not plan to do
         # anything with them as we're in the shutdown phase.
-        wait_with_poll
+        #
+        # What we do care however is the ability to still run revocation jobs in case anything
+        # would change in the cluster. We still want to notify the long-running jobs about changes
+        # that occurred in the cluster.
+        wait_polling(
+          wait_until: -> { @jobs_queue.empty?(@subscription_group.id) },
+          after_poll: -> { build_and_schedule_revoke_lost_partitions_jobs }
+        )
 
         # We do not want to schedule the shutdown jobs prior to finishing all the jobs
         # (including non-blocking) as there might be a long-running job with a shutdown and then
@@ -139,7 +158,20 @@ module Karafka
         # as it could create a race-condition.
         build_and_schedule_shutdown_jobs
 
-        wait_with_poll
+        # Wait until all the shutdown jobs are done
+        wait_polling(wait_until: -> { @jobs_queue.empty?(@subscription_group.id) })
+
+        # Once all the work is done, we need to decrement counter of active subscription groups
+        # within this consumer group
+        @consumer_group_status.finish
+
+        # Wait if we're in the quiet mode
+        wait_polling(wait_until: -> { !Karafka::App.quieting? })
+
+        # We need to wait until all the work in the whole consumer group (local to the process)
+        # is done. Otherwise we may end up with locks and `Timed out LeaveGroupRequest in flight`
+        # warning notifications.
+        wait_polling(wait_until: -> { !@consumer_group_status.working? })
 
         shutdown
 
@@ -256,10 +288,21 @@ module Karafka
       end
 
       # Waits without blocking the polling
-      # This should be used only when we no longer plan to use any incoming data and we can safely
-      # discard it
-      def wait_with_poll
-        @client.batch_poll until @jobs_queue.empty?(@subscription_group.id)
+      #
+      # This should be used only when we no longer plan to use any incoming messages data and we
+      # can safely discard it. We can however use the rebalance information if needed.
+      #
+      # @param wait_until [Proc] until this evaluates to true, we will poll data
+      # @param after_poll [Proc] code that we want to run after each batch poll (if any)
+      #
+      # @note Performance of this is not relevant (in regards to blocks) because it is used only
+      #   on shutdown and quiet, hence not in the running mode
+      def wait_polling(wait_until:, after_poll: -> {})
+        until wait_until.call
+          @client.batch_poll
+
+          after_poll.call
+        end
       end
 
       # We can stop client without a problem, as it will reinitialize itself when running the

data/lib/karafka/connection/listeners_batch.rb

@@ -9,8 +9,18 @@ module Karafka
       # @param jobs_queue [JobsQueue]
       # @return [ListenersBatch]
       def initialize(jobs_queue)
-        @batch = App.subscription_groups.map do |subscription_group|
-          Connection::Listener.new(subscription_group, jobs_queue)
+        @batch = App.subscription_groups.flat_map do |_consumer_group, subscription_groups|
+          consumer_group_status = Connection::ConsumerGroupStatus.new(
+            subscription_groups.size
+          )
+
+          subscription_groups.map do |subscription_group|
+            Connection::Listener.new(
+              consumer_group_status,
+              subscription_group,
+              jobs_queue
+            )
+          end
         end
       end
 

data/lib/karafka/embedded.rb

@@ -18,6 +18,16 @@ module Karafka
         # Stop needs to be blocking to wait for all the things to finalize
         Karafka::Server.stop
       end
+
+      # Quiets Karafka upon any event
+      #
+      # @note This method is not blocking and will not wait for Karafka to fully quiet.
+      # It will trigger the quiet procedure but won't wait.
+      #
+      # @note Please keep in mind you need to `#stop` to actually stop the server anyhow.
+      def quiet
+        Karafka::Server.quiet
+      end
     end
   end
 end

data/lib/karafka/errors.rb

@@ -44,6 +44,9 @@ module Karafka
     # This should never happen. Please open an issue if it does.
     InvalidCoordinatorStateError = Class.new(BaseError)
 
+    # This should never happen. Please open an issue if it does.
+    InvalidConsumerGroupStatusError = Class.new(BaseError)
+
     # This should never happen. Please open an issue if it does.
     StrategyNotFoundError = Class.new(BaseError)
 

data/lib/karafka/instrumentation/logger_listener.rb

@@ -18,7 +18,7 @@ module Karafka
 
       # Logs each messages fetching attempt
       #
-      # @param event [Dry::Events::Event] event details including payload
+      # @param event [Karafka::Core::Monitoring::Event] event details including payload
       def on_connection_listener_fetch_loop(event)
         listener = event[:caller]
         debug "[#{listener.id}] Polling messages..."
@@ -26,7 +26,7 @@ module Karafka
 
       # Logs about messages that we've received from Kafka
       #
-      # @param event [Dry::Events::Event] event details including payload
+      # @param event [Karafka::Core::Monitoring::Event] event details including payload
       def on_connection_listener_fetch_loop_received(event)
         listener = event[:caller]
         time = event[:time]
@@ -42,7 +42,7 @@ module Karafka
 
       # Prints info about the fact that a given job has started
       #
-      # @param event [Dry::Events::Event] event details including payload
+      # @param event [Karafka::Core::Monitoring::Event] event details including payload
       def on_worker_process(event)
         job = event[:job]
         job_type = job.class.to_s.split('::').last
@@ -53,7 +53,7 @@ module Karafka
 
       # Prints info about the fact that a given job has finished
       #
-      # @param event [Dry::Events::Event] event details including payload
+      # @param event [Karafka::Core::Monitoring::Event] event details including payload
       def on_worker_processed(event)
         job = event[:job]
         time = event[:time]
@@ -66,7 +66,7 @@ module Karafka
       # Logs info about system signals that Karafka received and prints backtrace for threads in
       # case of ttin
       #
-      # @param event [Dry::Events::Event] event details including payload
+      # @param event [Karafka::Core::Monitoring::Event] event details including payload
       def on_process_notice_signal(event)
         info "Received #{event[:signal]} system signal"
 
@@ -89,7 +89,7 @@ module Karafka
 
       # Logs info that we're running Karafka app.
       #
-      # @param _event [Dry::Events::Event] event details including payload
+      # @param _event [Karafka::Core::Monitoring::Event] event details including payload
       def on_app_running(_event)
         info "Running in #{RUBY_DESCRIPTION}"
         info "Running Karafka #{Karafka::VERSION} server"
@@ -99,23 +99,28 @@ module Karafka
         info 'See LICENSE and the LGPL-3.0 for licensing details.'
       end
 
+      # @param _event [Karafka::Core::Monitoring::Event] event details including payload
+      def on_app_quieting(_event)
+        info 'Switching to quiet mode. New messages will not be processed.'
+      end
+
       # Logs info that we're going to stop the Karafka server.
       #
-      # @param _event [Dry::Events::Event] event details including payload
+      # @param _event [Karafka::Core::Monitoring::Event] event details including payload
       def on_app_stopping(_event)
         info 'Stopping Karafka server'
       end
 
       # Logs info that we stopped the Karafka server.
       #
-      # @param _event [Dry::Events::Event] event details including payload
+      # @param _event [Karafka::Core::Monitoring::Event] event details including payload
       def on_app_stopped(_event)
         info 'Stopped Karafka server'
       end
 
       # Logs info when we have dispatched a message the the DLQ
       #
-      # @param event [Dry::Events::Event] event details including payload
+      # @param event [Karafka::Core::Monitoring::Event] event details including payload
       def on_dead_letter_queue_dispatched(event)
         message = event[:message]
         offset = message.offset
@@ -123,12 +128,12 @@ module Karafka
         dlq_topic = event[:caller].topic.dead_letter_queue.topic
         partition = message.partition
 
-        info "Dispatched message #{offset} from #{topic}/#{partition} to DQL topic: #{dlq_topic}"
+        info "Dispatched message #{offset} from #{topic}/#{partition} to DLQ topic: #{dlq_topic}"
       end
 
       # There are many types of errors that can occur in many places, but we provide a single
       # handler for all of them to simplify error instrumentation.
-      # @param event [Dry::Events::Event] event details including payload
+      # @param event [Karafka::Core::Monitoring::Event] event details including payload
       def on_error_occurred(event)
         type = event[:type]
         error = event[:error]

data/lib/karafka/instrumentation/notifications.rb

@@ -19,6 +19,7 @@ module Karafka
       EVENTS = %w[
         app.initialized
         app.running
+        app.quieting
         app.stopping
         app.stopped
 

data/lib/karafka/instrumentation/vendors/datadog/logger_listener.rb

@@ -42,7 +42,7 @@ module Karafka
 
           # Prints info about the fact that a given job has started
           #
-          # @param event [Dry::Events::Event] event details including payload
+          # @param event [Karafka::Core::Monitoring::Event] event details including payload
           def on_worker_process(event)
             current_span = client.trace('karafka.consumer')
             push_tags
@@ -60,7 +60,7 @@ module Karafka
 
           # Prints info about the fact that a given job has finished
           #
-          # @param event [Dry::Events::Event] event details including payload
+          # @param event [Karafka::Core::Monitoring::Event] event details including payload
           def on_worker_processed(event)
             push_tags
 
@@ -80,7 +80,7 @@ module Karafka
 
           # There are many types of errors that can occur in many places, but we provide a single
           # handler for all of them to simplify error instrumentation.
-          # @param event [Dry::Events::Event] event details including payload
+          # @param event [Karafka::Core::Monitoring::Event] event details including payload
           def on_error_occurred(event)
             push_tags
 

data/lib/karafka/pro/processing/coordinator.rb

@@ -20,10 +20,8 @@ module Karafka
         # @param args [Object] anything the base coordinator accepts
         def initialize(*args)
           super
-          @on_enqueued_invoked = false
-          @on_started_invoked = false
-          @on_finished_invoked = false
-          @on_revoked_invoked = false
+
+          @executed = []
           @flow_lock = Mutex.new
         end
 
@@ -34,9 +32,7 @@ module Karafka
           super
 
           @mutex.synchronize do
-            @on_enqueued_invoked = false
-            @on_started_invoked = false
-            @on_finished_invoked = false
+            @executed.clear
             @last_message = messages.last
           end
         end
@@ -50,9 +46,7 @@ module Karafka
         # enqueued
         def on_enqueued
           @flow_lock.synchronize do
-            return if @on_enqueued_invoked
-
-            @on_enqueued_invoked = true
+            return unless executable?(:on_enqueued)
 
             yield(@last_message)
           end
@@ -61,9 +55,7 @@ module Karafka
         # Runs given code only once per all the coordinated jobs upon starting first of them
         def on_started
           @flow_lock.synchronize do
-            return if @on_started_invoked
-
-            @on_started_invoked = true
+            return unless executable?(:on_started)
 
             yield(@last_message)
           end
@@ -75,25 +67,36 @@ module Karafka
         def on_finished
           @flow_lock.synchronize do
             return unless finished?
-            return if @on_finished_invoked
-
-            @on_finished_invoked = true
+            return unless executable?(:on_finished)
 
             yield(@last_message)
           end
         end
 
-        # Runs once when a partition is revoked
+        # Runs once after a partition is revoked
         def on_revoked
           @flow_lock.synchronize do
-            return unless finished?
-            return if @on_revoked_invoked
-
-            @on_revoked_invoked = true
+            return unless executable?(:on_revoked)
 
             yield(@last_message)
           end
         end
+
+        private
+
+        # Checks if given action is executable once. If it is and true is returned, this method
+        # will return false next time it is used.
+        #
+        # @param action [Symbol] what action we want to perform
+        # @return [Boolean] true if we can
+        # @note This method needs to run behind a mutex.
+        def executable?(action)
+          return false if @executed.include?(action)
+
+          @executed << action
+
+          true
+        end
       end
     end
   end

data/lib/karafka/pro/processing/strategies/dlq.rb

@@ -68,9 +68,9 @@ module Karafka
               payload: skippable_message.raw_payload,
               key: skippable_message.partition.to_s,
               headers: skippable_message.headers.merge(
-                'original-topic' => topic.name,
-                'original-partition' => skippable_message.partition.to_s,
-                'original-offset' => skippable_message.offset.to_s
+                'original_topic' => topic.name,
+                'original_partition' => skippable_message.partition.to_s,
+                'original_offset' => skippable_message.offset.to_s
               )
             )
 

data/lib/karafka/process.rb

@@ -10,6 +10,7 @@ module Karafka
       SIGQUIT
       SIGTERM
       SIGTTIN
+      SIGTSTP
     ].freeze
 
     HANDLED_SIGNALS.each do |signal|
@@ -48,21 +49,23 @@ module Karafka
 
     # Traps a single signal and performs callbacks (if any) or just ignores this signal
     # @param [Symbol] signal type that we want to catch
+    # @note Since we do a lot of threading and queuing, we don't want to handle signals from the
+    # trap context s some things may not work there as expected, that is why we spawn a separate
+    # thread to handle the signals process
     def trap_signal(signal)
       trap(signal) do
-        notice_signal(signal)
-        (@callbacks[signal] || []).each(&:call)
+        Thread.new do
+          notice_signal(signal)
+
+          (@callbacks[signal] || []).each(&:call)
+        end
       end
     end
 
     # Informs monitoring about trapped signal
     # @param [Symbol] signal type that we received
-    # @note We cannot perform logging from trap context, that's why
-    #   we have to spin up a new thread to do this
     def notice_signal(signal)
-      Thread.new do
-        Karafka.monitor.instrument('process.notice_signal', caller: self, signal: signal)
-      end
+      Karafka.monitor.instrument('process.notice_signal', caller: self, signal: signal)
     end
   end
 end

data/lib/karafka/processing/jobs_queue.rb

@@ -20,7 +20,7 @@ module Karafka
         # scheduled by Ruby hundreds of thousands of times per group.
         # We cannot use a single semaphore as it could potentially block in listeners that should
         # process with their data and also could unlock when a given group needs to remain locked
-        @semaphores = Hash.new { |h, k| h[k] = Queue.new }
+        @semaphores = Concurrent::Map.new { |h, k| h[k] = Queue.new }
         @in_processing = Hash.new { |h, k| h[k] = [] }
         @mutex = Mutex.new
       end
@@ -47,9 +47,9 @@ module Karafka
           raise(Errors::JobsQueueSynchronizationError, job.group_id) if group.include?(job)
 
           group << job
-        end
 
-        @queue << job
+          @queue << job
+        end
       end
 
       # @return [Jobs::Base, nil] waits for a job from the main queue and returns it once available
@@ -105,7 +105,9 @@ module Karafka
       # @return [Boolean] tell us if we have anything in the processing (or for processing) from
       # a given group.
       def empty?(group_id)
-        @in_processing[group_id].empty?
+        @mutex.synchronize do
+          @in_processing[group_id].empty?
+        end
       end
 
       # Blocks when there are things in the queue in a given group and waits until all the blocking

data/lib/karafka/railtie.rb

@@ -17,9 +17,6 @@ rescue LoadError
 end
 
 if rails
-  # Load Karafka
-  require 'karafka'
-
   # Load ActiveJob adapter
   require 'active_job/karafka'
 

data/lib/karafka/routing/features/base.rb

@@ -13,10 +13,10 @@ module Karafka
         class << self
           # Extends topic and builder with given feature API
           def activate
-            Topic.prepend(self::Topic) if const_defined?('Topic')
-            Proxy.prepend(self::Builder) if const_defined?('Builder')
-            Builder.prepend(self::Builder) if const_defined?('Builder')
-            Builder.prepend(Base::Expander.new(self)) if const_defined?('Contract')
+            Topic.prepend(self::Topic) if const_defined?('Topic', false)
+            Proxy.prepend(self::Builder) if const_defined?('Builder', false)
+            Builder.prepend(self::Builder) if const_defined?('Builder', false)
+            Builder.prepend(Base::Expander.new(self)) if const_defined?('Contract', false)
           end
 
           # Loads all the features and activates them

data/lib/karafka/routing/features/dead_letter_queue/topic.rb

@@ -4,7 +4,7 @@ module Karafka
   module Routing
     module Features
       class DeadLetterQueue < Base
-        # DQL topic extensions
+        # DLQ topic extensions
         module Topic
           # After how many retries should be move data to DLQ
           DEFAULT_MAX_RETRIES = 3

data/lib/karafka/server.rb

@@ -25,12 +25,10 @@ module Karafka
 
       # Method which runs app
       def run
-        # Since we do a lot of threading and queuing, we don't want to stop from the trap context
-        # as some things may not work there as expected, that is why we spawn a separate thread to
-        # handle the stopping process
-        process.on_sigint { Thread.new { stop } }
-        process.on_sigquit { Thread.new { stop } }
-        process.on_sigterm { Thread.new { stop } }
+        process.on_sigint { stop }
+        process.on_sigquit { stop }
+        process.on_sigterm { stop }
+        process.on_sigtstp { quiet }
         process.supervise
 
         # Start is blocking until stop is called and when we stop, it will wait until
@@ -74,7 +72,8 @@ module Karafka
       #   please start a separate thread to do so.
       def stop
         # Initialize the stopping process only if Karafka was running
-        return if Karafka::App.stopping? || Karafka::App.stopped?
+        return if Karafka::App.stopping?
+        return if Karafka::App.stopped?
 
         Karafka::App.stop!
 
@@ -125,6 +124,18 @@ module Karafka
         Karafka::App.stopped! if timeout
       end
 
+      # Quiets the Karafka server.
+      # Karafka will stop processing but won't quiet to consumer group, so no rebalance will be
+      # triggered until final shutdown.
+      def quiet
+        # If we are already quieting or in the stop procedures, we should not do it again.
+        return if Karafka::App.quieting?
+        return if Karafka::App.stopping?
+        return if Karafka::App.stopped?
+
+        Karafka::App.quiet!
+      end
+
       private
 
       # @return [Karafka::Process] process wrapper instance used to catch system signal calls

data/lib/karafka/status.rb

@@ -8,6 +8,7 @@ module Karafka
       initializing: :initialize!,
       initialized: :initialized!,
       running: :run!,
+      quieting: :quiet!,
       stopping: :stop!,
       stopped: :stopped!
     }.freeze

data/lib/karafka/version.rb

@@ -3,5 +3,5 @@
 # Main module namespace
 module Karafka
   # Current Karafka version
-  VERSION = '2.0.16'
+  VERSION = '2.0.18'
 end

data/lib/karafka.rb

@@ -86,6 +86,9 @@ end
 loader = Zeitwerk::Loader.for_gem
 # Do not load Rails extensions by default, this will be handled by Railtie if they are needed
 loader.ignore(Karafka.gem_root.join('lib/active_job'))
+# Do not load Railtie. It will load if after everything is ready, so we don't have to load any
+# Karafka components when we require this railtie. Railtie needs to be loaded last.
+loader.ignore(Karafka.gem_root.join('lib/karafka/railtie'))
 # Do not load pro components as they will be loaded if needed and allowed
 loader.ignore(Karafka.core_root.join('pro/'))
 # Do not load vendors instrumentation components. Those need to be required manually if needed
@@ -96,3 +99,6 @@ loader.eager_load
 # This will load features but since Pro are not loaded automatically, they will not be visible
 # nor included here
 ::Karafka::Routing::Features::Base.load_all
+
+# Load railtie after everything else is ready so we know we can rely on it.
+require 'karafka/railtie'

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: karafka
 version: !ruby/object:Gem::Version
-  version: 2.0.16
+  version: 2.0.18
 platform: ruby
 authors:
 - Maciej Mensfeld
@@ -35,7 +35,7 @@ cert_chain:
   Qf04B9ceLUaC4fPVEz10FyobjaFoY4i32xRto3XnrzeAgfEe4swLq8bQsR3w/EF3
   MGU0FeSV2Yj7Xc2x/7BzLK8xQn5l7Yy75iPF+KP3vVmDHnNl
   -----END CERTIFICATE-----
-date: 2022-11-09 00:00:00.000000000 Z
+date: 2022-11-18 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: karafka-core
@@ -176,6 +176,7 @@ files:
 - lib/karafka/cli/install.rb
 - lib/karafka/cli/server.rb
 - lib/karafka/connection/client.rb
+- lib/karafka/connection/consumer_group_status.rb
 - lib/karafka/connection/listener.rb
 - lib/karafka/connection/listeners_batch.rb
 - lib/karafka/connection/messages_buffer.rb

metadata.gz.sig

@@ -1,3 +1,2 @@
-��'���?w+b�<Ə`Xd����#�>�i���x�k�
%P9�����L�w�i�j%+�yz\X�?A�����T�*=�0�2�b���%B���(�B�i`%�Y>����w��D���j��m�e
-3�4�9^a�r��4�m�O��K�f�Hu:c�PPT��o�����8�"�r� �lᩉ&ef��o)��9K�����b��'2U>���蹞�B�\V=�
]ޯ�{u��y%8Q;=R���.ɽ��O20��*ڔ
-6QFof�}�Sb�>rj�UL�3��R�`����Z?6�
+��7j�]��Q�"�:|fO�l�qpA�lhe�+��ᜄfK�X񳒗�
+�]#E�_?�� �k��;qeN猅�9���ح�f���BɣY��8՟�x̗-֞�ݢ�fگ݆#�i����ÔFw@�u���/�U���BC�-��w���j_�{��������/�^r�h|��
��x�