karafka 2.0.28 → 2.0.30

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: c69f25e46161a6f20e57262b4eb67a852b2345e4cea05b4919bca43105fa2ca2
-  data.tar.gz: fb065461afccd8d2416bd509dfa6f04b47df07dd6b6ffcbcde9d266f35b71412
+  metadata.gz: 2c983b3fbf07db3e91f3c051a8b4de4b2d88619877fbf66a4e434e38bf67c96b
+  data.tar.gz: babb3c513e6bb3fd0610363e3bbd9e7fcf07b1820f02fdd80ccacc8a109aab08
 SHA512:
-  metadata.gz: 1819faf0d836a42cc6d05adde4cc896a31ec58d3b0de19f4f62d224fd16d1b3ebaa816afc45fc67025de579ac63c81cc09e18ffa4dacb8e2f05f636b21df24b5
-  data.tar.gz: 65437e05a791e8e0f86a17d08ab331ca52dfdadbb3979109b2358b6ba700ca63e6e956d0125784da0228adabf2aa92d18058f7f3440e72ec98b3659c9b127ba1
+  metadata.gz: ea000c13a834ffba97c1837fa1f19dabbf78dbe8958354a8200bb5d2c0940852343521e6740113e07890f97a7e6edd108d8595a66688c4c14dbf313e86ec715f
+  data.tar.gz: 81ae980ef7950c6cae7d1bea2751d1826e5bbf4c1384be4736af895977fb7ff38f70c24a9b7b248cb18744e042ffb0ba0dc154171c29dcc73262733d87c31ea1

data/CHANGELOG.md

@@ -1,8 +1,23 @@
 # Karafka framework changelog
 
+## 2.0.30 (2022-01-31)
+- [Improvement] Alias `--consumer-groups` with `--include-consumer-groups`
+- [Improvement] Alias `--subscription-groups` with `--include-subscription-groups`
+- [Improvement] Alias `--topics` with `--include-topics`
+- [Improvement] Introduce `--exclude-consumer-groups` for ability to exclude certain consumer groups from running
+- [Improvement] Introduce `--exclude-subscription-groups` for ability to exclude certain subscription groups from running
+- [Improvement] Introduce `--exclude-topics` for ability to exclude certain topics from running
+
+## 2.0.29 (2023-01-30)
+- [Improvement] Make sure, that the `Karafka#producer` instance has the `LoggerListener` enabled in the install template, so Karafka by default prints both consumer and producer info.
+- [Improvement] Extract the code loading capabilities of Karafka console from the executable, so web can use it to provide CLI commands.
+- [Fix] Fix for: running karafka console results in NameError with Rails (#1280)
+- [Fix] Make sure, that the `caller` for async errors is being published.
+- [Change] Make sure that WaterDrop `2.4.10` or higher is used with this release to support Web-UI.
+
 ## 2.0.28 (2023-01-25)
 - **[Feature]** Provide the ability to use Dead Letter Queue with Virtual Partitions.
-- [Improvement] Collapse Virtual Partitions upon retryable error to a single partition. This allows dead letter queue to operate and mitigate issues arising from work virtualization. This removes uncertainties upon errors that can be retried and processed. Affects given topic partition virtualization only for multi-topic and mulit-partition parallelization. It also minimizes potential "flickering" where given data set has potentially many corrupted messages. The collapse will last until all the messages from the collective corrupted batch are processed. After that, virtualization will resume.
+- [Improvement] Collapse Virtual Partitions upon retryable error to a single partition. This allows dead letter queue to operate and mitigate issues arising from work virtualization. This removes uncertainties upon errors that can be retried and processed. Affects given topic partition virtualization only for multi-topic and multi-partition parallelization. It also minimizes potential "flickering" where given data set has potentially many corrupted messages. The collapse will last until all the messages from the collective corrupted batch are processed. After that, virtualization will resume.
 - [Improvement] Introduce `#collapsed?` consumer method available for consumers using Virtual Partitions.
 - [Improvement] Allow for customization of DLQ dispatched message details in Pro (#1266) via the `#enhance_dlq_message` consumer method.
 - [Improvement] Include `original_consumer_group` in the DLQ dispatched messages in Pro.

data/Gemfile.lock

@@ -1,31 +1,31 @@
 PATH
   remote: .
   specs:
-    karafka (2.0.28)
+    karafka (2.0.30)
       karafka-core (>= 2.0.9, < 3.0.0)
       thor (>= 0.20)
-      waterdrop (>= 2.4.9, < 3.0.0)
+      waterdrop (>= 2.4.10, < 3.0.0)
       zeitwerk (~> 2.3)
 
 GEM
   remote: https://rubygems.org/
   specs:
-    activejob (7.0.4.1)
-      activesupport (= 7.0.4.1)
+    activejob (7.0.4.2)
+      activesupport (= 7.0.4.2)
       globalid (>= 0.3.6)
-    activesupport (7.0.4.1)
+    activesupport (7.0.4.2)
       concurrent-ruby (~> 1.0, >= 1.0.2)
       i18n (>= 1.6, < 2)
       minitest (>= 5.1)
       tzinfo (~> 2.0)
     byebug (11.1.3)
-    concurrent-ruby (1.1.10)
+    concurrent-ruby (1.2.0)
     diff-lcs (1.5.0)
     docile (1.4.0)
     factory_bot (6.2.1)
       activesupport (>= 5.0.0)
     ffi (1.15.5)
-    globalid (1.0.1)
+    globalid (1.1.0)
       activesupport (>= 5.0)
     i18n (1.12.0)
       concurrent-ruby (~> 1.0)
@@ -59,14 +59,15 @@ GEM
     simplecov-html (0.12.3)
     simplecov_json_formatter (0.1.4)
     thor (1.2.1)
-    tzinfo (2.0.5)
+    tzinfo (2.0.6)
       concurrent-ruby (~> 1.0)
-    waterdrop (2.4.9)
+    waterdrop (2.4.10)
       karafka-core (>= 2.0.9, < 3.0.0)
       zeitwerk (~> 2.3)
     zeitwerk (2.6.6)
 
 PLATFORMS
+  arm64-darwin-21
   x86_64-linux
 
 DEPENDENCIES
@@ -78,4 +79,4 @@ DEPENDENCIES
   simplecov
 
 BUNDLED WITH
-   2.4.2
+   2.4.5

data/README.md

@@ -10,6 +10,7 @@
 
 Karafka is a Ruby and Rails multi-threaded efficient Kafka processing framework that:
 
+- Has a built-in [Web UI](https://karafka.io/docs/Web-UI-Features/) providing a convenient way to monitor and manage Karafka-based applications.
 - Supports parallel processing in [multiple threads](https://karafka.io/docs/Concurrency-and-multithreading) (also for a [single topic partition](https://karafka.io/docs/Pro-Virtual-Partitions) work)
 - [Automatically integrates](https://karafka.io/docs/Integrating-with-Ruby-on-Rails-and-other-frameworks#integrating-with-ruby-on-rails) with Ruby on Rails
 - Has [ActiveJob backend](https://karafka.io/docs/Active-Job) support (including [ordered jobs](https://karafka.io/docs/Pro-Enhanced-Active-Job#ordered-jobs))
@@ -58,7 +59,8 @@ We also maintain many [integration specs](https://github.com/karafka/karafka/tre
 1. Add and install Karafka:
 
 ```bash
-bundle add karafka
+# Make sure to install Karafka 2.0
+bundle add karafka --version ">= 2.0.28"
 
 bundle exec karafka install
 ```
@@ -82,11 +84,11 @@ bundle exec karafka server
 
 ## Want to Upgrade? LGPL is not for you? Want to help?
 
-I also sell Karafka Pro subscriptions. It includes a commercial-friendly license, priority support, architecture consultations, and high throughput data processing-related features (virtual partitions, long-running jobs, and more).
+I also sell Karafka Pro subscriptions. It includes a commercial-friendly license, priority support, architecture consultations, enhanced Web UI and high throughput data processing-related features (virtual partitions, long-running jobs, and more).
 
 **20%** of the income will be distributed back to other OSS projects that Karafka uses under the hood.
 
-Help me provide high-quality open-source software. Please see the Karafka [homepage](https://karafka.io) for more details.
+Help me provide high-quality open-source software. Please see the Karafka [homepage](https://karafka.io/#become-pro) for more details.
 
 ## Support
 

data/bin/karafka

@@ -6,24 +6,6 @@ require 'karafka'
 # our bin/karafka cli
 ENV['KARAFKA_CLI'] = 'true'
 
-# If there is a boot file, we need to require it as we expect it to contain
-# Karafka app setup, routes, etc
-if File.exist?(Karafka.boot_file)
-  rails_env_rb = File.join(Dir.pwd, 'config/environment.rb')
-
-  # Load Rails environment file that starts Rails, so we can reference consumers and other things
-  # from `karafka.rb` file. This will work only for Rails, for non-rails a manual setup is needed
-  require rails_env_rb if Kernel.const_defined?(:Rails) && File.exist?(rails_env_rb)
-
-  require Karafka.boot_file.to_s
-else
-  # However when it is unavailable, we still want to be able to run help command
-  # and install command as they don't require configured app itself to run
-  raise(
-    Karafka::Errors::MissingBootFileError,
-    Karafka.boot_file
-  ) unless %w[-h install].include?(ARGV[0])
-end
-
+Karafka::Cli::Base.load
 Karafka::Cli.prepare
 Karafka::Cli.start

data/bin/verify_license_integrity

@@ -11,6 +11,8 @@ set -e
 if [ "$MODE" != "after" ]; then
   # Check the remote license prior to bundle installing
   curl \
+    --retry 5 \
+    --retry-delay 1 \
     --fail \
     -u $KARAFKA_PRO_USERNAME:$KARAFKA_PRO_PASSWORD \
     https://gems.karafka.io/gems/karafka-license-$KARAFKA_PRO_VERSION.gem \

data/karafka.gemspec

@@ -23,7 +23,7 @@ Gem::Specification.new do |spec|
 
   spec.add_dependency 'karafka-core', '>= 2.0.9', '< 3.0.0'
   spec.add_dependency 'thor', '>= 0.20'
-  spec.add_dependency 'waterdrop', '>= 2.4.9', '< 3.0.0'
+  spec.add_dependency 'waterdrop', '>= 2.4.10', '< 3.0.0'
   spec.add_dependency 'zeitwerk', '~> 2.3'
 
   if $PROGRAM_NAME.end_with?('gem')

data/lib/karafka/cli/base.rb

@@ -34,6 +34,26 @@ module Karafka
       end
 
       class << self
+        # Loads proper environment with what is needed to run the CLI
+        def load
+          # If there is a boot file, we need to require it as we expect it to contain
+          # Karafka app setup, routes, etc
+          if File.exist?(::Karafka.boot_file)
+            rails_env_rb = File.join(Dir.pwd, 'config/environment.rb')
+
+            # Load Rails environment file that starts Rails, so we can reference consumers and
+            # other things from `karafka.rb` file. This will work only for Rails, for non-rails
+            # a manual setup is needed
+            require rails_env_rb if Kernel.const_defined?(:Rails) && File.exist?(rails_env_rb)
+
+            require Karafka.boot_file.to_s
+          # However when it is unavailable, we still want to be able to run help command
+          # and install command as they don't require configured app itself to run
+          elsif %w[-h install].none? { |cmd| cmd == ARGV[0] }
+            raise ::Karafka::Errors::MissingBootFileError, ::Karafka.boot_file
+          end
+        end
+
         # Allows to set options for Thor cli
         # @see https://github.com/erikhuda/thor
         # @param option Single option details

data/lib/karafka/cli/console.rb

@@ -9,22 +9,27 @@ module Karafka
       option aliases: 'c'
 
       class << self
-        # @return [String] Console executing command
+        # @return [String] Console executing command for non-Rails setup
         # @example
         #   Karafka::Cli::Console.command #=> 'KARAFKA_CONSOLE=true bundle exec irb...'
-        def command
-          envs = [
-            "IRBRC='#{Karafka.gem_root}/.console_irbrc'",
-            'KARAFKA_CONSOLE=true'
-          ]
-          "#{envs.join(' ')} bundle exec irb -r #{Karafka.boot_file}"
+        def console
+          "IRBRC='#{Karafka.gem_root}/.console_irbrc' bundle exec irb -r #{Karafka.boot_file}"
+        end
+
+        # @return [String] Console executing command for Rails setup
+        # @note In case of Rails, it has its own console, hence we can just defer to it
+        def rails_console
+          'bundle exec rails console'
         end
       end
 
       # Start the Karafka console
       def call
         cli.info
-        exec self.class.command
+
+        command = ::Karafka.rails? ? self.class.rails_console : self.class.console
+
+        exec "KARAFKA_CONSOLE=true #{command}"
       end
     end
   end

data/lib/karafka/cli/server.rb

@@ -7,27 +7,73 @@ module Karafka
     class Server < Base
       include Helpers::Colorize
 
+      # Types of things we can include / exclude from the routing via the CLI options
+      SUPPORTED_TYPES = ::Karafka::Routing::ActivityManager::SUPPORTED_TYPES
+
+      private_constant :SUPPORTED_TYPES
+
       desc 'Start the Karafka server (short-cut alias: "s")'
+
       option aliases: 's'
+
+      # Thor does not work well with many aliases combinations, hence we remap the aliases
+      # by ourselves in the code
       option :consumer_groups, type: :array, default: [], aliases: :g
       option :subscription_groups, type: :array, default: []
       option :topics, type: :array, default: []
 
+      %i[
+        include
+        exclude
+      ].each do |action|
+        SUPPORTED_TYPES.each do |type|
+          option(
+            "#{action}_#{type}",
+            type: :array,
+            default: []
+          )
+        end
+      end
+
       # Start the Karafka server
       def call
         # Print our banner and info in the dev mode
         print_marketing_info if Karafka::App.env.development?
 
-        active_routing_config = Karafka::App.config.internal.routing.active
-        active_routing_config.consumer_groups = cli.options[:consumer_groups]
-        active_routing_config.subscription_groups = cli.options[:subscription_groups]
-        active_routing_config.topics = cli.options[:topics]
+        register_inclusions(cli)
+        register_exclusions(cli)
 
         Karafka::Server.run
       end
 
       private
 
+      # Registers things we want to include (if defined)
+      # @param cli [Karafka::Cli] Thor cli handler
+      def register_inclusions(cli)
+        activities = ::Karafka::App.config.internal.routing.activity_manager
+
+        SUPPORTED_TYPES.each do |type|
+          v1 = cli.options[type] || []
+          v2 = cli.options[:"include_#{type}"] || []
+          names = v1 + v2
+
+          names.each { |name| activities.include(type, name) }
+        end
+      end
+
+      # Registers things we want to exclude (if defined)
+      # @param cli [Karafka::Cli] Thor cli handler
+      def register_exclusions(cli)
+        activities = ::Karafka::App.config.internal.routing.activity_manager
+
+        activities.class::SUPPORTED_TYPES.each do |type|
+          names = cli.options[:"exclude_#{type}"] || []
+
+          names.each { |name| activities.exclude(type, name) }
+        end
+      end
+
       # Prints marketing info
       def print_marketing_info
         Karafka.logger.info Info::BANNER

data/lib/karafka/contracts/server_cli_options.rb

@@ -12,69 +12,71 @@ module Karafka
         ).fetch('en').fetch('validations').fetch('server_cli_options')
       end
 
-      optional(:consumer_groups) { |cg| cg.is_a?(Array) }
-      optional(:subscription_groups) { |sg| sg.is_a?(Array) }
-      optional(:topics) { |topics| topics.is_a?(Array) }
+      %i[
+        include
+        exclude
+      ].each do |action|
+        optional(:"#{action}_consumer_groups") { |cg| cg.is_a?(Array) }
+        optional(:"#{action}_subscription_groups") { |sg| sg.is_a?(Array) }
+        optional(:"#{action}_topics") { |topics| topics.is_a?(Array) }
 
-      virtual do |data, errors|
-        next unless errors.empty?
-        next unless data.key?(:consumer_groups)
+        virtual do |data, errors|
+          next unless errors.empty?
 
-        value = data.fetch(:consumer_groups)
+          value = data.fetch(:"#{action}_consumer_groups")
 
-        # If there were no consumer_groups declared in the server cli, it means that we will
-        # run all of them and no need to validate them here at all
-        next if value.empty?
-        next if (value - Karafka::App.consumer_groups.map(&:name)).empty?
+          # If there were no consumer_groups declared in the server cli, it means that we will
+          # run all of them and no need to validate them here at all
+          next if value.empty?
+          next if (value - Karafka::App.consumer_groups.map(&:name)).empty?
 
-        # Found unknown consumer groups
-        [[%i[consumer_groups], :consumer_groups_inclusion]]
-      end
+          # Found unknown consumer groups
+          [[[:"#{action}_consumer_groups"], :consumer_groups_inclusion]]
+        end
 
-      virtual do |data, errors|
-        next unless errors.empty?
-        next unless data.key?(:subscription_groups)
+        virtual do |data, errors|
+          next unless errors.empty?
 
-        value = data.fetch(:subscription_groups)
+          value = data.fetch(:"#{action}_subscription_groups")
 
-        # If there were no subscription_groups declared in the server cli, it means that we will
-        # run all of them and no need to validate them here at all
-        next if value.empty?
+          # If there were no subscription_groups declared in the server cli, it means that we will
+          # run all of them and no need to validate them here at all
+          next if value.empty?
 
-        subscription_groups = Karafka::App
-                              .consumer_groups
-                              .map(&:subscription_groups)
-                              .flatten
-                              .map(&:name)
+          subscription_groups = Karafka::App
+                                .consumer_groups
+                                .map(&:subscription_groups)
+                                .flatten
+                                .map(&:name)
 
-        next if (value - subscription_groups).empty?
+          next if (value - subscription_groups).empty?
 
-        # Found unknown subscription groups
-        [[%i[subscription_groups], :subscription_groups_inclusion]]
-      end
+          # Found unknown subscription groups
+          [[[:"#{action}_subscription_groups"], :subscription_groups_inclusion]]
+        end
 
-      virtual do |data, errors|
-        next unless errors.empty?
-        next unless data.key?(:topics)
+        virtual do |data, errors|
+          next unless errors.empty?
 
-        value = data.fetch(:topics)
+          value = data.fetch(:"#{action}_topics")
 
-        # If there were no topics declared in the server cli, it means that we will
-        # run all of them and no need to validate them here at all
-        next if value.empty?
+          # If there were no topics declared in the server cli, it means that we will
+          # run all of them and no need to validate them here at all
+          next if value.empty?
 
-        topics = Karafka::App
-                 .consumer_groups
-                 .map(&:subscription_groups)
-                 .flatten
-                 .map(&:topics)
-                 .map { |gtopics| gtopics.map(&:name) }
-                 .flatten
+          topics = Karafka::App
+                   .consumer_groups
+                   .map(&:subscription_groups)
+                   .flatten
+                   .map(&:topics)
+                   .map { |gtopics| gtopics.map(&:name) }
+                   .flatten
 
-        next if (value - topics).empty?
+          next if (value - topics).empty?
 
-        # Found unknown topics
-        [[%i[topics], :topics_inclusion]]
+          # Found unknown topics
+          [[[:"#{action}_topics"], :topics_inclusion]]
+        end
       end
 
       # Makes sure we have anything to subscribe to when we start the server
@@ -83,7 +85,7 @@ module Karafka
 
         next unless Karafka::App.subscription_groups.empty?
 
-        [[%i[topics], :topics_missing]]
+        [[%i[include_topics], :topics_missing]]
       end
     end
   end

data/lib/karafka/instrumentation/callbacks/error.rb

@@ -28,6 +28,7 @@ module Karafka
 
           @monitor.instrument(
             'error.occurred',
+            caller: self,
             subscription_group_id: @subscription_group_id,
             consumer_group_id: @consumer_group_id,
             type: 'librdkafka.error',

data/lib/karafka/railtie.rb

@@ -2,14 +2,7 @@
 
 # This file contains Railtie for auto-configuration
 
-rails = false
-
-begin
-  # Do not load Rails again if already loaded
-  Object.const_defined?('Rails::Railtie') || require('rails')
-
-  rails = true
-rescue LoadError
+unless Karafka.rails?
   # Without defining this in any way, Zeitwerk ain't happy so we do it that way
   module Karafka
     class Railtie
@@ -17,7 +10,7 @@ rescue LoadError
   end
 end
 
-if rails
+if Karafka.rails?
   # Load ActiveJob adapter
   require 'active_job/karafka'
 

data/lib/karafka/routing/activity_manager.rb

@@ -0,0 +1,84 @@
+# frozen_string_literal: true
+
+module Karafka
+  module Routing
+    # Allows us to get track of which consumer groups, subscription groups and topics are enabled
+    # or disabled via CLI
+    class ActivityManager
+      # Supported types of inclusions and exclusions
+      SUPPORTED_TYPES = %i[
+        consumer_groups
+        subscription_groups
+        topics
+      ].freeze
+
+      def initialize
+        @included = Hash.new { |h, k| h[k] = [] }
+        @excluded = Hash.new { |h, k| h[k] = [] }
+      end
+
+      # Adds resource to included/active
+      # @param type [Symbol] type for inclusion
+      # @param name [String] name of the element
+      def include(type, name)
+        validate!(type)
+
+        @included[type] << name
+      end
+
+      # Adds resource to excluded
+      # @param type [Symbol] type for inclusion
+      # @param name [String] name of the element
+      def exclude(type, name)
+        validate!(type)
+
+        @excluded[type] << name
+      end
+
+      # @param type [Symbol] type for inclusion
+      # @param name [String] name of the element
+      # @return [Boolean] is the given resource active or not
+      def active?(type, name)
+        validate!(type)
+
+        included = @included[type]
+        excluded = @excluded[type]
+
+        # If nothing defined, all active by default
+        return true if included.empty? && excluded.empty?
+        # Inclusion supersedes exclusion in case someone wrote both
+        return true if !included.empty? && included.include?(name)
+
+        # If there are exclusions but our is not excluded and no inclusions or included, it's ok
+        !excluded.empty? &&
+          !excluded.include?(name) &&
+          (included.empty? || included.include?(name))
+      end
+
+      # @return [Hash] accumulated data in a hash for validations
+      def to_h
+        (
+          SUPPORTED_TYPES.map { |type| ["include_#{type}".to_sym, @included[type]] } +
+          SUPPORTED_TYPES.map { |type| ["exclude_#{type}".to_sym, @excluded[type]] }
+        ).to_h
+      end
+
+      # Clears the manager
+      def clear
+        @included.clear
+        @excluded.clear
+      end
+
+      private
+
+      # Checks if the type we want to register is supported
+      #
+      # @param type [Symbol] type for inclusion
+      def validate!(type)
+        return if SUPPORTED_TYPES.include?(type)
+
+        raise(::Karafka::Errors::UnsupportedCaseError, type)
+      end
+    end
+  end
+end

data/lib/karafka/routing/consumer_group.rb

@@ -31,10 +31,7 @@ module Karafka
 
       # @return [Boolean] true if this consumer group should be active in our current process
       def active?
-        cgs = Karafka::App.config.internal.routing.active.consumer_groups
-
-        # When empty it means no groups were specified, hence all should be used
-        cgs.empty? || cgs.include?(name)
+        Karafka::App.config.internal.routing.activity_manager.active?(:consumer_groups, name)
       end
 
       # Builds a topic representation inside of a current consumer group route

data/lib/karafka/routing/subscription_group.rb

@@ -58,10 +58,7 @@ module Karafka
 
       # @return [Boolean] is this subscription group one of active once
       def active?
-        sgs = Karafka::App.config.internal.routing.active.subscription_groups
-
-        # When empty it means no groups were specified, hence all should be used
-        sgs.empty? || sgs.include?(name)
+        Karafka::App.config.internal.routing.activity_manager.active?(:subscription_groups, name)
       end
 
       private

data/lib/karafka/routing/topic.rb

@@ -87,10 +87,7 @@ module Karafka
         # Never active if disabled via routing
         return false unless @active
 
-        topics = Karafka::App.config.internal.routing.active.topics
-
-        # When empty it means no topics were specified, hence all should be used
-        topics.empty? || topics.include?(name)
+        Karafka::App.config.internal.routing.activity_manager.active?(:topics, name)
       end
 
       # @return [Hash] hash with all the topic attributes

data/lib/karafka/server.rb

@@ -33,7 +33,7 @@ module Karafka
         # We cannot validate this during the start because config needs to be populated and routes
         # need to be defined.
         Contracts::ServerCliOptions.new.validate!(
-          Karafka::App.config.internal.routing.active.to_h
+          Karafka::App.config.internal.routing.activity_manager.to_h
         )
 
         process.on_sigint { stop }

data/lib/karafka/setup/config.rb

@@ -111,12 +111,8 @@ module Karafka
           setting :subscription_groups_builder, default: Routing::SubscriptionGroupsBuilder.new
 
           # Internally assigned list of limits on routings active for the current process
-          # This should be overwritten by the CLI command
-          setting :active do
-            setting :consumer_groups, default: [].freeze
-            setting :subscription_groups, default: [].freeze
-            setting :topics, default: [].freeze
-          end
+          # This can be altered by the CLI command
+          setting :activity_manager, default: Routing::ActivityManager.new
         end
 
         setting :processing do

data/lib/karafka/templates/karafka.rb.erb

@@ -40,6 +40,12 @@ class KarafkaApp < Karafka::App
   Karafka.monitor.subscribe(Karafka::Instrumentation::LoggerListener.new)
   # Karafka.monitor.subscribe(Karafka::Instrumentation::ProctitleListener.new)
 
+  # This logger prints the producer development info using the Karafka logger.
+  # It is similar to the consumer logger listener but producer oriented.
+  Karafka.producer.monitor.subscribe(
+    WaterDrop::Instrumentation::LoggerListener.new(Karafka.logger)
+  )
+
   routes.draw do
 <% if rails? -%>
     # Uncomment this if you use Karafka with ActiveJob

data/lib/karafka/version.rb

@@ -3,5 +3,5 @@
 # Main module namespace
 module Karafka
   # Current Karafka version
-  VERSION = '2.0.28'
+  VERSION = '2.0.30'
 end

data/lib/karafka.rb

@@ -70,6 +70,19 @@ module Karafka
       App.config.license.token != false
     end
 
+    # @return [Boolean] Do we run within/with Rails. We use this to initialize Railtie and proxy
+    #   the console invocation to Rails
+    def rails?
+      return @rails if instance_variable_defined?('@rails')
+
+      # Do not load Rails again if already loaded
+      Object.const_defined?('Rails::Railtie') || require('rails')
+
+      @rails = true
+    rescue LoadError
+      @rails = false
+    end
+
     # @return [String] path to a default file that contains booting procedure etc
     # @note By default it is a file called 'karafka.rb' but it can be specified as you wish if you
     #   have Karafka that is merged into a Sinatra/Rails app and karafka.rb is taken.

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: karafka
 version: !ruby/object:Gem::Version
-  version: 2.0.28
+  version: 2.0.30
 platform: ruby
 authors:
 - Maciej Mensfeld
@@ -35,7 +35,7 @@ cert_chain:
   Qf04B9ceLUaC4fPVEz10FyobjaFoY4i32xRto3XnrzeAgfEe4swLq8bQsR3w/EF3
   MGU0FeSV2Yj7Xc2x/7BzLK8xQn5l7Yy75iPF+KP3vVmDHnNl
   -----END CERTIFICATE-----
-date: 2023-01-25 00:00:00.000000000 Z
+date: 2023-01-31 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: karafka-core
@@ -77,7 +77,7 @@ dependencies:
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: 2.4.9
+        version: 2.4.10
     - - "<"
       - !ruby/object:Gem::Version
         version: 3.0.0
@@ -87,7 +87,7 @@ dependencies:
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: 2.4.9
+        version: 2.4.10
     - - "<"
       - !ruby/object:Gem::Version
         version: 3.0.0
@@ -289,6 +289,7 @@ files:
 - lib/karafka/processing/worker.rb
 - lib/karafka/processing/workers_batch.rb
 - lib/karafka/railtie.rb
+- lib/karafka/routing/activity_manager.rb
 - lib/karafka/routing/builder.rb
 - lib/karafka/routing/consumer_group.rb
 - lib/karafka/routing/consumer_mapper.rb