karafka 2.1.12 → 2.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: de7a5880846f3b3cdab696683f4753bebdb6d133648297e930a415949937d1f5
-  data.tar.gz: df12badb044151bccbecb3832302182794af543b1740d9a939f984b299bd1521
+  metadata.gz: 6ca6426dc8527aac122a1d9ebcaccf33a4eff1608133b210e68e66d3a5f5c2c7
+  data.tar.gz: 85233cc04e591e5a96d53b3b9d3fe203b4b56bbde24c564ffa3663daeb3673c6
 SHA512:
-  metadata.gz: 17fb8af9c36c3df7e6ef084aab4de9d796710cf6a324cbbac4be29a57dc09f254bac817ec0b44e2a0cf0d2b8aae69ab3834eb59c93bce4d4a540d8000808f31f
-  data.tar.gz: 8fedb0f5dfe436bd3854caf5a4ac0da367345be98e059e47ca24dd9d31d414aaf25debc66533965f9fdbdd3b5a78aeb4d719929b1ee274f5f3a36b3737ee2393
+  metadata.gz: e89b72adccdb6a622d571ab9c580191e725cdb98d88f5d86f59d9e4a9900eff74fa438b3838cfe104f7259c0c4cee0dc60f7fb6d16cf14b59ff9229170bae504
+  data.tar.gz: a8ba4b92e1002c6d1112837e1b96706208ba0d55fa39feaadaa4f04f4c616095eb406d72bffa50bd346fb1daff7ca45951efd2237a71c25a7539be5a4397a781

data/CHANGELOG.md

@@ -1,5 +1,43 @@
 # Karafka framework changelog
 
+## 2.2.0 (2023-09-01)
+- **[Feature]** Introduce dynamic topic subscriptions based on patterns [Pro].
+- [Enhancement] Allow for `Karafka::Admin` setup reconfiguration via `config.admin` scope.
+- [Enhancement] Make sure that consumer group used by `Karafka::Admin` obeys the `ConsumerMapper` setup.
+- [Fix] Fix a case where subscription group would not accept a symbol name.
+
+### Upgrade notes
+
+As always, please make sure you have upgraded to the most recent version of `2.1` before upgrading to `2.2`.
+
+If you are not using Kafka ACLs, there is no action you need to take.
+
+If you are using Kafka ACLs and you've set up permissions for `karafka_admin` group, please note that this name has now been changed and is subject to [Consumer Name Mapping](https://karafka.io/docs/Consumer-mappers/).
+
+That means you must ensure that the new consumer group that by default equals `CLIENT_ID_karafka_admin` has appropriate permissions. Please note that the Web UI also uses this group.
+
+`Karafka::Admin` now has its own set of configuration options available, and you can find more details about that [here](https://karafka.io/docs/Topics-management-and-administration/#configuration).
+
+If you want to maintain the `2.1` behavior, that is `karafka_admin` admin group, we recommend introducing this case inside your consumer mapper. Assuming you use the default one, the code will look as follows:
+
+```ruby
+  class MyMapper
+    def call(raw_consumer_group_name)
+      # If group is the admin one, use as it was in 2.1
+      return 'karafka_admin' if raw_consumer_group_name == 'karafka_admin'
+
+      # Otherwise use default karafka strategy for the rest
+      "#{Karafka::App.config.client_id}_#{raw_consumer_group_name}"
+    end
+  end
+```
+
+## 2.1.13 (2023-08-28)
+- **[Feature]** Introduce Cleaning API for much better memory management for iterative data processing [Pro].
+- [Enhancement] Automatically free message resources after processed for ActiveJob jobs [Pro]
+- [Enhancement] Free memory used by the raw payload as fast as possible after obtaining it from `karafka-rdkafka`.
+- [Enhancement] Support changing `service_name` in DataDog integration.
+
 ## 2.1.12 (2023-08-25)
 - [Fix] Fix a case where DLQ + VP without intermediate marking would mark earlier message then the last one.
 
@@ -9,7 +47,10 @@
 
 ## 2.1.10 (2023-08-21)
 - [Enhancement] Introduce `connection.client.rebalance_callback` event for instrumentation of rebalances.
+- [Enhancement] Introduce new `runner.before_call` monitor event.
 - [Refactor] Introduce low level commands proxy to handle deviation in how we want to run certain commands and how rdkafka-ruby runs that by design.
+- [Change] No longer validate excluded topics routing presence if patterns any as it does not match pattern subscriptions where you can exclude things that could be subscribed in the future.
+- [Fix] do not report negative lag stored in the DD listener.
 - [Fix] Do not report lags in the DD listener for cases where the assignment is not workable.
 - [Fix] Do not report negative lags in the DD listener.
 - [Fix] Extremely fast shutdown after boot in specs can cause process not to stop.

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    karafka (2.1.12)
+    karafka (2.2.0)
       karafka-core (>= 2.1.1, < 2.2.0)
       thor (>= 0.20)
       waterdrop (>= 2.6.6, < 3.0.0)

data/bin/record_rss

@@ -0,0 +1,50 @@
+#!/bin/bash
+
+# This script monitors and records the Resident Set Size (RSS) of a process given its PID.
+# The RSS is logged every second to the specified output file until the process terminates.
+#
+# Usage:
+# ./script_name.sh <PID> <OUTPUT_FILE>
+#
+# Arguments:
+# <PID>         - Process ID of the process you want to monitor.
+# <OUTPUT_FILE> - Name of the file where RSS values will be logged.
+#
+# The script first checks if the correct number of arguments are provided.
+# It then verifies if the given PID exists. If it does, it starts recording the RSS.
+# For every iteration, the script fetches the current RSS of the process using the 'ps' command,
+# then appends the RSS value along with a timestamp to the output file.
+# This recording is done every second.
+# The loop stops if the process with the given PID terminates.
+# An informative message is printed out when recording starts and when it stops.
+
+# Check if the correct number of arguments are passed
+if [ "$#" -ne 2 ]; then
+    echo "Usage: $0 <PID> <OUTPUT_FILE>"
+    exit 1
+fi
+
+PID=$1
+OUTPUT_FILE=$2
+
+# Check if the given PID exists
+if ! kill -0 $PID 2>/dev/null; then
+    echo "Error: PID $PID does not exist."
+    exit 1
+fi
+
+# Start recording the RSS
+echo "Recording RSS for PID $PID every second to $OUTPUT_FILE..."
+
+while kill -0 $PID 2>/dev/null; do
+    RSS=$(ps -o rss= -p $PID)
+    if [ -z "$RSS" ]; then
+        echo "Error: Failed to get RSS for PID $PID."
+        exit 1
+    fi
+    TIMESTAMP=$(date '+%Y-%m-%d %H:%M:%S')
+    echo "$TIMESTAMP: $RSS KB" >> $OUTPUT_FILE
+    sleep 1
+done
+
+echo "Process $PID has terminated. Stopping recording."

data/config/locales/errors.yml

@@ -35,6 +35,10 @@ en:
       key_must_be_a_symbol: All keys under the kafka settings scope need to be symbols
       max_timeout_vs_pause_max_timeout: pause_timeout must be less or equal to pause_max_timeout
       shutdown_timeout_vs_max_wait_time: shutdown_timeout must be more than max_wait_time
+      admin.kafka_format: needs to be a hash
+      admin.group_id_format: 'needs to be a string with a Kafka accepted format'
+      admin.max_wait_time_format: 'needs to be an integer bigger than 0'
+      admin.max_attempts_format: 'needs to be an integer bigger than 0'
 
     server_cli_options:
       missing: needs to be present

data/config/locales/pro_errors.yml

@@ -28,6 +28,20 @@ en:
       pause_with_exponential_backoff_format: needs to be either true or false
       pause_timeout_max_timeout_vs_pause_max_timeout: pause_timeout must be less or equal to pause_max_timeout
 
+      patterns.active_format: 'needs to be boolean'
+      patterns.type_format: 'needs to be :matcher, :discovered or :regular'
+
+    consumer_group:
+      patterns_format: must be an array with hashes
+      patterns_missing: needs to be present
+      patterns_regexps_not_unique: 'must be unique within consumer group'
+
+    pattern:
+      regexp_format: must be a regular expression
+      name_format: 'needs to be a string with a Kafka accepted format'
+      regexp_string_format: 'needs to be a string and start with ^'
+      missing: needs to be present
+
     config:
       encryption.active_format: 'needs to be either true or false'
       encryption.public_key_invalid: 'is not a valid public RSA key'
@@ -37,3 +51,6 @@ en:
       encryption.version_format: must be a non-empty string
       encryption.public_key_format: 'is not a valid public RSA key'
       encryption.private_keys_invalid: 'contains an invalid private RSA key string'
+
+      patterns.ttl_format: needs to be an integer bigger than 0
+      patterns.ttl_missing: needs to be present

data/lib/karafka/admin.rb

@@ -7,32 +7,9 @@ module Karafka
   #   Since admin actions are not performed that often, that should be ok.
   #
   # @note It always uses the primary defined cluster and does not support multi-cluster work.
-  #   If you need this, just replace the cluster info for the time you use this
+  #   Cluster on which operations are performed can be changed via `admin.kafka` config, however
+  #   there is no multi-cluster runtime support.
   module Admin
-    # We wait only for this amount of time before raising error as we intercept this error and
-    # retry after checking that the operation was finished or failed using external factor.
-    MAX_WAIT_TIMEOUT = 1
-
-    # How many times should be try. 1 x 60 => 60 seconds wait in total
-    MAX_ATTEMPTS = 60
-
-    # Defaults for config
-    CONFIG_DEFAULTS = {
-      'group.id': 'karafka_admin',
-      # We want to know when there is no more data not to end up with an endless loop
-      'enable.partition.eof': true,
-      'statistics.interval.ms': 0,
-      # Fetch at most 5 MBs when using admin
-      'fetch.message.max.bytes': 5 * 1_048_576,
-      # Do not commit offset automatically, this prevents offset tracking for operations involving
-      # a consumer instance
-      'enable.auto.commit': false,
-      # Make sure that topic metadata lookups do not create topics accidentally
-      'allow.auto.create.topics': false
-    }.freeze
-
-    private_constant :CONFIG_DEFAULTS, :MAX_WAIT_TIMEOUT, :MAX_ATTEMPTS
-
     class << self
       # Allows us to read messages from the topic
       #
@@ -123,7 +100,7 @@ module Karafka
           handler = admin.create_topic(name, partitions, replication_factor, topic_config)
 
           with_re_wait(
-            -> { handler.wait(max_wait_timeout: MAX_WAIT_TIMEOUT) },
+            -> { handler.wait(max_wait_timeout: app_config.admin.max_wait_time) },
             -> { topics_names.include?(name) }
           )
         end
@@ -137,7 +114,7 @@ module Karafka
           handler = admin.delete_topic(name)
 
           with_re_wait(
-            -> { handler.wait(max_wait_timeout: MAX_WAIT_TIMEOUT) },
+            -> { handler.wait(max_wait_timeout: app_config.admin.max_wait_time) },
             -> { !topics_names.include?(name) }
           )
         end
@@ -152,7 +129,7 @@ module Karafka
           handler = admin.create_partitions(name, partitions)
 
           with_re_wait(
-            -> { handler.wait(max_wait_timeout: MAX_WAIT_TIMEOUT) },
+            -> { handler.wait(max_wait_timeout: app_config.admin.max_wait_time) },
             -> { topic(name).fetch(:partition_count) >= partitions }
           )
         end
@@ -242,7 +219,7 @@ module Karafka
       rescue Rdkafka::AbstractHandle::WaitTimeoutError
         return if breaker.call
 
-        retry if attempt <= MAX_ATTEMPTS
+        retry if attempt <= app_config.admin.max_attempts
 
         raise
       end
@@ -251,12 +228,18 @@ module Karafka
       # @param settings [Hash] extra settings for config (if needed)
       # @return [::Rdkafka::Config] rdkafka config
       def config(type, settings)
-        config_hash = Karafka::Setup::AttributesMap.public_send(
-          type,
-          Karafka::App.config.kafka.dup.merge(CONFIG_DEFAULTS).merge!(settings)
+        group_id = app_config.consumer_mapper.call(
+          app_config.admin.group_id
         )
 
-        ::Rdkafka::Config.new(config_hash)
+        app_config
+          .kafka
+          .then(&:dup)
+          .merge(app_config.admin.kafka)
+          .merge!(settings)
+          .tap { |config| config[:'group.id'] = group_id }
+          .then { |config| Karafka::Setup::AttributesMap.public_send(type, config) }
+          .then { |config| ::Rdkafka::Config.new(config) }
       end
 
       # Resolves the offset if offset is in a time format. Otherwise returns the offset without
@@ -281,6 +264,11 @@ module Karafka
           offset
         end
       end
+
+      # @return [Karafka::Core::Configurable::Node] root node config
+      def app_config
+        ::Karafka::App.config
+      end
     end
   end
 end

data/lib/karafka/connection/client.rb

@@ -510,7 +510,7 @@ module Karafka
 
         # Subscription needs to happen after we assigned the rebalance callbacks just in case of
         # a race condition
-        consumer.subscribe(*@subscription_group.topics.map(&:name))
+        consumer.subscribe(*@subscription_group.subscriptions)
         consumer
       end
 

data/lib/karafka/contracts/config.rb

@@ -34,6 +34,14 @@ module Karafka
       required(:max_wait_time) { |val| val.is_a?(Integer) && val.positive? }
       required(:kafka) { |val| val.is_a?(Hash) && !val.empty? }
 
+      nested(:admin) do
+        # Can be empty because inherits values from the root kafka
+        required(:kafka) { |val| val.is_a?(Hash) }
+        required(:group_id) { |val| val.is_a?(String) && Contracts::TOPIC_REGEXP.match?(val) }
+        required(:max_wait_time) { |val| val.is_a?(Integer) && val.positive? }
+        required(:max_attempts) { |val| val.is_a?(Integer) && val.positive? }
+      end
+
       # We validate internals just to be sure, that they are present and working
       nested(:internal) do
         required(:status) { |val| !val.nil? }
@@ -74,6 +82,7 @@ module Karafka
         end
       end
 
+      # Ensure all root kafka keys are symbols
       virtual do |data, errors|
         next unless errors.empty?
 
@@ -88,6 +97,21 @@ module Karafka
         detected_errors
       end
 
+      # Ensure all admin kafka keys are symbols
+      virtual do |data, errors|
+        next unless errors.empty?
+
+        detected_errors = []
+
+        data.fetch(:admin).fetch(:kafka).each_key do |key|
+          next if key.is_a?(Symbol)
+
+          detected_errors << [[:admin, :kafka, key], :key_must_be_a_symbol]
+        end
+
+        detected_errors
+      end
+
       virtual do |data, errors|
         next unless errors.empty?
 

data/lib/karafka/errors.rb

@@ -41,6 +41,9 @@ module Karafka
     # Raised when the license token is not valid
     InvalidLicenseTokenError = Class.new(BaseError)
 
+    # Raised on attempt to deserializer a cleared message
+    MessageClearedError = Class.new(BaseError)
+
     # This should never happen. Please open an issue if it does.
     InvalidCoordinatorStateError = Class.new(BaseError)
 

data/lib/karafka/instrumentation/vendors/datadog/logger_listener.rb

@@ -12,11 +12,14 @@ module Karafka
           include ::Karafka::Core::Configurable
           extend Forwardable
 
-          def_delegators :config, :client
+          def_delegators :config, :client, :service_name
 
           # `Datadog::Tracing` client that we should use to trace stuff
           setting :client
 
+          # @see https://docs.datadoghq.com/tracing/trace_collection/dd_libraries/ruby
+          setting :service_name, default: nil
+
           configure
 
           # Log levels that we use in this particular listener
@@ -44,7 +47,7 @@ module Karafka
           #
           # @param event [Karafka::Core::Monitoring::Event] event details including payload
           def on_worker_process(event)
-            current_span = client.trace('karafka.consumer')
+            current_span = client.trace('karafka.consumer', service: service_name)
             push_tags
 
             job = event[:job]

data/lib/karafka/messages/builders/message.rb

@@ -23,11 +23,15 @@ module Karafka
               received_at: received_at
             ).freeze
 
+            # Get the raw payload
+            payload = kafka_message.payload
+
+            # And nullify it in the kafka message. This can save a lot of memory when used with
+            # the Pro Cleaner API
+            kafka_message.instance_variable_set('@payload', nil)
+
             # Karafka messages cannot be frozen because of the lazy deserialization feature
-            Karafka::Messages::Message.new(
-              kafka_message.payload,
-              metadata
-            )
+            Karafka::Messages::Message.new(payload, metadata)
           end
         end
       end

data/lib/karafka/pro/active_job/consumer.rb

@@ -25,7 +25,7 @@ module Karafka
       class Consumer < ::Karafka::ActiveJob::Consumer
         # Runs ActiveJob jobs processing and handles lrj if needed
         def consume
-          messages.each do |message|
+          messages.each(clean: true) do |message|
             # If for any reason we've lost this partition, not worth iterating over new messages
             # as they are no longer ours
             break if revoked?

data/lib/karafka/pro/cleaner/errors.rb

@@ -0,0 +1,27 @@
+# frozen_string_literal: true
+
+# This Karafka component is a Pro component under a commercial license.
+# This Karafka component is NOT licensed under LGPL.
+#
+# All of the commercial components are present in the lib/karafka/pro directory of this
+# repository and their usage requires commercial license agreement.
+#
+# Karafka has also commercial-friendly license, commercial support and commercial components.
+#
+# By sending a pull request to the pro components, you are agreeing to transfer the copyright of
+# your code to Maciej Mensfeld.
+
+module Karafka
+  module Pro
+    module Cleaner
+      # Cleaner related errors
+      module Errors
+        # Base for all the clearer errors
+        BaseError = Class.new(::Karafka::Errors::BaseError)
+
+        # Raised when trying to deserialize a message that has already been cleaned
+        MessageCleanedError = Class.new(BaseError)
+      end
+    end
+  end
+end

data/lib/karafka/pro/cleaner/messages/message.rb

@@ -0,0 +1,46 @@
+# frozen_string_literal: true
+
+# This Karafka component is a Pro component under a commercial license.
+# This Karafka component is NOT licensed under LGPL.
+#
+# All of the commercial components are present in the lib/karafka/pro directory of this
+# repository and their usage requires commercial license agreement.
+#
+# Karafka has also commercial-friendly license, commercial support and commercial components.
+#
+# By sending a pull request to the pro components, you are agreeing to transfer the copyright of
+# your code to Maciej Mensfeld.
+
+module Karafka
+  module Pro
+    module Cleaner
+      # Cleaner messages components related enhancements
+      module Messages
+        # Extensions to the message that allow for granular memory control on a per message basis
+        module Message
+          # @return [Object] lazy-deserialized data (deserialized upon first request)
+          def payload
+            # If message has already been cleaned, it cannot be deserialized again
+            cleaned? ? raise(Errors::MessageCleanedError) : super
+          end
+
+          # @return [Boolean] true if the message has been cleaned
+          def cleaned?
+            @raw_payload == false
+          end
+
+          # Cleans the message payload and removes the deserialized data references
+          # This is useful when working with big messages that take a lot of space.
+          #
+          # After the message content is no longer needed, it can be removed so it does not consume
+          # space anymore.
+          def clean!
+            @deserialized = false
+            @raw_payload = false
+            @payload = nil
+          end
+        end
+      end
+    end
+  end
+end

data/lib/karafka/pro/cleaner/messages/messages.rb

@@ -0,0 +1,42 @@
+# frozen_string_literal: true
+
+# This Karafka component is a Pro component under a commercial license.
+# This Karafka component is NOT licensed under LGPL.
+#
+# All of the commercial components are present in the lib/karafka/pro directory of this
+# repository and their usage requires commercial license agreement.
+#
+# Karafka has also commercial-friendly license, commercial support and commercial components.
+#
+# By sending a pull request to the pro components, you are agreeing to transfer the copyright of
+# your code to Maciej Mensfeld.
+
+module Karafka
+  module Pro
+    module Cleaner
+      module Messages
+        # Extensions to the messages batch allowing for automatic cleaning of each message after
+        # message is processed.
+        module Messages
+          # @param clean [Boolean] do we want to clean each message after we're done working with
+          #   it.
+          # @yield block we want to execute per each message
+          #
+          # @note Cleaning messages after we're done with each of them and did not fail does not
+          #   affect any other functionalities. The only thing that is crucial is to make sure,
+          #   that if DLQ is used, that we mark each message as consumed when using this API as
+          #   otherwise a cleaned message may be dispatched and that should never happen
+          def each(clean: false)
+            @messages_array.each do |message|
+              yield(message)
+
+              next unless clean
+
+              message.clean!
+            end
+          end
+        end
+      end
+    end
+  end
+end

data/lib/karafka/pro/cleaner.rb

@@ -0,0 +1,41 @@
+# frozen_string_literal: true
+
+# This Karafka component is a Pro component under a commercial license.
+# This Karafka component is NOT licensed under LGPL.
+#
+# All of the commercial components are present in the lib/karafka/pro directory of this
+# repository and their usage requires commercial license agreement.
+#
+# Karafka has also commercial-friendly license, commercial support and commercial components.
+#
+# By sending a pull request to the pro components, you are agreeing to transfer the copyright of
+# your code to Maciej Mensfeld.
+
+module Karafka
+  module Pro
+    # Feature that introduces a granular memory management for each message and messages iterator
+    #
+    # It allows for better resource allocation by providing an API to clear payload and raw payload
+    # from a message after those are no longer needed but before whole messages are freed and
+    # removed by Ruby GC.
+    #
+    # This can be useful when processing bigger batches or bigger messages one after another and
+    # wanting not to have all of the data loaded into memory.
+    #
+    # Can yield significant memory savings (up to 80%).
+    module Cleaner
+      class << self
+        # @param _config [Karafka::Core::Configurable::Node] root node config
+        def pre_setup(_config)
+          ::Karafka::Messages::Message.prepend(Messages::Message)
+          ::Karafka::Messages::Messages.prepend(Messages::Messages)
+        end
+
+        # @param _config [Karafka::Core::Configurable::Node] root node config
+        def post_setup(_config)
+          true
+        end
+      end
+    end
+  end
+end

data/lib/karafka/pro/contracts/base.rb

@@ -0,0 +1,23 @@
+# frozen_string_literal: true
+
+# This Karafka component is a Pro component under a commercial license.
+# This Karafka component is NOT licensed under LGPL.
+#
+# All of the commercial components are present in the lib/karafka/pro directory of this
+# repository and their usage requires commercial license agreement.
+#
+# Karafka has also commercial-friendly license, commercial support and commercial components.
+#
+# By sending a pull request to the pro components, you are agreeing to transfer the copyright of
+# your code to Maciej Mensfeld.
+
+module Karafka
+  module Pro
+    # Pro contracts that aim to replace or complement the general framework contracts
+    module Contracts
+      # Base for all the Pro contracts
+      class Base < ::Karafka::Contracts::Base
+      end
+    end
+  end
+end