karafka 2.5.6 → 2.5.9

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 11edff86c8615652130786431e179d242dcc33130e5aabbbf5e0f5ed6d4138fe
-  data.tar.gz: 69244021709283a153da19907a424c230a3597832beec6acbb39dbc02e738256
+  metadata.gz: 0a43645cb0713b1565aa8ff9b3779d5d19225f08d2449e816ab17aaa532a6ffd
+  data.tar.gz: 438e1dcb6d6b65fd29e2d4f2feb6cee80cf8c629cba273b3e2936637379cda75
 SHA512:
-  metadata.gz: 843548637c77ace03cde5c1f4150226c244b1b3343c6acd17d16c19cdc798c6d770611522eed7ebef76830f3e551c75afa176f2937c1fd63045e66ccb6276701
-  data.tar.gz: a0696c28b998c8e13d4b0e0d3f4252245f27e6f93e0d75d2153a93cc1dcc53b45fbe8416ba0a9e6025753ec5d107088d578f03d0a4cada3bb5ed2eb4e5341518
+  metadata.gz: 44a2d60d274f70321846a4190e007f4d5c75f22e995835fe92cf29a5a658ed7eb99bd96147d6cfa08e9821676ca7064fcd095ecf17b6ac4014c018d885621e27
+  data.tar.gz: 4eb201aeb1e1d416008913c58b7f39637dce2342506f0032d8556b285025cf23f1475f8cf08f2d91a4ee00a9241902e26fc8ca7297547a41cd0b1126d09029ac

data/CHANGELOG.md

@@ -1,5 +1,27 @@
 # Karafka Framework Changelog
 
+## 2.5.9 (2026-03-30)
+- [Enhancement] Validate that `statistics.interval.ms` is not zero when dynamic multiplexing is enabled (Pro).
+- [Fix] Fix swarm liveness reporting to also use `on_connection_listener_fetch_loop` so nodes stay alive when `statistics.interval.ms` is disabled.
+- [Fix] Fix `ConfigProxy#producer` when called as a reader resetting the producer to nil instead of returning its value (#3076).
+- [Fix] Fix `Replication#build_generate_command` using `--reassignment-json-file` with `--generate` instead of `--topics-to-move-json-file` and `--broker-list` ([#3087](https://github.com/karafka/karafka/issues/3087)).
+
+## 2.5.8 (2026-03-23)
+- **[Feature]** Add `Karafka::Admin::Recovery` for coordinator-bypass offset reading and consumer group migration when the Kafka group coordinator is in a FAILED state (Pro).
+
+## 2.5.7 (2026-03-16)
+- [Enhancement] Report detailed blocking information (active listeners, alive workers, and in-processing jobs) during forceful shutdown instead of only aggregate counts.
+- [Enhancement] Improve `ForcefulShutdownError` description to clearly explain when and why it is raised.
+- [Enhancement] Cache `messages.last` in `BatchMetadata` builder to avoid duplicate array traversal.
+- [Enhancement] Optimize `VirtualOffsetManager#mark` to use a single array scan instead of separate `include?` and `index` calls (Pro).
+- [Enhancement] Optimize `VirtualOffsetManager#materialize_real_offset` to use `keys.sort` instead of `to_a.sort_by` with tuple destructuring (Pro).
+- [Enhancement] Optimize `IntervalRunner#call` to use a single `monotonic_now` call instead of two per invocation.
+- [Enhancement] Support WaterDrop `:fd` mode in Swarm.
+- [Maintenance] Use both `:fd` and `:thread` producer backends in CI.
+- [Maintenance] Include spec file hash in integration test topic names for easier traceability in Kafka logs (#3056).
+- [Fix] Remove duplicate topic creation in multi-broker health integration specs (#3056).
+- [Fix] Preserve producer-specific kafka settings (e.g., `enable.idempotence`) when recreating the producer in swarm forks.
+
 ## 2.5.6 (2026-02-28)
 - **[Feature]** Add `karafka topics health` command to check Kafka topics for replication and durability issues, detecting no redundancy (RF=1), zero fault tolerance (RF≤min.insync), and low durability (min.insync=1) configurations with color-coded severity grouping and actionable recommendations (Pro).
 - [Enhancement] Optimize license loading process by reading license files directly from the gem directory instead of requiring the entire gem, reducing initialization overhead and adding support for user-defined License modules.

data/LICENSE-COMM

@@ -10,7 +10,7 @@ In order to use the Software under this Agreement, you must either: (a) receive
 
 1. License Grant
 
-1.1 General Use. This Agreement grants you a non-exclusive, non-transferable, limited license to the use rights for the Software, without the right to grant sublicenses, subject to the terms and conditions in this Agreement. The Software is licensed, not sold.
+1.1 General Use. This Agreement grants you a non-exclusive, non-transferable, limited license to the use rights for the Software, without the right to grant sublicenses, subject to the terms and conditions in this Agreement. The Software is licensed, not sold. No patent rights are granted under this Agreement. This license grant covers only the copyright rights necessary to use the Software as expressly permitted herein.
 
 1.1.1 Entity Definition. Where you have purchased or otherwise acquired the Software as or for an entity, "you" and "your" refer exclusively to the single legal entity (such as a corporation, limited liability company, partnership, or other legally recognized organization) that is the named licensee. This license does not extend to any parent company, subsidiary, affiliate, related entity, or other member of a corporate group, regardless of ownership structure or control relationships. Each separate legal entity requires its own license.
 
@@ -30,7 +30,9 @@ In order to use the Software under this Agreement, you must either: (a) receive
 
 3.2 UNDER NO CIRCUMSTANCES MAY YOU USE THE SOFTWARE AS PART OF A PRODUCT OR SERVICE THAT PROVIDES SIMILAR FUNCTIONALITY TO THE SOFTWARE ITSELF.
 
-3.3 ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING PROHIBITION. WITHOUT LIMITING THE GENERALITY OF SECTION 3.1, THE SOFTWARE AND ITS SOURCE CODE MAY NOT BE USED IN ANY MANNER TO TRAIN, DEVELOP, IMPROVE, OR VALIDATE ANY ARTIFICIAL INTELLIGENCE OR MACHINE LEARNING SYSTEM. THIS PROHIBITION APPLIES REGARDLESS OF WHETHER THE SOFTWARE IS ACCESSED DIRECTLY, THROUGH AUTOMATED MEANS, OR VIA THIRD-PARTY DATASETS OR SERVICES. ANY SUCH USE CONSTITUTES A MATERIAL BREACH OF THIS AGREEMENT AND MAY RESULT IN IMMEDIATE TERMINATION OF YOUR LICENSE AND PURSUIT OF ALL AVAILABLE LEGAL REMEDIES. For clarity, this prohibition applies to the training and development of AI systems, and does not restrict licensed users from utilizing AI-powered development tools in the ordinary course of developing software that integrates with the Software.
+3.3 ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING PROHIBITION. WITHOUT LIMITING THE GENERALITY OF SECTION 3.1, THE SOFTWARE AND ITS SOURCE CODE MAY NOT BE USED IN ANY MANNER TO TRAIN, DEVELOP, IMPROVE, OR VALIDATE ANY ARTIFICIAL INTELLIGENCE OR MACHINE LEARNING SYSTEM. THIS PROHIBITION APPLIES REGARDLESS OF WHETHER THE SOFTWARE IS ACCESSED DIRECTLY, THROUGH AUTOMATED MEANS, OR VIA THIRD-PARTY DATASETS OR SERVICES. ANY SUCH USE CONSTITUTES A MATERIAL BREACH OF THIS AGREEMENT AND MAY RESULT IN IMMEDIATE TERMINATION OF YOUR LICENSE AND PURSUIT OF ALL AVAILABLE LEGAL REMEDIES. For clarity, this prohibition applies to the training and development of AI systems, and does not restrict licensed users from utilizing AI-powered development tools in the ordinary course of developing software that integrates with the Software. Additionally, runtime retrieval of the Software's source code by AI assistants, coding agents, and retrieval-augmented generation (RAG) systems is permitted solely for the purpose of providing contextual help to Karafka users, provided that such retrieval does not result in the storage, reproduction, or use of the source code for training, fine-tuning, or improving any AI or machine learning system.
+
+3.4 Source-Available Access. The Software's source code is made available for reading, referencing, and linking for personal or commercial use. This permits viewing the source code for informational purposes, referencing it in discussions or documentation, and linking to the official repository. Such access does not constitute a license to use, copy, reproduce, modify, or distribute the Software or any portion thereof. All restrictions in Sections 3.1, 3.2, and 3.3 remain fully applicable regardless of how the source code was accessed. Receipt, viewing, or possession of the Software's source code does not convey or imply any license or right beyond those expressly granted in this Agreement.
 
 The Open Source version of the Software ("LGPL Version") is licensed under the terms of the GNU Lesser General Public License version 3.0 ("LGPL") and not under this EULA.
 

data/config/locales/pro_errors.yml

@@ -2,6 +2,7 @@ en:
   validations:
     routing:
       swarm_nodes_not_used: 'At least one of the nodes has no assignments'
+      multiplexing_statistics_required_for_dynamic_multiplexing: 'Dynamic multiplexing requires statistics to be enabled (statistics.interval.ms must be greater than 0)'
 
       topic:
         virtual_partitions.partitioner_respond_to_call: needs to be defined and needs to respond to `#call`

data/lib/karafka/admin/replication.rb

@@ -55,6 +55,7 @@ module Karafka
         :target_replication_factor,
         :partitions_assignment,
         :reassignment_json,
+        :topics_to_move_json,
         :execution_commands,
         :steps
       )
@@ -108,6 +109,7 @@ module Karafka
         @cluster_info = cluster_info
 
         generate_reassignment_json
+        generate_topics_to_move_json
         generate_execution_commands
         generate_steps
 
@@ -256,6 +258,15 @@ module Karafka
         file_path
       end
 
+      # Export the topics-to-move JSON to a file for use with kafka-reassign-partitions.sh
+      # --generate. This optional step lets you ask Kafka to propose its own reassignment plan
+      # so you can compare it against the plan Karafka generated before executing.
+      # @param file_path [String] path where to save the JSON file
+      def export_topics_to_move_file(file_path)
+        File.write(file_path, @topics_to_move_json)
+        file_path
+      end
+
       # @return [String] human-readable summary of the plan
       def summary
         broker_count = @cluster_info[:brokers].size
@@ -428,6 +439,19 @@ module Karafka
         @reassignment_json = JSON.pretty_generate(reassignment_data)
       end
 
+      # Generates the topics-to-move JSON required by kafka-reassign-partitions.sh --generate.
+      # This file is only needed when you want Kafka to propose its own reassignment plan for
+      # comparison against the plan Karafka already computed. It is not required for execution.
+      # @return [void]
+      def generate_topics_to_move_json
+        topics_data = {
+          version: 1,
+          topics: [{ topic: @topic }]
+        }
+
+        @topics_to_move_json = JSON.pretty_generate(topics_data)
+      end
+
       # Generates command templates for executing the reassignment plan
       # Builds generate, execute, and verify command templates with placeholders
       # @return [void]
@@ -439,11 +463,16 @@ module Karafka
         }
       end
 
-      # Builds the kafka-reassign-partitions.sh command for generating reassignment plan
-      # @return [String] command template with placeholder for broker addresses
+      # Builds the kafka-reassign-partitions.sh --generate command that asks Kafka to propose
+      # its own reassignment plan. This is optional — you can compare Kafka's suggestion against
+      # the plan Karafka already computed, or skip this step and go straight to --execute.
+      # @return [String] command template with placeholders for broker addresses and IDs
       def build_generate_command
+        broker_ids = @cluster_info[:brokers].map { |b| b[:node_id] }.sort.join(",")
+
         "kafka-reassign-partitions.sh --bootstrap-server <KAFKA_BROKERS> " \
-          "--reassignment-json-file reassignment.json --generate"
+          "--topics-to-move-json-file topics-to-move.json " \
+          "--broker-list #{broker_ids} --generate"
       end
 
       # Builds the kafka-reassign-partitions.sh command for executing reassignment
@@ -466,7 +495,10 @@ module Karafka
       def generate_steps
         @steps = [
           "1. Export the reassignment JSON using: plan.export_to_file('reassignment.json')",
-          "2. Validate the plan (optional): #{@execution_commands[:generate]}",
+          "2. (Optional) Compare against Kafka's own proposal:",
+          "   a. Export topics-to-move JSON: plan.export_topics_to_move_file('topics-to-move.json')",
+          "   b. Ask Kafka to generate its plan: #{@execution_commands[:generate]}",
+          "   c. Compare Kafka's output with the Karafka-generated reassignment.json above",
           "3. Execute the reassignment: #{@execution_commands[:execute]}",
           "4. Monitor progress: #{@execution_commands[:verify]}",
           "5. Verify completion by checking topic metadata",

data/lib/karafka/errors.rb

@@ -51,7 +51,9 @@ module Karafka
       end
     end
 
-    # Raised when we've waited enough for shutting down a non-responsive process
+    # Raised when the graceful shutdown timeout has been exceeded and Karafka must forcefully
+    # terminate remaining listeners and workers. This typically happens when consumer processing
+    # or shutdown jobs take longer than the configured `shutdown_timeout`.
     ForcefulShutdownError = Class.new(BaseError)
 
     # Raised when the jobs queue receives a job that should not be received as it would cause

data/lib/karafka/helpers/interval_runner.rb

@@ -26,9 +26,11 @@ module Karafka
 
       # Runs the requested code if it was not executed previously recently
       def call
-        return if monotonic_now - @last_called_at < @interval
+        now = monotonic_now
 
-        @last_called_at = monotonic_now
+        return if now - @last_called_at < @interval
+
+        @last_called_at = now
 
         @block.call
       end

data/lib/karafka/instrumentation/logger_listener.rb

@@ -385,21 +385,34 @@ module Karafka
           fatal "Runner crashed due to an error: #{details}"
           fatal backtrace
         when "app.stopping.error"
-          # Counts number of workers and listeners that were still active when forcing the
-          # shutdown. Please note, that unless all listeners are closed, workers will not finalize
-          # their operations as well.
-          # We need to check if listeners and workers are assigned as during super early stages of
-          # boot they are not.
-          listeners = Server.listeners ? Server.listeners.count(&:active?) : 0
-          workers = Server.workers ? Server.workers.count(&:alive?) : 0
+          active_listeners = event.payload[:active_listeners]
+          alive_workers = event.payload[:alive_workers]
+          in_processing = event.payload[:in_processing]
 
           message = <<~MSG.tr("\n", " ").strip!
             Forceful Karafka server stop with:
-            #{workers} active workers and
-            #{listeners} active listeners
+            #{alive_workers.size} active workers and
+            #{active_listeners.size} active listeners
           MSG
 
           error message
+
+          active_listeners.each do |listener|
+            error "Listener #{listener.id} for #{listener.subscription_group.name} still active"
+          end
+
+          in_processing.each do |group_id, jobs|
+            next if jobs.empty?
+
+            jobs.each do |job|
+              job_class = job.class.name.split("::").last
+              topic_name = job.executor.topic.name
+              partition = job.executor.partition
+
+              error "In processing: #{job_class} job for #{topic_name}/#{partition} " \
+                    "(group: #{group_id})"
+            end
+          end
         when "app.forceful_stopping.error"
           error "Forceful shutdown error occurred: #{details}"
           error backtrace

data/lib/karafka/instrumentation/vendors/datadog/logger_listener.rb

@@ -117,7 +117,12 @@ module Karafka
             when "runner.call.error"
               fatal "Runner crashed due to an error: #{error}"
             when "app.stopping.error"
-              error "Forceful Karafka server stop"
+              active_listeners = event.payload[:active_listeners]
+              alive_workers = event.payload[:alive_workers]
+
+              error "Forceful Karafka server stop with: " \
+                    "#{alive_workers.size} active workers and " \
+                    "#{active_listeners.size} active listeners"
             when "app.forceful_stopping.error"
               error "Forceful shutdown error occurred: #{error}"
             when "librdkafka.error"

data/lib/karafka/messages/builders/batch_metadata.rb

@@ -17,16 +17,18 @@ module Karafka
           # @note We do not set `processed_at` as this needs to be assigned when the batch is
           #   picked up for processing.
           def call(messages, topic, partition, scheduled_at)
+            last_message = messages.last
+
             Karafka::Messages::BatchMetadata.new(
               size: messages.size,
               first_offset: messages.first&.offset || -1001,
-              last_offset: messages.last&.offset || -1001,
+              last_offset: last_message&.offset || -1001,
               deserializers: topic.deserializers,
               partition: partition,
               topic: topic.name,
               # We go with the assumption that the creation of the whole batch is the last message
               # creation time
-              created_at: local_created_at(messages.last),
+              created_at: local_created_at(last_message),
               # When this batch was built and scheduled for execution
               scheduled_at: scheduled_at,
               # This needs to be set to a correct value prior to processing starting

data/lib/karafka/pro/active_job/consumer.rb

@@ -6,9 +6,14 @@
 # This software is NOT open source. It is source-available commercial software
 # requiring a paid license for use. It is NOT covered by LGPL.
 #
+# The author retains all right, title, and interest in this software,
+# including all copyrights, patents, and other intellectual property rights.
+# No patent rights are granted under this license.
+#
 # PROHIBITED:
 # - Use without a valid commercial license
 # - Redistribution, modification, or derivative works without authorization
+# - Reverse engineering, decompilation, or disassembly of this software
 # - Use as training data for AI/ML models or inclusion in datasets
 # - Scraping, crawling, or automated collection for any purpose
 #
@@ -17,6 +22,9 @@
 # - Runtime retrieval by AI assistants, coding agents, and RAG systems
 #   for the purpose of providing contextual help to Karafka users
 #
+# Receipt, viewing, or possession of this software does not convey or
+# imply any license or right beyond those expressly stated above.
+#
 # License: https://karafka.io/docs/Pro-License-Comm/
 # Contact: contact@karafka.io
 

data/lib/karafka/pro/active_job/dispatcher.rb

@@ -6,9 +6,14 @@
 # This software is NOT open source. It is source-available commercial software
 # requiring a paid license for use. It is NOT covered by LGPL.
 #
+# The author retains all right, title, and interest in this software,
+# including all copyrights, patents, and other intellectual property rights.
+# No patent rights are granted under this license.
+#
 # PROHIBITED:
 # - Use without a valid commercial license
 # - Redistribution, modification, or derivative works without authorization
+# - Reverse engineering, decompilation, or disassembly of this software
 # - Use as training data for AI/ML models or inclusion in datasets
 # - Scraping, crawling, or automated collection for any purpose
 #
@@ -17,6 +22,9 @@
 # - Runtime retrieval by AI assistants, coding agents, and RAG systems
 #   for the purpose of providing contextual help to Karafka users
 #
+# Receipt, viewing, or possession of this software does not convey or
+# imply any license or right beyond those expressly stated above.
+#
 # License: https://karafka.io/docs/Pro-License-Comm/
 # Contact: contact@karafka.io
 

data/lib/karafka/pro/active_job/job_options_contract.rb

@@ -6,9 +6,14 @@
 # This software is NOT open source. It is source-available commercial software
 # requiring a paid license for use. It is NOT covered by LGPL.
 #
+# The author retains all right, title, and interest in this software,
+# including all copyrights, patents, and other intellectual property rights.
+# No patent rights are granted under this license.
+#
 # PROHIBITED:
 # - Use without a valid commercial license
 # - Redistribution, modification, or derivative works without authorization
+# - Reverse engineering, decompilation, or disassembly of this software
 # - Use as training data for AI/ML models or inclusion in datasets
 # - Scraping, crawling, or automated collection for any purpose
 #
@@ -17,6 +22,9 @@
 # - Runtime retrieval by AI assistants, coding agents, and RAG systems
 #   for the purpose of providing contextual help to Karafka users
 #
+# Receipt, viewing, or possession of this software does not convey or
+# imply any license or right beyond those expressly stated above.
+#
 # License: https://karafka.io/docs/Pro-License-Comm/
 # Contact: contact@karafka.io
 

data/lib/karafka/pro/admin/recovery/errors.rb

@@ -0,0 +1,51 @@
+# frozen_string_literal: true
+
+# Karafka Pro - Source Available Commercial Software
+# Copyright (c) 2017-present Maciej Mensfeld. All rights reserved.
+#
+# This software is NOT open source. It is source-available commercial software
+# requiring a paid license for use. It is NOT covered by LGPL.
+#
+# The author retains all right, title, and interest in this software,
+# including all copyrights, patents, and other intellectual property rights.
+# No patent rights are granted under this license.
+#
+# PROHIBITED:
+# - Use without a valid commercial license
+# - Redistribution, modification, or derivative works without authorization
+# - Reverse engineering, decompilation, or disassembly of this software
+# - Use as training data for AI/ML models or inclusion in datasets
+# - Scraping, crawling, or automated collection for any purpose
+#
+# PERMITTED:
+# - Reading, referencing, and linking for personal or commercial use
+# - Runtime retrieval by AI assistants, coding agents, and RAG systems
+#   for the purpose of providing contextual help to Karafka users
+#
+# Receipt, viewing, or possession of this software does not convey or
+# imply any license or right beyond those expressly stated above.
+#
+# License: https://karafka.io/docs/Pro-License-Comm/
+# Contact: contact@karafka.io
+
+module Karafka
+  module Pro
+    # Pro Admin utilities
+    module Admin
+      class Recovery < Karafka::Admin
+        # Recovery related errors
+        module Errors
+          # Base for all the recovery errors
+          BaseError = Class.new(::Karafka::Errors::BaseError)
+
+          # Raised when required cluster metadata cannot be retrieved (topic, partition, or
+          # broker not found)
+          MetadataError = Class.new(BaseError)
+
+          # Raised when a partition number is outside the valid range for __consumer_offsets
+          PartitionOutOfRangeError = Class.new(BaseError)
+        end
+      end
+    end
+  end
+end